IETF - Delayed email processing – Incident details

All systems operational

Delayed email processing

Resolved
Operational
Started 15 days agoLasted about 16 hours

Affected

Mail Services

Operational from 11:01 PM to 3:16 PM

Updates
  • Resolved
    Resolved

    The mail processing system encountered reduced disk i/o capacity due to an incident in the underlying filesystem that took some time to recover. During this period of reduced i/o capacity, email submission to lists was very high - driven by the Internet-Draft submission deadline at 2359 UTC Monday. This led to very long queue time for messages going to mailing lists. The system drained the queue, delivering all messages, by 0700 UTC Tuesday and is now keeping up. We are looking into ways to improve the baseline I/O capacity of the system to provide better resiliency for the future.

  • Investigating
    Investigating

    Messages going through the IETF email processing system, particularly to lists, is encountering long delays, with some delays greater than 4 hours. We are investigating the cause and immediate workarounds.

    Messages to lists are typically entering the archives almost immediately - it's the delivery to the list subscribers that is delayed.