Distributed Interrupts and I/O Forwarding

Features:

In pre-10.0 systems, all device interrupts were forwarded to the monarch processor. This could lead to "interrupt saturation" on the monarch on some workloads. That is, the monarch processor could end up processing interrupts most of the time while the other processors remained relatively idle.

In 10.0 systems, the interrupts for some of the most heavily used disk and networking drivers (HPFL, scsi3, GSC SCSI, lan3 and lan6) are "distributed".

The interrupts corresponding to each instance of one of these drivers are assigned to the next available processor on a round robin basis. Normally this assignment is done automatically, either during I/O system configuration while booting up or on the first I/O operation. The drivers call an I/O services function to accomplish this.

Each instance of one of these drivers corresponds to an IO card. So, the interrupt distribution is on a per card basis and not on a per device basis.

With the introduction of distributed interrupts, an additional performance gain is possible by forwarding an I/O request to the same processor slated to handle the associated device interrupt. This takes advantage of the cache locality of I/O data structures, potentially reducing third-party cache hits and improving MP scaling.

The I/O forwarding feature is currently only implemented for NIO disk devices (HPFL, scsi3, and scsi1). The drivers call an I/O services function defined in the 800 context dependent I/O environment to forward the request to the appropriate processor.

Impact/Performance

Distributed interrupts may have a beneficial performance impact on some I/O intensive applications. This is especially true as the number of CPUs in the system is increased.

I/O forwarding may provide a performance gain on MP systems that use NIO HPFL and/or SCSI interfaces and disk devices.

For drivers that have not been modified to distribute interrupts, the interrupts will continue to be forwarded to the monarch processor. Likewise, drivers that do not enable I/O forwarding will start up an I/O request on the processor that initially received the request.

Limitations

The interrupts for various instances of the drivers are assigned to different processors on a round robin basis. This could result in an uneven distribution at times.

The interrupt distribution is on a per card basis. As the number of devices on a card increases, there is a possibility of interrupt saturation occurring on the CPU to which the card's interrupts are assigned.