You are here

Manually load balancing IO interups between cpus, after adding/removing cpus in a vpar

Hy,

the HP-UX kernel assigns I/O interrupts to CPU’s using a
round robin algorithm. As each I/O device is discovered and bound to its driver, the drivers interrupt
service routine is registered to execute on a specific CPU. When an I/O device signals completion of
a DMA operation, the CPU that it is assigned to is interrupted and the devices I/O interrupt handler is
invoked to service the interrupt.
When a CPU is removed from a vPar, the processes and I/O interrupts that are assigned to it are
distributed amongst the remaining CPU’s. These same set of tasks are used to unload a CPU when it is
moved from the default processor set to the real-time processor set. When a CPU is migrated into a
vPar, the CPU will be assigned a portion of the processing load but will not be assigned any I/O
interrupts. Over time, I/O interrupts from other CPU’s migrating out of the system may be assigned to
this CPU. If many CPU migrations take place, the majority of the I/O interrupts may land on too few
CPU’s or worse case on the boot CPU. If the I/O load is high enough, response time may increase
and throughput may decrease due to this imbalance.

Estado inicial con 4 cpus, los interrupts estan distribuidos entre las 4 cpus:

# intctl (03-24 11:49)
hw path class drv card cpu cpu intr intr card
name cell ID cell type ID description
===========================================================================================
0/0/2/1/0/4/0 fc fcd 0 10 1 L 1 HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
0/0/2/1/0/6/0 lan iether 0 0 1 L 1 HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter
0/0/4/1/0 fc td 0 2 1 L 1 HP Tachyon XL2 Fibre Channel Mass Storage Adapter
0/0/12/1/0 lan igelan 0 10 1 L 1 HP A6825-60101 PCI 1000Base-T Adapter
0/250/0 acpi_node acpi_node 0 0 1 L 1 Acpi Hardware
1/0/0/3/0 ext_bus c8xx 1 11 1 T 2 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/0 ext_bus c8xx 1 2 1 L 1 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/1 ext_bus c8xx 1 0 1 T 2 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/0/0/3/1 ext_bus c8xx 1 0 1 L 1 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/250/0 acpi_node acpi_node 1 0 1 L 1 Acpi Hardware
1/250/1 ipmi ipmi 1 11 1 L 1 IPMI Controller

Despues de algún tiempo, se han quitado las cpus y repuesto, para dar cpus a otras maquinas, de tal forma que todo el tiempo de cpu de la io está siendo manejada por la cpu de boot:

# intctl (03-24 11:51)
hw path class drv card cpu cpu intr intr card
name cell ID cell type ID description
===========================================================================================
0/0/2/1/0/4/0 fc fcd 0 0 1 L 1 HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
0/0/2/1/0/6/0 lan iether 0 0 1 L 1 HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter
0/0/4/1/0 fc td 0 0 1 L 1 HP Tachyon XL2 Fibre Channel Mass Storage Adapter
0/0/12/1/0 lan igelan 0 0 1 L 1 HP A6825-60101 PCI 1000Base-T Adapter
0/250/0 acpi_node acpi_node 0 0 1 L 1 Acpi Hardware
1/0/0/3/0 ext_bus c8xx 1 0 1 T 2 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/0 ext_bus c8xx 1 0 1 L 1 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/1 ext_bus c8xx 1 0 1 T 2 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/0/0/3/1 ext_bus c8xx 1 0 1 L 1 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/250/0 acpi_node acpi_node 1 0 1 L 1 Acpi Hardware
1/250/1 ipmi ipmi 1 0 1 L 1 IPMI Controller

Si la maquina tiene mucho IO, esto afecta mucho la rendimiento, por lo que es mejor redistribuir manualmente los interrupts entre las cpus:

# intctl -M -H 0/0/12/1/0 -I 1 -c 10 (03-24 11:56)
intctl: Moved the interrupt: 1, of card 0/0/12/1/0, driver igelan, from CPU:0 to CPU:10
# intctl -M -H 0/0/2/1/0/4/0 -I 1 -c 11 (03-24 11:57)
intctl: Moved the interrupt: 1, of card 0/0/2/1/0/4/0, driver fcd, from CPU:0 to CPU:11
# intctl -M -H 1/0/0/3/0 -I 1 -c 11 (03-24 11:59)
intctl: Moved the interrupt: 1, of card 1/0/0/3/0, driver c8xx, from CPU:0 to CPU:11
# intctl -M -H 1/0/0/3/0 -I 2 -c 10 (03-24 12:00)
intctl: Moved the interrupt: 2, of card 1/0/0/3/0, driver c8xx, from CPU:0 to CPU:10
# intctl -M -H 1/0/0/3/1 -I 2 -c 2 (03-24 12:00)
intctl: Moved the interrupt: 2, of card 1/0/0/3/1, driver c8xx, from CPU:0 to CPU:2
# intctl -M -H 1/0/0/3/1 -I 1 -c 2 (03-24 12:13)
intctl: Moved the interrupt: 1, of card 1/0/0/3/1, driver c8xx, from CPU:0 to CPU:2

dejando balanceado los interrupts de io entre la cpu, mejorando el rendimiento en caso de tener mucho IO:

hw path class drv card cpu cpu intr intr card
name cell ID cell type ID description
===========================================================================================
0/0/2/1/0/4/0 fc fcd 0 11 1 L 1 HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
0/0/2/1/0/6/0 lan iether 0 2 0 L 1 HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter
0/0/4/1/0 fc td 0 11 1 L 1 HP Tachyon XL2 Fibre Channel Mass Storage Adapter
0/0/12/1/0 lan igelan 0 10 1 L 1 HP A6825-60101 PCI 1000Base-T Adapter
0/250/0 acpi_node acpi_node 0 0 1 L 1 Acpi Hardware
1/0/0/3/0 ext_bus c8xx 1 10 1 T 2 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/0 ext_bus c8xx 1 11 1 L 1 SCSI C1010 Ultra160 Wide LVD A6793-60001
1/0/0/3/1 ext_bus c8xx 1 2 0 T 2 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/0/0/3/1 ext_bus c8xx 1 2 0 L 1 SCSI C1010 Ultra Wide Single-Ended A6793-60001
1/250/0 acpi_node acpi_node 1 0 1 L 1 Acpi Hardware
1/250/1 ipmi ipmi 1 0 1 L 1 IPMI Controller

Es recomendable dejar cada T.fibra por 1 cpu diferente.

Unix Systems: 

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.