The node

The V-Class server can contain four to 16 processors. The processors and the associated hardware comprise what is commonly called node. The terms node and system are used interchangeably in this book. The node uses a symmetric multiprocessor (SMP) design that can exploit fine-grain parallelism.

A conceptual block diagram of the system is shown in Figure 1-1 “Functional block diagram of a V-Class system”. Centrally located in the diagram is the HP Hyperplane crossbar that is comprised of four Exemplar Routing Attachment controllers (ERAC). The Hyperplane crossbar allows all of the processors to access all available memory. Processors are installed on Exemplar Processor Agent controllers (EPACs). An EPAC allows the processor and the I/O subsystem (the Exemplar PCI-bus Interface controller—EPIC) access to the Hyperplane crossbar. Also connected to the Hyperplane crossbar are the Exemplar Memory Access controllers (EMAC). Up to two processors are located on each EPAC. Memory is controlled by the EMAC. Input and output devices connect to the system through EPIC which is connected to the processor agents.

The Exemplar Core Utilities board (ECUB—commonly called the Utilities board) in the node contains a section of hardware called the core logic. It provides interrupts to all of the processors in the system through the core logic bus which connects to each processor agent. The ECUB attaches to the Exemplar system Routing board (ENRB) centrally located in the node.

Figure 1-1 Functional block diagram of a V-Class system

Control and status registers (CSRs)

System hardware is manipulated by control and status registers located in the processors and controllers.

CSRs provide control, status, or both to the processors and other hardware in the system. Each CSR is memory mapped and is available to all processors in the system. Many of the registers are described in detail by functional groups, such as system configuration, messaging and data copy, I/O, and so on. These descriptions appear throughout this book.

Description of functional blocks

Each block in Figure 1-1 “Functional block diagram of a V-Class system” is described in the following sections.

Exemplar processor agent controller

The EPAC can connect to zero, one, or two PA-8200 processors. It can also connect to zero or one EPIC (the I/O controller). With no processors, the EPAC serves as an I/O-only interface. The EPAC has the following buses:

Runway bus (0, 1)—Two each, 64-bit, bidirectional buses for processor 0 and processor 1, respectively. These buses have a raw bandwidth of 960 MBytes per second.
Hyperplane crossbar port bus (0, 1)—Four 32-bit, unidirectional buses connected to two Hyperplane crossbar ERACs, two in each direction. These buses have a total raw bandwidth of 1.9 GBytes per second.
I/O port—Two 16-bit or 32-bit, unidirectional interfaces to an I/O device, one for reading data and one for writing data. The width of the bus depends on the width of the I/O device connected. Each bus has a bandwidth of 120 MBytes per second or 240 MBytes per second, depending on the width of the interface.
Core Logic Bus interface—A single bidirectional bus that supports boot and support services.

The EPAC sends and receives transactions from the ERACs using four unidirectional data paths. There are four ERACs in the Hyperplane crossbar. Each processor agent, however, communicates with only two of the four ERACs.

The EPAC includes special hardware called the data mover for rapid message and data movement between memory within a node. This dedicated hardware greatly improves file I/O and networking over software versions.

Exemplar routing attachment controller—Hyperplane crossbar

The Hyperplane crossbar is comprised of four ERACs that provide an interconnect for each processor and I/O device to memory.

Each of the four ERACs has the following buses:

EPAC Port (A, B, C, D)—Eight 32-bit, unidirectional interfaces to four EPAC ports, four in each direction. Each port has simultaneous (input and output) bandwidth of 960 MBytes per second.
EMAC Port (A, B, C, D)—Eight 32-bit, unidirectional interfaces to four EMAC ports, four in each direction. Each port has simultaneous (input and output) bandwidth of 960 MBytes per second.

Figure 1-2 “ERAC interconnection” shows how the ERACs connect to each EPAC and EMAC.

Figure 1-2 ERAC interconnection

Exemplar memory access controller

The EMAC controls all accesses to memory. Each EMAC controls four banks of memory, allowing up to 32 banks in an eight-EMAC system. Memory banks consist of Single Inline Memory Modules (SIMMs) of Synchronous Dynamic Random Access Memory (SDRAM).

The EMAC has the following buses:

ERAC Port (A, B)—Four 32-bit, unidirectional interfaces, two in each direction. This interface supports a total simultaneous read-write bandwidth of 1.9 GBytes per second.
Even Memory—A single 88-bit, bidirectional interface to the even memory banks associated with the EMAC.
Odd Memory—A single 88-bit, bidirectional interface to the odd memory banks associated with the EMAC.

A processor accesses memory by sending a request, in the form of packets, to an ERAC. The request is then forwarded to one of the EMACs. The EMAC routes requests into even and odd pending queues. Some packets not destined for memory are routed from processor to processor through the EMAC. These packets are routed directly to the output ports.

The EMAC accesses one of four available memory banks, checking the Error Correction Code (ECC). The data accessed from memory is returned to the processor by sending a response back to the ERAC, which forwards the response to the EPAC.

ECUB and core logic bus

The ECUB, or Utilities board, connects to the core logic bus and contains two field-programmable gate arrays (FPGAs): the Exemplar Processor Utilities controller (EPUC) and the Exemplar Monitoring Utilities controller (EMUC). The EPUC allows processors access to the system core logic and booting firmware, and the EMUC processes the environmental state of the system and interrupts the processors when appropriate. V-Class servers use the core logic bus primarily to boot the system and to issue environmental interrupts.

The core logic bus is a low-bandwidth, multidrop bus that connects each processor to the control and interface logic (both RS232 and ethernet). A processor can write to control and status registers (CSRs) accessed using the core logic bus to initialize and configure the ERAC chips and Utilities board logic.