System Architecture

The HP XC architecture is designed as a clustered system with single system traits. From a user perspective, this architecture achieves a single system view, providing capabilities such as the following:

Single user login
Single file system namespace
Integrated view of system resources
Integrated program development environment
Integrated job submission environment

HP XC System Software

The HP XC System Software enables the nodes in the platform to run cohesively to achieve a single system view. You can determine the version of the HP XC System Software from the /etc/hptc-release file.

$ cat /etc/hptc-release
HP XC V#.# RCx PKn date

Where:

#.#

Is the version of the HP XC System Software

RCx

Is the release candidate of the version.

PKnn

Indicates there was a cumulative patch kit for the HP XC System Software installed.

For example, if PK02 appears in the output, it indicates that both cumulative patch kits PK01 and PK02 have been installed.

This field is blank if no patch kits are installed.

date

Is the date (in yyyymmdd format) the software was released.

Operating System

The HP XC system is a high-performance compute cluster that runs HP XC Linux for High Performance Computing Version 1.0 (HPC Linux) as its software base. Any serial or thread-parallel applications, or applications built shared with HP-MPI that run correctly on Red Hat Enterprise Linux Advanced Server Version 3.0, also run correctly on HPC Linux.

Node Platforms

The HP XC System Software is available on several platforms. You can determine the platform by examining the top few fields of the /proc/cpuinfo file, for example, by using the head command:

$ head /proc/cpuinfo

Table 1-1 presents the representative output for each of the platforms. This output may differ according to changes in models and so on.

Table 1-1 Determining the Node Platform

Platform Partial Output of /proc/cpuinfo

CP3000

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15 
model           : 3 
model name      : Intel(R) Xeon(TM)

CP4000

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15 
model           : 5 
model name      : AMD Opteron(tm)

CP6000

processor  : 0 
vendor     : GenuineIntel 
arch       : IA-64 
family     : Itanium 2 
model      : 1

CP300BL (Blade-only XC systems)

processor       : 0 
vendor_id       : GenuineIntel 
cpu family      : 15 
model           : 6 
model name      :        Intel(R) Xeon(TM) CPU 3.73GHz




	Note: The `/proc/cpuinfo` file is dynamic.

Node Specialization

The HP XC system is implemented as a sea-of-nodes. Each node in the system contains the same software image on its local disk. There are two types of nodes in the system — a head node and client nodes.

head node: The node is installed with the HP XC system software first — it is used to generate other HP XC (client) nodes. The head node is generally of interest only to the administrator of the HP XC system.
client nodes: All the other the nodes that make up the system. They are replicated from the head node and are usually given one or more specialized roles to perform various system functions, such as logging into the system or running jobs.

The HP XC system allows for the specialization of client nodes to enable efficient and flexible distribution of the workload. Nodes can be assigned one or more specialized roles that determine how a particular node is used and what system services it provides. Of the many different roles that can be assigned to a client node, the following roles contain services that are of special interest to the general user:

login role: The role most visible to users is on nodes that have the login role. Nodes with the login role are where you log in and interact with the system to perform various tasks. For example, once logged in to a node with login role, you can execute commands, build applications, or submit jobs to compute nodes for execution. There can be one or several nodes with the login role in an HP XC system, depending upon cluster size and requirements. Nodes with the login role are a part of the Linux Virtual Server ring, which distributes login requests from users. A node with the login role is referred to as a login node in this manual.
compute role: The compute role is assigned to nodes where jobs are to be distributed and run. Although all nodes in the HP XC system are capable of carrying out computations, the nodes with the compute role are the primary nodes used to run jobs. Nodes with the compute role become a part of the resource pool used by LSF-HPC and SLURM, which manage and distribute the job workload. Jobs that are submitted to compute nodes must be launched from nodes with the login role. Nodes with the compute role are referred to as compute nodes in this manual.

Storage and I/O

The HP XC system supports both shared (global) and private (local) disks and file systems. Shared file systems can be mounted on all the other nodes by means of Lustre™ or NFS. This gives users a single view of all the shared data on disks attached to the HP XC system.

SAN Storage

The HP XC system uses the HP StorageWorks Scalable File Share (HP StorageWorks SFS), which is based on Lustre technology and uses the Lustre File System from Cluster File Systems, Inc. This is a turnkey Lustre system from HP. It supplies access to Lustre file systems through Lustre client/server protocols over various system interconnects. The HP XC system is a client to the HP StorageWorks SFS server.

Local Storage

Local storage for each node holds the operating system, a copy of the HP XC System Software, and temporary space that can be used by jobs running on the node.

HP XC file systems are described in detail in “File System”.

File System

Each node of the HP XC system has its own local copy of all the HP XC System Software files including the Linux distribution; it also has its own local user files. Every node can also import files from NFS or Lustre file servers. HP XC System Software uses NFS 3, including both client and server functionality. HP XC System Software also enables Lustre client services for high-performance and high-availability file I/O. These Lustre client services require the separate installation of Lustre software, provided with the HP Storage Works Scalable File Share (SFS).

NFS files can be shared exclusively among the nodes of the HP XC System or can be shared between the HP XC and external systems. External NFS files can be shared with any node having a direct external network connection. It is also possible to set up NFS to import external files to HP XC nodes without external network connections, by routing through a node with an external network connection. Your system administrator can choose to use either the HP XC administrative network or the HP XC system interconnect for NFS operations. The HP XC system interconnect can potentially offer higher performance, but only at the potential decrease in the performance of application communications.

For high-performance or high-availability file I/O, the Lustre file system is available on HP XC. The Lustre file system uses POSIX-compliant syntax and semantics. The HP XC System Software includes kernel modifications required for Lustre client services which enables the operation of the separately installable Lustre client software. The Lustre file server product used on HP XC is the HP StorageWorks Scalable File Share (SFS), which fully supports the HP XC System Software.

The SFS also includes HP XC Lustre client software. The SFS can be integrated with the HP XC so that Lustre I/O is performed over the same high-speed system interconnect fabric used by the HP XC. So, for example, if the HP XC system interconnect is based on a Quadrics® QsNet II® switch, then the SFS will serve files over ports on that switch. The file operations are able to proceed at the full bandwidth of the HP XC system interconnect because these operations are implemented directly over the low-level communications libraries. Further optimizations of file I/O can be achieved at the application level using special file system commands – implemented as ioctls – which allow a program to interrogate the attributes of the file system, modify the stripe size and other attributes of new (zero-length) files, and so on. Some of these optimizations are implicit in the HP-MPI I/O library, which implements the MPI-2 file I/O standard.

File System Layout

In an HP XC system, the basic file system layout is the same as that of the Red Hat Advanced Server 3.0 Linux file system.

The HP XC file system is structured to separate cluster-specific files, base operating system files, and user-installed software files. This allows for flexibility and ease of potential upgrades of the system software and keeps software from conflicting with user installed software. Files are segregated into the following types and locations:

Software specific to HP XC is located in /opt/hptc
HP XC configuration data is located in /opt/hptc/etc
Clusterwide directory structure (file system) is located in /hptc_cluster

Be aware of the following information about the HP XC file system layout:

Open source software that by default would be installed under the /usr/local directory is instead installed in the /opt/hptc directory.
Software installed in the /opt/hptc directory is not intended to be updated by users.
Software packages are installed in directories under the /opt/hptc directory under their own names. The exception to this is third-party software, which usually goes in /opt/r.
Four directories under the /opt/hptc directory contain symbolic links to files included in the packages:
- /opt/hptc/bin
- /opt/hptc/sbin
- /opt/hptc/lib
- /opt/hptc/man
Each package directory should have a directory corresponding to each of these directories in which every file has a symbolic link created in the /opt/hptc/ directory.

System Interconnect Network

The HP XC system interconnect provides high-speed connectivity for parallel applications. The system interconnect network provides a high-speed communications path used primarily for user file service and for communications within user applications that are distributed among nodes of the system. The system interconnect network is a private network within the HP XC. Typically, every node in the HP XC is connected to the system interconnect.

Table 1-2 indicates the types of system interconnects that are used on HP XC systems.

Table 1-2 HP XC System Interconnects

	CP3000	CP4000	CP6000
Quadrics QSNet II®		X	X
Myrinet®	X	X
Gigabit Ethernet®	X	X	X
InfiniBand®	X	X	X

Additional information on supported system interconnects is provided in the HP XC Hardware Preparation Guide.

Network Address Translation (NAT)

The HP XC system uses Network Address Translation (NAT) to enable nodes in the HP XC system that do not have direct external network connections to open outbound network connections to external network resources.