Hot-plug operations target a particular slot using the name of the slot. All slots that are eligible for a hot-plug operation have a directory in the hot-plug file system that is mounted at /sys/bus/pci/slots. The name of the target slot is based on the hardware location of the slot in the system. For the SGI ProPack 6 release, slot directories are in the form that the lspci(8) command uses, that is, as follows:

segment:bus:slot

Change directory (cd), to the /sys/bus/pci/slots directory and use the ls or ls -l command to view the contents of the file, as follows:

pci/slots> ls
0001:00:01  0002:00:01  0002:00:02  0003:00:01  0004:00:01  0004:00:02

Slot is part of a PCI domain. On an SGI Altix system, a PCI domain is a functional entity that includes a root bridge, subordinate buses under the root bridge, and the peripheral devices it controls. For more information, see “PCI Domain Support for SGI Altix Systems”.

Each slots directory contains two files called path and power. For example, change directory to /sys/bus/pci/slots/0001:00:01 and perform the ls command, as follows:

slots/0001:00:01> ls
path  power 

The power file provides the current hot-plug status of the slot. A value of 0 indicates that the slot is powered-down, and a value of 1 indicates that the slot is powered-up.

The path file is the module ID where the brick resides.

A hot-plug insert operation first instructs the L1 hardware controller to power-up the slot and reset the card. The L1 controller then checks that the card to be inserted is compatible with the running bus. Compatible is defined, as follows:

Any L1 controller detected incompatibilities or errors are reported to the user and the insert operation fails.

Once the slot has been successfully powered-up by the L1 controller, the Linux hot-plug infrastructure notifies the driver of the card that the card is available and needs to be initialized. After the driver has initialized the card, the hot-plug insert operation is complete and card is ready for use.

Before initiating a hot-plug remove operation, the system administrator must manually terminate any processes using the target card.

---

Warning: Failure to properly terminate any outstanding accesses to the target card may result in a system failure or data corruption when the hot-plug operation is initiated.

---

For a hot-plug remove operation, the hot-plug infrastructure verifies that the target slot is eligible to be powered-down. The L1 hardware controller restrictions do not permit the last card to be removed from a bus running at 33 MHz and an attempt to remove the last card fails. The hot-plug infrastructure then notifies the driver of the card of a pending hot-plug remove operation and the driver deactivates the card. The L1 hardware controller is then instructed to power-down the slot.

Attempts to power-down a slot that is already powered-down, or power-up a slot that is already powered-up are ignored.

For SGI ProPack 3 and SGI ProPack 4 non-blade systems, the XSCSI name has the following forms.

For direct attached devices, the form of the XSCSI name is, as follows:

/dev/xscsi/pciBB.SS.F[-C]/targetX/lunL/partT

For SAN attached devices, it is, as follows:

/dev/xscsi/pciBB.SS.F[-P]/nodeWWN/portP/lunL/partT

Where:

BB is the PCI bus number

SS is the Slot

F is the Function

C is the Channel ("-C" for QLA12160 HBA cards only)

For direct attach devices:

X is the target number

For SAN attached devices:

WWN is the world wide node number of the device

P is the port number of the device

For either direct attach or SAN attach devices:

L is the logical unit number (LUN) in lunL

T is the partition number

There are two ways of handling multiple port host bus adapter (HBA) cards in PCI. One way is to have each port be a different function. The other way is to have one function but have multiple channels off of the function. Most HBA cards use multiple functions. Therefore, most HBA cards will have differing F (function) numbers and the -C will be absent. The QLA12160 (Qlogic Parallel SCSI) uses multiple channels. Therefore, it will have one function, "0" and multiple channels, that is, "0-1" and "0-2".

An example of a direct attached device with partition 1 is, as follows:

/dev/xscsi/pci01.02.0/target1/lun0/part1

The same device attached device off of a Qlogic SCSI HBA card (or base IO9 base I/O) would be, as follows:

/dev/xscsi/pci01.02.0/target1/lun0/part1

An example of a SAN attached device is, as follows:

/dev/xscsi/pci22.01.1/node20000004cf2c84de/port1/lun0/part1

All SGI Altix systems running SGI ProPack 6 for Linux use domain-based XSCSI names. The XSCSI names change to accommodate PCI Express. They basically have the same form except that the PCI numbering takes the following form:

/dev/xscsi/pciDDDD:BB:SS.F[-C]/...

Where:

DDDD is the domain number

BB is the Bridge number

SS is the slot number

F is the function

C is the Channel

An example of a direct attach device with partition 1 is, as follows:

/dev/xscsi/pci0001:00:03.0-1/target1/lun0/part1

An example of a SAN attached device with partition 1 is, as follows:

/dev/xscsi/pci0002:00:01.0/node20000004cf2c8d0c/port2/lun0/part1

Note that the device number (slot number), function WWN, logical unit, and port number are fixed. These will never change. However, the system bus number could change because of a hardware problem (such as the I/O brick not booting) or the system being reconfigured.

For non-PCI express host bus adapter (HBA) cards in an SGI Altix 4700 system, the domain number is equivalent to the old bus number and the bridge number is 0. For more information, see “PCI Domain Support for SGI Altix Systems”.

You can use SGI's NUMAlink technology and the XPC and XPMEM kernel modules to create a very low latency, very large, shared-memory cluster for optimized use of Message Passing Interface (MPI) software and logically shared, distributed memory access (SHMEM) routines. The globally addressable, cache coherent, shared memory is exploited by MPI and SHMEM to deliver high performance.

Another reason for partitioning a system is fault containment. In most cases, a single partition can be brought down (because of a hardware or software failure, or as part of a controlled shutdown) without affecting the rest of the system. Hardware memory protections prevent any unintentional accesses to physical memory on a different partition from reaching and corrupting that physical memory. For current fault containment caveats, see “Limitations of Partitioning”.

You can power off and “warm swap” a failing C-brick in a down partition while other partitions are powered up and booted. For information see “Adding or Replacing a PCI or PCI-X Card” in chapter 12, “Maintenance and Upgrade Procedures” in the SGI Altix 3000 User's Guide or see “PCI and PCI-X Cards” in Chapter 6, “Installing and Removing Customer-replaceable Units” in the SGI Altix 350 User's Guide or see the appropriate chapter in the new SGI Altix 4700 User's Guide.

Partitions can be of different sizes, and a particular system can be configured in more than one way. For example, a 128-processor system could be configured into four partitions of 32 processors each or configured into two partitions of 64 processors each. (See "Supported Configurations" for a list of supported configurations for system partitioning.)

Your choice of partition size and number of partitions affects both fault containment and scalability. For example, you may want to dedicate all 64 processors of a system to a single large application during the night, but then partition the system in two 32 processor systems for separate and isolated use during the day.

One of the fundamental factors that determines the performance of a high-end computer is the bandwidth and latency of the memory. The SGI NUMAflex technology gives an SGI ProPack partitioned, shared-memory cluster a huge performance advantage over a cluster of commodity Linux machines (white boxes). If a cluster of N white boxes, each with M CPUs is connected via Ethernet or Myrinet or InfinaBand, an SGI ProPack system with N partitions of M CPUs provides superior performance because of the significantly lower latency of the NUMAlink interconnect, which is exploited by the XPNET kernel module.

SGI ProPack for Linux servers have XP, XPC, XPNET, and XPMEM kernel modules installed by default to provide partitioning support. XPC and XPNET are configured off by default in the /etc/sysconfig/sgi-xpc and /etc/sysconfig/sgi-xpnet files, respectively. XPMEM is configured on by default in the /etc/sysconfig/sgi-xpmem file. To enable or disable any of these features, edit the appropriate /etc/sysconfig/ file and execute the /etc/init.d/sgi-xp script.

On SGI ProPack systems running SLES10 or SLES11, if you intend to use the cross-partition functionality of XPMEM, you will need to add xpc to the line in the /etc/sysconfig/kernel file that begins with MODULES_LOADED_ON_BOOT. Once that is added, you may either reboot the system or issue an modprobe xpc command to get the cross-partition functionality to start working. For more information on using modprobe, see the modprobe(8) man page.

To activate xpc on future boots on SGI ProPack 6 systems running RHEL5, you need to add a modprobe xpc line to the /etc/sysconfig/modules/sgi-propack.modules file instead of adding the module name to the MODULES_LOADED_ON_BOOT line.

The XP kernel module is a simple module which coordinates activities between XPC, XPMEM, and XPNET. All of the other cross-partition kernel modules require XP to function.

The XPC kernel module provides fault-tolerant, cross-partition communication channels over NUMAlink for use by the XPNET and XPMEM kernel modules.

The XPNET kernel module implements an Internet protocol (IP) interface on top of XPC to provide high-speed network access via NUMAlink. XPNET can be used by applications to communicate between partitions via NUMAlink, to mount file systems across partitions, and so on. The XPNET driver is configured using the ifconfig commands. For more information, see the ifconfig(1M) man page. The procedure for configuring the XPNET kernel module as a network driver is essentially the same as the procedure used to configure the Ethernet driver. You can configure the XPNET driver at boot time like the Ethernet interfaces by using the configuration files in /etc/sysconfig/network-scripts. To configure the XPNET driver as a network driver see the following procedure.

The procedure for configuring the XPNET driver as a network driver is essentially the same as the procedure used to configure the Ethernet driver (eth0), as follows:

The XPMEM kernel module provides direct access to memory located on other partitions. It uses XPC internally to communicate with XPMEM kernel modules on other partitions to accomplish this. XPMEM is currently used by SGI's Message Passing Toolkit (MPT) software (MPI and SHMEM).

Partitioning rules define the set of valid configurations for a partitioned system. Fault isolation is one of the major reasons for partitioning a system. A software or hardware failure in one partition should not cause a failure in another partition. This section describes restrictions are placed on partitions to accomplish this. This section covers these topics:

This section describes how to partition your system.

To partition your system, perform the following steps :

To partition your system, perform the following steps:

cranberry-192.168.11.92-L2>p 1,2 rst

To determine whether a system is partitioned or not, perform the following steps:

To access the console on a partition, perform the following steps:

When a system is partitioned, the L2 prompt shows the partition number of the partition you selected, as follows:

cranberry-001-L2>sel p 2
console input: partition 2, 001c24 console0
console output: any brick partition 2
cranberry-001-L2:p2>

To remove the partitions from a system, perform the following steps:

System partitioning is an administrative function. The system console is connected to the controller as required by the configuration selected when an SGI ProPack system is installed. For additional information or recabling, contact your service representative.