Before configuring and administering XVM logical volumes, you should be familiar with the concepts that underlie the administration commands. This chapter describes the tasks that the XVM Volume Manager performs on physical and logical disk resources. More complete descriptions of the xvm command line interface (CLI) commands are provided in Chapter 4, “XVM Administration Commands”, along with examples of each of the commands.
The major sections in this chapter are:
An XVM object can be one of the following:
An XVM physical volume has a domain, which can be cluster or local. An XVM physical volume with a cluster domain is owned by a CXFS cluster, and it can be controlled by any of the nodes in that cluster, which is defined by the CXFS Cluster Manager. An XVM physical volume with a local domain is owned by a single node, and it can be controlled only by that node.
Only the owner for an XVM physical volume can modify the configuration on that physical volume. There may be XVM physical volumes that are seen by a host, but owned by another host or another cluster. XVM recognizes these disks and marks them as foreign. Disks without an XVM label are shown as unlabeled.
Figure 2-1 illustrates a physical volume that is controlled by a local owner. In this example, the XVM physvol lucy has a local domain of node ricky. The node ricky is part of the CXFS cluster neighbors that also includes the node fred and the node ethel, but neither fred nor ethel can control lucy.
In the configuration illustrated in Figure 2-1, the node ricky can see and modify the configuration of physvol lucy. The nodes fred, ethel, wilma, and betty see lucy as a foreign disk, and display only the disk path and not the physvol name itself. (If necessary, you can execute the show command on a foreign disk to determine its physvol name, as described in “Displaying Physical Volumes with the show Command” in Chapter 4.)
Figure 2-2 illustrates a physical volume that has a cluster domain. In this example, the physvol lucy has an owner of cluster neighbors, which consists of the nodes ricky, fred, and ethyl.
In the configuration illustrated in Figure 2-2, the nodes ricky, fred, and ethel can see and modify the configuration of the physvol lucy. The nodes wilma and betty cannot modify the configuration of lucy, even though they are connected to lucy through a SAN network; they see lucy as a foreign disk, and can display only the disk path.
An XVM logical volume that spans physvols may not span domains on a running system. A logical volume that spans local and cluster domains is marked offline.
When you bring up the XVM Volume Manager with the xvm command when cluster services are enabled, the xvm:cluster> prompt appears by default, indicating that all XVM physical volumes that you create in this XVM session are in the cluster domain. When cluster services are not enabled, the xvm:local> prompt appears, indicating that all XVM physical volumes that you create in this XVM session are in the local domain.
You can change the current XVM domain by invoking XVM with the -domain option, as described in “Using the XVM CLI” in Chapter 3, or by using the set command of the XVM Volume Manager, as described in “Changing the Current Domain with the set Command” in Chapter 4.
When you are running the XVM Volume Manager in the cluster domain, by default you can see and modify only the XVM physvols that are also in the cluster domain, even if you are running from the node that is the owner of a local physvol. To see and modify local disks, you either change your domain to local with the set domain command, or you use the local: prefix when specifying a physvol name. Similarly, when you are running the XVM Volume Manager in the local domain, you must change your domain to cluster or specify a cluster: prefix when specifying the physvol that is owned by the cluster. For information on setting and specifying XVM domains, see “Using the XVM CLI” in Chapter 3.
You can change the owner of an existing XVM physvol by using the XVM give command to give that physvol to a different owner, whether that owner is a single node or a cluster. When the node or cluster that currently owns the physical volume is unable to execute the give command, you can use the steal command to change the domain of an XVM physical volume. For information on the give and steal commands, see “Changing the Domain of a Physical Volume with the give and steal Commands” in Chapter 4.
Underlying XVM logical volumes are the physical volumes that make up the logical volumes. As part of XVM logical volume administration, you create, manage, and destroy XVM physical volumes, as described in the following sections.
In order to create XVM logical volumes on a physical disk, you must label the disk as an XVM disk by using the label command of the XVM Volume Manager. This command writes out an XVM physical volume label on a disk and allows the XVM volume manager to control the partitioning on the disk. In a CXFS cluster, any XVM physical volumes that will be shared must be physically connected to all nodes in the cluster.
When you label an XVM disk, you can specify whether the disk is an option disk, a system disk with a combined root and usr filesystem, or a system disk with separate root and usr filesystems. XVM physical volumes are option disks by default. You must be administering XVM in the local domain when you create an XVM system disks, since system disks are always local to the node which boots from them. For information on labeling disks as XVM system disks, see “XVM System Disks” in Chapter 4.
 | Note: XVM on Linux does not support XVM system disks. |
If you add a new disk that has already been labeled as an XVM physical volume to a running system, you must manually probe the disk by using the probe command of the XVM Volume Manager in order for the system to recognize the disk as an XVM disk. You do not have to do this when you are labeling a new XVM disk on your system, however, since the XVM Volume Manager probes the disk as part of the label process. All disks are probed when the system is booted to determine which disks are XVM disks.
By default, you cannot label a disk as an XVM disk if the disk contains any partitions that are currently in use as mounted filesystems. You can override this restriction with the -nopartchk option of the label command. Use the -nopartchk option with caution, as data corruption or system panics can result from labeling disks with partitions that are in use.
| Note: Before you can label a disk as an XVM disk, it must be formatted as an SGI disk. Under IRIX, if your disk has not been initialized during factory set-up, use the fx(1M) command to initialize the disk.For information on formatting a disk for use in XVM under Linux, see “Preparing to Configure XVM Volumes under Linux” in Chapter 7. |
You can perform the following tasks on physical volumes:
Display the physical volume
Change the domain of the physical volume
Add a physical volume to a running system
Replace a physical volume
Rename a physical volume
Display statistics for a physical volume
Change a physical volume from a system disk to an option disk
These tasks are described in the following subsections.
Use the XVM show command of the XVM Volume Manager to display information about physical volumes, both labeled and unlabeled. You can also use the show command to display information about disks that are foreign to the current node. You can use this feature to determine the current owner of a disk that is foreign to you, as described in “Displaying Physical Volumes with the show Command” in Chapter 4.
Use the XVM give command to change the owner of an existing XVM physvol, giving that physvol to a different local or cluster owner. When the node or cluster that currently owns the physical volume is unable to execute the give command, you can use the steal command to change the domain of an XVM physical volume. For information on the give and steal commands, see “Changing the Domain of a Physical Volume with the give and steal Commands” in Chapter 4.
When you boot your system, all disks connected to the system are probed to determine whether they are XVM disks. If you add an XVM disk to a system that is already running, you must manually probe the disk by using the probe command of the XVM Volume Manager so that the kernel recognizes the disk as an XVM disk.
The XVM Volume Manager allows you to replace a disk on a running system without rebooting the system. When you do this, you must regenerate the XVM label on the replacement disk. Use the dump command of the XVM Volume Manager to dump the commands to a file that will regenerate a physical volume label.
Note that when you dump the commands to regenerate a physical volume label, you must separately and explicitly dump the commands to regenerate the volume element tree that leads to the physical volume, as described in “Reconstructing Volume Elements: Using the dump Command” in Chapter 4.
You can rename a physical volume with the name option of the change command.
The XVM Volume Manager can maintain statistics for physical volumes, subvolumes, stripes, concats, mirrors, and slices. You can use the stat option of the change command of the XVM Volume Manager to turn statistics on and off and to reset the statistics for a physical volume. See Chapter 8, “Statistics”, for information on the statistics XVM maintains.
In a clustered environment, statistics are maintained for the local node only.
It may be necessary to change an XVM system disk to an XVM option disk if you need to delete the logical volumes on an XVM system disk, since volumes marked as swap volumes cannot be deleted. You can change an XVM disk to an option disk with the change command. For information on deleting system disks, see “Deleting XVM System Disks” in Chapter 4.
To remove an XVM physical volume from a system, use the unlabel command of the XVM Volume Manager to remove the XVM physical volume label from an XVM disk and restore the original partitioning scheme. The unlabel command provides a -force option which deletes each slice that currently exists on the physical volume, even if the slice is part of an open subvolume and its deletion will cause the subvolume state to go offline.
The swap partition of an XVM system disk cannot be deleted. This is to ensure that the swap partition cannot be deleted accidentally and cause a system panic. If you need to delete the logical volumes on a system disk so that you can unlabel the disk, you must first use the XVM change command to change the disk from a system disk to an option disk, as described in “Deleting XVM System Disks” in Chapter 4.
After you have created the XVM physical volumes you will use for your logical volume, you can create the logical volume elements that will make up the logical volume.
XVM topologies can be built top-down or bottom-up. Any tree or subtree you create that does not end in a slice will not have labels written to disk, and therefore will not be persistent across reboots.
While you are building your XVM topology, you may find it useful to display the existing defined topology for a volume element by using the -topology option of the show command of the XVM Volume Manager.
When volume elements other than volumes are created, they must be associated with a volume. You can name and create the volume explicitly when you create the volume element, or you can specify that the volume be automatically generated with a temporary name. A subvolume of type data is automatically generated for the volume (unless the volume element you are creating is itself a subvolume of a different type). Automatic volume and subvolume generation ensures that when an object is constructed, it can be immediately used by an application such as mkfs to initialize a filesystem.
When you explicitly name a volume, the volume name is stored in the label space and remains persistent across machine reboots. When the system generates a volume and volume name automatically, a new and possibly different name will be generated when the system reboots. Slices, however, are a special case; when the system generates a volume name for a slice, the volume name is not temporary and remains persistent across reboots.
You can make a temporary volume name persistent across reboots by using the change command to rename the volume.
Volume elements that compose an XVM volume are named as follows:
Slices are named automatically when you create them.
Slice names remain persistent across machine reboots. This makes it convenient to reorganize and rebuild logical volumes using slices you have defined for each disk, even after you have rebooted the system.
You can name stripes, concats, and mirrors explicitly when you create them. If you do not name them explicitly, you must specify that a default temporary name should be generated.
When you name stripes, concats, and mirrors explicitly, the volume element name is stored in the label space and remains persistent across machine reboots. Information on setting the size of the label space is provided in “Assigning Disks to the XVM Volume Manager with the label Command” in Chapter 4.
You can name a subvolume explicitly only if it is of a user-defined type. Data subvolumes are named data and log subvolumes are named log.
As described in “Automatic Volume and Subvolume Creation”, volumes can be created and named when you create the elements within those volumes. You can also create an empty volume and give it a name explicitly. If you do not name an empty volume when you create it, you must specify that the system generate a temporary name; this practice is not recommended for general configuration.
You can make temporary volume element names persistent across reboots by using the change command to rename the volume element.
It is not necessary to use the name of a volume element when you manipulate it. You can use its relative position in the logical volume instead. These naming options as well as general information on the syntax of volume element names are described in “Object Names in XVM” in Chapter 3.
When you create XVM logical volumes by attaching volume elements to one another through volume element creation or through the attach command, the following rules and restrictions are enforced by the XVM Volume Manager:
The source of an attach must be a subvolume, or the child of a subvolume. You cannot attach a volume to another volume element.
Subvolumes can be attached only to volumes.
Subvolumes can have only one child.
A volume cannot have more than one system-defined subvolume of a given type. The system-defined subvolumes are data subvolumes, log subvolumes, and real-time subvolumes.
A mirror cannot have more than eight members.
If you specify a position when you create or attach a volume element to a target volume element, the target volume element must not already have a volume element in that position.
When attaching a volume element to a target that is part of an open subvolume, the attachment cannot change the way the data is laid out in the target or any ancestor of the target. Examples of attaches that can affect data layout are:
Appending to a stripe
Appending to a concat which results in the growth of an ancestor that is not the rightmost volume element under its parents
When you attach a volume element to a mirror, this initiates a mirror revive during which the system mirrors the data. A message is written to the SYSLOG when this process is complete. You cannot halt a mirror revive once it has begun except by detaching all but one of the pieces of the mirror.
When you use the -safe option of an xvm command, you cannot attach volume elements that change the way the data is laid out in the target or any ancestor of the target even if the target does not belong to an open subvolume.
When you attach multiple source volume elements to a single target volume element, they are attached one at a time, in turn. If an attach in the list fails, XVM attempts to restore the volume elements to their previous parents. If a volume element cannot be restored, a warning message is generated and manual intervention is needed.
Use the detach command of the XVM Volume Manager to detach a volume element from its parent. When you detach a volume element, a new volume (and possibly data subvolume) will be created, just as a volume is created when you create a volume element. You can name the generated volume explicitly, or you can specify that the volume be automatically generated with a temporary name. A subvolume of type data is automatically generated for the volume element you are detaching (unless the volume element you are detaching is itself a subvolume of a different type).
If the volume element you detach is part of an open subvolume, its detachment cannot cause the subvolume state to go offline. Any command that would reduce the address space of an open subvolume, such as detaching a slice that is not mirrored, will cause it to go offline. You cannot detach the last valid piece of an open mirror from that mirror, since this will cause the mirror to go offline.
The detach command provides a -force option to override the restriction that you cannot detach a volume element that will cause a subvolume to go offline. The detach command also provides a -safe option to impose this restriction even if the subvolume is not open. See “The detach Command” in Chapter 4 for examples of this command.
You cannot detach a mirror leg that is being revived without using the -force option. You cannot detach the last leg of a mirror without using the -force option. These restrictions hold whether or not the subvolume is open.
When you create stripes, mirrors, concats, subvolumes, and volumes, you have the option of not specifying which child volume elements will compose these volume elements. If you do not specify the child elements, an empty volume element is created and you can attach volume elements at a later time.
The XVM Volume Manager can maintain statistics for physical volumes, subvolumes, stripes, concats, mirrors, and slices. You can use the stat option of the change command of the XVM Volume Manager to turn statistics on and off and to reset the statistics for a volume element. See Chapter 8, “Statistics”, for information on the statistics XVM maintains.
In a clustered environment, statistics are maintained for the local node only.
Use the slice command to create a slice from a block range of an XVM physical volume. You can specify the starting block of a slice and you can specify the length of a slice. In addition, you can specify the following methods of creating slices:
You can create a slice out of all of the blocks of a physical volume
You can divide a specified address range into equal parts, with each part a different slice
You can slice multiple physical volumes at once
You can specify that a slice is a system slice of type root, swap, or usr. For information on creating system slices, see “Configuring System Disks with the slice Command” in Chapter 4.
Slices are named automatically and are persistent across machine reboots. You cannot rename slices.
The volume that is generated when you create a slice is persistent across machine reboots. You can specify the name of the volume that is created when you create a slice. By default, the volume name will be the same as the slice object name.
Use the concat command of the XVM Volume Manager to create a concat, which is a volume element that concatenates all of its child volume elements into one address space.
The XVM Volume Manager enforces the rules of attachment during concat creation, as described in “Attaching Volume Elements”.
Use the stripe command of the XVM Volume Manager to create a stripe, which is a volume element that stripes a set of volume elements across an address space.
You can create a stripe that is made up of volume elements of unequal size, although this will leave unused space on the larger volume elements.
The XVM Volume Manager enforces the rules of attachment during stripe creation, as described in “Attaching Volume Elements”.
For information on configuring stripes that span two host bus adaptors, see “XVM Logical Volumes and Failover” in Chapter 1.
Use the mirror command of the XVM Volume Manager to create a mirror, which is a volume element that mirrors all of its child volume elements.
| Note: To use the mirroring feature of the XVM Volume Manager, you must purchase and install the appropriate FLEXlm license on IRIX or LK license on SGI ProPack 6 for Linux. |
When you create a mirror that has more than one piece, a message is written to the SYSLOG indicating that the mirror is reviving. This indicates that the system is beginning the process of mirroring the data. Another message is written to the SYSLOG when this process is complete. Should the revive fail for any reason, a message will be written to the system console as well as to the SYSLOG.
For large mirror components, this revive process may take a long time. When you are creating a new mirror that does not need to be revived, you should consider using the -clean option of the mirror command, as described in ???. When you are creating a new mirror that you will use for scratch filesystems that will never need to be revived, you should consider using the -norevive option of the mirror command, as described in “ The -norevive Mirror Creation Option”.
You cannot halt a mirror revive once it has begun except by detaching all but one of the pieces of the mirror. For more information on mirror revives, see “ Mirror Revives” in Chapter 9.
The XVM Volume Manager enforces the rules of attachment during mirror creation, as described in “Attaching Volume Elements”.
When you create a mirror you have the options of setting the following characteristics for the mirror:
The read policy for the mirror
The primary leg for the mirror
Whether the mirror will be synchronized at creation (the -clean option)
Whether the mirror will be resynchronized when the system boots (the -norevive option)
The following sections describe each of these options.
The XVM Volume Manager allows you to specify one of the following read policies for a mirror:
| round-robin | Balance the I/O load among the members of the mirror, blindly reading in a round-robin fashion. | |
| sequential | Route sequential I/O operations to the same member of the mirror. |
Figure 2-3 illustrates how data is read from the legs of a mirror with a round-robin read policy. The wedges represent units of data that you are reading. The first read operation gets the unit of data from the first leg, the second read operation gets the next unit of data from the second leg, and the next read operation gets the next unit of data from the third leg. The next read operation gets the requested unit of data from the first leg again.
Figure 2-4 illustrates how data is read from the legs of a mirror with a sequential read policy, showing that the different mirror members are not accessed for a single sequential I/O operation.
After you have defined a mirror, you can change the read policy with the change command.
You can specify whether a particular leg of a mirror is to be preferred for reading by marking it as a primary leg. After you have defined a mirror, you can redefine whether a leg is a primary leg with the change command.
When you create a mirror, you can use the -clean option of the mirror command to specify that the legs of the mirror do not need to be revived on creation. This option is useful when the legs of the mirror are already mirrored or when the mirror is new and all data will be written before being read.
When you create a mirror, you can use the -norevive option of the mirror command to specify that the legs of the mirror do not need to be revived when the system boots. This option is useful when you are creating a mirror for a scratch filesystem such as /tmp or swap.
If you are creating a mirror for a swap filesystem that is in active use, you can specify -norevive (but not -clean). If you are creating a mirror for a swap filesystem that is not in active use, then you can specify both -norevive and -clean as there is no valid data on either mirror leg. If you are creating a mirror for a filesystem that has no yet been used (and therefore contains no valid data on either leg) then specifying -clean by itself eliminates the initial revive.
Use the volume command of the XVM Volume Manager to create an XVM logical volume explicitly. Volumes may also be created automatically when you create a volume element, as described in “Automatic Volume and Subvolume Creation”.
When you create a volume with the volume command, you can specify subvolumes to attach to the volume after it is created. When subvolumes are attached to a volume, the XVM Volume Manager enforces the rules of attachment described in “Attaching Volume Elements”.
Use the subvolume command of the XVM Volume Manager to create a subvolume explicitly. Subvolumes of type data may also be created automatically when you create a volume element, as described in “Automatic Volume and Subvolume Creation”.
When you create a subvolume with the subvolume command, you can specify the volume element to attach to the subvolume. The volume element to attach to the subvolume cannot be a volume or a subvolume. If you do not specify a volume element to attach, an empty subvolume is created.
When you create a subvolume with the subvolume command, you can specify the subvolume type. This can be a system-defined subvolume type or a user-defined subvolume type. There are three system-defined subvolume types:
There cannot be more than one subvolume having the same system-defined type under a volume, and you cannot specify a user-defined name for a system-defined subvolume type.
| Note: XVM on Linux does not support real-time subvolumes. |
A user-defined subvolume type is in the range 16 through 255 (0 through 15 are reserved for system-defined types). System-defined subvolume types are used to associate an application-dependent type with a subvolume. You can specify more than one subvolume of a specific user-defined type under a volume.
As you create a logical volume, you can use the attach and detach commands of the XVM Volume Manager to organize and reorganize its elements. Additionally, after you have created volume elements you can reorganize the volume elements by using the remake command of the XVM Volume Manager. The remake command collapses holes in a volume element or rearranges pieces under a volume element. You can use a single remake command as a convenient alternative to executing a series of attach and detach commands.
After you have created your logical resources, you can perform the following tasks:
Display volume elements
Disable volume elements
Bring volume elements online
Make online changes to volume elements
Save the logical volume configuration
The following subsections summarize these procedures.
Use the show command of the XVM Volume Manager to display information about volume elements.
A volume element can be in one or more of the following states:
You can use the change command of the XVM Volume Manager to manually disable a volume element. When you disable a volume element, no I/O can be done to that volume element until you explicitly enable the element, which you can also do with the change command. The object remains disabled until explicitly enabled, even across machine reboots.
The system kernel may disable a volume element and take that element offline. This could happen, for example, when a mirror member shows an I/O error. You can use the change command of the XVM Volume Manager to bring the volume element back online.
You can insert a mirror or a concat above another volume element using the insert command of the XVM Volume Manager. This command can be used to grow a volume element or to add a mirror to a running system, because the volume element you are inserting can be part of an open subvolume and can have active I/O occurring.
You can remove a layer from a tree by using the collapse command of the XVM Volume Manager. Generally you use a collapse command to reverse a previous insert operation.
To save an XVM logical volume configuration, use the dump command of the XVM Volume Manager to dump the commands to a file that will regenerate a configuration. This allows you to replace a disk in a running system and to regenerate the XVM configuration on the new disk without rebooting the system.
When you dump and regenerate a device, you do not regenerate the data on the disk you are replacing, but rather you regenerate the XVM configuration on the new disk.
Note that when you dump the commands to regenerate a volume element tree, you must separately and explicitly dump the commands to regenerate the physical volumes that the tree leads to, as described in “Reconstructing Volume Elements: Using the dump Command” in Chapter 4.
It is possible for XVM labels to become corrupted. A good practice is to backup your XVM configuration using the xvm dump command any time it is changed just in case you need to recover from potential problems. You should save the xvm dump output into a filesystem other than the one being dumped.
For example, the following will dump all xvm_labels to the file /var/xvm_config:
xvm dump -topology -f /var/xvm_config phys/'*' vol/'*' |
The following sections describe how to remove XVM elements, and how to remove configuration information from the kernel when an XVM disk becomes unavailable for I/O.
Use the delete command of the XVM Volume Manager to delete a volume element. Parents of deleted volume elements remain and have open slots.
In general, if a volume element contains any attached children, it cannot be deleted. However, you can specify that either all of the children or all of the children but the slices be deleted by using the -all or -nonslice options, respectively. When you specify the -nonslice option, the slices are detached and a volume and data subvolume are automatically generated for the slices.
If the volume element you delete is part of an open subvolume, its deletion cannot cause the subvolume state to go offline. The delete command provides a -force option to override this restriction.
The swap partition of an XVM system disk cannot be deleted. This is to ensure that the swap partition cannot be deleted accidentally and cause a system panic. If you need to delete the logical volumes on a system disk so that you can unlabel the disk, you must first use the XVM change command to change the disk from a system disk to an option disk, as described in “Deleting XVM System Disks” in Chapter 4.
When an XVM disk becomes physically unavailable, you may not be able to execute standard XVM configuration commands on logical volumes that include that disk. To recover from this situation, you can use the reprobe command of the XVM volume manager to remove previous configuration information from the kernel.
For information on using the reprobe command, see “Removing Configuration Information: Using the reprobe Command” in Chapter 4.