Preparing the disks
Prepare your disks for GlusterFS configuration.
You must use symlinks of the disks when you configure GlusterFS. You have the option to configure the disks as multipath volumes and to encrypt the volumes.
Multipath volumes
You can use multipath volumes for GlusterFS.
For more information about setting up multipathing, see the following documentation:
- For Ubuntu, see https://help.ubuntu.com/lts/serverguide/dm-multipath-chapter.html
- For Red Hat Enterprise Linux (RHEL), see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/index
- For SUSE Linux Enterprise Server (SLES), see https://www.suse.com/documentation/sles-12/stor_admin/data/cha_multipath.html .
Encrypted volumes
You can use encrypted volumes for GlusterFS. For more information about encrypting volumes, see Encrypting volumes by using dm-crypt.
The encrypted volumes are available at /dev/mapper/<encrypted-volume-name>
location.
Symlinks
You must use the symbolic link (symlink) to identify the GlusterFS storage device. Do not use device names, such as /dev/sdb
, because the name might change between system restarts.
Note: The special characters that Heketi allows to be used in the device name are ^/[a-zA-Z0-9_.:/-]+$
. If your device name or system-generated symlink has special characters that are not allowed by Heketi, then you
must manually create the symlink.
Use system-generated symlinks
To get the symlink that the system assigns to a device, complete these steps:
-
Identify the storage devices to use. You can list the available storage devices by entering this command:
fdisk -l
The output looks like this example:
root@icps-worker-4:~# fdisk -l Disk /dev/sdb: 48 GiB, 51539607552 bytes, 100663296 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sda: 40 GiB, 42949672960 bytes, 83886080 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xe96e93a3 Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 7813119 7811072 3.7G 82 Linux swap / Solaris /dev/sda2 7815166 83884031 76068866 36.3G 5 Extended /dev/sda5 7815168 83884031 76068864 36.3G 83 Linux
Note: If you use multipathed or encrypted devices, these devices are available at the
/dev/mapper/
location. - Identify devices that have at least 25 GB of storage capacity. In the example, the device
/dev/sdb
is considered. -
Erase all file system, raid, and partition-table signatures by using the
wipefs
command. For example, to erase the signatures on device/dev/sdb
, run the following command:sudo wipefs --all --force /dev/sdb
-
Get the symlink of the device by entering this command:
ls -altr /dev/disk/*
The output looks like this example:
root@icps-worker-4:~# ls -altr /dev/disk/* /dev/disk/by-path: total 0 drwxr-xr-x 2 root root 160 Oct 23 02:34 . lrwxrwxrwx 1 root root 9 Oct 23 02:34 pci-0000:00:10.0-scsi-0:0:1:0 -> ../../sdb lrwxrwxrwx 1 root root 9 Oct 23 02:34 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda lrwxrwxrwx 1 root root 9 Oct 23 02:34 pci-0000:02:01.0-ata-1 -> ../../sr0 drwxr-xr-x 6 root root 120 Oct 23 02:34 .. lrwxrwxrwx 1 root root 10 Oct 23 02:34 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 23 02:34 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 23 02:34 pci-0000:00:10.0-scsi-0:0:0:0-part5 -> ../../sda5 /dev/disk/by-id: total 0 drwxr-xr-x 2 root root 260 Oct 23 02:34 . lrwxrwxrwx 1 root root 9 Oct 23 02:34 wwn-0x6000c29ddc232994ce32cad1533c25e1 -> ../../sda lrwxrwxrwx 1 root root 9 Oct 23 02:34 wwn-0x6000c296507a2c4be3b79a74f921d283 -> ../../sdb lrwxrwxrwx 1 root root 9 Oct 23 02:34 scsi-36000c29ddc232994ce32cad1533c25e1 -> ../../sda lrwxrwxrwx 1 root root 9 Oct 23 02:34 scsi-36000c296507a2c4be3b79a74f921d283 -> ../../sdb lrwxrwxrwx 1 root root 9 Oct 23 02:34 ata-VMware_Virtual_SATA_CDRW_Drive_00000000000000000001 -> ../../sr0 drwxr-xr-x 6 root root 120 Oct 23 02:34 .. lrwxrwxrwx 1 root root 10 Oct 23 02:34 wwn-0x6000c29ddc232994ce32cad1533c25e1-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 23 02:34 wwn-0x6000c29ddc232994ce32cad1533c25e1-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 23 02:34 scsi-36000c29ddc232994ce32cad1533c25e1-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 23 02:34 scsi-36000c29ddc232994ce32cad1533c25e1-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 23 02:34 wwn-0x6000c29ddc232994ce32cad1533c25e1-part5 -> ../../sda5 lrwxrwxrwx 1 root root 10 Oct 23 02:34 scsi-36000c29ddc232994ce32cad1533c25e1-part5 -> ../../sda5 /dev/disk/by-uuid: total 0 lrwxrwxrwx 1 root root 9 Oct 23 02:34 2017-02-15-20-36-22-00 -> ../../sr0 drwxr-xr-x 6 root root 120 Oct 23 02:34 .. drwxr-xr-x 2 root root 100 Oct 23 02:34 . lrwxrwxrwx 1 root root 10 Oct 23 02:34 6d48a0d5-846b-48d3-bb0f-a626bb49c916 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 23 02:34 60f23142-a575-455d-8a99-6edca077d990 -> ../../sda5 /dev/disk/by-label: total 0 lrwxrwxrwx 1 root root 9 Oct 23 02:34 Ubuntu-Server\x2016.04.2\x20LTS\x20amd64 -> ../../sr0 drwxr-xr-x 6 root root 120 Oct 23 02:34 .. drwxr-xr-x 2 root root 60 Oct 23 02:34 . root@icps-worker-4:~#
The symlink of a device can be in any of these link paths:
/dev/disk/by-path
,/dev/disk/by-id
,/dev/disk/by-uuid
, or/dev/disk/by-label
. -
Make a note of the symlink and its link path. For the example device
sdb
,/dev/disk/by-id
is the link path andscsi-36000c296507a2c4be3b79a74f921d283
is the symlink. For each device that you are using for GlusterFS configuration, you need to add the<link path>/<symlink>
in theconfig.yaml
file. For the example devicesdb
, you would add/dev/disk/by-id/scsi-36000c296507a2c4be3b79a74f921d283
in theconfig.yaml
file.
Use manually created symlinks
In some environments, such as IBM Cloud VSI or SLES, no symlinks are automatically generated for the devices. You must manually create symlinks by writing custom udev (userspace /dev) rules. When you create the symlink, use attributes that are unique to the device.
Following example includes steps to manually generate symlinks. The steps might vary with operating systems and environments.
-
Get information about the device's attributes.
udevadm info --root --name=/dev/vdb
The output resembles the following code:
P: /devices/pci0000:00/0000:00:10.0/virtio4/block/vdb N: vdb E: DEVNAME=/dev/vdb E: DEVPATH=/devices/pci0000:00/0000:00:10.0/virtio4/block/vdb E: DEVTYPE=disk E: MAJOR=253 E: MINOR=16 E: SUBSYSTEM=block E: TAGS=:systemd: E: USEC_INITIALIZED=6705725 E: elevator=noop
Use the
DEVTYPE
,SUBSYSTEM
, andDEVPATH
attributes to create the symlink of the device. -
Create a custom udev rules file.
vi /lib/udev/rules.d/10-custom-icp.rules
Add these lines of code to the file. Replace the attribute values with your device attribute values.
ENV{DEVTYPE}=="disk", ENV{SUBSYSTEM}=="block", ENV{DEVPATH}=="/devices/pci0000:00/0000:00:10.0/virtio4/block/vdb" SYMLINK+="disk/gluster-disk-1"
-
Reload the udev rules to create the symlinks.
udevadm control --reload-rules udevadm trigger --type=devices --action=change
-
Verify that the symlinks are created.
ls -ltr /dev/disk/gluster-*
The output resembles the following code:
lrwxrwxrwx 1 root root 3 Jul 4 23:12 /dev/disk/gluster-disk-1 -> vdb