Mismatch of Heketi disk usage
Mismatch of actual disk space and the used disk space that is reported by Heketi.
When you create a persistent volume claim (PVC), you might see an error message that no disk space is available.
Resolving the issue
If you see a disk usage mismatch in the heketi topology information, run these commands to synchronize the disk usage.
-
Set up kubectl CLI. See Accessing your cluster from the kubectl CLI.
-
Get the Heketi deployment pod name.
kubectl -n kube-system get pods | grep heketi
Following is an example output of the command:
storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf 1/1 Running 0 18h
-
Log in to the Heketi pod and get the list of nodes.
kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin node list
Following is an example output of the command:
Id:10cc13f4cd18136fc9b6d2a6d1eac733 Cluster:34fe224a09d8022a3582da817c31a81b Id:157012bb7b0391b2c116b35de8d6e7ba Cluster:34fe224a09d8022a3582da817c31a81b Id:95c72d452bc2c21b3a1e2fa87e63f902 Cluster:34fe224a09d8022a3582da817c31a81b
-
Get information about the devices on the nodes. Repeat this step for each node.
kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin node info <node-ID>
Following is an example command and output:
kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin node info 10cc13f4cd18136fc9b6d2a6d1eac733 Node Id: 10cc13f4cd18136fc9b6d2a6d1eac733 State: online Cluster Id: 34fe224a09d8022a3582da817c31a81b Zone: 1 Management Hostname: 10.41.4.108 Storage Hostname: 10.41.4.108 Devices: Id:c7e93cd3d3d293a78dfb99cb2809f699 Name:/dev/disk/by-path/virtio-pci-0000_00_11_0 State:online Size (GiB):699 Used (GiB):497 Free (GiB):202 Bricks: Id:33cb391e0113ec96ccc546a4e4288018 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick Id:41a508db035b042b1d35839f0d40f0c5 Size (GiB):20 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick
-
Get information about the disk usage on the devices. Repeat this step for each device on each node.
kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device info <device-ID>
Following is an example command and output:
kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device info c7e93cd3d3d293a78dfb99cb2809f699 Device Id: c7e93cd3d3d293a78dfb99cb2809f699 Name: /dev/disk/by-path/virtio-pci-0000_00_11_0 State: online Size (GiB): 699 Used (GiB): 497 Free (GiB): 202 Bricks: Id:33cb391e0113ec96ccc546a4e4288018 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick Id:41a508db035b042b1d35839f0d40f0c5 Size (GiB):20 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick
-
Synchronize the device to reflect the actual disk usage. Repeat this step for each device on each node.
kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device resync <device-ID>
Following is an example command and output:
kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device resync c7e93cd3d3d293a78dfb99cb2809f699 Device updated
-
Verify that the disk usage information is synchronized.
kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device info <device-ID>
Following is an example command and output:
kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device info c7e93cd3d3d293a78dfb99cb2809f699 Device Id: c7e93cd3d3d293a78dfb99cb2809f699 Name: /dev/disk/by-path/virtio-pci-0000_00_11_0 State: online Size (GiB): 699 Used (GiB): 120 Free (GiB): 579 Bricks: Id:33cb391e0113ec96ccc546a4e4288018 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick Id:41a508db035b042b1d35839f0d40f0c5 Size (GiB):20 Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick