IBM Support

Mongodbdir-icp-mongodb PVC - Low disk space

Question & Answer


Question

You might notice that PVC used by icp-mongodb-1 is not running properly. Mongodbdir-icp-mongodb PersistentVolumeClaim) pod is full and there is low disk space:

PVC Capacity: 20GiB
Used: 19.55 GiB

As an example, you have 3 pods of icp-mongodb-(increment):
icp-mongodb-0
icp-mongodb-1
icp-mongodb-2
You wonder how to overcome this issue:
  • Should you do backup/restore for all of each?
  • Should you decrease the pod count to 1 before backup?

Answer

To overcome this issue, you can apply the following steps:
  1. Change default storage class
    a. Remove annotation from old storage class - > oc annotate sc <sc_name> storageclass.kubernetes.io/is-default-class-
    b. Add annotation to new storage class -> oc annotate sc <new_sc_name> storageclass.kubernetes.io/is-default-class=true
  2. Change custom resource of MongoDB
    a. Get yaml file and trim -> kubectl get mongodb ibm-mongodb -o yaml, trim after this line.
     apiVersion: operator.ibm.com/v1alpha1
kind: MongoDB
metadata:
  labels:
    operator.ibm.com/opreq-control: "true"
  name: ibm-mongodb
  namespace: ibm-common-services
spec:
  replicas: 3
  resources:
    limits:
      cpu: 1000m
      memory: 640Mi
    requests:
      cpu: 500m
      memory: 640Mi

b. Delete ibm-mongodb CR and recreate it using above yaml.
  1. Delete PVCs and restart pods
    a. Check if icp-mongodb-0 is primary (check pod is primary, rs.status() should have icp-mongodb-0 as primary else delete the pod which is primary and recheck. )
    b. Delete PVC icp-mongodb-2 PVC, it will be in stuck status. This will get deleted once we delete mongo pods.
    c. Delete icp-mongodb-2 pod.
    d. Repeat the step for icp-mongodb-1 and then for icp-mongodb-0

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"ARM Category":[{"code":"a8m0z0000001irmAAA","label":"Use-\u003EADW App Usage-\u003EMongo DB"}],"ARM Case Number":"TS008623736","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Product Synonym

Cloud Pak for Business Automation.ICP, IBM Cloud Private, Mongodb

Document Information

Modified date:
07 April 2022

UID

ibm16570307

Page Feedback