Setting up Elasticsearch storage

If you plan to use embedded Elasticsearch and Kibana (rather than an external Elasticsearch installation), and only in this case, you must create a persistent volume for each Elasticsearch data and master pods. Optionally, also create a persistent volume for Elasticsearch data snapshots.

Before you begin

Preferably enable NFS share on the IBM® Cloud Private master node to ensure fast read/write access.

This procedure assumes that the NFS shared folder is /export/NFS.

About this task

If no dynamic provisioning is set up, you must create the appropriate persistent volumes. Go through the subsequent steps to create persistent volumes that persist data into NFS folders that are shared by an NFS server.

Moreover, if you want to enable the capability to take snapshots of Elasticsearch data, you must also create an extra persistent volume that is shared by all Elasticsearch data and master pods. For more information about snapshots, see Taking snapshots of Elasticsearch data.

Tip: Use dynamic provisioning to create storage volumes on-demand. For more information, see Creating a storage class. A storage class must have administrator privileges.

For 18.0.0 and For 18.0.1 : If you use embedded Elasticsearch and Kibana, two Elasticsearch data pods are created by default. The following steps are based on the default.

New in 18.0.2 : If you use embedded Elasticsearch and Kibana, one Elasticsearch data pod and one Elasticsearch master pod are created by default. The following steps are based on this default number.

Procedure

Create the NFS shared folder for each persistent volume. The code samples use /export/NFS as the NFS shared folder.
The Elasticsearch user and group under ID 1000 must have read and write permissions to the folders.
The following lines set up the pv1 and pv2 folders on the NFS server.
```
mkdir /export/NFS/ibm-bai-ek-pv1
mkdir /export/NFS/ibm-bai-ek-pv2
chown -R 1000:1000 /export/NFS/ibm-bai-ek-pv1
chmod 770 /export/NFS/ibm-bai-ek-pv1
chown -R 1000:1000 /export/NFS/ibm-bai-ek-pv2
chmod 770 /export/NFS/ibm-bai-ek-pv2
```

Optional: To support taking snapshots of Elasticsearch data, enter the following lines.

mkdir /export/NFS/ibm-bai-ek-snapshots
chown -R 1000:1000 /export/NFS/ibm-bai-ek-snapshots
chmod 770 /export/NFS/ibm-bai-ek-snapshots

To create the persistent volumes, save theses lines to a YAML file and run the apply command.

kubectl apply -f myfile.yaml

It is safer to apply the Retain reclaim policy to make sure that data is kept on release.

This code sample creates persistent volumes.

apiVersion: v1
kind: PersistentVolume
metadata:
   name: ibm-bai-ek-pv-1
spec:
   accessModes:
   - ReadWriteOnce
   capacity:
      storage: 10Gi
   nfs:
      path: /export/NFS/ibm-bai-ek-pv1
      server: <server-ip>
   persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: ibm-bai-ek-pv-2
spec:   
   accessModes:
   - ReadWriteOnce
   capacity:
      storage: 10Gi
   nfs:
      path: /export/NFS/ibm-bai-ek-pv2
      server: <server-ip>
   persistentVolumeReclaimPolicy: Retain

Optional: For data snapshots, create a persistent volume as follows.

apiVersion: v1
kind: PersistentVolume
metadata:
   name: ibm-bai-ek-snapshots
spec:
   accessModes:
   - ReadWriteMany
   capacity:
      storage: 30Gi
   nfs:
      path: /export/NFS/ibm-bai-ek-snapshots
      server: <server-ip>
   persistentVolumeReclaimPolicy: Retain

Optional: If you want to refine how the persistent volumes are bound, provide a storageClassName value to the .yaml file of the persistent volume. Reference the storage class name when you configure your IBM Business Automation Insights installation.
For more information, see the Class section of the Kubernetes documentation.

For data snapshot storage, by default the Helm chart creates the persistent volume claim (PVC) that binds the persistent volume that you created. However, you can instead create a PVC that binds the persistent volume and reference it as the ibm-dba-ek.elasticsearch.data.snapshotStorage.existingClaimName value.