IBM Support

Unable to create an s3-compatible NooBaa MCG backingstore

Troubleshooting


Problem

Unable to create an s3-compatible NooBaa MCG backingstore

Symptom

As a cluster-admin, when creating an s3 compatible NooBaa backingstore using the command:
noobaa backingstore create s3-compatible example-s3-endpoint --endpoint=https://s3.example.com --secret-name=example-s3-secret --target-bucket=example
An initial status of TemporaryError got invalid endpoint while the command is in the Creating Phase is followed by INVALID_ENDPOINT when it ultimately fails after 5 minutes.

With the following entries in the NooBaa operator logs:
INFO[0300] ⏳ BackingStore "backing-store-name" Phase is "Creating": TemporaryError got invalid endpoint. requeue again
ERRO[0303] ❌ BackingStore "backing-store-name" Phase is "Rejected": INVALID_ENDPOINT BackingStore "backing-store-name" invalid external connection "INVALID_ENDPOINT"

Cause

The noobaa-operator creates a ConfigMap named noobaa-inject-ca which includes the config.openshift.io/inject-trusted-cabundle: 'true' label, this label prompts OpenShift to inject the CA trust bundle. When no custom CA has been added, this bundle only includes the default CAs contained in the Red Hat CoreOS trusted CA bundle.

Environment

  • IBM Storage Fusion Data Foundation 4.13
  • Red Hat OpenShift Data Foundation 4.13

Diagnosing The Problem

With the OpenShift client installed, and logged in to the cluster in question as a user with cluster-admin access, follow the steps below:
  1. Check the logs of the noobaa-core-0 pod in the openshift-storage project: 
    • $ oc logs noobaa-core-0 -n openshift-storage
  2. Inspect the log output, looking for messages similar to the extract below: 
    • [WebServer/38]  [WARN] core.server.system_services.account_server:: got error on listBuckets with params { name: '<name-of-endpoint-being-created>', endpoint_type: 'S3_COMPATIBLE', endpoint: '<endpoint-fqdn>', identity: <xyz-123>, auth_method: 'AWS_V4' }  error: NetworkingError: unable to verify the first certificate, code: NetworkingError, message: unable to verify the first certificate
(node:38) UnhandledPromiseRejectionWarning: Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1600:34)
    at TLSSocket.emit (node:events:517:28)
    at TLSSocket.emit (node:domain:489:12)
    at TLSSocket._finishInit (node:_tls_wrap:1017:8)
    at ssl.onhandshakedone (node:_tls_wrap:803:12)
As seen in the above extract, unable to verify the first certificate portion of the message indicates a CA trust issue. Updating the CA bundle should resolve this.

Note: NooBaa requires the full CA chain for this to work correctly. Should you instead find an error message containing unable to get issuer certificate see this solution.

Resolving The Problem

NooBaa injects the CA trust bundle using the OpenShift CA injection process. The cluster CA bundle needs to be updated to include the CA that signed the certificate used by the s3 endpoint. In order to update the CA bundle follow the steps in the Red Hat OpenShift documentation. After updating the CA bundle, the noobaa-core-0 pod in the openshift-storage project must be restarted.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB66","label":"Technology Lifecycle Services"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SSSEWFV","label":"Storage Fusion Data Foundation"},"ARM Category":[{"code":"a8m3p000000UoIPAA0","label":"Support Reference Guide"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
30 May 2025

UID

ibm17229773

Page Feedback