IBM Support

Error creating an s3-compatible NooBaa backingstore despite custom CA being defined.

Troubleshooting


Problem

Error creating an s3-compatible NooBaa backingstore despite custom CA being defined.

Symptom

  • The cluster CA bundle has been successfully updated to include the appropriate CAs as per the Red Hat OpenShift documentation.

  • As a cluster-admin, when creating an s3-compatible NooBaa backingstore using the command:

  • noobaa backingstore create s3-compatible example-s3-endpoint --endpoint=https://s3.example.com --secret-name=example-s3-secret --target-bucket=example
  • The following occurs:
    • An initial status of TemporaryError got invalid endpoint while the command is in the Creating Phase. With the following message in the noobaa operator logs:
    • INFO[0300] ⏳ BackingStore "backing-store-name" Phase is "Creating": TemporaryError got invalid endpoint. requeue again
    • Followed by a status of INVALID_ENDPOINT when it ultimately fails after 5 minutes. With the following message in the noobaa operator logs:
    • ERRO[0303] ❌ BackingStore "backing-store-name" Phase is "Rejected": INVALID_ENDPOINT BackingStore "backing-store-name" invalid external connection "INVALID_ENDPOINT"

Cause

NooBaa requires the full CA chain in order to validate the certificate served by the endpoint which it is connecting to.

Environment

  • IBM Storage Fusion Data Foundation 4.13
  • Red Hat OpenShift Data Foundation 4.13
  • Red Hat OpenShift Container Platform 4.13
  • Connecting NooBaa to an external S3 compatible provider

Diagnosing The Problem

With the OpenShift client installed, and logged in to the cluster in question as a user with cluster-admin access, follow the steps below:

  1. Check the logs of the noobaa-core-0 pod in the openshift-storage project: 
    • $ oc logs noobaa-core-0 -n openshift-storage
  2. Inspect the log output, looking for messages including "unable to get issuer certificate" as seen in the below extract: 
    • 2024-01-08T16:36:07.189620334Z Jan-8 16:36:07.189 [WebServer/37]  [WARN] core.server.system_services.account_server:: got error on listBuckets with params { name: '<name-of-endpoint-being-created>', endpoint_type: 'S3_COMPATIBLE', endpoint: '<endpoint-fqdn>', identity: <xyz-123>, auth_method: 'AWS_V4' }  error: NetworkingError: unable to get issuer certificate, code: NetworkingError, message: unable to get issuer certificate
2024-01-08T16:36:07.190433358Z (node:37) UnhandledPromiseRejectionWarning: Error: unable to get issuer certificate
2024-01-08T16:36:07.190433358Z     at TLSSocket.onConnectSecure (node:_tls_wrap:1600:34)
2024-01-08T16:36:07.190433358Z     at TLSSocket.emit (node:events:517:28)
2024-01-08T16:36:07.190433358Z     at TLSSocket.emit (node:domain:489:12)
2024-01-08T16:36:07.190433358Z     at TLSSocket._finishInit (node:_tls_wrap:1017:8)
2024-01-08T16:36:07.190433358Z     at ssl.onhandshakedone (node:_tls_wrap:803:12)
    • Note: If in this step, instead of an unable to get issuer certificate error message, you find an error message containing unable to verify the first certificate, see this solution since you likely have not correctly updated the CA bundle.
  3. Using the CA bundle and the certificate served by the S3 endpoint, use openssl to confirm that using the validation does not complete successfully: 
    • $ openssl verify -CAfile ca-bundle.crt s3.crt 
DC = com, DC = example, CN = My Intermediate CA
error 2 at 1 depth lookup: unable to get issuer certificate
error s3.crt: verification failed
  4. Re-run the previous command, forcing acceptance of a partial chain using the -partial_chain option. Confirming that when forcing acceptance of the partial chain that the validation completes successfully: 
    • $ openssl verify -partial_chain -CAfile ca-bundle.crt s3.crt 
s3.crt: OK

Resolving The Problem

The trusted CA bundle needs to be updated to include the full chain of the CA that signed the certificate served by the s3 endpoint. In order to update the CA bundle follow the steps in the Red Hat OpenShift documentation.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB66","label":"Technology Lifecycle Services"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SSSEWFV","label":"Storage Fusion Data Foundation"},"ARM Category":[{"code":"a8m3p000000UoIPAA0","label":"Support Reference Guide"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
21 April 2025

UID

ibm17229623

Page Feedback