IBM Support

IBM Cloud Databases for PostgreSQL was affected by a security vulnerability

Security Bulletin


Summary

IBM was informed by Wiz, a cloud security vendor, under IBM Product Security Incident Response Team (PSIRT), of an issue with IBM Cloud Databases (ICD) for PostgreSQL. Wiz disclosed they could acquire unauthorized access to ICD repositories that store software dependencies for PostgreSQL container images. By polluting ICD internal trusted repositories, a malicious user could potentially force customer PostgreSQL instances into running unauthorized code. IBM Cloud quickly mitigated the reported issue. IBM Security Operations Center identified and monitored Wiz's activities at the time, and there is no evidence to suggest IBM Cloud systems or services were exploited further or by other parties. Analysis of logs was also conducted and has determined the only activity associated with this issue was by the Wiz researchers.

Vulnerability Details

Description: IBM Cloud Databases for PostgreSQL was susceptible to a supply-chain vulnerability.
CVSS Base Score: 8.8
CVSS Temporal Score: For more information see Common Vulnerability Scoring System Calculator
CVSS Vector: (CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H)

Affected Products and Versions

All IBM Cloud Databases for PostgreSQL instances were potentially impacted by this security vulnerability, however no action is required by customers.

Remediation/Fixes

No action is required by customers to remediate this security vulnerability. IBM Cloud Databases for PostgreSQL automatically applied the fixes to your instances. There is no need to update your cloud database service for this vulnerability.
IBM's Response
IBM took the following steps after the IBM Security Operations Center detected reverse shell activity originating from a customer’s IBM Cloud Databases for PostgreSQL instance:
  1. IBM Cloud responded by isolating the customer database, disabling the cloud account and reaching out to the account owner.
    • The Wiz researchers then brought the initial Remote Code Execution (RCE) to our attention, and we invited them to continue their research.
  2. We took the following approach to address the reported RCE vulnerabilities in IBM Cloud Databases (ICD) for PostgreSQL:
    • Patched the reported SQL Injection vulnerability in ICD’s Logical Replication functions
    • Blocked superuser operations that directly manipulate the filesystem (copy to/from) or server settings
    • Enhanced process level access control policies to further limit what a process can do, compartmentalizing different processes and sub-processes of PostgreSQL
  3. We deployed these changes to new instances on August 22, 2022.
    • After the patch rollout, our security teams and Wiz validated the fixes.
  4. Wiz reported the following additional vulnerabilities in existing instances on August 24, 2022:
    • An over-privileged Kubernetes Service Account token mounted to customer database containers
    • Access to read-only imagePullSecrets for ICD’s private container registry
    • Access to secrets stored in ICD’s PostgreSQL image manifest, providing read-write access to trusted internal repositories
  5. On August 26, 2022, Wiz confirmed the mitigations IBM had deployed were effective. These measures included:
    • Removed the over-privileged Kubernetes Service Account token from all customer database containers
    • Removed secrets from image manifest
    • Revoked and rotated secrets
    • Scoped repository credentials to adhere to the principle of least privilege
    • Validated the integrity of internal repositories and checked logs for any unauthorized access to repositories
  6. We finished updating the entire fleet of existing instances and addressed any remaining issues on September 03, 2022.
  7. After this final patch rollout, our security teams and Wiz validated the fixes.

Technical Details

The following were the steps used to gain remote code execution and access to IBM Cloud Databases for PostgreSQL internal trusted repositories:

  1. SQL Injection in IBM Cloud Databases for PostgreSQL Logical Replication implementation allowed privilege escalation to superuser “ibm”.
    • After acquiring superuser privileges, Wiz leveraged the PostgreSQL COPY statement to execute arbitrary commands on their underlying database container to obtain a reverse shell.
  2. Due to an over-privileged Kubernetes Service Account token mounted on customer database containers, it was possible for Wiz to authenticate to the Kubernetes API server and request pod specifications for pods within their own customer namespace.
  3. Inside the pod specifications Wiz identified a reference to imagePullSecrets which they were able to obtain from the API Server.
    • The imagePullSecrets are read-only secrets to ICD’s private container registry storing images related to ICD for PostgreSQL.
    • Using the imagePullSecrets, Wiz identified additional credentials in image manifests.
  4. The additional set of credentials are leveraged during the build process and have read-write access to internal trusted repositories, storing software dependencies in which customer ICD for PostgreSQL container images utilize during their runtime.  
    • Wiz provided proof of read-write access to internal repositories by creating and deleting dummy files.

Wiz has posted a blog about this issue available here. We would like to thank Wiz, who found these issues and worked closely with IBM to help secure our customers.

Monitor IBM Cloud Status for Future Security Bulletins

Monitor the security notifications on the IBM Cloud Status page to be advised of future security bulletins.

References

Off

Acknowledgement

We would like to thank Wiz, who found these issues and worked closely with IBM to help secure our customers. For more information, please see their blog post https://www.wiz.io/blog/hells-keychain-supply-chain-attack-in-ibm-cloud-databases-for-postgresql.

*The CVSS Environment Score is customer environment specific and will ultimately impact the Overall CVSS Score. Customers can evaluate the impact of this vulnerability in their environments by accessing the links in the Reference section of this Security Bulletin.

Disclaimer

Review the IBM security bulletin disclaimer and definitions regarding your responsibilities for assessing potential impact of security vulnerabilities to your environment.

[{"Type":"MASTER","Line of Business":{"code":"","label":""},"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSCL36","label":"Databases for PostgreSQL"},"ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
01 December 2022

UID

ibm16842111