AWS S3, MinIO, and IBM Cloud Object Storage connectors

Deployment options: Netezza Performance Server for Cloud Pak for Data System Netezza Performance Server for Cloud Pak for Data

Starting from version 11.0.4.1, you can use a cloud connector (S3) to back up and restore database backups on cloud storage.

Starting from version 11.2.0.0, S3 Glacier and S3 Glacier Deep Archive are supported.

Starting from version 11.2.2.0, MinIO Object Storage is supported.

Starting from version 11.2.2.0, S3 Compatible Object Storage is supported.

Netezza Performance Server for Cloud Pak for Data System supports backup and restore by using cloud storage. The S3 connector provides support for:
  • AWS S3 (Amazon Web Service - Simple Storage Service)
  • IBM Cloud Object Storage
  • MinIO Object Storage
  • S3 Compatible Object Storage
Note: Single connector works with both storages with different arguments that are specified for nzbackup and nzrestore.

The S3 connector does not need any client software installation.

Arguments

The nzbackup and nzrestore commands provide cloud storage support through following command-line arguments:
-connector
  • For the AWS S3 cloud storage, use the value s3 or aws with this argument. For example, -connector s3.
  • For IBM Cloud Object Storage, use the value cos or ibmcos with this argument. For example, -connector cos.
  • For MinIO, use the value minio with this argument. For example, -connector minio.
-connectorArgs
For connecting to cloud storage, provide the following parameters in <key>=<value> pairs, which are separated by :.
Argument Required Description
UNIQUE_ID Mandatory This is a namespace that is used by customers to group data in the cloud bucket.
ACCESS_KEY_ID Mandatory Key that is generated on AWS or MinIO or IBM Cloud Object Storage.
SECRET_ACCESS_KEY Mandatory Secret access key that is generated on AWS or MinIO or IBM Cloud Object Storage.
DEFAULT_REGION Mandatory Region of the bucket.
BUCKET_URL Mandatory Name of the bucket.
MULTIPART_SIZE_MB Optional Default 105. This value is in MB. It indicates the size of each part in a multipart upload. You can upload a maximum of 105 MB of buffer in one request.
ENDPOINT
  • Optional for AWS.
  • Mandatory for MinIO.
  • Mandatory for IBM Cloud Object Storage.
 This is a well-known region URL to access your bucket. It is optional for AWS and mandatory for MinIO and IBM Cloud Object Storage.
STORAGE_CLASS Optional Storage class for backup valid values: STANDARD, GLACIER, DEEP_ARCHIVE Default set to STANDARD Applicable in Backup.
TIER Optional Restoration speed from GLACIER/DEEP_ARCHIVE to S3 Valid Values: Standard, Bulk, Expedited Default set to Standard Applicable in Restore https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html.
RETAIN_DAYS Optional

Retention Period (in number of days) for data restored from Glacier to S3 Default set to 0. If STORAGE_CLASS is other than STANDARD, this must be set to a nonzero value.

Applicable in Restore.

MULTIPART_SIZE_MB

AWS provides a facility to upload objects in multiple parts. The nzbackup command uses this functions to split and upload data in parts of size that is specified by the MULTIPART_SIZE_MB parameter. The field can be configured because it can affect cost and performance. For more, see https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpuploadpricing.

AWS mandates a limit of 10000 parts per object. That means a backup file can be uploaded in maximum 10000 parts. The default registry setting (host.bnrFileSizeLimitGB= 1024) in NPS limits backup file size at 1 TB. Hence the default value of MULTIPART_SIZE_MB is set at 105 so that 1 TB file can be uploaded in 10000 parts. If you changed bnrFileSizeLimitGBsetting in registry, you must set MULTIPART_SIZE_MB to a higher value.