Replicating analytics data to a 2DCDR warm-standby data center

If you have a two data center warm-standby deployment, you can configure the analytics subsystem in your active data center to offload incoming API event records to the analytics subsystem in your warm-standby data center.

1. In DC1, identify the analytics ingestion client secret, by running this command in the management subsystem namespace:

   kubectl -n <management namespace> get secrets | grep "\-client"

   The secret name is either analytics-ingestion-client or a7s-ing-client, possibly prefixed with other text.

2. In DC1, view the contents of the analytics ingestion client secret:

   kubectl -n <management namespace> get secret -o yaml <analytics ingestion secret>

   The output should look like this:

   apiVersion: v1
   data:
     ca.crt: <ca.crt encoded certificate string>
     tls.crt: <tls.crt ncoded certificate string>
     tls.key: <tls.key encoded certificate string>
   ...

3. In DC1, convert the tls.key value to PKCS #8 format.
   1. Base64 decode tls.key, and copy to a file called tls.key:

      echo <tls.key encoded certificate string> | base64 --decode > tls.key

   2. Convert the tls.key file contents to PKCS #8, and Base64 encode it to a file called converted_tls.key:

      openssl pkcs8 -topk8 -in tls.key -nocrypt | base64 -w 0 > converted_tls.key

4. In DC2, create or update your analytics_offload_certificates.yaml file, using the DC1 certificate strings and TLS key that you obtained in steps 2 and 3.
   If you did not previously configure analytics offload targets, then create a new file that is called analytics_offload_certificates.yaml and paste in the following contents:

   apiVersion: v1
   kind: Secret
   metadata:
     name: offload-certificates
   data:
     replication_ca.crt: <ca.crt string from ingestion secret>
     replication_tls.crt: <tls.crt string from ingestion secret>
     replication_tls.key: <converted_tls.key file contents>

   If you already have other offload targets configured, then edit your existing analytics_offload_certificates.yaml file and add the certificate strings from the previous step under the data section:

   apiVersion: v1
   kind: Secret
   metadata:
     name: offload-certificates
   data:
     replication_ca.crt: <ca.crt string from ingestion secret>
     replication_tls.crt: <tls.crt string from ingestion secret>
     replication_tls.key: <converted_tls.key file contents>
     existing_offload_target1_ca.crt: hGywkdWE...
     existing_offload_target1_tls.crt: kY7fW..
     ...
   ...

5. In DC2, apply the analytics_offload_certificates.yaml file to create the offload-certificates secret:

   kubectl -n <analytics namespace> apply -f analytics_offload_certificates.yaml

   A new secret that is called offload-certificates is created. Verify that this secret exists by running:

   kubectl -n <analytics namespace> get secrets

   If you previously created offload targets, then this secret is updated.

6. In DC2, identify the analytics ingestion client secret, by running this command in the management subsystem namespace:

   kubectl -n <management namespace> get secrets | grep "\-client"

   The secret name is either analytics-ingestion-client or a7s-ing-client, possibly prefixed with other text.

7. In DC2, view the contents of the analytics ingestion client secret:

   kubectl -n <management namespace> get secret -o yaml <analytics ingestion secret>

   The output should look like this:

   apiVersion: v1
   data:
     ca.crt: <ca.crt encoded certificate string>
     tls.crt: <tls.crt ncoded certificate string>
     tls.key: <tls.key encoded certificate string>
   ...

8. In DC2, convert the tls.key value to PKCS #8 format.
   1. Base64 decode tls.key, and copy to a file called tls.key:

      echo <tls.key encoded certificate string> | base64 --decode > tls.key

   2. Convert the tls.key file contents to PKCS #8, and Base64 encode it to a file called converted_tls.key:

      openssl pkcs8 -topk8 -in tls.key -nocrypt | base64 -w 0 > converted_tls.key

9. In DC1, create or update your analytics_offload_certificates.yaml file, by using the DC2 certificate strings and TLS key that you obtained in steps 7 and 8.
   If you did not previously configure analytics offload targets, then create a new file that is called analytics_offload_certificates.yaml and paste in the following contents:

   apiVersion: v1
   kind: Secret
   metadata:
     name: offload-certificates
   data:
     replication_ca.crt: <ca.crt string from ingestion secret>
     replication_tls.crt: <tls.crt string from ingestion secret>
     replication_tls.key: <converted_tls.key file contents>

   If you already have other offload targets configured, then edit your existing analytics_offload_certificates.yaml file and add the certificate strings from the previous step under the data section:

   apiVersion: v1
   kind: Secret
   metadata:
     name: offload-certificates
   data:
     replication_ca.crt: <ca.crt string from ingestion secret>
     replication_tls.crt: <tls.crt string from ingestion secret>
     replication_tls.key: <converted_tls.key file contents>
     existing_offload_target1_ca.crt: hGywkdWE...
     existing_offload_target1_tls.crt: kY7fW..
     ...
   ...

10. In DC1, apply the analytics_offload_certificates.yaml file to create the offload-certificates secret:

    kubectl -n <analytics namespace> apply -f analytics_offload_certificates.yaml

    A new secret that is called offload-certificates is created. Verify that this secret exists by running:

    kubectl -n <analytics namespace> get secrets

    If you previously created offload targets, then this secret is updated.

11. In DC1, edit the analytics CR, and set the spec.external.offload section as shown:

    kubectl edit a7s

    external:
        offload:
          enabled: true
          output: |
            if  [gateway_service_name] == "<DC1 gateway service name>" {
              http {
                url => "https://<DC2 analytics ingestion endpoint>/ingestion"
                http_method => "post"
                codec => "json"
                content_type => "application/json"
                id => "offload1_http"
                ssl_certificate_authorities => "/etc/velox/external_certs/offload/replication_ca.crt"
                ssl_certificate => "/etc/velox/external_certs/offload/replication_tls.crt"
                ssl_key => "/etc/velox/external_certs/offload/replication_tls.key"
              }
            }
          secretName: offload-certificates  

    Where,

    • <DC1 gateway service name> is the name of the gateway service in DC1.
    • <DC2 analytics ingestion endpoint> is the analytics ingestion endpoint of the analytics service in DC2.

    If you have multiple offload targets, then see Multiple offload targets.

    After you save the updated the CR, the analytics ingestion pod restarts, and new API events are sent to DC2.

    Note: If you later return to edit properties in the external.offload.output section, you might need to restart the ingestion pod manually:

    kubectl delete pod <analytics ingestion pod>

12. In DC2, edit the analytics CR, and set the spec.external.offload section as shown:

    kubectl edit a7s

    external:
        offload:
          enabled: true
          output: |
            if  [gateway_service_name] == "<DC2 gateway service name>" {
              http {
                url => "https://<DC1 analytics ingestion endpoint>/ingestion"
                http_method => "post"
                codec => "json"
                content_type => "application/json"
                id => "offload1_http"
                ssl_certificate_authorities => "/etc/velox/external_certs/offload/replication_ca.crt"
                ssl_certificate => "/etc/velox/external_certs/offload/replication_tls.crt"
                ssl_key => "/etc/velox/external_certs/offload/replication_tls.key"
              }
            }
          secretName: offload-certificates  

    Where,

    • <DC2 gateway service name> is the name of the gateway service in DC2.
    • <DC1 analytics ingestion endpoint> is the analytics ingestion endpoint of the analytics service in DC1.

    After saving the updated the CR, the analytics ingestion pod restarts, and new API events are sent to DC1.

13. Confirm that replication is working by making a test API call to your DC2 gateway service and confirming that you see the event in the Discover view of the API Manager UI in DC1.

    To verify replication from DC1 to DC2, you must complete a failover of the management subsystem so that you can access the API Manager UI in DC2 and see data from the analytics subsystem in DC2.

kubectl logs <analytics ingestion pod>