As a major release, Big Replicate 2.1.0 introduces incompatibilities with the network protocols and storage formats used by prior versions. Please contact IBM support for information on the upgrade mechanism appropriate for your environment.

Big Replicate 2.1.0 adds support for replicating data efficiently from Network File Systems (NFS) for NetApp devices to any mix of on-premises and cloud environments. This feature allows data replication at any scale from NFS to other Big Replicate zones.

The Big Replicate user interface now presents a logical view of the Big Replicate operational components, Big Replicate zones and bandwidth limit policies in place of the physical map of locations. This makes it easier to observe the deployment of complex solutions and navigate directly to detailed views of individual item configuration.

An improvement has been made to the mechanism used by the HDFS and HCFS client library to detect when a working Big Replicate server is unavailable. This allows clients to bypass the Big Replicate server when needed without waiting for a TCP connection loss or timeout.

Big Replicate 2.1.0 can be configured to replicate from storage system locations that do not provide write access for the identity used by the Big Replicate server.

Big Replicate configuration options now include custom S3 endpoints so that replication can occur to non-AWS S3 providers. Additionally, when Big Replicate is hosted in AWS EC2, replication can occur to an S3 endpoint that is in a region other than where the Big Replicate services reside.

The Big Replicate repair feature allows the transfer of initial content between Big Replicate zones that have not previously replicated, and can be used as a mechanism to perform once-off replication that remains consistent with other replication activity. Repair has been enhanced significantly in Big Replicate 2.10, including the following:

The IBM Big Replicate user interface can be accessed over HTTPS, and for that configuration to be performed independently of other SSL configuration.

You can choose to install Big Replicate 2.1.0 in a location other than the default /opt/wandisco.

Big Replicate 2.1.0 can operate in an environment where one Big Replicate zone does not allow inbound network connectivity. This is typical for a secured on-premises deployment, where it may be difficult to modify or establish corporate firewall rules to allow inbound TCP connections to the Big Replicate services.

ACL replication can be enabled to allow changes from local- and remote-originated zones to be replicated. ACL information will be represented in consistency check results as appropriate.

Among a range of minor improvements to logged information, Big Replicate 2.1.0 adds the ability to log the identity of the proxy user for which requests are made.

This feature introduces a mechanism to quickly prevent applications from using Big Replicate when interacting with the underlying file system, without the need to make configuration changes. The fusion.replicated.dir.exchange configuration property in core-site.xml specifies the location under which a directory named bypass can be created to trigger this. Subsequent client activity in that cluster will bypass coordination through Big Replicate.

The API to track the status of transfers under a replicated directory now allows that tracking to be limited to a sub-directory of a replicated location.

Big Replicate 2.1.0 can be installed as a non-root user with sufficient permissions ( sudo tar, sudo ambari-server, sudo cp).

The Big Replicate 2.1.0 client library for HDFS and HCFS compatibility ensures that classpath conflicts do not occur with any client application, allowing Big Replicate to be used by applications that use alternative versions of the Guava and Netty libraries.

Periods of extended network outage between Big Replicate zones can be accommodated by limits that allow Big Replicate servers to identify a sidelined node, ensuring that operation of other nodes can continue in its absence. Prior to this release, bringing a sidelined node back into operation was a completely manual process. Big Replicate 2.1.0 adds a mechanism by which sidelined nodes can be recovered and participate in ongoing activity.

To support operation in environments where minimal security privileges must be allocated, the Big Replicate server can now operate as a principal without HDFS superuser privileges.

A configuration option ( fusion.client.coordinate.read) is provided to allow coordination of open() requests, which by default is false.

This release provides an API by which a preferred writer node can be specified for a given replicated path. The writer node is the Big Replicate server instance responsible for executing modifications to the local zone’s metadata via the file system API.

License expiration allows continued operation for a short grace period (by default one month for production licenses), during which notifications are presented to the administrator about the license expiry. This is in addition to the existing warnings provided prior to license expiration.

Additionally, license expiry does not halt operation of the Big Replicate server, which remains available to service activities that occur in non-replicated locations.

Big Replicate has added support for the following new platforms since Big Replicate 2.9:

IBM certifies the interoperability of Big Replicate with a wide variety of systems, including Hadoop distributions, object storage platforms, cloud environments, and applications.

For information on the data source and data target compatibility supported on Big Replicate, see Big Replicate data compatibility.

IBM Big Replicate is architected for maximum compatibility and interoperability with applications that use standard Hadoop File System APIs. All applications that use the standard Hadoop Distributed File System API or any Hadoop-Compatible File System API should be interoperable with IBM Big Replicate, and will be treated as supported applications. Additionally, Big Replicate supports the replication of content with Amazon S3 and S3-compatible objects stores, locally-mounted file systems, and NetApp NFS devices, but does not require or provide application compatibility libraries for these storage services.

When your license limits are exceeded, Big Replicate will operate in a limited manner, but allows you to apply a new license to bring the system back to full operation. Once a license is no longer valid:

Each different type of license has different limits.

How much Big Replicate you need to deploy is not proportionate to the amount of data stored in your clusters, or the number of nodes in your clusters. You deploy Big Replicate/IHC server nodes in proportion to the data traffic between clusters; the more data traffic you need to handle, the more resources you need to put into the Big Replicate server software.

If you plan to locate both the Big Replicate and IHC servers on the same machine then check the Collocated Server requirements:

Big Replicate Client port: 8023
Port used by Big Replicate server to communicate with HCFS/HDFS clients. The port is generally only open to the local Big Replicate server, however you must make sure that it is open to edge nodes.

Big Replicate Server listening port: 8024
Port used by Big Replicate server to listen for connections from remote IHC servers. It is only used in unidirectional mode, but it’s always opened for listening. Remote IHCs connect to this port if the connection can’t be made in the other direction because of a firewall. The SSL configuration for this port is controlled by the same ihc.ssl.enabled property that is used for IHC connections performed from the other side. See Enable SSL for Big Replicate

IHC ports: 7000-range or 9000-range
7000 range, (exact port is determined at installation time based on what ports are available), used for data transfer between Big Replicate Server and IHC servers. Must be accessible from all Big Replicate nodes in the replicated system.
9000 range, exact port is determined at installation time based on available ports), used for an HTTP Server that exposes JMX metrics from the IHC server.

Comment out requiretty in /etc/sudoers
The installer’s use of sudo won’t work with some linux distributions (CentOS where /etc/sudoer sets enables requiretty, where sudo can only be invoked from a logged in terminal session, not through cron or a bash script. When enabled the installer will fail with an error:

keystore
A keystore (containing a private key / certificate chain) is used by an SSL server to encrypt the communication and create digital signatures.

truststore
A truststore is used by an SSL client for validating certificates sent by other servers. It simply contains certificates that are considered "trusted". For convenience you can use the same file as both the keystore and the truststore, you can also use the same file for multiple processes.

Workaround
Upgrade to Java 8u60 or greater, or ensure Big Replicate is able to make use of OpenSSL libraries instead of JDK. Requirements for this can be found at http://netty.io/wiki/requirements-for-4.x.html FUS-3041

This table shows the versions of Hadoop and Java that we currently support:

Supported Big Data applications my be noted here, as we complete testing:

We’ll now look at what you should know and do as you begin the installation.

If you are running Kerberos on your cluster you should consider the following requirements:

Before running the installer on a platform that is secured by Kerberos, you’ll need to run through the following steps: Setting up Kerberos.

For example, if you connect a Zone that runs CDH, which has superuser 'hdfs" with a zone running MapR, which has superuser 'mapr', you would need to create the user 'hdfs' on the MapR zone and 'mapr' on the CDH zone.

Before you start the installation you must ensure that there are no existing Big Replicate installations or Big Replicate components installed on your elected machines. If you are about to upgrade to a new version of Big Replicate you must first make sure that you run through the removal instructions provided in the Appendix - Cleanup Big Replicate.

then HADOOP_HOME must be set to

You need to match IBM’s Big Replicate installer file to each data center’s version of Hadoop. Installing the wrong version of Big Replicate will result in the IHC servers being misconfigured.

After completing an evaluation deployment, you will need to contact IBM about getting a license file for moving your deployment into production.

These tags are appended to /var/ibm/slmtags every 60 minutes, but the frequency of generation can be modified through the ibm.slm.tag.freq configuration property.

An example of the generated data:

IBM Big Replicate uses specific SubType metric entries that should be interpreted as follows:

Use the following steps to complete an installation using the installer file. This requires an administrator to enter details throughout the procedure. Once the initial settings are entered through the terminal session, the installation is then completed through a browser or alternatively, using a Silent Installation option to handle configuration Programmatically.

You are now given a summary of all the settings provided so far. If these settings are correct then enter "Y" to complete the installation of the Big Replicate server. . The package will now install

At this point the Big Replicate server and corresponding IHC server will be installed. The next step is to configure the Big Replicate UI through a browser or using the silent installation script.

Follow this section to complete the installation by configuring Big Replicate using a browser-based graphical user interface.

The silent installation process supports two levels: Unattended installation handles just the command line steps of the installation, leaving the web UI-based configuration steps in the hands of an administrator. See unattended installation.

Fully Automated also includes the steps to handle the configuration without the need for user interaction. See link:[Fully Automated installation].

Use the following command for an unattended installation where an administrator will complete the configuration steps using the browser UI.

There are a number of properties that need to be set up before the installer can be run:

This mode only automates the initial command line installation step, the configuration steps still need to be handled manually in the browser steps.

This mode is closer to a full "Silent" installation as it handles the configuration steps as well as the installation.

Properties that need to be set:

Cluster Manager Variables are deprecated
The cluster manager variables are mostly redundant as they generally get set in different processes though they currently remain in the installer code.

If this part of the installation fails it is possible to re-run the silent_installer part of the installation by running:

This procedure is useful for UI-only installations:

For every package of Big Replicate there’s both an env.sh and a .properties file. The env.sh sets environment variables that complete the initial command step of an installation. The env.sh also points to a properties file that is used to automate the browser-based portion of the installer. The properties files for the different installation types are provided below:

Be sure to point your replicated directory to your NFS mount, either directly or using a a symlink.

The following requirements come into play if you have deployed Big Replicate using with its native fusion:/// URI.

In order to store a Hive table in Big Replicate you specify a Big Replicate URI when creating a table. E.g. consider creating a table called log that will be stored in a replicated directory.

CREATE TABLE log(requestline string) stored as textfile location 'fusion:///repl1/hive/log'; Note: Replicating table storage without sharing the Hive metadata will create a logical discrepancy in the Hive catalog. For example, consider a case where a table is defined on one cluster and replicated on the HCFS to another cluster. A Hive user on the other cluster would need to define the table locally in order to make use of it.

Hive from CDH 5.3/5.4 does not work with Big Replicate, (because of HIVE-9991). To get it working with CDH 5.3 and 5.4. you need to modify the default Hive file system setting. In Cloudera Manager, add the following property to hive-site.xml:

This property should be added in 3 areas:

It’s possible to configure Hive to use Big Replicate URIs as output paths for storing data, to do this you must specify a Big Replicate URI when writing data back to the underlying Hadoop-compatible file system (HCFS). For example, consider writing data out from a table called log to a file stored in a replicated directory:

In this section we’ll describe how to configure Hive to use Big Replicate URIs as input paths for loading data.

It is not common to load data into a Hive table from a file using the Big Replicate URI. When loading data into Hive from files the core-site.xml setting fs.default.name must also be set to Big Replicate, which may not be desirable. It is much more common to load data from a local file using the LOCAL keyword:

If you do wish to use a Big Replicate URI as a load path, you must change the fs.defaultFS setting to use Big Replicate, as noted in a previous section. Then you may run:

Advanced configuration - please contact IBM before attempting
In this section we’ll describe how to share the Hive metastore between two clusters. Since IBM Big Replicate can replicate the file system that contains the Hive data storage, sharing the metadata presents a single logical view of Hive to users on both clusters.

When sharing the Hive metastore, note that Hive users on all clusters will know about all tables. If a table is not actually replicated, Hive users on other clusters will experience errors if they try to access that table.

There are two options available.

In this configuration, the Hive metastore is configured normally on one cluster. On other clusters, the metastore process points to a read-only copy of the metastore database. MySQL can be used in master-slave replication mode to provide the metastore.

In this configuration, the Hive metastore is writable on all clusters.

Standard

For Cloudera CDH: See Hive Metastore High Availability.

For Hortonworks/Ambari: High Availability for Hive Metastore.

Manual Replication

In order to manually replicate metastore data ensure that the DDLs are placed on two clusters, and perform a partitions rescan.

If you plan to use Big Replicate’s own fusion:/// URI, then you will need to use the provided parcel (see the screenshot, below for link in the Client Download section of the Settings screen):

Follow the same steps described for installing the Big Replicate client, downloading the parcel and SHA file, i.e.:

In order for Impala to load the Big Replicate Client jars, the user needs to make a small configuration change in their Impala service, through Cloudera Manager. In Cloudera Manager, the user needs to add an environment variable in the section Impala Service Environment Advanced Configuration Snippet (Safety Valve).

Presto is an open source distributed SQL query engine for running interactive analytic queries. It can query and interact with multiple data sources, and can be extended with plugins.

Presto requires the use of Java 8, and had internal dependencies on Java library versions that may conflict with those of the Hadoop distribution with which it communicates when using the “hive-hadoop2†plugin. For example, Presto makes use of guava-18.0.jar, while HDP 2.4 uses guava-11.0.2.jar.

IBM Big Replicate leverages a replacement client library when overriding the hdfs:// scheme for access to the cluster file system in order to coordinate that access among multiple clusters. This replacement library is provided in a collection of jar files in the /opt/wandisco/fusion/client/lib directory for a standard installation. These jar files need to be available to any process that accesses the file system using the com.wandisco.fusion.fs.client.FusionHdfs implementation of the Apache Hadoop FileSystem API.

Because Presto requires these classes to be available to the hive-hadoop2 plugin, they must reside in the plugins/hive-hadoop2 directory of the Presto installation. Additionally, the additional JARs made available in that directory must not provide conflicting versions of classes already used by the hive-hadoop2 plugin.

As Big Replicate uses conflicting versions of some classes (e.g. guava), the Big Replicate client library used by Presto must be repackaged to avoid such conflicts.

IBM has provided a repackaged version of the Big Replicate 2.9.3.1 client libraries to help demonstrate Big Replicate-Presto interoperability. Please take note that this is an early access release of the client library that is specifically for Presto DB deployment.

The repackaged client library consists of a single JAR file: fusion-client-bundle-2.9.3.1-hdp-2.4.0-SNAPSHOT-all.jar. This file consists of the classes otherwise made available through the JAR files in the /opt/wandisco/fusion/client/lib directory:

The three 3rd-party JARs have had their classes repackaged under "shadow" package names in order that they do not conflict with the alternative versions used by Presto. Additionally, the IBM Big Replicate client library has been adjusted to use those versions of the classes under their alternative package names.

IBM has made this repackaged client library available for Walmart on fd.wandisco.com.

It is also important to confirm that the Presto configuration includes the necessary properties to function correctly with the hive-hadoop2 plugin. E.g. (the specific values below will need to be adjusted for the actual environment), but include reference to the IBM replicated metastore, the HDP cluster configuration that includes Big Replicate configuration, and Kerberos-specific information to allow Presto to interoperate with a secured cluster.

Keytabs and principals will need to be configured correctly, and as the hive-hadoop2 Presto plugin uses YARN for operation, the /user/yarn directory must exist and be writable by the yarn user in all clusters in which Big Replicate operates.

Presto embeds Hadoop configuration defaults into the hive-hadoop2 plugin, including a core-default.xml file that specifies the following property entry:

Although Presto allows the hive-hadoop2 plugin to use additional configuration properties (by specifying an entry like the following in a .properties file in the etc/catalog directory:

This entry allows extra configuration properties to be loaded from a standard Hadoop configuration file, but those entries cannot override setting that are embedded in the core-default.xml that ships with the Presto hive-hadoop2 plugin.

In a kerberized implementation the Big Replicate client library relies on the ability to read the hadoop.security.authentication configuration property to determine if it should perform a secure handshake with the Big Replicate server. Without that property defined, the client and server will fail to perform their security handshake, and Presto queries will not succeed.

The solution to this issue is to update the core-default.xml file contained in the hive-hadoop2 plugin:

Edit the core-default.xml file to update the hadoop.security.authentication property so that its value is “kerberosâ€

Distribute the hadoop-apache2-0.10.jar to all Presto nodes, and restart the Presto coordinator.

Oracle:BDA is built on top of Cloudera’s Hadoop and requires some extra steps to support multi-homed network environment.

There’s a known issue concerning configuration and the Cloudera Navigator Metadata Server classpath.

Error message:

There’s no clear way to override the fs.hdfs.impl setting just for the Navigator Metadata server, as is required for running with Big Replicate.

Use the following fix script to overcome the problem:

The environment variables are provided here - navigator_env.txt

You need to put this in the configuration for the Cloudera Management Service under " Navigator Metadata Server Environment Advanced Configuration Snippet (Safety Valve)". This modification currently needs to be applied whenever you upgrade or downgrade Big Replicate.

In order to make Hive with Tez work, you need to append the Big Replicate jar files in tez.cluster.additional.classpath.prefix under the Advanced tez-site section:

e.g. Big Replicate tree

Running Hortonworks Data Platform, the tez.lib.uris parameter defaults to /hdp/apps/${hdp.version}/tez/tez.tar.gz. So, to add Big Replicate libs, there are two choices:

Option 1: Delete the above value, and instead have a list including the path where the above gz unpacks to, and the path where Big Replicate libs are. or Option 2: Unpack the above gz, repack with Big Replicate libs and re-upload to HDFS. Note that both changes are vulnerable to a platform (HDP) upgrade.

Getting set up with the default URI is simple, Solr just needs to be able to find the Big Replicate client jar files that contain the FusionHdfs class.

This is a minimal working solution with Solr on top of fusion.

If you want to set up Flume through Cloudera Manager follow these steps:

There is a known issue where Spark is not picking up Hive-Site.xml, See Hadoop configuration is not localised when submitting job in yarn-cluster mode (Fixed in version 1.4).

You need to manually add it in by either:

IBM Big Replicate uses a replacement client library when overriding the hdfs:// scheme for access to the cluster file system in order to coordinate that access among multiple clusters. This replacement library is provided in a collection of jar files in the /opt/wandisco/fusion/client/lib directory for a standard installation. These jar files need to be available to any process that accesses the file system using the com.wandisco.Big Replicate.fs.client.FusionHdfs implementation of the Apache Hadoop File System API.

Because Spark does not provide a configurable mechanism for making the Big Replicate classes available to the Spark history server, the Spark Executor or Spark Driver programs, IBM Big Replicate client library classes need to be made available in the existing Spark assembly jar that holds the classes used by these Spark components. This requires updating that assembly jar to incorporate the Big Replicate client library classes.

This is one of a number of methods that may be employed to provide Big Replicate-Spark integration. We hope to cover some alternate methods at a later date.

Then follow this process to update the Spark assembly jar.

With the updated spark assembly jar, Big Replicate does not require any modification to the cluster SPARK_CLASSPATH environment variable, the spark.driver.extraClassPath and spark.executor.extraClassPath can also be left unset. This will require removing any modifications made to the spark-env.sh script by the IBM Big Replicate installer, leaving it in its original state, and removing any Ambari configuration properties created or modified for Big Replicate in the Custom spark-defaults section.

Other use of Spark should function as normal. Spark jobs have full access to the cluster file system via HDFS, mediated and coordinated by IBM Big Replicate.

For HBase to run with Big Replicate, the following directories need to be created and permissioned, as shown below:

The steps below provide a method of handling a recovery using a cold back-up. Note that multiple HMaster/region servers restarts might be needed for certain steps, since hbck command generally requires master to be up, which may require fixing filesystem-level inconsistencies first.

Final step to correct metadata-related errors

The following procedure covers the installation and setup of Big Replicate deployed over the LocalFileSystem. This requires an administrator to enter details throughout the procedure. Once the initial settings are entered through the terminal session, the deployment to the LocalFileSystem is then completed through a browser.

Entry fields for zone properties

Configuration for the local file system:

The installer will validate that the directory given or that is set by default (if you leave the field blank), can be written to by Big Replicate. Configuration file path:: System path to the Kerberos configuration file, e.g. /etc/krb5.conf Keytab file path:: System path to your generated keytab file, e.g. /etc/krb5.keytab

To configure Big Replicate for access to encrypted zones, two aspects need to be considered:

The following items need to be considered within KMS configuration to ensure Big Replicate has access:

The kms-site configuration (such as Advanced kms-site in Ambari) contains its own auth_to_local type parameter called “hadoop.kms.authentication.kerberos.name.rules”

Ensure that any auth_to_local mapping used for the Big Replicate principal is also contained here. This can be most easily achieved via simple copy/paste from core-site.xml.

The kms-site configuration (such as Custom kms-site in Ambari) contains proxyuser paramaters such as

Entries should be created for the local Big Replicate user (after auth_to_local translation) to allow Big Replicate to proxy / impersonate other users requests. This could be as simple as.

In the dbks-site configuration, the parameter hadoop.kms.blacklist.DECRYPT_EEK exists. Ensure this does not contain the username that Big Replicate uses (after auth_to_local translation).

In the KMS ACLs, such as using Ranger KMS, ensure that the Big Replicate user (after auth_to_local translation) has "Get Metadata" and "Decrypt EEK" permissions to keys.

This could be granted access to all keys. This will avoid a need to review rules when new keys are added. However, Big Replicate will only need these permissions to keys that apply to zones that fall within a replicated path. Consideration is needed here based on the user that Big Replicate has been configured as - either "hdfs" will need access to EEKs, OR the fusion user will need access, OR the supergroup could be given access to EEKs (it is enabled by default on Ambari but disabled on CDH), and then make the Big Replicate user a member of the supergroup.

If you do not perform the correct configuration, both local operations (as performed by a client) and / or the replicated actions may fail when the Big Replicate client is invoked. This should only apply to replicated paths.

So to troubleshoot:

The Big Replicate installer doesn’t currently handle the installation of the client to the rest of the nodes in the cluster. You need to go through the following procedure:

In the Client Installation section of the installer you will see line "Download a list of your client nodes" along with links to the client RPM packages.

RPM package location
If you need to find the packages after leaving the installer page with the link, you can find them in your installation directory, here:

If you are installing the RPMs, download and install the package on each of the nodes that appear on the list from step 1.

Installing the client RPM is done in the usual way:

HDP2.1/Ambari 1.6: Start services after installation
When installing clients via RPM into HDP2.1/Ambari 1.6., ensure that you restart services in Ambari before continuing to the next step.

Debian not supported
Although Ubuntu uses Debian’s packaging system, currently Debian itself is not supported. Note: Hortonworks HDP does not support Debian.

If you are running with an Ubuntu Linux distribution, you need to go through the following procedure for installing the clients using Debian’s DEB package:

For deployments into Cloudera clusters, clients can be installed using Cloudera’s own packaging format: Parcels.

For deployments into Hortonworks HDP/Ambari/IBM BigInsights cluster, version 1.7 or later. Clients can be installed using Hortonwork’s own packaging format: HDP Stack. This approach always works for Pivotal HD.

Ambari 1.6 and earlier
If you are deploying with Ambari 1.6 or earlier, don’t use the provided Stacks, instead use the generic RPMs.

Ambari 1.7
If you are deploying with Ambari 1.7, take note of the requirement to perform some necessary restarts on Ambari before completing an installation.

Ambari 2.0
When adding a stack to Ambari 2.0 (any stack, not just Big Replicate client) there is a bug which causes the YARN parameter yarn.nodemanager.resource.memory-mb to reset to a default value for the YARN stack. This may result in the Java heap dropping from a manually-defined value, back to a low default value (2Gb). Note that this issue is fixed from Ambari 2.1.

Upgrading Ambari
When running Ambari prior to 2.0.1, we recommend that you remove and then reinstall the Big Replicate stack if you perform an update of Ambari. Prior to version 2.0.1, an upgraded Ambari refuses to restart the WD Big Replicate stack because the upgrade may wipe out the added services folder on the stack.

If you perform an Ambari upgrade and the Ambari server fails to restart , the workaround is to copy the Big Replicate service directory from the old to the new directory, so that it is picked up by the new version of Ambari, e.g.:

Again, this issue doesn’t occur once Ambari 2.0.1 is installed.

HDP 2.3/Ambari 2.1.1 install
There’s currently a problem that can block the installation of the WD Fusion client stack. If the installation of the client service gets stuck at the "Customize Service" step, you may need to use a workaround:

Make sure the service components are created and the configurations attached by making a GET call, e.g.

1. Add the service

2. Add the component

3. Get a list of the hosts

4. For each of the hosts in the list, add the FUSION_CLIENT component

5. Install the FUSION_CLIENT component

On MapR clusters, you need to copy Big Replicate configuration onto all other nodes in the cluster:

The following section is used when preparing to install Big Replicate on system that already has an earlier version of Big Replicate installed. Before you install an updated version of Big Replicate you need to ensure that components and configurartion for an earlier installation have been removed. Go through the following steps before installing a new version of Big Replicate:

There’s currently no uninstall function for our installer, so the system will have to be cleaned up manually. If you used the unified installer then use the following steps:

For a complete uninstallation, remove the following properties from the core-site.xml:

Reinstalling Big Replicate server only
If you reinstall the fusion-server without also reinstalling the fusion-ui-server, then you should restart the fusion-ui-server service to ensure the correct function of some parts of the UI. If the service is not restarted then you may find that the dashboard graphs stop working properly, along with the UI’s Stop/start controls. e.g. run:

This section runs through the installation of Big Replicate into an IBM Openstack environment using Swift storage. Currently this deployment is limited to an active-passive configuration that would be used to ingest data from your on-premises cluster to your Swift storage.

Before you begin an installation to an Openstack Swift cluster make sure that you have the following directories created and suitably permissioned. Examples:

The installation process runs through the following steps:

Follow this section to complete the installation by configuring WD Fusion on a server that will place data that is replicated from your on-premises cluster to your OpenStack Swift storage. This second node can also be on-premises or co-located with your OpenStack platform.

Open a web browser and point it at the provided URL. e.g

Some of the required information can be gathered from the Bluemix UI, in the Service Credentials section:

Swift password changes
During installation, the Swift password is encrypted for use with Big Replicate. This process doesn’t require any further interaction except for the case where the Swift password is changed. If you change your Swift password you need to do the following:

The installer checks that the Swift account details are valid for accessing the segment container. If the validation fails, you should recheck your Swift account credentials.

It’s now time to demonstrate data replication between the on-premises cluster and the IBM OpenStack / Swift storage. First we need to perform a synchronization to ensure that the data stored in both zones is in exactly the same state.

You can synchronize data in both directions:

The following guide covers the replication from on-premises to the OpenStack/Swift node.

You can complete an IBM Swift installation using the Silent Installation procedure, putting the necessary configuration in the swift_silent_installer.properties as described in the section that covers Silent Installation.

Environment Variables Required for Swift deployments:

silent_installer.properties File additional settings or specific required values listed here:

As an example (as root), running on the installer moved to /tmp.

If you have a large folder you can parallelize the initial repair using the Fusion API. This can be accomplished on a single file or a whole directory. Choosing a directory will push all files from the source to the target regardless of existence at the target.

Consider the following directory structure for a Big Replicate replicated folder /home

We could run a bulk resolve in the UI against the /home directory, however, to provide parallelism of the repair operations we can use the Big Replicate API to issue repairs against each folder and the individual files in the /home folder.

This will spawn simultaneous repairs increasing the performance of the initial synchronization. This is especially helpful when you have small file sizes to better saturate the network.

For files, the recursive parameter is ignored
You can use the file transfers view in the Big Replicate UI on the OpenStack-replicating node to monitor the incoming files.

You can use the Fusion Web UI to selectively choose which files to repair in the UI when you have a small number of files that exists on both sides and a decision needs to be made as to which one is the source of truth.

To start Big Replicate UI:

To shut down:

Shutdowns take some time

The shutdown script attempts to stop processes in order before completing, as a result you may find that (from Big Replicate 2.1.3) shutdowns may take up to a minute to complete.

The start-up script for persistent running of Big Replicate is in the /etc/init.d folder. Run the script with the help command to list the available commands:

Check the running status (with current process ID):

Check the version:

Currently you need to use the same username and password that are required for your platform manager, e.g. Cloudera Manager or Ambari. In a future release we will separate Big Replicate UI from the manager and use a new set of credentials.

The Big Replicate UI dashboard provides a view of Big Replicate’s status. From the world map you can identify which data centers are experiencing problems, track replication between data centers or monitor the usage of system resources.

UI Dashboard will indicate if there are problems with Big Replicate on your cluster.

Using Big Replicate UI’s log viewer (View Logs):

Unless configured differently, Big Replicate logs should be written to the following locations:

By default the log location properties are not exposed in the ui.properties file. If you need to update the UI server to look in different locations for the log files then you can add the following properties (in ui.properties). To be clear these entries do not set alternate locations for Big Replicate to write its logs, it only ensures that the UI server can still read the logs in the event that they are moved.:

The file is read by the UI server on start up so you will need to restart the server for changes to take affect. The ui.properties file is not replicated between nodes so you must currently set it manually on each node.

At startup the default log location is /dev/null. If there’s a problem before log4j has initialised this will result in important logs getting lost. You can set the log location to a filespace that preserve early logging.

Edit fusion_env.sh adding paths to the following properties:

The induction process performs some validation before running. If this validation failures you will quickly see a warning messages appear.

The logs on the Big Replicate server record events that relate to the data replication system.

The Big Replicate user interface layer, responsible for handling interactions between the administrator, Big Replicate and the Hadoop Management layer.

The UI logs will contain errors such as failed access to the user interface, connectivity errors between the user interface and Big Replicate Server REST API and other syntax errors between the user interface and the Big Replicate server’s REST API and other syntax errors whilst performing administrative actions across the UI.

Responsible for streaming files from the location of the client write to the Big Replicate server process in any remote cluster to which hadoop data is replicated.

This is the standard format of the IBM log messages within Big Replicate. It includes an ISO8601 formatted timestamp of the entry, the log level / priority, followed by the log entry itself. Log levels we provide in order of severity (highest to lowest) that you may observe:

For log analysis and reporting, logs with at the PANIC, SEVERE and ERROR levels should be investigated. The warning level messages indicate an unexpected result has been observed but one that hasn’t impacted the system’s continued operation. Additional levels may exist, but are used in cases when the logging level has been increased for specific debug purposes. At other times, other levels should be treated as informational (INFO).

One simple thing that can be done is to grep the log file for any instance of "exception" and/or "PANIC" - this will tell the administrator a great deal without much effort. Using something like:

You can find the talkback script located on the Big Replicate server’s installation directory:

You can run talkback as follows:

If a cluster has Kerberos security enabled (Talkback will detect this from Big Replicate’s configuration), you may be asked for Kerberos details needed to authenticate with the cluster.

You will be asked to complete the following details:

To run the talkback script, follow this procedure:

If you move objects onto the distributed file system you must make sure that you use the same URI on both the originating and destination paths. Otherwise you’d see an error like this:

If you use the fusion:/// URI on both paths it will work, e.g.

Note that since the non-replicated directory doesn’t yet exist in ZONE2 it will get created without the files it contains on the originating zone. _When running Big Replicate using the fusion:///, moving non-replicated directory to replicated directory will not work unless you use of the fusion:/// URI.

You can’t move files between replicated directories
Currently you can’t perform a straight move operation between two separate replicated directories.

Big Replicate’s replication technology ensures that changes to data are efficiently propagated to each zone. There are, however, a few cases where the consistency of objects in the distributed file system lose consistency. Big Replicate can be set to schedule periodic consistency checks, or an administrator can trigger a check from the Admin UI or via the REST API.

If an inconsistency is found then the administrator needs to use the repair functions available through the Big Replicate UI or manually repair the issue using whatever system tools correspond with the Hadoop application. This may require that up-to-date files are manually copied over from one zone to overwrite the corrupted version of the files. In some cases files will need to be deleted/removed in order to restore consistency. You will need to follow the guidelines and documentation that correspond with your underlying applications, e.g. MapR, Hive etc.

Consistency Checks look at file size, not content
The current implementation of the Consistency Check tool compares the size of files between zones. We’re looking carefully at how we can implement a qualitative check that can specifically identify file corruption while not greatly impacting performance.

Repairs on large files
Please note that when very large files are repaired, it may appear that the process has stalled with different numbers of appends getting reported, post-completion. We recommend that you allow repair operations plenty of time to complete.

Username Translation
If any nodes that take part in a consistency check have the Username Translation feature enabled, then inconsistencies in the "user" field will be ignored.

When looking at the transfer reporting, note that there are situations in which HFlush/early file transfer where transfer logs will appear incorrect. For example, the push threshold may appear to be ignored. This could happen if an originating file is closed and renamed before pulls are triggered by the HFlush lookup. Note that although this results in confusing logs, those logs are in fact correct; you would see only two appends, rather than the number determined by your push threshold - one in the very beginning, and one from the rename, which pulls the remainder of the file. What is happening is optimal; all the data is available to be pulled at that instant, so we might as well pull all of it at once instead of in chunks.

WANdisco’s patented replication engine, DConE, can be configured for different use cases, balancing between performance and resource costs. The following section looks at a number of tunable properties that can be used to optimize Big Replicate for your individual deployment.

Only one Big Replicate node per zone is allowed to write into a particular replicated directory. The node that is assigned to do the writing is called the writer. See more about the role of the writer.

Should the current writer suddenly become unavailable, then a re-election process begins for assigning the role to one of the remaining nodes. Although the re-election process is designed to balance speed against and system resource usage, there may be deployments where the processing speed is critical. For this reason, the reelection timing can be tuned with the following system:

Period of time between a writer going off-line and another writer is elected and starts picking up = writerCheckPeriod * writerCheckMultiple. i.e.

If you feel these default settings create cause the system to wait too long before kicking off a re-election then you can update them using an API call:

You can adjust these properties to be optimal for your deployment. However, consider the following pointers:

Generate an xml file (we’ll call it induction.xml) containing an induction ticket with the inductors details (Generally the inductor port should not change but this is the port that all DConE traffic uses. You can find this in your application.properties file as application_port)

Send the xml file to your inductee:

Requirements: A minimum of two nodes that have been inducted.

Steps:

Generate an xml file (we’ll call it membership.xml) containing a membership object. DConE supports various configuration of node roles but for the time being the Big Replicate UI only supports <Acceptor, Proposer, Learner> and <Proposer, Learner>. If you choose to have an even number of <Acceptor, Proposer, Learner> nodes you must specify a tiebreaker.

Send the xml file to one of your nodes:

Requirements: A minimum of two nodes inducted together and a membership created that contains them (you’ll want to make a note of the membership id of your chosen membership).

Steps:
Generate an xml file (we’ll call it statemachine.xml) containing a fsMapping object.

Send the xml file to one of your nodes:

Prior to JDK6 Update 27, Java fails to load the Kerberos ticket cache correctly when using MIT Kerberos 1.8.1 or later, even after a kinit.

The following exception will occur when attempting to access the Hadoop cluster.

The workaround is:

See below for a full explanation of each option:

Use the -d option to test an uninstallation. This option lets you test the effects of an installation, without any actual file changes being made. Use this option to be sure that your uninstallation will do what you expect.

The first step in setting up a replicated folder is the creation of a target folder:

If the path contains a leading slash "/", we assume it is an absolute path, if it contains no leading slash then we assume it is a relative path and the root directory will be added to the beginning of the exclusion.

If you click on the View link for a Replicated Folder, then you enter a tabbed UI:

Run through the following procedure to perform a repair:

SnapDiff is an internal Data ONTAP engine that quickly identifies the file and directory differences between two Snapshot copies. See What SnapDiff is.

The consistency check mechanism lets you verify that replicated HDFS data is consistent between sites. Read about Handling file inconsistencies.

Click on the report link to get more information about the current consistency check results.

Big Replicate’s Consistency Check tool includes a feature for resolving any inconsistencies that are detected across the distributed file system. Use the following procedure to resolve any such inconsistencies:

Records are not persisted and are cleared up on a restart. The log records are truncated to stop an unbounded use of memory, and the current implementation is as follows:
For each state machine, if there are more than 1,000 entries in its list of transfers we remove the oldest transfers ,sorted by complete time, which are in a terminal state ("completed", "failed" or "deleted") until the size of the list is equal to 1,000. The check on the number of records in the list is performed every hour.

It is currently not possible to delete memberships that are no longer required. Currently, removing memberships would potentially break the replication system.

For deployments that are run under an enterprise license, additional tools are available for monitoring and managing the amount of data transferred between zones.

Enterprise License only The Bandwidth Management tools are only enabled on clusters that are running on an Enterprise license. See the Deployment Checklist for details about License Types.

The bandwidth management tools provide two additional areas of functionality to support Enterprise deployments.

Any applicable bandwidth limits are replicated across your nodes and applied on a per-zone basis.

Big Replicate Nodes - when Enterprise license is in use.

The Big Replicate Nodes screen will display current incoming traffic for the local zone. You will need to log in to the Big Replicate UI on a node within each Zone to see all incoming traffic levels.

Use this procedure to set up bandwidth limits between your zones

Click on the Set bandwidth limit button for each corresponding zone.

The Maximum bandwidth dialog will open. For each remote zone you can set a maximum Outgoing to and Incoming from values. Entered values are in Megabits per second. These are converted into Gigabytes per hour and displayed in brackets after each entry field.

Maximum bandwidth entry dialog.

When you have set your bandwidth values, click Update to apply these settings to your deployment.

Maximum bandwidth entry dialog.

You can enable SSL encryption between the Big Replicate UI and your browser.

The server settings give you control over traffic encryption between Big Replicate and IHC servers.

What follows is a manual procedure for setting up SSL. In most cases it has been superseded by the above Big Replicate UI-driven method. If you make changed using the following method, you will need to restart the WD Big Replicate server in order for the changed to appear in on the Settings tab.

Create the keystores / truststores. Every Big Replicate Server and IHC server should have a KeyStore with a private key entry / certificate chain for encrypting and signing. Every Big Replicate Server and Big Replicate Client must also have a truststore for validating certificates in the path specific in “fusion.ssl.truststore”. The keystores and truststores can be the same file and may be shared amongst the processes.

Note that if you enable the Big Replicate Server to communicate over HTTPS-only, then you must also make the following changes so that the Big Replicate UI matches up:

Big Replicate is designed to fit into deployments that have far-flung data centers. The Node Location setting is used to identify where in the world the data center is situated, using standard global positioning system coordinates. These coordinates will be used by any connected WD Big Replicate nodes to correctly place the node’s location on the world map.

This section describes how to set up notification emails that will be triggered if one of the tracked system resources reaches a defined threshold.

Email Notification Settings are located in the Zone section of the settings.

Complete the following steps to enable email notification:

The server config tab contains the settings for the SMTP email server that you will use for relaying your notification emails. You need to complete and check the provided details are correct first, before your notification emails can be enabled.

Email Notification Settings are located in the Zone section of the settings

The recipients tab is used to store one or more email addresses that can be used when sending out notification emails. You can enter any number of addresses, although you will still need to associate an entered address with a specific notification before it will be used. See Adding recipients

Once you have working server settings valid recipient email addresses you can start to enable notification emails from the Alerts tab.

The Templates tab gives you access to the email default text, allowing you to review and customize with additional messaging.

Email templates

This is what an email notification looks like:

By default, handshake tokens are created in the user’s working directories, e.g. /user/jdoe. It is recommended that you create them elsewhere, using the following procedure:

Important: Known issue running Teragen and Terasort
There are known problems running Teragen and Terasort with FusionHdfs or FusionHcfs configurations. Some required directories are currently missing and will cause Terasort to hang. You can work around the problem by creating the following directories, then making sure that Yarn and MapR users are added and that they have access to the directories. E.g.,

The following procedures illustrate how to installing Kerberos, running with a single Key Distribution Center, under Ambari.

When to use kadmin.local and kadmin?
When performing the Kerberos commands in this procedure you can use kadmin.local or kadmin depending on your access and account:

Before you start, download and install the Java Cryptographic Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7.

Install the Kerberos server:

Edit /etc/krb5.conf and replace "EXAMPLE.COM" with your realm. E.g.

Edit /var/kerberos/krb5kdc/kdc.conf:

Edit the /var/kerberos/krb5kdc/kadm5.acl and replace EXAMPLE.COM with your principle.

To create a database, run

Start Kerberos service:

Prepare your kerberos clients. Run

Repeat this on all other machines in the cluster to make them kerberos workstations connecting to the KDC. E.g.

Copy the /etc/krb5.conf file from the kerberos server node to all kerberos client nodes

Create a user on all nodes: useradd -u 1050 testuser

Create principal and password for user (testuser):

During the Big Replicate Installation’s Kerberos step, set the configuration for an existing Kerberos setup.

The following procedures illustrate how to installing Kerberos, running with a single Key Distribution Center, under CDH.

When to use kadmin.local and kadmin?
When performing the Kerberos commands in this procedure you can use kadmin.local or kadmin depending on your access and account:

Create hdfs.keytab and move hdfs.keytab file in the /etc/cloudera-scm-server/ directory on the host where you are running the Cloudera Manager Server. Make sure that the hdfs.keytab file has readable permissions for all users:

The following sequence is an example procedure for creating the Cloudera Manager Server principal and keytab file for MIT Kerberos.

After obtaining or creating the Cloudera Manager Server principal and keytab, follow these instructions to deploy them:

Move the cmf.keytab file to the /etc/cloudera-scm-server/. This is the directory on the host where you are running the Cloudera Manager Server.

Ensure that the cmf.keytab file is only readable by the Cloudera Manager Server user account cloudera-scm.

Add the Cloudera Manager Server principal (cloudera-scm/admin@DOMAIN.COM) to a text file named cmf.principal and store the cmf.principal file in the /etc/cloudera-scm-server/ directory on the host where you are running the Cloudera Manager Server.

Make sure that the cmf.principal file is only readable by the Cloudera Manager Server user account cloudera-scm.

Note: For Single KDC copy cmf.keytab and cmf.principal to another CM node:

There are some extra steps required to overcome a class loading error that occurs when Big Replicate is used with at-rest encrypted folders. Specifically, cluster config changes described as follows:

In the keystore, the server certificate is associate with a key. By default, we look for a key named server to validate the certificate. If you use a key for the server with a different name, enter this in the SSL settings.

A complete debug of the SSL logging will be required to diagnose the problems. To capture the debugging, ensure that the variable debugSsl is set to "true".

To enable the logging of SSL implemented layer, turn the logging to FINEST for 'com.wandisco.platform.net' package.