Install IBM Financial Transactions Repository

Ensure that you have enabled security for your Hortonworks Data Platform (HDP) environment.

Procedure

Update the HDFS configuration.
1. Log in to the Ambari console.
  
  https://ambari.server:8081
2. Click HDFS, click the Configs tab, and then click the Advanced tab.
3. Expand Custom core-site, and click Add Property.
4. In the Properties box, enter the following:
```
hadoop.proxyuser.sifsuser.hosts=*
hadoop.proxyuser.sifsuser.groups=*
```
5. Click Add, and then click Save.
6. Click Hive, click the Configs tab, and then click the Advanced tab.
7. Expand Custom hive-site, and click Add Property.
8. In the Properties box, enter the following:
```
hive.users.in.admin.role=hive,sifsuser
hadoop.proxyuser.sifsuser.hosts =*
hadoop.proxyuser.sifsuser.groups=*
hive.server2.enable.doAs=true
```
9. Click Add, and then click Save.
10. Copy the hive-site.xml to the SPARK_HOME/conf directory on the Spark driver and worker node computers.
Run the following commands as the hbase user on the HBase master server node computer:
```
Hbase shell
```
```
grant ‘RWCA’, ‘sifsuser’
```
Copy the following JAR files to the /opt/ibm/ftr/lib directory on all HDP nodes.
- json-1.8.jar
- kafka-clients-0.10.1.2.6.4.0-91.jar
Go to the /opt/ibm/ftr directory.
Ensure that the scripts in this directory have the following permissions:

Figure 1. Script permissions
Run the following command:
```
./ftrsetup.sh
```
The artifacts are downloaded from Artifactory and extracted. The JAR files are copied to the relevant directories, and the Hbase and Hive tables are created.

Run the following commands to set up DAL and configure the Hive and HBase tables. Run the commands as the root user.

mkdir /home/sifsuser/ConfFiles

cp /usr/hdp/2.6.4.0-91/kafka/conf/kafka.client.keystore.jks /home/sifsuser/ConfFiles/

cp /usr/hdp/2.6.4.0-91/kafka/conf/kafka.client.truststore.jks /home/sifsuser/ConfFiles/

cp /usr/hdp/2.6.4.0-91/kafka/conf/kafka_jaas.conf /home/sifsuser/ConfFiles/

chown -R sifsuser:hadoop /home/sifsuser/ConfFiles

Configure the ExtractConfig.jar file.
1. Create a temporary directory:
```
mkdir temp
```
2. Extract ExtractConfig.jar to the temp directory:
```
unzip ExtractConfig.jar -d ./temp
```
3. In the extracted files, open the resource/config.properties file to set the REST API URL. Change the <Kube_manager_IP> value to match your environment:
```
API_URL= https://<Kube_manager_IP>:3001/config
```
4. Compress the file to create new ExtractConfig.jar file:
```
zip -r ExtractConfig.jar
```

Create Hive external tables by using beeline:

su - sifsuser

>beeline

beeline>!connect <JDBC hiveserver2 address> "" "";

beeline>create database if not exists ccsdb;

beeline>CREATE EXTERNAL TABLE ccsdb.tblCCSAudit(key string, streamId string, activityTimestamp string, Component string, description string, fileIndicator string, fileName string, system string,fileSize String,notification String,numberOfParsingErrors String,numberOfRecords String,originalFileName String,processingStage String,status String,tradingDate String) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "auditData:streamId,auditData:activityTimestamp,auditData:component,auditData:description,auditData:fileIndicator,auditData:fileName, auditData:system,auditData:fileSize,auditData:notification,auditData:numberOfParsingErrors,auditData:numberOfRecords,auditData:originalFileName,auditData:processingStage,auditData:status,auditData:tradingDate") TBLPROPERTIES("hbase.table.name" = "tblCCSAudit");

beeline>CREATE EXTERNAL TABLE ccsdb.tblFileVersion(key string, StreamId string, Ingestion_Timestamp string, Version string, fileCreatedDate string, fileHandler string, fileName string, system string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "metaData:streamId,metaData:activityTimestamp,metaData:version,metaData:tradingDate,metaData:fileHandler,metaData:fileName, metaData:system") TBLPROPERTIES("hbase.table.name" = "tblFileVersion");

beeline>create external table ccsdb.fr_data_spark(rowkey string, msgType string, streamId string ,ClOrdID string, Symbol string, TransactTime string, OrdType string, OrderQty int,Price double, Side string, PartyID string) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'with serdeproperties ("hbase.columns.mapping"=":key,messageHeader:MsgType,metaData:streamId, messageBody:ClOrdID, messageBody:Symbol,messageBody:TransactTime, messageBody:OrdType, messageBody:OrderQty, messageBody:Price, messageBody:Side, messageBody:PartyID") TBLPROPERTIES("hbase.table.name" = "tblHRData");

Edit the /opt/ibm/ftr/config/configuration.properties file to include the environment details:

Table 1. configuration.properties values
Property	Location or value
SERVER_ADDRESS	Ambari > Services > HDFS > Config (search for the following property and copy the value) > fs.defaultFS
HADOOP_USER_NAME	Hadoop
HIVE_ADDRESS	Ambari > Services > Hive > HiveServer2 JDBC URL
HDFS_SERVER	Ambari > Services > HDFS > Config (search for the following property and copy the value) > fs.defaultFS
HBASE_ZOOKEEPER_QUORUM_VALUE	Ambari > Services > HBASE > Config (search for the following property and copy the value) > hbase.zookeeper.quorum
HBASE_ZOOKEEPER_CLIENT_VALUE (Port)	Ambari > Services > HBASE > Config (search for the following property and copy the value) > hbase.zookeeper.property.clientPort
BOOTSTRAP_SERVERS_CONFIG_PRODUCER	Ambari > Services > Kafka > Config (search for the following property and copy the value) > `<Kafka Host name : Port Id>`. The port number is 6667 for an HDP environment.
BOOTSTRAP_SERVERS_CONFIG_CONSUMER	Ambari > Services > Kafka > Config (search for the following property and copy the value) > `<Kafka Host name : Port Id>`. The port number is 6667 for an HDP environment.
SSL_TRUSTSTORE_LOCATION_CONFIG	`<Include SSL truststore .jks file path here>`
SSL_TRUSTSTORE_PASSWORD_CONFIG	`<Trustore password>`
SSL_KEYSTORE_LOCATION_CONFIG	`<Include SSL keystore .jks file path here>`
SSL_KEYSTORE_PASSWORD_CONFIG	`<Keystore password>`
SSL_KEY_PASSWORD_CONFIG	`<Key password>`
SASL_KERBEROS_SERVICE_NAME	kafka
SASL_JAAS_CONFIG	com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab=”`<include kafka service keytab>` " principal="`<include principal for kafka service keytab>`" useTicketCache=true renewTicket=true serviceName="kafka";
SPARK_HOME	`<Include spark2 path here>`. For example, for an HDP environment: usr/hdp/2.6.4.0-91/spark2
KEYTAB	`<include kafka service keytab>`
PRINCIPAL	`<include principal for kafka service keytab>`
JAAS_CONFIG	-Djava.security.auth.login.config=`<include path for kafka JAAS file here>`
CONFIG_API	https://`<kubernetes master hostname>`:3001/SIFSServices/ftr/config/

Change to the sifuser user.
```
su sifsuser
```

Run the following commands:

nohup java -jar /opt/ibm/ftr/jar/Kafka-2.0.3-SNAPSHOT-jar-with-dependencies.jar > /home/sifsuser/dataingestion_log.$(date --iso).out &

nohup park-submit --master local --deploy-mode client --driver-memory 10G --executor-memory 1G --num-executors 5 --executor-cores 3 --conf spark.driver.extraClassPath="/usr/hdp/2.6.4.0-91/hive/lib/*:/usr/hdp/2.6.4.0-91/hbase/lib/*" --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=/home/sifsuser/ConfFiles/sifs-jaas.conf" --files /usr/hdp/2.6.4.0-91/hbase/conf/hbase-site.xml --keytab /etc/security/keytabs/sifsuser.keytab --principal sifsuser@IBM.COM --class com.ibm.ccs.kafka.notification.SendNotification /opt/ibm/ftr/jar/NotificationComp-2.0.3-SNAPSHOT.jar &