Question & Answer
Question
How to configure the federated server to access Spark data sources
Answer
Spark that are accessed through the JDBC API are referred to in this text as Spark data sources.
Before you begin
The spark JDBC driver must be installed and configured on the computer that acts as the federated server.
Spark provides an JDBC driver for the AIX, Linux operating systems.
For prerequisites and limitations that apply to all JDBC data sources, See Configuring access to JDBC data sources.
Restrictions
- IUD and Transparent IUD are not supported.
- Method 1 is not supported for NNSTAT.
- Spark type: Binary supports binary(256) for query and will be truncated when the lengh > 256.
- Spark type: Binay and String will be truncated when the lengh > 32767 in passthru-mode session.
- TINYINT, SMALLINT, INT, BIGINT, DECIMAL types return "-" when remote data overflow.
- COMMIT and ROLLBACK is not supported.
Additional restrictions apply to all JDBC data sources. See Configuring access to JDBC data sources.
Procedure
- Use the following method to prepare the federated server and federated database: Prepare the federated server to access data sources through JDBC.
There are still some limitations for spark jdbc driver to connect to hive. See Spark information center.
Now we take spark1.3.1 + hive1.2.1 on linuxamd64 for example:
1. Setup spark and then start thriftserver using $spark_path/sbin/start-thriftserver.sh
2. Find Jdbc driver:hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar and hadoop-common-2.6.0.2.2.4.2-2.jar, put them into the computer that acts as the federated server.
3. Set CLASSPATH for jdbc driver and then jdbc wrapper can load jdbc driver to connect to spark.
export CLASSPATH=/$dirver_location/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar:/$dirver_location/hadoop-common-2.6.0.2.2.4.2-2.jar:$CLASSPATH
4. Create Federation Jdbc Objects
1. Register the JDBC wrapper.
2. Register the server definitions for an JDBC data source.
3. Create a user mapping for an JDBC data source.
4. Test the connection to the JDBC data source server.
5. Register nicknames for JDBC data source tables and views.
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21984912