Apache HDFS connection

To access your data in Apache HDFS, create a connection asset for it.

Apache Hadoop Distributed File System (HDFS) is a distributed file system that is designed to run on commodity hardware. Apache HDFS was formerly Hortonworks HDFS.

Supported versions

Apache HDFS 3.1.

Prerequisites for Kerberos authentication

If you plan to use Kerberos authentication, complete the following requirements:

Configure the data source for Kerberos authentication. Optional: This connection supports Kerberos SSO with user impersonation, which requires additional configuration.
Confirm that the service that you plan to use the connection supports Kerberos. For more information, see Kerberos authentication in Cloud Pak for Data.
An administrator must complete one set of the following setup steps:
- Kerberos without SSO: Enabling platform connections to use Kerberos authentication
- Kerberos SSO: Configuration for Kerberos SSO

Create a connection to Apache HDFS

To create the connection asset, you need these connection details. The WebHDFS URL is required.
The available properties in the connection form depend on whether you select Connect to Apache Hive so that you can write tables to the Hive data source.

WebHDFS URL to access HDFS.
Hive host: Hostname or IP address of the Apache Hive server.
Hive database: The database in Apache Hive.
Hive port number: The port number of the Apache Hive server. The default value is 10000.
Hive HTTP path: The path of the endpoint such as gateway/default/hive when the server is configured for HTTP transport mode.
SSL certificate (if required by the Apache Hive server).

Authentication method

You can choose Kerberos credentials or Username and password.

For Kerberos credentials, you must complete the prerequisite for Kerberos authentication and you need the following connection details. You also need these details for Hive if you selected Connect to Apache Hive:
- Service principal name (SPN) that is configured for the data source.
- User principal name to connect to the Kerberized data source.
- The keytab file for the user principal name that is used to authenticate to the Key Distribution Center (KDC).
For Username and password, you also provide values for the Hive user and password if you connect to Apache Hive.

For Credentials and Certificates, you can use secrets if a vault is configured for the platform and the service supports vaults. For information, see Using secrets from vaults in connections.

Federal Information Processing Standards (FIPS) compliance

This connection is FIPS-compliant and can be used on a FIPS-enabled cluster.

Apache HDFS setup

Install and set up a Hadoop cluster

Supported file types

The Apache HDFS connection supports these file types: Avro, CSV, Delimited text, Excel, JSON, ORC, Parquet, SAS, SAV, SHP, and XML.

Table formats

In addition to Flat file, the Apache HDFS connection supports these Data Lake table formats: Delta Lake and Iceberg.