tHDFSProperties Standard properties
These properties are used to configure tHDFSProperties running in the Standard Job framework.
tHDFSProperties component belongs to the Big Data and the File families.
Either Built-in or Repository
Built-in: No property data stored centrally.
Repository: Select the repository file in which the
Use an existing connection
Select this check box and in the Component List click the HDFS connection component from which
Note that when a Job contains the parent Job and the child Job,
Select the cluster you are using from the drop-down list. The options in the
list vary depending on the component you are using. Among these options, the following
ones requires specific configuration:
Select the version of the Hadoop distribution you are using. The available
|Select the URI scheme of the file system to be used from the
Scheme drop-down list. This scheme could be
The schemes present on this list vary depending on the distribution you
Once a scheme is
If you have selected
ADLS, the connection parameters to be defined become:
Type in the URI of the Hadoop NameNode, the master node of a
Use kerberos authentication
If you are accessing the Hadoop cluster running
with Kerberos security, select this check box, then, enter the Kerberos
principal name for the NameNode in the field displayed. This enables you to use
your user name to authenticate against the credentials stored in Kerberos.
This check box is available depending on the Hadoop distribution you are
|Use a keytab to authenticate
Select the Use a keytab to authenticate
Note that the user that executes a keytab-enabled Job is not necessarily
The User name field is available when you are not using
Enter the membership including the authentication user under which the HDFS instances were
Schema and Edit schema
A schema is a row description. It defines the number of fields
The schema of this component is read-only. You can click
Browse to, or enter the path pointing to the data to be used in the file system.
Get file checksum
Select this check box to generate and output the MD5 information of the file
Note that this is an HDFS only checksum and not a true MD5 hash that can be
uses a default configuration for its engine to perform
operations in a Hadoop distribution. If you need to use a custom configuration in a specific
situation, complete this table with the property or properties to be customized. Then at
runtime, the customized property or properties will override those default ones.
For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you
are using or see Apache’s Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:
|Select this check box to gather the Job processing
metadata at a Job level as well as at each component level.
ERROR_MESSAGE: the error message generated by the
A Flow variable functions during the execution of a component while an After variable
To fill up a field or expression with a variable, press Ctrl +
For further information about variables, see
tHDFSProperties can be standalone
Click the [+] button to add a row in the
The Dynamic settings table is
For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic
The Hadoop distribution must be properly installed, so as to guarantee the interaction
For further information about how to install a Hadoop distribution, see the manuals
JRE 1.6+ is required.