tHDFSCopy
source if required.
tHDFSCopy Standard properties
These properties are used to configure tHDFSCopy running in the Standard Job framework.
The Standard
tHDFSCopy component belongs to the Big Data and the File families.
The component in this framework is available in all Talend products with Big Data
and in Talend Data Fabric.
Basic settings
Property type |
Either Built-in or Repository Built-in: No property data stored centrally.
Repository: Select the repository file in which the |
Use an existing connection |
Select this check box and in the Component List click the HDFS connection component from which Note that when a Job contains the parent Job and the child Job, |
Distribution |
Select the cluster you are using from the drop-down list. The options in the
list vary depending on the component you are using. Among these options, the following ones requires specific configuration:
|
Version |
Select the version of the Hadoop distribution you are using. The available |
Scheme | Select the URI scheme of the file system to be used from the Scheme drop-down list. This scheme could be
The schemes present on this list vary depending on the distribution you Once a scheme is If you have selected
ADLS, the connection parameters to be defined become:
For a |
NameNode URI |
Type in the URI of the Hadoop NameNode, the master node of a |
Use kerberos authentication |
If you are accessing the Hadoop cluster running
with Kerberos security, select this check box, then, enter the Kerberos principal name for the NameNode in the field displayed. This enables you to use your user name to authenticate against the credentials stored in Kerberos.
This check box is available depending on the Hadoop distribution you are |
Use a keytab to authenticate |
Select the Use a keytab to authenticate Note that the user that executes a keytab-enabled Job is not necessarily |
User name |
The User name field is available when you are not using |
Group |
Enter the membership including the authentication user under which the HDFS instances were |
Source file or directory |
Browse to, or enter the path pointing to the data to be used in the file system. |
Target location |
Browse to, or enter the directory in HDFS to which you need to copy the |
Rename |
To rename the file or folder copied to the target location, select this check |
Copy merge |
Select this check box to merge the part files generated at the end of a Once selecting it, you need to enter the name of the final merged file in the |
Remove source |
Select this check box to remove the source file or folder once this source is |
Override target file (This option does not override the |
Select this check box to override the file already existing in the target |
Advanced settings
Hadoop properties |
Talend Studio uses a default configuration for its engine to perform operations in a Hadoop distribution. If you need to use a custom configuration in a specific situation, complete this table with the property or properties to be customized. Then at runtime, the customized property or properties will override those default ones.
For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you are using or see Apache’s Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:
|
tStatCatcher Statistics |
Select this check box to gather the Job processing metadata at a Job level as well as at each component level. |
Global Variables
Global Variables |
DESTINATION_FILEPATH: the destination file path. This is
SOURCE_FILEPATH: the source file path. This is an After
ERROR_MESSAGE: the error message generated by the A Flow variable functions during the execution of a component while an After variable To fill up a field or expression with a variable, press Ctrl + For further information about variables, see |
Usage
Usage rule |
tHDFSCopy is a standalone |
Dynamic settings |
Click the [+] button to add a row in the The Dynamic settings table is For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic |
Prerequisites |
The Hadoop distribution must be properly installed, so as to guarantee the interaction
For further information about how to install a Hadoop distribution, see the manuals |
Limitation |
JRE 1.6+ is required. |
Related scenario
Related topic, see Procedure
Related topic, see Iterating on a HDFS directory