tHDFSCompare
Compares two files in HDFS and based on the read-only schema, generates a row flow
that presents the comparison information.
tHDFSCompare helps to control the quality of the data
processed.
tHDFSCompare Standard properties
These properties are used to configure tHDFSCompare running in the Standard Job framework.
The Standard
tHDFSCompare component belongs to the Big Data and the File families.
The component in this framework is available in all Talend products with Big Data
and in Talend Data Fabric.
Basic settings
Property type |
Either Built-in or Repository Built-in: No property data stored centrally.
Repository: Select the repository file in which the |
Use an existing connection |
Select this check box and in the Component List click the HDFS connection component from which Note that when a Job contains the parent Job and the child Job, |
Distribution |
Select the cluster you are using from the drop-down list. The options in the
list vary depending on the component you are using. Among these options, the following ones requires specific configuration:
|
Version |
Select the version of the Hadoop distribution you are using. The available |
Scheme | Select the URI scheme of the file system to be used from the Scheme drop-down list. This scheme could be
The schemes present on this list vary depending on the distribution you Once a scheme is If you have selected
ADLS, the connection parameters to be defined become:
For a |
NameNode URI |
Type in the URI of the Hadoop NameNode, the master node of a |
Use kerberos authentication |
If you are accessing the Hadoop cluster running
with Kerberos security, select this check box, then, enter the Kerberos principal name for the NameNode in the field displayed. This enables you to use your user name to authenticate against the credentials stored in Kerberos.
This check box is available depending on the Hadoop distribution you are |
Use a keytab to authenticate |
Select the Use a keytab to authenticate Note that the user that executes a keytab-enabled Job is not necessarily |
User name |
The User name field is available when you are not using |
Group |
Enter the membership including the authentication user under which the HDFS instances were |
Schema and Edit schema |
A schema is a row description. It defines the number of fields The schema of this component is read-only. You can click |
Comparison mode |
Select the mode to be applied on the comparison. |
File to compare |
Browse, or enter the path to the file in HDFS you need to check for quality |
Reference file |
Browse, or enter the path to the file in HDFS the comparison is based on. |
If differences detected, display and If no |
Type in a message to be displayed in the Run console |
Print to console |
Select this check box to display the message in the Run |
Advanced settings
Encoding |
Select the encoding from the list or select Custom |
Hadoop properties |
Talend Studio uses a default configuration for its engine to perform operations in a Hadoop distribution. If you need to use a custom configuration in a specific situation, complete this table with the property or properties to be customized. Then at runtime, the customized property or properties will override those default ones.
For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you are using or see Apache’s Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:
|
tStatCatcher Statistics |
Select this check box to gather the Job processing metadata at a Job level as well as at each component level. |
Global Variables
Global Variables |
DIFFERENCE: the result of the comparison. This is a Flow
ERROR_MESSAGE: the error message generated by the A Flow variable functions during the execution of a component while an After variable To fill up a field or expression with a variable, press Ctrl + For further information about variables, see |
Usage
Usage rule |
tHDFSCompare can be standalone |
Dynamic settings |
Click the [+] button to add a row in the The Dynamic settings table is For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic |
Prerequisites |
The Hadoop distribution must be properly installed, so as to guarantee the interaction
For further information about how to install a Hadoop distribution, see the manuals |
Limitation |
JRE 1.6+ is required. |
Related scenarios
No scenario is available for the Standard version of this component yet.