tGenKeyHadoop – Docs for ESB 5.x

Component family

Data Quality/Matching

This component is deprecated as tGenKey can be used now in both standard and
Map/Reduce Jobs. tGenKeyHadoop will
continue to work in Jobs you import from older releases.

Function

This component connects to a given Hadoop distribution to process
the data stored in HDFS. It enables you to apply several kinds of
algorithms on each input column and use the computed results to
generate a functional key. These algorithms can be key or optional
algorithms.

Purpose

This component generates a functional key from the input columns,
by applying different types of algorithms on each column and
grouping the computed results in one key. It outputs this key
together with the input columns.

This component helps, using the generated functional key, to
narrow down your data filter/matching results for example.

Basic settings

Property type

Either Built-in or Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

Schema and Edit
schema

A schema is a row description, it defines the number of fields to
be processed and passed on to the next component. The schema is
either Built-in or stored remotely
in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

Click Sync columns to retrieve
the schema from the previous component in the Job.

Built-in: You create and store
the schema locally for this component only. Related topic: see
Talend Studio User
Guide.

Repository: You have already
created and stored the schema in the Repository. You can reuse it in
other projects and job designs. Related topic: see Talend Studio User Guide .

Link with a tGenKeyHadoop

Select this check box to reuse the connection, created by another
tGenKeyHadoop, to a given file
in HDFS. From the Component list,
select the relevant tGenKeyHadoop
component to reuse the Hadoop connection details you already
defined.

Use an existing connection

Select this check box and in the Component List click the
HDFS connection component from which you want to reuse the connection details already
defined.

Version

Distribution

Select the cluster you are using from the drop-down list. The options in the list vary
depending on the component you are using. Among these options, the following ones requires
specific configuration:

Hadoop version

Select the version of the Hadoop distribution you are using. The available options vary
depending on the component you are using. Along with the evolution of Hadoop, please note
the following changes:

NameNode URI

Select this check box to indicate the location of the NameNode of the Hadoop cluster to be
used. The NameNode is the master node of a Hadoop cluster. For example, we assume that you
have chosen a machine called masternode as the NameNode
of an Apache Hadoop distribution, then the location is hdfs://masternode:portnumber.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial
in Apache’s Hadoop documentation on http://hadoop.apache.org.

JobTracker URI

Select this check box to indicate the location of the Jobtracker service within the Hadoop
cluster to be used. For example, we assume that you have chosen a machine called machine1 as the JobTracker, then set its location as machine1:portnumber. A Jobtracker is the service that assigns
Map/Reduce tasks to specific nodes in a Hadoop cluster. Note that the notion job in this
term JobTracker does not designate a Talend Job, but rather a Hadoop job
described as MR or MapReduce job in Apache’s Hadoop documentation on http://hadoop.apache.org.

If you use YARN in your Hadoop cluster such as Hortonworks Data
Platform V2.0.0 or Cloudera CDH4.3 + (YARN
mode), you need to specify the location of the Resource
Manager instead of the Jobtracker. Then you can continue to set the following
parameters depending on the configuration of the Hadoop cluster to be used (if you leave the
check box of a parameter clear, then at runtime, the configuration about this parameter in
the Hadoop cluster to be used will be ignored ):

For further information about these parameters, see the documentation or
contact the administrator of the Hadoop cluster to be used.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial
in Apache’s Hadoop documentation on http://hadoop.apache.org.

User name

Enter the Hadoop user authentication.

For some Hadoop versions, you also need to enter the name of the
supergroup to which the user belongs in the Group field that is displayed.

HDFS directory

Enter the HDFS directory where the data to be processed is.

At runtime, this component will clean up any data in this
directory if it exists, write the input data in and perform the
operations.

If you want to reuse the existing data in HDFS, use:

Click the import icon to select a match rule from the Studio
repository.

When you click the import icon, a [Match
Rule Selector] wizard is opened to help you import
blocking keys from match rules in the Studio repository and use them
in your Job.

You can only import rules created with the VSR algorithm. For
further information, see Importing match rules from the studio repository

Algorithm

Column

Select the column(s) from the main flow on which you want to
define certain algorithms to set the functional key

Pre-Algorithm

If required, select the relevant matching algorithm from the
list:

remove diacritical marks:removes
any diacritical mark.

remove diacritical marks and lower
case: removes any diacritical mark and converts to
lower case before generating the code of the column.

remove diacritical marks and upper case:
removes any diacritical mark and converts to upper case
before generating the code of the column.

lower case: converts the field to
lower case before applying the key algorithm.

upper case: converts the field to
upper case before applying the key algorithm.

add left position character:
enables you to add a character to the left of the column.

add right position character:
enables you to add a character to the right of the column.

Value

Set the algorithm value, where applicable.

Algorithm

Select the relevant algorithm from the list:

first character of each word:
includes in the functional key the first character of each word in
the column.

N first characters of each word:
includes in the functional key N first characters of each word in
the column.

first N characters of the string:
includes in the functional key N first characters of the
string.

last N characters of the string:
includes in the functional key N last characters of the
string.

first N consonants of the string:
includes in the functional key N first consonants of the
string.

first N vowels of the string:
includes in the functional key N first vowels of the string.

pick characters: includes in the
functional key the characters located at a fixed position
(corresponding to the set digital/range).

exact: includes in the functional
key the full string.

substring(a,b): includes in the
functional key character according to the set index.

soundex code: generates a code
according to a standard English phonetic algorithm. This code
represents the character string that will be included in the
functional key.

metaphone code: generates a code
according to the character pronunciation. This code represents the
character string that will be included in the functional key.

double-metaphone code: generates a
code according to the character pronunciation using a new version of
the Metaphone phonetic algorithm, that produces more accurate
results than the original algorithm. This code represents the
character string that will be included in the functional key.

fingerPrintkey: generates the
functional key from a string value through the following sequential process:

nGramkey: this algorithm is similar to the fingerPrintkey method described above.
But instead of using whitespace separated tokens, it uses n-grams,
where the n can be specified by the user. This
method generates the functional key from a string value through the
following sequential process:

colognPhonetic: a soundex
phonetic algorithm optimized for the German language. It encodes a
string into a Cologne phonetic value. This code represents the
character string that will be included in the functional key.

Value

Set the algorithm value, where applicable.

Post-Algorithm

If required, select the relevant matching algorithm from the
list:

use default value (string): enables
you to choose a string to replace null or empty data.

add left position character:
enables you to add a character to the left of the column.

add right position character:
enables you to add a character to the right of the column.

Value

Set the option value, where applicable.

Show help

Select this check box to display instructions on how to set
algorithms/options parameters.

Advanced settings

Hadoop Properties

Talend Studio uses a default configuration for its engine to perform
operations in a Hadoop distribution. If you need to use a custom configuration in a specific
situation, complete this table with the property or properties to be customized. Then at
runtime, the customized property or properties will override those default ones.

For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you
are using or see Apache’s Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:

This table will not be available when you select the Link with a tGenKeyHadoop check
box.

Keep data in Hadoop

Select this check box to keep the data processed by this component
in the HDFS file.

If you leave this check box unselected, the component processes
data and then retrieves it from the HDFS file and output it in the
Job flow.

Use existing HDFS file

Select this check box to enable the component to directly process
the data located in a given directory in HDFS. When this check box
is selected, this component can act as a start component in your
Job.

HDFS file URI: set the URI of the
HDFS file holding the data you want to process.

Field delimiter: set the
character used as a field delimiter in the HDFS file.

If you leave this check box unselected, this component receives
the data flow to be processed and loads it into an HDFS file
specified in the HDFS directory
field.

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Dynamic settings

Click the [+] button to add a row
in the table and fill the Code
field with a context variable to choose your HDFS connection
dynamically from multiple connections planned in your Job.

The Dynamic settings table is available only when the
Use an existing connection check box is selected in the
Basic settings view. Once a dynamic parameter is
defined, the Component List box in the Basic settings view becomes unusable.

For more information on Dynamic settings and context
variables, see Talend Studio User Guide.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component can be a start or an intermediary component to read
and process given data in HDFS. This component can be used with
other components, such as tMatchGroupHadoop, in order to create a blocking key
for the data stored in HDFS.

Limitation/prerequisite

You need to use Linux to run the Job using this component.