tJavaMR

Warning

This component will be available in the Palette of
Talend Studio on the condition that you have subscribed to one of
the Talend
solutions with Big Data.

tJavaMR properties

Component family	Custom Code
Function	tJavaMR enables you to enter personalized MapReduce code in order to integrate it in Talend program. You can execute this code only once. This component appears only when you are creating a Map/Reduce Job.
Purpose	tJavaMR makes it possible to extend the functionalities of a Talend Job through writing custom map and reduce methods.
Basic settings	Schema and Edit Schema	A schema is a row description. It defines the number of fields to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.
		Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.
		Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.
		Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.
	Map only	Select this check box to edit and use a custom mapper only. In that situation, only the Map code editing field and the Map advanced code area are available.
	Map code	Enter the body of the map method you want to execute. This component automatically defines the other parts of the map method and if you are not using Map only, tJavaMR also automatically uses the column names you specify in the mrKeyStruct and the mrValueStruct tables to instantiate the key/value pairs intermediate between the map and the reduce phases. For example, you put word as column name in the mrKeyStruct table; then when you write the code, you need to write mrKey.word to represent the corresponding key instance and at runtime, this instance is automatically constructed to be `mrKey_component_ID.word` such as mrKey_tJavaMR_1.word. Note that the text displayed above the Map code editing field indicates the parameters you can directly use in writing the code, such as mrKey, mrValue or outputRow. For further information about a map method and the intermediate key/value pairs it outputs, see https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/Mapper.html. For further information about Java functions syntax specific to Talend, see Talend Studio Help Contents (Help > Developer Guide > API Reference). For a complete Java reference, check http://docs.oracle.com/javaee/6/api/.
	mrKeyStruct and mrValueStruct	In these two tables, add the columns you want to use to compose the key/value pairs required by MapReduce computations.
	Reduce code	Enter the body of the reduce method you want to execute according to the task you need to perform. This component automatically defines the shuffle and sort phases and the other parts of the reduce method and uses the column names you specify in the mrKeyStruct and the mrValueStruct tables to instantiate the key/value pairs that have been shuffled and sorted. For example, you put word as column name in the mrKeyStruct table, then when you write the code, you need to write key.word to represent the corresponding key instance and at runtime, this instance is automatically constructed to be `key_component_ID.word` such as key_tJavaMR_1.word; you put count in the mrValueStruct table, then you have to write values.count to define the corresponding value instance and at runtime, this instance is constructed to be `values_component_ID.count` such as values_tJavaMR_1.count. Note that the text displayed above the Reduce code editing field indicates the parameters you can directly use in writing the code. For further information about a reduce method and its related phases and key/value pairs, see https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/Reducer.html. For further information about Java functions syntax specific to Talend, see Talend Studio Help Contents (Help > Developer Guide > API Reference). For a complete Java reference, check http://docs.oracle.com/javaee/6/api/.
Advanced settings	Map advanced code	This area allows you to define the classes, variables and methods that you want to use along with the map method defined in the Basic settings view. Note that the advanced code is not required for using tJavaMR. Three fields are available for this purpose: Implement the prepare code: select this check box and in the displayed field, define variables, methods and inner classes to be nested in the body of the public class of this component’s mapper. Implement the configure method: select this check box and in the displayed field, define the body of the configure method of this component’s mapper. Implement the close method: select this check box and in the displayed field, define the body of the close method of this component’s mapper.
	Reduce advanced code	This area allows you to define the classes, variables and methods that you want to use along with the reduce method defined in the Basic settings view. Note that the advanced code is not required for using tJavaMR. Three fields are available for this purpose: Implement the prepare code: select this check box and in the displayed field, define variables, methods and inner classes to be nested in the body of the public class of this component’s reducer. Implement the configure method: select this check box and in the displayed field, define the body of the configure method of this component’s reducer. Implement the close method: select this check box and in the displayed field, define the body of the close method of this component’s reducer.
Global Variables	ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box. A Flow variable functions during the execution of a component while an After variable functions after the execution of the component. To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it. For further information about variables, see Talend Studio User Guide.
Usage	Once a Map/Reduce Job is opened in the workspace, tJavaMR appears in the Palette of the Studio. It is used as an intermediate step in a Map/Reduce Job. Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs, and non Map/Reduce Jobs.
Hadoop Connection	You need to use the Hadoop Configuration tab in the Run view to define the connection to a given Hadoop distribution for the whole Job. This connection is effective on a per-Job basis.

Scenario: Counting words using custom map and reduce code

Inspired by the MapReduce example explained in Apache’s documentation on http://wiki.apache.org/hadoop/WordCount, this scenario demonstrates how to
use tJavaMR to create a MapReduce program to count
words.

The sample data to be used in this scenario reads as follows:

Hello world goodbye world
Hello hadoop bye Hadoop

1 2	Hello world goodbye world Hello hadoop bye Hadoop

Before starting to replicate this scenario, ensure that you have appropriate rights
and permissions to access the Hadoop distribution to be used. Then proceed as
follows:

Linking components

In the Integration perspective
of the Studio, create an empty Map/Reduce Job from the
Job Designs node in the Repository tree view.

For further information about how to create a Map/Reduce Job, see
Talend Big Data Getting Started Guide.
Drop a tHDFSInput component, a tJavaMR component, and a tHDFSOutput component in the workspace.

The tHDFSInput component reads data from
the Hadoop distribution to be used and the tHDFSOutput component writes processed data into a that
distribution.
Connect these components using the Row >
Main link.

Setting up Hadoop connection

Click Run to open its view and then click the
Hadoop Configuration tab to display its
view for configuring the Hadoop connection for this Job.

This view looks like the image below:
From the Property type list, select Built-in. If you have created the connection to be
used in Repository, then select Repository and thus the Studio will reuse that set of
connection information for this Job.

For further information about how to create an Hadoop connection in
Repository, see the chapter describing the Hadoop
cluster node of the Talend Big Data Getting Started Guide.
In the Version area, select the Hadoop
distribution to be used and its version. If you cannot find from the list the
distribution corresponding to yours, select Custom so as to connect to a Hadoop distribution not officially
supported in the Studio.

For a step-by-step example about how to use this Custom option, see Connecting to a custom Hadoop distribution.

Along with the evolution of Hadoop, please note the
following changes:
- If you use Hortonworks Data Platform
  V2.2, the configuration files of your cluster might be using
  environment variables such as ${hdp.version}. If this is your situation, you need to set
  the mapreduce.application.framework.path property in the
  Hadoop properties table with the path
  value explicitly pointing to the MapReduce framework archive of your
  cluster. For
  example:
  
  mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework
  
  1
  
  mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework
- If you use Hortonworks Data Platform
  V2.0.0, the type of the operating system for running the
  distribution and a Talend Job must be the same,
  such as Windows or Linux. Otherwise, you have to use Talend Jobserver to execute the Job in the same
  type of operating system in which the Hortonworks
  Data Platform V2.0.0 distribution you are using is run. For
  further information about Talend Jobserver, see
  Talend
  Installation and Upgrade Guide.
In the Name node field, enter the location of
the master node, the NameNode, of the distribution to be used. For example,
hdfs://tal-qa113.talend.lan:8020.

If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
client will take care of the rest on the fly for creating the connection. The
MapR client must be properly installed. For further information about how to set
up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client
In the Job tracker field, enter the location
of the JobTracker of your distribution. For example, tal-qa114.talend.lan:8050.

Note that the notion Job in this term JobTracker designates the MR or the
MapReduce jobs described in Apache’s documentation on http://hadoop.apache.org/.

If you use YARN in your Hadoop cluster such as Hortonworks Data Platform V2.0.0 or Cloudera CDH4.3 + (YARN mode), you need to specify the location
of the Resource Manager instead of the
Jobtracker. Then you can continue to set the following parameters depending on
the configuration of the Hadoop cluster to be used (if you leave the check box
of a parameter clear, then at runtime, the configuration about this parameter in
the Hadoop cluster to be used will be ignored ):
- Select the Set resourcemanager scheduler
  address check box and enter the Scheduler address in
  the field that appears.
- Select the Set jobhistory address
  check box and enter the location of the JobHistory server of the
  Hadoop cluster to be used. This allows the metrics information of
  the current Job to be stored in that JobHistory server.
- Select the Set staging directory
  check box and enter this directory defined in your Hadoop cluster
  for temporary files created by running programs. Typically, this
  directory can be found under the yarn.app.mapreduce.am.staging-dir property in the
  configuration files such as yarn-site.xml or mapred-site.xml of your distribution.
- Select the Use datanode hostname
  check box to allow the Job to access datanodes via their hostnames.
  This actually sets the dfs.client.use.datanode.hostname property to
  true. When connecting to a
  S3N filesystem, you must select this check box.
If you are accessing the Hadoop cluster running with Kerberos security, select this check
box, then, enter the Kerberos principal name for the NameNode in the field displayed. This
enables you to use your user name to authenticate against the credentials stored in
Kerberos.

In addition, since this component performs Map/Reduce computations, you also need to
authenticate the related services such as the Job history server and the Resource manager or
Jobtracker depending on your distribution in the corresponding field. These principals can
be found in the configuration files of your distribution. For example, in a CDH4
distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

If you need to use a Kerberos keytab file to log in, select Use a
keytab to authenticate. A keytab file contains pairs of Kerberos principals
and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the
Keytab field.

Note that the user that executes a keytab-enabled Job is not necessarily the one a
principal designates but must have the right to read the keytab file being used. For
example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.
In the User name field, enter the login user
name for your distribution. If you leave it empty, the user name of the machine
hosting the Studio will be used.
In the Temp folder field, enter the path in
HDFS to the folder where you store the temporary files generated during
Map/Reduce computations.
Leave the default value of the Path separator in server as
it is, unless you have changed the separator used by your Hadoop distribution’s host machine
for its PATH variable or in other words, that separator is not a colon (:). In that
situation, you must change this value to the one you are using in that host.
Leave the Clear temporary folder check box
selected, unless you want to keep those temporary files.
Leave the Compress intermediate map output to reduce
network traffic check box selected, so as to spend shorter time
to transfer the mapper task partitions to the multiple reducers.

However, if the data transfer in the Job is negligible, it is recommended to
clear this check box to deactivate the compression step, because this
compression consumes extra CPU resources.
If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
properties to be customized. Then at runtime, these changes will override the
corresponding default properties used by the Studio for its Hadoop
engine.

For further information about the properties required by Hadoop, see Apache’s
Hadoop documentation on http://hadoop.apache.org, or
the documentation of the Hadoop distribution you need to use.
If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks
Data Platform V1.3, you need to set proper memory allocations for the map and reduce
computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred
job map memory mb and the Mapred job reduce memory
mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the
computations.

If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and
ApplicationMaster (in Mb), accordingly. These fields
allow you to dynamically allocate memory to the map and the reduce computations and the
ApplicationMaster of YARN.

For further information about this Hadoop
Configuration tab, see the section describing how to configure the Hadoop
connection for a Talend Map/Reduce Job of the Talend Big Data Getting Started Guide.

For further information about the Resource Manager, its scheduler and the
ApplicationMaster, see YARN’s documentation such as http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/.

For further information about how to determine YARN and MapReduce memory configuration
settings, see the documentation of the distribution you are using, such as the following
link provided by Hortonworks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html.

Configuring tHDFSInput

Double-click tHDFSInput to open its
Component view.
Click the button next to Edit
schema to open the schema editor.
Click the button once to add one row and in the Column column, rename it, for example, to
record.
Click OK to validate these changes and
accept the propagation prompted by the pop-up dialog box.
In the Folder/File field, enter the path,
or browse to the source file you need the Job to read.

If this file is not in the HDFS system to be used, you have to place it in
that HDFS, for example, using tFileInputDelimited and tHDFSOutput in a Standard
Job. For further information about these components, see tFileInputDelimited and tHDFSOutput.

Creating the MapReduce program

Double-click tJavaMR to open its
Component view.
Under the mrKeyStruct table, click the button once to add one row.
Rename that row to word_mr. This is the
key part of the key/value pair to be used by the Map/Reduce program being
created. In the map method, you need to write mrKey.word_mr to represent the keys to be outputted to a
reducer.
Under the mrValueStruct table, click the button once to add one row.
Rename that row to count_mr. This is
the value part of the above-mentioned key/value pair. In the map method, you
need to write mrValue.count_mr to
represent the values to be outputted to a reducer.
Click the button next to Edit
schema to open the schema editor.
On the side of the schema of tJavaMR,
click the button to add two columns and name them to word_output and count_output, respectively. This defines the structure of
the data to be outputted.
In the Type column, select Integer for count_output.

In the Map code editing field, edit the
body of the map method. In this example, the code is as follows:

String line = value.record;
java.util.StringTokenizer tokenizer = new java.util.StringTokenizer(line);
while(tokenizer.hasMoreTokens()) {
   mrKey.word_mr = tokenizer.nextToken().toUpperCase();
   mrValue.count_mr = 1;
   output.collect(mrKey, mrValue);
}

String line = value.record;

java.util.StringTokenizer tokenizer = new java.util.StringTokenizer(line);

while(tokenizer.hasMoreTokens()) {

mrKey.word_mr = tokenizer.nextToken().toUpperCase();

mrValue.count_mr = 1;

output.collect(mrKey, mrValue);

}

This method is used to split the input data into words, change each word
to upper case and create and output key/value pairs such as (HELLO, 1) and (WORLD,
1) to the reducer.

Note that at runtime, these pairs are automatically shuffled and sorted to
take the form of (key, list of values) before being process by
the reduce method.

In the Reduce code editing field, edit
the body of the reduce method. In this example, the code is as
follows:

int count = 0;
while(values.hasNext()){
  mrValueStruct value = values.next();
  count += value.count_mr; 
}    
outputRow.word_output = key.word_mr;
outputRow.count_output = count;
output.collect(NULL, outputRow);

int count = 0;

while(values.hasNext()){

mrValueStruct value = values.next();

count += value.count_mr;

}

outputRow.word_output = key.word_mr;

outputRow.count_output = count;

output.collect(NULL, outputRow);

This reduce method is used to make the sum of the values of the list in
each (key, list of values) pair and map the results to the
columns of the output schema.

Writing results in HDFS

Double-click tHDFSOutput to open its
Component view.
In the Folder field, enter the path, or
browse to the folder you want to write the results in.
From the Type list, select the data format for the
results to be written. In this example, select Text
file.
From the Action list, select the
operation you need to perform on the file in question. If the file already
exists, select Overwrite; otherwise, select
Create.
Select the Merge result to single file
check box and enter the path, or browse to the file you need to write the
merged output data in.
If you need to remove the source data of the merge, select Remove source dir. In this scenario, select
it.
If the file for the merged data exists, select the Override target file check box to overwrite that
file.

Executing the Job

Then you can press F6 to run this Job.

Once done, view the merged result in the web console of the HDFS system being
used.

If you need to obtain more execution information of this Job, see the web console
of the Jobtracker of that HDFS system.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 5.x

0 Comments

Inline Feedbacks

View all comments

tJavaMR – Docs for ESB 5.x