tConvertType – Docs for ESB 5.x

tConvertType

1. Drop the following components from the Palette onto the design workspace: tConvertType, tMap, and
   tLogRow.

2. In the Repository tree view, expand Metadata and from File delimited
   drag the relevant node, JavaTypes in this
   scenario, to the design workspace.

   The [Components] dialog box
   displays.

3. From the component list, select tFileInputDelimited and click Ok.

   A tFileInputComponent called
   Java types displays in the design workspace.

4. Connect the components using Row > Main
   links.

   

1. Double-click tFileInputDelimited to enter
   its Basic settings view.

2. Set Property Type to Repository since the file details are stored in
   the repository. The fields to follow are pre-defined using the fetched
   data.

   

   The input file used in this scenario is called input.
   It is a text file that holds string, integer, and float java types.

   

   Fill in all other fields as needed. For more information, see tFileInputDelimited. In this scenario, the header and the
   footer are not set and there is no limit for the number of processed
   rows.

3. Click Edit schema to describe the data
   structure of this input file. In this scenario, the schema is made of three
   columns, StringtoInteger, IntegerField, and
   FloatToInteger.

   

4. Click Ok to close the dialog box.

5. Double-click tConvertType to enter its
   Basic settings view.

   

6. Set Schema Type to Built in, and click Sync columns
   to automatically retrieve the columns from the tFileInputDelimited component.

7. Click Edit schema to describe manually
   the data structure of this processing component.

   

   In this scenario, we want to convert a string type data into an integer
   type and a float type data into an integer type.

   Click OK to close the [Schema of tConvertType] dialog box.

8. Double-click tMap to open the Map
   editor.

   The Map editor displays the input metadata of the
   tFileInputDelimited component

   

9. In the Schema editor panel of the Map
   editor, click the plus button of the output table to add two rows and name
   them to StringToInteger and
   Sum.

10. In the Map editor, drag the StringToInteger row from
    the input table to the StringToInteger row in the
    output table.

11. In the Map editor, drag each of the IntegerField and
    the FloatToInteger rows from the input table to the
    Sum row in the output table and click OK to close the Map editor.

    

12. In the design workspace, select tLogRow
    and click the Component tab to define its
    basic settings. For more information, see tLogRow.

1. Press Ctrl+S to save the Job.

2. Press F6 to execute it.

   

   The string type data is converted into an integer type and displayed in
   the StringToInteger column on the console. The float
   type data is converted into an integer and added to the
   IntegerField value to give the addition result in
   the Sum column on the console.

1. In the Repository tree view of the Integration perspective of Talend Studio, right-click the
   Job you have created in the earlier scenario to open its contextual menu and
   select Edit properties.

   Then the [Edit properties] dialog box is
   displayed. Note that the Job must be closed before you are able to make any
   changes in this dialog box.

   This dialog box looks like the image below:

   

   Note that you can change the Job name as well as the other descriptive
   information about the Job from this dialog box.

2. Click Convert to Map/Reduce Job. Then a
   Map/Reduce Job using the same name appears under the Map/Reduce Jobs sub-node of the Job
   Design node.

1. Double-click this new Map/Reduce Job to open it in the workspace. The
   Map/Reduce components’ Palette is opened
   accordingly and in the workspace, the crossed-out components, if any,
   indicate that those components do not have the Map/Reduce version.

2. Right-click each of those components in question and select Delete to remove them from the workspace.

3. Drop a tHDFSInput component in the
   workspace. The tHDFSInput component reads
   data from the Hadoop distribution to be used.

   If from scratch, you have to drop tConvertType, tMap and
   tLogRow, too.

4. Connect tHDFSInput to tConvertType using the Row
   > Main link and accept to get the schema of tConvertType.

1. Click Run to open its view and then click the
   Hadoop Configuration tab to display its
   view for configuring the Hadoop connection for this Job.

   This view looks like the image below:

   

2. From the Property type list, select Built-in. If you have created the connection to be
   used in Repository, then select Repository and thus the Studio will reuse that set of
   connection information for this Job.

   For further information about how to create an Hadoop connection in
   Repository, see the chapter describing the Hadoop
   cluster node of the Talend Big Data Getting Started Guide.

3. In the Version area, select the Hadoop
   distribution to be used and its version. If you cannot find from the list the
   distribution corresponding to yours, select Custom so as to connect to a Hadoop distribution not officially
   supported in the Studio.

   For a step-by-step example about how to use this Custom option, see Connecting to a custom Hadoop distribution.

   Along with the evolution of Hadoop, please note the
   following changes:

   • If you use Hortonworks Data Platform
     V2.2, the configuration files of your cluster might be using
     environment variables such as ${hdp.version}. If this is your situation, you need to set
     the mapreduce.application.framework.path property in the
     Hadoop properties table with the path
     value explicitly pointing to the MapReduce framework archive of your
     cluster. For
     example:

     mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework

     1	mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework

   • If you use Hortonworks Data Platform
     V2.0.0, the type of the operating system for running the
     distribution and a Talend Job must be the same,
     such as Windows or Linux. Otherwise, you have to use Talend Jobserver to execute the Job in the same
     type of operating system in which the Hortonworks
     Data Platform V2.0.0 distribution you are using is run. For
     further information about Talend Jobserver, see
     Talend
     Installation and Upgrade Guide.

4. In the Name node field, enter the location of
   the master node, the NameNode, of the distribution to be used. For example,
   hdfs://tal-qa113.talend.lan:8020.

   If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
   client will take care of the rest on the fly for creating the connection. The
   MapR client must be properly installed. For further information about how to set
   up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client

5. In the Job tracker field, enter the location
   of the JobTracker of your distribution. For example, tal-qa114.talend.lan:8050.

   Note that the notion Job in this term JobTracker designates the MR or the
   MapReduce jobs described in Apache’s documentation on http://hadoop.apache.org/.

   If you use YARN in your Hadoop cluster such as Hortonworks Data Platform V2.0.0 or Cloudera CDH4.3 + (YARN mode), you need to specify the location
   of the Resource Manager instead of the
   Jobtracker. Then you can continue to set the following parameters depending on
   the configuration of the Hadoop cluster to be used (if you leave the check box
   of a parameter clear, then at runtime, the configuration about this parameter in
   the Hadoop cluster to be used will be ignored ):

   • Select the Set resourcemanager scheduler
     address check box and enter the Scheduler address in
     the field that appears.

   • Select the Set jobhistory address
     check box and enter the location of the JobHistory server of the
     Hadoop cluster to be used. This allows the metrics information of
     the current Job to be stored in that JobHistory server.

   • Select the Set staging directory
     check box and enter this directory defined in your Hadoop cluster
     for temporary files created by running programs. Typically, this
     directory can be found under the yarn.app.mapreduce.am.staging-dir property in the
     configuration files such as yarn-site.xml or mapred-site.xml of your distribution.

   • Select the Use datanode hostname
     check box to allow the Job to access datanodes via their hostnames.
     This actually sets the dfs.client.use.datanode.hostname property to
     true. When connecting to a
     S3N filesystem, you must select this check box.

6. If you are accessing the Hadoop cluster running with Kerberos security, select this check
   box, then, enter the Kerberos principal name for the NameNode in the field displayed. This
   enables you to use your user name to authenticate against the credentials stored in
   Kerberos.

   In addition, since this component performs Map/Reduce computations, you also need to
   authenticate the related services such as the Job history server and the Resource manager or
   Jobtracker depending on your distribution in the corresponding field. These principals can
   be found in the configuration files of your distribution. For example, in a CDH4
   distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

   If you need to use a Kerberos keytab file to log in, select Use a
   keytab to authenticate. A keytab file contains pairs of Kerberos principals
   and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the
   Keytab field.

   Note that the user that executes a keytab-enabled Job is not necessarily the one a
   principal designates but must have the right to read the keytab file being used. For
   example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

7. In the User name field, enter the login user
   name for your distribution. If you leave it empty, the user name of the machine
   hosting the Studio will be used.

8. In the Temp folder field, enter the path in
   HDFS to the folder where you store the temporary files generated during
   Map/Reduce computations.

9. Leave the default value of the Path separator in server as
   it is, unless you have changed the separator used by your Hadoop distribution’s host machine
   for its PATH variable or in other words, that separator is not a colon (:). In that
   situation, you must change this value to the one you are using in that host.

10. Leave the Clear temporary folder check box
    selected, unless you want to keep those temporary files.

11. Leave the Compress intermediate map output to reduce
    network traffic check box selected, so as to spend shorter time
    to transfer the mapper task partitions to the multiple reducers.

    However, if the data transfer in the Job is negligible, it is recommended to
    clear this check box to deactivate the compression step, because this
    compression consumes extra CPU resources.

12. If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
    properties to be customized. Then at runtime, these changes will override the
    corresponding default properties used by the Studio for its Hadoop
    engine.

    For further information about the properties required by Hadoop, see Apache’s
    Hadoop documentation on http://hadoop.apache.org, or
    the documentation of the Hadoop distribution you need to use.

13. If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks
    Data Platform V1.3, you need to set proper memory allocations for the map and reduce
    computations to be performed by the Hadoop system.

    In that situation, you need to enter the values you need in the Mapred
    job map memory mb and the Mapred job reduce memory
    mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the
    computations.

    If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and
    ApplicationMaster (in Mb), accordingly. These fields
    allow you to dynamically allocate memory to the map and the reduce computations and the
    ApplicationMaster of YARN.

Configuring tHDFSInput

1. Double-click tHDFSInput to open its
   Component view.

   

2. Click the button next to Edit
   schema to verify that the schema received in the earlier
   steps is properly defined.

   

   Note that if you are creating this Job from scratch, you need to click the button to manually define the schema; otherwise, if the
   schema has been defined in Repository, you
   can select the Repository option from the
   Schema list in the Basic settings view to reuse it. For further
   information about how to define a schema in Repository, see the chapter describing metadata management
   in the Talend Studio User Guide or the chapter describing the
   Hadoop cluster node in Repository of Talend Big Data Getting Started Guide.

3. If you make changes in the schema, click OK to validate these changes and accept the propagation
   prompted by the pop-up dialog box.

4. In the Folder/File field, enter the path,
   or browse to the source file you need the Job to read.

   If this file is not in the HDFS system to be used, you have to place it in
   that HDFS, for example, using tFileInputDelimited and tHDFSOutput in a Standard
   Job.

Reviewing the transformation component

• Double-click tConvertType to open its
  Component view.

  

  This component keeps its both Basic
  settings and Advanced
  settings used by the original Job. Therefore, as its original
  one does, it converts the string type and the float type into
  integer.

Reviewing tMap

• Double-click tMap to open its editor. The
  mapping configuration remains as it is in the original Job, that is to say,
  to output the converted StringtoInteger
  column and to make the sum of the IntegerField and the FloatToInteger columns.

  

1. If you want to configure the presentation mode on its Component view, double-click the tLogRow component of interest to open the
   Component view and in the Mode area, then, select the Table (print values in cells of a table) option.

2. Press F6 to run this Job.