These properties are used to configure tConvertType running in
the Standard Job framework.

The Standard
tConvertType component belongs to the Processing
family.

The component in this framework is available in all Talend
products.

Basic settings

Advanced settings

Global Variables

Usage

In this scenario, the input schemas for the input delimited file are stored in the
repository, you can simply drag and drop the relevant file node from Repository – Metadata –
File delimited onto the design workspace to
automatically retrieve the tFileInputDelimited
component’s setting. For more information, see
Talend Studio User
Guide.

1. Drop the following components from the Palette onto the design workspace: tConvertType, tMap, and
   tLogRow.
2. In the Repository tree view, expand Metadata and from File delimited
   drag the relevant node, JavaTypes in this
   scenario, to the design workspace.
   The Components dialog box
   displays.
3. From the component list, select tFileInputDelimited and click Ok.
   A tFileInputComponent called
   Java types displays in the design workspace.
4. Connect the components using Row > Main
   links.
   

1. Double-click tFileInputDelimited to enter
   its Basic settings view.
2. Set Property Type to Repository since the file details are stored in
   the repository. The fields to follow are pre-defined using the fetched
   data.
   

   The input file used in this scenario is called input.
   It is a text file that holds string, integer, and float java types.
   

   Fill in all other fields as needed. For more information, see tFileInputDelimited. In this scenario, the header and the
   footer are not set and there is no limit for the number of processed
   rows.
3. Click Edit schema to describe the data
   structure of this input file. In this scenario, the schema is made of three
   columns, StringtoInteger, IntegerField, and
   FloatToInteger.
   

4. Click Ok to close the dialog box.
5. Double-click tConvertType to enter its
   Basic settings view.
   

6. Set Schema Type to Built in, and click Sync columns
   to automatically retrieve the columns from the tFileInputDelimited component.
7. Click Edit schema to describe manually
   the data structure of this processing component.
   

   In this scenario, we want to convert a string type data into an integer
   type and a float type data into an integer type.
   Click OK to close the Schema of tConvertType dialog box.
8. Double-click tMap to open the Map
   editor.
   The Map editor displays the input metadata of the
   tFileInputDelimited component
   

9. In the Schema editor panel of the Map
   editor, click the plus button of the output table to add two rows and name
   them to StringToInteger and
   Sum.
10. In the Map editor, drag the StringToInteger row from
    the input table to the StringToInteger row in the
    output table.
11. In the Map editor, drag each of the IntegerField and
    the FloatToInteger rows from the input table to the
    Sum row in the output table and click OK to close the Map editor.
    

12. In the design workspace, select tLogRow
    and click the Component tab to define its
    basic settings. For more information, see tLogRow.

1. Press Ctrl+S to save the Job.
2. Press F6 to execute it.
   

   The string type data is converted into an integer type and displayed in
   the StringToInteger column on the console. The float
   type data is converted into an integer and added to the
   IntegerField value to give the addition result in
   the Sum column on the console.

These properties are used to configure tConvertType running in the MapReduce Job framework.

The MapReduce
tConvertType component belongs to the Processing family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.

Basic settings

Global Variables

Usage

1. In the Repository tree view, right-click the Job you have created in
   the earlier scenario to open its contextual menu and select Edit properties.
   Then the Edit properties dialog box is displayed. Note that the Job must
   be closed before you are able to make any changes in this dialog box.
   This dialog box looks like the image below:

   

   Note that you can change the Job name as well as the other
   descriptive information about the Job from this dialog box.
2. From the Job Type list, select
   Big Data Batch. Then a Map/Reduce Job
   using the same name appears under the Big Data
   Batch sub-node of the Job
   Design node.

1. Double-click this new Map/Reduce Job to open it in the workspace. The
   Map/Reduce components’ Palette is opened
   accordingly and in the workspace, the crossed-out components, if any,
   indicate that those components do not have the Map/Reduce version.
2. Right-click each of those components in question and select Delete to remove them from the workspace.
3. Drop a tHDFSInput component in the
   workspace. The tHDFSInput component reads
   data from the Hadoop distribution to be used.
   If from scratch, you have to drop tConvertType, tMap and
   tLogRow, too.
4. Connect tHDFSInput to tConvertType using the Row
   > Main link and accept to get the schema of tConvertType.

1. Click Run to open its view and then click the
   Hadoop Configuration tab to display its
   view for configuring the Hadoop connection for this Job.
2. From the Property type list,
   select Built-in. If you have created the
   connection to be used in Repository, then
   select Repository and thus the Studio will
   reuse that set of connection information for this Job.
3. In the Version area, select the
   Hadoop distribution to be used and its version.

   • If you use Google Cloud Dataproc, see Google Cloud Dataproc.

   • If you cannot
     find the Cloudera version to be used from this drop-down list, you can add your distribution
     via some dynamic distribution settings in the Studio.

   • If you cannot find from the list the distribution corresponding to
     yours, select Custom so as to connect to a
     Hadoop distribution not officially supported in the Studio. For a
     step-by-step example about how to use this
     Custom option, see Connecting to a custom Hadoop distribution.

4. In the Name node field, enter the location of
   the master node, the NameNode, of the distribution to be used. For example,
   hdfs://tal-qa113.talend.lan:8020.

   • If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
     client will take care of the rest on the fly for creating the connection. The
     MapR client must be properly installed. For further information about how to set
     up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client

   • If you are using WebHDFS, the location should be
     webhdfs://masternode:portnumber; WebHDFS with SSL is not
     supported yet.

5. In the Resource Manager field,
   enter the location of the ResourceManager of your distribution. For example,
   tal-qa114.talend.lan:8050.

   • Then you can continue to set the following parameters depending on the
     configuration of the Hadoop cluster to be used (if you leave the check
     box of a parameter clear, then at runtime, the configuration about this
     parameter in the Hadoop cluster to be used will be ignored):

     • Select the Set resourcemanager
       scheduler address check box and enter the Scheduler address in
       the field that appears.

     • Select the Set jobhistory
       address check box and enter the location of the JobHistory
       server of the Hadoop cluster to be used. This allows the metrics information of
       the current Job to be stored in that JobHistory server.

     • Select the Set staging
       directory check box and enter this directory defined in your
       Hadoop cluster for temporary files created by running programs. Typically, this
       directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files
       such as yarn-site.xml or mapred-site.xml of your distribution.

     • Select the Use datanode hostname check box to allow the
       Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname
       property to true. When connecting to a
       S3N filesystem, you must select this check box.

6. 
   If you are accessing the Hadoop cluster running
   with Kerberos security, select this check box, then, enter the Kerberos
   principal name for the NameNode in the field displayed. This enables you to use
   your user name to authenticate against the credentials stored in Kerberos.
   

   • If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the
     MapR ticket authentication configuration in addition or as an alternative by following
     the explanation in Connecting to a security-enabled MapR.

     Keep in mind that this configuration generates a new MapR security ticket for the username
     defined in the Job in each execution. If you need to reuse an existing ticket issued for the
     same username, leave both the Force MapR ticket
     authentication check box and the Use Kerberos
     authentication check box clear, and then MapR should be able to automatically
     find that ticket on the fly.

   In addition, since this component performs Map/Reduce computations, you
   also need to authenticate the related services such as the Job history server and
   the Resource manager or Jobtracker depending on your distribution in the
   corresponding field. These principals can be found in the configuration files of
   your distribution. For example, in a CDH4 distribution, the Resource manager
   principal is set in the yarn-site.xml file and the Job history
   principal in the mapred-site.xml file.

   If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains
   pairs of Kerberos principals and encrypted keys. You need to enter the principal to
   be used in the Principal field and the access
   path to the keytab file itself in the Keytab
   field. This keytab file must be stored in the machine in which your Job actually
   runs, for example, on a Talend
   Jobserver.

   Note that the user that executes a keytab-enabled Job is not necessarily
   the one a principal designates but must have the right to read the keytab file being
   used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this
   situation, ensure that user1 has the right to read the keytab
   file to be used.

7. In the User name field, enter the login user
   name for your distribution. If you leave it empty, the user name of the machine
   hosting the Studio will be used.
8. In the Temp folder field, enter the path in
   HDFS to the folder where you store the temporary files generated during
   Map/Reduce computations.
9. 
   Leave the default value of the Path separator in
   server as it is, unless you have changed the separator used by your
   Hadoop distribution’s host machine for its PATH variable or in other words, that
   separator is not a colon (:). In that situation, you must change this value to the
   one you are using in that host.
   
10. Leave the Clear temporary folder check box
    selected, unless you want to keep those temporary files.
11. Leave the Compress intermediate map output to reduce
    network traffic check box selected, so as to spend shorter time
    to transfer the mapper task partitions to the multiple reducers.
    However, if the data transfer in the Job is negligible, it is recommended to
    clear this check box to deactivate the compression step, because this
    compression consumes extra CPU resources.
12. If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
    properties to be customized. Then at runtime, these changes will override the
    corresponding default properties used by the Studio for its Hadoop
    engine.
    For further information about the properties required by Hadoop, see Apache’s
    Hadoop documentation on http://hadoop.apache.org, or
    the documentation of the Hadoop distribution you need to use.
13. 
    If the HDFS transparent encryption has been enabled in your cluster, select
    the Setup HDFS encryption configurations check
    box and in the HDFS encryption key provider field
    that is displayed, enter the location of the KMS proxy.
    

    For further information about the HDFS transparent encryption and its KMS proxy, see Transparent Encryption in HDFS.

14. You can tune the map and reduce computations by
    selecting the Set memory check box to set proper memory allocations
    for the computations to be performed by the Hadoop system.

    The memory parameters to be set are Map (in Mb),
    Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate
    memory to the map and the reduce computations and the ApplicationMaster of YARN.

    For further information about the Resource Manager, its scheduler and the
    ApplicationMaster, see YARN’s documentation such as http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/.

    For further information about how to determine YARN and MapReduce memory configuration
    settings, see the documentation of the distribution you are using, such as the following
    link provided by Hortonworks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html.

15. 
    If you are using Cloudera V5.5+, you can select the Use Cloudera Navigator check box to enable the Cloudera Navigator
    of your distribution to trace your Job lineage to the component level, including the
    schema changes between components.
    

    With this option activated, you need to set the following parameters:

    • Username and Password: this is the credentials you use to connect to your Cloudera
      Navigator.

    • Cloudera Navigator URL : enter the location of the
      Cloudera Navigator to be connected to.

    • Cloudera Navigator Metadata URL: enter the location
      of the Navigator Metadata.

    • Activate the autocommit option: select this check box
      to make Cloudera Navigator generate the lineage of the current Job at the end of the
      execution of this Job.

      Since this option actually forces Cloudera Navigator to generate lineages of
      all its available entities such as HDFS files and directories, Hive queries or Pig
      scripts, it is not recommended for the production environment because it will slow the
      Job.

    • Kill the job if Cloudera Navigator fails: select this check
      box to stop the execution of the Job when the connection to your Cloudera Navigator fails.

      Otherwise, leave it clear to allow your Job to continue to run.

    • Disable SSL validation: select this check box to
      make your Job to connect to Cloudera Navigator without the SSL validation
      process.

      This feature is meant to facilitate the test of your Job but is not
      recommended to be used in a production cluster.

16. 
    If you are using Hortonworks Data Platform V2.4.0 onwards and you have
    installed Atlas in your cluster, you can select the Use
    Atlas check box to enable Job lineage to the component level, including the
    schema changes between components.
    

    With this option activated, you need to set the following parameters:

    • Atlas URL: enter the location of the Atlas to be
      connected to. It is often http://name_of_your_atlas_node:port

    • Die on error: select this check box to stop the Job
      execution when Atlas-related issues occur, such as connection issues to Atlas.

      Otherwise, leave it clear to allow your Job to continue to run.

    In the Username and Password fields, enter the authentication information for access to
    Atlas.

Configuring tHDFSInput

Reviewing the transformation component

Double-click tConvertType to open its
Component view.

This component keeps its both Basic
settings and Advanced
settings used by the original Job. Therefore, as its original
one does, it converts the string type and the float type into
integer.

Reviewing tMap

Double-click tMap to open its editor. The
mapping configuration remains as it is in the original Job, that is to say,
to output the converted StringtoInteger
column and to make the sum of the IntegerField and the FloatToInteger columns.

1. If you want to configure the presentation mode on its Component view, double-click the tLogRow component of interest to open the
   Component view and in the Mode area, then, select the Table (print values in cells of a table) option.
2. Press F6 to run this Job.

During the execution, the Run view is
automatically opened, where you can read how this Job progresses, including the
status of the Map/Reduce computation the Job is performing.

In the meantime in the workspace, progress bars automatically appear under the
components performing Map/Reduce to graphically show the same status of the
Map/Reduce computation.

If you need to obtain more details about the Job, it is recommended to use the web
console of the Jobtracker provided by the Hadoop distribution you are using.

These properties are used to configure tConvertType running in the Spark Batch Job framework.

The Spark Batch
tConvertType component belongs to the Processing family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

Basic settings

Usage

These properties are used to configure tConvertType running in the Spark Streaming Job framework.

The Spark Streaming
tConvertType component belongs to the Processing family.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

Basic settings

Usage