tMahoutClustering – Docs for ESB 5.x

tMahoutClustering

1. Drop the following components from the Palette onto the design workspace: tMahoutClustering, three tMap, two tAggregateRow and
   two tHDFSOutput components.

   

2. Set the components as shown in the capture and connect them together using
   Main links.

1. Click Run to open its view and then click the
   Hadoop Configuration tab to display its
   view for configuring the Hadoop connection for this Job.

   This view looks like the image below:

   

2. From the Property type list, select Built-in. If you have created the connection to be
   used in Repository, then select Repository and thus the Studio will reuse that set of
   connection information for this Job.

   For further information about how to create an Hadoop connection in
   Repository, see the chapter describing the Hadoop
   cluster node of the Talend Big Data Getting Started Guide.

3. In the Version area, select the Hadoop
   distribution to be used and its version. If you cannot find from the list the
   distribution corresponding to yours, select Custom so as to connect to a Hadoop distribution not officially
   supported in the Studio.

   For a step-by-step example about how to use this Custom option, see Connecting to a custom Hadoop distribution.

   Along with the evolution of Hadoop, please note the
   following changes:

   • If you use Hortonworks Data Platform
     V2.2, the configuration files of your cluster might be using
     environment variables such as ${hdp.version}. If this is your situation, you need to set
     the mapreduce.application.framework.path property in the
     Hadoop properties table with the path
     value explicitly pointing to the MapReduce framework archive of your
     cluster. For
     example:

     mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework

     1	mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework

   • If you use Hortonworks Data Platform
     V2.0.0, the type of the operating system for running the
     distribution and a Talend Job must be the same,
     such as Windows or Linux. Otherwise, you have to use Talend Jobserver to execute the Job in the same
     type of operating system in which the Hortonworks
     Data Platform V2.0.0 distribution you are using is run. For
     further information about Talend Jobserver, see
     Talend
     Installation and Upgrade Guide.

4. In the Name node field, enter the location of
   the master node, the NameNode, of the distribution to be used. For example,
   hdfs://tal-qa113.talend.lan:8020.

   If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
   client will take care of the rest on the fly for creating the connection. The
   MapR client must be properly installed. For further information about how to set
   up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client

5. In the Job tracker field, enter the location
   of the JobTracker of your distribution. For example, tal-qa114.talend.lan:8050.

   Note that the notion Job in this term JobTracker designates the MR or the
   MapReduce jobs described in Apache’s documentation on http://hadoop.apache.org/.

   If you use YARN in your Hadoop cluster such as Hortonworks Data Platform V2.0.0 or Cloudera CDH4.3 + (YARN mode), you need to specify the location
   of the Resource Manager instead of the
   Jobtracker. Then you can continue to set the following parameters depending on
   the configuration of the Hadoop cluster to be used (if you leave the check box
   of a parameter clear, then at runtime, the configuration about this parameter in
   the Hadoop cluster to be used will be ignored ):

   • Select the Set resourcemanager scheduler
     address check box and enter the Scheduler address in
     the field that appears.

   • Select the Set jobhistory address
     check box and enter the location of the JobHistory server of the
     Hadoop cluster to be used. This allows the metrics information of
     the current Job to be stored in that JobHistory server.

   • Select the Set staging directory
     check box and enter this directory defined in your Hadoop cluster
     for temporary files created by running programs. Typically, this
     directory can be found under the yarn.app.mapreduce.am.staging-dir property in the
     configuration files such as yarn-site.xml or mapred-site.xml of your distribution.

   • Select the Use datanode hostname
     check box to allow the Job to access datanodes via their hostnames.
     This actually sets the dfs.client.use.datanode.hostname property to
     true. When connecting to a
     S3N filesystem, you must select this check box.

6. If you are accessing the Hadoop cluster running with Kerberos security, select this check
   box, then, enter the Kerberos principal name for the NameNode in the field displayed. This
   enables you to use your user name to authenticate against the credentials stored in
   Kerberos.

   In addition, since this component performs Map/Reduce computations, you also need to
   authenticate the related services such as the Job history server and the Resource manager or
   Jobtracker depending on your distribution in the corresponding field. These principals can
   be found in the configuration files of your distribution. For example, in a CDH4
   distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

   If you need to use a Kerberos keytab file to log in, select Use a
   keytab to authenticate. A keytab file contains pairs of Kerberos principals
   and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the
   Keytab field.

   Note that the user that executes a keytab-enabled Job is not necessarily the one a
   principal designates but must have the right to read the keytab file being used. For
   example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

7. In the User name field, enter the login user
   name for your distribution. If you leave it empty, the user name of the machine
   hosting the Studio will be used.

8. In the Temp folder field, enter the path in
   HDFS to the folder where you store the temporary files generated during
   Map/Reduce computations.

9. Leave the default value of the Path separator in server as
   it is, unless you have changed the separator used by your Hadoop distribution’s host machine
   for its PATH variable or in other words, that separator is not a colon (:). In that
   situation, you must change this value to the one you are using in that host.

10. Leave the Clear temporary folder check box
    selected, unless you want to keep those temporary files.

11. Leave the Compress intermediate map output to reduce
    network traffic check box selected, so as to spend shorter time
    to transfer the mapper task partitions to the multiple reducers.

    However, if the data transfer in the Job is negligible, it is recommended to
    clear this check box to deactivate the compression step, because this
    compression consumes extra CPU resources.

12. If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
    properties to be customized. Then at runtime, these changes will override the
    corresponding default properties used by the Studio for its Hadoop
    engine.

    For further information about the properties required by Hadoop, see Apache’s
    Hadoop documentation on http://hadoop.apache.org, or
    the documentation of the Hadoop distribution you need to use.

13. If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks
    Data Platform V1.3, you need to set proper memory allocations for the map and reduce
    computations to be performed by the Hadoop system.

    In that situation, you need to enter the values you need in the Mapred
    job map memory mb and the Mapred job reduce memory
    mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the
    computations.

    If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and
    ApplicationMaster (in Mb), accordingly. These fields
    allow you to dynamically allocate memory to the map and the reduce computations and the
    ApplicationMaster of YARN.

1. Double-click tMahoutClustering to open
   its Component view.

   

2. From the Schema list, select Built-In and then click the […] button next to Edit
   Schema and describe the data structure in the input
   file.

   

3. Add eight rows to the schema dialog box and define the input data as shown
   in the above capture.

   The component has one read-only column,
   clusterID.

4. Click OK.

5. In the File Configuration area:

   • Click the […] button next to
     the Input HDFS file and browse to
     the HDFS file on the Hadoop system that holds the input numerical
     data you want to cluster.

   • Set the field separator used to separate the columns in the
     clustered data.

   • In the Cluster columns table, add
     rows to the table and click in each row to select a column from the
     input schema.

6. In the Clustering Configuration
   area:

   • From the Clustering Type list,
     select what algorithm you want to use to cluster the numerical data,
     Fuzzy K-means in this
     example.

   • From the Distance Measure list,
     select the distance measure you want to use for clustering.

   • In the Number of clusters field,
     enter 3.

   • Leave the values in Max
     iterations and Convergence
     delta as they are.

1. Double-click tMap to open the Map Editor.

   

2. Drop the Region and the
   clusterID columns to the first output table that
   corresponds to the first tAggregateRow
   component.

   Drop the Channel and the
   clusterID columns to the second output table that
   corresponds to the second tAggregateRow
   component.

   Use the Schema editor section at the
   bottom of the editor to add necessary lines to the output tables.

3. Click OK to validate changes.

1. Double-click the first tAggregateRow to
   display its Basic settings view and define
   the component properties.

   

2. Click the […] button next to Edit schema and define the output flow.

   

3. Move the columns in the input schema to the output schema and then use the
   [+] button to add a new column in the
   output schema. Call it count.

   When done, click OK to close the dialog
   box.

4. In the Group by section, click the plus
   button to add an many lines as needed. Here you can define the group-by
   values.

   • Click in the first Output column
     row and select the output column that will hold the aggregated data,
     the region column in this example.

   • Click in the first Input column
     position row and select the input column from which
     you want to collect the values to be aggregated, the
     region column in this example.

5. In the Operations section, click the plus
   button to add rows for the columns that will hold the aggregated data. Here
   you can define the calculation values.

   • Click in the Output column row
     and select the destination column from the list, the
     count column in this example.

   • Click in the Function column row
     and select any of the listed operations.

     In this example, we want to count the number of clients, based on
     their regions, to be listed only once in the output column.

   • Click in the Input column
     position row and select the input column from which
     you want to collect the values to be aggregated, the
     region column in this example.

6. Double-click the second tAggregateRow
   component and define, the same way, its basic settings to count the number
   of clients in the second cluster based on the channel
   column.

   

1. Double-click the second tMap to open the
   Map Editor.

   

2. Drop the region, the clusterID
   and the count columns to the output table that
   corresponds to the first HDFS file.

3. Click OK to validate changes.

4. Double-click the third tMap to open the
   Map Editor.

   

5. Drop the channel, the clusterID
   and the count columns to the output table that
   corresponds to the second HDFS file.

6. Click OK to validate changes.

1. Double-click the first tHDFSOutput to
   open its Component view.

   

2. Click the […] button next to the
   Folder field and browse to the folder
   in which you want to write the region data.

3. From the Type list, select the data
   format for the records to be written. In this example, select Text file.

4. From the Action list, select the
   operation you need to perform on the file in question. If the file already
   exists, select Overwrite, otherwise select
   Create.

5. Select the Merge result to single file
   check box and enter the path, or browse to the file you need to write the
   merged output data in.

6. If the file for the merged data exists, select the Override target file check box to overwrite that
   file.

7. Double-click the second tHDFSOutput to
   open its Component view.

   

8. Define the component settings similarly to write the data about the client
   channels from the second cluster to an output HDFS folder.

• Save your Job and press F6 to execute
  it.

  The below figure shows part of the clustered data written to the HDFS
  folders.

  

  tMahoutClustering reads data from the
  given Hadoop system and groups customer records into clusters. The other
  components in the Job analyze clustering results, show the number of
  customers in each cluster grouped by channels or regions and gives the
  cluster identification.

Visualizing output data

You can build a bar chart on each of the clusters to visualize the number of
customers grouped by different regions or channels.

• Use tHDFSInput and tBarChart components in two Jobs to read the output HDFS
  files and generate a bar chart on the data to ease technical
  analysis.

  In the first Job, tHDFSInput reads the
  region output HDFS file and passes the flow to tBarChart. tBarChart reads
  data from the input flow and transforms it into a bar chart in a PNG image
  file.

  

  In the second Job, tHDFSInput reads the
  channel output HDFS file and passes the flow to tBarChart which transforms the input data into a bar chart
  in a PNG image file.

  

  Each bar chart has three columns, every column represents the number of
  records in one cluster.

  For further information about the tHDFSInput component,
  see tHDFSInput and for more information about the
  tBarChart, see tBarChart.