July 30, 2023

tMahoutClustering (deprecated) – Docs for ESB 7.x

tMahoutClustering (deprecated)

Groups unlabeled numerical data into clusters that can reveal interesting patterns
or helps identifying abnormal data items in the data set.

tMahoutClustering groups data together
into clusters based on some similarities. The component offers several similarity
methods that can be used in different clustering algorithms.

tMahoutClustering uses clustering
algorithms from Mahout libraries. All processes are run in a given distributed file
system.

Note:

Currently, the studio supports Mahout 0.9.

tMahoutClustering MapReduce properties (deprecated)

These properties are used to configure tMahoutClustering running in the MapReduce Job framework.

The MapReduce
tMahoutClustering component belongs to the MapReduce family.

This component is available in Talend Platform products with Big Data and
in Talend Data Fabric.

The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.

Basic settings

Schema and Edit schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema
to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

The output schema of tMahoutClustering provides one read-only column,
ClusterID.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Input HDFS file

Browse to the HDFS file that holds the numerical data to be
processed.

Field separator

Enter a character, string or regular expression to separate fields
in the input and output data.

Cluster columns

In the Input Column, select the
column(s) from the main flow on which you want to define clustering
algorithms. These columns are used to calculate the clusters.

You can add only numerical columns to this table.

Clustering type

Select the relevant clustering algorithm from the list:

Canopy: this algorithm uses an
approximate distance metric and two distance thresholds T 1 and T 2
,where T 1 >T 2. It starts with a set of data points in any order,
picks a point called the centroid of the cluster and approximately
measures its distance to all other points. It puts all points that are
within distance threshold T 1 into a canopy. It removes from the main
set all points that are within distance threshold T 2. This way points
that are very close to the centroid will avoid all further processing.
The algorithm then chooses a second centroid among the data points in
the principal set. It continues until the initial set is empty,
accumulating a set of Canopies, each containing one or more points. A
given point may occur in more than one Canopy.

Canopy clustering is often used as an initial step in more rigorous
clustering techniques, such as K-Means clustering . By starting with
Canopy clustering the number of more expensive distance measurements can
be significantly reduced by ignoring points outside of the initial
canopies.

K-Means: it sorts a given data set
into a number of clusters, the number of which you must define. The
algorithm chooses k random points,
used as centroids of k clusters.

The algorithm then associates each data point belonging to a given
data set to the nearest cluster center.

Fuzzy K-Means: also called Fuzzy C-Means: it belongs to the family of
fuzzy-logic clustering algorithms. It works like K-Means but recomputes the cluster centers using the
probability of a point belonging to two or more clusters.

Distance measure

Select from the list the distance measure you want to use for
clustering:

Euclidean: defines the “ordinary”
distance between two points, as if measured with a ruler.

Manhattan: defines the distance
between two points if a grid-like path is followed.

Chebyshev: defines the maximum
distance between two vectors taken on any of the coordinate
dimensions.

Cosine: uses the cosine of the
angle between the two vectors representing the points to be
compared.

Canopy threshold1

The threshold of distance T1 used for the Canopy algorithm.

Canopy threshold2

The threshold of distance T2 used for the Canopy algorithm.

Number of clusters

Enter the maximum number of clusters that can be generated by a
clustering algorithm. Some clusters may not have data.

Max iterations

Enter the maximum number of iterations to be carried out for a
clustering algorithm.

Convergence delta

Enter a rate of convergence for the algorithm. It must be between
0.0 and 1.0. The greater the rate is, the faster the algorithm is
but results will be less precise.

Fuzziness

Enter the fuzziness parameter for the Fuzzy
K-Means
algorithm. It must be greater or equal to
1.0.

When the fuzziness is close to 1, then the cluster center closest
to the point is given much more weight than the others, and the
algorithm is similar to K-Means.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space
to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

tMahoutClustering is
deprecated. You must use JDK 7 to be able to run migrated Jobs with
tMahoutClustering successfully. If you need to
execute clustering algorithms, it is recommended to create a Spark Batch
Job and use tKMeansModel instead in that Job.

Grouping customer numerical data into clusters on HDFS
(deprecated)

This scenario applies only to subscription-based Talend products with Big
Data
.

The scenario is inspired from a research paper on model-based clustering. Its data can
be found at Wholesale
customers Data Set
. The research paper is available at Enhancing
the selection of a model-based clustering with external categorical
variables
. This scenario is included in the Data Quality
Demos
project you can import into your
Talend Studio
.
For further information, see the
Talend Studio User
Guide
.

The Job in this scenario connects to a given Hadoop distributed file system (HDFS),
groups customers of a “wholesale distributor” into two clusters using the algorithms in
tMahoutClustering and outputs data on a given
HDFS.

The data set has 440 samples that refer to clients of a wholesale distributor. It
includes the annual spending in monetary units on diverse product categories like fresh
and grocery products or milk.

The data set refers to customers from different channels – Horeca
(Hotel/Restaurant/Cafe) or Retail (sale of goods in small quantities) channel, and from
different regions (Lisbon/Oporto/other).

tMahoutClustering (deprecated)_1.png

This Job uses:

  • tMahoutClustering to compute the clusters for
    the input data set.

  • two tAggregateRow components to count the
    number of clients in both clusters based on the region and
    channel columns.

  • three tMap components to map the channel and
    region input flows into two separate output flows. The components are also used
    to map the single clusterID column received from tMahoutClustering to two-column data flow that feed
    the region and the channel clusters.

  • two tHDFSOutput components to write data to
    HDFS in two output files.

Prerequisites: Before being able to use the tMahoutClustering component, you must have a functional
Hadoop system.

Setting up the Job

  1. Drop the following components from the Palette onto the design workspace: tMahoutClustering, three tMap, two tAggregateRow and
    two tHDFSOutput components.

    tMahoutClustering (deprecated)_2.png

  2. Set the components as shown in the capture and connect them together using
    Main links.

Setting up Hadoop connection

  1. Click Run to open its view and then click the
    Hadoop Configuration tab to display its
    view for configuring the Hadoop connection for this Job.
  2. From the Property type list,
    select Built-in. If you have created the
    connection to be used in Repository, then
    select Repository and thus the Studio will
    reuse that set of connection information for this Job.
  3. In the Version area, select the
    Hadoop distribution to be used and its version.

    • If you use Google Cloud Dataproc, see Google Cloud Dataproc.

    • If you cannot
      find the Cloudera version to be used from this drop-down list, you can add your distribution
      via some dynamic distribution settings in the Studio.

    • If you cannot find from the list the distribution corresponding to
      yours, select Custom so as to connect to a
      Hadoop distribution not officially supported in the Studio. For a
      step-by-step example about how to use this
      Custom option, see Connecting to a custom Hadoop distribution.

  4. In the Name node field, enter the location of
    the master node, the NameNode, of the distribution to be used. For example,
    hdfs://tal-qa113.talend.lan:8020.

    • If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
      client will take care of the rest on the fly for creating the connection. The
      MapR client must be properly installed. For further information about how to set
      up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client

    • If you are using WebHDFS, the location should be
      webhdfs://masternode:portnumber; WebHDFS with SSL is not
      supported yet.

  5. In the Resource Manager field,
    enter the location of the ResourceManager of your distribution. For example,
    tal-qa114.talend.lan:8050.

    • Then you can continue to set the following parameters depending on the
      configuration of the Hadoop cluster to be used (if you leave the check
      box of a parameter clear, then at runtime, the configuration about this
      parameter in the Hadoop cluster to be used will be ignored):

      • Select the Set resourcemanager
        scheduler address
        check box and enter the Scheduler address in
        the field that appears.

      • Select the Set jobhistory
        address
        check box and enter the location of the JobHistory
        server of the Hadoop cluster to be used. This allows the metrics information of
        the current Job to be stored in that JobHistory server.

      • Select the Set staging
        directory
        check box and enter this directory defined in your
        Hadoop cluster for temporary files created by running programs. Typically, this
        directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files
        such as yarn-site.xml or mapred-site.xml of your distribution.

      • Select the Use datanode hostname check box to allow the
        Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname
        property to true. When connecting to a
        S3N filesystem, you must select this check box.


  6. If you are accessing the Hadoop cluster running
    with Kerberos security, select this check box, then, enter the Kerberos
    principal name for the NameNode in the field displayed. This enables you to use
    your user name to authenticate against the credentials stored in Kerberos.

    • If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the
      MapR ticket authentication configuration in addition or as an alternative by following
      the explanation in Connecting to a security-enabled MapR.

      Keep in mind that this configuration generates a new MapR security ticket for the username
      defined in the Job in each execution. If you need to reuse an existing ticket issued for the
      same username, leave both the Force MapR ticket
      authentication
      check box and the Use Kerberos
      authentication
      check box clear, and then MapR should be able to automatically
      find that ticket on the fly.

    In addition, since this component performs Map/Reduce computations, you
    also need to authenticate the related services such as the Job history server and
    the Resource manager or Jobtracker depending on your distribution in the
    corresponding field. These principals can be found in the configuration files of
    your distribution. For example, in a CDH4 distribution, the Resource manager
    principal is set in the yarn-site.xml file and the Job history
    principal in the mapred-site.xml file.

    If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains
    pairs of Kerberos principals and encrypted keys. You need to enter the principal to
    be used in the Principal field and the access
    path to the keytab file itself in the Keytab
    field. This keytab file must be stored in the machine in which your Job actually
    runs, for example, on a Talend
    Jobserver.

    Note that the user that executes a keytab-enabled Job is not necessarily
    the one a principal designates but must have the right to read the keytab file being
    used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this
    situation, ensure that user1 has the right to read the keytab
    file to be used.

  7. In the User name field, enter the login user
    name for your distribution. If you leave it empty, the user name of the machine
    hosting the Studio will be used.
  8. In the Temp folder field, enter the path in
    HDFS to the folder where you store the temporary files generated during
    Map/Reduce computations.

  9. Leave the default value of the Path separator in
    server
    as it is, unless you have changed the separator used by your
    Hadoop distribution’s host machine for its PATH variable or in other words, that
    separator is not a colon (:). In that situation, you must change this value to the
    one you are using in that host.

  10. Leave the Clear temporary folder check box
    selected, unless you want to keep those temporary files.
  11. Leave the Compress intermediate map output to reduce
    network traffic
    check box selected, so as to spend shorter time
    to transfer the mapper task partitions to the multiple reducers.

    However, if the data transfer in the Job is negligible, it is recommended to
    clear this check box to deactivate the compression step, because this
    compression consumes extra CPU resources.
  12. If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
    properties to be customized. Then at runtime, these changes will override the
    corresponding default properties used by the Studio for its Hadoop
    engine.

    For further information about the properties required by Hadoop, see Apache’s
    Hadoop documentation on http://hadoop.apache.org, or
    the documentation of the Hadoop distribution you need to use.

  13. If the HDFS transparent encryption has been enabled in your cluster, select
    the Setup HDFS encryption configurations check
    box and in the HDFS encryption key provider field
    that is displayed, enter the location of the KMS proxy.

    For further information about the HDFS transparent encryption and its KMS proxy, see Transparent Encryption in HDFS.

  14. You can tune the map and reduce computations by
    selecting the Set memory check box to set proper memory allocations
    for the computations to be performed by the Hadoop system.

    The memory parameters to be set are Map (in Mb),
    Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate
    memory to the map and the reduce computations and the ApplicationMaster of YARN.

    For further information about the Resource Manager, its scheduler and the
    ApplicationMaster, see YARN’s documentation such as http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/.

    For further information about how to determine YARN and MapReduce memory configuration
    settings, see the documentation of the distribution you are using, such as the following
    link provided by Hortonworks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html.


  15. If you are using Cloudera V5.5+, you can select the Use Cloudera Navigator check box to enable the Cloudera Navigator
    of your distribution to trace your Job lineage to the component level, including the
    schema changes between components.

    With this option activated, you need to set the following parameters:

    • Username and Password: this is the credentials you use to connect to your Cloudera
      Navigator.

    • Cloudera Navigator URL : enter the location of the
      Cloudera Navigator to be connected to.

    • Cloudera Navigator Metadata URL: enter the location
      of the Navigator Metadata.

    • Activate the autocommit option: select this check box
      to make Cloudera Navigator generate the lineage of the current Job at the end of the
      execution of this Job.

      Since this option actually forces Cloudera Navigator to generate lineages of
      all its available entities such as HDFS files and directories, Hive queries or Pig
      scripts, it is not recommended for the production environment because it will slow the
      Job.

    • Kill the job if Cloudera Navigator fails: select this check
      box to stop the execution of the Job when the connection to your Cloudera Navigator fails.

      Otherwise, leave it clear to allow your Job to continue to run.

    • Disable SSL validation: select this check box to
      make your Job to connect to Cloudera Navigator without the SSL validation
      process.

      This feature is meant to facilitate the test of your Job but is not
      recommended to be used in a production cluster.


  16. If you are using Hortonworks Data Platform V2.4.0 onwards and you have
    installed Atlas in your cluster, you can select the Use
    Atlas
    check box to enable Job lineage to the component level, including the
    schema changes between components.

    With this option activated, you need to set the following parameters:

    • Atlas URL: enter the location of the Atlas to be
      connected to. It is often http://name_of_your_atlas_node:port

    • Die on error: select this check box to stop the Job
      execution when Atlas-related issues occur, such as connection issues to Atlas.

      Otherwise, leave it clear to allow your Job to continue to run.

    In the Username and Password fields, enter the authentication information for access to
    Atlas.

Configuring the clustering process

  1. Double-click tMahoutClustering to open
    its Component view.

    tMahoutClustering (deprecated)_3.png

  2. From the Schema list, select Built-In and then click the […] button next to Edit
    Schema
    and describe the data structure in the input
    file.

    tMahoutClustering (deprecated)_4.png

  3. Add eight rows to the schema dialog box and define the input data as shown
    in the above capture.

    The component has one read-only column,
    clusterID.
  4. Click OK.
  5. In the File Configuration area:

    • Click the […] button next to
      the Input HDFS file and browse to
      the HDFS file on the Hadoop system that holds the input numerical
      data you want to cluster.

    • Set the field separator used to separate the columns in the
      clustered data.

    • In the Cluster columns table, add
      rows to the table and click in each row to select a column from the
      input schema.

  6. In the Clustering Configuration
    area:

    • From the Clustering Type list,
      select what algorithm you want to use to cluster the numerical data,
      Fuzzy K-means in this
      example.

    • From the Distance Measure list,
      select the distance measure you want to use for clustering.

    • In the Number of clusters field,
      enter 3.

    • Leave the values in Max
      iterations
      and Convergence
      delta
      as they are.

Mapping data

  1. Double-click tMap to open the Map Editor.

    tMahoutClustering (deprecated)_5.png

  2. Drop the Region and the
    clusterID columns to the first output table that
    corresponds to the first tAggregateRow
    component.

    Drop the Channel and the
    clusterID columns to the second output table that
    corresponds to the second tAggregateRow
    component.
    Use the Schema editor section at the
    bottom of the editor to add necessary lines to the output tables.
  3. Click OK to validate changes.

Aggregating and calculating output data

  1. Double-click the first tAggregateRow to
    display its Basic settings view and define
    the component properties.

    tMahoutClustering (deprecated)_6.png

  2. Click the […] button next to Edit schema and define the output flow.

    tMahoutClustering (deprecated)_7.png

  3. Move the columns in the input schema to the output schema and then use the
    [+] button to add a new column in the
    output schema. Call it count.

    When done, click OK to close the dialog
    box.
  4. In the Group by section, click the plus
    button to add an many lines as needed. Here you can define the group-by
    values.

    • Click in the first Output column
      row and select the output column that will hold the aggregated data,
      the region column in this example.

    • Click in the first Input column
      position
      row and select the input column from which
      you want to collect the values to be aggregated, the
      region column in this example.

  5. In the Operations section, click the plus
    button to add rows for the columns that will hold the aggregated data. Here
    you can define the calculation values.

    • Click in the Output column row
      and select the destination column from the list, the
      count column in this example.

    • Click in the Function column row
      and select any of the listed operations.

      In this example, we want to count the number of clients, based on
      their regions, to be listed only once in the output column.

    • Click in the Input column
      position
      row and select the input column from which
      you want to collect the values to be aggregated, the
      region column in this example.

  6. Double-click the second tAggregateRow
    component and define, the same way, its basic settings to count the number
    of clients in the second cluster based on the channel
    column.

    tMahoutClustering (deprecated)_8.png

Mapping output data

  1. Double-click the second tMap to open the
    Map Editor.

    tMahoutClustering (deprecated)_9.png

  2. Drop the region, the clusterID
    and the count columns to the output table that
    corresponds to the first HDFS file.
  3. Click OK to validate changes.
  4. Double-click the third tMap to open the
    Map Editor.

    tMahoutClustering (deprecated)_10.png

  5. Drop the channel, the clusterID
    and the count columns to the output table that
    corresponds to the second HDFS file.
  6. Click OK to validate changes.

Writing output data in HDFS

  1. Double-click the first tHDFSOutput to
    open its Component view.

    tMahoutClustering (deprecated)_11.png

  2. Click the […] button next to the
    Folder field and browse to the folder
    in which you want to write the region data.
  3. From the Type list, select the data
    format for the records to be written. In this example, select Text file.
  4. From the Action list, select the
    operation you need to perform on the file in question. If the file already
    exists, select Overwrite, otherwise select
    Create.
  5. Select the Merge result to single file
    check box and enter the path, or browse to the file you need to write the
    merged output data in.
  6. If the file for the merged data exists, select the Override target file check box to overwrite that
    file.
  7. Double-click the second tHDFSOutput to
    open its Component view.

    tMahoutClustering (deprecated)_12.png

  8. Define the component settings similarly to write the data about the client
    channels from the second cluster to an output HDFS folder.

Finalizing and executing the Job

  1. Save your Job and press F6 to execute
    it.

    The below figure shows part of the clustered data written to the HDFS
    folders.

    tMahoutClustering reads data from the
    given Hadoop system and groups customer records into clusters. The other
    components in the Job analyze clustering results, show the number of
    customers in each cluster grouped by channels or regions and gives the
    cluster identification.

Visualizing output data

You can build a bar chart on each of the clusters to visualize the number of
customers grouped by different regions or channels.

Use tHDFSInput and tBarChart components in two Jobs to read the output HDFS
files and generate a bar chart on the data to ease technical
analysis.

In the first Job, tHDFSInput reads the
region output HDFS file and passes the flow to tBarChart. tBarChart reads
data from the input flow and transforms it into a bar chart in a PNG image
file.
tMahoutClustering (deprecated)_13.png

In the second Job, tHDFSInput reads the
channel output HDFS file and passes the flow to tBarChart which transforms the input data into a bar chart
in a PNG image file.
tMahoutClustering (deprecated)_14.png

Each bar chart has three columns, every column represents the number of
records in one cluster.
For further information about the tHDFSInput component,
see tHDFSInput and for more information about the
tBarChart, see tBarChart.


Document get from Talend https://help.talend.com
Thank you for watching.
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x