tMahoutClustering (deprecated)
Groups unlabeled numerical data into clusters that can reveal interesting patterns
or helps identifying abnormal data items in the data set.
tMahoutClustering groups data together
into clusters based on some similarities. The component offers several similarity
methods that can be used in different clustering algorithms.
tMahoutClustering uses clustering
algorithms from Mahout libraries. All processes are run in a given distributed file
system.
Currently, the studio supports Mahout 0.9.
tMahoutClustering MapReduce properties (deprecated)
These properties are used to configure tMahoutClustering running in the MapReduce Job framework.
The MapReduce
tMahoutClustering component belongs to the MapReduce family.
This component is available in Talend Platform products with Big Data and
in Talend Data Fabric.
The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.
Basic settings
Schema and Edit schema |
A schema is a row description. It defines the number of fields Click Edit
The output schema of tMahoutClustering provides one read-only column, |
 |
Built-In: You create and store the schema locally for this component |
 |
Repository: You have already created the schema and stored it in the |
Input HDFS file |
Browse to the HDFS file that holds the numerical data to be |
Field separator |
Enter a character, string or regular expression to separate fields |
Cluster columns |
In the Input Column, select the You can add only numerical columns to this table. |
Clustering type |
Select the relevant clustering algorithm from the list:
Canopy: this algorithm uses an Canopy clustering is often used as an initial step in more rigorous
K-Means: it sorts a given data set The algorithm then associates each data point belonging to a given
Fuzzy K-Means: also called Fuzzy C-Means: it belongs to the family of |
Distance measure |
Select from the list the distance measure you want to use for
Euclidean: defines the “ordinary”
Manhattan: defines the distance
Chebyshev: defines the maximum
Cosine: uses the cosine of the |
Canopy threshold1 |
The threshold of distance T1 used for the Canopy algorithm. |
Canopy threshold2 |
The threshold of distance T2 used for the Canopy algorithm. |
Number of clusters |
Enter the maximum number of clusters that can be generated by a |
Max iterations |
Enter the maximum number of iterations to be carried out for a |
Convergence delta |
Enter a rate of convergence for the algorithm. It must be between |
Fuzziness |
Enter the fuzziness parameter for the Fuzzy When the fuzziness is close to 1, then the cluster center closest |
Global Variables
Global Variables |
ERROR_MESSAGE: the error message generated by the A Flow variable functions during the execution of a component while an After variable To fill up a field or expression with a variable, press Ctrl + For further information about variables, see |
Usage
Usage rule |
tMahoutClustering is |
Grouping customer numerical data into clusters on HDFS
(deprecated)
This scenario applies only to subscription-based Talend products with Big
Data.
The scenario is inspired from a research paper on model-based clustering. Its data can
be found at Wholesale
customers Data Set. The research paper is available at Enhancing
the selection of a model-based clustering with external categorical
variables. This scenario is included in the Data Quality
Demos project you can import into your
Talend Studio
.
For further information, see the
Talend Studio User
Guide.
The Job in this scenario connects to a given Hadoop distributed file system (HDFS),
groups customers of a “wholesale distributor” into two clusters using the algorithms in
tMahoutClustering and outputs data on a given
HDFS.
The data set has 440 samples that refer to clients of a wholesale distributor. It
includes the annual spending in monetary units on diverse product categories like fresh
and grocery products or milk.
The data set refers to customers from different channels – Horeca
(Hotel/Restaurant/Cafe) or Retail (sale of goods in small quantities) channel, and from
different regions (Lisbon/Oporto/other).
This Job uses:
-
tMahoutClustering to compute the clusters for
the input data set. -
two tAggregateRow components to count the
number of clients in both clusters based on the region and
channel columns. -
three tMap components to map the channel and
region input flows into two separate output flows. The components are also used
to map the single clusterID column received from tMahoutClustering to two-column data flow that feed
the region and the channel clusters. -
two tHDFSOutput components to write data to
HDFS in two output files.
Prerequisites: Before being able to use the tMahoutClustering component, you must have a functional
Hadoop system.
Setting up the Job
-
Drop the following components from the Palette onto the design workspace: tMahoutClustering, three tMap, two tAggregateRow and
two tHDFSOutput components. -
Set the components as shown in the capture and connect them together using
Main links.
Setting up Hadoop connection
-
Click Run to open its view and then click the
Hadoop Configuration tab to display its
view for configuring the Hadoop connection for this Job. -
From the Property type list,
select Built-in. If you have created the
connection to be used in Repository, then
select Repository and thus the Studio will
reuse that set of connection information for this Job. -
In the Version area, select the
Hadoop distribution to be used and its version.-
If you use Google Cloud Dataproc, see Google Cloud Dataproc.
-
If you cannot
find the Cloudera version to be used from this drop-down list, you can add your distribution
via some dynamic distribution settings in the Studio. -
If you cannot find from the list the distribution corresponding to
yours, select Custom so as to connect to a
Hadoop distribution not officially supported in the Studio. For a
step-by-step example about how to use this
Custom option, see Connecting to a custom Hadoop distribution.
-
-
In the Name node field, enter the location of
the master node, the NameNode, of the distribution to be used. For example,
hdfs://tal-qa113.talend.lan:8020.-
If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
client will take care of the rest on the fly for creating the connection. The
MapR client must be properly installed. For further information about how to set
up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client -
If you are using WebHDFS, the location should be
webhdfs://masternode:portnumber; WebHDFS with SSL is not
supported yet.
-
-
In the Resource Manager field,
enter the location of the ResourceManager of your distribution. For example,
tal-qa114.talend.lan:8050.-
Then you can continue to set the following parameters depending on the
configuration of the Hadoop cluster to be used (if you leave the check
box of a parameter clear, then at runtime, the configuration about this
parameter in the Hadoop cluster to be used will be ignored):-
Select the Set resourcemanager
scheduler address check box and enter the Scheduler address in
the field that appears. -
Select the Set jobhistory
address check box and enter the location of the JobHistory
server of the Hadoop cluster to be used. This allows the metrics information of
the current Job to be stored in that JobHistory server. -
Select the Set staging
directory check box and enter this directory defined in your
Hadoop cluster for temporary files created by running programs. Typically, this
directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files
such as yarn-site.xml or mapred-site.xml of your distribution. -
Select the Use datanode hostname check box to allow the
Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname
property to true. When connecting to a
S3N filesystem, you must select this check box.
-
-
-
If you are accessing the Hadoop cluster running
with Kerberos security, select this check box, then, enter the Kerberos
principal name for the NameNode in the field displayed. This enables you to use
your user name to authenticate against the credentials stored in Kerberos.
-
If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the
MapR ticket authentication configuration in addition or as an alternative by following
the explanation in Connecting to a security-enabled MapR.Keep in mind that this configuration generates a new MapR security ticket for the username
defined in the Job in each execution. If you need to reuse an existing ticket issued for the
same username, leave both the Force MapR ticket
authentication check box and the Use Kerberos
authentication check box clear, and then MapR should be able to automatically
find that ticket on the fly.
In addition, since this component performs Map/Reduce computations, you
also need to authenticate the related services such as the Job history server and
the Resource manager or Jobtracker depending on your distribution in the
corresponding field. These principals can be found in the configuration files of
your distribution. For example, in a CDH4 distribution, the Resource manager
principal is set in the yarn-site.xml file and the Job history
principal in the mapred-site.xml file.If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains
pairs of Kerberos principals and encrypted keys. You need to enter the principal to
be used in the Principal field and the access
path to the keytab file itself in the Keytab
field. This keytab file must be stored in the machine in which your Job actually
runs, for example, on a Talend
Jobserver.Note that the user that executes a keytab-enabled Job is not necessarily
the one a principal designates but must have the right to read the keytab file being
used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this
situation, ensure that user1 has the right to read the keytab
file to be used. -
-
In the User name field, enter the login user
name for your distribution. If you leave it empty, the user name of the machine
hosting the Studio will be used. -
In the Temp folder field, enter the path in
HDFS to the folder where you store the temporary files generated during
Map/Reduce computations. -
Leave the default value of the Path separator in
server as it is, unless you have changed the separator used by your
Hadoop distribution’s host machine for its PATH variable or in other words, that
separator is not a colon (:). In that situation, you must change this value to the
one you are using in that host.
-
Leave the Clear temporary folder check box
selected, unless you want to keep those temporary files. -
Leave the Compress intermediate map output to reduce
network traffic check box selected, so as to spend shorter time
to transfer the mapper task partitions to the multiple reducers.However, if the data transfer in the Job is negligible, it is recommended to
clear this check box to deactivate the compression step, because this
compression consumes extra CPU resources. -
If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
properties to be customized. Then at runtime, these changes will override the
corresponding default properties used by the Studio for its Hadoop
engine.For further information about the properties required by Hadoop, see Apache’s
Hadoop documentation on http://hadoop.apache.org, or
the documentation of the Hadoop distribution you need to use. -
If the HDFS transparent encryption has been enabled in your cluster, select
the Setup HDFS encryption configurations check
box and in the HDFS encryption key provider field
that is displayed, enter the location of the KMS proxy.
For further information about the HDFS transparent encryption and its KMS proxy, see Transparent Encryption in HDFS.
-
You can tune the map and reduce computations by
selecting the Set memory check box to set proper memory allocations
for the computations to be performed by the Hadoop system.The memory parameters to be set are Map (in Mb),
Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate
memory to the map and the reduce computations and the ApplicationMaster of YARN.For further information about the Resource Manager, its scheduler and the
ApplicationMaster, see YARN’s documentation such as http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/.For further information about how to determine YARN and MapReduce memory configuration
settings, see the documentation of the distribution you are using, such as the following
link provided by Hortonworks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html. -
If you are using Cloudera V5.5+, you can select the Use Cloudera Navigator check box to enable the Cloudera Navigator
of your distribution to trace your Job lineage to the component level, including the
schema changes between components.
With this option activated, you need to set the following parameters:
-
Username and Password: this is the credentials you use to connect to your Cloudera
Navigator. -
Cloudera Navigator URL : enter the location of the
Cloudera Navigator to be connected to. -
Cloudera Navigator Metadata URL: enter the location
of the Navigator Metadata. -
Activate the autocommit option: select this check box
to make Cloudera Navigator generate the lineage of the current Job at the end of the
execution of this Job.Since this option actually forces Cloudera Navigator to generate lineages of
all its available entities such as HDFS files and directories, Hive queries or Pig
scripts, it is not recommended for the production environment because it will slow the
Job. -
Kill the job if Cloudera Navigator fails: select this check
box to stop the execution of the Job when the connection to your Cloudera Navigator fails.Otherwise, leave it clear to allow your Job to continue to run.
-
Disable SSL validation: select this check box to
make your Job to connect to Cloudera Navigator without the SSL validation
process.This feature is meant to facilitate the test of your Job but is not
recommended to be used in a production cluster.
-
-
If you are using Hortonworks Data Platform V2.4.0 onwards and you have
installed Atlas in your cluster, you can select the Use
Atlas check box to enable Job lineage to the component level, including the
schema changes between components.
With this option activated, you need to set the following parameters:
-
Atlas URL: enter the location of the Atlas to be
connected to. It is often http://name_of_your_atlas_node:port -
Die on error: select this check box to stop the Job
execution when Atlas-related issues occur, such as connection issues to Atlas.Otherwise, leave it clear to allow your Job to continue to run.
In the Username and Password fields, enter the authentication information for access to
Atlas. -
Configuring the clustering process
-
Double-click tMahoutClustering to open
its Component view. -
From the Schema list, select Built-In and then click the […] button next to Edit
Schema and describe the data structure in the input
file. -
Add eight rows to the schema dialog box and define the input data as shown
in the above capture.The component has one read-only column,
clusterID. - Click OK.
-
In the File Configuration area:
-
Click the […] button next to
the Input HDFS file and browse to
the HDFS file on the Hadoop system that holds the input numerical
data you want to cluster. -
Set the field separator used to separate the columns in the
clustered data. -
In the Cluster columns table, add
rows to the table and click in each row to select a column from the
input schema.
-
-
In the Clustering Configuration
area:-
From the Clustering Type list,
select what algorithm you want to use to cluster the numerical data,
Fuzzy K-means in this
example. -
From the Distance Measure list,
select the distance measure you want to use for clustering. -
In the Number of clusters field,
enter 3. -
Leave the values in Max
iterations and Convergence
delta as they are.
-
Mapping data
-
Double-click tMap to open the Map Editor.
-
Drop the Region and the
clusterID columns to the first output table that
corresponds to the first tAggregateRow
component.Drop the Channel and the
clusterID columns to the second output table that
corresponds to the second tAggregateRow
component.Use the Schema editor section at the
bottom of the editor to add necessary lines to the output tables. - Click OK to validate changes.
Aggregating and calculating output data
-
Double-click the first tAggregateRow to
display its Basic settings view and define
the component properties. -
Click the […] button next to Edit schema and define the output flow.
-
Move the columns in the input schema to the output schema and then use the
[+] button to add a new column in the
output schema. Call it count.When done, click OK to close the dialog
box. -
In the Group by section, click the plus
button to add an many lines as needed. Here you can define the group-by
values.-
Click in the first Output column
row and select the output column that will hold the aggregated data,
the region column in this example. -
Click in the first Input column
position row and select the input column from which
you want to collect the values to be aggregated, the
region column in this example.
-
-
In the Operations section, click the plus
button to add rows for the columns that will hold the aggregated data. Here
you can define the calculation values.-
Click in the Output column row
and select the destination column from the list, the
count column in this example. -
Click in the Function column row
and select any of the listed operations.In this example, we want to count the number of clients, based on
their regions, to be listed only once in the output column. -
Click in the Input column
position row and select the input column from which
you want to collect the values to be aggregated, the
region column in this example.
-
-
Double-click the second tAggregateRow
component and define, the same way, its basic settings to count the number
of clients in the second cluster based on the channel
column.
Mapping output data
-
Double-click the second tMap to open the
Map Editor. -
Drop the region, the clusterID
and the count columns to the output table that
corresponds to the first HDFS file. - Click OK to validate changes.
-
Double-click the third tMap to open the
Map Editor. -
Drop the channel, the clusterID
and the count columns to the output table that
corresponds to the second HDFS file. - Click OK to validate changes.
Writing output data in HDFS
-
Double-click the first tHDFSOutput to
open its Component view. -
Click the […] button next to the
Folder field and browse to the folder
in which you want to write the region data. -
From the Type list, select the data
format for the records to be written. In this example, select Text file. -
From the Action list, select the
operation you need to perform on the file in question. If the file already
exists, select Overwrite, otherwise select
Create. -
Select the Merge result to single file
check box and enter the path, or browse to the file you need to write the
merged output data in. -
If the file for the merged data exists, select the Override target file check box to overwrite that
file. -
Double-click the second tHDFSOutput to
open its Component view. -
Define the component settings similarly to write the data about the client
channels from the second cluster to an output HDFS folder.
Finalizing and executing the Job
-
Save your Job and press F6 to execute
it.The below figure shows part of the clustered data written to the HDFS
folders.tMahoutClustering reads data from the
given Hadoop system and groups customer records into clusters. The other
components in the Job analyze clustering results, show the number of
customers in each cluster grouped by channels or regions and gives the
cluster identification.
Visualizing output data
You can build a bar chart on each of the clusters to visualize the number of
customers grouped by different regions or channels.
files and generate a bar chart on the data to ease technical
analysis.
region output HDFS file and passes the flow to tBarChart. tBarChart reads
data from the input flow and transforms it into a bar chart in a PNG image
file.
_13.png)
channel output HDFS file and passes the flow to tBarChart which transforms the input data into a bar chart
in a PNG image file.
_14.png)
records in one cluster.
see tHDFSInput and for more information about the
tBarChart, see tBarChart.