These properties are used to configure tMatchPairing running in the Spark Batch Job framework.

The Spark Batch
tMatchPairing component belongs to the Data Quality family.

The component in this framework is available in all Talend Platform products with Big Data and in Talend Data Fabric.

Basic settings

Advanced settings

Usage

• compute pairs of suspect records based on a blocking key
  definition,

• creates a sample of suspect records representative of the data
  set,

• can optionally write this sample of suspect records into a Grouping campaign
  defined on the Talend Data Stewardship server,

• separates unique records from exact match records,

• generates a pairing model to be used with tMatchPredict.

• computes similarities between the records in each suspect
  pair,

• trains a classification model based on the Random Forest
  algorithm.

• labels suspect records automatically,

• groups suspect records which match the label(s) set in the
  component properties,

• separates the exact duplicates from unique records.

• You have been assigned in Talend Administration Center the Campaign
  Owner role which grants you access to the campaigns on the
  server.

• You have created the Grouping campaign in Talend Data Stewardship and defined the schema which corresponds to the structure
  of the education centers file.

1. Drop the following components from the Palette onto the
   design workspace: tFileInputDelimited,
   tMatchPairing, tLogRow and two
   tFileOutputDelimited.
   

2. Connect tFileInputDelimited to
   tMatchPairing using the Main link.
   tFileInputDelimited reads the source file and sends
   data to the next component.
3. Connect tMatchPairing to the output file components
   using the Pairs and Unique rows
   links, and to tLogRow using the Exact
   duplicates link.
   tMatchPairing pre-analyzes the data, computes pairs
   of suspect duplicates, unique rows and exact duplicates and generates a pairing
   model to be used with tMatchPredict

1. Click Run to open its view and then click the
   Spark Configuration tab to display its view
   for configuring the Spark connection.
2. Select the Use local mode check box to test your Job locally.

   In the local mode, the Studio builds the Spark environment in itself on the fly in order to
   run the Job in. Each processor of the local machine is used as a Spark
   worker to perform the computations.

   In this mode, your local file system is used; therefore, deactivate the
   configuration components such as tS3Configuration or
   tHDFSConfiguration that provides connection
   information to a remote file system, if you have placed these components
   in your Job.

   You can launch
   your Job without any further configuration.

3. Clear the Use local mode check box to display the
   list of the available Hadoop distributions and from this list, select
   the distribution corresponding to your Spark cluster to be used.
   This distribution could be:

   • Databricks

   • Qubole

   • Amazon EMR

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • Cloudera

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Google Cloud
     Dataproc

     For this distribution, Talend supports:

     • Yarn client

   • Hortonworks

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • MapR

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Microsoft HD
     Insight

     For this distribution, Talend supports:

     • Yarn cluster

   • Cloudera Altus

     For this distribution, Talend supports:

     • Yarn cluster

       Your Altus cluster should run on the following Cloud
       providers:

       • Azure

         The support for Altus on Azure is a technical
         preview feature.

       • AWS

   As a Job relies on Avro to move data among its components, it is recommended to set your
   cluster to use Kryo to handle the Avro types. This not only helps avoid
   this Avro known issue but also
   brings inherent preformance gains. The Spark property to be set in your
   cluster is:
   

   spark.serializer org.apache.spark.serializer.KryoSerializer

   1	spark.serializer org.apache.spark.serializer.KryoSerializer

   If you cannot find the distribution corresponding to yours from this
   drop-down list, this means the distribution you want to connect to is not officially
   supported by
   Talend
   . In this situation, you can select Custom, then select the Spark
   version of the cluster to be connected and click the
   [+] button to display the dialog box in which you can
   alternatively:

   1. Select Import from existing
      version to import an officially supported distribution as base
      and then add other required jar files which the base distribution does not
      provide.

   2. Select Import from zip to
      import the configuration zip for the custom distribution to be used. This zip
      file should contain the libraries of the different Hadoop/Spark elements and the
      index file of these libraries.

      In
      Talend
      
      Exchange, members of
      Talend
      community have shared some ready-for-use configuration zip files
      which you can download from this Hadoop configuration
      list and directly use them in your connection accordingly. However, because of
      the ongoing evolution of the different Hadoop-related projects, you might not be
      able to find the configuration zip corresponding to your distribution from this
      list; then it is recommended to use the Import from
      existing version option to take an existing distribution as base
      to add the jars required by your distribution.

      Note that custom versions are not officially supported by
      
      Talend
      .
      Talend
      and its community provide you with the opportunity to connect to
      custom versions from the Studio but cannot guarantee that the configuration of
      whichever version you choose will be easy. As such, you should only attempt to
      set up such a connection if you have sufficient Hadoop and Spark experience to
      handle any issues on your own.

   For a step-by-step example about how to connect to a custom
   distribution and share this connection, see Hortonworks.

1. Double-click tHDFSConfiguration to open its Component view.

   Spark uses this component to connect to the HDFS system to which the jar
   files dependent on the Job are transferred.

2. If you have defined the HDFS connection metadata under the Hadoop
   cluster node in Repository, select
   Repository from the Property
   type drop-down list and then click the
   […] button to select the HDFS connection you have
   defined from the Repository content wizard.

   For further information about setting up a reusable
   HDFS connection, search for centralizing HDFS metadata on Talend Help Center
   (https://help.talend.com).

   If you complete this step, you can skip the following steps about configuring
   tHDFSConfiguration because all the required fields
   should have been filled automatically.

3. In the Version area, select
   the Hadoop distribution you need to connect to and its version.
4. In the NameNode URI field,
   enter the location of the machine hosting the NameNode service of the cluster.
   If you are using WebHDFS, the location should be
   webhdfs://masternode:portnumber; WebHDFS with SSL is not
   supported yet.
5. In the Username field, enter
   the authentication information used to connect to the HDFS system to be used.
   Note that the user name must be the same as you have put in the Spark configuration tab.

1. Double-click tFileInputDelimited to open its
   Basic settings view.
   The input data must have duplicate records, otherwise the model generated will
   not give authentic results when used on the whole suspect pairs.
2. Click the […] button next to Edit
   schema and use the [+] button in the
   dialog box to add String type columns: Original_Id,
   Source, Site_name and
   Address.
3. Click OK in the dialog box and accept to propagate the
   changes when prompted.
4. In the Folder/File field, set the path to the input
   file.
5. Set the row and field separators in the corresponding fields and the header and
   footer, if any.

1. Double-click tMatchPairing to display
   the Basic settings view and define the component
   properties.
   

2. Click Sync columns to retrieve the schema defined in the
   input component.
3. In the Blocking Key table, click the
   [+] button to add a row. Select the column you want
   to use as a blocking key, Site_name in this
   example.
   The blocking key is constructed from the center name and is used to generate
   the suffixes used to group pairs of records.
4. In the Suffix array blocking parameters section:
   1. In the Min suffix length field, set the minimum
      suffix length you want to reach or stop at in each group.
   2. In the Max block size field, set the maximum
      number of the records you want to have in each block. This helps
      filtering data in large blocks where the suffix is too common.
5. In the Folder field, set the path to the local folder
   where you want to generate the pairing model file.
   If you want to store the model in a specific file system, for example S3 or
   HDFS, you must use the corresponding component in the Job and select the
   Define a storage configuration component check box in
   the component basic settings.
6. Select the Integration with Data Stewardship check box
   and set the connection parameters to the Talend Data Stewardship
   server.
   1. In the URL field, enter the address of the application suffixed with /data-stewardship/, for example http://localhost:19999/data-stewardship/.

      If you are working with Talend Cloud Data Stewardship, use one of the following
      addresses to access the application:

      • https://tds.us.cloud.talend.com/data-stewardship for the US
        data center.
      • https://tds.eu.cloud.talend.com/data-stewardship for the EU
        data center.
      • https://tds.ap.cloud.talend.com/data-stewardship for the
        Asia-Pacific data center.

   2. Enter your login information
      in
      the Username and Password
      fields.
      To enter your password, click … next to the field, enter your
      password between double quotes in the dialog box that opens and click
      OK.

      If you
      are working with Talend Cloud Data Stewardship and if:

      • SSO is enabled, enter an access
        token in the field.
      • SSO is not enabled, enter either
        an access token or your password in the field.
   3. 
      Click Find a
      campaign to open a dialog box which lists the campaigns defined in
      Talend Data Stewardship and for which you
      are the owner or you have the access rights.
      
   4. Select the Sites deduplication campaign in which
      to write the grouping tasks and click OK.
7. Click Advanced settings and set the below
   parameters:
   1. In the Filtering threshold field, enter a value
      between 0.2 and 0.85 to filter the pairs of suspect records based on the
      calculated scores.
      This value helps to exclude the pairs which are not very similar. The
      higher the value is, the more similar the records are.
   2. Leave the Set a random seed check box clear as
      you want to generate a different sample by each execution of the
      Job.
   3. In the Number of pairs field, enter the size of
      the suspect pairs sample you want to generate.
   4. When configured with Talend Data Stewardship,
      enter the maximum number of the tasks to load per a commit in the
      Max tasks per commit field.
      There are no limits for the batch size in Talend Data Stewardship (on
      premises). However, do not exceed 200 tasks per commit in Talend Cloud Data Stewardship,
      otherwise the Job fails.

1. Double-click the first tFileOutputDelimited component to
   display the Basic settings view and define the component
   properties.
   You have already accepted to propagate the schema to the output components
   when you defined the input component.
2. Clear the Define a storage configuration component check
   box to use the local system as your target file system.
3. In the Folder field, set the path to the folder which
   will hold the output data.
4. From the Action list, select the operation for writing
   data:
   • Select Create when you run the Job for the first
     time.
   • Select Overwrite to replace the file every time
     you run the Job.
5. Set the row and field separators in the corresponding fields.
6. Select the Merge results to single file check box, and
   in the Merge file path field set the path where to output
   the file of the suspect record pairs.
7. Double-click the second tFileOutputDelimited component
   and define the component properties in the Basic settings
   view, as you do with the first component.
   This component creates the file which holds the unique rows generated from the
   input data.
8. Double-click tLogRow component and define the component
   properties in the Basic settings view.
   This component writes the exact duplicates generated from the input data on
   the Studio console.

1. Drop the following components from the Palette onto the design workspace:
   tFileInputDelimited, tMatchPairing, three tFileOutputDelimited and tLogRow.
2. Connect tFileInputDelimited to
   tMatchPairing using the Main link.
3. Connect tMatchPairing to
   the tFileOutputDelimited components using the Pairs, Pairs
   sample and Unique rows links, and to the
   tLogRow component using the
   Exact duplicates link.

1. Click Run to open its view and then click the
   Spark Configuration tab to display its view
   for configuring the Spark connection.
2. Select the Use local mode check box to test your Job locally.

   In the local mode, the Studio builds the Spark environment in itself on the fly in order to
   run the Job in. Each processor of the local machine is used as a Spark
   worker to perform the computations.

   In this mode, your local file system is used; therefore, deactivate the
   configuration components such as tS3Configuration or
   tHDFSConfiguration that provides connection
   information to a remote file system, if you have placed these components
   in your Job.

   You can launch
   your Job without any further configuration.

3. Clear the Use local mode check box to display the
   list of the available Hadoop distributions and from this list, select
   the distribution corresponding to your Spark cluster to be used.
   This distribution could be:

   • Databricks

   • Qubole

   • Amazon EMR

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • Cloudera

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Google Cloud
     Dataproc

     For this distribution, Talend supports:

     • Yarn client

   • Hortonworks

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • MapR

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Microsoft HD
     Insight

     For this distribution, Talend supports:

     • Yarn cluster

   • Cloudera Altus

     For this distribution, Talend supports:

     • Yarn cluster

       Your Altus cluster should run on the following Cloud
       providers:

       • Azure

         The support for Altus on Azure is a technical
         preview feature.

       • AWS

   As a Job relies on Avro to move data among its components, it is recommended to set your
   cluster to use Kryo to handle the Avro types. This not only helps avoid
   this Avro known issue but also
   brings inherent preformance gains. The Spark property to be set in your
   cluster is:
   

   spark.serializer org.apache.spark.serializer.KryoSerializer

   1	spark.serializer org.apache.spark.serializer.KryoSerializer

   If you cannot find the distribution corresponding to yours from this
   drop-down list, this means the distribution you want to connect to is not officially
   supported by
   Talend
   . In this situation, you can select Custom, then select the Spark
   version of the cluster to be connected and click the
   [+] button to display the dialog box in which you can
   alternatively:

   1. Select Import from existing
      version to import an officially supported distribution as base
      and then add other required jar files which the base distribution does not
      provide.

   2. Select Import from zip to
      import the configuration zip for the custom distribution to be used. This zip
      file should contain the libraries of the different Hadoop/Spark elements and the
      index file of these libraries.

      In
      Talend
      
      Exchange, members of
      Talend
      community have shared some ready-for-use configuration zip files
      which you can download from this Hadoop configuration
      list and directly use them in your connection accordingly. However, because of
      the ongoing evolution of the different Hadoop-related projects, you might not be
      able to find the configuration zip corresponding to your distribution from this
      list; then it is recommended to use the Import from
      existing version option to take an existing distribution as base
      to add the jars required by your distribution.

      Note that custom versions are not officially supported by
      
      Talend
      .
      Talend
      and its community provide you with the opportunity to connect to
      custom versions from the Studio but cannot guarantee that the configuration of
      whichever version you choose will be easy. As such, you should only attempt to
      set up such a connection if you have sufficient Hadoop and Spark experience to
      handle any issues on your own.

   For a step-by-step example about how to connect to a custom
   distribution and share this connection, see Hortonworks.

1. Double-click tFileInputDelimited to open its
   Basic settings view.
   The input data must have duplicate records, otherwise the model generated will
   not give authentic results when used on the whole suspect pairs.
2. Click the […] button next to Edit
   schema and use the [+] button in the
   dialog box to add String type columns: Original_Id,
   Source, Site_name and
   Address.
3. Click OK in the dialog box and accept to propagate the
   changes when prompted.
4. In the Folder/File field, set the path to the input
   file.
5. Set the row and field separators in the corresponding fields and the header and
   footer, if any.

1. Double-click tMatchPairing to display
   the Basic settings view and define the component
   properties.
   

2. Click Sync columns to retrieve the schema defined in the
   input component.
3. In the Blocking Key table, click the
   [+] button to add a row. Select the column you want
   to use as a blocking key, Site_name in this
   example.
   The blocking key is constructed from the center name and is used to generate
   the suffixes used to group pairs of records.
4. In the Suffix array blocking parameters section:
   1. In the Min suffix length field, set the minimum
      suffix length you want to reach or stop at in each group.
   2. In the Max block size field, set the maximum
      number of the records you want to have in each block. This helps
      filtering data in large blocks where the suffix is too common.
5. In the Folder field, set the path to the local folder
   where you want to generate the pairing model file.
   If you want to store the model in a specific file system, for example S3 or
   HDFS, you must use the corresponding component in the Job and select the
   Define a storage configuration component check box in
   the component basic settings.
6. Click Advanced settings and set the below
   parameters:
   1. In the Filtering threshold field, enter a value
      between 0.2 and 0.85 to filter the pairs of suspect records based on the
      calculated scores.
      This value helps to exclude the pairs which are not very similar. The
      higher the value is, the more similar the records are.
   2. Leave the Set a random seed check box clear as
      you want to generate a different sample by each execution of the
      Job.
   3. In the Number of pairs field, enter the size of
      the suspect pairs sample you want to generate.
   4. When configured with Talend Data Stewardship,
      enter the maximum number of the tasks to load per a commit in the
      Max tasks per commit field.
      There are no limits for the batch size in Talend Data Stewardship (on
      premises). However, do not exceed 200 tasks per commit in Talend Cloud Data Stewardship,
      otherwise the Job fails.

1. Double-click the first tFileOutputDelimited component to
   display the Basic settings view and define the component
   properties.
   You have already accepted to propagate the schema to the output components
   when you defined the input component.
2. Clear the Define a storage configuration component check
   box to use the local system as your target file system.
3. In the Folder field, set the path to the folder which
   will hold the output data.
4. From the Action list, select the operation for writing
   data:
   • Select Create when you run the Job for the first
     time.
   • Select Overwrite to replace the file every time
     you run the Job.
5. Set the row and field separators in the corresponding fields.
6. Select the Merge results to single file check box, and
   in the Merge file path field set the path where to output
   the file of the suspect record pairs.
7. Double-click the other tFileOutputDelimited components to display the Basic
   settings view and define the component properties.

   For example, set the path where to output the sample data to
   C:/tmp/tmp/pairsSample and set the path where to
   output the file of the suspect sample to
   C:/tmp/pairing/SampleToLabel.csv.

   For example, set the path where to output the unique row to
   C:/tmp/tmp/uniqueRows and set the path where to
   output the file of the suspect pairs sample to
   C:/tmp/pairing/uniqueRows.csv.

1. Double-click tLogRow component and define the component
   properties in the Basic settings view.
   This component writes the exact duplicates generated from the input data on
   the Studio console.
2. Click Sync columns to retrieve the schema from the
   preceding component.
3. In the Mode area, select Table (print values
   in cells of a table) for better readability of the result.