• You generated a pairing model and computed pairs of suspect
  duplicates using tMatchPairing.
• You labeled a sample of the suspect pairs manually or using
  Talend Data Stewardship to generate a
  matching model with tMatchModel.
• You predicted labels on suspect pairs based on the pairing and
  matching models using tMatchPredict.
• You cleaned and deduplicated the data set using tRuleSurvivorship.

These properties are used to configure tMatchIndex running in
the Spark Batch Job framework.

The Spark Batch
tMatchIndex component belongs to the Data Quality
family.

The component in this framework is available in all Talend Platform products with Big Data and in Talend Data Fabric.

Basic settings

Advanced settings

Usage

• You generated a pairing model using tMatchPairing.

  You can find examples of how to generate a pairing
  model on Talend Help Center (https://help.talend.com).

• Make sure the input data you want to index is clean and deduplicated.

  You can find an example of how to clean and
  deduplicate a data set on Talend Help Center (https://help.talend.com).

• The Elasticsearch cluster must be running Elasticsearch 5+.

1. Drop the following components from the Palette onto the
   design workspace: tFileInputDelimited and
   tMatchIndex.
2. Connect the components using a Row > Main connection.

1. Click Run to open its view and then click the
   Spark Configuration tab to display its view
   for configuring the Spark connection.
2. Select the Use local mode check box to test your Job locally.

   In the local mode, the Studio builds the Spark environment in itself on the fly in order to
   run the Job in. Each processor of the local machine is used as a Spark
   worker to perform the computations.

   In this mode, your local file system is used; therefore, deactivate the
   configuration components such as tS3Configuration or
   tHDFSConfiguration that provides connection
   information to a remote file system, if you have placed these components
   in your Job.

   You can launch
   your Job without any further configuration.

3. Clear the Use local mode check box to display the
   list of the available Hadoop distributions and from this list, select
   the distribution corresponding to your Spark cluster to be used.
   This distribution could be:

   • Databricks

   • Qubole

   • Amazon EMR

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • Cloudera

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Google Cloud
     Dataproc

     For this distribution, Talend supports:

     • Yarn client

   • Hortonworks

     For this distribution, Talend supports:

     • Yarn client

     • Yarn cluster

   • MapR

     For this distribution, Talend supports:

     • Standalone

     • Yarn client

     • Yarn cluster

   • Microsoft HD
     Insight

     For this distribution, Talend supports:

     • Yarn cluster

   • Cloudera Altus

     For this distribution, Talend supports:

     • Yarn cluster

       Your Altus cluster should run on the following Cloud
       providers:

       • Azure

         The support for Altus on Azure is a technical
         preview feature.

       • AWS

   As a Job relies on Avro to move data among its components, it is recommended to set your
   cluster to use Kryo to handle the Avro types. This not only helps avoid
   this Avro known issue but also
   brings inherent preformance gains. The Spark property to be set in your
   cluster is:
   

   spark.serializer org.apache.spark.serializer.KryoSerializer

   1	spark.serializer org.apache.spark.serializer.KryoSerializer

   If you cannot find the distribution corresponding to yours from this
   drop-down list, this means the distribution you want to connect to is not officially
   supported by
   Talend
   . In this situation, you can select Custom, then select the Spark
   version of the cluster to be connected and click the
   [+] button to display the dialog box in which you can
   alternatively:

   1. Select Import from existing
      version to import an officially supported distribution as base
      and then add other required jar files which the base distribution does not
      provide.

   2. Select Import from zip to
      import the configuration zip for the custom distribution to be used. This zip
      file should contain the libraries of the different Hadoop/Spark elements and the
      index file of these libraries.

      In
      Talend
      
      Exchange, members of
      Talend
      community have shared some ready-for-use configuration zip files
      which you can download from this Hadoop configuration
      list and directly use them in your connection accordingly. However, because of
      the ongoing evolution of the different Hadoop-related projects, you might not be
      able to find the configuration zip corresponding to your distribution from this
      list; then it is recommended to use the Import from
      existing version option to take an existing distribution as base
      to add the jars required by your distribution.

      Note that custom versions are not officially supported by
      
      Talend
      .
      Talend
      and its community provide you with the opportunity to connect to
      custom versions from the Studio but cannot guarantee that the configuration of
      whichever version you choose will be easy. As such, you should only attempt to
      set up such a connection if you have sufficient Hadoop and Spark experience to
      handle any issues on your own.

   For a step-by-step example about how to connect to a custom
   distribution and share this connection, see Hortonworks.

1. Double-click tHDFSConfiguration to open its Component view.

   Spark uses this component to connect to the HDFS system to which the jar
   files dependent on the Job are transferred.

2. If you have defined the HDFS connection metadata under the Hadoop
   cluster node in Repository, select
   Repository from the Property
   type drop-down list and then click the
   […] button to select the HDFS connection you have
   defined from the Repository content wizard.

   For further information about setting up a reusable
   HDFS connection, search for centralizing HDFS metadata on Talend Help Center
   (https://help.talend.com).

   If you complete this step, you can skip the following steps about configuring
   tHDFSConfiguration because all the required fields
   should have been filled automatically.

3. In the Version area, select
   the Hadoop distribution you need to connect to and its version.
4. In the NameNode URI field,
   enter the location of the machine hosting the NameNode service of the cluster.
   If you are using WebHDFS, the location should be
   webhdfs://masternode:portnumber; WebHDFS with SSL is not
   supported yet.
5. In the Username field, enter
   the authentication information used to connect to the HDFS system to be used.
   Note that the user name must be the same as you have put in the Spark configuration tab.

1. Double-click the tFileInputDelimited component to open
   its Basic settings view and define its properties.
   

2. Click the […] button next to Edit
   schema and use the [+] button in the
   dialog box to add String type columns: Original_Id,
   Source, Site_name and
   Address.
3. In the Folder/File field, set the path to the input
   file.
4. Set the row and field separators in the corresponding fields and the header and
   footer, if any.

1. Double click the tMatchIndex component to open its
   Basic settings view and define its properties.
   

2. In the Elasticsearch configuration area, enter the
   location of the cluster hosting the Elasticsearch system to be used in the
   Nodes field, for example:

   "localhost:9200"

3. Enter the index to be created in Elasticsearch in the
   Index field, for example:

   education-agencies-chicago

4. If you need to clean the Elasticsearch index specified in the
   Index field, select the Reset
   index check box.
5. Enter the path to the local folder from where you want to retrieve the pairing
   model files in the Pairing model folder.
6. 
   Press F6 to save and execute the
   Job.
   

tMatchIndex created the
education-agencies-chicago index in Elasticsearch,
populated it with the clean data and computed the best suffixes based on the
blocking key values.

You can browse the index created by tMatchIndex using the
plugin Elasticsearch-head.

You can now use the indexed data as a reference data set for the
tMatchIndexPredict component.

You can find an example of how to do continuous matching
using tMatchIndexPredict on Talend Help Center (https://help.talend.com).