August 16, 2023

tMatchIndexPredict – Docs for ESB 6.x

  2. Talend for ESB Docs
  3. Docs 6.x
  4. tMatchIndexPredict – Docs for ESB 6.x

tMatchIndexPredict

Compares a new data set with a lookup data set stored in ElasticSearch, using
tMatchIndex. tMatchIndexPredict outputs unique
records and suspect duplicates in separate files.

In the potential duplicates output, each record contains the fields from the source
records and the fields from the potentially matching lookup records.

For more information about tMatchIndex, see
the tMatchIndex documentation on Talend Help Center (https://help.talend.com).

This component can run only with Spark 2.0+ and ElasticSearch 5+.

tMatchIndexPredict properties for Apache Spark Batch

These properties are used to configure tMatchIndexPredict
running in the Spark Batch Job framework.

The Spark Batch
tMatchIndexPredict component belongs to the Data Quality family.

The component in this framework is available when you have subscribed to any Talend Platform product with Big Data or Talend Data
Fabric.

Basic settings

Define a storage configuration component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job. For
example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Sync columns to retrieve the schema from
the previous component connected in the Job.

Click Edit schema to make changes to the schema.
If the current schema is of the Repository type, three
options are available:

  • View schema: choose this option to view the
    schema only.

  • Change to built-in property: choose this
    option to change the schema to Built-in for
    local changes.

  • Update repository connection: choose this
    option to change the schema stored in the repository and decide whether to propagate
    the changes to all the Jobs upon completion. If you just want to propagate the
    changes to the current Job, you can select No
    upon completion and choose this schema metadata again in the [Repository Content] window.

You need to manually edit the output schema to add the necessary columns
that hold the fields from the lookup data.

The output schema of this component contains a read-only column:

LABEL: used only with the
Possible matches output link.

 

Built-In: You create and store the
schema locally for this component only. Related topic: see
Talend Studio

User Guide.

 

Repository: You have already created
the schema and stored it in the Repository. You can reuse it in various projects and
Job designs. Related topic: see
Talend Studio

User Guide.

ElasticSearch configuration

Nodes: Enter the location
of the cluster hosting the ElasticSearch system to be used.

Index: Enter the name of the ElasticSearch index
where the lookup data is stored.

Models

Pairing model folder: Set the path to the folder
which has the model files generated by the
tMatchPairing component.

Matching model
location: Select from the list where to get the model
file generated by the classification Job with the tMatchModel component:

  • from file system: In the
    Matching model folder, set the path
    to the folder where the model file is generated by the
    tMatchModel component.
  • from current Job: From the
    Model name list, select the name of
    the model file generated by the classification component. You
    can use this option only if the classification Job with the
    tMatchModel component is integrated
    in the Job with the tMatchIndexPredict
    component.

Matching model folder: Set the
path to the folder which has the model files generated by the tMatchModel component.

No-match label: Enter the label used for the
unique records in the output.

If you want to store the model in a specific file system, for example S3
or HDFS, you must use the corresponding component in the Job and select
the Define a storage configuration component
check box in the component basic settings.

The button for browsing does not work with the Spark Local mode; if you are using the Spark Yarn or the Spark Standalone mode,
ensure that you have properly configured the connection in a configuration component in
the same Job, such as tHDFSConfiguration.

Usage

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Batch component Palette it belongs to, appears only
when you are creating a Spark Batch Job.

Spark Batch Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Scenario: Doing continuous matching using tMatchIndexPredict

This scenario applies only to a subscription-based Talend Platform solution with Big data or Talend Data Fabric.

After indexing lookup data in Elasticsearch using tMatchIndex, you do
not need to restart the matching process from scratch. The
tMatchIndexPredict component compares new data records with the
lookup stored in ElasticSearch.

In this example, a list of early childhood education centers in Chicago coming from ten
different source has been cleaned, deduplicated and indexed in Elasticsearch. You want
to match new records which contain information about early childhood education centers
in Chicago against the reference data set stored in Elasticsearch.

tMatchIndexPredict uses pairing and matching models to group together
records from the input data and the matching records from the reference data set indexed
in Elasticsearch and label the suspect pairs.

tMatchIndexPredict outputs potential duplicates and unique records in
separate files.

Before you begin:

  • You generated a pairing model.

    You can find an example of how to generate a pairing
    model on Talend Help Center (https://help.talend.com).

  • You generated a matching model.

    You can find an example of how to generate a
    matching model on Talend Help Center (https://help.talend.com).

  • Clean and deduplicated data has been indexed in Elasticsearch to match
    against new data records and determine whether they are unique records or
    suspect duplicates.

    You can find an example of how to index clean and
    deduplicated data in ElasticSearch on Talend Help Center (https://help.talend.com).

  • The Elasticsearch search cluster must be running ElasticSearch 5+.

Setting up the Job

  1. Drop the following components from the Palette onto the
    design workspace: tFileInputDelimited,
    tMatchIndexPredict and two
    tFileOutputDelimited components.
  2. Connect tFileInputDelimited to the
    tMatchIndexPredict using a Row > Main connection.
  3. Connect tMatchIndexPredict to the first
    tFileOutputDelimited using a Row > Suspect duplicates connection.
  4. Connect tMatchIndexPredict to the second
    tFileOutputDelimited using a Row > Unique rows connection.
use_case_tmatchindexpredict.png

Configuring the input component

  1. Double-click the tFileInputDelimited component to open
    its Basic settings view.

    use_case_tmatchindexpredict2.png

  2. Click the […] button next to Edit
    schema     and use the [+] button in the
    dialog box to add String type columns: Original_Id,
    Source, Site_name and
    Address.
  3. Click OK in the dialog box and accept to propagate the
    changes when prompted.
  4. In the Folder/File field, set the path to the input
    file.
  5. Set the row and field separators in the corresponding fields and the header and
    footer, if any.

Configuring the tMatchIndexPredict component

  1. Double-click the tMatchIndexPredict component to open
    its Basic settings view.

    use_case_tmatchindexpredict3.png

  2. In the ElasticSearch configuration area, enter the
    location of the cluster hosting the Elasticsearch system to be used in the
    Nodes field, for example:

    "localhost:9200"

  3. In the ElasticSearch configuration area, enter the name
    of the Elasticsearch index where the reference data is stored in the
    Index field, for example:

    "education-agencies-chicago"

  4. In the Models area, set the information about the
    pairing and matching models:

    1. Set the path to the folder containing the model files generated by the
      tMatchPairing component in the Pairing
      model folder       field.
    2. Select from the Matching model location list
      where to get the model file generated by the
      tMatchModel component.

      In this example, select from file system
      because the classification Job using the
      tMatchModel component is not integrated to
      the current Job.

    3. Set the path to the folder containing the model file generated by the
      tMatchModel component in the
      Matching model folder field.
    4. Set the label used for the unique records output in the
      No-match label field.

Computing suspect pairs and unique rows

  1. Double-click the first tFileOutputDelimited
    component to display the Basic settings view and
    define the component properties.

    You have already accepted to propagate the schema to the output
    components when you defined the input component.
  2. Clear the Define a storage configuration component
    check box to use the local system as your target file system.
  3. Click the […] button next to Edit
    schema     and use the [+] button in the
    dialog box to add the columns from the reference data set to the schema.

    You must add _ref at the end of the column names
    to be added to the suspect duplicates output. In this example:
    Original_id_ref,
    Source_ref,
    Site_name_ref and
    Address_ref.

    use_case_tmatchindexpredict6.png

  4. In the Folder field, set the path to the folder
    which will hold the output data.
  5. From the Action list, select the operation for
    writing data:

    • Select Create when you run the Job for the
      first time.

    • Select Overwrite to replace the file every
      time you run the Job.

  6. Set the row and field separators in the corresponding fields.
  7. Select the Merge results to single file check box,
    and in the Merge file path field set the path where
    to output the file of the suspect record pairs.
  8. Double-click the second tFileOutputDelimited
    component and define the component properties in the Basic
    settings     view, as you do with the first component.

    This component creates the file which holds the unique rows generated
    from the input data.


  9. Press F6 to save and execute the
    Job.
tMatchIndexPredict groups together records from the input data and
the matching records from the reference data set indexed in Elasticsearch and labels
the suspect pairs.

use_case_tmatchindexpredict7.png
tMatchIndexPredict excludes unique records to write them in
another file.

use_case_tmatchindexpredict8.png

You can now clean and deduplicate the unique rows and use
tMatchIndex to add them to the reference data set stored in
Elasticsearch.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 6.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x