August 17, 2023

tFuzzyJoin – Docs for ESB 5.x

  2. Talend for ESB Docs
  3. Docs 5.x
  4. tFuzzyJoin – Docs for ESB 5.x

tFuzzyJoin

tFuzzyJoin_icon32_white.png

Warning

This component will be available in the Palette of
Talend Studio on the condition that you have subscribed to one of
the Talend Platform products.

tFuzzyJoin properties

Component family

Data Quality

  

Function

tFuzzyJoin joins two tables by doing a fuzzy
match on several columns. It compares columns from the main flow with reference
columns from the lookup flow and outputs the main flow data and/or the rejected
data.

Purpose

Helps ensuring the data quality of any source data against a reference data
source.

Basic settings

Schema and Edit Schema

A schema is a row description, it defines the number of fields to be processed
and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

 

 

Built-in: You create and store the schema
locally for this component only. Related topic: see Talend Studio User
Guide.

 

 

Repository: You have already created and stored
the schema in the Repository. You can reuse it in other projects and job designs.
Related topic: see Talend Studio User Guide.

 

Include lookup columns in output

Select this check box to include the lookup columns you define in the output
flow.

 

Input key attribute

Select the column(s) from the main flow that needs to be checked against the
reference (lookup) key column.

 

Lookup key attribute

Select the lookup key columns that you will use as a reference against which to
compare the columns from the input flow.

 

Matching type

Select the relevant matching algorithm from the list:

Exact Match: matches each processed entry to all
possible reference entries with exactly the same value.

Levenshtein: Based on the edit distance theory. It
calculates the number of insertion, deletion or substitution required for an entry
to match the reference entry.

Metaphone: Based on a phonetic algorithm for
indexing entries by their pronunciation. It first loads the phonetics of all entries
of the lookup reference and checks all entries of the main flow against the entries
of the reference flow.

Double Metaphone: a new version of the Metaphone
phonetic algorithm, that produces more accurate results than the original algorithm.
It can return both a primary and a secondary code for a string. This accounts for
some ambiguous cases as well as for multiple variants of surnames with common
ancestry.

 

Case sensitive

Select this check box to consider the letter case.

 

Min. distance

Only for Levenshtein. Set the minimum number of
changes allowed to match the reference. If set to 0, only perfect matches(Exact Match) are returned.

Note

You can create and store context variables for the minimum and maximum
distances in order to start from a low max number to match rows and go up to
higher max number to match more possible rows. You can press Ctrl+Space to access the variable list and select the new context
variables. For more information about context variables, see Talend Studio User
Guide.

 

Max. distance

Only for Levenshtein. Set the maximum number of
changes allowed to match the reference.

 

Inner join (with reject output)

Select this check box to join the two tables first and gather the rejected data
from the main flow.

Advanced settings

tStatCatcher
Statistics

Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is not startable and it requires two input components and one or
more output components.

Limitation/prerequisite

n/a

Scenario: Doing a fuzzy match on two columns and outputting the main and rejected
data

This scenario describes a five-component Job aiming at: first checking the edit distance
between the IdClient column of an input file against the data of the
reference input file, and second checking all emails by their pronunciation in the
Email column against the data of the reference input file. The outputs
of these two matching types are written in two separate files.

In this scenario, we have already stored the input schemas of the input and reference
files in the Repository. For more information about storing schema metadata in the Repository tree view, see Talend Studio User
Guide.

Setting up the Job

  1. In the Repository tree view, expand Metadata and the file node where you have stored the input
    schemas and drop the relevant file onto the design workspace.

    The [Components] dialog box displays.

    Use_Case_tFuzzyJoin.png

  2. Select tFileInputDelimited from the list and click
    OK to close the dialog box.

    The tFileInputDelimited component displays in the
    workspace. The input file used in this scenario is called
    ClientSample. It holds 12 columns including the two columns
    IdCient and Email we want to do the fuzzy
    match on.

  3. Do the same for the second input file you want to use as a reference,
    ClientSample_Update in this scenario.

  4. Drop the following components from the Palette onto
    the design workspace: tFuzzyJoin, tFileOutputExcel, and tFileOutputDelimited.

    Use_Case_tFuzzyJoin1.png

  5. Connect the main and reference input files to tFuzzyJoin using Main links. The link
    between the reference input file and tFuzzyJoin
    displays as a Lookup link on the design workspace.

  6. Connect tFuzzyJoin to tFileOutputExcel using the Main link and
    tFuzzyJoin to tFileOutputDelimited using the Inner join
    reject     link.

  7. If needed, double-click the main and reference input files to display their
    Basic settings views. All their property fields are
    automatically filled in. If you do not define your input files in the Repository, fill
    in the details manually after selecting Built-in in the
    Property Type field.

Configuring the tFuzzyJoin component

  1. Double-click tFuzzyJoin to display its Basic settings view and define its properties.

    Use_Case_tFuzzyJoin3.png

  2. Click the Edit schema button to open a dialog box
    that displays the data structure of the input files and then define the data you want to
    pass to the output components, two columns in this scenario,
    IdClient and Email.

    Use_Case_tFuzzyJoin4.png

  3. Click OK to close the dialog box.

  4. In the Key definition area of the Basic settings view of tFuzzyJoin, click the plus button to add two columns to the list and then
    select the input columns and the output columns you want to do the fuzzy matching on
    from the Input key attribute and Lookup key attribute lists respectively, IdClient and
    Email in this example.

  5. Click in the first cell of the Matching type column
    and select from the list the method to be used to check the incoming data against the
    reference data. In this scenario, Levenshtein is the matching type
    to be used.

  6. Then set the minimum and maximum distances. In this method, the distance is the
    number of character changes (insertion, deletion or substitution) that needs to be
    carried out in order for the entry to fully match the reference. In this example, we
    want the minimum distance to be 0 and the maximum distance to be 2. This will output all
    entries in the IdClient that exactly match or that have maximum two
    character changes.

    Note

    You can create and store context variables for the minimum and maximum distances
    in order to start from a low max number to match rows and go up to higher max number
    to match more possible rows. You can press Ctrl+Space
    to access the variable list and select these new context variables. For more
    information about context variables, see Talend Studio User Guide.

  7. Set the matching type for the second column to be checked,
    Metaphone in this example. There is no minimum nor maximum
    distance to set because this matching method is based on phonetic discrepancies between
    the input main and reference data.

  8. Select the Inner join (with reject output) check
    box to define one of the outputs as inner join reject table.

Configuring the output component

  1. Double-click tFileOutputExcel to display its
    Basic settings view and define its properties.

    Use_Case_tFuzzyJoin5.png

  2. Set the destination file name as well as the Sheet name and select the Include header check box.

  3. Double-click tFileOutputDelimited to display its
    Basic settings view and define its properties.

    Use_Case_tFuzzyJoin6.png

  4. Set the destination file name as well as row and field separators in the
    corresponding fields and select the Include header
    check box.

Job execution

Save your Job and click F6 to execute it.

The output of the Levenshtein matching type on the IdClient column
is written in the defined excel file, and the output of the metaphone matching type on the
Email column is written in the defined delimited file.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 5.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x