July 30, 2023

tFuzzyMatch – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tFuzzyMatch – Docs for ESB 7.x

tFuzzyMatch

Compares a column from the main flow with a reference column from the lookup flow and
outputs the main flow data displaying the distance.

tFuzzyMatch Standard properties

These properties are used to configure tFuzzyMatch running in the Standard Job framework.

The Standard
tFuzzyMatch component belongs to the Data Quality family.

The component in this framework is available in all Talend
products.

Basic settings

Schema and Edit
schema

A schema is a row description, it defines the number of fields to
be processed and passed on to the next component. The schema is
either Built-in or stored remotely
in the Repository.

Two read-only columns, Value and Match are added to the output
schema automatically.

 

Built-in: The schema will be
created and stored locally for this component only. Related topic:
see
Talend Studio User
Guide.

 

Repository: The schema already
exists and is stored in the Repository, hence can be reused in
various projects and job designs. Related topic: see

Talend Studio User
Guide.

Matching type

Select the relevant matching algorithm among:

Levenshtein: Based on the edit
distance theory. It calculates the number of insertion, deletion or
substitution required for an entry to match the reference entry.

Metaphone: Based on a
phonetic algorithm for indexing entries by their pronunciation. It first loads
the phonetics of all entries of the lookup reference and checks all entries of
the main flow against the entries of the reference flow. It does not support Chinese characters.

Double Metaphone: a new
version of the Metaphone phonetic algorithm, that produces more accurate
results than the original algorithm. It can return both a primary and a
secondary code for a string. This accounts for some ambiguous cases as well as
for multiple variants of surnames with common ancestry. It does not support Chinese characters.

Min distance

(Levenshtein only) Set the minimum number of changes allowed to
match the reference. If set to 0, only perfect matches are returned.

Max distance

(Levenshtein only) Set the maximum number of changes allowed to
match the reference.

Matching column

Select the column of the main flow that needs to be checked
against the reference (lookup) key column

Unique matching

Select this check box if you want to get the best match possible,
in case several matches are available.

Matching item separator

In case several matches are available, all of them are displayed
unless the unique match box is selected. Define the delimiter
between all matches.

Advanced settings

tStatCatcher Statistics

Select this check box to collect log data at the component level.

Global Variables

Global Variables

NB_LINE: the number of rows read by an input component or
transferred to an output component. This is an After variable and it returns an
integer.

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

This component is not startable (green background) and it requires
two input components and an output component.

Checking the Levenshtein distance of 0 in first names

This scenario describes a four-component Job aiming at checking the edit distance
between the First Name column of an input file with the data of the
reference input file. The output of this Levenshtein type check is displayed along with
the content of the main flow on a table

tFuzzyMatch_1.png

Setting up the Job

  1. Drag and drop the following components from the Palette to the design workspace: tFileInputDelimited (x2), tFuzzyMatch, tLogRow.
  2. Link the first tFileInputDelimited
    component to the tFuzzyMatch component
    using a Row > Main connection.
  3. Link the second tFileInputDelimited
    component to the tFuzzyMatch using a
    Row > Main connection (which appears as a Lookup row on the design workspace).
  4. Link the tFuzzyMatch component to the
    standard output tLogRow using a Row > Main
    connection.

Configuring the components

  1. Define the first tFileInputDelimited in
    its Basic settings view. Browse the system
    to the input file to be analyzed.
  2. Define the schema of the component. In this example, the input schema has
    two columns, firstname and gender.
  3. Define the second tFileInputDelimited
    component the same way.

    Warning:

    Make sure the reference column is set as key column in the
    schema of the lookup flow.

    tFuzzyMatch_2.png

  4. Double-click the tFuzzyMatch component to
    open its Basic settings view, and check its
    schema.

    The Schema should match the Main input flow schema in order for the main flow
    to be checked against the reference.
    tFuzzyMatch_3.png

    Note that two columns, Value and
    Matching, are added to the output
    schema. These are standard matching information and are read-only.
  5. Select the method to be used to check the incoming data. In this scenario,
    Levenshtein is the Matching
    type     to be used.
  6. Then set the distance. In this method, the distance is the number of char
    changes (insertion, deletion or substitution) that needs to be carried out
    in order for the entry to fully match the reference.

    tFuzzyMatch_4.png

    In this use case, we set both the minimum distance and the maximum
    distance to 0. This means only the exact
    matches will be output.
  7. Also, clear the Case sensitive check
    box.
  8. Check that the matching column and look up column are correctly
    selected.
  9. Leave the other parameters as default.

Executing the Job

Save the Job and press F6 to execute the
Job.

tFuzzyMatch_5.png

As the edit distance has been set to 0 (min and max), the output shows the result
of a regular join between the main flow and the lookup (reference) flow, hence only
full matches with Value of 0 are displayed.

A more obvious example is with a minimum distance of 1 and a maximum distance of
2, see Procedure

Checking the Levenshtein distance of 1 or 2 in first names

This scenario is based on the scenario described above. Only the minimum and maximum
distance settings in the tFuzzyMatch component are
modified, which will change the output displayed.

Procedure

  1. In the Component view of the tFuzzyMatch, change the minimum distance from
    0 to 1. This excludes straight away the exact matches (which would show a
    distance of 0).
  2. Change also the maximum distance to 2. The
    output will provide all matching entries showing a discrepancy of 2 characters
    at most.

    tFuzzyMatch_6.png

    No other changes are required.
  3. Make sure the Matching item separator is
    defined, as several references might be matching the main flow entry.
  4. Save the new Job and press F6 to run
    it.

    tFuzzyMatch_7.png

    As the edit distance has been set to 2,
    some entries of the main flow match more than one reference entry.

You can also use another method, the metaphone, to assess the distance between the
main flow and the reference, which will be described in the next scenario.

Checking the Metaphonic distance in first name

This scenario is based on the scenario described above.

Procedure

  1. Change the Matching type to Metaphone. There is no minimum nor maximum distance
    to set as the matching method is based on the discrepancies with the phonetics
    of the reference.

    tFuzzyMatch_8.png

  2. Save the Job and press F6. The phonetics
    value is displayed along with the possible matches.

    tFuzzyMatch_9.png

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x