August 17, 2023

tBlockedFuzzyJoin – Docs for ESB 5.x

  2. Talend for ESB Docs
  3. Docs 5.x
  4. tBlockedFuzzyJoin – Docs for ESB 5.x

tBlockedFuzzyJoin

tFuzzyJoin_icon32_white.png

Warning

This component will be available in the Palette of
Talend Studio on the condition that you have subscribed to one of
the Talend Platform products.

tBlockedFuzzyJoin properties

Component
family

Data Quality

  

Function

tBlockedFuzzyJoin
joins two tables by doing a fuzzy match
on several columns. It compares columns from the
main flow with reference columns from the lookup
flow and outputs the match data, the possible
match data and the rejected data.

Purpose

Helps ensuring the data quality of any
source data against a reference data
source.

Basic
settings

Schema and
Edit Schema

A schema is a row description, it defines
the number of fields to be processed and passed on
to the next component. The schema is either
Built-in or
stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

 

 

Built-in:
You create and store the schema locally for this
component only. Related topic: see Talend Studio User Guide .

 

 

Repository:
You have already created and stored the schema in
the Repository. You can reuse it in other projects
and Job designs. Related topic: see Talend Studio User Guide .

 

Replace output column with lookup
column if matches or possible
matches

Select this check box to replace the output
column with the lookup column in case of match or
possible match values.

Key definition

Input key
attribute

Select the column(s) from the main flow that
needs to be checked against the reference (lookup)
key column.

 

Lookup key
attribute

Select the lookup key columns that you will
use as a reference against which to compare the
columns from the input flow.

 

Matching type

Select the relevant matching algorithm from
the list:

Exact Match:
matches each processed entry to all possible
reference entries that have exactly the same
value.

Levenshtein:
Based on the edit distance theory. It calculates
the number of insertion, deletion or substitution
required for an entry to match the reference
entry.

Metaphone: Based
on a phonetic algorithm for indexing entries by
their pronunciation. It first loads the phonetics
of all entries of the lookup reference and checks
all entries of the main flow against the entries
of the reference flow.

Double Metaphone:
a new version of the Metaphone phonetic algorithm,
that produces more accurate results than the
original algorithm. It can return both a primary
and a secondary code for a string. This accounts
for some ambiguous cases as well as for multiple
variants of surnames with common ancestry.

 

Case sensitive

Select this check box to consider the letter
case.

 

Min. distance

Only for Levenshtein. Set the minimum number of
changes allowed to match the reference. If set to
0, only perfect matches(Exact Match) are returned.

Note

You can create and store context variables
for the minimum and maximum distances and then
have your Job to loop on these values in order to
start from a low max number to match rows and go
up to higher max number to match more possible
rows. You can press Ctrl+Space to access the variable list
and select the new context variables. For more
information about context variables, see Talend Studio User Guide.

 

Max. distance

Only for Levenshtein. Set the maximum number of
changes allowed to match the reference.

Advanced
settings

tStatCatcher
Statistics

Select this check box to collect log data at
the component level.

Global
Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is not startable (green
background) and it requires two input components
and one or more output components.

Limitation/prerequisite

n/a

Scenario: Doing a fuzzy match on two columns and outputting the match,
possible match and non match values

This scenario describes a six-component Job that aims at:

  • matching each processed group number in the
    grp column against the entries
    that have exactly the same values in the reference input
    file,

  • checking the edit distance between the entries in the
    firstname column of an input
    file against those of the reference input file.

The outputs of these two matching types are written in three output files: the
first for match values, the second for possible match values and the third
for the values for which there are no matches in the lookup file.

In this scenario, we have already stored the main and reference input schemas
in the Repository. For more information about storing schema metadata in the
Repository, see Talend Studio User Guide.

The main input file contains four columns: grp,
gender, firstname and
count. The data in this input file have
problems such as duplication, first names spelled differently or wrongly,
different information for the same customer.

Use_Case_tMergeRow.png

Setting up the Job

  1. In the Repository tree view,
    expand Metadata and the
    FileExcel node
    where you have stored the main input schemas and then drop
    it onto the design workspace.

    A tFileInputExcel component
    holding your schema displays on the workspace.

  2. Do the same with the reference input schema to display the
    reference tFileInputExcel
    on the workspace.

  3. Drop a tBlockedFuzzyJoin and
    tLogRow (x3) from
    the Palette onto the design
    workspace.

    Use_Case_tBlockedFuzzyJoin.png

  4. Connect the main and reference input Excel files to tBlockedFuzzyJoin using
    Main links. The
    link between the reference input Excel file and tBlockedFuzzyJoin displays as
    a Lookup link on the design
    workspace.

  5. Connect tBlockedFuzzyJoin to
    the three tLogrow
    components using the Matches, Possible
    Matches     and Non
    Matches     links.

Configuring the input components

Double-click the main and reference input Excel file
components to display their Basic
settings views.

The capture below shows the properties of the main input
file.

Use_Case_tBlockedFuzzyJoin1.png

The capture below shows the properties of the reference input file.

Use_Case_tBlockedFuzzyJoin2.png

The property fields for both tFileInputExcel
components are automatically filled in. If you do not define your input
schemas locally in the Repository, fill in the details manually after
selecting Built-in in the Schema and Property
Type fields.

Configuring the tBlockedFuzzyJoin component

  1. Double-click tBlockedFuzzyJoin to display its
    Basic settings view
    and define its properties.

    Use_Case_tBlockedFuzzyJoin3.png

  2. Click the Edit schema button
    to open a dialog box. Here you can define the data you want
    to pass to the output components.

    In this example we want to pass the four input columns to the output
    components in addition to the new column
    ref_firstname.

    Use_Case_tBlockedFuzzyJoin4.png

  3. Click OK to close the dialog
    box and proceed to the next step.

  4. In the Key definition area of
    the Basic settings view of
    tBlockedFuzzyJoin,
    click the plus button to add two columns to the list.

  5. Select the input columns and the output columns you want to do
    the fuzzy matching on from the Input
    key attribute     and Lookup key attribute lists respectively,
    grp and
    firstname in this
    example.

  6. Click in the first cell of the Matching
    type     column and select from the list the
    method to be used to check the incoming data against the
    reference data, Exact match in this
    example. There is no minimum nor maximum distance to
    set.

  7. Set the matching type for the second column,
    Levenshtein in this
    example.

  8. Then set the minimum and maximum distances. In this method,
    the distance is the number of character changes (insertion,
    deletion or substitution) that needs to be carried out in
    order for the entry to fully match the reference. In this
    example, we want the min. distance to be 0 and the max.
    distance to be 2. This will output all entries in the
    firstname column that exactly
    match or that have maximum two character changes.

Job execution

  1. Double-click the first tLogRow
    component to display its Basic settings view and define its
    properties.

    Use_Case_tBlockedFuzzyJoin5.png

  2. In the Mode area, select
    Table to display
    the source file and the tBlockedFuzzyJoin results together to be
    able to compare them.

  3. Do the same for the other two
    tLogRow     components.

  4. Save your Job and press F6 to
    execute it.

Three output tables are written on the console. The first shows the match
entries, the second show the possible match entries and the third shows the
non match entries according to the used matching method in the defined
columns.

The figure below illustrates extractions of the three output tables.

use_case-tblockedfuzzyjoin.png
Document get from Talend https://help.talend.com
Thank you for watching.
Docs 5.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x