August 15, 2023

Configuring the duplicate data – Docs for ESB 6.x

  2. Talend for ESB Docs
  3. Docs 6.x
  4. Configuring the duplicate data – Docs for ESB 6.x

Configuring the duplicate data

  1. Double-click tDuplicateRow to display the
    Basic settings view and define the
    component properties.

    use_case-tduplicaterow5.png

  2. Click the Edit schema button to view the
    input and output columns and do any modifications in the output schema, if
    needed.

    use_case-tduplicaterow6.png

    The output schema of this component contains one read-only column,
    ORIGINAL_MARK. This column identifies, by
    true or false, if the record is an original or a
    duplicate record. There is only one original record per group of
    duplicates.
  3. In the Percentage of duplicated records
    field, enter the percentage of the duplicate rows you want to have in the output
    flow.
  4. In the Distribution of duplicates area,
    select the Bernoulli distribution (probability
    theory) you want to use to generate duplicates. Set an average of how many
    duplicate records to have in each group.
  5. Click the plus button below the Modifications
    table and add four lines in the table.

    This table enables you to define what values to change in a given column and
    how to change them in order to generate duplicates:
    1. In the Input Column, select the
      column from the input flow from which you want to generate duplicates,
      Name, City and
      DOB in this example.

      When you add a column twice in the table and select different
      functions, you generate duplicates from the same field with different
      values. For example, in this scenario you modify the duplicate names
      with the Soundex replace function with probability
      of 50%, and then you modify it again with the Exchange
      characters       function with the probability of 50%. This
      means, the Name field of the duplicate record may
      still not be modified after the second function with the following
      probability: (1-0.5)*(1-0.5) = 0.25.
      If you want to make sure that all the duplicated records are modified,
      you must set the Modification Rate to
      1.
    2. In the Modification Rate column,
      enter a rate of the duplicate records you want to generate from the
      input column.
    3. From the Function list, select the
      function that will decide what modification to do on a value to
      duplicate it.

      In this example, there will be duplicate names with different sounds
      and characters and duplicate city names with different sounds. Date
      values in the date of birth column will be randomly changed here.
    4. In the Max Modification Count column,
      enter a maximum number of the values to be modified in each
      field.
  6. Click the Advanced settings tab and enter a
    random number in the Seed for random generator
    field.

    By setting a number in this field, you will generate the same sample of
    duplicate data in each execution of the Job. Change the value if you want to
    generate a different sample.
Document get from Talend https://help.talend.com
Thank you for watching.
Docs 6.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x