tDuplicateRow

Warning

This component will be available in the Palette of
Talend Studio on the condition that you have subscribed to one of
the Talend Platform products.

tDuplicateRow properties

Component family	Data Quality
Function	tDuplicateRow generates duplicate data from an input flow. It groups similar duplicates together and identifies the original record of each group by `true`. This component can be used in combination with the tRowGenerator component to generate duplicate data.
Purpose	tDuplicateRow can be used to create duplicates with meaningful data for data quality functional testing purposes.
Basic settings	Schema and Edit schema	A schema is a row description. It defines the number of fields to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository. Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions. Click Sync columns to retrieve the schema from the previous component in the Job. The output schema of this component contains one read-only column, ORIGINAL_MARK. This column identifies by `true` or `false` if the record is an original record or a duplicate record respectively. There is only one original record per group.
		Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.
		Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.
	Percentage of duplicated records	Enter the percentage of the duplicate rows you want to have in the output flow.
	Distribution of duplicates	Name: Select the distribution (probability theory) you want to use to generate duplicates: Bernoulli distribution or Poisson distribution or Geometric distribution. Average group size: Set an average size for all the groups that contain duplicates.
	Modifications	Define in the table what fields to change in a row and how to change them: –Input Column: Select the column from the input flow from which you want to generate duplicates through modifying its values. –Modification Rate: Enter a rate of the modifications you want to have in the duplicate record generated from an input column. The rate is a value between 0 and 1. If you set the rate to 0, no modification will be done. If you set the rate to 0.5, modification will be done in average every two rows. But if you set the rate to 1, modification is done at each row. These modifications are based on the function you select in the Function column and the number of modifications you set in the Max Modification Count column. –Function: Select the function that will decide what modification to do on a value to duplicate it. For example, you can decide to have exact or approximate duplicate values through replacing or adding letters or numbers, replacing values with synonyms from an index file or deleting values by setting the function to null or blank. The Function list will vary according to the column type. For example, a column of a String type will have an Add letters option in the list while a column of an Integer type will have an Add digits option in the list. Also, the Function list for a Date column is date-specific, it allows you to decide the type of modification you want to do on date values. –Max Modification Count: Enter a maximum number of the values to be modified. –Synonym Index Path: Set the path to the index file from which you use the synonym. This field is available if you select the Synonym replace function which means that the value in the duplicate record is replaced by one of its synonym, according to the given the rate. You must use the tSynonymOutput component to create a Lucene index and feed it with synonyms. For further information about how to create a synonym index and define the reference entries, see tSynonymOutput.
Advanced settings	Seed for random generator	Set a random number if you want to generate the same sample of duplicates in each execution of the Job. Repeating the execution with a different value for the seed will result in a different duplicate sample being generated. Keep this field empty if you want to generate a different duplicate sample each time you execute the Job.
	tStat Catcher Statistics	Select this check box to collect log data at the component level.
Usage	This component helps you to generate duplicate data of an input flow according to certain criteria and use it for testing purposes. tDuplicateRow can not be used in Map/Reduce Jobs for the time being.
Limitation/prerequisite	n/a

Scenario: Generating duplicate data from an input flow

This scenario describes a basic Job that generates a sample of duplicate data from an
input flow by using probability theories and specific criteria on three columns:
Name, City and DOB (date of
birth).

Below is a capture of a sample data of the input flow:

Setting up the Job

Drop the following components from the Palette onto
the design workspace: tFileInputDelimited, tDuplicateRow and tFileOutputDelimited.
Connect all the components together using the Row
> Main link.

Configuring the input data

Double-click tFileInputDelimited to display the
Basic settings view and define the component
properties.
In the File name/Stream field, browse to the file
to be used as the main input.

This file provides some information about customers.
Define the row and field separators the header and footer in the corresponding
fields, if any.
Click the […] button next to Edit schema to open a dialog box and define the input schema.
According to the input file structure, the schema is made of ten columns.
Click the [+] button and define the input columns
in the dialog box as in the above figure. Click OK to
close the dialog box.
If needed, right-click tFileInputDelimited and
select Data Viewer to display a view of the input
data.

Configuring the duplicate data

Double-click tDuplicateRow to display the Basic settings view and define the component properties.
Click the Edit schema button to view the input and
output columns and do any modifications in the output schema, if needed.

The output schema of this component contains one read-only column,
ORIGINAL_MARK. This column identifies, by true or
false, if the record is an original or a duplicate record. There is only
one original record per group of duplicates.
In the Percentage of duplicated records field,
enter the percentage of the duplicate rows you want to have in the output flow.
In the Distribution of duplicates area, select the
distribution (probability theory) you want to use to generate duplicates,
Geometric distribution in this example. Then set an average of
how many duplicate records to have in each group.
Click the plus button below the Modifications table
and add four lines in the table.

This table enables you to define what values to change in a given column and how to
change them in order to generate duplicates:
1. In the Input Column, select the column from the
  input flow from which you want to generate duplicates, Name,
  City and DOB in this example.
  
  When you add a column twice in the table and select different functions, you
  generate duplicates from the same field with different values. For example, in this
  scenario you modify the duplicate names with the Soundex
  replace function with probability of 50%, and then you modify it again
  with the Exchange characters function with the probability of
  50%. This means, the Name field of the duplicate record may
  still not be modified after the second function with the following probability:
  (1-0.5)*(1-0.5) = 0.25.
  
  If you want to make sure that all the duplicated records are modified, you must
  set the Modification Rate to
  1.
2. In the Modification Rate column, enter a rate
  of the duplicate records you want to generate from the input column.
3. From the Function list, select the function
  that will decide what modification to do on a value to duplicate it.
  
  In this example, there will be duplicate names with different sounds and
  characters and duplicate city names with different sounds. Date values in the date
  of birth column will be randomly changed here.
4. In the Max Modification Count column, enter a
  maximum number of the values to be modified in each field.
Click the Advanced settings tab and enter a random
number in the Seed for random generator field.

By setting a number in this field, you will generate the same sample of duplicate
data in each execution of the Job. Change the value if you want to generate a diffierent
sample.

Configuring the output component

Double-click tFileOutputDelimited to display its
Basic settings view and define the component
properties.
In the File Name field, specify the path to the
file to which you want to write the duplicate data.
Define the row and field separators in the corresponding fields, if any.

Executing the Job

Save your Job and press F6 to execute it.

Duplicate data is generated and written to the output file.
Right-click the output component and select Data
Viewer to display the duplicate data.

Duplicate records have been marked as false in the
ORIGINAL_MARK column.

Some data has been modified in the Name,
City and DOB fields according to the
criteria you set in the Modifications table and
duplicate records have been generated based on these modifications.

For example, if you compare the original name Mrs Morgan Ross
and the duplicate name Mrs M rganosRiss, you will see that the two
functions have been used on this duplicate record: the letter o has
been exchanged with a space, and also the sound has been replaced in
Ross and Riss. However, the soundex code has
not been changed for the replaced sound.
In the tDuplicateRow basic settings and in the
Distribution of duplicates area, select a different
distribution, Bernoulli distribution for example, and run the
Job.

Different duplicates are generated from the same input flow according to the
selected distribution as shown in the below figure.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 5.x

0 Comments

Inline Feedbacks

View all comments

tDuplicateRow – Docs for ESB 5.x