tPartitioner

Warning

This component will be available in the Palette of
the Integration perspective of your studio on the
condition that you have subscribed to one of the Talend Platform
solutions or Big Data solutions.

Note that Talend Studio also enables the automatic implementation of parallelization across a
Job without use of the parallelization components and we recommend using that approach. For
further information, see the section describing how to enable parallelization of data flows
of the Talend Studio User Guide. However,
if you need to understand how to use these specific parallelization components, bear in mind
that the parallelization components work closely with each other to accomplish parallel
execution on given processes: the tPartitioner component
dispatches the input records into a specific number of threads; the tCollector component sends these threads to its following components for
parallel execution; the tDepartitioner component regroups
the outputs of the processed parallel threads; the tRecollector component captures the output of a given tDepartitioner component and sends the captured data to the next
component.

tPartitioner Properties

Component family	Orchestration
Function	This component splits the input records into a given number of threads which tCollector sends for parallel execution.
Purpose	This component partitions the input data before tCollector can transfer them to the parallel execution processes.
Basic settings	Schema and Edit Schema	A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository. Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window. Click Sync columns to retrieve the schema from the previous component connected in the Job.
	Number of Child Threads	Enter the number of threads you want to split the input records up into. We recommend that this number be N-1 where N is the total number of CPUs or cores on the machine processing the data.
	Buffer Size	Enter the number of rows to be processed before the memory is freed. This is the number of rows to cache for each of the threads generated.
	Use a key hash for partitions	Select this check box to use the hash mode to dispatch the input records into threads. Once selecting it, the Key Columns table appears, in which you set the column(s) you want to apply the hash mode on. In the hash mode, the records meeting the same criteria are dispatched into the same threads. If you leave this check box clear, the dispatch mode is Round-robin, meaning records are dispatched one-by-one to each thread, in a circular fashion, until the last record is dispatched. Be aware that this mode cannot guarantee that records meeting the same criteria go into the same threads.
Advanced settings	tStatCatcher Statistics	Select this check box to collect the log data at the component level.
Usage	This component should be put after the input component(s) and before tCollector. This component uses and can only use the Trigger > Start link to connect to tCollector.
Global Variables	ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box. NB_LINE: the number of rows processed. This is an After variable and it returns an integer. A Flow variable functions during the execution of a component while an After variable functions after the execution of the component. To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it. For further information about variables, see Talend Studio User Guide.
Connections	Outgoing links (from this component to another): Trigger: Start. Incoming links (from one component to this one): Row: Main. For further information regarding connections, see Talend Studio User Guide.
Log4j	The activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide. For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.
Limitation	n/a

Scenario: sorting the customer data of large size in parallel

The Job in this scenario puts in order 20 million customer records by running
parallelized executions.

Linking the components

In the Integration perspective
of your studio, create an empty Job from the Job
Designs node in the Repository tree view.

For further information about how to create a Job, see Talend Studio User Guide.
Drop the following components onto the workspace: tFileInputDelimited, tPartitioner, tCollector,
tSortRow, tDepartitioner, tRecollector, tFileOutputDelimited.

The tFileInputDelimited component
(labeled test file in this example) reads
the 20 million customer records from a .txt file generated by tRowGenerator.

For further information about the tRowGenerator component, see tRowGenerator

For further information about how to label a component, see Talend Studio User Guide.
Connect tPartitioner to tCollector using the Trigger > Starts link.
Do the same to connect tDepartitioner to
tRecollector.
Connect the other components using the Row >
Main link.

Splitting the input data flow

Configuring the input flow

Double-click tFileInputDelimited to open
its Component view.
In the File name/Stream field, browse to,
or enter the path to the file storing the customer records to be
read.
Click the button to open the schema editor where you need to
create the schema to reflect the structure of the customer data.
Click the button five times to add five rows and rename them as
follows: FirstName, LastName, City, Address and
ZipCode.

In this scenario, we leave the data types with their default value
String. In the real-world practice, you
can change them depending on the data types of your data to be
processed.
Click OK to validate these changes and
accept the propagation prompted by the pop-up dialog box.
If needs be, complete the other fields of the Component view with values corresponding to your data to be
processed. In this scenario, we leave them as is.

Configuring the tPartitioner component

Double-click tPartitioner to open its
Component view.
In the Number of Child Threads field,
enter the number of the threads you want to partition the data flow into. In
this example, enter 3 because we are
using 4 processors to run this Job.
If required, change the value in the Buffer
Size field to adapt the memory capacity. In this example, we
leave the default one.
Click the button next to Edit
schema to open the schema editor.
Select all the rows in the tPartitioner
component schema table using Ctrl or
Shift.

Then the on the toolbar is activated.
Click to copy the schema selected.

Sorting the input records

Configuring tCollector

Double-click tCollector to open its
Component view.
Click the button next to Edit
schema to open the schema editor.
Press Ctrl+V to paste the tPartitioner component schema.
Click OK to validate these changes and
accept the propagation prompted by the pop-up dialog box.

Configuring tSortRow

Double-click tSortRow to open its
Component view.
Under the Criteria table, click the button three times to add three rows to the
table.
In the Schema column column, select, for
each row, the schema column to be used as the sorting criterion. In this
example, select ZipCode, City and Address, sequentially.
In the Sort num or alpha? column, select
alpha for all the three rows.
In the Order asc or desc column, select
asc for all the three rows.
If the schema does not appear, click the Sync
columns button to retrieve the schema from the preceding
component.
Click Advanced settings to open its
view.
Select Sort on disk. Then the Temp data directory path field and the Create temp data directory if not exist check box
appear.
In Temp data directory path, enter the
path to, or browse to the folder you want to use to store the temporary data
processed by tSortRow. In this approach,
tSortRow is enabled to sort
considerably more data.

As the threads will overwrite each other if they are written in the same
directory, you need to create the folder for each thread to be processed
using its thread ID. To do this, you can drop directly the global variable
THREAD_ID of tCollector from the Outline
view into this field; then the corresponding code is generated
automatically, reading:

((Integer)globalMap.get("tCollector_1_THREAD_ID"))

1

((Integer)globalMap.get("tCollector_1_THREAD_ID"))

This
makes the path read like:

“E:/Studio/workspace/temp”+((Integer)globalMap.get(“tCollector_1_THREAD_ID”)).

If the Outline view does not appear in the Studio, you
can display it by selecting it from the [Show
view] dialog box. For further information, see Talend Studio User Guide.
Ensure that the Create temp data directory if not
exists check box is selected.

Verifying tDepartitioner

Double-click the tDepartitioner component
open its Component view.
If required, change the values in the Buffer
Size field to adapt the memory capacity. In this example, we
leave the default value.
Click the button next to Edit
schema to open the schema editor, then, check that all of the
columns you need to output appear in the schema table of tDepartitioner. In this scenario, we output all
the columns received from its preceding components.

Outputting the sorted data

Double-click the tRecollector component
to open its Component view.
Click the button next to Edit
schema to open the schema editor, then, paste the tPartitioner schema we copied earlier when we
were configuring tPartitioner.

This schema should be consistent with that of the tDepartitioner component offering data to the current
tRecollector.
Click OK to validate these changes and
accept the propagation prompted by the pop-up dialog box.
In the Linked Departitioner field, select
the tDepartitioner component you want this
component to receive data from. In this example, it is the tDepartitioner component labelled tDepartitioner_1. Therefore, select tDepartitioner_1.
Double click the tFileOutputDelimited
component to open its Component
view.
In the File Name field, browse to the
file, or enter the directory and the name of the file, that you want to
write the sorted data in. At runtime, this file will be created if it does
not exist.

Executing the Job

Then you can press F6 to run this Job.

Once done, you can check the file holding the sorted data and the temporary
folders created by tSortRow for sorting data on
disk. These folders were emptied once the sorting had been done.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 5.x

0 Comments

Inline Feedbacks

View all comments

tPartitioner – Docs for ESB 5.x