July 30, 2023

tPartitioner – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tPartitioner – Docs for ESB 7.x

tPartitioner

Partitions the input data before tCollector
can transfer them to the parallel execution processes.

The tPartitioner splits the input records into a given
number of threads which tCollector sends for parallel
execution.

Note that Talend Studio also enables the automatic implementation of parallelization
across a Job without use of the parallelization components and we recommend using that
approach. For further information, see the section describing how to enable
parallelization of data flows of the Talend Studio User Guide. However, if you
need to understand how to use these specific parallelization components, bear in mind
that the parallelization components work closely with each other to accomplish parallel
execution on given processes: the tPartitioner
component dispatches the input records into a specific number of threads; the tCollector component sends these threads to its following
components for parallel execution; the tDepartitioner
component regroups the outputs of the processed parallel threads; the tRecollector component captures the output of a given
tDepartitioner component and sends the captured
data to the next component.

tPartitioner Standard properties

These properties are used to configure tPartitioner running in the Standard Job framework.

The Standard
tPartitioner component belongs to the Orchestration family.

This component is available in Talend products with Big Data, Talend Data Management Platform, Talend Data Services Platform, Talend MDM Platform and Talend Data Fabric.

Basic settings

Schema and Edit
Schema

A schema is a row description, it defines the number of fields to be processed and
passed on to the next component. The schema is either Built-in or stored remotely in the
Repository.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

Click Sync
columns to retrieve the schema from the previous component connected in the
Job.

 Number of Child Threads

Enter the number of threads you want to split the input records up
into.

We recommend that this number be N-1 where N is the total number
of CPUs or cores on the machine processing the data.

Buffer Size

Enter the number of rows to be processed before the memory is freed.

This is the number of rows to cache for each of the threads
generated.

Use a key hash for partitions

Select this check box to use the hash mode to dispatch the input
records into threads.

Once selecting it, the Key
Columns table appears, in which you set the column(s)
you want to apply the hash mode on. In the hash mode, the records
meeting the same criteria are dispatched into the same threads.

If you leave this check box clear, the dispatch mode is
Round-robin, meaning records are dispatched one-by-one to each
thread, in a circular fashion, until the last record is dispatched.
Be aware that this mode cannot guarantee that records meeting the
same criteria go into the same threads.

Advanced settings

tStatCatcher Statistics

Select this check box to collect the log data at the component
level.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

NB_LINE: the number of rows processed. This is an After
variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

This component should be put after the input component(s) and
before tCollector.

This component uses and can only use the Trigger > Start link to connect to tCollector.

Connections

Outgoing links (from this component to another):

Trigger: Start.

Incoming links (from one component to this one):

Row: Main.

For further information regarding connections, see

Talend Studio User
Guide.

Sorting the customer data of large size in parallel

This scenario applies only to subscription-based Talend products and Talend products with Big Data.

The Job in this scenario puts in order 20 million customer records by running
parallelized executions.

tPartitioner_1.png

Linking the components

  1. In the
    Integration
     perspective
    of your studio, create an empty Job from the Job
    Designs     node in the Repository tree view.

    For further information about how to create a Job, see
    Talend Studio User Guide    .
  2. Drop the following components onto the workspace: tFileInputDelimited, tPartitioner, tCollector,
    tSortRow, tDepartitioner, tRecollector, tFileOutputDelimited.

    The tFileInputDelimited component
    (labeled test file in this example) reads
    the 20 million customer records from a .txt file generated by tRowGenerator.
    For further information about the tRowGenerator component, see tRowGenerator
    For further information about how to label a component, see
    Talend Studio User Guide    .
  3. Connect tPartitioner to tCollector using the Trigger > Starts link.
  4. Do the same to connect tDepartitioner to
    tRecollector.
  5. Connect the other components using the Row >
    Main     link.

Splitting the input data flow

Configuring the input flow

  1. Double-click tFileInputDelimited to open
    its Component view.

    tPartitioner_2.png

  2. In the File name/Stream field, browse to,
    or enter the path to the file storing the customer records to be
    read.
  3. Click the

    tPartitioner_3.png

    button to open the schema editor where you need to
    create the schema to reflect the structure of the customer data.

    tPartitioner_4.png

  4. Click the

    tPartitioner_5.png

    button five times to add five rows and rename them as
    follows: FirstName, LastName, City, Address and
    ZipCode.

    In this scenario, we leave the data types with their default value
    String. In the real-world practice, you
    can change them depending on the data types of your data to be
    processed.
  5. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.
  6. If needs be, complete the other fields of the Component view with values corresponding to your data to be
    processed. In this scenario, we leave them as is.

Configuring the tPartitioner component

  1. Double-click tPartitioner to open its
    Component view.

    tPartitioner_6.png

  2. In the Number of Child Threads field,
    enter the number of the threads you want to partition the data flow into. In
    this example, enter 3 because we are
    using 4 processors to run this Job.
  3. If required, change the value in the Buffer
    Size     field to adapt the memory capacity. In this example, we
    leave the default one.
  4. Click the

    tPartitioner_3.png

    button next to Edit
    schema     to open the schema editor.

    tPartitioner_8.png

  5. Select all the rows in the tPartitioner
    component schema table using Ctrl or
    Shift.

    Then the

    tPartitioner_9.png

    on the toolbar is activated.

  6. Click

    tPartitioner_9.png

    to copy the schema selected.

Sorting the input records

Configuring tCollector

  1. Double-click tCollector to open its
    Component view.

    tPartitioner_11.png

  2. Click the

    tPartitioner_3.png

    button next to Edit
    schema     to open the schema editor.

  3. Press Ctrl+V to paste the tPartitioner component schema.
  4. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.

Configuring tSortRow

  1. Double-click tSortRow to open its
    Component view.

    tPartitioner_13.png

  2. Under the Criteria table, click the

    tPartitioner_5.png

    button three times to add three rows to the
    table.

  3. In the Schema column column, select, for
    each row, the schema column to be used as the sorting criterion. In this
    example, select ZipCode, City and Address, sequentially.
  4. In the Sort num or alpha? column, select
    alpha for all the three rows.
  5. In the Order asc or desc column, select
    asc for all the three rows.
  6. If the schema does not appear, click the Sync
    columns     button to retrieve the schema from the preceding
    component.
  7. Click Advanced settings to open its
    view.

    tPartitioner_15.png

  8. Select Sort on disk. Then the Temp data directory path field and the Create temp data directory if not exist check box
    appear.
  9. In Temp data directory path, enter the
    path to, or browse to the folder you want to use to store the temporary data
    processed by tSortRow. In this approach,
    tSortRow is enabled to sort
    considerably more data.

    As the threads will overwrite each other if they are written in the same
    directory, you need to create the folder for each thread to be processed
    using its thread ID. To do this, you can drop directly the global variable
    THREAD_ID of tCollector from the Outline
    view into this field; then the corresponding code is generated
    automatically, reading:
    This
    makes the path read like:
    “E:/Studio/workspace/temp”+((Integer)globalMap.get(“tCollector_1_THREAD_ID”)).
    If the Outline view does not appear in the Studio, you
    can display it by selecting it from the Show
    view     dialog box. For further information, see
    Talend Studio User Guide    .
  10. Ensure that the Create temp data directory if not
    exists     check box is selected.

Verifying tDepartitioner

  1. Double-click the tDepartitioner component
    open its Component view.

    tPartitioner_16.png

  2. If required, change the values in the Buffer
    Size     field to adapt the memory capacity. In this example, we
    leave the default value.
  3. Click the

    tPartitioner_3.png

    button next to Edit
    schema     to open the schema editor, then, check that all of the
    columns you need to output appear in the schema table of tDepartitioner. In this scenario, we output all
    the columns received from its preceding components.

Outputting the sorted data

  1. Double-click the tRecollector component
    to open its Component view.

    tPartitioner_18.png

  2. Click the

    tPartitioner_3.png

    button next to Edit
    schema     to open the schema editor, then, paste the tPartitioner schema we copied earlier when we
    were configuring tPartitioner.

    This schema should be consistent with that of the tDepartitioner component offering data to the current
    tRecollector.
  3. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.
  4. In the Linked Departitioner field, select
    the tDepartitioner component you want this
    component to receive data from. In this example, it is the tDepartitioner component labelled tDepartitioner_1. Therefore, select tDepartitioner_1.
  5. Double click the tFileOutputDelimited
    component to open its Component
    view.

    tPartitioner_20.png

  6. In the File Name field, browse to the
    file, or enter the directory and the name of the file, that you want to
    write the sorted data in. At runtime, this file will be created if it does
    not exist.

Executing the Job

Then you can press F6 to run this Job.

Once done, you can check the file holding the sorted data and the temporary
folders created by tSortRow for sorting data on
disk. These folders were emptied once the sorting had been done.

tPartitioner_21.png

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x