August 15, 2023

cIdempotentConsumer – Docs for ESB 6.x


Identifies messages that have already been sent to the receiver and eliminates
them. Messages are still sent by the sender but are ignored by the receiver at the
delivery stage.

cIdempotentConsumer deduplicates
messages and thereby prevents the receiving message endpoint from
receiving duplicate messages.

cIdempotentConsumer Standard properties

These properties are used to configure cIdempotentConsumer running in the Standard Job framework.

The Standard
cIdempotentConsumer component belongs to the Routing family.

Basic settings

Repository Type

Message identifiers need to be stored in a repository. For new
incoming messages, identifiers are checked against the ones stored
in the repository to identify and drop duplicates. There are two
ways to store them:

Memory: messages identifiers are
stored temporarily.


The in-memory storage mode can easily run out of
memory and does not work in a clustered

File: messages identifiers are
stored in a file. Specify the path to this file in the File store field.

File store

Specify the path and name of the file storing messages

Cache Size

Type in the size of the cache, namely the number of message
identifiers to store.

Use language

Select this check box if you want to specify the language used in
the Predicate field to specify the
identifier of the messages. In the Language list, select from None, Bean,
Constant, CorrelationID, EL,
Groovy, Header, JavaScript,
JoSQL, JSonPath, JXPath,
Property, Python, Ruby,
Simple, SpEL, SQL,
XPath, and XQuery.


Type in the expression to use to specify the identifier of the

Add Namespaces

This option appears when XPath is
selected in the Language list.

Select this check box to add namespaces for the Xpath expression.
Click [+] to add as many namespaces
as required to the table and define the prefix and URI in the
corresponding columns.


Select this check box to detect duplicate messages even when
messages are currently in progress; clear it to detect duplicates
only when messages have successfully been processed.

By default, this check box is selected.


Select this check box to drop duplicates; clear it to ignore
duplicates so that all messages will be continued.

By default, this check box is selected.


Usage rule

cIdempotentConsumer is used as a
middle component in a Route.

Connections idemp: The idemp link
retrieves messages deduplicated by the cIdempotentConsumer
  Route: As an optional link, the
Route link retrieves all messages from the message



Scenario: Deduplicating messages while routing them

This scenario applies only to a Talend solution with ESB.

In this scenario, duplicated messages are filtered and only the unique one is routed to the

Three XML files that have the same content, as shown below, are used in this use

Dropping and linking the components

This use case requires one cFile component, one
cIdempotentComsumer component, and two
cProcessor components.


  1. From the Palette, expand the Connectivity folder, select the cFile component, and drop it onto the design workspace as
    the message source component.
  2. Expand the Routing folder, select the
    cIdempotentComsumer component and drop
    it onto the design workspace as the message deduplicator.
  3. Expand the Custom folder, drop two cProcessor components onto the design workspace, one as the
    consumer for deduplicated messages and another for all messages.
  4. Right-click the cFile component, select
    Row > Route from the contextual menu and click the cIdempotentComsumer component.
  5. Right-click the cIdempotentComsumer
    component, select Row > idemp from the contextual menu and click the
    cProcessor component on the top.
  6. Connect the cIdempotentComsumer component
    to the other cProcessor component using a
    Row > Route connection. This optional connection will retrieve all
    the messages coming from the source.
  7. Label the components to better identify their roles in the Route.

Configuring the components and connections

  1. Double-click the cFile component, which
    is labelled Source, to display its
    Basic settings view in the Component tab.


  2. In the Path field, specify the file path to the message source.

    From the Encoding list, select the encoding type of your source files, and leave all the other parameters as they are.
  3. Double-click the cIdempotentComsumer
    component, which is labelled Deduplicator, to display its Basic
    view in the Component tab.


  4. From the Repository Type list, select between Memory and File to specify where the message identifiers will be stored before the deduplication process. For this scenario, select File.

    In the File store field, specify the location of the file storing message identifiers.
    In the Expression field, enter an expression to filter the messages. In this scenario, enter the following expression to filter the messages according to the person node of the XML files: xpath("/people/person"), and leave all the other parameters as they are. Alternatively, you can select the Use language check box, select XPath from the Language list, and enter "/people/person" in the Predicate field.
  5. Double-click the cProcessor component
    labelled Unique to display its Basic settings view in the Component tab.


  6. In the Code area, customize the code to
    display the file name of the message that passes the deduplication:

  7. Repeat these steps to configure the other cProcessor component, which is labelled All, to display the file names of all the
    messages coming from the source:

  8. Press Ctrl+S to save your Route.

Viewing code and executing the Route

  1. Click the Code tab at the bottom of the
    design workspace to view the generated code.


    In this partially shown piece of code, messages from the Source
    are filtered according to the expression
    xpath("/people/person") and deduplicated by
  2. Click the Run view to display it and
    click the Run button to launch the
    execution of your Route. You can also press F6 to execute it.

    RESULT: When several files have the same content, only the first one is
    routed to the receiving endpoint.

Document get from Talend
Thank you for watching.
Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x