July 30, 2023

tSQSOutput – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tSQSOutput – Docs for ESB 7.x

tSQSOutput

Delivers one or more messages to an Amazon SQS (Simple Queue Service)
queue.

Note: This component supports AWS server-side encryption (SSE)
for SQS queues.

tSQSOutput Standard properties

These properties are used to configure tSQSOutput running in the Standard Job
framework.

The Standard tSQSOutput component belongs to the Cloud family.

The component in this framework is available in all Talend
products.

Basic settings

Use an existing connection

Select this check box and in the Component List click the relevant connection component to
reuse the connection details you already defined.

Access
Key and Secret
Key

Specify the access keys (the access key ID in the Access
Key field and the secret access key in the Secret
Key field) required to access the Amazon Web Services. For more
information on AWS access keys, see Access keys (access key ID and secret access key).

To enter the secret key, click the […] button next to
the secret key field, and then in the pop-up dialog box enter the password between double
quotes and click OK to save the settings.

Inherit credentials from AWS role

Select this check box to leverage the instance profile
credentials. These credentials can be used on Amazon EC2 instances, and are delivered
through the Amazon EC2 metadata service. To use this option, your Job must be running
within Amazon EC2 or other services that can leverage IAM Roles for access to resources.
For more information, see Using an IAM Role to Grant Permissions to
Applications Running on Amazon EC2 Instances.

Assume Role

If you temporarily need some access permissions associated
to an AWS IAM role that is not granted to your user account, select this check box to
assume that role. Then specify the values for the following parameters to create a new
assumed role session.

Ensure that access to this role has been
granted to your user account by the trust policy associated to this role. If you are not
certain about this, ask the owner of this role or your AWS administrator.

  • Role ARN: the Amazon Resource Name (ARN) of the role to assume. You
    can find this ARN name on the Summary page
    of the role to be used on your AWS portal, for example, this role ARN could read
    like am:aws:iam::[aws_account_number]:role/[role_name].

  • Role session name: enter the name you want to use to uniquely
    identify your assumed role session. This name can contain upper- and lower-case
    alphanumeric characters with no spaces. You can also include underscores or any of
    the following characters: =,.@-.

  • Session duration (minutes): the duration (in minutes) for which you
    want the assumed role session to be active. This duration cannot exceed the
    maximum duration which your AWS administrator has set.

For an example about an IAM role and its related policy types, see Create and Manage AWS IAM Roles from the AWS
documentation.

Region

Specify the AWS region by selecting a region name from the
list or entering a region between double quotation marks (for example “us-east-1”) in the list. For more information on AWS Regions,
see AWS Regions and Endpoints.

Queue (Name or URL)

Specify the name or the URL of the queue to which messages will be
delivered.

The name of a FIFO queue must end with the .fifo
suffix.

Schema and Edit schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

The schema of this component is predefined with the following columns.
You can click the […] button next to
Edit schema to view its schema and edit the
schema by adding some customized attributes for the messages to be
delivered.

  • Body: the body of the message to be
    delivered.

  • MessageGroupId: the specific message group
    that the message belongs to. Available only when the
    Use FIFO queues (first-in-first-out)
    check box is selected.

  • MesssageDeduplicationId: the token used
    for deduplication of the message. Available only when the
    Use FIFO queues (first-in-first-out)
    check box is selected and the Use Content Based
    Deduplication     check box is cleared.

For more information on each predefined column, see SendMessage.

Use batch mode

Select this check box to deliver messages to the queue using batch
request. In the Batch size field displayed, enter
the number of messages to be delivered in each batch. Note that the
maximum number of messages delivered in each batch is
10.

Use FIFO (first-in-first-out) queues

Select this check box to deliver messages to a FIFO queue.

For more information on the FIFO queue, see FIFO (First-In-First-Out)
Queues.

Use Content Based Deduplication

Select this check box to enable content-based deduplication. With this
check box selected, Amazon SQS will use a SHA-256 hash to generate the
message deduplication ID using the body of the message.

This property is available only when the Use FIFO
(first-in-first-out) queues check box is selected.

Die on error

Select the check box to stop the execution of the Job when an error occurs.

Clear the check box to skip any error and continue the Job execution process.

Advanced settings

Config
client
Select this check box and in the table displayed specify the
client configuration parameters. Click the [+] button below the table to add as many rows as needed, each row for a
client configuration parameter, and set the value of the following attributes for each
parameter:

  • Client Parameter: click the cell and select a parameter from the
    drop-down list.

  • Value: enter the value for the corresponding parameter.

STS
Endpoint

Select this check box and in the field displayed, specify the
AWS Security Token Service endpoint, for example, sts.amazonaws.com, where session credentials are retrieved from.

This service allows you to request temporary,
limited-privilege credentials for the AWS user you authenticate; therefore, you still
need to provide the access key and secret key to authenticate the AWS account to be
used.

For a list of the STS endpoints you can use, see
AWS Security Token Service. For further information about the
STS temporary credentials, see Temporary Security Credentials. Both articles are from the AWS
documentation.

This check box is available only when the Assume role check box is selected.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at the Job level
as well as at each component level.

Global variables

NB_LINE

The number of rows processed. This is an After variable and it returns an integer.

NB_LINE_SUCCESS

The number of rows successfully processed. This is an After variable and it returns an
integer.

NB_LINE_REJECTED

The number of rows rejected. This is an After variable and it returns an integer.

ERROR_MESSAGE

The error message generated by the component when an error occurs. This is an After
variable and it returns a string. This variable functions only if the Die on
error check box is selected.

Usage

Usage rule

This component is usually used as an end component of a Job or
subJob and it always needs an input link.

Delivering messages to an Amazon SQS queue

Here’s an example of using Talend components to create an Amazon SQS
(Simple Queue Service) queue, then deliver a couple of messages to the queue, finally get and
display the attributes of the queue on the console.

The input messages in this example are as follows, and each message
contains the message creator attribute and the message body information.

Creating a Job for delivering Amazon SQS messages

Create a Job to open a connection to access Amazon SQS, then create an
Amazon SQS queue and deliver a couple of messages to the queue, finally get and display the
attributes of the queue on the console.

tSQSOutput_1.png

  1. Create a new Job and add a tSQSConnection component, a tSQSQueueCreate component, a tFixedFlowInput component, a tSQSOutput component, a tSQSQueueAttributes
    component, and a tLogRow component by typing their names in
    the design workspace or dropping them from the Palette.
  2. Link the tFixedFlowInput component
    to the tSQSOutput component using a Row > Main
    connection.
  3. Do the same to link the tSQSQueueAttributes component to the
    tLogRow component.
  4. Link the tSQSConnection component
    to the tSQSQueueCreate component using a
    Trigger > OnSubjobOk connection.
  5. Do the same to link the tSQSQueueCreate component to the tFixedFlowInput component and the
    tFixedFlowInput component to the
    tSQSQueueAttributes component.

Opening a connection to access Amazon SQS

Configure the tSQSConnection
component to open a connection to access Amazon SQS.

  1. Double-click the tSQSConnection
    component to open its Basic settings view.

    tSQSOutput_2.png

  2. In the Access Key and Secret Key fields, specify the authentication
    information required to access Amazon SQS.
  3. Select an AWS region available for Amazon SQS from the Region drop-down list. In this example, it is
    Asia Pacific (Tokyo).

Creating an Amazon SQS queue

Configure the tSQSQueueCreate
component to create an Amazon SQS queue.

  1. Double-click the tSQSQueueCreate
    component to open its Basic settings view.

    tSQSOutput_3.png

  2. Specify the connection details required to access Amazon SQS. In this
    example, select the Use an existing connection
    check box and from the Component List drop-down
    list displayed, select the connection component to reuse its connection details you
    have already defined.
  3. In the Queue (Name or URL)
    field, specify the name for the queue to be created. In this example, it is
    talend.
  4. If needed, specify the attributes for the queue to be created in the
    Queue Attributes table. In this example, the
    MessageRetentionPeriod attribute is added and
    its value is set to 86400, which means Amazon SQS will retain
    the messages delivered to the queue for 86400 seconds (1 day) instead of the default
    345600 seconds (4 day).

Delivering messages to the Amazon SQS queue

Configure the tFixedFlowInput
component and the tSQSOutput component to deliver
messages to the new empty Amazon SQS queue.

  1. Double-click the tSQSOutput
    component to open its Basic settings
    view.

    tSQSOutput_4.png

  2. Specify the connection details required to access Amazon SQS. In
    this example, select the Use an existing
    connection     check box and from the Component List drop-down list displayed, select the connection
    component to reuse its connection details you have already defined.
  3. In the Queue (Name or URL)
    field, enter the name for the queue to which the messages will be delivered. In
    this example, it is talend.
  4. Click the tSQSOutput_5.png button next to Edit schema to open the schema dialog box.

    tSQSOutput_6.png

    You can see the schema of the tSQSOutput component is predefined with a single String column
    Body that stores the body of each
    message to be delivered.
  5. Click the tSQSOutput_7.png button in the right panel to add one
    column Creator of String type, which will hold the
    customized creator attribute value of each message.
  6. Click the tSQSOutput_8.png button to copy all columns from the output schema to the
    input schema. In the pop-up dialog box, click OK to accept the changes.
  7. Click OK to close the schema
    dialog box.
  8. Double-click the tFixedFlowInput component to open its Basic settings view.

    tSQSOutput_9.png

  9. In the Mode area, select
    Use Inline Content(delimited file).
    Then in the Content field displayed, enter
    the creator attribute and body data of the messages to be delivered to Amazon
    SQS. In this example, it is:

Getting attributes of the Amazon SQS queue

Configure the tSQSQueueAttributes component and the
tLogRow component to get and display the attributes of the
Amazon SQS queue on the console.

  1. Double-click the tSQSQueueAttributes component to open
    its Basic settings view.
  2. Specify the connection details required to access Amazon SQS. In this example,
    select the Use an existing connection check box and from
    the Component List drop-down list displayed, select the
    connection component to reuse its connection details you have already
    defined.
  3. In the Queue (Name or URL) field, enter the name for the
    queue whose attributes will be retrieved. In this example, it is
    talend.
  4. Double-click the tLogRow component to open its
    Basic settings view.
  5. In the Mode area, select Vertical
    to display each attribute and its value in a row.

Executing the Job to deliver Amazon SQS messages

After setting up the Job and configuring the components used in the
Job for delivering Amazon SQS messages, you can then execute the Job and verify the Job
execution result.

  1. Press Ctrl + S to save the Job and
    then F6 to execute the Job.

    tSQSOutput_10.png

    As shown above, the Job is executed successfully and the attributes
    of the queue is displayed on the console.
  2. View the queue details on the AWS console to validate the Job execution
    result.

    tSQSOutput_11.png

    You can see the queue talend is created
    successfully with the retention period for one day, and three messages are now
    available in the queue.
  3. Select the queue talend on the AWS console
    and then click Queue Actions >
    View/Delete Messages to view messages
    in the queue.

    tSQSOutput_12.png

    You can see the three messages are now available in the
    queue.
  4. Click More Details for one of the
    messages to view the detailed information of the message.

    tSQSOutput_13.png

    You can see the customized Creator attribute
    value for the message is listed on the Message
    Attributes     tab.
Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x