August 17, 2023

tS3List – Docs for ESB 5.x

tS3List

tS3List_icon32_white.png

tS3List properties

Component family

Cloud/AmazonS3

 

Function

Lists the files on Amazon S3 based on the bucket/file prefix
settings.

Purpose

tS3List is designed to list the
files on Amazon S3 based on the bucket/file prefix settings.

Basic settings

Use existing connection

Select this check box and in the Component List click the
relevant connection component to reuse the connection details you already defined.

 

Access Key

The Access Key ID that uniquely identifies an AWS Account. For how
to get your Access Key and Access Secret, visit Getting Your AWS Access Keys.

 

Access Secret

The Secret Access Key, constituting the security credentials in
combination with the access Key.

To enter the secret key, click the […] button next to
the secret key field, and then in the pop-up dialog box enter the password between double
quotes and click OK to save the settings.

 

Region

Specify the AWS region by selecting a region name from the list or entering a region
between double quotation marks (e.g. “us-east-1”) in the
list. For more information about the AWS Region, see Regions
and Endpoints
.

 

List all bucket objects

Select this check box to list all the files on the S3
server.

Key prefix: enter the prefix of
files to be listed. This way, only files with that prefix will be
listed.

 

Bucket

Click the [+] button to add one
or more lines for defining the buckets and file prefixes.

Bucket name: name of the bucket
whose files will be listed.

Key prefix: prefix of files to be
listed.

Not available when List all bucket
objects
is selected.

 

Die on error

This check box is cleared by default, meaning to skip the row on
error and to complete the process for error-free rows.

Advanced settings

Config client

Select this check box to configure client parameters.

Client parameter: select client
parameters from the list.

Value: enter the parameter
value.

Not available when Use existing
connection
is selected.

 

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Dynamic settings

Click the [+] button to add a row in the table and fill
the Code field with a context variable to choose your
database connection dynamically from multiple connections planned in your Job. This feature
is useful when you need to access database tables having the same data structure but in
different databases, especially when you are working in an environment where you cannot
change your Job settings, for example, when your Job has to be deployed and executed
independent of Talend Studio.

Once a dynamic parameter is defined, the Component List
box in the Basic settings view becomes unusable.

For more information on Dynamic settings and context
variables, see Talend Studio User Guide.

Global Variables

CURRENT_BUCKET: the current bucket name. This is a Flow
variable and it returns a string.

CURRENT_KEY: the current key. This is a Flow variable and
it returns a string.

NB_BUCKET: the number of buckets. This is an After
variable and it returns an integer.

NB_BUCKET_OBJECT: the number of objects in all the
buckets. This is an After variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space
to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component can be used alone or with other S3 components, e.g.
tS3Delete.

Log4j

The activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User
Guide
.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

Due to license incompatibility, one or more JARs required to use this component are not
provided. You can install the missing JARs for this particular component by clicking the
Install button on the Component tab view. You can also find out and add all missing JARs easily on
the Modules tab in the Integration perspective
of your studio. For details, see https://help.talend.com/display/KB/How+to+install+external+modules+in+the+Talend+products
or the section describing how to configure the Studio in the Talend Installation and Upgrade
Guide
.

Scenario: Listing files with the same prefix from a bucket

In this scenario, tS3List is used to list all the
files in a bucket which have the same prefix.

There are such files in this bucket:

use_case_ts3list_1.png

For how to create a bucket and put files into it, see Scenario: Verifing the absence of a bucket, creating it and listing all the S3
buckets
and Scenario: File exchanges with Amazon S3 .

Linking the components

  1. Drop tS3Connection, tS3List, tIterateToFlow,
    tLogRow and tS3Close onto the workspace.

  2. Link tS3Connection to tS3List using the OnSubjobOk trigger.

  3. Link tS3List to tIterateToFlow using the Row >
    Iterate
    connection.

  4. Link tIterateToFlow to tLogRow using the Row >
    Main
    connection.

  5. Link tS3List to tS3Close using the OnSubjobOk trigger.

    use_case_ts3list_2.png

Configuring the components

  1. Double-click tS3Connection to open its
    Basic settings view.

    use_case_ts3list_3.png
  2. In the Access Key and Secret Key fields, enter the authentication
    credentials.

  3. Double-click tS3List to open its
    Basic settings view.

    use_case_ts3list_4.png
  4. Select the Use existing connection check
    box to reuse the connection.

  5. In the Bucket area, click the [+] button to add one line.

  6. In the Bucket name and Key prefix fields, enter the bucket name and file
    prefix.

    This way, only files with the specified prefix will be listed.

  7. Double-click tIterateToFlow to open its
    Basic settings view.

    use_case_ts3list_5.png
  8. Click Edit schema to open the schema
    editor.

    use_case_ts3list_6.png

    Click the [+] button to add one column,
    namely file_list of the String
    type.

    Click Ok to validate the setup and close
    the schema editor.

  9. In the Mapping area, press Ctrl + Space in the Value field to choose the variable tS3List_1_CURRENT_KEY.

  10. Double-click tLogRow to open its
    Basic settings view.

    use_case_ts3list_7.png

    Select Table (print values in cells of a
    table)
    for a better display of the results.

  11. Double-click tS3Close to open its
    Basic settings view.

    use_case_ts3list_8.png

    There is no need to select a connection component as the only one is
    selected by default.

Executing the Job

  1. Press Ctrl + S to save the Job.

  2. Press F6 to run the Job.

    use_case_ts3list_9.png

    As shown above, only the files with the prefix “in” are listed.


Document get from Talend https://help.talend.com
Thank you for watching.
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x