tS3List

Lists the files on Amazon S3 based on the bucket/file prefix settings.

tS3List Standard properties

These properties are used to configure tS3List running in the Standard Job framework.

The Standard
tS3List component belongs to the Cloud family.

The component in this framework is available in all Talend
products.

Basic settings

Use existing connection	Select this check box and in the Component List click the relevant connection component to reuse the connection details you already defined.
Access Key	The Access Key ID that uniquely identifies an AWS Account. For how to get your Access Key and Access Secret, visit Getting Your AWS Access Keys.
Secret Key	The Secret Access Key, constituting the security credentials in combination with the access Key. To enter the secret key, click the […] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.
Inherit credentials from AWS role	Select this check box to obtain AWS security credentials from Amazon EC2 instance metadata. To use this option, the Amazon EC2 instance must be started and your Job must be running on Amazon EC2. For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.
Assume role	If you temporarily need some access permissions associated to an AWS IAM role that is not granted to your user account, select this check box to assume that role. Then specify the values for the following parameters to create a new assumed role session.
Region	Specify the AWS region by selecting a region name from the list or entering a region between double quotation marks (e.g. “us-east-1”) in the list. For more information about the AWS Region, see Regions and Endpoints.
List all bucket objects	Select this check box to list all the files on the S3 server. Key prefix: enter the prefix of files to be listed. This way, only files with that prefix will be listed.
Bucket	Click the [+] button to add one or more lines for defining the buckets and file prefixes. Bucket name: name of the bucket, namely the top level folder, whose files will be listed. Key prefix: prefix of files to be listed. Not available when List all bucket objects is selected.
Die on error	This check box is cleared by default, meaning to skip the row on error and to complete the process for error-free rows.

Advanced settings

Config client	Select this check box to configure client parameters. Client parameter: select client parameters from the list. Value: enter the parameter value. Not available when Use existing connection is selected.
STS Endpoint	Select this check box and in the field displayed, specify the AWS Security Token Service endpoint, for example, `sts.amazonaws.com`, where session credentials are retrieved from. This check box is available only when the Assume role check box is selected.
tStatCatcher Statistics	Select this check box to collect log data at the component level.

Config client

Select this check box to configure client parameters.

Client parameter: select client
parameters from the list.

Value: enter the parameter
value.

Not available when Use existing
connection is selected.

STS Endpoint

Select this check box and in the field displayed, specify the
AWS Security Token Service endpoint, for example, sts.amazonaws.com, where session credentials are retrieved from.

This check box is available only when the Assume role check box is selected.

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Global Variables

CURRENT_BUCKET_NAME	The current bucket name. This is a Flow variable and it returns a string.
CURRENT_KEY	The current key. This is a Flow variable and it returns a string.
NB_BUCKET	The number of buckets. This is an After variable and it returns an integer.
NB_BUCKET_OBJECT	The number of objects in all buckets. This is an After variable and it returns an integer.
ERROR_MESSAGE	The error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is selected.

Usage

Usage rule	This component can be used alone or with other S3 components, e.g. tS3Delete.
Dynamic settings	Click the [+] button to add a row in the table and fill the Code field with a context variable to choose your database connection dynamically from multiple connections planned in your Job. This feature is useful when you need to access database tables having the same data structure but in different databases, especially when you are working in an environment where you cannot change your Job settings, for example, when your Job has to be deployed and executed independent of Talend Studio. Once a dynamic parameter is defined, the Component List box in the Basic settings view becomes unusable. For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic settings and context variables, see Talend Studio User Guide.

Usage rule

This component can be used alone or with other S3 components,
e.g. tS3Delete.

Dynamic settings

Click the [+] button to add a row in the table
and fill the Code field with a context
variable to choose your database connection dynamically from multiple
connections planned in your Job. This feature is useful when you need to
access database tables having the same data structure but in different
databases, especially when you are working in an environment where you
cannot change your Job settings, for example, when your Job has to be
deployed and executed independent of Talend Studio.

Once a dynamic parameter is defined, the Component List box in the Basic
settings view becomes unusable.

For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic
settings and context variables, see Talend Studio
User Guide.

Listing files with the same prefix from a bucket

In this scenario, tS3List is used to
list all the files in a bucket which have the same prefix.

There are such files in this bucket:

For how to create a bucket and put files into it, see Verifing the absence of a bucket, creating it and listing all the S3 buckets and Exchange files with Amazon S3.

Linking the components

Drop tS3Connection, tS3List, tIterateToFlow,
tLogRow and tS3Close onto the workspace.
Link tS3Connection to tS3List using the OnSubjobOk trigger.
Link tS3List to tIterateToFlow using the Row >
Iterate connection.
Link tIterateToFlow to tLogRow using the Row >
Main connection.
Link tS3List to tS3Close using the OnSubjobOk trigger.

Configuring the components

Double-click tS3Connection to open its
Basic settings view.
In the Access Key and Secret Key fields, enter the authentication
credentials.
Double-click tS3List to open its
Basic settings view.
Select the Use existing connection check
box to reuse the connection.
In the Bucket area, click the [+] button to add one line.
In the Bucket name and Key prefix fields, enter the bucket name and file
prefix.

This way, only files with the specified prefix will be listed.
Double-click tIterateToFlow to open its
Basic settings view.
Click Edit schema to open the schema
editor.

Click the [+] button to add one column,
namely file_list of the String
type.

Click Ok to validate the setup and close
the schema editor.
In the Mapping area, press Ctrl + Space in the Value field to choose the variable tS3List_1_CURRENT_KEY.
Double-click tLogRow to open its
Basic settings view.

Select Table (print values in cells of a
table) for a better display of the results.
Double-click tS3Close to open its
Basic settings view.

There is no need to select a connection component as the only one is
selected by default.