tS3Copy

Copies an Amazon S3 object from a source bucket to a destination
bucket.

tS3Copy Standard properties

These properties are used to configure tS3Copy running in the Standard Job
framework.

The Standard
tS3Copy component belongs to the Cloud family.

The component in this framework is available in all Talend
products.

Basic settings

Use an existing connection	Select this check box and in the Component List click the relevant connection component to reuse the connection details you already defined.
Access Key	Specify the Access Key ID that uniquely identifies an AWS Account. For how to get your Access Key and Access Secret, visit Getting Your AWS Access Keys.
Secret Key	Specify the Secret Access Key, constituting the security credentials in combination with the access Key. To enter the secret key, click the […] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.
Inherit credentials from AWS role	Select this check box to obtain AWS security credentials from Amazon EC2 instance metadata. To use this option, the Amazon EC2 instance must be started and your Job must be running on Amazon EC2. For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.
Assume role	If you temporarily need some access permissions associated to an AWS IAM role that is not granted to your user account, select this check box to assume that role. Then specify the values for the following parameters to create a new assumed role session.
Region	Specify the AWS region by selecting a region name from the list or entering a region between double quotation marks (e.g. “us-east-1”) in the list. For more information about the AWS Region, see Regions and Endpoints.
Source Bucket	Specify the name of the source bucket that contains the object to be copied.
Source Key	Specify the path to the object to be copied. Warning: This field must not start with a slash (/).
Destination Bucket	Specify the name of the destination bucket to which the object will be copied.
Destination Key	Specify the path to the destination object, using slash (/) to separate folders in the path. If: you only specify a file name in this field, the file will be in the bucket; the path contains folders that do not exist, the folders will be created; the file already exists, it will be overwritten. Warning: This field must not start with a slash (/).
Server-Side Encryption	Select this check box to enable server-side encryption with Amazon S3-Managed Encryption Keys (SSE-S3) and use the 256-bit Advanced Encryption Standard (AES-256) cipher to encrypt your data. For more information about the server-side encryption, see Protecting Data Using Server-Side Encryption.
Use KMS	Select this check box to enable server-side encryption with AWS KMS-Managed Keys (SSE-KMS) instead of Amazon S3-Managed Encryption Keys (SSE-S3). This property is available when the Server-Side Encryption check box is selected.
Customer Master Key	Specify your own customer master key (CMK) that is created in the IAM console using AWS Key Management Service for SSE-KMS encryption. If not specified, the default CMK, which is created the first time you add an SSE-KMS encrypted object to the defined bucket in your region, will be used for SSE-KMS encryption. This property is available when the Use KMS check box is selected.
Die on error	Select this check box to stop the execution of the Job when an error occurs. Clear the check box to skip any rows on error and complete the process for error-free rows.

Advanced settings

Config client	Select this check box and specify the client paramter(s) by clicking the [+] button to add as many rows as needed, each row for a client parameter, and then setting the value of the following fields for each parameter: Client Parameter: click the cell and from the drop-down list displayed select the client parameter. Value: enter the value for the selected parameter. This check box is available only when the Use an existing connection check box is cleared.
Part size (Mb) for file larger than 5GB	Specify the part size in MB. This parameter takes effect only when you copy an Amazon S3 object that is larger than 5 GB through multiple parts. Note that in the copy operation, the object’s metadata may drop during transition.
STS Endpoint	Select this check box and in the field displayed, specify the AWS Security Token Service endpoint, for example, `sts.amazonaws.com`, where session credentials are retrieved from. This check box is available only when the Assume role check box is selected.
tStatCatcher Statistics	Select this check box to gather the Job processing metadata at the Job level as well as at each component level.

Global Variables

ERROR_MESSAGE	The error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is selected.

Usage

Usage rule	This component can be used as a standalone component.

Copying an S3 object from one bucket to another

This scenario describes a Job that uploads a new object to an existing empty S3 bucket
bucket-src, then copies the object from
the bucket bucket-src to another existing empty
S3 bucket bucket-dst, finally lists the object in
the bucket bucket-dst to see whether the object
is successfully copied.

Setting up the Job

Create a new Job and add a tS3Connection component, a tS3Put component, a tS3Copy component, a
tS3List component,
a tIterateToFlow component,
and a tLogRow component by
typing their names on the design workspace or dropping them
from the Palette.
Link the tS3List component to
the tIterateToFlow
component using a Row >
Iterate
connection.
Link the tIterateToFlow
component to the tLogRow
component using a Row >
Main
connection.
Link the tS3Connection
component to the tS3Put
component using a Trigger >
On Subjob Ok
connection.
Do the same to link the tS3Put component to the tS3Copy component and the
tS3Copy component
to the tS3List
component.

Configuring the components

Creating a connection to Amazon S3

Double-click the tS3Connection component to open its
Basic settings view
on the Component
tab.
In the Access Key and
Secret Key fields,
enter the authentication credentials required to access
Amazon S3.
From the Region drop-down
list, select an AWS region where the object will be uploaded
and copied. In this example, we keep the default
setting.

Uploading an object to an Amazon S3 bucket

Double-click the tS3Put
component to open its Basic
settings view on the Component tab.
Select the Use an existing
connection check box to reuse the Amazon
S3 connection information you have defined in the tS3Connection
component.
In the Bucket field, enter
the name of the S3 bucket where the object will be uploaded.
In this example, it is bucket-src that already exists in Amazon
S3.
In the Key field, enter the
key for the object to be uploaded. In this example, it is
tS3Copy_icon32_src.png.
In the File field, browse to
or enter the path to the object to be uploaded. In this
example, it is D:/tS3Copy_icon32.png.

Copying the uploaded object to another Amazon S3 bucket

Double-click the tS3Copy
component to open its Basic
settings view on the Component tab.
Select the Use an existing
connection check box to reuse the Amazon
S3 connection information you have defined in the tS3Connection
component.
In the Bucket field in the
Source
Configuration area, enter the name of the
bucket which contains the object to be copied. In this
example, it is bucket-src.
In the Key field in the
Source
Configuration area, enter the key of the
object to be copied. In this example, it is tS3Copy_icon32_src.png.
In the Bucket field in the
Destination
Configuration area, enter the name of the
bucket to which the object will be copied. In this example,
it is the empty one bucket-dst that already exists in Amazon
S3.
In the Key field in the
Destination
Configuration area, enter the new key for
the object after being copied to the destination bucket. In
this example, it is tS3Copy_icon32_dst.png.

Listing the object in the destination bucket

Double-click the tS3List
component to open its Basic
settings view on the Component tab.
Select the Use an existing
connection check box to reuse the Amazon
S3 connection information you have defined in the tS3Connection
component.
Clear the List all buckets
objects check box, and then click the
[+] button to add
one row in the Bucket table
displayed and set the value for each column. In this
example, bucket-dst for
the Bucket name column and
empty value for the Key
prefix column, this way only the objects
in the bucket-dst bucket
will be listed.
Double-click the tIterateToFlow component to open its
Basic settings view
on the Component
tab.
Click the […] button next
to Edit schema and in the
pop-up schema dialog box define the schema by adding one
column ObjectList of
String type.
Click OK to save the changes
and in the pop-up dialog box click Yes to accept the propagation.
Double-click the tLogRow
component to open its Basic
settings view on the Component tab.
In the Mode area, select
Table (print values in cells
of a table) for better readability of the
result.

Saving and executing the Job

Press Ctrl + S to save the
Job.
Press F6 to run the
Job.

As shown above, the uploaded object has been copied to the
destination bucket successfully.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 7.x

0 Comments

Inline Feedbacks

View all comments

tS3Copy – Docs for ESB 7.x