tS3Connection

Establishes a connection to Amazon S3 to store and retrieve data.

tS3Connection Standard properties

These properties are used to configure tS3Connection running in the Standard Job
framework.

The Standard
tS3Connection component belongs to the Cloud family.

The component in this framework is available in all Talend
products.

Basic settings

Access Key	The Access Key ID that uniquely identifies an AWS Account. For how to get your Access Key and Access Secret, visit Getting Your AWS Access Keys.
Secret Key	The Secret Access Key, constituting the security credentials in combination with the access Key. To enter the secret key, click the […] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.
Inherit credentials from AWS role	Select this check box to obtain AWS security credentials from Amazon EC2 instance metadata. To use this option, the Amazon EC2 instance must be started and your Job must be running on Amazon EC2. For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.
Assume Role	If you temporarily need some access permissions associated to an AWS IAM role that is not granted to your user account, select this check box to assume that role. Then specify the values for the following parameters to create a new assumed role session. Ensure that access to this role has been granted to your user account by the trust policy associated to this role. If you are not certain about this, ask the owner of this role or your AWS administrator. Role ARN: the Amazon Resource Name (ARN) of the role to assume. You can find this ARN name on the Summary page of the role to be used on your AWS portal, for example, this role ARN could read like `am:aws:iam::[aws_account_number]:role/[role_name]`. Role session name: enter the name you want to use to uniquely identify your assumed role session. This name can contain upper- and lower-case alphanumeric characters with no spaces. You can also include underscores or any of the following characters: =,.@-. Session duration (minutes): the duration (in minutes) for which you want the assumed role session to be active. This duration cannot exceed the maximum duration which your AWS administrator has set. For an example about an IAM role and its related policy types, see Create and Manage AWS IAM Roles from the AWS documentation.
Region	Specify the AWS region by selecting a region name from the list or entering a region between double quotation marks (e.g. “us-east-1”) in the list. For more information about the AWS Region, see Regions and Endpoints.
Encrypt	Select this check box and from the Key type drop-down list displayed, select one of the following three options for encrypting the data on the client-side before sending to Amazon S3. For more information, see Protecting Data Using Client-Side Encryption. KMS-managed customer master key: use a KMS-managed customer master key (CMK) for the client-side data encryption. In the Key field, you need to specify the AWS KMS customer master key ID (CMK ID). Symmetric Master Key: use a symmetric master key (256-bit AES secret key) for the client-side data encryption. Algorithm: select the algorithm associated with the key from the list. By default, there is only one algorithm named AES. Encoding: select the encoding type associated with the key from the list, either Base64 or X509. Key or Key file: specify the key or the path to the file that stores the key. Asymmetric Master Key: use an asymmetric master key (a 1024-bit RSA key pair) for the client-side data encryption. Algorithm: select the algorithm associated with the key from the list. By default, there is only one algorithm named RSA. Public key file: specify the path to the public key file. Private key file: specify the path to the private key file.

Advanced settings

Use a custom region endpoint	Select this check box to use a custom endpoint and in the field displayed, specify the URL of the custom endpoint to be used.
Config client	Select this check box if you want to use customized client configuration other than the default. Client Parameter: select client parameters from the list. Value: enter the parameter value. For related information, go to Client Configuration.
Check S3 Accessibility	Leave this check box selected so that the component verifies the credentials to be used for this connection request to S3 before proceeding to further actionst. It is recommended to use the default By Account Owner option for this verification. The By Bucket Configuration option employs an old verification approach which could significantly increase your network load in some circumstances.
Enable Accelerate Mode	Select this check box to enable fast, easy and secure transfers of files over long distances between your client and an S3 bucket. To take it into account, you should enable this acceleration mode on the S3 bucket in advance.
STS Endpoint	Select this check box and in the field displayed, specify the AWS Security Token Service endpoint, for example, `sts.amazonaws.com`, where session credentials are retrieved from. This service allows you to request temporary, limited-privilege credentials for the AWS user you authenticate; therefore, you still need to provide the access key and secret key to authenticate the AWS account to be used. For a list of the STS endpoints you can use, see AWS Security Token Service. For further information about the STS temporary credentials, see Temporary Security Credentials. Both articles are from the AWS documentation. This check box is available only when the Assume role check box is selected.
tStatCatcher Statistics	Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE	The error message generated by the component when an error occurs. This is an After variable and it returns a string.

Usage

Usage rule	As a start component, this component is to be used along with other S3 components.
Dynamic settings	Click the [+] button to add a row in the table and fill the Code field with a context variable to choose your database connection dynamically from multiple connections planned in your Job. This feature is useful when you need to access database tables having the same data structure but in different databases, especially when you are working in an environment where you cannot change your Job settings, for example, when your Job has to be deployed and executed independent of Talend Studio. Once a dynamic parameter is defined, the Component List box in the Basic settings view becomes unusable. For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic settings and context variables, see Talend Studio User Guide.

Usage rule

As a start component, this component is to be
used along with other S3 components.

Dynamic settings

Click the [+] button to add a row in the table
and fill the Code field with a context
variable to choose your database connection dynamically from multiple
connections planned in your Job. This feature is useful when you need to
access database tables having the same data structure but in different
databases, especially when you are working in an environment where you
cannot change your Job settings, for example, when your Job has to be
deployed and executed independent of Talend Studio.

Once a dynamic parameter is defined, the Component List box in the Basic
settings view becomes unusable.

For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic
settings and context variables, see Talend Studio
User Guide.

Creating an IAM role on AWS

You need an IAM role to delegate permissions to the AWS service to be used by your Job. If this IAM role does not exist, define it on AWS.

You have the appropriate rights and permissions to create a new role on AWS.

Log in to your account on AWS and navigate to the AWS console.
Select IAM.
In the navigation pane of the IAM console, select Roles, and then select Create role.
Select AWS service and in the Choose the service that will use this role section, select the AWS service to be run with your Job. For example, select Redshift.
Select the use case to be used for this service. An use case in terms of AWS is defined by the service to include the trust policy that this service requires. Depending on the service and the use case that you selected, the available options vary. For example, with Redshift, you can choose an use case from:
- Redshift (with a pre-defined Amazon Redshift
  Service Linked Role Policy);
- Redshift – Customizable. In this use case, you are prompted to select either read-only policies or full-access policies.
In the Role name field, enter the name to be used for the role being created.
Select Create role.

A custom role has been created to delegate permissions to an AWS service. For the
full documentation about creating a role on AWS, see Role creation from the AWS
documentation.

Setting up SSE KMS for your EMR cluster

If required by the security policy of your organization, you need to set up SSE KMS, the server-side encryption service of Amazon, for the EMR cluster to be used, before creating this cluster.

This procedure explains only the
SSE KMS related operations for getting started with the security configuration for EMR.
If you need the complete information about all the available EMR security configurations
provided by AWS, see Create a Security Configuration from the
Amazon documentation.

If not yet done, go to https://console.aws.amazon.com/kms
to create a customer managed CMK to be used by the SSE KMS service. For detailed
instructions about how to do this, see this tutorial from the AWS
documentation.
- When adding roles, among other roles to be added depending on your
  security policy, you must add the EMR_EC2_DefaultRole role.
  
  The EMR_EC2_DefaultRole role allows your
  Jobs for Apache Spark to read or write files encrypted with SSE-KMS on
  S3.
  
  This role is a default AWS role that is
  automatically created along with the creation of your first EMR
  cluster. If this role and its associated policies do not exist in
  your account, see Use Default IAM Roles and
  Managed Policies from the AWS documentation
On the Amazon EMR page of
AWS, select the Security configurations
tab and click Create to open the
Create security configuration
view.
Select the At-rest encryption check box
to enable SSE KMS.
Under S3 data encryption, select
SSE-KMS for Encryption mode
and select the CMK key mentioned at the beginning of this procedure for
AWS KMS Key.
Under Local disk encryption, select AWS
KMS for Key provider type and select the
CMK key mentioned at the beginning of this procedure for AWS KMS
Key.
Click Create to validate your security configuration.

In the real-world practice, you can also configure the other security options such as Kerberos and IAM roles for EMRFS before clicking this Create button.
Click Clusters and once the Create Cluster page is open, click Go to advanced options to start creating the EMR cluster step by step.
At the last step called Security, in the Authentication and
encryption section, select the Security Configuration created in the previous steps.

Setting up SSE KMS for your S3 bucket

If required by the security policy of your organization, you need to set up SSE KMS for the S3 bucket to be used.

Prerequisite: you must have created the CMK key to be used. For detailed
instructions about how to do this, see this tutorial from the AWS
documentation.

Open your S3 service at https://s3.console.aws.amazon.com/.
From the S3 bucket list, select the bucket to be used. Ensure
that you have proper rights and permissions to access this bucket.
Select the Properties tab
and then Default encryption.
Select AWS-KMS.
Select the KMS CMK key to be used.
Select the Permissions tab, then select
Bucket Policy and enter your policy in the
console.

This article from AWS provides detailed explanations and a simple policy
example: How to Prevent Uploads of Unencrypted Objects
to Amazon S3.
Click Save to save your policy.

Now your bucket policy is set up. When you need to use this bucket with a Job, enter
the following parameter about AWS signature versions to the JVM argument list of this Job:

1 2	-Dcom.amazonaws.services.s3.enableV4

For further information about AWS Signature Versions, see Specifying the Signature Version in Request
Authentication.

Related scenario

For tS3Connection related scenarios, see Exchange files with Amazon S3.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 7.x

0 Comments

Inline Feedbacks

View all comments

tS3Connection – Docs for ESB 7.x