These properties are used to configure tS3Configuration running in the Spark
Batch Job framework.

The Spark Batch
tS3Configuration component belongs to the Storage family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

Basic settings

Advanced settings

Usage

• You have the appropriate rights and permissions to create a new role on AWS.

For further information about AWS Signature Versions, see Specifying the Signature Version in Request
Authentication.

In this scenario, you create a Spark Batch Job using
tS3Configuration and the Parquet components to write data on
S3 and then read the data from S3.

This scenario applies only to subscription-based Talend products with Big
Data.

1. In the
   Integration
   perspective of the Studio, create an empty
   Spark Batch Job from the Job Designs node in
   the Repository tree view.
2. In the workspace, enter the name of the component to be used and select this
   component from the list that appears. In this scenario, the components are
   tS3Configuration,
   tFixedFlowInput,
   tFileOutputParquet,
   tFileInputParquet and
   tLogRow.
   The tFixedFlowInput component is used to load the
   sample data into the data flow. In the real-world practice, you could use the
   File input components, as well as the processing components, to design a
   sophisticated process to prepare your data to be processed.
3. Connect tFixedFlowInput to tFileOutputParquet using the Row > Main link.
4. Connect tFileInputParquet to tLogRow using the Row > Main link.
5. Connect tFixedFlowInput to tFileInputParquet using the Trigger > OnSubjobOk link.
6. Leave tS3Configuration alone without any
   connection.

1. Double-click tS3Configuration to open its Component view.
   Spark uses this component to connect to the S3 system in which your Job writes
   the actual business data. If you place neither
   tS3Configuration nor any other configuration
   component that supports Databricks on AWS, this business data is written in the
   Databricks Filesystem (DBFS).
   

2. In the Access key and the Secret
   key fields, enter the keys to be used to authenticate to
   S3.
3. In the Bucket name field, enter the name of the bucket
   and the folder in this bucket to be used to store the business data, for
   example, mybucket/myfolder. This folder is created on the
   fly if it does not exist but the bucket must already exist at runtime.

1. Double-click the tFixedFlowIput component to
   open its Component view.
   

2. Click the […] button next to Edit schema to open the schema editor.
3. Click the [+] button to add the schema
   columns as shown in this image.
   

4. Click OK to validate these changes and accept
   the propagation prompted by the pop-up dialog box.
5. In the Mode area, select the Use Inline
   Content radio button and paste the previously mentioned sample data
   into the Content field that is displayed.
6. In the Field separator field, enter a
   semicolon (;).
7. Double-click the tFileOutputParquet component to
   open its Component view.
   
8. Select the Define a storage configuration component
   check box and then select the tS3Configuration component
   you configured in the previous steps.
9. Click Sync columns to ensure that
   tFileOutputParquet has the same schema as
   tFixedFlowInput.
10. In the Folder/File field, enter the name of the S3
    folder to be used to store the sample data. For example, enter
    /sample_user, then as you have specified
    my_bucket/my_folder to use in
    tS3Configuration to store the business data on S3,
    the eventual directory on S3 becomes
    my_bucket/my_folder/sample_user.
11. From the Action drop-down list, select
    Create if the sample_user folder
    does not exist yet; if this folder already exists, select
    Overwrite.

1. Double-click tFileInputParquet to open its
   Component view.
   

2. Select the Define a storage configuration component
   check box and then select the tS3Configuration
   component you configured in the previous steps.
3. Click the […] button next to Edit
   schema to open the schema editor.
4. Click the [+] button to add the schema columns for
   output as shown in this image.
   

5. Click OK to validate these changes and accept the
   propagation prompted by the pop-up dialog box.
6. In the Folder/File field, enter the name of the folder
   from which you need to read data. In this scenario, it is
   sample_user.
7. Double click tLogRow to open its
   Component view and select the Table radio button to present the result in a table.
8. Press F6 to run this Job.

Once done, you can find your Job on the Job page on the Web UI
of your Databricks cluster and then check the execution log of your Job.

If the AWS SSE-KMS encryption (at-rest encryption) service is
enabled to set Default encryption to protect data
on the S3A system of your EMR cluster, select the SSE-KMS option in tS3Configuration when writing data to that S3A
system.

The Job calculates the occurrence of each incident type.

• The S3 system to be used is S3A.
• The SSE-KMS encryption service on AWS is enabled with the
  Default encryption feature and a
  customer managed CMK has been specified for it.
• The EMR cluster to be used is created with SSE-KMS and the
  EMR_EC2_DefaultRole role has been added to the above-mentioned CMK.
• The administrator of your EMR cluster has granted the appropriate
  rights and permissions to the AWS account you are using in your Jobs.
• Your EMR cluster has been properly set up and is running.
• A Talend
  Jobserver has been deployed on an instance within the network of your EMR
  cluster, such as the instance for the master of your cluster.

• In your Studio or in Talend Administration Center, define this Jobserver as the execution server of your
  Jobs.

1. In the
   Integration
   perspective of the Studio, create an empty Spark Batch Job from the Job
   Designs node in the Repository tree view.
2. In the workspace, enter the name of the component to be used and select this
   component from the list that appears. In this scenario, the components are
   tHDFSConfiguration (labeled emr_hdfs), tS3Configuration, tFixedFlowInput, tAggregateRow and tFileOutputParquet.
   The tFixedFlowInput component is used to load
   the sample data into the data flow. In the real-world practice, use the input component specific to the data format or the source system to be used instead of tFixedFlowInput.
3. Connect tFixedFlowInput, tAggregateRow and
   tFileOutputParquet using the Row > Main link.
4. Leave the tHDFSConfiguration component and the tS3Configuration component alone
   without any connection.

1. Double-click tHDFSConfiguration to open its Component view.

   Spark uses this component to connect to the HDFS system to
   which the jar files dependent on the Job are transferred.

2. If you have defined the HDFS connection metadata under the
   Hadoop cluster node in Repository, select Repository from the Property
   type drop-down list and then click the […] button to select the HDFS connection you
   have defined from the Repository content
   wizard.

   For further information about setting up a reusable HDFS
   connection, search for centralizing HDFS metadata on Talend Help Center (https://help.talend.com).

   If you complete this step, you can skip the following steps
   about configuring tHDFSConfiguration
   because all the required fields should have been filled automatically.

3. In the Version area,
   select the Hadoop distribution you need to connect to and its version.
4. In the NameNode URI
   field, enter the location of the machine hosting the NameNode service of the
   cluster. If you are using WebHDFS, the location should be
   webhdfs://masternode:portnumber; WebHDFS with SSL is not
   supported yet.
5. In the Username field,
   enter the authentication information used to connect to the HDFS system to be
   used. Note that the user name must be the same as you have put in the Spark configuration tab.

1. Double-click tS3Configuration to open its Component view.

   Spark uses this component to connect to the S3 system to which the business data is stored. In this scenario, the sample data about the street incidents is written and ecrypted on S3.

2. Select the Use s3a Filesystem check box. The Inherit credentials from AWS role check box and the use SSE-KMS encryption
3. Enter the access credentials of the AWS account to be used.
   • If allowed by the security policy of your organization, in the Access Key and the Secret Key fields, enter the credentials.

     If you do not know the credentials to be used, contact the administrator of your AWS system or check Getting Your AWS Access
     Keys from the AWS documentation.

   • If the security policy of your organization does not allow you to expose the credentials in a client application, select Inherit credentials from AWS role to obtain the role-based temporary AWS security credentials from your EMR instance metadata. An IAM role must have been specified to associate with this EMR instance.

     For further information about using an IAM role to grant permissions, see Using IAM roles from the AWS documentation.

4. Select Use SSE-KMS encryption check box to enable the Job to verify and use the SSE-KMS encryption service of your cluster.
5. In the Bucket name field, enter the name of the bucket to be used to store the sample data. This bucket must have existed when you launch your Job. For example, enter my_bucket/my_folder.

1. Double-click tFixedFlowInput to display its
   Basic settings view.
   

2. Click the […] button next to Edit
   Schema to open the schema editor.
   

3. Click the [+] button to add four columns, namely
   id, address,
   incident_type and description.
4. Click Ok to close the schema editor and accept
   propagating the changes when prompted by the system.

5. In the Mode area, select the Use Inline Content(delimited file) radio button to display the Content area and enter the sample data.

1. Double-click tAggregateRow to open its
   Component view.
   

2. Click the […] button next to Edit schema to open the schema editor.
3. On the output side (right), click the [+]
   button twice to add two rows and in the Column column, rename them to incident_type and incident_number, respectively.
   

4. In the Type column of the incident_number row of the output side, select Integer.
5. Click OK to validate these changes and accept
   the propagation prompted by the pop-up dialog box.
6. In the Group by table, add one row by
   clicking the [+] button and configure
   this row as follows to group the outputted data:

   • Output column:
     select the columns from the output schema to be used as the
     conditions to group the outputted data. In this example, it is the
     incident_type from the output schema.

   • Input column
     position: select the columns from the input schema
     to send data to the output columns you have selected in the
     Output
     column column. In this scenario, it is the
     incident_type column from the input schema.

7. In the Operations table, add one row by clicking the [+] button once and configure this
   row as follows to calculate the occurrence of each incident type:

   • Output column:
     select the column from the output schema to store the calculation
     results. In this scenario, it is incident_number.

   • Function: select
     the function to be used to process the incoming data. In this
     scenario, select count. It counts the frequency of each
     incident.

   • Input column
     position: select the column from the input schema to
     provide the data to be processed. In this scenario, it is incident_type.

1. Double-click the tFileOutputParquet component to
   open its Component view.
   
2. Select the Define a storage
   configuration component check box and then select the tS3Configuration component you configured in the
   previous steps.
3. Click Sync columns to
   ensure that tFileOutputParquet retrieve the
   schema from the output side of tAggregateRow.
4. In the Folder/File
   field, enter the name of the folder to be used to store the aggregated data in
   the S3 bucket specified in tS3Configuration. For example,
   enter /sample_user, then at runtime, the folder called
   sample_user at the root of the bucket is used to
   store the output of your Job.
5. From the Action
   drop-down list, select Create if the
   folder to be used does not exist yet in the bucket to be used; if this folder
   already exists, select Overwrite.
6. Click Run to open its view and then click the
   Spark Configuration tab to display its view
   for configuring the Spark connection.
7. Select the Use local mode check box to test your Job locally before eventually submitting it to the remote Spark cluster.

   In the local mode, the Studio builds the Spark environment in itself on the fly in order to
   run the Job in. Each processor of the local machine is used as a Spark
   worker to perform the computations.

8. In this mode, your local file system is used; therefore, deactivate the
   configuration components such as tS3Configuration or
   tHDFSConfiguration that provides connection
   information to a remote file system, if you have placed these components
   in your Job.
9. In the Component view of tFileOutputParquet, change the file path in the Folder/File field to a local directory and adapt the action to be taken on the Action drop-down list, that is to say, creating a new folder or overwriting the existing one.
10. On the Run tab, click Basic
    Run and in this view, click Run to execute your Job locally to test its design
    logic.
11. When your Job runs successfully, clear the Use local
    mode check box in the Spark Configuration
    view of the Run tab, then in the design workspace of your
    Job, activate the configuration components and revert the changes you just made
    in tFileOutputParquet for the local test.

1. Enter the basic connection information to EMR:

   Yarn client

   The Studio runs the Spark driver to orchestrate how the Job should be performed and then send the orchestration to the Yarn service of a given Hadoop cluster so that the Resource Manager of this Yarn service requests execution resources accordingly. If you are using the Yarn client mode, you need to set the following parameters in their corresponding fields (if you leave the check box of a service clear, then at runtime, the configuration about this parameter in the Hadoop cluster to be used will be ignored): In the Resource managerUse datanode field, enter the address of the ResourceManager service of the Hadoop cluster to be used. Select the Set resourcemanager scheduler address check box and enter the Scheduler address in the field that appears. Select the Set jobhistory address check box and enter the location of the JobHistory server of the Hadoop cluster to be used. This allows the metrics information of the current Job to be stored in that JobHistory server. Select the Set staging directory check box and enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution. If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal names for the ResourceManager service and the JobHistory service in the displayed fields. This enables you to use your user name to authenticate against the credentials stored in Kerberos. These principals can be found in the configuration files of your distribution, such as in yarn-site.xml and in mapred-site.xml. If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend Jobserver. Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used. The User name field is available when you are not using Kerberos to authenticate. In the User name field, enter the login user name for your distribution. If you leave it empty, the user name of the machine hosting the Studio will be used. If the Spark cluster cannot recognize the machine in which the Job is launched, select this Define the driver hostname or IP address check box and enter the host name or the IP address of this machine. This allows the Spark master and its workers to recognize this machine to find the Job and thus its driver. Note that in this situation, you also need to add the name and the IP address of this machine to its host file.

   Yarn cluster

   The Spark driver runs in your Yarn cluster to orchestrate how the Job should be performed. If you are using the Yarn cluster mode, you need to define the following parameters in their corresponding fields (if you leave the check box of a service clear, then at runtime, the configuration about this parameter in the Hadoop cluster to be used will be ignored): In the Resource managerUse datanode field, enter the address of the ResourceManager service of the Hadoop cluster to be used. Select the Set resourcemanager scheduler address check box and enter the Scheduler address in the field that appears. Select the Set jobhistory address check box and enter the location of the JobHistory server of the Hadoop cluster to be used. This allows the metrics information of the current Job to be stored in that JobHistory server. Select the Set staging directory check box and enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution. Set path to custom Hadoop configuration JAR: if you are using connections defined in Repository to connect to your Cloudera or Hortonworks cluster, you can select this check box in the Repository wizard and in the field that is displayed, specify the path to the JAR file that provides the connection parameters of your Hadoop environment. Note that this file must be accessible from the machine where you Job is launched. This kind of Hadoop configuration JAR file is automatically generated when you build a Big Data Job from the Studio. This JAR file is by default named with this pattern: hadoop-conf-[name_of_the_metadata_in_the_repository]_[name_of_the_context].jar 1 hadoop-conf-[name_of_the_metadata_in_the_repository]_[name_of_the_context].jar You can also download this JAR file from the web console of your cluster or simply create a JAR file yourself by putting the configuration files in the root of your JAR file. For example: hdfs-sidt.xml core-site.xml 1 2 hdfs-sidt.xml core-site.xml The parameters from your custom JAR file override the parameters you put in the Spark configuration field. They also override the configuration you set in the configuration components such as tHDFSConfiguration or tHBaseConfiguration when the related storage system such as HDFS, HBase or Hive are native to Hadoop. But they do not override the configuration set in the configuration components for the third-party storage system such as tAzureFSConfiguration. If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal names for the ResourceManager service and the JobHistory service in the displayed fields. This enables you to use your user name to authenticate against the credentials stored in Kerberos. These principals can be found in the configuration files of your distribution, such as in yarn-site.xml and in mapred-site.xml. If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend Jobserver. Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used. The User name field is available when you are not using Kerberos to authenticate. In the User name field, enter the login user name for your distribution. If you leave it empty, the user name of the machine hosting the Studio will be used. Select the Wait for the Job to complete check box to make your Studio or, if you use Talend Jobserver, your Job JVM keep monitoring the Job until the execution of the Job is over. By selecting this check box, you actually set the spark.yarn.submit.waitAppCompletion property to be true. While it is generally useful to select this check box when running a Spark Batch Job, it makes more sense to keep this check box clear when running a Spark Streaming Job.

   Ensure that the user name in the Yarn
   client mode is the same one you put in
   tS3Configuration, the component used to provides S3
   connection information to Spark.

2. 
   With the Yarn client mode, the
   Property type list is displayed to allow you
   to select an established Hadoop connection from the Repository, on the condition that you have created this connection
   in the Repository. Then the Studio will reuse
   that set of connection information for this Job.
   
3. 
   If you need to launch from Windows, it is recommended to specify where
   the winutils.exe program to be used is stored.
   

   • If you know where to find your winutils.exe file and you want to use it, select the Define the Hadoop home directory check box
     and enter the directory where your winutils.exe is
     stored.

   • Otherwise, leave this check box clear, the Studio generates one
     by itself and automatically uses it for this Job.

4. 
   In the Spark “scratch” directory
   field, enter the directory in which the Studio stores in the local system the
   temporary files such as the jar files to be transferred. If you launch the Job
   on Windows, the default disk is C:. So if you leave /tmp in this field, this directory is C:/tmp.
   

1. Press Ctrl+S to save your Job.
2. In the Run tab, click Run to execute the Job.

These properties are used to configure tS3Configuration running in the Spark Streaming Job framework.

The Spark Streaming
tS3Configuration component belongs to the Storage family.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

Basic settings

Advanced settings

Usage