tSparkLoad – Docs for ESB 5.x

tSparkLoad

1. In the Integration perspective
   of the Studio, create an empty Job from the Job
   Designs node in the Repository tree view.

   For further information about how to create a Job, see Talend Studio User Guide.

2. In the workspace, enter the name of the component to be used and select
   this component from the list that opens. In this scenario, the components
   are tSparkConnection, tSparkLoad, tSparkNormalize, tSparkFilterRow and tSparkLog.

3. Connect tSparkConnection to tSparkLoad using the Trigger > On Subjob Ok link.

4. Connect the other components using the Row > Spark
   combine link.

1. Double-click tSparkConnection to open its
   Component view.

   

2. From the Spark mode list, select the mode
   that fits the Spark system you need to use. In this scenario, select
   Standalone since the Spark system to be
   used is installed in a standalone Hadoop cluster. Since this Job needs to
   read contents from a given Twitter account, ensure that the cluster you are
   connecting has the access to the Internet.

   For further details about the different Spark modes available in this
   component, see tSparkConnection. You can as well read
   Apache’s documentation about Spark or the documentation of the Hadoop
   distribution you are using for more relevant details.

3. In the Distribution and the Version lists, select the options that
   corresponds to the Hadoop cluster to be used.

   If the distribution you need to use is not yet officially supported by the
   Spark components, you need to select Custom
   and use the […] button that is displayed
   to set up the related configuration. For further information about how to
   configure a custom Hadoop connection, see Connecting to a custom Hadoop distribution.

4. In the Spark host field, enter the URI of
   the Spark master node.

5. In the Spark home field, enter the path
   to the Spark executables and libraries in the Hadoop cluster to be used.
   This path is the value of the SPARK_HOME variable.

6. Select the Define the driver hostname or IP
   address check box and in the field that is displayed, enter
   the IP address of the machine in which the Job is to be run.

   This value is actually the value of the spark.driver.host
   property. For further information about this property, see Apache’s
   documentation about Spark.

7. Select the Execute this Job as a streaming
   application check box to switch the Job to the streaming
   mode. Then the Batch size field and the
   Define a streaming timeout check box
   are displayed to allow you to configure this mode.

   As explained previously, at least two cores are required in the cluster to
   be used to run the streaming mode.

8. In the Batch size field, enter the time
   interval at the end of which you want to read the new ingestion of data from
   the Twitter account to be used. In this scenario, enter 1000, meaning 1000 milliseconds.

   You need to use an appropriate time interval to keep up with the ingestion
   rate of the data streams, which can depend on a number of variables. For
   further information, see Apache’s Spark documentation about streaming
   programming.

9. In the Define a streaming timeout field,
   enter the time frame at the end of which the Job stops running. For example,
   enter 60000, meaning 60000
   milliseconds.

1. Double-click tSparkLoad to open its
   Component view.

   

2. From the Spark connection list, select
   the connection to be used.

3. Click the [+] button next to Edit schema to open the schema editor.

4. Click the [+] button three times to add
   three rows and rename them to id,
   username and hashtag, respectively.

   

5. Click OK to validate these changes and
   accept the propagation prompted by the pop-up dialog box.

6. In the Storage source field, select the
   type of the source data to be processed. In this scenario, select Twitter feed. Then the Twitter configuration area is displayed.

7. In the following fields, enter the authentication information for the
   Twitter account to be accessed: Consumer
   key, Consumer secret,
   Access token, Secret token. You need to obtain this group of information
   from the Twitter side.

8. In the Filters field, enter the phrases
   to filter the tweets you want to select. In this scenario, enter hadoop,talend,spark,streaming between double
   quotation marks.

   This filter selects the tweets using any of the phrases put in this
   field.

9. In the Data mapping table, the columns
   you defined in the schema have been automatically added to the Column column; then in the Properties column, you need to select which property of
   Twitter data you want each column to receive.

   In this scenario, select Id for the
   id column, Username for usrename and
   Hashtags for hashtag.

1. Double-click tSparkNormalize to open its
   Component view.

   

2. From the Column to normalize list, select
   the column the normalization is based on. In this scenario, it is hashtag.

3. In the Item separator field, enter the
   separator you need to use to normalize the hashtags such that each row
   contains one hashtag. In this example, enter the comma (,) because the hashtags retrieved in each batch
   are automatically separated by the comma.

1. Double-click tSparkFilterRow to open its
   Component view.

   

2. In the Filter configuration table, click
   the [+] button once to add one row and then
   configure this row as follows:

   • Logical: leave the default
     logical operator as is because it is ignored in the
     execution.

   • Column: select the column
     which the filtering is based on. It is hashtag in this scenario.

   • Operator: select the function
     you need to use to filter. It is Not
     equal in this example.

   • Value: enter the value used
     as the criteria of the filtering function. In this scenario,
     enter a pair of double quotation marks, meaning the value is
     empty.

   This filter allows you select the rows in which the hashtag
   is of the String type and is not empty.