July 30, 2023

tAvroInput – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tAvroInput – Docs for ESB 7.x

tAvroInput

Extracts records from any given Avro format files for other components to process
the records.

tAvroInput parses Avro format files in
a given distributed file system and loads data to a data flow to pass the data to the
transformation component that follows.

Depending on the Talend
product you are using, this component can be used in one, some or all of the following
Job frameworks:

tAvroInput MapReduce properties (deprecated)

These properties are used to configure tAvroInput running in the MapReduce Job framework.

The MapReduce
tAvroInput component belongs to the MapReduce family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.

Basic settings

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.
 

Built-In: You create and store the schema locally for this component
only.
 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will read all of the files stored in that folder, for example,/user/talend/in; if sub-folders exist, the sub-folders are automatically ignored unless you define the property mapreduce.input.fileinputformat.input.dir.recursive to be true in the Hadoop properties table in the Hadoop configuration tab.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

Note that you need
to ensure you have properly configured the connection to the Hadoop
distribution to be used in the Hadoop
configuration tab in the Run view.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

In a
Talend
 Map/Reduce Job, it is used as a start component and requires
a transformation component as output link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tAvroInput as well as the MapReduce family appears in the Palette of the Studio.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
 data integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

Filtering Avro format employee data

This scenario applies only to subscription-based Talend products with Big
Data.

This scenario illustrates how to create a
Talend
 Map/Reduce Job to
read, transform and write Avro format data by using Map/Reduce components. This Job
generates Map/Reduce code and directly runs in Hadoop. In addition, the Map bar in the workspace indicates that only a mapper will be
used in this Job and at runtime, it shows the progress of the Map computation.

tAvroInput_1.png

Note that the
Talend
 Map/Reduce components are available to
subscription-based Big Data users only and this scenario can be replicated only with
Map/Reduce components.

The sample data to be used in this scenario is employee information of a company with
records virtually reading as follows but actually only visible as Avro format files:

Before starting to replicate this scenario, ensure that you have appropriate rights
and permissions to access the Hadoop distribution to be used. Then proceed as
follows:

Linking the components

  1. In the
    Integration
     perspective
    of the Studio, create an empty Map/Reduce Job from the
    Job Designs     node in the Repository tree view.

    For further information about how to create a Map/Reduce Job, see

    Talend Open Studio for Big Data Getting Started Guide
    .
  2. Drop tAvroInput, tMap, tHDFSOutput and
    tAvroOutput onto the workspace.
  3. Connect tAvroInput to tMap using the Row >
    Main     link.
  4. Do the same to connect tMap to tHDFSOutput and tAvroOutput, respectively. In doing it, you are prompted to
    name each link. In this example, name the link to tHDFSOutput to out1 and
    the link to tAvroOutput to reject.

Setting up Hadoop connection

  1. Click Run to open its view and then click the
    Hadoop Configuration tab to display its
    view for configuring the Hadoop connection for this Job.
  2. From the Property type list,
    select Built-in. If you have created the
    connection to be used in Repository, then
    select Repository and thus the Studio will
    reuse that set of connection information for this Job.
  3. In the Version area, select the
    Hadoop distribution to be used and its version.

    • If you use Google Cloud Dataproc, see Google Cloud Dataproc.

    • If you cannot
      find the Cloudera version to be used from this drop-down list, you can add your distribution
      via some dynamic distribution settings in the Studio.

    • If you cannot find from the list the distribution corresponding to
      yours, select Custom so as to connect to a
      Hadoop distribution not officially supported in the Studio. For a
      step-by-step example about how to use this
      Custom option, see Connecting to a custom Hadoop distribution.

  4. In the Name node field, enter the location of
    the master node, the NameNode, of the distribution to be used. For example,
    hdfs://tal-qa113.talend.lan:8020.

    • If you are using a MapR distribution, you can simply leave maprfs:/// as it is in this field; then the MapR
      client will take care of the rest on the fly for creating the connection. The
      MapR client must be properly installed. For further information about how to set
      up a MapR client, see the following link in MapR’s documentation: http://doc.mapr.com/display/MapR/Setting+Up+the+Client

    • If you are using WebHDFS, the location should be
      webhdfs://masternode:portnumber; WebHDFS with SSL is not
      supported yet.

  5. In the Resource Manager field,
    enter the location of the ResourceManager of your distribution. For example,
    tal-qa114.talend.lan:8050.

    • Then you can continue to set the following parameters depending on the
      configuration of the Hadoop cluster to be used (if you leave the check
      box of a parameter clear, then at runtime, the configuration about this
      parameter in the Hadoop cluster to be used will be ignored):

      • Select the Set resourcemanager
        scheduler address         check box and enter the Scheduler address in
        the field that appears.

      • Select the Set jobhistory
        address         check box and enter the location of the JobHistory
        server of the Hadoop cluster to be used. This allows the metrics information of
        the current Job to be stored in that JobHistory server.

      • Select the Set staging
        directory         check box and enter this directory defined in your
        Hadoop cluster for temporary files created by running programs. Typically, this
        directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files
        such as yarn-site.xml or mapred-site.xml of your distribution.

      • Select the Use datanode hostname check box to allow the
        Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname
        property to true. When connecting to a
        S3N filesystem, you must select this check box.


  6. If you are accessing the Hadoop cluster running
    with Kerberos security, select this check box, then, enter the Kerberos
    principal name for the NameNode in the field displayed. This enables you to use
    your user name to authenticate against the credentials stored in Kerberos.

    • If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the
      MapR ticket authentication configuration in addition or as an alternative by following
      the explanation in Connecting to a security-enabled MapR.

      Keep in mind that this configuration generates a new MapR security ticket for the username
      defined in the Job in each execution. If you need to reuse an existing ticket issued for the
      same username, leave both the Force MapR ticket
      authentication       check box and the Use Kerberos
      authentication       check box clear, and then MapR should be able to automatically
      find that ticket on the fly.

    In addition, since this component performs Map/Reduce computations, you
    also need to authenticate the related services such as the Job history server and
    the Resource manager or Jobtracker depending on your distribution in the
    corresponding field. These principals can be found in the configuration files of
    your distribution. For example, in a CDH4 distribution, the Resource manager
    principal is set in the yarn-site.xml file and the Job history
    principal in the mapred-site.xml file.

    If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains
    pairs of Kerberos principals and encrypted keys. You need to enter the principal to
    be used in the Principal field and the access
    path to the keytab file itself in the Keytab
    field. This keytab file must be stored in the machine in which your Job actually
    runs, for example, on a Talend
    Jobserver.

    Note that the user that executes a keytab-enabled Job is not necessarily
    the one a principal designates but must have the right to read the keytab file being
    used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this
    situation, ensure that user1 has the right to read the keytab
    file to be used.

  7. In the User name field, enter the login user
    name for your distribution. If you leave it empty, the user name of the machine
    hosting the Studio will be used.
  8. In the Temp folder field, enter the path in
    HDFS to the folder where you store the temporary files generated during
    Map/Reduce computations.

  9. Leave the default value of the Path separator in
    server     as it is, unless you have changed the separator used by your
    Hadoop distribution’s host machine for its PATH variable or in other words, that
    separator is not a colon (:). In that situation, you must change this value to the
    one you are using in that host.
  10. Leave the Clear temporary folder check box
    selected, unless you want to keep those temporary files.
  11. Leave the Compress intermediate map output to reduce
    network traffic     check box selected, so as to spend shorter time
    to transfer the mapper task partitions to the multiple reducers.

    However, if the data transfer in the Job is negligible, it is recommended to
    clear this check box to deactivate the compression step, because this
    compression consumes extra CPU resources.
  12. If you need to use custom Hadoop properties, complete the Hadoop properties table with the property or
    properties to be customized. Then at runtime, these changes will override the
    corresponding default properties used by the Studio for its Hadoop
    engine.

    For further information about the properties required by Hadoop, see Apache’s
    Hadoop documentation on http://hadoop.apache.org, or
    the documentation of the Hadoop distribution you need to use.

  13. If the HDFS transparent encryption has been enabled in your cluster, select
    the Setup HDFS encryption configurations check
    box and in the HDFS encryption key provider field
    that is displayed, enter the location of the KMS proxy.

    For further information about the HDFS transparent encryption and its KMS proxy, see Transparent Encryption in HDFS.

  14. You can tune the map and reduce computations by
    selecting the Set memory check box to set proper memory allocations
    for the computations to be performed by the Hadoop system.

    The memory parameters to be set are Map (in Mb),
    Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate
    memory to the map and the reduce computations and the ApplicationMaster of YARN.

    For further information about the Resource Manager, its scheduler and the
    ApplicationMaster, see YARN’s documentation such as http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/.

    For further information about how to determine YARN and MapReduce memory configuration
    settings, see the documentation of the distribution you are using, such as the following
    link provided by Hortonworks: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html.


  15. If you are using Cloudera V5.5+, you can select the Use Cloudera Navigator check box to enable the Cloudera Navigator
    of your distribution to trace your Job lineage to the component level, including the
    schema changes between components.

    With this option activated, you need to set the following parameters:

    • Username and Password: this is the credentials you use to connect to your Cloudera
      Navigator.

    • Cloudera Navigator URL : enter the location of the
      Cloudera Navigator to be connected to.

    • Cloudera Navigator Metadata URL: enter the location
      of the Navigator Metadata.

    • Activate the autocommit option: select this check box
      to make Cloudera Navigator generate the lineage of the current Job at the end of the
      execution of this Job.

      Since this option actually forces Cloudera Navigator to generate lineages of
      all its available entities such as HDFS files and directories, Hive queries or Pig
      scripts, it is not recommended for the production environment because it will slow the
      Job.

    • Kill the job if Cloudera Navigator fails: select this check
      box to stop the execution of the Job when the connection to your Cloudera Navigator fails.

      Otherwise, leave it clear to allow your Job to continue to run.

    • Disable SSL validation: select this check box to
      make your Job to connect to Cloudera Navigator without the SSL validation
      process.

      This feature is meant to facilitate the test of your Job but is not
      recommended to be used in a production cluster.


  16. If you are using Hortonworks Data Platform V2.4.0 onwards and you have
    installed Atlas in your cluster, you can select the Use
    Atlas     check box to enable Job lineage to the component level, including the
    schema changes between components.

    With this option activated, you need to set the following parameters:

    • Atlas URL: enter the location of the Atlas to be
      connected to. It is often http://name_of_your_atlas_node:port

    • Die on error: select this check box to stop the Job
      execution when Atlas-related issues occur, such as connection issues to Atlas.

      Otherwise, leave it clear to allow your Job to continue to run.

    In the Username and Password fields, enter the authentication information for access to
    Atlas.

Reading Avro data

Configuring tAvroInput
  1. Double-click tAvroInput to open its
    Component view.

    tAvroInput_2.png

  2. Click the

    tAvroInput_3.png

    button next to Edit
    schema    to open the schema editor.

  3. Click the

    tAvroInput_4.png

    button four times to add four rows and in the Column column, rename them to Id, FirstName, LastName and
    Reg_date, respectively.

    tAvroInput_5.png

  4. In the Type column, select Integer for Id
    and Date for Reg_date. The date pattern used in this scenario is
    dd-MM-yyyy.
  5. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.
  6. In the Folder/File field, enter the path,
    or browse to the source file you need the Job to read.

Configuring tAvroInput

  1. Double-click tAvroInput to open its
    Component view.

    tAvroInput_2.png

  2. Click the

    tAvroInput_3.png

    button next to Edit
    schema    to open the schema editor.

  3. Click the

    tAvroInput_4.png

    button four times to add four rows and in the Column column, rename them to Id, FirstName, LastName and
    Reg_date, respectively.

    tAvroInput_5.png

  4. In the Type column, select Integer for Id
    and Date for Reg_date. The date pattern used in this scenario is
    dd-MM-yyyy.
  5. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.
  6. In the Folder/File field, enter the path,
    or browse to the source file you need the Job to read.

Transforming the data

Configuring tMap
  1. Double-click tMap to open the Map Editor.

    tAvroInput_10.png

  2. Drop the four columns of the input schema from the input side (left) into
    each of the output flows of the output side (right), that is to say,
    out1 and reject. This way the input flow and the output flows are
    mapped.
  3. In the table representing the out1
    flow, click

    tAvroInput_11.png

    to display the filter editing area and enter the
    following expression to select the employee records that were registered
    before January 1st, 2008 (01-01-2008).
    row1.Reg_date.before( new Date(108,0,1))

  4. In the table representing the reject
    flow, click

    tAvroInput_12.png

    to display the property settings panel.

  5. In the Value field of the Catch output reject row, click the

    tAvroInput_3.png

    and select true in the
    pop-up dialog box. This allows you to output the records rejected by the
    out1 flow.

  6. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.

Configuring tMap

  1. Double-click tMap to open the Map Editor.

    tAvroInput_10.png

  2. Drop the four columns of the input schema from the input side (left) into
    each of the output flows of the output side (right), that is to say,
    out1 and reject. This way the input flow and the output flows are
    mapped.
  3. In the table representing the out1
    flow, click

    tAvroInput_11.png

    to display the filter editing area and enter the
    following expression to select the employee records that were registered
    before January 1st, 2008 (01-01-2008).
    row1.Reg_date.before( new Date(108,0,1))

  4. In the table representing the reject
    flow, click

    tAvroInput_12.png

    to display the property settings panel.

  5. In the Value field of the Catch output reject row, click the

    tAvroInput_3.png

    and select true in the
    pop-up dialog box. This allows you to output the records rejected by the
    out1 flow.

  6. Click OK to validate these changes and
    accept the propagation prompted by the pop-up dialog box.

Writing data in HDFS

Configuring the selected employee data

  1. Double-click tHDFSOutput to open its
    Component view.

    tAvroInput_18.png

  2. In the Folder field, enter the path, or
    browse to the folder you want to write the employee records registered
    before January 1st, 2008.
  3. From the Type list, select the data
    format for the records to be written. In this example, select Text file.
  4. From the Action list, select the
    operation you need to perform on the file in question. If the file already
    exists, select Overwrite; otherwise, select
    Create.
  5. Select the Merge result to single file
    check box and enter the path, or browse to the file you need to write the
    merged output data in.
  6. If the file for the merged data exists, select the Override target file check box to overwrite that
    file.

Configuring the rejected employee data

  1. Double-click tAvroOutput to open its
    Component view.

    tAvroInput_19.png

  2. In the Folder field, enter the path, or
    browse to the folder you want to write the employee records registered after
    January 1st, 2008.
  3. From the Action list, select the
    operation you need to perform on the folder in question. If the folder
    already exists, select Overwrite;
    otherwise, select Create.

Executing the Job

Then you can press F6 to run this Job and the
Map bar in the workspace shows the progress of
the Map computation.

Once done, you can check the results in the web console of the Hadoop distribution
being used.

The records in the out1 flow is outputted and
merged into one text file.

tAvroInput_20.png

The records in the reject flow is outputted as
Avro format files.

tAvroInput_21.png

If you need to obtain more execution information about this Job, you can check the
web console of the Jobtracker of the Hadoop distribution being used.

tAvroInput properties for Apache Spark Batch

These properties are used to configure tAvroInput running in the Spark Batch Job framework.

The Spark Batch
tAvroInput component belongs to the File family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

Basic settings

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.
 

Built-In: You create and store the schema locally for this component
only.
 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will
read all of the files stored in that folder, for example,
/user/talend/in; if sub-folders exist, the sub-folders are automatically
ignored unless you define the property
spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive to be
true in the Advanced properties table in the
Spark configuration tab.

  • Depending on the filesystem to be used, properly configure the corresponding
    configuration component placed in your Job, for example, a
    tHDFSConfiguration component for HDFS, a
    tS3Configuration component for S3 and a
    tAzureFSConfiguration for Azure Storage and Azure Data Lake
    Storage.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Advanced settings

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

When you want to control the partition number, you can generally set at least as many partitions as
the number of executors for parallelism, while bearing in mind the available memory and the
data transfer pressure on your network.

Use hierarchical mode

Select this check box to map the binary (including hierarchical) Avro schema to the
flat schema defined in the schema editor of the current component. If the Avro
message to be processed is flat, leave this check box clear.

Once selecting it, you need set the following parameter(s):

  • Mapping: create the map
    between the schema columns of the current component and the data stored
    in the hierarchical Avro message to be handled. In the
    Node column, you need to
    enter the JSON path pointing to the data to be read from the
    Avro message.

Usage

Usage rule

This component is used as a start component and requires an output
link..

This component, along with the Spark Batch component Palette it belongs to,
appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a
scenario presents only Standard Jobs, that is to
say traditional
Talend
 data integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

  • Yarn mode (Yarn client or Yarn cluster):

    • When using Google Dataproc, specify a bucket in the
      Google Storage staging bucket
      field in the Spark configuration
      tab.

    • When using HDInsight, specify the blob to be used for Job
      deployment in the Windows Azure Storage
      configuration       area in the Spark
      configuration       tab.

    • When using Altus, specify the S3 bucket or the Azure
      Data Lake Storage for Job deployment in the Spark
      configuration       tab.
    • When using Qubole, add a
      tS3Configuration to your Job to write
      your actual business data in the S3 system with Qubole. Without
      tS3Configuration, this business data is
      written in the Qubole HDFS system and destroyed once you shut
      down your cluster.

    • When using on-premise
      distributions, use the configuration component corresponding
      to the file system your cluster is using. Typically, this
      system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the
    configuration component corresponding to the file system your cluster is
    using, such as tHDFSConfiguration or
    tS3Configuration.

    If you are using Databricks without any configuration component present
    in your Job, your business data is written directly in DBFS (Databricks
    Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Batch version of this component
yet.

tAvroInput properties for Apache Spark Streaming

These properties are used to configure tAvroInput running in the Spark Streaming Job framework.

The Spark Streaming
tAvroInput component belongs to the File family.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

Basic settings

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.
 

Built-In: You create and store the schema locally for this component
only.
 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will
read all of the files stored in that folder, for example,
/user/talend/in; if sub-folders exist, the sub-folders are automatically
ignored unless you define the property
spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive to be
true in the Advanced properties table in the
Spark configuration tab.

  • Depending on the filesystem to be used, properly configure the corresponding
    configuration component placed in your Job, for example, a
    tHDFSConfiguration component for HDFS, a
    tS3Configuration component for S3 and a
    tAzureFSConfiguration for Azure Storage and Azure Data Lake
    Storage.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Advanced settings

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

When you want to control the partition number, you can generally set at least as many partitions as
the number of executors for parallelism, while bearing in mind the available memory and the
data transfer pressure on your network.

Use hierarchical mode

Select this check box to map the binary (including hierarchical) Avro schema to the
flat schema defined in the schema editor of the current component. If the Avro
message to be processed is flat, leave this check box clear.

Once selecting it, you need set the following parameter(s):

  • Mapping: create the map
    between the schema columns of the current component and the data stored
    in the hierarchical Avro message to be handled. In the
    Node column, you need to
    enter the JSON path pointing to the data to be read from the
    Avro message.

Usage

Usage rule

This component is used as a start component and requires an output link.

This component is only used to provide the lookup flow (the right side of a join
operation) to the main flow of a tMap component. In this
situation, the lookup model used by this tMap must be
Load once.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
 data
integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

  • Yarn mode (Yarn client or Yarn cluster):

    • When using Google Dataproc, specify a bucket in the
      Google Storage staging bucket
      field in the Spark configuration
      tab.

    • When using HDInsight, specify the blob to be used for Job
      deployment in the Windows Azure Storage
      configuration       area in the Spark
      configuration       tab.

    • When using Altus, specify the S3 bucket or the Azure
      Data Lake Storage for Job deployment in the Spark
      configuration       tab.
    • When using Qubole, add a
      tS3Configuration to your Job to write
      your actual business data in the S3 system with Qubole. Without
      tS3Configuration, this business data is
      written in the Qubole HDFS system and destroyed once you shut
      down your cluster.

    • When using on-premise
      distributions, use the configuration component corresponding
      to the file system your cluster is using. Typically, this
      system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the
    configuration component corresponding to the file system your cluster is
    using, such as tHDFSConfiguration or
    tS3Configuration.

    If you are using Databricks without any configuration component present
    in your Job, your business data is written directly in DBFS (Databricks
    Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Streaming version of this component
yet.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x