August 16, 2023

tTachyonConfiguration – Docs for ESB 6.x

  2. Talend for ESB Docs
  3. Docs 6.x
  4. tTachyonConfiguration – Docs for ESB 6.x

tTachyonConfiguration

Defines a connection to Tachyon storage system and enables the reuse of the
configuration in the same Job.

You define a connection to Tachyon in tTachyonConfiguration and configure the file system
related components to reused this connection. At runtime, the Spark
cluster to be used reads this configuration to connect to
Tachyon.

Depending on the Talend solution you
are using, this component can be used in one, some or all of the following Job
frameworks:

tTachyonConfiguration properties for Apache Spark Batch

These properties are used to configure tTachyonConfiguration running in the Spark Batch Job framework.

The Spark Batch
tTachyonConfiguration component belongs to the Storage family.

The component in this framework is available only if you have subscribed to one
of the
Talend
 solutions with Big Data.

Basic settings

Tachyon master URI

Enter the address of the master server of the Tachyon cluster to be used.

This information can be found in the conf/tachyon-env.sh file of your Tachyon system.

For details about the version compatibility between Tachyon and Spark, see Tachyon
documentation at http://tachyon-project.org/documentation/Running-Spark-on-Tachyon.html.

UnderFS username

Enter the credential required by the file system (underlayer storage system in terms of Tachyon) used by your Tachyon cluster. The default file system is HDFS.

This information can be found in the conf/tachyon-env.sh file of your Tachyon system.

Usage

Usage rule

This component is used with no need to be connected to other components.

You need to drop tTachyonConfiguration along with the file system related Subjob to be run in the
same Job so that the configuration is used by the whole Job at runtime.

This component, along with the Spark Batch component Palette it belongs to, appears only
when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
 data integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Related scenarios

For a scenario about how to use the same type of component in a Spark Batch Job, see Writing and reading data from MongoDB using a Spark Batch Job.

tTachyonConfiguration properties for Apache Spark Streaming

These properties are used to configure tTachyonConfiguration running in the Spark Streaming Job framework.

The Spark Streaming
tTachyonConfiguration component belongs to the Storage family.

The component in this framework is available only if you have subscribed to Talend Real-time Big Data Platform or Talend Data
Fabric.

Basic settings

Tachyon master URI

Enter the address of the master server of the Tachyon cluster to be used.

This information can be found in the conf/tachyon-env.sh file of your Tachyon system.

For details about the version compatibility between Tachyon and Spark, see Tachyon
documentation at http://tachyon-project.org/documentation/Running-Spark-on-Tachyon.html.

UnderFS username

Enter the credential required by the file system (underlayer storage system in terms of Tachyon) used by your Tachyon cluster. The default file system is HDFS.

This information can be found in the conf/tachyon-env.sh file of your Tachyon system.

Usage

Usage rule

This component is used with no need to be connected to other components.

You need to drop tTachyonConfiguration along with the file system related Subjob to be run in the
same Job so that the configuration is used by the whole Job at runtime.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
 data
integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Related scenarios

For a scenario about how to use the same type of component in a Spark Streaming Job, see
Reading and writing data in MongoDB using a Spark Streaming Job.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 6.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x