tHiveConfiguration
Enables the reuse of the connection configuration to Hive in the same
Job.
tHiveConfiguration provides Hive
connection information for the Hive related components used in the same
Spark Job. The Spark cluster to be used reads this configuration to
eventually connect to Hive.
Depending on the Talend solution you
are using, this component can be used in one, some or all of the following Job
frameworks:
-
Spark Batch: see tHiveConfiguration properties for Apache Spark Batch.
The component in this framework is available only if you have subscribed to one
of the
Talend
solutions with Big Data. -
Spark Streaming: see tHiveConfiguration properties for Apache Spark Streaming.
The component in this framework is available only if you have subscribed to Talend Real-time Big Data Platform or Talend Data
Fabric.
tHiveConfiguration properties for Apache Spark Batch
These properties are used to configure tHiveConfiguration running in the Spark Batch Job framework.
The Spark Batch
tHiveConfiguration component belongs to the Storage family.
The component in this framework is available only if you have subscribed to one
of the
Talend
solutions with Big Data.
Usage
|
Usage rule |
This component is used with no need to be connected to other components. You need to drop tHiveConfiguration along This component, along with the Spark Batch component Palette it belongs to, appears only Note that in this documentation, unless otherwise |
|
Spark Connection |
You need to use the Spark Configuration tab in
the Run view to define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files:
This connection is effective on a per-Job basis. |
Related scenarios
For a scenario about how to use the same type of component in a Spark Batch Job, see Writing and reading data from MongoDB using a Spark Batch Job.
tHiveConfiguration properties for Apache Spark Streaming
These properties are used to configure tHiveConfiguration running in the Spark Streaming Job framework.
The Spark Streaming
tHiveConfiguration component belongs to the Storage family.
The component in this framework is available only if you have subscribed to Talend Real-time Big Data Platform or Talend Data
Fabric.
Usage
|
Usage rule |
This component is used with no need to be connected to other components. You need to drop tHiveConfiguration along This component, along with the Spark Streaming component Palette it belongs to, appears Note that in this documentation, unless otherwise explicitly stated, a scenario presents |
|
Spark Connection |
You need to use the Spark Configuration tab in
the Run view to define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files:
This connection is effective on a per-Job basis. |
Related scenarios
For a scenario about how to use the same type of component in a Spark Streaming Job, see
Reading and writing data in MongoDB using a Spark Streaming Job.