|
Component family
|
Big Data / Spark
|
|
|
Function
|
tSparkJoin receives datasets from
two Spark flows and joins them using the mode you select.
|
|
Purpose
|
tSparkJoin joins different Spark
flows.
|
|
Basic settings
|
Schema and Edit
Schema
|
A schema is a row description. It defines the number of fields to be processed and passed on
to the next component. The schema is either Built-In or
stored remotely in the Repository.
Click Edit schema to make changes to the schema. If the
current schema is of the Repository type, three options are
available:
-
View schema: choose this option to view the
schema only.
-
Change to built-in property: choose this option
to change the schema to Built-in for local
changes.
-
Update repository connection: choose this option to change
the schema stored in the repository and decide whether to propagate the changes to
all the Jobs upon completion. If you just want to propagate the changes to the
current Job, you can select No upon completion and
choose this schema metadata again in the [Repository
Content] window.
When editing the output schema, you can name each column of this
schema as needs be but you must respect the following rules:
-
The output schema must contain all of the columns of
the two input schema.
-
The schema of the lookup flow must be appended to the
schema of the main flow and each of them must keep the
original order of their columns.
|
|
|
Spark connection
|
Select the Spark connection component to be used from the drop-down list in order to reuse
the connection created by that component.
|
| |
Join key
|
Add one or more rows to define the key columns to be used to join
the input flows.
|
| |
Join mode
|
Select the mode of the join you need to perform.
|
|
Advanced settings
|
tStatCatcher Statistics
|
Select this check box to collect log data at the component
level.
|
|
Global Variables
|
ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.
A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.
To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.
For further information about variables, see Talend Studio
User Guide.
|
|
Usage
|
This component is an intermediate component that is able to
combine multiple Spark flows.
|
|
Limitations
|
This component does not support the streaming mode and thus cannot
be used in a streaming-enable Spark process.For further information
about how to use the streaming mode, see tSparkConnection.
It is strongly recommended to use this component in a Spark-only Job, that is to say, to
design and run a Spark Job separately from the non Spark components or Jobs. For example, it
is not recommended to use the tRunJob component to
coordinate a Spark Job and a non Spark Job, or to use the tHDFSPut component along with the Spark components in the same Job.
|
