August 17, 2023

tSparkUnion – Docs for ESB 5.x




This component will be available in the Palette of
Talend Studio on the condition that you have subscribed to one of
the Talend
solutions with Big Data.

tSparkUnion properties

Component family

Big Data / Spark



tSparkUnion merges datasets from
various sources, based on a common schema.


tSparkUnion centralized datasets
from different sources using the same schema.

Basic settings

Schema and Edit

A schema is a row description. It defines the number of fields to be processed and passed on
to the next component. The schema is either Built-In or
stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the
current schema is of the Repository type, three options are

  • View schema: choose this option to view the
    schema only.

  • Change to built-in property: choose this option
    to change the schema to Built-in for local

  • Update repository connection: choose this option to change
    the schema stored in the repository and decide whether to propagate the changes to
    all the Jobs upon completion. If you just want to propagate the changes to the
    current Job, you can select No upon completion and
    choose this schema metadata again in the [Repository


Spark connection

Select the Spark connection component to be used from the drop-down list in order to reuse
the connection created by that component.

Advanced settings

tStatCatcher Statistics

Select this check box to collect log data at the component

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.


This component is an intermediate component that is able combine
multiple Spark flows using the same schema.


This component does not support the streaming mode and thus cannot be used in a
streaming-enable Spark process. For further information about how to
use the streaming mode, see tSparkConnection.

It is strongly recommended to use this component in a Spark-only Job, that is to say, to
design and run a Spark Job separately from the non Spark components or Jobs. For example, it
is not recommended to use the tRunJob component to
coordinate a Spark Job and a non Spark Job, or to use the tHDFSPut component along with the Spark components in the same Job.

Related scenario

No scenario is available for this component yet.

Document get from Talend
Thank you for watching.
Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x