tSparkStore – Docs for ESB 5.x

Component family

Big Data / Spark

Function

tSparkStore uses the Spark
connection created by a given tSparkConnection component and writes the datasets
it receives from its preceding Spark component into a specific
target, such as an HDFS system.

Purpose

tSparkStore ends the Spark
process you are designing and writes the processed datasets into a
specific file system.

Basic settings

Spark connection

Select the Spark connection component to be used from the drop-down list in order to reuse
the connection created by that component.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields to be processed and passed on
to the next component. The schema is either Built-In or
stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the
current schema is of the Repository type, three options are
available:

Storage target

Select the type of the target system you write the processed data
in.

Result folder URI

Enter the directory in which you need to write the data in the
target system. Along with this location parameter, you need to set
the following parameter about the target data:

Note that this file system cannot be the Windows system.

Remove result directory if exists

When the Storage target to be
used is Local, you can select this
check box to remove the folder for the result if it already
exists.

Advanced settings

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is the end component of a Spark process.

Limitations

It is strongly recommended to use this component in a Spark-only Job, that is to say, to
design and run a Spark Job separately from the non Spark components or Jobs. For example, it
is not recommended to use the tRunJob component to
coordinate a Spark Job and a non Spark Job, or to use the tHDFSPut component along with the Spark components in the same Job.