Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

Note that you need
to ensure you have properly configured the connection to the Hadoop
distribution to be used in the Hadoop
configuration tab in the Run view.

Action

Select an operation for writing data:

Create: Creates a file and write
data in it.

Overwrite: Overwrites the file
existing in the directory specified in the Folder field.

Compression

By default, the Uncompressed
option is active. But you can select the Gzip or the Snappy
option to compress the output data.

Hadoop provides different compression formats that help reduce the space needed for
storing files and speed up data transfer. When reading a compressed file, the Studio needs
to uncompress it before being able to feed it to the input flow.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage rule

In a
Talend
Map/Reduce Job, it is used as an end component and requires
a transformation component as input link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tParquetOutput as well as the MapReduce
family appears in the Palette of
the Studio.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
data integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

Define a storage configuration component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

This component does not support the Object type and the
List type.

Spark automatically infers
data types for the columns in a PARQUET schema. In a Talend Job for Apache Spark, the Date type is inferred
and stored as int96.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Action

Select an operation for writing data:

Create: Creates a file and write data in it.

Overwrite: Overwrites the file existing in the directory
specified in the Folder field.

Compression

By default, the Uncompressed option is active. But you
can select the Gzip or the Snappy
option to compress the output data.

Select the check box to output dates, hours, minutes and seconds contained in your
Date-type data. If you clear this check box, only years, months and days are
outputted.

The format used by Deltalake is yyyy-MM-dd HH:mm:ss.

Usage rule

This component is used as an end component and requires an input link.

This component, along with the Spark Batch component Palette it belongs to,
appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a
scenario presents only Standard Jobs, that is to
say traditional
Talend
data integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.

Define a storage configuration component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

This component does not support the Object type and the
List type.

Spark automatically infers
data types for the columns in a PARQUET schema. In a Talend Job for Apache Spark, the Date type is inferred
and stored as int96.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Action

Select an operation for writing data:

Create: Creates a file and write data in it.

Overwrite: Overwrites the file existing in the directory
specified in the Folder field.

Compression

By default, the Uncompressed option is active. But you
can select the Gzip or the Snappy
option to compress the output data.

For further information about when this is desirable
behavior, see this discussion.

Select the check box to output dates, hours, minutes and seconds contained in your
Date-type data. If you clear this check box, only years, months and days are
outputted.

The format used by Deltalake is yyyy-MM-dd HH:mm:ss.

Usage rule

This component is used as an end component and requires an input link.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
data
integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.