Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

Select this check box to reuse an existing dynamic schema to
handle data from unknown columns.

When this check box is selected, a Component
list appears allowing you to select the component
used to set the dynamic schema.

Use Output Stream

Select this check box process the data flow of interest. Once you
have selected it, the Output Stream
field displays and you can type in the data flow of interest.

The data flow to be processed must be added to the flow in order
for this component to fetch these data via the corresponding
representative variable.

This variable could be already pre-defined in your Studio or
provided by the context or the components you are using along with
this component; otherwise, you could define it manually and use it
according to the design of your Job, for example, using tJava or tJavaFlex.

In order to avoid the inconvenience of hand writing, you could
select the variable of interest from the auto-completion list
(Ctrl+Space) to fill the
current field on condition that this variable has been properly
defined.

For further information about how to use a stream, see Reading data from a remote file in streaming mode.

File Name

Name or path to the file to be processed and or the variable to be
used.

This field becomes unavailable once you have selected the
Use Output Stream check
box.

For further information about how to define and use a variable in
a Job, see
Talend Studio

User Guide.

Schema
and
Edit Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

This
component offers the advantage of the dynamic schema feature. This allows you to
retrieve unknown columns from source files or to copy batches of columns from a source
without mapping each column individually. For further information about dynamic schemas,
see
Talend Studio

User Guide.

This
dynamic schema feature is designed for the purpose of retrieving unknown columns of a
table and is recommended to be used for this purpose only; it is not recommended for the
use of creating tables.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Row separator

The separator used to identify the end of a row.

Append

Select this check box to add the new rows at the end of the
file.

Include header

Select this check box to include the column header to the
file.

Compress as zip file

Select this check box to compress the output file in zip
format.

Formats

Customize the positional file data format and fill in the columns
in the Formats table.

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Type in between
quotes the padding characters used. A space by default.

Alignment: Select the appropriate
alignment parameter.

Keep: If the data in the column or
in the field are too long, select the part you want to keep.

Advanced separator (for numbers)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Thousands separator: define
separators for thousands.

Decimal separator: define
separators for decimals.

Use byte length as the cardinality

Select this check box to add support of double-byte character to
this component. JDK 1.6 is required for this feature.

Create directory if not exists

This check box is selected by default. It creates a directory to
hold the output table if it does not exist.

Custom the flush buffer size

Select this check box to define the number of lines to write
before emptying the buffer.

Row Number: set the number of
lines to write.

Output in row mode

Writes in row mode.

Encoding

Select the encoding from the list or select Custom
and define it manually. This field is compulsory for database data handling. The
supported encodings depend on the JVM that you are using. For more information, see
https://docs.oracle.com.

Don’t generate empty file

Select this check box if you do not want to generate empty
files.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at a
Job level as well as at each component level.

Global Variables

NB_LINE: the number of rows read by an input component or
transferred to an output component. This is an After variable and it returns an
integer.

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage rule

Use this component to read a file and separate the fields using
the specified separator.

Dynamic settings

Click the [+] button to add a row in the
table and fill the Code field with a context variable
to choose your HDFS connection dynamically from multiple connections planned in your
Job. This feature is useful when you need to access files in different HDFS systems or
different distributions, especially when you are working in an environment where you
cannot change your Job settings, for example, when your Job has to be deployed and
executed independent of
Talend Studio
.

The Dynamic settings table is
available only when the Use an existing
connection check box is selected in the Basic settings view. Once a dynamic parameter is
defined, the Component List box in the
Basic settings view becomes unusable.

For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic
settings and context variables, see Talend Studio
User Guide.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Click this icon to open a database connection wizard and store the
database connection parameters you set in the component Basic
settings view.

For more information about setting up and storing database
connection parameters, see Talend Studio User Guide.

Schema and Edit Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder

Browse to, or enter the path pointing to the data to be used in the file system.

This path must point to a folder rather than a file, because a

Talend
Map/Reduce Job need to write in its target folder not only the
final result but also multiple part- files generated in
performing Map/Reduce computations.

Note that you need
to ensure you have properly configured the connection to the Hadoop
distribution to be used in the Hadoop
configuration tab in the Run view.

Action

Select an operation for writing data:

Create: Creates a file and write data in
it.

Overwrite: Overwrites the file existing
in the directory specified in the Folder
field.

Compress the data

Select the Compress the data check box to compress the
output data.

Hadoop provides different compression formats that help reduce the space needed for
storing files and speed up data transfer. When reading a compressed file, the Studio needs
to uncompress it before being able to feed it to the input flow.

Formats

Customize the positional file data format and fill in the
columns in the Formats table.

Column: Select the column you want to
customize.

Size: Enter the column size.

Padding char: Type in between quotes the
padding characters used. A space by default.

Alignment: Select the appropriate
alignment parameter.

Keep: If the data in the column or in
the field are too long, select the part you want to keep.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage rule

In a
Talend
Map/Reduce Job, it is used as an end component and requires
a transformation component as input link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tFileOutputPositional as well as the MapReduce
family appears in the Palette of the
Studio.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
data integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Click this icon to open a database connection wizard and store the
database connection parameters you set in the component Basic
settings view.

For more information about setting up and storing database
connection parameters, see Talend Studio User Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder

Browse to, or enter the path pointing to the data to be used in the file system.

This path must point to a folder rather than a file.

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Action

Select an operation for writing data:

Create: Creates a file and write
data in it.

Overwrite: Overwrites the file
existing in the directory specified in the Folder field.

Compress the data

Select the Compress the data check box to compress the
output data.

Row separator

The separator used to identify the end of a row.

Include header

Select this check box to include the column header to the
file.

Custom encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Merge result to single file

Select this check box to merge the final part files into a single file and put that file in a
specified directory.

Once selecting it, you need to enter the path to, or browse to the
folder you want to store the merged file in. This directory is
automatically created if it does not exist.

This option is not available for a Sequence file.

Formats

Customize the positional file data format and fill in the columns
in the Formats table.

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Type in between
quotes the padding characters used. A space by default.

Alignment: Select the appropriate
alignment parameter.

Keep: If the data in the column or
in the field are too long, select the part you want to keep.

Advanced separator (for numbers)

Select this check box to modify the separators used for
numbers:

Thousands separator: define separators
for thousands.

Decimal separator: define separators for
decimals.

Usage rule

This component is used as an end component and requires an input link.

This component, along with the Spark Batch component Palette it belongs to,
appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a
scenario presents only Standard Jobs, that is to
say traditional
Talend
data integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job.
For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Click this icon to open a database connection wizard and store the
database connection parameters you set in the component Basic
settings view.

For more information about setting up and storing database
connection parameters, see Talend Studio User Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this
  option to view the schema only.

• Change to built-in property:
  choose this option to change the schema to Built-in for local changes.

• Update repository connection:
  choose this option to change the schema stored in the repository and decide whether
  to propagate the changes to all the Jobs upon completion. If you just want to
  propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
  again in the Repository Content
  window.

Built-In: You create and store the schema locally for this component
only.

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Folder

Browse to, or enter the path pointing to the data to be used in the file system.

This path must point to a folder rather than a file.

The button for browsing does not work with the Spark
Local mode; if you are
using the other Spark Yarn
modes that the Studio supports with your distribution, ensure that you have properly
configured the connection in a configuration component in the same Job, such as

tHDFSConfiguration
. Use the
configuration component depending on the filesystem to be used.

Action

Select an operation for writing data:

Create: Creates a file and write data
in it.

Overwrite: Overwrites the file
existing in the directory specified in the Folder field.

Compress the data

Select the Compress the data check box to compress the
output data.

Hadoop provides different compression formats that help reduce the space needed for
storing files and speed up data transfer. When reading a compressed file, the Studio needs
to uncompress it before being able to feed it to the input flow.

Row separator

The separator used to identify the end of a row.

Include header

Select this check box to include the column header to the file.

Custom encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Formats

Customize the data format of the positional file for each column
from the input schema.

• Column: Select the column you
  want to customize.

• Size: Enter the column size.

• Padding char: Type in between
  quotes the padding characters used. A space by default.

• Alignment: Select the appropriate
  alignment parameter.

• Keep: If the data in the column
  or in the field are too long, select the part you want to keep.

Advanced separator (for numbers)

Select this check box to modify the separators used for
numbers:

Thousands separator: define
separators for thousands.

Decimal separator: define
separators for decimals.

For further information about when this is desirable
behavior, see this discussion.

Usage rule

This component is used as an end component and requires an input link.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
data
integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.