tAvroOutput – Docs for ESB 5.x

Component family

MapReduce

Function

tAvroOutput receives data flows
from the processing component placed ahead of it and writes the data
into Avro format files in a given distributed file system.

This component, along with the MapReduce family it belongs to, appears only when you are
creating a Map/Reduce Job.

Purpose

tAvroOutput creates Avro format
files in a given distributed file system.

Basic settings

Property type

Either Built-in or Repository.

Built-in: no property data stored
centrally.

Repository: reuse properties
stored centrally under the Hadoop
Cluster node of the Repository tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see Talend Big Data Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields to be processed and passed on
to the next component. The schema is either Built-In or
stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the
current schema is of the Repository type, three options are
available:

Built-In: You create and store the schema locally for this
component only. Related topic: see Talend Studio
User Guide.

Repository: You have already created the schema and
stored it in the Repository. You can reuse it in various projects and Job designs. Related
topic: see Talend Studio User Guide.

Folder

Browse to, or enter the directory in HDFS where the data you need to use is.

This path must point to a folder rather than a file, because a
Talend Map/Reduce
Job need to write in its target folder not only the final result but
also multiple part- files
generated in performing Map/Reduce computations.

Note that you need
to ensure you have properly configured the connection to the Hadoop
distribution to be used in the Hadoop
configuration tab in the Run view.

Action

Select an operation for writing data:

Create: Creates a file and write
data in it.

Overwrite: Overwrites the file
existing in the directory specified in the Folder field.

Compression

Select the Compress data blocks
check box to compress the output data.

Hadoop provides different compression formats that help reduce the space needed for
storing files and speed up data transfer. When reading a compressed file, the Studio needs
to uncompress it before being able to feed it to the input flow.

Advanced settings

tStatCatcher Statistics

Select this check box to collect log data at the component level.
Note that this check box is not available in the Map/Reduce version
of the component.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

In a Talend Map/Reduce Job, it is used as an end component and requires
a transformation component as input link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tAvroOutput as well as the MapReduce
family appears in the Palette of
the Studio.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional Talend data
integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.