tPigStoreResult – Docs for ESB 5.x

Component family

Big Data / Hadoop

Function

This component allows you to store the result of your Pig Job into
a defined data storage space.

Purpose

The tPigStoreResult component is
used to store the result into defined data storage space.

Basic settings

Property type

Either Repository or Built-in.

The Repository option allows you
to reuse the connection properties centrally stored under the
Hadoop cluster node of the
Repository tree. Once selecting
it, the button appears, then you can click it to
display the list of the stored properties and from that list, select
the properties you need to use. Once done, the appropriate
parameters are automatically set

Otherwise, if you select Built-in, you need to manually set each of the
parameters.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields to be processed and passed on
to the next component. The schema is either Built-In or
stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

Click Edit schema to make changes to the schema. If the
current schema is of the Repository type, three options are
available:

Built-In: You create and store the schema locally for this
component only. Related topic: see Talend Studio
User Guide.

Repository: You have already created the schema and
stored it in the Repository. You can reuse it in various projects and Job designs. Related
topic: see Talend Studio User Guide.

Use S3 endpoint

Select this check box to write data into a given Amazon S3 bucket
folder.

Once this Use S3 endpoint check box is selected, you need
to enter the following parameters in the fields that appear:

Note that the format of the S3 file is S3N (S3 Native Filesystem).

Result folder URI

Select the path to the result file in which data is stored.

Remove result directory if exists

Select this check box to remove an existing result directory.

Store function

Select a store function for data to be stored:

Note that when the file format to be used is PARQUET, you
might be prompted to find the specific Parquet jar file and install it into the Studio.

This jar file can be downloaded from Apache’s site. For further information
about how to install an external jar file, see https://help.talend.com/display/KB/How+to+install+external+modules+in+the+Talend+products.

HCataLog Configuration

Fill the following fields to configure HCataLog managed tables on
HDFS (Hadoop distributed file system):

Distribution and Version:

Select the Hadoop distribution to which you have defined the connection in the tPigLoad component, used in the same Pig process of the active
tPigStoreResult.

If that tPigLoad component connects to a custom Hadoop
distribution, you must select Custom for this tPigStoreResult component, too. Then the Custom jar table appears, in which, you need to add only the jar files
required by the selected Store function.

HCat metastore: Enter the
location of the HCatalog’s metastore, which is actually Hive’s
metastore.

Database: The database in which
tables are placed.

Table: The table in which data is
stored.

Partition filter: Fill this field
with the partition keys to list partitions by filter.

This area is available to the HBaseStorage function. The
parameters to be set are:

Distribution and Version:

Select the Hadoop distribution to which you have defined the connection in the tPigLoad component, used in the same Pig process of the active
tPigStoreResult.

If that tPigLoad component connects to a custom Hadoop
distribution, you must select Custom for this tPigStoreResult component, too. Then the Custom jar table appears, in which, you need to add only the jar files
required by the selected Store function.

Zookeeper quorum:

Type in the name or the URL of the Zookeeper service you use to coordinate the transaction
between Talend and HBase. Note that when you configure the Zookeeper, you
might need to set the zookeeper.znode.parent property to
define the root of the relative path of an HBase’s Zookeeper file; then select the Set Zookeeper znode parent check box to define this
property.

Zookeeper client port:

Type in the number of the client listening port of the Zookeeper service you are
using.

Table name:

Enter the name of the HBase table you need to store data in. The
table must exist in the target HBase.

Row key column:

Select the column used as the row key column of the HBase
table.

Store row key column to Hbase
column:

Select this check box to make the row key column an HBase column
belonging to a specific column family.

Mapping:

Complete this table to map the columns of the HBase table to be used with the schema
columns you have defined for the data flow to be processed.

The Column column of this table is automatically filled
once you have defined the schema; the syntax of the Column
family:qualifier column requires each HBase column name (qualifier) to be
paired with its corresponding family name, for example, in an HBase table, if a Paris column belongs to a France family, then you need to write it as France:Paris.

Enter character, string or regular expression to separate fields for the transferred
data.

Sequence Storage configuration

This area is available only to the SequenceFileStorage function. Since a SequenceFile
record consists of binary key/value pairs, the parameters to be set
are:

Key column:

Select the Key column of a key/value record.

Value column

Select the Value column of a key/value record.

Advanced settings

Register jar

Click the button to add rows to the table and from these rows, browse to the jar
files to be added. For example, in order to register a jar file called piggybank.jar, click the button once to add one row, then click this row to display the browse button, and click this button to browse to the piggybank.jar file following the [Select
Module] wizard.

Add and set more HBaseStorage storer options in this table. The
options are:

loadKey: enter true to store the
row key as the first column of the result schema, otherwise, enter
false;

gt: the minimum key value;

lt: the maximum key value;

gte: the minimum key value
(included);

lte: the maximum key value
(included);

limit: maxum number of rows to
retrieve per region;

caching: number of rows to
cache;

caster: the converter to use for
writing values to HBase. For example, Utf8StorageConverter.

HCatalog Configuration

Define the jars to register

This check box appears when you are using tHCatStorer, while by default, you can leave it
clear as the Studio registers the required jar files automatically.
In case any jar file is missing, you can select this check box to
display the Register jar for
HCatalog table and set the correct path to that
missing jar.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at the
Job level as well as at each component level.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is always used to end a Pig process and needs
tPigLoad at the beginning of
that chain to provide data

This component reuses automatically the connection created by the
tPigLoad component in that Pig
process.

Note that if you use Hortonworks Data
Platform V2.0.0, the type of the operating system for
running the distribution and a Talend Job must be the same, such as
Windows or Linux. Otherwise, you have to use Talend Jobserver to execute the Job in the
same type of operating system in which the Hortonworks Data Platform V2.0.0 distribution you
are using is run. For further information about Talend Jobserver, see Talend
Installation and Upgrade Guide.

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction
with Talend Studio. The following list presents MapR related information for
example.

For further information about how to install a Hadoop distribution, see the manuals
corresponding to the Hadoop distribution you are using.

Log4j

The activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User
Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

Knowledge of Pig scripts is required. If you select HCatStorer as the store function,
knowledge of HCataLog DDL(HCataLog Data Definition Language, a
subset of Hive Data Definition Language) is required. For further
information about HCataLog DDL, see https://cwiki.apache.org/confluence/display/Hive/HCatalog.