tSqoopImport – Docs for ESB 5.x

Component family

Big Data / Sqoop

Function

tSqoopImport calls Sqoop to
transfer data from a relational database management system (RDBMS)
such as MySQL or Oracle into the Hadoop Distributed File System
(HDFS).

Purpose

tSqoopImport is used to define
the arguments required by Sqoop for writing the data of your
interest into HDFS.

Basic settings

Mode

Select the mode in which Sqoop is called in a Job execution.

Use Commandline: the Sqoop shell is used to call Sqoop.
In this mode, you have to deploy and run the Job in the host where Sqoop is installed.
Therefore, if you are a subscription-based user, we recommend installing and using a
Jobserver provided by Talend in that host to run the Job; if you are using one of the
Talend solutions
with Big Data, you have to ensure that the Studio and the Sqoop to be used are in the same
machine. For further information about how to install a Jobserver, see Talend Installation
and Upgrade Guide.

Use Java API: the Java API is used to call Sqoop. In this
mode, the Job can be run locally in the Studio but you need to configure the connection to
the Hadoop distribution to be used. Note that JDK is required to execute the Job in the Java
API mode and the versions of the JDK kits installed in both machines must be compatible with
each other; for example, the versions are the same or the JDK version of the Hadoop machine
is more recent.

Hadoop properties

Either Built-in or Repository:

Version

Distribution

Select the cluster you are using from the drop-down list. The options in the list vary
depending on the component you are using. Among these options, the following ones requires
specific configuration:

Hadoop Version

Select the version of the Hadoop distribution you are using. The available options vary
depending on the component you are using. Along with the evolution of Hadoop, please note
the following changes:

Configuration

NameNode URI

Select this check box to indicate the location of the NameNode of the Hadoop cluster to be
used. The NameNode is the master node of a Hadoop cluster. For example, we assume that you
have chosen a machine called masternode as the NameNode
of an Apache Hadoop distribution, then the location is hdfs://masternode:portnumber.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial
in Apache’s Hadoop documentation on http://hadoop.apache.org.

JobTracker Host

Select this check box to indicate the location of the Jobtracker service within the Hadoop
cluster to be used. For example, we assume that you have chosen a machine called machine1 as the JobTracker, then set its location as machine1:portnumber. A Jobtracker is the service that assigns
Map/Reduce tasks to specific nodes in a Hadoop cluster. Note that the notion job in this
term JobTracker does not designate a Talend Job, but rather a Hadoop job
described as MR or MapReduce job in Apache’s Hadoop documentation on http://hadoop.apache.org.

This property is required when the query you want to use is executed in Windows and it is
a Select query. For example, SELECT your_column_name FROM
your_table_name

If you use YARN in your Hadoop cluster such as Hortonworks Data
Platform V2.0.0 or Cloudera CDH4.3 + (YARN
mode), you need to specify the location of the Resource
Manager instead of the Jobtracker. Then you can continue to set the following
parameters depending on the configuration of the Hadoop cluster to be used (if you leave the
check box of a parameter clear, then at runtime, the configuration about this parameter in
the Hadoop cluster to be used will be ignored ):

For further information about these parameters, see the documentation or
contact the administrator of the Hadoop cluster to be used.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial
in Apache’s Hadoop documentation on http://hadoop.apache.org.

 Authentication

Use kerberos authentication

If you are accessing the Hadoop cluster running with Kerberos security, select this check
box, then, enter the Kerberos principal name for the NameNode in the field displayed. This
enables you to use your user name to authenticate against the credentials stored in
Kerberos.

In addition, since this component performs Map/Reduce computations, you also need to
authenticate the related services such as the Job history server and the Resource manager or
Jobtracker depending on your distribution in the corresponding field. These principals can
be found in the configuration files of your distribution. For example, in a CDH4
distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

This check box is available depending on the Hadoop distribution you are connecting
to.

Select the Use a keytab to authenticate check box to log
into a Kerberos-enabled Hadoop system using a given keytab file. A keytab file contains
pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used
in the Principal field and the access path to the keytab
file itself in the Keytab field.

Note that the user that executes a keytab-enabled Job is not necessarily the one a
principal designates but must have the right to read the keytab file being used. For
example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

Hadoop user name

Enter the user name under which you want to execute the Job. Since a file or a directory in
Hadoop has its specific owner with appropriate read or write rights, this field allows you
to execute the Job directly under the user name that has the appropriate rights to access
the file or directory to be processed. Note that this field is available depending on the
distribution you are using.

JDBC property

Either Built-in or Repository:

Connection

Enter the JDBC URL used to connect to the database where the source data is stored.

User name and
Password

Enter the authentication information used to connect to the source database.

To enter the password, click the […] button next to the
password field, and then in the pop-up dialog box enter the password between double quotes
and click OK to save the settings.

If your password is stored in a file, select the The password is stored
in a file check box and enter the path to that file in the File path field that is displayed.

Note that this feature is available depending on the Sqoop version you are
using.

Driver JAR

In either the Use Commandline mode or the Java API mode, you must add the driver file of the database to be
used to the lib folder of the Hadoop distribution you are
using. For that purpose, use this Driver JAR table to add
that driver file for the current Job you are designing.

Table Name

Type in the name of the table to be transferred into HDFS.

This field is not available when you are using the free-form query
mode by selecting the Use query
check box.

File format

Select a file format for the data to be transferred. By default, the file format is
textfile, but you can as well choose the sequencefile or the Avro file
format instead.

Delete target directory

Select this check box to remove the target directory of the
transfer.

Append

Select this check box to append transferred data to an existing
dataset in HDFS.

Compress

Select this check box to enable compression.

Direct

Select this check box to use the import fast path.

Specify columns

Select this check box to display the column table where you can
specify the columns you want to transfer into HDFS.

Use WHERE clause

Select this check box to use a WHERE clause that controls the rows
to be transferred. In the field displayed, you can type in the
condition used to select the rows you want. For example, type in
id >400 to import only the rows where
the id column has a value greater than
400.

Query

Use query

Select this check box to use the free-form query mode provided by
Sqoop.

Once selecting it, you are able to enter the free-form query you
need to use.

Then, you must specify the target directory and if the Sqoop
imports data in parallel, specify as well the Split by
argument.

Specify Target Dir

Select this check box to enter the path to the target location, in
HDFS, where you want to transfer the source data to.

This location should be a new directory; otherwise, you must
select the Append check box.

Specify Split by

Select this check box, then, enter the table column you need and
are able to use as the splitting column to split the
workload.

For example, for a table where the id column is the key column, enter tablename.id. Then Sqoop will split the
data to be transferred according to their ID values and imports them
in parallel.

Specify Number of Mappers

Select this check box to indicate the number of map tasks (parallel processes) used to
perform the data transfer.

If you do not want Sqoop to work in parallel, enter 1
in the displayed field.

Print Log

Select this check box to activate the Verbose check
box.

Verbose

Select this check box to print more information while working, for example, the debugging
information.

Advanced settings

Use MySQL default delimiters

Select this check box to use MySQL’s default delimiter set. This check box is available
only to the Commandline mode.

Define Java mapping

Sqoop provides default configuration that maps most SQL types to appropriate Java types.
If you need to use your custom map to overwrite the default ones at runtime, select this
check box and define the map(s) you want to use in the table that appears.

Define Hive mapping

Sqoop provides default configuration that maps most SQL types to appropriate Hive types.
If you need to use your custom map to overwrite the default ones at runtime, select this
check box and define the map(s) you want to use in the table that appears.

Additional arguments

Complete this table to use additional arguments if needs be.

By adding additional arguments, you are able to perform multiple operations in one single
transaction. For example, you can use --hive-import and
--hive-table in the Commandline mode or hive.import and
hive.table.name in the Java API mode to create Hive table and write data in
at the runtime of the transaction writing data in HDFS. For further information about the
available Sqoop arguments in the Commandline mode and the Java API mode, respectively, see
Additional arguments.

Connector specific
configuration 

Use speed parallel data transfers

Select this check box to enable quick parallel data transfers between the Teradata
database and the Hortonworks Hadoop distribution. Then the Specific
params table and the Use additional params
check box appear to allow you to specify the Teradata parameters required by parallel transfers.

You must ensure that the Hortonworks Connector for Teradata has been
installed in your Hortonworks cluster. The latest connector can be downloaded from the
website of Hortonworks and installed by following the explanations from http://hortonworks.com/wp-content/uploads/2014/02/bk_HortonworksConnectorForTeradata.pdf.
In the same document, you can as well find the detailed explanations about each parameter
that is available for the parallel transfer purpose.

Available in the Use Commandline
mode only.

Hadoop properties

Talend Studio uses a default configuration for its engine to perform
operations in a Hadoop distribution. If you need to use a custom configuration in a specific
situation, complete this table with the property or properties to be customized. Then at
runtime, the customized property or properties will override those default ones.

For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you
are using or see Apache’s Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:

Mapred job map memory mb and
Mapred job reduce memory
mb

If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks
Data Platform V1.3, you need to set proper memory allocations for the map and reduce
computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred
job map memory mb and the Mapred job reduce memory
mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the
computations.

If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and
ApplicationMaster (in Mb), accordingly. These fields
allow you to dynamically allocate memory to the map and the reduce computations and the
ApplicationMaster of YARN.

Path separator in server

Leave the default value of the Path separator in server as
it is, unless you have changed the separator used by your Hadoop distribution’s host machine
for its PATH variable or in other words, that separator is not a colon (:). In that
situation, you must change this value to the one you are using in that host.

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

EXIT_CODE: the exit code of the remote command. This is
an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is used standalone. It respects the Sqoop prerequisites. You need necessary
knowledge about Sqoop to use it.

We recommend using the Sqoop of version 1.4+ in order to benefit the full functions of
these components.

For further information about Sqoop, see the Sqoop manual on: http://sqoop.apache.org/docs/

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction
with Talend Studio. The following list presents MapR related information for
example.

For further information about how to install a Hadoop distribution, see the manuals
corresponding to the Hadoop distribution you are using.

Log4j

The activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User
Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

If you have selected the Use Commandline mode, you need
to use the host where Sqoop is installed to run the Job using this component.

Connections

Outgoing links (from this component to another):

Trigger: Run if; On Subjob Ok; On
Subjob Error.

Incoming links (from one component to this one):

Row: Iterate;

Trigger: Run if; On Subjob Ok; On
Subjob Error; On Component Ok; On Component Error

For further information regarding connections, see Talend Studio User Guide.