–driver

jdbc.driver.class

–direct-split-size

import.direct.split.size

–inline-lob-limit

import.max.inline.lob.size

–split-by

db.split.column

–warehouse-dir

hdfs.warehouse.dir

–enclosed-by

codegen.output.delimiters.enclose

–escaped-by

codegen.output.delimiters.escape

–fields-terminated-by

codegen.output.delimiters.field

–lines-terminated-by

codegen.output.delimiters.record

–optionally-enclosed-by

codegen.output.delimiters.required

–input-enclosed-by

codegen.input.delimiters.enclose

–input-escaped-by

codegen.input.delimiters.escape

–input-fields-terminated-by

codegen.input.delimiters.field

–input-lines-terminated-by

codegen.input.delimiters.record

–input-optionally-enclosed-by

codegen.input.delimiters.required

–hive-home

hive.home

–hive-import

hive.import

–hive-overwrite

hive.overwrite.table

–hive-table

hive.table.name

–class-name

codegen.java.classname

–jar-file

codegen.jar.file

–outdir

codegen.output.dir

–package-name

codegen.java.packagename

Mode

Select the mode in which Sqoop is called in a Job execution.

Use Commandline: the Sqoop shell is used to call Sqoop. In
this mode, you have to deploy and run the Job in the host where Sqoop is installed.
Therefore, if you are a subscription-based user, we recommend installing and using a
Jobserver provided by
Talend
in that host to run the Job; if you are using one of the

Talend
solutions
with Big Data, you have to ensure that the Studio and the Sqoop to be used are in the same
machine.

Use Java API: the Java API is used to call Sqoop. In this
mode, the Job can be run locally in the Studio but you need to configure the connection to
the Hadoop distribution to be used. Note that JDK is required to execute the Job in the Java
API mode and the versions of the JDK kits installed in both machines must be compatible with
each other; for example, the versions are the same or the JDK version of the Hadoop machine
is more recent.

Hadoop properties

Distribution

1. Select Import from existing
   version to import an officially supported distribution as base
   and then add other required jar files which the base distribution does not
   provide.

2. Select Import from zip to
   import the configuration zip for the custom distribution to be used. This zip
   file should contain the libraries of the different Hadoop elements and the index
   file of these libraries.

   In
   Talend
   
   Exchange, members of
   Talend
   community have shared some ready-for-use configuration zip files
   which you can download from this Hadoop configuration
   list and directly use them in your connection accordingly. However, because of
   the ongoing evolution of the different Hadoop-related projects, you might not be
   able to find the configuration zip corresponding to your distribution from this
   list; then it is recommended to use the Import from
   existing version option to take an existing distribution as base
   to add the jars required by your distribution.

   Note that custom versions are not officially supported by
   
   Talend
   .
   Talend
   and its community provide you with the opportunity to connect to
   custom versions from the Studio but cannot guarantee that the configuration of
   whichever version you choose will be easy, due to the wide range of different
   Hadoop distributions and versions that are available. As such, you should only
   attempt to set up such a connection if you have sufficient Hadoop experience to
   handle any issues on your own.

   Note:

   In this dialog box, the active check box must be kept
   selected so as to import the jar files pertinent to the connection to be
   created between the custom distribution and this component.

   For a step-by-step example about how to connect to a custom
   distribution and share this connection, see Connecting to a custom Hadoop distribution.

Hadoop Version

NameNode URI

Type in the URI of the Hadoop NameNode, the master node of a Hadoop system. For
example, we assume that you have chosen a machine called masternode as the NameNode, then the location is hdfs://masternode:portnumber. If you are using WebHDFS, the location should be
webhdfs://masternode:portnumber; if this WebHDFS is secured
with SSL, the scheme should be swebhdfs and you need to use
a tLibraryLoad in the Job to load the library required by
the secured WebHDFS.

JobTracker Host

Select this check box and in the displayed field, enter the location of the
ResourceManager of your distribution. For example, tal-qa114.talend.lan:8050.

This property is required when the query you want to use is executed in Windows
and it is a Select query. For example, SELECT your_column_name FROM your_table_name

For further information about the Hadoop Map/Reduce framework, see the
Map/Reduce tutorial in Apache’s Hadoop documentation on http://hadoop.apache.org.

Use kerberos authentication

In addition, since this component performs Map/Reduce computations, you
also need to authenticate the related services such as the Job history server and
the Resource manager or Jobtracker depending on your distribution in the
corresponding field. These principals can be found in the configuration files of
your distribution. For example, in a CDH4 distribution, the Resource manager
principal is set in the yarn-site.xml file and the Job history
principal in the mapred-site.xml file.

This check box is available depending on the Hadoop distribution you are
connecting to.

Select the Use a keytab to authenticate
check box to log into a Kerberos-enabled system using a given keytab file. A keytab
file contains pairs of Kerberos principals and encrypted keys. You need to enter the
principal to be used in the Principal field and
the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in
which your Job actually runs, for example, on a Talend Jobserver.

Note that the user that executes a keytab-enabled Job is not necessarily
the one a principal designates but must have the right to read the keytab file being
used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this
situation, ensure that user1 has the right to read the keytab
file to be used.

Hadoop user name

Enter the user name under which you want to execute the Job. Since a file or a directory in
Hadoop has its specific owner with appropriate read or write rights, this field allows you
to execute the Job directly under the user name that has the appropriate rights to access
the file or directory to be processed. Note that this field is available depending on the
distribution you are using.

JDBC property

Connection

Type in the JDBC URL used to connect to the target
database.

User name and
Password

Type in the authentication information to the target
database.

To enter the password, click the […] button next to the
password field, and then in the pop-up dialog box enter the password between double quotes
and click OK to save the settings.

Driver JAR

In either the Use Commandline mode or the Java API mode, you must add the driver file of the database to be
used to the lib folder of the Hadoop distribution you are
using. For that purpose, use this Driver JAR table to add
that driver file for the current Job you are designing.

Table Name

Type in the name of the target table to which data is transferred
from HDFS. This table must already exist in the target database. The
input files are read and parsed into a set of records according to
the user-specified delimiters.

Export Dir

Enter the path to the source data to be transferred in
HDFS.

Direct

Select this check box to use the export fast path.

Specify Number of Mappers

Select this check box to indicate the number of map tasks (parallel processes) used to
perform the data transfer.

If you do not want Sqoop to work in parallel, enter 1
in the displayed field.

Call a stored procedure

Select this check box to enable the component to call a specific
store procedure to write data into the target database.

You need to enter the name of the stored procedure to be used in
the field that is displayed.

Syntax and capabilities of stored procedures vary among different
databases; for this reason, you are recommended to consult the
documentation of the database you want to use for details about how
a given stored procedure could be called.

Use batch mode

Select this check box to execute the statements in batch mode
instead of running a multi-row INSERT statement to write multiple
records in the target database.

Clear staging table

If you are using a specific staging table for the desired data
transfer, select this check box to ensure that the staging table is
empty when the data transfer runs.

Define a staging table

Select this check box to create a staging table for the data to be
transferred. The transferred data is staged within this table before
being written into the target table so as to avoid only a part of
the data being committed to the target table when the transfer
fails.

For further information about whether a staging table is supported
for a given data transfer, see Apache’s documentation for
Sqoop.

Specify how updates are performed when new
rows are found with non-match keys in database

Use column for update

Select this check box and in the table that is displayed, add the
columns to be used as the update key.

Print Log

Select this check box to activate the Verbose check
box.

Verbose

Select this check box to print more information while working, for example, the debugging
information.

Use MySQL default delimiters

Select this check box to use MySQL’s default delimiter set. This check box is available
only to the Commandline mode.

Define Java mapping

Sqoop provides default configuration that maps most SQL types to appropriate Java types.
If you need to use your custom map to overwrite the default ones at runtime, select this
check box and define the map(s) you want to use in the table that appears.

Additional arguments

Complete this table to use additional arguments if needs be.

By adding additional arguments, you are able to perform multiple operations in one single
transaction. For example, you can use --hive-import and
--hive-table in the Commandline mode or hive.import and
hive.table.name in the Java API mode to create Hive table and write data in
at the runtime of the transaction writing data in HDFS. For further information about the
available Sqoop arguments in the Commandline mode and the Java API mode, respectively, see
Additional arguments.

Use speed parallel data transfers

Available in the Use Commandline
mode only.

Hadoop properties

Mapred job map memory mb and
Mapred job reduce memory
mb

If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks
Data Platform V1.3, you need to set proper memory allocations for the map and reduce
computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred
job map memory mb and the Mapred job reduce memory
mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the
computations.

If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and
ApplicationMaster (in Mb), accordingly. These fields
allow you to dynamically allocate memory to the map and the reduce computations and the
ApplicationMaster of YARN.

Path separator in server

Leave the default value of the Path separator in
server as it is, unless you have changed the separator used by your
Hadoop distribution’s host machine for its PATH variable or in other words, that
separator is not a colon (:). In that situation, you must change this value to the
one you are using in that host.

tStatCatcher Statistics

Select this check box to collect log data at the component
level.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

EXIT_CODE: the exit code of the remote command. This is
an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage rule

This component is used standalone. It respects the Sqoop prerequisites. You need necessary
knowledge about Sqoop to use it.

We recommend using the Sqoop of version 1.4+ in order to benefit the full functions of
these components.

For further information about Sqoop, see the Sqoop manual on: http://sqoop.apache.org/docs/

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction
with
Talend Studio
. The following list presents MapR related information for
example.

• Ensure that you have installed the MapR client in the machine where the Studio is,
  and added the MapR client library to the PATH variable of that machine. According
  to MapR’s documentation, the library or libraries of a MapR client corresponding to
  each OS version can be found under MAPR_INSTALL
  hadoophadoop-VERSIONlib
  ative. For example, the library for
  Windows is lib
  ativeMapRClient.dll in the MapR
  client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.

  Without adding the specified library or libraries, you may encounter the following
  error: no MapRClient in java.library.path.

• Set the -Djava.library.path argument, for example, in the Job Run VM arguments area
  of the Run/Debug view in the [Preferences] dialog box in the Window menu. This argument provides to the Studio the path to the
  native library of that MapR client. This allows the subscription-based users to make
  full use of the Data viewer to view locally in the
  Studio the data stored in MapR.

For further information about how to install a Hadoop distribution, see the manuals
corresponding to the Hadoop distribution you are using.

Limitation

If you have selected the Use Commandline mode, you need
to use the host where Sqoop is installed to run the Job using this component.

Connections

Outgoing links (from this component to another):

Trigger: Run if; On Subjob Ok; On
Subjob Error.

Incoming links (from one component to this one):

Row: Iterate;

Trigger: Run if; On Subjob Ok; On
Subjob Error; On Component Ok; On Component Error

For further information regarding connections, see

Talend Studio
User Guide.