Schema and Edit
schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Note that the schema of this component is read-only. It stores the
messages to be published.

Use an existing connection

Select this check box and in the Component List click the relevant connection component to
reuse the connection details you already defined.

Version

Select the version of the Kafka cluster to be used.

Broker list

Enter the addresses of the broker nodes of the Kafka cluster to be used.

The form of this address should be hostname:port. This
information is the name and the port of the hosting node in this Kafka cluster.

If you need to specify several addresses, separate them using a comma (,).

Enter the name of the topic you want to publish messages to. This topic must already
exist.

Compress the data

Select the Compress the data check box to compress the
output data.

Use SSL/TLS

Select this check box to enable the SSL or TLS encrypted connection.

Then you need to use the tSetKeystore
component in the same Job to specify the encryption information.

This check box is available since Kafka 0.9.0.1.

Use Kerberos authentication

If the Kafka cluster to be used is secured with Kerberos, select this
check box to display the related parameters to be defined:

• JAAS configuration path: enter the
  path, or browse to the JAAS configuration file to be used by the Job to
  authenticate as a client to Kafka.

  This JAAS file describes how the clients, the Kafka-related
  Jobs in terms of
  Talend
  , can connect to the Kafka broker nodes, using either the kinit mode or
  the keytab mode. It must be stored in the machine where these Jobs are
  executed.

  
  Talend
  , Kerberos or Kafka does not provide this JAAS file. You need to create
  it by following the explanation in Configuring Kafka
  client depending on the security strategy of your
  organization.

• Kafka brokers principal name: enter
  the primary part of the Kerberos principal you defined for the brokers when
  you were creating the broker cluster. For example, in this principal kafka/kafka1.hostname.com@EXAMPLE.COM, the primary
  part to be used to fill in this field is kafka.

• Set kinit command path: Kerberos
  uses a default path to its kinit executable. If you have changed this path,
  select this check box and enter the custom access path.

  If you leave this check box clear, the default path is
  used.

• Set Kerberos configuration path:
  Kerberos uses a default path to its configuration file, the krb5.conf file (or krb5.ini
  in Windows) for Kerberos 5 for example. If you have changed this path,
  select this check box and enter the custom access path to the Kerberos
  configuration file.

  If you leave this check box clear, a given strategy is applied
  by Kerberos to attempt to find the configuration information it requires.
  For details about this strategy, see the Locating the
  krb5.conf Configuration File section in Kerberos
  requirements.

For further information about how a Kafka cluster is secured with
Kerberos, see Authenticating using
SASL.

This check box is available since Kafka 0.9.0.1.

Kafka properties

Add the Kafka new producer properties you need to customize to this table.

For further information about the new producer properties you can define in this table,
see the section describing the new producer configuration in Kafka’s documentation in http://kafka.apache.org/documentation.html#newproducerconfigs.

Select this check box to add headers to messages to be
sent.

This feature is available to Kafka 1.1.0 onwards.

tStatCatcher Statistics

Select this check box to gather the processing metadata at the Job
level as well as at each component level.

Usage rule

This component is an end component. It requires a tJavaRow or tJava component to transform the incoming data into
serialized byte arrays.

The following sample shows how to construct a statement to perform
this transformation:

In this code, the output_row
variable represents the schema of the data to be output to tKafkaOutput and output_row.serializedValue the single read-only column
of that schema; the input_row
variable represents the schema of the incoming data and input_row.users the input column called
users to be transformed to byte
arrays by the getBytes()
method.

Schema and Edit
schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Note that the schema of this component is read-only. It stores the
messages to be published.

Broker list

Enter the addresses of the broker nodes of the Kafka cluster to be used.

The form of this address should be hostname:port. This
information is the name and the port of the hosting node in this Kafka cluster.

If you need to specify several addresses, separate them using a comma (,).

Enter the name of the topic you want to publish messages to. This topic must already
exist.

Compress the data

Select the Compress the data check box to compress the
output data.

Kafka properties

Add the Kafka new producer properties you need to customize to this table.

For further information about the new producer properties you can define in this table,
see the section describing the new producer configuration in Kafka’s documentation in http://kafka.apache.org/documentation.html#newproducerconfigs.

Connection pool

In this area, you configure, for each Spark executor, the connection pool used to control
the number of connections that stay open simultaneously. The default values given to the
following connection pool parameters are good enough for most use cases.

• Max total number of connections: enter the maximum number
  of connections (idle or active) that are allowed to stay open simultaneously.

  The default number is 8. If you enter -1, you allow unlimited number of open connections at the same
  time.

• Max waiting time (ms): enter the maximum amount of time
  at the end of which the response to a demand for using a connection should be returned by
  the connection pool. By default, it is -1, that is to say, infinite.

• Min number of idle connections: enter the minimum number
  of idle connections (connections not used) maintained in the connection pool.

• Max number of idle connections: enter the maximum number
  of idle connections (connections not used) maintained in the connection pool.

Evict connections

Select this check box to define criteria to destroy connections in the connection pool. The
following fields are displayed once you have selected it.

• Time between two eviction runs: enter the time interval
  (in milliseconds) at the end of which the component checks the status of the connections and
  destroys the idle ones.

• Min idle time for a connection to be eligible to
  eviction: enter the time interval (in milliseconds) at the end of which the idle
  connections are destroyed.

• Soft min idle time for a connection to be eligible to
  eviction: this parameter works the same way as Min idle
  time for a connection to be eligible to eviction but it keeps the minimum number
  of idle connections, the number you define in the Min number of idle
  connections field.

Usage rule

This component is used as an end component and requires an input link.

This component needs a Write component such as tWriteJSONField to define a serializedValue column in the input schema to send serialized data.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
data
integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.