July 30, 2023

tHBaseLookupInput – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tHBaseLookupInput – Docs for ESB 7.x

tHBaseLookupInput

Provides lookup data to the main flow of a streaming Job.

tHBaseLookupInput extracts columns
corresponding to schema definition. Then it passes the extracted data to
the next component.

It passes on the extracted data to tMap in order to
provide the lookup data to the main flow. It must be directly connected to a tMap component and requires this tMap to use Reload at each row or Reload at each row (cache) for the lookup flow.

HBase filters

This table presents the HBase filters available in
Talend Studio
 and the
parameters required by those filters.

Filter type

Filter column

 Filter family Filter operation Filter value Filter comparator type Objective

Single Column Value Filter

Yes

Yes

Yes

Yes

Yes

It compares the values of a given column against the value defined
for the Filter value parameter. If
the filtering condition is met, all columns of the row will be
returned.

Family filter

 

Yes

Yes

  

Yes

It returns the columns of the family that meets the filtering
condition.

Qualifier filter

Yes

  

Yes

  

Yes

It returns the columns whose column qualifiers match the filtering
condition.

Column prefix filter

Yes

Yes

      

It returns all columns of which the qualifiers have the prefix
defined for the Filter column parameter.

Multiple column prefix filter

Yes (Multiple prefixes are separated by comma, for
example, id,id_1,id_2)

Yes

      

It works the same way as a Column prefix
filter does but allows specifying multiple
prefixes.

Column range filter

Yes (The ends of a range are separated by comma. )

Yes

       It allows intra row scanning and returns all matching columns of a
scanned row.

Row filter

   

Yes

Yes

 Yes It filters on row keys and returns all rows that matches the
filtering condition.

Value filter

   

Yes

Yes

Yes

 It returns only columns that have a specific value.

The use explained above of the listed HBase filters is subject to revisions made by
Apache in its Apache HBase project; therefore, in order to fully understand how to use
these HBase filters, we recommend reading Apache’s HBase documentation.

tHBaseLookupInput properties for Apache Spark Streaming

These properties are used to configure tHBaseLookupInput running in the Spark Streaming Job framework.

The Spark Streaming
tHBaseLookupInput component belongs to the Databases family.

The streaming version of this component is available in Talend Real Time Big Data Platform and in
Talend Data Fabric.

Basic settings

Storage configuration

Select the tHBaseConfiguration component from which the
Spark system to be used reads the configuration information to connect to HBase.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the
properties are stored.

tHBaseLookupInput_1.png

Click this icon to open a database connection wizard and store the
database connection parameters you set in the component Basic
settings view.

For more information about setting up and storing database
connection parameters, see Talend Studio User Guide.

Schema et Edit schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Table name

Type in the name of the table from which you need to extract columns.

Define a row selection

Select this check box and then in the Start row and the
End row fields, enter the corresponding row keys to
specify the range of the rows you want the current component to extract.

Different from the filters you can set using Is by
filter requiring the loading of all records before filtering the ones to be
used, this feature allows you to directly select only the rows to be used.
Mapping

Complete this table to map the columns of the table to be used with the schema columns you
have defined for the data flow to be processed.

Is by filter

Select this check box to use filters to perform fine-grained data selection from your
database, such as selection of keys, or values, based on regular expressions.

Once selecting it, the Filter table that is used to
define filtering conditions becomes available.

This feature leverages filters provided by HBase and subject to constraints explained in
Apache HBase documentation. Therefore, advanced knowledge of HBase is required to make full
use of these filters.

Logical operation
Select the operator you need to use to define the logical relation between filters. This
available operators are:

  • And: every defined filtering conditions must
    be satisfied. It represents the relationship
    FilterList.Operator.MUST_PASS_ALL

  • Or: at least one of the defined filtering
    conditions must be satisfied. It represents the relationship:
    FilterList.Operator.MUST_PASS_ONE

Filter
Click the button under this table to add as many rows as required, each row representing a
filter. The parameters you may need to set for a filter are:

  • Filter type: the drop-down list presents
    pre-existing filter types that are already defined by HBase. Select the type of
    the filter you need to use.

  • Filter column: enter the column qualifier on
    which you need to apply the active filter. This parameter becomes mandatory
    depending on the type of the filter and of the comparator you are using. For
    example, it is not used by the Row Filter type
    but is required by the Single Column Value
    Filter     type.

  • Filter family: enter the column family on
    which you need to apply the active filter. This parameter becomes mandatory
    depending on the type of the filter and of the comparator you are using. For
    example, it is not used by the Row Filter type
    but is required by the Single Column Value
    Filter     type.

  • Filter operation: select from the drop-down
    list the operation to be used for the active filter.

  • Filter Value: enter the value on which you
    want to use the operator selected from the Filter
    operation     drop-down list.

  • Filter comparator type: select the type of
    the comparator to be combined with the filter you are using.

Depending on the Filter type you are using,
some or each of the parameters become mandatory. For further information, see HBase filters

Usage

Usage rule

This component is used as a start component and requires an output link.

This component uses a tHBaseConfiguration component present in the same Job to connect to
HBase.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
 data
integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

  • Yarn mode (Yarn client or Yarn cluster):

    • When using Google Dataproc, specify a bucket in the
      Google Storage staging bucket
      field in the Spark configuration
      tab.

    • When using HDInsight, specify the blob to be used for Job
      deployment in the Windows Azure Storage
      configuration       area in the Spark
      configuration       tab.

    • When using Altus, specify the S3 bucket or the Azure
      Data Lake Storage for Job deployment in the Spark
      configuration       tab.
    • When using Qubole, add a
      tS3Configuration to your Job to write
      your actual business data in the S3 system with Qubole. Without
      tS3Configuration, this business data is
      written in the Qubole HDFS system and destroyed once you shut
      down your cluster.

    • When using on-premise
      distributions, use the configuration component corresponding
      to the file system your cluster is using. Typically, this
      system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the
    configuration component corresponding to the file system your cluster is
    using, such as tHDFSConfiguration or
    tS3Configuration.

    If you are using Databricks without any configuration component present
    in your Job, your business data is written directly in DBFS (Databricks
    Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

For a scenario about how to use the same type of component in a Spark Streaming Job, see
Reading and writing data in MongoDB using a Spark Streaming Job.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x