July 30, 2023

tExtractRegexFields – Docs for ESB 7.x

  2. Talend for ESB Docs
  3. Docs 7.x
  4. tExtractRegexFields – Docs for ESB 7.x

tExtractRegexFields

Extracts data and generates multiple columns from a formatted string using regex
matching.

Depending on the Talend
product you are using, this component can be used in one, some or all of the following
Job frameworks:

tExtractRegexFields Standard properties

These properties are used to configure tExtractRegexFields running in the Standard Job framework.

The Standard
tExtractRegexFields component belongs to the Data Quality and the Processing families.

The component in this framework is available in all Talend
products.

Basic settings

Field to split

Select an incoming field from the Field to
split list to split.

Regex

Enter a regular expression according to the programming language
you are using.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

Click Sync columns to retrieve
the schema from the previous component connected in the Job.

Warning:

Make sure that the output schema does not contain any column
with the same name as the input column to be split. Otherwise,
the regular expression will not work as expected.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Advanced settings

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Check each row structure against
schema

Select this check box to check whether the total number of columns
in each row is consistent with the schema. If not consistent, an
error message will be displayed on the console.

tStatCatcher Statistics

Select this check box to gather the processing metadata at the Job
level as well as at each component level.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

This component handles flow of data therefore it requires input
and output components. It allows you to extract data from a
delimited field, using a Row >
Main link, and enables you to
create a reject flow filtering data which type does not match the
defined type.

Extracting name, domain and TLD from e-mail addresses

This scenario describes a three-component Job where tExtractRegexFields is used to specify a regular
expression that corresponds to one column in the input data, email. The tExtractRegexFields component is
used to perform the actual regular expression matching. This regular expression includes
field identifiers for user name, domain name and Top-Level Domain (TLD) name portions in
each e-mail address. If the given e-mail address is valid, the name, domain and TLD are
extracted and displayed on the console in three separate columns. Data in the other two
input columns, id and age is extracted and
routed to destination as well.

tExtractRegexFields_1.png

Setting up the Job

  1. Drop the following components from the Palette onto the design workspace: tFileInputDelimited, tExtractRegexFields, and tLogRow.
  2. Connect tFileInputDelimited to tExtractRegexFields using a Row > Main link, and do
    the same to connect tExtractRegexFields to
    tLogRow.

Configuring the components

  1. Double-click the tFileInputDelimited
    component to open its Basic settings view
    in the Component tab.

    tExtractRegexFields_2.png

  2. Click the […] button next to the
    File name/Stream field to browse to the
    file where you want to extract information from.

    The input file used in this scenario is called test4.
    It is a text file that holds three columns: id,
    email, and
    age.
    For more information, see tFileInputDelimited.
  3. Click Edit schema to define the data
    structure of this input file.
  4. Double-click the tExtractRegexFields
    component to open its Basic settings
    view.

    tExtractRegexFields_3.png

  5. Select the column to split from the Field to
    split     list: email in this
    scenario.
  6. Enter the regular expression you want to use to perform data matching in
    the Regex panel. In this scenario, the
    regular expression "([a-z]*)@([a-z]*).([a-z]*)" is used to
    match the three parts of an email address: user name, domain name and TLD
    name.

    For more information about the regular expression, see http://en.wikipedia.org/wiki/Regular_expression.
  7. Click Edit schema to open the Schema of tExtractRegexFields dialog box, and
    click the plus button to add five columns for the output schema.

    In this scenario, we want to split the input email
    column into three columns in the output flow, name,
    domain,     and tld. The two other input
    columns will be extracted as they are.
    tExtractRegexFields_4.png

  8. Double-click the tLogRow component to
    open its Component view.
  9. In the Mode area, select Table (print values in cells of a table).

Saving and executing the Job

  1. Press Ctrl+S to save your Job.
  2. Execute the Job by pressing F6 or
    clicking Run on the Run tab.

    tExtractRegexFields_5.png

The tExtractRegexFields component matches all
given e-mail addresses with the defined regular expression and extracts the name,
domain, and TLD names and displays them on the console in three separate columns.
The two other columns, id and age, are
extracted as they are.

tExtractRegexFields MapReduce properties (deprecated)

These properties are used to configure tExtractRegexFields running in the MapReduce Job framework.

The MapReduce
tExtractRegexFields component belongs to the Processing family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.

Basic settings

Field to split

Select an incoming field from the Field to
split list to split.

Regex

Enter a regular expression according to the programming language
you are using.

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

Click Sync columns to retrieve
the schema from the previous component connected in the Job.

Warning:

Make sure that the output schema does not contain any column
with the same name as the input column to be split. Otherwise,
the regular expression will not work as expected.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Advanced settings

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Encoding

Select the encoding from the list or select Custom
and define it manually. This field is compulsory for database data handling. The
supported encodings depend on the JVM that you are using. For more information, see
https://docs.oracle.com.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

In a
Talend
 Map/Reduce Job, this component is used as an intermediate
step and other components used along with it must be Map/Reduce components, too. They
generate native Map/Reduce code that can be executed directly in Hadoop.

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

For further information about a
Talend
 Map/Reduce Job, see the sections
describing how to create, convert and configure a
Talend
 Map/Reduce Job of the

Talend Open Studio for Big Data Getting Started Guide
.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
 data integration Jobs, and non Map/Reduce Jobs.

Related scenarios

No scenario is available for the Map/Reduce version of this component yet.

tExtractRegexFields properties for Apache Spark Batch

These properties are used to configure tExtractRegexFields running in the Spark Batch Job framework.

The Spark Batch
tExtractRegexFields component belongs to the Processing family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

Basic settings

Prev.Comp.Column list

Select the column you need to extract data from.

Regex

Enter a regular expression according to the programming language you
are using.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

Click Sync columns to retrieve the
schema from the previous component connected in the Job.

Warning:

Make sure that the output schema does not contain any column with
the same name as the input column to be split. Otherwise, the
regular expression will not work as expected.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Advanced settings

Encoding

Select the encoding from the list or select Custom
and define it manually. This field is compulsory for database data handling. The
supported encodings depend on the JVM that you are using. For more information, see
https://docs.oracle.com.

Usage

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Batch component Palette it belongs to,
appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a
scenario presents only Standard Jobs, that is to
say traditional
Talend
 data integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

  • Yarn mode (Yarn client or Yarn cluster):

    • When using Google Dataproc, specify a bucket in the
      Google Storage staging bucket
      field in the Spark configuration
      tab.

    • When using HDInsight, specify the blob to be used for Job
      deployment in the Windows Azure Storage
      configuration       area in the Spark
      configuration       tab.

    • When using Altus, specify the S3 bucket or the Azure
      Data Lake Storage for Job deployment in the Spark
      configuration       tab.
    • When using Qubole, add a
      tS3Configuration to your Job to write
      your actual business data in the S3 system with Qubole. Without
      tS3Configuration, this business data is
      written in the Qubole HDFS system and destroyed once you shut
      down your cluster.

    • When using on-premise
      distributions, use the configuration component corresponding
      to the file system your cluster is using. Typically, this
      system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the
    configuration component corresponding to the file system your cluster is
    using, such as tHDFSConfiguration or
    tS3Configuration.

    If you are using Databricks without any configuration component present
    in your Job, your business data is written directly in DBFS (Databricks
    Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Batch version of this component
yet.

tExtractRegexFields properties for Apache Spark Streaming

These properties are used to configure tExtractRegexFields running in the Spark Streaming Job framework.

The Spark Streaming
tExtractRegexFields component belongs to the Processing family.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

Basic settings

Prev.Comp.Column list

Select the column you need to extract data from.

Regex

Enter a regular expression according to the programming language you
are using.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Click Edit
schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this
    option to view the schema only.

  • Change to built-in property:
    choose this option to change the schema to Built-in for local changes.

  • Update repository connection:
    choose this option to change the schema stored in the repository and decide whether
    to propagate the changes to all the Jobs upon completion. If you just want to
    propagate the changes to the current Job, you can select No upon completion and choose this schema metadata
    again in the Repository Content
    window.

Click Sync columns to retrieve the
schema from the previous component connected in the Job.

Warning:

Make sure that the output schema does not contain any column with
the same name as the input column to be split. Otherwise, the
regular expression will not work as expected.

 

Built-In: You create and store the schema locally for this component
only.

 

Repository: You have already created the schema and stored it in the
Repository. You can reuse it in various projects and Job designs.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Advanced settings

Encoding

Select the encoding from the list or select Custom
and define it manually. This field is compulsory for database data handling. The
supported encodings depend on the JVM that you are using. For more information, see
https://docs.oracle.com.

Usage

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
 data
integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

  • Yarn mode (Yarn client or Yarn cluster):

    • When using Google Dataproc, specify a bucket in the
      Google Storage staging bucket
      field in the Spark configuration
      tab.

    • When using HDInsight, specify the blob to be used for Job
      deployment in the Windows Azure Storage
      configuration       area in the Spark
      configuration       tab.

    • When using Altus, specify the S3 bucket or the Azure
      Data Lake Storage for Job deployment in the Spark
      configuration       tab.
    • When using Qubole, add a
      tS3Configuration to your Job to write
      your actual business data in the S3 system with Qubole. Without
      tS3Configuration, this business data is
      written in the Qubole HDFS system and destroyed once you shut
      down your cluster.

    • When using on-premise
      distributions, use the configuration component corresponding
      to the file system your cluster is using. Typically, this
      system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the
    configuration component corresponding to the file system your cluster is
    using, such as tHDFSConfiguration or
    tS3Configuration.

    If you are using Databricks without any configuration component present
    in your Job, your business data is written directly in DBFS (Databricks
    Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Streaming version of this component
yet.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 7.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x