August 16, 2023

tNLPPreprocessing – Docs for ESB 6.x

  2. Talend for ESB Docs
  3. Docs 6.x
  4. tNLPPreprocessing – Docs for ESB 6.x

tNLPPreprocessing

Prepares a text sample and divides it into tokens, which can be words, numbers or
punctuation marks.

tNLPPreprocessing outputs a column containing all the tokens for the
input text, separated by tabs. You can convert the output to the CoNLL format and
manually annotate the text. Then, you can use it to train a model and design features
with the tNLPModel component.

This component can run only with Spark 1.6 and 2.0.

tNLPPreprocessing properties for Apache Spark Batch

These properties are used to configure tNLPPreprocessing
running in the Spark Batch Job framework.

The Spark Batch
tNLPPreprocessing component belongs to the Natural Language Processing family.

The component in this framework is available when you have subscribed to any Talend Platform product with Big Data or Talend Data
Fabric.

Basic settings

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Sync columns to retrieve the schema from
the previous component connected in the Job.

Click Edit schema to make changes to the schema.
If the current schema is of the Repository type, three
options are available:

  • View schema: choose this option to view the
    schema only.

  • Change to built-in property: choose this
    option to change the schema to Built-in for
    local changes.

  • Update repository connection: choose this
    option to change the schema stored in the repository and decide whether to propagate
    the changes to all the Jobs upon completion. If you just want to propagate the
    changes to the current Job, you can select No
    upon completion and choose this schema metadata again in the [Repository Content] window.

The output schema of this component contains a read-only column:

tokens: This column holds the tokens for each row
of the input data.

 

Built-In: You create and store the
schema locally for this component only. Related topic: see
Talend Studio

User Guide.

 

Repository: You have already created
the schema and stored it in the Repository. You can reuse it in various projects and
Job designs. Related topic: see
Talend Studio

User Guide.

NLP Library

From this list, select the library for text preprocessing between
ScalaNLP and Stanford
CoreNLP.

Clean all HTML tags

Select this check box to remove all the tags from the text.

Column to preprocess

Select the column from the input schema containing the text to be divided
into tokens.

Usage

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Batch component Palette it belongs to, appears only
when you are creating a Spark Batch Job.

Spark Batch Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Natural Language Processing using Talend Studio

Using Talend Studio and machine learning on Spark, you can teach computers to
understand how humans learn and use natural language.

What is natural language
processing?

Natural language processing tasks include:

  • text tokenization, which divides a text into basic units such as words or
    punctuation marks;

  • sentence splitting, which divides the input into sentences, based on
    ending characters, such as periods or question marks; and

  • named entity recognition, which finds and classify person names, dates,
    locations and organizations in a text.

Natural language processing is useful to:

  • extract person names or company names from textual resources;

  • group forum discussions together by topics;

  • find discussions where people are mentioned but don’t participate to the
    discussion; or

  • link entities.

Natural language processing can help you create links between user profiles
and mentions in the text, between persons and organizations, or between persons and any
other information that may be used for re-identification.

Workflow

Machine learning with Spark is usually two phases: the first phase computes a model
based on historical data and mathematical heuristics, and the second phase applies
the model on text data. In Talend Studio, the first phase is
implemented by two Jobs:

  • the first one with the tNLPPreprocessing and the
    tNormalize components; and

  • the second one with the tNLPModel component.

While the second phase is implemented by a third Job with the
tNLPPredict component.

nlp_workflow.png
In this workflow, tNLPPreprocessing:

  • divides a text sample in tokens; and

  • cleans the text sample by removing all HTML tags.

Then, tNormalize converts tokens to the CoNLL format.

You can then manually label the tokens and add optional features by editing the
files. For example, you can label person names with PER:

nlp_workflow2.png

Next, you can use the tokenized sample text you labeled with
tNLPModel in the second Job where
tNLPModel:

  • generates fatures for each token; and

  • trains a classification model.

tNLPPredict labels text data automatically using the
classification model generated by tNLPModel.

For example, you can extract named entities with <PER>
labels:

nlp_workflow3.png

Scenario: Preparing a text sample to be used for learning a model

This scenario applies only to a subscription-based Talend Platform solution with Big data or Talend Data Fabric.

This Job uses tNLPPreprocessing to divide the input text into tokens.
Then, the tokens are converted to the CoNLL format using tNormalize.
You will be able to use this CoNLL file to learn a classification model for extracting
named entities in text data.

Extracting names entities from text data is a three-phase operation:

  1. Preparing a text sample by dividing it into tokens. The tokens will be used
    for training a classification model.

  2. Learning a classification model, designing the features and evaluating the
    model.

    You can find an example of how to generate a named
    entity recognition model on Talend Help Center (https://help.talend.com).

  3. Applying the model on the full text to extract named entities using
    tNLPPredict.

    You can find an example of how to extract named
    entities using a classification model on Talend Help Center (https://help.talend.com).

You can find more information about natural language processing on
Talend Help Center (https://help.talend.com).

Creating a Job to divide the input text into tokens in CoNLL format

This Job uses tNLPPreprocessing to divide a text sample in XML
format into tokens. Then, tokens are converted to the CoNLL format using
tNormalize.

  1. Drop the following components from the Palette onto the
    design workspace: tXMLFileInput,
    tNLPPreprocessing,
    tFilterColumns, tNormalize and
    tFileOutputDelimited.
  2. Connect the components using Row > Main connections.
use_case_tnlppreprocessing.png

Selecting the Spark mode

Depending on the Spark cluster to be used, select a Spark mode for your Job.
  1. Click Run to open its view and then click the
    Spark Configuration tab to display its view
    for configuring the Spark connection.
  2. Select the Use local mode check box to test your Job locally.

    In the local mode, the Studio builds the Spark environment in itself on the fly in order to
    run the Job in. Each processor of the local machine is used as a Spark
    worker to perform the computations.

    In this mode, your local file system is used; therefore, deactivate the
    configuration components such as tS3Configuration or
    tHDFSConfiguration that provides connection
    information to a remote file system, if you have placed these components
    in your Job.

    You can launch
    your Job without any further configuration.

  3. Clear the Use local mode check box to display the
    list of the available Hadoop distributions and from this list, select
    the distribution corresponding to your Spark cluster to be used.

    If you cannot find the distribution corresponding to yours from this
    drop-down list, this means the distribution you want to connect to is not officially
    supported by
    Talend
    . In this situation, you can select Custom, then select the Spark
    version     of the cluster to be connected and click the
    [+] button to display the dialog box in which you can
    alternatively:

    1. Select Import from existing
      version       to import an officially supported distribution as base
      and then add other required jar files which the base distribution does not
      provide.

    2. Select Import from zip to
      import the configuration zip for the custom distribution to be used. This zip
      file should contain the libraries of the different Hadoop/Spark elements and the
      index file of these libraries.

      In
      Talend

      Exchange, members of
      Talend
       community have shared some ready-for-use configuration zip files
      which you can download from this Hadoop configuration
      list and directly use them in your connection accordingly. However, because of
      the ongoing evolution of the different Hadoop-related projects, you might not be
      able to find the configuration zip corresponding to your distribution from this
      list; then it is recommended to use the Import from
      existing version       option to take an existing distribution as base
      to add the jars required by your distribution.

      Note that custom versions are not officially supported by

      Talend
      .
      Talend
       and its community provide you with the opportunity to connect to
      custom versions from the Studio but cannot guarantee that the configuration of
      whichever version you choose will be easy. As such, you should only attempt to
      set up such a connection if you have sufficient Hadoop and Spark experience to
      handle any issues on your own.

    For a step-by-step example about how to connect to a custom
    distribution and share this connection, see Connecting to a custom Hadoop distribution.

Configuring the input component

The tFileInputXML component is used to load the text to be
processed.

  1. Double click the tFileInputXML component to open its
    Basic settings view and define its properties.

    use_case_tnlppreprocessing2.png

    1. Click the […] button next to Edit
      schema       to add the necessary columns to hold the input
      data.
    2. In the File name field, specify the path to the
      file to be processed.
    3. In the Element to extract, enter
      "row".
    4. In the Loop XPath query field, enter the XPath
      query expression between double quotation marks to specify the node on
      which the loop is based.
    5. In the XPath query column of the
      Mapping table, specify the fields to be
      queried between double quotation marks.

  2. In the Advanced settings view of the
    component, select the Custom encoding check box if you
    encounter issues when processing the data.

  3. From the Encoding list, select the encoding
    to be used, UTF-8 in this example.

Converting the tokenized text to the CoNLL format

To be able to learn a classification model from a text, you must divide this text
into tokens and convert it to the CoNLL format using
tNormalize.

  1. Double click the tNLPPreprocessing component to open its
    Basic settings view and define its properties.

    use_case_tnlppreprocessing3.png


    1. Click Sync columns to retrieve the
      schema from the previous component connected in the Job.
    1. From the NLP Library list, select the library to
      be used for tokenization. In this example,
      ScalaNLP is used.
  2. From the Column to preprocess list, select the column
    that holds the text to be divided into tokens, which is
    message in this example.
  3. Double click the tFilterColumns component to open its
    Basic settings view and define its properties.
  4. Click Edit schema to add the
    tokens column in the output schema because this is
    the column to be normalized, and click OK to
    validate.
  5. Double click the tNormalize component to open its
    Basic settings view and define its properties.

    use_case_tnlppreprocessing5.png


    1. Click Sync columns to retrieve the
      schema from the previous component connected in the Job.
    2. From the Column to normalize list, select
      tokens.
    3. In the Item separator field, enter
      " " to separate tokens using a tab in the
      output file.
  6. Double click the tFileOutputDelimited component to open
    its Basic settings view and define its properties.

    use_case_tnlppreprocessing6.png


    1. Click Sync columns to retrieve the
      schema from the previous component connected in the Job.
    2. In the Folder field, specify the path to the
      folder where the CoNLL files will be stored.
    3. In the Row Separator field, enter
      "
      "      .
    4. In the Field Separator field, enter
      " " to separate fields with a tab.

  7. Press F6 to save and execute the
    Job.

The output files are created in the specified folder. The files contain a single
column with one token per row.

use_case_tnlppreprocessing7.png

You can then manually label person names with PER and the
other tokens with O before you can learn a classification
model from this text data:

use_case_tnlppreprocessing8.png

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 6.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x