August 15, 2023

tFileInputPositional – Docs for ESB 6.x

  2. Talend for ESB Docs
  3. Docs 6.x
  4. tFileInputPositional – Docs for ESB 6.x

tFileInputPositional

Reads a positional file row by row to split them up into fields based on a given
pattern and then sends the fields as defined in the schema to the next
component.

Depending on the Talend solution you
are using, this component can be used in one, some or all of the following Job
frameworks:

tFileInputPositional Standard properties

These properties are used to configure tFileInputPositional running in the Standard Job framework.

The Standard
tFileInputPositional component belongs to the File family.

The component in this framework is generally available.

Basic settings

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.

 

Repository: Select the repository file where the
properties are stored.
Use existing dynamic

Select this check box to reuse an existing dynamic schema to
handle data from unknown columns.

When this check box is selected, a Component
list appears allowing you to select the component
used to set the dynamic schema.

File Name/Stream

File name: Name and path of the
file to be processed.

Stream: The data flow to be
processed. The data must be added to the flow in order for tFileInputPositional to fetch these data
via the corresponding representative variable.

This variable could be already pre-defined in your Studio or
provided by the context or the components you are using along with
this component, for example, the INPUT_STREAM
variable of tFileFetch; otherwise,
you could define it manually and use it according to the design of
your Job, for example, using tJava
or tJavaFlex.

In order to avoid the inconvenience of hand writing, you could
select the variable of interest from the auto-completion list
(Ctrl+Space) to fill the
current field on condition that this variable has been properly
defined.

Related topic to the available variables: see
Talend Studio User Guide.

Related scenario to the input stream, see Scenario 2: Reading data from a remote file in streaming mode.

Row separator

The separator used to identify the end of a row.

Use byte length as the cardinality

Select this check box to enable the support of double-byte
character to this component. JDK 1.6 is required for this
feature.

Customize

Select this check box to customize the data format of the
positional file and define the table columns:

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Enter, between double quotation marks,
the padding charater you need to remove from the field. A space by
default.

Alignment: Select the appropriate
alignment parameter.

Pattern

Length values separated by commas, interpreted as a string between
quotes. Make sure the values entered in this field are consistent
with the schema defined.

Skip empty rows

Select this check box to skip the empty rows.

Uncompress as zip file

Select this check box to uncompress the input file.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Header

Enter the number of rows to be skipped in the beginning of file.

Footer

Number of rows to be skipped at the end of the file.

Limit

Maximum number of rows to be processed. If Limit = 0, no row is
read or processed.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the schema.
If the current schema is of the Repository type, three
options are available:

  • View schema: choose this option to view the
    schema only.

  • Change to built-in property: choose this
    option to change the schema to Built-in for
    local changes.

  • Update repository connection: choose this
    option to change the schema stored in the repository and decide whether to propagate
    the changes to all the Jobs upon completion. If you just want to propagate the
    changes to the current Job, you can select No
    upon completion and choose this schema metadata again in the [Repository Content] window.

This component offers the
advantage of the dynamic schema feature. This allows you to retrieve unknown columns
from source files or to copy batches of columns from a source without mapping each
column individually. For further information about dynamic schemas, see
Talend Studio

User Guide.

This dynamic schema
feature is designed for the purpose of retrieving unknown columns of a table and is
recommended to be used for this purpose only; it is not recommended for the use of
creating tables.

This component must work with tSetDynamicSchema to leverage the dynamic schema
feature.

 

Built-in: The schema will be
created and stored locally for this component only. Related topic:
see
Talend Studio User Guide.

 

Repository: The schema already
exists and is stored in the Repository, hence can be reused in
various projects and Job flowcharts. Related topic: see

Talend Studio User
Guide.

Advanced settings

Needed to process rows longer than 100 000
characters

Select this check box if the rows to be processed in the input
file are longer than 100 000 characters.

Advanced separator (for numbers)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Thousands separator: define
separators for thousands.

Decimal separator: define
separators for decimals.

Trim all column

Select this check box to remove leading and trailing whitespaces
from defined columns.

Validate date

Select this check box to check the date format strictly against the input schema.

Encoding

Select the encoding from the list or select Custom and define it manually. This field is compulsory for database
data handling.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at a
Job level as well as at each component level.

Global Variables

Global Variables

NB_LINE: the number of rows processed. This is an After
variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

Use this component to read a file and separate fields using a
position separator value. You can also create a rejection flow using
a Row > Reject link to filter
the data which does not correspond to the type defined. For an
example of how to use these two links, see Procedure.

From Positional to XML file

The following scenario describes a two-component Job, which aims at reading data from
an input file that contains contract numbers, customer references, and insurance numbers
as shown below, and outputting the selected data (according to the data position) into
an XML
file.

Dropping and linking components

Use_Case_tFileInputPos1.png

  1. Drop a tFileInputPositional component
    from the Palette to the design workspace.
  2. Drop a tFileOutputXML component as well.
    This file is meant to receive the references in a structured way.
  3. Right-click the tFileInputPositional
    component and select Row
    >

    Main. Then drag it onto the
    tFileOutputXML component and release
    when the plug symbol shows up.

Configuring data input

  1. Double-click the tFileInputPositional
    component to show its Basic settings view
    and define its properties.

    use_case-tfileinputpositional1-tos.png

  2. Define the Job Property type if needed. For this scenario, we use the
    built-in Property type.

    As opposed to the Repository, this means that the Property type is set for
    this station only.
  3. Fill in a path to the input file in the File
    Name     field. This field is mandatory.
  4. Define the Row separator identifying the
    end of a row if needed, by default, a carriage return.
  5. If required, select the Use byte length as the
    cardinality     check box to enable the support of double-byte
    character.
  6. Define the Pattern to delimit fields in a
    row. The pattern is a series of length values corresponding to the values of
    your input files. The values should be entered between quotes, and separated
    by a comma. Make sure the values you enter match the schema defined.
  7. Fill in the Header, Footer and Limit fields
    according to your input file structure and your need. In this scenario, we
    only need to skip the first row when reading the input file. To do this,
    fill the Header field with 1 and leave the other fields as they are.
  8. Next to Schema, select Repository if the input schema is stored in the
    Repository. In this use case, we use a Built-In input schema to define the data to pass on to the
    tFileOutputXML component.
  9. You can load and/or edit the schema via the Edit
    Schema     function. For this schema, define three columns,
    respectively Contract, CustomerRef
    and InsuranceNr matching the structure of the input
    file. Then, click OK to close the [Schema] dialog box and propagate the
    changes.

    Use_Case_tFileInputPos2.png

Configuring data output

  1. Double-click tFileOutputXML to show its
    Basic settings view.

    use_case-tfileinputpositional3-tos.png

  2. Enter the XML output file path.
  3. Define the row tag that will wrap each row of data, in this use case
    ContractRef.
  4. Click the three-dot button next to Edit
    schema     to view the data structure, and click Sync columns to retrieve the data structure from
    the input component if needed.
  5. Switch to the Advanced settings tab view
    to define other settings for the XML output.

    use_case-tfileinputpos_xml_advsettings1.png

  6. Click the plus button to add a line in the Root
    tags     table, and enter a root tag (or more) to wrap the XML
    output structure, in this case ContractsList.
  7. Define parameters in the Output format
    table if needed. For example, select the As
    attribute     check box for a column if you want to use its name
    and value as an attribute for the parent XML element, clear the Use schema column name check box for a column to
    reuse the column label from the input schema as the tag label. In this use
    case, we keep all the default output format settings as they are.
  8. To group output rows according to the contract number, select the
    Use dynamic grouping check box, add a
    line in the Group by table, select
    Contract from the Column list field, and enter an attribute for it in the
    Attribute label field.

    use_case-tfileinputpos_xml_advsettings2.png

  9. Leave all the other parameters as they are.

Saving and executing the Job

  1. Press Ctrl+S to save your Job to ensure
    that all the configured parameters take effect.
  2. Press F6 or click Run on the Run tab to
    execute the Job.

    The file is read row by row based on the length values defined in the
    Pattern field and output as an XML file
    as defined in the output settings. You can open it using any standard XML
    editor.
    Use_Case_tFileInputPos4.png

Scenario 2: Handling a positional file based on a dynamic schema

This scenario applies only to a subscription-based Talend solution.

This scenario describes a four-component Job that reads data from a positional file,
writes the data to another positional file, and replaces the padding characters with
space. The schema column details are not defined in the positional file components;
instead, they leverages a reusable dynamic schema. The input file used in this scenario
is as follows:

Dropping and linking components

  1. Drop the following components from the Palette onto the design workspace: tFixedFlowInput, tSetDynamicSchema, tFileInputPositional, and tFileOutputPositional.
  2. Connect the tFixedFlowInput component to
    the tSetDynamicSchema using a Row > Main
    connection to form a subjob. This subjob will define a reusable dynamic
    schema.
  3. Connect the tFileInputPositional
    component to the tFileOutputPositional
    component using a Row > Main connection to form another subjob. This
    subjob will read data from the input positional file and write the data to
    another positional file based on the dynamic schema set in the previous
    subjob.
  4. Connect the tFixedFlowInput component to
    the tFileInputPositional component using a
    Trigger > On
    Subjob Ok     connection to link the two subjobs together.

    use_case-tfileinputpos2-1.png

Configuring the first subjob: creating a dynamic schema

  1. Double-click the tFixedFlowInput
    component to show its Basic settings view
    and define its properties.

    use_case_tfileinputpos2-2.png

  2. Click the […] button next to Edit schema to open the [Schema] dialog box.

    use_case_tfileinputpos2-3.png

  3. Click the [+] button to add three
    columns: ColumnName, ColumnType, and ColumnLength, and set their types to String, String, and
    Integer respectively to define the
    minimum properties required for a positional file schema. Then, click
    OK to close the dialog box.
  4. Select the Use Inline Table option, click
    the [+] button three times to add three
    lines, give them a name in the ColumnName
    field, according to the actual columns of the input file to read: ID, Name,
    and City, set their types in the
    corresponding ColumnType field: id_Interger for column ID, and id_String for
    columns Name and City, and set the length values of the columns in the
    corresponding ColumnLength field. Note that
    the column names you give in this table will compose the header of the
    output file.
  5. Double-click the tSetDynamicSchema
    component to open its Basic settings view.

    use_case_tfileinputpos2-4.png

  6. Click Sync columns to ensure that the
    schema structure is properly retrieved from the preceding component.
  7. Under the Parameters table, click the
    [+] button to add three lines in the
    table.
  8. Click in the Property field for each
    line, and select ColumnName, Type, and Length
    respectively.
  9. Click in the Value field for each line,
    and select ColumnName, ColumnType, and ColumnLength respectively.

    Now, with the values set in the inline table of the tFixedFlowInput component retrieved, the following data
    structure is defined in the dynamic schema:
    Column Name Type Length
    ID Integer 6
    Name String 12
    City String 12

Configuring the second subjob: reading and writing positional data

  1. Double-click the tFileInputPositional
    component to open its Basic settings
    view.

    use_case_tfileinputpos2-5.png

    Warning:

    The dynamic schema feature is only supported in Built-In mode and requires the input file
    to have a header row.

  2. Select the Use existing dynamic check
    box, and in from the Component List that
    appears, select the tSetDynamicSchema
    component you use to create the dynamic schema. In this use case, only one
    tSetDynamicSchema component is used, so
    it is automatically selected.
  3. In the File name/Stream field, enter the
    path to the input positional file, or browse to the file path by clicking
    the […] button.
  4. Fill in the Header, Footer and Limit fields
    according to your input file structure and your need. In this scenario, we
    only need to skip the first row when reading the input file. To do this,
    fill the Header field with 1 and leave the other fields as they are.
  5. Click the […] button next to Edit schema to open the [Schema] dialog box, define only one column, dyn in this example, and select Dynamic from the Type list. Then, click OK
    to close the [Schema] dialog box and
    propagate the changes.

    use_case_tfileinputpos2-6.png

  6. Select the Customize check box, enter
    '-' in the Padding char
    field, and keep the other settings as they are.
  7. Double-click the tFileOutputPositional
    component to open its Basic settings
    view.

    use_case_tfileinputpos2-7.png

  8. Select the Use existing dynamic check
    box, specify the output file path, and select the Include header check box.
  9. In the Padding char field, enter '
    '     so that the padding characters will be replaced with space in
    the output file.

Saving and executing the Job

  1. Press Ctrl+S to save your Job to ensure
    that all the configured parameters take effect.
  2. Press F6 or click Run on the Run tab to
    execute the Job.

    use_case_tfileinputpos2-8.png

    The data is read from the input positional file and written into the
    output positional file, with the padding characters replaced by
    space.

tFileInputPositional MapReduce properties

These properties are used to configure tFileInputPositional running in the MapReduce Job framework.

The MapReduce
tFileInputPositional component belongs to the MapReduce family.

The component in this framework is available only if you have subscribed to one
of the
Talend
 solutions with Big Data.

Basic settings

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the
schema. Note that if you make changes, the schema automatically becomes built-in.
 

Built-In: You create and store the
schema locally for this component only. Related topic: see
Talend Studio

User Guide.
 

Repository: You have already created
the schema and stored it in the Repository. You can reuse it in various projects and
Job designs. Related topic: see
Talend Studio

User Guide.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will read all of the files stored in that folder, for example,/user/talend/in; if sub-folders exist, the sub-folders are automatically ignored unless you define the property mapreduce.input.fileinputformat.input.dir.recursive to be true in the Hadoop properties table in the Hadoop configuration tab.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

If the file to be read is a compressed one, enter the file name
with its extension; then tFileInputPositional automatically decompresses it
at runtime. The supported compression formats and their
corresponding extensions are:

  • DEFLATE: *.deflate

  • gzip: *.gz

  • bzip2: *.bz2

  • LZO: *.lzo

Note that you need
to ensure you have properly configured the connection to the Hadoop
distribution to be used in the Hadoop
configuration tab in the Run view.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Clear the check box to skip any rows on error and complete the process for
error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Row separator

The separator used to identify the end of a row.

Customize

Select this check box to customize the data format of the
positional file and define the table columns:

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Enter, between double quotation marks,
the padding charater you need to remove from the field. A space by
default.

Alignment: Select the appropriate
alignment parameter.

Pattern

Enter between double quotes the length values separated by commas, interpreted as a
string. Make sure the values entered in this field are consistent
with the schema defined.

Header

Enter the number of rows to be skipped in the beginning of file.

For example, enter 0 to ignore
no rows for the data without header and set 1 for the data with header at the first row.

Skip empty rows

Select this check box to skip the empty rows.

Advanced settings

Custom Encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Then select the encoding to be used from the list or select
Custom and define it
manually.

Advanced separator (for number)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Trim columns

Select this check box to remove the leading and trailing whitespaces from all
columns. When this check box is cleared, the Check column to
trim table is displayed, which lets you select particular columns to
trim.

Global Variables

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule

In a
Talend
 Map/Reduce Job, it is used as a start component and requires
a transformation component as output link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tFileInputDelimited as well as the
MapReduce family appears in the Palette of the Studio.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
 data integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Map/Reduce version of this component yet.

tFileInputPositional properties for Apache Spark Batch

These properties are used to configure tFileInputPositional running in the Spark Batch Job framework.

The Spark Batch
tFileInputPositional component belongs to the File family.

The component in this framework is available only if you have subscribed to one
of the
Talend
 solutions with Big Data.

Basic settings

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job. For
example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the
schema. Note that if you make changes, the schema automatically becomes built-in.
 

Built-In: You create and store the
schema locally for this component only. Related topic: see
Talend Studio

User Guide.
 

Repository: You have already created
the schema and stored it in the Repository. You can reuse it in various projects and
Job designs. Related topic: see
Talend Studio

User Guide.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will read all of the files stored in that folder, for example, /user/talend/in; if sub-folders exist, the sub-folders are automatically ignored unless you define the property
spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive to be true in the Advanced properties table in the Spark configuration tab.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

If the file to be read is a compressed one, enter the file name
with its extension; then tFileInputPositional automatically decompresses it
at runtime. The supported compression formats and their
corresponding extensions are:

  • DEFLATE: *.deflate

  • gzip: *.gz

  • bzip2: *.bz2

  • LZO: *.lzo

The button for browsing does not work with the Spark Local mode; if you are using the Spark Yarn or the Spark Standalone mode,
ensure that you have properly configured the connection in a configuration component in
the same Job, such as tHDFSConfiguration.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Row separator

The separator used to identify the end of a row.

Customize

Select this check box to customize the data format of the
positional file and define the table columns:

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Enter, between double quotation marks,
the padding charater you need to remove from the field. A space by
default.

Alignment: Select the appropriate
alignment parameter.

Pattern

Enter between double quotes the length values separated by commas, interpreted as a
string. Make sure the values entered in this field are consistent
with the schema defined.

Header

Enter the number of rows to be skipped in the beginning of file.

For example, enter 0 to ignore
no rows for the data without header and set 1 for the data with header at the first row.

Skip empty rows

Select this check box to skip the empty rows.

Advanced settings

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

When you want to control the partition number, you can generally set at least as many partitions as
the number of executors for parallelism, while bearing in mind the available memory and the
data transfer pressure on your network.

Custom Encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Advanced separator (for number)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Trim columns

Select this check box to remove the leading and trailing whitespaces from all
columns. When this check box is cleared, the Check column to
trim table is displayed, which lets you select particular columns to
trim.

Usage

Usage rule

This component is used as a start component and requires an output link..

This component, along with the Spark Batch component Palette it belongs to, appears only
when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
 data integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Batch version of this component
yet.

tFileInputPositional properties for Apache Spark Streaming

These properties are used to configure tFileInputPositional running in the Spark Streaming Job framework.

The Spark Streaming
tFileInputPositional component belongs to the File family.

The component in this framework is available only if you have subscribed to Talend Real-time Big Data Platform or Talend Data
Fabric.

Basic settings

Define a storage configuration
component

Select the configuration component to be used to provide the configuration
information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local
system.

The configuration component to be used must be present in the same Job. For
example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write
the result in a given HDFS system.

Property type

Either Built-In or Repository.

 

Built-In: No property data stored centrally.
 

Repository: Select the repository file where the
properties are stored.

The properties are stored centrally under the Hadoop
Cluster node of the Repository
tree.

The fields that come after are pre-filled in using the fetched
data.

For further information about the Hadoop
Cluster node, see the Getting Started Guide.

Schema and Edit
Schema

A schema is a row description. It defines the number of fields (columns) to
be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the
schema. Note that if you make changes, the schema automatically becomes built-in.
 

Built-In: You create and store the
schema locally for this component only. Related topic: see
Talend Studio

User Guide.
 

Repository: You have already created
the schema and stored it in the Repository. You can reuse it in various projects and
Job designs. Related topic: see
Talend Studio

User Guide.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

If the path you set points to a folder, this component will read all of the files stored in that folder, for example, /user/talend/in; if sub-folders exist, the sub-folders are automatically ignored unless you define the property
spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive to be true in the Advanced properties table in the Spark configuration tab.

If you want to specify more than one files or directories in this
field, separate each path using a comma (,).

If the file to be read is a compressed one, enter the file name
with its extension; then tFileInputPositional automatically decompresses it
at runtime. The supported compression formats and their
corresponding extensions are:

  • DEFLATE: *.deflate

  • gzip: *.gz

  • bzip2: *.bz2

  • LZO: *.lzo

The button for browsing does not work with the Spark Local mode; if you are using the Spark Yarn or the Spark Standalone mode,
ensure that you have properly configured the connection in a configuration component in
the same Job, such as tHDFSConfiguration.

Die on error

Select the check box to stop the execution of the Job when an error
occurs.

Row separator

The separator used to identify the end of a row.

Customize

Select this check box to customize the data format of the
positional file and define the table columns:

Column: Select the column you
want to customize.

Size: Enter the column
size.

Padding char: Enter, between double quotation marks,
the padding charater you need to remove from the field. A space by
default.

Alignment: Select the appropriate
alignment parameter.

Pattern

Enter between double quotes the length values separated by commas, interpreted as a
string. Make sure the values entered in this field are consistent
with the schema defined.

Header

Enter the number of rows to be skipped in the beginning of file.

For example, enter 0 to ignore
no rows for the data without header and set 1 for the data with header at the first row.

Skip empty rows

Select this check box to skip the empty rows.

Advanced settings

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

When you want to control the partition number, you can generally set at least as many partitions as
the number of executors for parallelism, while bearing in mind the available memory and the
data transfer pressure on your network.

Custom Encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Advanced separator (for number)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Trim columns

Select this check box to remove the leading and trailing whitespaces from all
columns. When this check box is cleared, the Check column to
trim table is displayed, which lets you select particular columns to
trim.

Usage

Usage rule

This component is used as a start component and requires an output link.

This component is only used to provide the lookup flow (the right side of a join
operation) to the main flow of a tMap component. In this
situation, the lookup model used by this tMap must be
Load once.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
 data
integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in
the Run view to define the connection to a given
Spark cluster for the whole Job. In addition, since the Job expects its dependent jar
files for execution, you must specify the directory in the file system to which these
jar files are transferred so that Spark can access these files:

  • Yarn mode: when using Google
    Dataproc, specify a bucket in the Google Storage staging
    bucket     field in the Spark
    configuration     tab; when using other distributions, use a
    tHDFSConfiguration
    component to specify the directory.

  • Standalone mode: you need to choose
    the configuration component depending on the file system you are using, such
    as tHDFSConfiguration
    or tS3Configuration.

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Streaming version of this component
yet.

Document get from Talend https://help.talend.com
Thank you for watching.
Docs 6.x
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x