tSchemaComplianceCheck – Docs for ESB 5.x

Component family

Data Quality

Function

Validates all input rows against a reference schema or check
types, nullability, length of rows against reference values. The
validation can be carried out in full or partly.

Purpose

Helps to ensure the data quality of any source data against a
reference data source.

Basic settings

Base Schema and Edit
schema

A schema is a row description, it defines the number of fields to be processed and
passed on to the next component. The schema is either Built-in or stored remotely in the
Repository.

Since version 5.6, both the Built-In mode and the Repository mode are
available in any of the Talend solutions.

Describe the structure and nature of your data to be processed as
it is.

Built-in: The schema will be
created and stored locally for this component only. Related topic:
see Talend Studio User Guide.

Repository: The schema already
exists and is stored in the Repository, hence can be reused in
various projects and Job designs. Related topic: see
Talend Studio User
Guide.

Check all columns from schema

Select this option to carry out all checks on all columns against
the base schema.

Custom defined

Select this option to carry out particular checks on particular
columns. When this option is selected, the Checked Columns table shows.

Checked Columns

In this table, define what checks are to be carried out on which
columns.

Column: Displays the columns
names.

Type: Select the type of data
each column is supposed to contain. This validation is mandatory for
all columns.

Date pattern: Define the expected
date format for each column with the data type of
Date.

Nullable: Select the check box in
an individual column to define the column to be nullable, that is,
to allow empty rows in this column to go to the output flow
regardless of the base schema definition. To define all columns to
be nullable, select the check box in the table header.

Undefined or empty: Select the
check box in an individual column to reject empty rows in this
column while the column is not nullable in the base schema
definition. To carry out this verification on all the columns,
select the check box in the table header.

Max length: Select the check box
in an individual column to verify the data length of the column
against the length definition of the base schema. To carry out this
verification on all the columns, select the check box in the table
header.

Use another schema for compliance check

Define a reference schema as you expect the data to be, in order
to reject the non-compliant data.

It can be restrictive on data type, null values, and/or
length.

Trim the excess content of column when length checking
chosen and the length is greater than defined length

With any of the three modes of tSchemaComplianceCheck, select this check box to
truncate the data that exceeds the length specified rather than
reject it.

Advanced settings

Use Fastest Date Check

Select this check box to perform a fast date format check using
the TalendDate.isDate() method of the
TalendDate system routine if
Date pattern is not defined. For more
information about routines, see Talend Studio User
Guide.

Ignore TimeZone when Check Date

Select this check box to ignore the time zone setup upon date
check.

Not available when the Check all columns
from schema mode is selected.

Treat all empty string as NULL

Select this check box to treat any empty fields in any columns as
null values, instead of empty strings.

By default, this check box is selected. When it is cleared, the
Choose Column(s) table shows to
let you select individual columns.

tStatCatcher Statistics

Select this check box to collect log data at the component level.
Note that this check box is not available in the Map/Reduce version
of the component.

Global Variables

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio
User Guide.

Usage

This component is an intermediary step in the flow allowing to
exclude from the main flow the non-compliant data. This component
cannot be a start component as it requires an input flow. It also
requires at least one output component to gather the validated flow,
and possibly a second output component for rejected data using
Rejects link. For more information, see Talend Studio User Guide.

Usage in Map/Reduce Jobs

If you have subscribed to one of the Talend solutions with Big Data, you can also
use this component as a Map/Reduce component. In a Talend Map/Reduce Job, this
component is used as an intermediate step and other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

It does not support data of the Object and the List
types.

For further information about a Talend Map/Reduce Job, see the sections
describing how to create, convert and configure a Talend Map/Reduce Job of the
Talend Big Data Getting Started Guide.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional Talend data
integration Jobs, and non Map/Reduce Jobs.