tFileInputJSON

Extracts JSON data from a file and transfers the data to a file, a database table,
etc.

Depending on the Talend
product you are using, this component can be used in one, some or all of the following
Job frameworks:

Standard: see tFileInputJSON Standard properties.

The component in this framework is available in all Talend
products.
MapReduce: see tFileInputJSON MapReduce properties (deprecated).

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.
Spark Batch: see tFileInputJSON properties for Apache Spark Batch.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.
Spark Streaming: see tFileInputJSON properties for Apache Spark Streaming.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

tFileInputJSON Standard properties

These properties are used to configure tFileInputJSON running in the Standard Job framework.

The Standard
tFileInputJSON component belongs to the Internet and the File families.

The component in this framework is available in all Talend
products.

Basic settings

Property Type	Either Built-In or Repository.
	Built-In: No property data stored centrally.
	Repository: Select the repository file where the properties are stored.
Schema and Edit schema	A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word `line` when naming the fields. Built-In: You create and store the schema locally for this component only. Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Click Edit schema to make changes to the schema. Note: If you make changes, the schema automatically becomes built-in. View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
Read By	Select a way of extracting JSON data in the file. JsonPath: Extracts JSON data based on the JSONPath query. With this option selected, you need to select a JSONPath API version from the API version drop-down list. It is recommended to read data by JSONPath in order to gain better performance. Xpath: Extracts JSON data based on the XPath query. JsonPath without loop: Extracts JSON data based on the JSONPath query without setting a loop node.
Use Url	Select this check box to retrieve data directly from the Web.
URL	Enter the URL path from which you will retrieve data. This field is available only when the Use Url check box is selected.
Filename	Specify the file from which you will retrieve data. This field is not visible if the Use Url check box is selected. Warning: Use absolute path (instead of relative path) for this field to avoid possible errors.
Loop Jsonpath query	Enter the path pointing to the node within the JSON field, on which the loop is based. Note if you have selected Xpath from the Read by drop-down list, the Loop Xpath query field is displayed instead.
Mapping	Complete this table to map the columns defined in the schema to the corresponding JSON nodes. Column: The Column cells are automatically filled with the defined schema column names. Json query/JSONPath query: Specify the JSONPath node that holds the desired data. For more information about JSONPath expressions, see http://goessner.net/articles/JsonPath/. This column is available only when JsonPath is selected from the Read By list. XPath query: Specify the XPath node that holds the desired data. This column is available only when Xpath is selected from the Read By list. Get Nodes: Select this check box to extract the JSON data of all the nodes or select the check box next to a specific node to extract the data of that node. This column is available only when Xpath is selected from the Read By list.
Die on error	Select this check box to stop the execution of the Job when an error occurs. Clear the check box to skip the row on error and complete the process for error-free rows. If needed, you can collect the rows on error using a Row > Reject link.

Advanced settings

Advanced separator (for numbers)	Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.). Thousands separator: define separators for thousands. Decimal separator: define separators for decimals.
Use the loop node as root	Select this check box to use the loop node as the root for querying the file. The loop node is set in the Loop Json query text frame in the Basic Settings view. If this option is checked, only the child elements of the loop node are available for querying; otherwise, both the parent elements and the child elements of the loop node can be queried. You can specify a parent element through JSON path syntax. This check box is available only when JsonPath is selected in the Read By drop-down list of the Basic settings view.
Validate date	Select this check box to check the date format strictly against the input schema. This check box is available only if the Read By XPath check box is selected.
Encoding	Select the encoding type from the list or select Custom and define it manually. This field is compulsory for DB data handling.
tStatCatcher Statistics	Select this check box to gather the Job processing metadata at a Job level as well as at each component level.

Global Variables

Global Variables	NB_LINE: the number of rows processed. This is an After variable and it returns an integer. ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box. A Flow variable functions during the execution of a component while an After variable functions after the execution of the component. To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it. For further information about variables, see Talend Studio User Guide.

NB_LINE: the number of rows processed. This is an After
variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the
component when an error occurs. This is an After variable and it returns a string. This
variable functions only if the Die on error check box is
cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule	This component is a start component of a Job and always needs an output link.

Extracting JSON data from a file using JSONPath without setting a loop
node

This scenario describes a two-component Job that extracts data from the JSON file
Store.json by specifying the complete JSON path for
each node of interest and displays the flat data extracted on the console.

The JSON file Store.json contains information about a
department store and the content of the file is as follows:

{"store": {
    "name": "Sunshine Department Store",
    "address": "Wangfujing Street",
    "goods": {
        "book": [
            {
                "category": "Reference",
                "title": "Sayings of the Century",
                "author": "Nigel Rees",
                "price": 8.88
            },
            {
                "category": "Fiction",
                "title": "Sword of Honour",
                "author": "Evelyn Waugh",
                "price": 12.66
            }
        ],
        "bicycle": {
            "type": "GIANT OCR2600",
            "color": "White",
            "price": 276
        }
    }
}}

{"store": {

"name": "Sunshine Department Store",

"address": "Wangfujing Street",

"goods": {

"book": [

{

"category": "Reference",

"title": "Sayings of the Century",

"author": "Nigel Rees",

"price": 8.88

{

"category": "Fiction",

"title": "Sword of Honour",

"author": "Evelyn Waugh",

"price": 12.66

}

"bicycle": {

"type": "GIANT OCR2600",

"color": "White",

"price": 276

}

}}

In the following example, we will extract the store name, the store address, and the
bicycle information from this file.

Adding and linking the components

Create a new Job and add a tFileInputJSON
component and a tLogRow component by typing
their names in the design workspace or dropping them from the Palette.
Link the tFileInputJSON component to the
tLogRow component using a Row > Main
connection.

Configuring the components

Double-click the tFileInputJSON component to
open its Basic settings view.
Select JsonPath without loop from the
Read By drop-down list. With this option,
you need to specify the complete JSON path for each node of interest in the
JSONPath query fields of the Mapping table.
Click the […] button next to Edit schema to open the schema editor.
Click the [+] button to add five columns,
store_name, store_address, bicycle_type,
and bicycle_color of String type, and bicycle_price of Double
type.

Click OK to close the schema editor. In the
pop-up dialog box, click Yes to propagate the
schema to the subsequent component.
In the Filename field, specify the path to
the JSON file that contains the data to be extracted. In this example, it is
“E:/Store.json”.
In the Mapping table, the Column fields are automatically filled with the
schema columns you have defined.

In the JSONPath query fields, enter the
JSONPath query expressions between double quotation marks to specify the nodes
that hold the desired data.
- For the columns store_name and
  store_address, enter the JSONPath
  query expressions “$.store.name” and
  “$.store.address” relative to the
  nodes name and address respectively.
- For the columns bicycle_type,
  bicycle_color, and bicycle_price, enter the JSONPath query
  expressions “$.store.goods.bicycle.type”, “$.store.goods.bicycle.color”, and “$.store.goods.bicycle.price” relative to
  the child nodes type, color, and price of the bicycle
  node respectively.
Double-click the tLogRow component to display
its Basic settings view.
In the Mode area, select Table (print values in cells of a table) for better
readability of the result.

Executing the Job

Press Ctrl+S to save the Job.
Press F6 to execute the Job.

As shown above, the store name, the store address, and the bicycle information
are extracted from the source JSON data and displayed in a flat table on the
console.

Extracting JSON data from a file using JSONPath

Based on Extracting JSON data from a file using JSONPath without setting a loop node, this scenario
extracts data under the book array of the JSON file
Store.json by specifying a loop node and the relative
JSON path for each node of interest, and then displays the flat data extracted on the
console.

Procedure

In the Studio, open the Job used in Extracting JSON data from a file using JSONPath without setting a loop node to display it in
the design workspace.
Double-click the tFileInputJSON component to open
its Basic settings view.
Select JsonPath from the Read By drop-down list.
In the Loop Json query field, enter the JSONPath
query expression between double quotation marks to specify the node on which the
loop is based. In this example, it is “$.store.goods.book[*]”.
Click the […] button next to Edit schema to open the schema editor.

Select the five columns added previously and click the x button to remove all of them.

Click the [+] button to add four columns,
book_title, book_category, and book_author of
String type, and book_price of Double type.

Click OK to close the schema editor. In the
pop-up dialog box, click Yes to propagate the
schema to the subsequent component.
In the Json query fields of the Mapping table, enter the JSONPath query expressions between double
quotation marks to specify the nodes that hold the desired data. In this example,
enter the JSONPath query expressions “title”,
“category”, “author”, and “price” relative to
the four child nodes of the book node
respectively.
Press Ctrl+S to save the Job.
Press F6 to execute the Job.

As shown above, the book information is extracted from the source JSON data and displayed
in a flat table on the console.

Extracting JSON data from a file using XPath

Based on Extracting JSON data from a file using JSONPath without setting a loop node, this
scenario extracts the store name and the book information from the JSON file Store.json using XPath queries and displays the flat data
extracted on the console.

Procedure

In the Studio, open the Job used in Extracting JSON data from a file using JSONPath without setting a loop node to display it in
the design workspace.
Double-click the tFileInputJSON component to open
its Basic settings view.
Select Xpath from the Read
By drop-down list.
Click the […] button next to Edit schema to open the schema editor.

Select the five columns added previously and click the x button to remove all of them.

Click the [+] button to add five columns,
store_name, book_title, book_category, and
book_author of String type, and book_price of
Double type.

Click OK to close the schema editor. In the
pop-up dialog box, click Yes to propagate the
schema to the subsequent component.
In the Loop XPath query field, enter the XPath
query expression between double quotation marks to specify the node on which the
loop is based. In this example, it is “/store/goods/book”.
In the XPath query fields of the Mapping table, enter the XPath query expressions between
double quotation marks to specify the nodes that hold the desired data.
- For the column store_name, enter the
  XPath query “../../name” relative to the
  name node.
- For the columns book_title, book_category, book_author, and book_price, enter the XPath query expressions “title”, “category”, “author”, and
  “price” relative to the four child
  nodes of the book node
  respectively.
Press Ctrl+S to save the Job.
Press F6 to execute the Job.

As shown above, the store name and the book information are extracted from the
source JSON data and displayed in a flat table on the console.

Extracting JSON data from a URL

In this scenario, tFileInputJSON retrieves data of the
friends node from the JSON file facebook.json on the Web that contains the data of a Facebook
user and tExtractJSONFields extracts the data from the
friends node for flat data output.

The JSON file facebook.json is deployed on the Tomcat
server, specifically, located in the folder <tomcat
path>/webapps/docs, and the content of the file is as follows:

{"user": {
    "id": "9999912398",
    "name": "Kelly Clarkson",
    "friends": [
        {
            "name": "Tom Cruise",
            "id": "55555555555555",
            "likes": {"data": [
                {
                    "category": "Movie",
                    "name": "The Shawshank Redemption",
                    "id": "103636093053996",
                    "created_time": "2012-11-20T15:52:07+0000"
                },
                {
                    "category": "Community",
                    "name": "Positiveretribution",
                    "id": "471389562899413",
                    "created_time": "2012-12-16T21:13:26+0000"
                }
            ]}
        },
        {
            "name": "Tom Hanks",
            "id": "88888888888888",
            "likes": {"data": [
                {
                    "category": "Journalist",
                    "name": "Janelle Wang",
                    "id": "136009823148851",
                    "created_time": "2013-01-01T08:22:17+0000"
                },
                {
                    "category": "Tv show",
                    "name": "Now With Alex Wagner",
                    "id": "305948749433410",
                    "created_time": "2012-11-20T06:14:10+0000"
                }
            ]}
        }
    ]
}}

{"user": {

"id": "9999912398",

"name": "Kelly Clarkson",

"friends": [

{

"name": "Tom Cruise",

"id": "55555555555555",

"likes": {"data": [

{

"category": "Movie",

"name": "The Shawshank Redemption",

"id": "103636093053996",

"created_time": "2012-11-20T15:52:07+0000"

{

"category": "Community",

"name": "Positiveretribution",

"id": "471389562899413",

"created_time": "2012-12-16T21:13:26+0000"

}

]}

{

"name": "Tom Hanks",

"id": "88888888888888",

"likes": {"data": [

{

"category": "Journalist",

"name": "Janelle Wang",

"id": "136009823148851",

"created_time": "2013-01-01T08:22:17+0000"

{

"category": "Tv show",

"name": "Now With Alex Wagner",

"id": "305948749433410",

"created_time": "2012-11-20T06:14:10+0000"

}

]}

}

]

}}

Adding and linking the components

Create a new Job and add a tFileInputJSON
component, a tExtractJSONFields component, and
two tLogRow components by typing their names in
the design workspace or dropping them from the Palette.
Link the tFileInputJSON component to the
first tLogRow component using a Row > Main
connection.
Link the first tLogRow component to the
tExtractJSONFields component using a
Row > Main connection.
Link the tExtractJSONFields component to the
second tLogRow component using a Row > Main
connection.

Configuring the components

Double-click the tFileInputJSON component to
open its Basic settings view.
Select JsonPath without
loop from the Read By
drop-down list. Then select the Use Url
check box and in the URL field displayed
enter the URL of the file facebook.json from which the data
will be retrieved. In this example, it is
http://localhost:8080/docs/facebook.json.
Click the […] button
next to Edit schema and in the Schema dialog box define the schema by adding one
column friends of String type.

Click OK to close the
dialog box and accept the propogation prompted by the pop-up dialog box.
In the Mapping table,
enter the JSONPath query "$.user.friends[*]" next to the
friends column to retrieve the entire
friends node from the source file.
Double-click tExtractJSONFields to open its
Basic settings view.
Select Xpath from the Read By drop-down list.
In the Loop XPath query
field, enter the XPath expression between double quotation marks to specify the
node on which the loop is based. In this example, it is
"/likes/data".
Click the […] button
next to Edit schema and in the Schema dialog box define the schema by adding five
columns of String type,
id, name,
like_id, like_name, and
like_category, which will hold the data of relevant nodes under the JSON field
friends.

Click OK to close the
dialog box and accept the propogation prompted by the pop-up dialog box.
In the XPath query
fields of the Mapping table, type in the
XPath query expressions between double quotation marks to specify the JSON nodes
that hold the desired data. In this example:
- "../../id" (querying the
  “/friends/id” node) for the column id,
- "../../name" (querying the
  “/friends/name” node) for the column name,
- "id" for the column
  like_id,
- "name" for the column
  like_name, and
- "category" for the column
  like_category.
Double-click the second tLogRow component to
open its Basic settings view.

In the Mode area, select Table (print values in cells of a table) for better
readability of the result.

Executing the Job

Press Ctrl + S to save the Job.
Click F6 to execute the Job.

As shown above, the friends data in the JSON file specified using the URL is
extracted and then the data from the node friends is extracted and displayed in a flat table.

tFileInputJSON MapReduce properties (deprecated)

These properties are used to configure tFileInputJSON running in the MapReduce Job framework.

The MapReduce
tFileInputJSON component belongs to the MapReduce family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

The MapReduce framework is deprecated from Talend 7.3 onwards. Use Talend Jobs for Apache Spark to accomplish your integration tasks.

Basic settings

Property type	Either Built-In or Repository.
	Built-In: No property data stored centrally.
	Repository: Select the repository file where the properties are stored. The fields that come after are pre-filled in using the fetched data. For further information about the File Json node, see the section about setting up a JSON file schema in Talend Studio User Guide.
Schema et Edit Schema	A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word `line` when naming the fields. Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
	Built-In: You create and store the schema locally for this component only.
	Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Read by	Select a way of extracting the JSON data in the file. Xpath: Extracts the JSON data based on the XPath query. JsonPath: Extracts the JSON data based on the JSONPath query. Note that it is recommended to read the data by JSONPath in order to gain better performance.
Folder/File	Enter the path to the file or folder on HDFS from which the data will be extracted. If the path you entered points to a folder, all files stored in that folder will be read. If the file to be read is a compressed one, enter the file name with its extension; then tFileInputJSON automatically decompresses it at runtime. The supported compression formats and their corresponding extensions are: DEFLATE: .deflate gzip: .gz bzip2: .bz2 LZO: .lzo Note that you need to ensure you have properly configured the connection to the Hadoop distribution to be used in the Hadoop configuration tab in the Run view.
Die on error	Select the check box to stop the execution of the Job when an error occurs. Clear the check box to skip any rows on error and complete the process for error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.
Loop Jsonpath query	Enter the path pointing to the node within the JSON field, on which the loop is based. Note if you have selected Xpath from the Read by drop-down list, the Loop Xpath query field is displayed instead.
Mapping	Complete this table to map the columns defined in the schema to the corresponding JSON nodes. Column: The Column cells are automatically filled with the defined schema column names. Json query/JSONPath query: Specify the JSONPath node that holds the desired data. For more information about JSONPath expressions, see http://goessner.net/articles/JsonPath/. This column is available only when JsonPath is selected from the Read By list. XPath query: Specify the XPath node that holds the desired data. This column is available only when Xpath is selected from the Read By list. Get Nodes: Select this check box to extract the JSON data of all the nodes or select the check box next to a specific node to extract the data of that node. This column is available only when Xpath is selected from the Read By list.

Advanced settings

Advanced separator (for number)	Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).
Validate date	Select this check box to check the date format strictly against the input schema.
Encoding	Select the encoding from the list or select Custom and define it manually.

Global Variables

Global Variables	ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box. A Flow variable functions during the execution of a component while an After variable functions after the execution of the component. To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it. For further information about variables, see Talend Studio User Guide.

A Flow variable functions during the execution of a component while an After variable
functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl +
Space to access the variable list and choose the variable to use from it.

For further information about variables, see
Talend Studio

User Guide.

Usage

Usage rule	In a Talend Map/Reduce Job, it is used as a start component and requires a transformation component as output link. The other components used along with it must be Map/Reduce components, too. They generate native Map/Reduce code that can be executed directly in Hadoop. Once a Map/Reduce Job is opened in the workspace, tFileInputJSON as well as the MapReduce family appears in the Palette of the Studio. For further information about a Talend Map/Reduce Job, see the sections describing how to create, convert and configure a Talend Map/Reduce Job of the Talend Open Studio for Big Data Getting Started Guide . Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs, and non Map/Reduce Jobs.
Hadoop Connection	You need to use the Hadoop Configuration tab in the Run view to define the connection to a given Hadoop distribution for the whole Job. This connection is effective on a per-Job basis.
Prerequisites	The Hadoop distribution must be properly installed, so as to guarantee the interaction with Talend Studio . The following list presents MapR related information for example. Ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to MapR’s documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL hadoophadoop-VERSIONlib ative. For example, the library for Windows is lib ativeMapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr. Without adding the specified library or libraries, you may encounter the following error: `no MapRClient in java.library.path`. Set the `-Djava.library.path` argument, for example, in the Job Run VM arguments area of the Run/Debug view in the Preferences dialog box in the Window menu. This argument provides to the Studio the path to the native library of that MapR client. This allows the subscription-based users to make full use of the Data viewer to view locally in the Studio the data stored in MapR. For further information about how to install a Hadoop distribution, see the manuals corresponding to the Hadoop distribution you are using.

Usage rule

In a
Talend
Map/Reduce Job, it is used as a start component and requires
a transformation component as output link. The other components used along with it must be
Map/Reduce components, too. They generate native Map/Reduce code that can be executed
directly in Hadoop.

Once a Map/Reduce Job is opened in the workspace, tFileInputJSON as well as the MapReduce
family appears in the Palette of
the Studio.

For further information about a
Talend
Map/Reduce Job, see the sections
describing how to create, convert and configure a
Talend
Map/Reduce Job of the

Talend Open Studio for Big Data Getting Started Guide
.

Note that in this documentation, unless otherwise
explicitly stated, a scenario presents only Standard Jobs,
that is to say traditional
Talend
data integration Jobs, and non Map/Reduce Jobs.

Hadoop Connection

You need to use the Hadoop Configuration tab in the
Run view to define the connection to a given Hadoop
distribution for the whole Job.

This connection is effective on a per-Job basis.

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction
with
Talend Studio
. The following list presents MapR related information for
example.

Ensure that you have installed the MapR client in the machine where the Studio is,
and added the MapR client library to the PATH variable of that machine. According
to MapR’s documentation, the library or libraries of a MapR client corresponding to
each OS version can be found under MAPR_INSTALL
hadoophadoop-VERSIONlib
ative. For example, the library for
Windows is lib
ativeMapRClient.dll in the MapR
client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.

Without adding the specified library or libraries, you may encounter the following
error: no MapRClient in java.library.path.
Set the -Djava.library.path argument, for example, in the Job Run VM arguments area
of the Run/Debug view in the Preferences dialog box in the Window menu. This argument provides to the Studio the path to the
native library of that MapR client. This allows the subscription-based users to make
full use of the Data viewer to view locally in the
Studio the data stored in MapR.

For further information about how to install a Hadoop distribution, see the manuals
corresponding to the Hadoop distribution you are using.

Related scenarios

No scenario is available for the Map/Reduce version of this component yet.

tFileInputJSON properties for Apache Spark Batch

These properties are used to configure tFileInputJSON running in the Spark Batch Job framework.

The Spark Batch
tFileInputJSON component belongs to the File family.

The component in this framework is available in all subscription-based Talend products with Big Data
and Talend Data Fabric.

Basic settings

Define a storage configuration component	Select the configuration component to be used to provide the configuration information for the connection to the target file system such as HDFS. If you leave this check box clear, the target file system is the local system. The configuration component to be used must be present in the same Job. For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write the result in a given HDFS system.
Property type	Either Built-In or Repository.
	Built-In: No property data stored centrally.
	Repository: Select the repository file where the properties are stored. The fields that come after are pre-filled in using the fetched data. For further information about the File Json node, see the section about setting up a JSON file schema in Talend Studio User Guide.
Schema et Edit Schema	A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word `line` when naming the fields. Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
	Built-In: You create and store the schema locally for this component only.
	Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Read by	Select a way of extracting the JSON data in the file. Xpath: Extracts the JSON data based on the XPath query. JsonPath: Extracts the JSON data based on the JSONPath query. Note that it is recommended to read the data by JSONPath in order to gain better performance.
Folder/File	Browse to, or enter the path pointing to the data to be used in the file system. If the path you entered points to a folder, all files stored in that folder will be read. If the file to be read is a compressed one, enter the file name with its extension; then tFileInputJSON automatically decompresses it at runtime. The supported compression formats and their corresponding extensions are: DEFLATE: .deflate gzip: .gz bzip2: .bz2 LZO: .lzo The button for browsing does not work with the Spark Local mode; if you are using the other Spark Yarn modes that the Studio supports with your distribution, ensure that you have properly configured the connection in a configuration component in the same Job, such as tHDFSConfiguration . Use the configuration component depending on the filesystem to be used.
Die on error	Select the check box to stop the execution of the Job when an error occurs. Clear the check box to skip any rows on error and complete the process for error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.
Loop Jsonpath query	Enter the path pointing to the node within the JSON field, on which the loop is based. Note if you have selected Xpath from the Read by drop-down list, the Loop Xpath query field is displayed instead.
Mapping	Complete this table to map the columns defined in the schema to the corresponding JSON nodes. Column: The Column cells are automatically filled with the defined schema column names. Json query/JSONPath query: Specify the JSONPath node that holds the desired data. For more information about JSONPath expressions, see http://goessner.net/articles/JsonPath/. This column is available only when JsonPath is selected from the Read By list. XPath query: Specify the XPath node that holds the desired data. This column is available only when Xpath is selected from the Read By list. Get Nodes: Select this check box to extract the JSON data of all the nodes or select the check box next to a specific node to extract the data of that node. This column is available only when Xpath is selected from the Read By list.

Advanced settings

Set minimum partitions	Select this check box to control the number of partitions to be created from the input data over the default partitioning behavior of Spark. In the displayed field, enter, without quotation marks, the minimum number of partitions you want to obtain. When you want to control the partition number, you can generally set at least as many partitions as the number of executors for parallelism, while bearing in mind the available memory and the data transfer pressure on your network.
Advanced separator (for number)	Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).
Encoding	You may encounter encoding issues when you process the stored data. In that situation, select this check box to display the Encoding list. Select the encoding from the list or select Custom and define it manually.

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

When you want to control the partition number, you can generally set at least as many partitions as
the number of executors for parallelism, while bearing in mind the available memory and the
data transfer pressure on your network.

Advanced separator (for number)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Usage

Usage rule	This component is used as a start component and requires an output link.. This component, along with the Spark Batch component Palette it belongs to, appears only when you are creating a Spark Batch Job. Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.
Spark Connection	In the Spark Configuration tab in the Run view, define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files: Yarn mode (Yarn client or Yarn cluster): When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab. When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab. When using Altus, specify the S3 bucket or the Azure Data Lake Storage for Job deployment in the Spark configuration tab. When using Qubole, add a tS3Configuration to your Job to write your actual business data in the S3 system with Qubole. Without tS3Configuration, this business data is written in the Qubole HDFS system and destroyed once you shut down your cluster. When using on-premise distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration. Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration or tS3Configuration. If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem). This connection is effective on a per-Job basis.

Usage rule

This component is used as a start component and requires an output
link..

This component, along with the Spark Batch component Palette it belongs to,
appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a
scenario presents only Standard Jobs, that is to
say traditional
Talend
data integration Jobs.

Spark Connection

In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

Yarn mode (Yarn client or Yarn cluster):
- When using Google Dataproc, specify a bucket in the
  Google Storage staging bucket
  field in the Spark configuration
  tab.
- When using HDInsight, specify the blob to be used for Job
  deployment in the Windows Azure Storage
  configuration area in the Spark
  configuration tab.
- When using Altus, specify the S3 bucket or the Azure
  Data Lake Storage for Job deployment in the Spark
  configuration tab.
- When using Qubole, add a
  tS3Configuration to your Job to write
  your actual business data in the S3 system with Qubole. Without
  tS3Configuration, this business data is
  written in the Qubole HDFS system and destroyed once you shut
  down your cluster.
- When using on-premise
  distributions, use the configuration component corresponding
  to the file system your cluster is using. Typically, this
  system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the
configuration component corresponding to the file system your cluster is
using, such as tHDFSConfiguration or
tS3Configuration.

If you are using Databricks without any configuration component present
in your Job, your business data is written directly in DBFS (Databricks
Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Batch version of this component
yet.

tFileInputJSON properties for Apache Spark Streaming

These properties are used to configure tFileInputJSON running in the Spark Streaming Job framework.

The Spark Streaming
tFileInputJSON component belongs to the File family.

This component is available in Talend Real Time Big Data Platform and Talend Data Fabric.

Basic settings

Define a storage configuration component	Select the configuration component to be used to provide the configuration information for the connection to the target file system such as HDFS. If you leave this check box clear, the target file system is the local system. The configuration component to be used must be present in the same Job. For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write the result in a given HDFS system.
Property type	Either Built-In or Repository.
	Built-In: No property data stored centrally.
	Repository: Select the repository file where the properties are stored. The fields that come after are pre-filled in using the fetched data. For further information about the File Json node, see the section about setting up a JSON file schema in Talend Studio User Guide.
Schema et Edit Schema	A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word `line` when naming the fields. Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
	Built-In: You create and store the schema locally for this component only.
	Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Read by	Select a way of extracting the JSON data in the file. Xpath: Extracts the JSON data based on the XPath query. JsonPath: Extracts the JSON data based on the JSONPath query. Note that it is recommended to read the data by JSONPath in order to gain better performance.
Folder/File	Browse to, or enter the path pointing to the data to be used in the file system. If the path you entered points to a folder, all files stored in that folder will be read. If the file to be read is a compressed one, enter the file name with its extension; then tFileInputJSON automatically decompresses it at runtime. The supported compression formats and their corresponding extensions are: DEFLATE: .deflate gzip: .gz bzip2: .bz2 LZO: .lzo The button for browsing does not work with the Spark Local mode; if you are using the other Spark Yarn modes that the Studio supports with your distribution, ensure that you have properly configured the connection in a configuration component in the same Job, such as tHDFSConfiguration . Use the configuration component depending on the filesystem to be used.
Die on error	Select the check box to stop the execution of the Job when an error occurs.
Loop Jsonpath query	Enter the path pointing to the node within the JSON field, on which the loop is based. Note if you have selected Xpath from the Read by drop-down list, the Loop Xpath query field is displayed instead.
Mapping	Complete this table to map the columns defined in the schema to the corresponding JSON nodes. Column: The Column cells are automatically filled with the defined schema column names. Json query/JSONPath query: Specify the JSONPath node that holds the desired data. For more information about JSONPath expressions, see http://goessner.net/articles/JsonPath/. This column is available only when JsonPath is selected from the Read By list. XPath query: Specify the XPath node that holds the desired data. This column is available only when Xpath is selected from the Read By list. Get Nodes: Select this check box to extract the JSON data of all the nodes or select the check box next to a specific node to extract the data of that node. This column is available only when Xpath is selected from the Read By list.

Advanced settings

Set minimum partitions	Select this check box to control the number of partitions to be created from the input data over the default partitioning behavior of Spark. In the displayed field, enter, without quotation marks, the minimum number of partitions you want to obtain. When you want to control the partition number, you can generally set at least as many partitions as the number of executors for parallelism, while bearing in mind the available memory and the data transfer pressure on your network.
Advanced separator (for number)	Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).
Encoding	You may encounter encoding issues when you process the stored data. In that situation, select this check box to display the Encoding list. Select the encoding from the list or select Custom and define it manually.

Set minimum partitions

Select this check box to control the number of partitions to be created from the input
data over the default partitioning behavior of Spark.

In the displayed field, enter, without quotation marks, the minimum number of partitions
you want to obtain.

Advanced separator (for number)

Select this check box to change the separator used for numbers. By default, the thousands separator is a comma (,) and the decimal separator is a period (.).

Encoding

You may encounter encoding issues when you process the stored data. In that
situation, select this check box to display the Encoding list.

Select the encoding from the list or select Custom and define it manually.

Usage

Usage rule	This component is used as a start component and requires an output link. This component is only used to provide the lookup flow (the right side of a join operation) to the main flow of a tMap component. In this situation, the lookup model used by this tMap must be Load once. This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job. Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.
Spark Connection	In the Spark Configuration tab in the Run view, define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files: Yarn mode (Yarn client or Yarn cluster): When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab. When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab. When using Altus, specify the S3 bucket or the Azure Data Lake Storage for Job deployment in the Spark configuration tab. When using Qubole, add a tS3Configuration to your Job to write your actual business data in the S3 system with Qubole. Without tS3Configuration, this business data is written in the Qubole HDFS system and destroyed once you shut down your cluster. When using on-premise distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration. Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration or tS3Configuration. If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem). This connection is effective on a per-Job basis.

Usage rule

This component is used as a start component and requires an output link.

This component is only used to provide the lookup flow (the right side of a join
operation) to the main flow of a tMap component. In this
situation, the lookup model used by this tMap must be
Load once.

This component, along with the Spark Streaming component Palette it belongs to, appears
only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents
only Standard Jobs, that is to say traditional
Talend
data
integration Jobs.

Spark Connection

Yarn mode (Yarn client or Yarn cluster):
- When using Google Dataproc, specify a bucket in the
  Google Storage staging bucket
  field in the Spark configuration
  tab.
- When using HDInsight, specify the blob to be used for Job
  deployment in the Windows Azure Storage
  configuration area in the Spark
  configuration tab.
- When using Altus, specify the S3 bucket or the Azure
  Data Lake Storage for Job deployment in the Spark
  configuration tab.
- When using Qubole, add a
  tS3Configuration to your Job to write
  your actual business data in the S3 system with Qubole. Without
  tS3Configuration, this business data is
  written in the Qubole HDFS system and destroyed once you shut
  down your cluster.
- When using on-premise
  distributions, use the configuration component corresponding
  to the file system your cluster is using. Typically, this
  system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the
configuration component corresponding to the file system your cluster is
using, such as tHDFSConfiguration or
tS3Configuration.

If you are using Databricks without any configuration component present
in your Job, your business data is written directly in DBFS (Databricks
Filesystem).

This connection is effective on a per-Job basis.

Related scenarios

No scenario is available for the Spark Streaming version of this component
yet.

Document get from Talend https://help.talend.com

Thank you for watching.

Docs 7.x

0 Comments

Inline Feedbacks

View all comments

tFileInputJSON – Docs for ESB 7.x

tFileInputJSON

tFileInputJSON Standard properties

Basic settings

Advanced settings

Global Variables

Usage

Extracting JSON data from a file using JSONPath without setting a loop
node

Adding and linking the components

Configuring the components

Executing the Job

Extracting JSON data from a file using JSONPath

Procedure

Extracting JSON data from a file using XPath

Procedure

Extracting JSON data from a URL

Adding and linking the components

Configuring the components

Executing the Job

tFileInputJSON MapReduce properties (deprecated)

Basic settings

Advanced settings

Global Variables

Usage

Related scenarios

tFileInputJSON properties for Apache Spark Batch

Basic settings

Advanced settings

Usage

Related scenarios

tFileInputJSON properties for Apache Spark Streaming

Basic settings

Advanced settings

Usage

Related scenarios

My Website Links

Tags