Elasticsearch

Allows the connection to Elasticsearch to automatically publish data to an index, in batch or streaming, with automatic date sub-indexing.

Elasticsearch pipes are write-only, there is no Elasticsearch input.

Useful links:

Common

The following fields are available for all Elasticsearch components:

Name	Mandatory	Description	Example	Default
index	Yes	The Elasticsearch index to write to. A date suffix computed from date_field and sub_index_date_pattern is appended to it.	index = "my-index"
date_field	Yes	The field of the dataset holding the document date, used to build the date suffix of the index.	date_field = "documentDate"
sub_index_date_pattern	No	The date pattern used to build the date suffix of the index.	sub_index_date_pattern = "yyyy.MM.dd"	yyyy.MM
mode	Yes	The writing mode.	mode = "append"
options	No	Elasticsearch options, as key = value pairs. These are passed as options to the Spark connector for Elasticsearch. Note that es.nodes and es.port are mandatory.	options { "es.nodes" = "my-elasticsearch-host.mycompany.net", "es.port" = "9200" }

No batch input is currently available for Elasticsearch in Data I/O.

Type: com.amadeus.dataio.pipes.elasticsearch.batch.ElasticsearchOutput

No streaming input is currently available for Elasticsearch in Data I/O.

Type: com.amadeus.dataio.pipes.elasticsearch.streaming.ElasticsearchOutput

Name	Mandatory	Description	Example
trigger	No	Sets the trigger for the stream query. Can be AvailableNow, Continuous or empty. Controls the trigger() Spark function.	trigger = "AvailableNow"
duration	No	Sets the trigger for the stream query. Controls the trigger() Spark function. In case no trigger is defined, will set a ProcessingTime trigger.	duration = "60 seconds"
timeout	No	Controls the amount of time before returning from the streaming query. In case it is not defined, the query is not awaited.	timeout = "24 hours"