15.11. The Elasticsearch Export Connector

Documentation

VoltDB Home » Documentation » Using VoltDB

15.11. The Elasticsearch Export Connector

The Elasticsearch connector receives serialized data from the export streams and inserts it into an Elasticsearch server or cluster. Elasticsearch is an open-source full-text search engine built on top of Apache Lucene™. By exporting selected streams from your VoltDB database to Elasticsearch you can perform extensive full-text searches on the data not possible with VoltDB alone.

Before using the Elasticsearch connector, we recommend reading the Elasticsearch documentation and becoming familiar with the software. The instructions in this section assume a working knowledge of Elasticsearch, its configuration and its capabilities.

The only required property when configuring Elasticsearch is the endpoint, which identifies the location of the Elasticsearch server and what index to use when inserting records into the target system. The structure of the Elasticsearch endpoint is the following:

<protocol>://<server>:<port>//<index-name>//<type-name>

For example, if the target Elasticsearch service is on the server esearch.lan using the default port (9200) and the exported records are being inserted into the employees index as documents of type person, the endpoint declaration would look like this:

<property name="endpoint">
   http://esearch.lan:9200/employees/person
</property>

You can use placeholders in the endpoint that are replaced at runtime with information from the export data, such as the stream name (%t), the partition ID (%p), the export generation (%g), and the date and hour (%d). For example, to use the stream name as the index name, the endpoint might look like the following:

<property name="endpoint">
   http://esearch.lan:9200/%t/person
</property>

When you export to Elasticsearch, the export connector creates the necessary index names and types in Elasticsearch (if they do not already exist) and inserts each record as a separate document with the appropriate metadata. Table 15.6, “Elasticsearch Export Properties” lists the supported properties for the Elasticsearch connector.

Table 15.6. Elasticsearch Export Properties

PropertyAllowable ValuesDescription
endpoint*string

Specifies the root URL of the RESTful interface for the Elasticsearch cluster to which you want to export the data. The endpoint should include the protocol, host name or IP address, port, and path. The path is assumed to include an index name and a type identifier.

The export connector will use the Elasticsearch RESTful API to communicate with the server and insert records into the specified locations. You can use placeholders to replace portions of the endpoint with data from the exported records at runtime, including the stream name (%t), the partition ID (%p), the date and hour (%d), and the export generation (%g).

batch.modetrue, falseSpecifies whether to send multiple rows as a single request or send each export row separately. The default is true.
timezonestringThe time zone to use when formatting timestamps. Specify the time zone as a Java timezone identifier. The default is the local time zone.

*Required