15.14. The Kinesis Importer

Documentation

VoltDB Home » Documentation » Using VoltDB

15.14. The Kinesis Importer

The Kinesis importer connects to the specified Amazon Kinesis stream and writes the records into the VoltDB database. Kinesis streams let you aggregate data from multiple sources, such as click streams and media feeds, which is then pushed as streaming data to the application. The VoltDB Kinesis importer acts as a target application for the Kinesis Stream. The data is decoded according to the specified format — comma-separated values (CSV) by default — and is inserted into the VoltDB database using the specified stored procedure.

When import starts, the importer first checks to make sure the specified stored procedure exists in the database schema. If not (for example, when you first create a database and before a schema is loaded), the importer issues periodic warnings to the console.

Once the specified stored procedure is declared, the importer looks for the specified Kinesis stream. If the stream cannot be found or accessed (for example, if the keys don't match), the importer reports an error and stops. You will need to restart import once this error condition is corrected. You can restart import using any of the following methods:

  • Stop and restart the database

  • Pause and resume the database using the voltadmin pause and voltadmin resume commands

  • Update the configuration using the voltadmin update command or the web-based VoltDB Management Center

If the stream is found and can be accessed, the importer starts fetching data and submitting it to the stored procedure to insert into the database. In the simplest case, you can use the default insert procedure for a table to insert records into a single table. For more complex data you can write your own import stored procedure to interpret the data and insert it into the appropriate table(s).

Table 15.8, “Kinesis Import Properties” lists the allowable properties for the Kinesis importer. You can also specify properties associated with the formatter, as described in Table 15.9, “CSV and TSV Formatter Properties”.

Table 15.8. Kinesis Import Properties

PropertyAllowable ValuesDescription
app.name*stringA user-defined name that is used by Kinesis to track the application's current position in the stream.
procedure*stringThe stored procedure to invoke to insert the incoming data into the database.
region*stringThe Amazon region where the Kinesis stream service is running.
stream.name*stringThe name of the Kinesis stream.
access.key*stringThe Amazon access key for permitting access to the stream.
secret.key*stringThe Amazon secret key for permitting access to the stream.
​max.read.batch.sizeintegerThe maximum number of records to read in a single batch. The default batch size is size 10,000 records.

*Required