Package org.voltdb.client.topics
Class VoltDBKafkaPartitioner
java.lang.Object
org.apache.kafka.clients.producer.internals.DefaultPartitioner
org.voltdb.client.topics.VoltDBKafkaPartitioner
- All Implemented Interfaces:
Closeable
,AutoCloseable
,org.apache.kafka.clients.producer.Partitioner
,org.apache.kafka.common.Configurable
public class VoltDBKafkaPartitioner
extends org.apache.kafka.clients.producer.internals.DefaultPartitioner
VoltDBKafkaPartitioner is a partitioner to calculate VoltDB partition id from the key value.
Here is a simple example of using the producer to send records with VoltDBKafkaPartitioner
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("bootstrap.servers.voltdb", "localhost:21212");
props.put("enable.idempotence", "false");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, VoltDBKafkaPartitioner.class.getName());
Producer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 100; i++) {
producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));
}
producer.close();
The bootstrap.servers.voltdb
is required to calculate partition id from the key.
Kafka client properties used by the partitioner
ProducerConfig.BOOTSTRAP_SERVERS_CONFIG
- Ifbootstrap.servers.voltdb
is not setCommonClientConfigs.SECURITY_PROTOCOL_CONFIG
SaslConfigs.SASL_MECHANISM
- Must bePLAIN
SaslConfigs.SASL_JAAS_CONFIG
SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG
SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG
-
Field Summary
Modifier and TypeFieldDescriptionstatic final String
Configuration for a VoltDB client to connect to VoltDB cluster: comma separated list of the form server[:port]protected org.voltdb.client.ClientImpl
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
void
protected ClientConfig
createClientConfig
(org.voltdb.client.topics.VoltDBKafkaPartitioner.PartitionConfig configs) Create ClientConfig for client connection to VoltDB clusterprotected void
int
partition
(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, org.apache.kafka.common.Cluster cluster) Use DefaultPartitioner for opaque topics, otherwise use VoltDB hash mechanism for partition calculation.Methods inherited from class org.apache.kafka.clients.producer.internals.DefaultPartitioner
onNewBatch, partition
-
Field Details
-
BOOTSTRAP_SERVERS_VOLTDB
Configuration for a VoltDB client to connect to VoltDB cluster: comma separated list of the form server[:port]
- See Also:
-
m_client
protected org.voltdb.client.ClientImpl m_client
-
-
Constructor Details
-
VoltDBKafkaPartitioner
public VoltDBKafkaPartitioner()
-
-
Method Details
-
configure
- Specified by:
configure
in interfaceorg.apache.kafka.common.Configurable
- Overrides:
configure
in classorg.apache.kafka.clients.producer.internals.DefaultPartitioner
-
partition
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, org.apache.kafka.common.Cluster cluster) Use DefaultPartitioner for opaque topics, otherwise use VoltDB hash mechanism for partition calculation. Under the following corner cases that the partitioner could direct a producer to wrong partitions:- When opaque topics are dropped, then recreated as non-opaque topics, or vice versa
- When a VoltDB cluster is elastically expanded or reduced, the hash mechanism on VoltDB client is not promptly updated.
- Specified by:
partition
in interfaceorg.apache.kafka.clients.producer.Partitioner
- Overrides:
partition
in classorg.apache.kafka.clients.producer.internals.DefaultPartitioner
-
close
public void close()- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Specified by:
close
in interfaceorg.apache.kafka.clients.producer.Partitioner
- Overrides:
close
in classorg.apache.kafka.clients.producer.internals.DefaultPartitioner
-
loadTopics
protected void loadTopics() -
createClientConfig
protected ClientConfig createClientConfig(org.voltdb.client.topics.VoltDBKafkaPartitioner.PartitionConfig configs) Create ClientConfig for client connection to VoltDB cluster
- Parameters:
configs
- Configuration properties from KafkaProducer- Returns:
- ClientConfig
-