If a node rejoined a paused cluster, the --pause option could be ignored and the rejoining node would be active, not paused. The same could happen on Kubernetes when setting maintenance mode on or off, leaving the cluster in an inconsistent state. This issue has been resolved.

Previously, rejoining a node could be much slower than expected when tables contained many deleted rows, because the rejoining node spent extra time reproducing the empty space they left. This issue has been resolved.

Previously, a node could fail to rejoin the cluster if a table had more than about 33 million rows in one partition, counting space left by deleted rows. The rejoin timed out, and the rejoin snapshot could stay in progress on the other nodes. This also prevented in-service upgrades. This issue has been resolved.

On Kubernetes, if stopping a node was canceled just as the operation was being planned, that node could stop communicating with other nodes in the cluster. This could happen as a result of an error or a request to pause during an in-service upgrade. This issue has been resolved.

If a node failed while partition leaders were being rebalanced, the cluster could move a partition leader onto a node that was still rejoining, resulting in the rejoin never completing. This issue has been resolved.

If a rejoining node failed while it was receiving data, a running node could keep the rejoin snapshot open indefinitely, blocking later rejoins and command log truncation snapshots. The running node now ends the rejoin snapshot within about 10 minutes of the rejoining node failing. This issue has been resolved.

On Kubernetes, if a port scanner or other application connected to port 11235, which VoltDB uses to respond to liveness and readiness probes, the port could be blocked. As a result, the pod could not respond and Kubernetes would attempt to restart the healthy pod. This issue has been resolved and the pod disconnects any connections to port 11235 that do not respond after five seconds.

Previously, once a table's data in a single partition exceeded 4 GB, the TUPLE_DATA_MEMORY column of @Statistics TABLE and the TUPLEDATA column of @Statistics MEMORY reported a value much smaller than the actual memory in use. This issue has been resolved.

Previously, if a bulk load into a replicated table (for example, using csvloader) included a row that violated a NOT NULL constraint, the rejected row could remain in the table. The row was counted by queries but missing from the table's indexes, and a later TRUNCATE TABLE could crash the node or a node rejoin could fail. This issue has been resolved.

Previously, each insert that failed because of a NOT NULL constraint violation or a DR buffer overflow could lower the STRING_DATA_MEMORY column of @Statistics TABLE and the STRINGMEMORY column of @Statistics MEMORY. Over time these values under-reported string memory and could even become negative. This issue has been resolved.

Previously, if a user snapshot was saved into a subdirectory of the configured snapshots or command log snapshot directory (for example, voltadmin save /path/to/snapshots/mydir mynonce), a subsequent recover could select that snapshot as the most recent one and then fail to load it, reporting "No snapshot related digests files found" and stopping the cluster before the command logs were replayed. This issue has been resolved.

Previously, if the nodes of a cluster could not agree on the state of database replication when a node rejoined the cluster, the rejoining node could wait indefinitely while starting XDCR. A rejoining node now waits only while the agreement is making progress. If there is no progress for 60 seconds, the rejoin is cancelled.

There was an issue where, if a cluster stops abruptly while writing a command log snapshot and then recovers using that snapshot, older snapshots were not cleaned up correctly. Some older snapshots were left and the newest complete snapshot could be deleted, leaving the cluster unable to recover. This issue has been resolved. Snapshots actively being written and the latest complete snapshot are never deleted in this situation.

Previously, when a cluster joined an existing XDCR mesh of two or more clusters, it was possible for individual nodes on the joining cluster to request a synchronization snapshot from different clusters in the XDCR mesh. On VoltDB V13.3.18 and earlier, this discrepancy could result in the XDCR producer stalling indefinitely, blocking replication between the other clusters in the mesh. On VoltDB V14.3.7 and earlier, it would break the XDCR connections. This issue has been resolved. All nodes of a joining cluster now deterministically select the same cluster as the source of the synchronization snapshot.

Changing the DR consumer queue limit (the dr.consumerlimit maxsize or maxbuffers setting) on a running XDCR cluster reported success but had no effect; the cluster kept using the previous limit until each node was restarted. This issue has been resolved and the new limit now takes effect without a restart.

If the cluster node coordinating a schema change stopped before the change finished, rejoining nodes could time out waiting for the configuration update. The issue has been resolved.

Previously, a node could rejoin the cluster while an elastic expansion (adding nodes) was migrating data, which could cause various issues and threaten stability of the cluster. This issue has been resolved and the rejoining node now waits for elastic expansion to finish.

Under certain race conditions, if a node in an XDCR cluster stopped just as an XDCR consumer from another cluster was connecting to it, that consumer node could temporarily stop applying changes from the producer cluster for up to twice the configured XDCR receive timeout limit. The issue has been resolved.

Previously, if the server hit a fatal error during voltdb init (for example, a configured command log path that could not be created), it exited with only a generic error message and no explanation on the console or in the log. This issue has been resolved: fatal errors during initialization are now reported on the console and in the log, and a directory that cannot be created is named along with the cause.

Previously, the @Statistics XDCR (or XDCR_READINESS) selector could report a consumer state of RECEIVE, a producer state of ACTIVE, and dynamic schema support even when the cluster had no DR consumer or producer at all. This issue has been resolved: those state columns are now null and DYNAMIC_SCHEMA is false when no DR connection exists.

Previously, when a node finished rejoining the cluster, the "Node data recovery completed" log message always reported 0 megabytes transferred at a rate of 0.0 megabytes/sec. This issue has been resolved and the message now shows the actual amount of data streamed to the rejoining node and the transfer rate.

Client latency during an in-service upgrade (ISSU) has been optimized. Previously, when a partition's leadership migrated, a topology-aware Java client could briefly keep routing transactions to the former leader until its next periodic topology refresh. The server redirects these transactions to the correct partition leader, increasing latency and causing occasional timeouts as a consequence. Now, the client updates its topology mapping as soon as it detects a misrouted transaction, reducing latency spikes during leadership migration. This optimization is performed by the Java client, so requires the application be linked to the latest version of the client library before taking effect. Also, a node being stopped for an in-service upgrade or rolling maintenance no longer waits on a synchronous fault-log disk write while its partition leadership moves away, removing a stall during the controlled stop.

When a node stops, if it is a partition leader, leadership is transitioned to one of the remaining nodes. There was a race condition that could stop the new leader from completing the transition. As a result the partition (and any multi-partition transactions) would stall until the next leadership transition. This issue has been resolved.

Previously, the Prometheus metrics endpoint emitted histogram metric families in a format that strict OpenMetrics consumers, such as Google Managed Prometheus, rejected, causing the affected metrics to be dropped on ingestion. This issue has been resolved.

If a cluster is reset with the voltadmin DR RESET command, then the cluster attempts to reestablish an XDCR connection but the synchronization snapshot fails, a second attempt to reset the cluster (DR RESET) and reinitialize would fail and the synchronization never happen. This issue has been resolved.

There was a unique situation where a failed cluster could stall during recovery. This would happen if, just before the cluster failure, a node was rejoining the cluster and the snapshot used to restore the node was being taken while the node rejoined. This issue has been resolved.

Previously, histogram metrics recorded concurrently, such as non-transactional procedure execution times, could report incorrect sample counts, totals, and minimum or maximum values. This issue has been resolved.

There was an issue where an I/O error on the command log could cause the server to deadlock, neither continuing to run properly or shutting down. This issue has been resolved and the server process now exits properly.

Under certain circumstances, if a cluster received XDCR data and a schema change simultaneously, where the new schema conflicts with the contents of the binary log, the XDCR insert was rejected. The problem is that subsequent inserts to that cluster succeeded but were no longer propagated to the other participating clusters, resulting in divergence. This issue has been resolved.

If a node was stopped during a maintenance operation while a command log truncation snapshot was in progress, the server process could fail with a fatal error "Unexpected exception while attempting to create truncation snapshot." This issue is resolved and the node now performs an orderly exit from the cluster as expected.

Previously, when security was enabled, log messages recording schema changes (for example, CREATE TABLE or DROP PROCEDURE) identified the client connection but not the user who made the change. The log message now includes the name of the authenticated user who issued the schema update.

During a node rejoin, partition leaders would wait for acknowledgement from the rejoining node before completing write transactions. However, the response was often delayed by contention with snapshot processing. To avoid the resulting latency spike this caused, partition leaders no longer wait on the rejoining node. As a result, optimal transaction latency is maintained during the rejoin and the affected node processes the pending transactions once the snapshot restore is complete.

The internal operation of the voltadmin DR RESET command has changed. It still behaves like before from the user's perspective and issues the same messages. However, the command now avoids potential delays and stalls waiting for other processes on the cluster.

Previously, clock skew measurement was a one-way test that could be impacted by network latency, JVM garbage collection pauses, and scheduling delay. The resulting value could over-report the skew and occasionally trigger spurious "clock skew too high" warnings — or, in rare cases — fail a node join even when the clocks were well synchronized. This measurement now corrects for any network round-trip delay. During an in-service upgrade to the new method, all nodes fall back to the previous measurement behavior until the upgrade is complete.

Client latency during an in-service upgrade (ISSU) has been optimized. Previously, when a partition's leadership migrated, a topology-aware Java client could briefly keep routing transactions to the former leader until its next periodic topology refresh. The server redirects these transactions to the correct partition leader, increasing latency and causing occasional timeouts as a consequence. Now, the client updates its topology mapping as soon as it detects a misrouted transaction, reducing latency spikes during leadership migration. This optimization is performed by the Java client, so requires the application be linked to the latest version of the client library before taking effect.

You can now change the metrics collection interval on a running cluster without restarting it. Previously, any change to the metrics configuration required a cluster restart to take effect. Note that other configuration changes such as enabling or disabling metrics, or changing the buffer size, still require a restart.

VoltDB now supports Multus and similar services that provide access to multiple network interfaces for Kubernetes pods. VoltDB can advertise separate addresses for client and XDCR traffic, configured per connection type (by interface name or CIDR) through the new `network` block in the Volt Operator's Helm chart — and optionally bind each listener to its interface. This feature requires Volt Operator 3.18.0 or later as well as support in the associated VoltDB software version.

There was a situation, usually associated with an extended period of high volume XDCR traffic, where the consuming cluster would incorrectly assume the buffer for inbound transactions was full and would stop queuing further XDCR traffic. Replication would stop until the cluster was restarted. This issue has been resolved.

During an in-service upgrade, under certain circumstances, a call to @Statistics DRROLE could complete without returning any information. Or, alternatively, DRROLE might report PARTITIONS_MATCH as false when replication partitioning actually matched. This issue has been resolved.

If a node failed while taking a command log truncation snapshot, the snapshot was never reported as complete and therefore blocked any further truncation snapshots. As a result, the remaining nodes stopped truncating their command logs. This issue has been resolved and the interrupted snapshot is now canceled so subsequent snapshots can be scheduled and completed properly.

If a large cluster (A) attempts to join an existing XDCR cluster with fewer nodes (B), not all nodes of cluster A will participate in receiving the initial synchronization snapshot. Previously, the larger cluster would break replication at the end of the snapshot, with one of the uninvolved nodes reporting an illegal state exception. This issue has been resolved and XDCR initialization now completes successfully.

There was a race condition where, if a node was stopped (via a voltadmin STOP command, @StopNode call, or as part of an in-service upgrade) while a multi-partition transaction was being processed on that node, the transaction might not be correctly terminated, blocking the queue and resulting in all subsequent multi-partition work (including snapshots) being stalled. This issue has been resolved.

Previously, if nodes failed concurrently and interrupt a multi-partition transaction, the cluster's attempt to repair the transaction would fail. Consequently, all subsequent multi-partition transactions, including snapshots and node rejoins, were blocked. This issue has now been resolved.

The Volt Java client API captures latency statistics. Previously, these statistics included both user transactions and service calls to system procedures the API makes to track server topology changes. During transitions such as node failures, rejoins, or in-service upgrades, these service calls can take longer than usual, resulting in misleadingly high latency statistics while user transactions are completing normally. The Java client now captures statistics for user transactions and service invocations separately to better reflect actual application performance. Note this change is in the Java client library and requires the application be linked against the latest client library to take effect.

There was a separate issue with the Volt Java client that resulted in incorrect procedure statistics. The client had an upper limit for how long it could track a procedure's latency. Any procedure running over that limit resulted in two separate records: one for the limit (10 seconds) and one for any carry over. This behavior resulted in both incorrect averages and maximum values for latency as well as over counting the total number of transactions. This issue has been resolved

Previously, TLS clients attempting to renegotiate or update keys on an existing connection to the database could cause a loop in a network thread, leading to the affected node rejecting new client connections until it was restarted. This issue has been resolved.

Previously, under asynchronous command logging, if the command log disk ran out of space or became unresponsive due to excessive load, transactions would stall indefinitely waiting for the command log to catch up. Two follow-on consequences were that the transaction queues remained blocked even after the disk responded, and any  multi-partition transactions in the queue would stall the entire cluster until the node was restarted. These issues have been resolved. Now, the blocked transaction queues resume as soon as the command log drains. Also if the command log makes no progress for an extended period of time (two minutes by default), the affected node deliberately shuts down so the rest of the cluster can continue. Warnings are logged while execution queues are waiting on the command log, making the disk issues visible in the log.

Previously, if a node hosting a partition leader was terminated abruptly (for example, a Kubernetes pod deletion during a rolling restart), XDCR could transmit a transaction that had not yet been fully committed on the local cluster. The same DR sequence number would then be reused for a different transaction, causing the remote XDCR cluster to stop replication with the error "Master and replica databases do not agree - Replica is ahead of master." This issue has been resolved.

If a voltadmin DR RESET command is issued before XDCR is fully established between two clusters, the command will appear to succeed but communication will not be reset for any local consumer nodes that were still attempting to connect to the remote producers. Those nodes would continue to report their XDCR status as "active". XDCR could not be reestablished between the clusters until the falsely "active" nodes were restarted. This issue has now been resolved and DR RESET now stops any incomplete connections so all nodes can be reset.

While a node was rejoining a K-safe cluster — most notably during an in-service (rolling) upgrade — one or more of the surviving nodes could show a sustained spike in CPU usage that continued after the rejoin had completed. Throughput and latency were unaffected, but the extra CPU could persist until the next membership change. This issue has been resolved.

Previously, if @CancelPrepareStopNode was issued while @PrepareStopNode was still moving partition leaders off the target node, the cancellation did not fully unwind the leader-migration state. Partition leaders were left unbalanced. This issue has been resolved. Cancelling a prepare-stop-node now clears the in-progress migration state and rebalances partition leaders, so subsequent @PrepareStopNode requests are accepted.

Previously, if two stored procedures shared the same SQL statement object under different field names, invoking one of those procedures while metrics were enabled could cause a fatal error that stopped all copies of the affected partition at once, making the cluster unviable. This issue has been resolved. Instead of throwing an error, the metrics system now skips statistics for statements it cannot identify.

Previously, if a snapshot from an older version (version 13 or higher) was restored to a running cluster, once the restore was complete, the cluster would only support the capabilities of the earlier version. In other words, if a feature was added after the earlier version, it was no longer available to the cluster. This situation was the result of a mistaken application of in-service upgrade logic outside of an in-service upgrade, and has now been corrected.

Due to a race condition during a rolling restart or in-service upgrade to an XDCR cluster, it was possible for the cluster to mistakenly stop consuming data from other clusters, thinking there was a mismatch between partitions on the two clusters. This issue has been resolved.

Previously, an application using the Volt Java client (Client2) could encounter a NullPointerException if it requested client statistics or attempted to open a connection after the client had been closed. This issue has been resolved and these operations now complete cleanly.

Previously, after selecting a different server from the server list in the Volt Management Center (VMC), the CPU and memory monitoring charts could continue to show the previously selected server's data, making the per-server graphs appear identical. This issue has been resolved. VMC now keeps a separate, continuously updated history for every server, so switching servers immediately shows that server's own CPU and memory history without gaps.

There was a rare race condition where if a node failed during a schema or configuration update, the cluster could encounter a hash mismatch, causing it to crash. This issue has been resolved.

Previously, different nodes in a cluster could return rows in a different order if the query involved ordering by a non-unique index with duplicate key values. This issue affected physical row order only; no data was lost or returned incorrectly. This issue has been resolved and query results ordered by non-unique indexes are returned in a deterministic way by all nodes in the cluster.

Previously, collecting the IOSTATS and TLS statistics selectors required the network threads to assemble per-connection counters on demand, blocking the statistics request until each network thread responded. On a busy node with many client connections and frequent statistics polling, these requests could time out — reporting "timed out retrieving stats from network thread" in the log file and returning an error to the client. This issue has been resolved. The network threads now publish these counters to a cache at a regular interval, which the statistics request reads, eliminating the wait for the network threads. IOSTATS and TLS statistics collection no longer times out under load.

There was a rare race condition during rolling restart that could cause replication to fail with the error "DRAGENT: Terminating replication: null java.util.NoSuchElementException". This issue has been resolved.

In version 14.2.0 the snapshot buffer size was increased to 1.75MB to reduce the time required to complete a snapshot. However, the time each individual snapshot task took to process the larger buffer caused a noticeable increase in the latency of user transactions during the snapshot. Consequently, the buffer size has been reset to its original size (512K) to achieve a balance between transaction latency and overall snapshot duration.

Messages reporting that a procedure or query is taking a long time to execute were previously always logged at the INFO level; they are now raised to WARN once the operation has been running for more than two seconds, making prolonged operations easier to detect with monitoring tools.

Previously, the Volt Management Console (VMC) did not always change the information displayed after selecting a different server from the server list. This issue has been resolved.

In certain situations, performing a DR RESET would fail to restart the XDCR producer for the reset cluster. So the cluster could receive XDCR transactions from other clusters, but would not replicate its own transactions to the other participating clusters. This issue has been resolved.

There is a rare case where, during an in-service upgrade if multi-partition transactions are being processed by an XDCR consumer while the partition leader is being moved from one node to another, the transaction might fail with the error NoSuchElementException, breaking replication. This issue has been resolved.

In the database configuration, the memory compaction interval must be greater than zero. Previously, setting the compaction value to zero caused a fatal exception. This restriction is now enforced and reports a meaningful error.

In certain rare cases where TLS/SSL is enabled, a race condition could cause a network thread to stall, resulting in a connection timeout when using sqlcmd. The issue has been resolved.

In the Volt Management Center (VMC) sizing worksheet, changing the number of nodes or the K-factor did not affect the per-node memory estimates. This issue has been resolved and the estimates now correctly account for these settings.

It was possible, during an in-service upgrade, for the node being upgraded to fail with a bogus error stating that migrating partition leadership was blocked by an elastic operation. This issue has been resolved.

In certain situations during an in-service upgrade on Kubernetes, if the Volt operator starts to upgrade a node, but then detects an error, and aborts the upgrade before the node reboots, the cluster could end up in an unbalanced state. The issue is that the operator does not cancel the "stop node" operation on the node currently being upgraded, so it never returns partition leadership to that node, or redirects client transactions back to the node. This issue has been resolved. However, the fix requires updates to VoltDB (this release), the Volt Java client,  and the Volt Operator (version 3.16.0). It also means that the fix does not take effect until performing an in-service upgrade from the new version to a subsequent version.

The @Statistics INITIATOR output includes a new column, FORWARDED, that reports the number of transactions that are forwarded from the node that receives the request to a different node for coordination.

Under some certain race conditions, partition leader migration could be blocked. The issue has been resolved.

In-service upgrades from V13.3.8 though V13.3.12 to previous LTS releases of V14 (that is, V14.3.0 through V14.3.3) would fail with an index out of bounds error. Customers upgrading from these V13 releases should choose V14.3.4 or later as the target for their upgrade.

There is a rare condition where, under an extremely heavy XDCR workload, a DR consumer could stall while processing a multi-partition transaction, resulting in the node no longer consuming binary logs. This is another instance, although even less common,  of a similar bug addressed in 13.3.8. This issue has been resolved.

When running the VMC service on bare metal, setting the --logdir option to the current working directory prevented the vmc.log file from being created. This issue has been resolved.

Previously, client connections were periodically notified of cluster topology changes. Now client connections are notified immediately about partition leader migrations.

The Volt enterprise and VMC containers now use the Red Hat Universal Base Image version 10 (ubi10-minimal). Previous releases used version 9.

A rejoining node did not accept client transactions until the rejoin finished and a truncation snapshot was completed. This has changed. A rejoining node now accepts transactions as soon as the rejoin operation is complete. It no longer waits for the truncation snapshot to finish.

The HTTP/JSON programming interface returns a session cookie that lets the calling program reuse the client connection. A new section in the chapter on Using Other Programming Languages in the Using VoltDB guide explains how to use this cookie to manage client connections for applications that make frequent calls to the HTTP/JSON API.

The file descriptor count in the result of the LIMITS and LIVECLIENTS statistics now reflects the current in-use count, rather than the count seen in the most recent check, which could be up to 10 minutes before the stats call.

VMC session handling has been changed to avoid creating unneeded session objects when VMC cannot connect to VoltDB.

There was a slow heap memory leak in the metrics/statistics tracking of snapshots. This issue has been resolved.

If, during a rejoin, the rejoin snapshot fails for any reason, all subsequent attempts to rejoin the cluster, or to initialize a snapshot on the cluster, will fail. When this happens, the subsequent failures were reported in the log with the message "Unable to remove outdated rejoin snapshot request." This issue has been resolved.

In recent releases, the Volt Management Center (VMC) charts for CPU and memory usage did not display any data. This issue has been resolved.

In rare cases, if multiple read-only multi-partition transactions are executing simultaneously during a cluster topology change (that is, a node failing or rejoining), they could trigger a race condition, causing a deadlock. This issue has been resolved.

Under certain rare conditions during cluster restart, it was possible for a multi-partition transaction being replayed from the command log to generate a null pointer exception. This issue has been resolved.

In rare cases, if a node fails, then fails again while rejoining the cluster but before receiving the synchronization snapshot, any further attempts to rejoin would also fail. This issue has been resolved.

The VMC service container images (voltdb/volt-vmc-svc) for versions 14.3.1 and 13.3.8 only support AMD64 CPUs; ARM64 support was accidentally omitted. This release corrects that omission.

Previously, if a stored procedure queued more than 32,767 statements before calling voltExecuteSQL(), it exceeded the allowable limit of procedure results. This error condition was not caught; instead, the database reported a hash mismatch. The incorrect error message has been resolved. However, the limit on procedure results remains and the database now reports a more meaningful error.

An attempt to set the value of a FLOAT column to '-Infinity' incorrectly converted the value to null. This has been corrected. Note that arithmetic operations that produce a result less than about -1.7E+308 may now evaluate to negative infinity instead of the previous incorrect null result. Volt does not recommend computation with infinities.

There was a race condition when initializing XDCR clusters. In certain rare cases, the initialization could fail when the cluster attempts to process incoming data after receiving the synchronization snapshot but before initialization is complete. This issue has been resolved.

Under certain rare circumstances the MpInitiator stack could overflow when loading stored procedure definitions. This issue has been resolved.

In certain situations, if more than one Volt cluster was started within a single Kubernetes namespace, the second or subsequent Volt clusters could be started with incorrect configuration values for properties such as XDCR, placement groups, and topics. This issue has been resolved.

There was an issue where compaction was inefficient for tables with indexes containing string columns. The compaction algorithm has been changed to correct this situation. Specifically, the maximum relocation count for these tables is now limited to 1 block, regardless of the overall configuration setting. In addition, compaction will only occur if the fragmentation threshold reaches 60% instead of 95%.

The "voltadmin dr reset" command has been optimized to improve performance and reduce the time it takes to complete, particularly on large clusters.

When metrics are enabled, there was a slow memory leak in the Java heap associated with frequent ad hoc read-only multi-partition queries or frequent schema changes. This issue has been resolved.

Under certain rare conditions, XDCR and Export tasks could interfere with each other for extended periods, often resulting in the message "The multipartition procedure {procedure-name} is taking a long time...". This issue has been resolved.

Previously, if a DELETE statement did not contain a WHERE clause, it was treated like a TRUNCATE statement, deleting all data in the table on the current cluster but not creating the binary logs needed to synchronize with other XDCR clusters. This behavior has been corrected so that DELETE always creates the necessary logs to maintain consistency across XDCR clusters.

In certain rare instances, particularly involving frequent multi-partition procedures, XDCR flow control could stall, resulting in the affected cluster no longer processing incoming XDCR data. This issue has been resolved.

When using a cert-manager secret to configure TLS/SSL for Volt, the properties cluster.clusterSpec.ssl.certificateFile and cluster.clusterSpec.ssl.clientCertFile are ignored. There are no plans to remove this limitation.

The "voltadmin status --dr" command is meant to show the status of replication for all cooperating XDCR clusters. However, there are cases where the command cannot retrieve the information from the remote clusters -- most notably, when the remote nodes do not all use the same client and admin port numbers. Previously, when this happened, the command incorrectly displayed the current cluster's information multiple times. This issue has been resolved and the command now detects the problem and only displays information for the clusters it can reach.

There was an issue with handling multiple certificates in a single PEM trust store. This issue has been resolved.

Under certain rare conditions, the metrics subsystem could receive too much data internally, resulting in the illegal state exception "would overflow integer count", causing the server process to fail with the additional warning "encountered an unexpected error and will die." This issue has been resolved.

Previously, after a node stopped and then rejoined the cluster, the "voltadmin resize --test" command would report the original host IDs of the cluster, not the new assignment for the rejoined node. This issue has been resolved.

The use of Java Key Store (JKS) files for managing TLS/SSL encryption is now deprecated in favor of Privacy Enhanced Mail (PEM) format. Although use of JKS is still possible in some cases, the third-party module required to support JKS in Python no longer installs successfully under pip 23.1 and later. And for Java, JKS was deprecated in Java 9. PEM, on the other hand, is supported in the current releases of both languages.

Command logging uses a pool of segment files for managing logs prior to their being written to a snapshot. The segment files are reused to avoid the overhead of deleting and recreating them. However, it is possible for a spike in logging to create additional files that, under normal conditions, are not needed. Now, to avoid unnecessary disk usage, Volt deletes excess segments files if they remain unused for over 24 hours.

A new system procedure, @AdjustLogging, dynamically changes the level of a specified logger in the running cluster. The "voltadmin log4j" command also supports this as an alternative to providing the entire logging configuration in a file.

VoltDB assigns each client an ID internally. That ID is now accessible by the client application through a new method on the Java client API, clientId(), which returns the ID as an integer.

The @QueryStats system procedure and associated querystats directive in sqlcmd are now deprecated and will be removed in a future major release.

When an initialized cluster joins XDCR for the first time, it receives a synchronization snapshot containing the current contents of the XDCR environment. Previously, the restoration of this snapshot was recorded in the command logs. However, this is not necessary, since the joining cluster is not operational until the snapshot is finished. Now, to avoid unnecessary work, command logging does not begin until the snapshot is fully restored.

A new column, PARTITIONS_MATCH, has been added to the DRCONSUMER and DRROLE selectors for the @Statistics system procedure. This column indicates whether the two clusters have the same number of unique partitions or not.

VoltDB now supports mutual TLS (mTLS) and certificate revocation lists (CRLs). See the sections on configuring TLS/SSL encryption in the Using VoltDB and Kubernetes Administrator's Guide for details.

There was an issue with the Meshmonitor tool that prevented Prometheus from scraping metrics data. This issue has been resolved.

Previously, if a function that returns a string value (such as SUBSTRING()) was included as part of an index, the index could fail to find all matching tuples, under certain conditions. The issue has been resolved.

There was a race condition in the Volt Management Console (VMC) where, if security was enabled and a user logged in, clicking on the menu item to log out could give the appearance that the user was logged out, but occasionally still provide access to the database. This issue has been resolved and logging out consistently clears the account credentials from the browser cache.

There was an issue where, in certain unique cases, an index containing a function could result in the database crashing when the index is accessed at runtime. This issue has been resolved.

When using XDCR between heterogeneous clusters (that is, clusters with a differing number of unique partitions), it was possible for the joining cluster to run out of heap memory while processing the initial synchronizing snapshot. This issue has been resolved.

Previously, enabling TLS/SSL encryption required a combination of JKS and PEM files. Starting with VoltDB V14.2.0 and Volt Operator 3.7.0 it is now possible to set up TLS/SSL using only PEM files. To create the necessary PEM files for a self-signed certificate, you can use the openssl command. Then everywhere where you specify a certificate or key in the server configuration, operator configuration, client, or command line utilities, you can use the corresponding PEM file. (On Kubernetes, the resulting configuration will save the PEM file with a ".jks" extension, but the actual contents are in PEM format. This inconsistency will be corrected in a future release.)

These features will be described in more detail in an upcoming update to the documentation.

A new feature has been added to manage the performance of database snapshots. Autotuning allows the system to dynamically adjust the execution of snapshot tasks based on the current workload. Snapshot autotuning is enabled by setting the deployment.systemsettings.snapshot.autotune.enabled configuration property to true. See the chapter on saving and restoring the database in the Using VoltDB guide for more information.

In addition, the @Statistics system procedure has a new selector, SNAPSHOTAUTOTUNING, that provides feedback on the effect of autotuning.

The resource monitoring configuration is now reported as part of the daily information included when the log file rolls over.

The Volt Kubernetes Diagnostics Pod now includes a network monitoring tool, Mesh Monitor. See the appendix on diagnostic tools in the Volt Kubernetes Administrator's Guide for more information.

The VoltDB Enterprise V14 container for Kubernetes now uses a Java 21 runtime.

The Kinesis importer is being deprecated and has been removed from the VoltDB docker image on Kubernetes because the version of the Kinesis library it uses is no longer supported.

The online help for the Volt Management Console (VMC) has been rewritten to provide better guidance on the use of VMC with TLS/SSL encryption.

Openshift 4.17 has been tested and qualified as a Kubernetes base platform for VoltDB.

When nodes leave or join the cluster, it is possible for command log snapshot files to be left behind. VoltDB now periodically scans the command log snapshot directories and deletes old snapshots that are no longer needed.

Some functions of the Volt Management Console (VMC) failed when running under Java 17 or later. This issue has been resolved.

Under certain circumstances, believed to be related to network instability, the Volt Java clients may start flooding the server with requests for topology information. These requests end up consuming large amounts of heap memory, ultimately causing the system to become unresponsive. This issue has been resolved by limiting both the number of statistics requests the server will accept and the number the clients will send. This means that to receive both parts of the fix, you must update both the server software and java client library.

Sometimes, if the DR_OVERFLOW device ran out of space, all I/O processing was affected, blocking stored procedure execution and in some cases possibly generating multi-partition deadlocks. This issue has been resolved.

In rare situations for XDCR with three or more clusters, issuing the voltadmin DR RESET command without the --force qualifier could leave XDCR replication in an unstable state. One symptom of this situation is files in the dr_overflow directory continue to accumulate and re-issuing the command with the --force qualifier does not clear the condition. This issue has now been resolved. Customers who are not able to upgrade immediately should always use ‘--force’ with the DR RESET command to avoid this issue.

In previous releases, the @SnapshotDelete system procedure returned an empty table, even if snapshots had been deleted. This issue has been resolved.

In rare cases, if there was insufficient memory when Volt attempted to start a snapshot, the database could crash with a SIGSEGV error. This issue has been resolved.

There was an issue where using the command log statistics in a call to the @QueryStats system procedure or querystats directive within sqlcmd could result in a null pointer exception. This was due to the handling of a null value for the last back pressure. This issue has been resolved. Now, if there is no previous back pressure, the value is reported as the lowest possible UNIX time, January 1, 1970.

Under normal operation, it is not possible to convert a non-XDCR cluster into an XDCR cluster without reinitializing the cluster. However, if you manually edit the on-disk configuration file or change the DR role while the database is running in Kubernetes, the next time the cluster restarts, it will try to start as an XDCR cluster. In previous releases, the cluster would fail because the export connectors for XDCR conflict logs are not created. The cluster failure has been resolved and now the cluster starts with a warning. However, the conflict logs are still not set up correctly. The proper corrective action, to ensure the cluster is configured as desired, is to save a snapshot, reinitialize the cluster, and restore the snapshot to the freshly configured cluster.

There was a situation specific to one unique cluster configuration where, after several nodes failed and rejoined a four-node cluster with K=2, partition placement would fail to select the optimal arrangement. This issue has been resolved.

It should be possible to dynamically update the configuration property deployment.dr.exportconficts. However, in V14.1.0, updating this property on a running database had no effect. This issue has been resolved.

There was a race condition in XDCR that could cause a consumer cluster to report that it was "ahead" of the producer or that it received a transaction ID "twice", causing replication to stop. This issue has been resolved.

A recent release fixed an issue to improve a performance regression in certain indexes. Unfortunately, this fix could mistakenly result in a fatal error, “Failed to remove tuple” from an index. This issue has been resolved.

Under certain rare situations, it was possible for a third-party package (Snappy) to fail with a buffer overflow while decompressing data. This issue has been resolved.

Under certain circumstances, if a node starts to rejoin the cluster but then fails, the snapshot used to load data for the rejoining node also fails, blocking any subsequent snapshots. This issue has been resolved.

Several new fields have been added to the DRCONSUMER and DRPRODUCER selectors for the @Statistics system procedure. These fields provide information on both the number of items in the execution queue and the specific tasks at the head of the queue.

SNMP update. The snmp4j library used by Volt for SNMP messaging has been updated to version 3.8.2.

It is now possible to update the license on a running cluster to a new license that adds features, such as XDCR.

The 'engine id' of the SNMP agent is now written to the log file on startup. The id is always 8000137004766f6c746462.

The documentation and examples have been rewritten to emphasize use of Client2, a modern and improved Java API for accessing VoltDB programmatically. Documentation of the original Client API is available in the Upgrade Guide and the javadoc.

It is now possible to disable XDCR conflict logging. Normally, XDCR records any data conflicts to disk. If you do not design your application to avoid conflicts, this can result in excessive disk I/O and storage use. If you do not need the conflict logs, you can turn them off by setting the property deployment.dr.exportconflicts to false. Note, however, if you do disable conflict logging, you lose information about the specific conflicts that do occur. This setting does not affect the counts of conflicts shown in the DRCONFLICTS Statistics.

New YAML configuration available for network settings. There is a new argument to the volt start command, --network, that lets you specify alternate network addresses and ports, such as for the client and admin ports, as a single YAML file. See the description of the VoltDB command in the Using VoltDB manual for details.

An issue was introduced in the V11.4 Java client API that could result in client calls reporting back pressure prematurely. This issue has been resolved. Note this is a client, not a server issue. So client applications must be run against the updated client JAR files to correct the problem.

Under certain circumstances, notifications to DR producer for committed transactions could be missed, which could prevent DR producer from being drained. The issue has been resolved.

If the command logs are corrupted such that certain files are missing, Volt would throw an exception during recovery. This issue has been resolved and Volt now reports a meaningful error if this occurs.

Previously, if during a cluster recovery from command logs the server cannot resolve the path to the command log segments, the process would stop with a null pointer exception. This issue has been resolved. Now the recovery will continue, but because the command log segments cannot be accessed, the completed recovery will be missing any final transactions not in the last command log snapshot.

Previously, the voltdb check command did not report that Java 8 is no longer supported. This issue has been resolved.

There was an issue where attempting to assign a decimal fraction (with a scale larger than the Volt default of 12) to a Volt BIGDECIMAL variable would cause a rounding exception. This issue has been resolved.

When using YAML to configure VoltDB, some properties of the DR connection were incorrectly parsed. The issue is now fixed.

It was possible, in rare cases during an in-service upgrade, that if a node in the cluster failed while another node was performing the upgrade, the upgrading node would fail as well. This issue has been resolved.

Previously, configuring a nodeport for a Kubernetes service definition would use the node IP address if no external address was already defined. In recent releases, the fall back to the node IP address stopped working on OpenShift. This issue has been resolved.

Previously, calling a user-defined function that aggregates on a column that is not part of the GROUP BY clause could result in the server failing with an "unsupported internal operation" exception. This issue has been resolved.

There was a degradation in performance starting in VoltDB V9, specifically for SQL queries using an index where the requested data did not exist within the database. This issue has been resolved.

Under certain situations, when a Volt cluster was shut down, CPU usage of the management console (VMC) connected to that cluster would grow, potentially using all available CPU. This issue has been resolved.

Improvements have been made to the network partition detection for clusters with a K-safety factor of two or more.

There was a rare race condition in XDCR clusters where, if someone performs a schema change while the initial synchronization snapshot is being sent to one of the clusters, it was possible for XDCR communication to fail and replication stop. This issue has been resolved.

Several minor issues related to the new YAML configuration parsing have been corrected, including:

Previously, configuring a nodeport for a Kubernetes service definition would use the node IP address if no external address was already defined. In recent releases, the fall back to the node IP address stopped working on OpenShift. This issue has been resolved.

Embedded Volt Management Center (VMC) and HTTP JSON API

Standalone Prometheus agent

By default, the Client2 API manages connections to the cluster; for example, reconnecting to failed hosts after they rejoin. The configuration option disableConnectionMgmt was provided to disable this behavior, but a bug in previous releases prevented automatic management from actually being disabled. This issue has been resolved.

There was an issue with memory compaction on replica clusters in passive DR. Compaction was not being scheduled on the replica, so memory usage could continually increase, potentially leading to an out of memory error. This issue has been resolved.

Previously, the IS_READY column of the @Statistics system procedure XDCR selector output could give the wrong result and report "false" for all hosts if any of the hosts had a partition that was not active. This issue has been resolved and it now reports false for only those hosts that are affected.

Specify which interface to use for internal and external ports, respectively, using the --internalinterface and --externalinterface arguments when starting the database process with the voltdb start command.

For an individual port, specify the interface and port on the command line. For example voltdb start --client=32.31.30.29:21212.

Java 8 — release 8u32-b05

Java 11 — release 11.0.20+8

Java 17 — release 17.0.8+7