Previously, a node could rejoin the cluster while an elastic expansion (adding nodes) was still migrating data. Node rejoin now waits for elastic expansion to finish.

Previously, once a table's data in a single partition exceeded 4 GB, the TUPLE_DATA_MEMORY column of @Statistics TABLE and the TUPLEDATA column of @Statistics MEMORY reported a value much smaller than the actual memory in use. This issue has been resolved.

Previously, if a bulk load into a replicated table (for example, using csvloader) included a row that violated a NOT NULL constraint, the rejected row could remain in the table. The row was counted by queries but missing from the table's indexes, and a later TRUNCATE TABLE could crash the node or a node rejoin could fail. This issue has been resolved.

Previously, each insert that failed because of a NOT NULL constraint violation or a DR buffer overflow could lower the STRING_DATA_MEMORY column of @Statistics TABLE and the STRINGMEMORY column of @Statistics MEMORY. Over time these values under-reported string memory and could even become negative. This issue has been resolved.

Previously, if a user snapshot was saved into a subdirectory of the configured snapshots or command log snapshot directory (for example, voltadmin save /path/to/snapshots/mydir mynonce), a subsequent recover could select that snapshot as the most recent one and then fail to load it, reporting "No snapshot related digests files found" and stopping the cluster before the command logs were replayed. This issue has been resolved.

Previously, if the nodes of a cluster could not agree on the state of database replication when a node rejoined the cluster, the rejoining node could wait indefinitely while starting XDCR. A rejoining node now waits only while the agreement is making progress. If there is no progress for 60 seconds, the rejoin is cancelled.

There as an issue where, if a cluster stops abruptly while writing a command log snapshot and then recovers using that snapshot, older snapshots were not cleaned up correctly. Some older snapshots were left and the newest complete snapshot could be deleted, leaving the cluster unable to recover. This issue has been resolved. Snapshots actively being written and the latest complete snapshot are never deleted in this situation.

Previously, when a cluster joined an existing XDCR mesh of two or more clusters, it was possible for individual nodes on the joining cluster to request a synchronization snapshot from different clusters in the XDCR mesh. On VoltDB V13.3.18 and earlier, this discrepancy could result in the XDCR producer stalling indefinitely, blocking replication between the other clusters in the mesh. On VoltDB V14.3.7 and earlier, it would break the XDCR connections. This issue has been resolved. All nodes of a joining cluster now deterministically select the same cluster as the source of the synchronization snapshot.

Changing the DR consumer queue limit (the dr.consumerlimit maxsize or maxbuffers setting) on a running XDCR cluster, using voltadmin update or, on Kubernetes, helm upgrade, reported success but had no effect: the cluster kept using the previous limit until each node was restarted. This issue has been resolved and the new limit now takes effect without a restart.

If the cluster node coordinating a schema change stopped before the change finished, rejoining nodes could time out waiting for the configuration update. The issue has been resolved.

Under certain race conditions, if a node in an XDCR cluster stopped just as XDCR consumer from another cluster was connecting to it, that consumer node could temporarily stop applying changes from the producer cluster for up to twice the configured XDCR receive timeout limit. The issue has been resolved.

When a node stops, if it is a partition leader, leadership is transitioned to one of the remaining nodes. There was a race condition that could stop the new leader from completing the transition. As a result the partition (and any multi-partition transactions) would stall until the next leadership transition. This issue has been resolved.

Previously, the Prometheus metrics endpoint emitted histogram metric families in a format that strict OpenMetrics consumers, such as Google Managed Prometheus, rejected, causing the affected metrics to be dropped on ingestion. This issue has been resolved.

If a cluster is reset with the voltadmin DR RESET command, then the cluster attempts to reestablish an XDCR connection but the synchronization snapshot fails, a second attempt to reset the cluster (DR RESET) and reinitialize would fail and the synchronization never happen. This issue has been resolved.

There was a unique situation where a failed cluster could stall during recovery. This would happen if, just before the cluster failure, a node was rejoining the cluster and the snapshot used to restore the node was being taken while the node rejoined. This issue has been resolved.

Previously, histogram metrics recorded concurrently, such as non-transactional procedure execution times, could report incorrect sample counts, totals, and minimum or maximum values. This issue has been resolved.

Previously, if the server hit a fatal error during voltdb init (for example, a configured command log path that could not be created), it exited with only a generic error message and no explanation on the console or in the log. This issue has been resolved: fatal errors during initialization are now reported on the console and in the log, and a directory that cannot be created is named along with the cause.

Previously, the @Statistics XDCR (or XDCR_READINESS) selector could report a consumer state of RECEIVE, a producer state of ACTIVE, and dynamic schema support even when the cluster had no DR consumer or producer at all. This issue has been resolved: those state columns are now null and DYNAMIC_SCHEMA is false when no DR connection exists.

There was a rare race condition during rolling restart that could cause replication to fail with the error "DRAGENT: Terminating replication: null java.util.NoSuchElementException". This issue has been resolved.

Previously, when a node finished rejoining the cluster, the "Node data recovery completed" log message always reported 0 megabytes transferred at a rate of 0.0 megabytes/sec. This issue has been resolved and the message now shows the actual amount of data streamed to the rejoining node and the transfer rate.

Previously, if the node coordinating multi-partition XDCR transactions changed while one of those transactions was still being applied, for example during a rolling restart or an in-service upgrade, the consumer cluster could stop replication with the error "DRAGENT: Terminating replication: null" and a java.util.NoSuchElementException. This issue has been resolved.

There was a situation, usually associated with an extended period of high volume XDCR traffic, where the consuming cluster would incorrectly assume the buffer for inbound transactions was full and would stop queuing further XDCR traffic. Replication would stop until the cluster was restarted. This issue has been resolved.

If a node failed while taking a command log truncation snapshot, the snapshot was never reported as complete and therefore blocked any further truncation snapshots. As a result, the remaining nodes stopped truncating their command logs. This issue has been resolved and the interrupted snapshot is now canceled so subsequent snapshots can be scheduled and completed properly.

Under certain circumstances, if a cluster received XDCR data and a schema change simultaneously, where the new schema conflicts with the contents of the binary log, the XDCR insert was rejected. The problem is that subsequent inserts to that cluster succeeded but were no longer propagated to the other participating clusters, resulting in divergence. This issue has been resolved.

If a node was stopped during a maintenance operation while a command log truncation snapshot was in progress, the server process could fail with a fatal error "Unexpected exception while attempting to create truncation snapshot." This issue is resolved and the node now performs an orderly exit from the cluster as expected.

There was a race condition where, if a node was stopped (via a voltadmin STOP command, @StopNode call, or as part of an in-service upgrade) while a multi-partition transaction was being processed on that node, the transaction might not be correctly terminated, blocking the queue and resulting in all subsequent multi-partition work (including snapshots) being stalled. This issue has been resolved.

Previously, if nodes failed concurrently and interrupt a multi-partition transaction, the cluster's attempt to repair the transaction would fail. Consequently, all subsequent multi-partition transactions, including snapshots and node rejoins, were blocked. This issue has now been resolved.

The Volt Java client API captures latency statistics. Previously, these statistics included both user transactions and service calls to system procedures the API makes to track server topology changes. During transitions such as node failures, rejoins, or in-service upgrades, these service calls can take longer than usual, resulting in misleadingly high latency statistics while user transactions are completing normally. The Java client now captures statistics for user transactions and service invocations separately to better reflect actual application performance. Note this change is in the Java client library and requires the application be linked against the latest client library to take effect.

There was a separate issue with the Volt Java client that resulted in incorrect procedure statistics. The client had an upper limit for how long it could track a procedure's latency. Any procedure running over that limit resulted in two separate records: one for the limit (10 seconds) and one for any carry over. This behavior resulted in both incorrect averages and maximum values for latency as well as over counting the total number of transactions. This issue has been resolved

Previously, if a node hosting a partition leader was terminated abruptly (for example, a Kubernetes pod deletion during a rolling restart), XDCR could transmit a transaction that had not yet been fully committed on the local cluster. The same DR sequence number would then be reused for a different transaction, causing the remote XDCR cluster to stop replication with the error "Master and replica databases do not agree - Replica is ahead of master." This issue has been resolved.

Previously, when security was enabled, log messages recording schema changes (for example, CREATE TABLE or DROP PROCEDURE) identified the client connection but not the user who made the change. The log message now includes the name of the authenticated user who issued the schema update.

The internal operation of the voltadmin DR RESET command has changed. It still behaves like before from the user's perspective and issues the same messages. However, the command now avoids potential delays and stalls waiting for other processes on the cluster.

If a large cluster (A) attempts to join an existing XDCR cluster with fewer nodes (B), not all nodes of cluster A will participate in receiving the initial synchronization snapshot. Previously, the larger cluster would break replication at the end of the snapshot, with one of the uninvolved nodes reporting an illegal state exception. This issue has been resolved and XDCR initialization now completes successfully.

Previously, TLS clients attempting to renegotiate or update keys on an existing connection to the database could cause a loop in a network thread, leading to the affected node rejecting new client connections until it was restarted. This issue has been resolved.

Previously, under asynchronous command logging, if the command log disk ran out of space or became unresponsive due to excessive load, transactions would stall indefinitely waiting for the command log to catch up. Two follow-on consequences were that the transaction queues remained blocked even after the disk responded, and any  multi-partition transactions in the queue would stall the entire cluster until the node was restarted. These issues have been resolved. Now, the blocked transaction queues resume as soon as the command log drains. Also if the command log makes no progress for an extended period of time (two minutes by default), the affected node deliberately shuts down so the rest of the cluster can continue. Warnings are logged while execution queues are waiting on the command log, making the disk issues visible in the log.

If a voltadmin DR RESET command is issued before XDCR is fully established between two clusters, the command will appear to succeed but communication will not be reset for any local consumer nodes that were still attempting to connect to the remote producers. Those nodes would continue to report their XDCR status as "active". XDCR could not be reestablished between the clusters until the falsely "active" nodes were restarted. This issue has now been resolved and DR RESET now stops any incomplete connections so all nodes can be reset.

While a node was rejoining a K-safe cluster — most notably during an in-service (rolling) upgrade — one or more of the surviving nodes could show a sustained spike in CPU usage that continued after the rejoin had completed. Throughput and latency were unaffected, but the extra CPU could persist until the next membership change. This issue has been resolved.

Previously, if @CancelPrepareStopNode was issued while @PrepareStopNode was still moving partition leaders off the target node, the cancellation did not fully unwind the leader-migration state. Partition leaders were left unbalanced. This issue has been resolved. Cancelling a prepare-stop-node now clears the in-progress migration state and rebalances partition leaders, so subsequent @PrepareStopNode requests are accepted.

There was a rare race condition where if a node failed during a schema or configuration update, the cluster could encounter a hash mismatch, causing it to crash. This issue has been resolved.

The internal operation of the voltadmin DR RESET command has changed. It still behaves like before from the user's perspective and issues the same messages. However, the command now avoids potential delays and stalls waiting for other processes on the cluster.

Previously, clock skew measurement was a one-way test that could be impacted by network latency, JVM garbage collection pauses, and scheduling delay. The resulting value could over-report the skew and occasionally trigger spurious "clock skew too high" warnings — or, in rare cases — fail a node join even when the clocks were well synchronized. This measurement now corrects for any network round-trip delay. During an in-service upgrade to the new method, all nodes fall back to the previous measurement behavior until the upgrade is complete.

Client latency during an in-service upgrade (ISSU) has been optimized. Previously, when a partition's leadership migrated, a topology-aware Java client could briefly keep routing transactions to the former leader until its next periodic topology refresh. The server redirects these transactions to the correct partition leader, increasing latency and causing occasional timeouts as a consequence. Now, the client updates its topology mapping as soon as it detects a misrouted transaction, reducing latency spikes during leadership migration. This optimization is performed by the Java client, so requires the application be linked to the latest version of the client library before taking effect.

VoltDB now supports Multus and similar services that provide access to multiple network interfaces for Kubernetes pods. VoltDB can advertise separate addresses for client and XDCR traffic, configured per connection type (by interface name or CIDR) through the new `network` block in the Volt Operator's Helm chart — and optionally bind each listener to its interface. This feature requires Volt Operator 3.18.0 or later as well as support in the associated VoltDB software version.

Messages reporting that a procedure or query is taking a long time to execute were previously always logged at the informational (INFO) level. These messages are now raised to warning (WARN) once the operation has been running for more than two seconds, making prolonged operations easier to detect with monitoring tools.

Previously, if two stored procedures shared the same SQL statement object under different field names, invoking one of those procedures while metrics were enabled could cause a fatal error that stopped all copies of the affected partition at once, making the cluster unviable. This issue has been resolved. Instead of throwing an error, the metrics system now skips statistics for statements it cannot identify.

Previously, if a snapshot from an older version (version 13 or higher) was restored to a running cluster, once the restore was complete, the cluster would only support the capabilities of the earlier version. In other words, if a feature was added after the earlier version, it was no longer available to the cluster. This situation was the result of a mistaken application of in-service upgrade logic outside of an in-service upgrade, and has now been corrected.

Due to a race condition during a rolling restart or in-service upgrade to an XDCR cluster, it was possible for the cluster to mistakenly stop consuming data from other clusters, thinking there was a mismatch between partitions on the two clusters. This issue has been resolved.

Previously, an application using the Volt Java client (Client2) could encounter a NullPointerException if it requested client statistics or attempted to open a connection after the client had been closed. This issue has been resolved and these operations now complete cleanly.

Previously, after selecting a different server from the server list in the Volt Management Center (VMC), the CPU and memory monitoring charts could continue to show the previously selected server's data, making the per-server graphs appear identical. This issue has been resolved. VMC now keeps a separate, continuously updated history for every server, so switching servers immediately shows that server's own CPU and memory history without gaps.

Previously, different nodes in a cluster could return rows in a different order if the query involved ordering by a non-unique index with duplicate key values. This issue affected physical row order only; no data was lost or returned incorrectly. This issue has been resolved and query results ordered by non-unique indexes are returned in a deterministic way by all nodes in the cluster.

Previously, collecting the IOSTATS and TLS statistics selectors required the network threads to assemble per-connection counters on demand, blocking the statistics request until each network thread responded. On a busy node with many client connections and frequent statistics polling, these requests could time out — reporting "timed out retrieving stats from network thread" in the log file and returning an error to the client. This issue has been resolved. The network threads now publish these counters to a cache at a regular interval, which the statistics request reads, eliminating the wait for the network threads. IOSTATS and TLS statistics collection no longer times out under load.

In certain situations, performing a DR RESET would fail to restart the XDCR producer for the reset cluster. So the cluster could receive XDCR transactions from other clusters, but would not replicate its own transactions to the other participating clusters. This issue has been resolved.

There is a rare case where, during an in-service upgrade if multi-partition transactions are being processed by an XDCR consumer while the partition leader is being moved from one node to another, the transaction might fail with the error NoSuchElementException, breaking replication. This issue has been resolved.

In certain rare cases, during a node shut down, the command log process could throw an alarming although ultimately harmless error that the Java printStackTrace method could not be invoked. This issue has been resolved.

Previously, if the VMC HTTP service was disabled in deployment.xml, starting the database with the --http option could still open the HTTP port. This issue has been resolved.

In the Volt Management Center (VMC), selecting a server in the "Servers" view could fail because VMC attempted to connect directly to the selected pod IP address. This issue has been resolved.

In the database configuration, the memory compaction interval must be greater than zero. Previously, setting the compaction value to zero caused a fatal exception. This restriction is now enforced and reports a meaningful error.

In certain rare cases where TLS/SSL is enabled, a race condition could cause a network thread to stall, resulting in a connection timeout when using sqlcmd. The issue has been resolved.

In the Volt Management Center (VMC) sizing worksheet, changing the number of nodes or the K-factor did not affect the per-node memory estimates. This issue has been resolved and the estimates now correctly account for these settings.

It was possible, during an in-service upgrade, for the node being upgraded to fail with a bogus error stating that migrating partition leadership was blocked by an elastic operation. This issue has been resolved.

In certain situations during an in-service upgrade on Kubernetes, if the Volt operator starts to upgrade a node, but then detects an error, and aborts the upgrade before the node reboots, the cluster could end up in an unbalanced state. The issue is that the operator does not cancel the "stop node" operation on the node currently being upgraded, so it never returns partition leadership to that node, or redirects client transactions back to the node.

This issue has been resolved. However, the fix requires updates to VoltDB (this release), the Volt Java client,  and the Volt Operator (version 3.16.0). It also means that the fix does not take effect until performing an in-service upgrade from the new version to a subsequent version.

The Volt enterprise, prometheus agent, and VMC containers now use the Red Hat Universal Base Image version 10 (ubi10-minimal). Previous releases used version 9.

There is a rare condition where, under an extremely heavy XDCR workload, a DR consumer could stall while processing a multi-partition transaction, resulting in the node no longer consuming binary logs. This is another instance, although even less common,  of a similar bug addressed in 13.3.8. This issue has been resolved.

When running the VMC service on bare metal, setting the --logdir option to the current working directory prevented the vmc.log file from being created. This issue has been resolved.

A new column, FORWARDED, was added to statistics INITIATOR, to track the number of transactions clients send to a host which forwards to other hosts for coordination.

Previously, client connections were periodically notified of cluster topology changes. Now client connections are notified immediately about partition leader migrations.

A rejoining node did not accept client transactions until the rejoin finished and a truncation snapshot was completed. As a result, client applications could see elevated latencies for this period. This has changed. A rejoining node now accepts transactions as soon as the rejoin operation is complete. It no longer waits for the truncation snapshot to finish.

Previously, the metric voltdb_procedure_forward_total produced an incorrect count. This issue has been resolved and the count for this metric is now accurate.

There was a rare race condition during rolling restart that could cause replication to fail with the error "DRAGENT: Terminating replication: null java.util.NoSuchElementException". This issue has been resolved.

In rare circumstances, a Volt node shutting down with a FATAL error could exit before flushing command log buffers to disk. While the number of transactions held in memory was generally small, all exit conditions now write to disk before the process stops.

The HTTP/JSON programming interface returns a session cookie that lets the calling program reuse the client connection. A new section in the chapter on Using Other Programming Languages in the Using VoltDB guide explains how to use this cookie to manage client connections for applications that make frequent calls to the HTTP/JSON API.

The file descriptor count in the result of the LIMITS and LIVECLIENTS statistics now reflects the current in-use count, rather than the count seen in the most recent check, which could be up to 10 minutes before the stats call.

When an initialized cluster joins XDCR for the first time, it receives a synchronization snapshot containing the current contents of the XDCR environment. Previously, the restoration of this snapshot was recorded in the command logs. However, this is not necessary, since the joining cluster is not operational until the snapshot is finished. Now, to avoid unnecessary work, command logging does not begin until the snapshot is fully restored.

VMC session handling has been changed to avoid creating unneeded session objects when VMC cannot connect to VoltDB.

There was a slow heap memory leak in the metrics/statistics tracking of snapshots. This issue has been resolved.

If, during a rejoin, the rejoin snapshot fails for any reason, all subsequent attempts to rejoin the cluster, or to initialize a snapshot on the cluster, will fail. When this happens, the subsequent failures were reported in the log with the message "Unable to remove outdated rejoin snapshot request." This issue has been resolved.

In rare cases, if multiple read-only multi-partition transactions are executing simultaneously during a cluster topology change (that is, a node failing or rejoining), they could trigger a race condition, causing a deadlock. This issue has been resolved.

Under certain rare conditions during cluster restart, it was possible for a multi-partition transaction being replayed from the command log to generate a null pointer exception. This issue has been resolved.

Attempting to shutdown a single cluster node while under a heavy workload could result in excessive log messages reporting "Incoming transactions after housekeeping started". These messages could, in turn, impact transaction latency. This issue has been resolved.

In rare cases, if a node fails, then fails again while rejoining the cluster but before receiving the synchronization snapshot, any further attempts to rejoin would also fail. This issue has been resolved.

The VMC service container images (voltdb/volt-vmc-svc) for versions 14.3.1 and 13.3.8 only support AMD64 CPUs; ARM64 support was accidentally omitted. This release corrects that omission.

Previously, if a stored procedure queued more than 32,767 statements before calling voltExecuteSQL(), it exceeded the allowable limit of procedure results. However, this error condition was not caught; instead, the database reported a hash mismatch. The incorrect error message has been resolved. However, the limit on procedure results remains and the database now reports a more meaningful error.

There was a race condition when initializing XDCR clusters. In certain rare cases, the initialization could fail when the cluster attempts to process incoming data after receiving the synchronization snapshot but before initialization is complete. This issue has been resolved.

Under certain rare circumstances the MpInitiator stack could overflow when loading stored procedure definitions. This issue has been resolved.

In certain situations, if more than one Volt cluster was started within a single Kubernetes namespace, the second or subsequent Volt clusters could be started with incorrect configuration values for properties such as XDCR, placement groups, and topics. This issue has been resolved.

There was an issue where compaction was inefficient for tables with indexes containing string columns. The compaction algorithm has been changed to correct this situation. Specifically, the maximum relocation count for these tables is now limited to 1 block, regardless of the overall configuration setting. In addition, compaction will only occur if the fragmentation threshold reaches 60% instead of 95%.

When metrics are enabled, there was a slow memory leak in the Java heap associated with frequent ad hoc read-only multi-partition queries or frequent schema changes. This issue has been resolved.

Under certain rare conditions, XDCR and Export tasks could interfere with each other for extended periods, often resulting in the message "The multipartition procedure {procedure-name} is taking a long time..." This issue has been resolved.

This release addresses additional cases where the "voltadmin resize --test" command could return the wrong host ID.

Previously, if a DELETE statement did not contain a WHERE clause, it was treated like a TRUNCATE statement, deleting all data in the table on the current cluster but not creating the binary logs needed to synchronize with other XDCR clusters. This behavior has been corrected so that DELETE always creates the necessary logs to maintain consistency across XDCR clusters.

In certain rare instances, particularly involving frequent multi-partition procedures, XDCR flow control could stall, resulting in the affected cluster no longer processing incoming XDCR data. This issue has been resolved.

The "voltadmin dr reset" command has been optimized to improve performance and reduce the time it takes to complete, particularly on large clusters.

The "voltadmin status --dr" command is meant to show the status of replication for all cooperating XDCR clusters. However, there are cases where the command cannot retrieve the information from the remote clusters -- most notably, when the remote nodes do not all use the same client and admin port numbers. Previously, when this happened, the command incorrectly displayed the current cluster's information multiple times. This issue has been resolved and the command now detects the problem and only displays information for the clusters it can reach.

Under certain rare conditions, the metrics subsystem could receive too much data internally, resulting in the illegal state exception "would overflow integer count", causing the server process to fail with the additional warning "encountered an unexpected error and will die."

Previously, after a node stopped and then rejoined the cluster, the "voltadmin resize --test" command would report the original host IDs of the cluster, not the new assignment for the rejoined node. This issue has been resolved.

There was an issue with the Meshmonitor tool that prevented Prometheus from scraping metrics data from the Diagnostics pod. This issue has been resolved.

Previously, if a function that returns a string value (such as SUBSTRING()) was included as part of an index, the index could fail to find all matching tuples, under certain conditions. The issue has been resolved.

VoltDB assigns each client an ID internally. That ID is now accessible by the client application through a new method on the Java client API, clientId(), which returns the ID as an integer.

This release provides support on Kubernetes for Mesh Monitor, a network monitoring tool recently added to the Volt Diagnostics Pod. See the appendix on diagnostic tools in the Volt Kubernetes Administrator's Guide for more information.

The Kinesis importer has been deprecated and removed from the VoltDB docker image on Kubernetes because the version of the Kinesis library it uses is no longer supported. As a result, the security vulnerability CVE-2024-7254 is no longer present.

There was a race condition in the Volt Management Console (VMC) where, if security was enabled and a user logged in, clicking on the menu item to log out could give the appearance that the user was logged out, but occasionally still provide access to the database. This issue has been resolved and logging out consistently clears the account credentials from the browser cache.

There was an issue where, in certain unique cases, an index containing a function could result in the database crashing when the index is accessed at runtime. This issue has been resolved.

When using XDCR between heterogeneous clusters (that is, clusters with a differing number of unique partitions), it was possible for the joining cluster to encounter an out of memory error while processing the initial synchronizing snapshot. This issue has been resolved.

When nodes leave or join the cluster, it is possible for command log snapshot files to be left behind. VoltDB now periodically scans the command log snapshot directories and deletes old snapshots that are no longer needed.

Previously, if XDCR replication failed due to an illegal argument exception, attempts to reset the cluster with voltadmin dr reset would also fail. This issue has been resolved.

Some functions of the Volt Management Console (VMC) failed when running under Java 17 or later. This issue has been resolved.

There were cases where a cluster establishing an XDCR connection starts generating repeated warnings saying "Can not cover ExpectedNextDRId...". This warning was spurious and despite what it might look like, there was no problem with the cluster. This issue has been corrected and the misleading messages no longer appear.

Under certain circumstances, believed to be related to network instability, the Volt Java clients may start flooding the server with requests for topology information. These requests end up consuming large amounts of heap memory, ultimately causing the system to become unresponsive. This issue has been resolved by limiting both the number of statistics requests the server will accept and the number the clients will send. This means that to receive both parts of the fix, you must update both the server software and java client library.

Sometimes, if the DR_OVERFLOW device ran out of space, all I/O processing was affected, blocking stored procedure execution and in some cases possibly generating multi-partition deadlocks. This issue has been resolved.

In rare situations for XDCR with three or more clusters, issuing the voltadmin DR RESET command without the --force qualifier could leave XDCR replication in an unstable state. One symptom of this situation is files in the dr_overflow directory continue to accumulate and re-issuing the command with the --force qualifier does not clear the condition. This issue has now been resolved. Customers who are not able to upgrade immediately should always use ‘--force’ with the DR RESET command to avoid this issue.

In previous releases, the @SnapshotDelete system procedure returned an empty table, even if snapshots had been deleted. This issue has been resolved.

In rare cases, if there was insufficient memory when Volt attempted to start a snapshot, the database could crash with a SIGSEGV error. This issue has been resolved.

There was an issue where using the command log statistics in a call to the @QueryStats system procedure or querystats directive within sqlcmd could result in a null pointer exception. This was due to the handling of a null value for the last back pressure. This issue has been resolved. Now, if there is no previous back pressure, the value is reported as the lowest possible UNIX time, January 1, 1970.

The resource monitoring configuration is now reported as part of the daily information included when the log file rolls over.

Several new fields have been added to the DRCONSUMER and DRPRODUCER selectors for the @Statistics system procedure. These fields provide information on both the number of items in the execution queue and the specific tasks at the head of the queue.

It is now possible to update the license on a running cluster to a new license that adds features, such as XDCR.

In rare cases, if there was insufficient memory when Volt attempted to start a snapshot, the database could crash with a SISEGV error. This issue has been resolved.

There was a race condition in XDCR that could cause a consumer cluster to report that it was "ahead" of the producer or that it received a transaction ID "twice", causing replication to stop. This issue has been resolved.

A recent release (13.3.3) fixed an issue to improve a performance regression in certain indexes. Unfortunately, this fix could mistakenly result in a fatal error, “Failed to remove tuple” from an index. This issue has been resolved.

Under certain rare situations, it was possible for a third-party package (Snappy) to fail with a buffer overflow while decompressing data. This issue has been resolved.

An issue was introduced in the V11.4 Java client API that could result in client calls reporting back pressure prematurely. This issue has been resolved. Note this is a client, not a server issue. So client applications must be run against the updated client JAR files to correct the problem.

Under certain circumstances, if a node starts to rejoin the cluster but then fails, the snapshot used to load data for the rejoining node also fails, blocking any subsequent snapshots. This issue has been resolved.

Under certain circumstances, notifications to the XDCR producer for committed transactions could be missed, which prevents the producer from draining properly. The issue has been resolved.

Attempting to recover an empty database (with no schema defined) from command logs could cause the recovery to hang. This issue has been resolved.

Improvements have been made to the network partition detection for clusters with a K-safety factor of two or more.

Previously, configuring a nodeport for a Kubernetes service definition would use the node IP address if no external address was already defined. In recent releases, the fall back to the node IP address stopped working on OpenShift. This issue has been resolved.

There was an issue with memory compaction on replica clusters in passive DR. Compaction was not being scheduled on the replica, so memory usage could continually increase, potentially leading to an out of memory error. This issue has been resolved.

There was a degradation in performance starting in VoltDB V9, specifically for SQL queries using an index where the requested data did not exist within the database. This issue has been resolved.

It was possible, in rare cases during an in-service upgrade, that if a node in the cluster failed while another node was performing the upgrade, the upgrading node would fail as well. This issue has been resolved.

Previously, calling a user-defined function that aggregates on a column that is not part of the GROUP BY clause could result in the server failing with an "unsupported internal operation" exception. This issue has been resolved.

There was an issue where attempting to assign a decimal fraction (with a scale larger than the Volt default of 12) to a Volt BIGDECIMAL variable would cause a rounding exception. This issue has been resolved.

Previously, if during a cluster recovery from command logs the server cannot resolve the path to the command log segments, the process would stop with a null pointer exception. This issue has been resolved. Now the recovery will continue, but because the command log segments cannot be accessed, the completed recovery will be missing any final transactions not in the last command log snapshot.

There was a rare race condition in XDCR clusters where, if someone performs a schema change while the initial synchronization snapshot is being sent to one of the clusters, it was possible for XDCR communication to fail and replication stop. This issue has been resolved.

Previously, there was an issue in Kubernetes where cycling the database cluster pods could cause the VMC service to consume all available CPU. Specifically, if — while the VMC service is processing ongoing HTTP API requests — the cluster nodes are stopped, restarted, then stopped again, while the cluster is stopped API requests to the VMC service cause CPU usage to spike. This issue has been resolved.

There was a situation where, if the procedure call timeout was set to a fraction of a second on the client call, the timeout response could be delayed if it coincided with a topology change in the cluster (that is, a node dropping out of rejoining. This was caused by the timeout and topology tracking running on the same thread. The problem was also specific to the version 1 Java client only. This issue has been resolved.

Under certain conditions, XDCR clusters could generate a large volume of log messages warning of export gaps being recovered. This was caused by two issues: an unusually large number of XDCR conflicts, and a bug in the handling of the conflict log when a cluster node drops and rejoins. In rare cases a race condition could exacerbate the situation and cause nodes to crash.

The internal handling of the conflict log and resulting race condition have been corrected. However, it is important for the integrity of your data to recognize (and correct) the application logic if it is causing frequent and persistent transaction conflicts in an XDCR environment.

The HTTP error codes returned by the VMC and HTTP API service have been improved. For example, when the cluster nodes become unavailable (either due to downtime or network failures) the VMC service now returns HTTP code 503, service unavailable.

The standalone VMC service supports the HTTP JSON API. However, in previous releases, the API returned call results in version 1.0 format for both V1.0 and V2.0 requests. This issue has been resolved and V2 API requests are now returned in V2.0 format.

In Kubernetes, Volt Active Data now supports control group V2 resource management when determining how much memory is available to the pods. Previously, Volt used control group V1, which could return incorrect values on recent releases of Kubernetes that use V2 by default.

There was a problem with the Volt Active Data kits for the Macintosh M1 architecture (aarch64) where the JNI library was in the wrong location. The result was that the server would fail to start reporting that it could not load a shared library. This issue has been resolved.

In earlier releases of Volt Active Data, there was an issue where a schema change during an in-service upgrade from 13.2.x to 13.3.x could cause the upgrade to fail. This issue is resolved in the current releases. However, even though it can succeed, in general initiating a schema change during an in-service upgrade is not recommended.

Due to functionality that was removed from the latest release of Python, earlier releases of the Volt command line utilities voltdb and voltadmin did not work properly under Python 3.12. This issue has been resolved.

The separate VMC and HTTP service uses cookies to maintain session state, which works with web browsers. However, for other HTTP invocations (such as curl or programmatic calls to the HTTP client API) this caused the server to create a new session for each call, leaking memory. This issue has been resolved.

Under certain circumstances, the cluster would report an logging error "failed to determine log rollover time" once a day. When this happened, the usual list of system settings was not added to the daily log. This issue has been resolved.

When TLS/SSL is enabled, the voltdb command needs the Python SSL module installed to communicate with the server, and will report an error if it is not found. However, previously, the error message was obscure and the command incorrectly attempted to load the module when it wasn't needed. Now the SSL module is only loaded when necessary and a meaningful error message is returned if the library is not found.

Previously, the web server used by the Volt Management Console (VMC) reported details concerning the server software version in the HTTP header. This information has been removed from the server messages.

In Kubernetes, under certain conditions, when using an IPv6 address for XDCR, the networking protocols would not be configured correctly, causing the cluster pods to fail with a Java illegal argument exception. This issue has been resolved.

Previously, attempting to create an XDCR environment using IPv6 addresses in the <connection> source attribute would fail because the initial snapshot was never sent to the cooperating clusters. This issue has been resolved.

Volt Active Data changed how procedure timeout conditions are handled in V12.0. However, it turns out this change conflicts with the time limit for logging long-running procedures when it was set to a very small value, such as 20 milliseconds (for example, cluster.config.deployment.systemsettings.procedure.loginfo=20 in Kubernetes or <systemsettings><procedure loginfo="20"/></systemsettings> in the configuration file). The symptom of the problem was frequent, spurious log messages warning of a multi-partition procedure taking too long. This issue has been resolved.

During a rejoin or schema update, the server loads classes associated with the database (such as stored procedures). Previously, if the load failed because Java ran out of memory, the error was caught but the update operation was not completely canceled and the schema could be left in an incomplete state. This issue has been resolved. Now if Java runs out of memory during a rejoin, the rejoin fails. During a schema or configuration update, the update is rejected and the database retains its previous state.

A race condition was discovered in XDCR which, in a few rare cases, could be triggered by a topology change in the cluster such as occurs during a node failure and rejoin, elastic expansion or contraction, or an in-service upgrade. The issue is that it is possible for the cluster to lose a binary log it receives from another XDCR cluster. The possible symptoms are that replication refuses to drain during shutdown or the other cluster ultimately breaks off replication because it never receives confirmation of the lost transaction. This issue has been resolved.

The default XDCR consumer queue size was previously applied inconsistently and would select the maximum number of buffers rather than bytes if both were configured. Setting the queue size in bytes is recommended, so queue size now always defaults to a maximum number of bytes (100MB by default) and the size in bytes takes precedence where both are set.

With SNMP enabled, when a DR overflow limit was exceeded, another SNMP resource limit for export overflow was erroneously cleared. This issue has been resolved.

It was possible, when using the voltadmin stop command and requesting one server to stop another server (for example, voltadmin stop --host=svr1 svr2) for Volt to attempt to use the wrong network address when contacting the first server and the command would fail. This issue has been resolved.

It was possible for Volt metrics to erroneously report a negative value for a partition's idle time. This issue has been resolved.

Performing an in-service upgrade from V13.1.4 to V13.2.2 could result in a null pointer exception on an XDCR cluster. This issue has been resolved.

As a result of the switch from Alpine to UBI as the base image for VoltDB on Kubernetes, some network and productivity tools — such as gzip, nc, netstat, and vim — were not included in the Docker image. These utilities have been added back into the base image.

There was an issue where one specific Kafka client could not publish records to a Volt opaque topic. The cause of the problem was that the topic was reporting an incorrect offset back to the client. This issue has been resolved.

The Java client library JAR released with Volt 13.2.0 is missing a required Java class. This issue has been resolved. This release is a client library update only. If you downloaded the V13.2.0 Java client library, you must update to the latest release.

Ease of Use — You only need to specify the version of VoltDB to use, you no longer need to correlate which operator and chart versions are compatible with which software versions.

Reliability — There is a single operator for all currently supported versions of VoltDB; consequently, fixes to the operator itself automatically apply to all versions of the product.

Performance — the service can be run on a database server or on a separate server, so as not to compete with the other database functions.

Flexibility — the service can be run on systems running Java 8, 11, or later, including most Linux systems and even Microsoft Windows.

Scalability — you can run as many copies of the VMC service as you like. When serving just VMC, one service instance per cluster is usually sufficient. If your application makes significant use of the JSON API, you can run multiple copies for increased capacity and availability.

Reliability — As a separate service, VMC can easily be updated in case of bug fixes or security vulnerabilities without having to update the database software.

Previously, it was not possible to change all of the available flush intervals while the database was running. Only the DR flush interval could be updated dynamically. This issue has been resolved and all flush intervals can now be changed with the voltadmin update command.

In certain rare situations, if a schema change coincides with a node's attempt to rejoin the cluster, a race condition could result in the rejoin request being rejected. This issue has been resolved.

There were situations where Volt failed to allocate memory, but rather than reporting a meaningful error, the server process failed with a segmentation violation (SIGSEGV). The original error is now caught and an appropriate error that sufficient memory could not be allocated is reported.

On rare occasions, if a node dropped out of the cluster, then during the initial rejoin it failed again or was interrupted, subsequent attempts to rejoin the cluster would fail claiming there was a snapshot in progress. Rejoining a new node (or the same node re-initialized) would clear the condition. This issue has been resolved.

There was an issue with Active(N) cross datacenter replication (XDCR), where if a producer node stops and restarts, it could take up to a minute for the consumer to reconnect after the producer comes back up. This issue has been resolved.

In previous releases, if an Active(N) conflict occurs due to a row containing an invalid timestamp, replication would be broken and the server reported an SQL error related to the CAST() function. Now the conflict is recorded correctly in the conflict log and replication continues.

Previously, when removing one cluster from a 3-way XDCR environment, it was possible to drop the cluster from the mesh by setting the DR state to "none". However, subsequent attempts to restart the cluster would result in a warning of a possible mutlipartition transaction deadlock. This issue has been resolved.

Earlier versions of the example Grafana dashboards incorrectly calculated the number of procedure errors, resulting in the dashboard sometimes showing no errors when errors existed. This issue has been resolved.

In recent releases, if the database was configured to use external TLS/SSL encryption but security was not turned on, a Java client trying to connect without the appropriate SSL configuration would generate a stack dump (illegal argument exception) rather than receiving a meaningful error. This also applied to starting sqlcmd without the --ssl argument. This issue has been resolved.

Previously, errors in the filename or path specified in the TLS/SSL configuration were not identified or reported until the database started. Now the paths are tested and any errors reported during the initialization of the database root directory.

In certain situations where a cluster topology change coincides with the execution of a stored procedure, the procedure may end up being restarted. When the interrupted procedure was a run-everywhere procedure intended for all partitions, it was possible for the restart to incorrectly treat the procedure as single-partitioned This issue has been resolved.

In very rare cases, it was possible that after several snapshot restores, a subsequent restore could fail to load all of the data due to a cached index being out of date. This issue has been resolved.

There was an issue related to elastically expanding a cluster where export was enabled or XDCR replication was enabled, even if XDCR was not currently connected to another cluster. If, after the expansion, the cluster performed a stop node action followed by a rejoin, there was a potential conflict over which nodes were controlling which partitions. In the case of export, multiple nodes might attempt to export the same data. In the case of XDCR, attempting to perform an in-service upgrade would fail when the second node being updated would not shutdown. This issue has been resolved.

There was a problem with in-service upgrades where if an invalid image was specified, the upgrade would fail in such a way it could not recover. This issue has been resolved. The invalid image is now detected, the upgrade fails, but the user can simply revert to the previous image specification to roll back to the original software.

There was an issue with Active(N) cross datacenter replication (XDCR), where if a producer node stops and restarts, it could take up to a minute for the consumer to reconnect after the producer comes back up. This effect was amplified during an in-service upgrade of an XDCR cluster, because producer nodes are stopped and restarted in series, in some cases causing XDCR replication to stall for the duration of the upgrade. This issue has been resolved.

In some circumstances, if a pod failed to start with the new version during an in-service upgrade, subsequent attempts to rollback to the previous version would also fail. This issue has been resolved.

Under certain circumstances, it was possible to trigger a small but persistent memory leak in the DR port if the port was configured for TLS/SSL encryption. Specifically, when programs such as port scanners or load balancers repeatedly test the DR port, creating new connections each time, memory usage would grow. This issue has been resolved.

There were situations where a topology-aware client might fail to acknowledge all change announcements, leading to the client not connecting to all available servers. This was most likely to occur with large clusters after the cluster restarts or where the client application starts before the cluster starts.

This issue has been resolved. However, it is important to note that the fix is in the Java client, not the server software. So to resolve this problem your client applications must be run against the latest version (V13.1.2) of the Volt Client Library JAR.

In some circumstances, if a pod failed to start with the new version during an in-service upgrade, subsequent attempts to rollback to the previous version would also fail. This issue has been resolved.

In Kubernetes, a bug in the Go language processor was causing the Volt Operator to mistakenly update pod settings repeatedly. Although normally this would go unnoticed, in the case of services using the Border Gateway Protocol (BGP) -- such as certain load balancers used for negotiating XDCR connections -- the repeated updates could result in network issues, excessive latency, and, ultimately, XDCR connection failures. This issue has been resolved.

As a result of extensive internal testing, a few rare edge cases that could cause node rejoins to fail have been identified and corrected.

The @SystemInformatin OVERVIEW system procedure no longer reports on individual licensed features, but conitnues to report on the license type and expiration date.

The @SystemInformation LICENSE system procedure now reports on all licensed features, not just command logging and database replication.

In recent releases (starting with V12.3), events in the VMC service were not being logged properly under Kubernetes. This issue has been resolved

Volt no longer checks if TCP offload or generic receive offload are disabled. Previously, these settings had to be disabled for VoltDB to start. By eliminating the check, it is possible to run VoltDB in environments, most notably Kubernetes, where these OS features may be enabled.

A number of edge cases related to elastically resizing the database cluster have been detected and corrected, significantly improving the reliability and robustness of elastic operations.

Extra snapshot files manually copied into the snapshot folders in the database root directory could result in nodes failing to rejoin the cluster or the cluster failing to start after a crash or shutdown. This issue has been resolved. However, manually copying additional files into the root directory structure is strongly discouraged and can cause unpredictable behavior, including failures.

Rigorous testing of failure scenarios uncovered a rare condition associated with tables declared with the EXPORT TO TARGET or EXPORT TO TOPIC clause and actively exporting during a sequence of node failures and rejoins. A race condition while nodes are dropping and rejoining could potentially result in the table not queueing all appropriate rows to the associated target. This issue has been resolved.

There was a race condition associated with the TTL (time to live) feature during a rejoin operation. It was possible, under certain limited circumstances, for a TTL operation during a rejoin to cause data to silently diverge. This issue has been resolved.

Another race condition, in command logging this time, could cause the cluster to crash on startup. This condition could only be triggered if the command logs contained a schema update in the binary logs. This issue has been resolved.

There are reports that VoltDB does not operate correctly when the networking environment is configured to use jumbo frames. The symptoms are that TCP packets with MTU>9000 are dropped. However, this is not a Volt-specific issue. When configuring the network to use jumbo frames, it is crucial to thoroughly test the network to guarantee that TCP packets are not fragmented or have their segments reordered during reassembly. If not, there is a danger of lost packets in the network layer, unrelated to the specific application involved.

Several errors logged by the new metrics subsystem were discovered and corrected.

Under certain circumstances, when resizing a database cluster to reduce the number of nodes, a flurry of informational messages reporting that an invocation request was rejected could fill up the log files. This issue has been resolved.

In releases of Volt Active Data since V12.0, when restoring a snapshot from a larger cluster to a smaller cluster (for example, moving a snapshot from a five node to a three node cluster), it was possible for the restore operation to run out of heap memory, even though the new cluster had sufficient memory for the saved data. This issue has been resolved.

Previously, TLS/SSL encryption was enabled for the new metrics port whenever SSL was enabled for any other VoltDB port on the server, whether internal or external. TLS/SSL should only be enabled for the metrics port when it is enabled for external ports. This issue has been resolved.

There was a minor memory leak associated with closing client connections with SSL/TLS enabled. Although trivial under normal conditions, for applications that connect and disconnect repeatedly in rapid succession, the cumulative effect can ultimately use up all available memory. This issue has been resolved.

https://github.com/VoltDB/volt-monitoring/tree/main/dashboards/Volt-V13.x/new-metrics

Volt V12.3 introduced the ability to alter DR tables without pausing the database. Unfortunately, after altering the table, it was possible in certain cases for subsequent tuple updates (UPDATE or DELETE) to generate unnecessary and potentially misleading conflicts in the DR conflict log. This issue has been resolved.

In recent versions of Volt (starting with V12.0), if a schema change coincided with a snapshot save operation, it was possible under certain circumstances for the schema change to hang and never complete. This issue has been resolved.

Specify which interface to use for internal and external ports, respectively, using the --internalinterface and --externalinterface arguments when starting the database process with the voltdb start command.

For an individual port, specify the interface and port on the command line. For example voltdb start --client=32.31.30.29:21212.

Java 8 — release 8u32-b05

Java 11 — release 11.0.20+8

Java 17 — release 17.0.8+7