Vitess v19.0.6

Release of Vitess v19.0.6

The entire changelog for this release can be found here.

The release includes 21 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @shlomi-noach, @systay, @vitess-bot

Vitess v18.0.7

Release of Vitess v18.0.7

The entire changelog for this release can be found here.

Known Issues

VTTablet runs an unnecessary DDL

In this release, with MySQL 8.0 whenever VTTablet's tabletserver goes through initialization, it finds
a diff in its internal schemacopy table and tries to apply a DDL to fix it. The DDL
is a no-op and this loop of applying the DDL continues to run.
If this problem is encountered, the following PR should be reverted to resolve it - #15859.

The release includes 23 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @frouioui, @shlomi-noach, @vitess-bot

Vitess v20.0.1

Release of Vitess v20.0.1

The entire changelog for this release can be found here.

The release includes 21 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @frouioui, @shlomi-noach, @vitess-bot

Vitess v19.0.5

Release of Vitess v19.0.5

The entire changelog for this release can be found here.

The release includes 40 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @harshit-gangal, @systay, @vitess-bot

Vitess v18.0.6

Release of Vitess v18.0.6

The entire changelog for this release can be found here.

Known Issues

VTTablet runs an unnecessary DDL

In this release, with MySQL 8.0 whenever VTTablet's tabletserver goes through initialization, it finds
a diff in its internal schemacopy table and tries to apply a DDL to fix it. The DDL
is a no-op and this loop of applying the DDL continues to run.
If this problem is encountered, the following PR should be reverted to resolve it - #15859.

The release includes 28 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @arthurschreiber, @shlomi-noach, @vitess-bot

Vitess v20.0.0

Release of Vitess v20.0.0

Summary

Table of Contents

• Major Changes

  • Deletions
    • --vreplication_tablet_type flag
    • Pool Capacity Flags
    • vitess/base and vitess/k8s Docker images
    • gh-ost binary and endtoend tests
    • Legacy EmergencyReparentShard stats
  • Breaking changes
    • Metric Name Changes in VTOrc
    • ENUM and SET column handling in VTGate VStream API
    • shutdown_grace_period Default Change
    • New unmanaged Flag and disable_active_reparents deprecation
    • recovery-period-block-duration Flag deprecation
    • mysqlctld onterm-timeout Default Change
    • MoveTables now removes auto_increment clauses by default when moving tables from an unsharded keyspace to a sharded one
    • Durabler interface method renaming
  • Query Compatibility
    • Vindex Hints
    • Update with Limit Support
    • Update with Multi Table Support
    • Update with Multi Target Support
    • Delete with Subquery Support
    • Delete with Multi Target Support
    • User Defined Functions Support
    • Insert Row Alias Support
  • VReplication
    • Multi-tenant Imports
    • VDiff Support For OnlineDDL Migrations
  • Query Timeout
  • Flag changes
    • pprof-http default change
    • New healthcheck-dial-concurrency flag
    • New minimum for --buffer_min_time_between_failovers
    • New track-udfs vtgate flag
    • Help text fix for --lock-timeout
    • New --querylog-sample-rate flag
    • New --tablet-filter-tags flag

• Minor Changes

  • New Stats
    • VTTablet Query Cache Hits and Misses
    • VTGate and VTTablet Query Text Characters Processed
  • SIGHUP reload of gRPC client static auth creds
  • VTAdmin
    • Updated to node v20.12.2
    • Replaced highcharts with d3

Major Changes

Deletion

--vreplication_tablet_type flag

The previously deprecated flag --vreplication_tablet_type has been deleted.

Pool Capacity Flags

The previously deprecated flags --queryserver-config-query-pool-waiter-cap, --queryserver-config-stream-pool-waiter-cap and --queryserver-config-txpool-waiter-cap have been deleted.

vitess/base and vitess/k8s Docker images

Since we have deleted MySQL from our vitess/lite image, we are removing the vitess/base and vitess/k8s images.

These images are no longer useful since we can use vitess/lite as the base of many other Docker images (vitess/vtgate, vitess/vtgate, ...).

gh-ost binary and endtoend tests

Vitess 20.0 drops support for gh-ost DDL strategy.

vttablet binary no longer embeds a gh-ost binary. Users of gh-ost DDL strategy will need to supply a gh-ost binary on the vttablet host or pod. Vitess will look for the gh-ost binary in the system PATH; otherwise the user should supply vttablet --gh-ost-path.

Vitess' endtoend tests no longer use nor test gh-ost migrations.

Legacy EmergencyReparentShard stats

The following EmergencyReparentShard stats were deprecated in Vitess 18.0 and are removed in Vitess 20.0:

• ers_counter
• ers_success_counter
• ers_failure_counter

These counters are replaced by the following stats (introduced in Vitess 18.0):

• emergency_reparent_counts - Number of times EmergencyReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.
• planned_reparent_counts - Number of times PlannedReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.

Also, the reparent_shard_operation_timings stat was added to provide per-operation timings of reparent operations.

Breaking Changes

Metric Name Changes in VTOrc

The following metric names have been changed in VTOrc. The old metrics are still available in /debug/vars for this release, but will be removed in later releases. The new metric names and the deprecated metric names resolve to the same metric name on prometheus, so there is no change there.

Old Metric Name	New Metric Name	Name in Prometheus
analysis.change.write	AnalysisChangeWrite	vtorc_analysis_change_write
audit.write	AuditWrite	vtorc_audit_write
discoveries.attempt	DiscoveriesAttempt	vtorc_discoveries_attempt
discoveries.fail	DiscoveriesFail	vtorc_discoveries_fail
discoveries.instance_poll_seconds_exceeded	DiscoveriesInstancePollSecondsExceeded	vtorc_discoveries_instance_poll_seconds_exceeded
discoveries.queue_length	DiscoveriesQueueLength	vtorc_discoveries_queue_length
discoveries.recent_count	DiscoveriesRecentCount	vtorc_discoveries_recent_count
instance.read	InstanceRead	vtorc_instance_read
instance.read_topology	InstanceReadTopology	vtorc_instance_read_topology
emergency_reparent_counts	EmergencyReparentCounts	vtorc_emergency_reparent_counts
planned_reparent_counts	PlannedReparentCounts	vtorc_planned_reparent_counts
reparent_shard_operation_timings	ReparentShardOperationTimings	vtorc_reparent_shard_operation_timings_bucket

ENUM and SET column handling in VTGate VStream API

The VTGate VStream API now returns ENUM and SET column type values in VEvent messages (in the embedded RowChange messages) as their string values instead of the integer based ones — in both the copy/snapshot phase and the streaming phase. This change was done to make the VStream API more user-friendly, intuitive, and to align the behavior across both phases. Before this change the values for ENUM and SET columns were string values in the copy phase but integer values (which only have an internal meaning to MySQL) in the streaming phase. This inconsistency led to various challenges and issues for each VStream client/consumer (e.g. the Debezium Vitess connector failed to properly perform a snapshot for tables containing these column types). Now the behavior is intuitive — clients need the string values as the eventual sink is often not MySQL so each consumer needed to perform the mappings themselves — and consistent. While this is a (potentially) breaking change, a new boolean field has been added to the FieldEvent message called EnumSetStringValues. When that field is false (in Vitess v19 and older) then the consumer will need to perform the mappings during streaming phase, but not during copy phase. When this field is true, then no mapping is required. This will help to ensure a smooth transition for all consumers over time. To demonstrate, let's look at the textual output (printing the received VEvents as strings) when streaming a single enum_set_test table from the unsharded commerce keyspace so that we can see what the VStream looks like before and after when we star...

Vitess v20.0.0-rc2

Release of Vitess v20.0.0

Summary

Table of Contents

• Major Changes

  • Deletions
    • --vreplication_tablet_type flag
    • Pool Capacity Flags
    • vitess/base and vitess/k8s Docker images
    • gh-ost binary and endtoend tests
    • Legacy EmergencyReparentShard stats
  • Breaking changes
    • Metric Name Changes in VTOrc
    • ENUM and SET column handling in VTGate VStream API
    • shutdown_grace_period Default Change
    • New unmanaged Flag and disable_active_reparents deprecation
    • recovery-period-block-duration Flag deprecation
    • mysqlctld onterm-timeout Default Change
    • MoveTables now removes auto_increment clauses by default when moving tables from an unsharded keyspace to a sharded one
    • Durabler interface method renaming
  • Query Compatibility
    • Vindex Hints
    • Update with Limit Support
    • Update with Multi Table Support
    • Update with Multi Target Support
    • Delete with Subquery Support
    • Delete with Multi Target Support
    • User Defined Functions Support
    • Insert Row Alias Support
  • Query Timeout
  • Flag changes
    • pprof-http default change
    • New healthcheck-dial-concurrency flag
    • New minimum for --buffer_min_time_between_failovers
    • New track-udfs vtgate flag
    • Help text fix for --lock-timeout
    • New --querylog-sample-rate flag
    • New --tablet-filter-tags flag

• Minor Changes

  • New Stats
    • VTTablet Query Cache Hits and Misses
    • VTGate and VTTablet Query Text Characters Processed
  • SIGHUP reload of gRPC client static auth creds
  • VTAdmin
    • Updated to node v20.12.2
    • Replaced highcharts with d3

Major Changes

Deletion

--vreplication_tablet_type flag

The previously deprecated flag --vreplication_tablet_type has been deleted.

Pool Capacity Flags

The previously deprecated flags --queryserver-config-query-pool-waiter-cap, --queryserver-config-stream-pool-waiter-cap and --queryserver-config-txpool-waiter-cap have been deleted.

vitess/base and vitess/k8s Docker images

Since we have deleted MySQL from our vitess/lite image, we are removing the vitess/base and vitess/k8s images.

These images are no longer useful since we can use vitess/lite as the base of many other Docker images (vitess/vtgate, vitess/vtgate, ...).

gh-ost binary and endtoend tests

Vitess 20.0 drops support for gh-ost DDL strategy.

vttablet binary no longer embeds a gh-ost binary. Users of gh-ost DDL strategy will need to supply a gh-ost binary on the vttablet host or pod. Vitess will look for the gh-ost binary in the system PATH; otherwise the user should supply vttablet --gh-ost-path.

Vitess' endtoend tests no longer use nor test gh-ost migrations.

Legacy EmergencyReparentShard stats

The following EmergencyReparentShard stats were deprecated in Vitess 18.0 and are removed in Vitess 20.0:

• ers_counter
• ers_success_counter
• ers_failure_counter

These counters are replaced by the following stats (introduced in Vitess 18.0):

• emergency_reparent_counts - Number of times EmergencyReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.
• planned_reparent_counts - Number of times PlannedReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.

Also, the reparent_shard_operation_timings stat was added to provide per-operation timings of reparent operations.

Breaking Changes

Metric Name Changes in VTOrc

The following metric names have been changed in VTOrc. The old metrics are still available in /debug/vars for this release, but will be removed in later releases. The new metric names and the deprecated metric names resolve to the same metric name on prometheus, so there is no change there.

Old Metric Name	New Metric Name	Name in Prometheus
analysis.change.write	AnalysisChangeWrite	vtorc_analysis_change_write
audit.write	AuditWrite	vtorc_audit_write
discoveries.attempt	DiscoveriesAttempt	vtorc_discoveries_attempt
discoveries.fail	DiscoveriesFail	vtorc_discoveries_fail
discoveries.instance_poll_seconds_exceeded	DiscoveriesInstancePollSecondsExceeded	vtorc_discoveries_instance_poll_seconds_exceeded
discoveries.queue_length	DiscoveriesQueueLength	vtorc_discoveries_queue_length
discoveries.recent_count	DiscoveriesRecentCount	vtorc_discoveries_recent_count
instance.read	InstanceRead	vtorc_instance_read
instance.read_topology	InstanceReadTopology	vtorc_instance_read_topology
emergency_reparent_counts	EmergencyReparentCounts	vtorc_emergency_reparent_counts
planned_reparent_counts	PlannedReparentCounts	vtorc_planned_reparent_counts
reparent_shard_operation_timings	ReparentShardOperationTimings	vtorc_reparent_shard_operation_timings_bucket

ENUM and SET column handling in VTGate VStream API

The VTGate VStream API now returns ENUM and SET column type values in VEvent messages (in the embedded RowChange messages) as their string values instead of the integer based ones — in both the copy/snapshot phase and the streaming phase. This change was done to make the VStream API more user-friendly, intuitive, and to align the behavior across both phases. Before this change the values for ENUM and SET columns were string values in the copy phase but integer values (which only have an internal meaning to MySQL) in the streaming phase. This inconsistency led to various challenges and issues for each VStream client/consumer (e.g. the Debezium Vitess connector failed to properly perform a snapshot for tables containing these column types). Now the behavior is intuitive — clients need the string values as the eventual sink is often not MySQL so each consumer needed to perform the mappings themselves — and consistent. While this is a (potentially) breaking change, a new boolean field has been added to the FieldEvent message called EnumSetStringValues. When that field is false (in Vitess v19 and older) then the consumer will need to perform the mappings during streaming phase, but not during copy phase. When this field is true, then no mapping is required. This will help to ensure a smooth transition for all consumers over time. To demonstrate, let's look at the textual output (printing the received VEvents as strings) when streaming a single enum_set_test table from the unsharded commerce keyspace so that we can see what the VStream looks like before and after when we start a new VStream in copy/snapshot mode and then transition to streaming mode for the following table:

CREATE TABLE `enum_set_test` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(120) DEFAULT NULL,
  `shirt_size` enum('small','medium','large','xlarge'...

Vitess v20.0.0-rc1

Release of Vitess v20.0.0

Summary

Table of Contents

• Major Changes

  • Deletions
    • --vreplication_tablet_type flag
    • Pool Capacity Flags
    • vitess/base and vitess/k8s Docker images
    • gh-ost binary and endtoend tests
    • Legacy EmergencyReparentShard stats
  • Breaking changes
    • Metric Name Changes in VTOrc
    • ENUM and SET column handling in VTGate VStream API
    • shutdown_grace_period Default Change
    • New unmanaged Flag and disable_active_reparents deprecation
    • recovery-period-block-duration Flag deprecation
    • mysqlctld onterm-timeout Default Change
    • MoveTables now removes auto_increment clauses by default when moving tables from an unsharded keyspace to a sharded one
    • Durabler interface method renaming
  • Query Compatibility
    • Vindex Hints
    • Update with Limit Support
    • Update with Multi Table Support
    • Update with Multi Target Support
    • Delete with Subquery Support
    • Delete with Multi Target Support
    • User Defined Functions Support
    • Insert Row Alias Support
  • Query Timeout
  • Flag changes
    • pprof-http default change
    • New healthcheck-dial-concurrency flag
    • New minimum for --buffer_min_time_between_failovers
    • New track-udfs vtgate flag
    • Help text fix for --lock-timeout
    • New --querylog-sample-rate flag
    • New --tablet-filter-tags flag

• Minor Changes

  • New Stats
    • VTTablet Query Cache Hits and Misses
    • VTGate and VTTablet Query Text Characters Processed
  • SIGHUP reload of gRPC client static auth creds
  • VTAdmin
    • Updated to node v20.12.2
    • Replaced highcharts with d3

Major Changes

Deletion

--vreplication_tablet_type flag

The previously deprecated flag --vreplication_tablet_type has been deleted.

Pool Capacity Flags

The previously deprecated flags --queryserver-config-query-pool-waiter-cap, --queryserver-config-stream-pool-waiter-cap and --queryserver-config-txpool-waiter-cap have been deleted.

vitess/base and vitess/k8s Docker images

Since we have deleted MySQL from our vitess/lite image, we are removing the vitess/base and vitess/k8s images.

These images are no longer useful since we can use vitess/lite as the base of many other Docker images (vitess/vtgate, vitess/vtgate, ...).

gh-ost binary and endtoend tests

Vitess 20.0 drops support for gh-ost DDL strategy.

vttablet binary no longer embeds a gh-ost binary. Users of gh-ost DDL strategy will need to supply a gh-ost binary on the vttablet host or pod. Vitess will look for the gh-ost binary in the system PATH; otherwise the user should supply vttablet --gh-ost-path.

Vitess' endtoend tests no longer use nor test gh-ost migrations.

Legacy EmergencyReparentShard stats

The following EmergencyReparentShard stats were deprecated in Vitess 18.0 and are removed in Vitess 20.0:

• ers_counter
• ers_success_counter
• ers_failure_counter

These counters are replaced by the following stats (introduced in Vitess 18.0):

• emergency_reparent_counts - Number of times EmergencyReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.
• planned_reparent_counts - Number of times PlannedReparentShard has been run. It is further subdivided by the keyspace, shard and the result of the operation.

Also, the reparent_shard_operation_timings stat was added to provide per-operation timings of reparent operations.

Breaking Changes

Metric Name Changes in VTOrc

The following metric names have been changed in VTOrc. The old metrics are still available in /debug/vars for this release, but will be removed in later releases. The new metric names and the deprecated metric names resolve to the same metric name on prometheus, so there is no change there.

Old Metric Name	New Metric Name	Name in Prometheus
analysis.change.write	AnalysisChangeWrite	vtorc_analysis_change_write
audit.write	AuditWrite	vtorc_audit_write
discoveries.attempt	DiscoveriesAttempt	vtorc_discoveries_attempt
discoveries.fail	DiscoveriesFail	vtorc_discoveries_fail
discoveries.instance_poll_seconds_exceeded	DiscoveriesInstancePollSecondsExceeded	vtorc_discoveries_instance_poll_seconds_exceeded
discoveries.queue_length	DiscoveriesQueueLength	vtorc_discoveries_queue_length
discoveries.recent_count	DiscoveriesRecentCount	vtorc_discoveries_recent_count
instance.read	InstanceRead	vtorc_instance_read
instance.read_topology	InstanceReadTopology	vtorc_instance_read_topology
emergency_reparent_counts	EmergencyReparentCounts	vtorc_emergency_reparent_counts
planned_reparent_counts	PlannedReparentCounts	vtorc_planned_reparent_counts
reparent_shard_operation_timings	ReparentShardOperationTimings	vtorc_reparent_shard_operation_timings_bucket

ENUM and SET column handling in VTGate VStream API

The VTGate VStream API now returns ENUM and SET column type values in VEvent messages (in the embedded RowChange messages) as their string values instead of the integer based ones — in both the copy/snapshot phase and the streaming phase. This change was done to make the VStream API more user-friendly, intuitive, and to align the behavior across both phases. Before this change the values for ENUM and SET columns were string values in the copy phase but integer values (which only have an internal meaning to MySQL) in the streaming phase. This inconsistency led to various challenges and issues for each VStream client/consumer (e.g. the Debezium Vitess connector failed to properly perform a snapshot for tables containing these column types). Now the behavior is intuitive — clients need the string values as the eventual sink is often not MySQL so each consumer needed to perform the mappings themselves — and consistent. While this is a (potentially) breaking change, a new boolean field has been added to the FieldEvent message called EnumSetStringValues. When that field is false (in Vitess v19 and older) then the consumer will need to perform the mappings during streaming phase, but not during copy phase. When this field is true, then no mapping is required. This will help to ensure a smooth transition for all consumers over time. To demonstrate, let's look at the textual output (printing the received VEvents as strings) when streaming a single enum_set_test table from the unsharded commerce keyspace so that we can see what the VStream looks like before and after when we start a new VStream in copy/snapshot mode and then transition to streaming mode for the following table:

CREATE TABLE `enum_set_test` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(120) DEFAULT NULL,
  `shirt_size` enum('small','medium','large','xlarge'...

Vitess v19.0.4

Release of Vitess v19.0.4

The entire changelog for this release can be found here.

The release includes 19 merged Pull Requests.

Thanks to all our contributors: @GuptaManan100, @app/vitess-bot, @systay, @vitess-bot

Vitess v18.0.5

Release of Vitess v18.0.5

The entire changelog for this release can be found here.

The release includes 14 merged Pull Requests.

Thanks to all our contributors: @app/vitess-bot, @notfelineit, @rohit-nayak-ps, @shlomi-noach, @vitess-bot