Allow instance groups to change zones when repairing VMs (#2914)
## Summary
Every Linux and Windows agent VM runs a single job and then powers off
(`disconnect-after-job=true` + `ExecStopPost=/bin/systemctl poweroff`).
The instance group then repairs the VM by recreating it, and by default
a repair must recreate the VM **in the same zone**. When that zone runs
out of capacity, recreated VMs get stuck retrying there
(`ZONE_RESOURCE_POOL_EXHAUSTED`), even if other zones in the region have
room. That's what happened to `bk-docker-arm64` and
`bk-testing-docker-arm64` on 2026-10-01: all stuck VMs sat in
us-central1-c.
This change sets
[`--on-repair-allow-changing-zone=yes`](https://cloud.google.com/compute/docs/instance-groups/autohealing-instances-in-migs)
for all regional instance groups so a repair can recreate the VM in
another zone. GCE requires `--force-update-on-repair` for that, which
also makes repaired VMs pick up the group's latest configuration.
## Changes
- `create_instances.py`: new `on_repair_allow_changing_zone` option.
When it's set on a regional group, the MIG is created with
`--force-update-on-repair --on-repair-allow-changing-zone yes`.
- `instances.yml`: enable it in `default_vm`, so it applies to all
instance groups.
## Notes
- This doesn't reserve capacity, it only avoids getting stuck in one
zone. Reservations would be needed to hold capacity across recreates.
- Already enabled manually on `bk-testing-docker-arm64` and
`bk-docker-arm64`. Other groups get it the next time
`create_instances.py` recreates them.
## Testing
Ran `create_instance_group()` against the real `instances.yml` with
`gcloud` stubbed out. All 9 groups get both flags on the MIG create
command, and instance templates are unchanged. `black --check` passes.
diff --git a/buildkite/create_instances.py b/buildkite/create_instances.py
index 4974d7a..00ff023 100755
--- a/buildkite/create_instances.py
+++ b/buildkite/create_instances.py
@@ -38,6 +38,7 @@
target_distribution_shape = config.pop("target_distribution_shape", None)
health_check = config.pop("health_check", None)
initial_delay = config.pop("initial_delay", None)
+ on_repair_allow_changing_zone = config.pop("on_repair_allow_changing_zone", False)
if not project:
raise Exception("Invalid instance config, no project name set")
@@ -82,6 +83,12 @@
kwargs["region"] = region
if target_distribution_shape:
kwargs["target_distribution_shape"] = target_distribution_shape
+ if on_repair_allow_changing_zone:
+ # VMs power off after each job and the MIG repairs them by recreating them.
+ # Let the MIG recreate them in another zone if their zone ran out of capacity.
+ # GCE requires force_update_on_repair for this.
+ kwargs["force_update_on_repair"] = True
+ kwargs["on_repair_allow_changing_zone"] = "yes"
if health_check:
kwargs["health_check"] = health_check
if initial_delay:
diff --git a/buildkite/instances.yml b/buildkite/instances.yml
index 7b3abc3..c66df03 100644
--- a/buildkite/instances.yml
+++ b/buildkite/instances.yml
@@ -18,6 +18,8 @@
network: default
region: us-central1
target_distribution_shape: ANY
+ # Let the MIG recreate a VM in a different zone when its zone runs out of capacity.
+ on_repair_allow_changing_zone: true
restart-on-failure: False
scopes: cloud-platform
image_project: bazel-public