Migrate GCE runner pools to Regional MIGs (RMIG) across us-central1 (#2870)

Migrate Linux (x86) and Windows runner instance groups from zonal MIGs
(in us-central1-c) to regional MIGs in us-central1 with ANY distribution
shape to prevent capacity stockouts.

### Problem
Upgrading `default_vm` in single-zone `us-central1-c` caused severe
capacity exhaustion across Linux Docker and Windows runners, draining
the active runner fleet as runners terminated after each job.

### Solution
- Migrate runner pools to Regional Managed Instance Groups (RMIGs)
across `us-central1` using `--target-distribution-shape=ANY`. GCE will
dynamically distribute VMs across `us-central1-a`, `b`, `c`, and `f`
based on real-time hardware capacity.
- Inherit `region: us-central1` from `default_vm` for `bk-docker`,
`bk-windows`, and their testing/trusted variants.
- Retain explicit `zone: us-central1-c` on ARM64 pools
(`c4a-standard-8`).
- In `buildkite/create_instances.py`, set
`target_distribution_shape="ANY"` for regional MIG creation and
automatically check for and delete legacy zonal MIGs with the same name
to prevent naming and resource conflicts during migration.

---------

Co-authored-by: Salma Samy <16399431+SalmaSamy@users.noreply.github.com>

buildkite/create_instance_template.py

diff --git a/buildkite/create_instance_template.py b/buildkite/create_instance_template.py
index 1938b28..32dfbea 100755
--- a/buildkite/create_instance_template.py
+++ b/buildkite/create_instance_template.py
@@ -41,6 +41,7 @@
             project = item.pop("project")
             zone = item.pop("zone", None)
             region = item.pop("region", None)
+            item.pop("target_distribution_shape", None)
             del item["health_check"]
             del item["initial_delay"]
 

buildkite/create_instances.py

diff --git a/buildkite/create_instances.py b/buildkite/create_instances.py
index 286e62c..4974d7a 100755
--- a/buildkite/create_instances.py
+++ b/buildkite/create_instances.py
@@ -35,6 +35,7 @@
         project = config.pop("project")
         zone = config.pop("zone", None)
         region = config.pop("region", None)
+        target_distribution_shape = config.pop("target_distribution_shape", None)
         health_check = config.pop("health_check", None)
         initial_delay = config.pop("initial_delay", None)
 
@@ -79,6 +80,8 @@
             kwargs["zone"] = zone
         elif region:
             kwargs["region"] = region
+            if target_distribution_shape:
+                kwargs["target_distribution_shape"] = target_distribution_shape
         if health_check:
             kwargs["health_check"] = health_check
         if initial_delay:

buildkite/instances.yml

diff --git a/buildkite/instances.yml b/buildkite/instances.yml
index bcbcb82..85d79be 100644
--- a/buildkite/instances.yml
+++ b/buildkite/instances.yml
@@ -17,9 +17,9 @@
   machine_type: c2-standard-30
   network: default
   region: us-central1
+  target_distribution_shape: ANY
   restart-on-failure: False
   scopes: cloud-platform
-  zone: us-central1-c
   image_project: bazel-public
   tags: health-check
   health_check: buildkite-check
@@ -76,14 +76,12 @@
     service_account: buildkite@bazel-untrusted.iam.gserviceaccount.com
     image_family: bk-windows
     metadata_from_file: windows-startup-script-ps1=startup-windows-pdssd.ps1
-    zone: us-central1-c
   - name: bk-testing-windows
     count: 5
     project: bazel-untrusted
     service_account: buildkite-testing@bazel-untrusted.iam.gserviceaccount.com
     image_family: bk-testing-windows
     metadata_from_file: windows-startup-script-ps1=startup-windows-pdssd.ps1
-    zone: us-central1-c
   - name: bk-trusted-windows
     count: 10
     project: bazel-public

buildkite/update_instances.py

diff --git a/buildkite/update_instances.py b/buildkite/update_instances.py
index 7e92f5d..15563ec 100755
--- a/buildkite/update_instances.py
+++ b/buildkite/update_instances.py
@@ -41,6 +41,7 @@
             project = item.pop("project")
             zone = item.pop("zone", None)
             region = item.pop("region", None)
+            item.pop("target_distribution_shape", None)
             del item["health_check"]
             del item["initial_delay"]