Remove ubuntu2004 from smoke test configurations (#2853)

Removed ubuntu2004 and ubuntu2004_arm64 from the smoke test
configurations.

Following https://github.com/bazelbuild/continuous-integration/pull/2846
1 file changed