[Backport 8.1.x] release: Run promote finalize once, and make tag creation retry-safe (#981)

Backport ebaf2a2 from #979

The v8.1.1335 promotion ran finalize twice, and the run the release
worker watches lost the race:

| Build | Commit | Started by | Promote finalize |
|---|---|---|---|
|
[#3093](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3093)
| `4d8b729` "release: v8.1.1335" (internal config commit) | creating
`promote/release-v8.1.1335` | **passed** at 07:14: tagged the promote
branch head `a018051`, pushed `release-v8.1.1335`, promoted on the
release registry |
|
[#3094](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3094)
| `21ae663` image update | push | test failed (no config) |
|
[#3095](https://buildkite.com/sourcegraph/deploy-sourcegraph-helm/builds/3095)
| `a018051` "promote-release: v8.1.1335" | push | **failed** 4×: `fatal:
tag 'v8.1.1335' already exists` |

Creating the promote branch from the internal branch builds the internal
config commit, and `--config-from-commit` accepts its config, so that
build runs the promote jobs too. Its finalize tags whatever the promote
branch head is when it gets there. This time that was the right commit,
only because the tests ahead of it took about 7 minutes. Build #3095 is
the one the release worker monitors (sourcegraph/sourcegraph#17343), so
the promote workflow failed and never closed the release PRs or opened
the "Update main" PR. The release itself is fine: tag `v8.1.1335` is on
`a018051` with public images, and the registry shows it as public.
v8.0.0 had the same race (#2990 passed finalize, #2993 failed it), but
nothing monitored the promote build then.

This PR changes two things:

- **Promote jobs run only for the promote config commit.** `(promote)
Release: test` and `finalize` now also require `build.message =~
/^promote-release:/`. Both the release worker and `release.yaml`'s
manual flow write that message.
- **`git:tag` is retry-safe.** If `vX` already points at the promote
branch head, it skips creating the tag and continues. If it points
anywhere else, it fails. The release registry's promote endpoint already
treats re-promoting as a no-op (200), so a retried finalize now succeeds
end to end.

## How this was tested

- **Job conditions.** I checked the old and new conditions against the
real branch and commit message of each v8.1.1335 build:

  | Build | Promote jobs before | after |
  |---|---|---|
  | #3093 `release: v8.1.1335` | run | **skip** |
  | #3094 `Update Helm chart images…` | run | **skip** |
  | #3095 `promote-release: v8.1.1335` | run | run |
  | #978 `promote/…-update-main` | skip | skip |

This PR's own Buildkite build validates that the pipeline, with its new
conditionals, uploads.
- **`git:tag`.** I extracted the step from `release.yaml`, stubbed `gh`
and `buildkite-agent`, and ran it in fresh clones against a scratch bare
remote:
- **First finalize:** exits 0; the tag and `release-vX` are created on
the promote head.
- **Retried or duplicate finalize** (tag already on the promote head):
exits 0 with "already points at … not recreating it"; refs unchanged.
- **Tag on a different commit:** exits 1 with "already exists at <sha>,
not at promote/release-vX (<sha>)".
- **Control:** `main`'s step on the retry case exits 128 with `fatal:
tag 'v9.9.9' already exists`, the v8.1.1335 failure.

## Rollout

Buildkite reads `pipeline.yaml` and `release.yaml` from the commit it
builds, so patch releases need this on their `X.Y.x` branch. It's
labelled `backport 8.1.x`, which applies cleanly. `8.0.x` predates the
`-update-main` exclusion, so it would need a manual backport if it gets
more patches.

The same change is open for the other deploy repos:
sourcegraph/deploy-sourcegraph-k8s#497 (k8s failed v8.1.1335 the same
way) and sourcegraph/deploy-sourcegraph-docker#1449 (Docker avoided it
only because the extra build was cancelled).

The v8.1.1335 cleanup is done by hand: #976 and #977 are closed, and
#978 is the "Update main" PR.

Co-authored-by: Warren Gifford <warrenbruceg@gmail.com>

Author: sourcegraph-release-bot

.buildkite/pipeline.yaml

@@ -47,8 +47,11 @@ steps:
 
       sg release run test --workdir=. --config-from-commit
 
+  # Promote jobs run only for the promote config commit. Creating the promote
+  # branch also builds its first commit, the internal config commit, which
+  # --config-from-commit would otherwise accept, running finalize twice.
   - label: "(promote) Release: test"
-    if: build.branch =~ /^promote\/release-.*/ && build.branch !~ /-update-main$/
+    if: build.branch =~ /^promote\/release-.*/ && build.branch !~ /-update-main$/ && build.message =~ /^promote-release:/
     plugins:
       - ssh://git@github.com/sourcegraph/sg-buildkite-plugin.git#main: ~
     command: |
@@ -71,7 +74,7 @@ steps:
 
       sg release run internal finalize --workdir=. --config-from-commit
   - label: "(promote) Release: finalize"
-    if: build.branch =~ /^promote\/release-.*/ && build.branch !~ /-update-main$/
+    if: build.branch =~ /^promote\/release-.*/ && build.branch !~ /-update-main$/ && build.message =~ /^promote-release:/
     plugins:
       - ssh://git@github.com/sourcegraph/sg-buildkite-plugin.git#main: ~
     command: |

release.yaml

@@ -476,7 +476,16 @@ promoteToPublic:
           # Create the final branch holding the tagged commit
           git checkout "${promote_branch}"
           git switch -c "${release_branch}"
-          git tag {{version}}
+          # A retried finalize finds its own tag; only a tag elsewhere is an error
+          if tagged=$(git rev-parse -q --verify "refs/tags/{{version}}^{commit}"); then
+            if [ "${tagged}" != "$(git rev-parse HEAD)" ]; then
+              echo "Tag {{version}} already exists at ${tagged}, not at ${promote_branch} ($(git rev-parse HEAD))" >&2
+              exit 1
+            fi
+            echo "Tag {{version}} already points at ${promote_branch}, not recreating it"
+          else
+            git tag {{version}}
+          fi
           git push origin ${release_branch} --tags
 
           # Web URL to the tag