diff --git a/eng/doc/AdditionalFeatures.md b/eng/doc/AdditionalFeatures.md index b109c7aab8..f6081880aa 100644 --- a/eng/doc/AdditionalFeatures.md +++ b/eng/doc/AdditionalFeatures.md @@ -11,7 +11,8 @@ If it's critical to you to understand the exact implementation details of all ch The Microsoft build of Go collects [opt-out (enabled by default) toolset telemetry](https://devblogs.microsoft.com/go/microsoft-go-telemetry/). -See the [Data Collection policy for the Microsoft build of Go](/README.md#data-collection). +See the [Telemetry documentation](/eng/doc/Telemetry.md) for details on what is collected and how to opt out. +See also the [Data Collection policy for the Microsoft build of Go](/README.md#data-collection). ## System-provided cryptography backend diff --git a/eng/doc/MigrationGuide.md b/eng/doc/MigrationGuide.md index c4918c6351..2792447ea4 100644 --- a/eng/doc/MigrationGuide.md +++ b/eng/doc/MigrationGuide.md @@ -8,7 +8,8 @@ It's a fork, and some runtime behavior slightly differs, but in most cases it ha We expect that most projects don't require any Go code changes to work with the Microsoft build of Go. Note that the Microsoft build of Go has [toolset telemetry enabled by default](https://devblogs.microsoft.com/go/microsoft-go-telemetry/) (opt-out telemetry). -See [the Data Collection policy for the Microsoft build of Go](/README.md#data-collection). +See the [Telemetry documentation](/eng/doc/Telemetry.md) for details on what is collected and how to opt out. +See also [the Data Collection policy for the Microsoft build of Go](/README.md#data-collection). ## Quick start diff --git a/eng/doc/README.md b/eng/doc/README.md index aaed7c9848..c7641610b3 100644 --- a/eng/doc/README.md +++ b/eng/doc/README.md @@ -4,6 +4,7 @@ by reading the infrastructure code itself. For cryptography information, see the [CrossPlatformCryptography.md](CrossPlatformCryptography.md) doc. For dev scenario documentation, see the [DeveloperGuide.md](DeveloperGuide.md) doc. +For telemetry details, see the [Telemetry.md](Telemetry.md) doc. The [Downloads.md](Downloads.md) doc contains a table of links to the latest assets for each supported Go release branch. The [release-branch-links.json](release-branch-links.json) file contains the same data in JSON format suitable for parsing. diff --git a/eng/doc/Telemetry.md b/eng/doc/Telemetry.md new file mode 100644 index 0000000000..8c70e60f25 --- /dev/null +++ b/eng/doc/Telemetry.md @@ -0,0 +1,75 @@ +# Telemetry in the Microsoft build of Go + +The Microsoft build of Go collects opt-out (enabled by default) toolset telemetry. +Telemetry is sent to Microsoft via Azure Application Insights when `go build`, `go install`, or `go run` is executed. +No telemetry is collected for other `go` subcommands (e.g. `go test`, `go vet`, `go fmt`). + +For the blog post announcing this feature, see [Microsoft Go Telemetry](https://devblogs.microsoft.com/go/microsoft-go-telemetry/). + +## What is collected + +All telemetry data consists of **counters** — simple event names that are incremented. +No source code, file paths, module contents, or personally identifiable information is collected. +Each telemetry session is assigned a random session ID that cannot be linked back to a user. + +The counters collected are defined in the [telemetry upload configuration](/go/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.json) and are listed below. + +### Invocation counter + +| Counter | Description | +|---|---| +| `go/invocations` | Incremented once per `go` command invocation. | + +### Subcommand + +| Counter | Description | +|---|---| +| `go/subcommand:{build,install,run}` | Which subcommand was used. | + +### Target platform + +| Counter | Description | +|---|---| +| `go/platform/target/port:-` | The target platform port (e.g. `linux-amd64`, `windows-arm64`). | +| `go/cgo:{enabled,disabled}` | Whether cgo was enabled for the build. | + +### GOEXPERIMENT settings + +| Counter | Description | +|---|---| +| `go/goexperiment:` | Each active `GOEXPERIMENT` value that differs from the baseline. Only Microsoft-specific experiments are uploaded: `systemcrypto`, `nosystemcrypto`, `opensslcrypto`, `cngcrypto`, `darwincrypto`, `ms_nocgo_opensslcrypto`, and `ms_tls_config_schannel`. | + +### CI environment detection + +| Counter | Description | +|---|---| +| `msgo/ci:` | Detected CI provider, if any. Values: `azdo`, `github`, `gitlab`, `appveyor`, `travis`, `circleci`, `aws_codebuild`, `jenkins`, `teamcity`, `google_cloud_build`. Detection is based on well-known environment variables set by each CI system. | + +### System cryptography + +| Counter | Description | +|---|---| +| `msgo/systemcrypto:{enabled,disabled}` | Whether the system cryptography backend is enabled on platforms that support it. | + +## What is NOT collected + +- No source code or file contents. +- No file paths from the local file system. +- No environment variable values (only the presence of specific CI-related variables is checked). +- No network or authentication information. +- No personally identifiable information (PII). + +## How to opt out + +Set the `MS_GOTOOLCHAIN_TELEMETRY_ENABLED` environment variable to `0`. + +This completely disables Microsoft telemetry collection. When disabled, no data is sent to Application Insights. + +> [!NOTE] +> This setting only controls Microsoft-specific telemetry. The upstream Go toolchain has its own [telemetry system](https://go.dev/doc/telemetry) controlled separately via `go telemetry`. + +## Privacy + +Microsoft's privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. + +See the [Data Collection policy](/README.md#data-collection) in the repository README for more information. diff --git a/patches/0001-Vendor-external-dependencies.patch b/patches/0001-Vendor-external-dependencies.patch index 7bcc747a11..d379ef6f3d 100644 --- a/patches/0001-Vendor-external-dependencies.patch +++ b/patches/0001-Vendor-external-dependencies.patch @@ -13,9 +13,10 @@ Use a 'go' that was recently built by the current branch to ensure stable result .../internal/telemetry/counter/deps_ignore.go | 17 + .../microsoft/go-infra/telemetry/LICENSE | 21 + .../microsoft/go-infra/telemetry/README.md | 9 + + .../microsoft/go-infra/telemetry/ci.go | 97 + .../go-infra/telemetry/config/LICENSE | 21 + .../go-infra/telemetry/config/config.go | 11 + - .../go-infra/telemetry/config/config.json | 76 + + .../go-infra/telemetry/config/config.json | 79 + .../go-infra/telemetry/counter/counter.go | 63 + .../telemetry/internal/appinsights/README.md | 12 + .../telemetry/internal/appinsights/client.go | 192 ++ @@ -270,10 +271,11 @@ Use a 'go' that was recently built by the current branch to ensure stable result .../internal/subtle/aliasing.go | 32 + .../internal/sysdll/sys_windows.go | 55 + src/vendor/modules.txt | 23 + - 262 files changed, 35630 insertions(+), 7 deletions(-) + 263 files changed, 35730 insertions(+), 7 deletions(-) create mode 100644 src/cmd/internal/telemetry/counter/deps_ignore.go create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/LICENSE create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/README.md + create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/ci.go create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/LICENSE create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.go create mode 100644 src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.json @@ -527,30 +529,30 @@ Use a 'go' that was recently built by the current branch to ensure stable result create mode 100644 src/vendor/github.com/microsoft/go-crypto-winnative/internal/sysdll/sys_windows.go diff --git a/src/cmd/go.mod b/src/cmd/go.mod -index 29b78693abd787..dbc87f66957dcb 100644 +index 29b78693abd787..ffcaa3d7848408 100644 --- a/src/cmd/go.mod +++ b/src/cmd/go.mod @@ -4,6 +4,8 @@ go 1.27 require ( github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83 -+ github.com/microsoft/go-infra/telemetry v0.0.0-20260513084116-8db604e72b76 -+ github.com/microsoft/go-infra/telemetry/config v0.0.0-20260513084116-8db604e72b76 ++ github.com/microsoft/go-infra/telemetry v0.0.0-20260522114235-13e33af6063a ++ github.com/microsoft/go-infra/telemetry/config v0.0.0-20260522114235-13e33af6063a golang.org/x/arch v0.27.1-0.20260521044007-9c1a596a2c97 golang.org/x/build v0.0.0-20260122183339-3ba88df37c64 golang.org/x/mod v0.36.1-0.20260513122029-343ee60345a1 diff --git a/src/cmd/go.sum b/src/cmd/go.sum -index 929e1c34dcbec1..b873d102f0a9f7 100644 +index 929e1c34dcbec1..68262698d63570 100644 --- a/src/cmd/go.sum +++ b/src/cmd/go.sum @@ -4,6 +4,10 @@ github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83 h1:z2ogiKUYzX5Is6zr/v github.com/google/pprof v0.0.0-20260115054156-294ebfa9ad83/go.mod h1:MxpfABSjhmINe3F1It9d+8exIHFvUqtLIRCdOGNXqiI= github.com/ianlancetaylor/demangle v0.0.0-20250417193237-f615e6bd150b h1:ogbOPx86mIhFy764gGkqnkFC8m5PJA7sPzlk9ppLVQA= github.com/ianlancetaylor/demangle v0.0.0-20250417193237-f615e6bd150b/go.mod h1:gx7rwoVhcfuVKG5uya9Hs3Sxj7EIvldVofAWIUtGouw= -+github.com/microsoft/go-infra/telemetry v0.0.0-20260513084116-8db604e72b76 h1:nP1tNhLNK4ceYOGVEfAY44tj/5Q0vq/pOqZggUzXHXc= -+github.com/microsoft/go-infra/telemetry v0.0.0-20260513084116-8db604e72b76/go.mod h1:LxxLUDlqi1gwmGrnh1slAFqnEhZnKjn37MABZ6xJs44= -+github.com/microsoft/go-infra/telemetry/config v0.0.0-20260513084116-8db604e72b76 h1:LESU567RNJA+E3w8PyUdwbswh/PY8vF0xvsf3fUERMI= -+github.com/microsoft/go-infra/telemetry/config v0.0.0-20260513084116-8db604e72b76/go.mod h1:t6u8QcO4tExYT4+NEB0XqR0ObiUvLe0D8ZpxFobGuA8= ++github.com/microsoft/go-infra/telemetry v0.0.0-20260522114235-13e33af6063a h1:1yIH/VJLOUUWw2yVBEqAqDsakqIsx1TyjHv3rQuVv2A= ++github.com/microsoft/go-infra/telemetry v0.0.0-20260522114235-13e33af6063a/go.mod h1:LxxLUDlqi1gwmGrnh1slAFqnEhZnKjn37MABZ6xJs44= ++github.com/microsoft/go-infra/telemetry/config v0.0.0-20260522114235-13e33af6063a h1:F/bJDa61Hj7ISaUMyErZ9m0GIgHSaLMrhf2f1Pja9Io= ++github.com/microsoft/go-infra/telemetry/config v0.0.0-20260522114235-13e33af6063a/go.mod h1:t6u8QcO4tExYT4+NEB0XqR0ObiUvLe0D8ZpxFobGuA8= github.com/yuin/goldmark v1.6.0 h1:boZcn2GTjpsynOsC0iJHnBWa4Bi0qzfJjthwauItG68= github.com/yuin/goldmark v1.6.0/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= golang.org/x/arch v0.27.1-0.20260521044007-9c1a596a2c97 h1:OEbDVxixMxnrAI3whhcFkCb0rPrEHwxeSSUMdN0V414= @@ -619,6 +621,109 @@ index 00000000000000..5fc9356268f64a + +The [`appinsights`](appinsights) package is an alternative client that can send more arbitrary telemetry event data to Application Insights. +It only supports a few features of Application Insights that are used in other projects maintained by the Microsoft build of Go team. +diff --git a/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/ci.go b/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/ci.go +new file mode 100644 +index 00000000000000..bb3188f26ead88 +--- /dev/null ++++ b/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/ci.go +@@ -0,0 +1,97 @@ ++// Copyright (c) Microsoft Corporation. ++// Licensed under the MIT License. ++ ++package telemetry ++ ++import "strings" ++ ++// DetectCI inspects the given environment variables to determine which CI ++// system is most likely in use, if any. The env parameter should be in the same ++// format as os.Environ() (i.e. each entry is "KEY=VALUE"). It returns a short ++// identifier matching the go/ci counter values, or "" if no CI system is ++// detected. ++func DetectCI(env []string) string { ++ m := envMap(env) ++ ++ // Azure Pipelines ++ // https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables#system-variables-devops-services ++ if isTrue(m["TF_BUILD"]) { ++ return "azdo" ++ } ++ ++ // GitHub Actions ++ // https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables ++ if isTrue(m["GITHUB_ACTIONS"]) { ++ return "github" ++ } ++ ++ // GitLab CI ++ // https://docs.gitlab.com/ee/ci/variables/predefined_variables.html ++ if m["GITLAB_CI"] != "" { ++ return "gitlab" ++ } ++ ++ // AppVeyor ++ // https://www.appveyor.com/docs/environment-variables/ ++ if isTrue(m["APPVEYOR"]) { ++ return "appveyor" ++ } ++ ++ // Travis CI ++ // https://docs.travis-ci.com/user/environment-variables/#default-environment-variables ++ if isTrue(m["TRAVIS"]) { ++ return "travis" ++ } ++ ++ // CircleCI ++ // https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables ++ if isTrue(m["CIRCLECI"]) { ++ return "circleci" ++ } ++ ++ // AWS CodeBuild ++ // https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html ++ if m["CODEBUILD_BUILD_ID"] != "" && m["AWS_REGION"] != "" { ++ return "aws_codebuild" ++ } ++ ++ // TeamCity ++ // https://www.jetbrains.com/help/teamcity/predefined-build-parameters.html#Predefined+Server+Build+Parameters ++ if m["TEAMCITY_VERSION"] != "" { ++ return "teamcity" ++ } ++ ++ // Jenkins ++ // https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/jenkins/model/CoreEnvironmentContributor/buildEnv.groovy ++ if m["BUILD_ID"] != "" && m["BUILD_URL"] != "" { ++ return "jenkins" ++ } ++ ++ // Google Cloud Build ++ // https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values#using_default_substitutions ++ if m["BUILD_ID"] != "" && m["PROJECT_ID"] != "" { ++ return "google_cloud_build" ++ } ++ ++ return "" ++} ++ ++// envMap converts an os.Environ()-style slice into a map for fast lookup. ++func envMap(env []string) map[string]string { ++ m := make(map[string]string, len(env)) ++ for _, e := range env { ++ if k, v, ok := strings.Cut(e, "="); ok { ++ m[k] = v ++ } ++ } ++ return m ++} ++ ++// isTrue reports whether the value is a common boolean-true string. ++// Matches the logic in dotnet/sdk EnvironmentVariableParser.ParseBool. ++func isTrue(v string) bool { ++ return v == "1" || ++ strings.EqualFold(v, "true") || ++ strings.EqualFold(v, "yes") || ++ strings.EqualFold(v, "on") ++} diff --git a/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/LICENSE b/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/LICENSE new file mode 100644 index 00000000000000..9e841e7a26e4eb @@ -665,10 +770,10 @@ index 00000000000000..044268a046a54d +var Config []byte diff --git a/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.json b/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.json new file mode 100644 -index 00000000000000..4b16ad97f35d1a +index 00000000000000..89ce84ff435575 --- /dev/null +++ b/src/cmd/vendor/github.com/microsoft/go-infra/telemetry/config/config.json -@@ -0,0 +1,76 @@ +@@ -0,0 +1,79 @@ +{ + "GOOS": [ + "aix", @@ -730,16 +835,19 @@ index 00000000000000..4b16ad97f35d1a + "Name": "go/subcommand:{build,install,run}" + }, + { -+ "Name": "go/platform/target/goos:{aix,android,darwin,dragonfly,freebsd,hurd,illumos,ios,js,linux,nacl,netbsd,openbsd,plan9,solaris,wasip1,windows,zos}" ++ "Name": "go/platform/target/port:{aix-ppc64,android-386,android-amd64,android-arm,android-arm64,darwin-amd64,darwin-arm64,dragonfly-amd64,freebsd-386,freebsd-amd64,freebsd-arm,freebsd-arm64,freebsd-riscv64,illumos-amd64,ios-amd64,ios-arm64,js-wasm,linux-386,linux-amd64,linux-arm,linux-arm64,linux-loong64,linux-mips,linux-mips64,linux-mips64le,linux-mipsle,linux-ppc64,linux-ppc64le,linux-riscv64,linux-s390x,linux-sparc64,netbsd-386,netbsd-amd64,netbsd-arm,netbsd-arm64,openbsd-386,openbsd-amd64,openbsd-arm,openbsd-arm64,openbsd-ppc64,openbsd-riscv64,plan9-386,plan9-amd64,plan9-arm,solaris-amd64,wasip1-wasm,windows-386,windows-amd64,windows-arm64}" + }, + { -+ "Name": "go/platform/target/goarch:{386,amd64,amd64p32,arm,arm64,arm64be,armbe,loong64,mips,mips64,mips64le,mips64p32,mips64p32le,mipsle,ppc,ppc64,ppc64le,riscv,riscv64,s390,s390x,sparc,sparc64,wasm}" ++ "Name": "go/cgo:{enabled,disabled}" + }, + { -+ "Name": "go/cgo:{enabled,disabled}" ++ "Name": "msgo/ci:{appveyor,aws_codebuild,azdo,circleci,github,gitlab,google_cloud_build,jenkins,teamcity,travis}" + }, + { + "Name": "msgo/systemcrypto:{enabled,disabled}" ++ }, ++ { ++ "Name": "msgo/module:*" + } + ] + } @@ -2220,14 +2328,14 @@ index 00000000000000..e50deaa4189b18 + return false +} diff --git a/src/cmd/vendor/modules.txt b/src/cmd/vendor/modules.txt -index f3ed9c985e7bf5..96de94b746e601 100644 +index f3ed9c985e7bf5..67e1651ba7e3c5 100644 --- a/src/cmd/vendor/modules.txt +++ b/src/cmd/vendor/modules.txt @@ -16,6 +16,17 @@ github.com/google/pprof/third_party/svgpan # github.com/ianlancetaylor/demangle v0.0.0-20250417193237-f615e6bd150b ## explicit; go 1.13 github.com/ianlancetaylor/demangle -+# github.com/microsoft/go-infra/telemetry v0.0.0-20260513084116-8db604e72b76 ++# github.com/microsoft/go-infra/telemetry v0.0.0-20260522114235-13e33af6063a +## explicit; go 1.25 +github.com/microsoft/go-infra/telemetry +github.com/microsoft/go-infra/telemetry/counter @@ -2235,7 +2343,7 @@ index f3ed9c985e7bf5..96de94b746e601 100644 +github.com/microsoft/go-infra/telemetry/internal/appinsights/internal/contracts +github.com/microsoft/go-infra/telemetry/internal/config +github.com/microsoft/go-infra/telemetry/internal/telemetry -+# github.com/microsoft/go-infra/telemetry/config v0.0.0-20260513084116-8db604e72b76 ++# github.com/microsoft/go-infra/telemetry/config v0.0.0-20260522114235-13e33af6063a +## explicit; go 1.24 +github.com/microsoft/go-infra/telemetry/config # golang.org/x/arch v0.27.1-0.20260521044007-9c1a596a2c97 diff --git a/patches/0009-Add-appinsights-telemetry.patch b/patches/0009-Add-appinsights-telemetry.patch index 9bb8d64efb..53fc64f914 100644 --- a/patches/0009-Add-appinsights-telemetry.patch +++ b/patches/0009-Add-appinsights-telemetry.patch @@ -5,14 +5,14 @@ Subject: [PATCH] Add appinsights telemetry --- README.md | 15 +++++ - .../internal/telemetrystats/telemetrystats.go | 10 ++++ + .../internal/telemetrystats/telemetrystats.go | 17 ++++++ src/cmd/go/main.go | 8 +++ src/cmd/go/mstelemetry_test.go | 56 +++++++++++++++++++ src/cmd/go/script_test.go | 10 ++++ src/cmd/go/scriptcmds_test.go | 22 ++++++++ src/cmd/internal/telemetry/counter/counter.go | 54 +++++++++++++++++- .../telemetry/counter/counter_bootstrap.go | 3 + - 8 files changed, 177 insertions(+), 1 deletion(-) + 8 files changed, 184 insertions(+), 1 deletion(-) create mode 100644 src/cmd/go/mstelemetry_test.go diff --git a/README.md b/README.md @@ -40,22 +40,30 @@ index 71c9d1dc299388..8cab5daec4da54 100644 +these practices. \ No newline at end of file diff --git a/src/cmd/go/internal/telemetrystats/telemetrystats.go b/src/cmd/go/internal/telemetrystats/telemetrystats.go -index c6e58935bb0ee2..5aec1694aaae96 100644 +index c6e58935bb0ee2..0cd7d3582c9ee6 100644 --- a/src/cmd/go/internal/telemetrystats/telemetrystats.go +++ b/src/cmd/go/internal/telemetrystats/telemetrystats.go -@@ -11,6 +11,8 @@ import ( +@@ -11,7 +11,12 @@ import ( "cmd/go/internal/cfg" "cmd/go/internal/modload" "cmd/internal/telemetry/counter" + "internal/buildcfg" + "internal/platform" ++ "os" "strings" ++ ++ "github.com/microsoft/go-infra/telemetry" ) -@@ -39,6 +41,14 @@ func incrementConfig() { + func Increment() { +@@ -39,6 +44,18 @@ func incrementConfig() { counter.Inc("go/cgo:disabled") } ++ if ci := telemetry.DetectCI(os.Environ()); ci != "" { ++ counter.Inc("msgo/ci:" + ci) ++ } ++ + if platform.SystemCryptoSupported(cfg.Goos, cfg.Goarch) { + if buildcfg.SystemCryptoDisabled { + counter.Inc("msgo/systemcrypto:disabled") @@ -68,7 +76,7 @@ index c6e58935bb0ee2..5aec1694aaae96 100644 counter.Inc("go/platform/target/goarch:" + cfg.Goarch) counter.Inc("go/platform/target/port:" + cfg.Goos + "-" + cfg.Goarch) diff --git a/src/cmd/go/main.go b/src/cmd/go/main.go -index 8cdfd9196e4cb1..3f4a4d8f912969 100644 +index bba11a2fdeaaa6..cb7801588662a2 100644 --- a/src/cmd/go/main.go +++ b/src/cmd/go/main.go @@ -110,6 +110,14 @@ func main() {