x/build: build failures due to 410 Gone responses from symbolic-datum-552 GOPROXY

eit6fx6z  于 3个月前  发布在  Go
关注(0)|答案(9)|浏览(46)
:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "run" "golang.org/x/benchmarks/cmd/bench"] and env ["HOSTNAME=buildlet-linux-amd64-perf-rn8418630" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=0s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64-perf" "GOROOT_BOOTSTRAP=/go1.4" "BENCH_BASELINE_GOROOT=/workdir/gobaseline" "BENCH_BRANCH=master" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/workdir/gopath/src/golang.org/x/benchmarks"] in dir /workdir/gopath/src/golang.org/x/benchmarks
…
[sweet] Setting up benchmark: go-build
[sweet] error: build go-build for experiment: exit status 1. stderr:
go: github.com/golang-migrate/migrate/v4@v4.6.2 requires
	cloud.google.com/go@v0.37.4 requires
	google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20190404172233-64821d5d2107.mod: 410 Gone
	server response: not found: temporarily unavailable
go: downloading github.com/go-redis/redis/v8 v8.0.0
go: downloading github.com/google/safehtml v0.0.2
go: downloading cloud.google.com/go v0.66.0
go: downloading cloud.google.com/go/storage v1.10.0
go: downloading golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
go: downloading cloud.google.com/go/logging v1.0.0
go: downloading contrib.go.opencensus.io/integrations/ocsql v0.1.4
go: downloading github.com/jackc/pgx/v4 v4.12.0
go: downloading google.golang.org/genproto v0.0.0-20200923140941-5646d36feee1
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.1.0
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.4
go: downloading go.opencensus.io v0.22.4
go: downloading golang.org/x/mod v0.3.1-0.20200828183125-ce943fd02449
go: downloading github.com/go-redis/redis_rate/v9 v9.0.2
go: downloading google.golang.org/api v0.32.0
go: downloading github.com/microcosm-cc/bluemonday v1.0.5
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/yuin/goldmark v1.4.0
go: downloading github.com/yuin/goldmark-emoji v1.0.1
go: downloading github.com/golang/protobuf v1.4.2
go: downloading google.golang.org/grpc v1.32.0
go: downloading github.com/Masterminds/squirrel v1.4.0
go: downloading github.com/golang-migrate/migrate/v4 v4.6.2
go: downloading github.com/google/licensecheck v0.3.1
go: downloading github.com/lib/pq v1.10.2
go: downloading golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
go: downloading github.com/jackc/pgconn v1.9.0
go: downloading github.com/go-git/go-billy/v5 v5.0.0
go: downloading github.com/go-git/go-git/v5 v5.1.0
go: downloading github.com/evanw/esbuild v0.8.57
go: downloading google.golang.org/protobuf v1.25.0
go: downloading github.com/google/go-cmp v0.5.2
go: downloading golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
go: downloading github.com/aws/aws-sdk-go v1.34.29
go: github.com/golang-migrate/migrate/v4@v4.6.2 requires
	cloud.google.com/go@v0.37.4 requires
	google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/genproto/@v/v0.0.0-20190404172233-64821d5d2107.mod: 410 Gone
	server response: not found: temporarily unavailable

greplogs -l -e '\[sweet\] error: build .*(?:\n.*)*410 Gone' --since=2022-01-01
2022-06-22T16:18:44-0fc6e7c-92c9b81/linux-amd64-perf
对我来说,symbolic-datum-552代理是如何填充的,或者x/benchmarks/sweet模块依赖关系是否以某种方式是非封装的或不稳定的,这一点并不明显。
(注意@prattmic@mknyszek;CC @golang/release)

pvcm50d1

pvcm50d11#

我在x/tools trybots(Linux和Windows)上在2022年6月23日15:22 UTC左右的https://go.dev/cl/413578上遇到了非常相似的失败。不同的软件包,没有涉及到x/benchmarks,但出现了相同的410错误:
https://storage.googleapis.com/go-build-log/2e773a38/linux-amd64_f7fad36c.log

linux-amd64 at 2e773a3894fba7af744090d7d42968f4993018e2 building build at d36736901e40cbd378c11bb2535ea9e16f39a7a3

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "test" "-short" "golang.org/x/build/..."] and env ["HOSTNAME=buildlet-linux-bullseye-rnf676e4b" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=0s" "GO_STAGE0_DL_DELAY=100ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-amd64" "GOROOT_BOOTSTRAP=/go1.4" "GO_DISABLE_OUTBOUND_NETWORK=1" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157" "GO111MODULE=on" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/workdir/gopath/src/golang.org/x/build"] in dir /workdir/gopath/src/golang.org/x/build

go: downloading cloud.google.com/go/compute v1.3.0
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading github.com/google/go-cmp v0.5.8
go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
go: downloading go4.org v0.0.0-20180809161055-417644f6feb5
go: downloading google.golang.org/grpc v1.44.0
go: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
go: downloading cloud.google.com/go/storage v1.21.0
go: downloading golang.org/x/crypto v0.0.0-20220307211146-efcb8507fb70
go: downloading github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
go: downloading github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: downloading google.golang.org/api v0.70.0
go: downloading cloud.google.com/go/datastore v1.1.0
go: downloading github.com/NYTimes/gziphandler v1.1.1
go: downloading github.com/jackc/pgx/v4 v4.13.0
go: downloading github.com/shurcooL/githubv4 v0.0.0-20220520033151-0b4e3294ff00
go: downloading github.com/golang/protobuf v1.5.2
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.3.0
go: downloading contrib.go.opencensus.io/exporter/stackdriver v0.13.5
go: downloading go.opencensus.io v0.23.0
go: downloading cloud.google.com/go v0.100.2
go: downloading google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf
go: downloading golang.org/x/text v0.3.7
go: downloading cloud.google.com/go/errorreporting v0.2.0
go: downloading github.com/gliderlabs/ssh v0.3.3
go: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
go: downloading golang.org/x/perf v0.0.0-20220408152633-b570d1a8e7a7
go: downloading cloud.google.com/go/secretmanager v1.3.0
go: downloading github.com/influxdata/influxdb-client-go/v2 v2.8.0
go: downloading cloud.google.com/go/bigquery v1.8.0
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8
go: downloading github.com/creack/pty v1.1.15
go: downloading golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
go: downloading github.com/aws/aws-sdk-go v1.30.15
go: downloading google.golang.org/protobuf v1.27.1
go: downloading github.com/google/uuid v1.2.0
go: downloading github.com/googleapis/gax-go/v2 v2.1.1
go: downloading github.com/jackc/pgconn v1.11.0
go: downloading github.com/go-sql-driver/mysql v1.5.0
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/mattn/go-sqlite3 v1.14.6
go: downloading github.com/aclements/go-gg v0.0.0-20170118225347-6dbb4e4fefb0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f
go: downloading github.com/dghubble/oauth1 v0.7.0
go: downloading github.com/esimov/stackblur-go v1.1.0
go: downloading github.com/sendgrid/sendgrid-go v3.11.1+incompatible
go: downloading github.com/yuin/goldmark v1.4.12
go: downloading golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d
go: downloading github.com/golang-migrate/migrate/v4 v4.15.0-beta.3
go: downloading github.com/julienschmidt/httprouter v1.3.0
go: downloading golang.org/x/sys v0.0.0-20220209214540-3681064d5158
go: downloading github.com/shurcooL/graphql v0.0.0-20220520033453-bdb1221e171e
go: downloading github.com/jackc/pgproto3/v2 v2.2.0
go: downloading github.com/jackc/puddle v1.1.3
go: downloading github.com/google/go-querystring v1.0.0
go: downloading github.com/prometheus/client_golang v1.9.0
go: downloading github.com/prometheus/statsd_exporter v0.20.0
go: downloading cloud.google.com/go/monitoring v1.4.0
go: downloading cloud.google.com/go/trace v1.2.0
go: downloading github.com/census-instrumentation/opencensus-proto v0.2.1
go: downloading cloud.google.com/go/iam v0.1.1
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
go: downloading github.com/deepmap/oapi-codegen v1.8.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/jackc/pgio v1.0.0
go: downloading github.com/jackc/pgtype v1.8.1
go: downloading github.com/jackc/chunkreader/v2 v2.0.1
go: downloading github.com/jackc/pgpassfile v1.0.0
go: downloading github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b
go: downloading github.com/aclements/go-moremath v0.0.0-20210112150236-f10218a38794
go: downloading gonum.org/v1/plot v0.10.0
go: downloading github.com/sendgrid/rest v2.6.9+incompatible
go: downloading github.com/hashicorp/golang-lru v0.5.4
go: downloading github.com/prometheus/common v0.15.0
go: downloading cloud.google.com/go/container v1.2.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/procfs v0.2.0
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading github.com/jackc/pgerrcode v0.0.0-20201024163028-a0d42d470451
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839
go: downloading github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9
go: downloading github.com/fogleman/gg v1.3.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/go-fonts/liberation v0.2.0
go: downloading github.com/go-latex/latex v0.0.0-20210823091927-c0d11ff05a81
go: downloading github.com/ajstarks/svgo v0.0.0-20210923152817-c3b6e2f0c527
go: downloading github.com/go-pdf/fpdf v0.5.0
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac
go: downloading github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82
go: downloading github.com/gonum/internal v0.0.0-20181124074243-f884aa714029
go: downloading github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9
go: downloading github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
google.golang.org/grpc@v1.44.0: reading http://gk3-services-nap-jetxd907-8f582974-sbzf.c.symbolic-datum-552.internal:30157/google.golang.org/grpc/@v/v1.44.0.mod: 410 Gone
	server response: not found: temporarily unavailable
xiozqbni

xiozqbni2#

也许x/benchmarks/sweet模块的依赖关系在某种程度上是非封装的或不稳定的。我不这么认为。go-build的这部分可能是https://cs.opensource.google/go/x/benchmarks/+/master:sweet/harnesses/go-build.go;l=93。也就是说,我们只是对kubelet、istioctl或pkgsite前端的一个固定提交进行了简单的go build(我们无法记录是哪一个;我会修复这个问题)。我认为这应该是稳定的,除非有什么奇怪的事情发生。

hkmswyz6

hkmswyz63#

https://go.dev/cl/414396提到了这个问题:sweet/harnesses: log which go-build package fails to build

ux6nzvsh

ux6nzvsh4#

不同的软件包,没有涉及到x/基准测试,但相同的结果:410 Gone。
实际上,这是另一个正在构建x/build的示例:https://build.golang.org/log/c532106b8e7169a892bbdc77978e904684480c53
更新搜索正则表达式,我们得到:
greplogs -l -e 'reading .*: 410 Gone\n\s+server response: not found: temporarily unavailable'
2022-06-24T23:50:20-60cf787-41e1d90/linux-amd64
2022-06-22T16:18:44-0fc6e7c-92c9b81/linux-amd64-perf
我不确定这是否是symbolic-datum-552代理或其连接的后端(也许是proxy.golang.org?)的问题。

dxpyg8gm

dxpyg8gm5#

默认的GOPROXY值是:https://proxy.golang.org,direct。使用这个设置,当go命令下载一个模块或模块元数据时,它会首先向proxy.golang.org发送请求...如果代理返回404(未找到)或410(已消失)状态,go命令将尝试直接连接到提供该模块的版本控制系统。
协调者在没有direct回退的情况下设置GOPROXY:https://cs.opensource.google/go/x/build/+/master:cmd/coordinator/buildstatus.go;l=1244。这解释了为什么410会导致失败。但对我来说(1)不清楚是否故意不设置直接回退,或者(2)为什么这么长时间以来都没有发生这种情况。

ncecgwcz

ncecgwcz6#

我认为我们没有直接的后备方案,因为如果GO_DISABLE_OUTBOUND_NETWORK正在运行(可能没有,参见#51444),那么防火墙无论如何都会拒绝任何直接的后备连接。

lf3rwulv

lf3rwulv7#

谢谢,这很有道理。当我意识到我们只是为proxy.golang.org提供一个反向代理时,我猜想可能是这种情况。
如果他们知道代理的更改可能触发了这个问题,请@heschi@suzmue。
一个可能的解释是,随着时间的推移,我们获取的包版本在全球范围内变得足够不受欢迎,以至于它们现在更有可能从代理中的缓存中消失。
到目前为止,我们收到410状态码的版本有:

  • github.com/docker/docker@v17.12.0-ce-rc1.0.20210128214336-420b1d36250f+incompatible
  • google.golang.org/genproto@v0.0.0-20190404172233-64821d5d2107
gupuwyp2

gupuwyp28#

所以看起来这条路径正在访问 proxy.golang.org ?那么问题是,为什么代理对于这些URL返回410状态码而不是一直阻塞直到它们被获取?
(也许与模块获取路径上的某些错误或故障的不良交互?但在那时,我希望看到描述获取错误的内容,而不仅仅是 temporarily unavailable ...)

mlnl4t2r

mlnl4t2r9#

这是一个半知名的模块代理中的bug。@suzmue应该知道更多。

相关问题