Skip to content

Instantly share code, notes, and snippets.

@everpeace

everpeace/00_README.md

Last active July 14, 2020 13:52
Show Gist options
  • Save everpeace/ea9582347e2c9d0bb519cdae1b438ead to your computer and use it in GitHub Desktop.
Save everpeace/ea9582347e2c9d0bb519cdae1b438ead to your computer and use it in GitHub Desktop.
Download ZIP
Buildkit TCP service behind Envoy Proxy
Raw
00_README.md

Architecture

Envoy filters out Prune/DiskUsage method. see envoy-config.yaml below.

              +-------+     +-----------+
buidctl <---->+ Envoy +<--->+ Buildkitd +
              +-------+     +-----------+

How to Try

$ git clone https://gist.github.com/ea9582347e2c9d0bb519cdae1b438ead.git envoy-builkitd
$ cd envoy-builkitd
$ docker-compose up
...you see logs...
  • envoy listens: localhost:1234
  • envoy forwards requests to buildkitd
  • buildkitd also listens localhost:11234 for debug

Issue

builctl with direct access to buildkit works.

$ buildctl --debug --addr tcp://127.0.0.1:11234 debug workers
ID                              PLATFORMS
xflodox4is8qyxb9ewq9bl1l3       linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6

But buildctl can't connect to buildkit thgourh envoy.

$ buildctl --debug --addr tcp://127.0.0.1:1234 debug workers
error: rpc error: code = Unavailable desc = transport is closing
failed to list workers
github.com/moby/buildkit/client.(*Client).ListWorkers
        /src/client/workers.go:32
github.com/moby/buildkit/cmd/buildctl/debug.listWorkers
        /src/cmd/buildctl/debug/workers.go:41
github.com/urfave/cli.HandleAction
        /src/vendor/github.com/urfave/cli/app.go:523
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:174
github.com/urfave/cli.(*App).RunAsSubcommand
        /src/vendor/github.com/urfave/cli/app.go:404
github.com/urfave/cli.Command.startApp
        /src/vendor/github.com/urfave/cli/command.go:373
github.com/urfave/cli.Command.Run
        /src/vendor/github.com/urfave/cli/command.go:102
github.com/urfave/cli.(*App).Run
        /src/vendor/github.com/urfave/cli/app.go:276
main.main
        /src/cmd/buildctl/main.go:100
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

Envoy's debug lots are like this. invalid frame: Invalid HTTP header field was received on stream 1 is the direct cause.

envoy_1      | [2020-07-14 13:51:17.913][14][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C8] new connection
envoy_1      | [2020-07-14 13:51:17.914][14][debug][http2] [source/common/http/http2/codec_impl.cc:970] [C8] updating connection-level initial window size to 268435456
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http] [source/common/http/conn_manager_impl.cc:268] [C8] new stream
envoy_1      | [2020-07-14 13:51:17.916][14][debug][http2] [source/common/http/http2/codec_impl.cc:670] [C8] invalid frame: Invalid HTTP header field was received on stream 1
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:283] [C8] dispatch error: The user callback function failed
envoy_1      | [2020-07-14 13:51:17.917][14][debug][http] [source/common/http/conn_manager_impl.cc:1936] [C8][S4570179258856159313] stream reset
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl.cc:109] [C8] closing data_to_write=0 type=2
envoy_1      | [2020-07-14 13:51:17.917][14][debug][connection] [source/common/network/connection_impl_base.cc:30] [C8] setting delayed close timer with timeout 1000 ms
envoy_1      | [2020-07-14T13:51:17.916Z] "- - HTTP/2" 0 DPE 0 0 1 - "-" "-" "-" "-" "-"
envoy_1      | [2020-07-14 13:51:18.919][14][debug][connection] [source/common/network/connection_impl_base.cc:44] [C8] triggered delayed close
envoy_1      | [2020-07-14 13:51:18.920][14][debug][connection] [source/common/network/connection_impl.cc:200] [C8] closing socket: 1
envoy_1      | [2020-07-14 13:51:18.920][14][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C8] adding to cleanup list
envoy_1      | [2020-07-14 13:51:18.924][12][debug][conn_handler] [source/server/connection_handler_impl.cc:372] [C9] new connection
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:558] [C9] remote close
envoy_1      | [2020-07-14 13:51:18.926][12][debug][connection] [source/common/network/connection_impl.cc:200] [C9] closing socket: 0
envoy_1      | [2020-07-14 13:51:18.927][12][debug][conn_handler] [source/server/connection_handler_impl.cc:86] [C9] adding to cleanup list
envoy_1      | [2020-07-14 13:51:19.212][1][debug][main] [source/server/server.cc:177] flushing stats
Raw
docker-compose.yml
version: '3'
services:
envoy:
image: envoyproxy/envoy:v1.14.2
command: "--config-path /etc/envoy/envoy-config.yaml --log-level debug"
volumes:
- .:/etc/envoy
ports:
- "1234:1234" # buidkitd
- "8001:8001" # envoy admin
links:
- buildkitd
buildkitd:
image: moby/buildkit:rootless
# listens on 0.0.0.0 for debug
command: "--debug --addr=tcp://0.0.0.0:11234 --oci-worker-no-process-sandbox"
ports:
- "11234:11234" # direct access (for clarifying buildkitd works)
security_opt:
- seccomp:unconfined
- apparmor:unconfined
Raw
envoy-config.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 1234
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
http2_protocol_options: {}
route_config:
name: local_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
safe_regex:
google_re2: {}
regex: "\/moby.buildkit.v1.Control\/(Status|Solve|Session|ListWorkers).*"
route:
cluster: local_service
timeout: "0s"
access_log:
- name: envoy.file_access_log
config:
path: "/dev/stdout"
http_filters:
- name: envoy.router
config: {}
clusters:
- name: local_service
http2_protocol_options: {}
connect_timeout: 0.250s
type: strict_dns
lb_policy: round_robin
health_checks:
- timeout: 5s
interval: 10s
unhealthy_threshold: 2
healthy_threshold: 2
tcp_health_check: {}
hosts:
- socket_address:
address: buildkitd
port_value: 11234
admin:
access_log_path: /dev/stdout
address:
socket_address:
address: 0.0.0.0
port_value: 8001
@dio
Copy link

dio commented Jul 10, 2020

Try this as your envoy-config.yaml

static_resources:
  listeners:
    - name: listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 1234
      filter_chains:
        - filters:
            - name: envoy.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
                stat_prefix: tcp_local_service
                cluster: local_service
  clusters:
    - name: local_service
      connect_timeout: 0.250s
      type: strict_dns
      lb_policy: round_robin
      hosts:
        - socket_address:
            address: buildkitd
            port_value: 11234
admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

@everpeace
Copy link
Author

everpeace commented Jul 14, 2020 •
edited
Loading

Thanks. yeah, tcp_proxy works as a charm!! However, I wanted to filter-out several grpc methods in envoy. So I think I need to realize http connection manager. note: I updated my envoy-config.yaml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment