Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File based dynamic configuration, cluster not updating #13853

Closed
dili91 opened this issue Nov 1, 2020 · 12 comments · Fixed by #13854
Closed

File based dynamic configuration, cluster not updating #13853

dili91 opened this issue Nov 1, 2020 · 12 comments · Fixed by #13854
Assignees
Labels

Comments

@dili91
Copy link
Contributor

dili91 commented Nov 1, 2020

I've been struggling a few hours to make this sample work, but no luck. Originally I implemented something very similar on my own: I could see the cds.yaml file properly updated across host and container's filesystem, but envoy did not pick the updated file at runtime.

This is the only update I could spot on proxy logs soon after updating the file:

proxy_1     | [2020-11-01 15:04:02.971][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 2 file: cds.yaml
proxy_1     | [2020-11-01 15:04:02.972][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 2 file: cds.yaml
proxy_1     | [2020-11-01 15:04:02.991][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 2 file: cds.yaml
proxy_1     | [2020-11-01 15:04:02.991][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:72] notification: fd: 1 mask: 2 file: cds.yaml
proxy_1     | [2020-11-01 15:04:04.718][7][debug][main] [source/server/server.cc:190] flushing stats

Finally, I also cloned this repo and ran this project but I got the same results: even though I can see the file properly updated on both host and containers file systems, it seems envoy is not acknowledging any changes.

below verify.sh logs (I had to remove the -i option on sed command in order to use it on my Mac)

$ ./verify.sh

> [dynamic-config-fs] Pull the images (.)
Pulling service1 ... done
Pulling service2 ... done
Pulling proxy    ... done


> [dynamic-config-fs] Bring up services (.)
Creating network "dynamic-config-fs_default" with the default driver
Building proxy
Step 1/4 : FROM envoyproxy/envoy-dev:latest
 ---> 89853ab39357
Step 2/4 : COPY ./envoy.yaml /etc/envoy.yaml
 ---> Using cache
 ---> 475b0028d480
Step 3/4 : RUN chmod go+r /etc/envoy.yaml
 ---> Using cache
 ---> 0208205b021a
Step 4/4 : CMD ["/usr/local/bin/envoy", "-c /etc/envoy.yaml", "-l", "debug"]
 ---> Using cache
 ---> d68dc0f44dec

Successfully built d68dc0f44dec
Successfully tagged dynamic-config-fs_proxy:latest
Creating dynamic-config-fs_service1_1 ... done
Creating dynamic-config-fs_service2_1 ... done
Creating dynamic-config-fs_proxy_1    ... done

            Name                          Command               State                  Ports
-----------------------------------------------------------------------------------------------------------
dynamic-config-fs_proxy_1      /docker-entrypoint.sh /usr ...   Up      0.0.0.0:10000->10000/tcp,
                                                                        0.0.0.0:19000->19000/tcp
dynamic-config-fs_service1_1   /bin/echo-server                 Up      8080/tcp
dynamic-config-fs_service2_1   /bin/echo-server                 Up      8080/tcp
Attaching to dynamic-config-fs_proxy_1, dynamic-config-fs_service2_1, dynamic-config-fs_service1_1
proxy_1     | [2020-11-01 14:54:47.028][7][info][main] [source/server/server.cc:305] initializing epoch 0 (base id=0, hot restart version=11.104)
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:307] statically linked extensions:
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.udp_listeners: quiche_quic_listener, raw_udp_listener
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.squash, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, envoy.squash
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.transports: auto, framed, header, unframed
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.compression.decompressor: envoy.compression.gzip.decompressor
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.route_matchers: default
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.http.cache: envoy.extensions.http.cache.simple
proxy_1     | [2020-11-01 14:54:47.029][7][info][main] [source/server/server.cc:309]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
proxy_1     | [2020-11-01 14:54:47.030][7][info][main] [source/server/server.cc:309]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.kafka_broker, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.mysql_proxy, envoy.filters.network.postgres_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.rocketmq_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
proxy_1     | [2020-11-01 14:54:47.030][7][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
proxy_1     | [2020-11-01 14:54:47.030][7][info][main] [source/server/server.cc:309]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
proxy_1     | [2020-11-01 14:54:47.030][7][info][main] [source/server/server.cc:309]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.health_checkers: envoy.health_checkers.redis
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.quic_server_codec: quiche
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.serializers: dubbo.hessian2
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.compression.compressor: envoy.compression.gzip.compressor
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.dubbo_proxy.protocols: dubbo
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
service2_1  | Echo server listening on port 8080.
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.thrift_proxy.filters: envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.bootstrap: envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.stats_sinks: envoy.dog_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.resolvers: envoy.ip
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.udp_packet_writers: udp_default_writer, udp_gso_batch_writer
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.upstreams: envoy.filters.connection_pools.http.generic, envoy.filters.connection_pools.http.http, envoy.filters.connection_pools.http.tcp
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.quic_client_codec: quiche
proxy_1     | [2020-11-01 14:54:47.031][7][info][main] [source/server/server.cc:309]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, tls
proxy_1     | [2020-11-01 14:54:47.036][7][info][main] [source/server/server.cc:325] HTTP header map info:
proxy_1     | [2020-11-01 14:54:47.037][7][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
proxy_1     | [2020-11-01 14:54:47.037][7][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
proxy_1     | [2020-11-01 14:54:47.038][7][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
proxy_1     | [2020-11-01 14:54:47.038][7][warning][runtime] [source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
proxy_1     | [2020-11-01 14:54:47.038][7][info][main] [source/server/server.cc:328]   request header map: 608 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-proto,x-ot-span-context,x-request-id
proxy_1     | [2020-11-01 14:54:47.038][7][info][main] [source/server/server.cc:328]   request trailer map: 128 bytes:
proxy_1     | [2020-11-01 14:54:47.038][7][info][main] [source/server/server.cc:328]   response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
proxy_1     | [2020-11-01 14:54:47.038][7][info][main] [source/server/server.cc:328]   response trailer map: 152 bytes: grpc-message,grpc-status
proxy_1     | [2020-11-01 14:54:47.039][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.shrink_heap.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][debug][main] [source/server/overload_manager_impl.cc:378] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
proxy_1     | [2020-11-01 14:54:47.040][7][info][main] [source/server/server.cc:448] admin address: 0.0.0.0:19000
proxy_1     | [2020-11-01 14:54:47.042][7][info][main] [source/server/server.cc:583] runtime: layers:
proxy_1     |   - name: base
proxy_1     |     static_layer:
proxy_1     |       {}
proxy_1     |   - name: admin
proxy_1     |     admin_layer:
proxy_1     |       {}
service1_1  | Echo server listening on port 8080.
proxy_1     | [2020-11-01 14:54:47.042][7][info][config] [source/server/configuration_impl.cc:95] loading tracing configuration
proxy_1     | [2020-11-01 14:54:47.042][7][info][config] [source/server/configuration_impl.cc:70] loading 0 static secret(s)
proxy_1     | [2020-11-01 14:54:47.042][7][info][config] [source/server/configuration_impl.cc:76] loading 0 cluster(s)
proxy_1     | [2020-11-01 14:54:47.043][11][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.045][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'cds.yaml' fd: 1
proxy_1     | [2020-11-01 14:54:47.045][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:127] maybe finish initialize state: 1
proxy_1     | [2020-11-01 14:54:47.045][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:136] maybe finish initialize primary init clusters empty: true
proxy_1     | [2020-11-01 14:54:47.045][7][info][config] [source/server/configuration_impl.cc:80] loading 0 listener(s)
proxy_1     | [2020-11-01 14:54:47.045][7][info][config] [source/server/configuration_impl.cc:121] loading stats sink configuration
proxy_1     | [2020-11-01 14:54:47.047][7][debug][file] [source/common/filesystem/inotify/watcher_impl.cc:47] added watch for directory: '/var/lib/envoy' file: 'lds.yaml' fd: 1
proxy_1     | [2020-11-01 14:54:47.047][7][debug][init] [source/common/init/manager_impl.cc:24] added target LDS to init manager Server
proxy_1     | [2020-11-01 14:54:47.047][7][debug][init] [source/common/init/manager_impl.cc:49] init manager RTDS contains no targets
proxy_1     | [2020-11-01 14:54:47.047][7][debug][init] [source/common/init/watcher_impl.cc:14] init manager RTDS initialized, notifying RTDS
proxy_1     | [2020-11-01 14:54:47.047][7][info][runtime] [source/common/runtime/runtime_impl.cc:425] RTDS has finished initialization
proxy_1     | [2020-11-01 14:54:47.047][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:196] continue initializing secondary clusters
proxy_1     | [2020-11-01 14:54:47.047][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:127] maybe finish initialize state: 2
proxy_1     | [2020-11-01 14:54:47.047][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:136] maybe finish initialize primary init clusters empty: true
proxy_1     | [2020-11-01 14:54:47.047][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:151] maybe finish initialize secondary init clusters empty: true
proxy_1     | [2020-11-01 14:54:47.047][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
proxy_1     | [2020-11-01 14:54:47.047][7][info][upstream] [source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
proxy_1     | [2020-11-01 14:54:47.047][7][debug][config] [source/common/config/filesystem_subscription_impl.cc:61] Filesystem config refresh for /var/lib/envoy/cds.yaml
proxy_1     | [2020-11-01 14:54:47.053][7][info][upstream] [source/common/upstream/cds_api_impl.cc:64] cds: add 1 cluster(s), remove 0 cluster(s)
proxy_1     | [2020-11-01 14:54:47.055][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:634] add/update cluster example_proxy_cluster during init
proxy_1     | [2020-11-01 14:54:47.055][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:669] adding TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.056][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:79] cm init: adding: cluster=example_proxy_cluster primary=1 secondary=0
proxy_1     | [2020-11-01 14:54:47.056][7][info][upstream] [source/common/upstream/cds_api_impl.cc:79] cds: add/update cluster 'example_proxy_cluster'
proxy_1     | [2020-11-01 14:54:47.056][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:127] maybe finish initialize state: 4
proxy_1     | [2020-11-01 14:54:47.056][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:136] maybe finish initialize primary init clusters empty: false
proxy_1     | [2020-11-01 14:54:47.057][7][debug][config] [source/common/config/filesystem_subscription_impl.cc:71] Filesystem config update accepted for /var/lib/envoy/cds.yaml: resources {
proxy_1     |   [type.googleapis.com/envoy.config.cluster.v3.Cluster] {
proxy_1     |     name: "example_proxy_cluster"
proxy_1     |     type: STRICT_DNS
proxy_1     |     connect_timeout {
proxy_1     |       seconds: 1
proxy_1     |     }
proxy_1     |     http2_protocol_options {
proxy_1     |     }
proxy_1     |     load_assignment {
proxy_1     |       cluster_name: "example_proxy_cluster"
proxy_1     |       endpoints {
proxy_1     |         lb_endpoints {
proxy_1     |           endpoint {
proxy_1     |             address {
proxy_1     |               socket_address {
proxy_1     |                 address: "service1"
proxy_1     |                 port_value: 8080
proxy_1     |               }
proxy_1     |             }
proxy_1     |           }
proxy_1     |         }
proxy_1     |       }
proxy_1     |     }
proxy_1     |   }
proxy_1     | }
proxy_1     | 183412668: "envoy.api.v2.DiscoveryResponse"
proxy_1     |
proxy_1     | [2020-11-01 14:54:47.057][7][warning][main] [source/server/server.cc:565] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
proxy_1     | [2020-11-01 14:54:47.059][7][info][main] [source/server/server.cc:679] starting main dispatch loop
proxy_1     | [2020-11-01 14:54:47.059][7][debug][upstream] [source/common/upstream/upstream_impl.cc:286] transport socket match, socket default selected for host with address 192.168.96.2:8080
proxy_1     | [2020-11-01 14:54:47.059][7][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:146] DNS hosts have changed for service1
proxy_1     | [2020-11-01 14:54:47.059][7][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for service1, refresh rate 5000 ms
proxy_1     | [2020-11-01 14:54:47.059][7][debug][upstream] [source/common/upstream/upstream_impl.cc:991] initializing Primary cluster example_proxy_cluster completed
proxy_1     | [2020-11-01 14:54:47.059][7][debug][init] [source/common/init/manager_impl.cc:49] init manager Cluster example_proxy_cluster contains no targets
proxy_1     | [2020-11-01 14:54:47.059][7][debug][init] [source/common/init/watcher_impl.cc:14] init manager Cluster example_proxy_cluster initialized, notifying ClusterImplBase
proxy_1     | [2020-11-01 14:54:47.059][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.060][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:107] cm init: init complete: cluster=example_proxy_cluster primary=0 secondary=0
proxy_1     | [2020-11-01 14:54:47.060][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:127] maybe finish initialize state: 4
proxy_1     | [2020-11-01 14:54:47.060][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:136] maybe finish initialize primary init clusters empty: true
proxy_1     | [2020-11-01 14:54:47.060][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:151] maybe finish initialize secondary init clusters empty: true
proxy_1     | [2020-11-01 14:54:47.060][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
proxy_1     | [2020-11-01 14:54:47.060][7][info][upstream] [source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
proxy_1     | [2020-11-01 14:54:47.060][7][info][main] [source/server/server.cc:660] all clusters initialized. initializing init manager
proxy_1     | [2020-11-01 14:54:47.060][7][debug][init] [source/common/init/manager_impl.cc:53] init manager Server initializing
proxy_1     | [2020-11-01 14:54:47.060][7][debug][init] [source/common/init/target_impl.cc:15] init manager Server initializing target LDS
proxy_1     | [2020-11-01 14:54:47.060][7][debug][config] [source/common/config/filesystem_subscription_impl.cc:61] Filesystem config refresh for /var/lib/envoy/lds.yaml
proxy_1     | [2020-11-01 14:54:47.068][7][debug][config] [source/server/listener_manager_impl.cc:395] begin add/update listener: name=listener_0 hash=7423914739769031304
proxy_1     | [2020-11-01 14:54:47.068][7][debug][config] [source/server/listener_manager_impl.cc:432] use full listener update path for listener name=listener_0 hash=7423914739769031304
proxy_1     | [2020-11-01 14:54:47.068][7][debug][config] [source/server/listener_manager_impl.cc:95]   filter #0:
proxy_1     | [2020-11-01 14:54:47.068][7][debug][config] [source/server/listener_manager_impl.cc:96]     name: envoy.http_connection_manager
proxy_1     | [2020-11-01 14:54:47.068][7][debug][config] [source/server/listener_manager_impl.cc:103]   config: {
proxy_1     |  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
proxy_1     |  "route_config": {
proxy_1     |   "name": "local_route",
proxy_1     |   "virtual_hosts": [
proxy_1     |    {
proxy_1     |     "routes": [
proxy_1     |      {
proxy_1     |       "route": {
proxy_1     |        "cluster": "example_proxy_cluster"
proxy_1     |       },
proxy_1     |       "match": {
proxy_1     |        "prefix": "/"
proxy_1     |       }
proxy_1     |      }
proxy_1     |     ],
proxy_1     |     "name": "local_service",
proxy_1     |     "domains": [
proxy_1     |      "*"
proxy_1     |     ]
proxy_1     |    }
proxy_1     |   ]
proxy_1     |  },
proxy_1     |  "stat_prefix": "ingress_http",
proxy_1     |  "http_filters": [
proxy_1     |   {
proxy_1     |    "name": "envoy.router"
proxy_1     |   }
proxy_1     |  ]
proxy_1     | }
proxy_1     |
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:482]     http filter #0
proxy_1     | [2020-11-01 14:54:47.073][7][warning][misc] [bazel-out/k8-opt/bin/source/extensions/common/_virtual_includes/utility_lib/extensions/common/utility.h:65] Using deprecated extension name 'envoy.router' for 'envoy.filters.http.router'. This name will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details.
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:504]       name: envoy.router
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:511]     config: {}
proxy_1     |
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/server/filter_chain_manager_impl.cc:218] new fc_contexts has 1 filter chains, including 1 newly built
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/target_impl.cc:15] init manager Server initializing target Listener-init-target listener_0
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/manager_impl.cc:49] init manager Listener-local-init-manager listener_0 7423914739769031304 contains no targets
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/watcher_impl.cc:14] init manager Listener-local-init-manager listener_0 7423914739769031304 initialized, notifying Listener-local-init-watcher listener_0
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/watcher_impl.cc:14] target Listener-init-target listener_0 initialized, notifying init manager Server
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/server/listener_impl.cc:107] Create listen socket for listener listener_0 on address 0.0.0.0:10000
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/server/listener_impl.cc:97] Set listener listener_0 socket factory local address to 0.0.0.0:10000
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/server/listener_impl.cc:656] add active listener: name=listener_0, hash=7423914739769031304, address=0.0.0.0:10000
proxy_1     | [2020-11-01 14:54:47.073][7][info][upstream] [source/server/lds_api.cc:79] lds: add/update listener 'listener_0'
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/watcher_impl.cc:14] target LDS initialized, notifying init manager Server
proxy_1     | [2020-11-01 14:54:47.073][7][debug][init] [source/common/init/watcher_impl.cc:14] init manager Server initialized, notifying RunHelper
proxy_1     | [2020-11-01 14:54:47.073][7][info][config] [source/server/listener_manager_impl.cc:888] all dependencies initialized. starting workers
proxy_1     | [2020-11-01 14:54:47.073][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 0
proxy_1     | [2020-11-01 14:54:47.074][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 1
proxy_1     | [2020-11-01 14:54:47.074][14][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.074][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 2
proxy_1     | [2020-11-01 14:54:47.074][15][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.075][14][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.075][16][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.075][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 3
proxy_1     | [2020-11-01 14:54:47.075][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.075][14][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.075][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.075][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.075][17][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.075][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 4
proxy_1     | [2020-11-01 14:54:47.075][19][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.075][18][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.075][14][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.075][7][debug][config] [source/server/listener_manager_impl.cc:899] starting worker 5
proxy_1     | [2020-11-01 14:54:47.075][20][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.076][21][debug][main] [source/server/worker_impl.cc:127] worker entering dispatch loop
proxy_1     | [2020-11-01 14:54:47.077][19][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.077][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.077][7][debug][config] [source/common/config/filesystem_subscription_impl.cc:71] Filesystem config update accepted for /var/lib/envoy/lds.yaml: resources {
proxy_1     |   [type.googleapis.com/envoy.config.listener.v3.Listener] {
proxy_1     |     name: "listener_0"
proxy_1     |     address {
proxy_1     |       socket_address {
proxy_1     |         address: "0.0.0.0"
proxy_1     |         port_value: 10000
proxy_1     |       }
proxy_1     |     }
proxy_1     |     filter_chains {
proxy_1     |       filters {
proxy_1     |         name: "envoy.http_connection_manager"
proxy_1     |         typed_config {
proxy_1     |           [type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager] {
proxy_1     |             stat_prefix: "ingress_http"
proxy_1     |             route_config {
proxy_1     |               name: "local_route"
proxy_1     |               virtual_hosts {
proxy_1     |                 name: "local_service"
proxy_1     |                 domains: "*"
proxy_1     |                 routes {
proxy_1     |                   match {
proxy_1     |                     prefix: "/"
proxy_1     |                   }
proxy_1     |                   route {
proxy_1     |                     cluster: "example_proxy_cluster"
proxy_1     |                   }
proxy_1     |                 }
proxy_1     |               }
proxy_1     |             }
proxy_1     |             http_filters {
proxy_1     |               name: "envoy.router"
proxy_1     |             }
proxy_1     |           }
proxy_1     |         }
proxy_1     |       }
proxy_1     |     }
proxy_1     |   }
proxy_1     | }
proxy_1     | 183412668: "envoy.api.v2.DiscoveryResponse"
proxy_1     |
proxy_1     | [2020-11-01 14:54:47.078][20][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.078][20][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.078][20][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.078][24][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.078][23][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running
proxy_1     | [2020-11-01 14:54:47.078][21][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.078][17][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1040] adding TLS initial cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.078][19][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.079][21][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.079][17][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:667] updating TLS cluster example_proxy_cluster
proxy_1     | [2020-11-01 14:54:47.079][19][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.079][21][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.079][17][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1194] membership update for TLS cluster example_proxy_cluster added 1 removed 0
proxy_1     | [2020-11-01 14:54:47.078][25][debug][grpc] [source/common/grpc/google_async_client_impl.cc:50] completionThread running

> [dynamic-config-fs] Check for response comes from service1 upstream
Request served by service1

> [dynamic-config-fs] Check config for active clusters pointing to service1
                      "address": "service1",

> [dynamic-config-fs] Set upstream to service2
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: example_proxy_cluster
  connect_timeout: 1s
  type: strict_dns
  http2_protocol_options: {}
  load_assignment:
    cluster_name: example_proxy_cluster
    endpoints:
    - lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: service2
              port_value: 8080

> [dynamic-config-fs] Check for response comes from service2 upstream
ERROR: curl expected (http://localhost:10000): Request served by service2

> [dynamic-config-fs] Cleanup (.)
Stopping dynamic-config-fs_proxy_1    ... done
Stopping dynamic-config-fs_service2_1 ... done
Stopping dynamic-config-fs_service1_1 ... done
Removing dynamic-config-fs_proxy_1    ... done
Removing dynamic-config-fs_service2_1 ... done
Removing dynamic-config-fs_service1_1 ... done
Removing network dynamic-config-fs_default

Last but not least, my docker setup info, if required:

$ docker version 

Client: Docker Engine - Community
 Cloud integration  0.1.18
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 16:58:31 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker-compose version

docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

I tried and noticed the above issues on both envoyproxy/envoy-dev:latest and envoyproxy/envoy:v1.16-latest Docker images.

Thank you.
Andrea

@dili91 dili91 added bug triage Issue requires triage labels Nov 1, 2020
@phlax
Copy link
Member

phlax commented Nov 1, 2020

hi @dili91 not sure why its not working on your system exactly, but keen to figure out. It wfm locally and in CI

I have the same version of docker as you so i dont think that is the problem.

regarding removing -i from the sed command - does this not just prevent it from updating ? certainly that would be the effect on linux

@phlax
Copy link
Member

phlax commented Nov 1, 2020

...also if you test this sandbox with the ./verify.sh locally (and the sed command is working) then you will need to do something like:

git checkout configs/cds.yaml
./verify.sh

if you want to test it more than once

@phlax
Copy link
Member

phlax commented Nov 1, 2020

/assign phlax

@mattklein123 mattklein123 added area/docs help wanted Needs help! and removed bug triage Issue requires triage labels Nov 1, 2020
@dili91
Copy link
Contributor Author

dili91 commented Nov 1, 2020

Hey @phlax thanks for the prompt reply. Mac comes with a slightly different version of sed compared to GNU's one.

Below command inside verify.sh returns an error and does not replace service1 with service2 on cds.yaml file.
sed -i s/service1/service2/ configs/cds.yaml

I can easily overcome this by either replacing -i with -e or using gsed, that is GNU sed. Earlier I wrote removing, but actually I meant changing the above command to this:

sed -e s/service1/service2/ configs/cds.yaml

That said, I ran the script multiple times as well as testing on my own (manually changing the values) but the updated file is not picked up by the envoy process running on the Docker container.

I'm keen to provide extra troubleshooting details if any

@phlax
Copy link
Member

phlax commented Nov 1, 2020

@dili91 so 2 things

fixing the sed command

im keen to update the verify.sh script to make it work on mac (i have no easy access to any versions atm)

this has always been a bit of a mess on mac, and seems to still be - see https://stackoverflow.com/questions/4247068/sed-command-with-i-option-failing-on-mac-but-works-on-linux

i dont think -e cuts it - it just outputs to the term.

does a variant of this work ?

sed -i'.bak' -e 's/old_link/new_link/g' *

if i can get a decent command to work on both ill update the script

debugging your env

can you confirm that after running verify.sh the cds file has changed

@phlax
Copy link
Member

phlax commented Nov 1, 2020

if the file is getting updated and the example is still not working then my guess is its something to do with file inodes and envoy not seeing the file as changed (its the only thing i can think of atm)

if this is the case, the first question would be whether the timestamp has changed inside the container

@phlax
Copy link
Member

phlax commented Nov 1, 2020

maybe relevant moby/moby#15793 (comment)

@dili91
Copy link
Contributor Author

dili91 commented Nov 1, 2020

Hey, so:

i dont think -e cuts it - it just outputs to the term.

Yeah sorry, you're right. But afterwards I manually changed the file with vim.

does a variant of this work ?
sed -i'.bak' -e 's/old_link/new_link/g' *

This one applies the desired changes to the file: sed -i'.bak' -e 's/service1/service2/g' configs/cds.yaml. But envoy process in the container does not pick it up

Yes, timestamp is updated:

$ docker exec dynamic-config-fs_proxy_1 ls -alh /var/lib/envoy


total 16K
drwxrwxrwx 5 root root  160 Nov  1 17:22 .
drwxr-xr-x 1 root root 4.0K Nov  1 17:22 ..
-rw-r--r-- 1 root root  396 Nov  1 17:22 cds.yaml
-rw-r--r-- 1 root root  396 Nov  1 17:21 cds.yaml.bak
-rw-r--r-- 1 root root  752 Nov  1 14:36 lds.yaml

Last but not least, during my manual changes I used vim. I just found out that vim and inotify seem not to be happy toghether. See this thread as reference.

I solved by updating the file like this, from the host system (my Mac):

docker exec dynamic-config-fs_proxy_1 sed -i s/service1/service2/ /var/lib/envoy/cds.yaml

I also tried to replace the original sed command in the verify.sh script with the above command and the verification passed. If interested, I could raise a PR to trigger this update on the container rather than updating the host file.
There is a last outstanding issue likely linked to inodes - as you said - but I'm OK with what I found out. I would just mention in the documentation to make sure to use sed or other tools that override the file like sed does.

@phlax
Copy link
Member

phlax commented Nov 1, 2020

I solved by updating the file like this, from the host system (my Mac):

great solution!

this also removes the need to give the umask warning in the docs

can you PR to update ?

if not, im happy to

@phlax
Copy link
Member

phlax commented Nov 1, 2020

(...just read rest of post...)

yep please! do PR!

@phlax
Copy link
Member

phlax commented Nov 1, 2020

on the exec its better to add -T for ci

@dili91
Copy link
Contributor Author

dili91 commented Nov 1, 2020

Sure, will raise a PR then !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants