Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The dfget daemon program channel is created repeatedly #3471

Open
The-half opened this issue Sep 2, 2024 · 1 comment
Open

The dfget daemon program channel is created repeatedly #3471

The-half opened this issue Sep 2, 2024 · 1 comment
Labels
bug

Comments

@The-half
Copy link

The-half commented Sep 2, 2024 •
edited
Loading

Bug report:

在本地执行dfget daemon代理程序，用来代理gitlab 的lfs 仓库，执行命令为:"dfget daemon --config=dfget-peer.yaml --verbose" 但执行后一直报出上述问题，channnel 每隔十秒重试连接；补充， dragonfly 服务端均已使用二进制部署完成

  • dfget daemon 的日志如下：
2024-09-02T15:45:10.316+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel created
2024-09-02T15:45:10.316+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"passthrough", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/10.10.101.123:8002", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2024-09-02T15:45:10.316+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel authority set to "10.10.101.123:8002"
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Resolver state updated: {
  "Addresses": [
    {
      "Addr": "10.10.101.123:8002",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "10.10.101.123:8002",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel switches to new LB policy "pick_first"
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel created
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel Connectivity change to CONNECTING
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel exiting idle mode
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel Connectivity change to CONNECTING
2024-09-02T15:45:10.317+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel picks a new address "10.10.101.123:8002" to connect
2024-09-02T15:45:10.319+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel Connectivity change to READY
2024-09-02T15:45:10.319+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel Connectivity change to READY
2024-09-02T15:45:10.321+0800    DEBUG   zap/client_interceptors.go:52   finished client unary call      {"system": "grpc", "span.kind": "client", "grpc.service": "grpc.health.v1.Health", "grpc.method": "Check", "grpc.code": "OK", "grpc.time_ms": 4.392}
2024-09-02T15:45:10.321+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel Connectivity change to SHUTDOWN
2024-09-02T15:45:10.322+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Closing the name resolver
2024-09-02T15:45:10.322+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]ccBalancerWrapper: closing
2024-09-02T15:45:10.322+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel Connectivity change to SHUTDOWN
2024-09-02T15:45:10.322+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51 SubChannel #52]Subchannel deleted
2024-09-02T15:45:10.322+0800    INFO    grpclog/grpclog.go:37   [core][Channel #51]Channel deleted
  • dfget-peer.yaml的配置文件如下:
aliveTime: 0s
gcInterval: 1m0s
workHome: ''
workHomeMode: 0755
logDir: ''
cacheDir: ''
cacheDirMode: 0755
pluginDir: ''
dataDir: ''
dataDirMode: 0755
keepStorage: false
console: true
verbose: true
pprof-port: -1
jaeger: ''
scheduler:
  manager:
    enable: true
    netAddrs:
      - type: tcp
        addr: 10.10.101.123:65003
    refreshInterval: 10m
    seedPeer:
      enable: true
      type: super
      clusterID: 1
      keepAlive:
        internal: 5s
  scheduleTimeout: 30s
  disableAutoBackSource: false
host:
  location: ''
  idc: ''
download:
  calculateDigest: false
  totalRateLimit: 1024Mi
  perPeerRateLimit: 512Mi
  pieceDownloadTimeout: 30s
  prefetch: false
  transportOption:
    dialTimeout: 2s
    keepAlive: 30s
    maxIdleConns: 100
    idleConnTimeout: 90s
    responseHeaderTimeout: 2s
    tlsHandshakeTimeout: 1s
    expectContinueTimeout: 2s
  resourceClients:
    https:
      dialTimeout: 30s
      keepAlive: 30s
      maxIdleConns: 100
      idleConnTimeout: 90s
      responseHeaderTimeout: 30s
      tlsHandshakeTimeout: 30s
      expectContinueTimeout: 10s
      insecureSkipVerify: true
    http:
      dialTimeout: 30s
      keepAlive: 30s
      maxIdleConns: 100
      idleConnTimeout: 90s
      responseHeaderTimeout: 30s
      tlsHandshakeTimeout: 30s
      expectContinueTimeout: 10s
    oras:
      dialTimeout: 30s
      keepAlive: 30s
      maxIdleConns: 100
      idleConnTimeout: 90s
      responseHeaderTimeout: 30s
      tlsHandshakeTimeout: 30s
      expectContinueTimeout: 10s
      insecureSkipVerify: true
  concurrent:
    thresholdSize: 10M
    thresholdSpeed: 2M
    goroutineCount: 4
    initBackoff: 0.5
    maxBackoff: 3
    maxAttempts: 3
  downloadGRPC:
    security:
      insecure: true
      cacert: ''
      cert: ''
      key: ''
      tlsVerify: true
      tlsConfig: null
    unixListen:
      socket: ''
  peerGRPC:
    security:
      insecure: true
      cacert: ''
      cert: ''
      key: ''
      tlsVerify: true
    tcpListen:
      port: 65000
upload:
  rateLimit: 1024Mi
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    port: 65002
objectStorage:
  enable: false
  filter: 'Expires&Signature&ns'
  maxReplicas: 3
  security:
    insecure: true
    tlsVerify: true
  tcpListen:
    port: 65004
storage:
  taskExpireTime: 6h
  strategy: io.d7y.storage.v2.simple
  diskGCThreshold: 50Gi
  diskGCThresholdPercent: 80
  multiplex: true
health:
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    path: /server/ping
    port: 40901
proxy:
  defaultFilter: 'Expires&Signature&ns'
  defaultTag: ''
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    namespace: ''
    port: 65001
  registryMirror:
    dynamic: false
    url: https://index.docker.io
    insecure: true
    certs: []
    direct: false
    useProxies: false
  proxies:
    - regx: blobs/sha256.*
    - regx: some-registry/
      useHTTPS: true
    - regx: no-proxy-reg
      direct: true
    - regx: some-registry
      redirect: another-registry
    - regx: ^http://some-registry/(.*)
      redirect: http://another-registry/$1
    - regex: '.*gitlab.example.com:32415'
      use_https: true
      insecure: true
      redirect: 'http://localhost:65002'
  hijackHTTPS:
    cert: ''
    key: ''
    hosts:
      - regx: gitlab.example.com:32415
        use_https: true
        insecure: true
        certs: []
  maxConcurrency: 0
  whiteList:
    - host: ''
      regx:
      ports:
  basicAuth:
    username: 'admin'
    password: 'password'
security:
  autoIssueCert: false
  caCert: ''
  tlsVerify: false
  tlsPolicy: 'prefer'
  certSpec:
    dnsNames:
    ipAddresses:
    validityPeriod: 4320h
network:
  enableIPv6: false
announcer:
  schedulerInterval: 30s
networkTopology:
  enable: true
  probe:
    interval: 20m
  • 其中dragonfly 的manager 程序的 /var/log/dragonfly/manger/core.log 的部分日志字段为
{"level":"warn","ts":"2024-09-02 11:50:44.193","caller":"rpcserver/manager_server_v2.go:682","msg":"manager:applications cache miss because of cache: key is missing","hostname":"k8s-master","ip":"10.10.101.123","stacktrace":"d7y.io/dragonfly/v2/manager/rpcserver.(*managerServerV2).ListApplications\n\t/home/runner/work/Dragonfly2/Dragonfly2/manager/rpcserver/manager_server_v2.go:682\nd7y.io/api/v2/pkg/apis/manager/v2._Manager_ListApplications_Handler.func1\n\t/home/runner/go/pkg/mod/d7y.io/api/v2@v2.0.148/pkg/apis/manager/v2/manager_grpc.pb.go:366\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func5\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/logging/zap/server_interceptors.go:31\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7.1\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.UnaryServerInterceptor.func4\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/ratelimit/ratelimit.go:24\nd7y.io/dragonfly/v2/pkg/rpc/manager/server.New.ChainUnaryServer.func7\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\nd7y.io/api/v2/pkg/apis/manager/v2._Manager_ListApplications_Handler\n\t/home/runner/go/pkg/mod/d7y.io/api/v2@v2.0.148/pkg/apis/manager/v2/manager_grpc.pb.go:368\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1379\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1790\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029"}
image

会不会 是dragonfly 的服务端的listApplications 访问不到，导致的客户端grpc 连接重试

Environment:

  • Dragonfly version: v2.1.55
  • OS: centos 7.9
  • Kernel (e.g. uname -a): 3.10.0-693.el7.x86_64
  • Others: 使用 二进制部署的dragonfly 服务端，服务端均正常，可以正常访问网页
@The-half The-half added the bug label Sep 2, 2024
@gaius-qi
Copy link
Member

@The-half Refer to https://d7y.io/docs/next/operations/integrations/git-lfs/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gaius-qi @The-half