Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lotus daemon --api setting overrides LOTUS_API_LISTENADDRESS without warning #12545

Open
5 of 11 tasks
jakobilobi opened this issue Oct 2, 2024 · 2 comments
Open
5 of 11 tasks

Lotus daemon --api setting overrides LOTUS_API_LISTENADDRESS without warning #12545

jakobilobi opened this issue Oct 2, 2024 · 2 comments
Labels
good first issue Good for newcomers

Comments

@jakobilobi
Copy link

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus fvm/fevm - Lotus FVM and FEVM interactions
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt/WinningPoSt)
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

Daemon: lotus version 1.29.1+mainnet+git.3a3395d

Repro Steps

  1. Start the lotus daemon with env variable LOTUS_API_LISTENADDRESS set to /ip4/0.0.0.0/tcp/1234/http.
  2. Also set the daemon --api setting as --api=1234.
  3. Attempt to reach the API endpoint from an external source, e.g. another machine, on the same network.
  4. Get Failed to connect to <redacted IP>: Connection refused.

Describe the Bug

Hi there!
I'm new to this space and I found a potential bug when setting up a Lotus full node. My issue was that when I tried to connect my Prometheus (running on another machine) to the Lotus node, to scrape its /debug/metrics endpoint, it was not getting through.

What I found was that the --api=1234 CLI arg to the lotus daemon command overrides the env var I had set for the service running the daemon. It both overrides the env var's port and address, even though the setting only is for the port. E.g. even though I had specified 0.0.0.0 for the listen address in the env, it still ended up being 127.0.0.1:1234.

I don't know if this is intended or not, but either way it would be helpful if the lotus daemon prints what address/port it listens to when starting (kind of like the gateway prints setting up API endpoint at 0.0.0.0:2346). And perhaps even more helpful if a warning is printed in case an override occurs.

There aren't really any logs directly connected to the issue so I'll put the startup message from lotus daemon below.

Logging Information

lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        main        lotus/daemon.go:230        lotus repo: /home/lotus/.lotus
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-61fa69f38b9cc771ba27b670124714b4ea77fbeae05e377fb859c4a43b73a30c.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-fb9e095bebdd77511c0269b967b4d87ba8b8a525edaa0e165de23ba454510194.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-0-0-92180959e1918d26350b8e6cfe217bbdd0a2d8de51ebec269078b364b715ad63.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-2-102e1444a7e9a97ebf1e3d6855dcc77e66c011ea66f936d9b2c508f87f2f83a7.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-empty-sector-update-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok
lotus[42951]: 2024-10-02T12:32:15.461Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok
lotus[42951]: 2024-10-02T12:32:15.462Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok
lotus[42951]: 2024-10-02T12:32:15.462Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok
lotus[42951]: 2024-10-02T12:32:15.462Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok
lotus[42951]: 2024-10-02T12:32:15.462Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok
lotus[42951]: 2024-10-02T12:32:15.462Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok
lotus[42951]: 2024-10-02T12:32:15.464Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok
lotus[42951]: 2024-10-02T12:32:15.467Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok
lotus[42951]: 2024-10-02T12:32:15.795Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:209        Parameter file /var/tmp/filecoin-proof-parameters/v28-fil-inner-product-v1.srs is ok
lotus[42951]: 2024-10-02T12:32:15.795Z        INFO        paramfetch        go-paramfetch@v0.0.4/paramfetch.go:233        parameter and key-fetching complete
lotus[42951]: 2024-10-02T12:32:15.801Z        INFO        modules        modules/core.go:63        memory limits initialized        {"max_mem_heap": 0, "total_system_mem": 41721327616, "effective_mem_limit": 41721327616}
lotus[42951]: 2024-10-02T12:32:15.802Z        WARN        modules        modules/core.go:123        failed to initialize cgroup-driven watchdog; err: failed to load cgroup for process: cgroups: cgroup mountpoint does not exist
lotus[42951]: 2024-10-02T12:32:15.802Z        WARN        modules        modules/core.go:124        trying a system-driven watchdog
lotus[42951]: 2024-10-02T12:32:15.802Z        INFO        watchdog        go-watchdog@v1.3.0/watermarks.go:16        initialized watermark watchdog policy; watermarks: []; thresholds: [20860663808 25032796569 29204929331 35463128473 37549194854 38592228044 39635261235]
lotus[42951]: 2024-10-02T12:32:15.802Z        INFO        watchdog        go-watchdog@v1.3.0/watchdog.go:472        initialized heap profile capture; threshold: 0.900000; max captures: 10; dir: /home/lotus/.lotus/heapprof
lotus[42951]: 2024-10-02T12:32:15.802Z        INFO        modules        modules/core.go:129        initialized system-driven watchdog
lotus[42951]: 2024-10-02T12:32:15.802Z        ERROR        alerting        alerting/alerts.go:106        alert raised        {"type": {"System":"process","Subsystem":"udp-buffer-size"}, "message": {"current_size":212992,"help":"See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.","message":"UDP buffer size is low","wanted_size":2097152}}
lotus[42951]: 2024-10-02T12:32:15.803Z        INFO        p2pnode        lp2p/rcmgr.go:42        libp2p resource manager is enabled
lotus[42951]: 2024-10-02T12:32:15.803Z        DEBUG        rcmgr        resource-manager/limit.go:84        initializing new limiter with config        {"limits": {}}
lotus[42951]: 2024-10-02T12:32:15.828Z        INFO        badgerbs        v2@v2.2007.4/levels.go:183        All 0 tables opened in 0s
lotus[42951]: 2024-10-02T12:32:15.831Z        INFO        badgerbs        v2@v2.2007.4/value.go:1158        Replaying file id: 0 at offset: 0
lotus[42951]: 2024-10-02T12:32:15.831Z        INFO        badgerbs        v2@v2.2007.4/value.go:1178        Replay took: 6.011µs
lotus[42951]: 2024-10-02T12:32:15.834Z        INFO        badgerbs        v2@v2.2007.4/levels.go:183        All 1 tables opened in 2ms
lotus[42951]: 2024-10-02T12:32:15.834Z        INFO        badgerbs        v2@v2.2007.4/value.go:1158        Replaying file id: 0 at offset: 60312581
lotus[42951]: 2024-10-02T12:32:15.834Z        INFO        badgerbs        v2@v2.2007.4/value.go:1178        Replay took: 2.396µs
lotus[42951]: 2024/10/02 12:32:15 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
lotus[42951]: 2024-10-02T12:32:15.840Z        INFO        p2pnode        lp2p/addrs.go:101        Swarm listening at: [/ip4/127.0.0.1/tcp/43627 /ip4/10.36.157.81/tcp/43627 /ip6/::1/tcp/35453 /ip4/127.0.0.1/udp/58977/quic-v1 /ip4/10.36.157.81/udp/58977/quic-v1 /ip6/::1/udp/43833/quic-v1 /ip4/127.0.0.1/udp/39079/quic-v1/webtransport/certhash/uEiDo95d0bp5cTIVIJ-Mja9F-eVKPGJmfaIG9XVR00jZSnQ/certhash/uEiAQ5RbNkLkwSRqrEdhwskFhDx5w8bR8roKSHkcazJdzaA /ip4/10.36.157.81/udp/39079/quic-v1/webtransport/certhash/uEiDo95d0bp5cTIVIJ-Mja9F-eVKPGJmfaIG9XVR00jZSnQ/certhash/uEiAQ5RbNkLkwSRqrEdhwskFhDx5w8bR8roKSHkcazJdzaA /ip6/::1/udp/36268/quic-v1/webtransport/certhash/uEiDo95d0bp5cTIVIJ-Mja9F-eVKPGJmfaIG9XVR00jZSnQ/certhash/uEiAQ5RbNkLkwSRqrEdhwskFhDx5w8bR8roKSHkcazJdzaA]
lotus[42951]: 2024-10-02T12:32:15.894Z        INFO        modules        modules/services.go:153        subscribing to pubsub topic /fil/blocks/testnetnet
lotus[42951]: 2024-10-02T12:32:15.895Z        INFO        messagepool        messagepool/messagepool.go:447        mpool ready
lotus[42951]: 2024-10-02T12:32:15.896Z        INFO        modules        modules/services.go:218        relaying messages for pubsub topic /indexer/ingest/mainnet
lotus[42951]: 2024-10-02T12:32:15.898Z        INFO        splitstore        splitstore/splitstore.go:792        starting splitstore        {"baseEpoch": "0", "warmupEpoch": 0}
lotus[42951]: 2024-10-02T12:32:15.898Z        INFO        f3/dynamic-manifest        manifest/dynamic_manifest_provider.go:97        starting a dynamic manifest provider        {"manifestServerID": "12D3KooWENMwUF9YxvQxar7uBWJtZkA6amvK4xWmKXfSiHUo2Qq7"}
lotus[42951]: 2024-10-02T12:32:15.900Z        INFO        f3        go-f3@v0.2.0/f3.go:249        starting f3 reconfiguration
lotus[42951]: 2024-10-02T12:32:15.900Z        INFO        f3        go-f3@v0.2.0/f3.go:299        stopping F3 internals
lotus[42951]: 2024-10-02T12:32:15.900Z        INFO        f3        go-f3@v0.2.0/f3.go:190        F3 is starting        {"initialDelay": 0, "hasPendingManifest": false, "NetworkName": "testnetnet/32"}
lotus[42951]: 2024-10-02T12:32:15.904Z        INFO        healthcheck        node/health.go:57        waiting 1s before starting ChainNotify channel
lotus[42951]: 2024-10-02T12:32:16.905Z        INFO        healthcheck        node/health.go:70        started ChainNotify channel
lotus[42951]: 2024-10-02T12:32:20.826Z        INFO        basichost        basic/natmgr.go:112        DiscoverNAT error:no NAT found
lotus[42951]: 2024-10-02T12:32:20.894Z        INFO        peermgr        peermgr/peermgr.go:209        connecting to bootstrap peers
lotus[42951]: 2024-10-02T12:32:21.688Z        WARN        hello        hello/hello.go:117        other peer hasnt completed libp2p identify, waiting a bit
lotus[42951]: 2024-10-02T12:32:21.862Z        WARN        chain        chain/sync_manager.go:354        caution, multiple distinct chains seen during head selections
@jakobilobi jakobilobi added the kind/bug Kind: Bug label Oct 2, 2024
@rjan90
Copy link
Contributor

rjan90 commented Oct 8, 2024

2024-10-08

Thanks for writing down this issue @jakobilobi. We discussed this in our triage meeting and think logging a warning, so that it is easier to see debug or find what happened. We will put it on our board as a P2

@rjan90 rjan90 added good first issue Good for newcomers and removed kind/bug Kind: Bug labels Oct 8, 2024
@jakobilobi
Copy link
Author

2024-10-08

Thanks for writing down this issue @jakobilobi. We discussed this in our triage meeting and think logging a warning, so that it is easier to see debug or find what happened. We will put it on our board as a P2

YW. Glad you brought it up, and thanks for getting back to me 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers
Projects
FilOz
Status: 🐱 Todo
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rjan90 @jakobilobi