TCP connection floods overwhelm the client

[zzzorgo]

Describe the bug
One of my rathole clients (not sure about the others) creates a lot of TCP connections without closing them. The half of the connections are in CLOSE_WAIT state and the other half in ESTABLISHED. This could be as bad as 1300+ simultaneous TCP connections for only two services.

To Reproduce
Steps to reproduce the behavior:

1. Start the server and the client
2. Wait for about 24 hours

Configuration
Configuration used to reproduce the behavior:

# client.toml
[client]
remote_addr = "XXXX:XXXX"

[client.transport]
type = "tls"

[client.transport.tls]
trusted_root = "XXXX.pem"

[client.services.local_balancer]
token = "XXXX"
local_addr = "127.0.0.1:8081"

[client.services.local_ssh]
token = "XXXX"
local_addr = "127.0.0.1:22"

# server.toml
[server]
bind_addr = "XXXX:XXXX"

[server.transport]
type = "tls"

[server.transport.tls]
pkcs12 = "XXXX.pfx"
pkcs12_password = "XXXX"

[server.services.local_balancer]
token = "XXXX"
bind_addr = "0.0.0.0:64115"

[server.services.local_ssh]
token = "XXXX"
bind_addr = "0.0.0.0:64116"

[server.services.local_balancer_bryansk]
token = "XXXX"
bind_addr = "0.0.0.0:64117"

[server.services.local_ssh_bryansk]
token = "XXXX"
bind_addr = "0.0.0.0:64118"

Logs
Rathole log
log.txt

Netstat output when the flood is there
netstat.log

Netstat output after rathole process restarting
netstat-after-restarting.log

Environment:

• OS: Ubuntu 22.04.4 LTS
• rathole --version output:

rathole 
Build Timestamp:     2024-05-19T09:14:41.914676647Z
Build Version:       0.5.0
Commit SHA:          None
Commit Date:         None
Commit Branch:       None
cargo Target Triple: x86_64-unknown-linux-musl
cargo Profile:       release
cargo Features:      base64,client,default,futures_core,futures_sink,hot_reload,native_tls,noise,notify,server,snowstorm,tokio_native_tls,tokio_tungstenite,tokio_util,websocket_native_tls

• CPU architecture: x86
• rustc version: not sure, most probably 1.78 (official docker image, compiled 18 days ago)

[aman-sharma08]

Having similar issue in my deployment. Can someone assist?

[rucciva]

Hey @zzzorgo, do you think if rathole could use http2 bidirectional streaming as transport would solve this problem? Since single tcp connection could cary multiple http2 stream.

Related to #392

[zzzorgo]

Hi @rucciva, I am far from being an expert in the field but I don't think that it is a practical solution (but again I am just a user of this app). You see, both http2 and rathole are using TCP connections as a transport. And if I am getting it right you are suggesting putting rathole on top of http2 on top of TCP which will add one extra layer of abstraction (read as extra complexity) and most probably will require a big change to the app.

The alternative is just fixing the damn bug with TCP connections not being terminated (or reused?) properly :D