Are there any plans to support distributed networking?

[dehoej]

Lots of webservices require all kinds of networking involved. Configuring this manually using ports is both unsecure and difficult to automate. Therefore i am thinking if you are planning to implement support for distributed network or in docker terms: overlay networking which is encrypted and cross-node. Would be extremely useful, but i don't know if its possible to use outside docker swarm.

[scart88]

I managed to workaround this with multiple bare-metal servers with Proxmox installed within a Private VLAN.

Node 1

/etc/network/interfaces

auto lo
iface lo inet loopback

iface enp5s0 inet manual
iface enp6s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address [public-ipV4]/28
        gateway [public-ipV4-gateway]
        bridge-ports enp5s0
        bridge-stp off
        bridge-fd 0
        up ip route add [secondary-public-ipv4]/32 dev vmbr0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/enp5s0/proxy_arp

auto vmbr0
iface vmbr0 inet static
        address 10.1.0.2/24
        post-up iptables -t nat -A POSTROUTING -s '10.1.0.1/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.1.0.1/24' -o vmbr0 -j MASQUERADE

iface vmbr0 inet6 static
        address [ipv6]::1000/64
        gateway [ipv6]::1

Node 2

/etc/network/interfaces

auto lo
iface lo inet loopback
iface enp2s0f0 inet manual
iface enp2s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address [public-ipV4]/28
        gateway [public-ipV4-gateway]
        bridge-ports enp2s0f0
        bridge-stp off
        bridge-fd 0
        up ip route add [secondary-public-ipv4]/32 dev vmbr0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/enp2s0f0/proxy_arp

auto vmbr0
iface vmbr0 inet static
        address 10.1.0.3/24
        post-up iptables -t nat -A POSTROUTING -s '10.1.0.1/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.1.0.1/24' -o vmbr0 -j MASQUERADE

iface vmbr0 inet6 static
        address [ipv6]::1001/64
        gateway [ipv6]::1

Now you can create private containers that cam communicate on the private network you just created.
Each Proxmox VM/Container can have any ip address in the /24 subnet.
For example on Node 1, you can assign the private ipv4 10.1.0.100/24 with gateway 10.1.0.2
On Node 2, you can assign the private ipv4 10.1.0.101 with gateway 10.1.0.3

Now, you can protect it even further by adding a datacenter firewall and block ssh.

You can also try Nebula, defined[dot]net, tailscale, netmaker to create a mesh network, however, this is faster than a mesh network.

Then you can deploy MRSK using a jump host or you can create a container, add Tailscale to that container and you will have access to to the private network and containers inside your home network.

sudo tailscale up --advertise-routes=10.1.0.0/24 --accept-routes
tailscale status

Add Tailscale to your device and you have access to all your private VMs in the 10.1.0.0/24 subnet from your home network.

If you want try the Nebula route, here is a great introductory video:
https://www.youtube.com/watch?v=aImSCypCsuw

[dehoej]

I guess the docker containers will not get an ip on that private network, or am i wrong here. That is what i am searching for. An isolated container network.

[scart88]

You can use host and add-host

Here you can find a few examples:

accessories:
  redis:
    image: redis:7.0
    host: 10.1.0.60
    port: "10.1.0.60:6379:6379"
    add-host: 10.1.0.60
    directories:
      - redis:/data
  db:
    image: postgres:15.2
    host: 10.1.0.61
    port: "10.1.0.61:5432:5432"
    add-host: 10.1.0.61
    env:
      clear:
        POSTGRES_USER: <%= ENV["POSTGRES_USER"] %>
        POSTGRES_DB: <%= ENV["POSTGRES_DB"] %>
        POSTGRES_PASSWORD: <%= ENV["POSTGRES_PASSWORD"] %>
    directories:
      - db:/var/lib/postgresql/data
    files:
      - db/postgresql.conf:/etc/postgresql/postgresql.conf
    cmd: postgres -c 'config_file=/etc/postgresql/postgresql.conf'
   
  duplicati:
    image: lscr.io/linuxserver/duplicati:2.0.7
    host: 10.1.0.62
    port: "10.1.0.62:8200:8200"
    add-host: 10.1.0.62
    env:
      clear:
        PUID: 0
        PGID: 0
        TZ: Etc/UTC
    volumes:
      - /:/source/hostfiles
    directories:
      - duplicati/appdata/config:/config
      - duplicati/backups:/backups

Here is a more complex example for Redis Sentinel
#382

Here you can find more general examples:
#355

Please note: If you are using DigitalOcean, Vultr, Linode, Hetzner or any hosting provider you will get a private ipv4 address on each vm/droplet you create and they can communicate within the VPC network in the same region. You don't really need Proxmox or a Private VLAN to make this work, you can use the private IPs you get on your VMs.

However, you will get more performance/value with Proxmox on a bare-metal server.
The only downside is that you have to manage that yourself and learn all the networking stuff.

For example, I migrated from a managed kubernetes cluster with 32gb RAM and 16 vCPUs to 4x bare metal servers for the same price. Now I'm using MRSK on these bare-metal servers and have 256GB Ram and 108 vCPUs: 1x Ryzen 9 7900 (24 threads) and 3x Xeon E5-2680 v4 (28 threads).

[dehoej]

I don't see any documentation on the add-host feature. Is it an equivalent of the docker --add-host parameter? Does this mean it will be added to all hosts like this:
service-name ?

[scart88]

Yes. Is the equivalent of the docker --add-host parameter.

You can learn more from this article:
https://www.stefanwienert.de/blog/2023/03/30/rails-mrsk-deployment-migrating-from-ansible-capistrano-like-to-single-host-docker-deployment-using-mrsk/