Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UDP Memleak when running multiple parallel instances #52

Open
margau opened this issue Jan 11, 2020 · 3 comments
Open

UDP Memleak when running multiple parallel instances #52

margau opened this issue Jan 11, 2020 · 3 comments

Comments

@margau
Copy link
Contributor

margau commented Jan 11, 2020

Hello,
our workaround for #49 is running multiple parallel instances of mesh-announce, one per domain:

python3 /opt/mesh-announce/respondd.py -d /opt/mesh-announce/providers -i dom2-br -i dom2-tp -b dom2-bat
(Other Domains with the same scheme)

Basically it works, but unfortunately, after a few hours we have a mem-leak blocking all UDP communication on the system (including fastd): https://prometheus.ffm.freifunk.net/graph?g0.range_input=1h&g0.end_input=2020-01-09%2008%3A44&g0.expr=(node_sockstat_UDP_mem_bytes%7Binstance%3D~%22gw05.*%22%7D%20%2F%201024%20%2F%201024)%20%3E%20100&g0.tab=0

This doesn't happen if we have only one mesh-announce instance running.

As my knowledge with Python and the ThreadingUDPServer is very limited, I currently not sure how to debug this problem.

Python Version is Python 3.7.3 (default, Apr 3 2019, 05:39:12) on Stretch 5.4.0-1-amd64 #1 SMP Debian 5.4.6-1 (2019-12-27)

Anyone has some ideas?
Thanks!
margau

@tackin
Copy link

tackin commented Feb 5, 2020

Same problem for me.
Running 5 services of this fork only on our GW-erai ( https://github.com/freifunktrier/mesh-announce ) and the map ( http://maps.freifunk-trier.de/#/de/map#!/de/map/2661965025dc ) shows all kinds of stupid things with every respond-update.

  • bad uptime in one service and the correct one in the other service
  • wrong neighbours
  • no neighbours

I also tried all our net-segments with one service but failed as well.

@AiyionPrime
Copy link
Contributor

@margau on what commit are you guys currently running?
There is a nasty upstream issue with python udp-server and python 3.7 apparenty.
It got hotfixed on master, just wanted to be sure you're not affected by that.

Please share the result of this command in your mesh-announce directory:
grep -rni server.daemon_threads

@margau
Copy link
Contributor Author

margau commented Apr 7, 2020

Currently running on 8deeec1

respondd.py:104: server.daemon_threads = True

Our workaround was to increase
udp_mem in sysctl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants