[tech] Mussel

[Sam Moore]

Mussel isn't running very well lately. Unfortunately we haven't caught it 
yet.

tl;dr That's all you really need to know

We think the problem is due to OOM caused by apache spawning too many 
processes. Currently apache uses about 40% of memory with no load, and has 
about 90 threads running. When something actually does request a lot of 
stuff, mussel runs out of memory.

Changing /etc/apache2/apache2.conf to reduce the number of processes 
doesn't seem to work, because there are apache processes running under the 
user 'ffxsync' (about 50 of them, 20% of mussel's memory). Also there are 
a bunch running under 'www-data' more than my maths* says should be 
allowed.

There is a script running under screen on mussel which prints thread, CPU 
and memory counts for apache as well as changes in the error or access 
logs. Hopefully the next time mussel crashes this will help determine 
whether apache is solely to blame, or something else is using the other 
60% of the memory just before it crashes.


* Maths done by matches at 1am are not gauranteed to be correct

[SZM]