Based on repository activity, growth velocity and community engagement.
40
Growth9/30
Activity2/25
Popularity13/25
Trust17/20
1,288
Stars
High
Sentiment
Votes
1,288
README.md
nohang
nohang package provides a highly configurable daemon for Linux which is able to correctly prevent out of memory (OOM) and keep system responsiveness in low memory conditions.
The package also includes additional diagnostic tools (oom-sort, psi2log, psi-top).
What is the problem?
OOM conditions may cause freezes, livelocks, drop caches and processes to be killed (via sending SIGKILL) instead of trying to terminate them correctly (via sending SIGTERM or takes other corrective action). Some applications may crash if it's impossible to allocate memory.
Here are the statements of some users:
"How do I prevent Linux from freezing when out of memory?
Today I (accidentally) ran some program on my Linux box that quickly used a lot of memory. My system froze, became unresponsive and thus I was unable to kill the offender.
How can I prevent this in the future? Can't it at least keep a responsive core or something running?"
"With or without swap it still freezes before the OOM killer gets run automatically. This is really a kernel bug that should be fixed (i.e. run OOM killer earlier, before dropping all disk cache). Unfortunately kernel developers and a lot of other folk fail to see the problem. Common suggestions such as disable/enable swap, buy more RAM, run less processes, set limits etc. do not address the underlying problem that the kernel's low memory handling sucks camel's balls."
"The traditional Linux OOM killer works fine in some cases, but in others it kicks in too late, resulting in the system entering a livelock for an indeterminate period."
earlyoom: This is a simple, stable and tiny OOM prevention daemon written in C (the best choice for emedded and old servers). It has a minimum dependencies and can work with oldest kernels. It is enabled by default on Fedora 32 Workstation (and F33 KDE).
oomd: This is a userspace OOM killer for linux systems written in C++ and developed by Facebook. This is the best choice for use in large data centers. It needs Linux 4.20+.
systemd-oomd: Provided by systemd as systemd-oomd.service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take action on processes before an OOM occurs in kernel space. It's used by default on desktop versions of Fedora 34.
nohang: nohang is earlyoom on steroids and has many useful features, see below. Maybe this is a good choice for modern desktops and servers if you need fine-tuning. Previously it was used by default on Garuda Linux.
Use these tools to improve responsiveness during heavy swapping:
MGLRU patchset is merged in Linux 6.1. Setting min_ttl_ms > 50 can help you.
le9-patch: [PATCH] mm: Protect clean file pages under memory pressure to prevent thrashing, avoid high latency and prevent livelock in near-OOM conditions. It's kernel-side solution that can fix the OOM killer behavior.
prelockd: Lock executables and shared libraries in memory to improve system responsiveness under low-memory conditions.
memavaild: Keep amount of available memory by evicting memory of selected cgroups into swap space.
uresourced: This daemon will give resource allocations to active graphical users. It's enabled by default on Fedora 33 Workstation.
Sending the SIGTERM signal is default corrective action. If the victim does not respond to SIGTERM, with a further drop in the level of memory it gets SIGKILL;
Customizing victim selection: impact on the badness of processes via matching their names, cgroups, exe realpathes, environs, cmdlines and euids with specified regular expressions;
Customizing corrective actions: if the name or control group of the victim matches a certain regex pattern, you can run any command instead of sending the SIGTERM signal (the default corrective action) to the victim. For example:
systemctl restart foo;
kill -INT $PID (you can override the signal sent to the victim, $PID will be replaced by the victim's PID).
GUI notifications:
Notification of corrective actions taken and displaying the name and PID of the victim;
https://youtu.be/PLVWgNrVNlc – opening multiple chromium tabs with 2.3 GiB memory and 1.8 GiB swap space on zram.
Requirements
For basic usage:
Linux (>= 3.14, since MemAvailable appeared in /proc/meminfo)
Python (>= 3.3)
To respond to PSI metrics (optional):
Linux (>= 4.20) with CONFIG_PSI=y
To show GUI notifications (optional):
notification server (most of desktop environments use their own implementations)
libnotify (Arch Linux, Fedora, openSUSE) or libnotify-bin (Debian GNU/Linux, Ubuntu)
sudo or doas if nohang started with UID=0.
Memory and CPU usage
VmRSS is about 10–14 MiB instead of the settings, about 10–11 MiB by default (with Python <= 3.8), about 16–17 MiB with Python 3.9.
CPU usage depends on the level of available memory and monitoring intensity.
Warnings
the daemon runs with super-user privileges and has full access to all private memory of all processes and sensitive user data;
the daemon does not forbid you to shoot yourself in the foot: with some settings, unwanted killings of processes can occur;
the daemon is not a panacea: there are no universal settings that reliably protect against all types of threats.
Known problems
The documentation is terrible.
The ZFS ARC cache is memory-reclaimable, like the Linux buffer cache. However, in contrast to the buffer cache, it currently does not count to MemAvailable (see openzfs/zfs#10255). See also https://github.com/rfjakob/earlyoom/pull/191 and https://github.com/hakavlad/nohang/issues/89.
Linux kernels without CONFIG_CGROUP_CPUACCT=y (linux-ck, for example) provide incorrect PSI metrics, see issue.
nohang vs nohang-desktop
nohang comes with two configs: nohang.conf and nohang-desktop.conf. nohang comes with two systemd service unit files: nohang.service and nohang-desktop.service. Choose one.
nohang.conf provides vanilla default settings without PSI checking enabled, without any badness correction and without GUI notifications enabled.
nohang-desktop.conf provides default settings optimized for desktop usage.
$ nohang -h
usage: nohang [-h|--help] [-v|--version] [-m|--memload]
[-c|--config CONFIG] [--check] [--monitor] [--tasks]
optional arguments:
-h, --help show this help message and exit
-v, --version show version of installed package and exit
-m, --memload consume memory until 40 MiB (MemAvailable + SwapFree)
remain free, and terminate the process
-c CONFIG, --config CONFIG
path to the config file. This should only be used
with one of the following options:
--monitor, --tasks, --check
--check check and show the configuration and exit. This should
only be used with -c/--config CONFIG option
--monitor start monitoring. This should only be used with
-c/--config CONFIG option
--tasks show tasks state and exit. This should only be used
with -c/--config CONFIG option
How to configure
The program can be configured by editing the config file. The configuration includes the following sections:
Checking kernel messages for OOM events;
Common zram settings;
Common PSI settings;
Poll rate;
Warnings and notifications;
Soft threshold;
Hard threshold;
Customize victim selection;
Customize soft corrective actions;
Misc settings;
Verbosity, debug, logging.
Just read the description of the parameters and edit the values. Please restart the daemon to apply the changes.
How to test nohang
The safest way is to run nohang --memload. This causes memory consumption, and the process will exits before OOM occurs.
Another way is to run tail /dev/zero. This causes fast memory comsumption and causes OOM at the end.
If testing occurs while nohang is running, these processes should be terminated before OOM occurs.
Tasks state
Run sudo nohang -c/--config CONFIG --tasks to see the table of prosesses with their badness values, oom_scores, names, UIDs etc.
To view the latest entries in the log (for systemd users):
sudo journalctl -eu nohang.service
#### or
sudo journalctl -eu nohang-desktop.service
You can also enable separate_log in the config to logging in /var/log/nohang/nohang.log.
oom-sort
oom-sort is an additional diagnostic tool that will be installed with nohang package. It sorts the processes in descending order of their oom_score and also displays oom_score_adj, Uid, Pid, Name, VmRSS, VmSwap and optionally cmdline. Run oom-sort --help for more info. Man page: oom-sort.manpage.md.
Usage:
oom-sort
oom_score oom_score_adj UID PID Name VmRSS VmSwap cmdline
--------- ------------- ---- ----- --------------- ------- -------- -------
23 0 0 964 Xorg 58 M 22 M /usr/libexec/Xorg -background none :0 vt01 -nolisten tcp -novtswitch -auth /var/run/lxdm/lxdm-:0.auth
13 0 1000 1365 pcmanfm 38 M 10 M pcmanfm --desktop --profile LXDE
10 0 1000 1408 dnfdragora-upda 9 M 27 M /usr/bin/python3 /bin/dnfdragora-updater
5 0 0 822 firewalld 0 M 19 M /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid
5 0 1000 1364 lxpanel 18 M 2 M lxpanel --profile LXDE
5 0 1000 1685 nm-applet 6 M 12 M nm-applet
5 0 1000 1862 lxterminal 16 M 2 M lxterminal
4 0 996 890 polkitd 8 M 6 M /usr/lib/polkit-1/polkitd --no-debug
4 0 1000 1703 pnmixer 6 M 11 M pnmixer
3 0 0 649 systemd-journal 10 M 1 M /usr/lib/systemd/systemd-journald
3 0 1000 1360 openbox 9 M 2 M openbox --config-file /home/user/.config/openbox/lxde-rc.xml
3 0 1000 1363 notification-da 3 M 10 M /usr/libexec/notification-daemon
2 0 1000 1744 clipit 5 M 3 M clipit
2 0 1000 2619 python3 9 M 0 M python3 /bin/oom-sort
1 0 0 809 rsyslogd 3 M 3 M /usr/sbin/rsyslogd -n
1 0 0 825 udisksd 2 M 2 M /usr/libexec/udisks2/udisksd
1 0 0 873 sssd_nss 4 M 1 M /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
1 0 0 876 systemd-logind 2 M 2 M /usr/lib/systemd/systemd-logind
1 0 0 907 abrt-dump-journ 2 M 1 M /usr/bin/abrt-dump-journal-oops -fxtD
1 0 0 920 NetworkManager 3 M 2 M /usr/sbin/NetworkManager --no-daemon
1 0 1000 1115 systemd 4 M 1 M /usr/lib/systemd/systemd --user
1 0 1000 1118 (sd-pam) 0 M 5 M (sd-pam)
1 0 1000 1366 xscreensaver 5 M 0 M xscreensaver -no-splash
1 0 1000 1851 gvfsd-trash 3 M 1 M /usr/libexec/gvfsd-trash --spawner :1.6 /org/gtk/gvfs/exec_spaw/0
1 0 1000 1969 gvfsd-metadata 6 M 0 M /usr/libexec/gvfsd-metadata
1 0 1000 2262 bash 5 M 0 M bash
0 -1000 0 675 systemd-udevd 0 M 4 M /usr/lib/systemd/systemd-udevd
0 -1000 0 787 auditd 0 M 1 M /sbin/auditd
0 0 0 807 ModemManager 0 M 1 M /usr/sbin/ModemManager
0 0 0 808 smartd 0 M 1 M /usr/sbin/smartd -n -q never
0 0 0 810 alsactl 0 M 0 M /usr/sbin/alsactl -s -n 19 -c -E ALSA_CONFIG_PATH=/etc/alsa/alsactl.conf --initfile=/lib/alsa/init/00main rdaemon
0 0 0 811 mcelog 0 M 0 M /usr/sbin/mcelog --ignorenodev --daemon --foreground
0 0 172 813 rtkit-daemon 0 M 0 M /usr/libexec/rtkit-daemon
0 0 0 814 VBoxService 0 M 1 M /usr/sbin/VBoxService -f
0 0 0 817 rngd 0 M 1 M /sbin/rngd -f
0 -900 81 818 dbus-daemon 3 M 0 M /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
0 0 0 823 irqbalance 0 M 0 M /usr/sbin/irqbalance --foreground
0 0 70 824 avahi-daemon 0 M 0 M avahi-daemon: running [linux.local]
0 0 0 826 sssd 0 M 2 M /usr/sbin/sssd -i --logger=files
0 0 995 838 chronyd 1 M 0 M /usr/sbin/chronyd
0 0 0 849 gssproxy 0 M 1 M /usr/sbin/gssproxy -D
0 0 0 866 abrtd 0 M 2 M /usr/sbin/abrtd -d -s
0 0 70 870 avahi-daemon 0 M 0 M avahi-daemon: chroot helper
0 0 0 871 sssd_be 0 M 2 M /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
0 0 0 875 accounts-daemon 0 M 1 M /usr/libexec/accounts-daemon
0 0 0 906 abrt-dump-journ 1 M 2 M /usr/bin/abrt-dump-journal-core -D -T -f -e
0 0 0 908 abrt-dump-journ 1 M 2 M /usr/bin/abrt-dump-journal-xorg -fxtD
0 0 0 950 crond 2 M 1 M /usr/sbin/crond -n
0 0 0 951 atd 0 M 0 M /usr/sbin/atd -f
0 0 0 953 lxdm-binary 0 M 0 M /usr/sbin/lxdm-binary
0 0 0 1060 dhclient 0 M 2 M /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp0s3.pid -lf /var/lib/NetworkManager/dhclient-939eab05-4796-3792-af24-9f76cf53ca7f-enp0s3.lease -cf /var/lib/NetworkManager/dhclient-enp0s3.conf enp0s3
0 0 0 1105 lxdm-session 0 M 1 M /usr/libexec/lxdm-session
0 0 1000 1123 pulseaudio 0 M 3 M /usr/bin/pulseaudio --daemonize=no
0 0 1000 1124 lxsession 1 M 2 M /usr/bin/lxsession -s LXDE -e LXDE
0 0 1000 1134 dbus-daemon 2 M 0 M /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
0 0 1000 1215 imsettings-daem 0 M 1 M /usr/libexec/imsettings-daemon
0 0 1000 1218 gvfsd 3 M 1 M /usr/libexec/gvfsd
0 0 1000 1223 gvfsd-fuse 0 M 1 M /usr/libexec/gvfsd-fuse /run/user/1000/gvfs -f -o big_writes
0 0 1000 1309 VBoxClient 0 M 0 M /usr/bin/VBoxClient --display
0 0 1000 1310 VBoxClient 0 M 0 M /usr/bin/VBoxClient --clipboard
0 0 1000 1311 VBoxClient 0 M 0 M /usr/bin/VBoxClient --draganddrop
0 0 1000 1312 VBoxClient 0 M 0 M /usr/bin/VBoxClient --display
0 0 1000 1313 VBoxClient 1 M 0 M /usr/bin/VBoxClient --clipboard
0 0 1000 1316 VBoxClient 0 M 0 M /usr/bin/VBoxClient --seamless
0 0 1000 1318 VBoxClient 0 M 0 M /usr/bin/VBoxClient --seamless
0 0 1000 1320 VBoxClient 0 M 0 M /usr/bin/VBoxClient --draganddrop
0 0 1000 1334 ssh-agent 0 M 0 M /usr/bin/ssh-agent /bin/sh -c exec -l bash -c "/usr/bin/startlxde"
0 0 1000 1362 lxpolkit 0 M 1 M lxpolkit
0 0 1000 1370 lxclipboard 0 M 1 M lxclipboard
0 0 1000 1373 ssh-agent 0 M 1 M /usr/bin/ssh-agent -s
0 0 1000 1485 agent 0 M 1 M /usr/libexec/geoclue-2.0/demos/agent
0 0 1000 1751 menu-cached 0 M 1 M /usr/libexec/menu-cache/menu-cached /run/user/1000/menu-cached-:0
0 0 1000 1780 at-spi-bus-laun 0 M 1 M /usr/libexec/at-spi-bus-launcher
0 0 1000 1786 dbus-daemon 1 M 0 M /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
0 0 1000 1792 at-spi2-registr 1 M 1 M /usr/libexec/at-spi2-registryd --use-gnome-session
0 0 1000 1840 gvfs-udisks2-vo 0 M 2 M /usr/libexec/gvfs-udisks2-volume-monitor
0 0 1000 1863 gnome-pty-helpe 1 M 0 M gnome-pty-helper
0 0 1000 1864 bash 0 M 1 M bash
0 0 0 1899 sudo 0 M 1 M sudo -i
0 0 0 1901 bash 0 M 1 M -bash
0 0 0 1953 oomd_bin 0 M 0 M oomd_bin -f /sys/fs/cgroup/unified
0 -600 0 2562 python3 10 M 0 M python3 /usr/sbin/nohang --config /etc/nohang/nohang.conf
Kthreads, zombies and Pid 1 will not be displayed.
psi-top
psi-top is script that prints the PSI metrics values for every cgroup. It requires Linux >= 4.20 with CONFIG_PSI=y. Man page: psi-top.manpage.md.
$ psi-top
cgroup2 mountpoint: /sys/fs/cgroup
avg10 avg60 avg300 avg10 avg60 avg300 cgroup2
----- ----- ------ ----- ----- ------ ---------
some 0.00 0.21 1.56 | full 0.00 0.16 1.14 [SYSTEM_WIDE]
some 0.00 0.21 1.56 | full 0.00 0.16 1.14
some 0.00 0.15 1.11 | full 0.00 0.12 0.89 /user.slice
some 45.92 28.77 20.19 | full 45.05 28.17 19.56 /user.slice/user-1000.slice
some 1.44 4.67 9.24 | full 1.44 4.65 9.20 /user.slice/user-1000.slice/user@1000.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/pulseaudio.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-daemon.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/dbus.socket
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-udisks2-volume-monitor.service
some 0.25 1.97 4.05 | full 0.25 1.96 4.03 /user.slice/user-1000.slice/user@1000.service/xfce4-notifyd.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/init.scope
some 0.00 0.66 1.99 | full 0.00 0.66 1.97 /user.slice/user-1000.slice/user@1000.service/gpg-agent.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-gphoto2-volume-monitor.service
some 0.93 0.75 0.20 | full 0.93 0.75 0.20 /user.slice/user-1000.slice/user@1000.service/at-spi-dbus-bus.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-metadata.service
some 0.00 2.44 6.78 | full 0.00 2.43 6.74 /user.slice/user-1000.slice/user@1000.service/dbus.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-mtp-volume-monitor.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /user.slice/user-1000.slice/user@1000.service/gvfs-afc-volume-monitor.service
some 44.99 28.30 19.41 | full 44.10 27.70 18.79 /user.slice/user-1000.slice/session-2.scope
some 0.00 0.31 0.53 | full 0.00 0.31 0.53 /init.scope
some 7.25 11.40 13.34 | full 7.23 11.32 13.24 /system.slice
some 0.00 0.01 0.02 | full 0.00 0.01 0.02 /system.slice/systemd-udevd.service
some 0.00 0.58 1.55 | full 0.00 0.58 1.55 /system.slice/cronie.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/sys-kernel-config.mount
some 0.00 0.22 0.35 | full 0.00 0.22 0.35 /system.slice/polkit.service
some 0.00 0.06 0.20 | full 0.00 0.06 0.20 /system.slice/rtkit-daemon.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/sys-kernel-debug.mount
some 0.00 0.14 0.62 | full 0.00 0.14 0.62 /system.slice/accounts-daemon.service
some 7.86 11.48 12.56 | full 7.84 11.42 12.51 /system.slice/lightdm.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/ModemManager.service
some 0.00 1.82 5.47 | full 0.00 1.81 5.43 /system.slice/systemd-journald.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/dev-mqueue.mount
some 0.00 1.64 4.07 | full 0.00 1.64 4.07 /system.slice/NetworkManager.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/tmp.mount
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/lvm2-lvmetad.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/dev-disk-by\x2duuid-5d7355c0\x2dc131\x2d40c5\x2d8541\x2d1e04ad7c8b8d.swap
some 0.00 0.09 0.11 | full 0.00 0.09 0.11 /system.slice/upower.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/udisks2.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/dev-hugepages.mount
some 0.00 0.27 0.49 | full 0.00 0.27 0.48 /system.slice/dbus.service
some 0.00 0.00 0.00 | full 0.00 0.00 0.00 /system.slice/system-getty.slice
some 0.00 0.12 0.20 | full 0.00 0.12 0.20 /system.slice/avahi-daemon.service
some 0.00 0.18 0.30 | full 0.00 0.18 0.30 /system.slice/systemd-logind.service
psi2log
psi2log is a CLI tool that can check and log PSI metrics from specified target. It requires Linux >= 4.20 with CONFIG_PSI=y. Man page: psi2log.manpage.md.
