Skip to content

Imposing temporary CPU and memory resource limits on a user on Ubuntu 16.04

Suppose, not entirely hypothetically, that you sometimes have users
on your primary login server who accidentally run big CPU-consuming
and memory-eating compute jobs that will adversely impact the
machine. You could kill their process or their entire login session,
but that's both a drastic impact and potentially not a sure cure,
and life gets complicated if they're running something involving
multiple processes. In an ideal world you would probably want to
configure this shared login server so that all users are confined
with reasonable per-user resource limits. Unfortunately systemd
cannot do that today; you need to put limits on the user-${UID}.slice
unit that systemd creates for each user, but you can't template
this unit
to add
your own settings.

Without always-on per-user resource limits, what you'd like to do
is impose per-user resource limits on your runaway user on the fly,
so that they can't use more than, say, half the CPUs or three
quarters of the memory or the like (pick your own reasonable limits).
Systemd can do this, in a way similar to using systemd-run to
limit something's RAM consumption
, but
on Ubuntu 16.04 this requires a little bit more work than you would
expect.

The basic approach is to set limits on the user's user-${UID}.slice
slice unit:

systemctl --runtime set-property user-915.slice CPUQuota=200% MemoryLimit=8G

With --runtime, these limits will not persist over the next reboot
(although that may be quite some time in the future, depending on
how you manage your machines; ours tend to be up for quite a while).

In theory this should be all that you need to do. In practice, on
Ubuntu 16.04 the first problem is that this will limit new login
sessions for the user but not existing ones. Of course existing
ones are the ones that you care about right now, because the user
is already logged on and already running those CPU-eaters. The
problem appears to be that just setting these properties does not
turn on CPUAccounting and MemoryAccounting for existing sessions,
so nothing is actually enforcing those limits.

The obvious fix here is to explicitly turn on these for the
user-${UID}.slice unit we already manipulated. Sadly this
has no effect. Instead the magic fix appears to be to find one
of the user's scopes (use 'systemctl status ' for one
of the CPU-eating processes) and then set the limit on that
scope:

systemctl --runtime set-property scope-c178012.scope CPUAccounting=true MemoryAccounting=true

In my testing, the moment that I set these on for any current scope,
all of the user's current login sessions were affected. If I sort
of understand what systemd is doing with cgroups, this is probably
because setting these on a single scope causes (or forces) systemd
to ripple this up to parent units. Taken from the
systemd.resource-control manpage
:

Note that turning on CPU accounting for one unit will also implicitly
turn it on for all units contained in the same slice and for all its
parent slices and the units contained therein.

It's possible that this will turn on global per-user fair share
scheduling
all by itself. This is
probably not such a bad thing on the kind of shared login server
where we'd want to do this.

If you think you're going to need to add these on the fly limits,
an obvious thing to do is to pre-enable CPU and memory accounting,
so that all user slices and login scopes will be created ready for
you to add limits. The basic idea works, but several ways to achieve
it do not, despite looking like they should. What appears to be the
requirement in Ubuntu 16.04 is that you force systemd to adjust its
current in-memory configuration. The most straightforward way is this:

systemctl --runtime set-property user.slice CPUAccounting=true MemoryAccounting=true

Doing this works, but it definitely has the side effect that it
turns on per-user fair share CPU scheduling. Hopefully this is a
feature for you (it probably is for us).

The following two methods don't work, or at least they don't persist
over reboots (they may initially appear to work because they're
also causing systemd to adjust its current in-memory configuration):

  • Enabling DefaultCPUAccounting and DefaultMemoryAccounting in
    user.conf via a file in /etc/systemd/user.conf.d, contrary
    to how I thought you'd set up per-user fair share scheduling
    . There is no obvious reason why
    this shouldn't work and it's even documented as working, it just
    doesn't in the Ubuntu 16.04 version of systemd (nominally version
    229). If you do 'systemctl daemon-reload' they may initially
    appear to work, but if you reboot they will quietly do nothing.
  • Permanently enabling CPUAccounting and MemoryAccounting on
    user.slice with, for example, 'systemctl set-property user.slice
    CPUAccounting=true MemoryAccounting=true'. This will create some
    files in /etc/systemd/system/user.slice.d, but much like the
    user.conf change, they will do nothing after a reboot.

I can only assume that this is a systemd bug, but I don't expect
it to ever be fixed in Ubuntu 16.04's version and I have no idea
if it's fixed in upstream systemd (and I have little capability to
report a bug, given the version number issue covered here).

(There is presumably some sign in /sys/fs/cgroup/* to show whether
per-user fair share CPU scheduling is on or off, but I have no idea
what it might be. Alternately, if the presence of user-${UID}.slice
directories in /sys/fs/cgroup/cpu,cpuacct/user.slice means that
per-user fair share scheduling is on, it's somehow wound up being
turned on on quite a few of our machines.)

In general I've wound up feeling that this area of systemd is
badly underdocumented
. All of
the manpage stuff appears to be written for people who already
understand everything that systemd is doing internally to manage
resource limits, or at least for people who understand much more
of how systemd operates here than I do.

0
sfy39587p00