After some funky behaviour on one of my Linux server VMs, I realized that the disk space was critically low, caused by the build up of logs from syslog and the systemd journal. Here’s some steps to resolve the issue, and ensure you keep a safe amount of disk space free.

The system is Ubuntu 22.04, running rsyslog and systemd-journald, which were both growing in size over time, eventually leading to minimal free space remaining. The evidence was in the large file sizes of /var/log/syslog and /var/log/journal. This VM acts as a reverse-proxy running Caddy, so I provisioned a rather lean disk allotment of 16gb many years ago.

Before proceeding, it is worth investigating the logs to see if there are consistent issues which end up filling the logs themselves. Take a look, or at least tail a good chunk of lines to see if deeper issues exist. By fixing those issues first, if any, the chances that the log will grow in size quickly are reduced (in most cases, but not all!). That said, it’s still wise to configure your log retention policies to be optimal for your investigative and disk space management needs.

In my case, this server runs Ubuntu 22.04, so I used the logrotate service to modify the policy for syslog as follows, in /etc/logrotate.d/rsyslog (the file may also be named syslog instead of rsyslog):

/var/log/syslog
/var/log/mail.info
...
/var/log/messages
{
  rotate 7
  daily
  size 100M
  missingok
  notifempty
  compress
  delaycompress
  sharedscripts
  postrotate
          /usr/lib/rsyslog/rsyslog-rotate
  endscript
}

Important values for you to check are rotate, size, and daily. From the logrotate man pages:

  • rotate count

    • Log files are rotated count times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather than rotated. If count is -1, old logs are not removed at all, except they are affected by maxage (use with caution, may waste performance and disk space). Default is 0.

  • daily

    • Log files are rotated every day.

  • size size

    • Log files are rotated only if they grow bigger than size bytes. If size is followed by k, the size is assumed to be in kilobytes. If M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G are all valid.
    • This option is mutually exclusive with the time interval options, and it causes log files to be rotated without regard for the last rotation time, if specified after the time criteria (the last specified option takes the precedence).

Of course you will need to define values that are specific to your own needs, and I suggest you read through man logrotate for more details on how to determine those best values for you.

Next up, systemd-journal. The config file is located at /etc/systemd/journald.conf, and you can adjust these two values under the [Journal] section:

[Journal]
SystemMaxUse=512M
SystemKeepFree=8G

In my case above, I’ve set the logs to consume a maximum of 512mb, while also ensuring that the system itself keeps a minimum of 8gb free.

By tuning these values in both configs above, and (re)starting the services via systemctl start logrotate.service and systemctl restart systemd-journald.service, I immediately saw the logs get pruned and disk space was restored. Hope this helps if you’ve stumbled upon my site while searching for such an answer!