DISCLAIMER: all the information in this article is based on my home lab environment and is not advised to be used in production without thorough impact analysis.
Occasionally, you can get alarms in vCenter Server UI about low disk space on some disks. Normally, you would start with accessing VAMI to check which disk is over 78%. In my case it was /var/log and I found plenty of KB articles about potential cause, but none of them was relevant to my issue. After some digging, I found out that
- Login through SSH to your vCSA using root account and enable shell
- Check the size of /var/log subfolders with this command: du -sh /var/log/* | sort -h
- Check the size of journal log files: journalctl –disk-usage
- If the size of /var/log/journal is significant, you can purge it with command: journalctl –vacuum-time=10d (or alternatively use vacuum-size=500M switch.
This will delete all the logs older than 10 days.
You can also permanently cap the journal log file with this command:
echo SystemMaxUse=500M | sudo tee -a /etc/systemd/journald.conf