Linux system housekeeping 101: Managing file storage

One of your many duties as a system administrator is the often daunting task of keeping your system's filesystems clear of clutter. It's not an easy task, is it? This first article in a short housekeeping series explores some basic system housekeeping concepts that will keep your systems healthy and your users responsible.

System health rules

There's an unwritten rule in system housekeeping that I'm about to write down. System health is your first responsibility. There, I wrote it. Some people have the mistaken opinion that keeping your users happy is your first responsibility. It isn't, and I can prove it with the following scenario.

You manage a Linux system that services 50 users. You have two users who are notorious about filling up shared filesystems with hundreds of gigabytes of personal files, preventing the other 48 system users from storing business files. You hear a lot of complaints from your user base.

You've spoken to the culprits, emailed them, and even threatened to erase their data. They continue to fill the common spaces with personal data. The health of the system takes precedence over the "needs" of the two people who violate rules of common courtesy and professionalism. You have 48 other users who do not abuse the system's available space. The health of the system suffers because of those two rogue users. It's your job to fix it.

These are the droids you're looking for

So, what are these "common" file storage areas that plague system administrators? They are, in order of importance: / (root), /tmp, and /home. These are the default filesystems to which users have or might have write access. I placed these filesystems in order of importance because that's how I view them.

The root (/) filesystem

The root (/) filesystem is the most critical for sysadmin deep cleaning. Through the magic of internet-powered virtual reality, I can hear you saying, "No sysadmin in their right mind would allow users to write to the root filesystem." You're correct. You're also probably not realizing that if your /tmp, /home, /var, /opt, and other filesystems aren't mounted on separate volumes or partitions, then you have allowed write access to the root filesystem.

I suggest that you mount /tmp, /var, /opt, and /home on separate volumes or partitions so that your root filesystem stays clean. Any user-writable directory mounted on the same volume or partition as the root filesystem is at risk of becoming filled up, which can have very negative consequences for your system. For example, logs won't write to the /var/log directory.

The rule: Mount any user-writable directory onto a separate partition.

The /tmp filesystem

One of the first articles I wrote for Enable Sysadmin was: Manage the Linux /tmp directory like a boss. This article gives you a thorough introduction to managing the /tmp directory. There are no hard and fast rules for creating a /tmp directory with a specific size. The size of /tmp is based on how busy your server is. I know that's not much help, but it's a start. If you have a reasonably busy server with only a few responsible users, then 1-2 GB might be fine. In contrast, if you have a busy system with 50 users, you should bump that /tmp directory into the double-digit size range.

Users have write access to the /tmp directory, which can be a bad thing if they figure it out. They might store large files, or many hundreds of files in this system use only space.

The rule: Size the /tmp directory based on how busy the system is.

The /home filesystem

Users feel that not only do they have unlimited space, but that every other user also has unlimited space. These assumptions cause a lot of disk space pain for system administrators. Users copy data to their home directories without regard to how much space they're using and that their use depletes available space for everyone. Users also leave data on a shared space for an undetermined amount of time.

If you automate a search to find files that have over 90 days since the last accessed time and then warn users that non-accessed files need to be archived somewhere other than their /home directories or they'll be removed, you'll have some unhappy users. However, as mentioned earlier, you can't just allow users to upload files to a shared space area and leave them indefinitely. If they're not accessing and using the files, they should be archived or removed.

If you supply each user 1 GB of disk space in /home, which is quite generous, you'll need to also set up quotas for the /home filesystem. Quotas are out of scope for this article, but we will create a quotas post in the near future to augment this article's information.

The rule: Limit /home directory space per user.

Other shared spaces

There are sysadmins who, for one reason or another, won't create any new directories on a filesystem such as /shared, /public, /users, etc. They are purists who make everyone use the /opt directory, which is hopefully mounted on its own volume or partition separate from /. To manage these shared spaces, you'll need a policy and a plan. A policy gives your housekeeping management buy-in, which means that your unpopular enforcement of space limitation will be automatically approved. A plan means that you have a reasonably calculated idea of how much space should be given to users.

Each user believes themselves to be the exception to every rule, so you have to be somewhat heavy-handed in enforcing rules and policies. It's not pleasant to do so, but you can't make exceptions, or your entire system will become an exception. You should have your users explore alternative storage if their needs violate policy.

The rule: Draft a use policy and stick to it.

Wrapping up

Being a system administrator means that not only do you need to be a thought leader in your company, but you also have to be the space police and a housekeeper. Housekeeping often means performing the undesirable task of telling users "No" and enforcing rules such as disk space restrictions. Your responsibility is to all users, not just one. System health is your number one priority, which must be protected at any cost.

The rule: Be respectful but responsible.

[ Want to test your sysadmin skills? Take a skills assessment today. ]