How to manage the size of large /var/log/* Log files in Linux

How to manage the size of large /var/log/* Log files in Linux

If you use Linux, you may have come across problems to do with storage space shortage, especially if you have constrained disk space allocated to your Linux distribution. In some cases, you may know exactly what the storage culprit is, maybe a large file you downloaded to the ~/Downloads/, ~/Documents/, /home/, etc  directory. However, in other cases, you may try to diagnose what is eating up much space in an effort to free up some with no luck. In such a case it is apt to check your Log files in the “/var/log/” directory.

 

[Check the Size of Log files of Interest]

In the /var/log/ directory, not each file is of interest. The Log files that take up much space are majorly only three; “user.log“, “syslog“, and “messages“. Depending on how much time you have actively used your Linux distribution, the file size may vary. You can find any of these files taking up a few hundred MBs to a few GBs. To check for your case, you can run the several commands:

‘df’ utility:

df /var

 

‘wc’ utility:

wc -l /var/log/syslog /var/log/messages /var/log/user.log

 

‘ls’ utility:

ls -s -h -1 /var/log/syslog /var/log/messages /var/log/user.log

ls -lSsh /var/log

I recommend the last utility as it shows the space used in a human-readable format! The first instance of ‘ls‘ shows the size of specific files of interest while the last instance shows the size of files in the /var/log/ directory sorted according to descending order of size.

Depending on how much space you can spare for your logs, a few hundred MBs of logs can be too much. However, logs are an important way of monitoring a system and ground zero in troubleshooting whatever errors in any system, Linux or not. For this reason, you do not want to quit logging but rather manage the size of the log files just in case you need them. There is no standard size of Log files when you want to reduce their size, so it is totally dependent on your decision. When it comes to this disk management, there are several methods sorted according to recommendation by the author, that is, “Method 1” should be the first option while “Method 3” should be the last option;

 

Method 1: Using the “logrotate” utility

The “logrotate” utility is used to manage many Log files easily on a specified routine timeline in conjunction with cron jobs in Linux. To understand its usage, check its manual page;

man logrotate

 

[“logrotate” configuration]

To use ‘logrotate‘ you need to understand its configuration as specified in the “/etc/logrotate.conf” file.

 

[“logrotate” Log files syntax]

The instructions in the above file are applied in files stored in the “/etc/logrotate.d/” in a format like;

/path/to/logfile {
    configuration file Directives
}

Since there are already Log files managed using the ‘logrotate‘ utility, we can check some to get the syntax and apply later. Below are the logrotate configuration files “/etc/logrotate.d/wtmp” and “/etc/logrotate.d/dpkg” used to manage the respective Log files, “/var/log/wtmp” and “/var/log/dpkg.log“;

 

[Custom “logrotate” configuration files]

With the syntax and usage in mind, we can create our own custom logrotate configuration files for our Log files of interest, that is “/var/log/syslog“, “/var/log/messages“, and “/var/log/user.log“. The ‘missingok‘ directive implies that absence of a Log file does not generate an error. The ‘weekly‘ directive implies that the frequency of rotation and then deletion is after 7 days. The ‘create‘ directive implies that a new Log file with a name similar to the one just rotated is created with specified:- mode, owner and group. The ‘minsize‘ directive implies that the Log file is rotated after the specified size is reached but only after the specified interval of rotation. The ‘rotate‘ directive implies that the Log file is rotated (renamed a number of specified times before deletion).

For “/var/log/syslog” the logrotate configuration file can be as shown below. ;

 

For “/var/log/messages” the logrotate configuration file can be as shown below. ;

 

For “/var/log/user.log” the logrotate configuration file can be as shown below. ;

 

[Setting & Testing “logrotate” cron jobs]

The above logrotate configurations will be run as cron jobs at a frequency of one week. To be sure that they will be run, you can test them by using the “logrotate” command with a ‘–force‘ or ‘-f’ parameter and a logrotate configuration file argument. This forces the configuration file to be run as if the set interval has been reached. Note that a the original file ‘syslog‘ file is renamed to “syslog.1” as part of the rotation with a new “syslog” file created with 0 bytes. The command is as shown below;

logrotate --force /etc/logrotate.d/syslog

 

In case you want to view what is actually happening in the background, you can add the ‘–verbose‘ parameter, like;

logrotate --force --verbose /etc/logrotate.d/syslog

 

Method 2: Using the “truncate” utility

The “truncate” utility is inbuilt in Linux distributions and is meant to Extend/Shrink the size of a file. To check its usage, you can check its manual page using the command;

man truncate

Say for instance that you want to shrink the space of the “/var/log/messages” file. You first need to know the space taken by the file;

ls -s -h /var/log/messages

 Using the ‘truncate‘ utility, the command to shrink the file can be like;

truncate -s 800MB /var/log/messages

To confirm that the file size was shrinked, again run the command;

ls -s -h /var/log/messages

It is clear and evident that the file was actually shrinked! You can repeat this process for any Log file you intend to shrink in size. This however means that you have to do this the next time you see the Log files have grown too large, which implies that this is not a permanent solution for managing the problem.

 

Method 3: Delete the Log files

This is not exactly a recommended option, but it works nevertheless. The reason it works is because, by default, new Log files are created automatically by the system if they are not found to exist in the locations/directories they are expected to live in. For instance, to remove the three main Log files of interest;

rm /var/log/syslog /var/log/messages /var/log/user.log

The option is included because it works. It should however be the last option since getting rid of a Log file entirely can lead to problems when troubleshooting a crashed or problematic system.

 

 

How to manage the size of large /var/log/* Log files in Linux
Hacking | thetqweb