How to Use Logrotate to Manage Log Files in Linux

It is important to control the sizes of log files on a Linux server because their size always grows over time. Every server has limited resources and too large logs can lead to performance and memory problems, not to mention the loss of precious storage space. This problem is typically solved through log rotation, a process that involves renaming or compressing a log file before it gets too large, and cleaning up old logs to reclaim storage. The logrotate program is the weapon of choice for log rotation in most Linux distributions, and it’s what we’ll be working with through out this tutorial.

By reading through this article, you will learn how to do the following:

• View and edit the configuration for the `logrotate` utility.
• Learn the differences between the general and application-specific configuration.
• Create a standard `logrotate` configuration for your application.
• Create a system-independent log rotation schedule for your application.

Prerequisites

Ensure that you’ve met the following prerequisites before proceeding with the rest of this tutorial:

• Basic knowledge of working with the Linux command line.
• A Linux server that includes the non-root user with sudo access. We’ll be using Ubuntu 20.04 in this guide.
• Basic of knowledge of Linux system log files.

🔭 Want to centralize and monitor your Linux logs?

Head over to Logtail and start ingesting your logs in 5 minutes.

Step 1 — Viewing Logrotate configuration

The logrotate daemon is pre-installed and active by default in Ubuntu. This daemon uses configuration files that specify all the log rotation details for an application. The default setup consists of the following aspects:

• /etc/logrotate.conf: this is the general configuration file for logrotate.
• /etc/logrotate.d: this directory includes files that configure a specific application.

We will examine both configuration possibilities below.

The general configuration

First off, let’s view the general configuration file at /etc/logrotate.conf. Go ahead and print its contents with the cat utility:

cat /etc/logrotate.conf

The command above prints the entire contents of this file:

 Output

# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

Here’s a description of what each configuration directive means:

• weekly: logs are rotated every week. Alternatively, you can specify another time interval (daily, weekly, monthly, or yearly). You can also rotate the logs every hour (hourly), but note that you’ll need to set up a cron job if a shorter rotation frequency than daily is desired.
• su root adm: log rotation is performed with the root user and admin group.
• rotate 4: log files are rotated four times before old files are removed. If rotate is set to zero, then the old versions are removed immediately.
• create: immediately after rotation, create a new log file with the same name as the one just rotated.
• compress: this rule determines whether old log files should be compressed or not. Log compression is turned off by default.
• include: this specifies the directory for application-specific configuration.

Application-specific configuration

Let’s view the contents of the /etc/logrotate.d directory that contains application-specific configuration for logrotate.

ls /etc/logrotate.d/

The command above lists all files in this directory:

 Output

alternatives  bootlog      dpkg               ubuntu-advantage-tools
apache2       btmp         ppp                ufw
apport        certbot      rsyslog            unattended-upgrades
apt           cups-daemon  speech-dispatcher  wtmp

You can observe that quite a few programs have their configuration files in this directory. For example, let’s take a look at the first 15 files in the config file for rsyslog through the head command.

head -n 15 /etc/logrotate.d/rsyslog

You’ll see the program’s output appear on the screen:

 Output

Output
/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

/var/log/mail.info
/var/log/mail.warn

The output shows that the /var/log/syslog file is rotated daily and keeps seven compressed backups. It also includes the following directives:

• missingok: do not report any error if the log file is missing.
• notifempty: do not rotate the log if it is empty.
• delaycompress: postpone compression of the previous log file to the next rotation cycle when a program is using it.
• postrotate/endscript: the lines between postrotate and endscript are executed after the log file is rotated.

A Logrotate configuration may include various other directives. For example, you can specify that older log files are uploaded to another server or archived, or you can rotate a log file when it exceeds a predefined size. All possible directives are described in logrotate manual pages which can be examined by executing man logrotate.

Step 2 — Creating a log file for a custom application

In this step, we will create a log file for a fictional custom application called my-custom-app. In subsequent sections, we will set up a log rotation policy for the logs produced by this application.

Create a new subdirectory in /var/log with mkdir:

sudo mkdir /var/log/my-custom-app

The mkdir command creates a new directory where we will place our custom log file.

Go ahead and create a log file called backup.log in the new my-custom-app directory, and open it in your text editor.

sudo nano /var/log/my-custom-app/backup.log

Fill this file with some fictional logging data:

 Output

Apr  7 10:00:00 Fictional application start.
Apr  7 11:00:00 Fictional application error.
Apr  7 12:00:00 Fictional application exit.

Note that the exact contents of this file is not important for the purpose of this tutorial. In a real-world scenario, the log entries in the file will be produced by the application itself.

Save and close the file, then head to the next section to learn how to configure a log rotation policy for this application’s logs.

Step 3 — Creating a standard Logrotate configuration

In this step, we will create a standard logrotate job for our application’s logs. After we set up a new logrotate configuration file, it will be executed with all the other system log rotation jobs once a day. We will use the my-custom-app example from the previous step.

Create a new my-custom-app file in the /etc/logrotate.d directory with your text editor:

sudo nano /etc/logrotate.d/my-custom-app

Add the following code to the file:

 Output

/var/log/my-custom-app/*.log {
    daily
    missingok
    rotate 7
    compress
    notifempty
}

The configuration above applies to all the files ending with .log in the /var/log/my-custom-app directory. We’ve already discussed what each directive means in step 1, so we won’t go over that again.

Let’s go ahead and test the new configuration by executing the logrotate command. The --debug option ensures that logrotate operates in test mode where only debug messages are printed.

sudo logrotate /etc/logrotate.conf --debug

The logrotate program reads the configuration file at /etc/logrotate.conf and prints some messages. You should spot an entry for the my-custom-app configuration that looks similar to what is displayed below:

 Output

. . .

rotating pattern: /var/log/my-custom-app/*.log  after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 4
considering log /var/log/my-custom-app/backup.log
Creating new state
  Now: 2021-04-07 12:27
  Last rotated at 2021-04-07 12:00
  log does not need rotating (log has been already rotated)
switching euid to 0 and egid to 0

. . .

The above output indicates that the new configuration file at /etc/logrotate.d/my-custom-app has been found by the logrotate program. Therefore, the log files in the my-custom-app directory will now be rotated according to the policy defined within it along with the other system and application logs.

Step 4 — Change the ownership of the log directory

An alternative to the approach taken in the previous step involves creating a configuration file that is system-independent. Such a configuration will not be included in the /etc/logrotate.d/ directory. Instead, we will create a cron job that will execute the configuration file at custom time interval.

There are two main reasons for using a system-independent configuration:

• The logrotate program cannot rotate logs at an interval of less than once per day with the standard configuration.
• The standard configuration cannot rotate logs as a non-root user due to directory permissions.

Similar to the previous step, we will use the my-custom-app example to demonstrate a system-independent configuration.

First off, let’s confirm the owner of its log files with the ls command (the -l option shows access rights to the file):

ls -l /var/log/my-custom-app/backup.log

The program’s output should appear on the screen:

 Output

-rw-r--r-- 1 root root 134 Feb 10 09:16 /var/log/my-custom-app/backup.log

The output shows that the owner of this log file is root. Let’s go ahead and change the ownership of the my-custom-app directory and all its contents to the current user instead of the root user so that we’ll be able to modify them without root privileges. Replace the <username> placeholder below with the current logged in user:

sudo chown -R <username>: /var/log/my-custom-app

Afterward, execute the previous ls command again.

ls -l /var/log/my-custom-app/backup.log

You’ll notice that the ownership has changed (ayo is the username on my server):

 Output

-rw-r--r-- 1 ayo ayo 134 Feb 10 09:16 /var/log/my-custom-app/backup.log

Step 5 — Creating a custom Logrotate configuration

In this step, we’ll create a custom Logrotate configuration in the user’s home directory. Go ahead and create it with your text editor:

nano /home/<user>/logrotate.conf

Set up a logrotate configuration for the my-custom-app application by adding the following lines to the file:

 Output

/var/log/my-custom-app/*.log {
    hourly
    missingok
    rotate 7
    compress
    notifempty
}

We’ve defined that the above configuration should apply to all files in the /var/log/my-custom-app directory that end with the .log extension. Note that the time interval is specified as hourly which is not supported in the standard configuration.

Logrotate does not know about this new configuration because it is not included in the /etc/logrotate.conf file. So instead of adding the new configuration there, we will use a logrotate state file. This file records what the logrotate did last time it ran, and it informs the program of what to do the next time it runs.

The default status file for Logrotate is located in /var/lib/logrotate/status. However, we will create a custom one through the command below:

logrotate /home/<username>/logrotate.conf --state /home/<username>/custom-state

The --state option tells logrotate to use an alternative state file located at /home/<username>/custom-state. The logrotate CLI will create this file if it doesn’t exist. You can view its content with cat:

cat /home/<username>/custom-state

You’ll see the program’s output appear on the screen:

 Output

logrotate state -- version 2
"/var/log/my-custom-app/backup.log" 2022-2-10-10:0:0

The output indicates that logrotate identified the backup.log file in the /var/log/my-custom-app directory. Notice that we no longer need root access to execute the logrotate command. That being said, it won’t run again unless we intervene manually. Therefore, in the last step for this tutorial, we will create a cron job that executes the command periodically after some time has passed (every hour to be specific).

Step 6 — Running Logrotate periodically with Cron

Go ahead and open the cron jobs configuration file by executing crontab -e in your terminal:

crontab -e

The -e option is used to edit the current user’s cron jobs using the editor specified by the $VISUAL or $EDITOR environmental variables. The above command should open a configuration file in your preferred text editor specified by these variables.

Go to the bottom of the file and add the following line:

 Output

0 * * * * /usr/sbin/logrotate /home/john/logrotate.conf --state /home/john/custom-state

This new line specifies that the cron job will be executed every hour (at minute 0) and the logrotate command will run with our custom configuration and state file.

Save and close the modified file. You will observe the following output:

 Output

crontab: installing new crontab

Now that your system-independent log rotation policy is all set up, you can view the /var/log/my-custom-app directory after an hour to confirm that the log files therein are rotated according to the defined policy. For more details about cron jobs see the following documentation or type man cron in your terminal.

Conclusion and next steps

In this tutorial, you learned about log rotation in Linux and how it is achieved through the logrotate program. We started by examining its configuration files, and then we discussed several of the common directives you are likely to encounter. We also created a standard logrotate configuration for our custom application, before pivoting to one that is system-independent.

To learn more about logrotate and everything that you can do with it, do check out its manual page by running man logrotate in your terminal