Raspberry Pi Centralized Log Server

Setting up a Pi to be a centralized log store is amazingly simple. If you are using Raspbian it comes with rsyslog installed by default, so all that’s left is to setup the config and tailor log rotation.

First you should create a directory under /var/log for all the remote logs.

sudo mkdir /var/log/central

Then edit the configuration for rsyslog, which can be found in /etc/rsyslog.conf
Find and uncomment the following line(s):

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

Then create a template file under /etc/rsyslog.d/central.conf with the following contents:

$template RemoteLogs,"/var/log/central/%HOSTNAME%.log" 
*.*  ?RemoteLogs
& ~

The first line tells rsyslog to place incoming logs into their own file based on hostname. The RemoteLogs line is just a name for the template, it can be anyname, the official documentation uses RemoteHost in the examples. You can use whatever makes sense. However it should match the preceding line. The third line tells rsyslog to stop processing messages and not write anything to the local files.

For any configuration change to become active you will need to restart the daemon

service rsyslog restart

This will open port 514 on your machine. If you have a firewall configured, then adjust it accordingly.

If you do not set up log rotation, you will eventually have very large files and a full disk. This is easily remedied by configuring logrotate.

Create a configuration file so logrotate knows about your logs:

sudo vim /etc/logrotate.d/central

Add the configuration details for your environment. Here is an excerpt of my config file.

/var/log/central/pfsense.fakelabs.org.log
{
rotate 52
weekly
size 25M
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
  invoke-rc.d rsyslog rotate > /dev/null
endscript
}

/var/log/central/switch0.log
/var/log/central/fileserver.log
/var/log/central/logpi.log
{
rotate 12
monthly
size 10M
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
  invoke-rc.d rsyslog rotate > /dev/null
endscript
}

The script /etc/cron.daily/logrotate is run daily by cron, so no need to restart anything for changes to take effect. If you want to validate the changes, you can run the following command:

sudo logrotate --debug

This will not rotate your logs, but it will give you a sanity check of your configuration.

Once you are satisfied with the above, you are all done. Now you need to configure all your devices to send their logs to the Raspberry Pi.

If you are using syslog, edit /etc/syslog.conf and add the following at the end of the file:

*.* @log.fakelabs.local

if you have rsyslog, edit /etc/rsyslog.conf and append the following :

*.* @log.fakelabs.local
*.* @@log.fakelabs.local #TCP

if you have syslog-ng, edit /etc/syslog-ng.conf and insert the following:

# Collect system and internal logs
source src {
  system();
  internal();
};

# Where to send the logs and the transport
destination logbox {
  udp("1og.fakelabs.local" port(514));
};

# What to send and where
log {
  source(src);
  destination(logbox);
};

Change domain to your log server IP or domain name, then restart your log daemon.

After each host you configure, you should see the logs under /var/log/central on the Raspberry Pi.

NOTE:

I’m not encrypting the logs sent to the pi on my LAN. However, the logs are being encrypted from my network out to my remote splunk server in the cloud. Maybe in another blog post I’ll cover encrypted remote logging.

That’s all there is to it! Thanks for reading.

 

 

6 thoughts on “Raspberry Pi Centralized Log Server”

    • Jon, I have a Splunk Universal Forwarder on the Raspberry Pi that sends the logs to a Splunk Indexer. Splunk has a web UI so I use that to view the logs in aggregate. You can do the same thing using the Elastic Stack (Kibana, Logstash, Elasticsearch). If you just want a quick and dirty way to see the logs from windows you can ssh into the raspberry pi using putty or from the command prompt and just do a ‘tail -qf /var/log/central/*.log’

      Reply
  1. Can you specify where you’re actually changing the config to redirect the log stream to the directory you created?

    Reply
    • Josh, good catch. I created a template file under /etc/rsyslog.d/central.conf with the following contents:
      $template RemoteLogs,”/var/log/central/%HOSTNAME%.log”
      *.* ?RemoteLogs
      & ~

      I will update the blog to reflect that missing content. Merry Christmas!

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: