How to Set up Splunk with Syslog-ng and a FIFO
Joe Reeves
03.30.2006
Contents:
Assumptions
You have a supported Splunk platform and root access to the server.
What is a FIFO?
A FIFO is an old programming term that stands for First In First Out. A more accurate name for what you're setting up is a Named Pipe. Basically, what it is, is a special file on your UNIX system that receives input from some program and sends that information to another program. You can think of it as a stack. One program puts things on the stack, and another program takes stuff off the stack, but it starts with the first item that went on.
Why Syslog-ng?
Well basically because the standard syslog daemon sucks. Seriously, the standard syslog daemon that comes with your typical UNIX is very limited. Syslog-ng is very flexible and includes support for many things that the standard syslog daemon does not, including, but not limited to: multiple sources and destinations, extensive filtering including regex support, the ability to read and write to pipes, and much more. It's also extremely efficient on system resources.
Syslog-ng install
Depending on your platform, the procedure for installing syslog-ng may vary widely. On Linux platforms, you will most likely be using RPM or Debs. On Solaris, it's quite likely that Sun Freeware will have a package that will work for you. Mac OS X users: As far as I can tell, Mac OS X is not a supported platform. Sorry about that!
Examples for common platforms:
redhat variant# yum install syslog-ng
debian variant# apt-get install syslog-ng
gentoo# emerge syslog-ng
Creating the FIFO
Select a place on your filesystem for the FIFO to live. It doesn't matter much where as the FIFO will only buffer a limited amount of lines that will consume an insignificant amount of resources. I would suggest /var as that is where most sockets and pipes on the system are located by default. I gave mine a subdirectory called syslog-ng for the sake of tidiness.
Example command to create the FIFO:
myserver# mkfifo /var/syslog-ng/syslog_fifo
Remember this location as you will need to put it in your syslog-ng and splunk config files.
Configuring syslog-ng
In most installs of syslog-ng, your main configuration file will be located in /etc/syslog-ng/syslog-ng.conf. At a minimum, you will need to set up a source that accepts remote connections [assuming that you wish to send logs from multiple systems to be included in your Splunk index], and a destination for your FIFO. It would also be wise to log to a location on the filesystem as well. Here's the first bit:
source remote {
udp();
};
destination splunk {
pipe("/var/syslog-ng/syslog_fifo");
};
log {
source(remote);
destination(splunk);
};
That will be sufficient to send the logs only to your named
pipe. However, I would recommend that you also set up syslog-ng
to log to a location on the local filesystem like so:
destination hosts {
file("/var/log/hosts/$HOST/messages"
owner(root) group(logs) perm(0640) dir_perm(0750) create_dirs(yes));
};
log {
source(remote);
destination(hosts);
};
Note that the $HOST moniker is a magic variable in syslog-ng that is replaced by the hostname of the system that is sending the logs. It's quite handy. The create_dirs bit automatically creates the subdirectories if they do not exist.
Configuring Splunk
For this portion, I will assume that Splunk was installed in it's default location of /opt. If you have installed in a different location, adjust the paths accordingly. You will need to edit the XML config file for fifoInput that is located at /opt/splunk/etc/modules/fifoInput/config.xml. NOTE: In newer versions of Splunk [1.2+] you will first need to copy the config.xml.disabled to config.xml.
Alternately, you may download a copy of my config.xml and make the necessary changes to it.
IMPORTANT: It is only necessary to edit the following portion of the config file. Do not remove anything else from the config, as it may cause the fifo plugin to break!
Note that the hostname extraction portion will allow Splunk to
extract hostnames from the data stream:
<!--Copyright (C) 2006 Splunk Inc. All Rights Reserved. Version 1.2 -->
<pipeline name="fifoInput" type="startup">
<processor name="fifoReader" plugin="fifoinputprocessor">
<config>
<field>_raw</field>
<fifo>/var/syslog-ng/syslog_fifo</fifo>
<sourceType>syslog</sourceType>
<host>master</host>
<index>default</index>
</config>
</processor>
<processor name="regexreplacement" plugin="regexextractionprocessor"
configReference="$$SPLUNK_HOME]]/etc/myinstall/pluginConfs/syslogRegexHostExtraction.xml">
</processor>
Note that master is the name of my syslog-ng server. You will need to replace that part with the name of your server. If the path to your fifo is different, modify the <fifo> line accordingly.
