Setting up Splunk 2.0 with Syslog-ng and a FIFO

Joe Reeves

Last Modified: 07.27.2006

Contents:

• Assumptions

• What is a FIFO?

• Why Syslog-ng?

• Syslog-ng install

• Creating the FIFO

• Configuring syslog-ng

• Configuring Splunk

• Debugging a FIFO

Assumptions

• You have a supported Splunk platform and root access to the server.
• You are installing Splunk 2.0 or higher.

What is a FIFO?

A FIFO is an old programming term that stands for First In First Out. A more accurate name for what you're setting up is a Named Pipe. Basically, what it is, is a special file on your UNIX system that receives input from some program and sends that information to another program. You can think of it as a stack. One program puts things on the stack, and another program takes stuff off the stack, but it starts with the first item that went on.

Why Syslog-ng?

Well basically because the standard syslog daemon sucks. Seriously, the standard syslog daemon that comes with your typical UNIX is very limited. Syslog-ng is very flexible and includes support for many things that the standard syslog daemon does not, including, but not limited to: multiple sources and destinations, extensive filtering including regex support, the ability to read and write to pipes, and much more. It's also extremely efficient on system resources.

Syslog-ng install

Depending on your platform, the procedure for installing syslog-ng may vary widely. On Linux platforms, you will most likely be using RPM or Debs. On Solaris, it's quite likely that Sun Freeware will have a package that will work for you. Mac OS X users: As far as I can tell, Mac OS X is not a supported platform. Sorry about that!

Examples for common platforms:

Creating the FIFO

Select a place on your filesystem for the FIFO to live. It doesn't matter much where as the FIFO will only buffer a limited amount of lines that will consume an insignificant amount of resources. I would suggest /var as that is where most sockets and pipes on the system are located by default. I gave mine a subdirectory called syslog-ng for the sake of tidiness.

Example command to create the FIFO:

    myserver# mkfifo /var/syslog-ng/syslog_fifo
    

Remember this location as you will need to put it in your syslog-ng and splunk config files.

Configuring syslog-ng

In most installs of syslog-ng, your main configuration file will be located in /etc/syslog-ng/syslog-ng.conf. At a minimum, you will need to set up a source that accepts remote connections [assuming that you wish to send logs from multiple systems to be included in your Splunk index], and a destination for your FIFO. It would also be wise to log to a location on the filesystem as well. Here's the first bit:

    source remote {
      udp();
    };

    destination splunk {
      pipe("/var/syslog-ng/syslog_fifo");
    };

    log {
      source(remote);
      destination(splunk);
    };
    

That will be sufficient to send the logs only to your named pipe. However, I would recommend that you also set up syslog-ng to log to a location on the local filesystem like so:

 
    destination hosts {
      file("/var/log/hosts/$HOST/messages"
      owner(root) group(logs) perm(0640) dir_perm(0750) create_dirs(yes));
    };
    
    log {
      source(remote);
      destination(hosts);
    };
    

Note that the $HOST moniker is a magic variable in syslog-ng that is replaced by the hostname of the system that is sending the logs. It's quite handy. The create_dirs bit automatically creates the subdirectories if they do not exist.

Configuring Splunk

For this portion, I will assume that Splunk was installed in it's default location of /opt. If you have installed in a different location, adjust the paths accordingly.

Change directory to /opt/splunk/etc/modules/input/FIFO and copy the config.xml.disabled file to config.xml. If config.xml already exists, then your input might already be configured. Check to make sure that you cannot see the input from the main page of the Splunk web interface before proceeding. If your FIFO input isn't working, you'll need to open the config.xml file and perform the following edits.

Alternately, you may download a copy of my config.xml and make the necessary changes to it.

Note that the hostname extraction part that was required in previous releases to grab the hostnames from the data stream is no longer necessary as this is built into the new parsing queue. Hostname extraction should happen automagically.

<!--Copyright (C) 2006 Splunk Inc. All Rights Reserved. Version 1.2 -->

<pipeline name="FIFOinput" type="startup">
	<processor name="fifoReader" plugin="fifoinputprocessor">
        	<config>
			<field>_raw</field> 
			<fifo>/var/syslog-ng/syslog_fifo</fifo>
			<sourceType>syslog</sourceType>
                        <host>master</host>
                        <index>default</index>
                </config>
        </processor>

        <processor name="sendOut" plugin="queueoutputprocessor">
		<config>
			<queueName>parsingQueue</queueName>
		</config>
        </processor>
</pipeline>

Note that master is the name of my syslog-ng server. You will need to replace that part with the name of your server. If the path to your fifo is different, modify the <fifo> line accordingly.

Debugging FIFOs

If you are having problems with your FIFO, the first thing that you should do is to verify that it it getting data put into it by syslog-ng. I have written a small tool called Piper that can help you do this. Download Piper and run the following command:

Then check your debug.log to make sure that it contains data. If it does contain data, then syslog-ng and the FIFO that you created are working properly and you should re-examine your Splunk config. If it looks correct, contact Splunk support for assistance. If your debug.log does not contain any data, then something is most likely wrong with your syslog-ng config. First try restarting syslog-ng to make sure that your changes have applied. If that doesn't do the trick, examine your syslog-ng configuration closely and make sure that you didn't miss any semicolons or other necessary syntax.

Questions? comments? Flames?

Email me