Monitoring hidden files on Linux

[Peter_B]

I'm having a problem trying to monitor the .bash_history file. I've set up a monitor for /home with a whitelist of ".bash_hsitory". When I run "splunk list monitor" it shows no files. If I take out the whitelist it shows all the files under /home except for those with a dot prefix.

Does anyone know how to make Splunk find these hidden files?

Thanks.

[jrodman]

The 4.0.x monitor input, at least (currently unsure about 4.1.x), doesn't handle dotfiles. A customer reported this around the turn of the year, and it was logged as an enhancement request. I have a vague memory of special features surrounding dotfiles dating back to very early versions of Splunk. There certainly are cases where people might want to monitor a directory of files and NOT monitor dotfiles in said directory, although I would prefer our blacklist feature be used to get this behavior.

(Splunk 1.x was very focused on "handy" features for the solo admin to help her or him get a lot done with little effort. By Splunk 4 the focused has changed drastically towards straightforward behavior and reliable, investigatable configuration. This is a scale of users thing.)

You may be able, on 4.1.x, to defeat this rule by symlinking to it. On 4.0.x the symlink would likely not be successful.

[Peter_B]

Thanks for that. I can monitor a particular file, e.g. root's .bash_history, explicitly; but as Lowell pointed out above it is of limited use. It does provide some information extra to what we get from ps output and from the syscall logging by auditd, so we'll probably use it that way - monitor some explicitly named bash_history files rather than look for all under /home.

[Lowell]

It seems like there are a number of items to address here. As far as I know splunk does not do any special processing or blocking of hidden (or ".*" files).

The first think you may want to check is simply file permissions. From within splunk's web interface, you should be able to see evidence of this kind of problem with the following search:

index=_internal sourcetype="splunkd" FileClassifierManager

Also, a more generally useful approach is to login as the 'splunk' user, and run the the command to see how splunk will process an individual input file.

splunk test sourcetype /home/someuser/.bash_history

When I try this on my system, for example, and got the message, which indicates a permissions issue.

WARN  FileClassifierManager - Unable to open /home/someuser/.bash_history.
WARN  FileClassifierManager - Invalid file: /home/someuser/.bash_history, reason: cannot_read.

When I tried this with a file that could be read by the splunk user, then it created a new source type called "bash_history", but looking over the automatically assigned settings, I can tell you that it will not look quite right out of the box due to line breaking. The other issue you will have is with timestamping. My local copy of .bash_history has no timestamps in it, and if I understand how bash works properly, this file is only ever updated when you logout, therefore you will see a number of new entires showing up all at once.... also I don't think this file is "rotated" in standard log fashion, which could cause duplicate entries (I'd have to test it to see). The bottom line is that this probably isn't going to work right out of the box, and I'm not sure it's going to give you want you are looking for. Which begs the question, what exactly are you actually trying to get out of indexing this file?

If you want to track user activity, there are other options. Also if this is your only means of tracking user activity, then your users can easily circumvent your means of tracking fairly easily by either messing with .bash_history directly (since they own the file), or my simply running a different shell (zsh, ksh, csh, and so on...).

You may wan to check out the ps scripted input in the unix application. That's already setup to capture commands running on the system, and you may find it to be a better option for you. If you really want to get nit picky about whats running, you probably will want to check out the linux audit system. Splunk also has a scripted input for reading files written by auditd.

[Lowell]

Have you tried explicitly monitoring for a single file for a single user? Something like [monitor:///home/someuser/.bash_history], while this isn't a long-term solution you can at least prove or disprove the usability of this source. Good luck. (Oh, and please post an update letting us know if indexing this info ends up being helpful or not. I'm quite curious.)

[Peter_B]

Thanks for your response. I tried splunk test sourcetype as you suggest - it doesn't indicate any permissions problems. I agree with you about the problems with bash_history (or *history to cover other shells), but I think when it's used in conjunction with auditd and the ps input it may provide additional useful information. Unfortunately I can't try it out because I can't get a monitor on /home to find any files with a dot prefix.
Thanks again for the splunk test sourceype tip, I hadn't heard of that before.