Monitoring Splunk

Find changed files and diff the first and last versions

skooby
Explorer

I am writing app and searches that will find changed files and diff the first and last version of the file:

(This is in place of fschange which I understand is now deprecated).


inputs.conf:

#############################################################

# xml

#############################################################

[monitor:///.../*.xml]

sourcetype=my_xml

index = Main

crcSalt=< SOURCE >TEST1

props.conf:

[my_xml]

CHECK_METHOD = entire_md5

TRUNCATE = 0

LINE_BREAKER = (?!)

I have composed some searches that get the oldest and newest file....


index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1

and

index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| tail 1

These pull back the desired info BUT but display the desired file details in the "Statistics" tab (in the Splunk UI).

If I try and feed this into a subsearch, e.g.


[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1]



then Splunk disregards the "head -1" part and spits out all 8 file versions to the "Events" tab.

Perhaps because of this, my diff also fails (having said that, I'm not sure if diff is meant to be used like this, feel free to put me straight here!)


diff pos1=[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1] pos2=[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| tail 1]

gives the error:


Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the right hand side.

The search job has failed due to an error. You may be able view the job in the Job Inspector.

Can anyone give me any pointers, please?

Tags (4)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

In this case, fschange is safe to continue to use for some time. It may have been deprecated as of Splunk 5.0, but as of Splunk 6.1 it is still supported. I would think it continues to be "safe to use as-is" even though it is definitely not being improved.

Looking for things that do fschange-like-stuff, an obvious contender is OSSEC (http://ossec-docs.readthedocs.org/en/latest/manual/non-technical-overview.html) which fits nicely into splunk.

But, more specific to your question -- an approach like below might be more reliable for getting your first and last events for diffing.

index="main" source="/*xml" 
| streamstats count 
| eventstats max(count) as high, min(count) as low
| where ( (count=high) OR (count=low) )

With this, streamstats attaches a count to each event seen. First event is '1', second '2', etc. Then eventstats tacks on the highest and lowest values of count seen to the original events, giving you the ability to filter.

The diff command expects TWO events (and only two events).

View solution in original post

0 Karma

dwaddle
SplunkTrust
SplunkTrust

In this case, fschange is safe to continue to use for some time. It may have been deprecated as of Splunk 5.0, but as of Splunk 6.1 it is still supported. I would think it continues to be "safe to use as-is" even though it is definitely not being improved.

Looking for things that do fschange-like-stuff, an obvious contender is OSSEC (http://ossec-docs.readthedocs.org/en/latest/manual/non-technical-overview.html) which fits nicely into splunk.

But, more specific to your question -- an approach like below might be more reliable for getting your first and last events for diffing.

index="main" source="/*xml" 
| streamstats count 
| eventstats max(count) as high, min(count) as low
| where ( (count=high) OR (count=low) )

With this, streamstats attaches a count to each event seen. First event is '1', second '2', etc. Then eventstats tacks on the highest and lowest values of count seen to the original events, giving you the ability to filter.

The diff command expects TWO events (and only two events).

0 Karma

skooby
Explorer

Yay! that works!

My final search was:

[search index="m*" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc ] | streamstats count
| eventstats max(count) as high, min(count) as low
| where ( (count=high) OR (count=low) ) | diff pos1=1 pos2=2

Thank you dwaddle!

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...