Getting Data In

props.conf stanza and zip files containing logs

jlvix1
Communicator

Hi everyone, yesterday I spent most of the day battling through transforms.conf and props.conf - with lucrative results.

Today however, a slight anomaly occurred, please see below ... This works well for logs

**<transforms.conf>**
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \d+\/\d+\/\d\d\d\d\s\d+:\d\d:\d\d\s(AM|PM)\s(Error)\s
DEST_KEY = queue
FORMAT = indexQueue


**<props.conf>**
[source::source-to-break]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = \d+\/\d+\/\d\d\d\d\s\d+:\d\d:\d\d\s(AM|PM)\s

[source::C:\\SplunkFwdTest\\*.log]
TRANSFORMS-set = setnull,setparsing

It worked brilliantly for just .log files, however when I placed in .zip files with .log files within, it skipped both the [setnull] and [setparsing] and just imported all the lines anyway - the BREAK_ONLY_BEFORE regex was respected to parse the events, I just found myself with thousands of unwanted events that were extracted (non-errors), it's almost as if there is a glitch with .zip files where the contents are treated differently, do I need to put another [source::whatever] section in the file to account for the zip files and their contents or is this a bug?

I was expecting it to work, I'm now running a test where the line looks as follows:

[source::C:\\SplunkFwdTest\\*.(log|zip)]

This should respect both .zip and .log files and discard anything that isn't an error - still waiting for results, but I may be doing this incorrectly even if it works?

Cheers
J.

0 Karma

jlvix1
Communicator

Hi, just did another test - it will match the zip file itself by name - it then disregards how to treat the contents.

So, I have a zip file: salestransaction.zip

The contents do not match the whitelist, however what has happened is that these files have been imported anyway.

0 Karma

jlvix1
Communicator

Hi, thanks for the quick response.

Not really but it would be nice to be able to dump zip files in the path from our archives, I just discovered this when dumping all sorts of files in to the forwarders path I mapped to make sure it worked ok (it skipped certain files correctly), also with the following white list on the data input:

\.*(salestransaction)\.*|\.*(fulfillment)\.*

An example zip file name is cbe-salesinformation.2016-08-03.2.zip and may contain one or more files whose names conform to the whitelist as well.

What I would expect is that the files are treated like the other non-zipped files and correctly matched to the queues specified in my transforms.conf, e.g. the source reads as follows when looking at the Data Summary: and it is these files that get imported fully.

C:\SplunkFwdTest\APP005V\cbe-fulfillment.zip:.\cbe-fulfillment.log

What I am pointing out is that perhaps zip files are treated differently? In theory, I think that the zip and its contents conform to the rules I have put in, and that anything but errors should route to the nullqueue.

0 Karma

lguinn2
Legend

You said "What I would expect is that the files are treated like the other non-zipped files and correctly matched to the queues specified in my transforms.conf, e.g. the source reads as follows when looking at the Data Summary: and it is these files that get imported fully."

But that isn't how Splunk works, sorry. If a zipped file matches the monitor stanza and is not filtered by the whitelist/blacklists, it is unzipped and indexed. Here is more information on how Splunk monitors archived files.

0 Karma

jlvix1
Communicator

Hi, what I was expecting was that the zip and the contents are both indexed as per the white/blacklist rules, and that the parsing I defined in props.conf would be carried out, none of this has happened... The zip file name itself was picked up however.

The page you have pointed out has a very brief description that only indicates that zip files are processed and not how they are handled differently from other files.

With most of this understood now, is anyone able to explain why the parsing is not working in props.conf?

Thanks.

0 Karma

lguinn2
Legend

Is there some reason that you must "splunk" the .zip files? Splunk will work better if you allow it to read and parse the uncompressed log files. Then you can blacklist the compressed (.zip) versions of the files in inputs.conf

This will make Splunk work faster and it will be able to deal with all the files in exactly the same way.

BTW, it is also a best practice to use a log rotation tool of some sort to rotate and compress logs. When doing this, you should really keep the current/active log file in uncompressed form, and one prior version in uncompressed form. This will avoid any glitches that could happen when the log files are being simultaneously rolled, compressed and parsed! Here is an example inputs.conf

[monitor:///var/log/]
blacklist = (\.zip|\.tgz)$
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...