Hi All,
Just trying to work out how to use eventgen for multiline logs such as oracles' hideous audit file.
Audit file /ora/app/oracle/admin/cdh/adump/oracle_audit_log.aud
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
ORACLE_HOME = /ora/app/oracle/product/11.2.0.4/db_1
System name: SunOS
Node name: hostname-oracle4
Release: 5.10
Version: Generic_150400-26
Machine: sun4v
Instance name: this_instance
Redo thread mounted by this instance: 1
Oracle process number: 90
Unix process pid: 5542, image: oracle@hostname-oracle4
Mon Nov 2 11:33:30 2015 +11:00
LENGTH: "223"
SESSIONID:[8] "23491997" ENTRYID:[1] "1" USERID:[6] "SIEM_TEST1" ACTION:[3] "44" RETURNCODE:[1] "0" LOGOFF$PREAD:[1] "0" LOGOFF$LREAD:[2] "24" LOGOFF$LWRITE:[1] "0" LOGOFF$DEAD:[1] "0" DBID:[9] "847685182" SESSIONCPU:[1] "4"
Mon Nov 2 11:33:33 2015 +11:00
LENGTH: "223"
SESSIONID:[8] "23491997" ENTRYID:[1] "1" USERID:[6] "SIEM_TEST1" ACTION:[3] "115" RETURNCODE:[1] "0" LOGOFF$PREAD:[1] "0" LOGOFF$LREAD:[2] "24" LOGOFF$LWRITE:[1] "0" LOGOFF$DEAD:[1] "0" DBID:[9] "847685182" SESSIONCPU:[1] "4"
I have tried this in evengen.conf
[oracle_audit_log.aud]
disabled = false
mode = replay
index=sec_database
sourcetype=oracle:audit:text
# breaker= ^\r\n
bundlelines = true
## Generate all events in sample
count = 0
earliest = -5m
latest = now
interval = 300
outputMode=spool
spoolFile = sample.oracle
# host.token
# host.replacement = hosts.list
## Replace timestamp
token.0.token = \w{3}\s+\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\d{4}
token.0.replacementType = timestamp
token.0.replacement = %a %b %d %H:%M:%S %Y
But it throws this error
2015-11-03 13:55:51,350 INFO Retrieving eventgen configurations from /configs/eventgen
2015-11-03 13:55:51,894 INFO Creating timer object for sample 'oracle_audit_log.aud' in app 'TA_ob-3_oracle_eventgen'
2015-11-03 13:55:51,896 INFO Starting timers
2015-11-03 13:55:51,898 ERROR Can't find a timestamp (using patterns '['\\w{3}\\s+\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2}\\s+\\d{4}']') in this event: 'LENGTH: "223"
'.
2015-11-03 13:55:51,901 ERROR Exception in sample: oracle_audit_log.aud
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py", line 47, in run
partialInterval = self.sample.gen()
File "/opt/splunk/etc/apps/SA-Eventgen/lib/eventgensamples.py", line 506, in gen
self._lastts = self._getTSFromEvent(self._rpevents[self._currentevent])
File "/opt/splunk/etc/apps/SA-Eventgen/lib/eventgensamples.py", line 702, in _getTSFromEvent
raise ValueError("Can't find a timestamp (using patterns '%s') in this event: '%s'." % (formats, event))
ValueError: Can't find a timestamp (using patterns '['\\w{3}\\s+\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2}\\s+\\d{4}']') in this event: 'LENGTH: "223"
'.
I'm guessing it is trying to find a timestamp on every line, but obviously one does not exist.
I even tried messing around with breaker.
I even removed all the junk at the top of the log with no luck.
Any ideas would be appreciated.
Your breaker is wrong and it's looking for the timestamp in this one line:
LENGTH: "223"
Use this as your breaker and discard the header data in the sample log:
([\n\r]+)\w{3}\s+\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\d{4}
Its been a while since I looked at this but I should note the error is in the eventgen code not in the Splunk ingestion config.
Eventgen can't handle generating sample data where it is multiple lines. It is breaking as eventgen tries to make fake logs from this sample data.
You are right though if I was having problems ingesting this data into Splunk. Changing the breaker would make this work just fine. However changing the breaker will have no effect on eventgen as far as I know.
I meant with eventgen, not on data ingestion. The error message is saying it's failing to parse the timestamp on the length line... and I have little knowledge about event gen but figured if you put the proper breaker in there it could work.
Good to know. I haven't looked much at Eventgen since I posted this ~2 years ago it has likely changed a bit since then.
https://github.com/splunk/eventgen
Back then if eventgen didn't find a timestamp on every line it borked out as shown above.
Reviving this old thread... I am encountering a similar issue trying to replay a log file containing multi-line event.
Anyone has any suggestions on how to make this work?
Old threads should not be "revived". It's better to post a new question describing the problem you are having.
Reviving this old thread... I am encountering a similar issue trying to replay a log file containing multi-line event.
Anyone has any suggestions on how to make this work?
You should create your own question and provide the same level of details as the original poster. We cant tell you how to fix your problem without all of the same details and therefore you should start a new question.
Will do so... Thanks
can you confirm if you have used token.0.replacementType = timestamp stanza which covers all possible timeformats available in your multi-line event data?
Also, if you are in app, say myapps with your sample file, mysample.csv, you can run to check the actual error ( should also be in splunkd.log)
/opt/splunk/etc/apps/myapps/samples$ /opt/splunk/bin/python ../../eventgen/bin/eventgen.py -v -s mysample.csv ../local/eventgen.conf
could you do one of these two?
I am also facing the same issue with eventgen when i am trying to generate Windows HostMon data. I think the breaker is the causing the problem. Any suggetions for mutliline event generation ?