Getting Data In

getting multiple lines when monitoring a directory with subdirectories and xml files

monzy
Communicator

i added my Adium chat logs to be monitored by splunk. i see multiple repeats for any given log event. i verified the Adium logs to ensure that there aren't multiples. i added the logs like so:

/opt/splunk/bin/splunk add monitor "/Users/myuser/Library/Application Support/Adium 2.0/Users/Default/Logs" -sourcetype chat -index chat

mage showing results from grep and splunk search.
http://dl.dropbox.com/u/67500730/adiumLog.jpg

(cant figure out the markup. keeps escaping the xml in the raw result).

jrodman
Splunk Employee
Splunk Employee

"XML Logging" is kind of an oxymoron.
Most xml logging programs don't append to the end of the file, they instead re-create the file with new contents every time they go to add an item to the "log".

This results in really really bad I/O patterns, and also means the file can't be monitored by watching the end of it with eg tail -f.

So it's not really good for performance, and isn't really externally watchable, so it defeats two goals of logging right off the bat.

It can be workable if the application is willing to log single items as xml, and leave the document unclosed until the file closes, but this causes xml parsers to be unhappy with the intermediate state.

For splunk, "xml logging" tends to cause indigestion because our content tracking assumes logfiles don't change their already-written contents (because it doesn't make sense to ever do so.) If there are some important apps, we may have to introduce tricky logic to rewind our idea of EOF to before the closing tags, but this would likely have to be configurable, and putting an xml parser in the tailer is not very exciting.

Of course Adium is built aroud libpurple, and I happen to know it support plaintext logging because I've configured my installs of gaim -> pidgin -> adium to do plaintext logging for around a decade now, because it's easier to parse, easier to grep, easier to read, faster, cheaper, safer, and generally better. I recommend you do the same.

0 Karma

monzy
Communicator

hey Josh. can you share how you are getting plain text logs from the current version of Adium. i poked around and i don't see a clear path to that. googling reveals this quote:
"We will never add plain text logging back to Adium."
https://trac.adium.im/ticket/74

its an old post and perhaps they changed their mind later.

0 Karma

monzy
Communicator

i don't have a props.conf stanza. i just specified a sourcetype with splunk add monitor. perhaps the link for the image might provide some insight. i appreciate you looking into this.

0 Karma

vincesesto
Communicator

Hey monzy, can you please send the details of the 'chat' source type, eg; how is it currently set up in the props.conf file, as this could be an issue.
Regards Vince

0 Karma

monzy
Communicator

"/Users/mmrza/Library/Application Support/Adium 2.0/Users/Default/Logs/IRC.monzy/#splunk/#splunk (2013-03-10T16.17.38-0600).chatlog/#splunk (2013-03-10T16.17.38-0600).xml"

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

What's the source file on those events in splunk?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...