We are not getting extracted fields for some events and there's no apparent pattern as to why. These are all simple extractions and they usually work. This is very problematic as will result in false statistics.
Extracted:
2014-09-03T10:59:59.316-0400 myAction="CachePut" myActualContext="services.ServiceService" myCacheType="remote" myDurationNanos="2209789" myRecordedTimestamp="2014/09/03 10:59:59.316 EDT" myRequestedContext="services.MemberGetSystemMap" {myUow=c17df9e5-1261-4d2d-907a-12ca954ce11f}
2014-09-03T10:59:59.224-0400 ihAction="CachePut" myActualContext="null" myCacheType="local" myDurationNanos="426969" myRecordedTimestamp="2014/09/03 10:59:59.224 EDT" myRequestedContext="Sxc.getMemberEffectiveDates" {myUow=c17df9e5-1261-4d2d-907a-12ca954ce11f}
Not Extracted:
2014-09-03T10:59:59.264-0400 myAction="CachePut" myActualContext="null" myCacheType="remote" myDurationNanos="2293969" myRecordedTimestamp="2014/09/03 10:59:59.264 EDT" myRequestedContext="Power.getMemberEffectiveDates" {myUow=fadcb445-3722-4289-8821-04c6874942e5}
Is this a known bug and/or is there a way we can get debug why the extraction is not occurring for some events?
I still do not see what is not working with your events. If you do not know if you have any manual field extractions configured you probably don't have any. I have encountered sourcetypes where Splunk did not work in 100% of the cases. What I usually do then is switch to a manually configured extraction.
If you do not know about props & transforms yet -> read this documentation
There is a GUI field extractor that might help -> documented here I don't use it so I can't tell if it will work for your problem
If you have access to the file system of your Splunk server you can either create or add the following stanzas to props.conf & transforms.conf in
props.conf
[replaceWithYourSourcetype]
KV_MODE=none
REPORT-test=delims
transforms.conf
[delims]
DELIMS = " ", "="
I still do not see what is not working with your events. If you do not know if you have any manual field extractions configured you probably don't have any. I have encountered sourcetypes where Splunk did not work in 100% of the cases. What I usually do then is switch to a manually configured extraction.
If you do not know about props & transforms yet -> read this documentation
There is a GUI field extractor that might help -> documented here I don't use it so I can't tell if it will work for your problem
If you have access to the file system of your Splunk server you can either create or add the following stanzas to props.conf & transforms.conf in
props.conf
[replaceWithYourSourcetype]
KV_MODE=none
REPORT-test=delims
transforms.conf
[delims]
DELIMS = " ", "="
I guess this is the best answer I will get for this. I appreciate your help but frankly it leaves me a little cold. I'm pretty new to Splunk. Is it typical to see things not really work right and you are on your own to work around it? I'm not accustomed to this approach with paid software (IBM aside.)
Update: Although telling the search to extract (what I think it's always supposed to do anyway) seemed to help, I've found more examples where this doesn't have any effect and no fields are being extracted from events similar to those shown above.
So now I'm getting partially extracted data where some fields are always pulled but the durations in nanos are sometimes not getting pulled out (based on a cursory look)
Almost forgot: Thanks for your help.
I do not know the answer to your problem but here are some questions that might help: Did you set up any manual extractions (using props & transforms)? Are none of the fields extracted or only some? All the events have the same sourcetype right? If not does splunk btool props list display KV_MODE=none for one sourcetype? There aren't any unescaped or extra " in there events that do not work,right (I do not see any in your sample)? Does adding a "| extract " to the end of your search change anything?
http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchReference/Extract