I have a strange one. I'm migrating hosts to a different indexer to redistribute my workload. Last week from the command line on the boxes in question I did: ./splunk add forward-server 10.11.12.13:9997 -auth blah:blah ./splunk remove forward-server 10.9.8.7:9997 -auth blah:blah ./splunk restart I just went back to check things today and I still see the host sending data to the removed server "10.9.8.7" even after a restart (I restarted again just to be sure). Further I tried to do the remove again and it says: In handler 'tcpout-server': 10.21.168.37:9997 forwarded-server not found So I'm sure it's not in the config the 'normal' way. Searching through the splunkbase hasn't turned up much that's relevant. The clients in question are all running 4.1.x versions of splunk, so I'm going to try and upgrade to see if that fixes it, but I've never seen this before where it continues to forward events 4 days after the indexer was removed from the config. Any thoughts on what's going on? Cheers, Rich ... View more

I have some data sources in splunk that are XML formated. The initial request: <query id=12345-54321> <Request_1 initialQueryTime="1294214640000"><SearchParms startNum="1" numResults="14"> <TextQuery type="Title">Children's Hour</TextQuery> </Request_1> </query> The Response: <query id=c12345-54321> <?xml version="1.0" encoding="ISO-8859-1"?> <Response_1> <NumberOfMatches>0</NumberOfMatches> <SearchResults> </SearchResults> </Response_1> </query> or: <tvsquery id=12345-54321> <?xml version="1.0" encoding="ISO-8859-1"?> <Response_1> <NumberOfMatches>2</NumberOfMatches> <SearchResults> <VODShow closedCaptioning="Y" dolby="Y" hd="Y" entitled="Y"> <ResultTitle>Graffiti Bridge HD</ResultTitle> <AssetID>INTL0609000006823480</AssetID> </VODShow> <VODShow closedCaptioning="Y" dolby="Y" entitled="Y"> <ResultTitle>Graffiti Bridge HD</ResultTitle> <AssetID>INTL0609000005278648</AssetID </VODShow> </SearchResults> </Response_1> </query> So I have a field id that should be a transaction. If I do a simple search: host="srchhost*" src="/path/to/tomcat/log/qlogs/*" | transaction id The transactions marry up correctly by id. If I try to look for transactions with specific attributes, I don't get the results I"m expecting. I've tried several variations of the following (with both the transaction command and join command: host="srchhost*" src="/path/to/tomcat/log/qlogs/*" NumberOfMatches < 0 | transaction id [search host="srchhost*" src="/path/to/tomcat/log/qlogs/*" "Request_1" | top TextQuery ] I think I'm not looking at the problem the right way and am hoping someone out there can put me on the right path. I have a field id that is common to the search and the result set. I want to know what TextQuery people used that returned zero NumberOfMatches. So I ask for a result set of id values for zero matches. I then want to return the TextQuery field for every one of those IDs. The individual queries work. I can get a list of the zero result IDs. I can get the TextQuery values for a given ID. I just can't do it in a single search pipeline. Cheers, Rich ... View more

I'm still sifting through the 'realated questsions' proposed in "Ask a Question" (great feature btw), but I don't think my senario is covered. I have a search set to run every 24 hours to sumarize the previous 24 hours stats. host="HOSTSBLAH*" (source="/usr/local/tvs/apache-tomcat/logs/qlogger/*" NOT source="*.gz") | lookup Market_by_Controller_ID Controller_ID as Controller_ID OUTPUT Market as Market | eval QueryFirstTwo=substr(TextQuery,1,2) | transaction MAC, QueryFirstTwo maxspan=5m maxpause=1m delim="," mvlist=TextQuery | eval LastQuery=mvindex(TextQuery, -1) | fillnull value=0 forward | eval MAC="salt".MAC | eval MAC=md5(MAC) | stats count(LastQuery) by Market, Controller_ID, StreamingServerID, forward | fields count LastQuery, Controller_ID StreamingServerID Market forward MAC | collect addtime=true index=dashboard_summarize Data is getting into the index, but none of the fields that have been 'EVAL'ed at some stage. So in this example, LastQuery, forward and MAC are NULL in the summary index. If I take the stats stanza out, the data is collected. It seems odd/bug-like to me that STATS would some how null out EVAL'ed fields. I've taken the stats stanza out and am moving on with my life. I'll generate the stats in a separate query, but again, I'm puzzled why they'd be NULL in the summary. Are there other cases like this that I need to watch out for? Cheers, Rich ... View more

I have what I think is a routine problem, but I don't know how to solve it. I have a log file that has mixed content in it. Some lines have XML and I'm trying to find statistics for it. The issue is I keep hitting limits within splunk. Either a 10,000 event or 50,000 event limit depending on how I write the query. There has to be a way to do this or splunk would be useless as a BI tool, so I'm hoping someone can let me in on the secret for large data sets. The lines look like this (the xml is well formed, the submission form is munging it): 2010-09-12 03:14:55,550 INFO [QELogger] A9 XML: < A9_Request_1 initialQueryTime="2010-09-12T03:14Z" > < SearchParms startNum="1" numResults="14" > < Sort type="Relevance" > < Relevance /> </ Sort > </ SearchParms > < Filter hd="ALL" mode="TextQuery" > < TextQuery type="Keyword" >GREEK < /TextQuery > < Linear forward="14400"/ > < VOD/ > < /Filter > < STBInfo > < MAC > 123456789ABC < /MAC > < Controller_ID > 2583 < /Controller_ID > < ChannelMapID > 2201 < /ChannelMapID > < StreamingServerID > 19501 < /StreamingServerID > < /STBInfo > < Debug docVersion="1.0" searchAgent="hostname.ofserver.net" > TVGI C< /Debug > < /A9_Request_1 > So what we do is a search like this: host="srchqenmana*" "< A9_Request" AND NOT ("FFFFFFFFFFFF" OR "000013ED3AEB" OR "Agent.007") | xmlkv | stats count(MAC) which only pulls the XML lines and filters out some known monitoring MAC addresses and parses the XML. We then pipe that to various stat counts, such as | stats count(MAC) or the like. Some times we count unique macs or other fields. But it never return a full data set. Typically it stops at 50,000 events. There's no warning or alert that the datasent is incomplete, it just finishes normally. So there must be something I should be doing to get all the data. Can someone point me in the right direction? Our traffic levels are such now that we can't get 24 hours worth of counts anymore. Even 'last four hours' tops out at 50,000 at some points. Any help would be appreciated. Rich ... View more