Hi, I have a number of searches structured where I'm comparing again summary indexes and lookup tables or results I don't want. From the docs my searches are formatted like: index=some_index NOT [search index=another_index | fields field_val] where the summary index another_index is only updated once a day. These searches are used in my dashboards several times and seem to run very slowly. Is there a way to explicitly save sub search so that it doesn't have to be rerun? Or this done automatically? ... View more

Hi, Last week I defined a custom search command, and after reading the documentation I decided that I should reference it in: $SPLUNK_HOME/etc/app/<app_name>/local where my <app_name> is search because its a search command. So I created $SPLUNK_HOME/etc/app/<app_name>/local/commands.conf and put in a single santza for my command. Saved it, restarted splunk and tested it. After debugging worked great, so did all the normal search commands. Over the thanksgiving weekend my splunk started crashing. I think it unrelated and has todo with some security changes we made to the server, but wanted to double check that I didn't my "guess" about where to put it isn't destroying things. (btw its are still crashing once it remove the new commands.conf and restart so I'm fairly sure its not the problem). -Tristan ... View more

So I'm very much not an export on these things, but I think something has gone horribly wrong with my ports.... about a week ago my splunk we stopped loading unless I explicitly specified the port in the URL like http://myurl.com:8000/en-US/manager/search I'm not periodically having splunk web go down all together and getting: 503 Service Unavailable Return to Splunk home page The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running. When I check what ports are being used (with netstat -a) I see none of the normal localhost:8089 lines. I can restart Splunk and have it come back up fine (well using port 8000...) but it will go down again seemingly randomly. Any help (or where to start looking / googling) would be great. Thanks ... View more

I'm running a scheduled search that uses the script command to call a python script, which generates a file. I'd really like to name the file something useful like: output_[earliest]_to_[latest].dat where [earliest] and [latest] are the bounds of the search window. I'm currently just grabbing the first and last result times and using those, but that has two problems. First they aren't the bounds of the search so you loose the information about there being no results between their time and the search window. Second even though my script is set to be non-streaming it seems to run multiple times with partial results while before it finishes so I get multiple files (which I would rather just overwrite as I go). Thanks ... View more

I'm having trouble understanding the math rules on the search line, so instead of continuing to guess what might work and having splunk tell me my formatting is wrong, or give me non sensical results, I'm just going to ask for help. I have a lot of data that is aggregated in to 1 minute bins. For each 1 minute bins I have x_i: mean value of my variable as measured in the 1 minute bin n_i: number of times I measured it in the 1 minute bin sigma_i: standard deviation of the variable measured in the 1 minute bin I'd then like to move to 1 hour bins so I want to calculate a weighted average of my one minute bins. Mathematically this is very straight forward x_ weighted_ average = sum(weight_ i * x_ i) / sum(weight_i) where traditionally weight_ i = 1 / error_ i^2 and for counting statistics error_ i = stand_ deviation_ i / sqrt(n_ i) putting that together we get x_ weighted_ average = sum (x_ i * n_ i / sigma_ i^2) / sum(n_ i) What is the best way to do this kind of math in the search line while aggregating? Or should I just pass it to a python script and back out? Thanks, Tristan ... View more

Hi, I have a weird data structure I'm trying to figure out a better way to handle. The data I'm getting uses category and type fields to tell me what the counts mean. Right now its set up to work something like this, where I do a fair amount of logic to get the counts I want. (but there are actually a lot more categories, types, and logic): index=counts | eval count_a=case(category=1 AND type='A', count1, category=2 AND type='A', count2) | eval count_b=case(category=1 AND type='B', count3, category=2 AND type='B', count4) | table _time, count_a, count_b This gives me results like _time, count_a, count_b 1 10/26/13 8:15:00.201 PM, [BLANK], 50 2 10/26/13 8:15:00.201 PM, 65, [BLANK] 3 10/26/13 8:00:00.201 PM, [BLANK] , 43 4 10/26/13 8:00:00.201 PM, 78, BLANK].... The [BLANK]s are literally blank, because not every line if the file has data for each type of count I'm interested in. After all the logic I have count fields for all the counts and every time step has a value for each count, but they are split over multiple lines. Is there a way to combine them so that I can get table with out blanks in it? Like _time, count_a, count_b 1 10/26/13 8:15:00.201 PM, 65, 50 2 10/26/13 8:00:00.201 PM, 78 , 43.... Thanks, Tristan ... View more

Hi, I'm taking over splunk management for a company I just joined and have found some errors in the way the data was being processed. Consequently I've needed to reindex the whole thing. Is there any easy way to schedule a sequence of commands to run one after the other? I'd like to step through the whole pipeline as if the normally daily cron jobs are being run day1, day2, day3... the way they were schedule, but I'm not seeing an easy way to do that. I mean I can schedule them to run more often, but I don't see an easy way to set the time ranges, that way. Thanks, Tristan ... View more

Hi, I'm seeing a very weird behavior from splunk and wondering if anyone knows whats going on. My input is a cvs file with unix time stamps, which splunk seems to be reading in properly one of my time stamps is 1382037457.102446000 both in the csv file and as read by splunk splunk gives an event time of: 10/19/13 6:38:12.700 AM which seems weird sense this is from a live feed so it should have just happened. If i google a general unix time stamp converter I get: 10 / 17 / 13 @ 11:17:37am PST which seems about right. This isn't happening all the time, on a different search I get ~ 116k events and see 205 or .2% with weird times. For those events the times 1381940863.218676000 1382044524.744784000 which convert to give times of 10 / 16 / 13 @ 8:27:43am PST 10 / 17 / 13 @ 1:15:24pm PST but splunk give times of 10 / 17 / 13 @ 2:55:00.000 PM (PST) 10 / 17 / 13 @ 2:15:24.000 PM (PST) so yeah, I'm more then a little confused. Anyone know whats going on? -Tristan ... View more

I have two indexes one contains objects of interest and things I don't care about, all_results, and the other is a list of things I know I don't care about, bad_results, I want to return elements of the first list that don't exist in the second. I was able to make it working by using. Is there a simpler / better way to do this then something like: index=all_results OR index=bad_results | eval isBad=if(index="bad_results", key, 0) | where key!=isBad | transaction key maxspan=1d maxevents=-1 Thanks ... View more

Hi, So I'm inheriting some splunk code that I'm going through and cleaning up. It contains: rex field=source "/data/splunk/(?<key>.*)/" The search runs fine, when I navigate to it in the browser, (bread crumbs: splunk > manager >> Searches and reports >> [the search name] ) and modify a completely unrelated part of the search and go to save it it kicks the error Your entry was not saved. The following error was reported: SyntaxError: Unexpected token <. I'm fairly sure it's not escaping <> inside of quotes properly, so I can't save my search. I can however create a new search that has the exact same text in it. Thanks, Tristan ... View more