About gcoles

gcoles · ‎08-14-2014

FYI - you can reload the jbridge server by updating the JVM Command Line Options under the "Splunk DB Connect Configuration" dialog (http://your.splunk:8000/en-US/manager/dbx/apps/local/dbx/setup?action=edit). I typically just change the heap size between two equivalent sets of units (eg "-Xmx2048m" and "-Xmx2G") and save to restart the java bridge. You can also reload the java bridge by hitting the debug/refresh endpoint as follows: http://your.splunk:8000/en-US/debug/refresh?entity=admin/dbx-dblookups&entity=admin/dbx-dboutput Verify that this worked by looking at the Java Bridge Status UI's uptime info.

gcoles · ‎04-30-2013

Aha, I just found the optional fields= parameter for splunk.Intersplunk.outputResults() -- will give that a go to see if I can use it to specify column order. outputResults(results, messages=None, fields=None, mvdelim='\n', outputfile=<open file '<stdout>', mode 'w'>)

gcoles · ‎04-30-2013

Thanks sideview, that definitely is the issue. The chart module only works properly when _time is the first field returned. The problem is that the custom command is using the splunk API to return the results as a list of dictionaries (a dict for each row of results), and dictionaries do not allow fieldorder to be set. Do you (or anybody) know of any way to specify the fieldorder via the API as results are returned, aside from switching to csv output?

gcoles · ‎04-29-2013

I've written a custom (generating) Splunk command that retrieves data from Carbon/Graphite, a numerical data-logging tool similar to RRD (but better), for the time period specified in the time-picker. The command works as I had hoped, quickly retrieving the data and providing it in columnar format, with _time , _span , and each requesed column of data provided as outputs. The problem is that when I click the "Results chart" button above the search results, three out of four times the chart doesn't use _time as the x-axis, instead choosing one of the other columns for this, and tries to use _time as the value being plotted. This problem shows up consistently for the same results over a fixed time-period (same number of points, same columns, field values, and _time and _span fields). To validate my work, I wrote another command that dumps the raw field data to a log file, and can't see anything changing in the output of my command between each execution, yet the chart handler behaves inconsistently. I also compared my results to those of timechart with the same span, and they look identical in the raw data dump. Does anyone of of any magic that timechart does to prepare data for the Chart module, outside of the result fields themselves? EDIT: The answer was to use the fields= parameter with splunk.Intersplunk.outputResults() to specify column order. I just needed to create a list of the columns, starting with _time and _span , and append the names of each generated column to the list, then provide that as the second argument: outputResults(results, fields=column_list)

gcoles · ‎03-15-2013

I'm not sure about the dashboard part of the request, but you'd find it easier if you set KV_MODE=JSON in a props.conf entry referencing your sourcetype (on the indexer/search heads). I like to set the sourcetype=json for the json input, and then I only need to set up one props.conf entry for all data that I'll be ingesting later. The KV_MODE automatically runs spath-like field extractions over the entire event, so you can run a search like: sourcetype=json source=xyz timings | stats count by timings.cat or sourcetype=json source=xyz timings.cpu>1000000 Also, if you run your search in "smart" or "verbose" mode, you'll see all your contextual fields on the left with statistics (eg. timings.subtimings.find.subtimings.cat ). With that in place, it will be easier to glob on fields for charting purposes. I think your actual data structure is somewhat cumbersome, though. You may want to have an entity like timings.subtimings.operation = find , rather than find being a key underneath subtimings , that way you can run stats over the values of timings.subtimings.operation , which you could do using your structure. The same principle applies further in the nest. Hope that helps a little bit.

gcoles · ‎01-07-2013

In your case the time (in seconds) field comes three positions after the request details. Since the request is always enclosed in quotes in apache logs, I like to use them as an anchor point for the regex, rather than the start of the log message (it allows for a simpler regex). Also, you have three numeric fields following a double-quote and preceding another double-quote, so I would write a regex like this: \" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?) Note that this forum won't let me use the correct syntax above, so if you use this regex with either rex or the field extraction UI, remove the spaces on either side of response_time. I am not sure if the %T field may include decimal points or not, so I made the regex support them optionally. You can use the regex with either rex (inline pipe to rex), or you can save this field extraction for your apache sourcetype (Manager > Fields > Field Extractions). Here is what a search might look like with Field Extraction configured: index=xxx sourcetype=access_combined response_time>10 And with inline rex: index=xxx sourcetype=access_combined | rex "\" \d+ \d+ (?P< response_time >\d+(?:\.\d+)?)" | where response_time>10 I should note that you also have the %D parameter in your apache configuration, which is the response time in microseconds. I would personally use this field as it has better resolution and its at the end of the log, which makes it easier to anchor within the string: (?P< response_time >\d+)$ Then just adjust your where/search syntax for the precision, response_time>10000000 . As I mentioned before, make sure to remove the spaces around 'response_time' above, when you use it in any of the splunk UI's.

gcoles · ‎12-19-2012

Thanks dart, this seems to work well for me. I'll try to make it into a macro so that the span time can be supplied as an argument and the macro used as a replacement for bucket .

gcoles · ‎12-04-2012

Thats true, sideview, but timechart handle output columns differently than stats (a column will exist for each permutation of the field being split upon rather than a single column with a row for each permutation), and also doesn't let you segment by multiple fields, like stats does (count by a,b,c). Since most of my work involves nested calls to stats, eventstats, and streamstats, using timechart anywhere in the search flattens the data too much to be useful. However, it could be used as the generating command I was talking about, since it could create the _time fields.

gcoles · ‎12-04-2012

I also ran into this problem yesterday. I think the best way would be to create a generating command (creates synthetic events) that outputs a fake event for every day (or configurable timespan) in the timerange for which it is called, then use a join type=outer _time [search your_search] to join your timechart or bucket'ed stats results by time. Then you can use eval to find events where the field(s) from the join is/are null ( eval zeroresults=if(isnull(myfield),"true","false") ). If writing generating commands is too difficult, you could create a csv file with a column for _time, and set the timestamp for each row to the day boundary (in seconds since 1970), then use the inputlookup command to generate the synthetic events and perform the join mentioned earlier. Thinking of this, I'm surprised Splunk wouldn't already have a command to do this. May need to dig deeper in the commands documentation.

gcoles · ‎12-03-2012

Just filed an ER for it.

gcoles · ‎12-03-2012

I am writing a search that looks at weighted moving averages of data points summarized and logged at 2 minute intervals. I need to bucket the data into two minute spans, in a window of ten minutes. The search will run every minute, and look at the past ten minutes worth of data, thus, there should always be five buckets of 2 minutes each. You'd think this would be as easy as: earliest=-10m@m latest=@m *base_search* | bucket _time span=2m | stats xxx by _time However, the bucket command (and timechart, etc) always make bucket boundaries snap to even numbered time boundaries, rather than being relative to the search time boundaries. To elaborate, if the search is made at 10:10:23, there are five buckets, for 10:00, 10:02, 10:04, 10:06, and 10:00, and if the search is run at 10:11:xx, there are six buckets: 10:00, 10:02 ... 10:10, with the first and last bucket containing one minutes' worth of data each (half the data). What I think should happen with the 10:11 search is five buckets, the first being 10:01, then 10:03, etc. Has anyone found a way to do this that still lets them sleep at night? Maybe a call to eval that segments time similarly to the bucket command (could be a macro).

gcoles · ‎11-02-2012

Almost, you have to search the _audit index for the audit logs: index=_audit action=search search=*

gcoles · ‎11-02-2012

In the scripted input, you could pipe the output of the curl command through a command like hxnormalize, which is part of the html-xml-utils package in Linux.

gcoles · ‎11-02-2012

No problem, can you paste in a sample log record or your apache LogFormat directive?Apache log files typically don't include field=value pairs (just values), so regular expressions must be position-sensitive (eg populate this field with the fifth element in the message).

gcoles · ‎11-02-2012

This can be difficult, since so many top level domains are possible. If you have a fixed list of tld's that are common in your result set, you could use a regex like this to extract the website name (eg, google) from an existing field (eg, referer😞 (?P< website>[^. ]*)\.(?:co[m]?|net|cn|ca) (Note that the space in front of the field name, website, is only there because the forum code required them for parsing, remove them in your rex command and/or transforms). Example usage (with the extra space): index=weblogs referer=* | rex field=referer "(?P< referer_website>[^. ]*)\.(?:co[m]?|net|cn|ca)" | top referer_website

gcoles · ‎11-02-2012

As hexx recommended, posting this as an answer: The following commands instantly fixed the CPU usage by the universal forwarder: /etc/init.d/ntp stop (date +"%H:%M:%S" |perl -pe 'chomp';echo `date +"%N"` / 999999999|bc -l) | sudo perl -ne 'chomp;system ("date","-s",$_);' /etc/init.d/ntp start Note that a simpler date command has been referenced in other articles on the web, but we found it to be less accurate than the above (which includes microseconds).

gcoles · ‎11-02-2012

You can find out whether or not it is Splunk listening on udp:162 with the following command on any splunk server: $SPLUNK_HOME/splunk list udp It is hard to answer further without knowing what your setup is, aka do you use forwarders, do you have an snmptrapd instance logging to files that are ingested by Splunk on the indexer or forwarders, etc. If you use forwarders, it is possible that they have an inputs.conf file somewhere in $SPLUNK_HOME/etc/apps that specifies listening on udp:162, which won't show up in your web UI on the indexer and/or search head(s).

gcoles · ‎11-01-2012

I have no AIX systems to test with, hope you found a fix, jnhth!

gcoles · ‎11-01-2012

I did write my own 'printf' splunk command that let me inject newlines at will, by as Ayn noted, I would then need css modifcations or some other magic to make it work in the HTML UI, which seemed cumbersome. Rather than bending Splunk to my will, but I found that I could get what I was looking for by altering the search to split by permutations (one event returned per permutation) instead of trying to list out all the permutations with line breaks inside of a single event.

gcoles · ‎11-01-2012

Do you already have a Field Extraction set up for the response_time field? If you do, you can simply use greater and less than signs in the search: index=weblogs response_time>10

gcoles · ‎11-01-2012

I can verify that this is happening with our instances after the 5.0 upgrade, as well. Could be a change to the $SPLUNK_HOME/etc/log.cfg file, but I don't have an old copy handy to compare against.

gcoles · ‎11-01-2012

I recently got this working on my heavy forwarders by creating $SPLUNK_HOME/etc/system/local/outputs.conf and adding the following lines: [tcpout] forwardedindex.filter.disable = true You can read more in the outputs.conf documentation. UPDATE If data is placed in main rather than _internal, the inputs.conf definitions must have either been changed or duplicated, with either no index, or the main index specified (because main is the fallback). Double check $SPLUNK_HOME/etc/system/default/inputs.conf -- it should look like this: [monitor://$SPLUNK_HOME/var/log/splunk] index = _internal If you're on Linux, find all input stanzas for $SPLUNK_HOME/var/log/splunk with: sudo find /opt/splunk/etc -name 'inputs.conf' -exec grep -H "/var/log/splunk" {} \; Do this on the indexer and forwarders and examine the files to ensure that index = _internal

gcoles · ‎10-25-2012

Anyone get Scripted Auth working in 4.3 (or any other version) where the user's email address is populated? I can't find any way to do this in the documentation, but seems necessary because so much of what can be done with Splunk assumes that you have an email address. What makes this much worse is that users who have been populated with Scripted Auth cannot be edited from within the Manager UI, so I am unable to set the users' email addresses manually, as well. I am currently using PAM auth (by necessity -- don't ask), and I can easily use an algorithm to set the user's email address, if only I knew where. In that past I've used Splunk with AD/LDAP auth and had no trouble with email address population.

gcoles · ‎08-03-2012

Us as well. We are using Splunk for graphing website performance and analytics, and I would LOVE to put annotations on the graph to show whenever we push new code to the website, so that we can try to better visualize the impacts of those pushes in near-real time (which may occur several times per day).

gcoles · ‎07-08-2012

This sounds like the same thing that happened to us on debian squeeze. Check out this post. To fix, on the ubuntu server having issues, try these commands: /etc/init.d/ntp stop (date +"%H:%M:%S" |perl -pe 'chomp';echo `date +"%N"` / 999999999|bc -l) | sudo perl -ne 'chomp;system ("date","-s",$_);' /etc/init.d/ntp start If you don't have perl installed for some reason, you can replace the middle date command with a simpler one referenced in the post above.

Posts	44
Solutions	3
Karma Given	40
Karma Received	42
Member Since	‎02-21-2011

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Problem with custom search command vs timechart in...

Bucket command time boundary issues

Scripted Auth in 4.3 with User Email

Universal Forwarder High CPU after Leap Second Cor...

Add Line Breaks with Eval

Re: DB Connect configuration: Unable to save datab...

Re: Problem with custom search command vs timechar...

Re: Problem with custom search command vs timechar...

Problem with custom search command vs timechart in...

Re: Parsing JSON data

Re: Tracing Response time of apache using splunk

Re: Bucket command time boundary issues

Re: Find days with no events

Re: Find days with no events

Re: Bucket command time boundary issues

Bucket command time boundary issues

Re: No longer logging to searches.log since upgrad...

Re: Scripted input with cURL

Re: Tracing Response time of apache using splunk

Re: how does splunk analyse URL？

Re: Universal Forwarder High CPU after Leap Second...

Re: Receiving SNMP alerts without a Data Input con...

Re: Universal Forwarder High CPU after Leap Second...

Re: Add Line Breaks with Eval

Re: Tracing Response time of apache using splunk

Re: No longer logging to searches.log since upgrad...

Re: Can't write data to _internal index

Scripted Auth in 4.3 with User Email

Re: Annotation chart over line chart overlay

Re: High Splunk Universal Forwarder CPU usage