Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About EricLloyd79
EricLloyd79
Builder
Member since:
11-15-2013
06-05-2020
Community Statistics
| Posts | 192 |
| Solutions | 11 |
| Karma Given | 12 |
| Karma Received | 12 |
| Member Since | 11-15-2013 |
Activity Feed
- Karma Re: In Splunk IT Service Intelligence, why can I not see a preview of data in my Aggregate Threshold Views window? for skoelpin. 06-05-2020 12:50 AM
- Got Karma for Re: How do I handle customized time ranges in Splunk IT Service Intelligence (ITSI)?. 06-05-2020 12:50 AM
- Got Karma for Why is backfill not working in Splunk IT Service Intelligence?. 06-05-2020 12:50 AM
- Got Karma for In Splunk IT Service Intelligence, why can I not see a preview of data in my Aggregate Threshold Views window?. 06-05-2020 12:50 AM
- Got Karma for Re: What is the difference in ITSI thresholds between Preview Aggregate Threshold values compared to the configure thresholds values?. 06-05-2020 12:50 AM
- Got Karma for Re: What is the difference in ITSI thresholds between Preview Aggregate Threshold values compared to the configure thresholds values?. 06-05-2020 12:50 AM
- Got Karma for Can I get a list of reports from a splunk query?. 06-05-2020 12:50 AM
- Got Karma for Can I create a KPI in ITSI that is a sum of two other KPIS?. 06-05-2020 12:50 AM
- Got Karma for Why is my Splunk IT Service Intelligence (ITSI) service health score not being reflected correctly?. 06-05-2020 12:50 AM
- Got Karma for Re: Why is my Splunk IT Service Intelligence (ITSI) service health score not being reflected correctly?. 06-05-2020 12:50 AM
- Got Karma for Re: Splunk refuses to ingest particular variable. 06-05-2020 12:49 AM
- Got Karma for Why am I getting this error: splunklib.binding.HTTPError: HTTP 404 Not Found -- Unknown sid when using the Python SDK. 06-05-2020 12:49 AM
- Karma Re: Complex subsearch comparing time periods for micahkemp. 06-05-2020 12:48 AM
- Karma Re: Returning the bytes of each log for aweitzman. 06-05-2020 12:47 AM
- Karma Re: Creating Pivot Chart with two sums? for okrabbe. 06-05-2020 12:47 AM
- Karma Re: How to create a timechart using a root search with data models and pivots? for lguinn2. 06-05-2020 12:47 AM
- Karma Re: How to add the latest field value from two hosts? for lguinn2. 06-05-2020 12:47 AM
- Got Karma for How to create a timechart using a root search with data models and pivots?. 06-05-2020 12:47 AM
- Karma Re: snap to 5 minute increments in timerange for charleswheelus. 06-05-2020 12:46 AM
- Karma Re: Extract string and separate results by different strings for martin_mueller. 06-05-2020 12:46 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 1 | |||
| 1 | |||
| 0 |
06-03-2019
01:42 PM
Thank you! That makes sense now! Because I see hosts on the first query that actually doesn't index anything we use but Splunk metrics but the second always gets the hosts which actually indexes data we use. Perfect thank you now I know which one to use.
Sadly, for some reason, we have historical indexing data for the first query back 30 days but for the second query it only goes back 12 days.
... View more
06-03-2019
01:13 PM
First off, before I even ask, let me state that using Splunk on Splunk is not a solution for us as we are trying to produce a report using the Python SDK, run some transformations on the data received and email it out.
Now that the default answer to this question is out of the picture, I am seeing some unusual differences between running two types of indexing queries:
(1) index="_internal" source="*metrics.log" per_host_thruput | stats sum(kb) by series
(2) index=_internal type=Usage st!=splunk_metrics | stats sum(b) by h
The first query produces almost double the total indexing of the second one (yes I did the conversion because the first one is in kb and the second is in bytes). I am also seeing hosts appear in the list for the first query that does not appear in the second query and vice versa.
Can anyone clarify the differences in these two queries?
Should I expect the amount of indexing returned from them to be the same? If not, why shouldn't I?
... View more
03-19-2019
09:30 AM
As a side note, when I print results as a table, it gets to about 431207 then halts for a second and then starts over at 1 and goes to about 3000.
ex:
431200 20190306 abc 5551212
431201 20190306 abc 5551212
431202 20190306 abc 5551212
431203 20190306 abc 5551212
431204 20190306 abc 5551212
431205 20190306 abc 5551212
431206 20190306 abc 5551212
431207 20190306 abc 5551212
row date fullItemID mdn
1 20190306 abc 5551212
2 20190306 abc 5551212
3 20190306 abc 5551212
4 20190306 abc 5551212
5 20190306 abc 5551212
...
... View more
03-19-2019
09:11 AM
I tried it without the regex and it produced the same ~3000 rows only.
... View more
03-19-2019
09:09 AM
Sure.
/opt/splunk/bin/splunk search 'sourcetype=xyz url=/foo/bar.jsp kpi01 earliest="03/06/2019:00:00:00" latest="03/06/2019:23:59:59" | rex field=_raw "(?:[^=\n]*=){2}(?P[^ ]+)" | eval date=strftime(_time, "%Y%m%d") | table date, mdn, fullItemID | streamstats count as row | fields row *' -maxout 0 >> output15.log
... View more
03-19-2019
08:15 AM
First, yes, we are using "-maxout 0"
For some reason, on one of our hosts, when we run a particular query that pipes into a table, it only returns around ~3000 rows depending on the 24 hour period we are searching for.
Here are some of the unusual things we've noticed:
(1) Its not a resource issue. We have upgraded the host to 32 GB memory and 12 cores.
(2) If we run the query and just have it return raw events, it works as expected.
(3) The query works on another host in our cluster as expected so query is good.
(4) If we run for a window that is 3 hours or LESS it will return the number of rows as expected. 4 hours or more and it returns only around 3000 rows.
(5) I did notice that around 3 hours, it returns just under 1 million rows. Im wondering if there is a 1 million rows limit for tables in the CLI.
(6) This runs as expected in the Splunk web GUI.
Any ideas?
... View more
03-14-2019
10:46 AM
I am having an issue connecting to a Splunk search head with the Splunk PHP SDK:
http://x.x.x.x
I get the error below. It is attempt to connect to https://.. and then gives me an SSL error. When I attempt to connect to https://x.x.x.x through my web browser it gives me a similar error so clearly I need to connect to this search head using http, not https.
I cannot figure out how to tell the SDK to connect with http: instead of https. Help?
PHP Fatal error: Uncaught exception 'Splunk_ConnectException' with message 'SSL connect error' in /var/www/html/eventcount/src/SplunkIndexMonitor/Splunk/Splunk/Http.php:196
Stack trace:
0 /var/www/html/eventcount/src/SplunkIndexMonitor/Splunk/Splunk/Http.php(94): Splunk_Http->requestWithCurl('post', 'https://x.x.x...', Array, 'username=abc&pa...')
1 /var/www/html/eventcount/src/SplunkIndexMonitor/Splunk/Splunk/Http.php(45): Splunk_Http->request('post', 'https://x.x.x...', Array, 'username=abc&pa...')
2 /var/www/html/eventcount/src/SplunkIndexMonitor/Splunk/Splunk/Context.php(94): Splunk_Http->post('https://x.x.x...', Array)
3 /var/www/html/eventcount/src/SplunkIndexMonitor/abc-prod04-run.php(127): Splunk_Context->login()
4 {main}
thrown in /var/www/html/eventcount/src/SplunkIndexMonitor/Splunk/Splunk/Http.php on line 196
... View more
03-07-2019
01:58 PM
Perhaps I am just misunderstanding the concept behind Status over Time but I set a KPI to trigger if it is at Critical 90% of the time for the last 24 hours and when I open it to set that 90% (see screenshot) it shows that in the last 24 hours it was WELL below 90% but this notable event triggers over and over again non-stop almost.
Am I misunderstanding the concept behind an alert that triggers if the KPI is at Critical 90% of the last 60 mins?
... View more
11-20-2018
12:45 PM
I have the query seen in the screenshot and it seems if I use eventstats it returns an incorrect value far larger than it should be for my summation of the difference of two values. If I use stats instead of eventstats, it works as expected.
Can anyone give me any insight why eventstats would be doing this?
Ive attached a screenshot using stats and one with eventstats.
(I need to use eventstats as this is a query for ITSI)
... View more
11-20-2018
12:01 PM
1 Karma
Yes it seemed to work this time. Thanks.
... View more
11-20-2018
11:41 AM
Odd. I've tried this and it didn't work. Let me try again. Im doing it creating a KPI Base Search.
... View more
11-20-2018
11:10 AM
So it seems ITSI provides 4 calculation windows:
Last Minute
Last 5 Minutes
Last 15 Minutes
Last 24 Hours
Has anyone found a way to circumvent this surprising limitation? The majority of my KPIs are measured from the start of the current day until the present moment.
(Is ITSI 4.0 going to address this?)
... View more
10-25-2018
01:50 PM
So I have noticed the backfill data is exactly 3178 for the last 7 days.
When I run in itsi and ask for the last 24 hours, it returns exactly 3178.
So that means that somehow the backfill is not computing for a 5 min window every minute but is doing it for the last 24 hours every minute. Or thats one heck of a coincidence.
... View more
10-25-2018
01:47 PM
Yes, Im not sure if you saw where I wrote this before. I ran it without the time modifier. It did indeed backfill but it filled with values in the 3000s range so its incorrect.
I went into itsi_summary and confirmed. In the last 5 mins, it shows it at 11-13 range. When I searched in the backfilled section of yesterday, it showed in the 3000s range.
So there is an issue with the calculations of the backfill data.
... View more
10-25-2018
12:42 PM
Darn! I thought I had it. I included the timeframe for calculation in the actual query in order to get backfill to work properly but it didn't work. In fact, it didn't backfill at all.
I created it with this:
sourcetype=cpu SCP1_CPU earliest=-5m latest=now
| eventstats sum(SCP1_CPU) as sum_scp1
| eventstats sum(SCP2_CPU) as sum_scp2
| eventstats sum(SCP3_CPU) as sum_scp3
| eventstats sum(SCP4_CPU) as sum_scp4
| eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4
Not being able to get backfill is a limitation I can work around for now. I still really wish I could see the threshold preview but at least I can still set thresholds and then see them propogate in the deep dive. I can't imagine why it would be returning 3000+ values when its always expected to be 11-13 within a 5 min calculation window.
... View more
10-25-2018
12:31 PM
well this is my generated search.
sourcetype=cpu SCP1_CPU
| eventstats sum(SCP1_CPU) as sum_scp1
| eventstats sum(SCP2_CPU) as sum_scp2
| eventstats sum(SCP3_CPU) as sum_scp3
| eventstats sum(SCP4_CPU) as sum_scp4
| eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4 | aggregate_raw_into_service(avg, avg_scp_cpu) | assess_severity(1e9057dc-4f5d-4abf-a773-e85349dd8a84, 840141769da338d4b0e16cdd, true, true) | eval kpi="scp cpu util3", urgency="5", alert_period="5", serviceid="1e9057dc-4f5d-4abf-a773-e85349dd8a84" | assess_urgency
When I press ctrl+shift+e it doesnt do anything at all. Maybe its cause Im on a mac?
I ran that generated search and did a search for yesterday and indeed, the alert_value is 3174. But when I run it for last 5 mins, it shows the expected 11.
This makes me wonder if somehow during backfill with this query, it is summing up the complete days worth of values. Im going to test this.
... View more
10-25-2018
10:39 AM
Okay, so changing it to 5m span, I still see 11 as the number of cpu util value.
Im not sure how to do this "open the kpi calculation" you mention. I see the macro you listed (is that for mac as well) but Im unsure where in the process I am suppose to press that but it sounds like a good path of troubleshooting.
Can you detail a bit more how to open the kpi calculation and expand out the macro? I tried doing so in the services configuration screen but its just a dropdown box for the kpi calculation and doesnt do anything when I press ctl+shift+e
... View more
10-25-2018
10:32 AM
As a side note, I noticed even though the backfilled data is in the 3000s, after a few mins, the data goes down to the 11-13 range that I expect.
... View more
10-25-2018
10:29 AM
Yes, I did. I ran:
sourcetype=cpu SCP1_CPU
| eventstats sum(SCP1_CPU) as sum_scp1
| eventstats sum(SCP2_CPU) as sum_scp2
| eventstats sum(SCP3_CPU) as sum_scp3
| eventstats sum(SCP4_CPU) as sum_scp4
| eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4
| timechart span=1m max(avg_scp_cpu)
in search first and the values are all 11, what I am hoping for. Running the same query in Ad-hoc KPI for ITSI and backfilling 7 days gives me values in the 3000s with the Average calculation even. I am running it every minute and calculating on the last 5 mins. Could that be the issue?
... View more
10-25-2018
10:20 AM
What did you use your your Calculation when creating the KPI? I used Maximum since its already doing the calculation in the query itself...
... View more
10-25-2018
10:18 AM
Well I copied your query into an Ad-hoc KPI and did backfill of 7 hours and it filled it in with data that was all in the 3000s whereas the range for the data is 11-13. So it must be calculating it wrong somehow.
... View more
10-25-2018
10:09 AM
Thats very odd because when I run my query, I see the avg_scp_cpu field in the Interesting Fields column when I run my query:
sourcetype=cpu SCP1_CPU | eventstats sum(SCP1_CPU) as sum_scp1 sum(SCP2_CPU) as sum_scp2 sum(SCP3_CPU) as sum_scp3 sum(SCP4_CPU) as sum_scp4 | eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4
in a regular search so it is creating the avg_scp_cpu field for me. I also see in there fields for sum_scp1, sum_scp2, sum_scp3, and sum_scp4 in the Interesting fields so it getting those as well... Im not sure what you are talking about.
I will try your new version with 4 eventstats to see if the backfill works for me.
Did you see threshold previews for your new query as well?
... View more
10-24-2018
02:31 PM
Sort of good news. Even though I could not see the preview in Thresholding, I can apply it and see it reflected in the Deep Dive so I can at least use Thresholding.
... View more
10-24-2018
02:20 PM
Thank you for this query. Yes, when I put that in for itsi_summary, I see values ranging from 11-13 as expected and as I see in Deep Dive. This is not reflected in the preview for ITSI Thresholds. But I think I can set a static threshold since they the range is so small and see if it will at least work that way.
I am going to need Splunk professional services to troubleshoot with me why I cannot get a backfill from eventstats or eval queries I think.
... View more
10-24-2018
02:18 PM
Ugh I copied the wrong one here.
Here is the one I am using. I have one I am using and seeing the unusual behavior for:
sourcetype=cpu SCP1_CPU | eventstats sum(SCP1_CPU) as sum_scp1 sum(SCP2_CPU) as sum_scp2 sum(SCP3_CPU) as sum_scp3 sum(SCP4_CPU) as sum_scp4 | eval avg_scp_cpu = (sum_scp1 + sum_scp2 + sum_scp3 + sum_scp4) / 4
I tried to create one with stats hoping it would work but the one with stats didnt return anything at all.
... View more
