Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About pestatp
pestatp
Path Finder
Member since:
03-02-2020
11-04-2021
Community Statistics
| Posts | 12 |
| Solutions | 1 |
| Karma Given | 3 |
| Karma Received | 1 |
| Member Since | 03-02-2020 |
Activity Feed
- Tagged Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds on Getting Data In. 11-04-2021 06:28 AM
- Posted Re: Replace space in timestamp with a 0 (with datetime.xml possibly?) on Getting Data In. 11-04-2021 05:54 AM
- Karma Re: Replace space in timestamp with a 0 (with datetime.xml possibly?) for scelikok. 01-08-2021 08:38 AM
- Posted Re: Replace space in timestamp with a 0 (with datetime.xml possibly?) on Getting Data In. 01-08-2021 08:34 AM
- Posted Re: Replace space in timestamp with a 0 on Getting Data In. 01-08-2021 06:30 AM
- Posted Re: Replace space in timestamp with a 0 on Getting Data In. 12-02-2020 07:55 AM
- Posted Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds on Getting Data In. 12-02-2020 06:18 AM
- Tagged Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds on Getting Data In. 12-02-2020 06:18 AM
- Tagged Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds on Getting Data In. 12-02-2020 06:18 AM
- Tagged Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds on Getting Data In. 12-02-2020 06:18 AM
- Karma Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second for to4kawa. 06-05-2020 12:51 AM
- Got Karma for Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second. 06-05-2020 12:51 AM
- Karma Calculate total bps from NetFlow data for masato_sekiguch. 06-05-2020 12:50 AM
- Posted Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-05-2020 11:11 AM
- Posted Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-05-2020 11:07 AM
- Posted Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-03-2020 04:04 AM
- Posted Re: Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-03-2020 03:29 AM
- Posted Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-02-2020 09:51 AM
- Tagged Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-02-2020 09:51 AM
- Tagged Given Netflow data with start/end time and total bits transfered, calculate total bandwith used per time second on Getting Data In. 03-02-2020 09:51 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 |
11-04-2021
05:54 AM
I have still not been able to figure this out. I haven't seen anything in newer versions that would help with this, anyone know of anything?
... View more
01-08-2021
08:34 AM
@scelikok That is the closest I have gotten it so far, but it doesn't look like it's treating the spaces as zeros, it's just ignoring them. It indexed 01-08-2021:11.28.23. 8213 as 1/8/21 11:28:23.821 AM - that is better than what it was doing before, but still not quite right. It should be indexed as 1/8/21 11:28:23.008213 AM
... View more
01-08-2021
06:30 AM
After some more searching, I came across datetime.xml. It looks like that can be used to do custom datetime extraction although, I am a bit confused by the documentation. Does anyone know if that would work for this scenario?
... View more
12-02-2020
07:55 AM
No, I cannot ignore it. These are SIP messages and they need to be in as precise order as possible to ensure the call flow is correct. Many of the messages are within just a few thousandths of a second of each other.
... View more
12-02-2020
06:18 AM
I have events that unfortunately use a space instead of a 0 in their timestamp field. The timestamp goes down to 6 decimal places, so there can be as many as 5 leading spaces in the decimal seconds section. Each event starts with the timestamp as below. As you can see, it has a leading space and I'd like to change that to a 0 [12-02-2020:08.31.44. 15133] SIP IN: I have tried using SEDCMD in the props.conf, but it didn't seem to work on the newly indexed events, is my regex not correct or am I way off? [sip_sbc]
SEDCMD-replace_space=s/^(\[[0-9-:\.]{20}\]) ()/\10\2/g Edit: This is my current props.conf settings. This works fine for the timestamps that have 6 digits after the . but any of them that have leading spaces fail to get the proper timestamp [sip_sbc]
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
LINE_BREAKER = ----------------------------------------------------------------------------------------([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %m-%d-%Y:%H.%M.%S.%6N
TIME_PREFIX = [
category = Custom
pulldown_type = 1
BREAK_ONLY_BEFORE_TIME =
disabled = false
MUST_BREAK_AFTER = Edit No. 2: I have added a custom datetime.xml file for the app. It does have an effect, but it's not working quite right. It doesn't pad the leading spaces to 0, it just removes the space and therefor causes the subseconds to be much higher than they are supposed to be on some timestamps. <datetime>
<define name="custom_dateformat" extract="month, day, year">
<text><![CDATA[\[(\d+)-(\d+)-(\d+)]]></text>
</define>
<define name="custom_timeformat" extract="hour, minute, second, subsecond">
<text><![CDATA[\[\d+-\d+-\d+:(\d+).(\d+).(\d+).\s*(\d+)]]></text>
</define>
<timePatterns>
<use name="custom_timeformat" />
</timePatterns>
<datePatterns>
<use name="custom_dateformat" />
</datePatterns>
</datetime> This datetime.xml caused an event with the timestamp: 01-08-2021:11.28.23. 8213 (note the 2 spaces before 8213) to be parsed as 1/8/21 11:28:23.821 (this is 8 10ths of a second after the timestamp should be) It should be 1/8/21 11:28:23.008213
... View more
Labels
- Labels:
-
props.conf
03-05-2020
11:11 AM
The search that works the best for me in this scenario which is modified from to4kawa's answer is:
| makeresults
| eval _raw="{\"endtime\":\"2020-03-02T17:35:31.850000Z\",\"timestamp\":\"2020-03-02T17:04:51.630000Z\",\"bytes_in\":64,\"dest_ip\":\"xxx.xxx.187.28\",\"dest_mask\":0,\"dest_port\":5061,\"dest_sysnum\":0,\"event_name\":\"netFlowData\",\"exporter_ip\":\"10.136.57.2\",\"exporter_sampling_interval\":1000,\"exporter_sampling_mode\":1,\"exporter_time\":\"2020-Mar-02 17:35:22\",\"exporter_uptime\":1553552496,\"flow_end_rel\":1553562346,\"flow_start_rel\":1551722126,\"ingress_vlan\":103,\"input_snmpidx\":114,\"netflow_version\":9,\"nexthop_addr\":\"0.0.0.0\",\"observation_domain_id\":0,\"output_snmpidx\":0,\"packets_in\":1,\"protoid\":6,\"seqnumber\":54418,\"src_ip\":\"10.136.216.199\",\"src_mask\":0,\"src_port\":1028,\"src_sysnum\":0,\"tcp_flags\":16,\"tos\":184}#
{\"endtime\":\"2020-03-02T17:35:31.820000Z\",\"timestamp\":\"2020-03-02T16:54:11.510000Z\",\"bytes_in\":68,\"dest_ip\":\"xxx.xxx.187.28\",\"dest_mask\":0,\"dest_port\":5061,\"dest_sysnum\":0,\"event_name\":\"netFlowData\",\"exporter_ip\":\"10.136.57.2\",\"exporter_sampling_interval\":1000,\"exporter_sampling_mode\":1,\"exporter_time\":\"2020-Mar-02 17:35:32\",\"exporter_uptime\":1553562496,\"flow_end_rel\":1553562316,\"flow_start_rel\":1551082006,\"ingress_vlan\":54,\"input_snmpidx\":49,\"netflow_version\":9,\"nexthop_addr\":\"0.0.0.0\",\"observation_domain_id\":0,\"output_snmpidx\":0,\"packets_in\":1,\"protoid\":6,\"seqnumber\":54509,\"src_ip\":\"10.136.189.15\",\"src_mask\":0,\"src_port\":1028,\"src_sysnum\":0,\"tcp_flags\":16,\"tos\":0}"
| makemv delim="#" _raw
| stats count by _raw
| rename COMMENT as "this is sample"
| spath
| fields - _* count
| dedup src_ip,src_port,dest_ip,dest_port,exporter_ip,timestamp
| eval start_time = strptime(timestamp . "-0000", "%FT%T.%6QZ%z")
| eval end_time = strptime(endtime . "-0000", "%FT%T.%6QZ%z")
| eval diff_secs = (end_time-start_time)+1
| eval diff = tostring((diff_secs), "duration")
| eval bps=if(isnull(bytes_in/diff_secs),0,bytes_in/diff_secs)
| addinfo
| eval start_time_adj=if(start_time<info_min_time,info_min_time,start_time)
| eval temp=mvrange(start_time_adj,end_time+1)
| table exporter_ip bps temp
| eval bps=bps
| mvexpand temp
| rename temp AS _time
| bucket span=1s _time
| timechart cont=f partial=f sum(bps) as total_bps by exporter_ip
A couple of the changes involve the mvrange start time. If you don't use the start time from your selected time range, then your timechart will display blank times way back to when the first start timestamp in your data and in mine, that is always significantly before the time range I want to see. I also split the information by exporter_ip which correlates to the IP of the network device sending the data.
... View more
03-05-2020
11:07 AM
1 Karma
My solution was to modify your answer a little bit, but the biggest thing I did was upgrade our Splunk indexer. The original server was getting quite old and too slow for a query like this. Just the upgrade decreased the job time from 48 seconds to less than 2 for the same query.
... View more
03-03-2020
04:04 AM
Using stats by prevented it from truncating results and it seems to be a bit faster, but still fairly slow.
This query took 48 seconds for 15 minutes worth of data:
sourcetype=stream:netflow
| fields - _* count
| dedup src_ip,src_port,dest_ip,dest_port,exporter_ip,timestamp
| eval start_time = strptime(timestamp . "-0000", "%FT%T.%6QZ%z")
| eval end_time = strptime(endtime . "-0000", "%FT%T.%6QZ%z")
| eval diff_secs = (end_time-start_time)+1
| eval diff = tostring((diff_secs), "duration")
| eval bps=if(isnull(bytes_in/diff_secs),0,bytes_in/diff_secs)
| addinfo
| eval start_time_adj=if(start_time<info_min_time,info_min_time,start_time)
| eval temp=mvrange(start_time_adj,end_time+1)
| table exporter_ip bps temp
| stats values(bps) as bps by temp, exporter_ip
| rename temp AS _time
| bucket span=1s _time
| timechart cont=f partial=f sum(bps) as total_bps by exporter_ip
It's too bad that there isn't something like concurrency that can be used to count a field instead of just the number of events.
... View more
03-03-2020
03:29 AM
The reason I said it only works with a limited number of events is because mvexpand generates a seriously large number of results. Using this on my data this morning for 15 minutes, it creates 12 million results from ~32,000 actual Netflow events. If I attempt to view anything longer than 15 minutes or so, then my results get truncated due to memory. I already upped the mvexpand memory to 2,048.
I was really hoping to find a less "expensive" way to accomplish this.
... View more
03-02-2020
09:51 AM
I am looking for an efficient way to calculate the total bandwidth used per second on a device from our netflow data. The netflow data we receive contains a start and end time for the flow(timestamp and endtime respectively) as well as the total bytes that have been transferred. It is simple enough to calculate BPS for each flow, but I cannot figure out how to calculate total bandwidth in a usable manor.
Example netflow data:
{"endtime":"2020-03-02T17:35:31.850000Z","timestamp":"2020-03-02T17:04:51.630000Z","bytes_in":64,"dest_ip":"xxx.xxx.187.28","dest_mask":0,"dest_port":5061,"dest_sysnum":0,"event_name":"netFlowData","exporter_ip":"10.136.57.2","exporter_sampling_interval":1000,"exporter_sampling_mode":1,"exporter_time":"2020-Mar-02 17:35:22","exporter_uptime":1553552496,"flow_end_rel":1553562346,"flow_start_rel":1551722126,"ingress_vlan":103,"input_snmpidx":114,"netflow_version":9,"nexthop_addr":"0.0.0.0","observation_domain_id":0,"output_snmpidx":0,"packets_in":1,"protoid":6,"seqnumber":54418,"src_ip":"10.136.216.199","src_mask":0,"src_port":1028,"src_sysnum":0,"tcp_flags":16,"tos":184}
{"endtime":"2020-03-02T17:35:31.820000Z","timestamp":"2020-03-02T16:54:11.510000Z","bytes_in":68,"dest_ip":"xxx.xxx.187.28","dest_mask":0,"dest_port":5061,"dest_sysnum":0,"event_name":"netFlowData","exporter_ip":"10.136.57.2","exporter_sampling_interval":1000,"exporter_sampling_mode":1,"exporter_time":"2020-Mar-02 17:35:32","exporter_uptime":1553562496,"flow_end_rel":1553562316,"flow_start_rel":1551082006,"ingress_vlan":54,"input_snmpidx":49,"netflow_version":9,"nexthop_addr":"0.0.0.0","observation_domain_id":0,"output_snmpidx":0,"packets_in":1,"protoid":6,"seqnumber":54509,"src_ip":"10.136.189.15","src_mask":0,"src_port":1028,"src_sysnum":0,"tcp_flags":16,"tos":0}
I have been able to come up with a solution, but it only works with very small timeframes. I would like something that is significantly more robust. The code below will only work with a very limited number of events:
sourcetype=stream:netflow
| dedup src_ip,src_port,dest_ip,dest_port,timestamp,exporter_ip
| eval start_time = strptime(timestamp . "-0000", "%FT%T.%6QZ%z")
| eval end_time = strptime(endtime . "-0000", "%FT%T.%6QZ%z")
| eval diff_secs = end_time-start_time
| eval diff = tostring((diff_secs), "duration")
| eval bps=if(isnull(bytes_in/diff_secs),0,bytes_in/diff_secs)
| addinfo
| eval start_time_adj=if(start_time<info_min_time,info_min_time,start_time)
| eval temp=mvrange(start_time_adj,end_time)
| mvexpand temp
| rename temp AS _time
| bucket span=1s _time
| timechart sum(bps) as total_bps
... View more
03-02-2020
06:02 AM
This still wouldn't provide a total bandwidth from multiple events, correct?
I took the original question to be asking how to get the total bandwidth for a particular time from multiple events with total bytes and a time range.
... View more
03-02-2020
04:52 AM
Did you ever come up with a solution to this? I am attempting to solve the same problem and can't quite figure it out.
Unfortunately I don't have the option to use Kafka, so a SPL solution would be the best for me.
... View more
