This search take only a few second to come back index=* sourcetype=* (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 and return 81,000 records but once I add below to the search I get the search job terminated unexpectedly.
index=* sourcetype=* (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 | timechart span=5min sum(field*) as AA* | addtotals | table _time,Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total
Here is a sample of the data and is written every 5 minutes. When I was writing the data every 15 minutes, it seem to work ok.
2017-12-31 23:55:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Any help to help improve my search is very much appreciated.
Thanks,
Stephen Robinson
This appeared to improve my search dramatically, thank you for the help
Thanks,
Stephen Robinson
This appeared to improve my search dramatically, thank you for the help
Thanks,
Stephen Robinson
can you please write search query which is getting terminated in 101010
sample code for a better understanding of query. Also, i think you need to optimize your search query
give us sample input with the field name and what output you want?
index= sourcetype= (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 | timechart span=5min sum(field*) as AA* | addtotals | table _time,Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total
This provides me the output I want, but the job doesn't seem to efficient. The search creates fields AA1-121 per event and then does all the calculations it needs to do.
This is the sample of data that I have.
2017-12-31 23:55:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2018-01-01 00:00:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2018-01-01 00:05:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
....
Thanks,
Stephen Robinson
See if this helps
index= sourcetype= (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1
| eval Total=0 | foreach field* [eval Total=Total+'<<FIELD>>' ]
| timechart span=5min sum(Total) as Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total