Searches taking long?

[suhprano]

I'm not sure if I am storing the streams properly in order for the search to process optimally. How do I know if the extracted fields are indexed? OR should I be indexing extracted fields per sourcetype to make the search go faster?

It takes me 23 minutes to search through a source returning only 13k events, scanning over 7 million events for the past 48hr time frame. Is that about right in regards to standard performance and speed?

Also, there are 6 other log sources monitored and stored in the same main index, with different sourcetypes and field extractions. Currently my main index holds about 140GB, should this search be taking this long?

Events indexed 3,849,887,346

UPDATED:

Main query - very long, 20+ min.

source="/mnt/journal_logs/current.log" earliest=-48h [search index=cs_logs earliest=-24h |dedup id|fields id] |convert ctime(_time) as GMTTime|geoip clientip|table id,accountid, clientip,score, clientip_countryname, GMTTime |dedup clientip consecutive=true sortby id

Subsearch - very fast, returns 2K events

index=cs_logs earliest=-24h |dedup wowid |fields wowid

[fox]

sounds like a long time, however it is hard to compare as it all depends on your hardware and any concurrent processes running...

tips for speed: always explicitly name the index first and then any other known filters such as sourcetype and source. then reduce the to only the fields that you need (...|fields + ....) this reduces the work load for following piped operations. Then i would build the search up incrementally to see if any one operation is the beast.

If you still need to improve - you may want to consider adding a metadata extracted field at index time...

[netwrkr]

"It takes me 23 minutes to search through a source returning only 13k events, scanning over 7 million events for the past 48hr time frame. Is that about right in regards to standard performance and speed?"

Heck no, that's 'grep' speed (or slower). Lets see your query.

[netwrkr]

hrm. I'm not a search guru but I'm thinking ~13K queries to geoip could be causing some of your pain. I'm still using version 3 of Splunk but found that by changing the geoip script to use memory cache it significantly sped up my searches. In the old geoip script I did

gi = GeoIP.open(gi_db, GeoIP.GEOIP_MEMORY_CACHE)

Which makes it load the entire geoip DB once.

You don't mention what version of Splunk you're using but I found with version 3, if I separate the indexes by year it also made searches significantly faster. I don't think this is an issue with v4 though.

HTH

[suhprano]

I'm testing if this search is faster by moving the index off the main, which has almost 4 billion events for the past week of logs collected...