Subsearch within a bucket

[nickhills]

I am looking to correlate events from two different sources whereby a rare event in source A, (in a 1 hour window) searches for events in source B (in the same 1 hour window)

These are mobile phone records, and source A lists the Phone Numbers I want to seach for in the CDR logs (source B).

This is what I have so far:

eventtype=phone_logs flagged=true |bucket  _time span=1h|stats dc(phone_log_cli) by _time phone_log_cli |join max=0 phone_log_cli[search sourcetype=cdr SCRIPT_RES="d*" BDA="*" BDUR>5]|dedup ID|eval AOA2=substr(AOA,1, 4)+"xxxxxx"| eval BDA2=substr(BDA,1,6)+"xxxxxxxx"| eval CallTime=ACONDATE|rename SCRIPT_RES as "CallType" |table _time CallTime AOA2 BDA2 BDUR BREASON customer country Network CallType

This returns bucket windows for source A, but as the search runs it populates the first bucket where a cli appears with ALL the CDR records of the same device.

That is to say (over a 4 hour search window 08:00-12:00), if a phone made 1 call an hour in each of the 4 hours, then all 4 calls show up in the 08:00 result set.

I want to run this query over 30 days, and at the moment, every call gets returned in the first window in which its CLI was flagged in source A.

How can I restrict the 'join search' to only search within the same time window as the bucket from first search?
I have looked at localize - map, but this seems problematic also.

[landen99]

max=0 does not bypass the subsearch hard limit of 50k results. Also, there are subsearch timeout considerations as well.

As for your search, I avoid joins. But your join has not reporting command so I have no idea what information you are looking to add to your base search, so I cannot suggest anything specific to improve the search. Usually, it is best to search both sources at the beginning and then bring them together with stats

[nickhills]

Ok, so...
The Subsearch works as per your example, but i want to pull some fields out of source A which are used in a table.

using your example:

source=B [search source=A phoneno=XXX blaha=YYY| dedup phoneno date_mday date_hour| fields + date_time date_mday phone no username]| table srcA_phoneno srcA_username srcB_AOA srcB_BDA

obviously, adding 'username' to the fieldset for the subsearch results in the main search looking for matches on the username also (which will not exist in source B)

how can i get additional fields from the sub search to be available for post processing along with results from the main search?

[kristian_kolb]

As far as I know it the subsearch approach I outlined earlier would not do the job required. The results from the subsearch would be added as part of the main search - and if 'username' is not present there, the search would essentially fail - so subsearch is not the way to go.

/k

[kristian_kolb]

Not trying to go into your question in depth, so I concentrating on your first paragraph.

Could you pick out date_hour and date_mday for your rare events and use it to limit the search among the more common events.

---

UPDATE:

I was thinking along the lines of a subsearch, but as I said I haven't really looked at your question, so I'm not sure it would apply. My line of thinking, based on your first paragraph was;

source=B [search source=A phoneno=XXX blaha=YYY| dedup phoneno date_mday date_hour| fields + date_time date_mday phoneno]| the_rest_of_your_search_here

The inner search would look at your rare events and return unique combinations of phoneno date_mday and date_hour, which would be used as part of the outer search in source B.

Hope this helps at least a little bit.

/K

[nickhills]

OK, so this makes sense, but how do i pass date_hour and date_mday into the join search?
Do i have to build this into a full time format?