Merge logs to fill data model

[mertox]

Hey Guys,

in case of a fortiwaf, the device is sending one log message for each connection coming in. When one of the IDS-Signatures matches and the connection is getting blocked, a second msg containing the actual ids information is getting generated. In my understanding, filling the network traffic data model, both of these msgs has to be combined to determine the actual action (allowed/blocked). I could solve this by schedule a saved search each x minutes and | collect the data into a second index. This would increase the storage requirements a lot and comes with an additional time lag as well as mem/cpu load. Therefor: is there another way to get the data into the datamodel without using stashed data? Are there any underlying 'acceleration queries' which are in charge of filling the metadata necessary to get data into a model? Maybe I could add/alter those to get my own data in.

--- Example ---
1st msg (the status represents the package processing result not the action taken by the device)

Oct 23 15:01:00 10.20.0.1 date=2018-10-23 time=15:01:00 log_id=30000000 msg_id=000094175065 device_id=XXXXXXXX vd="root" timezone="(GMT+1:00)Amsterdam,Berlin,Bern,Rome,Stockholm,Vienna" type=traffic subtype="http" pri=notice proto=tcp service=https status=success reason=none policy=policy-01 src=AAA.AAA.AAA.AAA src_port=BBBB dst=CCC.CCC.CCC.CCC dst_port=DDD http_request_time=0 http_response_time=0 http_request_bytes=122 http_response_bytes=234 http_method=get http_url="/" http_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" http_retcode=404 msg="HTTPS get request from AAA.AAA.AAA.AAA:BBBB to CCC.CCC.CCC.CCC:DDD" srccountry="Japan" content_switch_name="none" server_pool_name="some_server_pool" http_host="EEE.EEE.EEE.EEE" user_name="Unknown" http_refer="none" http_version="1.x" dev_id=none

2nd msg:

Oct 23 15:01:00 10.20.0.1 date=2018-10-23 time=15:00:59 log_id=20000002 msg_id=000094175064 device_id=XXXXXXXX vd="root" timezone="(GMT+1:00)Amsterdam,Berlin,Bern,Rome,Stockholm,Vienna" type=attack pri=alert main_type="Protected Hostnames" sub_type="N/A" trigger_policy="" severity_level=Low proto=tcp service=https action=Alert_Deny policy="policy-01" src=AAA.AAA.AAA.AAA src_port=BBBB dst=CCC.CCC.CCC.CCC dst_port=DDD http_method=get http_url="/" http_host="EEE.EEE.EEE.EEE" http_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" http_session_id=none msg="HTTP Host Violation" signature_subclass="N/A" signature_id="N/A" srccountry="Japan" content_switch_name="none" server_pool_name="some_server_pool" false_positive_mitigation="none" user_name="Unknown" monitor_status="Enabled" http_refer="none" http_version="1.x" dev_id="none" threat_weight=0 history_threat_weight=0 threat_level=Off ftp_mode="N/A" ftp_cmd="N/A"

Thanks in advance!