Hello @PavelP, thank you for your support for my issue. It seems that I have not mentioned that we need to filter out those logs (drop to null queue) to save the license. We are doing filtering in props / transforms on HF where the streamer is installed. Below are the logs that appeared after each applied filter. rec_type=71 src_port=40519 src_bytes=80 dns_rec_id=1 ssl_ticket_id=0000000000000000000000000000000000000000 ips_count=0 fw_policy= event_usec=0 ssl_cert_fingerprint=0000000000000000000000000000000000000000 fw_rule_reason=N/A ip_proto=UDP sec_zone_ingress=  dns_query= dest_ip= src_ip= ssl_server_name="" file_count=0 url_reputation="Risk unknown" event_desc="Flow Statistics" dest_bytes=0 connection_id=20829 ssl_policy_id=00000000000000000000000000000000 iface_egress= netbios_domain="" security_context=00000000000000000000000000000000 iface_ingress= src_tos=0 ssl_url_category=0 ssl_expected_action=Unknown dns_resp_id=0 dest_port=53 first_pkt_sec=1590133282 http_response=0 sensor= user_agent="" dns_ttl=0 event_type=1003 event_subtype=1 netflow_src=00000000-0000-0000-0000-000000000000 rec_type_desc="Connection Statistics" event_sec=1590133283 client_app="DNS client" src_pkts=1 last_pkt_sec=0 http_referrer="" has_ipv6=1 web_app=Unknown sec_zone_egress= url_category=Unknown ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ip_layer=0 dns_record_name=A ssl_flow_flags=0 rec_type_simple=RNA referenced_host="" dest_pkts=0 src_ip_country=unknown monitor_rule_7=N/A sec_intel_event=No app_proto=DNS ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL client_version="" instance_id=8 dns_record_desc="a host address" url="" fw_rule= dest_ip_country=unknown fw_rule_action=Allow sinkhole_uuid=00000000-0000-0000-0000-000000000000 rec_type=71 monitor_rule_7=N/A fw_rule_action=Allow src_tos=0 dns_resp_id=0 event_sec=1589960731 sec_intel_event=No first_pkt_sec=1589960731 ip_layer=0 sec_zone_ingress= ssl_url_category=0 ssl_policy_id=00000000000000000000000000000000 instance_id=7 fw_policy= connection_id=26787 src_bytes=152 ssl_ticket_id=0000000000000000000000000000000000000000 ssl_expected_action=Unknown iface_ingress= file_count=0 user_agent="" dest_bytes=0 client_version="" client_app="DNS client" dns_rec_id=1 url_category=Unknown ssl_cert_fingerprint=0000000000000000000000000000000000000000 dns_query= src_ip_country="united states" src_ip= app_proto=DNS ssl_server_name="" dns_ttl=0 ssl_flow_flags=0 src_port=50792 url_reputation="Risk unknown" dns_record_desc="a host address" event_usec=0 netflow_src=00000000-0000-0000-0000-000000000000 event_subtype=1 rec_type_simple=RNA http_response=0 sensor= web_app=Unknown ip_proto=UDP fw_rule== event_desc="Flow Statistics" security_context=00000000000000000000000000000000 ips_count=0 dest_ip_country="united states" src_pkts=1 rec_type_desc="Connection Statistics" dns_record_name=A dest_pkts=0 fw_rule_reason=N/A sec_zone_egress= netbios_domain="" last_pkt_sec=0 dest_port=53 dest_ip=209.244.0.3 event_type=1003 has_ipv6=1 url="" http_referrer="" referenced_host="" ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL iface_egress= rec_type=71 ssl_server_name="" sec_zone_ingress= event_subtype=1 dns_record_name=A src_ip_country= dest_ip_country= dns_record_desc="a host address" src_tos=0 dns_ttl=0 netflow_src=00000000-0000-0000-0000-000000000000 ssl_session_id=0000000000000000000000000000000000000000000000000000000000000000 iface_egress= ssl_cipher_suite=TLS_NULL_WITH_NULL_NULL dest_bytes=0 src_bytes=152 dns_resp_id=0 url="" monitor_rule_7=N/A src_port=50467 src_ip= sec_intel_event=No web_app=Unknown netbios_domain="" rec_type_desc="Connection Statistics" dest_ip= ssl_ticket_id=0000000000000000000000000000000000000000 ssl_url_category=0 fw_rule_reason=N/A event_sec=1590047131 url_category=Unknown has_ipv6=1 connection_id=35377 fw_policy= ssl_policy_id=00000000000000000000000000000000 referenced_host="" ssl_expected_action=Unknown sensor= event_type=1003 client_version="" first_pkt_sec=1590047130 user_agent="" last_pkt_sec=0 security_context=00000000000000000000000000000000 ips_count=0 ip_layer=0 dest_port=53 sec_zone_egress=S iface_ingress= fw_rule= file_count=0 dest_pkts=0 event_desc="Flow Statistics" app_proto=DNS event_usec=0 http_referrer="" fw_rule_action=Allow http_response=0 rec_type_simple=RNA src_pkts=1 ssl_cert_fingerprint=0000000000000000000000000000000000000000 instance_id=4 url_reputation="Risk unknown" ip_proto=UDP dns_query= ssl_flow_flags=0 client_app="DNS client" dns_rec_id=1 Thank you. ... View more

Hello @PavelP We created a regex that filters out DNS based on the order below (I'm talking about raw events): dns_record_name src_ip dest_ip dest_port rec_type=71\sssl_server_name=\S+\ssec_zone_ingress=\S+\sevent_subtype=1\sdns_record_name=\S+\s.+src_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?dest_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?(dest_port=53).+ After that the remaining DNS logs started to appear in order below, which we also covered with regex: src_ip dest_port dns_record_name dest_port dest_ip .src_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3})\s[\s\S]?(dns_record_name=\S+)\s[\s\S]?(dest_port=53)\s[\s\S]?dest_ip=(10.\d{1,3}.\d{1,3}.\d{1,3}|192.168.\d{1,3}.\d{1,3}|172.(1[6-9]|2[0-9]|3[0-1]).\d{1,3}.\d{1,3}).* Now the DNS rules appear in order below, which requires yet another regex ... dest_ip src_ip dest_port dns_record_name Each time we filter out traffic, new order logs appear for DNS... Regards, Dawid ... View more

Hello @PavelP We wanted to filter out logs that arrive from internal hosts to internal DNS servers. What we are doing atm is filtering using src_ip dns_record_name dest_ip to get requests done by internal hosts to internal servers. To do that we need a few regexes since there is multiple order, in which those fields appear in logs. it's true that the logs are parsed by Splunk, but when we want to parse the logs using regex they should be in the same order or we would have to implement multiple conditions that cause an error in Splunk. I have checked and unfortunately, the referenced_host filed is often empty. The question is: a) why the log order is different for some of the messages? I have not seen this issue in the case of other log sources. b) what would be the correct approach to tackle this? Thank you for your help. ... View more