I have more than 20 alerts about network security. Such as: Port_Scan、Web_Attack、Host_Attack。
The number of fields and field names for these alarm outputs are different. However, there is a field that contains the attacker IP.
Although their field names are different
Of course, these events from the index is not the same
for example:
Port_Scan The attacker's IP field is 'scan_sip' (port_scan event come from the index 'firewall_logs')
Web_Attack The attacker's IP field is 'web_attack_sip' (Web_Attack event come from the index 'apache_access')
Host_Attack The attacker's IP field is 'host_attack_sip' (Host_Attack event come from the index 'secure'
ok. So I have the following two questions
1、If these alert are triggered and output results, that is to say, the results are not empty.How to bring together the alert results togethe? That is, the results are aggregated in an index.
Use summary index? However, there are many problems with the summary index.
a、summary index is triggered in real-time, Regardless of whether the result is empty, that will be displayed in the Triggered Alert
b、Because in the cluster,If the alert is triggered and the result is not empty, there are many repeated results in summary index.
for example: The ‘’port_scan‘’ alert has been triggered.I try to search:
index = summary source = port_scan
There are five identical results. There will be a lot of duplicate data.
2、How to bring all the attacker's IP in a field (for example, the field: hack_ip).
for example:
sorry .please forgive my English
First, ensure you are putting a _time
or alert_time
into the summary index, for use in analysis.
Also, preferably, you should create a specific summary index for this particular kind of data, rather than using the default one.
Second, for your own sanity, please resolve to regularize your naming conventions (and other formatting). Having a different name in each alert for the same conceptual data field is going to cause your code to become ridiculous over time.
Third, look into field aliasing. If you alias the various attacker IP fields in the summary index, then you won't have to deal with individually coding them in your searches. See reference - http://docs.splunk.com/Documentation/Splunk/6.6.2/Knowledge/Addaliasestofields
Fourth, for now, you can use coalesce (either coalesce
or mvappend
) to get the variously named fields into a single field. Use mvappend
only if there may be more than one of the fields and you want to retain the multiple values. That looks like either...
| eval hacker_ip = coalesce(scan_sip, web_attack_sip, host_attack_sip,....)
...or...
| eval hacker_ip = mvappend(scan_sip, web_attack_sip, host_attack_sip,....)
Fifth, assuming you've already tightened your timeframe to include only relevant events, then you can get rid of the _time field and use uniq
to kill any complete dups.
| fields - _*
| uniq
Alternately, you could use a separate dedup
for each type of record, with keepempty=t
so that only that type of record will be deduped by those parameters.
| dedup scan_sip scan_dip scan_dport scan_type status keepempty=t
| dedup web_attack_sip server attack_type request_link keepempty=t
| dedup host_attack_sip server attack_type keepempty=t
| dedup ...
Sixth, you will then feed your hacker_ip
into iplocation
to determine Country
etc. See reference - https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/SearchReference/Iplocation
Finally, a starting place for your aggregation report...
earliest=foo index=summary (source=port_scan OR source=web_attack OR source=host_attack ...)
| dedup scan_sip scan_dip scan_dport scan_type status keepempty=t
| dedup web_attack_sip server attack_type request_link keepempty=t
| dedup host_attack_sip server attack_type keepempty=t
| dedup ...
| eval hacker_ip = coalesce(scan_sip, web_attack_sip, host_attack_sip,....)
| eval unit=1
| stats sum(unit) as hack_count by hacker_ip
| iplocation hacker_ip
| table hacker_ip Country City hack_count
| rename hack_count as "Count of Hack Attempts"
First, ensure you are putting a _time
or alert_time
into the summary index, for use in analysis.
Also, preferably, you should create a specific summary index for this particular kind of data, rather than using the default one.
Second, for your own sanity, please resolve to regularize your naming conventions (and other formatting). Having a different name in each alert for the same conceptual data field is going to cause your code to become ridiculous over time.
Third, look into field aliasing. If you alias the various attacker IP fields in the summary index, then you won't have to deal with individually coding them in your searches. See reference - http://docs.splunk.com/Documentation/Splunk/6.6.2/Knowledge/Addaliasestofields
Fourth, for now, you can use coalesce (either coalesce
or mvappend
) to get the variously named fields into a single field. Use mvappend
only if there may be more than one of the fields and you want to retain the multiple values. That looks like either...
| eval hacker_ip = coalesce(scan_sip, web_attack_sip, host_attack_sip,....)
...or...
| eval hacker_ip = mvappend(scan_sip, web_attack_sip, host_attack_sip,....)
Fifth, assuming you've already tightened your timeframe to include only relevant events, then you can get rid of the _time field and use uniq
to kill any complete dups.
| fields - _*
| uniq
Alternately, you could use a separate dedup
for each type of record, with keepempty=t
so that only that type of record will be deduped by those parameters.
| dedup scan_sip scan_dip scan_dport scan_type status keepempty=t
| dedup web_attack_sip server attack_type request_link keepempty=t
| dedup host_attack_sip server attack_type keepempty=t
| dedup ...
Sixth, you will then feed your hacker_ip
into iplocation
to determine Country
etc. See reference - https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/SearchReference/Iplocation
Finally, a starting place for your aggregation report...
earliest=foo index=summary (source=port_scan OR source=web_attack OR source=host_attack ...)
| dedup scan_sip scan_dip scan_dport scan_type status keepempty=t
| dedup web_attack_sip server attack_type request_link keepempty=t
| dedup host_attack_sip server attack_type keepempty=t
| dedup ...
| eval hacker_ip = coalesce(scan_sip, web_attack_sip, host_attack_sip,....)
| eval unit=1
| stats sum(unit) as hack_count by hacker_ip
| iplocation hacker_ip
| table hacker_ip Country City hack_count
| rename hack_count as "Count of Hack Attempts"
hi DalJeanis~Thank you for your help. But I have a problem that is not clear enough.
When an alert(for example: port_scan) is triggered, I search "index=summary source=port_scan ", There are 5 identical that alert in summay index. why? The splunk_server of these 5 events is one of the index servers in the indexer cluster.Host is the search head for the current search execution. This is also a headache for me to solve
In addition, since the summary index is real-time search and has too many summary indexes, it will prompt a message: Splunk tells me that the number of alerts currently triggered is too much. It's against me. Message probably said so
@xsstest - see the other question for information about multiple duplicate results in this kind of search.
https://answers.splunk.com/answers/558452/too-many-search-jobs-found-in-the-dispatch-directo-3.html
@DalJeanis
...alert search... | rename scan_sip as hacker_ip
Web_Attack: ...alert search... | rename web_attack_sip as hacker_ip
Host_Attack: ...alert search... | rename host_attack_sip as hacker_ip