Compare field in search

[nidet]

I have a folder which stores .txt files. I need to compare the data that is inside the files. Not only accumulate the data for each search

1234.txt
4563.txt
8523.txt

The 3 files have similar data but at the same time new data.

What code can be used to compare 3 files?

Example

Three different .txt files are generated during the month, all month.
What we want is to collect data compare equal as a single vulnerability if they have the same IP and same vulnerability.

Do not take that as a cumulative total would be 6, but 3 are repeated with the same data only date difference.
The process would stay on 3 vulnerabilities and compared in total

All files is in the same folder in splunk.

2014-01-04.txt

response_datetime="2014-01-04 20:46:15" scan_date="2014-01-04 20:46:15" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

2014-01-25.txt

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=45863214 ip=65.196.59.130 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.3
Target Hostname: 192.168.1.3

response_datetime="2014-01-25 20:45:33" scan_date="2014-01-25 20:45:33" org_id=25336984 scan_id=78963512 host_id=4D2F5C05 ip=192.168.1.4 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.4

2014-01-18.txt

response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.2 testid=9162 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name="Scan Information" port=0 protocol=tcp results="Scanner IP: 10.250.83.152
Target IP: 192.168.1.2
Target Hostname: 192.168.1.2

response_datetime="2014-01-18 20:45:14" scan_date="2014-01-18 20:45:14" org_id=25336984 scan_id=78963512 host_id=45863214 ip=192.168.1.3 testid=772 vuln_type="Preliminary Analysis" vuln_risk=0 vuln_name=Services port=23 protocol=tcp results="A telnet server seems to be running on this port
Target IP: 192.168.1.3

[somesoni2 · Accepted solution]

The simplest way would be to just do a dedup on the keys/fields which uniquely identify a vulnerability - host/ip combination. An example search would be like this

your base search from all there files | dedup host ip vuln_type vuln_name results 

This will remove all the duplicates/repeating vulnerability from all there files and give only the unique ones. After that you can perform your post process.

[nidet]

This is a type of code that we use to analyze the .txt files
As the commentary would add this code to compare files that are inside the month.

sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low | replace 78963512 with "Prueba" in scan_id | rename scan_id as "Scan Name" | fields "Scan Name", High, Medium, Low | addtotals

[vasanthmss]

Try this,

 index=<your index> sourcetype=<sourcetype> | stats count as vulnerabilities by ip,vuln_type,source | sort -count | table vuln_type,vulnerabilities

[nidet]

Hi vasanthmss

Do you know?
As I can use the code that you registered with this code is the use for this part specifically vulnerabilities

sourcetype="---" org_id="25336984" NOT vuln_risk=0 | chart count over scan_id by vuln_risk | rename 8 as High, 4 as Medium, 1 as Low

i need any suggestion.

[vasanthmss]

Hi Nidet,

Share some some sample logs / details which are available in the files, and kindly share what you want to compare exactly.

[nidet]

I shared information, about the question. Thanks for your support

[aljohnson_splun]

Have you gone through the Search Tutorial? I think you'll find your answer by the end. Its kind of hard to try an explain so much information on the answers site.