Splunk Search

How do I find all unique IP addresses in a file?

bfaber
Communicator

If I have data that looks like

(date) srcip=x.x.x.x dstip=y.y.y.y

How can I create a single list of all unique IPs regardless of src/dst?

I imagine this is some sort of funky stats option...

Tags (4)
1 Solution

Lowell
Super Champion

You could use a search like this:

source=my_file | rex max_match=100 "\b(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b" | stats values(ip) as ip_list

That should make a multivalued field called ip and populates it with any IP-like values found in the event's raw text. Then the stats command will build a single list of unique values of your ip addresses.

Regex hint: Note that the regex "\b" is for boundary matching. It should match an "=" or a space before the IP address, and should also allow for a comma after the IP address; all of which may be common values before/after an ip address. Also, \b also matches the very beginning and very end of an event.


If you already have your ip address fields defined and you have different names for different sourcetype (which tends to happen), you can use the eval command to combine them. (You can also setup a field alias, but sometimes that may not always be preferable.) For example, say you had fields called dst, DST, dest, and dstip , you could pull them into a single field using a command like so:

| eval new_destip=coalesce(dst,DST,dest,dstip)

So if you want to look at both a source ip address and a dest ip address and then combine them, you could use the same approach for both fields, then use some ugly tricks to convert that into a single multi-value field, and then you can use the stats command to get your list of unique IPs....

| eval d=coalesce(dst,DST,dest,dstip,"") | eval s=coalesce(src,SRC,srcip,"") | eval ips=s.";".d | eval ips=split(ips, ";") | stats values(ips) as ip

Note: the eval split() function is new in Splunk 4.1.

View solution in original post

Simeon
Splunk Employee
Splunk Employee

In theory, Splunk should have automatically extracted the srcip and dstip as fields. The basic commands to get a list of unique values is to use the chart and dedup command. However, you want to list those individual fields as the same field which could require some eval and case statements. For just a single field, you could probably do this:

source=/your/log/file.txt | dedup srcip | chart count by srcip
0 Karma

Lowell
Super Champion

Hmm. I think you either want just dedup to get a single list of values. Or use the chart command. Using both, as shown, the count will always be 1 for each value of scrip

0 Karma

erydberg
Splunk Employee
Splunk Employee

I think an easy way to do it is to do a field extraction of the ip addresses, and then do a

"... | dedup ip | fields ip | fields - _*"

to remove dupes and get only the ip address field.

0 Karma

Lowell
Super Champion

Keep in mind that using dedup will probably not work as you would expect when dealing with multi-valued fields. The stats command will multi-valued fields properly. So | stats values(ip) is probably preferable.

Lowell
Super Champion

You could use a search like this:

source=my_file | rex max_match=100 "\b(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b" | stats values(ip) as ip_list

That should make a multivalued field called ip and populates it with any IP-like values found in the event's raw text. Then the stats command will build a single list of unique values of your ip addresses.

Regex hint: Note that the regex "\b" is for boundary matching. It should match an "=" or a space before the IP address, and should also allow for a comma after the IP address; all of which may be common values before/after an ip address. Also, \b also matches the very beginning and very end of an event.


If you already have your ip address fields defined and you have different names for different sourcetype (which tends to happen), you can use the eval command to combine them. (You can also setup a field alias, but sometimes that may not always be preferable.) For example, say you had fields called dst, DST, dest, and dstip , you could pull them into a single field using a command like so:

| eval new_destip=coalesce(dst,DST,dest,dstip)

So if you want to look at both a source ip address and a dest ip address and then combine them, you could use the same approach for both fields, then use some ugly tricks to convert that into a single multi-value field, and then you can use the stats command to get your list of unique IPs....

| eval d=coalesce(dst,DST,dest,dstip,"") | eval s=coalesce(src,SRC,srcip,"") | eval ips=s.";".d | eval ips=split(ips, ";") | stats values(ips) as ip

Note: the eval split() function is new in Splunk 4.1.

gkanapathy
Splunk Employee
Splunk Employee

eval split is new in 4.1, but older versions can use makemv to do the same thing.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...