Splunk Enterprise Security: New Domain Analysis won't populate with Web data from Stream

[jhigginsmq]

We are in the process of configuring Enterprise Security on our system. We don't have a lot of data sources so the only data we have that populates the "Web" data model is the stream:http source from the Splunk Stream app. This looks to be sufficient for most Web-related dashboards however the "New Domain Analysis" under "Web Intelligence" fails to populate at all. I've configured everything required to use the "whois_system" modular input for this dashboard, in the absence of a domaintools API subscription, however this has had no effect.

I've noticed that the searches tied to this dashboard appear to assume the data will have a full domain name for the Web.dest field, and that from looking at a demo-data sandbox version of ES this appears to be the case for the non-stream sources. However the stream:http source has the destination ip address for the Web.dest field. I think this is the problem, but even if not; is this perhaps a deviation from CIM-compliance that should be fixed in the stream app?

[vshcherbakov_sp · Accepted solution]

I'm not a CIM expert, but it seems like CIM doc defines Web.dest field as

The destination of the network traffic (the remote host). You can alias this from more specific fields, such as dest_host, dest_ip, or dest_name.

This IMO allows mapping dest_ip as dest (which is what Stream TA does), although to your point for http traffic the FQDN seems like a better alternative. I believe you can easily tune it by dropping a couple of props/transforms.conf stanzas that alias stream's http site field as dest on your SH.

[jhigginsmq]

Ok thanks for that, yes I see what you mean: strictly speaking the Stream TA does what it is required to do. It still seems to be a bit of a disconnect from what ES expects, in this particular case. I've put stanzas in props/transforms.conf in the Stream TAs local directory to alias the 'site' field as 'dest' for the http sourcetype, and the domain analysis dashboard is now partially populating.

I guess that answers the question I was asking, but just to expand a bit; I'm now getting data for a domain type of "newly seen", but not "newly registered". My understanding is that this relies on the 'whois_domaintools' or 'whois_system' modular input, and I'm still struggling to see why 'whois_system' doesn't seem to be working - specifically the 'whois' index isn't populating despite having files in the /splunk/var/lib/splunk/modinputs/whois directory.

[vshcherbakov_sp]

I agree that Stream's current behavior is suboptimal; I've created a ticket to change it as you've proposed, so hopefully it'll get fixed in one of the future stream releases. Re: whois_* stuff - I'm not familiar with that part of the product, so unfortunately I have no suggestions here..

[jhigginsmq]

Great thanks. Yeah no problem re: whois, I'll ask another question on here if I'm still having problems when I get back round to it.