All Apps and Add-ons

How to find interesting fields which are common in every app and TA i installed

Rocky31
Path Finder

We have around 10 apps installed but I want to find out the common interesting fields in it.

I went through the sourcetype's every app has different interesting fields. Is it I can search for any common and similar fields without spending too much time on each app.

Any suggestion is appreciated.

Tags (1)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

If I am understanding the question completely, I think the answer you seek is in CIM compliance.

If the apps are all CIM compliant, the fields will end up standardized. So, "source_port" and "src_port" and "SRCPT" and all the other possible ways an app's logging could specify src_port will end up being a field (or being aliased to a field) that's called src_port. Destination ports end up being dest_port. The user initiating an interaction will be known as "user" and so on.

It sounds like your apps aren't CIM compliant, though. When it's because Splunk's simple CIM doesn't have a "place" to put that data, well then shame on Splunk. But if there exists a location to put it (e.g. it fits into one of the many CIM categories linked in above) and the App's author didn't do that layer of compliance, then it's Shame on them.

Unfortunately, there's no easy way to "fix" this. Someone - you it seems - will have to go through one by one finding interesting fields and mapping them into CIM.

Or, you could just go through one by one and write down any interesting fields you've found - but you already did this and it's tedious, and you have to repeat it each time.

So better is to map fields to something like the CIM, then at least you only have to do it once.

I'm sorry this isn't a more fun or easier answer, but it is what it is. If you get stuck in particulars, or would like to provide a more concrete example that maybe we can help with, I'm sure you'll get someone to help.

Otherswise, I hope this helps!
Happy splunking!
-Rich

View solution in original post

0 Karma

Richfez
SplunkTrust
SplunkTrust

If I am understanding the question completely, I think the answer you seek is in CIM compliance.

If the apps are all CIM compliant, the fields will end up standardized. So, "source_port" and "src_port" and "SRCPT" and all the other possible ways an app's logging could specify src_port will end up being a field (or being aliased to a field) that's called src_port. Destination ports end up being dest_port. The user initiating an interaction will be known as "user" and so on.

It sounds like your apps aren't CIM compliant, though. When it's because Splunk's simple CIM doesn't have a "place" to put that data, well then shame on Splunk. But if there exists a location to put it (e.g. it fits into one of the many CIM categories linked in above) and the App's author didn't do that layer of compliance, then it's Shame on them.

Unfortunately, there's no easy way to "fix" this. Someone - you it seems - will have to go through one by one finding interesting fields and mapping them into CIM.

Or, you could just go through one by one and write down any interesting fields you've found - but you already did this and it's tedious, and you have to repeat it each time.

So better is to map fields to something like the CIM, then at least you only have to do it once.

I'm sorry this isn't a more fun or easier answer, but it is what it is. If you get stuck in particulars, or would like to provide a more concrete example that maybe we can help with, I'm sure you'll get someone to help.

Otherswise, I hope this helps!
Happy splunking!
-Rich

0 Karma

Richfez
SplunkTrust
SplunkTrust

Oh, I should have mentioned there ARE a few tools to help with this process a bit - probably the most interesting is the Splunk command fieldsummary which will give information about fields and using the Patterns tab (which uses cluster behind the scenes, if I remember rightly).

If you run that against each index/sourcetype, then compare the results, you might find some commonalities. Obviously you'll have to use some sense in deciding if it's really related and not just accidentally related, but it's at least pretty good at telling you when two fields aren't similar, in any case.

For instance, if I ran

index="fw"  
| fieldsummary
| fields - values

It would give me a list of the types of things found in my home firewall logs. I do a | fields - values only because for my purposes that's noise in these logs - you can try it both with that or without that and see which you find more useful. Anyway, from that list I can see that, for instance, SPT and DPT are both numbers, both range between 1 and about 65000, both appear in practically all records, and both have a distinct_count of 500 (or really close to each other, anyway). So, it's possible these two may cover the same sort of domain (Which they do, it's destination and source port for network traffic).

For the Patterns tab, it's less useful for comparing data, but not always useless - a couple of times it's found similar events in two data sources (though admittedly those data sources are usually very similar ones anyway). So you'd craft a search like index=A OR index=B and adjust your time frame to something where there's a few thousand events. Then click on the tab labeled 'Patterns' and give it a bit of time to load, then take a look at what it's telling you.

So, hopefully that helps a little bit, even though I know finding how data is related to one another is a tricky thing that requires a lot of thinking. For what it's worth, I don't really know of any system that finds proper interrelationships like this - even the best are just guessing based on information like what you might get from fieldsummary and require a lot of thinking and more study to determine if that's actually the case.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...