All Apps and Add-ons

indexes best practice?

mikefg
Communicator

Is there a best practice for specifying indexes for the Windows add-on? I am building a fresh install, so no legacy versions. Specify one index for all Windows events or multiple indexes like it was in the earlier version?

woodcock
Esteemed Legend

It is fully arbitrary. The 3 things to consider, in order of importance are:
1: Roles-based access (RBA): If the data must be made invisible to different groups of users, then the data MUST be in it's own index (or in an index that has the same access rules/roles).
2: Retention: The size and age of event data is enforced on an index-by-index basis.
3: Efficiency/Speed: The more isolated the data is, the fewer false positives and less data needs to be sorted through so the more efficient and faster searches will be.

0 Karma

mikefg
Communicator

What about working with the Splunk App for Windows Infrastructure? Doesn't it expect to find data in these indexes? Or is it based on Table A or custom indexes as discussed later?

The Windows event logs get indexed into the wineventlog index.
The performance monitor logs get indexed into the perfmon index.
The Active Directory data gets indexed into the msad index.

https://docs.splunk.com/Documentation/MSApp/1.5.2/MSInfra/WhatdatatheSplunkAppforWindowsInfrastructu...

Table A or custom indexes
https://docs.splunk.com/Documentation/MSApp/1.5.2/MSInfra/DownloadandconfiguretheSplunkAdd-onforWind...

0 Karma

woodcock
Esteemed Legend

I would do what makes sense FOR YOU based on the 3 priorities I laid out. If you sill have flexibility after having considered them, then by all means, accommodate the presuppositions of the apps that you intend to use. Even so, the index values that they use are easily changed and really should not be hard-coded anyway (many app developers are removing hard-coded index/sourcetype values and defaulting them inside of macros that you are supposed to update).

0 Karma

dstaulcu
Builder

Some of the things to consider in creation/configuration of indexes and routing of windows data among them include access control, retention, performance, and cost recovery objectives.

Indexes are a point of administration where you can separate search access based on roles. For example, you might route sensitive data such as windows security event logs into an index which is separate from non-sensitive windows logs.

You can also set retention and tiered storage strategies on an per-index basis.. Maybe you want to keep security related indexes searchable for years and maybe you want to keep performance/metrics logs searchable for months. Among those logs you want to keep searchable for a long time, maybe you want to keep more recent events in faster storage and older events on cheaper/slower storage.

Regarding performance, many of the events produced by the windows add-on are derived from performance counters. You might want to transform/reroute those outputs from a standard index to an index optimized for metrics to reduce search costs.

Last, if you provide Splunk as a service in your organization having multiple business units feeding you windows-based data, you might want to consider routing windows data into indexes that are specific to each business unit so that you are better positioned recover costs based on storage displacement and search history.

Definitely tracking this to see where the conversation goes.

0 Karma

mikefg
Communicator

Besides the considerations you listed (thank you) are there any for functionality? Or can I set indexes per stanza and the app will still work correctly?

0 Karma

dstaulcu
Builder

Knowledge objects (searches, dashboards, workflow actions, automated lookups, etc.) shipped in apps you would download from splunkbase should not reference indexes by name. Instead, best practice is to query events by source or perhaps sourcetype. Index names are yours to set as you see fit in your environment and splunkbase app developers should not make assumptions about what those index names will be.

Changing an index (into which perfmon-based inputs route) to a metrics data store would likely break things in the windows-add on unless you also customize the app to handle index-time transforms which fit the perfmon data into a metrics mold. You would also need to replace search commands in windows-add-on knowledge objects to metrics based search commands. I imagine you probably shouldn't go there just yet. 🙂

0 Karma

nareshinsvu
Builder

If your retention is too big (say 60days), then having multiple small indexes is better than having one large index.
Else, for smaller retentions (say 7 days), you can go for one index as per your requirements.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...