Knowledge Management

App development best practice: new index, or not?

Jason
Motivator

When developing an App for SplunkBase for widespread use, is it a good practice to put all of my app's data in a new index which my app creates? Or let the inputs go whereever the user decides (likely to default)?

Some apps, like the unix app, create one, many others don't - just looking for some feedback.

Tags (3)
1 Solution

ziegfried
Influencer

IMO this is a matter of taste. I tend to not creating a dedicated index because the settings you choose (like max. total size, etc) may not fit for the user, that installs the app.

View solution in original post

mw
Splunk Employee
Splunk Employee

Just stumbled on this older post, but thought I'd add my 2 cents anyway. My opinion is that it's best for an app to use its own index. However, I think it's more important for searches to be constructed in such a way that it's easy for the end-user to control whether the app uses a custom index. I like having the app create an index because the end-user can choose not to see that data during regular searches by not making it one of their default indexes (of their role). Not to mention that they can clean the index safely, and define their aging and access policies properly. Certainly, they should ensure that the index settings are proper (I punt on this and leave everything empty).

When people create apps, they often don't leave room for the end-user to add an index if they want one. There are a lot of app searches out there like:

sourcetype=mytype src_ip=1.2.3.4 | top dest_ip

...or whatever, thrown around in savedsearches.conf or within views. As the end-user, if I want to have this app use another index, I now have to go and modify every single search to add index=my_index. It's easy for the end-user to set their preferred index settings, but it's less easy for them to modify every search in my app. Again though, I guess I'd say that the best practice is to make it easy to do either. I've started doing this:

# savedsearches.conf
[myapp]
search = index=my_index

[real_search]
search = savedsearch=myapp field=foo field=bar | ...

Now, if the user wants to change the index, they just override that single "myapp" stanza in whatever way they need to. In fact, they can easily send data to multiple indexes, and just override that stanza with "search = index=indexA OR index=indexB ...".

You could also stuff this into a macro -- I choose not to because the "savedsearch=..." syntax actually limits them to adding term/boolean search expressions, which is my way of forcing them to think harder about whether they really want to modify all searches as heavily as a macro or full-fledged savedsearch (i.e. | savedsearch myapp) would allow.

At the end of the day, adding an index adds more flexibility, while not adding one offers less.

dwaddle
SplunkTrust
SplunkTrust

mw -- at one of the 2011 Splunk.conf sessions, Will Hayes talked about this exact issue. The best-practice-advice given then was for apps to define a couple of macros, basically as follows:

[myapp_sourcetype]
definition= "sourcetype=mysourcetype"

[myapp_index]
definition= "index=myindex"

As long as your views and such all follow this advice, then it becomes easy for the consumer of your app to make single-place adjustments for index and sourcetype to match their environment.

tpsplunk
Communicator

i totally agree- it should be easy for the splunk admin to configure the app to use an existing or new indexes. for summary indexing in particular the data collection should add a tag to the data and the searches should reference that tag and not just the summary index name (if for example you decided to dump a bunch of different search data into the same summary index).

0 Karma

dwaddle
SplunkTrust
SplunkTrust

If an app is going to expect a certain index, then then a sensible-default indexes.conf needs to ship with the app. Some splunkbase apps have all of their searches/queries defined with "index=xyz" but the index is not defined with the app.

ziegfried
Influencer

IMO this is a matter of taste. I tend to not creating a dedicated index because the settings you choose (like max. total size, etc) may not fit for the user, that installs the app.

araitz
Splunk Employee
Splunk Employee

I agree. I like to let the admin choose whether or not the data requires a separate index.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...