Knowledge Management

What is your best practice for search slicing by host environment/role?

Skorfulose
Explorer

Hey there!

I did not find an optimal solution for myself yet. But I guess many of you have similar use cases, so maybe you can explain how you handle such situations:

Let's say you have an application consisting of:

  • load balancers
  • web servers
  • application servers
  • database servers

And you have separate environments for:

  • production
  • staging
  • development

So you probably want to slice your searches different ways:
Either by server role: Show me any errors on web servers (no matter whether prod, staging or dev)
Or by server environment: Show me any errors in production environment (for all: lb, web, app, db)

So I started tagging my hosts like this:

[host=server1]
production = enabled
webserver = enabled

[host=server2]
production = enabled
database = enabled

[host=server3]
staging = enabled
webserver = enabled
...

Now I can quickly perform normal searches like index=perfmon tag=webserver or index=perfmon tag=production.

I know one can also use eventtypes, macros or lookups to achieve almost the same. The reason I prefer tags over the others is I can assign multiple attributes to one host easily. Doing the same with eventtypes or macros gets tedious quickly. If you add a webserver to production, you have to change two macros. And if you have more attributes (pci etc), it doesn't scale well.

So here is my current problem:
I have built a KV Store lookup storing the currently deployed application builds by host: _key=host and value=buildnumber.
If I e.g. want to query the buildnumber for a specific environment I cannot use my tags:

| inputlookup my_buildnumbers where tag=webserver would be great because the filtering would be done directly in KV Store / MongoDB. But obviously this doesn't work because Mongo doesn't know my tags.

| inputlookup my_buildnumbers | eval host=_key | search tag=webserver unfortunately doesn't work either. I guess because tags aren't added for inputlookups. Only for real events.

| inputlookup my_buildnumbers | search `webservers` with the macro expanding to (host=server1 OR host=server2) would probably work. But as mentioned I dislike the macro approach for the reason above.

Does anybody have a good idea? How do you handle such things?

Cheers!
Thomas

0 Karma
1 Solution

sundareshr
Legend

How about a sub-search

| inputlookup my_buildnumbers | search [search tag=webservers | stats count by host | table host]

View solution in original post

0 Karma

sundareshr
Legend

How about a sub-search

| inputlookup my_buildnumbers | search [search tag=webservers | stats count by host | table host]
0 Karma

Skorfulose
Explorer

Hello sundareshr,

thanks, this works. But depending on the search time range the subsearch can be quite slow. I think I need to find some optimizations for it.

Regards,
Thomas

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...