Hello All,
We have recently cleaned up our Splunk eco-system and developing process / procedures for our we manage it. I wanted to get some ideas of questions Splunk Admins ask teams during the data on-boarding phase. Questions that make sure teams know their data and will get value out of the tool as opposed to using Splunk as a dumping ground.
Here are some of the questions I use:
These are the only critical ones I have come up with so far. I would love to hear what you all ask!
Thanks.
One of the most important questions I've learned to ask users who are asking for data onboarding:
"Do you have a service architecture diagram that you can share with me?"
Knowing where the system(s) are in relation to the service, and how data is passed around, is more than half the battle in building a complete service/ops/security package.
Other questions I like for all data sources:
"Where are the logs located on your system?"
"What are the log files you want monitored, and what are their contents?"
Asking these questions is equal to having a sample of the log, because it tells you what the end user thinks is going on, and you can look at reality to see if they are right, close. or have no clue. This is important to know if you are going to be building reports and dashboards for them.
"Pretty" version of this document.
https://github.com/LTRand/Splunk-Management-Docs/blob/master/Central%20Log%20Request%20Form.docx
There is a wide range of states between a dumping ground and a perfect Splunk implementation and I think most of us live somewhere in between. The "common" challenge, in my mind, is to move from the dumping ground scenario to the perfect Splunk implementation.
Here is a sample form that I love to start with with. (Shout out to @LTRand for hooking me up with this list initially - see his github below for another take on an onboarding form)
This is not an all encompassing list, but should get you moving in the right direction.
Implementing this into a self serve form in a portal is a great idea as well! (ie. SNOW, Remedy, whatever you use for workorders)
Allowing the groups to complete the form when it suits them, should trigger a fulfillment request or ticket and a quick working session with the requestor once the admin has had a chance to review.
Having a getting started package that links to any internal chat rooms/community, docs and training you have, or to splunk fundamentals 1 free training, and most importantly, a use case showcase showing how teams are extracting value is a great idea as well!
The main thing is to stress that onboarding is the just the beginning of the process. Once the data is in, the real fun can start!
Name/Owner: __________________________________________________
Title/Role: __________________________________________________
Team: __________________________________________________
A data sample.
Description of the data:
Sourcetype suggestion: _________________________
How are events broken? ___ single-line ___ multi-line (events start with: _________________________)
Is there a date/timestamp? ___ yes ___ no ___ >1 (pick one: _________________________)
What time zone is in use? _________________________
What fields are interesting? ______________________________________________________________________
Uses for the data:
Searches
___ I want to search using keywords for troubleshooting
___ I want real-time searches
___ I want to compute statistics over the last _____ mins/hours/days/months
___ I want to know the top n of something over the last _____ mins/hours/days/months
___ I want to create and save my own searches
Reports
___ I want to create charts/tables/gauges over the last _____ mins/hours/days/months
___ I want real-time reports
___ Please give me a dashboard
___ I want to create and save my own reports
___ I am building reports over long periods of time and want data summarized
Alerts
___ I want Splunk to send me alerts via email every _____ mins/hours/days
Clues on data collection:
Where is it located? server(s) _________________________ path _________________________
How should it be collected? ___ Splunk Universal Forwarder ___ syslog ___ other: _______________
Hints on retention policy:
Keep it for this long: _____ days/months/years
Store this much of it: _____ MB/GB/TB
Who should have access to the data:
Team / LDAP Group: ____________________
Apply the Common Information Model:
Is there a TA available (look on Splunk Apps)?
Validate success of data on-boarding
@mmodestino - a fabulous document!!!