Hey Gang,
Here are the basics: We are running Splunk Enterprise 6.5.1. I have a distributed architecture that has two separate search heads, 4 indexers with AutoLB (but no clustering) and a deployment server (all 6.5.1 running on RedHat).
Now for the actual question. We have 70 or more websphere servers that are all similarly named (i.e. prdwas01, prdwas02, prdwas03...tstwas01, tstwas02, tstwas03....stgwas01, stgwas02, stgwas03....etc.etc.etc.). I have a series of extracted fields that I pull from the "source" value (see below):
EXTRACT-sourcefields = (?<WAS_Cluster_All>(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?<WAS_JVM_name>[a-zA-Z0-9]++)_(?[a-zA-Z0-9]++) in source
Basically, I'm pulling out some cluster and JVM characteristics from the file path of the source. Now, I would like to apply this across all 70 websphere servers. As near as I can tell, you can only specify field extractions for host, source or sourcetype. Well, I want to be able to pull these values across 70 or more different servers without having to enter in over 70 separate stanzas. If I could apply it based on index I'd be fine, but that's not an option.
I have seen some articles on answers that reference using regex as part of the stanza title for these extractions in source and sourcetype, and I have attempted some of those, and not gotten them to work. Ideally, I would like to have a stanza that said something along the lines of:
[host::.{3}was\d\d]
That would represent any hostname that had 3 characters, then the letters 'was' and then two digits and have it then apply the included field extractions. I spoke with my sales engineer, and he claimed that it wasn't possible to use regex as part of the stanza header, and I wasn't able to get any of the examples from answers to work, so I decided to ask a question that specifically dealt with what I was trying to do.
Any thoughts or information would be very appreciated.
Thanks,
Matthew Granger
Hi Matthew,
I think you are asking for a way to include the hostname in the field names... I would think there's a better way to do that on the fly rather than trying to extract a tonne of fields with the hostname hard coded...that makes future homework as servers are added/decommissioned... who wants to miss monitoring on the new server because poor overworked Matthew forgot to add the new server to the list...
What about something like this:
start with this as a set of base data to illustrate the point:
| makeresults | eval data="1507063392.123,was01,17,18,19,20|1507063394.345,was01,21,22,23,24|1507063396.567,was02,25,26,27,28"
| makemv delim="|" data
| mvexpand data
| rex field=data "^(?<_time>[^\,]+)\,(?<host>[^\,]+)\,(?<datapoint1>\d+)\,(?<datapoint2>\d+)\,(?<datapoint3>\d+)\,(?<datapoint4>\d+)"
| table _time host datapoint*
so i have multiple data points for each host... like so:
_time host datapoint1 datapoint2 datapoint3 datapoint4
2017-10-03 16:43:12.123 was01 17 18 19 20
2017-10-03 16:43:14.345 was01 21 22 23 24
2017-10-03 16:43:16.567 was02 25 26 27 28
From your question, I read that you want to get each datapoint1 value to be hostname-datapoint1
So to make that happen, do this:
| makeresults | eval data="1507063392.123,was01,17,18,19,20|1507063394.345,was01,21,22,23,24|1507063396.567,was02,25,26,27,28"
| makemv delim="|" data
| mvexpand data
| rex field=data "^(?<_time>[^\,]+)\,(?<host>[^\,]+)\,(?<datapoint1>\d+)\,(?<datapoint2>\d+)\,(?<datapoint3>\d+)\,(?<datapoint4>\d+)"
| table _time host datapoint*
| eval measurename=host+"-datapoint1"
| fields + _time measurename datapoint1
| chart avg(datapoint1) by measurename
Which outputs...:
measurename AVG
---------------- ---
was01-datapoint1 19
was02-datapoint1 25
That's not at all what I'm trying to do. I have all the host names. The host names are not the issue. The issue is that I want to apply the SAME field extraction to multiple hosts without having to create multiple field extraction entries in props.conf. Right now, in props.conf I have to have the following stanzas:
[host::prdwas05]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source
[host::prdwas06]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source
The extractions work fine. I have no problem with the extractions at all. What I'm asking is, right now I have to have a separate stanza for EVERY websphere server, and I have over 70 websphere servers. So, instead of having one stanza that is [host::prdwas05] and ANOTHER stanza for [host::prdwas06] and ANOTHER and ANOTHER and ANOTHER.....(for up to 70 of them), I would like to have a single stanza that pattern matches the name of the server, and then applies the field extractions. Something like:
[host::.{3}was\d\d]
EXTRACT-sourcefields = (?(?<=logs\/)[a-zA-Z0-9.]++) in source
EXTRACT-sourcefileds = \/logs\/(?[a-zA-Z0-9]++)(?[a-zA-Z0-9]++) in source
However, that doesn't work.
Okay, HTML encoding screwed up my sample extractions (darn less than and greater than symbols), but the point stands, My extractions are working when I apply them specifically to one server. I'm trying to find a way to apply the same field extractions to Multiple servers based on a hostname pattern match.