Not sure if this is possible on a single server instance of a Splunk setup but I have all my ESXi logs forwarding to my Splunk server over TCP:1514. I did some digging and found references to the props.conf file and adding a regex filter there. So I did some digging and found multiple copies of this config file but I think (and tell me if I am wrong here) that I need to modify the copy found under:
\etc\system\local
If that is the case I just need some guidance no how to filter out everything but logs that contain the string "dfwpktlogs"
I am trying to filter out the rest of the logs as ESXi is very chatty and it eats into the license and I have to set the index it feeds into to only keep logs for a month because it just fills up so fast.
hey
So you want to keep specific event and discard the rest.
follow this steps to do that : same is written in the doc as well
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_...
step 1: Edit props.conf and add the following:you will do this on /local/props.conf of the same path i.e. /opt/splunk/etc/app/<appname>/local
[<specify_sourcetype_name>]
TRANSFORMS-set= setnull,setparsing
step 2: Edit transforms.conf and add the following:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = dfwpktlogs
DEST_KEY = queue
FORMAT = indexQueue
step 3: Restart Splunk Enterprise.
Also, now you want to set the retention period of 1 month i.e. 30 days
so find that index in mostly in /opt/splunk/etc/<appname>/default/indexes.conf
and copy the stanza in local/indexes.conf
and add this attribute to that stanza
frozenTimePeriodInSecs = 2592000
Find more information in indexes.conf and props.conf
Let me know if it helps !
hey
So you want to keep specific event and discard the rest.
follow this steps to do that : same is written in the doc as well
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_...
step 1: Edit props.conf and add the following:you will do this on /local/props.conf of the same path i.e. /opt/splunk/etc/app/<appname>/local
[<specify_sourcetype_name>]
TRANSFORMS-set= setnull,setparsing
step 2: Edit transforms.conf and add the following:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = dfwpktlogs
DEST_KEY = queue
FORMAT = indexQueue
step 3: Restart Splunk Enterprise.
Also, now you want to set the retention period of 1 month i.e. 30 days
so find that index in mostly in /opt/splunk/etc/<appname>/default/indexes.conf
and copy the stanza in local/indexes.conf
and add this attribute to that stanza
frozenTimePeriodInSecs = 2592000
Find more information in indexes.conf and props.conf
Let me know if it helps !
Okay just testing the suggested settings and just made sure to put the new settings at the top of the props.config file and did a restart and it worked!
Thank you for your help!
@mayurr98, thank you for the detailed post. I am still a Splunk novice so I just need to clarify a couple things before I make the change in production.
The proper path for the props.conf file if I am not using a specific app for our ESXi logs will be
/opt/splunk/etc/app/search/local
?
Also according to the link, you provided it mentions putting the setting you provided at the top of the prop.config file. Just wanted to verify the location where I put the settings in the file matters.
yes you are right./opt/splunk/etc/system/local/ defines the global path. you should make changes to /app//local is a best practice.