I have data coming into SPLUNK [service] , but i only need the file name not the data in the file.
The data is getting in, but i need to reduce it.
So i am trying to reduce the data with REGEX before it hits the INDEX. For example i think i would have to take one character from each file so it will register the file and i can use the file name. I have the below REGEX but its not working. Any ideas?
transforms.conf
[NoInfo_100]
REGEX = .$
DEST_KEY = queue
FORMAT = nullQueue
props.conf
[service]
TRANSFORMS-filter = NoInfo_100
Thanks in Advance
Robert Lynch
Your configuration above routes lines to nullQueue
, and therefore would skip indexing entirely. That's not what you described you're looking for. The below should rewrite the log lines to just the first character of the line.
[NoInfo_100]
REGEX = (.)
DEST_KEY = _raw
FORMAT = $1
If there is a header, or any other record that will ALWAYS be there in small but nonzero numbers, then use @mayurr98's solution to route everything but the header to the nullqueue.
If there is no qualifying type of records, then perhaps one character per record might be the best you can do.
This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue
Here is a basic example that will drop everything except events that contain the string login
props.conf
[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set = setnull, setparsing
In transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue
Let me know if this helps!
Your configuration above routes lines to nullQueue
, and therefore would skip indexing entirely. That's not what you described you're looking for. The below should rewrite the log lines to just the first character of the line.
[NoInfo_100]
REGEX = (.)
DEST_KEY = _raw
FORMAT = $1
This is really great guys, thanks.
Do you think it is possible to take in only one character per file, not per line?
In the perfect world i just want to look at the data in the filename, so the data inside the file is not usefull.
Got this by adding
[service]
BREAK_ONLY_BEFORE=ererererererer
TRANSFORMS-filter = NoInfo_100
Would it be easier just to run a script that prints out the file names in this directory?
Your current regex would remove the last character of each line in the file.
I really think you should use a script to list the names instead of using the indexing pipeline to transform the data.