Using this which ignores header (and also retrieves field names from header): http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime
But how to ignore footer from forwarding, now it is ending up with this error as footers doesn't have date/time.
07-08-2014 16:35:11.591 +0000 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jul 5 15:54:07 2014). Context: source::/usr/PATHOFFILE/filename_c105.log.20140704-163507-00|host::105|file_type_access|50
I'd like both header and footer to be removed, and have the following in props.conf
[sourcetype_access]
HEADER_FIELD_LINE_NUMBER = 2
FIELD_HEADER_REGEX = ^#Fields:\s(.*)
FIELD_DELIMITER = \t
TIMESTAMP_FIELDS = date, time
PREAMBLE_REGEX = #.*
TRANSFORMS-to_trash1 = remove_comments
in transforms.conf
[remove_comments]
DEST_KEY = queue
REGEX = ^(?:#)
FORMAT = nullQueue
It removes only the header but not footer. I tried using REGEX instead of PREAMBLE_REGEX but that doesn't help. What should be done to remove both header and footer?
Lets say the footer looks like this:
# Hey, Im a footer #
Use this in your props.conf:
TRANSFORMS-null = sourcetype_NullQueue
Use this in your transforms.conf:
[sourcetype_NullQueue]
REGEX=^#\sHey\,Im.a.footer.#$
DEST_KEY = queue
FORMAT = nullQueue
Then restart and test.
Yes, please give examples of your header and footer.
We don't know what your footer looks like, so it's hard to say. What jkat54 has said is how I get rid of headers/footers.
Please refer my comment, I'd like both header and footer to be removed, and use the fields from header.