Getting Data In

Support Apache Tomcat Valves Extended Access Log

mldeschenes
Explorer

I can't seem to get Splunk to auto/detect our current Apache Tomcat 6.x or 7.x logs.
Please help and appreciate the support, I have tried all I can so far. New to Splunk and not yet SME with this tool ... 🙂

Log source/format (Apache Tomcat 6.x – org.apache.catalina.valves.ExtendedAccessLogValve)

<Valve className="org.apache.catalina.valves.ExtendedAccessLogValve" directory="E:\folder-Logs" pattern="date time c-ip s-ip cs-method cs-uri-stem cs-uri-query sc-status bytes time-taken cs(User-Agent)    cs(Cookie) cs(Referer) cs(HOST)" prefix="${tomcat.instance.name}-" resolveHosts="false" suffix=".log"/>

Sample scrubbed http access log:

#Fields: date time c-ip s-ip cs-method cs-uri-stem cs-uri-query sc-status bytes time-taken cs(User-Agent)   cs(Cookie) cs(Referer) cs(HOST)
#Version: 2.0
#Software: Apache Tomcat/6.0.32
2014-05-06 04:04:09 7x.2xx.3x.5x 10.5x.7x.6x POST /folder/ajax/get.action - 200 79782 0.890 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.54.16 (KHTML, like Gecko) Version/5.1.4 Safari/534.54.16'    'JSESSIONID=BXA; CookiesEnabled=1; Sx7xFE=1xxxx.2xxxx.0000;' 'hxxps://client1.domain.com/folder/do.action?content=mypage=1' 'client1.skillport.com'
Tags (2)
0 Karma

lguinn2
Legend

I don't know what you mean by "autodetect", but this is the inputs.conf you probably need

[monitor://E:\folder-Logs]
sourcetype=access_combined_extended

For props.conf on the indexer, I would use

[access_combined_extended]
REPORT-ace=access_combined_base_fields
EXTRACT-aceExt1=\'(?<cs_User_Agent>.*?)\'.*?\'(?<cs_Cookie>.*?)\'.*?\'(?<cs_Referer>.*?)\'.*?\'(?<cs_Host>.*?)\'.
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 30

And for transforms.conf on the indexer

[access_combined_base_fields]
DELIMS = " "
FIELDS = date, time, c_ip, s_ip, cs_method, cs_uri_stem, cs_uri_query, sc_status, bytes, time_taken

Note: there shouldn't be any linebreak on the EXTRACT line above. Or the FIELDS line.

I just made up the sourcetype called access_combined_extended, because your data doesn't exactly match the common Apache formats I see. And I also set a few attributes in props.conf that you don't strictly need, but specifying them will help Splunk parse your data more efficiently.

0 Karma

lguinn2
Legend

create each of the files named above in

$SPLUNK_HOME/etc/system/local

Probably only the inputs.conf file will already exist. But for any file that already exists, simply copy and paste the above at the end of the file.

After copying the files, then restart Splunk.

You should probably walk through the Splunk Tutorial at
http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial

0 Karma

mldeschenes
Explorer

Sorry I can't seem to figure this out, please provide me exact files/path if all possible. I have fresh 6.1 install, don't care of any existing data as we are running poc/pilot.

0 Karma

mldeschenes
Explorer

Appreciate the support, I am rather new to Splunk. Will give this a shot, is it possible to send me the files and I can simply copy/past? I'm assuming I simply need to modify existing files and add the info you provided?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...