I create two virtual indexes within Hunk that reads from two separate HDFS directory. One is for Cisco ASA logs, and the other is for IIS logs. Each HDFS directory contains a bunch of *.log files. Clicking on 'search' for either index starts to index/read all of the log files, but the sourcetype is set wrong for both indexes.
How do I assign the correct sourcetype to each index?
Thx
Try this option - in props.conf use the source:: and HDFS location :
Go to /hunk/etc/apps/search/local -> Create props.conf
[source::/user/xyz/ciscologfiles/...]
sourcetype = cisco_syslog
[source::/user/xyz/iislogfiles/...]
sourcetype = iis
Thx for the reply and info.
Created a props.conf file, which now reads:
[source::hdfs://hostname:8020/logs/firewall] sourcetype = cisco_syslog
[source::hdfs://hostname:8020/logs/web] sourcetype = iis
Restarted splunk and when I search on IIS or the ASA logs, they're still not parsing correctly
You need to remove the "hdfs://hostname:8020" part from the stanza, ie try replacing them with the following verbatim (no need to restart)
[source::/logs/firewall/...]
sourcetype = cisco_syslog
[source::/logs/web/...]
sourcetype = iis
Thx. Now reads:
[source::logs/firewall/...]
sourcetype = cisco_syslog
[source::logs/web/...]
sourcetype = iis
Still not parsing (not seeing source/dst IPs, and so on)
What are the events being sourcetyped as by Hunk?
The IIS logs are being tagged correctly (sourcetype = iis), where as the Cisco ASA logs have no sourcetype associated with them at all.
If I go the 'Explore Data' route and select sourcetype = iis the preview data screen shows the logs being parsed correctly, but once I save and then search the fields aren't parsed like they should be.
When I do 'Explore Data' re: Cisco, I set the sourcetype to cisco:asa, but preview data screen doesn't show the ASA logs being parsed as they should. I can switch between System Defaults, Syslog, and cisco:asa and the files never change how they're parsed.
Looking at your stanzas again you seem to be missing the leading / in both of them - can you please add that?
As for the iis logs being sourcetyped correctly but not being parsed properly, it seems like the root cause here is that iis log parsing uses index time rules which are not usable in Hunk - here's how the iis sourcetype is defined. Can you post the first few lines (anonimized) of the iis log files?
[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
INDEXED_EXTRACTIONS = w3c
detect_trailing_nulls = auto
Fixed props.conf to:
[source::/logs/firewall/...]
sourcetype = cisco_syslog
[source::/logs/web/...]
sourcetype = iis
[source::/logs/web/ex140401.log]
sourcetype = iis
anonimized iis log files:
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/ImapRedirect.aspx - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /favicon.ico - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 404 0 2
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:02 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/dir_name/Default.aspx - 443 - 96.235.28.93 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:28.0)+Gecko/20100101+Firefox/28.0 302 0 0
2014-04-01 04:00:04 W3SVC1 127.0.0.1 POST /dir_name/AuthProviderSoapBinding.asmx - 80 - 127.0.0.1 Plumtree+OpenHTTP+Library+(version+2.0) 200 0 0
2014-04-01 04:00:04 W3SVC1 x.x.x.x POST /portal/server.pt - 443 - 24.3.69.132 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:27.0)+Gecko/20100101+Firefox/27.0 302 0 0
Can you also please include the header line that contains the list of fields? Once you have the fields you should be able to configure parsing using props/transforms.conf to extract the fields
props.conf
[source::/logs/web/ex140401.log]
sourcetype = new-iis
[new-iis]
REPORT-manual-iis = manual-iis
transforms.conf
[manual-iis]
FIELDS = <comma delimited list of fields from file header>
DELIMS = " "
header line:
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
Okay, so give this a try:
props.conf
[source::/logs/web/ex140401.log]
sourcetype = new-iis
[new-iis]
REPORT-manual-iis = manual-iis
transforms.conf
[manual-iis]
FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-user-agent, sc-status, sc-substatus, sc-win32-status
DELIMS = " "
# if the fields are tab delimited try the following
#DELIMS = "\t"
props.conf now reads:
[source::/logs/firewall/...]
sourcetype = cisco_syslog
[source::/logs/web/...]
sourcetype = iis
[new-iis]
REPORT-manual-iis = manual-iis
[source::/logs/web/ex140401.log]
sourcetype = iis
(pleas note - this is added to the file when I do 'Explore Data')
Do I create the transforms.conf file in /hunk/etc/apps/search/local?
Thx for all of your help Ledion
Yes, both (props/transforms) go in $SPLUNK_HOME/etc/apps/search/local/
And to be safe I restart Splunk after every change
props.conf:
[source::/logs/firewall/...]
sourcetype = cisco_syslog
[source::/logs/web/...]
sourcetype = iis
[new-iis]
REPORT-manual-iis = manual-iis
[source::/logs/web/ex140401.log]
sourcetype = iis
transforms.conf:
[manual-iis]
FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-user-agent, sc-status, sc-substatus, sc-win32-status
DELIMS = "\t"
(I tried with DELIMS = " " as well)
Still not parsing. Anything to be gleaned from the search log?
Please setup the props.conf correctly as below, to wire up transforms correctly:
[source::/logs/web/ex140401.log]
sourcetype = manual-iis
Ledion,
Hope you had a nice Thanksgiving holiday.
Set the props.conf as follows:
cat /opt/hunk/etc/apps/search/local/props.conf
[source::/logs/firewall/...]
sourcetype = cisco_syslog
[source::/logs/web/...]
sourcetype = iis
[new-iis]
REPORT-manual-iis = manual-iis
[source::/logs/web/ex140401.log]
sourcetype = manual-iis
(note - I see two props.conf files being modified. One is /opt/hunk/etc/apps/search/local/props.conf - and the other is /opt/hunk/etc/users/admin/search/local/props.conf)
Restarted Splunk and searched on web logs in Virtual Indexes and the events still aren't being parsed correctly.
Thx
You have another mistake in your props.conf - sourcetype=new-iis (not sourcetype=manual-iis) - can you please just copy and paste the below and try running this search: index=vix source=/logs/web/ex140401.log? What is the sourcetype of the events that come back?
/opt/hunk/etc/apps/search/local/props.conf
[source::/logs/web/ex140401.log]
sourcetype = new-iis
priority = 10
[new-iis]
REPORT-manual-iis = manual-iis
/opt/hunk/etc/apps/search/local/transforms.conf
[manual-iis]
FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-user-agent, sc-status, sc-substatus, sc-win32-status
DELIMS = " "
# if the fields are tab delimited try the following
#DELIMS = "\t"
Ledion,
Was wondering if you might have time to take a look at a different issue re: key/value pairs?
I download the IAS app and got it to work, and it really saved a ton of time for parsing the Microsoft NPS Radius logs. However, I did notice there are some errors and/or missing key/value pairs. For example, in the transforms.conf file I see the following stanza:
[ias-attr-45]
SOURCE_KEY = ias_message
REGEX = ^(([^,]+),){6}(([^,]+),([^,]+),)*(45),([^,]+)
FORMAT = acct_authentic::$7
I found a parsing of the IAS/NPS logs and for the acct_authentic key/value, I see the following:
'enum' => { '0' => 'None', '1' => 'RADIUS', '2' => 'Local', '3' => 'Remote' },
'name' => 'Acct-Authentic',
How do I modify the transforms.conf stanza to include the values (0 = 'None', 1 = 'Radius', 2 = 'Local', 3 = 'Remote')?
Link is http://answers.splunk.com/answers/217719/field-extractions-for-microsoft-ias-how-to-modify.html
Thx