About JacobCarrell

JacobCarrell · ‎07-28-2020

I've got an issue where a significant portion of my ingested Log4Net_xml sourcetype logs have duplicate events. I'm currently using a file monitor on a Ubuntu 16.04 LTS, Splunk 7.3.0 machine with a local Monitor stanza to ingest the logs which looks like this: [monitor:///mnt/CALogs/.../*.log] disabled = 0 host_segment = 4 index = ca sourcetype = log4net_xml #initCrcLength = 1024 # Tried this, it started reindexing all the log file. Our Log4Net configs looks like this: <appender name="RollingFile" type="log4net.Appender.RollingFileAppender"> <file type="log4net.Util.PatternString" value="\\server1\ca\Logs\Application\%property{log4net:HostName}\Application.log" /> <appendToFile value="true" /> <rollingStyle value="Size" /> <maximumFileSize value="5MB" /> <staticLogFileName value="true" /> <maxSizeRollBackups value="10" /> <layout type="log4net.Layout.XMLLayout" /> </appender> I have that directory mounted over SMB/CIFS with the following entry in fstab: //server1/LogsCA /mnt/CALogs cifs username=user,password=Password,domain=somedomain.dev,sec=ntlm 0 0 Can anybody spot what I'm doing wrong? Here's a picture of the search that I'm using to estimate duplicates:

JacobCarrell · ‎10-29-2017

I'm building a Splunk App and I'd like my users to be able to point the import a single folder and have it accurately import each type of IRC log. However, Even with a well defined source type of: [ZNC] pulldown_type = true category = IRC LINE_BREAKER = \r{0,1}\n SHOULD_LINEMERGE = false MAX_TIMESTAMP_LOOKAHEAD = 15 EXTRACT-baseinfo = (?P<zncusername>\w+)[\\\/](?P<network>\w+)[\\\/](?P<channel>#.+)[\\\/](?:.+\.log) in source EXTRACT-IRCmessage = (?:\[.+\]) <(?P<channeluser>.+)> (?P<message>.+) EXTRACT-userjoinquitpart = (\*\*\*) (?P<event>.+): (?P<channeluser>.+) \((?P<ident>.+)@(?P<userhost>.+?)\)( \((?P<eventreason>.+?)\)){0,1} EXTRACT-topic = \*\*\* (?P<uername>.+? ).+? (?P<event>.+?) to (?P<topic>\'.+') EXTRACT-kicked = \*\*\* (?P<uername>.+?) was (?P<event>.+? )by (?P<eventintiatedby>.+?) \((?P<eventreason>.+)\) LOOKUP-IRC_actions = EventLookup action AS event OUTPUT IRCevent as event It will still randomly pick up partial dates as the source type. I've read about a field where you can specify source file regex for each sourcetype, but several of my planned source types have identical naming schemes. So, What can I do to give hints to the autotype that this is type A vs type B?

JacobCarrell · ‎10-25-2017

This works as is. Just have to make sure to place the file in the right spot, restart the UF and WAIT. It took a little while for events with the correct filter to come in. I'm guessing there was a queue somewhere between sending to Splunk and applying the props/transforms.

JacobCarrell · ‎10-25-2017

Thanks, I didn't know you could do this. It turns out I'm using an app that provides predefined sourcetypes. I ran out of oomph last night after rewriting the regex to grab the IP from the first IP and STILL getting the same results, so somewhere in one of my props.conf or transforms.conf is a bad regex. I'll have to track it down tonight. For the record, this is syslogs from PFSense.

JacobCarrell · ‎10-24-2017

Forgot to add, Here's what Splunk captures for the IP: Initiated with [AF_INET]66.103.224.123

JacobCarrell · ‎10-24-2017

I've got a regex that's working in Regex101's editor, but when I paste it into Splunk I get garbage or no results: Regex: ^(?:[^ \n]* ){5}\[(?P<username>\w+)(?:\].*\])(?P<IP>[^:]+) Sample entries: Oct 24 18:43:57 openvpn[36372]: 66.103.224.123:59349 [username] Peer Connection Initiated with [AF_INET]66.103.224.123:59349 Oct 24 18:28:54 openvpn[21337]: 66.103.224.123:50873 [username] Peer Connection Initiated with [AF_INET]66.103.224.123:50873 URL: https://regex101.com/r/by1mOW/7

JacobCarrell · ‎09-28-2017

I've found many entries on the subject of filtering IIS logs, with people saying X has worked. However, I'm not able to get it fully working. If I copy an IIS log that should be filtered to the server and import it manually it works (as far as I can tell, I only went to preview) but if I use a UF from a server 2003 (so older UF version) box, to the Splunk server on windows 2012 (6.6.3), it doesn't get filtered. Any help here? Props.conf: [iis] TRANSFORMS-ignoredpages= iis_ignoredpages Transforms.conf: [iis_ignoredpages] #SOURCE_KEY=field:cs_uri_stem REGEX=(Page1|Page2) DEST_KEY= queue FORMAT=nullQueue Page1 and Page2 are only part of the cs-uri-stem (that's its name in the IIS logs, but Splunk seems to turn it into cs_uri_stem), instead they're like companyname.product.page1/service.asmx or companyname.product/page2.asmx I've tried placing the props and transforms files on both the system/local directory of the UF and the Splunk receiver, restarted both and it continued to process the unwanted pages. I understand that it looks like UF itself can't filter these lines, but that it processes them sufficiently to get past props and transforms on the Splunk machine. I assume there's a way I can make Universal Fowarder send the logs RAW and the Spunk box will go "OH, W3C, process normally," but how do I do that? ---- Less relevant ---- Filtering out these pages is absolutely critical as they're hundreds of thousands of internal calls that would spam the Splunk logs, and overwhelm our 500mb/day limit that I need to stay under for proof of concept.

Posts	7
Solutions	1
Karma Given	1
Karma Received	1
Member Since	‎09-26-2017

Online Status	Offline
Date Last Visited	‎12-03-2020 01:41 PM

Duplicated Events: Local Log4net_xml Monitor

How to improve automatic sourcetype extraction

Splunk Regex - Works in regex101 but not Splunk

IIS filter transform not processing when forwarded...

Duplicated Events: Local Log4net_xml Monitor

How to improve automatic sourcetype extraction

Re: IIS filter transform not processing when forwa...

Re: Splunk Regex - Works in regex101 but not Splun...

Re: Splunk Regex - Works in regex101 but not Splun...

Splunk Regex - Works in regex101 but not Splunk

IIS filter transform not processing when forwarded...