Getting Data In

breaking multiline forefront events

justinhart
Path Finder

Hello,

I'm trying to break logs collected from Microsoft Forefront Client Security into separate events. Here is a sample of the logs:

--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 08:57:48
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
************************************************************
Microsoft Forefront Client Security Log, (c) 2006
Stopped On Fri Feb 10 2012 09:20:43 (Exit Code = 0x0)
************************************************************
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 09:20:45
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
************************************************************

Begin Resource Scan
Scan ID:{4D865233-FA4A-42C5-87A3-962CC4E6B95A}
Scan Source:4
Start Time:Mon May 09 2011 15:10:29
End Time:Mon May 09 2011 15:10:30
Explicit resource to scan
Resource Schema:taskscheduler
Resource Path:@S-1-5-21-3171008878-1269407668-386151811-1006\C:\WINDOWS\tasks\MP Scheduled Quick Scan.job
Result Count:1
Known File
Number of Resources:1
Resource Schema:file
Resource Path:c:\Program Files\Microsoft Forefront\Client Security\Client\Antimalware\MpCmdRun.exe
Extended Info:481036337152
End Scan
************************************************************

I'd like for the data in between the asterisks to be one event.

I've tried several versions of the props.conf stanza with nothing working all of the time. What is the best and most efficient way to do this. Thanks.

0 Karma
1 Solution

justinhart
Path Finder

Just an update on what actually solved this problem. I wasn't real clear in my original question, leaving out the fact that the production data, that I was having problems with, was coming from a forwarder. This is why the changes made to the props.conf on the indexer did not fix the production data. It did fix the test data which was not coming from a forwarder.

I applied the props stanza below on each of the forwarders and my issue is resolved. Thanks for everybody's help.

[ms_forefront]
CHARSET = UTF-16LE
LINE_BREAKER = ([\r\n]+)(\*{60})
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = 1
pulldown_type = 1

View solution in original post

0 Karma

justinhart
Path Finder

Just an update on what actually solved this problem. I wasn't real clear in my original question, leaving out the fact that the production data, that I was having problems with, was coming from a forwarder. This is why the changes made to the props.conf on the indexer did not fix the production data. It did fix the test data which was not coming from a forwarder.

I applied the props stanza below on each of the forwarders and my issue is resolved. Thanks for everybody's help.

[ms_forefront]
CHARSET = UTF-16LE
LINE_BREAKER = ([\r\n]+)(\*{60})
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = 1
pulldown_type = 1
0 Karma

justinhart
Path Finder

What I did, after using your props.conf stanza, added CHARSET=UTF-16LE to the stanza and removed my SEDCMD entry. This pulled it in without the \x00 "null" characters and for the most part broke the events as they are supposed to be. There are just a few that didn't break. Now, I think, it's just a matter of fine-tuning the line break commands. Thanks for your help.

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

How are you getting these logs to Splunk? Is it with a forwarder or with FTP?

0 Karma

justinhart
Path Finder

Actually, just doing a monitor of a UNC path where the files are located.

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Not sure, take it out and try since that is an at index time command.

0 Karma

justinhart
Path Finder

It does have something to do with it. I created a second log file and copied/pasted log data into it and it broke the lines fine.

If I remove the sedcmd then i get an \x00 in between each character of my log. I guess I need to look and see exactly what the sedcmd is doing.

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Mine are line breaking correctly. Are you sure you deleted them and reindexed? Did you add a crcSalt=<SOURCE> to inputs.conf?

1   »  2/10/12
9:29:47.000 AM  
************************************************************
2   »  2/10/12
9:29:47.000 AM  
************************************************************
Begin Resource Scan
Scan ID:{4D865233-FA4A-42C5-87A3-962CC4E6B95A}
Scan Source:4
Start Time:Mon May 09 2011 15:10:29
End Time:Mon May 09 2011 15:10:30
Explicit resource to scan
Resource Schema:taskscheduler
Resource Path:@S-1-5-21-3171008878-1269407668-386151811-1006\C:\WINDOWS\tasks\MP Scheduled Quick Scan.job
Result Count:1
Show all 16 lines
3   »  2/10/12
9:29:47.000 AM  
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
4   »  2/10/12
9:29:47.000 AM  
************************************************************
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 09:20:45
5   »  2/10/12
9:29:47.000 AM  
************************************************************
Microsoft Forefront Client Security Log, (c) 2006
Stopped On Fri Feb 10 2012 09:20:43 (Exit Code = 0x0)
6   »  2/10/12
9:29:47.000 AM  
************************************************************
Product Version: 1.5.1941.0
Engine Version: 1.1.8001.0
AS Signature Version: 1.119.1683.0
AV Signature Version: 1.119.1683.0
7   »  2/10/12
9:29:47.000 AM  
--------------------------------------------------------------------------------
Microsoft Forefront Client Security Log, (c) 2006
Started On Fri Feb 10 2012 08:57:48
0 Karma

justinhart
Path Finder

I am not deleting them and reindexing, I'm making the changes, restarting splunk, and then triggering more events to be written to my forefront logs.

Yes, I do have crcSalt = in my input.

Would the SEDCMD that I have to put in my props.conf affect anything?

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Try this:

[forefront]
BREAK_ONLY_BEFORE = ^\*.+?$
BREAK_ONLY_BEFORE_DATE = FALSE
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
pulldown_type = 1
EXTRACT-productVersion = ^Product\sVersion:\s(?P<productVersion>.+?)$
EXTRACT-engineVersion = ^Engine\sVersion:\s(?P<engineVersion>.+?)$
EXTRACT-ASSignature = ^AS\sSignature\sVersion:\s(?P<ASSignatureVersion>.+?)$
EXTRACT-AVSignature = ^AV\sSignature\sVersion:\s(?P<AVSignatureVersion>.+?)$
EXTRACT-ScanID = ^Scan\sID:(?P<scanID>.+?)$
EXTRACT-ScanSource = ^Scan\sSource:(?P<scanSource>.+?)$
EXTRACT-StartTime = ^Start\sTime:(?P<startTime>.+?)$
EXTRACT-EndTime = ^End\sTime:(?P<endTime>.+?)$
EXTRACT-ResourceSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
EXTRACT-ResourcePath = ^Resource\sPath:(?P<resourcePath>.+?)$
EXTRACT-ResultCount = ^Result\sCount:(?P<resultCount>.+?)$
EXTRACT-NumberOfResources = ^Number\sof\sResources:(?P<numberOfResources>.+?)$
EXTRACT-Resource\sSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
EXTRACT-Resource\sPath = ^Resource\sPath:(?P<resourcePath>.+?)$
EXTRACT-Extended\sInfo = ^Extended\sInfo:(?P<extendedInfo>.+?)$
0 Karma

justinhart
Path Finder

This is the event in Splunk:
Microsoft Forefront Client Security Log, (c) 2006


Microsoft Forefront Client Security Log, (c) 2006

Started On Fri Feb 10 2012 14:15:44


Product Version: 1.5.1941.0

Engine Version: 1.1.8001.0

AS Signature Version: 1.119.1709.0

AV Signature Version: 1.119.1709.0


The second and third row of -----'s are actualy asterisks, not sure why the comment changed them.

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Something like this:

inputs.conf

[monitor:///Users/dmaislin/Desktop/forefront]
disabled = false
followTail = 0
index = forefront
sourcetype = forefront

props.conf

 [forefront]
    BREAK_ONLY_BEFORE = ---
    NO_BINARY_CHECK = 1
    SHOULD_LINEMERGE = true
    pulldown_type = 1
    EXTRACT-productVersion = ^Product\sVersion:\s(?P<productVersion>.+?)$
    EXTRACT-engineVersion = ^Engine\sVersion:\s(?P<engineVersion>.+?)$
    EXTRACT-ASSignature = ^AS\sSignature\sVersion:\s(?P<ASSignatureVersion>.+?)$
    EXTRACT-AVSignature = ^AV\sSignature\sVersion:\s(?P<AVSignatureVersion>.+?)$
    EXTRACT-ScanID = ^Scan\sID:(?P<scanID>.+?)$
    EXTRACT-ScanSource = ^Scan\sSource:(?P<scanSource>.+?)$
    EXTRACT-StartTime = ^Start\sTime:(?P<startTime>.+?)$
    EXTRACT-EndTime = ^End\sTime:(?P<endTime>.+?)$
    EXTRACT-ResourceSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
    EXTRACT-ResourcePath = ^Resource\sPath:(?P<resourcePath>.+?)$
    EXTRACT-ResultCount = ^Result\sCount:(?P<resultCount>.+?)$
    EXTRACT-NumberOfResources = ^Number\sof\sResources:(?P<numberOfResources>.+?)$
    EXTRACT-Resource\sSchema = ^Resource\sSchema:(?P<resourceSchema>.+?)$
    EXTRACT-Resource\sPath = ^Resource\sPath:(?P<resourcePath>.+?)$
    EXTRACT-Extended\sInfo = ^Extended\sInfo:(?P<extendedInfo>.+?)$
0 Karma

justinhart
Path Finder

Thanks dmaislin, but that didn't seem to do what I'm wanting.

Each line in between the "***************" lines should be one event. Here is the props.conf stanza that I'm using right now that isn't working

[ms_forefront]
NO_BINARY_CHECK = true
SEDCMD-stripnull = s/\\x00//g
LINE_BREAKER = ([\r\n]+)(\*{60})
SHOULD_LINEMERGE = true
MUST_BREAK_AFTER = (\*{60})
BREAK_ONLY_BEFORE_DATE = false

I have to wonder if the SEDCMD has something to do with it. Without the sedcmd, I get a "\x00" inserted in between each character of my log.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...