Splunk Search

Best way to manage extra field from raw log

samlinsongguo
Communicator

I imported some custom log for file auditing. each log message is very long, it has 7 type of messages. To normalize /extra useful field from the raw log, I wrote 7 separate regex to fully extra every line of the log file. so props.conf file end up like this.

My question is : Is this a right/good way to manage field extraction in this situation, or I should write an app to manage this imperatively. Will this causing any performance issue?
Thanks

[customlog]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_FORMAT = %y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = TimeCreated SystemTime=
category = Custom
 disabled = false
pulldown_type = 1
SHOULD_LINEMERGE = false
TZ = Australia/Canberra
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$1 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessList>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessMask>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<DesiredAccess>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?(?=<)(?<Attributes>)|(?<Attribute>[^<]+))<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$2 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthPackageName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$3 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$4 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\$\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<InformationRequested>[^<]+)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$5 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\w\$\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthenticationPackageName>[\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$6 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<OldPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<NewPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<Attributes>)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$7 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<ObjectName>[^<]+)<\/\w+><[\w\"\=\s]+>(?<WriteOffset>\d+)<\/\w+><[\w\"\=\s]+>(?<WriteCount>\d+)
Tags (2)
0 Karma

peterchenadded
Path Finder

Wow, probably better to try and convert the message into a proper XML message and have splunk automatically extract the tags for you.

You can then get rid of all the regex and setup field alises if you need the fields to be different names to the tags.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...