How to extract the same fields from multiple log formats?

[SridharS]

Hi,

I have a 3 different log files and there are 8 different formats in them. All formats have the same fields in them (cpu,memory etc.) and regex is similar. Hence, based on the system name, I should get its values (level, OS, primary_drive). Below is the sample log:

Nov 08 12:55:07 servername: {"cpu_cores":"4","ram_size":"6","system_name":"NAME1","level":"info","OS":"WINDOWS","primary_drive":"C"}
Nov 08 12:54:07 servername: {"cpu_cores":"2","ram_size":"6","BIOS_Version":"A12","system_model":"Opti","system_name":"NAME2","level":"info","OS":"WINDOWS","primary_drive":"D"}
Nov 08 12:52:07 servername: {"cpu_cores":"4","ram_size":"4","system_name":"NAME3","level":"info","OS":"WINDOWS","primary_drive":""}

What I did here is I gave the 2 different regex for 2 log formats in props.conf. Hence I get both these log formats in and other 6 formats are ignored

props.conf

[source::/source/file name]
TRANSFORMS-set = outside,inside

transforms.conf

[outside]
REGEX =.
DEST_KEY = queue
FORMAT = nullQueue

[inside]
REGEX = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>

REGEX =  {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w) 

DEST_KEY = queue
FORMAT = indexQueue

Also for field extraction, I made the change in props.conf as below, but I get the field extraction for only regex2. The field extraction for regex 1 is not working, hence it's not getting extracted. I am not sure whether same field getting extracted for different patterns might be a problem. Can someone help me, so that I should get field extractions for both regex types?

props.conf

[my_sourcetype]
EXTRACT-fields = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>

EXTRACT-fields =  {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w) 

[SridharS · Accepted solution]

Hi, finally I found it. Instead of giving EXTRACT-fields = regex, I gave the all the field names .

EXTRACT-field1, field2, field3....... = regex 1
EXTRACT-field1, field2, field3....... = regex 2

[bkumarm]

Sridhar,
I had come across a similar problem. I am explaining the approach I have taken, may be useful for you.
1. Combine all the three log files into an eventtype.
Example: eventtype myevent="index=* host=hostname source="source1.log" OR source="source2.log"
2. in each log file define field extracts for the fields as you are currently doing. Use same field name across files (or use field aliases)
3. search for the common fields using eventtype.

example: eventtype="myevent" ID=*
this query will give you values for ID from all three files.
-Bharath

[sundareshr]

How about trying something like this

REGEX ="([^"]+)":"([^"]*),?
FORMAT = $1::$2

[SridharS]

This works. But in my case I need to provide 2 different regex patterns for a single log file. When I give the 2 patterns in props.conf it extracts only one in result(where the fields for both patterns are same).