Hi,
I have a 3 different log files and there are 8 different formats in them. All formats have the same fields in them (cpu,memory etc.) and regex is similar. Hence, based on the system name, I should get its values (level, OS, primary_drive). Below is the sample log:
Nov 08 12:55:07 servername: {"cpu_cores":"4","ram_size":"6","system_name":"NAME1","level":"info","OS":"WINDOWS","primary_drive":"C"}
Nov 08 12:54:07 servername: {"cpu_cores":"2","ram_size":"6","BIOS_Version":"A12","system_model":"Opti","system_name":"NAME2","level":"info","OS":"WINDOWS","primary_drive":"D"}
Nov 08 12:52:07 servername: {"cpu_cores":"4","ram_size":"4","system_name":"NAME3","level":"info","OS":"WINDOWS","primary_drive":""}
What I did here is I gave the 2 different regex for 2 log formats in props.conf. Hence I get both these log formats in and other 6 formats are ignored
props.conf
[source::/source/file name]
TRANSFORMS-set = outside,inside
transforms.conf
[outside]
REGEX =.
DEST_KEY = queue
FORMAT = nullQueue
[inside]
REGEX = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>
REGEX = {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)
DEST_KEY = queue
FORMAT = indexQueue
Also for field extraction, I made the change in props.conf as below, but I get the field extraction for only regex2. The field extraction for regex 1 is not working, hence it's not getting extracted. I am not sure whether same field getting extracted for different patterns might be a problem. Can someone help me, so that I should get field extractions for both regex types?
props.conf
[my_sourcetype]
EXTRACT-fields = <{\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)>
EXTRACT-fields = {\"(?:cpu_cores\")\:\"(?<cpu_cores>.+)\"\,\"(?:ram_size\")\:\"(?<ram_size>.+)\"\,\"(?:BIOS_Version\")\:\"(?<BIOS_Version>.+)\"\,\"(?:system_model\")\:\"(?<system_model>.+)\"\,\"(?:system_name\")\:\"(?<system_name>.+)\"\,\"(?:level\")\:\"(?<level>.+)\"\,\"(?:OS\")\:\"(?<OS>.+)\,\"(?:primary_drive\")\:\"(?<primary_drive>\w)
Hi, finally I found it. Instead of giving EXTRACT-fields = regex, I gave the all the field names .
EXTRACT-field1, field2, field3....... = regex 1
EXTRACT-field1, field2, field3....... = regex 2
Hi, finally I found it. Instead of giving EXTRACT-fields = regex, I gave the all the field names .
EXTRACT-field1, field2, field3....... = regex 1
EXTRACT-field1, field2, field3....... = regex 2
Sridhar,
I had come across a similar problem. I am explaining the approach I have taken, may be useful for you.
1. Combine all the three log files into an eventtype.
Example: eventtype myevent="index=* host=hostname source="source1.log" OR source="source2.log"
2. in each log file define field extracts for the fields as you are currently doing. Use same field name across files (or use field aliases)
3. search for the common fields using eventtype.
example: eventtype="myevent" ID=*
this query will give you values for ID from all three files.
-Bharath
How about trying something like this
REGEX ="([^"]+)":"([^"]*),?
FORMAT = $1::$2
This works. But in my case I need to provide 2 different regex patterns for a single log file. When I give the 2 patterns in props.conf it extracts only one in result(where the fields for both patterns are same).