Hi fellow splunkers,
I ran into a problem regarding "Overwriting of an existing sourcetype via props and transforms".
Let me tell you more about my current scenario:
I have to connect some loadbalancers via syslog udp to a heavy forwarder.
I have the following stanza in inputs.conf
[udp://:1514]
connection_host=dns
index=loadbalancer
sourcetype=loadbalancer_syslog
So far the events get collected in the index=loadbalancer and sourcetype=loadbalancer_syslog.
I then noticed that the syslog-events contain a lot of messages from different system-deamons. So for me there is the task to split every format of every deamon into a different sourcetype.
So I did the following to the props.conf on the heavy forwarder:
[loadbalancer_syslog]
TRUNCATE = 10000
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+)
TRANSFORMS-changeSourcetype1 = logger-set-sourcetype, httpd-set-sourcetype, sshd-set-sourcetype, crond-set-sourcetype, tmm-set-sourcetype, mcpd-set-sourcetype, syslog-ng-set-sourcetype
[loadbalancer_logger]
[loadbalancer_httpd]
[loadbalancer_sshd]
[loadbalancer_crond]
[loadbalancer_tmm]
[loadbalancer_mcpd]
[loadbalancer_syslog-ng]
Then I did the corresponding stanz in the transforms.conf on the HF:
[logger-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+\s+?\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\s+?\w+\s+?logger)
FORMAT = loadbalancer_logger
[httpd-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+\s+?\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\s+?\w+\s+?httpd)
FORMAT = loadbalancer_httpd
...[shortened]
When I search the data based on the index there are distinct values for the sourcetypes in the sourcetype-field, but I can't search for the sourcetype-values directly. It returns "0" events.
I can find data by searching via the original "loadbalancer_syslog" sourcetype, but not with the newly created ones. (very strange)
Anyone had this problem before?
Help is much apprechiated!
Regards,
pyro_wood
Hi,
had a similar problem and the FORMAT = is kinda confusing in the documentation.
FORMAT = sourcetype:: ...
should do the trick. Notice the double "::". The extracted values from REGEX should be inserted properly then.
Skalli
Hi,
had a similar problem and the FORMAT = is kinda confusing in the documentation.
FORMAT = sourcetype:: ...
should do the trick. Notice the double "::". The extracted values from REGEX should be inserted properly then.
Skalli
Worked like a charm!
Thanks!