How to log to metric with unstructured data?

[brandy81]

Hello Guru,

I would like to do "log to metric" on unstructured data. Let's say the data is "access_combined.log".
I would like to extract last 3 digit as "code" file name.

On FW,
inputs.conf
[monitor:///home/ec2-user/access_combined.log]
index = metric_test
sourcetype = metric_access

props.conf
[metric_access]
TRANSFORMS-metricname = metric_name
TRANSFORMS-metricvalue = metric_value
METRIC-SCHEMA-TRANSFORMS = metric-schema:extract_metrics

transforms.conf
[metric_name]
REGEX = (.*)
FORMAT = $1 metric_name::code
WRITE_META = true

[metric_value]
REGEX = (\d+)$
FORMAT = _value::$1
WRITE_META = true

[metric-schema:extract_metrics]
METRIC-SCHEMA-MEASURES-queue = ALLNUMS

What's wrong with this?

[DalJeanis]

You didn't tell the transforms where to look for the information was that it was trying to extract. Add a SOURCE_KEY field to your extraction stanzas.

Also, the METRIC_NAME FORMAT looks odd.

[DalJeanis]

@brandy81 - Take a look at that $1 in your METRIC_NAME FORMAT and tell me what it will do. What is the value that was extracted? What will the system do with it? Does it belong there at all?

It seems like you are trying to use metric indexes backwards, using the entire _raw as the metric name, and an arbitrary 3-digit value as the "value", which makes no sense. Not only will you not get any speed benefit out of it, you will probably encounter all kinds of odd glitches based on the unstructured data, and the statistical advantages of a metric index will have no meaning.

How, precisely, do you intend to use the metric index? Give us an example of two records, and what the expected metric name and metric value would be.

[brandy81]

@DalJeanis Thank you for reply. I made a mistake. When I apply those configurations on Indexer, It is working as I intended. But I encountered another issue: I am not able to extract two field-value.

Actually, my intention was to apply "log-to-metric" on unstructured data. The log below is just sample.

176.212.0.44 - - [29/Jan/2017:19:12:30] "POST /cart.do?action=purchase&itemId=EST-17&JSESSIONID=SD9SL8FF10ADFF5186 HTTP 1.1" 200 853 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-17&categoryId=SPORTS&productId=CU-..." "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3" 517
476.212.0.44 - - [29/Jan/2017:19:12:30] "POST /cart.do?action=purchase&itemId=EST-17&JSESSIONID=SD9SL8FF10ADFF5186 HTTP 1.1" 200 853 "http://www.buttercupgames.com/cart.do?action=addtocart&itemId=EST-17&categoryId=SPORTS&productId=CU-..." "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3" 517

176.212.0.44 - - [29/Jan/2017:19:12:31] "POST /cart/success.do?JSESSIONID=SD9SL8FF10ADFF5186 HTTP 1.1" 200 2472 "http://www.buttercupgames.com/cart.do?action=purchase&itemId=EST-17" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3" 299

Let's say I want to extract 3 digit at the end of log and 3 digit at the beginning of the log (I know it does not make sense.) How can adjust conf files? I have searched and tried many times, but it seems impossible... Do you have any idea?

[DalJeanis]

Okay, @brandy81, so, to be clear:

As an exercise only, you want to extract the 3-digit number at the start and the 3-digit number at the end of each record. You want one of those to be used as a dimension, and the other one to be a metric value? Or did you want the first one to be a metric "name", and the second one a metric value? Or did you want each one of those to be a different metric value?

[brandy81]

@DalJeanis Thanks. The default value of SOURCE_KEY is _raw so don't need to specify. METRIC_NAME FORMAT is correct because I intended to do hard-cording field name with "code", not extracting from the log. That's way I used the syntax. Thanks anyway.