I need some help to figure out how to extract or make sure all the products were shown.
index=main sourcetype=appserver custid=xbhfhsls | table products
It's showing only a couple of products, but in the log, we see the customer has purchased multiple products.
yes i'm referring to auto field discovery which splunk applies....i have the raw log if you can help me with regex...
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h78789" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="Crochet Pencil, Blush" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h7980" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="Jersey T-Shirt, Black" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h78876" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="T-shirt, Blue" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
yes i'm referring to auto field discovery which splunk applies....i have the raw log if you can help me with regex...
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h78789" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="Crochet Pencil, Blush" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h7980" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="Jersey T-Shirt, Black" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
<line_item registry_id="" transaction_type="A" item_id="" item_id="56 7h78876" sku_id="skuxxxxx" catalog_id="HJH78" catalog_item="KJDN78" Ksku_id="xxxxxxxxx" product="T-shirt, Blue" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
I wrote your regular expression in my comment above
Is each line_item a separate event? or Did you paste a single multiline event?
nope it's not separate event, it's a single multiline event.
Yep this is the issue. I was assuming each block represented a different event. You could either linebreak these blocks into seperate events in your props.conf or you could do it at search time like jplumsdaine22 suggested
As @skoelpin says - break these into separate events if it makes sense to do so. I t will make your life easier.
Thanks much Jplumsdaine and skoelpin...it's working
index=main sourcetype=server.log | rex field=_raw (?P(?<=product_name=)\".*(?=quantity)) max_match=0 | table Products _raw
can you please let me know how i can break this events using props.conf on a heavy forwarder...?
Ah that's the issue. You need to have the following option in your | rex
command max_match=0
See http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex
It should then create a multivalue field called product
Are you referring to the auto field discovery Splunk applies when you search? If this isn't working for you then you may want to create a regular expression to extract all the values from that field. You can do it at search time or create a new field which you can reuse later. Splunk also has a regex builder but it's unreliable in my opinion. Provide some events and I'll help write the regex for you
Here's some regex which will grab the product name.. It does a lookbehind for product and does a lookahead for quantity and grabs everything in between. Go to 'Extract New Field' then 'I prefer to write my own regular expression' and enter this
(?P<Product>(?<=product\=)\".*(?=quantity))
Thanks skoelpin, this regex is creating a new filed product, but it's displaying the entire log starting form product
"Crochet Pencil, Blush" quantity="1" service_code="XXX" gift_wrap_code="" price_each="XXX.00" tax_each="XX.XX" other_each="0.00" freight_each="0.00" duty_each="0.00" markdown_each="," arrive_by_date="" buy_now_gift_later_flag="N" promotion_code="," promotion_key="XXXXX," gift_message1="" gift_message2="" gift_message3="" gift_message4="" gift_message5="" current_status="" current_status_date="" tracking_number="" giftcard_email="" store_fulfill_flag="N" reporting_code="" pickup_store_number="" selected_replenishment_interval="" replenishment_contract_id="" shipto_store_number="" dynamic_image_url="" facility_ID="" line_discount_amount="" employee_store_call="" special_instruction_flag="" special_instruction_price="" selection_set_id="" bogo_proration=""/>
Try this.. I removed the lookbehind.
(?P<Product>(?<=product\=)\"((\w+\,\s\w+\")|(\w+\s+\w+\,\s\w+\")))
somehow it's not working and also the actual format is
product_name="jersey T shirt, black"
I'm sorry i have made some changes to raw log(given product instead product_name)due to confidentiality. Do we need to make any tweaks for the above format...?
Yep that matters. I have a lookahead which anchors in on the product text. Below is updated regex which includes product_name in the lookahead. I also added the lookbehind again and included a \d and \s which should cut it off and not get all that extra data after the product info (Hate when that happens). It works good in my Regex tester
(?P<Product>(?<=product\_name\=)\".+(?=quantity="\d"\s))
This regex is working fine, but i'm having the same issue, it's just looking up the first product_name in the entire log.
The raw log which i have is a single log under a single timestamp, it's only capturing the first product leaving the rest of the products bought by the single customer.
This is a reply from your comment below..
If you want to modify your linebreaking NOT at search time, then you want to modify the props.conf file on the indexer, not the forwarder.
Go to your Splunk\etc\system\local\Props.conf file
Insert this stanza (Your's may vary)
[Your Sourcetype/host]
TRUNCATE = 20000
MAX_EVENTS = 20000
TIME_FORMAT = %Y-%m-%d %H:%M:%S,$3N
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE = ^(?:\w+\s+)?\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2},\d{3}
I would definitely try this on a Indexer, i thought we also have this ability to do on Heavy forwarder. Appreciate your help..!!
Can you please accept this answer if it was helpful for you?
Can you provide what the raw log looks like and post your props.conf.
Thanks for the reply, i didn't extract this field using props.conf, it's auto key-value pair which is shown in the fields. in this situation it's showing only first 2 items under product