Security

How to extract all sub strings ends with .csv in a string?

SumitPan
Explorer

Sorry I'm new to regex. I'm trying to get some meaning full data from the log files.

I want all the sub-strings ending with .csv in my log file at any given point of time. Below is the the log file preview. Any leads would be highly appreciated.

alt text

Tags (2)
0 Karma
1 Solution

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*

View solution in original post

0 Karma

SumitPan
Explorer

it fixed the problem.......

rex field=_raw ".*\s(?P.*\.csv)$" |search CSVFiles=*
0 Karma

lakromani
Builder

This rex does not work with the above data.
It can not be complete here?

PS you do not need to specify field=_raw, if omitted, _raw is used by default.

0 Karma

SumitPan
Explorer

I have tried both the rex and both seems to be working fine.

0 Karma

lakromani
Builder

(?P..csv) this does not extract anything. If it was more like (?<CSVFiles>\w+\.csv) it will work.

0 Karma

lakromani
Builder

It's better if you past the text in stead of a picture of the text.

0 Karma

SumitPan
Explorer

Below are the logs. Need to list down all files names ending with .csv. e.g.:
1. adn_attribute_set.csv
2. adn_navigation_attributes.csv

host=mdc1vr1002 sourcetype=MCOM_ETL_OUT

2016-11-06 19:42:35,800 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_attribute_set.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_attribute_set.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_attribute_set.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Appending smaller file: adn_navigation_attributes.csv
2016-11-06 19:42:35,801 | DEBUG | main:ConcatNCopy | Copy: adn_navigation_attributes.csv to /opt/pim/ETL/MCOM/etlc/output/site/adn_navigation_attributes.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:35,809 | DEBUG | main:ConcatNCopy | Appending smaller file: archived_products.csv
2016-11-06 19:42:35,830 | DEBUG | main:ConcatNCopy | Copy: archived_products.csv to /opt/pim/ETL/MCOM/etlc/output/site/archived_products.csv, size: 2768026, elapsed ms: 21
2016-11-06 19:42:35,853 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_cat.csv
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Copy: attr_cat.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_cat.csv, size: 201223799, elapsed ms: 190
2016-11-06 19:42:36,043 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_category_exclusion.csv
2016-11-06 19:42:36,044 | DEBUG | main:ConcatNCopy | Copy: attr_category_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_category_exclusion.csv, size: 16705, elapsed ms: 1
2016-11-06 19:42:36,045 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_page_media.csv
2016-11-06 19:42:36,079 | DEBUG | main:ConcatNCopy | Copy: attr_page_media.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_page_media.csv, size: 38563205, elapsed ms: 34
2016-11-06 19:42:36,125 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_brand_ship.csv
2016-11-06 19:42:36,169 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_copy_reg.csv
2016-11-06 19:42:36,359 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_data_source.csv
2016-11-06 19:42:36,366 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_forced_new.csv
2016-11-06 19:42:36,422 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_site_search.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_trigger_data.csv
2016-11-06 19:42:36,773 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_tuple_data.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod.csv, size: 876915462, elapsed ms: 852
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod2.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod2.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod2.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod3.csv
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod3.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod3.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod4.csv
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Copy: attr_prod4.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod4.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,978 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod5.csv
2016-11-06 19:42:36,984 | DEBUG | main:ConcatNCopy | Copy: attr_prod5.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod5.csv, size: 6903099, elapsed ms: 6
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod7.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod7.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod7.csv, size: 0, elapsed ms: 1
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod8.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod8.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod8.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_prod_colorway.csv
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Copy: attr_prod_colorway.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_prod_colorway.csv, size: 11624, elapsed ms: 0
2016-11-06 19:42:36,985 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_product_exclusion.csv
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Copy: attr_product_exclusion.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_product_exclusion.csv, size: 5340406, elapsed ms: 6
2016-11-06 19:42:36,991 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_promo.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_promo.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_promo.csv, size: 577403, elapsed ms: 1
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_req.csv
2016-11-06 19:42:36,992 | DEBUG | main:ConcatNCopy | Copy: attr_req.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_req.csv, size: 52738, elapsed ms: 0
2016-11-06 19:42:37,010 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_upc.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_upc.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_upc.csv, size: 478738319, elapsed ms: 485
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vl.csv
2016-11-06 19:42:37,495 | DEBUG | main:ConcatNCopy | Copy: attr_vl.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vl.csv, size: 21284, elapsed ms: 0
2016-11-06 19:42:37,496 | DEBUG | main:ConcatNCopy | Appending smaller file: attr_vlitems.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attr_vlitems.csv to /opt/pim/ETL/MCOM/etlc/output/site/attr_vlitems.csv, size: 1181621, elapsed ms: 1
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: attribute.csv
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Copy: attribute.csv to /opt/pim/ETL/MCOM/etlc/output/site/attribute.csv, size: 173351, elapsed ms: 0
2016-11-06 19:42:37,497 | DEBUG | main:ConcatNCopy | Appending smaller file: brand.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand.csv, size: 117929, elapsed ms: 1
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: brand_constraint_val.csv
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Copy: brand_constraint_val.csv to /opt/pim/ETL/MCOM/etlc/output/site/brand_constraint_val.csv, size: 0, elapsed ms: 0
2016-11-06 19:42:37,498 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_pools.csv
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Copy: cat_pools.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_pools.csv, size: 4362512, elapsed ms: 4
2016-11-06 19:42:37,502 | DEBUG | main:ConcatNCopy | Appending smaller file: cat_prod.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: cat_prod.csv to /opt/pim/ETL/MCOM/etlc/output/site/cat_prod.csv, size: 54444, elapsed ms: 1
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog.csv, size: 9310, elapsed ms: 0
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Appending smaller file: catalog_context.csv
2016-11-06 19:42:37,503 | DEBUG | main:ConcatNCopy | Copy: catalog_context.csv to /opt/pim/ETL/MCOM/etlc/output/site/catalog_context.csv, size: 31, elapsed ms: 0
2016-11-06 19:42:37,504 | DEBUG | main:ConcatNCopy | Appending smaller file: category.csv
2016-11-06 19:42:37,512 | DEBUG | main:ConcatNCopy | Copy: category.csv to /opt/pim/ETL/MCOM/etlc/output/site/category.csv, size: 9478833, elapsed ms: 8
2016-11-06 19:42:37,513 | DEBUG | main:ConcatNCopy | Appending smaller file: category_facet.csv
2016-11-06 19:42:37,543 | DEBUG | main:ConcatNCopy | Copy: category_facet.csv to /opt/pim/ETL/MCOM/etlc/output/site/category_facet.csv, size: 36649061, elapsed ms: 30
2016-11-06 19:42:37,544 | DEBUG | main:ConcatNCopy | Appending smaller file: contextual_category.csv
2016-11-06 19:42:37,547 | DEBUG | main:ConcatNCopy | Copy: contextual_category.csv to /opt/pim/ETL/MCOM/etlc/output/site/contextual_category.csv, size: 2614776, elapsed ms: 3

0 Karma

lakromani
Builder

Then this should do:
your search | rex "(?<file>\w+\.csv)"

Sames as rich7177 posted. So if this works, accept his answer.

PS some lines have more than one file name, this rex gets them all.

2016-11-06 19:42:36,977 | DEBUG | main:ConcatNCopy | Copy: attr_prod_brand_ship.csv attr_prod_copy_reg.csv attr_prod_data_source.csv attr_prod_forced_new.csv attr_prod_site_search.csv attr_prod_trigger_data.csv attr_prod_tuple_data.csv to
0 Karma

Richfez
SplunkTrust
SplunkTrust

Try..

 ... | rex "(?<MyCSVFile>\w+\.csv)" 

Change the name MyCSVFile to whatever you want to call it. Here you can see it in regex101.com.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...