We have a text file to log row counts for CSV files used in an ETL job. The format of the file is like this:
08/30/15 16:08:51
acme_file.csv: 0
coyote_file.csv: 0
anvil_file.csv: 88835
roadrunner_file.csv: 330333
I want to break this into one line per filename from line 2 onward, into fields {filename}.csv: {rowcount} Note that the first line in the .txt file is the datetime for all of the events.
I think the linebreaking is fairly straightforward via props.conf on my indexers, but that first line with the datetime is in a different format; I want to use that and apply it to all subsequent lines. Little help!
Based on this clarification:
Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt
You can do it like this:
<define name="datetimefromfile" extract="month, day, year, hour, minute, second">
<text><![CDATA[source::.*?(\d{2})-(\d{2})-(\d{4})_(\d{2})(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
<use name="datetimefromfile"/>
</timePatterns>
<datePatterns>
<use name="datetimefromfile"/>
</datePatterns>
</datetime>
[YourSourcetypeHere]
DATETIME_CONFIG = /etc/apps/MyApp/default/datetime.xml
Can you have the timestamp added to the filename? If so, you can use datetime.xml
easily to do this.
Yes, the timestamp is in the filename: 08-30-2015_162551__summary.txt