Hi,
I want to know how to configure offset for et/lt in indexes.conf for virtual index in Hunk.
I understand the basic configuration where HDFS path contains the following date information.
/data/test/20140213/07/some.log.gz
/data/test/20140213/08/some.log.gz
/data/test/20140213/09/some.log.gz
vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.offset = 0
vix.input.1.et.regex = test/(\d+)/(\d+)/
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.offset = 3600
vix.input.1.lt.regex = test/(\d+)/(\d+)/
vix.input.1.path = /data/test/${yyyymmdd}/${hh}/...
It is clear that the offset is 1 hour = 3600
However, I don't know how to configure this:
/data/test/2014/some.log.gz
/data/test/2013/some.log.gz
/data/test/2012/some.log.gz
vix.input.1.et.format = yyyy
vix.input.1.et.offset = 0
vix.input.1.et.regex = test/(\d+)/
vix.input.1.lt.format = yyyy
vix.input.1.lt.offset = <ummmm I don't know>
vix.input.1.lt.regex = test/(\d+)/(\d+)/
vix.input.1.path = /data/test/${yyyymmdd}/${hh}/...
If the path has only year information, then do I need to consider leap year?
If the path has year and month, do I also have to consider number of days in a month? 28/29/30/31??
Thanks,
For variable partitions like month/year level,
you should set the offsets such that they include the maximum possible time range
e.g.
month = 31 days = 2678400 seconds
year = 366 days = 31622400 seconds
For variable partitions like month/year level,
you should set the offsets such that they include the maximum possible time range
e.g.
month = 31 days = 2678400 seconds
year = 366 days = 31622400 seconds