We have logs , where first few lines needs to be omitted from ingesting.
We only need to on-board the events , that start with the date/time in the following format:
"%m/%d/%Y@%H:%M"
Appreciate all the ideas and suggestions.
Here is the log example (there are also empty lines
before first "#-----------------------------------------" and
after last "#-----------------------------------------"):
#-----------------------------------------
#DATE CREATED: 11/02/2021@04:16
#SUBJECT: REPORT ON THE GENERAL STATUS OF AUTOSYS JOBS
#ENVIRONMENT: CBA
#-----------------------------------------
11/02/2021@04:16,CBA,OTHER,CBA_CLIENT_REPORT_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_copy_file_job,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_SCHEDULER_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_REPORT_BOX,OI
@mlevsh , This question is already answered in the past.
Can you please look into below link and see that helps.
Ignoring-any-data-record-that-begins-with-a-quot-quot-character
You can do this with props and Transforms as mentioned in the above post.