Lines starting with semicolon(;) needs to discarded completely from indexing . My props and transforms not working

[NAVEEN_CTS]

I would like to remove any lines that start with semicolon(;) from indexing. Below are my config files and sample data. Im not receiving logs to my splunk. Please help

Sample Log data:
123 123 12123
;123 123 123 123 123
; 123 123 123 123
214121 ; 214 ; 1212 ; 33

My inputs.conf
[monitor://$SPLUNK_HOME/etc/apps/testapp/log]
index=test
sourcetype=test

My props.conf
[source:://$SPLUNK_HOME/etc/apps/testapp/log]                  
TRANSFORMS-null= setnull

My transforms.conf
[setnull]
REGEX = ^;.*$
DEST_KEY = queue
FORMAT = nullQueue
  

[somesoni2]

Give this a try

My props.conf

[source:://$SPLUNK_HOME/etc/apps/testapp/log]            
SHOULD_LINEMERGE = false       
TRANSFORMS-null= setnull_colons

My transforms.conf

[setnull_colons]
REGEX = ^\;.+
DEST_KEY = queue
FORMAT = nullQueue

[NAVEEN_CTS]

this didn't work as well.

Let me review my config first.

1) I have placed my log file at HF and it has full permission
2) inputs .conf is placed in /apps/local
3) props.conf and transforms.conf is placed at idx --> /apps/local/

My inputs.conf

[monitor://$SPLUNK_HOME/etc/apps/app_name/log/test.txt]
index=test
sourcetype=test

My props:
[source::/$SPLUNK_HOME/etc/apps/app_name/log/test.txt]
SHOULD_LINEMERGE = false
TRANSFORMS-drop = delLines

My Transforms:
[delLines]
REGEX = ^[^\;].+
DEST_KEY = queue
FORMAT = nullQueue

I have 2 problems
1) I get all the lines as single event.
2) Lines starting with (;) is not removed

Please let me know the missing config here. Thanks in advance

[somesoni2]

The props and transforms should be in the first Splunk Enterprise instance in your data flow. If you've heavy forwarders in front of indexes, then heavy forwarders should have that config. Also, do remember to restart HF after applying those configurations.

For your line breaking, could you post sample events and show what are your event boundaries?

[NAVEEN_CTS]

@somesoni2 .

My log file looks like this. In splunk im seeing it as a single line And the line that is starting with ; needs to be removed

Sample Log:

;*************
; X ABCDEF
;*************
xxxxxxxx BE A X.XX.XX.XXX
xxxxxxxx BE A X.XX.XX.XXX
XXXXXXXXXXXXXXXXXXXXXXXXX IN A X.XX.XX.XXX
XXXXXXXXXXXXXXXX BE A XX.XX.XX.XXX
XXXXXXXXXXXXXXX BE A XX.XX.XX.XXX
XXXXXXX BE A XX.XX.XX.XXX

[niketn]

@NAVEEN_CTS do you have event breaking in props.conf for breaking every line? Do you have timestamp in the data? Does time get identified correctly? With nullQueue not working are you seeing each line as separate event with correct event raw data and correct time stamp?

[NAVEEN_CTS]

@niketnilay no i have not set the event breaking and also my log doesn't have a timestamp.....it is just a dump of some report which i would like to index and use it as a lookup. But it has lot of junk data ....which i want to remove lines that starts with ;

So not setting the event breaking is the problem?

[NAVEEN_CTS]

Could you please help with line break regex? Also where should we keep these props and transforms? Both HF and IDX? Right now im keeping it in idx alone. Log file is monitored in HF

[vnravikumar]

Hi

Try with this regex

^[^\;].+

[NAVEEN_CTS]

@vnravikumar No it didnt work .....same as before. I get all the lines as a single event .... may be i have to try event breaking