Hello Splunk Team,
How can I write/schedule a program (java/python) to clean the eventdata?
My use case is:
Please provide your input on same.
Thanks in advance.
You can simply do this using a shell script. Write Shell script with 3 commands
Command 1: splunk stop
Command 2: splunk clean eventdata -index
Command 3: splunk start
Schedule this script to run before you index new data.
Same logic can be used in python script also. As per my knowledge you have to schedule that python program to run using some shell script.
Since the no of records are less, and it need to be updated frequently (daily), why don't you use lookup table file to store this metadata instead of Splunk Index. You can use outputlookup after your dbx command to updated the lookup table file from search.
Something like this
your dbx command | outputlookup YourLookupName.csv append=false
Append=false will ensure data is overwritten, so you'll always have the latest data.
somesoni2 - Thank you for your response.
You can simply do this using a shell script. Write Shell script with 3 commands
Command 1: splunk stop
Command 2: splunk clean eventdata -index
Command 3: splunk start
Schedule this script to run before you index new data.
Same logic can be used in python script also. As per my knowledge you have to schedule that python program to run using some shell script.
Strive- Yes, there is not other way to clean the index. I am using mentioned script/commands to clean the index...
As per splunk documentation, splunk recommends to clean the data by stopping splunk. But i have not tried cleaning event data without stopping splunk.. So i do not know the impact.
Strive- I am looking other solution. I don't wanna stop/start the server since this is my production server (enterprise app).