Getting Data In

Configurations that Filters Existing Orders from AS/400 File

_gkollias
Builder

I'm looking to come up with some configurations that filter out existing orders from files I (currently) manually copy to a local directory where Splunk then picks them up and indexes the order info for that hour.

Each file that comes out from the AS/400 has a total of all orders with various information on that order(CustomerNumber, PONumber, Date, Time, etc.) up to that particular hour.

Basically every time Splunk Picks up one of those files, I want it so that Splunk only indexes the NEW orders, rather than indexing the same order data from the previous hours. Otherwise, there will be a large amount of duplicate data being indexed.

Is there a way I can do this? Let me know if any information is needed to dig in to this further.

Thanks in advance!

Tags (2)
0 Karma

lguinn2
Legend

This might be possible if you

(1) Keep the same file name. In other words, overwrite the old file with the new file each hour.

(2) Make sure that the beginning of the file (up to the point of the new data) has not changed.

But if Splunk figures out that this is a different file, it will index it from the beginning, causing the duplication that you are trying to avoid.

There is no way for Splunk to compare inbound data with existing data before indexing. However, it is possible to "dedup" data being retreived during a search - although you have to do it explicitly with the uniq command.

I would test it out.

_gkollias
Builder

Awesome - thanks for your suggestions. I will some testing around it a bit more and come back with some feedback

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...