Splunk Search

Is there a way to dedup then use a time picker?

jeffsegal
Explorer

I am trying to create a report that would tell me if an item that should be available within a certain timeframe (i.e. last 30 days) has not been available. Splunk feeds is a full set of results everyday, so although the _time would be different, then computername would have to be rid of duplicates before specifying a time.

this example does not throw an error, but also does not provide content. from a manual review, I know that content exists.
example: index=test sourcetype=computernames |dedup computername |search latest=-30d

thank you in advance for your help.

0 Karma

woodcock
Esteemed Legend

Do it like this:

YOUR SEARCH HERE
| appendpipe [ |inputlookup YourLookupHere ]
| dedup computernames
| outputlookup YourLookupHere

This will take the new stuff in preference over the old stuff if both exist, but keep the old stuff if nothing new is found in the search.

mdsnmss
SplunkTrust
SplunkTrust

So you want an asset list of computernames over a 30 day period or so and then be able to search a different period and see if it exists within that time frame? How does Splunk get the full set of results everyday? It may be worth feeding that daily feed into a lookup with an append=true. Then you have a running list for computernames. If that's not what you are looking for can you clarify a little more?

0 Karma

jeffsegal
Explorer

Yes to your first question.

it gets a daily dump using dbconnect with their last report in time, I then create another sourcetype using their last report time as _time.

your suggestion is interesting, but I wonder if I could create a lookup for the last 30 days, then somehow remove the computernames it finds.

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

I'm not sure how your database stores the information so am not sure if you can make it time relative. If the record in your database has a timestamp to it you should be able to do it via lookup. You can create a lookup directly from DBConnect and could could run a search to lookup with your dedup and where time<30 days in the search. An alternative is to schedule a search each day after your DB dump comes in that runs your search over the last thirty days with the dedup and output to the lookup from that. Then you can run your reference search against the lookup that should have your computername list going back the last 30 days.

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

DBConnect Lookups: http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaselookups. I think if you can create the lookup right here it would be best. That would save on having an additional saved search to populate the lookup. You can either do a database lookup or have it output to csv for a Splunk lookup.

0 Karma

jeffsegal
Explorer

So I created a lookup for computernames that have reported in, in the last 30 days, but do you know how I would go about removing computernames from the original search that are in the lookup?

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Try index=test sourcetype=computernames NOT [| inputlookup <lookup_name> | fields computernames] | dedup computernames.

Alternatively, you could add a flag to your lookup using lookup definitions. Under advanced options for the definition you can set "Minimum matches" equal to 1 and default to 0. Then you could run index=test sourcetype=computernames | dedup computernames | lookup computernames as computernames | search match=0

I might be a little off in syntax, but that should be the idea.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...