I need to search for users who clicked on totally new urls seen in last 24 hrs. If user has clicked on a link which was already previously seen i do not need those events.
Any inputs will be really helpful
You need to maintain a lookup file
with something like ... | stats min(_time) AS _time BY url | outputlookup firstTimeURLs.csv
and then xref with ... | lookup firstTimeURLs.csv url OUPUT url AS seen | where isnull(seen)
against your data.
the csv of daily access list for each user.
Is the proxy log in Splunk?
Yes proxy log is in Splunk. There cannot be a specific daily access list for each user because numbers of users are high and no restriction on the number of url accessed
index=your_index sourcetype=proxy
| rex field=url "https?:\/\/(?<domain>[^\/]+)"
| stats values(eval(if(status=200,domain,NULL))) as domain
| mvexpand domain
| eval flag="access"
| outputcsv domain.csv
First, create access domain csv, and
index=your_index sourcetype=proxy
| rex field=url "https?:\/\/(?<domain>[^\/]+)"
| inputlookup append=t domain.csv
| where flag!="access"
| stats values(domain) as domain by user
Second, check new access domian by user.
It seems impossible to create a list of access domains for each user.