I run this command:
index=dccmtdit sourcetype=DCCMT_Log4J_JSON | transaction DpsNum maxevents=-1
It returns: 4,999 events (before 11/11/14 11:34:05.000 AM)
I would expect the number of events returned to be the same as the distinct count of events returned by the following command:
index=dccmtdit sourcetype=DCCMT_Log4J_JSON | stats dc(DpsNum)
However, the number of events returned from the second command is:
dc(DpsNum)
45733
Is there a limit somewhere which prevents me from having all of the events in the transaction? I have various messages coming in which update the status of the event and it is possible that these messages may come in out of order.
It is actually a limit of the number of open transaction handles. The parameters (also in limits.conf) is maxopentxn. Playing with this variables, though, seems to have crashed my splunkd. I think I am going to have to seek an alternative to transactions for what I want to do. I really just want to combine events which have the same ID, so maybe dedup will allow me to do that.
I ran into this. Try adding the "keepevicted" (boolean) option to the transaction command. In your example simply :
... | transaction DpsNum maxevents=-1 keepevicted
That got me from ~5000 to ~35000.
It is actually a limit of the number of open transaction handles. The parameters (also in limits.conf) is maxopentxn. Playing with this variables, though, seems to have crashed my splunkd. I think I am going to have to seek an alternative to transactions for what I want to do. I really just want to combine events which have the same ID, so maybe dedup will allow me to do that.