How to search for events that have null values for a field?

[abelnation]

I have json log lines that sometimes contain a request object of the form

{
timestamp: ts_val,
app: "my_app",
request: {
method: "GET",
status: 200,
}
}


I am trying to query for events that do not have the request value using isnull/isnotnull, but it doesn't have the expected effect;

app="my_app" | where isnull(request)

Still returns the full set of events. Can someone clarify what splunk actually treats as NULL? I've seen some weird behavior with nested fields. Are there links to documentation of those conditions?

Thanks!

---

I have also tried:

app="my_app" request=*

[abelnation · Accepted solution]

The key difference to my question is the fact that request points to a nested object.

For simple fields whose values are literal values (string, boolean, int), any of the following would solve the simple case to find events where a top-level field, testField is null:

app="my_app" NOT testField="*"
app="my_app" | where isnull(testField)

However, as my original post explained, I am trying to test for all events where request is not present, where request normally pointed to an object, not a literal value.

If I query either:

app="my_app" NOT request="*"
app="my_app" | where isnull(request)

It returns all events, effectively treating an object value in request as equivalent to NULL. HOWEVER, if I query specifically on a field I know to always be non-null inside the request object, e.g. request.method, I can get what I wanted:

app="my_app" NOT request.method="*"
app="my_app" | where isnull('request.method')

NOTE THE SINGLE QUOTES in the isnull call. Inside where/eval statements, splunk does not handle complex field names, well, and requires you wrap them in SINGLE quotes.

Hopefully this answer makes sense and is helpful.

[opsbbgl]

Damn, those single quotes got me...