Splunk Search

Specific Search not working after upgrade to Splunk 5.0

tiny3001
Path Finder

I've got a specific search, that generates two time ranges on a timechart, using the instructions found on this Splunk blog entry:
Compare two time ranges in one report

This search worked perfectly... until yesterday, when we upgraded our search heads to Splunk 5.0.2.

I've confirmed this, by running the exact search on another search head that is still running Splunk 4.3.3. The search works on Splunk 4, does not work on Splunk 5.

Here is the (edited for privacy) search:

index="summary" search_name="Event Count Summary" earliest=-28d@d latest=@d
| eval Month="This Month"
| append [search index="summary" search_name="Event Count Summary" earliest=-56d@d latest=-28d@d
  | eval Month="Last Month"
  | eval new_time=_time+2419200]
| eval _time=if(isnotnull(new_time), new_time, _time)

On the 5.0.2 search head, I only get results for "Last Month" and only "Last Month" gets graphed. On the 4.3.3 search head, everything gets graphed perfectly.

I suspected possible timeouts in the subsearches, but the limits.conf between the two search heads are almost identical.

How can a Splunk upgrade break a search? Where do I need to look?

Edit: (2013/04/15)
The comment below is correct, this definitely has to do with the 'append' command. Still no answer on this though, but in the meantime, I have taken it up with Splunk Enterprise support. Haven't heard from them in over a month now...

Tags (3)
1 Solution

daniel_splunk
Splunk Employee
Splunk Employee

If your indexer is still under 4.3.x, you may hit this.

Searches that contain subsearches do not return data in environments where search heads are running version 5.0.x and indexers are running version 4.3.x. To work around this issue, upgrade indexers to the same version as the search heads. (SPL-62457)

This is documented in the known issue.

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

View solution in original post

daniel_splunk
Splunk Employee
Splunk Employee

If your indexer is still under 4.3.x, you may hit this.

Searches that contain subsearches do not return data in environments where search heads are running version 5.0.x and indexers are running version 4.3.x. To work around this issue, upgrade indexers to the same version as the search heads. (SPL-62457)

This is documented in the known issue.

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

tiny3001
Path Finder

Also, closing the Splunk Enterprise support call... not sure why they didn't pick up on this? Maybe my question wasn't clear enough?

0 Karma

tiny3001
Path Finder

Absolutely the correct answer. Ironically, the one thing that could have solved our problem (upgrading the 4.3.3 indexer) was the one thing that could have solved our problem. We were too scared to upgrade after we picked up the issue and wanted to understand it completely before we upgraded everything.

0 Karma

jevenson
Path Finder

I have a similar issue, and I'm thinking it may have something to do with the 'append' command. My issue can be seen here: http://splunk-base.splunk.com/answers/74749/fillnull-not-working-on-my-search

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...