Splunk cannot find files

[fjp2485]

Hi Splunk,

We use Splunk Enterprise 7.2.3.

In our environment there are 49 XML files in subfolders which have to be found by splunk. (With file explorer the files are visible.)Name of 45 files look like this:
GDSNightlyStagingInstallation_*.xml and 4 of them look like this: GDSNightlyStagingSIT_*.xml. (With date and time info in place of '*'.) Each file's document element node has name: BundleResult and has an attribute named NetDurationMins.

We use this search string:
source="\GDS\*NightlyStaging" | search (BundleResult AND NetDurationMins="*") | dedup LogFile

This search string can find only the 45 GDSNightlyStagingInstallation_*.xml files but the remaining 4 GDSNightlyStagingSIT_*.xml files are not retrieved. Could you explain where is the error, please?

Thanks in advance,
Sandor Michaletzky

[fjp2485]

Hi nyoung,
Many thanks for dealing with my problem so much.

I ran the search string you've written:
" | metadata type=sources index=sambaresultindex | search source=*GDSNightlyStagingSIT_* "
but it didn't retrieve anything.

Then I removed the SIT_ suffix to check if installation results files are retrieved. They were also not retrieved.

Then I changed index to newsambalogindex and tried again with this search string (to check if installation log files are retrieved):
" | metadata type=sources index=newsambalogindex | search source=*GDSNightlyStaging* "
This search retrieved all installation log files but the test result files didn't appear again.

Then I tried to retrieve only the test result files:
" | metadata type=sources index=newsambalogindex | search source=*GDSNightlyStagingSIT_* "
and this also didn't retrieve anything - of course.

What do I do wrong?

[fjp2485]

Hello,
I've found an index which works for our query and I've rerun the searches.

• This search:
  | tstats count where index=newsambalogindex source=\\bafs1\Builds\DashboardData\GDS\\GDSNightlyStaging
  resulted 372983 events (so much because there was no filtering for content).

• This search:
  | tstats count where index=newsambalogindex source=\\bafs1\Builds\DashboardData\GDS\\GDSNightlyStagingInstallation
  resulted the same number of events: 372983.

• This search:
  | tstats count where index=newsambalogindex source=\\bafs1\Builds\DashboardData\GDS\\GDSNightlyStagingSIT
  resulted 0.

Please tell me if I should run other tests in order to continue investigation.
Regards,
fjp2485

[fjp2485]

The two kinds of files (i.e. NightyStagingInstallation* and NightlyStagingSIT*) are produced by the same program in the same run of that. In addition the program does not give any special or different permissions to those files.

[nyoung_splunk]

I would also check permissions on those 4, make sure they didn't carry permissions from another device.

[richgalloway]

What is the monitor statement (from inputs.conf) for the files?
If you run this query: | tstats count where index=* source=GDSNightlyStagingSIT_* do the four source files appear in the results?

[fjp2485]

I tried to run the tstats, this was its result:

Reduced buckets were found in index='sambaresultindex'. Tstats searches are not supported on reduced buckets. Search results will be incorrect.

And of course result count was 0.

But the same was the result with GDSNightlyStagingInstallation*.

[richgalloway]

Try this query, instead. | metadata type=sources index=sambaresultindex | search source=GDSNightlyStagingSIT_*.