After installing the File/Directory Information Input add-on, why are no logs being indexed?

[smudge797]

After installing File/Directory Information Input add-on on Splunk 6.5.1 for Windows and configuring to UNC path, I see the following:

INFO Time is later than filter, st_mtime=1459251861.8578942, must_be_later_than=None

[LukeMurphey]

This happens when there is no mapping between the account names and the SIDs available on the host. See http://www.rebeladmin.com/2016/01/how-to-fix-error-no-mapping-between-account-names-and-security-ids... for information.

I am going to modify the input so that it proceeds even if SID lookup fails. This will done under this ticket and will be released in version 1.1.1.

Update

I just released version 1.1.1 which should allow the input to work even if the SID and account name mapping doesn't exist on the host. You won't get the Windows ACL data if this condition exists, but the input will still run.

Please let me know if that fixes the problem (or just accept this answer).

[smudge797]

Using latest version I'm still having issues. Only seeing occasional logging to the correctly configured index on restarting the service. The interval is being ignored here.

Looking at internal logs:
Index=_internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"

Seeing events like:

INFO Completed retrieval of file data....
WARNING Unable to get the ACL data, reason=(5, 'GetFileSecurity', 'Access is denied.')...
INFO Time is later than filter, st_mtime=1322859631.165719, must_be_later_than=None, path='...
INFO Time is later than filter, st_ctime=1330974351.030764, must_be_later_than=None, path=...

[LukeMurphey]

A few things to look into:

1. Do the "Time is later than filter" logs include the files that you want logs for? This would indicate that the input is skipping the files because it doesn't detect that they have changed.
2. Are you sure that Splunk has access to the files you want it to monitor? The permission errors seen previously might be an indicator that Splunk doesn't have access to the files.
3. You might try disabling the option to only include new results to see if you get the results you want.

[woodcock]

Make sure that NTP is set on both your forwarder (the Windows machine) and the Indexers. Things cannot happen in the future.

[smudge797]

NTP is working.

[aaraneta_splunk]

@smudge797 - Please provide more information and context as to what you need help with as it is not clear. Generally, the more information you provide, the better chance of being answered by experts in the Answers community. Thank you.

[smudge797]

This is the app: https://splunkbase.splunk.com/app/2776/

This is the configured input:

[file_meta_data://ACETestFolder]
file_hash_limit = 500MB
file_path = \\acetest
include_file_hash = 0
interval = 15m
only_if_changed = 1
recurse = 1

There is an initial pull of events seen in the main index, some 3k events
similar to this:
time="Tue Jan 24 16:37:39 2017" is_directory=1 file_count=0 directory_count=16 path=\\acetest atime="Thu Dec 22 12:52:26 2016" atime_epoch=1482429146.82 ctime="Thu Oct 15 16:15:49 2015" ctime_epoch=1444940149.33 dev=0 gid=0 ino=0 mode=16895 mtime="Thu Dec 22 12:52:26 2016" mtime_epoch=1482429146.82 nlink=0 size=4096 uid=0 owner=Administrators\BUILTIN owner...(lots more fields)

All events have same timestamp time="Tue Jan 24 16:37:39 2017" which is correctly indexed in main.

In the index= _internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"
Events like:
2017-01-24 16:37:34,867 INFO Time is later than filter, st_mtime=1482429146.8163483, must_be_later_than=0, path='\\\acetest'

Have second input:

[file_meta_data://fileTest]
file_hash_limit = 500MB
file_path = \\someshare_archive06$
include_file_hash = 0
interval = 15m
only_if_changed = 1
recurse = 1
disabled = 0

No events in main.

index=_internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"

2017-01-24 17:35:24,240 INFO Time is later than filter, st_mtime=1333460403.592, must_be_later_than=0, path="\\\someshare_archive06$\~filedetails.xlsm3.xlsm"

Also lost of these:

2017-01-24 17:29:38,009 ERROR Error when processing path="blah", reason="(1332, 'LookupAccountSid', 'No mapping between account names and security IDs was done.')" Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\file_meta_data\bin\file_meta_data.py", line 381, in get_file_data windows_acl_info = cls.get_windows_acl_data(file_path, logger) File "C:\Program Files\Splunk\etc\apps\file_meta_data\bin\file_meta_data.py", line 254, in get_windows_acl_data sid_resolved = win32security.LookupAccountSid(None, sid) error: (1332, 'LookupAccountSid', 'No mapping between account names and security IDs was done.')

[LukeMurphey]

The only thing I can think of is that something in your environment isn't allowing the account SID to be looked up.

I'm going to make input succeed even if the SID lookup fails.