I'm having som issues with the application log on some of our windows servers getting spammed with the following messages:     Faulting application name: splunk-winevtlog.exe, version: 1794.768.23581.39240, time stamp: 0x5c1d9d74 Faulting module name: KERNELBASE.dll, version: 6.3.9600.19724, time stamp: 0x5ec5262a Exception code: 0xeeab5254 Fault offset: 0x0000000000007afc Faulting process id: 0x3258 Faulting application start time: 0x01d787a1d9f141cd Faulting application path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe Faulting module path: C:\Windows\system32\KERNELBASE.dll Report Id: 18687572-f395-11eb-8131-005056b32672 Faulting package full name: Faulting package-relative application ID:       Always followed by a 1001 information event like so:       Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0 Problem signature: P1: splunk-winevtlog.exe P2: 1794.768.23581.39240 P3: 5c1d9d74 P4: KERNELBASE.dll P5: 6.3.9600.19724 P6: 5ec5262a P7: eeab5254 P8: 0000000000007afc P9: P10: Attached files: These files may be available here: C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_splunk-winevtlog_32b957db7bcb27fbdcdd5be64aea86e1b639666_0170a0ed_a993dd7e Analysis symbol: Rechecking for solution: 0 Report Id: 18687572-f395-11eb-8131-005056b32672 Report Status: 4100 Hashed bucket:       I've tried a lot of changes to the Universal Forwarder configuration but nothing i do removes these message. The only thing i've noticed that can helt to remove these messages is by lowering the memory consumption on the server. So far the servers i've seen with these message in the application log are running at 70% and more memory consumption. But 70% memory consumption seems to be normal and i don't see why this should cause the splunk-winevtlog.exe to crash (as often as every minute).   Our version of Splunk Universal Forwarder is 7.2.3. I've checked the "known issues" on splunk docs but can't fint anything related to memory issues for this version. I'm thinking about upgrading the Universal Forwarder to a newer version, but that's just because i can't think og anything else to try. Do anyone else experience this and know what can be done? As a side note: Splunk internal shows absolutely nothing. There are no warnings or errors at all in the internal log on these servers. But the event spamming (crashes) are still logged in the windows application log. Splunk itself does not log or detect a crash it seems? ... View more

I'm having som issues with how Splunk is handling event breaking for CSV files. A sample of the CSV file in question: USER;SYSTEMINFO;DATE;RT;VT;MESSAGE USER1;Windows 10 Enterprise;20201128 06:05:27,862;5;0;"File not accessible ("90s"). Error: "Try again"." USER2;Windows 10 Enterprise;20201128 06:05:29,288;15.9;5;"" USER3;Windows 10 Enterprise;20201128 06:15:25,463;5;0;"File not accessible ("90s"). Error: "Try again"." USER4;Windows 10 Enterprise;20201128 06:15:26,830;21.3;0;""   In props.conf i have tried the following (this is a single instance installation of splunk with no additional forwarder and the input is local in this instance of Splunk): [MyCSV] FIELD_DELIMITER = ; FIELD_QUOTE = " HEADER_FIELD_DELIMITER = ; HEADER_FIELD_QUOTE = None SHOULD_LINEMERGE = false KV_MODE = none LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Structured description = CSV input disabled = false pulldown_type = true   This works perfect in the cases where MESSAGE contains two double quotes. in the cases (like the example i provided) where the MESSAGE field contains multiple double quotes Splunk can't seem to break the event properly. One event would end up like this: USER1;Windows 10 Enterprise;20201128 06:05:27,862;5;0;"File not accessible ("90s"). Error: "Try again"." USER2;Windows 10 Enterprise;20201128 06:05:29,288;15.9;5;"" USER3;Windows 10 Enterprise;20201128 06:15:25,463;5;0;"File not accessible ("90s"). Error: "Try again"." But i would expect that to be broken into three separate events. From what i can gather Splunk has issues with the multiple double quotation marks and completely ignores the line_breaker regex. If i change FIELD_QUOTE = " and instead put FIELD_QUOTE = None it does seem to break the events like it should, but the auto-extracted MESSAGE field will then contain quotation marks. With FIELD_QUOTE=" Splunk removes the double quotes around the event but fails to break the event properly. I have also tried to change the double quotes inside the outer double quotes with SEDCMD like this: SEDCMD-removeinnerquotes = s/(?<!;)"(?![\r\n]|$)/'/g This is working so the indexed MESSAGE field is changed from:  "File not accessible ("90s"). Error: "Try again"." to: "File not accessible ('90s'). Error: 'Try again'." However: Splunk is still auto extracting the MESSAGE field BEFORE my SEDCMD kicks in. So one event will now look like this: USER1;Windows 10 Enterprise;20201128 06:05:27,862;5;0;"File not accessible ('90s'). Error: 'Try again'." USER2;Windows 10 Enterprise;20201128 06:05:29,288;15.9;5;"" USER3;Windows 10 Enterprise;20201128 06:15:25,463;5;0;"File not accessible ('90s'). Error: 'Try again'."   And the MESSAGE field in the left side meny under "Interesting fields" will have the value like this: File not accessible ("90s"). Error: "Try again".   So because this is not working i tried to remove all double quotes with SEDCMD like this: SEDCMD-removequotes = s/(?<!;)"(?![\r\n]|$)/'/g s/"//g FIELD_QUOTE=None   Causing event breaking to be done correctly (because of FIELD_QUOTE = None i suspect) and one event will now be correctly: USER1;Windows 10 Enterprise;20201128 06:05:27,862;5;0;File not accessible ('90s'). Error: 'Try again'. HOWEVER: The MESSAGE field on the left hand side under "Interesting fields" still contains double quotes. Despite the event above where all double quotes is either changed or removed, the MESSAGE field will have the value like this: "File not accessible ("90s"). Error: "Try again"."   - This seem to indicate that whatever i try Splunk will extract the field before any change i do and therefore make it near impossible to fix this issue. I have not set INDEXED_EXTRACTION = CSV and have left KV_MODE = none to try to keep Splunk from extracting fields at both indexing and search time, but it still extracts the fields. My guess is that this has to do with the header. As long as Splunk is looking for and using the header in the CSV file it will also auto-extract the fields. I have not found a way to change this behavior.  In my desperation i also tried to write a transforms stanza to remove the header, then remove the HEADER_FIELD_DELIMITER and HEADER_FIELD_QUOTE settings,  but STILL Splunk is extracting the header and the fields that goes with them.  There is the HEADER_FIELD_LINE_NUMBER setting, but this can't be set to "none" and defaults to 0. Even if i set this to HEADER_FIELD_LINE_NUMBER = 1 and have the transforms to remove the header, Splunk extracts the HEADER fields automatically.  Does anyone have any tips for fixing this issue? What i want is the events to be broken down correctly with 1 event per line and to have the MESSAGE field values without quotation marks. The obvious and best solution would be for the system to change it's logger and never include multiple double quotation marks in the MESSAGE text, but this is not possible i'm afraid, so i'm stuck trying to find a solution in Splunk. Note again that this is a single instance Splunk Enterprise installation with no forwarder (local input). ... View more

I'm having some issues when trying to share KO (field extractions) with other roles and users. I have field extractions which reside in props.conf and transforms.conf for App A. What I want to do is share these extractions with role B. Role B only have access to an app called App B. So the question: Is it possible to share the field extraction from app A with role B, so that when role B does a search in app B they will see the field extractions? I have tried the following: Settings -> Fields -> Field extractions -> found the correct props stanza for the field extractions and checked "all apps" and given read permission for role B. Same procedure for "field transformations" and the corresponding transforms stanza. When a user with role B logs in they still can't see the fields even though the extraction was set to "global" and they have read access to them. Local.meta then looks something like this: [transforms/REPORT-csv_A] access = read : [ admin, B ], write : [ admin ] export = system owner = admin [props/somesourcetype/REPORT-csv_A] access = read : [ admin, B ], write : [ admin ] export = system owner = admin [] access = read : [ admin ], write : [ admin ] export = none Then I tried to share "all" KO's, and not only the field extractions by editing app A and selecting "All apps" for "Set permissions for configurations that have been copied over or added to config files rather than created through the UI. Objects defined in config files only (not in the UI) should appear in" Resulting in the following change in the local.meta: [] access = read : [ admin ], write : [ admin ] export = system And this actually works. Having shared ALL KO's a user with role B can search in app B and get the field extractions from app A. So I'm struggling to understand why the sharing of only filed extraction is not working? Does anyone have any ideas as to what is happening? Have I forgotten something? I might also add that the default.meta in app A looks like this: [app/ui] version = 7.2.3 modtime = x [app/launcher] version = 7.2.3 modtime = x [] access = read : [ * ], write : [ admin ] ### EVENT TYPES [eventtypes] export = system ### PROPS [props] export = system ### TRANSFORMS [transforms] export = system ### LOOKUPS [lookups] export = system ### VIEWSTATES: even normal users should be able to create shared viewstates [viewstates] access = read : [ * ], write : [ * ] export = system And that giving "read" access to app A for role B also works. Role B can then search in app B and still get all the field extractions from app A. The problem with this one is that app A is also visible to role B, which I do not want. One more small detail: The sourcetype used is defined in app A default/props.conf. Could this be why it is working when sharing ALL KO's (also sharing the sourcetype in default)? ... View more