We've always had a physical load balancer (F5) handling our HEC load balancing duties. But behind that we've tried 2 scenarios. Originally, we were direct to indexers back in the 6.x and 7.x days. When a user would "overlog" this would cause problems on the indexers. The solution was to disable the token in play, but doing so through config files required a restart of the cluster. Likewise, any CRUD to HEC tokens would also require a cluster restart. To address this major issue we installed a set of VMs to sit in front of the indexing tier acting as Heavyweight Forwarders (HWFs). All HEC deliveries happen there, and they forward to the indexing tier. This seemed like a win win. I could perform token management easily without interrupting the much more vital indexing services. And the indexers were isolated from aggressive loggers. However, there were drawbacks lurking under the surface. Performance under heavy load was still not great. Due to the way HWFs "bake" the data, it's actually MORE data being delivered to the indexers. And it bypasses the normal queuing process, jumping ahead. This can cause perf issues not seen in the optimized, normal stream processing of data. And in higher volume scenarios we'd see the indexers queues backing up eventually to the point of impacting HEC queues as well. You also have the "lensing" effect to account for. The HECs act as concentrators, delivering to 1 or 2 indexers at a time (depending on your pipeline count). If many HECs happen to land on one indexer, they could Real Genius -style laser beam it out of existence. Finally, managing another set of .conf files was a not insignificant layer of additional complexity I really didn't need. 8.1.2 brought good news: CRUD operations on HEC inputs no longer trigger a rolling restart on indexers. As this was one of our primary reasons for the HEC tier, it seemed obvious to question the need. After some testing, we removed the HEC tier from one of bigger environments, having the LB deliver direct to indexers. I've seen fewer queue spikes, and literally no negative impact on indexer load, and even hints of exactly the opposite. The environment footprint was reduced by 20-some instances, and complexity is reduced. Now *that* is a win-win. 🙂 (FWIW, for both the HEC farm and the indexing tier, we have persistent queue enabled on every token.)   EDIT: typos ... View more

The regex is simple. You'd have to answer if it's appropriate or not. Any log event with "notice" anywhere in the event will match.   Pattern matching in the `[source::]` qualifier works like it does with inputs. `*` matches anything but file delimiters, and `...` matches anything. Something like this might work    `[source::/folder/folder/logs/firewall-*/*/*/*/local*.log]` ... View more

No. Don't do it. Here's my story. Our COVID19 work from home barage started up and our execs wanted VPN stats pronto. Security guys said they'd point their syslog at where we wanted. I quickly built a VM and threw a UF on it. Next, I set-up a UDP input on 514, and configured the props, indexes, etc. Finally I lit it up, boom! Data coming in. For a few days we worked on reports. Then I started noticing missing events here and there. As I dug in I found duckfez's post on tracking UDP errors. And oh man were there a lot of errors. Like thousands per second. We're definitely dropping events. I set-up rsyslog to receive instead. It's writing to disk, and splunk is reading from there. I also set-up logrotate to clean up cuz these logs are gonna be big. With some tweaking of my props and transforms, I got everything to match the slightly different appearance of the logs. End results with the exact same stream of data being thrown at the server: While using Splunk to receive directly: 2,500 events/sec 10,000 UDP rcv buf errors/sec While using rsyslog to receive, and Splunk reads from disk: 25,000 events/sec 0 UDP rcv buf errors/sec No other changes were made to the host or the log stream being shoved at it. Don't use Splunk to receive syslog. ... View more

According to user-prefs.conf.spec: It also means that values in another app will never be used unless they are exported globally (to system scope) or to the user-prefs app So you need to create the meta file entry to export this setting to system. In appname/metadata/default.meta include this: [user-prefs/role_myrole] export = system Your entry in user-prefs.conf: [role_myrole] default_namespace = appname Will now behave as expected. ... View more