About MichalG1

MichalG1

Hello Team, Deployment with: - HF with ACK when sending to Indexer - HEC on HF with ACK - application sending events via HEC on HF with ACK Still in this model there is a chance that some of the events will be missed. Application might get ACK from HEC, but if the event is still on the HF output queue (not yet sent to the indexer) and we have non-gracefull reboot of HF (so that it could not flush out it's output queue). Can you confirm ? What would be the best way to address it ? So that once the application receives ACK we do have end to end guarantee that event is indexed ? Thanks, Michal

MichalG1 · ‎03-31-2024

Thanks a lot @richgalloway That behavior of SH seems to be unnecessarily complicated. Instead of sending all of those KO bundles to indexers, could not SH first expand SPL query (to resolve all of the names/variables which are search-time) and then sent it to indexers ? Thanks, Michal

MichalG1 · ‎03-31-2024

Thanks @richgalloway So just to confirm: "To know what results to return to the SH, the peers need to know the values of the tags, eventtypes, and macros used in the query. " Example: "index=_audit eventtype=splunk_access". Since event type extraction is search-time (not index-time) indexer does not have definition for that event type. Because of this SH need to push to indexer definition for that event type: [splunk_access] search = index=_audit "action=login attempt" NOT "action=search" Once that is done, indexer will actually expand original SQL query to "index=_audit index=_audit action=login attempt NOT action=search" and will be able to execute the query correctly. The same would happen with most of the other Knowledge Objects. Including all the search time field extractions. So the summary would be: Search Head needs to push Knowledge Objects to indexer, because for indexer those are "unknown variables/names". Indexer does not have those definitions and does not know how to expand/execute SQL queries using those KOs. This is applicable only to search-time operations/objects defined on SH (index-time related configurations like TRANSFORMS should be already on the indexer). Could you please confirm @richgalloway all of this is correct ? Thanks !

MichalG1 · ‎03-29-2024

Hello Team, As per https://docs.splunk.com/Documentation/Splunk/9.2.0/DistSearch/Knowledgebundlereplication "The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf" Or "The knowledge bundle consists of a set of files that the search peers ordinarily need in order to perform their searches" Could you please give me one example why we really need it ? I had the impression that to return search results to SH indexer just need SPL query and it's locally indexed data + metadata. One of my guesses for a good example were: lookup files, but i guess indexer should not need any lookup files since that job is done be search head, not indexer. The same with other KO objects like tags, event types, macros etc...-> those objects should not be needed on the indexer to perform search, those are used by search head to enrich data returned by the indexer. Another theory: we distribute those files not to help with searching, but with parsing and indexing (for example using props.conf and transforms.conf). Maybe that is the case ? Extra question: the conf files delivered in the bundle: if i do understand correctly those settings are in memory only, not modifying any existing conf files on the indexer ? But at the same modifying memory settings for (for example) index.conf ? If so - i should be able to run "splunk btool indexes list" to see something different then "splunk show" -> to compare the diff between current configuration files versus those sent from bundle and applied in memory ? What are the best practices here ? What am i missing ? Thanks, Michal

MichalG1 · ‎10-15-2023

I can login there (using my instructor account), but see 0 events (both public and invited), any way to get the access there ? Thanks @gcusello !

MichalG1 · ‎10-15-2023

Looks great, will test it, thx !

MichalG1 · ‎10-15-2023

Hello Team, Pre staging environment (not production), a single server with 12 CPU + 24 GB or memory + raid0 nvme (2.5GB/s write, 5GB/s read). All in one deployment (SH + indexer). CPU cores with HT on dedicated server (6 cores with HT = 12 CPU -> but not used by any other VM). Splunk 9.1.1 and ES 7.1.1. Fresh install. NO data ingested (0 events in most of the indexes including main, notable, risk etc...) - so basically no data yet to be processed. Default ES configuration, i have not yet tuned any correlation searches etc. Defaults. And already performance problems: 1. MC Scheduler Activity Instance showing 22% skipped. 2. ESX reporting minimal CPU usage (the same with memory): 3. MC showing more details, many different Accelerated DM tasks are skipped, all the time: Questions: 1. obviously the first recommendation would be to disable many of correlation searches/accelerated DMs, but that not what i would like do because the aim is to test complete ES functionality (by generating a small number of different types of events). Why do i have those problems in a first place ? I can see that all the tasks are very short, finishes in 1 second, just few takes several seconds. And that is expected since i have 0 events everywhere and i do always expect to have a small number of events on this test deployment. What should i do to tune it and make sure there are no problems with skipped jobs ? Shall i increase max_searches_per_cpu base_max_searches Any other ideas ? Overall that seems weird,

MichalG1 · ‎10-15-2023

Hello Team, I have my training Splunk instance with ES 7.1. I have used it for training and experiments. Question: is there any "training/example" data i could use to import it there ? The point is: i do not want to configure dozens of TA's/collectors, create real labs with real cybersecurity attacks, instead would love to have a test data so i could learn/experiment/review Splunk ES capabilities. Anything ?

Posts	8
Solutions	0
Karma Given	4
Karma Received	0
Member Since	‎04-14-2023

Online Status	Offline
Date Last Visited	2 weeks ago

Lost events on HF HEC

Why indexer really needs knowledge objects from SH...

Hitting limit: maximum number of concurrent auto-s...

Example data/incidents for ES