About stranjer

stranjer · ‎02-03-2021

Disk Quota Limits, Search API Endpoint Differences and Parameters Looking for better clarity and deeper understanding to better solve a recurring issue I'm seeing. We have a script performing searches using the API. Currently, the flow works like this: Start search by doing POST /services/search/jobs with search as parameter, get back search id (sid). Run loop to do GET /services/search/jobs/{sid} and check the search job status until done Pull back in results of search by doing GET /services/search/jobs/{sid}/results After results are pulled back, do POST /services/search/jobs/{sid}/control to cancel search and delete the result cache The script is designed to not try to run more than a few searches at a time, and to wait until earlier searches have been canceled to start the next search. However, we are still sometimes hitting the search disk quota limitation. We've increased this limit a few times well past the default, which has reduced frequency but issue still comes up. We do NOT want to change this to unlimited, nor keep increasing it every time it gets hit.There are a few questions I'm not able to find documentation on when trying to figure out solutions: Is there normally delay after a search has successfully been cancelled before the results cache is removed? Or somehow until the disk usage quota is updated to reflect the cleared space? Would doing a DELETE /services/search/jobs/{sid} clear up space quicker? Would switching to using the /services/search/jobs/export endpoint help? If the results are streaming, do they also still persist on disk? The Python & Java SDK docs say export searches '...return results in a stream, rather than as a search job that is saved on the server.' But I'm not sure that means the result cache isn't saved. Does setting a low 'timeout' value in the search/jobs parameter clear the disk space after that value has passed? With the 'auto_cancel' parameter what counts as 'inactivity'? checking the status of the SID? retrieving results? If accidently set to do 'search index=* ' for all time, does this stop it before completion? ( I assume so, but wanted confirmation ) The documentation is unclear on what some phases mean (like 'inactivity', or 'rather than saved on the server' in the SDK docs ), and some other parts are likely simplifications/abstracts of concepts I need to understand more in-depth (cancelling/deleting jobs, clearing disk space). Trying to avoid just putting band-aids on a bullet wound, but need more details to determine the right treatment.

stranjer · ‎05-04-2020

I'm trying to use the REST API to update a large number of alerts/saved searches across multiple environments. Specifically, I want to update to ensure that CSV's are attached to emails. I'm testing this in a lab on Splunk 8.0.2. I've tried using both of these: 'curl -u splunk_user:splunk_pass -X POST -sk "https://splunk_ip:8089/services/saved/searches/(search title URL encoded)" -d action.email.sendcsv=1' 'curl -u splunk_user:splunk_pass -X POST -sk "https://splunk_ip:8089/servicesNS/(owner)/(app)/saved/searches/(search title URL encoded)" -d action.email.sendcsv=1' However, what seems to be happening is that the alerts are being cloned into a report instead of being updated themselves. The name is identical, and the search is being copied over despite not being in the curl request. Not sure what is going wrong with these API calls. They look correct by my reading of the API documentation, but I may be overlooking something.

stranjer · ‎03-01-2019

Thanks for the response. I double checked to confirm, but the retention policy/searchable indexes/searchable terms changes all don't fit as I can see the search terms returning similar events from both indexes prior to the missing one. I have tried a few searches for deleted events (index=_audit command="delete", and some others I found in:https://answers.splunk.com/answers/526983/how-can-i-find-what-users-deleted-specific-events.html) and I'm not seeing any hits that look like anything was deleted from either index. If there was a specific query you'd suggest or have had success with, I'm willing to try it. I'm not that familiar with what an event deleted by the system or a user would look like or what fields it'd have. For timestamps, I checked the specific host+user+group combo for +/- 2 days, and the notable index for the saved search name for +/- 2 days for anything that matched username, with no luck.

stranjer · ‎02-27-2019

Thanks for the responses lakshman239 and tiagofbmm. Here is some additional information. I see notable events using the macro, and just searching the index. I see them from before and after the missing notable event, both in general and for the specific alert. The KV store is running and shows no error in status. The notable index and Windows index have longer retention policies setup, are not full, and have events prior to the ones that are missing. Indexer cluster shows no issues with search or replication issues. I don't seen the original events that would have triggered the alert in Splunk, which is part of the issue I'm trying to understand. The alert should trigger off of Windows events for group additions. I can see the following info: - In Splunk, I see Windows events from the specific host, where other users were added to the specific group, before and after. - In Splunk, I see Windows events from the specific host, where the specific user was added to other groups, before and after. - The notification email sent does not have details on the event configured, it just sent in the alert name. However, the person originally looking described a specific user being added to a specific group on a specific host at a specific time based on what they were seeing in Splunk. They did not have access to the Windows host. - On the local host itself, we are able to confirm that the specific event happened with the details the other person originally saw in Splunk, at the time they mentioned. - The day after making those notes, neither the original user nor anyone else could find the notable event or the Window event that generated it. The information appears to have been in Splunk, but is now missing. Original person described details they couldn't have known without it being available in Splunk, and those details were verified so its not like they were just mistaken.

stranjer · ‎02-27-2019

Thanks for your response - When I run notable I am seeing notable events. If I look for the particular search, I am seeing events from that search, on days before and days after the missing notable event. If search for the original Windows events that caused the alert, I do not see them in the Windows Index related to the host. KV store status comes back good, no issues seen there. To make things clearer, the alert is for when users are added to particular groups. A user was added to a group that matches the criteria on the host, and an alert triggered. I see: * Splunk showing group add events from the specific host, before and after the event in question * Splunk showing the specific user being added to other groups, before and after * Splunk showing other users being added to the specific group, on the specific host, both before and after * Local Windows event on the host showing the specific user being added to the specific group * Notable events for other instances where the alert was triggered for other cases of the activity, before and after. * Email notification sent and reviewed. Person working noted the specific user being added to the specific group on the specific host, but none of this was in the email notification (the email info is pretty bland, just the alert name, just to generate ticket). What I am not seeing as of now * Splunk showing the Windows Event showing the specific user being added to the specific group * Splunk showing the alert that triggered in the Notable Events The person who worked the alert had covered details that were not in the notification email about the event that wouldn't have been possible to know unless the event was in Splunk, but later neither them nor anyone else could find the event again in Splunk. So it looks like the event was there, then vanished.

stranjer · ‎02-27-2019

We have an alert that we had setup to create a notable event and email a notification when a particular Windows Event occurs. A few weeks ago, we received an email of the event, and originally saw the event in the Splunk environment, and verified the event occurred on the Windows host referenced. However, when we went to review the event later, we can no longer find that event or the Notable Event that should have been generated when the activity occurred. All search peers are up and search and replication factors are being met Reran the original alert search logic to confirm, still not seeing the exact event that triggered it, but nothing in search logic exclude it Tried to search for just the artifacts I know occurred, Event ID and host involved. Checked Index retention settings, set for 1 year and I'm seeing plenty of space in the index and similar events from earlier, so shouldn't have rolled. Looked through Notable Index around the time for the specific alert that triggered, and did not see the event. I did searches for the event over multiple days range thinking there might be time difference issues, did not find events that matched and when we looked for the event on the host with Event Viewer, it doesn't look like there was much of a delay from when it occurred locally to when Splunk indexed it and sent an email. I could understand if a search peer was down, there were replication issues, index got full and was rolling to frozen, or the event just never came into the indexers, but none of those appear to be the case and I'm not sure what other explanations could be. Any suggestions on what to look for would be welcome.

Posts	6
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎02-27-2019

Online Status	Offline
Date Last Visited	‎02-05-2021 06:49 PM

Disk Quota Limits, Search API Endpoint Differences...

REST API Saved Searches POST Duplicating Searches

Events now Missing from Regular/Notable Index

Disk Quota Limits, Search API Endpoint Differences...

REST API Saved Searches POST Duplicating Searches

Re: Events now Missing from Regular/Notable Index

Re: Events now Missing from Regular/Notable Index

Re: Events now Missing from Regular/Notable Index

Events now Missing from Regular/Notable Index