About BenjaminWyatt

BenjaminWyatt · ‎11-20-2022

Hi all, Here is the use case I'm dealing with. We have a large virtual environment in which a lot of teams like to just clone one VM to another, meaning that the forwarder hostname and guid gets cloned, which messes with our reporting. I am trying to write a simple script that does the following: 1. Detects if a UF's hostname is correct or not 2. Runs a simple scripted input to clear out any cloned configs 3. Restarts the forwarder so that the new configs are picked up. #3 is causing me trouble. If I try to put a "splunk restart" command in the main body of the script, then Splunk will stop, kill the scripted input, and never restart. I've also tried creating a "wrapper" script that invokes a separate script to do the restart, but with no success - Splunk will stop but not start back up. Is there a better way to do this? All hosts are AWS Linux.

BenjaminWyatt · ‎09-21-2020

Yes, that's the one - sounds like that won't be a possibility then. Out of curiosity, what is the default ordering for statuses in the dropdown? It looks like they are listed by order of their status ID, but it would be good to confirm.

BenjaminWyatt · ‎09-15-2020

Hi everyone, I have a request from our security team to reorder our notable event statuses in the dropdown. We have a lot of custom statuses that supplant the "In Progress" label, and at present they all appear at the very end of the dropdown list, which isn't terribly convenient for our analysts. I can see how to edit the status names, but I can't find a way to change the order the appear in the dropdown. Is there a way to do this?

BenjaminWyatt · ‎05-05-2018

I agree that it's a suspicious multiple. Upload times are in ms as far as I can see...this is one of the most heavily taxed databases in the environment, so it's going to be a bit higher than one would like. We have confirmed that Splunk is not reading the appropriate amount of records. We are missing entries.

BenjaminWyatt · ‎05-04-2018

I have a DBX 3.1.2 job that's failing at some point along the way. I don't get any error messages (everything is set to DEBUG levels), just the following message in the metrics logs: 2018-05-03 12:06:37.976 -0400 INFO c.s.dbx.server.task.listeners.JobMetricsListener - action=collect_job_metrics connection=my_db_connection jdbc_url=null record_read_success_count=3444 db_read_time=397794 record_read_error_count=1 hec_upload_time=102 hec_record_process_time=13 format_hec_success_count=3444 hec_upload_bytes=1631645 status=FAILED input_name=my_db_input batch_size=1000 error_threshold=N/A is_jmx_monitoring=false start_time=2018-05-03_12:00:00 end_time=2018-05-03_12:06:37 duration=397965 read_count=3444 write_count=3000 filtered_count=0 error_count=0 As you can see, not everything in the read_count field is making it into the write_count field. But when I search for error messages related to this input, I don't get anything beyond this. Has anybody else had this problem? Where did you look?

BenjaminWyatt · ‎12-31-2015

Done. See above.

BenjaminWyatt · ‎12-29-2015

Thanks. So if it fails to move a bucket, will the process continue attempting to freeze other buckets? As stated above, we're seeing our entire colddb filled with "frozen" buckets.

BenjaminWyatt · ‎12-29-2015

Hi all, I've noticed some weird behavior on one (and only one) of my indexers. A customer complained about data "suddenly disappearing" in the middle of the day. When my team investigated, they found that all the buckets in cold resembled "frozen" buckets - they contain only a rawdata directory with journal.gz inside. When I search for a representative bucket in the logs, I see only the following two messages, repeated over and over again: 12-30-2015 00:13:53.533 +0900 INFO DbMaxSizeManager - Will freeze bucket=/opt/splunk/var/lib/splunk/jcare/colddb/db_1449606229_1449572203_26083 LT=1449606229 size=30715904 (29MB) 12-30-2015 00:13:55.068 +0900 ERROR BucketMover - aborting move because recursive copy from src='/opt/splunk/var/lib/splunk/jcare/colddb/db_1449606229_1449572203_26083' to dst='/opt/splunk/var/lib/splunk/frozen/jcare/inflight-db_1449606229_1449572203_26083' failed (reason='No space left on device') The disk space error referenced is due to a known problem with our backups client that we're working on. However, that still leaves me with two questions: If Splunk fails to move a bucket to frozen, why does it continue freezing other buckets in cold? Shouldn't it just stop altogether? If Splunk fails to move a bucket to frozen, why does it leave the bucket in "frozen" status on cold? My understanding was that the bucket was first moved, then frozen. Curious if anyone else has seen this and has suggestions as to why this is happening. Again, the root cause appears to be disk-related, but we are dependent upon a support ticket with our Backups vendor to resolve this, and I'd love to see any ameliorative suggestions that we can implement in the meantime. Adding relevant lines from indexes.conf here: [default] maxTotalDataSizeMB = 1272700 [volume:tstats] path = /opt/splunk/var/lib/splunk/tstats [volume:ctx_hot] path = /opt/splunk/var/lib/splunk/ctx_hot maxVolumeDataSizeMB = 36000 [volume:ctx_cold] path = /opt/splunk/var/lib/splunk/ctx_cold maxVolumeDataSizeMB = 74000 [jcare] homePath = $SPLUNK_DB/jcare/db coldPath = $SPLUNK_DB/jcare/colddb thawedPath = $SPLUNK_DB/jcare/thaweddb tstatsHomePath = volume:tstats/jcare coldToFrozenDir = $SPLUNK_DB/frozen/jcare homePath.maxDataSizeMB = 185000 coldPath.maxDataSizeMB = 237000 maxHotBuckets = 3 maxDataSize = auto

BenjaminWyatt · ‎11-27-2015

I'm trying to set up a scraper using the Website Inputs app. For this particular web page, the data I need is contained in a "pre" HTML element. For some reason, when I try to use this as my CSS selector, I get a "no matches found" message. Is there a bug with this particular element? I'm happy to provide more details about the webpage I'm trying to scrape if that makes things easier.

BenjaminWyatt · ‎03-18-2015

Based on my reading of the inputs.conf spec file, you could do this if you wanted. However, why not control this by specifying the default routing group in outputs.conf instead? This seems to be the more natural way to create a "default" forwarding group.

BenjaminWyatt · ‎09-11-2014

Hi all, I'm working on building a few force-directed graphs for some use cases I have. I'm using the Splunk Web Framework Toolkit (with the components ported over to Enterprise Security) in order to build the visualizations. I've followed the tutorials located here: http://dev.splunk.com/view/webframework-htmldashboards/SP-CAAAEV6, and I've been trying to adapt them to cover my force-directed graph. However, it seems like no matter what I try, I can't get the page to load properly! I've tried looking at the example provided in the Web Framework Toolkit, but it doesn't seem like it will actually let me dig into the underlying HTML - my old trick of adding "?showsource=1" to the end of the URL is failing, and I can't find any sort of a base view for that example anywhere within Splunk. I also don't have a lot of web development experience, which I know is part of the problem, but I also feel like I don't have any good way to get started building these dashboards. Does anybody have a working example of a force-directed graph template that they'd be willing to share? Or, if examples of working HTML5 code are available somewhere and I've missed them, would someone mind pointing me in the right direction? Thanks so much!

BenjaminWyatt · ‎07-17-2014

Perfect, the problem fixed itself. Thanks!

BenjaminWyatt · ‎07-17-2014

Also, we are currently using v10.1 of the Netscaler.

BenjaminWyatt · ‎07-16-2014

I just installed the components needed to use the Splunk App for Citrix Netscaler with AppFlow. The ns_log sourcetype shows up just fine, but for the appflow sourcetype, all I see are logs with this message: TimeStamp="2014-07-16T21:00:04"; Template="264"; Observer="1"; Address="10.2.41.254"; Port="2203"; ParseError="Template not known (yet)."; Does anybody know what this means? I can't seem to find information on it in the docs...

BenjaminWyatt · ‎05-08-2014

Oh, whoops, formatting error. What I meant to type was: mysearch NOT refRepId=* (refrepid=* OR refRepid=*) I'm not sure I understand it needing to be case-insensitive. Field names are case-sensitive in the search string...

BenjaminWyatt · ‎05-08-2014

What about the fixedrange option for timechart? From the docs page (http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Timechart?r=searchtip) fixedrange Syntax: fixedrange= Description: (Not valid for 4.2) Specify whether or not to enforce the earliest and latest times of the search. Setting it to false allows the timechart to constrict to just the time range with valid data. Default is True | T. I know in the latest version of Splunk this defaults to True, but if you're not on the latest version of Splunk, then it might not be set to "true" by default. I've had similar issues, and this has fixed the issue.

BenjaminWyatt · ‎05-08-2014

Hmmm...what about something like... mysearch NOT refRepId=* (refrepid=* OR refRepid=*) It's not the most elegant solution, but you will filter out all the values with the casing you don't want, and enforce that the two casings you do want have values in them.

BenjaminWyatt · ‎04-17-2014

So it looks like I can see results now, but the searches are taking far longer than they should with data model acceleration - is there a "quick and easy" way to diagnose and troubleshoot issues with data model acceleration?

BenjaminWyatt · ‎04-16-2014

We recently upgraded our Enterprise Security instance to v3.0 from v2.4. After the upgrade, I noticed that Correlation Searches relying on TSIDX instances (that is, searches that begin with a command of the form "| tstats count FROM datamodel=X") do not appear to be functioning properly. I can verify that the data models are building properly, and when I try to return the same data with a "| pivot" command, it works just fine. I'd like to know why this is happening, and see if we can remediate it...I know I can convert all my correlation searches to "| pivot" commands, but I'm hoping there's a better way. If it makes any difference, we have changed the default home directory of the data model summaries; instead of living in /opt/splunk/var/lib/splunk/index-name, they live in /opt/splunk/var/lib/splunk/index-name/db. I first noticed this problem after the change, so that may have something to do with it...

BenjaminWyatt · ‎04-15-2014

Bump. Has anybody ever figured this out?

BenjaminWyatt · ‎03-04-2014

General Splunk guidelines state there isn't much benefit to splitting out the search head/indexer functions into separate servers until you have about 4 indexers. You should be fine as you are.

BenjaminWyatt · ‎01-20-2014

From the config files for the TAs: [monitor://C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\MessageTracking] whitelist=.log$|.LOG$ sourcetype=MSExchange:2010:MessageTracking queue=parsingQueue index=msexchange disabled=false [monitor://C:\inetpub\logs\LogFiles\W3SVC1*.log] sourcetype=MSWindows:2008R2:IIS queue=parsingQueue index=msexchange disabled=false ignoreOlderThan = 24h crcSalt = _TCP_ROUTING = New_Splunk_Indexers So Message Tracking and IIS logs are the primary log types. There is also a Powershell script that collects admin logs (if you have those enabled in your environment). Note that other statistics are collected through perfmon and scripted inputs, so your total consumed volume will be higher than that. Compression ratio depends entirely on your environment - Splunk will typically tell you to expect an average compression ratio of 0.5, but this all depends on your data. In our environment, it's typically about twice as good as that; that is, we see compression ratios from 0.25.

BenjaminWyatt · ‎12-10-2013

To add my own two cents: I think the issue for me is simply that it's annoying to have perfmon data residing in different indexes. Given that there aren't really any security constraints around perfmon data, and whatever data retention policies exist are probably going to be standard across most of the servers you are collecting perfmon data from, there just doesn't seem to be a good reason to have them sent to different indexes "out of the box". I grant you that this is easy enough to fix at the app level - it just seems like an inelegant design to have to work with.

BenjaminWyatt · ‎12-09-2013

I figured out the problem. Posting my answer here in case it's helpful to others. Essentially, it was an issue where the script was working, but for various reasons it looked kind of broken. First of all, the script doesn't appear to pull historical alert data (you only get data from when you first turn on the script). This made me think that the script wasn't working, because I was expecting to get historical data. Second, because the default polling interval is 15 seconds, the calls to the IPS API frequently return no alerts. Now, when you just hit the API in your browser, you get a ton of XML output, which includes alerts, healtchecks, etc. The script tosses the healthcheck information away and just keeps the alerts. This isn't a problem in and of itself, but there is no documentation in the app or logging in the script to indicate that is the expected behavior. So we were comparing what we saw in Splunk to what was in the webpage when we saw it in our browser, and that made us think that we were having a problem with the collection. We didn't figure out what was going on until we forced the script to log the raw XML output. Once we figured out that the script was just filtering out all the healthcheck data, we let the script run for a few hours and monitored for events - sure enough, they started showing up. To any Splunk folks reading this - some official documentation to walk us through how the app worked, and what we could have expected in terms of data volume, would have been really helpful and saved us a considerable amount of time and frustration.

BenjaminWyatt · ‎12-05-2013

We are currently attempting to use the Splunk for Cisco IPS App (http://apps.splunk.com/app/528/) to pull data from our IPS devices into Splunk. However, we have run into the following problem: while the script successfully authenticates against the IPS device, it does not seem to pull any data from the /cgi-bin/sdee-server endpoint. When I hit this endpoint in my browser (using the same credentials as the script is using), I see the log data, so I know it exists and is accessible by this account. Nor are any exceptions raised in the logs; it appears the script is simply returning no data from the web request. Any thoughts on what might be causing this problem?

Posts	68
Solutions	7
Karma Given	13
Karma Received	24
Member Since	‎07-18-2012

Online Status	Offline
Date Last Visited	‎11-21-2022 10:22 AM

Restarting Splunk with Scripted Input?

Customize Notable Status Order in Dropdown

Why does the DBX read count and write count doesn'...

Why are we seeing Frozen Buckets in Cold?

Web-Page Inputs App: CSS selector not working for ...

Does anybody have a working example of a force-dir...

Citrix Netscaler - Appflow - Template not known (y...

Splunk Enterprise Security - TSIDX-dependent Corre...

Splunk Cisco IPS App - no data being pulled?

Find changes that require a restart of Splunk?

Restarting Splunk with Scripted Input?

Re: Customize Notable Status Order in Dropdown

Customize Notable Status Order in Dropdown

Re: Why does the DBX read count and write count do...

Why does the DBX read count and write count doesn'...

Re: Why are we seeing Frozen Buckets in Cold?

Re: Why are we seeing Frozen Buckets in Cold?

Why are we seeing Frozen Buckets in Cold?

Web-Page Inputs App: CSS selector not working for ...

Re: Can I set _TCP_ROUTING in the default stanza f...

Does anybody have a working example of a force-dir...

Re: Citrix Netscaler - Appflow - Template not know...

Re: Citrix Netscaler - Appflow - Template not know...

Citrix Netscaler - Appflow - Template not known (y...

Re: Splunk Case-Sensitive Search

Re: Min/Max of Time Axis on Timechart Does Not Alw...

Re: Splunk Case-Sensitive Search

Re: Splunk Enterprise Security - TSIDX-dependent C...

Splunk Enterprise Security - TSIDX-dependent Corre...

Re: WARN LMTracker - skipping directive -> Any id...

Re: If we have a single indexer, is there any adva...

Re: Exchange 2007 and 2010 which log files are col...

Re: Why don't the splunk for windows and splunk fo...

Re: Splunk Cisco IPS App - no data being pulled?

Splunk Cisco IPS App - no data being pulled?