About laytonj76

laytonj76 · ‎07-07-2017

I'm seeing the same behavior; however, I'm running 6.6.0. Was your issue resolved in time as tfletcher suggested?

laytonj76 · ‎05-08-2017

My customer has a dashboard that used to display 90 or so events in a bar chart with x-axis labels displayed for all. We just upgraded from 6.2.2 to 6.5.2 and are unable to see labels when rendering more than 60 or so events. From what I've read in answers, and what I've witnessed using different browsers on different systems, it seems to be browser or system-dependent. For example, in one environment I could see labels when I limited returned results to 62 (using 'head 62' for debug purposes). In another browser I could see labels when limiting results to 63 ... not a significant delta, but enough to believe it's a configuration external to Splunk. Is there anything that you've seen or implemented that we can do to force the labels to display?

laytonj76 · ‎07-18-2016

Have you confirmed successful forwarding from that forwarder before? Also, I don't recall off the top of my head, but if you modified inputs.conf you may need to restart splunk for the change to become effective. The docs should tell you which changes require a restart.

laytonj76 · ‎07-18-2016

Admittedly, this is a hack and we haven't identified the root cause but it got us going and got Splunk back to a healthy status. We had an issue where Splunk appeared to be trying to write to an index directory that no longer existed. The index still existed, but the bucket was removed (maybe retention configuration rolled it off before Splunk processes could complete its tasks). We determined this by running 'htop' on the indexer and identifying the process id of the splunkd that was spiking so high. Then, we ran an 'strace' on that process, wrote the output to a local file, and found the error. Lastly, we created the directory it couldn't find and it resolved itself. Again, a hack, but we're still working to identify the root cause. So, more specifically, we logged onto the indexer that was congested (a Linux box) and executed the following steps: Execute 'htop' command Identify the process id of the splunk process that's 'spiking' Execute 'strace process_id > strace.out' (NOTE: process_id is the process id identified in the previous step and strace.out is just a random filename to store the output) You will see some limited action, but let this run for 5 seconds or so and then kill it (ctrl-C) Open strace.out and grep for 'directory or file not found' (or something to that affect) Once you've identified the directory, make the directory it couldn't find Execute 'htop' If that was the only directory it couldn't find, the processing utilization for the splunk process(es) should decrease

laytonj76 · ‎02-16-2016

Thanks for the response. I forgot to mention the boxes in question will be Indexers. We used the sizing app and that was very helpful. Our question comes up as the sizing app doesn't have anything that would indicate whether there's an optimal HW configuration (and I'm not sure if this is something it can or should do). In any case, thanks for the response. I suppose a follow up question is, does splunk scale out well on larger boxes. For example, if Splunk is running on a 14 core server, will it use all 14 cores or is it limited for any particular reason?

laytonj76 · ‎02-16-2016

We are in the middle of designing an Integration environment that we ultimately want to replace our Production environment. We have determined sizing based on our daily ingestion, replication, search factor, RAID configuration, etc. The question I have is whether it's recommended to have more servers with less storage or less servers with more storage? Specifically, if we acquire servers with 12 bays and 2TB drives, we'd need 10 servers. Alternatively, if we acquire servers with 24 bays and 2TB drives we'd need 6 servers. We can support acquisition of either, but is there a recommendation between the two in terms of Splunk performance?

laytonj76 · ‎02-11-2016

This post was helpful as I'm researching the JMS TA since we have inherited an existing set up using it. I have a follow up to this question. In my customer's current implementation, the TA was manually installed on all 3 of our indexers instead of on a forwarder. We're now seeing indexing issues (i.e. only 1 of our 3 indexers is actively ingesting per the DMC). Do you think the TA manually installed on the 3 indexers could be the issue? By the way, we are planning to move from this current implementation to utilize a heavy forwarder as recommended; we just want to develop a clear understanding of why we'd see the indexer issues we're seeing.

laytonj76 · ‎12-28-2015

The link must not have saved; my apologies. Here is the link: https://answers.splunk.com/answers/177494/is-my-big-csv-lookup-file-indexed-in-memory-by-spl.html

laytonj76 · ‎12-28-2015

jkat54, thanks for the input; very much appreciated. Your recommendations were right in line with what we were thinking early on as well. However, when we found that there was only one copy on the entire server we were at a loss. I just added a comment that explains what we had to do that solved our problem in the near term. We have some long term updates to make, but we now understand the root cause. As for the lookups being 'indexed', Splunk actually will create an index but not in the way it would a conventional index. Splunk actually will create a ".index" directory in the same directory as the lookup which contains the tsidx file and a few other things. Here's a link to a question here in Splunk Answers. Thanks again for the input; provided some food for thought.

laytonj76 · ‎12-28-2015

Thanks to sel105, my coworker referenced below, for reaching out to our Splunk rep to help identify our particular issue. We have a SH cluster integrated with an Index cluster. What we learned is that for searches to execute across our search peers (i.e. our index cluster), the Search Heads must send a bundle containing the knowledge objects. As the knowledge objects change, new bundles are pushed out to the search peers. When we executed our query our search peers, not our Search Heads, were referencing old lookup files. A trick shared by the Splunk rep to determine whether the local lookup file returned something different was to add "local=t" option to the lookup command. When we did that, we got our expected results which confirmed that the local copy was correct and the search peers had an old version. I'm still doing some research, but since the changes were made locally on one of our SHs and since we don't have SH replicated configured completely, that change was not propagated across our SH cluster. Since the change did not make it to our Captain, which I'm surmising determines when to send updated bundles to search peers, the knowledge object changes were not pushed to the search peers. The short term fix was to put the correct copy on the SH Cluster's elected captain and the change was propagated to the search peers. Our long term objective will be to identify what we want replicated across our SH cluster and create a whitelist to facilitate such. This seems to have solved our immediate problem; however, if there is anything I've misstated, please chime in.

laytonj76 · ‎12-23-2015

I have a lookup file in a particular app that I use to enrich data from a particular index. This file, lookup_file.csv for example, changes each month, so we make those changes as necessary and upload a new file via the GUI. Recently, I noticed that the lookup command does not return all results in search; whereas the inputlookup command does. I know the two function differently and there could be a legitimate explanation (i.e. lookup could be looking at data in memory vs. on disk); however, I'm not sure how to force a fresh read of the file. The file has changed over time to exceed the default 10MB max_memtable_bytes in limits.conf. I assume this means that the file will no longer be stored in memory, but rather indexed. However, I am not seeing the .index directory I would expect if that were the case. Additionally, I can substantially change the contents of the file (i.e. I removed 100k of the 116k rows) and still get the same results returned as before. Also, oddly enough, if I remove the file altogether, Splunk returns an error. I assume all of this means that there is a reference stored in Splunk somewhere to old data and not refreshing when we make our changes. I'm not sure where that is or how I would refresh that reference. I have restarted splunkd AND rebooted the search head, assuming that the in-memory reference would refresh or the file would be indexed. Unfortunately, neither has happened. Any thoughts on what could be happening? Any references that may help me understand what's going on? Lastly, if it's of consequence, the file is a lookup stored in the user_dashboards app ... on a SH cluster.

Posts	11
Solutions	0
Karma Given	3
Karma Received	1
Member Since	‎09-10-2015

Online Status	Offline
Date Last Visited	‎03-02-2021 12:52 PM

How can I show labels on bar charts for long chart...

Should we have more servers with less storage or l...

Why do I see old data in my lookup table in a sear...

Re: After a clean install of Splunk IT Service Int...

How can I show labels on bar charts for long chart...

Re: Why am I getting "access denied" when trying t...

Re: Where do I start with troubleshooting full que...

Re: Should we have more servers with less storage ...

Should we have more servers with less storage or l...

Re: Where do I install the JMS Modular Input in my...

Re: Why do I see old data in my lookup table in a ...

Re: Why do I see old data in my lookup table in a ...

Re: Why do I see old data in my lookup table in a ...

Why do I see old data in my lookup table in a sear...