About ashleyherbert

ashleyherbert · ‎03-27-2019

Haven't noticed this before, but yes I can see it happening on mine as well. For some reason when the dashboard is refreshing, it's running through the functions and not finding any pre-populated tokens. I'm not sure why this is, I'll do some playing around and see if I can get it working.

ashleyherbert · ‎02-27-2018

Thanks heaps for the info chitturics! Really appreciate you taking the time to post this. I'm thinking we'll end up having to do something similar, just getting the default stuff in initially and then will work through what's required for our broad user base.

ashleyherbert · ‎02-17-2018

Hi all, My company is building a new private cloud platform, and will be offering F5 virtual servers as a standard offering for each application that onboards to the platform. As part of this, using Splunk we want to give visibility of the F5 devices for the F5 support team, as well as giving visibility to consumers of the platform to their own virtual servers. We are a large company and will end up with hundreds of apps on this platform. I've been playing with the multi tenancy in the F5 iApp, but I have a few concerns with the way it works: The number of indexes will become quite large, and I'm a little concerned about the scalability & performance of separating all the data into hundreds of indexes. For the platform consumers, I think the dashboards might be too much information to display, when mostly all they want to see is the status of their VIPs and pool members. So my question is, do others offer this app to their consumers, or just use it for dedicated support of the F5s? If so, do you find it consumable by your average application support team members? I'm leaning towards making it a single tenant model and filtering by app, then just building some simple dashboards to cover the consumers use cases. We already provide a standardised Splunk app to consumers of Splunk, so I'd just extend that. We'd then only allow the F5 support team access to this app. Thanks heaps in advance! Ash

ashleyherbert · ‎02-16-2018

Just thought I'd post an alternative way of doing this. I'm not a big fan of adding javascript to my dashboards, so I've found a way to do this by adjusting the token for the input using the change & eval attributes. See the example below from one of our dashboards: <input type="multiselect" token="host" searchWhenChanged="true"> <label>Hosts</label> <search base="lookup"> <query>search $env$ | dedup host</query> </search> <valuePrefix>host=</valuePrefix> <delimiter> OR </delimiter> <default>*</default> <choice value="*">All</choice> <fieldForLabel>host</fieldForLabel> <fieldForValue>host</fieldForValue> <change> <eval token="form.host">if(mvcount('form.host')=0,"*",if(mvcount('form.host')!=1,mvfilter('form.host'!="*"),'form.host'))</eval> </change> </input> This will reset the token to 'All' if no options are selected, and will remove the 'All' option if more than 1 are selected. UPDATE: One of my colleagues found this didn't work properly. When you select a couple of hosts, the only way to get back to 'All' is to remove all your previous selections, at which point 'All' will re-appear. Below is the fix he added, this is the commentary from our internal Jira system: Basically, because of the '*', the regex doesn't quite work, so we need to change the <choice value="*">All</choice> to <choice value="All">All</choice> so the regex mvfind works as expected. Then, set a second token dependent on the value of 'form.blah', which is then used in any post-process search [decided to change the name of the original multi-select form token, and then set a token with the old name, so it remains transparent to the post-process searches that use the original token]. So: - it defaults to 'All' (All is alone at the start, index=0) - if another env is chosen, 'All' is removed - if 'All' is chosen, all other envs are removed (ie All is always alone at the start, index=0) - if everything is removed, 'All' reappears (ie All is always alone at the start, index=0) <input type="multiselect" token="env2" searchWhenChanged="true"> <label>Environment</label> <search base="lookup"> <query>dedup environment</query> </search> <default>All</default> <valuePrefix>environment=</valuePrefix> <delimiter> OR </delimiter> <choice value="All">All</choice> <fieldForLabel>environment</fieldForLabel> <fieldForValue>environment</fieldForValue> <change> <eval token="form.env2">case(mvcount('form.env2')=0,"All",mvcount('form.env2')>1 AND mvfind('form.env2',"All")>0,"All",mvcount('form.env2')>1 AND mvfind('form.env2',"All")=0,mvfilter('form.env2'!="All"),1==1,'form.env2')</eval> <eval token="env">if(mvfind('form.env2',"All")=0,"environment=*",$env2$)</eval> </change> </input> ...still use $env$ where appropriate <input type="dropdown" token="host" searchWhenChanged="true"> <label>Select a Host:</label> <selectFirstChoice>true</selectFirstChoice> <search base="lookup"> <query>search $env$ | dedup host</query> </search> Hope this helps, cheers.

ashleyherbert · ‎02-16-2018

Just thought I'd post an alternative way of doing this. I'm not a big fan of adding javascript to my dashboards, so I've found a way to do this by adjusting the token for the input using the change & eval attributes. See the example below from one of our dashboards: <input type="multiselect" token="host" searchWhenChanged="true"> <label>Hosts</label> <search base="lookup"> <query>search $env$ | dedup host</query> </search> <valuePrefix>host=</valuePrefix> <delimiter> OR </delimiter> <default>*</default> <choice value="*">All</choice> <fieldForLabel>host</fieldForLabel> <fieldForValue>host</fieldForValue> <change> <eval token="form.host">if(mvcount('form.host')=0,"*",if(mvcount('form.host')!=1,mvfilter('form.host'!="*"),'form.host'))</eval> </change> </input> This will reset the token to 'All' if no options are selected, and will remove the 'All' option if more than 1 are selected.

ashleyherbert · ‎06-14-2016

Thanks for the quick response Damien, that's a better response than I'd be able to give before I've had my coffee!! Yeah wanted to check if this had been considered before we started developing something new. I'm thinking it might be better to just have a forwarder running somewhere to retrieve the messages. It's not highly available, but at least it won't be affected by maintenance on the Splunk servers...

ashleyherbert · ‎06-14-2016

Hi, We currently use the JMS Messaging Modular Input to retrieve JMS message from MQ queues, and the jms_ta runs on all 4 indexers in our indexer cluster. We have a new requirement to retrieve messages from a Tibco EMS Topic, but if we use the same setup (the jms_ta running on 4 indexers), I'm assuming we will get multiple copies of the messages in Splunk as each of the indexers acts as a different topic subscriber. Is there any way for the jms_ta modules to know about each other to stop it receiving multiple copies? Or any suggestions on how to have highly available consumers of the JMS topic? Thanks, Ash

ashleyherbert · ‎02-01-2016

Hello We have implemented a search head cluster behind an F5 LTM. I've created an LTM VIP with cookie persistence for the REST API (hitting port 8089 obviously), but we're having an issue with our REST clients. When the REST client authenticates, it gets a sessionkey back. Then when it makes the next request passing the sessionkey, if the request hits a different Splunk server, then it gets a HTTP 401 and the response 'call not properly authenticated'. I thought initially that the second request is hitting a different server before the session is replicated around the cluster, but I've found in my testing that the sessionkey never gets replicated at all, and I can only use that sessionkey against the one Splunk server. Is this expected behaviour? I would expect that it would function the same way as the Splunk Web component, where the session cookies are accepted by every server in the cluster. Is there a way I can get these sessionkeys to replicate around the cluster? I can change the persistence to be source based, but even then there's no guarantee that the next requests will always hit the same server. Thanks, Ash

ashleyherbert · ‎11-15-2015

Hi Luke, thanks very much, I didn't notice that there was another version there. We're downloading it now and will test it over the next couple of days. I'll let you know if we find any problems. Cheers, Ash

ashleyherbert · ‎11-11-2015

Hi Luke, Just wondering if you're still looking to get this working with SH Clustering? We use the lookup editor quite heavily to manage recipients of alerts (combined with the sendresults command). The app is really handy, but we've just changed our test environment to use SH clustering and found we can no longer use this. Just want to find out if we should hold fire on redesigning our solution, in case you have an update planned? Thanks, Ashley

ashleyherbert · ‎09-24-2015

I haven't tried out the 6.3 forwarder yet, but my best bet would be that it is trying to connect to the cluster master. There is a new feature called 'Indexer Auto-Discovery' which connects to the cluster master to determine which indexers are available. See here: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/indexerdiscovery Could that be turned on by default?

ashleyherbert · ‎08-17-2015

Thanks! Yes this is exactly the issue, thanks for spotting it. I'd be curious as to what would perform better, if I use the INDEXED_VALUE setting or if I was to do a transform on the source data to separate the fields at index time (such as adding commas) to allow it to easily find the values. This data is going to be heavily searched upon so I'm very conscious of performance.

ashleyherbert · ‎08-17-2015

Hi Jeff, I've defined them in props.conf. Yes I've used rex to test them, but obviously that works OK as I have to search on the fields after the rex command - eg. | rex " | search field=value The issue only appears when it's searching for the fields within the original search, if i search for them after the first pipe then it's fine.

ashleyherbert · ‎08-16-2015

We've found the cause of this issue, and it's actually nothing to do with Splunk or the JMS Modular Input. The problem is that we were using a VirtualBox VM for Splunk with the network configuration set to NAT. When we changed the network to 'Bridged' mode, the performance is good with both SSL & non-SSL. As soon as we switch NAT back on, the performance using SSL immediately drops by about 90%. Using non-SSL is also fine with both network modes. I've done a bit of googling to see if it's a known VirtualBox issue, but haven't been able to find anything. Good to know it's not an issue with the JMS Modular Input, thanks heaps for your help Damien!

ashleyherbert · ‎08-16-2015

Yeah I've also tried with quotes, but it's the same result. If I use Type=*GCCC* it does actually work (with or without quotes), but I need to be able to search for the complete string without the wildcards as it makes it very inefficient.

ashleyherbert · ‎08-16-2015

Hi, I'm seeing some very unusual behavior when extracting fields in Splunk 6.2. Basically I can see the fields are extracted successfully, but I can't use them to search. I have the following sample data: 101STUS NVLGCCCPDRf4cc5a8023ce40e28c9f260c376dabe9032134120864 032000123456789 191820013550000000000000000ESBtesSSP191820013550000000000000000abcdefSSD00071468C4875691F2CC0000000102763400095 C02763400095 20150721112211485002KO-001HIHI12345 6_1 ABCD02 20150721102147122201507211754000000000000000007400 AU S2015072120150721CRN MH48A 0201 ACSP 20150721112211485ACSC 2015072111221148511215201121520 BIS 0000000000 00000000 This is a fixed length field log file (from mainframe), with no field separators. Therefore, I am using the following regular expression to extract the fields, which basically just extracts them from their position in the log file: .{11}(?<Type>.{4}).{1}(?<Direction>.{2}).{32}(?<InputMainID>.{6})(?<InputSecondaryID>.{28})(?<OutputMainID>.{6})(?<OutputSecondaryID>.{28})(?<BusinessID>.{36})(?<TransID>.{36})(?<SystemID>.{20})(?<SequenceNo>.{8})(?<CustomerKey>.{15}).{33}(?<BusinessChannel>.{3})(?<SourceChannel>.{6}).{132}(?<SubmitTime>.{17}).{17}0*(?<Figure>\d{1,16}).{90}(?<OverallStatus>.{4}).{4}(?<UpdateTime>.{17})(?<Status>.{4}).{4}(?<TransUpdateTime>.{17}) Now when I search for the log in Splunk, I can see all the fields created with the correct values. index=main sourcetype=mytype However, if I try to add the fields to the search string, I am unable to see any results. For example: index=main sourcetype=mytype Type=GCCC I've found that if I put a * on either side of the field value, it does find them, which I find strange: index=main sourcetype=mytype Type=*GCCC* This indicates that there may be whitespace around the value, but it doesn't appear that way when I look at the values. I've also found that I can successfully search for the fields if I add it as an extra search function after the main search: index=main sourcetype=mytype | search Type=GCCC This looks like it doesn't run the field extraction until after the main search, however I can see in a lot of other sourcetypes I have that this isn't the case, as I can search for those. I've also tried a number of other things to try to get this working: Separating the regexes so each field has it's own extractions. I still get the same issue. Extracting only 1 field from the data to simplify it. I still get the same issue. Adding the following Calculated Field. This works, but I don't want to add an EVAL for every field as I'm sure there will be performance implications [mytype] EVAL-Status = Status Has anyone seen this before? I've played around a lot with the regex, but could there be a problem with this? Is there a better way to extract the fields for a fixed length file? I suspect that it's partly because the fields have no separators, therefore Splunk isn't able to do keyword searches on partial matches, can anyone confirm? Thanks in advance. Ashley

ashleyherbert · ‎07-08-2015

Thanks, yeah I originally had it without the "-D" but it didn't work either. I'll play around with this to get it going, but the most pressing issue is the performance. I'm going to try getting different versions of the client jars (newer & older) to see if there's any bugs in the version we're using. I'll also try different versions of Java on the client side. We've also been working with our MQ experts to review the config, and there doesn't appear to be anything on the Queue Manager or Queue that would cause this performance issue. Any other suggestions on things to try? Should I try setting up the Local handler that you have, would this behave any differently?

ashleyherbert · ‎07-06-2015

Hi Damien, We're using the client jars from MQ V8.0.0.0 (not WAS. In fact, the jars were copied from the queue manager that we're connecting to). And I'm using JRE 1.8 to connect to it. I also had issues trying to get the JVM System Properties to be picked up, so the keyStore setting isn't working. I've ended up just putting the trust cert into cacerts, but I'll need to change that (We'll need to use SSL MA once I've got this working correctly). Below is my inputs.conf setup: [jms://queue/:PAYMENTMSG] browse_frequency = -1 browse_mode = all browse_queue_only = 0 durable = 0 index = main index_message_header = 0 index_message_properties = 0 init_mode = jndi jms_connection_factory_name = PAYMON jndi_initialcontext_factory = com.sun.jndi.fscontext.RefFSContextFactory jndi_provider_url = file:/opt/splunk/indexer/etc/apps/jms_ta/mq/ sourcetype = mq strip_newlines = 1 disabled = 0 jvm_system_properties = -Djavax.net.ssl.keyStore=/opt/splunk/indexer/etc/apps/jms_ta/mq/clientjks.jks,-Djavax.net.ssl.keyStorePassword=xxxxxxxx

ashleyherbert · ‎07-06-2015

Hi, We're testing the JMS messaging modular input to pull messages from a Websphere MQ V8 queue, and have just got it working over SSL. However, we're seeing a massive performance degradation when connecting via SSL. Has anyone else got this working? Using an unencrypted connection, we can pull close to 200 messages per second from the queue. Once we turn on SSL, that value drops below 20 messages per second. Trying to find out what the bottleneck is, as it doesn't appear to be re-establishing connections every time. Are there any settings to improve this? Thanks, Ashley

ashleyherbert · ‎05-30-2015

Thanks for the reply mockd, I was on the right path but didn't know where I'd see that output. I've added the line in and I can see the debug output in the search log in the job inspector, so it looks good. Now I just have to wait for it to happen again! Thanks heaps for your help!

ashleyherbert · ‎05-28-2015

Hi, We use the sendresults command for some alerting, and we are having an issue where occasionally the sendresults.py script is hanging, which causes the scheduled search to never finish, and then the subsequent scheduled searches are not run. This happens a couple of times per week, and is causing our alerting to be very unreliable. I suspect it's a problem with our SMTP server not responding, but I want to turn on some debug logging inside the sendresults.py to figure out where this is hanging. I've figured out I can change the "level=logger.INFO" to DEBUG and that prints out extra debug statements for the Splunk stuff, but it doesn't print out anything about the SMTP connection. The library that is being used (smtplib) has a function "SMTP.set_debuglevel(True)" which should do this. Do you know if it is possible to add this setting and have it log to the sendresults.log? Appreciate the help in advance. Thanks, Ash

ashleyherbert · ‎12-22-2014

Yeah the panel is entirely visible, and there's no gaps or weird layout. I am actually surprised the search still runs & refreshes with the depends=$nothing$ tag, but it definitely does. And yes, I just tried removing the refresh.auto.interval tags from all the other single values and they still all refresh.

ashleyherbert · ‎12-22-2014

Great idea, and it works! I added the following extra single value in the same panel. Even though the module is hidden (due to the depends token missing), it still refreshes the search. <single depends="$nothing$"> <search id="rmiperfSearch"> <query>index=sovereign sourcetype=rmiperf action=lookup_startup | stats latest(action_time) as action_time by host | rangemap field=action_time low=0-10000 elevated=10001-30000 severe=30001-100000 default=severe</query> <earliest>-3m</earliest> <latest>now</latest> </search> <option name="linkView">search</option> <option name="drilldown">none</option> <option name="refresh.auto.interval">60</option> <option name="field">action_time</option> </single>

ashleyherbert · ‎12-22-2014

Hi All, Using the new 6.2 search module, I'm trying to create a dashboard with a global search and multiple single values presented from it using post-process searches. I need the single values to refresh every 60 seconds, however since they're driven off the global search, the refresh.auto.interval option does not re-run the global search. Is there a way to refresh the global search individually? I know I can refresh the whole dashboard, but I plan to have a few extra searches on this dashboard which will need to be refreshed separately. I could also make it a scheduled saved search, but that means it's running all the time when nobody is looking at it. Here's my XML: <dashboard> <label>Sovereign Performance</label> <search id="rmiperfSearch"> <query>index=sovereign sourcetype=rmiperf action=lookup_startup | stats latest(action_time) as action_time by host | rangemap field=action_time low=0-10000 elevated=10001-30000 severe=30001-100000 default=severe</query> <earliest>-3m</earliest> <latest>now</latest> </search> <row> <panel> <title>RMI lookup_startup latest response time in Kogarah</title> <single> <title>K1 - VIRNTAPP035</title> <search base="rmiperfSearch"> <query>search host=virntapp035</query> </search> <option name="underLabel">K1</option> <option name="linkView">search</option> <option name="drilldown">none</option> <option name="afterLabel">ms</option> <option name="refresh.auto.interval">60</option> <option name="field">action_time</option> <option name="classField">range</option> </single> <single> <title>K2 - VIRNTAPP036</title> <search base="rmiperfSearch"> <query>search host=virntapp036</query> </search> <option name="underLabel">K2</option> <option name="linkView">search</option> <option name="drilldown">none</option> <option name="afterLabel">ms</option> <option name="refresh.auto.interval">60</option> <option name="field">action_time</option> <option name="classField">range</option> </single> </panel> <panel> <title>RMI lookup_startup latest response time in WSDC</title> <single> <title>G1 - VIRNTAPP042</title> <search base="rmiperfSearch"> <query>search host=virntapp042</query> </search> <option name="underLabel">G1</option> <option name="linkView">search</option> <option name="drilldown">none</option> <option name="afterLabel">ms</option> <option name="refresh.auto.interval">60</option> <option name="field">action_time</option> <option name="classField">range</option> </single> <single> <title>G2 - VIRNTAPP043</title> <search base="rmiperfSearch"> <query>search host=virntapp043</query> </search> <option name="underLabel">G2</option> <option name="linkView">search</option> <option name="drilldown">none</option> <option name="afterLabel">ms</option> <option name="refresh.auto.interval">60</option> <option name="field">action_time</option> <option name="classField">range</option> </single> </panel> </row> <row> <panel> <title>Response times by server for the past 4 hours</title> <chart> <search> <query>index=sovereign sourcetype=rmiperf action=lookup_startup | timechart span=1m avg(action_time) by host</query> <earliest>-4h@m</earliest> <latest>now</latest> </search> <option name="charting.chart">line</option> <option name="charting.axisY2.enabled">undefined</option> <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option> <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option> <option name="charting.axisTitleX.visibility">collapsed</option> <option name="charting.axisTitleY.visibility">visible</option> <option name="charting.axisTitleY2.visibility">visible</option> <option name="charting.axisX.scale">linear</option> <option name="charting.axisY.scale">linear</option> <option name="charting.axisY2.scale">inherit</option> <option name="charting.chart.bubbleMaximumSize">50</option> <option name="charting.chart.bubbleMinimumSize">10</option> <option name="charting.chart.bubbleSizeBy">area</option> <option name="charting.chart.nullValueMode">gaps</option> <option name="charting.chart.sliceCollapsingThreshold">0.01</option> <option name="charting.chart.stackMode">default</option> <option name="charting.chart.style">shiny</option> <option name="charting.drilldown">all</option> <option name="charting.layout.splitSeries">0</option> <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option> <option name="charting.legend.placement">right</option> <option name="refresh.auto.interval">60</option> <option name="height">400px</option> </chart> </panel> </row> </dashboard>

ashleyherbert · ‎06-30-2014

Hi All, I'm looking at a way to setup a DR process for a license master. We currently have our license master running in the same instance as our deployment server, and we have a standby deployment/license server in a 2nd site. To fail over the license master to our standby, can I have the licenses pre-installed on the standby server? I'm concerned that we will get duplicate license errors as the deployment server needs an enterprise license to work. Or should the process be to manually install the license files at the time of a DR invocation? What do others out there do? I can automatically fail over the communications to the standby license master, as these all go via F5 load balancers at the moment, but I'm not sure whether I can have the licenses running in both servers at the same time? Cheers, Ash

Posts	46
Solutions	2
Karma Given	5
Karma Received	53
Member Since	‎11-20-2011

Online Status	Offline
Date Last Visited	‎05-19-2021 05:25 PM

Use of F5 Network Analytics app for customers

JMS Messaging Modular Input: How to prevent indexi...

Using REST API load balancing in a Search Head Clu...

Unable to search for regex extracted fields in fix...

JMS Messaging Modular: Why are we seeing slow perf...

sendresults debug logging

Is there a way to Auto Refresh a global search ind...

License Master Disaster Recovery Process

Report Acceleration on subsearch

index_name variable to use in all index paths

Re: Remove "All" from Multiselect Input in Dashboa...

Re: Use of F5 Network Analytics app for customers

Use of F5 Network Analytics app for customers

Re: Remove "All" from Multiselect Input in Dashboa...

Re: Make a SimpleXML multiselect revert to default...

Re: JMS Messaging Modular Input: How to prevent in...

JMS Messaging Modular Input: How to prevent indexi...

Using REST API load balancing in a Search Head Clu...

Re: Lookup File Editor with Search Head Clustering...

Re: Lookup File Editor with Search Head Clustering...

Re: Changes to forwarder->indexer connectivity in ...

Re: Unable to search for regex extracted fields in...

Re: Unable to search for regex extracted fields in...

Re: JMS Messaging Modular: Why are we seeing slow ...

Re: Unable to search for regex extracted fields in...

Unable to search for regex extracted fields in fix...

Re: JMS Messaging Modular: Why are we seeing slow ...

Re: JMS Messaging Modular: Why are we seeing slow ...

JMS Messaging Modular: Why are we seeing slow perf...

Re: sendresults debug logging

sendresults debug logging

Re: Is there a way to Auto Refresh a global search...

Re: Is there a way to Auto Refresh a global search...

Is there a way to Auto Refresh a global search ind...

License Master Disaster Recovery Process