Can we run a scripted input on demand?

[caagrawal]

We have a Splunk dashboard - one of the panels (search) uses splunk dbmon:dump to fetch the details from an Oracle DB. Now, due to some problems, we want to get away from dbmon and instead have our own script (scripted input) to fetch the data from the database. I have written the script and it works well, however...

1. It does not run on demand. What I mean is, the search on the dashboard uses the data which is generated by this script. Now, if the script runs at, say every 15 mins, then I would have to wait for 15 mins to get the real-time data. In case of dbmon, I used to get data immediately after any updates to the db.
2. Is there any other way to achieve this?

[lguinn2]

So as I now understand it:

The Oracle database is running on a server that has a Universal Forwarder (UF).
When you run a search on the search head, you want that search to do the following
1 - trigger a scripted input to run on the forwarder that collects data about the Oracle database
2 - wait until the data is collected and indexed
3 - analyze the collected data in some way with the search language

There are many reasons why this is probably not possible, at least as you have described it:
1 - Your script would need to exist on the search head in order to be triggered by a search, but it would need to collect data on the universal forwarder and have that data indexed by the indexing tier.
2 - There is no way to make a search wait for a script completion + indexing, AFAIK.

If you only want to collect the data just before you run a search, the only way I can think of to do it, is to make this into two manual steps:
1 - launch the data collection script and wait for it to complete.
2 - run a search and have it use only the most recent data.

I still think that you need to have the data collection script write to a file. The resulting file could be on the UF, and then a simple file monitor input on the UF could collect the data immediately and forward it to the indexer(s). You could have the script overwrite the existing file each time or create a brand-new file - there are a variety of possibilities. BUT a "Splunk scripted input" must be scheduled. It can be scheduled to run periodically or it can be scheduled to run "all the time." So you need to figure out how to launch the script only when you want the data collected; you can't use a Splunk scripted input.

If you don't want to keep past data about the Oracle DB, put the data into a small index and set the retention time on that index to a week or something. Delete the data collection files routinely (or overwrite).

The final search should be easy to write. You just can't run the search until the data collection is complete; or perhaps I should say "the most recent data will not show up on the dashboard until the data collection is complete."

Once you get this going manually, you might be able to think of some clever ways to automate or coordinate the various pieces using scripted alert actions. Or you could add a link to the data collection script to your Splunk menu.

[lguinn2]

Here is what I would do:

Set up your script on the UF so that it runs all the time, or loops every minute or whatever you want. Whenever the script notices that something happened, have it write to a file.
Do not set up the script as a scripted input in Splunk. Instead, have it launched by init.d or as a Windows service or whatever works best in your environment. (It is possible to have it launched by Splunk, but just never return any value to Splunk and never stop, but probably better to use the OS.)

In Splunk, set up a file monitor input to watch the file that the script writes. Unless your forwarder is very busy, it will pick up the information from the file and forward it so that you can search it within a few seconds. Whenever you load the search or the dashboard, the most recent information will already be there - you don't have to launch a script to do the data collection.

This should give you better response time in general. Also the file provides a natural way to trace back what happened and to recover from a failure.

Forget the "search command" - that will only work on a search head. You need for your script to be running on the same server as the DB, which means that it needs to run on the UF.

[caagrawal]

Thanks for your suggestion. However the query I would be firing is time consuming. I can't run it every now and then through the script. That's why I said I want to run it on demand means only when my search will execute. Also, I dont want incremental data. I want point in time data when the search would run. There is no need to have that data written to the file.
Hence, I think this approach might not fully suffice to my requirements. Let me know what you think.

[MuS]

Why not create a modular input http://docs.splunk.com/Documentation/Splunk/6.3.2/AdvancedDev/ModInputsScripts from this script? This way it will be controlled and managed from within Splunk, also it can log any errors or additional infos into splunkd.log or its own log. Yes, modular inputs do work on UF as long as you have a Python available from the OS and don't need any fancy/strange libraries for it to run. Running the rest_ta for example on an UF 😉

[lguinn2]

@MuS - the problem is, the modular input is a configurator for a scripted input. That scripted input still has to run on a schedule, no matter where it is located (UF, indexer, etc.)

I think that this question is "how to run a scripted input (on a UF) that is triggered by a search (not a schedule), where the inputs collected become part of the search results?"

[caagrawal]

@Iguinn - Exactly, that is the problem I want to solve.

[somesoni2]

This should help you. You can make your custom script as Search Command and can invoke on demand from search (similar to dbmon or dbquery)

http://docs.splunk.com/Documentation/Splunk/6.1/AdvancedDev/Searchscripts

[caagrawal]

Thanks for your help. Does it work for Universal forwarders also?
I quickly tried following steps but its not working -
1. Created small perl script mydata.pl to fetch data from DB. (Note script must run on one of UF as DB connectivity is open from particular host only)
2. Kept script in /opt/splunk/etc/apps//bin directory
3. Created commands.conf in /opt/splunk/etc/apps//local/ directory and added stanza
[mydata]
type = perl
filename = mydata.pl
4. Restarted splunk on UF and Indexer
5. Tried executing search "| script perl mydata". It says script mydata does not exist in the custom commands database.

[somesoni2]

The search command is used in Search Heads and that's where you should be setting it up. So if it has to be setup in UF, then Search command is not that we can use. If your script captures incremental data, then you can set it as scripted input and run more frequently. Again, it depends upon your script's performance and how much delay is acceptable will decide the interval for the scripted input.