Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About carlalldis
carlalldis
Explorer
Member since:
09-14-2015
09-18-2020
Community Statistics
| Posts | 8 |
| Solutions | 0 |
| Karma Given | 1 |
| Karma Received | 2 |
| Member Since | 09-14-2015 |
Activity Feed
- Got Karma for Slow splunkweb startup caused by splunk_instrumentation. 06-05-2020 12:48 AM
- Got Karma for Slow splunkweb startup caused by splunk_instrumentation. 06-05-2020 12:48 AM
- Karma CIDR in a lookup table - no access to transforms.conf? for mbolostk. 06-05-2020 12:47 AM
- Posted Re: Is there a systemd unit file for Splunk? on Splunk Search. 12-31-2018 05:19 AM
- Posted Re: Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-15-2017 12:56 AM
- Posted Re: Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-14-2017 01:08 AM
- Posted Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Tagged Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Tagged Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Tagged Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Tagged Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Tagged Slow splunkweb startup caused by splunk_instrumentation on Splunk Dev. 06-06-2017 06:37 AM
- Posted Re: Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-27-2016 07:40 AM
- Posted Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Tagged Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Tagged Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Tagged Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Tagged Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Tagged Splunk_TA_nix multiple inputs configurations on All Apps and Add-ons. 07-25-2016 04:20 AM
- Posted Re: Splunk DB Connect 2: Why is Splunk not indexing SQL data that has epoch timestamps? on All Apps and Add-ons. 09-21-2015 02:54 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 2 | |||
| 0 | |||
| 0 |
12-31-2018
05:19 AM
I had to remove this line for it to work:
action.lookup("unit") == "Splunkd.service" &&
for some reason my "splunk start/stop" commands are not passing the unit variable (i did a log test which shows this as undefined). This is insecure for the moment but works for my test environment. Does anyone have any better suggestions?
... View more
06-15-2017
12:56 AM
Red Hat 7. I also get the issue on a personal Centos 7 instance. Both are fully up to date.
... View more
06-14-2017
01:08 AM
Great, I thought I was going mad. I also have a support case ongoing with Splunk who cannot reproduce the issue as of yet. I am doing some diagnostic traces for them to review. Hopefully we will find the cause of the bug as it seems to be environment related.
... View more
06-06-2017
06:37 AM
2 Karma
Hi,
After an upgrade from 6.3 to 6.6.1 I am experiencing very slow startups (around 5 minutes) on all search heads, indexers, heavy forwarder, license master and deployment server, while 'Waiting for web server at http://127.0.0.1:8000 to be available' . This also seems to be the case on a standalone dev instance.
I have tracked the issue down to splunk_instrumentation. See the large gap of time in web_service.log without debug logging:
2017-06-06 13:27:04,519 INFO [59369f954b7fb8451b27d0] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 13:31:19,367 INFO [59369f954b7fb8451b27d0] custom:211 - Registering custom app endpoint: splunk_instrumentation/instrumentation_controller
2017-06-06 13:31:19,454 INFO [59369f954b7fb8451b27d0] custom:211 - Registering custom app endpoint: Splunk_TA_nix/setup
And with python debug logging on:
2017-06-06 13:40:04,899 INFO [5936a2a4407fa13c5cd7d0] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 13:40:05,001 DEBUG [5936a2a4407fa13c5cd7d0] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.>
2017-06-06 13:40:05,002 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/instrumentation_controller.py
2017-06-06 13:40:05,002 DEBUG [5936a2a4407fa13c5cd7d0] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.splunk_instrumentation>
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/cherrypy.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/traceback.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/traceback.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/json.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/json.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/logging.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/logging.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/zipfile.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/zipfile.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/StringIO.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/StringIO.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/datetime.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/datetime.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/cherrypy.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk_instrumentation.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk_instrumentation.py
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/string.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/string.py
2017-06-06 13:44:19,655 INFO [5936a2a4407fa13c5cd7d0] custom:211 - Registering custom app endpoint: splunk_instrumentation/instrumentation_controller
Moving the app folder to somewhere else and starting splunk results in an near instant startup:
2017-06-06 14:29:02,563 INFO [5936ae1c147f2297de8790] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/json.py'
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/json.py
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/setup.py
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.Splunk_TA_nix>
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.>
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/splunk.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/splunk.py
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/cherrypy.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/cherrypy.py
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/sys.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/sys.py
2017-06-06 14:29:02,906 DEBUG [5936ae1c147f2297de8790] root:452 - No shared storage location configured, using: /opt/splunk/var/run/splunk/rss
2017-06-06 14:29:02,906 DEBUG [5936ae1c147f2297de8790] root:446 - Checking for shared storage location
2017-06-06 14:29:02,906 INFO [5936ae1c147f2297de8790] custom:211 - Registering custom app endpoint: Splunk_TA_nix/setup
I have managed to workaround the issue for now by creating /opt/splunk/etc/apps/splunk_instrumentation/local/app.conf with the following parameters:
[install]
allows_disable = true
state = disabled
Hopefully this helps anyone else with slow startup issues, and perhaps someone can shed some light as to why this app, which is supposed to do background telemetry, is causing such an impact.
Thanks,
Carl
... View more
07-27-2016
07:40 AM
Thanks tsweet; I never considered using that. Very helpful.
... View more
07-25-2016
04:20 AM
Hi,
I am looking to deploy the Splunk_TA_nix to multiple servers. I would like to have different inputs.conf depending on the server type, all deployed from the same deployment server.
There are a few options I have considered:
Renaming the Splunk_TA_nix app depending on the server type (e.g. Splunk_TA_nix_dns, Splunk_TA_nix_sql). The problem with this is that there are a few references within the app to the app name itself, and I do not want this to cause issues.
Creating a new app per server type with just the inputs.conf. The issue with this is that the script inputs use the relative path (e.g. ./cpu.sh); if I use this same stanza name in my app, it will not work. If the absolute path is used, it works fine, but this is bad practice as the forwarder paths may differ depending on installation type (e.g. /opt/splunk or /opt/splunkforwarder).
Has anyone had any experience with this who might be able to lend some advice please?
Thank you, Carl
... View more
09-21-2015
02:54 AM
Thanks for the replace. I tried forcing this in the conf (input_timestamp_format = epoch) however then I just get the error:
09/21/2015 10:26:18 [ERROR] [ws.py] [DBInput Service] ERROR: Input timestamp format is invalid as [Illegal pattern character 'e']..
09/21/2015 10:26:18 [ERROR] [websocket.py] ERROR: Input timestamp format is invalid as [Illegal pattern character 'e']..
From the spec you posted, it looks like the _timestam_format fields only accept a java datetime, of which 'epoch' is not. I assume that to specify epoch you have to omit the field from the conf altogether. I adjusted the conf accordingly, removing both fields:
[mi_input://PNET07_WIN_CPU]
connection = PNET07
index = pnet07
input_timestamp_column_name = TOTIME
input_timestamp_column_number = 3
interval = 300
max_rows = 100
mode = batch
query = SELECT * FROM "storm_CCA-SCI-PNET07"."dba"."PATROLMS_HW_MAIN_MS_HW_CPUCORE_CONT_MS_HW_CPUCORE_RT_VIEW"
source = PNET07_WIN_CPU
sourcetype = WIN_CPU
ui_query_catalog = storm_CCA-SCI-PNET07
ui_query_mode = advanced
ui_query_schema = dba
ui_query_table = _PATROL_MS_HW_MAIN_MS_HW_ENCLOSURE_MS_HW_LOGICALDISK_CONT_MS_HW_LOGICALDISK_STATS_19
however now I get:
09/21/2015 10:33:37 [ERROR] [ws.py] [DBInput Service] ERROR: The datatype of output timestamp column number [3] is invalid as [int]..
09/21/2015 10:33:37 [ERROR] [websocket.py] ERROR: The datatype of output timestamp column number [3] is invalid as [int]..
so I then removed the input_timestamp_column_number fields (although the above specifies output_timestamp_column_number, which is actually commented out in the python code from what I can see?) however I still get the same issue:
09/21/2015 10:50:38 [ERROR] [ws.py] [DBInput Service] ERROR: The datatype of output timestamp column number [3] is invalid as [int]..
09/21/2015 10:50:38 [ERROR] [websocket.py] ERROR: The datatype of output timestamp column number [3] is invalid as [int]..
Thus I still have no idea why this is failing. As I say, I looked through some of the code and can find no active references for 'output_timestamp' or 'output timestamp'
Thanks
... View more
09-17-2015
02:49 AM
Hi,
I am having an issue with Splunk not indexing SQL data (from Sybase SQL Anywhere 11) that has epoch timestamps. Here is an example of the results returned by the SQL query:
STATUS_AVG STATUS_LOW TOTIME MOINSTID FROMTIME STATUS_HIGH
0 0 1425240000 12669 1425236400 0
0 0 1425240000 12666 1425236400 0
0 0 1425243600 12673 1425240000 0
0 0 1425243600 12672 1425240000 0
0 0 1425243600 12671 1425240000 0
0 0 1425243600 12670 1425240000 0
0 0 1425243600 12669 1425240000 0
0 0 1425243600 12668 1425240000 0
0 0 1425243600 12667 1425240000 0
0 0 1425243600 12666 1425240000 0
In the gui, I configured the input as follows (also, the GUI validates the connection as ok, and the query runs successfully):
Name: PNET07_WIN_CPU
App: Splunk DB Connect v2
Connection: PNET07
Advanced Query: SELECT * FROM "storm_CCA-SCI-PNET07"."dba"."_PATROL__MS_HW_MAIN_MS_HW_CPUCORE_CONT_MS_HW_CPUCORE_RT_VIEW"
Type: Batch Input
Max Rows: 100
Timestamp: Choose Column
Specify Timestamp Column: TOTIME
Timestamp Format: Epoch time
Output Timestamp Format: Epoch time
Execution Frequency: 300
Source: PNET07_WIN_CPU
Sourcetype: WIN_CPU
Index: pnet07
If I look at the stanza generated in inputs.conf, it is as follows:
[mi_input://PNET07_WIN_CPU]
connection = PNET07
index = pnet07
input_timestamp_column_name = TOTIME
input_timestamp_column_number = 3
interval = 300
max_rows = 100
mode = batch
output_timestamp_format = epoch
query = SELECT * FROM "storm_CCA-SCI-PNET07"."dba"."_PATROL__MS_HW_MAIN_MS_HW_CPUCORE_CONT_MS_HW_CPUCORE_RT_VIEW"
source = PNET07_WIN_CPU
sourcetype = WIN_CPU
ui_query_catalog = storm_CCA-SCI-PNET07
ui_query_mode = advanced
ui_query_schema = dba
ui_query_table = _PATROL__MS_HW_MAIN_MS_HW_ENCLOSURE_MS_HW_LOGICALDISK_CONT_MS_HW_LOGICALDISK_STATS_19
This all seems to be ok, however, when the input runs on the schedule specified, the following is logged in dbx2.log:
09/17/2015 10:26:14 [ERROR] [ws.py] [DBInput Service] ERROR: Output timestamp format is invalid as [Illegal pattern character 'e']..
09/17/2015 10:26:14 [ERROR] [websocket.py] ERROR: Output timestamp format is invalid as [Illegal pattern character 'e']..
I have discovered that this seems to be related to the inputs.conf line output_timestamp_format = epoch , as if I change the value from epoch to test in the conf, restart Splunk, the error in dbx2.log is then:
09/17/2015 10:26:14 [ERROR] [ws.py] [DBInput Service] ERROR: Output timestamp format is invalid as [Illegal pattern character 't']..
09/17/2015 10:26:14 [ERROR] [websocket.py] ERROR: Output timestamp format is invalid as [Illegal pattern character 't']..
Interestingly, this issue does not affect my other inputs; these are not attempting to extract timestamp information from the rows, but just use the index time, however, have the exact same config for output_timestamp_format. Example stanzas:
[mi_input://PNET07_VIEWS]
connection = PNET07_DB
index = pnet07
interval = 86400
max_rows = 1000000
mode = batch
output_timestamp_format = epoch
query = SELECT * FROM "storm_CCA-SCI-PNET07"."sys"."SYSVIEWS"
source = sysviews
sourcetype = views
ui_query_catalog = storm_CCA-SCI-PNET07
ui_query_mode = simple
ui_query_schema = sys
ui_query_table = SYSVIEWS
[mi_input://PNET07_TABLES]
connection = PNET07_DB
index = pnet07
interval = 86400
max_rows = 1000000
mode = batch
output_timestamp_format = epoch
query = select table_name, count from systable where primary_root<>0 and creator=1 order by 1
source = systable
sourcetype = tables
ui_query_catalog = storm_CCA-SCI-PNET07
ui_query_mode = advanced
ui_query_schema = sys
ui_query_table = SYSVIEWS
Has anyone come across this bizarre issue before?
Thank you,
Carl
... View more
