I'm using the Splunk Add-on for Kafka to collect events from a kafka topic. The kafka server was on the same machine with splunk, so network should not be an issue. There is a shell script continuously populating events into a certain kafka topic with 16 partitions (let's say topic "A"), when I use kafka-console-consumer.sh to check topic "A", it prints events continuously which is just as expected. So I thought my kafka setup was correct. Now the problem is, When I was populating the events into the topic "A" in normal speed (50-80 events per minute), the splunk add-on for Kafka fetches messages in a strange behavior, it got hundreds of events in 1 minute but "sleep" for another 6-7 minutes. Just like the chart below, But, when I increased the populating speed to 1000 msg/s or 5000 msg/s, splunk got those messages immediately and in expected throughput, please see the screenshots below, It seems like there is some "buffer" in the add-on which will wait for certain count of messages to flush into splunk... btw, the Kafka version is: kafka_2.12-0.10.2.0 the inputs.conf content: [kafka_mod] interval = 5 [kafka_mod://kfk1] index = aaa kafka_cluster = bbb kafka_partition_offset = earliest kafka_topic = A kafka_topic_group = splunk kafka_partition = 0 [kafka_mod://kfk2] index = aaa kafka_cluster = bbb kafka_partition_offset = earliest kafka_topic = A kafka_topic_group = splunk kafka_partition = 1 [kafka_mod://kfk3] index = aaa kafka_cluster = bbb kafka_partition_offset = earliest kafka_topic = A kafka_topic_group = splunk kafka_partition = 2 ... //16 partitions in total ... ... View more

In the document I found: With raw data, ...... This is particularly useful for sending data to a non-Splunk system. In my case, the server which forwarder was installed has heavy load, and all the log files are zipped. So the forwarder has to decompress the packages then read the log files and send data to indexer, this consumes extra CPU time, and leads to slow data forwarding. So I wonder is there a way to tell the forwarder don't decompress the ZIP packages, just send them to indexer and let indexers do the unzip job? I tried adding "sendCookedData=false" to inputs.conf and restarted the forwarder, but then the indexer refused the connection from the forwarder.... Thanks for anyone may help! ... View more

I set up an alert to trigger a script (sample " echo.bat " in $SPLUNK_HOME\bin\scripts\ ) The alert fired normal, but the script was not executed. When I check the splunkd.log, it says: ERROR ScriptRunner - Couldn't start child process. script="E:\Program Files\Splunk\etc\apps\search\bin\runshellscript.py" ERROR script - Script execution failed for external search command 'runshellscript.' I tried to add python.exe path to %PATH , but it didn't help, strange..... splunk version: 4.3-115073-x64 windows: server 2008 64-bit ... View more

Is there anyway to tell splunk to judge whether some error codes appear in 10 continuous events? The key point is "continuous 10 events", means if some normal event breaks in, then don't need to trigger alert. For example: if return code is in "ER1111" "222222" "GT9999" "empty" or less than zero, and such events appeared continuous for 10-times, then raise a alert. Logs like this should alert because 10 continuous events all match the condition: [2012-05-09 11:10:02] ## XXXXX return=ER1111 [2012-05-09 11:10:03] ## XXXXX return=ER1111 [2012-05-09 11:10:04] ## XXXXX return=222222 [2012-05-09 11:10:05] ## XXXXX return=ER1111 [2012-05-09 11:10:06] ## XXXXX return=GT9999 [2012-05-09 11:10:07] ## XXXXX return=ER1111 [2012-05-09 11:10:08] ## XXXXX return= [2012-05-09 11:10:09] ## XXXXX return=ER1111 [2012-05-09 11:10:10] ## XXXXX return=-3131 [2012-05-09 11:10:11] ## XXXXX return=GT9999 But logs like this should not alert because an event with return=000000 breaked in line 5: [2012-05-09 11:10:13] ## XXXXX return=ER1111 [2012-05-09 11:10:14] ## XXXXX return=-99433 [2012-05-09 11:10:15] ## XXXXX return=-3041 [2012-05-09 11:10:16] ## XXXXX return=222222 [2012-05-09 11:10:17] ## XXXXX return=000000 [2012-05-09 11:10:18] ## XXXXX return=ER1111 [2012-05-09 11:10:19] ## XXXXX return=ER1111 [2012-05-09 11:10:22] ## XXXXX return=222222 [2012-05-09 11:10:26] ## XXXXX return=GT9999 [2012-05-09 11:10:28] ## XXXXX return=222222 [2012-05-09 11:10:31] ## XXXXX return= [2012-05-09 11:10:33] ## XXXXX return=-796 [2012-05-09 11:10:37] ## XXXXX return=-4176 Would be appreciate if someone can help~~ ... View more

Hi Guys Recently I have been dealing with some application logs and met some difficulties with field extraction. Every fieldname is started with a "#", such as #field1=xxx#field2=yyy#field3=zzz#... Then splunk extracted fields like: field1="xxx#field2=yyy#field3=zzz". How to tell splunk to threat "#" as a comma or space (means ignore the #) ? Would someone help me ? Thanks in advance! The log format is like: 2012-04-16 17:41:27:087 [Inner-Tans-90013849] request:a60061152 102600 #oper_flag=1#user_type=2#user_id_type=1#user_id=1234567#user_pwd=897985d09890f41a4300#login_ip=192.168.1.1#bank_no=0000#net_envionment=2#net_agent=1# 2012-04-16 17:41:27:087 [Inner-Tans-90013849] 1.Start:com.klink.btrs.product.system.imp.T9006 2012-04-16 17:41:27:089 [Inner-Tans-90013849] com.klink.btrs.BtrsException: Invalid Username or password！ 2012-04-16 17:41:27:095 [Inner-Tans-10023849] response：a60061152 102600 HJ9999 #rsp_msg=Invalid Username or password！# ... View more

Hi Splunk, Recently we meet the issue of license limit but we did not notice this until we triggered the license violations for 7 times. So the search function was disabled, with a warning message on the top of the browser - "reached the maximum license violations for this time period". Today I updated the license from default trial license (500MB) to 5GB, and restarted splunk, but the search still disabled and we are continuing get this message. Here is the new license info: # ./splunk show license Splunk username: admin Password: Current Daily Usage Amount: 1844873670 Expiration date: 2010-09-29T13:30:42+0800 Expiration State: ok License level: 5120 MB Product: Enterprise License violations: 2010-08-31T00:00:28+0800 License violation #7 2010-08-30T00:04:20+0800 License violation #6 2010-08-29T00:00:21+0800 License violation #5 2010-08-28T00:00:53+0800 License violation #4 2010-08-27T00:03:06+0800 License violation #3 2010-08-26T00:01:19+0800 License violation #2 2010-08-21T00:03:35+0800 License violation #1 Max Violations: 5 Peak usage: 3526 MB Days remaining: 29 day(s) Violation Period: 30 And the error message always there: Error in 'UnifiedSearch': Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting www.splunk.com/store or calling 866.GET.SPLUNK. Please help us! Thanks in advance! Soni Jin Netis Technologies ... View more