I have an HTTP Event Collector input collecting JSON data via syslog forwarder. The syslog-ng message looks like:   body("{ \"source\": \"${.splunk.source}\", \"event\": ${MSG} }")   I can see the message and the proper source in my indexer. But time extraction is the problem. Because there is often a delay between the log and the time syslog receives it, I want to use a field in the message to grab the timestamp... A message looks like this:   {"data":"stuff","time_stamp":"2022-05-10 17:14:23Z","value1":"more_stuff"}   So, I create a props.conf on my indexer cluster that looks like:   [my_sourcetype] DATETIME_CONFIG = MAX_TIMESTAMP_LOOKAHEAD = 30 TIME_PREFIX = time_stamp\":\" TIME_FORMAT = %Y-%m-%d %h:%M:%S%Z TZ = UTC   I've confirmed the sourcetype is correct as I define it in the inputs for HEC and they match. But for the life of me, I can't seem to get Splunk to find the time. I've tried looking for errors using this search:   index = _internal log_level = WARN OR log_level =ERROR "timestamp"   (found in an older community post, so thanks to the author) but I find nothing. I tried playing in the UI and creating a new data type (the add new data widget). In that UI, my props.conf should work. But for some reason on the cluster it doesnt. Are there any other troubleshooting steps I can follow? Am I missing something that might help this work better?   ... View more

I have an indexer cluster and a search head environment. I've deployed the Splunk_TA_fortinet_fortigate app on both the search head and the cluster. Logs come in via syslog to syslog-ng where they are shipped to the indexer via HTTP Event Collector's raw endpoint. Logs come in via the fortigate_log sourcetype. Then the TA sets the sourcetype to the correct type via a transform. Those sourcetypes then specify TIME_PREFIX=^ in the props.conf. However, this doesn't work as there is no date field for Splunk to get the date from. There is a time field though... (see sample data below). What I want is to use the eventtime field as the timestamp. So, I create a local/props.conf file that looks like this:   [fortigate_log] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N [fgt_log] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N [fortigate_traffic] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N [fortigate_utm] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N [fortigate_anomaly] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N [fortigate_event] MAX_TIMESTAMP_LOOKAHEAD = 0 TIME_PREFIX = eventtime= TIME_FORMAT = %s%9N   The idea comes from a few places, notably these documents: https://splunkbase.splunk.com/app/2800/#/details (there is a section which has that) https://community.splunk.com/t5/Getting-Data-In/How-to-configure-int64-epoch-nanosecond-timestamp-as-time/m-p/26881 I deployed it to the indexer cluster first just in the traffic, utm blocks thinking that I need to override what is in the default props.conf. But that didn't help. So I added it to the incoming fortigate_log sourcetype hoping it would do the time extraction earlier in the ingestion process. Neither seems to do anything. I also tried putting it on the search head thinking that the configuration bundle it sends to the cluster may be overriding my config. Still nothing. What am I doing wrong? Any ideas? Thanks, Scott Sample data:   time=23:59:59 devname="hostname" devid="devid" slot=1 logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1630645200040167048 tz="-0500" srcip=1.2.3.4 srcport=35847 srcintf="port25" srcintfrole="undefined" dstip=2.3.4.5 dstport=49164 dstintf="port26" dstintfrole="undefined" srcuuid="xxx" dstuuid="xxx" sessionid=455176702 proto=17 action="accept" policyid=10873 policytype="policy" poluuid="xxxx" service="udp/49164" dstcountry="United States" srccountry="United States" trandisp="noop" duration=11699117 sentbyte=49457564285 rcvdbyte=0 sentpkt=164980295 rcvdpkt=0 appcat="unscanned" sentdelta=49457564285 rcvddelta=0     ... View more

I'm trying to use the python sdk to build a custom search command. In my commands.conf, I have "chunked = true" set. I should therefore be using version 2 of the search command language. In my python script, I have the following lines: from splunklib.searchcommands import dispatch, StreamingCommand, Configuration @Configuration(distributed=False,type='streaming') When I remove the type variable from the configuration decoration, the command tries to execute on all my indexers. However, I want it to run local on the search head for now so I can test it and fix issues without having to deploy it all over the place. When I add the type variable, I get the following error messages. There is a valueerror saying that I can't set the type variable. Is there a way around this problem? 01-29-2020 20:07:03.600 INFO ChunkedExternProcessor - Running process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/myapp/bin/myapp.py 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last): 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/etc/apps/myapp/bin/myapp.py", line 9, in <module> 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: class myappCommand(StreamingCommand): 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/etc/apps/myapp/bin/lib/splunklib/searchcommands/decorators.py", line 84, in __call__ 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: ConfigurationSetting.fix_up(o.ConfigurationSettings, self.settings) 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: File "/opt/splunk/etc/apps/myapp/bin/lib/splunklib/searchcommands/decorators.py", line 193, in fix_up 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: raise ValueError('The value of configuration setting {} is fixed'.format(name)) 01-29-2020 20:07:03.710 ERROR ChunkedExternProcessor - stderr: ValueError: The value of configuration setting type is fixed 01-29-2020 20:07:03.720 ERROR ChunkedExternProcessor - EOF while attempting to read transport header 01-29-2020 20:07:03.721 ERROR ChunkedExternProcessor - Error in 'myapp' command: External search command exited unexpectedly with non-zero error code 1. 01-29-2020 20:07:03.721 ERROR SearchPhaseGenerator - Fallback to two phase search failed:Error in 'myapp' command: External search command exited unexpectedly with non-zero error code 1. 01-29-2020 20:07:03.722 ERROR SearchOrchestrator - Error in 'myapp' command: External search command exited unexpectedly with non-zero error code 1. 01-29-2020 20:07:03.722 ERROR SearchStatusEnforcer - sid:1580328423.9 Error in 'myapp' command: External search command exited unexpectedly with non-zero error code 1. 01-29-2020 20:07:03.722 INFO SearchStatusEnforcer - State changed to FAILED due to: Error in 'myapp' command: External search command exited unexpectedly with non-zero error code 1. ... View more

I am creating a modular input. My input is a CSV and I convert it to JSON to be imported as a new event in Splunk. Several of the fields have newlines in the data. However, once indexed the newlines are removed. Here is the code that does it: csvdata = [row for row in csv.reader(data.splitlines())] header = csvdata.pop(0) for row in csvdata: e = {} for col, val in zip( header, row ): col = col.replace( " ", "_" ) e[col] = val event_time = calendar.timegm(time.strptime(e["timefield"], time_pattern)) event = helper.new_event(data=json.dumps(e), time=event_time, index=index, unbroken=True) ew.write_event(event) One thing I've tried is adding the SHOULD_LINEMERGE=0 to props.conf which didn't work. Is there a way to tell Splunk not to remove the newlines from fields? Thanks! ... View more