Here's my local props.conf.
[tmweb@app1.splunkdev.jetdev2.syseng.tmcs ~]$ cat /opt/splunk-efr/splunk/etc/system/local/props.conf
[default]
TRUNCATE=100
TIME_PREFIX = datetime=
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
SHOULD_LINEMERGE = False
TZ=Canada/Eastern
I don't understand why those settings are not considered. No matter what I'm doing, Splunk does not consider my setting.
Please, I'm really upset about this. I'm using a new version of splunk, and it was working very nicely before. Now, sometime it's works, sometimes not.
Here's a result in Splunk. As you see, date time is not considered, nor truncation.
08/12/2015 10:23:15.000
severity="INFO" host="app1.jdas.jetdev2.syseng.tmcs" service_version="1.4.47" client_host="app2.jdat.jetdev2.syseng.tmcs" client_version="1.5.97" Correlation-ID="a3b3fde3-e657-42bc-804e-4677ce4de1a7" client_rid="73891577-0da4-4f7f-ac07-b1497ab68246" rid="249cce85-4b42-43e7-8ea6-7e7b951993fc" sid="8d2c2fc8-9981-4b90-a6e1-b5190c1874d7_143" thread="http-8080-2" category="com.ticketmaster.platform.bam.strategies.PayloadBAMStrategy" datetime="2015-12-08T15:23:12.805Z" bam="payload" appCode="outbound.response.rest" activity="FindAttractions" seq="108697" payload="{'@type':'{http://www.ticketmaster.com/jdas/authoring/attraction/service/operations/1.0}FindAttractionResults','record':[{'@type':'{http://www.ticketmaster.com/jdas/authoring/attraction/service/operations/1.0}FindAttractionRecord','attractionVersion':{'@type':'{http://www.ticketmaster.com/jdas/authoring/attraction/model/1.0}AttractionVersion','attractionVersionId':'dataAdmin-attrVer-00000000027b9fe9','attractionId':'dataAdmin-attraction-000000000083d829'}}],'totalRecords':326,'nbRecords':1}" headers="{Response-Code=[200], TMPS-Request-Id=[249cce85-4b42-43e7-8ea6-7e7b951993fc], TMPS-Correlation-Id=[a3b3fde3-e657-42bc-804e-4677ce4de1a7], TMPS-Service-Version=[1.4.47], TMPS-Hostname=[app1.jdas.jetdev2.syseng.tmcs], Cache-Control=[no-cache], Date=[Tue, 08 Dec 2015 15:23:12 GMT]}"
question : are you using an universal or lightweight forwarder, and are those props on he indexer or on the forwarder ?
Because in general, the events are parsed by the indexers, so you may want to put those props on :
- indexers
- intermediary heavy forwarder (if any)
First of all, you should not be modifying that file, but you should create a new file inside $SPLUNK_HOME/etc/apps/YourApp/default/props.conf
. Second of all, you should not be using the [default]
stanza header, but something that is specifically targetted to your events. Thirdly, I suspect that the Z
means Zulu
and is a TZ
specifier and suspect that your TZ
value is wrong. Lastly, I am highly skeptical that this was working just fine before (or at any time). In any case, make the packaging changes that I described and then try this:
[YourSourcetypeHere]
TIME_PREFIX = datetime="
MAX_TIMESTAMP_LOOKAHEAD = 11
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
SHOULD_LINEMERGE = false
TZ=Canada/Eastern
Then deploy to your Indexers, restart all splunk instances there and verify events that are forwarders AFTER the restarts are OK.
MMM I understand skepticism. I'm not very good in splunk configuration. And also my english is not perfect to.
So your're saying that I should not modify that file $SPLUNK_HOME/etc/system/local/default/props.conf ? This is really strange because is what I took from this link:
https://answers.splunk.com/answers/44041/editing-props-conf-transforms-conf-configuration-file.html
By the way, I try to edit $SPLUNK_HOME/etc/apps/search/default/props.conf, but nothing has change. Looks like my configuration is not considered ???
Any other's idea ?
It is true that /local
is almost always preferrable to /default
, which is the point of the link that you referenced. It is also true that $SPLUNK_HOME/etc/apps/
is almost always better than $SPLUNK_HOME/etc/system
. You should not make making a non-system-wide (global-scope) change inside of $SPLUNK_HOME/etc/system
, which is my point. To make it worse, you are using the [defaiult]
stanza which makes your local-scope configurations applied to everything everywhere, which is definitely a bad idea, which again is my point. Try to package your configurations in an app so that they apply ONLY to your input. It is MUCH easier to debug this way and you won't run the (HUGE) risk of breaking other stuff.
Ok, I've done everythings that you said, but nothing has changed.
From a brand new installation of indexer and heavy forwarder, I only add this as configuration of my indexer and splunk always merge line ???
/opt/splunk-efr/splunk/etc/apps/search/default/props.conf
[source::/app/local/log/payload_log]
SHOULD_LINEMERGE = False
Here's 2 lines merged as example and there are coming from heavy forwarder
06:58:30.000
severity="INFO" host="app1.cops.intqa102.syseng.tmcs" service_version="1.4.10" client_host="" client_version="2.0.9" Correlation-ID="20151217123221_f4f77d69-828e-41e2-acf9-eedf2d9d72ba" client_rid="" rid="10bfe7d0-3b39-436b-b612-824b630e1dc9" sid="" thread="http-8080-2" category="com.ticketmaster.platform.bam.strategies.PayloadBAMStrategy" datetime="2015-12-18T14:58:28.838Z" bam="payload" appCode="request.rest" activity="" seq="81289" payload="{'common':{'id':{'tmonline-us':{'id':'516110'},'primary':{'id':'dataAdmin-venue-000000000008d36a'},'jetson':{'id':'dataAdmin-venue-000000000008d36a'}},'name':{'es-us':'BUFFALO_ROSE ID = 516110 modified at 09:15','en-au':'BUFFALO_ROSE ID = 516110 modified at 09:15','es-mx':'BUFFALO_ROSE ID = 516110 modified at 09:15','fr-ca':'BUFFALO_ROSE ID = 516110 modifié a 13:31','en-nz':'BUFFALO_ROSE ID = 516110 modified at 09:15','en-ca':'BUFFALO_ROSE ID = 516110 modified at 09:15','en-us':'BUFFALO_ROSE ID = 516110 modified at 09:15','en-mx':'BUFFALO_ROSE ID = 516110 modified at 09:15'},'isTest':false,'source':'tmonline-us','image':[{'width':650,'height':366,'url':'http://qaip10.ticketmaster.net:8001/dbimages/17751v.jpg','path':'/dbimages/17751v.jpg','fallback':fa... Unidos','en-au':'United States Of America','es-mx':'Estados Unidos','fr-ca':'États-Unis','en-nz':'United States Of America','en-ca':'United States Of America','en-us':'United States Of America','en-mx':'United States Of America'},'countryCode':'US'},'state':{'name':{'es-us':'New York','en-au':'New York','es-mx':'New York','fr-ca':'New York','en-nz':'New York','en-ca':'New York','en-us':'New York','en-mx':'New York'},'stateCode':'NY'},'city':{'name':{'es-us':'new york','en-au':'new york','es-mx':'new york','fr-ca':'new york','en-nz':'new york','en-ca':'new york','en-us':'new york','se-eu':'new york','en-mx':'new york'}},'postalCode':'12345','zipCode':'12345','address':{'line1':{'es-us':'1495, du Golf','en-au':'1495, du Golf','es-mx':'1495, du Golf','fr-ca':'1495, du Golf','en-nz':'1495, du Golf','en-ca':'1495, du Golf','en-us':'1495, du Golf','se-eu':'1495, du Golf','en-mx':'1495, du Golf'},'line2':{'es-us':'Adress line 2','en-au':'Adress line 2','es-mx':'Adress line 2','fr-ca':'Adress line 2','en-nz':'Adress line 2','en-ca':'Adress line 2','en-us':'Adress line 2','se-eu':'Adress line 2','en-mx':'Adress line 2'}},'location':{'latitude':-72.12345675,'longitude':72.12345675},'timezone':'America/Phoenix','dma':[{'id':281}],'market':[{'id':{'tmonline-us':{'id':'23'},'primary':{'id':'23'}}}],'currency':'USD','isAvailableForNewEvents':true,'hostData':{'host':'PHX','venueNumber':14}}" headers="{Address=[http://app.cops.intqa102.syseng.tmcs:8080/rest/content-persistence-service/1-0/persist/unifiedVenue], Http-Method=[POST], Accept=[text/plain, application/json, application/+json, */], accept-charset=[big5, big5-hkscs, cesu-8, euc-jp, euc-kr, gb18030, gb2312, gbk, ibm-thai, ibm00858, ibm01140, ibm01141, ibm01142, ibm01143, ibm01144, ibm01145, ibm01146, ibm01147, ibm01148, ibm01149, ibm037, ibm1026, ibm1047, ibm273, ibm277, ibm278, ibm280, ibm284, ibm285, ibm290, ibm297, ibm420, ibm424, ibm437, ibm500, ibm775, ibm850, ibm852, ibm855, ibm857, ibm860, ibm861, ibm862, ibm863, ibm864, ibm865, ibm866, ibm868, ibm869, ibm870, ibm871, ibm918, iso-2022-cn, iso-2022-jp, iso-2022-jp-2, iso-2022-kr, iso-8859-1, iso-8859-13, iso-8859-15, iso-8859-2, iso-8859-3, iso-8859-4, iso-8859-5, iso-8859-6, iso-8859-7, iso-8859-8, iso-8859-9, jis_x0201, jis_x0212-1990, koi8-r, koi8-u, shift_jis, tis-620, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-32be, utf-32le, utf-8, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258, windows-31j, x-big5-hkscs-2001, x-big5-solaris, x-compound_text, x-euc-jp-linux, x-euc-tw, x-eucjp-open, x-ibm1006, x-ibm1025, x-ibm1046, x-ibm1097, x-ibm1098, x-ibm1112, x-ibm1122, x-ibm1123, x-ibm1124, x-ibm1364, x-ibm1381, x-ibm1383, x-ibm300, x-ibm33722, x-ibm737, x-ibm833, x-ibm834, x-ibm856, x-ibm874, x-ibm875, x-ibm921, x-ibm922, x-ibm930, x-ibm933, x-ibm935, x-ibm937, x-ibm939, x-ibm942, x-ibm942c, x-ibm943, x-ibm943c, x-ibm948, x-ibm949, x-ibm949c, x-ibm950, x-ibm964, x-ibm970, x-iscii91, x-iso-2022-cn-cns, x-iso-2022-cn-gb, x-iso-8859-11, x-jis0208, x-jisautodetect, x-johab, x-macarabic, x-maccentraleurope, x-maccroatian, x-maccyrillic, x-macdingbat, x-macgreek, x-machebrew, x-maciceland, x-macroman, x-macromania, x-macsymbol, x-macthai, x-macturkish, x-macukraine, x-ms932_0213, x-ms950-hkscs, x-ms950-hkscs-xp, x-mswin-936, x-pck, x-sjis_0213, x-utf-16le-bom, x-utf-32be-bom, x-utf-32le-bom, x-windows-50220, x-windows-50221, x-windows-874, x-windows-949, x-windows-950, x-windows-iso2022jp], accept-encoding=[gzip,deflate], accept_encoding=[gzip, deflate], connection=[Keep-Alive], Content-Length=[2229], content-type=[application/json;charset=UTF-8], host=[app.cops.intqa102.syseng.tmcs:8080], tmps-client-app-version=[2.0.9], tmps-client-hostname=[], tmps-correlation-id=[20151217123221_f4f77d69-828e-41e2-acf9-eedf2d9d72ba], tmps-request-id=[10bfe7d0-3b39-436b-b612-824b630e1dc9], user-agent=[Apache-HttpClient/4.3.5 (java 1.5)]}"
severity="INFO" host="app1.cops.intqa102.syseng.tmcs" service_version="1.4.10" client_host="" client_version="2.0.9" Correlation-ID="20151217123221_f4f77d69-828e-41e2-acf9-eedf2d9d72ba" client_rid="" rid="10bfe7d0-3b39-436b-b612-824b630e1dc9" sid="" thread="http-8080-2" category="com.ticketmaster.platform.bam.strategies.PayloadBAMStrategy" datetime="2015-12-18T14:58:28.859Z" bam="payload" appCode="outbound.response.rest" activity="PersistUnifiedVenue" seq="81296" payload="{'persistedId':'dataAdmin-venue-000000000008d36a','persistenceStatus':'Matched','modifiableStatus':false}" headers="{Response-Code=[200], Date=[Fri, 18 Dec 2015 14:58:28 GMT]}"
If I add the data from ui, really easy to configure. There's something that I really don't understand with props.conf files ?
Hi @efrenette11
To provide more information for users to help you, can you share what version of Splunk you were using before and what version you upgraded to exactly? You mentioned your props.conf worked before, but after upgrading, that's when you started having an issue. Always include as much detail as you can in your post for users to have a full picture.