Getting Data In

How do I edit props.conf on my indexer to prevent automatic sorting of events?

seetharamanss
Explorer

Hi,

I have a problem when indexing the events through a forwarder. The forwarder is listening to a log file with first event (first record) as header event. But I could see the events are not coming as is into the indexer. It is getting sorted on datetime.

Log file:

REQID~INPCD~RQTPCD~WKFWNM~INSTNM~NODENM~PRTNCT~STRTTS~ENDTS~DURMSCT~VOLCT~REVOLCT~APPNM~RULENO~NMSCORE~CONMSCORE~ADDRSCORE~PHSCORE~SSNSCORE~YOBSCORE~MATCHIND
4472276843203022016174103072                                ~RT ~RT ~node01~RUNINSTANCE4~PCIS_ABCDPB_01~1~2016-03-02 17:41:02.771~2016-03-02 17:41:03.072
~1~1~1~1 
4477576843003022016175856617                                ~RT ~RT ~node01~RUNINSTANCE2~PCIS_ABCDPB_01~1~2016-03-02 17:58:56.022~2016-03-02 17:58:56.617
~1~1~1~1                          ~NA~NA~NA~NA~NA~NA~NA~N

In indexer:

4477576843003022016175856617                                ~RT ~RT ~node01~RUNINSTANCE2~PCIS_ABCDPB_01~1~2016-03-02 17:58:56.022~2016-03-02 17:58:56.617
~1~1~1~1                          ~NA~NA~NA~NA~NA~NA~NA~N
4472276843203022016174103072                                ~RT ~RT ~node01~RUNINSTANCE4~PCIS_ABCDPB_01~1~2016-03-02 17:41:02.771~2016-03-02 17:41:03.072
~1~1~1~1 

Props.conf in indexer

NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d
TRUNCATE=300000
pulldown_type = 1

Please advise the correct configuration to get the event as is from the log file.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Splunk Indexes data as soon as they are received and they might not know when your file writing finishes. These events are persisted to disk in their original arrival order. However, in search results events are retrieved in inverse time order to show the latest events first.

You can sort events during search to see events in arrival order.

Happy Splunking!
0 Karma

seetharamanss
Explorer

The first record in the file is header record. In the search head,I'm trying to extract the fields by creating the props.conf and transforms.conf in $splunk_HOME/etc/apps/appname/local . But I couldn't see the fields getting extracted ,Is it because of order I'm not able to extract the fields ? Please advise.

props.conf

[12345678_abcd_Analytic_logs]
SHOULD_LINEMERGE=false
KV_MODE = none
CHECK_FOR_HEADER = TRUE
REPORT-ext = detail_abcd_fields
pulldown_type = 1

transforms.conf

[detail_abcd_fields]
DELIMS = "~"
FIELDS = REQ_ID,INP_CHAN_CD,REQ_TYP_CD,INFA_WKFLOW_NM,INFA_INSTNC_NM,INFA_NODE_NM,INFA_PRTN_CT,PRCS_STRT_TS,PRCS_END_TS,PRCS_DUR__CT,INP_VOL_CT,REFER_VOL_CT,APPL_NM
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

The order of indexing should not affect the extraction process. From the above configuration, it more looks like an indexer time extraction and for that you have to place the configuration on indexers.

If you just want search time extractions, try with splunk web

http://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/Managesearch-timefieldextractions

If you still want to do search time field extractions using config files, refer to http://docs.splunk.com/Documentation/Splunk/6.4.1/Knowledge/Createandmaintainsearch-timefieldextract...

Also remember, whenever you do configurations using files, then you need to restart splunk and with the web, it's not required.

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...