Getting Data In

long xmls are split into multiple events in splunk?

madhanbaskar
Explorer

Transport : GoodTransport
System : ESS
JMS Message ID : ID:414d512042542e51e37d79596dde3421
Queue JNDI Name : MQ_ESRequest
Event ID : 31ae5b1#15dbb8ab#-688
Request Send Start Time : Mon Aug 07 08:08:23 2017
Request Send End Time : Mon Aug 07 08:08:23 2017
Status : Success
-------------------------------------------REQUEST XML-------------------------------------------
256 lines
------------------------------------------END OF REQUEST-----------------------------------------

In the above XML Transport and System fields are split into 1 event, JMS ID Queue and EventID are split into another event and also in the XML there are many splits in between.

How should I write the props.conf so that I can get complete xml as a single event..?
Also I should modify the props.conf which is present in SPLUNK_HOME/etc/system/local right?

Please help guys...!

Tags (1)
0 Karma

woodcock
Esteemed Legend

You need TRUNCATE. See this excellent answer by @yannK (be sure to UpVote him):
https://answers.splunk.com/answers/90586/can-i-change-truncate-and-max-events-to-unlimited.html

0 Karma

madhanbaskar
Explorer

@woodcock

Props.conf :
[ofs_logs]
TRUNCATE = 50900
MAX_EVENTS = 260

I have added this in my /opt/splunk/etc/system/local/props.conf file

Even then I couldn't find the events getting rightly split.

Please help!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What sourcetype are you using for this event? What are the props.conf settings for that sourcetype?

Yes, modify the local file, never default.

---
If this reply helps you, Karma would be appreciated.
0 Karma

madhanbaskar
Explorer

@niketnilay,

Can you please help me on this query?

0 Karma

niketn
Legend

@madhanbaskar, it would be a bit tough to assist without having a look at your data. If you can anonymize/mask sensitive information in your data and paste at-least one complete event using code button on Splunk Answers, it would be easier for Splunkers here to assist you.

Essentially, Splunk indexes Time Series Data, so it is interested in two things
1) Identify Event Timestamp correctly and
2) Identify Events with correct line breaking.

You can take a sample event (or maybe handful events) from your MQ logs put it a dummy file and upload the same in Preview mode on a test Splunk server or may be local machine with Splunk Enterprise. You can try using following documentations to come up with proper Timestamp Extraction and Event Breaking: https://docs.splunk.com/Documentation/Splunk/latest/Data/Setsourcetype#Adjust_time_stamps_and_event_...
https://docs.splunk.com/Documentation/Splunk/latest/Data/Modifyeventprocessing
https://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf#Line_breaking

Since your first event is getting 256 lines, I expect you have not handled Timestamp recognition and in your data Timestamp is not present in first 256 characters. So you should consider supplying the regular expression to find the same.

Further, to identify events correctly you would need either event beginning pattern or event ending pattern, whichever is possible. Preview mode with sample data will let you know whether the events are getting extracted properly with correct timestamp or not.

Worst case if you can not post the mocked up sample data here or not able to figure out the props.conf settings yourself you can also reach out to Splunk Support with your Splunk License Entitlement so that they can assist you further.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

madhanbaskar
Explorer

@niketnilay

below is one such example ...

Not sure how to write props.conf for this kinda(----) event start and event end.

please help!

we can use ------------------------------------------END OF REQUEST----------------------------------------- for event breaking (break after) because this would be same for all the xml.


Transport : Transport
System : EC
JMS Message ID : ID:41436d25
Queue JNDI Name : MQ_ESC
Event ID : -170f26811
Event context : USR:'e2e'-SYS:'System'-TX:'Packager'-LOC:''-SEQ:''-CID:''-E2EData:'19=1.1,15=O,16=ESC,13=P,11=REQ
Request Send Start Time : Fri Aug 18 06:45:24 2017
Request Send End Time : Fri Aug 18 06:45:24 2017
Status : Success
-------------------------------------------REQUEST XML-------------------------------------------

<stan:e2e>
 <stan:E2EDATA>1c:3xj4g4iav2</stan:E2EDATA>
</stan:e2e>
<stan:serviceState>
 <stan:stateCode></stan:stateCode>
</stan:serviceState>
<stan:serviceAddressing>
 <stan:from>by</stan:from>
 <stan:to>
  <stan:address>om</stan:address>
 </stan:to>
 <stan:replyTo>
  <stan:address>by</stan:address>
 </stan:replyTo>
 <stan:faultTo>
  <stan:address></stan:address>
 </stan:faultTo>
 <stan:serviceName>2</stan:serviceName>
 <stan:action>fil</stan:action>
</stan:serviceAddressing>
<stan:serviceSpecification>
 <stan:payloadFormat>XML</stan:payloadFormat>
 <stan:version>1</stan:version>
</stan:serviceSpecification>

8088

<ns:OrderIdentifier>80964</ns:OrderIdentifier>
<ns:CustomerOrderNumber>VO</ns:CustomerOrderNumber>
<ns:OriginationSystemOrderRef>VO</ns:OriginationSystemOrderRef> <ns:OrderAction>MODIFY</ns:OrderAction>
<party:customerSupplyOrderParty>
 <party:CustSupplyOrderPartyRole>DEL</party:CustSupplyOrderPartyRole>
 <party:CustSupplyOrderPartyName>Mr</party:CustSupplyOrderPartyName>
</party:customerSupplyOrderParty>
<party:customerSupplyOrderParty>
 <party:CustSupplyOrderPartyRole>LER</party:CustSupplyOrderPartyRole>
 <party:CustSupplyOrderPartyName>200</party:CustSupplyOrderPartyName>
</party:customerSupplyOrderParty>

------------------------------------------END OF REQUEST-----------------------------------------

0 Karma

madhanbaskar
Explorer

@niketnilay

Also this is another xml where I need to split it across timestamp,

what regex should I supply for the this kinda split?

2017082127689978;SALES;E;MAJOR;Order3-3LOL013-2LCMOHQ21/08/2017 7:41:29:978;;0
2017082127690046;SALES;I;INFO;Manager.NotifyKCIEvent;Inputs.GetChildCount()21/08/2017 7:41:30:46;;0;
2017082127690053;SALES;I;INFO; Strategic Manager.KCI Inputs -- [ProductFamily] ='WLR'||[ProdSubType] ='Service Product'||[EntityVal] ='Complete'||[ProdFullFillType] ='OTHER'||[EntityVal3] ='Add'||[ParentEntityId]||[CustomerId]|[Product]|='Required'||[EntityId]||[EntityType]||
2017082127690060;SALES;I;INFO;In Validate - Search Spec Built is :; [Entity Type] ='OrderLineItem' AND [Entity Value1] ='Complete' AND ([Product Family]

0 Karma

madhanbaskar
Explorer

@richgalloway,
Props.conf :
[ofs_logs]
TRUNCATE = 50900
MAX_EVENTS = 260

I have added this in my /opt/splunk/etc/system/local/props.conf file

Even then I couldn't find the events getting rightly split.

Please help!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...