Getting Data In

What configuration is required to index a single log with one event only, transforms.conf or props.conf?

AdsicSplunk
New Member

Hi,
My query is that Splunk indexer is indexing a single log with two separate events whereas it should be one event only.
The issue is that I am receiving two timestamps in a single log and I need Splunk to index it as a single event only.

Full Event Expected:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>
          <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
          <receiptNumber>987654321</receiptNumber>
        </DEF>
      </tickets>
    </A>
  </ABC>

Received Event 1:-

[2018-01-31T15:23:25.470+04:00]...abc.....def...........ghi...........
......................................................................
......................................................................
  <ABC xmlns="http://tempuri.org/">
    <A>
      <ID>1234567</ID>
      <tickets>
        <DEF>
          <ticketNumber>12345</ticketNumber>
          <paidAmount>100</paidAmount>

Received Event 2:-

      <paymentDateTime>2015-02-10T15:25:19Z</paymentDateTime>
      <receiptNumber>987654321</receiptNumber>
    </DEF>
  </tickets>
</A>

Could anyone please suggest me how to proceed with this and what parameters to use for configuring props.conf or ?transforms.conf(if required)?

0 Karma

adonio
Ultra Champion

hello there:

in inputs.conf:

[monitor://path.to.file]
index = index
sourcetype = your_sourcetype

in props.conf on indexer or heavy forwarder:

 [your_sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m%dT%H:%M%s.%3N:%z
TIME_PREFIX=[
MAX_TIMESTAMP_LOOKAHEAD=30
BREAK_ONLY_BEFORE=\

further reading regarding where to place files and which configurations goes in each file here:
https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

hope it helps

0 Karma

AdsicSplunk
New Member

Hi Adonio,

Thank you for your support!!
i tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @AdsicSplunk,

Please try below configuration in props.conf on Indexer/Heavy Forwarder whichever comes first.

props.conf

[yoursourcetype]
TIME_PREFIX = ^\[
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%:z
MAX_TIMESTAMP_LOOKAHEAD = 29

Restart splunk on Indexer/Heavy Forwarder.

I hope this helps.

Thanks,
Harshil

0 Karma

AdsicSplunk
New Member

Hi Harsmarvania,

Thank you for your support!!

I tried this config in props.conf but it got worse for me. Now, my indexer is creating even more events breaking each line and putting each line in a separate event. My question is that my event should not break into 2 events but should create one event only ignoring the second timestamp coming inside the event. Please read my questions, if you need some clarifications on this. please feel free to ask me questions.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @AdsicSplunk,

I tried with below configuration in splunk

props.conf

[mysourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%:z
TIME_PREFIX=^\[
MAX_TIMESTAMP_LOOKAHEAD=29

And it is working perfectly fine.

Please refer screenshot https://imgur.com/a/BnQJ9

If this does not work for you, can you please let us know whether do you have any whitespace before [2018-01-31T15:23:25.470+04:00] ?

0 Karma

AdsicSplunk
New Member

Hi @harsmarvania57,

This is the same config that Adonio provided.
I tried this config but now the events are created not based on [2018-01-31T15:23:25.470+04:00] which was being picked without props.conf config earlier. Now, this config has scattered the events and displaying the results which are not at all good. My requirement is to create events based on [2018-01-31T15:23:25.470+04:00] not based on 2015-02-10T15:25:19Z. Please provide your inputs.

Note:- There is no space in the timestamp. The data begins with "[" only.

0 Karma

AdsicSplunk
New Member

The screenshot is not accessible. Could you please share again?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If you look at TIME_FORMAT parameter closely in config which is provided by @adonio and config which I have provided then there are difference . You can see screenshot here as well https://prnt.sc/i8i709

0 Karma

AdsicSplunk
New Member

I tried with both configs, still I am not getting what is required. Anyway, Thank you for your support, I will try again to get to the desired requirement.

Really appreciate your support.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...