Events posted to HEC successfully, not showing up in index

[Cuyose]

I am able to use the supplied official documentation on posting to a configured HEC, and I receive the correct success response, however the events are never indexed.

[Cuyose]

Ok,

I got this to work. It would have been much easier with a one sentence edit in the documentation. The documented examples are in json, however when setting up the token you get to also select a sourcetype. I selected _json and it did not work. It was looking for a timestamp. You need to select _json_no_timestamp for the examples to work as documented.

[gjanders]

however when setting up the token you
get to also select a sourcetype

Wouldn't the issue here be that your using a field extracted sourcetype that requires a time field? I believe a non-field extracted sourcetype would have worked just fine (or even not specifying the sourcetype at all).

However you could provide feedback to the docs team around this

[Cuyose]

That's what I thought, however when providing a time in the event Json, and selecting the _json structured sourcetype, it would not work, as none of the examples spoke to the importance of the selected sourcetype when following the provided examples, this wasted a lot of troubleshooting time.

I would suggest that the sample curl in the doc's include a time element, as that is key to how Splunk works, rather than simply sending a string message, which shows a successful post but does not get indexed.

[woodcock]

Please do click Accept on your answer and do include all of this feedback on the docs page.

[gjanders]

As per woodcock please accept your answer, however I've used that test curl command before without issue which implies that it might be something related to your chosen sourcetype.

Anyway, the docs team is open to feedback!

[andrewb_splunk]

Cuyose, the Splunk doc team is very receptive to feedback. If you submit your suggestion using the feedback form on the bottom of the topic that needs improvement, the doc team will receive it and can take action on it. Sorry that the process was more difficult than it needed to be!

[PowerPacked]

Hi @Cuyose

check if the events are going to the default -- main index.

& u can see in - settings - data - indexes - your indexname --- & see if it recieved any events.

& also settings - data - data inputs - HTTP Event Collector - to see what was the index & token & many other settings.

Thanks

[prasannjeet]

Hey PowerPacked, thanks for your answer. I see that the events are logged successfully in the index. It has indeed received all the events that I sent. However, I don't know how to see those events. I am logged in as admin. Any ideas? Thanks!

[prasannjeet]

Nevermind, I found the problem. Firstly, it was because the index was not set to "main", and the other was the host url. It wasn't "https". Fixing both made the logs pop up in my dashboard. Thank you!

[sujith549]

Hi @Cuyose

Do you got fixed the issue?, we are facing same issue verified all the Token,Index names. Able to see event got logged into _introspection.log but not able see the event info at any index( main or customIndex)

[Cuyose]

Thanks, all of this was checked, data is not showing up in any index. My best assumption is it may be permissions related, either having access to view or within the backend with permissions to write. As I stated _introspection shows the metrics of the event it saw come in successfully, but on the index management page within splunk, no events are being added to any index.

Is there some condition where an event will not get written to an index, and not log an error in _internal?

[Cuyose]

I am not seeing any warnings or errors anywhere. Just unable to find the test messages in any index at all anywhere after the successful response that the event was posted.

[pradeepkumarg]

Is your HEC instance a heavy forwarder or indexer? Is the data going to a new index? Is the index created?
Search for a larger time range?

[Cuyose]

This is on standalone enterprise test instance. indexes are created and searchable.

[Cuyose]

I am even seeing it was indexed in the _introspection logs, but just cant search it
{"datetime":"07-09-2018 11:59:27.627 -0700","log_level":"INFO","component":"HttpEventCollector","data":{"token_name":"hectest","series":"http_event_collector_token","transport":"http","format":"json","total_bytes_received":50,"total_bytes_indexed":14,"num_of_requests":1,"num_of_events":1,"num_of_errors":0,"num_of_parser_errors":0,"num_of_requests_to_disabled_token":0,"num_of_requests_in_mint_format":0}}

[pradeepkumarg]

Did you expand your time range and search for a longer period to account for any timestamp/timezone issues?

[Cuyose]

Searching all time, still nothing

[Cuyose]

I should note, I am admin, full privilages