How to view events from HTTP Event Connector?

[rajpraba]

I have installed AWS Splunk enterprise. We plan to redirect our Node JS application logs to Splunk. I have setup http event connector and got a token. I am able to send event logs from console (curl) as well as from application. But I dont see it anywhere.

The search/dashboard shows 0 events. But the Index says 1 MB...

curl -k https://54.xx.xx.xx:8088/services/collector -H 'Authorization: Splunk xxxxx  -d '{"source": "test-token-1", "sourcetype": "_json", "index":"sample", "event":"Hello, World!"}' 
{"text":"Success","code":0}

The name of the input token is 'test-token-1'. this is configured as source type _json and default index as 'sample'>

The search shows 0 events

I ssh to the box and see the folder

[ec2-user@ip-172-31-36-245 ~]$ sudo find / -name sample -type d
/opt/splunk/var/lib/splunk/sample

but don't have permission to see the content.

Any help is approeciated.

thanks
Raj

[rajpraba · Accepted solution]

Since, The response from console was
{"text":"Success","code":0}, It appears the event is logged.
After playing with this all day, I got a message,
'Disk Monitor: Cannot write data to index path '/opt/splunk/var/lib/splunk/_internaldb/db' because you are low on disk space on partition '/'. Indexing has been paused. Free disk space above 5000MB to resume indexing.'

However the search is showing 0. So, I would think the admin does not have rights to see any of the events logged. I also connected my Node JS app server logs. They seem be to logging but my "search" is empty. shows waiting for data. And shows 0 events in http event conenctor dahsboard

[starcher]

Sounds like you answered it. Indexing paused due to space.

[rajpraba]

starcher:
Thank you so much for leading me into the solution. The aws micro medium instance I used which quickly filled with logs. I used small instance to play with before committing to c3.xlarge instance which works flawlessly.

Thanks

[rajpraba]

Starcher:

Thanks. Let me answer what I had done. If I need to do something else for the steps please let me know.

1. I see the Index named 'sample' in Setting->Indexes, I see a row for sample ($SPLUNK_DB/sample/db, 0 events, enabled)
2. Settings->Http Event Collector->test-token-1 edit->Default Index ->Sample. (hope this is what is intended configuration)
3. I am logged in as admin. I add all the roles available. Settings->Access Control -> Admin

Thanks

[starcher]

Are you:
1. sure index = sample exists?
2. the HEC token was given permission to the index?
3. that you have permission to search index = sample if 1 and 2 above are true.