Hi
Little background
I have a EKS cluster and On Premise splunk cluster. We have 5-10 application micro-service running on EKS.
I want ingest logs into the Splunk from EKS K8.
Splunk Connector has been configured and able to ingest logs.
At present all the K8 pods logs converged into splunk-objects pods and object pod logs are mapped to one index (kube_obj-index)in the splunk.
index="kube_obj-index" namespace="myapplication1" "GET" | collect index=myapplication1-logs
index="kube_obj-index" namespace="myapplication2" "GET" | collect index=myapplication2-logs
index="kube_obj-index" namespace="myapplication3" "GET" | collect index=myapplication3-logs
I need to help to modify yaml files so that i want each Kubernetes Namespace logs goes to separate index in the Splunk.
have you solved this, or do you still need help? Our dev branch has annotation support in preview now as well.
@richgalloway
Sorry I don't see your comment. Would you please post it again.Thanks
does the Namespace to Index Routing section on the readme help?
https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/README.md
@maciep Thanks for your comment. It worked
For example: Consider the following kubernetes namespace to splunk index topology.
(Namespace) -> (Splunk Index)
kube-system -> kube-system
kube-public -> kube-public
default -> indexRoutingDefaultIndex For this topology to work appropriately we have to create the splunk indexes "kube-system", "kube-public" and the value of indexRoutingDefaultIndex.
I have created below indexes
kube-system
kube-public
namespace1
namespace2
HEC_TOKEN is mapped to an index in the splunk. I used that index as a default index.
I am getting below error.
19-09-17 17:20:47 +0000 [info]: #0 fluentd worker is now running worker=0
19-09-17 17:20:53 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:00 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:07 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:14 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:21 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:28 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
19-09-17 17:21:34 +0000 [error]: #0 Failed POST to https://test.solutions.company.com/services/collector, response: {"text":"Incorrect index","code":7,"invalid-event-number":1}
Any thought and do you think i am doing anything wrong.
Thanks.
Are you specifying "allowed" indexes for that token on the HEC. I think that's when I've seen the "Incorrect index" problem before - the index you're sending to isn't in the allowed list.
I think typically the approach is to not define any allowed indexes and rely on the default index to catch unexpected data.
@maciep
Thanks I will try and let you know.
Hello @rmurali4u I'm running into the same issue. I'm running 1.4.7 version connector. Any inputs would be helpful.