Splunk 6 RTO in cluster environment

[hopnscotch]

Where do you install the real-time output app within a clustered environment, search head or indexers?

Thanks!

[dfronck]

We have a cluster and I installed this on a dedicated search head. It forwards the events just fine but not in a format ArcSight can really use.

I've only spent about an hour on this so maybe I just don't know what I'm doing but here's my primary issue.

The CEF Output Assistant doesn't seem to work. I do a search and can see the generic CEF mapping.

Apr 15 15:45:18 CEF:0|Splunk|syslog|1.0|100000|generic event|5|rt=1397591119 src=TheRightIP dvchost=TheRightHost shost=TheRightHostName

But when I drag in the CEF Field "cef_dproduct" it says "provide a static mapping" and if I enter a name "VENDOR" or one of the Splunk Fields that it extracts, it still says Splunk in the Vendor field. When ArcSight gets the event, it is that same generic event that I have above.

Also, when I save and go back into the rule, it has deleted all those mappings.

[dfronck]

Alex, I used info from this http://answers.splunk.com/answers/95299/is-there-more-detailed-information-about-how-the-cef-field-m... to learn how to manually map fields and it kinda works.

index=bit9 "Console user login" | eval cef_field_map="2:cef_severity,Bit9:cef_dvendor,Parity:cef_dproduct,Process:cs2Label,Type:cs1Label,date:start,host:dvchost,misc:msg,process:cs2,server_version:cef_dversion,src_host:shost,src_ip:src,src_user:suser,subtype:cef_name,date:rt,type:cs1"

ArcSight wants msg not message.
None of the cef_ replacement stuff works.

[dfronck]

Splunk 6.0.2. RTO 1.0.4b.

I've tried dragging Splunk fields and just doing static text. For example, cef_dvendor=Bit9,cef_name=subtype

These are in the current mappings but the output displayed at the bottom never changes. It still says

"Apr 16 07:08:59 CEF:0|Splunk|syslog|1.0|100000|generic event|5|"

And when ArcSight gets the event, it still has Splunk as the Vendor and "generic event" as the name.

And, once I click Apply and OK to save it, it seems to go away. When I look in realtimeoutput.conf, only the

search = index=bit9 "Console user login" is there.

[araitz]

Try dragging a field from the left hand box of Splunk fields to the middle box of CEF fields.

What version of Splunk are you using? What version of the app?

[araitz]

If you have a modest use case - say, less than 50 GB per day - the search head should be fine. However, any more than that, and the search head will eventually become resource constrained - depending on what else is going on on the search head, what type of hardware it is running, etc.

After that point, you are better off installing it on each of your indexers. It is a lot harder to manage configurations, and you'll have each indexer creating CEF data and potentially connecting to your universal connector, but it will scale better.

[araitz]

I've never tested with clustering, but I don't think that it would affect data being replicated to the instance since the real-time output depends on real-time search, which only has accesss to data as it is being indexed (not replicated).

[hopnscotch]

Thanks for you reply araitz.

I mainly questioned it because I read in the "Integrating Splunk with Arcsight" pdf that "First, the real-time search API is used to inspect events just before they are indexed." .. so that would lead me to believe it's installed on indexers. However, would there end up being duplicate data if installed on the indexers in a cluster where there is replication happening? This isn't being managed by the master, so what prevents it from searching replicated data?

Thanks