Different fields in apps for same sourcetype?

[echojacques]

Hello,

One of my sourcetypes is bcoat_proxysg (BlueCoat). Within the Search app, I have all of the correct/expected fields. Within the Enterprise Security app, there is a different set of fields and none of the relevant BlueCoat fields (like http_referrer and http_user_agent) are available. It's almost as if the sourcetype is broken within ES (but it's fine in Search). None of my other sourcetypes have this problem so I'm really confused about why the fields are different between the two apps.

Update: I tried the suggested answer below last night and made the changes to TA-bluecoat but this didn't make any difference. So currently in my Search app, the sourcetype=bcoat_proxysg fields are correct but in my ES app the fields are incorrect. I'm looking for guidance on where I should begin to troubleshoot this... I'm really confused why this is happening in the first place... I thought that a sourcetype (once defined) would have the same fields available in all apps.

Do I need to do field extractions twice... once for Search and once for ES???

Thanks!

[echojacques · Accepted solution]

After a lot of testing, I had to revert from ES version 3.0 back to ES version 2.4 to get my sourcetypes and their fields to work properly again within ES. What I learned is that I will need to modify my field extractions/transformations in order to make them compatible with ES 3 - otherwise, the fields/dashboards/and views within ES 3 will not have good data.

[echojacques]

Hi, I decided to revert because almost all of the built-in dashboards/views within ES 3 either had missing data or incorrect data. Then I also found that my Checkpoint/opsec-lea sourcetype had incorrect fields (just like the bluecoat issue) as well.

We are setup to read all of our data from a syslog instead of directly from the devices. I think this may be the issue since we have many customized extractions/transformations that ES 3 may not know how to interpret. That said, if it works within ES 2.4 then ES 3.0 should know how to handle it during an upgrade...

Thanks again for your help.

[jcoates_splunk]

there were some changes between the two to switch from proxy to the web datamodel, but this is the first i've heard of an upgrade not working right. A support ticket might be in order to figure out what's not working for you. Reverting the whole ES install shouldn't be necessary either, though I appreciate that restoring the backup is easier than troubleshooting.

[jcoates_splunk]

seems like we're online at the same times today 🙂

TA-bluecoat expects a certain field order (and sourcetype = bluecoat). I'd edit its props.conf and eventtypes.conf to look for your sourcetype instead. Then I'd use Splunk Add-on for Weblogs to generate a more efficient field extraction that matches your data... Since you're on ES 3, make sure you reference the Web data model instead of one of the canned field sets.

[jcoates_splunk]

so, no idea what you actually did, just guessing here 🙂 But my guess is that you got bitten by app isolation. The easiest solution is probably to take the props and transforms generated by weblog add-on and copy them to the TA-bluecoat/local directory. Alternatively, you might need to restart the search head or even rerun ES's setup so it will know to inherit knowledge from the new TA.

To your question on doing extractions twice, we actually consider the ability to do different extractions for different apps from the same raw data a feature 🙂

[echojacques]

Unfortunately, making the modifications to the .conf files didn't work...

[echojacques]

Thanks, that makes sense. I'll edit the files and test it again.

Yes, I guess we're on a similar schedule today!