Splunk App for Web Analytics: Why am I getting error "[subsearch]: Failed to fetch REST endpoint uri=..."?

[cbader]

All of the configs appear to be correct. When looking at the search for the Data Model Audit, it is returning the following:

[subsearch]: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SplunkAppForWebAnalytics/admin/summarization/tstats:DM_SplunkAppForWebAnalytics_Web?count=0 from server=https://127.0.0.1:8089 

I tried to access this URI on the server where Splunk is installed and I get the following result:

<response>
   <messages>
      <msg type="ERROR">
         Application does not exist:SplunkAppforWebAnalytics
     </msg>
  </messages>
</response>

Not sure what I am missing? Any help would be appreciated. Thanks

[jbjerke_splunk]

Great progress!

Can you check that the other users can see data when searching for tag=web? They also need access to the datamodel web. You can browse this by clicking the Pivot link in the menu.

j

[bfsplunkdl]

my regular user (who is actually a member of splunk admins also) does get results when running "tag=web"

The Web datamodel is set to Everyone Read AND Write.

my regular user can see RealTime stats on the App dashboard, but if I select, for instance, Traffic->Traffic Center, none of the dashboards return any results.

Opening 1 Dashboard in Search, I try to cut the query down to any part that actually returns results.

This full query (from the Traffic dashboard, top left, Request Count By Type) returns no results:

| tstats summariesonly=t prestats=t count AS Requests FROM datamodel=Web WHERE Web.site="*" "Web.eventtype"=pageview OR "Web.eventtype"=non-pageview GROUPBY "Web.eventtype" _time span=1h | search Web.eventtype=pageview OR Web.eventtype=non-pageview | timechart span=1h count by Web.eventtype

This edited query DOES return results:

| tstats count AS Requests FROM datamodel=Web WHERE Web.site="*" "Web.eventtype"=pageview OR "Web.eventtype"=non-pageview GROUPBY "Web.eventtype" _time span=1h | search Web.eventtype=pageview OR Web.eventtype=non-pageview | timechart span=1h count by Web.eventtype

Does this provide info on where my permissions may be misaligned? summaries? prestats? I'm not familar.

FYI - Today I even rebuilt the Data Model using my regular user login, per the Documentation page (turn off accelleration, re-run Generate Sessions, turn on accelleration, run "Rebuild" - it did not change the behavior.

Thanks,

[jbjerke_splunk]

Hi

Regarding the http_request, this field is populated from a field alias as the corresponding field is be called cs_uri_stem. Can you go into Settings->Fields->Field Alias and make sure they all match up with your ISS logs?

By default the app ships with these:

FIELDALIAS-clientip = c_ip AS clientip
FIELDALIAS-cookie = cs_Cookie AS cookie
FIELDALIAS-http_user_agent = cs_User_Agent AS http_user_agent
FIELDALIAS-bytes = cs_bytes AS bytes
FIELDALIAS-http_method = cs_method AS http_method
FIELDALIAS-uri_query = cs_uri_query AS uri_query
FIELDALIAS-cs_uri_stem = cs_uri_stem AS uri
FIELDALIAS-uri = cs_uri_stem AS http_request
FIELDALIAS-user = cs_username AS user
FIELDALIAS-version = cs_version AS version
FIELDALIAS-status = sc_status AS status

The above can also be seen in the props.conf file in the default directory.

j

[bfsplunkdl]

Update: Today I scanned all my splunk servers and changed all old references to sourcetype=iisw3c to sourcetype=iis. I commented out any old usage of props or transforms associated with these sourcetypes.

Basically, since 6.x now natively supports w3c better than pre-6.x, I removed all old conf entries and made it "default" to match 6.x version.

I am now seeing initial data - uninstalling and reinstalling your app let me save a website quickly, get green checkmarks in a few places on the setup/documentation page(s). I am already seeing a few hits on "real time" dashboard, and the "generate pages" tool is currently running and finally showing entries - it has never, until now, even gotten the first entry.

So....very good progress today. I will update this after some deeper research but I suspect it will fall in line correctly.

For anyone else having this issue, if you have upgraded from a pre 6.x splunk installation, or if you have used older iisw3c props or transforms, you might want to convert these to the newer standard iis sourcetypes.

Thanks!

[bfsplunkdl]

Great progress overnight as the data models built - I'm able to see nearly everything I've tried and I believe it's good except for some more setup and configuration.

However, now I am having a permissions issue.

I installed/configured the app as user "admin".

Now my other users cannot see data on the dashboards, pages, or features. My normal user is a member of "admin" (and power_users, etc), so that user should be able to see everything "admin" can.

I have edited app permissions to everyone read, and everyone write, still no results.

I can see data and results on the realtime page, but none of the other dashboards return any data.

Thanks again for the help so far, good looking app now that I can see some data.

[bfsplunkdl]

That's very helpful, thank you. I am beginning to understand and see how the underlying changes should be working, and I feel I'm almost there, or getting closer.

The only place "http_request" appears in my entire \etc directory is in the $SPLUNK_HOME\etc\apps\SplunkAppForWebAnalytics\default\props.conf file

Here are the contents of that section for iis in cs_Referer
EVAL-http_referer = if(isnull(cs_Referer),"-",cs_Referer)
FIELDALIAS-clientip = c_ip AS clientip
FIELDALIAS-cookie = cs_Cookie AS cookie
FIELDALIAS-http_user_agent = cs_User_Agent AS http_user_agent
FIELDALIAS-bytes = cs_bytes AS bytes
FIELDALIAS-http_method = cs_method AS http_method
FIELDALIAS-uri_query = cs_uri_query AS uri_query
FIELDALIAS-cs_uri_stem = cs_uri_stem AS uri
FIELDALIAS-uri = cs_uri_stem AS http_request
FIELDALIAS-user = cs_username AS user
FIELDALIAS-version = cs_version AS version
FIELDALIAS-status = sc_status AS status
EXTRACT-file = .*/ in cs_uri_stem

[jbjerke_splunk]

Almost there 🙂 !

Can you check that you have the fields pre-pended with "cs_" in your data? You mentioned earlier you had a field called "sc_status" and not "cs_status" as the app wants.

If you modify the last entry to this it should hopefully extract the file field correctly:

EXTRACT-file = .*[/](?<file>.+\.\w+) in cs_uri_stem

I would recommend you make a copy of this props.conf file and place it in the local folder instead of the default folder. Then rename and adjust as you please to make it work.

j

[jbjerke_splunk]

Oh wait, the app is expecting sc_status. My bad. Just double check you have the other fields in your source data. If not you might need to do more field extractions.

[bfsplunkdl]

I'm starting to realize I have a possible conflict with sourcetypes from UF, to HWF, to Indexer, to Search head. My Splunk infrastructure has been upgraded more than once and the new (6.0+) sourcetypes for IIS vs IISW3C are different. I suspect I have old methods conflicting with new.

I say this because if I go just to my index (not within the app) and search "sourcetype=iis" I get different results than if I search "sourcetype=iisw3c" and the results are not necessarily consistent.

For what it's worth, sourcetype=iisw3c DOES return results and DOES have a LOT of fields, most of the ones your FIELDALIASEs are looking for.

I'll report back when I've confirmed consistent sourcetype data from end to end on my side.

[jbjerke_splunk]

The app is designed to work with sourcetype=iis only. You could potentially use sourcetype renaming for iisw3c->iis in the context of the app or alternatively duplicate all configs linked to iis in props.conf and modify eventtypes.conf to include iisw3c.

j

[cbader]

The "file" field does not appear to be defined. I can create this field using the extract, but it would be helpful to know what the Web Analytics is expecting to see for this?

[jbjerke_splunk]

The filename+suffix:

index.php
image.jpg
...

If there is no file and just a path in the URL, this field should be empty.

j

[jbjerke_splunk]

Hi

The problem is with this eventtype:

"non-pageview 100%"

That means that all your requests are interpreted as non pageviews (i.e. js, jpg, css requests) and not actually visits to a page. That eventtype is defined by looking at a couple of fields - for instance the "file" field. Can you check if you have that field extracted? If you haven't, can you create an extracting using the Extract Fields guide?

j

[cbader]

Not making any progress with this. tag=web returns results with the following eventtype values
external-referer 100%
non-pageview 100%
visitor-type-direct 100%
web-traffic 100%
web-traffic external 100%

This install was originally setup with Splunk installed on CentOS 6.X with Splunk Forwarders on a couple of Mac OS X (10.11.1). I went back and installed it directly on the Mac OS X system, reinstalled Web Analytics and am getting the same results. Is this possibly an issue with the way that Apple has their Apache Logs formatted?

[jbjerke_splunk]

Hi

There should be a new version of the app that fixes that error - v1.6.

The issue is that it is trying to run a REST search on the indexers and not only on the search head. Its fixed by adding the parameter

splunk_server=local

to the "| rest" search command

j

[bfsplunkdl]

Deleted, reinstalled as the main splunk admin onto my search head.

• Setup -> Websites, page already had log names/sourcetypes loaded(loading) in the right hand pane
• Added 1 Website - waited - huge font "Website saved successfully"
• Searched "tag=web" - got hits
• Still no "file" event type
• Went to eventtypes, copied this and ran it from search - eventtype=web-traffic status=200 NOT (eventtype=web-uri-nonpage OR eventtype=ua-bot OR eventtype=exclude-pageview OR eventtype=clientip-internal) (http_method=GET OR NOT http_method=*)
• That search returned 0 results
• changed to (status=200 OR sc_status=200) and it DID return results
• Same behavior as I mention in posts above

Why is the file extract not working?
Why is the fieldalias for status<=>sc_status not working?

[jbjerke_splunk]

Hi

As you can customize the log format for IIS, the app might not extract all the fields correctly. Can you try and manually extract the field "file" from your IIS logs? This is not an eventtype it is a field called "file" and it should have the values of the file that was requested from the server.

Do a search for tag=web, find the first event and expand it. Click on "Event Actions" and open up the "Extract Fields" wizard. It should take you through the steps on how to extract that field.

You can read the docs for how it works here:
http://docs.splunk.com/Documentation/Splunk/6.3.3/Knowledge/ExtractfieldsinteractivelywithIFX

Alternatively, can you post one event here so I can have a look at it?

j

[bfsplunkdl]

If I open the "generate pages" in search, I see this:
eventtype=pageview [| inputlookup WA_settings | fields value | dedup value | rename value AS site] | top limit=100 http_request by site | fields site http_request | table site http_request | outputlookup WA_pages createinapp=t

If I cut that back to see which piece works, it returns no results because of "http_request"

Searching the app objects, I see "http_request" as an access_comb* extraction - that seems to be Apache, not IIS.

I don't see a comparable IIS extraciton, object, or setting for the equivalent "http_request".

I am running IISW3C extended format, with every value selected and every value being ingested in splunk.

[bfsplunkdl]

Thanks. It's not optimal yet but I extracted the field "file" as the entire uri_stem.

Instead of "button.jpg" as the "file" i have \path\to\button.jpg - if I can get the basics working I can fix that field extraction.

However, the eventtype=pageview is still not working out of the box due to "status=200" is not showing up. I have manually edited to include "OR sc_status=200".

I now finally see some (not entirely valid) data on "Real Time" dashboard, and I'm currently running "Generate Pages" and "Sessions" for last 30 days to see if they can complete and get me the KV extracts for the data.

Getting closer I hope. Once I get the little things sorted out, I will likely remove/reinstall and do it clean from scratch (if that's needed).

I'll let you know, and thanks for the reply.

[bfsplunkdl]

grr. forum formatting fail.

i have path-to-button-slash-button.jpg ( you probably got what I meant already) - full relative path, not just the actual filename.