Track users journeys through an app and map out the pages they view

[Guitaraholic]

So I have a bespoke Java app running in tomcat logging out different events which correlate to different sections on the app. Each different page is logged into a different log file so I have multiple sources all under one sourcetype.
There is a key value pair field called 'user' on every line which represents the logged in users email address.

I'm able to isolate an event in each source which shows the user has visited that page in the app.

I want to be able to create a report and/or visualisation that can show the order in which the users moved around the app at a high level as a Proof of Concept. I need to be able to visualize multiple users and variations in the journey as its non-linear.

This is a rough version of my query atm.

index=prod user=foo@user.com NOT message="cache"

| dedup _raw,host,_time
| transaction source maxspan=1m
| rex field=source "/var/log/tomcat/(?.*).txt"

| table _time,user,page
| chart count(user) over _time by page
| chart count(userjounrney) over _time by page

Any ideas on how we could visualize this in a way it shows the progression of the pages that a specific user hit at what time?

UPDATE:

I've tried adding this to the end of the search and it visualizes the pages BUT not showing the order or time at which users visited them

| eval Page = if(page="acs","ACS",if(page="home","Home",if(page="my-bills","My-Bills",if(page="ebill","eBill",if(page="direct-debit","Direct-Debit",if(page="my-apps","My-Apps",if(page="my-profile","My-Profile",if(page="createprofile","Create-Profile",if(page="my-offers","My-Offers",if(page="faults","Service-Status",if(page="trackorder","Track Order",0))))))))))) | chart count over Page by user usenull=f useother=f

[lguinn2]

I have a couple of suggestions. First, I would create a lookup table that maps the name of the source to a page name. I think that this will ultimately be more flexible. The CSV for the lookup table might look like this:

source,page,title
/var/log/tomcat/acs.txt.=,acs,"ACS"
/var/log/tomcat/home.txt,home,"Home"
/var/log/tomcat/ebill.txt,ebill,"E-bill"

In my example below, I will assume that the lookup is defined with the name page_map

You could follow the "path" of the users, in a text format, with

index=prod user=* NOT message="cache" 
| lookup page_map source
| stats list(title) by user

Although this would not show times, it would show the page titles in the order they were visited.

If you wanted to see this by minute, you could do something like this (be sure to set the search timerange to a very short period of time, like 10-15 minutes).

index=prod user=* NOT message="cache" 
| eval minute=relative_time(_time,"@m")
| lookup page_map source
| stats list(title) by user, minute
| fieldformat minute=strftime(minute, "%x %H:%M")

Neither of these is a chart or graph - I don't know of a good way to map a path using the Splunk commands. If you want to use another program (maybe in Java 🙂 ?) to do the visualization, you could do the following search, which compiles the data into a table and exports it as a CSV file:

index=prod user=* NOT message="cache" 
| lookup page_map source
| sort user _time
| table _time user title source
| outputcsv useractions

[lguinn2]

The trouble is that Splunk requires a numeric value on the y-axis. In fact, the y-axis must be a number that is the result of one of the stats, timechart or chart functions.

While there are a few ways to "fake things" with Splunk chart, I know of no way to get around this problem. At least one axis must always be numeric.

[Guitaraholic]

OK thats definitely putting me in the right direction.....

Wondering if there is a way to have a timechart with the line being the 'user' and the y-axis showing which pages the user went too?