Dashboards & Visualizations

Splunk for a boat race?

overboard
Explorer

Hello!

I am a bit of a lurker here. I am considering using Splunk as both the frontend and backend for a support crew telemetry website for a yacht race.

The boats transmit frequent but irregular telemetry data.

Given the competitive nature of the endeavour security is a paramount concern 😉

As the support crews are not necessarily IT literate, I propose that the site is basically one big dashboard, with very limited access to the underlying Splunk functionality.

So, here's the question(s):

  1. For security reasons, I propose to allocate a separate index to each boat. Is this the best approach?
  2. For simplicity, let's say there are two types of telemetry data, depending on the boats' equipment. I'm going to develop two dashboards, each displaying optimum information based on the raw data.

Do I need to replicate the dashboard/ app across each of the relevant indices? Or can I use the same app across each of the relevant indices?

  1. If I have to replicate the dashboard across each, is there some easy way to abstract the source data from the dashboard? What I mean, is that if I have index="boat1", index="boat2", etc, do I have to change all my queries, or can I have app1 written to expect a source of "boat_telemetry1 and abstract/ or 'rename' in one place boat1= boat_telemetry1, boat10=boat_telemetry2, etc? Does that make sense?

  2. Given the irregular nature of the comms, is there anything I should bear in mind?

Any advice appreciated.

OB.

PS. I'm considering ammaps. Has anyone used it for non-IP location data?

OB.

Tags (2)

MuS
SplunkTrust
SplunkTrust

Hi overboard,

just got curious, because Americas Cup is currently running....
Have you ever used Splunk for this?

Cheers, MuS

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

If security is your most important concern, you should ABSOLUTELY use a separate index per boat and have individual users only have access to their boat. The administrative user can have access to all indexes and hence can report across all boats at the same time, for example:

index=boat* | stats ... by index

You do not need a separate dashboard per boat. Each user's dashboard will only reference the data in the index they have access to. It is very, very important here to NOT schedule the search for the dashboard. Shared scheduled searches (for dashboards) will run as the owner, but the results will be available to all users with access. In this case, if the administrative user shares the scheduled search, all users will see data about all boats.

For bursty and sparse data, the biggest challenge is that a simple timechart. If you know when the bursts of data start, you can use a time-based lookup table to assign a run id (and perhaps start time to data in each burst. This can be done after the fact. Then you can make it easier to retrieve and summarize each batch of data.

platform_pie
New Member

Is there any way to ensure that the data from each boat has not been manipulated in any way before it got to your indexer?

0 Karma

overboard
Explorer

Thanks Stephen, I'll let you know how I get on. OB.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

Your dashboard should just refer to source=boat_telemetry_A. Users visiting the dashboard will see the indexes they have access to, so users with access to (only) index=boat1 will see (only) that source for index=boat1.

0 Karma

overboard
Explorer

How do I link index="boat1" to the dashboard expecting a source of "boat_telemetry_A"? I downloaded the Splunk for Cisco app and noticed that the dashboard complained that I didn't have any ironport sources. Given that everyone's not going to rename their sources to ironport, I assume I can make this link some how. Sorry about asking again. Just trying to get me head around this.

Regards,

OB.

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

You can search index=_audit to see which users ran searches. The user activity dashboard in the Search app provides an example of this.

0 Karma

overboard
Explorer

With regard to 'irregular', I was referring to the timing of received data. I'll always get it in the correct format, but some days I'll get multiple updates, some days none at all. That's what I meant by frequent but irregular. Sorry for any confusion.
On a separate but related subject, what would the search look like to see what teams have logged onto the dashboards, when, etc?
Thanks,
OB.

0 Karma

overboard
Explorer

Thanks for that Stephen (et al). So let's say I have 10 indices, index="boat1", index="boat2", etc and I have two dashboards, one expecting source type "boat_telemetry_A", the other "boat_telemetry_B". How do I link the boat's raw telemetry data to the appropriate dashboard? I hope I am explaining that clearly. I don't expect Splunk to do it automatically, but for each boat, I would hope to edit the minimum number of variables to get the dashboards to work. Otherwise it won't scale, (I hope to have more than 10 boats).

0 Karma

zscgeek
Path Finder

One critical note about security: be sure to disable typeahead for your users. The suggestions won't take in account the search filters that are in place and depending on the data layout and the EQ (evil quotient) of your users this can cause problems as they can extract info from this source.

Stephen_Sorkin
Splunk Employee
Splunk Employee

This isn't really accurate if you make a separate index per boat. Typeahead only will access indexes that the user has access to. This is one of the key benefits of using indexes to segregate data.

bbingham
Builder

So, here's the question(s):

For security reasons, I propose to allocate a separate index to each boat. Is this the best approach?

Not needed, you can have each boat in the same index, use a sourcetype flag and then limit each users search to their respected boats.

Do I need to replicate the dashboard/ app across each of the relevant indices? Or can I use the same app across each of the relevant indices?

Nope, you can use the same app and once again user their username to drive the saved search for the dashboard. If you store everything in a different index, you can also set a default index for each user, and then their search will be constrained to that index.

If I have to replicate the dashboard across each, is there some easy way to abstract the source data from the dashboard? What I mean, is that if I have index="boat1", index="boat2", etc, do I have to change all my queries, or can I have app1 written to expect a source of "boat_telemetry1 and abstract/ or 'rename' in one place boat1= boat_telemetry1, boat10=boat_telemetry2, etc? Does that make sense?

This is the main reason I feel you should do this all under 1 index. You'll be able to pull a correlation accross all boat times etc easily if they are contained in one index, and then set host="boata" host="boatb" etc. This will allow you to have very fine control over boats who's comms are similar, and those who are very unique. You'll be able to set each sourcetype with the field extraction for the unique data type, and the host tag can filter out down to a unique boat.

Given the irregular nature of the comms, is there anything I should bear in mind?

Since you don't fully know what data is coming it, it's hard to build any extraction type. By setting up a unique file for each boat and then controlling the inputs.conf for each file, you'll be able to account for any unique comms as well as set a global override. Given with what you're trying to do, I'd use the "host" flag for each boat, and then try to standardize the sourcetype based on each major comm style you receive. You'll be able to lock down each user to a specific host under the user permissions and should account for any major concerns you've mentioned. The only other major thing to make sure of, each "event" will need a date/time stamp. After that, splunk can manipulate the data any way you'd like!

Hope I've been of some help!

bbingham
Builder

Isn't searching across multiple indexes slower? This is for my own information, but I thought once a search crosses over to a new index it has to search the entire specified timespan for each index vs. if all the information was in 1 index, the main search only runs one time, am I mistaken?

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

I think that you're mistaken about multiple indexes. You can search (and correlate) across many indexes if you have access to several indexes using either index=boat* or index=boat1 OR index=boat2 OR ....

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...