All Apps and Add-ons

Group lookup being overwritten

pauliohughes
Explorer

Hi,

We're trialing Cisco CDR, and one issue we keep running into is the groups.csv is being overwritten and set back to the default value.
I've not been able to nail down whats causing this, it could happen after a few minutes of re-uploading through the sideview utils app, or it could be the next day.

Has anyone seen this before, or have any suggestions on how to debug this further?

Thanks

1 Solution

sideview
SplunkTrust
SplunkTrust

UPDATE:

My apologies. There is a bug. Here are several different ways you can fix it.

1) Create your groups lookup with all four fields defined number, name, group and subgroup. The docs used to say you can create just the first three and skip the fourth but a recent bug causes the system to discard your rows if the fourth field is missing.

2) Go to Settings > Advanced Search > Macros. Edit the "ftr_create_groups_lookup" macro and replace it's long definition with stats count

3) Wait about a week and download a new build of the Cisco CDR app. We'll be shipping the current maintenance release asap, including this fix, and it'll be up on the site by the end of this week.


Initial Answer:
I do know what is doing this, and I can give you a workaround to prevent it from happening.
I'm still working on the "why its suddenly doing it" unfortunately.

Short version

Where we just fix it. From the Cisco CDR app, Click "Settings > Advanced Search > Macros."

Then scroll down or page down to the "ftr_create_groups_lookup" lookup.
Replace it's long fussy looking definition with something that does nothing, like
noop
or
stats count

This will disable the thing that is (somehow) being bad when it shouldn't.

Long version.
There's a longstanding Catch-22 in app development, around using lookups where the end-user has to fill out the values.

  • If you ship the physical lookup file, than as soon as the customer updates the app, they will clobber the local file they created.
  • If you do not ship the actual lookup file, but you do ship the config around it, then the first time your new users load the app's UI, they'll see a veritable flood of cryptic red error messages about lookup files being missing. (NB They'll mentally give up on your app before they're halfway through reading the first one. )

In practice app developers seem to do one of two hacks to get around this.
1) hack a scripted input that is not actually "inputting" anything but that instead just creates the file(s) the first time it runs and then disables/deletes itself. this is a) horrible. b) doing filesystem operations from python which is bad (and which will cause you appcert troubles these days)

(Also you can do FTR-UI hackery where a setup wizard process in python does the same thing. )

2) Search language hackery where you do
| inputlookup groups| <stuff to create the rows if there's nothing there, but not damage existing rows if there is> | outputlookup groups
This is a) horrible, but b) ....... well until the last month or so there was no b. This has actually worked pretty well for years and we have about 70 customers deployed with exactly this happening every time they load the homepage. It still seems to work great in the lab, but you're the second or third customer reporting this kind of problem in the last month, and I don't think it's a coincidence that 6.5's release was a bit more than that long ago.

Leading followup question - are you on 6.5?

Other bonus question - any chance there's something unusual about the file ownership or perms on the groups.csv file after you're customizing it but before it gets overwritten? (I'm eager to reproduce locally so I can fix).

Also... When it clobbers your file, is it a 0 byte file specifically, or if you open the file is there a header row for "number,name,group,subgroup" and a single row of placeholder values? That would help narrow down what's doing it as well.

View solution in original post

sideview
SplunkTrust
SplunkTrust

UPDATE:

My apologies. There is a bug. Here are several different ways you can fix it.

1) Create your groups lookup with all four fields defined number, name, group and subgroup. The docs used to say you can create just the first three and skip the fourth but a recent bug causes the system to discard your rows if the fourth field is missing.

2) Go to Settings > Advanced Search > Macros. Edit the "ftr_create_groups_lookup" macro and replace it's long definition with stats count

3) Wait about a week and download a new build of the Cisco CDR app. We'll be shipping the current maintenance release asap, including this fix, and it'll be up on the site by the end of this week.


Initial Answer:
I do know what is doing this, and I can give you a workaround to prevent it from happening.
I'm still working on the "why its suddenly doing it" unfortunately.

Short version

Where we just fix it. From the Cisco CDR app, Click "Settings > Advanced Search > Macros."

Then scroll down or page down to the "ftr_create_groups_lookup" lookup.
Replace it's long fussy looking definition with something that does nothing, like
noop
or
stats count

This will disable the thing that is (somehow) being bad when it shouldn't.

Long version.
There's a longstanding Catch-22 in app development, around using lookups where the end-user has to fill out the values.

  • If you ship the physical lookup file, than as soon as the customer updates the app, they will clobber the local file they created.
  • If you do not ship the actual lookup file, but you do ship the config around it, then the first time your new users load the app's UI, they'll see a veritable flood of cryptic red error messages about lookup files being missing. (NB They'll mentally give up on your app before they're halfway through reading the first one. )

In practice app developers seem to do one of two hacks to get around this.
1) hack a scripted input that is not actually "inputting" anything but that instead just creates the file(s) the first time it runs and then disables/deletes itself. this is a) horrible. b) doing filesystem operations from python which is bad (and which will cause you appcert troubles these days)

(Also you can do FTR-UI hackery where a setup wizard process in python does the same thing. )

2) Search language hackery where you do
| inputlookup groups| <stuff to create the rows if there's nothing there, but not damage existing rows if there is> | outputlookup groups
This is a) horrible, but b) ....... well until the last month or so there was no b. This has actually worked pretty well for years and we have about 70 customers deployed with exactly this happening every time they load the homepage. It still seems to work great in the lab, but you're the second or third customer reporting this kind of problem in the last month, and I don't think it's a coincidence that 6.5's release was a bit more than that long ago.

Leading followup question - are you on 6.5?

Other bonus question - any chance there's something unusual about the file ownership or perms on the groups.csv file after you're customizing it but before it gets overwritten? (I'm eager to reproduce locally so I can fix).

Also... When it clobbers your file, is it a 0 byte file specifically, or if you open the file is there a header row for "number,name,group,subgroup" and a single row of placeholder values? That would help narrow down what's doing it as well.

pauliohughes
Explorer

Hi,

Thanks for the detailed reply. We're on 6.5, but we have previously been running this on 6.4 too and we kept running into the same issue.

When the file gets overwritten we are getting a 104 byte file with the headers and sample values in.

number,name,group,subgroup
"_PLACEHOLDER","PLACEHOLDER_NAME",PLACEHOLDER_GROUP","PLACEHOLDER_SUBGROUP"

The only odd thing with our file is that we aren't using subgroups, that column is blank. We just have number, name, group.

Could that be the issue?

Thanks and Regards,

0 Karma

sideview
SplunkTrust
SplunkTrust

Oh dear. Yes that is indeed what is causing it. The docs say you can either add the subgroup column and leave it blank, or you can omit it entirely and it'll be fine. However that last part is false. The code on the homepage that guards against the lookup being missing entirely, has a bug. If it sees an existing lookup without all four fields- number, name, group and subgroup, then it throws away the existing rows.

I'll update the top of my answer with the options you have to fix or workaround.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...