Splunk Search

What's the best way to version control a lookuptable?

bhawkins1
Communicator

I have a Splunk app I'm building that will eventually be bundled ( .tgz ). The app has an optional csv file that the user can insert or ignore. If the csv file is inserted, I need a lookup definition for it, and optional data model fields that use that lookup.

What's the best way to implement this when developing the app? Can I version-control the .csv_[0-9]+.index/ directory or will this cause problems if the .csv is removed and later added?

Masa
Splunk Employee
Splunk Employee

I'm still not understanding why you need version control for a lookup file and .index file. So, my answer might not be proper to your question.

If your app requires to keep track of all changes users updated lookup csv files, you need your own way to do it. Current Splunk does not have such feature.

Splunk automatically create index files (tsidx files) when csv file is large. tsidx file will improve search performance. If a lookup is updated, Splunk needs to create a new tsidx files. Splunk will not keep those tsidx files. tsidx files. They are binary files. So, when you say version control, I assume you keep all of them because diff would be really difficult. So, size of the app will eventually become very large depending on how much version records will be kept.

What is purpose of version control? Do you have a real use cases and examples? Do you need to revert back to different versions? Do you need something to check diff or merge lookup files by your own app's feature, not Splunk feature?

Because I do not understand your use case for requiring "version control" of lookup files, I believe my answer is not proper.

0 Karma

bhawkins1
Communicator

@aaraneta_splunk if you edit my post can you please provide a summary of changes? Thanks

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@bhawkins1 - As @MuS mentioned, you can click the little gear icon to the right of question title, you can see the revision history. You'll see that I provided formatting/capitalization changes and updated tags to your original post. In reference to your comment below @thomasb42's answer: No information was removed from your original post.

bhawkins1
Communicator

Thanks, and sorry for the confusion. I think when I misread my own original post when I was writing that comment, and missed that the stuff about .csv_[0-9]+.index/ was still there.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

No problem, glad it's sorted 🙂

MuS
SplunkTrust
SplunkTrust

@bhawkins1 you can click on the little gear symbol to the right of the question title and select See revisions to what has been modified.

cheers, MuS

thomasb42
Engager

Not sure what you mean with this .index directory. A lookup would be in lookups/something.csv, I don't see on what this csv_[0-9]+.index/ regex shall match.
From my point of view, in the bundled app there should be: a.) the CSV put in the right place, b.) have all stanzas in default transforms.conf that are needed for the lookup, c.) have the optional field in the data model and d.) set the lookup stanzas to disabled = 1 in default. Enabling the lookup should need as less work as possible. The user should only need to add a disabled = 0 in the local transforms to have everything up and running.
Reason: The higher the amount of manual work, the higher the amount of time needed for a proper upgrade, as every manual change needs to be verified by the user if it still works with the new version. Imagine a user needs to manually copy over csv files or has to decide which one of the two possible data model json files (with and without the optional field) shall be used. If the user isn't careful enough, maybe his setup breaks during an upgrade.

bhawkins1
Communicator

@thomasb42 thanks for the response. @aaraneta_splunk has modified my original question and removed some info about Lookup Definitions , which, when configured, produces a directory alongside lookup q.csv called q.csv_23491502.index/ . I need to version control the optional existence of both the lookup file ( q.csv ), and the definition index, ( q.csv_23491502.index/ ).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...