Getting Data In

I'm struggling with how I SHOULD be doing inputs and also props/transforms/etc stuff within Splunk Cloud.

khourihan_splun
Splunk Employee
Splunk Employee

from a customer:

I'm struggling with how I SHOULD be doing inputs and also props/transforms/etc stuff within Splunk Cloud.

In short, I am used to using shell access and I want to know what's the best practice on how to live within the GUI to accomplish what I usually do through "vi".

1 Solution

khourihan_splun
Splunk Employee
Splunk Employee

Presently with Splunk Cloud you have GUI/UI access only. For search time operations like on-the-fly field extractions, or report transformations -- they can indeed be done using the GUI.

For example, my insurgency app, which creates statistics for the video game Insurgency, I want to extract player names from the logs.

Extracts are a good way to do field extractions in one fell-swoop (making a change in one file)

If your a CLI guy, normally you'd pop into a shell and fire up emacs on a props.conf in the the $SPLUNK_HOME/etc/apps/insurgency_stats/local/props.conf and put this in there:

[insurgency]
EXTRACT-killed = killed \"(?<killed>.*?)<
EXTRACT-killer = :\s+\"(?<killer>.*?)<

If you wanted to do the same using the GUI you'd login and goto Settings-> Fields-> Field Extractions.

alt text

If you wanted to do something slightly more complicated, where say you had an app that you've had installed in Splunk Cloud and you wanted to change its props AND transforms.

Take for example the F5 app, which has the both props and transforms. Lets say you wanted to view or modify how its extracting fields. If you were a GUI CLI guy you'd go into $SPLUNK_HOME/etc/app/SplunkforF5Networks/default and look the props.conf and transforms.conf files.

To edit props.conf settings from the GUI, you'd go to Settings-> Fields.

The entries in the props.conf stanza's line up to different links on this page:

alt text

So for example, lets look at the six REPORT-s in the props.conf and we'll find them in Settings-> Fields-> Field extractions:

alt text
Above you see hows the Reports from the props.conf correspond to entries in Field Extractions. Note I filtered the field extractions in the GUI to just show Reports in this apps context.

Now for look at the F5:LTM:Syslog:REPORT, note its Extraction/Transform. That will have a corresponding entry in transforms.conf.

To view it in the GUI, I go back to Settings-> Fields -> Field transformations and filter for the f5-syslog-eventcode
alt text

and if I click on the name, I will see the transformation's details:

alt text

If I look in my $SPLUNK_HOME/etc/app/SplunkforF5Networks/default/transforms.conf, I find this corresponding entry:

#transforms.conf
[CHOP]
[f5-syslog-eventcode]
REGEX = \]:\s(........:.):\sPool\s(\S+)\smember\s(\S+)\smonitor\sstatus\s(\S+)
\.\s\[\swas\s(\S+)\sfor\s(\S+)
FORMAT = event_code::$1 ltm_pool::$2 ltm_member::$3 ltm_monitor_status::$4 ltm
_prevstatus::$5 ltm_prevstatus_time::$6
[CHOP]

If I were to make a change via the GUI, changes would not be written to $SPLUNK_HOME/etc/app/SplunkforF5Networks/default/transforms.conf. Can you guess why?

Because in Splunk, you never change files in the default directory. Best practices states if you are going to change a setting, that you copy your props or transforms into the local directory and then edit. (or better yet, only copy the bits that you plan to change from default).

The GUI back end follows this best practice for you, so any changes would be written to $SPLUNK_HOME/etc/app/SplunkforF5Networks/local/transforms.conf.

So you can see that while UI is different that CLI, its possible to do search time props and transforms configuration of Splunk Cloud via the UI.

Its important to note, that if you are doing index time transforms such as sourcetype overriding or reformatting of data, those changes need to be requested via ticket. The present UI will let you install those changes, but you will notice the output won't be change.

Got a question about whether you can or can't make a change? just ask cloudse@splunk.com or contact support.

Thats all for now folks.

View solution in original post

khourihan_splun
Splunk Employee
Splunk Employee

Presently with Splunk Cloud you have GUI/UI access only. For search time operations like on-the-fly field extractions, or report transformations -- they can indeed be done using the GUI.

For example, my insurgency app, which creates statistics for the video game Insurgency, I want to extract player names from the logs.

Extracts are a good way to do field extractions in one fell-swoop (making a change in one file)

If your a CLI guy, normally you'd pop into a shell and fire up emacs on a props.conf in the the $SPLUNK_HOME/etc/apps/insurgency_stats/local/props.conf and put this in there:

[insurgency]
EXTRACT-killed = killed \"(?<killed>.*?)<
EXTRACT-killer = :\s+\"(?<killer>.*?)<

If you wanted to do the same using the GUI you'd login and goto Settings-> Fields-> Field Extractions.

alt text

If you wanted to do something slightly more complicated, where say you had an app that you've had installed in Splunk Cloud and you wanted to change its props AND transforms.

Take for example the F5 app, which has the both props and transforms. Lets say you wanted to view or modify how its extracting fields. If you were a GUI CLI guy you'd go into $SPLUNK_HOME/etc/app/SplunkforF5Networks/default and look the props.conf and transforms.conf files.

To edit props.conf settings from the GUI, you'd go to Settings-> Fields.

The entries in the props.conf stanza's line up to different links on this page:

alt text

So for example, lets look at the six REPORT-s in the props.conf and we'll find them in Settings-> Fields-> Field extractions:

alt text
Above you see hows the Reports from the props.conf correspond to entries in Field Extractions. Note I filtered the field extractions in the GUI to just show Reports in this apps context.

Now for look at the F5:LTM:Syslog:REPORT, note its Extraction/Transform. That will have a corresponding entry in transforms.conf.

To view it in the GUI, I go back to Settings-> Fields -> Field transformations and filter for the f5-syslog-eventcode
alt text

and if I click on the name, I will see the transformation's details:

alt text

If I look in my $SPLUNK_HOME/etc/app/SplunkforF5Networks/default/transforms.conf, I find this corresponding entry:

#transforms.conf
[CHOP]
[f5-syslog-eventcode]
REGEX = \]:\s(........:.):\sPool\s(\S+)\smember\s(\S+)\smonitor\sstatus\s(\S+)
\.\s\[\swas\s(\S+)\sfor\s(\S+)
FORMAT = event_code::$1 ltm_pool::$2 ltm_member::$3 ltm_monitor_status::$4 ltm
_prevstatus::$5 ltm_prevstatus_time::$6
[CHOP]

If I were to make a change via the GUI, changes would not be written to $SPLUNK_HOME/etc/app/SplunkforF5Networks/default/transforms.conf. Can you guess why?

Because in Splunk, you never change files in the default directory. Best practices states if you are going to change a setting, that you copy your props or transforms into the local directory and then edit. (or better yet, only copy the bits that you plan to change from default).

The GUI back end follows this best practice for you, so any changes would be written to $SPLUNK_HOME/etc/app/SplunkforF5Networks/local/transforms.conf.

So you can see that while UI is different that CLI, its possible to do search time props and transforms configuration of Splunk Cloud via the UI.

Its important to note, that if you are doing index time transforms such as sourcetype overriding or reformatting of data, those changes need to be requested via ticket. The present UI will let you install those changes, but you will notice the output won't be change.

Got a question about whether you can or can't make a change? just ask cloudse@splunk.com or contact support.

Thats all for now folks.

zella
Explorer

@khourihan_splunk, so are you saying this app is good for masking data independently of Splunk Support, or is this something that will require a ticket?

0 Karma

khourihan_splun
Splunk Employee
Splunk Employee

You need to use transforms.conf to mask the data, so that's not something you can do in the UI presently.  However, you can create a configuration app and, now with Splunk 7.2+ you can upload both the props and transforms and install them to both Seach Head and indexers, via a new Splunk API call that applies cluster bundles to the indexers via the clustermaster.  All that is done behind the scenes.

 

Check this out: https://docs.splunk.com/Documentation/SplunkCloud/8.0.2006/User/PrivateApps

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...