Splunk Search

Restricting search results to or excluding Extracted Fields

Carmageddon
New Member

Hello,

I would like to filter a search result, of irrelevant data, to display less information so its easier to spot.

I have extracted a field, the regex was created and test worked fine, I called it "LocalTime".

Then, I went to run a search: eventtype="ConnectionLogs" | fields -LocalTime

I still got the field in the search results.

I tried to run eventtype="ConnectionLogs" | fields LocalTime

I still got all fields showing up.

Any advice on how to filter out fields correctly? perhaps I didnt extract the field the right way? or extracting fields != assigning names to fields? (pretty sure it is though).

Thanks!

Tags (2)
0 Karma

Genti
Splunk Employee
Splunk Employee

Ok, usually by default the fields that show up in the events (underneath the event itself) are Host, index, and source. The rest of the extracted fields are to the side (left)

If you have set up your LocalTime as a field, either it is showing on the left handside menu or is under the "ALL xx Fields" and you need to add it in order to show it.

If it is indeed in the left hand side menu, then just running:
eventtype="ConnectionLogs" | fields - LocalTime
should remove it from the list and place it in the "All xx fields" Other wise there is no change that happens.

So, if you want to remove fields under the event, or if you want to remove it so it doesnt show from the left hand side, you need to use:
eventtype="ConnectionLogs" | fields - LocalTime

(note the space between the - sign and LocalTime)
Hope this helped.

.gz

Lowell
Super Champion

You are correct. Eventtype definitions cannot include pipes, they must be simple search expressions only. Note the docs page: http://www.splunk.com/base/Documentation/4.1.3/Knowledge/Configureeventtypes

0 Karma

Lowell
Super Champion

You don't want | fields - host - source - sourcetype , you would want to do | fields - host source sourcetype (You don't repeat -)

0 Karma

Carmageddon
New Member

Another problem is, that I cant seem to use pipes in eventtypes 😞 how can I add these parameters to a forced eventtype upon certain role?

0 Karma

Carmageddon
New Member

You are right, it indeed just removes from the fields list on the left, but NOT from the result.
What I want, is REMOVE a field from display result... like, trim down the result so it shows less information, and therefore relevant info is easier to spot.

This is a good progress anyway, now I know how to | fields - host - source - sourcetype to not display that silly lines on each result 🙂

but I still want to trim the result, just like in SQL you can do select field1, field1 from someTable;
This is essentially what I want to do - deselect (or select specific) fields.

0 Karma

Lowell
Super Champion

I think you want a space.

Instead of:

fields -LocalTime

Try using:

fields - LocalTime


Based on a comment you made. It sounds like you not only want to hide the value from the shown field values (shown bellow the event) and also want to hide the portion of the raw text where that field was extracted from. (In other words, hide data in your raw event.) This is a very different question, and not something splunk will do for you out of the box.

And you generally you wouldn't want splunk to behave this way, and I think a simple example would demonstrate this. Say you have an event like: 2010-06-23 10:00:00 user joe@example.com : login successful. And you have an extracted field called user that contains the value joe and an extracted field called login that contains the value joe@example.com. So since these fields overlap, if you said something like | fields + login | fields + user, would you see just @example.com.... Anyways, the bottom line is splunk doesn't work this way, and splunk never screws with your raw event text by default. That said, you can explicitly instruct splunk to do this.....

Since we do not know how LocalTime is being extracted and we don't have a sample event, it's impossible to give you a working example. So here is the best I can do. This is an example that demonstrates how you could go about "hiding" certain portions of an event. Let's say you have an event that looks like this:

 Jun 23 09:10:01 stageftp1 /USR/SBIN/CRON[2559]: (root) CMD (   ntpdate-debian -p 5 -s)

And you really want to remove the pid field and hide it from the event text as well. (The pid value is 2959, but we will want to go ahead and remove the [] too.)

 my search | fields - pid | rex mode=sed "s/^(.*?)\[\d+\](.*)$/\1\2/"

The basic idea here is to use a "sed"-like expression using a simple "s/find/replace/" syntax. The rex command finds and captures all the text before "[2559]" and all the text after it (captured as group 1 and 2 respectively). The "replace" portion of our expression is \1\2 which simply says to keep our two capture groups (which simply excludes the value we are dropping). So the resulting text looks like this:

 Jun 23 09:10:01 stageftp1 /USR/SBIN/CRON: (root) CMD (   ntpdate-debian -p 5 -s)

You can use this same type of approach to cut out the unloved parts of your raw text. Obviously your regular expression will have to be updated to find and remove your LocalTime value, wherever that is being extracted from in your event.


If you simply want a table showing an explicit set of columns, then try building a dashboard with a <table> element. You can explicitly set which fields you would like to have displayed.

0 Karma

Lowell
Super Champion

If you post an example event I can give you an example rex command. (Please add the example to your question with the "edit" link, don't post it in the comments because the formatting will get messed up.) You can get regex help here: http://www.regular-expressions.info/, keep in mind that the s/find/replace/ is not a REGEX per se, it's a commonly known format supported by splunk and made populate by the sed unix command. So if you copy the whole expression, and regex tool may not understand it.

0 Karma

Carmageddon
New Member

Thanks, I am at home now, was busy day at work so I quickly tested your solution (or rather - the REGEX part of it) in Expresso (a REGEX design application) - it does not appear to be catching your example input (I copy pasted the part between the outer "/" slashes.

How did you design this REGEX expression?

0 Karma

Lowell
Super Champion

I've updated my answer to include additional example of how to remove text from the raw event.

0 Karma

Carmageddon
New Member

Thanks for the syntax, but its not the full solution

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...