Splunk Search

Does Splunk have an easier way to get FileSize to human readable format?

tb5821
Communicator

I'm surprised splunk doesn't have an easier way to get a human readable format by passing it the field you want it to work its magic on al-la du -h

So here's my question, I have a field called fs which is file size but its in bytes. I want to convert that to the proper Kb,Mb,Gb format... whats the best way to do this?

Labels (1)
Tags (4)
0 Karma
1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

I have this setup. You can put it into $SPLUNK_HOME/etc/system/local/props.conf and it should work. You may want to have your field called "b" or "bytes" for it to work, or change the config to match your fields. This will auto-calculate kilos, megas and gigas automatically.

[host::*]
priority = 100
EVAL-kilobytes = if(isnotnull(kilobytes),kilobytes,bytes/1024)
EVAL-megabytes = if(isnotnull(megabytes),megabytes,bytes/1024/1024)
EVAL-gigabytes = if(isnotnull(gigabytes),gigabytes,bytes/1024/1024/1024)
EVAL-kb = if(isnotnull(kb),kb,b/1024)
EVAL-mb = if(isnotnull(mb),mb,b/1024/1024)
EVAL-gb = if(isnotnull(gb),gb,b/1024/1024/1024)

You can also use a macro (in the Search UI):
Manager -> Advanced Search -> Search Macros

Name: resize(1)
Arguments: bytes
definition = eval kilobytes = $bytes$/1024| eval megabytes=kilobytes/1024|eval gigabytes/1024

Then you can do your_search | resize(fs)

View solution in original post

jameswatts
Explorer

How about just installing Humanize?

Convert numbers, bytes, and timestamps into fuzzy, human-friendly units! Using the humanize library from https://github.com/jmoiron/humanize

Here are some example searches:
index=_internal | head | eval foo=random() | humanize command=naturalsize foo
index=_internal earliest=-7h | bin _time span=10m | eval foo=_time | stats first(foo) as foo by _time | humanize command=naturaltime foo

https://splunkbase.splunk.com/app/3104/

0 Karma

jpvlsmv
Path Finder

Ancient thread necropsy, but here's a better macro (IMO). It's ugly but it works just like the -h option on many GNU tools.
Usage:

| eval readable_size=`readable(size)`

Definition: (as seen in Settings -> Adv Search -> Search macros -> new:

if( $num$ < 1024, tostring($num$), if ( (floor($num$/pow(1024,floor(log($num$,1024))))) < 10
     , ( (tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + ".") + tostring(round((($num$/pow(1024,floor(log($num$,1024))))-(floor($num$/pow(1024,floor(log($num$,1024))))))*10))) + (substr("KMGTPEZY",floor(log($num$,1024)),1))
     , ( tostring((floor($num$/pow(1024,floor(log($num$,1024)))))) + (substr("KMGTPEZY",floor(log($num$,1024)),1)) )
   ) )

Not an eval-based definition (unchecked)
Arguments: num
Validation Expression: !isnum($num$)
Validation Error Message: Numeric value required

My key observation for the algorithm is that the log base 1024 will give you the "scale"-- KB or PB or whatever, by dropping the fractional part (i.e. log_10(5.6MB) = 2 -> M).

In working on this, I used meaningful names and replace-all'd them to fundamental eval functions. Here's the pseudocode:

if $num$ < 1024:
  printf("%4d", $num$)
else
  if $num$ reduces to a single digit
    # print in the form x.yS
    printf( "%d.%d%c", whole_part(reduction), 1st digit of frac_part(reduction), KMGTPEZY suffix appropriate for this scale
  else # This is actually the most common case.  The result is just the whole part of the reduction and the suffix
    printf("%3d%s", whole_part(reduction), suffix)

Hope this helps somebody
--Joe

alacercogitatus
SplunkTrust
SplunkTrust

I have this setup. You can put it into $SPLUNK_HOME/etc/system/local/props.conf and it should work. You may want to have your field called "b" or "bytes" for it to work, or change the config to match your fields. This will auto-calculate kilos, megas and gigas automatically.

[host::*]
priority = 100
EVAL-kilobytes = if(isnotnull(kilobytes),kilobytes,bytes/1024)
EVAL-megabytes = if(isnotnull(megabytes),megabytes,bytes/1024/1024)
EVAL-gigabytes = if(isnotnull(gigabytes),gigabytes,bytes/1024/1024/1024)
EVAL-kb = if(isnotnull(kb),kb,b/1024)
EVAL-mb = if(isnotnull(mb),mb,b/1024/1024)
EVAL-gb = if(isnotnull(gb),gb,b/1024/1024/1024)

You can also use a macro (in the Search UI):
Manager -> Advanced Search -> Search Macros

Name: resize(1)
Arguments: bytes
definition = eval kilobytes = $bytes$/1024| eval megabytes=kilobytes/1024|eval gigabytes/1024

Then you can do your_search | resize(fs)

alacercogitatus
SplunkTrust
SplunkTrust

weird. anyway, glad that worked for you!

0 Karma

tb5821
Communicator

Found the issue, the definition needs to be:

eval kilobytes=($fs$/1024) | eval megabytes=kilobytes/1024 |eval gigabytes=megabytes/1024

0 Karma

tb5821
Communicator

now getting:
Error in 'eval' command: Failed to parse the provided arguments. Usage: eval dest_key = expression

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

you need the "backticks". They are the key under Esc on the keyboard.

0 Karma

tb5821
Communicator

Error in 'SearchParser': Missing a search command before '''.

...| 'resize(fs)'

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

See my edit. You will need backticks around "resize(fs)". The answer system stripped them for me.

0 Karma

tb5821
Communicator

Can I do it in search? I don't have access to anything but the search UI

tfujita_splunk
Splunk Employee
Splunk Employee

This could be solution for you.

https://community.splunk.com/t5/Dashboards-Visualizations/Smart-conversion-of-large-numbers-to-human...

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",printf("%.2f",round(bytes/pow(1024,if(bytes=0,0,floor(min(log(bytes,1024),10)))),2)).case(bytes=0 OR log(bytes,1024)<1,"B ", log(bytes,1024)<2,"KiB", log(bytes,1024)<3,"MiB", log(bytes,1024)<4,"GiB", log(bytes,1024)<5,"TiB", log(bytes,1024)<6,"PiB", log(bytes,1024)<7,"EiB", log(bytes,1024)<8,"ZiB", log(bytes,1024)<9,"YiB", log(bytes,1024)<10,"RiB", log(bytes,1024)<11,"QiB", 1=1, "QiB"))

 

If you can install app or ask admin on your to install app,

installing add-on Numeral system macros for Splunk enables you to use macros numeral_binary_symbol(1) or numeral_binary_symbol(2).

Example

| makeresults count=35
```THIS SECTION IS JUST CREATING SAMPLE VALUES.```
| streamstats count as digit
| eval val=pow(10,digit-1), val=val+random()%val
| foreach bytes [eval <<FIELD>>=val]
| table digit val bytes
| fieldformat val=tostring(val,"commas")

```THE FOLLOWING LINES MAY BE WHAT ACHIEVES THE FORMAT YOU ARE LOOKING FOR.```
| fieldformat bytes=printf("% 10s",`numeral_binary_symbol(bytes,2)`)

 

Numeral system macros for Splunk

https://splunkbase.splunk.com/app/6595


Usage:

How to convert a large number to string with expressions of long and short scales, or neither.

https://community.splunk.com/t5/Splunk-Search/How-to-convert-a-large-number-to-string-with-expressio...

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...