I have two different sets of data coming in Splunk:
Dec 1 08:43:07 a4-hpc2-2.llnl.gov logger: dom0stat42 : timestamp=08:43:02 pool=General2 hardware=a4-hpc2-2 dom0_one_min_load=0.50 dom0_free_memory=114 dom0_total_memory=1024 dom0_used_memory=910 xen_free_memory=12507 xen_total_memory=49149 xen_used_memory=36642 dom0_total_vmos_mb=3355444 dom0_used_vmos_mb=2684594 dom0_free_vmos_mb=670850 stolen_cpu_ticks= steal_time=0.20
Dec 1 08:42:12 a4-hpc2-2.llnl.gov logger: gvmstats timestamp=08:42:02 pool=General2 hardware=a4-hpc2-2 gvm=splatint0007 memory=2048 vcpu=1 cpu_seconds=244751.0 vnc_console=a4-hpc2-2:5906
I am trying to search through them and pull out some key information:
index=unix gvmstats OR dom0stat42 hardware=a4-hpc2-2
| eval xen_free_memory_GB=round(xen_free_memory/1024,2)
| stats values(xen_free_memory_GB), values(gvm), values(vcpu), values(memory), values(vnc_console) by pool hardware
All the data gets combined, but scrambled:
pool hardware values(xen_free_memory_GB) values(gvm) values(vcpu) values(memory) values(vnc_console)
General2 a4-hpc2-2 12.21 Domain-0 1 1024 a4-hpc2-2:5900
dbdev0003 2 2048 a4-hpc2-2:5901
oidev0001 4 4096 a4-hpc2-2:5902
oidvqa0001 8 8192 a4-hpc2-2:5903
` ` savidev0006 a4-hpc2-2:5904
saviqa0010 a4-hpc2-2:5905
secwsint0003 a4-hpc2-2:5906
splatint0007 a4-hpc2-2:5907
The pool, hardware, and "xen_free_memory_GB" are correct. Every thing past that is all screwed up. Domain-0 has 1024 (correct, but only by luck), 12vcpu, and no vnc_console.
Any ideas of how to fix this?
Table gives me all the data, and formats it correctly. But I get ALL the data, not just the "last" values. using last() does not seem to work. Ideas?
Last as in "most recent"? Remember that Splunk naturally sorts that data in reverse chronological order, so the first results are the most recent.
Your stats command is saying "for each combination of pool and hardware, show me ALL the values for the other fields."
Stats is used to summarize. I think you may prefer the table command for this case:
...
| table pool hardware xen_free_memory_GB gvm vcpu memory) vnc_console
Or maybe you can describe the output that you want in more detail...
If you just want to see the most recent event, do this
...
| table pool hardware xen_free_memory_GB gvm vcpu memory) vnc_console
| head 1
Or, perhaps you want this
...
| table pool hardware xen_free_memory_GB gvm vcpu memory) vnc_console
| tail 1