Why is my ps.sh command truncating output even after props.conf change?

[jreesnc]

Running Splunk 7.1.1 on RHEL 7

We are monitoring some applications that use the universal forwarder and the *nix app to send ps data to our indexer.

When I search the index, using ...

host = "myhostname.com" index=os sourcetype=ps

... we see only processes from root. When I run ./ps.sh from the bin directory, I see the missing processes.
I have copied props.conf from the default directory to the local for the *nix app and changed the truncate parameter to 0
ie.

[source::...linux.ps]
sourcetype = ps
HEADER_MODE = always
SHOULD_LINEMERGE = false

[ps]
SHOULD_LINEMERGE=false
LINE_BREAKER=(^$|[\r\n]+[\r\n]+)
TRUNCATE=0

Still no output.

Any other ideas?

[JohnWilly]

Is this fixed?  We are facing similar issues. Few processes and randomly not reporting.

[jreesnc]

Here is what I also tried ( added on all my indexers)

/opt/splunk/etc/apps/Splunk_TA_nix/local

added props.conf

Put here in it
[ps]
TRUNCATE = 0
MAX_EVENTS = 2000

Still I get truncated indexing on my PS output and cannot see the processes I am try8ng to set alert actions on.
What am I missing

[jreesnc]

Any other ideas?
Certainly, others with servers running a lot of processes has seen this?

[harsmarvania57]

Have you restarted splunk after configuration changes?

[jreesnc]

Yes. I have restarted it after the change. This one is really got me scratching my head

[jreesnc]

I guessed, and here si where I put the suggested inputs on my indexer:

[splunk@dpydalspl0101 local]$ pwd
/opt/splunk/etc/system/local
[splunk@dpydalspl0101 local]$ cat props.conf
[ps]
TRUNCATE = 0
MAX_EVENTS = 1000
[splunk@dpydalspl0101 local]$ 

[jreesnc]

Example:
[splunk@dpydalspl0101 apps]$ pwd
/opt/splunk/etc/apps
[splunk@dpydalspl0101 apps]$ find . -name props.conf
./learned/local/props.conf
./splunk_archiver/default/props.conf
./monitoringwincontainers/default/props.conf
./Perficient_TM1_App/default/props.conf
./sample_app/default/props.conf
./SplunkLightForwarder/default/props.conf
./search/default/props.conf
./legacy/default/props.conf
./splunk_instrumentation/default/props.conf
./splunk_monitoring_console/default/props.conf
./monitoringdocker/default/props.conf
./Splunk_TA_Infrastructure/default/props.conf

[jreesnc]

Here is the output (sorry for delay)

[tm1adm@dpydaltm1001 etc]$ cd ..
[tm1adm@dpydaltm1001 splunkforwarder]$ cd bin
[tm1adm@dpydaltm1001 bin]$ ./splunk cmd btool props list --debug ps
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf [ps]
/opt/splunkforwarder/etc/system/default/props.conf           ANNOTATE_PUNCT = True
/opt/splunkforwarder/etc/system/default/props.conf           AUTO_KV_JSON = true
/opt/splunkforwarder/etc/system/default/props.conf           BREAK_ONLY_BEFORE = 
/opt/splunkforwarder/etc/system/default/props.conf           BREAK_ONLY_BEFORE_DATE = True
/opt/splunkforwarder/etc/system/default/props.conf           CHARSET = UTF-8
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf DATETIME_CONFIG = CURRENT
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-UsedBytes = RSZ_KB*1024
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-app = if(ARGS!="<noArgs>", COMMAND." ".ARGS,COMMAND)
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-cpu_time = replace(CPUTIME, "^00:[0]{0,1}", "")
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-mem_used = RSZ_KB*1024
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-process = if(ARGS!="<noArgs>", COMMAND." ".ARGS,COMMAND)
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-process_mem_used = RSZ_KB*1024
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-process_name = replace(COMMAND, "[\[\]()]", "")
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf EVAL-time = replace(CPUTIME, "^00:[0]{0,1}", "")
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-cpu_load_percent_for_ps = pctCPU AS PercentProcessorTime,pctCPU as cpu_load_percent
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-dest_for_ps = host as dest
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-percentmemory_for_ps = pctMEM AS PercentMemory
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-process_cpu_used_percent = pctCPU as process_cpu_used_percent
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-process_id_for_ps = PID AS pid,PID as process_id
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-rss_for_ps = RSZ_KB AS rss
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-src_for_ps = host as src
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-stat_for_ps = S AS stat
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-tty_for_ps = TTY AS tty
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-user_for_ps = USER AS user
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf FIELDALIAS-vsz_for_ps = VSZ_KB AS vsz
/opt/splunkforwarder/etc/system/default/props.conf           HEADER_MODE = 
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf KV_MODE = multi
/opt/splunkforwarder/etc/system/default/props.conf           LEARN_MODEL = true
/opt/splunkforwarder/etc/system/default/props.conf           LEARN_SOURCETYPE = true
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf LINE_BREAKER = (^$|[\r\n]+[\r\n]+)
/opt/splunkforwarder/etc/system/default/props.conf           LINE_BREAKER_LOOKBEHIND = 100
/opt/splunkforwarder/etc/system/default/props.conf           MATCH_LIMIT = 100000
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DAYS_AGO = 2000
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DAYS_HENCE = 2
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DIFF_SECS_AGO = 3600
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DIFF_SECS_HENCE = 604800
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf MAX_EVENTS = 1000
/opt/splunkforwarder/etc/system/default/props.conf           MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunkforwarder/etc/system/default/props.conf           MUST_BREAK_AFTER = 
/opt/splunkforwarder/etc/system/default/props.conf           MUST_NOT_BREAK_AFTER = 
/opt/splunkforwarder/etc/system/default/props.conf           MUST_NOT_BREAK_BEFORE = 
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION = indexing
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-all = full
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-inner = inner
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-outer = outer
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-raw = none
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-standard = standard
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf SHOULD_LINEMERGE = false
/opt/splunkforwarder/etc/system/default/props.conf           TRANSFORMS = 
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/props.conf TRUNCATE = 0
/opt/splunkforwarder/etc/system/default/props.conf           detect_trailing_nulls = false
/opt/splunkforwarder/etc/system/default/props.conf           maxDist = 100
/opt/splunkforwarder/etc/system/default/props.conf           priority = 
/opt/splunkforwarder/etc/system/default/props.conf           sourcetype = 
/opt/splunkforwarder/etc/system/default/props.conf           [psv]
/opt/splunkforwarder/etc/system/default/props.conf           ANNOTATE_PUNCT = True
/opt/splunkforwarder/etc/system/default/props.conf           AUTO_KV_JSON = true
/opt/splunkforwarder/etc/system/default/props.conf           BREAK_ONLY_BEFORE = 
/opt/splunkforwarder/etc/system/default/props.conf           BREAK_ONLY_BEFORE_DATE = True
/opt/splunkforwarder/etc/system/default/props.conf           CHARSET = UTF-8
/opt/splunkforwarder/etc/system/default/props.conf           DATETIME_CONFIG = /etc/datetime.xml
/opt/splunkforwarder/etc/system/default/props.conf           FIELD_DELIMITER = |
/opt/splunkforwarder/etc/system/default/props.conf           HEADER_FIELD_DELIMITER = |
/opt/splunkforwarder/etc/system/default/props.conf           HEADER_MODE = 
/opt/splunkforwarder/etc/system/default/props.conf           INDEXED_EXTRACTIONS = psv
/opt/splunkforwarder/etc/system/default/props.conf           KV_MODE = none
/opt/splunkforwarder/etc/system/default/props.conf           LEARN_MODEL = true
/opt/splunkforwarder/etc/system/default/props.conf           LEARN_SOURCETYPE = true
/opt/splunkforwarder/etc/system/default/props.conf           LINE_BREAKER_LOOKBEHIND = 100
/opt/splunkforwarder/etc/system/default/props.conf           MATCH_LIMIT = 100000
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DAYS_AGO = 2000
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DAYS_HENCE = 2
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DIFF_SECS_AGO = 3600
/opt/splunkforwarder/etc/system/default/props.conf           MAX_DIFF_SECS_HENCE = 604800
/opt/splunkforwarder/etc/system/default/props.conf           MAX_EVENTS = 256
/opt/splunkforwarder/etc/system/default/props.conf           MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunkforwarder/etc/system/default/props.conf           MUST_BREAK_AFTER = 
/opt/splunkforwarder/etc/system/default/props.conf           MUST_NOT_BREAK_AFTER = 
/opt/splunkforwarder/etc/system/default/props.conf           MUST_NOT_BREAK_BEFORE = 
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION = indexing
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-all = full
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-inner = inner
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-outer = outer
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-raw = none
/opt/splunkforwarder/etc/system/default/props.conf           SEGMENTATION-standard = standard
/opt/splunkforwarder/etc/system/default/props.conf           SHOULD_LINEMERGE = False
/opt/splunkforwarder/etc/system/default/props.conf           TRANSFORMS = 
/opt/splunkforwarder/etc/system/default/props.conf           TRUNCATE = 10000
/opt/splunkforwarder/etc/system/default/props.conf           category = Structured
/opt/splunkforwarder/etc/system/default/props.conf           description = Pipe-separated value format. Set header and other settings in "Delimited Settings"
/opt/splunkforwarder/etc/system/default/props.conf           detect_trailing_nulls = false
/opt/splunkforwarder/etc/system/default/props.conf           maxDist = 100
/opt/splunkforwarder/etc/system/default/props.conf           priority = 
/opt/splunkforwarder/etc/system/default/props.conf           pulldown_type = true
/opt/splunkforwarder/etc/system/default/props.conf           sourcetype = 
[tm1adm@dpydaltm1001 bin]$ 

[harsmarvania57]

You need to apply MAX_EVENTS setting on Indexer or Heavy Forwarder not on Universal Forwarder (UF).

[jreesnc]

OK, that is helpful. I have been making the change on the universal forwarder on the source machine.

I have never modified a property on our indexer before, where would I look for this file, there are several. We do not use heavy forwarders

[harsmarvania57]

I'll suggest you to install Splunk_TA_nix on Indexer because this add-on contains Index field extraction so this add-on is require on Indexer as well. If you do not want to monitor any inputs (like ps, cpu) on Indexer then you can remove Splunk_TA_nix/local/inputs.conf from Add-on.

And put props.conf configuration in Splunk_TA_nix/local/props.conf on Indexer and then restart splunk on Indexer.

[harsmarvania57]

Hi,

Which version of Splunk_TA_nix are you running ? While looking at Splunk_TA_nix version 5.2.4, it is running below command on RHEL and it is generating correct output.

ps -wweo uname,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args

Can you please run above command on RHEL 7 and check whether are you getting all running processes or not?

[jreesnc]

I am running "Splunk Add-on for Unix and Linux version 5.2.4"

Output is below.
Looks ok to me

[mesadmin@dpydaltm1001 ~]$  ps -wweo uname,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args
USER        PID PSR %CPU     TIME %MEM   RSZ    VSZ TT       S     ELAPSED COMMAND
root          1  27  0.0 00:03:01  0.0 11084 197780 ?        S  2-17:04:33 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root          2  11  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [kthreadd]
root          3   0  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [ksoftirqd/0]
root          5   0  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [kworker/0:0H]
root          7   0  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [migration/0]
root          8   0  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [rcu_bh]
root          9  24  0.1 00:07:27  0.0     0      0 ?        S  2-17:04:33 [rcu_sched]
root         10   0  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [lru-add-drain]
root         11   0  0.0 00:00:01  0.0     0      0 ?        S  2-17:04:33 [watchdog/0]
root         12   1  0.0 00:00:01  0.0     0      0 ?        S  2-17:04:33 [watchdog/1]
root         13   1  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [migration/1]
root         14   1  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [ksoftirqd/1]
root         16   1  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [kworker/1:0H]
root         17   2  0.0 00:00:01  0.0     0      0 ?        S  2-17:04:33 [watchdog/2]
root         18   2  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [migration/2]
root         19   2  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [ksoftirqd/2]
root         21   2  0.0 00:00:00  0.0     0      0 ?        S  2-17:04:33 [kworker/2:0H]
root         22   3  0.0 00:00:01  0.0     0      0 ?        S  2-17:04:33 [watchdog/3]

[harsmarvania57]

So when you ran command you got all processes but when you are running ps.sh you are not getting all processes in output ?

[jreesnc]

To answer your question. I ran two tests:
Test 1.
I ran ../etc/apps/Splunk_TA_nix/bin/ps.sh
I got the full output, including the processes that are missing on splunk. The output is large, 509 lines out output.

Test 2.
I ran the command you suggested
'ps -wweo uname,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args'

I received the same output. Again, my processes appear.

In splunk, I only see processes from root , in the output of the command line ps , a non root process does not even appear until the 309th line. So, it looks, to me, like a clear case of the entire output stream is not getting forwarded. It certainly looks like it is getting truncated.

I am attaching the entire output here for reference

https://drive.google.com/file/d/14fhE90bWMQQNCsv4Kz0D1B6WEytQ3mTo/view?usp=sharing

[harsmarvania57]

Can you please try to set below config in Indexers/Heavy Forwarder whichever comes first from Universal Forwarder and then check whether it is truncating lines or not.

props.conf

[ps]
MAX_EVENTS = 1000

[jreesnc]

Done, still does not work. This is very frustrating. We wanted to set some alert actions based on the presences or absence of processes.
Here is the local props.conf
$ cat props.conf

[source::...linux.ps]
sourcetype = ps
HEADER_MODE = always
SHOULD_LINEMERGE = false

[ps]
MAX_EVENTS = 1000
SHOULD_LINEMERGE=false
LINE_BREAKER=(^$|[\r\n]+[\r\n]+)
TRUNCATE=0
DATETIME_CONFIG = CURRENT
KV_MODE = multi

FIELDALIAS-cpu_load_percent_for_ps = pctCPU AS PercentProcessorTime,pctCPU as cpu_load_percent
FIELDALIAS-dest_for_ps = host as dest
## The "start" field in this data is never used so no extractions applied here.
FIELDALIAS-process_id_for_ps = PID AS pid,PID as process_id
FIELDALIAS-percentmemory_for_ps = pctMEM AS PercentMemory
FIELDALIAS-rss_for_ps = RSZ_KB AS rss
FIELDALIAS-src_for_ps = host as src
FIELDALIAS-vsz_for_ps = VSZ_KB AS vsz
FIELDALIAS-tty_for_ps = TTY AS tty
FIELDALIAS-stat_for_ps = S AS stat
FIELDALIAS-user_for_ps = USER AS user
FIELDALIAS-process_cpu_used_percent = pctCPU as process_cpu_used_percent
EVAL-process_mem_used=RSZ_KB*1024

# The "app" field is the conjunction of COMMAND plus ARGS
# Note that the UNIX app joins arguments with an underscore.
EVAL-app = if(ARGS!="<noArgs>", COMMAND." ".ARGS,COMMAND)
EVAL-process = if(ARGS!="<noArgs>", COMMAND." ".ARGS,COMMAND)
EVAL-process_name = replace(COMMAND, "[\[\]()]", "")

# Truncate needless leading zeroes from the cumulative CPU time field.
EVAL-cpu_time = replace(CPUTIME, "^00:[0]{0,1}", "")
EVAL-time = replace(CPUTIME, "^00:[0]{0,1}", "")

# UsedBytes is calculated as RSZ_KB*1024. Previously it was calculated using 
# %MEM and the "Mem:" header from "top -bn 1", which tended to underestimate
# compared to this value. This is a rough measure of resident set size (i.e.,
# physical memory in use).
EVAL-mem_used=RSZ_KB*1024
EVAL-UsedBytes=RSZ_KB*1024

All I see is the processes from root. None of the other processes appear.
https://www.flickr.com/photos/jrees/30600350547/sizes/o/

[harsmarvania57]

Can you please run below command on Indexer/Heavy Forwarder to check MAX_EVENTS and TRUNCATE settings and can you please provide output of below command ?

$SPLUNK_HOME/bin/splunk cmd btool props list --debug ps

Also can you please check below error in $SPLUNK_HOME/var/log/splunk/splunkd.log on Indexer/Heavy Forwarder for ps sourcetype?

WARN  AggregatorMiningProcessor - Breaking event because limit of 

OR

WARN  LineBreakingProcessor - Truncating line because limit of

[jreesnc]

No warnings

[tm1adm@dpydaltm1001 log]$ cd splunk
[tm1adm@dpydaltm1001 splunk]$ cat splunkd.log | grep -i AggregatorMiningProcessor
[tm1adm@dpydaltm1001 splunk]$ cat splunkd.log | grep -i LineBreakingProcessor    
[tm1adm@dpydaltm1001 splunk]$ 

[harsmarvania57]

It looks like you are checking this ERROR on Universal Forwarder, you need to check those error on Indexer Or Heavy Forwarder whichever comes first from UF.