Please manually set "disabled = 0" for hadoopmon_cpu.sh and restart splunkd. Hopefully in about 5 minutes hadoop_host2maxcpu lookup would be generated
... View more
This one depends on the scripted input 'hadoopmon_cpu.sh' be invoked and run correctly.
1) check if 'source=cpu' returns events
2) check if executing the script 'Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh' in a terminal returns data like
CPU pctUser pctNice pctSystem pctIowait pctIdle
all 1.52 0.00 2.02 0.00 96.46
0 1.01 0.00 2.02 0.00 96.97
1 2.00 0.00 2.00 0.00 96.00
3) check with 'hopsconfig.sh --list-all' whether hadoopmon_cpu.sh is enabled
... View more
I suspect the mapred-site.xml that is needed to create the hadoop_host2mapred lookup is not being indexed by Splunk. The best advice I can give right now without spamming this thread anymore is to verify that mapred-site.xml is being indexed by Splunk.
... View more
The 3 enabling failures were probably caused by those files/directory not existed yet.
Now with local/inputs.conf generated,
1) do you see monitor:///usr/lib/hadoop/conf/*.xml enabled? If not, please manually add the corresponding entry (suitable for your cluster) to local/inputs.conf and restart splunkd. (see the example above)
2) is hadoop_host2mapred lookup still empty? if so, does running '|savedsearch __generate_lookup_hadoop_host2mapred' fix the problem?
... View more
1) You can safely delete eventgen.conf. It is not used.
2) Can you try the steps here http://docs.splunk.com/Documentation/HadoopOps/latest/HadoopOps/DeployandlaunchTA#Update_forwarders_with_changes_to_monitored_file_locations and see if it fixes the problem?
3) To be cautious... did you restart splunkd after making changes to inputs.conf?
... View more
For reference, below is a working example of local/inputs.conf of CDH3 namenode.
[monitor:///usr/lib/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs
[monitor:///var/log/hadoop/hadoop-cmf-hdfs1-NAMENODE-cdh1.tw.splunk.com*.out]
disabled = 0
sourcetype = hadoop_namenode
index = hadoopmon_logs
... View more
I should make a correction here, check if xml files in your HADOOP_CONF_DIR is being monitored in Splunk_TA_hadoopops/local/inputs.conf, something like
[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs
However, I just remembered you have host2hdfs run correctly, which means it should be there.
... View more
a quickfix worth trying is running this search query in 'HadoopOps'->'Search'
"|savedsearch __generate_lookup_hadoop_host2hdfs"
You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:
"|savedsearch __generate_lookup_hadoop_host2mapred"
... View more
This suggest that 'ps' data are not getting in from forwarder (Splunk_TA_hadoopops). Is [script://./bin/hadoopmon_ps.sh] enabled?
Edit:
As pierre4splunk replied below, core-site.xml and mapred-site.xml need to be monitored to create hdfs and mapred lookup, respectively. maxcpu lookup depends on results from hadoopmon_cpu.sh. Might be worth double-checking.
... View more