How to create event in Zenoss v.5 which listens only on HTTPS?

[kachi444]

Hello,

Zenoss v.5 listens only on HTTPS for any incoming event.
'Create_event' function of this add-on doesn't seem to handle SSL, so HTTPS connection fails.
Any update/work-around for this?

thanks,

[shaskell_splunk]

The current version of the app v2.0.2 is compatible with Zenoss 5.x

https://splunkbase.splunk.com/app/2766/#/details

[bandit]

I've created a shell script for Splunk to run as an alert trigger action which will loop through events and post to the Zenoss api.
Note: you may have to customize for fields relevant to your splunk and zenoss environments respectively.

If this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk
Alerts should reference zenoss-alert-forward-wrapper.sh which enables logging and calls zenoss-alert-forward.sh

Template Search:

index=main
| head 1 
| eval severity="INFO" 
| eval resource=host_fqdn 
| eval component=sourcetype+":"+source 
| eval evclass="/App/Splunk" 
| eval summary="Splunk to Zenoss test pulse" 
| eval comment="--------------------------------------------------------------------------------" 
| eval comment="valid values for severity: CRITICAL ERROR WARNING INFO DEBUG CLEAR" 
| eval comment="resource must be host_fqdn if wanting to link to a host object in Zenoss or we may have to have a Zenoss script to do the conversion?"
| eval comment="component is currently set to soucetype + source, however can be customized. Check with your Zenoss admin"
| eval comment="evclass should always be /App/Splunk unless the Zenoss admin asks you to change it" 
| eval comment="table command is order-sensitive i.e. last command in an alert sent to zenoss should always be | table severity resource component evclass summary" 
| eval comment="--------------------------------------------------------------------------------" 
| table severity resource component evclass summary
| fillnull value=missing_value

zenoss-alert-forward-wrapper.sh

#!/usr/bin/env sh
# export all vars so the child script will have access to them
# if this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk

export SEARCH_COUNT="$1"       # $1 - Number of events returned
export SEARCH_TERMS="$2"       # $2 - Search terms
export SEARCH_QUERY="$3"       # $3 - Fully qualified query string
export SEARCH_NAME="$4"        # $4 - Name of saved search
export SEARCH_REASON="$5"      # $5 - Reason saved search triggered
export SEARCH_URL="$6"         # $6 - URL/Permalink of saved search
export SEARCH_TAGS="$7"        # $7 - Always empty as of 4.1
export SEARCH_PATH="$8"        # $8 - Path to raw saved results in Splunk instance (advanced)

export RESULTS_GZ="$SEARCH_PATH"

SCRIPT_DIR=$SPLUNK_HOME/bin/scripts
LOG_FILE=$SPLUNK_HOME/var/log/splunk/zenoss-alert-forward.log

# execute script and log standard out and standard error 
$SCRIPT_DIR/zenoss-alert-forward.sh 2>&1 > $LOG_FILE

zenoss-alert-forward.sh

  #!/usr/bin/env sh
# zenoss-alert-forward.sh
# read through splunk results and generate an alert per splunk event to push to zenoss api via curl
# if this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk

# rob@jordan2000.com
# version 1.0.5
# created 8/23/2017
# updated 11/7/2018

# Parameters passed in from the alert.
# $1-$9 is the positional parameter list.

# --------------------------------------------------------------------------------
# *** CONFIG PARAMS
# --------------------------------------------------------------------------------

 ZENOSS_API_USER="your_zenoss_api_user_here"
 ZENOSS_API_PASS="your_zenoss_api_pass_here"
 ZENOSS_URL="https://your.zenoss.api.url.here:443"
ZENOSS_API_PATH="/zport/dmd/evconsole_router"
# maximum number of events that can be sent to Zenoss per Splunk alert
# each Splunk event will be sent as individual alert
# anything over the event limit will be truncated
EVENT_LIMIT=100

# in most cases, you shouldn't have to change parameters below this line
ZENOSS_API_URL="${ZENOSS_URL}${ZENOSS_API_PATH}"
ZENOSS_MESSAGE_BEGIN="'{\"action\":\"EventsRouter\",\"method\":\"add_event\",\"data\":[{"
ZENOSS_MESSAGE_END="}],\"type\":\"rpc\",\"tid\":1}'"

SCRIPT_DIR="$SPLUNK_HOME/bin/scripts"
LOG_FILE="$SPLUNK_HOME/var/log/splunk/zenoss-alert-forward.log"
TMP_DIR="/tmp/zenoss-alert-forward.$$"
CURL_POST_SCRIPT="$TMP_DIR/zenoss-alert-forward-curl-post.$$.sh"

echo "START SCRIPT"
echo "--------------------------------------------------------------------------------"
/bin/date

# create tmp dir so that this script can run multiple copies in parallel without collision
mkdir $TMP_DIR

# --------------------------------------------------------------------------------
# *** DETERMINE RESULTS LOCATION AND UNZIP
# --------------------------------------------------------------------------------
RESULTS_FILE=`echo $RESULTS_GZ|awk -F".gz" '{print $1}'`

echo "RESULTS_GZ: $RESULTS_GZ"
echo "RESULTS_FILE: $RESULTS_FILE"

echo "*** cd $TMP_DIR"
cd $TMP_DIR

echo "*** cp $RESULTS_GZ $TMP_DIR"
cp $RESULTS_GZ $TMP_DIR

echo "*** gunzip $RESULTS_FILE"
gunzip $RESULTS_GZ

# --------------------------------------------------------------------------------
# *** echo SCRIPT VARS
# --------------------------------------------------------------------------------

echo "SCRIPT_DIR: $SCRIPT_DIR"
echo "LOG_FILE: $LOG_FILE"
echo "TMP_DIR: $TMP_DIR"

# --------------------------------------------------------------------------------
# *** echo ZENOSS VARS
# --------------------------------------------------------------------------------

#echo "ZENOSS_API_USER: $ZENOSS_API_USER"
#echo "ZENOSS_API_PASS: $ZENOSS_API_PASS"
echo "ZENOSS_URL: $ZENOSS_URL"
echo "ZENOSS_API_PATH: $ZENOSS_API_PATH"
echo "ZENOSS_API_URL: $ZENOSS_API_URL"
echo "EVENT_LIMIT: $EVENT_LIMIT"
echo "SCRIPT_DIR: $SCRIPT_DIR"
echo "LOG_FILE: $LOG_FILE"
echo "TMP_DIR: $TMP_DIR"
echo "ZENOSS_MESSAGE_BEGIN: $ZENOSS_MESSAGE_BEGIN"
echo "ZENOSS_MESSAGE_END: $ZENOSS_MESSAGE_END"

# --------------------------------------------------------------------------------
# *** echo DEFAULT SPLUNK VARS
# --------------------------------------------------------------------------------
echo "SEARCH_COUNT: $SEARCH_COUNT"
#echo "SEARCH_TERMS: $SEARCH_TERMS"
echo "SEARCH_QUERY: $SEARCH_QUERY"
echo "SEARCH_NAME: $SEARCH_NAME"
echo "SEARCH_REASON: $SEARCH_REASON"
echo "SEARCH_URL: $SEARCH_URL"
#echo "SEARCH_TAGS: $SEARCH_TAGS"
echo "SEARCH_PATH: $SEARCH_PATH"

# --------------------------------------------------------------------------------
# *** READ THROUGH RESULTS RECORDS, PARSE & FORWARD ALERTS TO ZENOSS USING CURL
# --------------------------------------------------------------------------------
# this allows us to grab addtional Splunk fields written out from a scheduled alert as a record in the table format
# note: your SPLUNK query must be in the following format: base search | table SEVERITY RESOURCE COMPONENT EVCLASS SUMMARY otherwise you will need to mod this script to use a new format/order

echo "*** cat $RESULTS_FILE"

# enable line below for additional debug
# cat $RESULTS_FILE

# determine field positions from results header
HEADER_RECORD=`cat $RESULTS_FILE | head -1 | sed -e 's/,,/,/g'`
FIELD_01_NAME=`echo $HEADER_RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_02_NAME=`echo $HEADER_RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_03_NAME=`echo $HEADER_RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_04_NAME=`echo $HEADER_RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_05_NAME=`echo $HEADER_RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`

echo "HEADER_RECORD: $HEADER_RECORD"
echo "FIELD_01_NAME: $FIELD_01_NAME"
echo "FIELD_02_NAME: $FIELD_02_NAME"
echo "FIELD_03_NAME: $FIELD_03_NAME"
echo "FIELD_04_NAME: $FIELD_04_NAME"
echo "FIELD_05_NAME: $FIELD_05_NAME"

cat $RESULTS_FILE|grep -v "__"|head -$EVENT_LIMIT|while read RECORD
do
        # read through results records, set values and strip off any double quotes
        # skipping every other delimiter since splunk has null values between delimiters in the csv.gz it outputs
        # ie. value1,,value2,,value3,,value4,,value5,
        echo "RECORD: $RECORD"
        # replace double commas with single commas since stats and top commands change the event format
        RECORD=`echo $RECORD | sed -e 's/,,/,/g'`
        echo "RECORD: $RECORD"
        #SEVERITY=`echo $RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
        FIELD_01_VALUE=`echo $RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
        #RESOURCE=`echo $RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
        FIELD_02_VALUE=`echo $RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
        #COMPONENT=`echo $RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
        FIELD_03_VALUE=`echo $RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
        #EVCLASS=`echo $RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
        FIELD_04_VALUE=`echo $RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
        #SUMMARY=`echo $RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`
        FIELD_05_VALUE=`echo $RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`

        case $FIELD_01_NAME in
                severity)
                        SEVERITY=$FIELD_01_VALUE
                        ;;
                resource)
                        RESOURCE=$FIELD_01_VALUE
                        ;;
                component)
                        COMPONENT=$FIELD_01_VALUE
                        ;;
                evclass)
                        EVCLASS=$FIELD_01_VALUE
                        ;;
                summary)
                        SUMMARY=$FIELD_01_VALUE
                        ;;
                *)
                        echo "ERROR: FIELD_01_NAME value of $FIELD_01_NAME not expected!"
                        ;;
        esac

        case $FIELD_02_NAME in
                severity)
                        SEVERITY=$FIELD_02_VALUE
                        ;;
                resource)
                        RESOURCE=$FIELD_02_VALUE
                        ;;
                component)
                        COMPONENT=$FIELD_02_VALUE
                        ;;
                evclass)
                        EVCLASS=$FIELD_02_VALUE
                        ;;
                summary)
                        SUMMARY=$FIELD_02_VALUE
                        ;;
                *)
                        echo "ERROR: FIELD_02_NAME value of $FIELD_02_NAME not expected!"
                        ;;
        esac

        case $FIELD_03_NAME in
                severity)
                        SEVERITY=$FIELD_03_VALUE
                        ;;
                resource)
                        RESOURCE=$FIELD_03_VALUE
                        ;;
                component)
                        COMPONENT=$FIELD_03_VALUE
                        ;;
                evclass)
                        EVCLASS=$FIELD_03_VALUE
                        ;;
                summary)
                        SUMMARY=$FIELD_03_VALUE
                        ;;
                *)
                        echo "ERROR: FIELD_03_NAME value of $FIELD_03_NAME not expected!"
                        ;;
        esac

        case $FIELD_04_NAME in
                severity)
                        SEVERITY=$FIELD_04_VALUE
                        ;;
                resource)
                        RESOURCE=$FIELD_04_VALUE
                        ;;
                component)
                        COMPONENT=$FIELD_04_VALUE
                        ;;
                evclass)
                        EVCLASS=$FIELD_04_VALUE
                        ;;
                summary)
                        SUMMARY=$FIELD_04_VALUE
                        ;;
                *)
                        echo "ERROR: FIELD_04_NAME value of $FIELD_04_NAME not expected!"
                        ;;
        esac

        case $FIELD_05_NAME in
                severity)
                        SEVERITY=$FIELD_05_VALUE
                        ;;
                resource)
                        RESOURCE=$FIELD_05_VALUE
                        ;;
                component)
                        COMPONENT=$FIELD_05_VALUE
                        ;;
                evclass)
                        EVCLASS=$FIELD_05_VALUE
                        ;;
                summary)
                        SUMMARY=$FIELD_05_VALUE
                        ;;
                *)
                        echo "ERROR: FIELD_05_NAME value of $FIELD_05_NAME not expected!"
                        ;;
        esac

        # check for null values and set to "null"

        if [ "$SEVERITY" = "" ]
        then
                SEVERITY="null"
        fi

        if [ "$RESOURCE" = "" ]
        then
                RESOURCE="null"
        fi

        if [ "$COMPONENT" = "" ]
        then
                COMPONENT="null"
        fi

        if [ "$EVCLASS" = "" ]
        then
                EVCLASS="null"
        fi

        if [ "$SUMMARY" = "" ]
        then
                SUMMARY="null"
        fi

        # generate MSG_ID
        MSG_ID_DATE=`date '+%Y-%m-%d.%H.%M.%S'`
        MSG_ID="$MSG_ID_DATE.$$"

        # append MSG_ID to summary for tracing - uncomment line below if you want a unique msg_id to appear in the zenoss summary - makes it simple to trace the message back to the Splunk logs
        #SUMMARY="$SUMMARY msg_id: $MSG_ID"

        echo "MSG_ID: $MSG_ID"
        echo "SEVERITY: $SEVERITY" 
        echo "RESOURCE: $RESOURCE" 
        echo "COMPONENT: $COMPONENT" 
        echo "EVCLASS: $EVCLASS" 
        echo "SUMMARY: $SUMMARY" 

        if [ "$EVCLASS" != "/App/Splunk" ]
        then
                echo "ERROR: EVCLASS not parsed correctly and/or not set to /App/Splunk - This will result in an invalid message being pased to Zenoss!"
        fi

        # NOTE: zenoss does not seem to like the field name resource, although the zenoss UI shows resource, so changed the field name to device before sending the message
        #ZENOSS_MESSAGE_DATA="\"severity\":\"${SEVERITY}\",\"resource\":\"${RESOURCE}\",\"component\":\"${COMPONENT}\",\"evclass\":\"${EVCLASS}\",\"summary\":\"${SUMMARY}\",\"evclasskey\":\"\""
        ZENOSS_MESSAGE_DATA="\"severity\":\"${SEVERITY}\",\"device\":\"${RESOURCE}\",\"component\":\"${COMPONENT}\",\"evclass\":\"${EVCLASS}\",\"summary\":\"${SUMMARY}\",\"evclasskey\":\"\""
        echo "ZENOSS_MESSAGE_DATA: $ZENOSS_MESSAGE_DATA"

        ZENOSS_MESSAGE="$ZENOSS_MESSAGE_BEGIN$ZENOSS_MESSAGE_DATA$ZENOSS_MESSAGE_END"
        echo "ZENOSS_MESSAGE: $ZENOSS_MESSAGE"

        #CURL_COMMAND="curl -v -k -1 -i -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
        #CURL_COMMAND="curl -k -1 -i -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
        CURL_COMMAND="curl -k -1 -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
        echo "CURL_COMMAND: $CURL_COMMAND"

        POST_COMMAND="$CURL_COMMAND $ZENOSS_MESSAGE $ZENOSS_API_URL"
        echo "POST_COMMAND: $POST_COMMAND"

        # had some trouble with the shell/curl properly passing arguments to zenoss, opted to build and run a dynamic script
        # create script 
        echo "$POST_COMMAND" > $CURL_POST_SCRIPT

        # set execute perms on script
        chmod 700 $CURL_POST_SCRIPT

        # execute script
        $CURL_POST_SCRIPT 2>&1

        # remove script after execution
        rm -f $CURL_POST_SCRIPT
done

# --------------------------------------------------------------------------------
# *** CLEANUP
# --------------------------------------------------------------------------------

# remove tmp files generated by the script
echo
echo "*** rm -rf $TMP_DIR"
rm -rf $TMP_DIR

/bin/date
echo "--------------------------------------------------------------------------------"
echo "END SCRIPT"

exit