README_BB.md

readme_bb.md

For Big Brother, unfortunately we do not. Not the greatest tool. So for the OI monitoring content, I wrote a few Python scripts that grepped logs, did some comparison of strings, etc. Then I called those from a BASH script. The BASH script I had three strings, YELLOWLINES, GREENLINES, and REDLINES. So with the OI checking arrival time of products, I compared the ldmd.log file content to the nwwsoi.log file content grepping the individual product strings. Then the output of the app said how many products over 15 minutes and the average time to process. I have in the BASH, if the average time to process is > 20 seconds, go red, >10 go yellow, otherwise green.

Then at the bottom of the BASH SCRIPT, I have ################################################################################################################################

Final Message Section

################################################################################################################################

Calculate color

BB_MESSAGE=""

if [[ $REDLINES != "" ]]; then color="red" BB_MESSAGE="Error in NWWS OI Products Processing" elif [[ $YELLOWLINES != "" ]]; then color="yellow" BB_MESSAGE="NWWS OI Products Not Processing Cleanly" elif [[ $GREENLINES != "" ]]; then color="green" BB_MESSAGE="NWWS OI Products Functioning Normally." fi

$BB $BBDISP "status $MACHINE.$BBALERT_NAME $color date $BB_MESSAGE ${REDLINES}${YELLOWLINES}${GREENLINES}"

That sends the content over to Big Brother based on the strings I appended to either REDLINES, YELLOWLINES, or GREENLINES.

Here is a piece from earlier in that BASH script ###############################################################################################################################

LDM Processing Section

##############################################################################################################################

check that LDM is running

RET=/opt/ldm/bin/ldmadmin isrunning; echo $?

if [ $RET -ne 0 ] then
REDLINES="${REDLINES}
&red ldm is not running, contact onboarding Team" else GREENLINES="${GREENLINES} &green LDM is running" fi

That is where I check if LDM is running. The echo $? will echo 0 if it is. If it isn't then anything else

So I just append a new line and &red if it isn't working or newline and &green if it is green.

So I was thinking with Spot we could do something similar with a CURL response, maybe parse the JSON and then just update the strings with the details from the JSON output. 12:54 Send that into the redlines/greenlines/yellowlines 12:55 If you want to see the Big Brother stuff for OI,

ssh vm-bldr-nwwsoi-dev-processing.ncep.noaa.gov

sudo -u bbuser -s /bin/bash

cd ~/bb ; export BBHOME=pwd ; . ./etc/bbdef.sh

cd ext

(edited) 12:56 And the scripts are in there. I call the product lines python script and the ldm status script from the bb script 12:56 And then I put the BASH script into the Big Brother exttab configuration [bbuser@vm-bldr-nwwsoi-dev-processing ext]$ cat ../etc/bb-bbexttab

Ansible managed

vm-bldr-nwwsoi-dev-processing.ncep.noaa.gov : : bb-nwwsoiproc.sh 12:56 And we get monitoring 1:01 So the connect line is actually the $BB $BBDISP "status $MACHINE.$BBALERT_NAME $color date $BB_MESSAGE ${REDLINES}${YELLOWLINES}${GREENLINES}" 1:02 Which pulls those values from the environment variables in Big Brother. Which you can simulate with the cd ~/bb ; export BBHOME=pwd ; . ./etc/bbdef.sh line 1:02 Except for the REDLINES, YELLOWLINES,GREENLINES which is what I put in