Unbound stats graphed with Xymon

Refer to this overall guide for a master overview and below is an example hopefully others can find useful.  This is for the Unbound stats we wanted to monitor initially and it may change.

Script to collect data

On the Xymon client:

touch ~xymon/ext/unboundstats.sh && chown xymon:xymon unboundstats.sh && chmod 744 unboundstats.sh

unboundstatsh.sh:

#!/bin/sh

/usr/sbin/unbound-control stats | \
grep -E “^total.num.queries|^total.num.queries_ip_ratelimited|^total.num.cachehits|^total.num.cachemiss|^total.recursion.time.avg|num.query.tcp|^unwanted.queries|^unwanted.replies” | \
awk -F= -v OFS=: ‘{gsub(/\./, “”, $1); print}’ > /tmp/unbound.stats

#sed would remove decimal in the recursion time value
#sed s,=,:, | sed s,\\.,,g > /tmp/unbound.stats

$XYMON $XYMSRV “status $MACHINE.unboundstats green `date`

`cat /tmp/unbound.stats`
”

exit 0

You need to restart the Xymon server!  Generally this is systemctl restart xymon or /etc/init.d/xymonlaunch.cfg restart

I’m not sure if this is the right way of going about the permissions issue, but the Xymon user needs to be able to read the SSL key/pem…

chgrp xymon /etc/unbound/unbound_control.pem /etc/unbound/unbound_control.key /etc/unbound/unbound_server.pem /etc/unbound/unbound_server.key

Now make the Xymon client run the above script in ~xymon/etc/clientlaunch.cfg:

#this collects unbound stats
[unboundstats]
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
CMD $XYMONCLIENTHOME/ext/unboundstats.sh
INTERVAL 5m

On the Xymon server you will want to turn this test submission data into RRD.  In ~xymon/server/etc/xymonserver.cfg:

TEST2RRD=”cpu=la,defaultdefaultdefault,unboundstats=ncv”

SPLITNCV_unboundstats=”totalnumqueries:GAUGE,totalnumqueries_ip_ratelimited:GAUGE,totalnumcachehits:GAUGE,totalnumcachemiss:GAUGE,totalrecursiontimeavg:GAUGE,numquerytcp:GAUGE,numquerytcpout:GAUGE,unwantedqueries:GAUGE,unwantedreplies:GAUGE”

GRAPHS=”defaultdefaultdefault”

GRAPHS_unboundstats=”unboundstats,unboundstats1″ #note this requires xymon 4.3.20 https://lists.xymon.com/archive/2017-April/044560.html

After a few minutes you should see RRD files

# ls /var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,*
/var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,numquerytcpout.rrd
…

Add the graph definition (this translates from the RRD files to pretty graphs and once the file is edited/saved you can refresh the test page on Xymon/unboundstats for live viewing) in ~xymon/server/etc/graphs.cfg:

[unboundstats]
TITLE Unbound Stats
YAXIS Num
DEF:numqueries=unboundstats,totalnumqueries.rrd:lambda:AVERAGE
DEF:numqueriesiprate=unboundstats,totalnumqueries_ip_ratelimited.rrd:lambda:AVERAGE
DEF:numcachehits=unboundstats,totalnumcachehits.rrd:lambda:AVERAGE
DEF:totalnumcachemiss=unboundstats,totalnumcachemiss.rrd:lambda:AVERAGE

LINE2:numqueries#00CCCC:Total Queries
LINE2:numqueriesiprate#00FF00:Queries IP Rate
LINE2:numcachehits#FF0000:Total Cache Hits
LINE2:totalnumcachemiss#0000FF:Total Cache Miss

COMMENT:\n
GPRINT:numqueries:LAST:Total Queries \: %5.1lf%s (cur)
GPRINT:numqueries:MAX: \: %5.1lf%s (max)
GPRINT:numqueries:MIN: \: %5.1lf%s (min)
GPRINT:numqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numqueriesiprate:LAST:IP Rate Queries \: %5.1lf%s (cur)
GPRINT:numqueriesiprate:MAX: \: %5.1lf%s (max)
GPRINT:numqueriesiprate:MIN: \: %5.1lf%s (min)
GPRINT:numqueriesiprate:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numcachehits:LAST:Cache Hits\: %5.1lf%s (cur)
GPRINT:numcachehits:MAX: \: %5.1lf%s (max)
GPRINT:numcachehits:MIN: \: %5.1lf%s (min)
GPRINT:numcachehits:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:totalnumcachemiss:LAST:Cache Miss\: %5.1lf%s (cur)
GPRINT:totalnumcachemiss:MAX: \: %5.1lf%s (max)
GPRINT:totalnumcachemiss:MIN: \: %5.1lf%s (min)
GPRINT:totalnumcachemiss:AVERAGE: \: %5.1lf%s (avg)\n

[unboundstats1]
TITLE Unbound Stats
YAXIS Num
DEF:totalrecursiontimeavg=unboundstats,totalrecursiontimeavg.rrd:lambda:AVERAGE
DEF:numquerytcp=unboundstats,numquerytcp.rrd:lambda:AVERAGE
DEF:numquerytcpout=unboundstats,numquerytcpout.rrd:lambda:AVERAGE
DEF:unwantedqueries=unboundstats,unwantedqueries.rrd:lambda:AVERAGE
DEF:unwantedreplies=unboundstats,unwantedreplies.rrd:lambda:AVERAGE

LINE2:totalrecursiontimeavg#AAA1AA:Average Recusion Time
LINE2:numquerytcp#C6913B:Queries w TCP
LINE2:numquerytcpout#8BBFFF:Queries out w TCP
LINE2:unwantedqueries#FF69B4:Unwanted Queries
LINE2:unwantedreplies#FFFF00:Unwanted Replies

COMMENT:\n
GPRINT:totalrecursiontimeavg:LAST:Recursion Time Avg\: %5.1lf%s (cur)
GPRINT:totalrecursiontimeavg:MAX: \: %5.1lf%s (max)
GPRINT:totalrecursiontimeavg:MIN: \: %5.1lf%s (min)
GPRINT:totalrecursiontimeavg:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcp:LAST:TCP Queries\: %5.1lf%s (cur)
GPRINT:numquerytcp:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcp:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcp:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcpout:LAST:TCP Queries Out\: %5.1lf%s (cur)
GPRINT:numquerytcpout:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcpout:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcpout:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedqueries:LAST:Unwanted Queries\: %5.1lf%s (cur)
GPRINT:unwantedqueries:MAX: \: %5.1lf%s (max)
GPRINT:unwantedqueries:MIN: \: %5.1lf%s (min)
GPRINT:unwantedqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedreplies:LAST:Unwanted Replies\: %5.1lf%s (cur)
GPRINT:unwantedreplies:MAX: \: %5.1lf%s (max)
GPRINT:unwantedreplies:MIN: \: %5.1lf%s (min)
GPRINT:unwantedreplies:AVERAGE: \: %5.1lf%s (avg)\n