Blog of iam8up

Updating Centos 7.9 firmware for Raspbery Pi 4 Model B

I purchased a Raspberry Pi 4 Model B 2019 Quad Core 64 Bit WiFi Bluetooth (4GB). Every other Pi I use in production uses Centos 7 so of course I go to load it on a microSD and boot. I’m then halted with progress by this error:

start4.elf is not compatible this board requires newer software

The fix is very simple. Download this archive:
*http://bay.uchicago.edu/centos-altarch/7/isos/aarch64/images/CentOS-Userland-7-aarch64-RaspberryPI-Minimal-4-2009-sda.raw.xz
and then extract the raw image file.

Then run upd-image-fw from:
*https://forums.raspberrypi.com/viewtopic.php?p=1714655#p1714655
*(direct link to program) https://forums.raspberrypi.com/download/file.php?id=41880&sid=0a370ba73c65275426888fdfcf0b01f0

and then simply run:

sudo ./upd-image-fw CentOS-Userland-7-aarch64-RaspberryPI-Minimal-4-2009-sda.raw

I like to use balenaEtcher to write the image to SD.

Unbound stats graphed with Xymon

Refer to this overall guide for a master overview and below is an example hopefully others can find useful. This is for the Unbound stats we wanted to monitor initially and it may change.

Script to collect data

On the Xymon client:

touch ~xymon/ext/unboundstats.sh && chown xymon:xymon unboundstats.sh && chmod 744 unboundstats.sh

unboundstatsh.sh:

#!/bin/sh

/usr/sbin/unbound-control stats | \
grep -E “^total.num.queries|^total.num.queries_ip_ratelimited|^total.num.cachehits|^total.num.cachemiss|^total.recursion.time.avg|num.query.tcp|^unwanted.queries|^unwanted.replies” | \
awk -F= -v OFS=: ‘{gsub(/\./, “”, $1); print}’ > /tmp/unbound.stats

#sed would remove decimal in the recursion time value
#sed s,=,:, | sed s,\\.,,g > /tmp/unbound.stats

$XYMON $XYMSRV “status $MACHINE.unboundstats green `date`

`cat /tmp/unbound.stats`
”

exit 0

You need to restart the Xymon server! Generally this is systemctl restart xymon or /etc/init.d/xymonlaunch.cfg restart

I’m not sure if this is the right way of going about the permissions issue, but the Xymon user needs to be able to read the SSL key/pem…

chgrp xymon /etc/unbound/unbound_control.pem /etc/unbound/unbound_control.key /etc/unbound/unbound_server.pem /etc/unbound/unbound_server.key

Now make the Xymon client run the above script in ~xymon/etc/clientlaunch.cfg:

#this collects unbound stats
[unboundstats]
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
CMD $XYMONCLIENTHOME/ext/unboundstats.sh
INTERVAL 5m

On the Xymon server you will want to turn this test submission data into RRD. In ~xymon/server/etc/xymonserver.cfg:

TEST2RRD=”cpu=la,defaultdefaultdefault,unboundstats=ncv”

SPLITNCV_unboundstats=”totalnumqueries:GAUGE,totalnumqueries_ip_ratelimited:GAUGE,totalnumcachehits:GAUGE,totalnumcachemiss:GAUGE,totalrecursiontimeavg:GAUGE,numquerytcp:GAUGE,numquerytcpout:GAUGE,unwantedqueries:GAUGE,unwantedreplies:GAUGE”

GRAPHS=”defaultdefaultdefault”

GRAPHS_unboundstats=”unboundstats,unboundstats1″ #note this requires xymon 4.3.20 https://lists.xymon.com/archive/2017-April/044560.html

After a few minutes you should see RRD files

# ls /var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,*
/var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,numquerytcpout.rrd
…

Add the graph definition (this translates from the RRD files to pretty graphs and once the file is edited/saved you can refresh the test page on Xymon/unboundstats for live viewing) in ~xymon/server/etc/graphs.cfg:

[unboundstats]
TITLE Unbound Stats
YAXIS Num
DEF:numqueries=unboundstats,totalnumqueries.rrd:lambda:AVERAGE
DEF:numqueriesiprate=unboundstats,totalnumqueries_ip_ratelimited.rrd:lambda:AVERAGE
DEF:numcachehits=unboundstats,totalnumcachehits.rrd:lambda:AVERAGE
DEF:totalnumcachemiss=unboundstats,totalnumcachemiss.rrd:lambda:AVERAGE

LINE2:numqueries#00CCCC:Total Queries
LINE2:numqueriesiprate#00FF00:Queries IP Rate
LINE2:numcachehits#FF0000:Total Cache Hits
LINE2:totalnumcachemiss#0000FF:Total Cache Miss

COMMENT:\n
GPRINT:numqueries:LAST:Total Queries \: %5.1lf%s (cur)
GPRINT:numqueries:MAX: \: %5.1lf%s (max)
GPRINT:numqueries:MIN: \: %5.1lf%s (min)
GPRINT:numqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numqueriesiprate:LAST:IP Rate Queries \: %5.1lf%s (cur)
GPRINT:numqueriesiprate:MAX: \: %5.1lf%s (max)
GPRINT:numqueriesiprate:MIN: \: %5.1lf%s (min)
GPRINT:numqueriesiprate:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numcachehits:LAST:Cache Hits\: %5.1lf%s (cur)
GPRINT:numcachehits:MAX: \: %5.1lf%s (max)
GPRINT:numcachehits:MIN: \: %5.1lf%s (min)
GPRINT:numcachehits:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:totalnumcachemiss:LAST:Cache Miss\: %5.1lf%s (cur)
GPRINT:totalnumcachemiss:MAX: \: %5.1lf%s (max)
GPRINT:totalnumcachemiss:MIN: \: %5.1lf%s (min)
GPRINT:totalnumcachemiss:AVERAGE: \: %5.1lf%s (avg)\n

[unboundstats1]
TITLE Unbound Stats
YAXIS Num
DEF:totalrecursiontimeavg=unboundstats,totalrecursiontimeavg.rrd:lambda:AVERAGE
DEF:numquerytcp=unboundstats,numquerytcp.rrd:lambda:AVERAGE
DEF:numquerytcpout=unboundstats,numquerytcpout.rrd:lambda:AVERAGE
DEF:unwantedqueries=unboundstats,unwantedqueries.rrd:lambda:AVERAGE
DEF:unwantedreplies=unboundstats,unwantedreplies.rrd:lambda:AVERAGE

LINE2:totalrecursiontimeavg#AAA1AA:Average Recusion Time
LINE2:numquerytcp#C6913B:Queries w TCP
LINE2:numquerytcpout#8BBFFF:Queries out w TCP
LINE2:unwantedqueries#FF69B4:Unwanted Queries
LINE2:unwantedreplies#FFFF00:Unwanted Replies

COMMENT:\n
GPRINT:totalrecursiontimeavg:LAST:Recursion Time Avg\: %5.1lf%s (cur)
GPRINT:totalrecursiontimeavg:MAX: \: %5.1lf%s (max)
GPRINT:totalrecursiontimeavg:MIN: \: %5.1lf%s (min)
GPRINT:totalrecursiontimeavg:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcp:LAST:TCP Queries\: %5.1lf%s (cur)
GPRINT:numquerytcp:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcp:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcp:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcpout:LAST:TCP Queries Out\: %5.1lf%s (cur)
GPRINT:numquerytcpout:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcpout:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcpout:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedqueries:LAST:Unwanted Queries\: %5.1lf%s (cur)
GPRINT:unwantedqueries:MAX: \: %5.1lf%s (max)
GPRINT:unwantedqueries:MIN: \: %5.1lf%s (min)
GPRINT:unwantedqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedreplies:LAST:Unwanted Replies\: %5.1lf%s (cur)
GPRINT:unwantedreplies:MAX: \: %5.1lf%s (max)
GPRINT:unwantedreplies:MIN: \: %5.1lf%s (min)
GPRINT:unwantedreplies:AVERAGE: \: %5.1lf%s (avg)\n

Extract one column from a CSV

Assuming your CSV looks like this

“ID”,”DeviceName”,”IPAddress”,”LastStatusChange”
“211”,”customer name here”,”172.16.10.19″,”2018-08-1309:51:28″
“339”,”fue bah”,”10.13.0.1″,”2018-09-0619:28:30″

We can use Notepadd++ (or other tool) and some regex to extract the IP address.

Find what: ^(.*?),(.*?),(.*?),(.*?)$
Replace with: $3

Be sure to use regex if you’re on Windows/Notepad++! This works with four “column” documents, to adjust it for more or less be sure to mirror the number of commas and use one – (.*?) – for each column.

CentOS 6 with Python 2.7

https://danieleriksson.net/2017/02/08/how-to-install-latest-python-on-centos/

# Start by making sure your system is up-to-date:
yum update
# Compilers and related tools:
yum groupinstall-y”development tools”
# Libraries needed during compilation to enable all features of Python:
yum install-yzlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel
# If you are on a clean “minimal” install of CentOS you also need the wget tool:
yum install-ywget

wget http://python.org/ftp/python/2.7.14/Python-2.7.14.tar.xz
tar xf Python-2.7.14.tar.xz
cdPython-2.7.14
./configure–prefix=/usr/local–enable-unicode=ucs4–enable-shared LDFLAGS=”-Wl,-rpath /usr/local/lib”
make&&makealtinstall

wget https://bootstrap.pypa.io/get-pip.py

# Then execute it using Python 2.7 and/or Python 3.6:
python2.7 get-pip.py

# With pip installed you can now do things like this:
pip2.7 install [packagename]
pip2.7 install –upgrade [packagename]
pip2.7 uninstall [packagename]

Clean Config Output

One of the best alias ideas I've ever seen!  This strips all the comments and such out of a config file so you can see, generally, what it looks like to the program reading it.

# alias
alias cleanconfig='sed -e '\''s/#.*//'\'' -e '\''s/[ ^I]*$//'\'' -e '\''/^$/ d'\'''

# cleanconfig /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery

RRDTool dump and restore

This issue typically comes up when moving data from a 32bit to a 64bit system. The commands are very easy to work with for one file, but with many files and multiple directories (subdirectories) it can be cumbersome. I found this post in a forum and found it extremely useful and powerful.

SRC: http://www.linuxquestions.org/questions/programming-9/script-to-dump-and-restore-rrd-files-608407/

This should be a foo.sh file and you should run it in /usr/share/app/data where the files are within data or below that. It will convert and rename the file from rrd to xml. Transfer the data over to the new system and run the same command (either no argument or look at the bottom for ‘help’).

#!/bin/sh

ECHO="echo -e"

rrdump ()
{
	for rrd in `find . -type f -name "*.rrd"`
		do
			xml=`echo $rrd | sed 's/.rrd//g'`
			rrdtool dump $rrd > $xml.xml
			rm $rrd
		done
}	

xmlimport ()
{
	for xml in `find . -type f -name "*.xml"`
        do
                rrd=`echo $xml | sed 's/.xml//g'`
                rrdtool restore $xml $rrd.rrd
                rm $xml
        done
}


case "$1" in
        dump)
                rrdump
		;;
        import)
                xmlimport
                ;;
        *)
                $ECHO "$0 takes one of two arguments\n"
		$ECHO "$0 dump -- dumps contents to xml\n"
		$ECHO "$0 import -- imports xml files to rrd\n"
                ;;
esac

Network throughput on Linux seen on CLI

There are many options – http://www.binarytides.com/linux-commands-monitor-network/

nload is probably the simplest and what most people want though iftop is pretty cool.

Onboarding with SNMP via BASH

With the release of cnMaestro you can now ditch the CNS server *AFTER* you’ve upgraded your radios to 2.5+ The text files ap.txt and cpe.txt are simply one IP per line. I got these from my network management software (a simple SQL query for all Cambium ePMP AP and Cambium ePMP CPE). This was done in PC with:

SELECT e.DeviceType,
INET_NTOA(e.IPAddress + a.StartAddress) AS 'IP Address' FROM Equipment e
INNER JOIN Customer c ON c.CustomerID=e.EndUserID
LEFT OUTER JOIN AddressRange a ON e.IPType = a.AddressRangeID
WHERE e.Type LIKE '%Access Point%'
AND e.DeviceType LIKE '%Cambium ePMP AP%'
GROUP BY e.ID
ORDER BY c.CustomerID;

This simply requires SNMP access and snmputils to work.

#!/bin/bash

community=CHANGETHISCOMMUNITYHERE
server=https://cloud.cambiumnetworks.com
cambiumid=URCMPNYIDGOESHERE
onboardkey=048j03j03jf093jf0j309jf03jf

#use this to identify your radio versions (it needs 2.5+)
#while read -r radioip; do
# echo “Querying” $radioip
# snmpget -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.1.1.1.0
#done < “ap.txt”

#this is how you enable, set host/cambiumid/onboardkey, apply
while read -r radioip; do
echo “Querying” $radioip
snmpset -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.3.20.1.0 i 1
snmpset -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.3.20.2.0 s $server
snmpset -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.3.20.3.0 s $cambiumid
snmpset -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.3.20.4.0 s $onboardkey
snmpset -v2c -c$community $radioip .1.3.6.1.4.1.17713.21.4.4.0 i 1
done < “ap.txt” >> onboard.log

Migrating a Linux machine

In my case I needed to move a machine from a headless box to an ESXi server. The VMWare tool does not support software raid.

I started by installing the OS on the new box with the same partition layout. This takes care of all of the mbr and boot stuff. The critical part, though, is / with the configuration, installed programs and any amount of data. I used nc to do this and did it very painlessly.

#server you’re sending files to
nc -l 1024 > backup4.tgz

#client you’re backing up
tar -cvpz –one-file-system / | nc neon.imaginenetworksllc.com 1024

#boot the new box aka new client with a linux live cd and mount the partition
mkdir /mnt/sda3
mount /dev/sda3 /mnt/sda3
nc -l 1024 | tar -xvpzf – -C /mnt/sda3

#on that backup server
cat backup4.tgz | nc NEW.BOX.COM 1024

In my case I just installed the OS on the box so it had grub and the boot partition all done. What makes this box is the files in /.
Moving from ext3 partitions on md0 devices directly to sda1 I have to tell it to mount correctly.
*/etc/fstab
*eth0.bak has the ip config

*nload was nice to monitor network usage

*lots of help from https://help.ubuntu.com/community/BackupYourSystem/TAR

Visualizing disk usage in Linux – CLI/command line

http://gt5.sourceforge.net/

This project is simply amazing. The download includes the binary and other files. Simply execute gt5 from where it is extracted or more wisely place it in /bin/gt5 so you can run it from anywhere. You can specify any directory (ie: gt5 /).

*stores previously ran information and will show you the different (in bytes)
*very nice easy interface
*largest (percentage) at the top to narrow down where the disk usage is
*runs on nearly any system with included utilities (du for example)

Output is done in HTML and easily shown after parsing with the included tool, again with the simple execution of gt5 /

It took a bit of digging to find a good program that would work directly in the shell for headless machines. This one is simply fantastic.

EDIT:
gt5 is still pretty good but it’s not as easy/fast to get installed. Recently I’ve been using ncdu thanks to this question/answer. There are CentOS and Debian packages in their repos. ncdu homepage.