Smokeping Traceroute Alerts

Smokeping is an excellent monitoring tool used to test and alert on latency between servers. It’s an open source tool that has been around for a while.

Here at Bit Refinery, we are often tasked with helping a customer that is encountering latency or disconnects between infrastructure hosted in our environment and their locations. Sometimes the problem is with one of our many upstream providers and sometimes it’s with theirs.

We use Smokeping in addition to numerous other monitoring tools to help provide insight in the health of our network but often times when things do go bad, it’s hard to determine where the issue is. This is where Smokeping can be used to send an alert when a latency threshold is met. You can use the built in alerting or call a custom script. We wanted to share with you a very basic script we use to run a “mtr" command and email that so our NOC and have more details on the issue.

In the /etc/smokeping/config.d/Alerts script, replace the “to" line with this line to call a custom script named trace_alert.sh:

to = |/etc/smokeping/config.d/trace_alert.sh 2> /tmp/trace.log

The trace_alert.sh was put together very quickly and needs some work but it works well enough for now:

########################################################
# Script to email a mtr report on alert from Smokeping #
########################################################

alertname=$1
target=$2
losspattern=$3
rtt=$4
hostname=$5
email="monitoring@email.com"
smokename="BR-NYC-"

if [ “$losspattern" = “loss: 0%" ];
then
subject="Clear-${smokename}-Alert: $target host: ${hostname}"
else
subject="${smokename}Alert: ${target} – ${hostname}"
fi

echo “MTR Report for hostname: ${hostname}" > /tmp/mtr.txt
echo “" >> /tmp/mtr.txt
echo “sudo mtr -n –report ${hostname} "
sudo /usr/sbin/mtr -n –report ${hostname} >> /tmp/mtr.txt

echo “" >> /tmp/mtr.txt
echo “Name of Alert: " $alertname >> /tmp/mtr.txt
echo “Target: " $target >> /tmp/mtr.txt
echo “Loss Pattern: " $losspattern >> /tmp/mtr.txt
echo “RTT Pattern: " $rtt >> /tmp/mtr.txt
echo “Hostname: " $hostname >> /tmp/mtr.txt
echo “" >> /tmp/mtr.txt
echo “Full mtr command is: sudo /usr/sbin/mtr -n –report ${hostname}" >> /tmp/mtr.txt

echo “subject: " $subject
if [ -s /tmp/mtr.txt ] then
mailx -s “${subject}" $email
fi

You need mailx installed and the script needs to be owned and executable by the smokeping user. Also, in order to call sudo, you must have the following line in your /etc/sudoers:

smokeping ALL=NOPASSWD:/usr/sbin/mtr

And comment out the “Defaults requiretty" line.

That’s about it. Here is an example of what the email looks like:

MTR Report for hostname: xx.xx.xx.xx

HOST: smokeping-host Loss%   Snt   Last   Avg  Best  Wrst StDev

  1. 10.201.1.240 0.0%    10    0.3   0.3   0.2   0.4   0.0
  2. x.x.x.x 0.0%    10    0.5   3.3   0.5  11.2   3.7
  3. x.x.x.x 0.0%    10    0.4   2.8   0.4  10.9   3.6
  4. x.x.x.x          0.0%    10    1.0   1.1   1.0   1.1   0.0
  5. x.x.x.x 0.0%    10    2.0   2.0   1.9   2.3   0.1
  6. x.x.x.x 0.0%    10   26.0  26.0  25.9  26.1   0.0
  7. x.x.x.x 0.0%    10   26.0  25.9  25.8  26.0   0.1
  8. x.x.x.x 0.0%    10   24.0  24.1  24.0  24.4   0.2
  9. x.x.x.x 0.0%    10   50.6  49.5  47.7  50.9   1.2
  10. x.x.x.x 0.0%    10   47.7  48.9  47.1  50.8   1.3
  11. x.x.x.x 0.0%    10   47.6  49.0  47.1  51.0   1.4
  12. x.x.x.x 0.0%    10   49.0  49.8  48.9  50.7   0.6
  13. x.x.x.x 0.0%    10   52.9  53.2  52.3  54.4   0.8
  14. x.x.x.x 0.0%    10   57.5  57.0  55.7  58.9   0.9
  15. x.x.x.x 0.0%    10   55.2  53.7  53.3  55.2   0.6
  16. x.x.x.x 0.0%    10   51.9  52.0  51.8  52.4   0.2
  17. x.x.x.x         20.0%    10   59.4  59.9  59.4  61.0   0.7

Name of Alert:  packetloss

Target:  Smokeping Target Name

Loss Pattern:  loss: 30%

RTT Pattern:  rtt: 60ms

Hostname: x.x.x.x

Full mtr command is: sudo /usr/sbin/mtr -n –report x.x.x.x

There is still some work to do on the script but for those looking for a quick solution to add more details to latency alerts, this script really helps us out.

– Bit Refinery Team