Monitoring your services with Papertrail and Heroku

NOTE: This technical article was published long time ago. It may be not up to date. Please check for the newest versions of mentioned software. Also I am not able to provide technical support anymore. Thank you for your understanding.

Recently I have noticed some problems with my Internet connection, sometimes it was completely unavailable. I concluded that I need some precise information about the outages, so I can talk to my ISP. As at the same time I learned about the Paper Trail service, I decided to use it in my setup. Other thing, that I wanted to learn was Heroku, so I gave it a try. Both services have some free tier, which is perfectly sufficient for my needs.

The architecture of my solution is as follows: I have local server running on Raspberry Pi (as mentioned in one of the previous posts about the online radio streaming). On this server, I have a Python script, running as a cron job every 10 minutes. This script sends simple log message to my Papertrail account. Then, I have another Python script, that runs every 10 minutes as a scheduled job on Heroku. This script checks, if there are recent “heartbeat” logs on Papertrail and if not – sends an e-mail message. You can download the source code of both scripts from the github repository.

Collecting logs

To start collecting logs, you need to create free account on Papertrail. They give free 100MB for logs every month, which is way beyond what we need. After creating the account, you are provided with address and port number, which would be the destination of your log messages. Then you need any mean of sending logs to Papertrail, i.e. this symple Python script (based on the Papertrail tutorial):

# IMPORTANT NOTE: remember to set PAPERTRAIL_PORT enviroment variable
import logging, os

from logging.handlers import SysLogHandler
logger = logging.getLogger()
logger.setLevel(logging.INFO)
syslog = SysLogHandler(address=('logs.papertrailapp.com', int(os.environ['PAPERTRAIL_PORT'])))
formatter = logging.Formatter('%(name)s: %(levelname)s %(message)s')
syslog.setFormatter(formatter)
logger.addHandler(syslog)

logger.info("I'm alive!")

This simple script creates SysLogHandler, which sends logs to the Papetrail application host, with your custom port set as a environment variable (it is useful to set configuration parameters as environment variables, so you don’t have to deal with config files). To get the script working, you need to run it once every few (i.e. ten) minutes as a cron job:

PAPERTRAIL_PORT=YOUR_PORT_NUMBER
*/10 * * * * /usr/bin/python /home/pi/scripts/is-alive/send-heartbeat.py

Note, that you have to put port number assigned to you by Papertrail in the first line.

The only “log” message, that we are going to send is “I’m alive!” – the message from our local server that it’s indeed alive and connected to the internet. Now you can access them with very nice browser console, pictured below. You may want to change the name of your system (by default it is the IP number), which appears in the log.

Papertrail console

Papertrail console

Web console is convenient way to access logs, but we will need to use the Papertrail HTTP API to access them programmatically. You will find everything that you need about the API in its documentation. Simplest query, that shows all logs that are available is: https://papertrailapp.com/api/v1/events/search.json (you have to be logged in to the application to see the result).

Checking for logs

Next step is to prepare other script, that would query the API, check if there are any recent messages from our server and – if not – send an e-mail. I decided to run this script on Heroku – just to learn about this service.

# IMPORTANT NOTE: set the following environment variables:
# PAPERTRAIL_TOKEN, SMTP_HOST, SMTP_USER, SMTP_PASS
# EMAIL_FROM, EMAIL_TO

import httplib, urllib, time, os, json
INTERVAL = 10 * 60

conn = httplib.HTTPSConnection(host = 'papertrailapp.com')
conn.request(
    method = 'GET',
    url = '/api/v1/events/search.json?' + urllib.urlencode({
        'min_time' : str(int(time.time() - INTERVAL))
    }),
    headers = {
        'X-Papertrail-Token' : os.environ['PAPERTRAIL_TOKEN']
    }
)
response = conn.getresponse()

events_no = len(json.loads(response.read())['events'])

if events_no < 1:
    import smtplib
    from email.mime.text import MIMEText

    s = smtplib.SMTP(os.environ['SMTP_HOST'])
    s.login(os.environ['SMTP_USER'], os.environ['SMTP_PASS'])

    msg = MIMEText('Your host stopped calling home!\nTested at: '+time.ctime())
    msg['Subject'] = 'Your host stopped calling home!'
    msg['From'] = os.environ['EMAIL_FROM']
    msg['To'] = os.environ['EMAIL_TO']
    s.sendmail(os.environ['EMAIL_FROM'], [os.environ['EMAIL_TO']], msg.as_string())
    print "message sent"
else:
    print "OK"

This script is extremely simple too. It connects to the Papertrail HTTP API, sending query with parameter min_time with value of Unix time – INTERVAL (10 minutes). The result is JSON object with logs sent within the interval of last 10 minutes (note, that if you have more than one log source, you need to specify desired source in the query too). Then simple code is used to count number of events in the object – if there are no entries, it means that your server didn’t “call home” during the last 10 minutes. If such situation occurs – an e-mail is sent to you.

Heroku needs some magic to deploy to their platform. First, of course, you need to create free account. Then, you need to “verify” your account by providing your credit card number (don’t worry, they will not charge you anything). Then, you need to install Heroku Toolbelt – set of scripts that allows you to deploy your app to the Heroku platform. Under Debian/Ubuntu it is as easy as typing:

wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh

(of course, be aware, that this will run external code on your computer, so you may want to check, what’s inside) Then, you need to create Heroku specific files: requirements.txt and Procfile. Both of them should be empty, because our application doesn’t have any external dependencies, and it would run as scheduled script. (Anyway, you should read more about getting started with Python on Heroku here.)

Heroku needs your scripts to be inside git repository, so prepare it:

git init
git add .
git commit -m "initial commit"

To push your application to the Heroku, you would need to login to Heroku and create your application first. Do it by:

heroku login
heroku create
git push heroku master

To configure the application, you need to provide environment variables to it. You can do it by issuing heroku command:

heroku config:add PAPERTRAIL_TOKEN=... SMTP_HOST=... SMTP_USER=... SMTP_PASS=... EMAIL_FROM=... EMAIL_TO=...

As we want to run our app as a scheduled task (not web or worker), we need to configure scheduler. To do so, first add this add-on to your app:

heroku addons:add scheduler

Then, in Heroku dashboard click on your app and then on scheduler addon. There, add new job, running “python query.py” every 10 minutes.

That’s it 🙂 You can find further information on the Getting started with Python on Heroku tutorial page.

Homework

As a homework, you might want to add more features to your scripts. My propositions are:

  1. Send the warning e-mail only once, after the script notices, that the connection was lost.
  2. Send another e-mail, when the connection is restored.
  3. Monitor other factors, i.e. computers connected to the local network, etc.

UPDATE: As Troy Davis from Papertrail pointed me out, not always the e-mail notification has to be made by calling the HTTP API by external application. My solution bases on the looking for no event – however, Papertrail’s Alerts can be configured to notify you by an e-mail,  when specific event occurs (i.e. log message with matching text).