Here’s a really fascinating TED talk where Erez Lieberman Aiden and Jean-Baptiste Michel explain how they’ve used Google’s book digitization effort to identify cultural trends:

What’s really interesting is that their work resulted in the Google Labs tool Books Ngram Viewer which is like Google Trends, only for books, and dating back to year 1500. For instance, it looks like things aren’t as wonderful and glorious as they once were:

I’ll send a Spotify invitation to the first 50 impatient Americans who send an email to spotify-invite at this domain, or write a comment using an account with a valid email address :-)

Update: No invitations left. Thanks for playing.

I recently stumbled upon F.lux, an application for Windows, Mac OS X and Linux that changes the color temperature of your display depending on the time of day. Although changing your color temperature is nothing new, F.lux makes it completely seamless: Just set your location and your desired color temperatures (optional), then forget about it.

From the author:

Ever notice how people texting at night have that eerie blue glow? Or wake up ready to write down the Next Great Idea, and get blinded by your computer screen? During the day, computer screens look good—they’re designed to look like the sun. But, at 9PM, 10PM, or 3AM, you probably shouldn’t be looking at the sun.

F.lux fixes this: it makes the color of your computer’s display adapt to the time of day, warm at night and like sunlight during the day. It’s even possible that you’re staying up too late because of your computer. You could use f.lux because it makes you sleep better, or you could just use it just because it makes your computer look better.

Get it here.

EmailsFromFile is a Python script that finds all email addresses in a file. It shows one email address per line by default, but you can pick your own separator (i.e. a comma or space) if you want.

The script is very simple; the magic lies in the email regular expression (based on the RFC 2822 standard) which is, quite frankly, one of the craziest I’ve seen:

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

You can find more information about email regex here.

I just finished looking at some interesting slides about the Computational Complexity of Air Travel Planning by Carl de Marcken, when I happened upon the product portfolio of his company. What I found was Matrix 2, an advanced flight planner. It’s the best flight planner I’ve ever used.

Here are some screenshots to better illustrate what I’m talking about:

Step 1: Search

Step 2: Price Calendar

Step 3: Search Results and Itineraries

Voila!

You can’t book flights through Matrix 2, but it’s an extremely useful tool nonetheless. It’s rather advanced, too, if you need it to be:

Extra Search Options

Check it out for yourself, here.

Readability is a bookmarklet that sanitizes web pages not designed with readability in mind, like the vast majority of blogs. I can’t remember how many times I’ve caught myself reading eye-fraggingly small text on a site with a left-oriented, 10% width content column, and wondered why I need such a deliciously large monitor.

Go to the website, customize your design, drag the bookmarklet to your bookmarks bar, then go to a non-aesthetic web page and click the bookmark — voilà! Hopefully, you should feel no need to use the bookmarklet on this site. If you do, though, let me know why.

Check out the Arc90 Lab for more stuff from the authors of Readability; they make some pretty cool stuff. I especially like HashMask and HalfMask.

outSSIDer is the lonely little application that’s there for you when you drop your phone in the big city and you really need to twitter about it. Get the laptop, open outSSIDer, close the lid, then start walking around. When it makes a successful connection to a wireless network, it’ll make a ding sound. If it is then successful in retrieving a file (in this case, Google’s favicon) from the great internet, it’ll start dinging frantically. If not, well, it’s the hi-hat, of course.

Download it here.

I wasn’t joking about closing the laptop lid, by the way — you stand out a little when you walk around looking at an open laptop making cymbal noises.

outSSIDer is based on inSSIDer, a Wi-Fi diagnostics tool.

Staying ahead of IT service issues can be frustrating when you manage several servers, or even a single server with many services. Enterprise IT Infrastructure Monitoring Solutions (a fancy term for something that is really pretty simple) attempt to remedy the problem by repeatedly checking the status of machines and services on the network and alerting the responsible administrators as soon as something goes wrong, or even before there’s a problem.

It’s hard to argue against implementing a monitoring solution within the network, as it is much a setup-and-forget matter that adds negligible load. The monitoring solution itself is — or at least, should be — very low maintenance, yet provides very valuable insight into the health of the network.

Introducing Nagios

Nagios is an infrastructure monitoring solution that is both popular and open source. Apart from its obvious monitoring capabilities, it includes the ability to associate an event handler to an event, allowing you to fix a problem automatically. If — for example — one of your Python applications crashes, you can have Nagios do python /opt/myapp/myapp.py automatically, before any human administrators have the time to do so. Other features include the ability to create many kinds of reports, and to send notifications and alerts via email and SMS.

Nagios' web interface

Nagios is based primarily on C and shell scripts, which makes it light on performance but adds a slightly ‘hackish’ feel. It comes with a CGI-based web interface (which we’ll spice up a bit) that lets you view and manage Nagios, through what are known as External Commands.

I’d like to demonstrate how to set up rudimentary Nagios monitoring on a small farm of Linux servers, with an Ubuntu/Debian server running the primary Nagios process. In the end, we’ll be monitoring the states of various services on the servers, including the ones seen in the screenshot above (Apache processes, APT, Current Load, Current Users, Disk Space, Dovecot, FTP, HTTP, MySQL, SMTP, SSH, Swap, Total Processes, and Zombie Processes). We will also receive notifications by email whenever something goes wrong:

Please note that this guide is meant to get you up and running quickly, and that it’s not a substitute for the official Nagios documentation. If you want to know what all of the different configuration options do (or can do), please consult the (excellent) documentation.

Setting Up The Nagios Server

The steps in this section should just be done on the main Nagios server, not the clients it will be monitoring. We’ll get to those later!

This procedure should be quite similar on other distributions if you use their package managers (yum, yast, urpmi, etc.) or install Nagios from source, but no guarantees.

1. Let’s become root so we don’t have to prepend sudo to everything:

   sudo -s

2. If you want to make use of Nagios’ web interface and Apache isn’t already installed:

   apt-get install apache2

   It’s entirely possible to use something like nginx or lighttpd to serve the interface, but that is not covered in this guide.

3. Install Nagios from the package repositories:

   apt-get install nagios3 nagios-nrpe-plugin

Nagios should be accessible at http://nameofnagiosserver/nagios3 already! We still have some configuration to do, though.

4. Stop Nagios:

   /etc/init.d/nagios3 stop

5. Add a new user for the web interface, e.g. patrick. The default configuration grants all security permissions to the user nagiosadmin, but we’ll change that to the name of the new user, too:

   htpasswd -c /etc/nagios3/htpasswd.users patrick
   perl -p -i -e "s/nagiosadmin/patrick/g" /etc/nagios3/cgi.cfg

The perl command above replaces all occurrences of nagiosadmin with patrick in the file /etc/nagios3/cgi.cfg.

The users listed in /etc/nagios3/cgi.cfg are effectively global administrators. For regular users, you can still add them as users with htpasswd, but assign privileges by making them Contacts for certain hosts or hostgroups, instead. We’ll get to this later!

6. If you want to add more user accounts for the web interface:

   htpasswd /etc/nagios3/htpasswd.users john

7. And if you want to give them superuser privileges:

   perl -p -i -e "s/patrick/patrick, john/g" /etc/nagios3/cgi.cfg

   Go through /etc/nagios3/cgi.cfg manually to see what the different security options do, and to grant more fine-grained privileges to other administrators.

8. Edit /etc/nagios3/nagios.cfg and change check_external_commands=0 to 1 to allow monitoring commands to be issued through the web interface
9. On Debian/Ubuntu, run the following commands after setting check_external_commands=1:

   dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
   dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3

10. Edit /etc/nagios3/conf.d/contacts_nagios2.cfg to match your preferences. Example:

    define contact{
            contact_name                    patrick
            alias                           Patrick Mylund
            service_notification_period     24x7
            host_notification_period        24x7
            service_notification_options    w,u,c,r
            host_notification_options       d,r
            service_notification_commands   notify-service-by-email
            host_notification_commands      notify-host-by-email
            email                           my@emailaddress.com
            }

    And further down:

    define contactgroup{
            contactgroup_name       admins
            alias                   Nagios Administrators
            members                 patrick
            }

11. Make a host definition for a server you want to monitor by creating a matching config file, e.g for the server ‘tranquillity’, nano -w /etc/nagios3/conf.d/tranquillity_nagios2.cfg, then insert a declaration. Example:

    define host{
            use                     generic-host            ; Name of host template to use
            host_name               tranquillity
            alias                   PatrickMylund.com Web Server
            address                 209.20.82.6
            }

    You can put all of your host definitions in one file if you want, e.g. datacenter1_nagios2.cfg — just remember the _nagios2.cfg at the end of the file name, which is what tells Nagios to load that file (and in the proper format).

12. Repeat the step above to add a host definition for each server you want to monitor
13. Move some standard configs to make room for our configured ones:

    mv /etc/nagios3/conf.d/localhost_nagios2.cfg /etc/nagios3/conf.d/localhost_nagios2.cfg.old
    mv /etc/nagios3/conf.d/services_nagios2.cfg /etc/nagios3/conf.d/services_nagios2.cfg.old
    wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/services_nagios2.cfg -O /etc/nagios3/conf.d/services_nagios2.cfg

14. Edit /etc/nagios3/conf.d/hostgroups_nagios2.cfg. List which hosts (comma-separated) should belong to which groups (debian-servers, http-servers, ssh-servers, and ping-servers), and add some extra hostgroups: db-server, ftp-servers, and mail-servers:

    define hostgroup {
            hostgroup_name  db-servers
                    alias           Database servers
                    members         tranquillity, singularity
            }
     
    define hostgroup {
            hostgroup_name  ftp-servers
                    alias           FTP servers
                    members         tranquillity, singularity
            }
     
    define hostgroup {
            hostgroup_name  mail-servers
                    alias           IMAPS/SMTP servers
                    members         tranquillity
            }

    You can see which services are associated with which hostgroups by looking in /etc/nagios3/conf.d/services_nagios2.cfg.

We’re done with the Nagios server for now. Let’s look at the settings for the Linux servers we want to monitor.

Configuring Monitored Clients

The steps in this section should be done on each Linux host that you want to monitor.

1. Again, let’s become root:

   sudo -s

2. Install Nagios’ NRPE module:

   apt-get install nagios-nrpe-server

   Installing the NRPE module is optional, but you won’t be able to run any of Nagios’ scripts directly on the target client if you do not. This is necessary for monitoring system stats, and generally anything that cannot be probed from the outside over the network (by the main Nagios server).

   See the NRPE documentation (PDF) for manual installation instructions, as well as how to get information via SSH (get_by_ssh) instead of NRPE.

3. Stop NRPE:

   /etc/init.d/nagios-nrpe-server stop

4. Install a custom nrpe_local.cfg (this will save us some time later):

   mv /etc/nagios/nrpe_local.cfg /etc/nagios/nrpe_local.cfg.old
   wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/nrpe_local.cfg -O /etc/nagios/nrpe_local.cfg

   Go through /etc/nagios/nrpe_local.cfg to see the list of commands that Nagios will be able to execute on hosts running NRPE. By default, NRPE will only run the commands defined in this configuration file, and without any arbitrary arguments. I strongly recommend you stick to this for security purposes.

   On the main Nagios server, all service commands prefixed with check_nrpe_1arg in /etc/nagios3/services_nagios2.cfg are commands defined in /etc/nagios/nrpe_local.cfg on the monitored clients.

5. Define what hosts are going to be allowed to probe the NRPE module for information (comma-separated). For instance, if the main Nagios server has IP 192.168.1.105:

   perl -p -i -e "s/127.0.0.1/192.168.1.105/g" /etc/nagios/nrpe_local.cfg

6. If you have a firewall (iptables, ufw, etc.), you need to open for connections on port 5666 on the clients (for NRPE). If the main Nagios server has IP 192.168.1.105, you could do ufw allow proto tcp from 192.168.1.105 to any port 5666, or ufw allow 5666/tcp with Ubuntu’s Uncomplicated Firewall.
7. Start the NRPE module:

   /etc/init.d/nagios-nrpe-server start

We just about have a basic Nagios setup now!

Testing Nagios

Let’s see if what we’ve set up is working. On the main Nagios server, start the Nagios service:

/etc/init.d/nagios3 start

If all goes well, navigate to e.g. http://192.168.1.105, login with the user credentials you set up earlier, then click on Service Detail in the menu on the left. All of our services will be PENDING, meaning they’ll be checked shortly. You can speed this up by clicking on a service and clicking Re-schedule the next check of this service (this is what is called an External Command).

If any of the service states turn out to be CRITICAL or UNKNOWN, don’t panic — take a look at the different configuration files in /etc/nagios3/conf.d. The settings and commands are pretty straight-forward.

You can find examples of the resulting configuration files in nagios-conf-example.tar.gz. The configs are for a single server (singularity) with the IP address 192.168.2.3.

An Extra Touch

Nagios’ web interface doesn’t look very pretty. We can spice it up a little by changing the CSS. I’ve prepared a modified status.css for your convenience:

mv /etc/nagios3/stylesheets/status.css /etc/nagios3/stylesheets/status.css.old
wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/status.css -O /etc/nagios3/stylesheets/status.css

Now hit F5 in the web interface!

Bear In Mind

• The easiest way to monitor the Nagios server itself is to pretend it’s yet another server. Install NRPE, set the connection settings, and add it in the host declarations with the other servers.
• The exclamation mark (!) is meant to separate command arguments in Nagios configuration files. For instance, check_nrpe_1arg!check_swap would mean you’re running check_nrpe_1arg with the argument check_swap.
• All of the scripts and commands you can issue through Nagios are stand-alone scripts. When configuring Nagios, you can run each command, for instance check_smtp, manually instead of doing tons of trial-and-error with the configuration files:

  /usr/lib/nagios/plugins/check_smtp -H 192.168.1.105
  /usr/lib/nagios/plugins/check_smtp -h

• All lists in Nagios configuration files are comma-separated.
• You can set the contact_groups value on any service, host, or hostgroup declaration. Contact groups are defined in /etc/nagios3/conf.d/contacts_nagios2.cfg. Any person in a contact group that has a user account for the web interface (htpasswd.users) can automatically view any hosts and services associated with it.

  Example:

  define hostgroup {
          hostgroup_name  mail-servers
                  alias           IMAPS/SMTP servers
                  members         singularity
                  contact_groups  mailadmins
          }

Again, the best part about what we’ve set up now is that you can go right ahead and forget about it. You’ll receive an e-mail at the contact address specified whenever something is amiss, as well as when it gets better. If I’m right, though, you’ll want to tune your configuration a lot further. We’ve barely touched the surface; Nagios can do much more, and everything is thoroughly documented in the official documentation.

Other useful links:

• Official Nagios FAQ
• Official Nagios Wiki
• MonitoringExchange Nagios Plugins, Guides, Artwork, and more
• NagiosForge
• Another Nagios Wiki

Blind Search is one of my favorite new toys. It lets you search for something on Google, Bing and Yahoo simultaneously, and lines up the results in three columns. The catch? It doesn’t tell you which search engine each column of search results comes from — rather, it asks you which results you think are the best. I’m pretty sure the results are going to surprise you.

One thing is for sure; whenever I turn out voting for Bing, they’ll gain significantly more of my respect than when they tried to turn me over with a $1.15 meal and a ten grand hidden treasure, especially since the guy behind Blind Search works for Microsoft. It’s too bad that the site isn’t officially endorsed by the company — what better way to win over users and market your product than to empirically demonstrate its prowess?

FilesByWeek is a small script that counts the number of files in a folder that were created (or last modified) in X week of the year. It’s designed for use with Linux/Postfix Maildirs (and thus excludes the standard .Sent folder and any Dovecot/Courier IMAP files from the find query), but should work just fine on any kind of directory.

Get the latest version here. Or, if you just want to see what the fuzz is about:

#!/bin/bash
typeset -i YEAR WEEK COUNT
 
WEEK=$1
# Comment out the following line if the year starts on a Monday
WEEK=$((WEEK-1))
# Use current system year by default. This can be changed to e.g.: YEAR=2008
YEAR=`date +%Y`
TGTDIR=$2
COUNT=0
 
\find ${TGTDIR} \
-type d \( -name "*.sent" -o -name "*.Sent" -o -name "courierimapkeywords" -o -name "courierimaphieracl" \) -prune -o \
-type f \( ! -name "subscriptions" ! -name "courierimapsubscribed" ! -name "dovecot.index.log*" ! -name "dovecot.index" ! -name "maildirfolder" ! -name "dovecot-keywords" ! -name "dovecot.index.cache" ! -name "courierimapacl" ! -name "courierimapuiddb" ! -name "dovecot-uidlist" \) \
-print |
{
    while read FILENAME; do
        if [[ `\date +%Y-%W -r "${FILENAME}"` == ${YEAR}-${WEEK} ]]; then
            # Uncomment to show the names of matching files
            # echo ${FILENAME}
            let COUNT++
        fi
    done
 
    echo Week $1 -- ${TGTDIR}: ${COUNT}
}
 
exit 0