Many NAT firewalls time out idle sessions after a certain period of time to keep their trunks clean. Sometimes the interval between session drops is 24 hours, but on many commodity firewalls, connections are killed after as little as 300 seconds. To avoid having your SSH sessions become unresponsive after e.g. 5 minutes, do the following:

On Windows (PuTTY)

In your session properties, go to Connection and under Sending of null packets to keep session active, set Seconds between keepalives (0 to turn off) to e.g. 300 (5 minutes).

On Linux (ssh)

To enable the keep alive system-wide (root access required), edit /etc/ssh/ssh_config; to set the settings for just your user, edit ~/.ssh/config (create the file if it doesn’t exist). Insert the following:

Host *
    ServerAliveInterval 300
    ServerAliveCountMax 2

You can also make your OpenSSH server keep alive all connections with clients by adding the following to /etc/ssh/sshd_config:

ServerAliveInterval 300
ServerAliveCountMax 2

These settings will make the SSH client or server send a null packet to the other side every 300 seconds (5 minutes), and give up if it doesn’t receive any response after 2 tries, at which point the connection is likely to have been discarded anyway.

From the ssh_config man page:

ServerAliveCountMax
Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if the server becomes unresponsive, ssh will disconnect after approximately 45 seconds. This option applies to protocol version 2 only; in protocol version 1 there is no mechanism to request a response from the server to the server alive messages, so disconnection is the responsibility of the TCP stack.

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server, or 300 if the BatchMode option is set. This option applies to protocol version 2 only. ProtocolKeepAlives and SetupTimeOut are Debian-specific compatibility aliases for this option.

I have a Razer Deathadder. It’s a nice mouse. In Ubuntu, though, its polling rates are through the roof and the mouse is pretty much unusable, even with GNOME’s mouse sensitivity and acceleration settings at their lowest. Previously, this could be fixed by tweaking the mouse section of your X.Org configuration file, /etc/X11/xorg.conf, but in recent versions, a different measure is needed, as most devices are managed via HAL. Here’s how I regained my sanity and mouse slowness. The fix should work for any high-end mouse.

1. Open a terminal
2. Run the command: hal-device
3. In the output, locate the mouse’s hex format vendor and product ID’s as highlighted below:

   82: udi = '/org/freedesktop/Hal/devices/usb_device_1532_7_noserial_if0'
     linux.hotplug_type = 2  (0x2)  (int)
     linux.subsystem = 'usb'  (string)
     info.linux.driver = 'usbhid'  (string)
     info.subsystem = 'usb'  (string)
     info.product = 'USB HID Interface'  (string)
     info.udi = '/org/freedesktop/Hal/devices/usb_device_1532_7_noserial_if0'  (string)
     usb.linux.sysfs_path = '/sys/devices/pci0000:00/0000:00:1d.2/usb8/8-2/8-2:1.0'  (string)
     usb.configuration_value = 1  (0x1)  (int)
     usb.num_configurations = 1  (0x1)  (int)
     usb.num_interfaces = 1  (0x1)  (int)
     usb.device_class = 0  (0x0)  (int)
     usb.device_subclass = 0  (0x0)  (int)
     usb.device_protocol = 0  (0x0)  (int)
     usb.product_id = 7  (0x7)  (int)
     usb.vendor_id = 5426  (0x1532)  (int)
     usb.product = 'USB HID Interface'  (string)
     usb.vendor = 'Razer USA, Ltd'  (string)
     usb.num_ports = 0  (0x0)  (int)
     usb.max_power = 100  (0x64)  (int)
     usb.device_revision_bcd = 256  (0x100)  (int)
     usb.is_self_powered = false  (bool)
     usb.can_wake_up = true  (bool)
     usb.bus_number = 8  (0x8)  (int)
     usb.speed = 12  (double)
     usb.version = 2  (double)
     linux.sysfs_path = '/sys/devices/pci0000:00/0000:00:1d.2/usb8/8-2/8-2:1.0'  (string)
     info.parent = '/org/freedesktop/Hal/devices/usb_device_1532_7_noserial'  (string)
     usb.interface.number = 0  (0x0)  (int)
     usb.linux.device_number = 3  (0x3)  (int)
     usb.interface.subclass = 1  (0x1)  (int)
     usb.interface.class = 3  (0x3)  (int)
     usb.interface.protocol = 2  (0x2)  (int)

   In this case, my Product ID is 0×7 and my Vendor ID is 0×1532. Note that there can be more than one section containing the name of your mouse or its manufacturer — if you can’t find the product and vendor ID, look further down.

4. Edit the HAL policy file for input devices: sudo nano -w /etc/hal/fdi/policy/10-x11-input.fdi
5. Insert the following text:

<?xml version="1.0" encoding="UTF-8"?>
<deviceinfo version="0.2">
  <device>
    <match key="@input.originating_device:usb.vendor_id" int="0x1532">
      <match key="@input.originating_device:usb.product_id" int="0x7">
        <merge key="input.x11_options.ConstantDeceleration" type="string">5</merge>
      </match>
    </match>
  </device>
</deviceinfo>

Adjust your vendor_id and product_id to match what you noted down before. If the file is empty or doesn’t exist, don’t worry. If it already exists, omit the first line about xml.

6. Hit Ctrl + X, then Y to save the file and exit nano
7. Restart hald: sudo service hald restart
8. Restart X.Org (log out or reboot your computer)

That’s it! The “ConstantDeceleration” setting in /etc/hal/fdi/policy/10-x11-input.fdi is what does the trick. When set to a value of 5, the sensitivity will essentially be divided by 5. Oh, sweet sanity.

By far the easiest way to set up a simple connection proxy is to use the SSH tunneling feature of either PuTTY on Windows or SSH on Linux. This lets you establish connections to servers and ports that you might not be able to access (e.g. from work), as long as you can connect to your server’s SSH service (e.g. myserver.com port 22). This might be for privacy reasons, to connect to MSN from work, to browse a blocked website, et cetera.

On Windows Machines

1. Download and open PuTTY
2. In the fields ‘Address’ and ‘Port’, enter the address and port for your SSH server
3. Go to ‘Connection’ -> ‘SSH’ -> ‘Tunnels’ on the left-hand side
4. In ‘Source port’, enter 31337, then click the button ‘Dynamic’ and then ‘Add’
5. Go back to the main ‘Session’ screen
6. In the ‘Saved Sessions’ text box, enter e.g. “My Shell” and click ‘Save’
7. Double-click “My Shell” to establish a connection, then log in to your shell
8. In any application that supports connecting through a proxy, set the following settings:
   • Proxy Type: SOCKS 5
   • Proxy Server: 127.0.0.1
   • Proxy Port: 31337

You can also set these as your global proxy settings in Windows (via ‘Control Panel’ -> ‘Internet Properties’ -> ‘Connections’ -> ‘LAN settings’ -> “Use a proxy server for your LAN” -> ‘Advanced’ -> ‘Socks’: 127.0.0.1:31337. This will cause most applications to connect through the SSH tunnel to your server.

In the future, just open PuTTY and double-click “My Shell” to open your shell and activate the SSH tunneling.

On Linux Machines

1. Open a terminal
2. Enter e.g.: ssh -D31337 myuser@myserver.com -N
3. Log in to your shell
4. In any application that supports connecting through a proxy, set the following settings:
   • Proxy Type: SOCKS 5
   • Proxy Server: 127.0.0.1
   • Proxy Port: 31337

Alternatively, enter e.g.: ssh -L 31337:patrickmylund.com:80 myuser@myserver.com -N. Here, you specify the target host and port before-hand; the result is that all connections to 127.0.0.1 port 31337 will be tunneled through your server, myserver.com, using your username, myuser, to the target machine, patrickmylund.com, port 80.

The SSH tunnel will stay active until you close the terminal window or hit CTRL+C (Linux), or close PuTTY (Windows).

A branch called chromiumos was just added to the official Chromium git repository, and the contents of the index files there indicate that Chromium OS, and in effect Google Chrome OS, is based on Ubuntu Linux. From chromiumos.git/src/package_repo/repo_list_image.txt:

  71 fontconfig 2.6.0-1ubuntu12 optional utils pool/main/f/fontconfig/fontconfig_2.6.0-1ubuntu12_i386.deb
  72 fontconfig-config 2.6.0-1ubuntu12 optional libs pool/main/f/fontconfig/fontconfig-config_2.6.0-1ubuntu12_all.deb
  73 gcc-4.4-base 4.4.1-1ubuntu3 required libs pool/main/g/gcc-4.4/gcc-4.4-base_4.4.1-1ubuntu3_i386.deb

Here are some videos from the Chromium OS website that just went live:

What is Google Chrome OS?

Chromium OS Security

Chromium OS & Open Source

Chromium Fast Boot

There’s a live transcript of the Google Chrome OS event speech by Google VP of Product Management, Sundar Pichai, over at TechCrunch. They have also just posted a video demo of Google Chrome OS recorded at the announcement event in Mountain View, showing off the interface, bootup times, and more.

Update: Ars Technica has just published an excellent analysis of Chrome OS.

Google is apparently releasing their highly anticipated, lightweight, Linux-based operating system targeted at netbooks, Google Chrome OS, within a week, according to TechCrunch. TechCrunch also thinks we should expect shoddy driver support, which seems like a strange conclusion considering the maturity of the Linux kernel.

There’s really not that much information about the operating system, though it is safe to assume that there will be a strong emphasis on the Chrome browser and all of Google’s web-based services and applications.

Let’s wait and see.

Update: It seems that Google is planning a special Chrome OS event for Thursday, the 19th of November, which will include a demonstration and “complete overview” of Google Chrome OS. The product apparently won’t be released until (early) next year, but the wording “complete overview” gives me the impression that they are just polishing and testing now.

Update 2: Chromium OS is Out, and It is Based on Ubuntu Linux!

Staying ahead of IT service issues can be frustrating when you manage several servers, or even a single server with many services. Enterprise IT Infrastructure Monitoring Solutions (a fancy term for something that is really pretty simple) attempt to remedy the problem by repeatedly checking the status of machines and services on the network and alerting the responsible administrators as soon as something goes wrong, or even before there’s a problem.

It’s hard to argue against implementing a monitoring solution within the network, as it is much a setup-and-forget matter that adds negligible load. The monitoring solution itself is — or at least, should be — very low maintenance, yet provides very valuable insight into the health of the network.

Introducing Nagios

Nagios is an infrastructure monitoring solution that is both popular and open source. Apart from its obvious monitoring capabilities, it includes the ability to associate an event handler to an event, allowing you to fix a problem automatically. If — for example — one of your Python applications crashes, you can have Nagios do python /opt/myapp/myapp.py automatically, before any human administrators have the time to do so. Other features include the ability to create many kinds of reports, and to send notifications and alerts via email and SMS.

Nagios' web interface

Nagios is based primarily on C and shell scripts, which makes it light on performance but adds a slightly ‘hackish’ feel. It comes with a CGI-based web interface (which we’ll spice up a bit) that lets you view and manage Nagios, through what are known as External Commands.

I’d like to demonstrate how to set up rudimentary Nagios monitoring on a small farm of Linux servers, with an Ubuntu/Debian server running the primary Nagios process. In the end, we’ll be monitoring the states of various services on the servers, including the ones seen in the screenshot above (Apache processes, APT, Current Load, Current Users, Disk Space, Dovecot, FTP, HTTP, MySQL, SMTP, SSH, Swap, Total Processes, and Zombie Processes). We will also receive notifications by email whenever something goes wrong:

Please note that this guide is meant to get you up and running quickly, and that it’s not a substitute for the official Nagios documentation. If you want to know what all of the different configuration options do (or can do), please consult the (excellent) documentation.

Setting Up The Nagios Server

The steps in this section should just be done on the main Nagios server, not the clients it will be monitoring. We’ll get to those later!

This procedure should be quite similar on other distributions if you use their package managers (yum, yast, urpmi, etc.) or install Nagios from source, but no guarantees.

1. Let’s become root so we don’t have to prepend sudo to everything:

   sudo -s

2. If you want to make use of Nagios’ web interface and Apache isn’t already installed:

   aptitude install apache2

   It’s entirely possible to use something like nginx or lighttpd to serve the interface, but that is not covered in this guide.

3. Install Nagios from the package repositories:

   aptitude install nagios3 nagios-nrpe-plugin

Nagios should be accessible at http://nameofnagiosserver/nagios3 already! We still have some configuration to do, though.

4. Stop Nagios:

   /etc/init.d/nagios3 stop

5. Add a new user for the web interface, e.g. patrick. The default configuration grants all security permissions to the user nagiosadmin, but we’ll change that to the name of the new user, too:

   htpasswd -c /etc/nagios3/htpasswd.users patrick
   perl -p -i -e "s/nagiosadmin/patrick/g" /etc/nagios3/cgi.cfg

The perl command above replaces all occurrences of nagiosadmin with patrick in the file /etc/nagios3/cgi.cfg.

The users listed in /etc/nagios3/cgi.cfg are effectively global administrators. For regular users, you can still add them as users with htpasswd, but assign privileges by making them Contacts for certain hosts or hostgroups, instead. We’ll get to this later!

6. If you want to add more user accounts for the web interface:

   htpasswd /etc/nagios3/htpasswd.users john

7. And if you want to give them superuser privileges:

   perl -p -i -e "s/patrick/patrick, john/g" /etc/nagios3/cgi.cfg

   Go through /etc/nagios3/cgi.cfg manually to see what the different security options do, and to grant more fine-grained privileges to other administrators.

8. Edit /etc/nagios3/nagios.cfg and change check_external_commands=0 to 1 to allow monitoring commands to be issued through the web interface
9. On Debian/Ubuntu, run the following commands after setting check_external_commands=1:

   dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
   dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3

10. Edit /etc/nagios3/conf.d/contacts_nagios2.cfg to match your preferences. Example:

    define contact{
            contact_name                    patrick
            alias                           Patrick Mylund
            service_notification_period     24x7
            host_notification_period        24x7
            service_notification_options    w,u,c,r
            host_notification_options       d,r
            service_notification_commands   notify-service-by-email
            host_notification_commands      notify-host-by-email
            email                           my@emailaddress.com
            }

    And further down:

    define contactgroup{
            contactgroup_name       admins
            alias                   Nagios Administrators
            members                 patrick
            }

11. Make a host definition for a server you want to monitor by creating a matching config file, e.g for the server ‘tranquillity’, nano -w /etc/nagios3/conf.d/tranquillity_nagios2.cfg, then insert a declaration. Example:

    define host{
            use                     generic-host            ; Name of host template to use
            host_name               tranquillity
            alias                   PatrickMylund.com Web Server
            address                 209.20.82.6
            }

    You can put all of your host definitions in one file if you want, e.g. datacenter1_nagios2.cfg — just remember the _nagios2.cfg at the end of the file name, which is what tells Nagios to load that file (and in the proper format).

12. Repeat the step above to add a host definition for each server you want to monitor
13. Move some standard configs to make room for our configured ones:

    mv /etc/nagios3/conf.d/localhost_nagios2.cfg /etc/nagios3/conf.d/localhost_nagios2.cfg.old
    mv /etc/nagios3/conf.d/services_nagios2.cfg /etc/nagios3/conf.d/services_nagios2.cfg.old
    wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/services_nagios2.cfg -O /etc/nagios3/conf.d/services_nagios2.cfg

14. Edit /etc/nagios3/conf.d/hostgroups_nagios2.cfg. List which hosts (comma-separated) should belong to which groups (debian-servers, http-servers, ssh-servers, and ping-servers), and add some extra hostgroups: db-server, ftp-servers, and mail-servers:

    define hostgroup {
            hostgroup_name  db-servers
                    alias           Database servers
                    members         tranquillity, singularity
            }
     
    define hostgroup {
            hostgroup_name  ftp-servers
                    alias           FTP servers
                    members         tranquillity, singularity
            }
     
    define hostgroup {
            hostgroup_name  mail-servers
                    alias           IMAPS/SMTP servers
                    members         tranquillity
            }

    You can see which services are associated with which hostgroups by looking in /etc/nagios3/conf.d/services_nagios2.cfg.

We’re done with the Nagios server for now. Let’s look at the settings for the Linux servers we want to monitor.

Configuring Monitored Clients

The steps in this section should be done on each Linux host that you want to monitor.

1. Again, let’s become root:

   sudo -s

2. Install Nagios’ NRPE module:

   aptitude install nagios-nrpe-server

   Installing the NRPE module is optional, but you won’t be able to run any of Nagios’ scripts directly on the target client if you do not. This is necessary for monitoring system stats, and generally anything that cannot be probed from the outside over the network (by the main Nagios server).

   See the NRPE documentation (PDF) for manual installation instructions, as well as how to get information via SSH (get_by_ssh) instead of NRPE.

3. Stop NRPE:

   /etc/init.d/nagios-nrpe-server stop

4. Install a custom nrpe_local.cfg (this will save us some time later):

   mv /etc/nagios/nrpe_local.cfg /etc/nagios/nrpe_local.cfg.old
   wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/nrpe_local.cfg -O /etc/nagios/nrpe_local.cfg

   Go through /etc/nagios/nrpe_local.cfg to see the list of commands that Nagios will be able to execute on hosts running NRPE. By default, NRPE will only run the commands defined in this configuration file, and without any arbitrary arguments. I strongly recommend you stick to this for security purposes.

   On the main Nagios server, all service commands prefixed with check_nrpe_1arg in /etc/nagios3/services_nagios2.cfg are commands defined in /etc/nagios/nrpe_local.cfg on the monitored clients.

5. Define what hosts are going to be allowed to probe the NRPE module for information (comma-separated). For instance, if the main Nagios server has IP 192.168.1.105:

   perl -p -i -e "s/127.0.0.1/192.168.1.105/g" /etc/nagios/nrpe_local.cfg

6. If you have a firewall (iptables, ufw, etc.), you need to open for connections on port 5666 on the clients (for NRPE). If the main Nagios server has IP 192.168.1.105, you could do ufw allow proto tcp from 192.168.1.105 to any port 5666, or ufw allow 5666/tcp with Ubuntu’s Uncomplicated Firewall.
7. Start the NRPE module:

   /etc/init.d/nagios-nrpe-server start

We just about have a basic Nagios setup now!

Testing Nagios

Let’s see if what we’ve set up is working. On the main Nagios server, start the Nagios service:

/etc/init.d/nagios3 start

If all goes well, navigate to e.g. http://192.168.1.105, login with the user credentials you set up earlier, then click on Service Detail in the menu on the left. All of our services will be PENDING, meaning they’ll be checked shortly. You can speed this up by clicking on a service and clicking Re-schedule the next check of this service (this is what is called an External Command).

If any of the service states turn out to be CRITICAL or UNKNOWN, don’t panic — take a look at the different configuration files in /etc/nagios3/conf.d. The settings and commands are pretty straight-forward.

You can find examples of the resulting configuration files in nagios-conf-example.tar.gz. The configs are for a single server (singularity) with the IP address 192.168.2.3.

An Extra Touch

Nagios’ web interface doesn’t look very pretty. We can spice it up a little by changing the CSS. I’ve prepared a modified status.css for your convenience:

mv /etc/nagios3/stylesheets/status.css /etc/nagios3/stylesheets/status.css.old
wget http://patrickmylund.com/files/misc/1202-nagios_quickstart/status.css -O /etc/nagios3/stylesheets/status.css

Now hit F5 in the web interface!

Bear In Mind

• The easiest way to monitor the Nagios server itself is to pretend it’s yet another server. Install NRPE, set the connection settings, and add it in the host declarations with the other servers.
• The exclamation mark (!) is meant to separate command arguments in Nagios configuration files. For instance, check_nrpe_1arg!check_swap would mean you’re running check_nrpe_1arg with the argument check_swap.
• All of the scripts and commands you can issue through Nagios are stand-alone scripts. When configuring Nagios, you can run each command, for instance check_smtp, manually instead of doing tons of trial-and-error with the configuration files:

  /usr/lib/nagios/plugins/check_smtp -H 192.168.1.105
  /usr/lib/nagios/plugins/check_smtp -h

• All lists in Nagios configuration files are comma-separated.
• You can set the contact_groups value on any service, host, or hostgroup declaration. Contact groups are defined in /etc/nagios3/conf.d/contacts_nagios2.cfg. Any person in a contact group that has a user account for the web interface (htpasswd.users) can automatically view any hosts and services associated with it.

  Example:

  define hostgroup {
          hostgroup_name  mail-servers
                  alias           IMAPS/SMTP servers
                  members         singularity
                  contact_groups  mailadmins
          }

Again, the best part about what we’ve set up now is that you can go right ahead and forget about it. You’ll receive an e-mail at the contact address specified whenever something is amiss, as well as when it gets better. If I’m right, though, you’ll want to tune your configuration a lot further. We’ve barely touched the surface; Nagios can do much more, and everything is thoroughly documented in the official documentation.

Other useful links:

• Official Nagios FAQ
• Official Nagios Wiki
• MonitoringExchange Nagios Plugins, Guides, Artwork, and more
• NagiosForge
• Another Nagios Wiki

FilesByWeek is a small script that counts the number of files in a folder that were created (or last modified) in X week of the year. It’s designed for use with Linux/Postfix Maildirs (and thus excludes the standard .Sent folder and any Dovecot/Courier IMAP files from the find query), but should work just fine on any kind of directory.

Get the latest version here. Or, if you just want to see what the fuzz is about:

#!/bin/bash
typeset -i YEAR WEEK COUNT
 
WEEK=$1
# Comment out the following line if the year starts on a Monday
WEEK=$((WEEK-1))
# Use current system year by default. This can be changed to e.g.: YEAR=2008
YEAR=`date +%Y`
TGTDIR=$2
COUNT=0
 
\find ${TGTDIR} \
-type d \( -name "*.sent" -o -name "*.Sent" -o -name "courierimapkeywords" -o -name "courierimaphieracl" \) -prune -o \
-type f \( ! -name "subscriptions" ! -name "courierimapsubscribed" ! -name "dovecot.index.log*" ! -name "dovecot.index" ! -name "maildirfolder" ! -name "dovecot-keywords" ! -name "dovecot.index.cache" ! -name "courierimapacl" ! -name "courierimapuiddb" ! -name "dovecot-uidlist" \) \
-print |
{
    while read FILENAME; do
        if [[ `\date +%Y-%W -r "${FILENAME}"` == ${YEAR}-${WEEK} ]]; then
            # Uncomment to show the names of matching files
            # echo ${FILENAME}
            let COUNT++
        fi
    done
 
    echo Week $1 -- ${TGTDIR}: ${COUNT}
}
 
exit 0

Surprise, the warrant that was issued for a college student’s stuff on the basis that he was using an operating system with “white font on a black screen” has been thrown out, as no probable cause existed.

On May 21, 2009, Justice Botsford of the Supreme Judicial Court of Massachusetts granted our client Riccardo Calixte’s motion to quash the illegal search warrant with which it seized Calixte’s computers, phones, ipods, camera and other personal property. Not only is this an enormous victory for Calixte himself, but the ruling is also the highest state court opinion to repudiate the nascent law enforcement “trend” of charging internet users who violate websites’ terms of service as criminals. (Case page with background documents here.)

Interesting automated software test system concept. The goal is to design this for the Linux kernel, but it should be very useful in other regards as well.

On Friday, EFF and the law firm of Fish and Richardson filed an emergency motion to quash and for the return of seized property on behalf of a Boston College computer science student whose computers, cell phone, and other property were seized as part of an investigation into who sent an e-mail to a school mailing list identifying another student as gay. The problem? Not only is there no indication that any crime was committed, the investigating officer argued that the computer expertise of the student itself supported a finding of probable cause to seize the student’s property.

“[The Person] reported that Mr. Calixte uses two different operating systems to hide his illegal activities. One is the regular B.C. operating system and the other is a black screen with white font which he uses prompt commands on”. What a nice college.