Jeff Epler's blog

Every once in a while, Python is too slow

I recently acquired a USB-connected relay that controls a standard power outlet, called the "USB NET POWER 8800". I'd ordered this with the knowledge that there was a Python implementation of the control program that was said to run on Linux.

It does work, but I was stunned at how long it took to switch the outlet on or off: a couple of seconds! I should disclaim that it's running on a rather underpowered machine (the OLPC XO-1 with a Geode x86 CPU @ 430MHz), but this was way too long to be acceptable:

$ time py-usbpower query
Power: on

real    0m2.690s
user    0m1.484s
sys     0m0.096s

With some detective work, I discovered that ctypes is doing things like running the compiler, objdump(!) and ldconfig(!), several times (!!), at each invocation of the program. (this is python2.5, but it looks like Python 2.7's ctypes still does essentially the same thing, unfortunately)

I set about to code this in C. For my reward, I got a very fast-executing program:

$ time c-usbpower status
ON

real    0m0.007s
user    0m0.000s
sys     0m0.000s

That's something like a 384x speedup. Lesson? ctypes has some very substantial startup costs for loading a library. It may be enough to make it unsuitable for a short-lived program.

In case it's useful to you, I enclose a copy of my usbpower program. It's under the terms of the GNU GPLv2.

Files currently attached to this page:

usbpower.c

3.4kB

sorttop: show the biggest values as a program runs

When I am looking for big disk users, I don't care about all the little items. What would be great is to continuously see the biggest items encountered so far.

To this end, I've written a Python program I call 'sorttop'; it continually reads stdin, sorts the first column according to magic (i.e., human numbers like from "du -h" work as expected), and shows the top items read so far. Example usage: du -hx / | python sorttop.py

Update, Feb 20, 2012: Revised the script to work for more terminals; fixes a traceback on the line 'bol = curses.tparm(ti_hpa, 0)' when TERM=screen.

Files currently attached to this page:

sorttop.py

3.3kB

csql: perform sql queries on csv data

The websites I operate are flat-file based, so when I want to store data it's easiest to write it in csv format. But it's not particularly easy to make queries on this data without additional software.

I'd rather not fire up Open Office or try to figure out how to make awk parse a quoted excel-format file, so I wrote 'csql', which reads a csv file into an in-memory sql database, performs a query, and then writes the result also in csv format.

A short example:

rsvp.csv:

Buffy Summers,false,0
Xander Harris,true,2
Willow Rosenberg,true,1

rsvp.cols:

name text
attend boolean
count integer

Performing queries:

$ csql rsvp.csv 'select sum(count) from t where attend="true"'
3
$ csql rsvp.csv 'select name from t where attend="true"'
Xander Harris
Willow Rosenberg

Files currently attached to this page:

csql

3.4kB

tnef2mime: automatically extract parts from winmail.dat

In her professional life, my fiancée routinely gets emails with attachments in the 'winmail.dat' aka application/ms-tnef format. It does her no good to berate these prisoners of proprietary software, so I set out to provide her with an easy-to-use solution that would work on her end. Now, prompted by this blog entry I am posting it for others to use.

There are a number of programs that can rescue data from this proprietary attachment format, but I was not aware of a program that would automatically convert the winmail.dat attachment to (a series of) regular mime attachments. (it appears that ytnef_smtpd may be such a program)

So I wrote one, tnef2mime.py. tnef2mime reads a message on standard input, combs it for ms-tnef attachments, and uses the tnef(1) program to extract their contents. Then it takes the extracted files, adds them as new MIME parts, and prints the whole resulting message.

This is suitable for use in e.g. procmail:

:0fw
| tnef2mime

Several steps are taken in order to minimize the chances for data loss: Just in case it did an imperfect job, it leaves the existing ms-tnef attachment intact. If any problem is encountered, the unmodified message is written.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Files currently attached to this page:

tnef2mime.py

3.1kB

CropGUI 0.1.1: now with gtk

Gtk version of cropgui

Earlier this year, I released a program for lossless cropping of jpeg images. This week, based on feedback from Ingrid, I ported the program to gtk (it originally used tkinter).

Remebering values between runs of emc

This is a userspace hal component which stores values to disk so that it can be preserved from run to run.

Usage

loadusr -W remember v1 v2 f:floatv u:unsignedv

Show recent e-mail in screen title bar or status bar

"lastmail" parses a procmail log file and extracts information about the most recent received e-mail, printing it in the form definbox: Logwatch for lamp (Linux) (8 minutes ago). This is useful as a backtick substitution in screenrc:

backtick 1 30 30 lastmail
hardstatus string "%H %n%?: %t%? %h  %1`"

Files currently attached to this page:

lastmail

2.5kB

cropgui: A Linux GUI for Lossless JPEG Cropping

Cropping a tiny insect

Update: See the newer version of cropgui

Of the pictures from my recent trip I'd like to put online, I've found that in 75% of the cases where I want to retouch the photo, it's to crop it and nothing else. Since I shoot in jpeg, it's a lossy process to load the jpeg in gimp, crop it, and write the result.

But it turns out that debian's jpegtran has a "-crop" flag which performs lossless cropping of jpeg images as long as the crop is to a multiple of what the manpage calls the "iMCU boundary", a (usually?) 8x8 block of pixels. This feature was pioneered by Guido of jpegclub.org some years ago.

There's apparently a nice Windows front-end to this program, but I didn't find a Linux one. So I wrote one! It's pretty basic, but it gets the job done. You can download it below.