Jeff Epler's blog

Calling a Python function with a timeout

In Python 3.13+ with pyrepl, my rpn calculator can show the result of evaluating some code while the user is entering it at the prompt.

Most code is trivial (e.g., 3 4 5 * * to multiply some numbers together) but complex code including recursion is possible. This can give code that runs for quite a long time! It's one thing if this happens when you hit enter, but another when there's the possibility that the repl freezes in the middle of entering some code.

There are various recipes for running Python code with timeout, but I wrote my own. Then, just for extra fun, I made it properly type check with ty.

I didn't integrate it into pydc yet but that part shouldn't be hard.

See the docstring of the function for some additional caveats.

(Embedded not available - View kvghgghs/timeout.py on codeberg.org or download raw)

[permalink]

Python multiline input: Using _pyrepl in your own code

Colorized, multiline input in Python 3.14

Starting with Python 3.13, a new internal module called _pyrepl has been added to Python. It is used for the regular interactive prompt, and allows multiline editing. Python 3.14 adds syntax highlighting.

I thought these features would be great to use in my own program, an rpn calculator modeled after the classic unix dc.

Since _pyrepl is an internal module to Python, it's not intended for use by scripts. But, they can't stop us! Here's what I've learned...

Basics: Multiline editing

The dc language denotes strings with balanced square brackets, so that [x [a b] w] is a string. In terms of multiline editing, we want input to continue when there is an unbalanced open bracket character.

Here's a simplistic implementation of a function which checks some input to find out whether it's balanced:

def count_brackets(s):
    return s.count("[") - s.count("]")

However, what we need is a true-or-false predicate. In Python zero is false and nonzero numbers are true so we could use count_brackets directly, but I also wrote an actual predicate:

def more_lines(s):
    return count_brackets(s) > 0

Now, we can go ahead and do some multiline input:

from _pyrepl.readline import multiline_input  
while True:
    try:
        statement = multiline_input(more_lines, ps1, ps2)
    except EOFError:
        break
    print(repr(statement))

A session with this program might look like so:

pydc> 3 3 +
'3 3 +'
pydc> [a [
  ... b]
  ... c]
'[a [\nb]\nc]'

Advanced: Syntax colorization

From Python 3.14, the Python repl colorizes Python code. We can repurpose this for highlighting our own syntax rather than Python syntax.

In Python 3.14, it's necessary to monkey-patch the function _pyrepl.reader.gen_colors. In Python 3.16 alphas, that doesn't work. Instead, it is required to update the gen_colors property of the reader object. Happily, the same gen_colors routine works in the same way in each version.

gen_colors generates a sequence of ColorSpan objects. The span member is an inclusive range of characters, and the tag must be one of several predefined python syntax elements.

Rather than fully highlighting the dc language, I'll show a simple gen_colors implementation, which colors digits and uppercase letters the way pyrepl shows numbers, lowercase letters the way pyrepl shows strings, and everything else in the terminal default color:

if hasattr(_pyrepl.utils, 'ColorSpan'):
    from _pyrepl.utils import ColorSpan, Span
    import _pyrepl.readline

    def gen_colors(s):
        for i, c in enumerate(s):
            if c in string.ascii_lowercase:
                yield ColorSpan(Span(i, i), "string")
            elif c in string.digits or c in string.ascii_uppercase:
                yield ColorSpan(Span(i, i), "number")
            else:
                yield ColorSpan(Span(i, i), "reset")

    reader = _pyrepl.readline._wrapper.get_reader()
    if hasattr(reader, 'gen_colors'):
        reader.gen_colors = gen_colors
    else:
        import _pyrepl.reader
        _pyrepl.reader.gen_colors = gen_colors

It's worth noting that the builtin gen_colors also takes care to emit fewer spans whenever possible, something I ignored for this example.

Advanced II: Showing messages

Pyrepl includes support for an area below the entry area. This is for showing completions, but you can use it to show whatever you like.

This is done by monkeypatching the reader object's after_command:

reader = _pyrepl.readline._wrapper.get_reader()
original_after_command = reader.after_command
def after_command(self, cmd) -> None:
    original_after_command(cmd)

    if cmd.finish: return

    buffer = "".join(self.buffer)
    msg = repr(buffer)
    if msg != self.msg:
        self.msg = msg
        self.dirty = True
reader.after_command = lambda cmd: after_command(reader, cmd) # fake instance method

This has the effect of continuously previewing what will be shown after hitting enter.

The code

Putting it all together, here's the full script:

(Embedded not available - View gz0bnruv/mli.py on codeberg.org or download raw)

[permalink]

Variations on 'if TYPE_CHECKING'

Suggested by mypy documentation:

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from typing import …

Works in mypy, pyright, pyrefly. Found in setuptool_scm generated __version__.py:

TYPE_CHECKING = False
if TYPE_CHECKING:
    from typing import …

Works in mypy, pyright, pyrefly. Best variant for CircuitPython?

def const(x): return x
TYPE_CHECKING = const(0)
if TYPE_CHECKING:
    from typing import …

Works in mypy only. Does not work in pyright, pyrefly:

if False:
    from typing import …

[permalink]

Talking directly to in-process tcl/tk

To be clear the original post also calls out the situation where the tkinter part of python is not installed by default, and my technique would not be applicable there.

So what do you do if you like Tk but don't care for the high level abstraction provided by Tkinter? Well, you can import _tkinter and create a tkapp object with _tkinter.create().

The _tkinter module and the tkapp object is largely undocumented, but in Python 3.11 here are some useful methods:

• tkapp.createcommand: Create a callback into Python code. Takes a string and a callable. Creates a Tcl command with that name, that calls back into Python code: app.createcomand("cb", lambda *args: print("cb", args))
• tkapp.eval: Takes a command and evaluates it. Call it with a single string: app.eval("button .b -text HI -command {cb arg1 arg2}")
• tkapp.call: Takes a series of arguments, does proper Tcl quoting, and evaluates it: app.call("pack", ".b")

Python 3.11.2 (main, Aug 26 2024, 07:20:54) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _tkinter
>>> app = _tkinter.create()
>>> app.createcommand("cb", lambda *args: print("cb", args))
>>> app.eval("button .b -text HI -command {cb arg1 arg2}")
'.b'
>>> app.call("pack", ".b")
''
>>> # Now I click the button several times ...
>>> cb ('arg1', 'arg2')
cb ('arg1', 'arg2')
cb ('arg1', 'arg2')

[permalink]

NYT "Letter Boxed" solver in C++

Lately it's been trying to get me to solve a puzzle called "Letter Boxed".

In this puzzle, there are 12 letters arranged with 3 on each side of a square. Valid words have to be made from the letters, with the additional constraint that two consecutive letters may not be on the same side of the square. For example if the edges are "xoy", "tws", "apv" and "kri" then "air" is not a valid word because the consecutive letters "ir" come from the same side. "vow" is. The last constraint is that each consecutive word starts with the last letter of the previous word, so "vow" can be followed by "wok" but not "sat". (Words of any length are permitted, not just 3 letters)

There is no particular scoring to the game, though you're suggested to "try to solve it in # words"; you can also view the previous day's suggested solution.

To me, it seems that the best answer is in the fewest words, with ties broken by the fewest number of characters.

The following program (which assumes that there's a standard unix-style dictionary at /usr/share/dict/words) can find what appear to be "optimal" solutions in only a few milliseconds. Simply supply it with the 12 letters as the first commandline argument, and it will perform a breadth-first search (BFS) with up to 5 words in length. Each candidate printed is better scoring (shorter) than the previous one, so the last line is the best score.

$ ./a.out xoytwsavpkri
228 candidate words
proviso - oaks - sixty - yaw
provisos - sixty - yaw - wk
vow - warps - sixty - yak
Checked 3027570 sequences

I originally wrote a program in Python, but memory usage was high and speed was low. This version uses an efficient structure where each word is reduced to a 32-bit quantity that tracks the letters present, the word length, and the word's terminal character. A particular game play is characterized by a fixed-size data structure that includes the characters used so far, the number of words, the number of total characters, and an array of up to 5 words. The deque is initialized with one entry for each possible word. In the main loop, the item is taken from the front of the deque. Then, based on the terminal letter of the last word played it tries each word starting with that letter. If this potential solution is not lower-scoring than the best one, then it stops evaluating. Otherwise, if this word completes the puzzle, the solution is printed and the best known score values are updated. Otherwise, this puzzle state is added to the end of the deque. The program loops until all possibilities have been evaluated.

A dynamic-programming approach would probably beat the BFS but BFS is quite fast enough for the published puzzles I've solved. It might also be possible to work from both ends towards the middle.

I don't have enough experience with the puzzle to know if 5 words always suffice for published puzzles, but it seems likely. The two real puzzles I have tried have 2-word solutions from this program (13 characters, the minimum possible length for 2 words), while they were suggested to be solvable in 5 and 6 steps.

When it comes to randomly selected puzzles, there are possible boards for which I find a best answer of 7 words (using a modified version of the program)

knezcuvamybx
68 candidate words
zany - yak - kc - can - numb - beaux - xv 7/23
zany - ye - exec - cab - bevy - yuk - km 7/22
zany - yuk - km - me - eve - exec - cab 7/21
Checked 3516880 sequences

and other sequences which have no solutions up to 7 words and use a lot of RAM and time before giving no solution (11GB peak resident size, 16 seconds):

mtijacfshpwk
194 candidate words
Checked 636279365 sequences

I didn't go full dynamic-programming but I did track the best way to each each of the 12*4096 states and stop recursing if the new candidate doesn't reach a known state faster. This runs much faster and uses less memory; the 11GB & 16s example with no solution above is now <4MB and <.01s! Runs below are with that version. (and apparently using a different dictionary, some runs were on debian oldstable and some on debian stable)

mtijacfshpwk
214 candidate words
Checked 1944 sequences

I also found that there are 8-word puzzles:

kmujpocaziqh
69 candidate words
jam - ma - aqua - ah - ho - oz - zip - pick 8/22
Checked 854 sequences

dnijzaexkyrc
153 candidate words
jerk - kc - ca - ax - xi - icky - yd - dz - zen 9/23
Checked 1038 sequences

Source code (GPL-3.0 license) (build with g++ -std=c++20 -O2):

Faster version:

Original version:

[permalink]

A quick example of transforming Python with libcst

I had occasion to encounter a Python library that used assert with a side effect:

assert initialize_hardware(), "hardware failed to initialize"

"Aha", I said, "I bet I can fix this with an automated tool". In this round of investigation, I found LibCST and set about creating a program that would do what was needed, namely, to turn at assert into if not initialize_hardware(): raise RuntimeError("hardware failed to initialize").

While LibCST has an explicit facility for "codemodding", I didn't notice it at first and wrote in terms of transformers, with my own command-line driver program.

Unfortunately, while my transformer succeeded, attempting to format the CST back into code would result in an error without very many matches on my favorite search engine: Unexpected keyword argument 'default_semicolon'. That linked issue didn't provide an answer, but my further investigation did.

In the Python grammer as represented by LibCST, an assert statement is part of a SimpleStatementLine, while an if statement is not wrapped in a SimpleStatementLine. So if the transformation of an Assert node into an If node is done alone, the new If node lies inside a SimpleStatementLine node, and that is not valid. The problem is not detected until rendering the CST back into code. (It may be possible that using type checking would have found a problem, as this is essentially a type error)

The solution that I arrived at was to also transform any SimpleStatementLine which ended up containing an If node, by using the FlattenSentinel to do it. I think it might have been even more correct to directly perform the transformation within SimpleStatementLine, but what I ended up works now.

[permalink]

Don't wreck your system with miniconda/anaconda

I noticed that the little blob it deposits in ~/.bashrc can easily be surrounded with a function definition. So, now to activate anaconda in the current shell, but never replace/hide system python in a normal shell, I can just type "fml".

fml () {
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/jepler/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/jepler/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/jepler/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/jepler/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
}

Now I don't feel quite so worried that having it present on the system is going to interfere with system software or with software I've installed to work with system software via pip.

[permalink]

My experience adding type annotations to a 2.5k-line Python library

The annotations were added in two steps: See pull requests #7 and #8. Together, these PRs contained 320 insertions and 223 deletions across 14 python files, plus 6 insertions in 2 other files related to CI. I did the work during a part of a day, probably under 4 hours of time spent. Since the package currently contains exactly 2500 physical lines of Python code, adding type annotations touched or added over 10% of physical lines!

read more…

Using Adafruit Macropad as LinuxCNC Control Pendant

Quick CircuitPython Driver for ES100 WWVB Receiver

Si5351 Frequency Planner in Python

Precision vs Accuracy: A Clock

How do you check a signature on a PGP/Mime message

Red: the perfect unit of measurement?

SNTP from Python: getting server's esimate of time quality

Callcentric "click 2 dial" commandline client