2009-12-31

How to fix the Ctrl-Y Ctrl-Z inconsistency in GNOME Terminal with international keyboard layouts

This blog post describes and fixes an inconsistency between some keys with or without Ctrl in GNOME Terminal with international keyboard layouts.

The quick fix:

$ wget -O- -q http://pts-mini-gpl.googlecode.com/svn/trunk/pts-vtefix/vtefix.sh |
sudo bash

In GNOME Terminal, with the Hungarian and German keyboard layouts the keys Y and Z are switched with respect to the US English layout. So when the national layout is active, one of the keys yields the string "z", but when the same key is pressed with Ctrl, it yields "^Y", instead of the expected "^Z". All other applications (e.g. GIMP, Firefox, xterm) work as expected, except for those using VTE (e.g. GNOME Terminal).

Here is a script which fixes this on Ubuntu Hardy, Ubuntu Karmic and Ubuntu Lucid:

perl -we '
  use integer;
  use strict;
  my $fn = $ARGV[0];
  print STDERR "info: vtefix library filename: $fn\n";
  die "error: file not found: $fn\n" if !open F, "<", $fn;
  $_ = join "", ;
  my @L;
  while (/\xf6[\x80-\x8f].[\x09-\x0a]\0\0[\x00\x04]
         |\x80[\x78-\x7f].\0\x74.\xf6\x85.\xff\xff\xff[\x00\x04]
         |\x80\x7d\x32\x00\x74\x11\x41\xf6\xc5[\x00\x04]
          (?=(?:\x0f\x1f\x80\0\0\0\0)?\x0f\x85)
         /sgx) {
    push @L, pos($_) - 1;
  }
  # We patch only the first occurrence, the 2nd one is different.
  pop @L if @L == 2 and vec($_, $L[0] - 6, 8) == 0xf6 and
                        vec($_, $L[1] - 6, 8) == 0xf6;
  if (@L == 1) {
    if (vec($_, $L[0], 8) == 4) {  # need to patch
      print "info: patching at offset $L[0]\n";
      die "error: cannot open file for writing: $fn\n" if !open F, "+<", $fn;
      die if !sysseek F, $L[0], 0;
      die if !syswrite F, "\0";
      print "info: patching OK, no need to restart gnome-terminal\n";
    } else {
      print "info: already patched at offset $L[0]\n";
      exit 2;
    }
  } else {
    die "error: instruction to patch not found (@L)\n";
  }
' -- "${1:-/usr/lib/libvte.so.9}"

Download the script from: http://pts-mini-gpl.googlecode.com/svn/trunk/pts-vtefix/vtefix.sh

The script has to be run each time after the libvte9 package is upgraded.

The script modifies the i386 and amd64 compiled libvte binary:

# amd64 Ubuntu Hardy:
# $ objdump -d /usr/lib/libvte.so.9.2.17 | grep testb | grep '[$]0x4,'
# 2a786:       f6 80 88 0a 00 00 04    testb  $0x4,0xa88(%rax)
# 2a7ba:       f6 87 88 0a 00 00 04    testb  $0x4,0xa88(%rdi)
# 
# i386 Ubuntu Hardy:
# $ objdump -d /usr/lib/libvte.so.9.2.17 | grep testb | grep '[$]0x4,'
# 25cd8:       f6 80 4c 09 00 00 04    testb  $0x0,0x94c(%eax)
# 25d0d:       f6 81 4c 09 00 00 04    testb  $0x4,0x94c(%ecx)

This corresponds to the following snipped in vte-0.16.3/src/vte.c:

if (handled == FALSE && normal == NULL && special == NULL) {
  if (event->group &&
      (terminal->pvt->modifiers & GDK_CONTROL_MASK))
    keyval = vte_translate_national_ctrlkeys(event);
  keychar = gdk_keyval_to_unicode(keyval);

The script replaces GDK_CONTROL_MASK (== 4) by 0, so the condition is always false, and vte_translate_national_ctrlkeys(...) is never called, which fixes the problem. Due to memory mapping, it is not necessary to restart GNOME Terminal instances already running.

To find the spot to patch in the binary, grep for gdk_keyval_to_unicode in the output of objdump -d /usr/lib/libvte.so.9.

More information about the bug (problem):

2009-12-19

Experimental HTTP server using Stackless Python

This blog post documents my experiment to write a non-blocking HTTP server based on coroutines (tasklets) of Stackless Python. My goal was to write a minimalistic web server server which can handle cuncurrent requests by using non-blocking system calls, multiplexing with select(2) or epoll(2), returning a simple Hello, World page for each request, using the coroutines of Stackless Python. I've done this, and measured its speed using ApacheBench, and compared it to the Hello, World server of Node.js.

The code is here: http://code.google.com/p/pts-mini-gpl/source/browse/#svn/trunk/pts-stackless-httpd http://syncless.googlecode.com/svn/trunk/benchmark.old/

Relevant ApacheBench spee results (for ab -n 100000 -c 50 http://127.0.0.1:.../):

Notes about the speed measurements:
  • I was using a recently compiled Stackless Python 2.6 and a recently compiled psyco for JITting.
  • I was surpriesed that my experimental code using select(2) and Stackless Python is faster than Node.js (by a factor of 1.925 on average, and the worst-case times are faster as well).
  • The speed comparison is not fair since Node.js has a real HTTP server protocol implementation, with its overhead, and my code just skips the HTTP header without parsing it.
  • Setting the TCP socket listen queue size to 100 (using listen(2)) made a huge difference on the worst case connection time. Compared to the setting of 5, it reduced the worst-case connection time from 9200 ms to 23 ms (!) in the measurement.
  • The source code of both servers can be found in the repository above.
  • My conclusion about the speed measurements is that a HTTP server based on Stackless Python and epoll(2) can be a viable alternative of Node.js. It would be worthwhile implementing one, and then doing proper benchmarks.

The advantage of using Stackless Python over callback-based solutions (such as Node.js in JavaScript, Twisted and Tornado) is that one can implement a non-blocking TCP server without being forced to use callbacks.

The unique advantage of Node.js over other solutions is that in Node.js not only socket communication is non-blocking, but DNS lookups, local filesystem access and other system calls as well – Node.js is non-blocking by design, but with other frameworks the programmer has to be careful not to accidentally call a blocking function. Avoiding a blocking function is especially cumbersome if a library used only provides a blocking interface.

Update: I've created a HTTP server capable of running WSGI applications. I've also integrated dnspython as an asynchronous DNS resolver. See it as project Syncless.

Update: Added (web.py) and CherryPy support.

Update: I've realized that the functionality of Syncless has already been implemented many times in Python. Examples: Concurrence, eventlet, gevent. See the comparison.

Minimalistic client for Amazon S3 (AWS) in Python

I've written py-mini-s3, a minimalistic client for Amazon S3 (AWS) in pure Python. It supports the GET and PUT operations. See its source code.

The implementation is based on the documentation http://docs.amazonwebservices.com/AmazonS3/latest/index.html?RESTAuthentication.html

2009-12-15

Sex, Unix style

My instruction sequence compilation of the traditional Unix sex theme:

date; touch; kiss; grep; strip; unzip; finger; mount; fsck; more; yes; uptime; gasp; umount; sleep

2009-12-06

Polynomials in Python with operator overloading

This blog post is a fun example for implementing univariate polynomials with operator overloading in Python. It is almost a domain-specific language (DSL). Here is the code:

#! /usr/bin/python2.4

class Polynomial(object):

  __attrs__ = ['co']

  def __init__(self, co=None):
    if co:
      co = list(co)
      while len(co) > 1 and not co[-1]:
        co.pop()
      self.co = co
    else:
      self.co = [0]

  def __repr__(self):
    items = []
    for i in xrange(len(self.co)):
      if self.co[i]:
        if i == 0:
          items.append('%s' % self.co[i])
        else:
          if self.co[i] == 1:
            pre = ''
          else:
            pre = '%s * ' % self.co[i]
          if i == 1:
            post = ''
          else:
            post = ' ** %s' % i
          items.append('%sx%s' % (pre, post))
    return ' + '.join(items) or '0'

  __str__ = __repr__

  def __call__(self, x):
    i = len(self.co)
    v = 0
    while i > 0:
      i -= 1
      v = self.co[i] + x * v
    return v

  def __add__(self, p):
    if isinstance(p, Polynomial):
      q = self
      if len(p.co) > len(q.co):
        p, q = q, p
      co = list(q.co)
      for i in xrange(len(p.co)):
        co[i] += p.co[i]
    else:
      co = list(self.co)
      co[0] += p
    return type(self)(co)

  __radd__ = __add__

  def __mul__(self, p):
    if isinstance(p, Polynomial):
      co = []
      for i in xrange(len(self.co) + len(p.co) - 1):
        s = 0
        for j in xrange(max(0, i - len(p.co) + 1), min(i + 1, len(self.co))):
          s += self.co[j] * p.co[i - j]
        co.append(s)
      return type(self)(co)
    else:
      return type(self)([x * p for x in self.co])

  __rmul__ = __mul__

  def __pow__(self, n):
    assert isinstance(n, int)
    assert n >= 0
    if n == 0:
      return type(self)([1])
    co = type(self)(self.co)
    while n > 1:
      co = co * self
      n -= 1
    return co

  #__rpow__ = __pow__

  def __sub__(self, p):
    return self + (-1) * p

  def __rsub__(self, p):
    return (-1) * self + p

x = Polynomial([0, 1])

# --- Test

for p in (x * (x + 1) * (2 * x + 1),
          (x + 1) * (1 + x),
          (x + 2) * (2 + x),
          (x + 3) ** 5,
          0 * x,
          x ** 2 - 2 * x + 1,
          (x + 3) + (3 * x ** 2 + 6) + x ** 4):
  print 'p = ', p
  print 'p(0) = ', p(0)
  print 'p(7) = ', p(7)
  print

Similar code for symbolic calculation with polynomials in Perl.