2009-07-29

BibTeX cheat sheet and entry templates

See the most up-to-date version of the cheat sheet at http://www.inf.bme.hu/~pts/bibtex-cheat-sheet.txt . A quick copy:
* Based on information form http://en.wikipedia.org/wiki/BibTeX
* See also: http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibstyles.html
* Nonstandard entries: url=, annote=, crossref=,
* No url= field in standard types, should be put to note={URL \url{...}}.
* There are no comments in BibTeX .bib files, not even %
* ?NAME= describes an optional field.
* BibTeX silently ignores ignores a field whose name it doesn't know.

.bib file entry templates
"""""""""""""""""""""""""
@article{NAME,
author={},
title={},
journal={},
year=,
?volume={},
?number={},
?pages={},
?month=,
?note={},
?key={},
}

@book{NAME,
author/editor={},
title={},
publisher={},
year=,
?volume={},
?series={},
?address={},
?edition={},
?month=,
?note={},
?key={},
?pages={},
}

@booklet{NAME,
title={},
?author={},
?howpublished={},
?address={},
?month=,
?year=,
?note={},
?key={},
}

@conference{NAME,
author={},
title={},
booktitle={},
year=,
?editor={},
?pages={},
?organization={},
?publisher={},
?address={},
?month=,
?note={},
?key={},
}

@inbook{NAME,
author/editor={},
title={},
chapter/pages={},
publisher={},
year=,
?volume={},
?series={},
?address={},
?edition={},
?month=,
?note={},
?key={},
}

@incollection{NAME,
author={},
title={},
booktitle={},
year=,
?editor={},
?pages={},
?organization={},
?publisher={},
?address={},
?month=,
?note={},
?key={},
}

@inproceedings{NAME,
author={},
title={},
booktitle={},
year=,
?editor={},
?pages={},
?organization={},
?publisher={},
?address={},
?month=,
?note={},
?key={},
}

@manual{NAME,
title={},
?author={},
?organization={},
?address={},
?edition={},
?month=,
?year=,
?note={},
?key={},
}

@mastersthesis{NAME,
author={},
title={},
school={},
year=,
?address={},
?month=,
?note={},
?key={},
}

@misc{NAME,
?author={},
?title={},
?howpublished={},
?month=,
?year=,
?note={},
?key={},
}

@phdthesis{NAME,
author={},
title={},
school={},
year=,
?address={},
?month=,
?note={},
?key={},
}

@proceedings{NAME,
title={},
year=,
?editor={},
?publisher={},
?organization={},
?address={},
?month=,
?note={},
?key={},
}

@techreport{NAME,
author={},
title={},
institution={},
year=={},
?type={},
?number={},
?address={},
?month=,
?note={},
?key={},
}

@unpublished{NAME,
author={},
title={},
note={},
?month=,
?year=,
?key={},
}

Description of entries and fields
"""""""""""""""""""""""""""""""""
* @article: An article from a journal or magazine.
* @book: A book with an explicit publisher.
* @booklet: A work that is printed and bound, but without a named publisher or
sponsoring institution.
* @conference: The same as inproceedings, included for Scribe compatibility.
* @inbook: A part of a book, usually untitled. May be a chapter (or section or
whatever) and/or a range of pages.
* @incollection: A part of a book having its own title.
* @inproceedings: An article in a conference proceedings.
* @manual: Technical documentation.
* @mastersthesis: A Master's thesis.
* @misc: For use when nothing else fits.
* @phdthesis: A Ph.D. thesis.
* @proceedings: The proceedings of a conference.
* @techreport: A report published by a school or other institution, usually
numbered within a series.
* @unpublished: A document having an author and title, but not formally
published.
* address=: Publisher's address (usually just the city, but can be the full
address for lesser-known publishers)
* annote=: An annotation for annotated bibliography styles (not typical)
* author=: The name(s) of the author(s) (in the case of more than one author,
separated by and)
* booktitle=: The title of the book, if only part of it is being cited
* chapter=: The chapter number
* crossref=: The key of the cross-referenced entry
* edition=: The edition of a book, long form (such as "first" or "second")
* editor=: The name(s) of the editor(s)
* eprint=: A specification of an electronic publication, often a preprint or a
technical report
* howpublished=: How it was published, if the publishing method is nonstandard
* institution=: The institution that was involved in the publishing, but not
necessarily the publisher
* journal=: The journal or magazine the work was published in
* key=: A hidden field used for specifying or overriding the alphabetical order
of entries (when the "author" and "editor" fields are missing). Note that
this is very different from the key (mentioned just after this list) that is
used to cite or cross-reference the entry.
* month=: The month of publication (or, if unpublished, the month of creation).
Example 1=: month=jan. Example 2=: month="17~" # feb.
* note=: Miscellaneous extra information
* number=: The "number" of a journal, magazine, or tech-report, if applicable.
(Most publications have a "volume", but no "number" field.)
* organization=: The conference sponsor
* pages=: Page numbers, separated either by commas or double-hyphens. For books,
the total number of pages.
* publisher=: The publisher's name
* school=: The school where the thesis was written
* series=: The series of books the book was published in (e.g. "The Hardy Boys"
or "Lecture Notes in Computer Science")
* title=: The title of the work
* type=: The type of tech-report, for example, "Research Note"
* url=: The WWW address
* volume=: The volume of a journal or multi-volume book
* year=: The year of publication (or, if unpublished, the year of creation)

2009-07-11

How to create a bootable CD running GRUB4DOS on Linux (Ubuntu Hardy)

GRUB4DOS is a flexible, feature-extended version of the GRUB boot manager. This blog post explains how to create a bootable CD which boots GRUB4DOS, from which you can boot almost anything. To create a bootable floppy instead, see http://ptspts.blogspot.com/2009/07/how-to-create-bootable-floppy-running.html.

Run this shell script to create the CD image grldr.iso:
#! /bin/bash
# by pts@fazekas.hu at Sat Jul 11 16:24:31 CEST 2009
GRUB4DOS_ZIP=grub4dos-0.4.4-2009-06-20.zip
GRUB4DOS_DIR=grub4dos-0.4.4 # as extracted from $GRUB4DOS_ZIP
set -ex
TO_INSTALL=''
type -p mkisofs || TO_INSTALL="$TO_INSTALL mkisofs"
type -p wget || TO_INSTALL="$TO_INSTALL wget"
type -p unzip || TO_INSTALL="$TO_INSTALL unzip"
test "$TO_INSTALL" && sudo apt-get install $TO_INSTALL
wget -O "$GRUB4DOS_ZIP" http://download.gna.org/grub4dos/"$GRUB4DOS_ZIP"
unzip -o "$GRUB4DOS_ZIP"
test "$GRUB4DOS_DIR"/grldr
rm -rf grldr.iso.dir
mkdir grldr.iso.dir
cp "$GRUB4DOS_DIR"/{grldr,menu.lst} grldr.iso.dir
mkisofs -R -b grldr -no-emul-boot -boot-load-size 4 -o grldr.iso grldr.iso.dir
: All OK, CD image grldr.iso created.
Please note that you may want to adjust the GRUB4DOS_ZIP variable above in order to download a more recent version of GRUB4DOS, when its available. you can get the list of versions from http://download.gna.org/grub4dos/.

You may use the CD image grldr.iso in your virtualization software (such as VirtualBox), or you may burn it to a CD with:
$ sudo apt-get install growisofs
$ growisofs -dvd-compat -Z /dev/cdrom=grldr.iso

How to create a bootable floppy running GRUB4DOS on Linux (Ubuntu Hardy)

GRUB4DOS is a flexible, feature-extended version of the GRUB boot manager. This blog post explains how to create a bootable floppy which boots GRUB4DOS, from which you can boot almost anything. To create bootable CD instead, see http://ptspts.blogspot.com/2009/07/how-to-create-bootable-cd-running.html.

Run this shell script to create the 1.44MB floppy image grldr.img:
#! /bin/bash
# by pts@fazekas.hu at Sat Jul 11 11:02:14 CEST 2009
GRUB4DOS_ZIP=grub4dos-0.4.4-2009-06-20.zip
GRUB4DOS_DIR=grub4dos-0.4.4 # as extracted from $GRUB4DOS_ZIP
set -ex
TO_INSTALL=''
type -p wget || TO_INSTALL="$TO_INSTALL wget"
type -p mformat || TO_INSTALL="$TO_INSTALL mtools"
type -p unzip || TO_INSTALL="$TO_INSTALL unzip"
test "$TO_INSTALL" && sudo apt-get install $TO_INSTALL
wget -O "$GRUB4DOS_ZIP" http://download.gna.org/grub4dos/"$GRUB4DOS_ZIP"
unzip -o "$GRUB4DOS_ZIP"
dd if=/dev/zero of=grldr.img bs=1474560 count=1
mformat -i grldr.img -f1440 ::
"$GRUB4DOS_DIR"/bootlace.com --floppy --chs grldr.img
mcopy -i grldr.img "$GRUB4DOS_DIR/grldr" ::GRLDR
mcopy -i grldr.img "$GRUB4DOS_DIR/menu.lst" ::menu.lst
: All OK, 1.44MB floppy image grldr.img created.
Please note that you may want to adjust the GRUB4DOS_ZIP variable above in order to download a more recent version of GRUB4DOS, when its available. you can get the list of versions from http://download.gna.org/grub4dos/.

You may use the floppy image grldr.img in your virtualization software (such as VirtualBox), or you may write it to a floppy disk with:
$ cat grldr.img >/dev/fd0

2009-06-21

How to configure dynamic DNS (dyndns) with OpenWRT and LuCI

This blog post tells you how to configure dynamic DNS (using dyndns.org) on your router running OpenWRT Kamikaze. The instructions havene be tested on a Linksys WRT54GL running OpenWRT Kamikaze (r14417).

Go to the menu System / Software. If you don't see package descriptions, click on the link Update package lists. Make sure you don't have the obsolete packages updatedd or ipupdate or ez-ipupdate. (You'll still find some old, obsolete forum posts recommending them on the net.) Install package luci-app-ddns.

Reboot (power-cycle) the router, or run rm -rf /var/luci-* in the SSH prompt. Without this step, the menu Services / Dynamic DNS doesn't appear.

Go to the menu Services / Dynamic DNS. Adjust your settings. The package supports dyndns.org, changeip.com, zoneedit.com, no-ip.com, and freedns.afraid.org . Please note that if you specify a force update interval less then 28 hours (the default is 72 hours), then dyndns.org will ban you. Click on the button Save & Apply.

Reboot (power-cycle) the router, or run INTERFACE=wan ACTION=ifup sh /etc/hotplug.d/iface/25-ddns in the SSH prompt. Without this step, the updater daemon script (/bin/sh /usr/lib/ddns/dynamic_dns_updater.sh myddns 0) is not started. This script sends the IP address update to the dynamic DNS provider. Wait a few minutes for the DNS update to take effect, then try to ping your new host name.

2009-06-15

How to render SVG with sharp text and manual kerning

If you want to render SVG vector graphics with text (i.e. convert it to a PNG), you might experience some problems: 1. the antialiasing glitch: the edges of the characters (even horizontal and vertical edges) are blurry; 2. the manual kerning glitch: if you specify manual kerning (by pressing Alt-Left or Alt-Right between two characters) in Inkscape, this gets ignored upon rendering. This blog post gives a solution to these problems.

To get rid of the antialiasing glitch, make sure your text is a text object, not a path. To check this, try to edit it with the text tool of Inkscape. If you can edit the text, then it's a text object. (If you have your text as path, you'll get the antialiasing glitch.) Then render the SVG with rsvg or The GIMP. There will be no antialiasing glitch in the output PNG. (If you render by exporting from Inkscape 0.46 or earlier, you'll get the antialiasing glitch. We tested it on Linux and Windows.)

To get rid of the manual kerning glitch, export the SVG to PNG in Inkscape. However, this introduces the antialiasing glitch. To get rid of both, you should use a patched rsvg, i.e. downloading, patching, compiling and installing rsvg for yourself. Get the patch from http://code.google.com/p/pts-mini-gpl/source/browse/#svn/trunk/pts-librsvg-manual-kerning-patch. As of now, the patch is for librsvg-2.26.0, but it is reported to apply cleanly to librsvg-2.22.3 as well. If you use Ubuntu Intrepid on an x86 system, you can get .deb packages from the attachments on https://www.epointsystem.org/trac/vending_machine/wiki/HotSpotLogo.

An example rendered image which avoids both problems is visible on https://www.epointsystem.org/trac/vending_machine/wiki/HotSpotLogo .

2009-06-13

How to make Ghostscript embed a Type 1 font to the PDF in one part, without splitting

I was facing a problem with Ghostscript 8.54 -sDEVICE=pdfwrite. When embedding a Type 1 font to the PDF, Ghostscript splitted the font to two font objects (and the corresponding FontDescriptor and CFF stream objects as well), instead of generating only one font object. This blog post explains how I solved the problem.

Please note that the problem didn't happen with a newer Ghostscript (such as GPL Ghostscript 8.61 (2007-11-21), it always generated a single PDF font object.

In a few hours I was able to reduce the source font to two glyphs, Ghostscript still splitting it to two 1-glyph fonts. Then I had the idea to change the /Encoding array, putting the two glyph names next to each other. It solved the problem, Ghostscript didn't split the font anymore. In fact, it turned out that one of the glyphs was missing from the /Encoding, and once I added it (anywhere), the problem was solved.

See the source PostScript file on http://www.math.bme.hu/~pts/twob.ps.txt . It contains the 2-glyph font, demonstrates the problem and shows the solution as well (when compiled with -dfill_encoding=true).

2009-06-12

Using \romannumeral in TeX to do multiple macro expansions

This blog post demonstrates a TeX macro hack: using \romannumeral to expand many macros in a single expansion. The example goal is to define a macro \stars which (when called properly) expands to the specified number of stars as a single expansion. Using a single expansion only makes the macro work in an \expandafter\def...{...} context. The similar macro given in The TeXbook doesn't work in this context because it needs multiple expansions.
\documentclass{article} 
%** Usage: \romannumeral\stars{NUMSTARS}
%** This expands to a NUMSTARS stars (* tokens) in a single expansion step.
\def\stars#1{%
\expandafter\mtostar\expandafter{\expandafter}\romannumeral\number#1 000z}
\def\firstoftwo#1#2{#1}%
\def\secondoftwo#1#2{#2}%
\def\mtostar#1#2{%
\ifx#2z\expandafter\firstoftwo\else\expandafter\secondoftwo\fi
{0 #1}{\mtostar{#1*}}%
}
\begin{document}
\expandafter\def\expandafter\mystars\expandafter{\romannumeral\stars{4}}%
\texttt{(\meaning\mystars)}
\end{document}

2009-06-09

How to typeset text with bilingual fragments with LaTeX

This post gives a possible solution for typesetting two similar versions of the same LaTeX document differing only in some text fragments.

Here is the code:
\documentclass{article}

\makeatletter
\newif\ifsecondlanguage
\def\@bracedfirst#1#2{{#1}}
\def\@bracedsecond#1#2{{#2}}

\def\do@localized#1[#2]{%
\let\@@localized\@undefined
\newcommand\@@localized[#2]{#1}\@@localized}%

%** Typeset different text depending on the current value of \ifsecondlanguage.
%** Usage: \bilingual{LANG1PAT}{LANG2PAT}[ARGCOUNT]{ARG1}{ARG2}...
%** If \ifsecondlanguage is true, pattern LANG2PAT is used, else LANG1PAT is
%** used. Arguments #1, #2 etc. in the pattern are substituted with
%** ARG1, ARG2 etc.
\def\bilingual{%
\expandafter\expandafter\expandafter\do@localized\csname
@braced\ifsecondlanguage second\else first\fi\endcsname
}

\begin{document}

\secondlanguagetrue
\bilingual
{This is addition: #1; and this is multiplication: #2.}
{Translation to another language: #1 and another translation: #2.}
[2]{$x+y$}{$x*y$}

\secondlanguagefalse
\bilingual
{These are additions: #1, #2; and this is multiplication: #3.}
{Translation to another language: #1, #2 and another translation: #3.}
[3]{$x+y$}{$x\oplus y$}{$x*y$}

\end{document}
Select the language to use with \secondlanguagetrue or \secondlanguagefalse. Whenever you have a text fragment which needs to be typeset differently depending on the languagem, use the \bilingual command, as demonstrated above.

2009-06-01

How to migrate or merge a CVS or SVN repository to a remote SVN repository

Let's suppose you have one or more local CVS and/or SVN repositories, and you want to merge them (with their change history) to new directories of an existing, possibly remote target SVN repository. This blog post explains how to do this using Unix tools.

The following tools will be used:Please note that we won't use svnsync, because it requires the target repository to be empty. We won't use svn-merge-repos.pl much either, because it requires the target repository to be local. We won't use svnadmin load much either, because it requires the target repository to be local.

Access remote repositories for the first time

For some repositories, you have to specify your username (usually in the command line) and password (usually answering to an interactive prompt) in order to be able to connect. For each machine, you have to do it once, because SVN records your username and password to files under $HOME/.subversion/auth. To avoid problems connecting later, make sure you access the the remote repositories on each machine you'll be working on, so your credentials get saved. The easiest way to do it is to run
svn ls URL://TO/SVN/REPOSITORY --username MYUSER

Install svn-pusher

svn-pusher can add any SVN repository to another SVN repository (as a subdirectory) no matter local or remote, keeping the commit history of the specified revision interval. You can skip this installation step now and come back only if you are asked to use svn-pusher in some of the steps below.

svn-pusher is implemented as a Perl script using SVN's Perl bindings (SVN::Core). The easiest way to install svn-pusher is with root access on a Unix system. For example, on Debian or Ubuntu, run this as root:
# apt-get update
# apt-get install libsvn-core-perl
# echo no | cpan -i SVN::Pusher # this may take about a minute
# type -p svn-pusher
/usr/local/bin/svn-pusher
# svn-pusher help
For the sake of completeness we mention (but we recommend against using) svn-push as an alternative of svn-pusher. svn-push is a tool written in C, available in the SVN contrib directory, and it does something like svn-pusher, but it's dumber: it can commit only a single revision at once, and it needs the head revision number. Here is how to compile it:
$ wget http://svn.collab.net/repos/svn/trunk/contrib/client-side/svn-push/svn-push.c
$ sudo apt-get install libsvn-dev
$ gcc -W -Wall -I/usr/include/subversion-1 -I/usr/include/apr-1.0 \
-Doff64_t='unsigned long long' -o svn-push svn-push.c -lsvn_client-1
$ ./svn-push
Usage : svn-push -r N:M SRC_URL DEST_URL
For the sake of completeness, we also mention the SVN::Push Perl module, which provides the svnpush command. SVN::Pusher seems to be more up-to-date.

Convert the CVS repository to an SVN repository dump

You can skip this step if your source repository is not a CVS repository.

A CVS repository is a directory containing *,v files (possibly in subdirectories), and containing a directory named CVSROOT (or one of its parents containing the CVSROOT).

Download and install cvs2svn from here. It's a Python script, so install Python as well (2.4 or 2.5 should be OK). You don't need root access to run cvs2svn; in fact, it can be run as extracted from the tarball. Example:
$ wget http://cvs2svn.tigris.org/files/documents/1462/44372/cvs2svn-2.2.0.tar.gz
$ tar xzvf cvs2svn-2.2.0.tar.gz
$ cvs2svn-2.2.0/cvs2svn --help
You don't need Subversion itself for this step – cvs2svn (if run with --dumpfile=) needs only Python and the standard Unix sort utility.

Make sure you have your CVS repository on the same machine as cvs2svn. (Copy with scp -r or rsync if necessary.) It is a good and safe idea to make a copy and to use it for the purpose of the conversion. Make sure you have a neighbor or parent directory named CVSROOT next to the repository. The CVSROOT directory can be empty. Make sure that all files in your CVS repository directory are named *,v (i.e. their name ends with ,v). If you don't need all files or all directories, feel free to remove them now. The effect would be as if those files and/or directories have never been added to the CVS repository.

Run cvs2svn --dumpfile=PROJECT.dump PATH/TO/CVS/REPOSITORY . This creates the file PROJECT.dump, which contains all files in the CVS repository, with their full commit history.

Convert the SVN repository to an SVN repository dump

You can skip this step if your source repository is not a SVN repository.

If your source repository is local and you have read access to it, just run
svnadmin dump PATH/TO/SVN/REPOSITORY >PROJECT.dump
Otherwise use svn-pusher like this:
$ svnadmin create PROJECT.copy
$ svn-pusher push URL://OF/SVN/REPOSITORY PROJECT.copy
$ svnadmin dump PROJECT.copy >PROJECT.dump # this may take some time
$ rm -rf PROJECT.copy
As an alternative to svn-pusher, you can use svnsync as well (part of the standard SVN installation), see blog post Dump a SVN repository from a URL how to do it. Please note that both svn-pusher and svnsync are quite slow (as compared to svnadmin dump), and svnsync is better supported and documented since it is part of standard SVN.

Once you have your PROJECT.dump file, use svndumpfilter (part of standard SVN) to get rid of the unnecessary files and directories. You may also edit the file in a text editor to do some other modifications (such as renaming files). The file format should be self-explanatory.

If you expect a file name conflict between the repositories (between source1 and source2 or source1 and target), e.g. multiple repositories have trunk/version.h, it is safest to move/rename all the source repositories to their own directories, and once the merge is done, do a careful and safe svn mv. Here is how to rename everything (e.g. from DIR/TO/FILE to PROJECT.merge/DIR/TO/FILE) in a *.dump file:
$ perl -pi -e's@^(Node-path: )@${1}PROJECT.merge/@' PROJECT.dump
After that, please make sure you add the directory creation into revision 1 of PROJECT.dump. Use your text editor to insert the following lines just below the first PROPS-END line:
Node-path: PROJECT.merge
Node-kind: dir
Node-action: add

Merge an SVN repository dump to a local target SVN repository

You can skip this step if the target repository is not local.
The contents of the dump file PROJECT.dump can be added to an existing local target SVN repository using svnadmin load PATH/TO/TARGET/SVN/REPOSITROY <PROJECT.dump . An example for creating a new target SVN repostiory, and adding multiple projects to it:
$ svnadmin create myprojects
$ cvs2svn --dumpfile=myproject1.dump cvsrepo/dir/myproject1
$ svnadmin load myprojects <myproject1.dump
$ cvs2svn --dumpfile=myproject2.dump cvsrepo/dir/myproject2
$ # (edit myproject2.dump, see below)
$ svnadmin load myprojects <myproject2.dump
$ svn ls -R file://$PWD/myprojects
Please note that cvs2svn adds the creation of the directories trunk, tags and branches to the *.dump file. This will be a problem in svnadmin load myprojects <myproject2.dump, because this tries to add directory trunk, which already exists in repository mpyrojects, so the operation will fail. The solution is to edit the file myproject2.dump, and remove the following lines from near the beginning:
Node-path: trunk
Node-kind: dir
Node-action: add


Node-path: branches
Node-kind: dir
Node-action: add


Node-path: tags
Node-kind: dir
Node-action: add

Merge an SVN repository dump to a target SVN repository

This steps works for both local and remote target SVN repositories, but it's a lot slower than the svnadmin load method described above (which works only if the target SVN repository is local). Install svn-pusher (see the step for it above). It doesn't matter which machine you install svn-pusher to as long as it can connect to the target repository. Copy PROJECT.dump created above to the machine you've installed svn-pusher to. Then run
$ svnadmin create PROJECT.copy
$ svnadmin load PROJECT <PROJECT.dump
$ svn-pusher push file://$PWD/PROJECT.copy URL://TO/TARGET/SVN/REPOSITORY # slow
$ rm -rf PROJECT.copy
Please note that revision numbers in messages reported by svn-pusher are usually off by one, e.g. Committed revision 1 from revision 0. This is normal, the revision numbers will match perfectly in the target repository.

You can use svn-pusher multiple times on the same target repository to merge multiple source repositories. If the target repository is not empty by the time you start svn-pusher, it is safest to dump it first (to have a backup), and then please pay attention to the following facts. svn-pusher (unlike svnadmin load) reports a warning if a directory (e.g. trunk) has already been added. This warning is usally harmless. If svn-pusher wants to add a file which already exists, it will skip merging that source revision, but it will proceed merging subsequent source revisions. This is not always what you want, because you may want to fix the conflict first by renaming files, and only then proceed with subsequent revisions. Should this happen, you may have to rebuild the target repository from scratch using the backup.

2009-05-31

Syntax highlighted Python

Example syntax highlighted Python code: (please ignore)
def Hello(self, x, y): 
return x * y + "foo" # bar
The syntax highlighting was done in the borland style on http://pygments.org/demo/.

2009-05-21

Ruby on Rails 2.3 unit tests

The blog post Ruby on Rails Unit Tests explains why it is a bad idea to give the unit tests access to the database, and how to set up your Rails project so that you get an exception if you try to access the database with ActiveRecord model objects from a unit test. Unfortunately, that blog post doesn't work with the latest Rails. This blog post explains how to set up a Ruby on Rails 2.3 project to disallow database accees in the unit tests. The instructions were tested with Rails 2.3.2.

Step 1. Create file lib/tasks/unit_tests.rake with the following contents:
Rake::Task[:'test:units'].prerequisites.delete('db:test:prepare')
Step 2. Create file test/unit_test_helper.rb with the following contents:
# similar to autogenerated test/test_helper.rb
#
fail "some of the unit tests has loaded test_helper.rb. Please change " +
"(require 'test_helper') to (require 'unit_test_helper') in " +
"tests/unit/**/*.rb" if $".include?('test_helper.rb')
fail "some of the unit tests has loaded test_help.rb. Please make sure that " +
"the first line is (require 'unit_test_helper') in tests/unit/**/*.rb" if
$".include?('test_help.rb')
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
HIDE_ActiveRecord = self.class.send(:remove_const, :ActiveRecord)
require 'test_help' # rails 2.3.2 standard module
ActiveRecord = self.class.send(:remove_const, :HIDE_ActiveRecord)

class FakeConnection
class InvalidActionError < StandardError
end
COLUMNS = {}
def self.columns(table_name, name=nil)
if COLUMNS.has_key?(table_name)
COLUMNS[table_name]
else
raise InvalidActionError, "please create something like this first: " +
"FakeConnection::COLUMNS[#{table_name.inspect}] = [ " +
"ActiveRecord::ConnectionAdapters::Column.new(name=..., " +
"default=nil, sql_type=\"text\", null=false), ...]"
end
end
DB_ERROR_MSG = 'You cannot access the database from a unit test'
def self.quote_table_name(*args) # called from ActiveRecord::Base.find
raise InvalidActionError, DB_ERROR_MSG, caller
end
def self.quote_column_name(*args) # called from ActiveRecord::Base.delete
raise InvalidActionError, DB_ERROR_MSG, caller
end
def self.select_all(*args) # called from ActiveRecord::Base.find_by_sql
raise InvalidActionError, DB_ERROR_MSG, caller
end
def self.transaction(*args) # called from ActiveRecord::Base.save
raise InvalidActionError, DB_ERROR_MSG, caller
end
end
class << ActiveRecord::Base
def connection
FakeConnection
end
end
Step 3. Make sure all your unit tests (i.e. =*.rb= files in =test/unit=, recursively) start with the line require 'unit_test_helper' instead of require 'test_helper'. Don't forget any of the =*.rb= files, otherwise you'll get an error message at test file load time.

Step 4. If you need access to your model objects, add the necessary column declaration to your test file. For example, for model class Foo:
FakeConnection::COLUMNS['foos'] = [
ActiveRecord::ConnectionAdapters::Column.new(
name="bar1", default=nil, sql_type="varchar(255)", null=false),
ActiveRecord::ConnectionAdapters::Column.new(
name="bar2", default=nil, sql_type="integer", null=false),
]
After this, you can do a Foo.new in your tests – but you won't be able to save the object, and Foo.find, Foo.find_by_sql and Foo.delete won't work either. If you try any of those, a FakeConnection::InvalidAction gets raised. To test such a functionality, use an integration test instead of a unit test. To do so, create your test file as test/integration/*.rb instead of test/unit/*.rb.

Once all steps are done, run your unit tests with rake test:units. This will run all tests in all *.rb files in the test/units directory (recursively). If you get an exception, and you need the full backtrace, rerun with BACKTRACE=1 rake test:units. For an even longer backtrace, run rake --trace test:units.

2009-05-14

Java .class file converter which makes method and fields public available

ClassPublic.Java is a Java .class file converter which makes a Java class public and non-final, its fields public and its methods public and non-final. ClassPublic.java has a compact implementation, depending only on a J2SE 1.5.0 or newer. ClassPublic.java can be used to increase interoperability and code reusability of a .class file whose .java source is not available or it is not feasible to recompile.

Download classpublic-latest.zip.

2009-05-13

Java class disassembler with offset display available

jdisasm.py is a Java class file disassembler implemented as a Python script. jdisasm.py displays a java .class file in a human readable form, showing the class name, the field names and types, the method names, types and codes (including instruction mnemonics). For each item shown, the file offset is prepended. (Neither javap or jad can display the file offset.)

Download jdisasm-latest.tar.gz.

2009-05-05

How to deploy a Ruby on Rails application using Phusion Passenger and Rails Enterprise Edition on Debian Etch or Ubuntu Hardy

This post gives step-by-step instructions on deploying a Ruby on Rails application on a Linux-based server using Phusion Passenger and Ruby Enterprise Edition. The instructions are for a Debian Etch system, but they should work with minor modifications on any Linux system based on Debian or Ubuntu (including Debian Etch Mini, available as Minimal Gnome Desktop from http://www.visoracle.com/download/debian/.

It doesn't matter if you have Ruby installed on the server, because you'll compile and install Ruby Enterprise Edition (REE) 1.8.6 into its own directory, and run the deployed Rails application using REE. REE is Ruby with some garbage collection improvements to allow memory page reuse across fork()ed subprocesses.

So the Ruby interpreter, Ruby packages and Ruby gems already installed to the system won't be used or needed. But some libraries (such as SQLite client and MySQL client) will be used. You have to install those libraries including the development packages using the system's package manager. Example:
# apt-get install libsqlite3-dev
# apt-get install libpq-dev # optional, needed if Rails app connects to PostgreSQL
# apt-get install libmysqlclient15-dev # needed if app connects to MySQL
You also have to install some development packages in order to be able to compile and install REE. Do this:
# apt-get install wget gcc g++ make libc6-dev libreadline5-dev zlib1g-dev libssl-dev
Should other packages be missing, the installer script run below will tell you the apt-get install command to run.

You can get the REE installer tarball from http://www.rubyenterpriseedition.com/. Feel free to substitute any newer ruby-enterprise-*.tar.gz. filename to the command below. To download and install REE (including a bundled Ruby, rubygems, Rake, Rails and Ruby client modules for SQLite 3, MySQL and PostgreSQL), run this:
# cd /usr/src
# wget http://rubyforge.org/frs/download.php/55511/\
ruby-enterprise-1.8.6-20090421.tar.gz
# tar xzvf ruby-enterprise-1.8.6-20090421.tar.gz
# ruby-enterprise-1.8.6-20090421/installer \
-c--enable-pthread -a/usr/local/ruby-enterprise-1.8.6
# strip /usr/local/ruby-enterprise-1.8.6/bin/ruby
The installer is an interactive script, and it would have asked questions and waited for you to press Enter occasionally if you hadn't called it with the -a flag. Nevertheless, it show you some nice, colorful messages indicating progress. In a few minutes, it finishes compilation and installation to /usr/local/ruby-enterprise-1.8.6. Please note that the installer runs /usr/local/ruby-enterprise-1.8.6/gem install to download, compile and install some gems (Ruby modules). Please note that -c--enable-pthread is necessary to work around a bug in ruby-enterprise-1.8.6-20090421.tar.gz, which causes fork()ed subprocesses to exit early with SIGVTALRM.

Scroll up and check that the MySQL and PostgreSQL Ruby client modules were installed properly. If not (and you need those), then apt-get install the necessary libraries, and run the installer or gem install again. Example command lines (optional, needed only if you need MySQL or PostgreSQL and the installer failed the compile the relavant Ruby modules):
# apt-get install libmysqlclient15-dev
# /usr/local/ruby-enterprise-1.8.6/bin/gem install --no-ri --no-rdoc mysql

# apt-get install libpq-dev
# /usr/local/ruby-enterprise-1.8.6/bin/gem install --no-ri --no-rdoc postgres
If you have any other gems needed by your Rails application, install them now (similarly to the postgres gem above). Please note that all gems should be installed as root in the setup described in this tutorial, and gems are shared among Rails applications.

Create a user (non-root) and download your Rails application to the user's home (e.g. /home/myrails/myapp). The best way to download the application is the checkout function of a version control system (such as Subversion), because using that you can easily re-deploy the application later. (Please note that Capistrano can be used to fully automate Rails application deployment, but this is beyond the scope of this tutorial.) If you have multiple Rails applications, you can create as many users as you want, and map applications to users freely. The application would run in production as the UID of that user (myrails). As myrails, add these lines to /home/myrails/.bash_profile:
export GEM_HOME=/usr/local/ruby-enterprise-1.8.6/lib/ruby/gems/1.8
export PATH="/usr/local/ruby-enterprise-1.8.6/bin:$PATH"
Log in again, and check that type -p ruby gem rake rails prints a filename inside /usr/local/ruby-enterprise-1.8.6. Make sure that all scripts in /home/myrails/myapp/scrit/* start with /usr/bin/env ruby, and they don't have a specific Ruby interpreter (such as /usr/bin/ruby) hardcoded. Create the initial database of your Rails application (by running rake db:migrate etc.). Quickly try your application by starting
/home/myrails/myapp/script/server --environment=production
and visiting http://localhost:3000/ . Try some functionality which accesses the database. As soon as it works fine, stop the server script.

Now it is time to install Apache2 (please note that Apache 1.3 won't suffice, because Phusion Passenger, the Rails--Apache connector this tutorial uses needs Apache 2). Run this:
# apt-get install apache2 apache2-prefork-dev libapr1-dev
# /usr/local/ruby-enterprise-1.8.6/bin/passenger-install-apache2-module -a
The passenger-install script gives you nice, colorful instructions how to configure your Apache. Memorize those instructions, and feel free to use them in place of the instructions given in this tutorial. Configure your Apache2 as usual. (Setting up and populating a DocumentRoot, setting up SSL (https://) and setting up Apache VirtualHost entries is not covered in this tutorial.) Create file /etc/apache2/mods-available/passenger.conf containing the right paths, for example (type it without the line break):
PassengerRoot /usr/local/ruby-enterprise-1.8.6/lib/ruby/gems/1.8/gems
/passenger-2.2.2
PassengerRuby /usr/local/ruby-enterprise-1.8.6/bin/ruby
Crete file /etc/apache2/mods-available/passenger.load containing the right paths, for example (type it without the line break):
LoadModule passenger_module /usr/local/ruby-enterprise-1.8.6/lib/ruby/gems/
1.8/gems/passenger-2.2.2/ext/apache2/mod_passenger.so
Run this:
# ln -s ../mods-available/passenger.load /etc/apache2/mods-enabled/
# ln -s ../mods-available/passenger.conf /etc/apache2/mods-enabled/
Restart apache with
# /etc/init.d/apache2 restart
Wait about 60 seconds, and make sure you don't get any Passenger-related error messages at the end of /var/log/apache2/error.log.

If you want to deploy your Rails application to the root URI of an Apache VirtualHost, all you have to do is specifying
<VirtualHost server.name:80>
ServerName server.name
DocumentRoot /home/myrails/myapp/public
</VirtualHost>
in your /etc/apache2/sites-available/default. Do not specify RailsBaseUrl /, that wouldn't work. Restart Apache, and visit http://server.name/ . The first page download may take a few seconds, because Phusion Passenger starts the Rails application at that time. The latency of further downloads should be negligible. If you get a colorful, but unhelpful error message like The page you were looking for doesn't exist.; You may have mistyped the address or the page may have moved., then visit http://localhost/ instead, on which Phusion Passenger doesn't hide the exception raised by Rails. Examining /var/log/apache2/error.log can also help you diagnose the problem.

If you want to deploy your Rails application to a non-root URI (e.g. http://server.name/myapp), then first make sure that the necessary workarounds are applied. Follow the instructions on http://ptspts.blogspot.com/2009/05/how-to-fix-railsbaseuri-sub-uri-with.html of adding files /home/myrails/myapp/config/initializers/!fix_relative_url_root.rb and /home/myrails/myapp/config/initializers/!relative_url_for.rb. Make sure those files are added to your source code repository as well, if applicable. Then create a symlink to your application's public directory. Assuming that your Apache2 DocumentRoot is /var/www, create the symlink by running
# ln -s /home/myrails/myapp/public /var/www/myapp
Make sure you have these lines in your Apache2 configuration (most probably /etc/apache2/sites-available/default):
DocumentRoot /var/www
<Directory /var/www>
Options +FollowSymlinks
</Directory>
RailsBaseUri /myrails
It's OK to have multiple RailsBaseUri directives, both for multiple or a single application. Restart Apache2 if needed. Visit http://server.name/myrails . If you see an unhelpful error message instead of your application's main page, then visit http://localhost/myrails to get the details. Examining /var/log/apache2/error.log can also help you diagnose the problem.

Please note that if you have mutiple Rails applications (i.e. multiple public directories (after following symlinks)), Phusion Passenger will not
reuse the same Ruby worker process for running multiple applications. Please also note that Ruby worker processes spawned by Phusion Passenger are single-threaded. Phusion Passenger treats two Rails applications the same if they have the same public directory (after following symlinks), so the http:// and https:// versions and aliases of your Rails application may run on the same Ruby worker process.

Rails applications running under Phusion Passenger's control in the production environment don't pick up file changes automatically. So if you change your application's code or config, you have to restart the application: run (as myrails)
# touch /home/myrails/myapp/tmp/restart.txt
and visit the application's URL. The first download should take a few seconds, because Phusion Passenger is restarting your Rails application.

Please refer to Phusion Passenger's documentation for more information about running your Rails application with Phusion Passenger with an Apache2 frontend.

2009-05-04

How to fix RailsBaseUri (sub URI) with Phusion Passenger without setting relative_url_root

There is a bug which prevents correct routing with Phusion Passenger 2.2.2 and Rails 2.3.2 (>= 2.2.2) and RailsBaseUri. For example, if you have RailsBaseUri myapp, and you visit http://localhost/myapp/mycontroller/myview, then the request gets routed to controller myapp instead of mycontroller, and you most probably get the error message Routing Error; No route matches "/myapp/mycontroller/myview" with {:method=>:get}. If you visit http://myserver/myapp/mycontroller/myview, you'll get the error message The page you were looking for doesn't exist. You may have mistyped the address or the page may have moved.. This post gives instructions how to work around the bug.

The bug is mentioned in various places, such as
http://blog.ashchan.com/archive/2008/12/10/passengers-railsbaseuri-not-working/ and https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/1946-setting-a-relative-root-url-via-a- and http://github.com/rails/rails/commit/a87462afcb6c642e59bfcd2e11e3351e2b718c38 .

The manual workaround (according to what is suggested on the pages above) is adding the line below to config/environments/production.rb:
config.action_controller.relative_url_root = '/myapp'
We propose a completely automatic workaround here, which doesn't need hardcoding URIs to the Rails application configuration. To fix this, create a file config/initializers/!fix_relative_url_root.rb with the following contents:
# automatic relative_url_root fix
# for Phusion Passenger 2.2.2 and Rails 2.3.2 (>= 2.2.2)
# by pts@fazekas.hu at Mon May 4 20:48:38 CEST 2009
# from http://ptspts.blogspot.com/2009/05/how-to-fix-railsbaseuri-sub-uri-with.html
fail unless ActionController::Request # check loaded
module ActionController
class Request
def initialize(env)
@env = env # Rack::Request#initialize does only this
path = request_uri.to_s[/\A[^\?]*/]
sn = @env['SCRIPT_NAME']
if (RAILS_ENV == 'production' and
(sn.empty? or sn.starts_with?('/')) and
path == sn + @env['PATH_INFO'])
Base.relative_url_root = sn
end
end
end
end
In addition to the fix above, here is another fix for url_for and redirect_to so they automatically prepend ActionController::Base.relative_url_root when they get a string starting with a slash. To apply the fix, create file config/initializers/!relative_url_for.rb with the following contents:
# fix url_for and redirect_to to use ActionController::Base.relative_url_for
# fix for Rails 2.3.2
# by pts@fazekas.hu at Mon May 4 22:38:44 CEST 2009
# from http://ptspts.blogspot.com/2009/05/how-to-fix-railsbaseuri-sub-uri-with.html
fail unless ActionController::Base # check loaded
fail unless ActionView::Helpers::UrlHelper # check loaded
module ActionController
class Base
alias url_for__ptsroot__ url_for
def url_for(options = {})
options = Base.relative_url_root.to_s + options if
options.kind_of?(String) and options.starts_with?('/')
url_for__ptsroot__(options)
end
alias redirect_to__ptsroot__ redirect_to
def redirect_to(options = {})
options = Base.relative_url_root.to_s + options if
options.kind_of?(String) and options.starts_with?('/')
redirect_to__ptsroot__(options)
end
end
end
module ActionView
module Helpers
module UrlHelper
alias url_for__ptsroot__ url_for
def url_for(options = {})
return escape_once(
::ActionController::Base.relative_url_root.to_s + options) if
options.kind_of?(String) and options.starts_with?('/')
url_for__ptsroot__(options)
end
end
end
end

2009-04-22

How to fix a broken registry key if Windows XP is not booting

Recently I modified the Windows XP registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage\OEMCP from 437 to 65001, which resulted an immediate Windows XP system crash, and Windows didn't boot anymore, not even in safe mode. This post describes how I changed the key back without a working Windows on the machine.
  • I downloaded SystemRescueCD 1.1.7 from http://www.sysresccd.org/Download, burnt it to CD, booted the system from it.
  • I pressed Enter at the CD boot menu to boot with the default options.
  • I waited a few minutes for the SystemRescueCD Gentoo system to boot.
  • At the root@sysresccd /root % prompt (I pressed Alt-F2 at the help screen), I listed the partitions with the command fdisk -l. It turned out that my Windows XP partition was /dev/sda1.
  • I created a mount point directory with mkdir /mnt/, and mounted the Windows XP partition with mount -t ntfs-3g /dev/sda1 /mnt/p .
  • I edited the system registry with chntpw -e /mnt/p/WINDOWS/system32/config/system (see the transcript below).
  • I ensured that changes are written to disk sync.
  • I rebooted without the CD, and now my Windows XP worked.
Here is the transcript of the chntpw session:
chntpw version 0.99.5 070923 (decade), (c) Petter N Hagen
Hive name (from header):
ROOT KEY at offset: 0x001020 * Subkey indexing type is: 686c <lh>
Page at 0x54c000 is not 'hbin', assuming file contains garbage at end
File size 5767168 [580000] bytes, containing 1301 pages (+ 1 headerpage)
Used for data: 103727/5482832 blocks/bytes, unused: 2263/25616 blocks/bytes.

Simple registry editor. ? for help.

> ls
Node has 7 subkeys and 0 values
key name
<ControlSet001>
<ControlSet002>
<LastKnownGoodRecovery>
<MountedDevices>
<Select>
<Setup>
<WPA>

> cd \ControlSet001\Control\Nls\CodePage

\ControlSet001\Control\Nls\CodePage> cat OEMCP

Value <OEMCP> of type REG_SZ, data length 12 [0xc]
65001


\ControlSet001\Control\Nls\CodePage> ed OEMCP
EDIT: <OEMCP> of type REG_SZ with length 12 [0xc]
[ 0]: 65001

Now enter new strings, one by one.
Enter nothing to keep old.
[ 0]: 65001
-> 437
newkv->len: 8

\ControlSet001\Control\Nls\CodePage> q

Hives that have changed:
# Name
0 <system>
Write hive files? (y/n) [n] : y
0 <system> - OK
Please note that a similar procedure (with the exact same fdisk, mount, and chntpw comamnds) using the Knoppix 5.3.1 live CD instead of SystemRescueCD.

Please note that it is possible to edit any registry file with the Windows XP regedit. Here are some relevant links how to do this: http://smallvoid.com/article/winnt-offline-registry-edit.html; http://www.hardforum.com/showthread.php?t=1162302; http://www.911cd.net/forums//index.php?showtopic=7066.

Please note that it is possible to edit the registry with the regedit application shipping with the ERD Commander 2005 Windows XP live boot CD. Get the CD from http://www.fullandfree.info/software/erd-commander-2005/ . There is a Linux tool nrg2iso which can convert the .nrg file in the download for burning to CD. On Windows, it is possible to burn the .nrg file using ImgBurn.

Here are some methods which I tried, but they didn't work to edit the registry system registry file:
  • Using kregedit – it crashed opening the system registry file.
  • Running wine's /usr/bin/regedit – I wasn't able to specify which file to edit.
  • Running wine regedit.exe with the Windows XP regedit.exe – it failed to start up because it hasn't found some DLLs. I haven't bothered forcing it.

How to fix Ubuntu Hardy and Intrepid boot hang on a Philips laptop

If you have a Philips laptop and you want to boot Ubuntu Hardy and Intrepid on it, the system might hang indefinitely while booting (not reacting to any keypress, not even the Num Lock key). The reason for the hang may be the non-working driver for the notebook's built-in memory card reader (MMC). This post describes the solution: how to disable the driver for the memory card reader and thus get Ubuntu boot. (But you won't be able to use the memory card in Ubuntu.)

If you have another Linux boot CD at hand (such as Knoppix) which boots, and you are familiar with it, then boot it, open file /etc/modprobe.d/local on your Linux root partition, and add the line blacklist mmc_core . Reboot the machine to Ubuntu. Now it should boot normally, without hanging. By doing this you ensure that that the memory card reader driver kernel module won't ever be loaded in the future at boot time. This is a long-term solution, it survives kernel and distribution upgrades.

If you don't have an alternative boot CD at hand, then do the fixing like this. Reboot the system (power-cycle it if necessary). In the GRUB boot menu (a list of long lines starting with Ubuntu 8, press key E, in the new menu, select the line starting with kernel, press key E, press key Space, type init=/bin/bash rw , press key Enter, press key B. This will start the system boot process, giving you a shell soon. In a minute, you'll get a prompt ending with #, and the cursor blinking after that. Type bash -c 'echo blacklist mmc_core >>/etc/modprobe.d/local', press Enter. Type sync , press Enter. Wait a few seconds. Reboot the machine by pressing Ctrl-Alt-Del (or by any other means). Now Ubuntu should boot normally, without hanging.

If the instructions above don't work for you, here is a brute force, temporary fix, which won't survive some system upgrades. Do exactly as above, up to the # prompt. Then type rm -rf /lib/mmc , press Enter. Type mv -fT /lib/modules/`uname -r`/kernel/drivers/mmc /lib/mmc , press Enter. Continue with sync etc.

2009-04-18

How to set up an external monitor for watching movies with the nvidia proprietary x.org driver

This post describes how to set up your /etc/X11/xorg.conf on Linux if you have a laptop with an NVIDIA video card, and you want to attach an external monitor for watching movies with MPlayer. Please note that the external monitor is not for regular work, so normally you don't want to put windows there, or use the mouse or keyboard to interact with windows on the external monitor.

How to configure the X server

If you enable TwinView in /etc/X11/xorg.conf (possibly using the command nvidia-xconfig, you'll get one big X11 screen (DISPLAY=:0), with windows possibly spanning two two displays. This is not what you want now. What you want is to have two, independent X11 screens, the laptop's built-in LCD display (DISPLAY=:0.0) running your regular X11 session, and the external monitor (DISPLAY=:0.1) showing the movies played. (Sending audio to the external monitor's speakers is not described here.)

To configure the X.Org X11 server for this, you have to modify /etc/X11/xorg.conf the following way. You have to add a new "Monitor" section, a new "Device" section and a new "Screen" section. First make a backup of your config file: sudo cp /etc/X11/xorg.conf{,.single.good}. Then modify the config file like this:
# modifications to /etc/X11/xorg.conf

Section "Device"
# This is the original Section "Device"
Identifier "..."
Driver "nvidia"
Busid ...
Option ...
...
# ADD Screen 0.
Screen 0
EndSection

Section "Device"
Identifier "nvidia1"
Driver "nvidia"
# COPY Busid from original
Busid ...
# COPY Option(s) from original
Option ...
# ADD Screen 1.
Screen 1
EndSection

# ADD this Monitor section.
# The specified HorizSync and VertRefresh ranges are good for most
# external LCD monitors. You may want to widen them for your monitor.
Section "Monitor"
Identifier "monitor1"
Option "DPMS"
HorizSync 28-64
VertRefresh 43-60
EndSection

# ADD this Screen section.
Section "Screen"
Identifier "screen1"
Device "nvidia1"
Monitor "monitor1"
# If it doesn't work with 24, try changing to 32 (and below as well).
DefaultDepth 24
SubSection "Display"
Depth 24
# SET this to the preferred (maximum) resolution of your external monitor.
Modes "1920x1080"
EndSubSection
EndSection

# ADD or modify this ServerLayout section
Section "ServerLayout"
Identifier ...
# Multiple InputDevice entries are OK
InputDevice ...
...
# SET ... to the name of your original Section "Screen".
Screen 0 "..." 0 0
# MAKE sure that the number you specify is larger than
# the maximum width of your screens. Otherwise the mouse
# pointer may accidentally wrap one the edge of one screen
# to another.
Screen 1 "screen1" 1300 0
EndSection
Make sure that the position you specify for Screen 1 in ServerLayout is large enough, i.e. it is larger than the maximum width of your screens. http://users.tkk.fi/spniskan/switchscreen/ gives the same instructions: Define in the ServerLayout section the second screen's position to be farther away than the first screen's width.. Doing this makes sure that the mouse pointer won't accidentaly wrap from one screen to another when you move it out at the edge of any of the screens.

After modifying /etc/X11/xorg.conf, make a backup copy of the new version (sudo cp /etc/X11/xorg.conf{,.twoscreens.try}), close all applications, connect the external monitor, then restart your X11 session by pressing Ctrl-Alt-<BackSpace>. If it doesn't start up again, or you get some display configuration dialog, then there was something wrong with your modifications to /etc/X11/xorg.conf, and you have to fix it. To do so, press Ctrl-Alt-<F1> to switch to text mode, log in, then stop GDM by running sudo /etc/init.d/gdm stop, then have a look at the startup log file /var/log/Xorg.0.log , and edit /etc/X11/xorg.conf accordingly (you may have to copy it back from /etc/X11/xorg.conf.twoscreens.try first). To try your changes, run X (or sudo X), which displays a black-and-white dotted background on both screens, and an X-shaped mouse cursor on screen 0 (the laptop LCD). Exit by pressing Ctrl-Alt-<BackSpace>. If that one works, you may want to try startx to get a more interesting X session. Exit again by pressing Ctrl-Alt-<BackSpace>. Make a backup copy of the config file (sudo cp /etc/X11/xorg.conf{,.twoscreens.good}, and restart GDM: sudo /etc/X11/gdm restart.

How to prevent the GNOME panel and the icons from appearing on the second screen

Sorry, I don't know the right answer. The panels are drawn by gnome-panel. You can remove all but one, and GNOME will remember it upon relogin. If you remove all panels from a screen, GNOME will recreate the panels with the default configuration :-(. The desktop icons are drawn by nautilus. I have no idea how to disable it on the second screen. The window manager is metacity or compiz (use ps x to find out which is running). You can disable compiz on the second screen by appending the line COMPIZ_OPTIONS="$COMPIZ_OPTIONS --only-current-screen" to ~/.config/compiz/compiz-manager .

How to change the background image

Install ImageMagick with sudo apt-get install imagemagick. Then run display -window root -display :0.1 background.jpg. Please note that this will tile (repeat) copies of the image. You may want to resize the image to fit the screen size: display -window root -display :0.1 -resize 1920x1080 background.jpg, but this may ruin the aspect ratio. Use your favorite image editor (such as GIMP) to create an image of the right size. If you use GNOME, this trick might be useful: http://gnome-hacks.org/hacks.html?id=6 (but it sets the background for multiple screens).

How to move the mouse and switch the keyboard focus to the other X11 screen

There is a handy tool for that named switchscreen. Apparently there are two programs named switchscreen, one by Sampo Niskanen (http://users.tkk.fi/spniskan/switchscreen/) and one by David Antliff (http://en.gentoo-wiki.com/wiki/X.Org/Dual_Monitors#Moving_focus_between_screens and http://unlogical.net/files/scripts/switchscreen-0.4.tar.gz; alternate download for switchscreen.c: http://www.math.bme.hu/~pts/switchscreen.c). We are going the use the latter. Compile it with gcc -s -O2 -W -Wall -L/usr/X11R6/lib -lX11 -lXtst -lXext -o switchscreen switchscreen.c. (On Debian Etch and Ubuntu Hardy, you'll have to install some packages first: sudo apt-get install gcc libc6-dev x11-proto-core-dev x11proto-xext-dev x11proto-xext-dev .) Install with sudo cp switchscreen /usr/local/bin/ .

If you use GNOME, run switchscreen by pressing Ctrl-<F2>, and typing switchscreen plus Enter. You can use your desktop environment's configuration or xbindkeys to bind a key combination to switchscreen.

How to play a movie with mplayer on the external monitor

Run mplayer -display :0.1 -osdlevel 3 -fs movie.avi . If you don't see the the number of seconds elapsed, you may have to configure the MPlayer OSD font. If the OSD text is two large, append lines subfont-text-scale = 3 and subfont-osd-scale = 3 to the file ~/.mplayer/config . If the aspect ratio is wrong on the external monitor, but it is good on the LCD display, specify the flag -monitorpixelaspect X in the mplayer command line above, experimenting with values between 0.5 and 2.0 for X.

2009-04-07

How to prevent stderr log messages from interleaving on Linux

Let's suppose you have a program which forks subprocesses, and each subprocess writes log messages to the stderr they share. Since processes run concurrently, it may be possible that the bytes they write interleave (overlap). For example, process A calls write(2, "ABC\n", 4) == 4, and, at the same time process D calls write(2, "DEF\n", 4) == 4. You expect that stderr will contain "ABC\nDEF\n" or "DEF\nABC\n", depending on which write succeeds first. But this is not always the case (!): messages written can interleave, e.g. stderr might contain "ABDCEF\n\n" or something similar. This post gives some recepies how to prevent that, and thus always get either "ABC\nDEF\n" or "DEF\nABC\n".

The instructions given below were tested on Linux 2.6.22 and 2.6.24, but they may probably apply other Linux kernels and other Unices as well.
  • Use the write(2) system call to write your log messages. Avoid anything which can buffer or split the message.

    • In C, don't use printf(3), fwrite(3), fputs(3) etc.; use write(2)

    • In Java 6, don't use System.err.println; use System.err.print, and add a "\n" by hand.

    • In Python, don't use sys.stderr.write or print >>sys.stderr; use os.write(2, ...)
    • .
    • In Ruby, don't use $stderr.write or $stderr <<; use $stderr.syswrite. Don't forget to check the return value.

    • In Perl, don't use print(STDERR ...); use syswrite(STDERR, ...).

  • If stderr is a pipe, make sure you don't write more than 4096 bytes at a time. If you have a longer log message, split it to multiple log messages of at most 4096 bytes (including the newline). This size limit doesn't apply if stderr is a regular file. This size limit has been verified for the kernels above.

  • If stderr is a regular file, make sure that it is in append mode when you are writing it.

    • Without append mode, even data loss is possible, e.g. instead of "ABC\nDEF\n" you may get "ADBEF\n" (losing "C\n").

    • You can ensure append mode by using the append-redirection 2>>file.log instead of 2>file.log when invoking the program.

    • In C you can use fcntl(2) to put a file descriptor to append mode: fcntl(2, F_SETFL, fcntl(2, F_GETFL, 0) | O_APPEND). (Don't forget about checking the return values to detect errors.)

    • In Python you can use fcntl.fcntl to put a file descriptor to append mode: import fcntl; import os; fcntl.fcntl(2, fcntl.F_SETFL, fcntl.fcntl(2, fcntl.F_GETFL) | os.O_APPEND).

    • In Ruby you can use IO#fcntl to put a file descriptor to append mode: require 'fcntl'; $stderr.fcntl(Fcntl::F_SETFL, $stderr.fcntl(Fcntl::F_GETFL) | Fcntl::O_APPEND).

    • In Perl you can use Fcntl to put a file descriptor to append mode: use Fcntl; fcntl(STDERR, Fcntl::F_SETFL, fcntl(STDERR, Fcntl::F_GETFL, 0) | Fcntl::O_APPEND).

As a more generic solution (not specific to Unix), you may also consider using mutexes or semaphores to prevent concurrent writes to the same file.

2009-04-04

How to convert some LaTeX text to a high resolution PNG image

Create the TeX source file dump.tex like this:
\documentclass{article}
\pdfpagewidth2cm
\pdfpageheight1cm
\hoffset-2.3cm
\voffset-2.3cm
\begin{document}
\shipout\hbox{$\infty \sum \int$}
\end{document}

Convert it to PDF:
% pdflatex dump.tex

Make sure that pk files don't appear on the console output of pdflatex. If they happen to appear, choose a different font, e.g. \usepackage{lmodern}.

Convert the PDF to PNG with Ghostscript:
% gs -dBATCH -dNOPAUSE -sDEVICE=pngmono -r1000 -sOutputFile=dump.png dump.pdf

If you need a larger resolution image, increase the number after the -r flag above.

This solution generates a PNG with sharp edges (no smoothing). If you take a screenshot of the PDF in a PDF viewer, or you load the PDF directly into GIMP, you'll most probably get a smoothed (antialiased) image.

2009-03-31

How to set up ruby gems on Debian Etch as non-root

This tutorial describes how to install Ruby gems to your home directory. The gems you install there will not be visible to other user, and they won't affect the system default configuration. You'll need root access in the first few steps to install Ruby and the rubygems packaging framework itself. It is possible to install those without root access as well, but that's beyond the scope of this tutorial.

You need a working ruby and rubygems. You have to install it as root:
$ su -
# apt-get update
# apt-get install ruby1.8 rubygems
(if you want to install gems which need C compilation, e.g. ruby-sqlite:)
# apt-get install ruby1.8-dev gcc libc6-dev
# ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]

This is optional, to have the latest rubygems (you can do this any time later):
$ su -
# gem install rubygems-update
...
# /var/lib/gems/1.8/bin/update_rubygems
# rm -f /usr/bin/gem
# ln -s gem1.8 /usr/bin/gem
$ gem -v
1.3.1

All the rest works as non-root:
$ export GEM_HOME=$HOME/gems
$ rm -rf $GEM_HOME
$ mkdir $GEM_HOME{,/cache,/doc,/gems,/specifications}
$ cp -a /var/lib/gems/1.8/cache/sources-*.gem $GEM_HOME/cache/
$ cp -a /var/lib/gems/1.8/gems/sources-* $GEM_HOME/gems/
$ cp -a /var/lib/gems/1.8/specifications/sources-*.gemspec $GEM_HOME/specifications/
$ gem update
Updating installed gems...
Bulk updating Gem source index for: http://gems.rubyforge.org
(this takes a few minutes)
Gems: [] updated

Try querying or installing something:
$ gem search rake --remote
$ gem install rake
Successfully installed rake-0.8.4
Installing ri documentation for rake-0.8.4...
Installing RDoc documentation for rake-0.8.4...

Try installing rails:
$ gem install rails --include-dependencies
(... takes some time)
$ ~/gems/bin/rails -v
2.3.2

Try installing something which needs C code. Please note that you have to install the C prerequisite (libsqlite3) as root:
$ su -c 'apt-get install libsqlite3-dev'
$ gem search sqlite3 --remote
$ gem install --platform ruby sqlite3-ruby
Building native extensions. This could take a while...
Successfully installed sqlite3-ruby-1.2.4
1 gem installed
Installing ri documentation for sqlite3-ruby-1.2.4...
Installing RDoc documentation for sqlite3-ruby-1.2.4...

Please note that you need the GEM_HOME environment variable set for installing gems and for running applications requiring gems. To have this variable set for you, do this:
$ echo 'export GEM_HOME=$HOME/gems' >>~/.bashrc
$ echo 'export GEM_HOME=$HOME/gems' >>~/.bash_profile

If you need the rails command without having to specify ~/gems/bin/rails, make sure you have an export PATH=$HOME/gems/bin:$PATH in your .bashrc and/or .bash_profile.

2009-03-27

How to make Flash full screen work in Ubuntu Hardy and Firefox

We were using the Flash applet on Livescribe's web site in Firefox running on Ubuntu Hardy. We were using Flash 9.0 r124, and Firefox 3.0.7. It had a full screen button, but nothing happened. It was strange, because the full screen button worked a week ago. I've found a solution on http://ubuntulinuxtipstricks.blogspot.com/2007/12/fullscreen-youtube-now-available.html :
If you're using Compiz Fusion, turn off "Unredirect Fullscreen Windows" in the General section and turn off "Legacy Fullscreen Support" in the Workarounds plugin. If you don't, the controls for the player get cut off.
We've found the settings above in our Gnome desktop > System > Preferences > Advanced desktop effects > General options. Turning both settings off solved the problem for us.

2009-03-26

ASCII isdigit, isalpha and isxdigit macros in ANSI C with arithmetic operations

The naïve way of defining a C macro which tests whether a character is a digit (assuming an ASCII-based character set) is #define ISDIGIT(c) ((c) >= '0' && (c) <= '9'). There is a fundamental problem with this naïve definition: it evaluates its argument c more than once, so for example ISDIGIT(x++) will increment x by 2 in some cases. The question naturally arises if there is a macro definition #define ISDIGIT(c) ..., which uses c exactly once. Indeed, there is:
#define ISDIGIT(c) ((c) - '0' + 0U <= 9U). The capital Us in the expression enforce unsigned calculation, so if c is less than '0', then (c) - '0' + 0U becomes a large positive number instead of a negative number with small absolute value, so the comparison will (correctly) return false.

The following code contains solutions for ISDIGIT, ISALPHA and ISXDIGIT, the latter not being practical because of the excessive use of arithmetic operations.
/* by pts@fazekas.hu at Thu Mar 26 01:10:56 CET 2009
*
* 0123456789 ISDIGIT
* ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ISALPHA
* 0123456789ABCDEF ISCAPITALHEX
* 0123456789ABCDEFabcdef ISXDIGIT
*/
#include <stdio.h>

#define ISDIGIT(c) ((c) - '0' + 0U <= 9U)
#define ISALPHA(c) (((c) | 32) - 'a' + 0U <= 'z' - 'a' + 0U)
#define ISCAPITALHEX(c) ((((((c) - 48U) & 255) * 23 / 22 + 4) / 7 ^ 1) <= 2U)
#define ISXDIGIT(c) (((((((((c) - 48U) & 255) * 18 / 17 * 52 / 51 * 58 / 114 \
* 13 / 11 * 14 / 13 * 35 + 35) / 36 * 35 / 33 * 34 / 33 * 35 / 170 ^ 4) \
- 3) & 255) ^ 1) <= 2U)

int main(int argc, char **argv) {
int i;
(void)argc; (void)argv;
for (i = 0; i < 256; ++i) if (ISDIGIT(i)) putchar(i);
printf(" ISDIGIT\n");
for (i = 0; i < 256; ++i) if (ISALPHA(i)) putchar(i);
printf(" ISALPHA\n");
for (i = 0; i < 256; ++i) if (ISCAPITALHEX(i)) putchar(i);
printf(" ISCAPITALHEX\n");
for (i = 0; i < 256; ++i) if (ISXDIGIT(i)) putchar(i);
printf(" ISXDIGIT\n");
return 0;
}

2009-03-13

How to select an ALSA sound card and have concurrent, simultaneus playback using dmix

This blog post is tutorial which describes how to select a default sound card and run multiple playbacks simultaneously, using ALSA on Linux, without a sound server.

The quick trick is having
defaults.pcm.!card Headset
defaults.ctl.!card Headset
defaults.pcm.!device 0
defaults.ctl.!device 0
in your ~/.asoundrc. (Replace Headset with the name of the card on which you want to hear sound. Get the list of available sound cards with aplay -l | awk '/^card/{print$3}'|sort|uniq. To apply this setting for all users, add the lines above to /etc/asound.conf instead.) If this doesn't work for you, please continue reading.

Let's suppose you have a Linux system and many sound cards with an ALSA (>= 0.9) driver, and you don't use any sound server (e.g. pulse, ESD == esound or artsd). The dmix feature of ALSA does software sound mixing, so it makes it possible to run multiple playbacks simultaneously on the same card. ALSA, by default, enables dmix for all cards which don't support concurrent playback of more than one sound stream.

So you just run mplayer -ao alsa file1.mp3, and simultaneously (possibly in another terminal window) mplayer -ao alsa file2.mp3. You should hear both playbacks at the same time. (Please note that you can omit =-ao alsa= from the mplayer command line if you add ao=alsa to your ~/.mplayer/config file.) If the second mplayer doesn't start playback, but exists with an error message containing Device or resource busy, this means there is something wrong with your settings – this tutorial will help to fix that.

If even the first mplayer produces the Device or resource busy error, then close or kill all applications that might play sound. This includes the web browser (with flash), pulse, esd, artsd, mplayer, Skype, MPD and system sounds (disable everything in Gnome / System / Preferences / Sound / Sound). After that, mplayer -ao alsa file1.mp3 should start the first playback, and simultaneously, mplayer -ao alsa file2.mp3 should start the second playback.

If you have multiple sound cards (possibly the sound card in your computer, and an external USB headset), you can set the environment variable ALSA_CARD to direct playback to a specific card. Example 1: ALSA_CARD=Headset mplayer -ao alsa file1.mp3. Example 2: start Firefox as ALSA_CARD=Headset firefox to have the sound Flash movies played on the card named Headset.

You can get a list of sound cards (with some extra information) with aplay -l. For example, on my system, I get
$ aplay -l
card 0: Intel [HDA Intel], device 0: CONEXANT Analog [CONEXANT Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 1: Conexant Digital [Conexant Digital]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: TuxDroid [TuxDroid], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: TuxDroid [TuxDroid], device 1: USB Audio [USB Audio #1]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 2: Headset [Plantronics Headset], device 0: USB Audio [USB Audio]
Subdevices: 0/1
Subdevice #0: subdevice #0
Here is how I can select each card and device:
card 0: Intel [HDA Intel], device 0: CONEXANT Analog [CONEXANT Analog]
ALSA_CARD=0
ALSA_CARD=Intel
card 0: Intel [HDA Intel], device 1: Conexant Digital [Conexant Digital]
ALSA_CARD=0 ALSA_PCM_CARD=1
ALSA_CARD=Intel ALSA_PCM_CARD=1
card 1: TuxDroid [TuxDroid], device 0: USB Audio [USB Audio]
ALSA_CARD=1
ALSA_CARD=TuxDroid
card 1: TuxDroid [TuxDroid], device 1: USB Audio [USB Audio #1]
ALSA_CARD=1 ALSA_PCM_CARD=1
ALSA_CARD=TuxDroid ALSA_PCM_CARD=1
card 2: Headset [Plantronics Headset], device 0: USB Audio [USB Audio]
ALSA_CARD=2
ALSA_CARD=Headset
Once you have the proper ALSA_CARD and ALSA_PCM_CARD setting, you can add them to your ~/.bashrc and ~/.gnomerc and ~/.xprofile (and possibly to /etc/environment and /etc/X11/Xsession.d/* and /etc/gdm/Xsession). If you log out and log in to your graphic session, you'll have these environment variables by default.

It is possible to select the default sound card without changing setting the ALSA_CARD or ALSA_PCM_CARD environment variables. To do so, add these lines to your ~/.asoundrc:
defaults.pcm.!card Headset
defaults.ctl.!card Headset
defaults.pcm.!device 0
defaults.ctl.!device 0
Each !card line corresponds to the ALSA_CARD value, and each !device line corresponds to the ALSA_PCM_CARD value.

If you set the default in ~/.asoundrc and set the environment variables as well, then the environment variables take effect.

Please note that if you specify any specific ALSA device to any program (other than default), and the program starts playback, then automatic dmix would not work until that program finishes playback and closes the device. This implies that no other program will be able to start playback (but will yield Device or resource busy) until that happens. For example, Skype keeps the sound card open while it is running. So if you want to play sound not coming from Skype while Skype is running, you have to select Default device (default) in Options / Sound Devices for both Sound Out and Ringing. A similar restriction applies to music players and other software: if you specify the playback device for them in their command line or preferences, then the software will lock the sound card, and you lose dmix and concurrent playback. The only dmix-safe ways to select a sound card are the ALSA_CARD etc. environment variables and ~/asoundrc.

To get information about the ALSA cards, run aplay -l and aplay -v -v -L.

ALSA provides OSS emulation (i.e. /dev/dsp, /dev/dsp1), but dmix doesn't work with OSS emulation. To get it work, please run the software which needs OSS using the aoss wrapper, e.g. aoss mplayer -ao oss file1.oss, and concurrently, aoss mplayer -ao oss file2.oss or mplayer -ao alsa file2.oss . If you get the error message /dev/dsp: Device or resource busy from a program, then you'll either have to change it to use ALSA, or run it within aoss.

Here are some sine wave sound generators if you don't have an MP3 ready to test playback with:
perl -e 'print pack("v",32000*sin($_/34))."\0\0"; ++$_ while 1' | aplay -f dat  # Left ear
perl -e 'print "\0\0".pack("v",32000*sin($_/34)); ++$_ while 1' | aplay -f dat # Right ear
If you have the asoundconf utility, you can use it to set up the default sound card in your ~/.asoundrc. For example, after removing ~/.asoundrc and running
asoundconf set-default-card Headset, you'll get a line <:/home/USERNAME/.asoundrc.asoundconf> in file ~/.asoundrc, and the file ~/.asoundrc.asoundconf would contain more than 50 config lines, the essential ones being
!defaults.pcm.card Headset
defaults.ctl.card Headset
defaults.pcm.device 0
defaults.pcm.subdevice -1
defaults.pcm.nonblock 1
defaults.pcm.ipc_key 5678293
defaults.pcm.ipc_gid audio
defaults.pcm.ipc_perm 0660
defaults.pcm.dmix.max_periods 0
defaults.pcm.dmix.rate 48000
defaults.pcm.dmix.format S16_LE
defaults.pcm.dmix.card defaults.pcm.card
defaults.pcm.dmix.device defaults.pcm.device
defaults.pcm.dsnoop.card defaults.pcm.card
defaults.pcm.dsnoop.device defaults.pcm.device
defaults.namehint.extended off
This seems to be too much compared to the 4 lines the beginning of this tutorial suggests.

Random notes

With ALSA 1.0.15, only plughw: works for tuxdroid (so it cannot be used with ALSA_CARD, which implies hw:). Here is how to play: mplayer -ao alsa:device=plughw=TuxDroid file1.mp3 or aplay -D plughw:TuxDroid </dev/urandom. Please note that this restriction applies to both mplayer and aplay. (Maybe that's because dmix was busy when the tuxdroid was connected -- and if we reload ALSA, maybe it will work if I connect the tuxdroid first, and then load the ALSA modules?) The reason why it doesn't work seems to be that the U8 sample format needed by the tuxdroid was introduced only in ALSA 1.0.16. I've verified with libasound 1.0.16 installed (with the same old ALSA kernel) in a chroot, and dmix works with the tuxdroid.

The setting ALSA_CARD=Foo ALSA_PCM_CARD=2 corresponds to aplay -D hw:Foo,2 and mplayer -ao alsa:device=hw=Foo.2. Please note that there is no corresponding environment variable setting for plughw instead of hw. Please also note that mplayer won't use dmix (thus it won't be able to run multiple playbacks concurrently) if you specify any other ALSA setting than mplayer -ao alsa or mplayer -ao alsa:device=default . A similar restriction applies to aplay -D: if you specify any device other than default there, it won't use dmix.

An example full mplayer error message when dmix didn't work:
[AO_ALSA] alsa-lib: pcm_hw.c:1099:(snd_pcm_hw_open) open /dev/snd/pcmC2D0p failed: Device or resource busy
[AO_ALSA] Playback open error: Device or resource busy
Could not open/initialize audio device -> no sound.
Audio: no sound

2008-09-03

Using pytz to display a timestamp in a time zone

Download pytz from http://pytz.sourceforge.net/. You can try out the code snippet below without installing pytz. Just create your Python script in the directory where pytz's setup.py lives.

import time  
import pytz  
import datetime  
dt = datetime.datetime.fromtimestamp(time.time(), pytz.utc)  
#tz = pytz.timezone('EST')  
tz = pytz.timezone('CET')  
print tz.normalize(dt.astimezone(tz)).strftime('%Y-%m-%d %H:%M:%S %Z(%z)')