tag:blogger.com,1999:blog-88218468187140293932024-03-05T08:27:51.281+01:00pts.blogUnknownnoreply@blogger.comBlogger230125tag:blogger.com,1999:blog-8821846818714029393.post-89512997800794621992021-10-30T11:08:00.000+02:002021-10-30T11:08:08.307+02:00How to use HSQLDB for the first time<p>This blog post is a beginner-level tutorial explaining how to use <a href="http://hsqldb.org/">HSQLDB</a>, an SQL relational database system (RDBMS) written in Java.
<p>HSQLDB supports both the client-server model (i.e. the server opens the database files) and the embedded model (i.e. opening the database files directly). In both cases we mean use the term <i>client</i> meaning the process which initiates the SQL statement.
<p>Each HSQLDB table can have storage type <i>memory</i>, <i>cached</i> and <i>text</i>. Data in <i>memory</i> table is loaded to memory (from the <i>.script</i> file, parsed as SQL <i>INSERT INTO</i> statements at load time) when the database is opened, and saved (written back to the <i>.script</i> file) when the client disconnects. Data in <i>cached</i> table isn't loaded to memory in its entirety, but parts of it are read from the binary <i>.data</i> file (also called as the cache file) at query time; subsequent changes are kept in memory, and are written to the <i>.data</i> file when the client disconnects. Thus a huge number of inserts done by a single client uses a huge amount of memory. Data in a <i>.text</i> table is stored in a separate <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV file</a>.
<p>HSQLDB stores the database in a few files with different suffixes: <i>.lck</i>, <i>.script</i>, <i>.properties</i>, <i>.data</i>, <i>.log</i>, <i>.backup</i> and <i>.lobs</i> . Uncommitted data is not stored in any of the files. Committed data is first appended to the <i>.log</i> file as SQL <i>INSERT INTO</i> and <i>DELETE FROM</i> statements. When the client disconnects (either by exiting or invoking the <i>SHUTDOWN;</i> SQL statement), committed data is moved the the <i>.script</i> file (for table type <i>memory</i>, as SQL <i>INSERT INTO</i> statements with deleted rows omitted) or the <i>.data</i> file (for table type <i>cached</i>, as binary data which is faster to query). The <i>.log</i> and <i>.script</i> files are text files containing SQL statements, separated by newlines, no semicolon at the end. Data definition statements are first stored in the <i>.log</i> file (e.g. <i>CREATE TABLE</i>), and when the client disconnects, they are moved to the <i>.script</i> file (normalized as <i>CREATE CACHED TABLE</i> and <i>CREATE MEMORY TABLE</i>, table and column names uppercased).
<p>Follow these steps to run your first few SQL statements:
<ul>
<li>Install Java (the JRE). The following command (without the leading <code>$</code>) should work after successful installation:
<pre>
$ java -version
openjdk version "11.0.11" 2021-04-20
...
</pre>
<li>Download HSQLDB. When writing the blog post, <i>.jar</i> files were downloaded from <a href="http://hsqldb.org/download/hsqldb_251_jdk6/">http://hsqldb.org/download/hsqldb_251_jdk6/</a>. You may use a later version. You need 2 <i>.jar</i> files: <i>hsqldb-*.jar</i> (not the <i>hsqldb-*-sources.jar</i>) and <i>sqltool-*.jar</i>. Rename the downloaded files to <i>hsqldb.jar</i> and <i>sqltool.jar</i>. It's important that the filename of the former is exactly <i>hsqldb.jar</i>, because <i>sqltool.jar</i> tries to find it by that name.
<li>In your download directory, create a text file named <i>sqltool.rc</i> containing these 2 lines:
<pre>
urlid first
url jdbc:hsqldb:file:/tmp/first.hsql;hsqldb.default_table_type=cached;hsqldb.script_format=3
</pre>
<p>On Windows, use <code>C:/Windows/Temp</code> (with forward slashes) or something similar instead of <code>/tmp</code> as the directory name.</p>
<p>There is no need to create the files, HSQLDB will create them automatically.</p>
<li>Open a terminal window, <i>cd</i> to your download directory, and run:
<pre>java -jar sqltool.jar --rcFile=sqltool.rc first</pre>
It displays a long welcome message, and shows you the <code>sql></code> prompt.
<li>Type some SQL statements (without the leading <code>sql></code> prompt):
<pre>
sql> CREATE TABLE names (first VARCHAR(255), last VARCHAR(255));
sql> INSERT INTO names VALUES ('first1', 'last1');
sql> INSERT INTO names VALUES ('first2', 'last2');
sql> COMMIT;
sql> INSERT INTO names VALUES ('first3', 'last3');
sql> COMMIT;
sql> SELECT * FROM names;
FIRST LAST
------ -----
first1 last1
first2 last2
first3 last3
Fetched 3 rows.
sql> DELETE FROM names WHERE last='last2';
sql> CREATE INDEX myindex ON names (last);
sql> COMMIT;
sql> SELECT * FROM names;
FIRST LAST
------ -----
first1 last1
first3 last3
Fetched 2 rows.
sql> SHUTDOWN;
sql> \q
</pre>
<li>At this point, metadata will be saved to compressed file <code>/tmp/first.hsql.script</code>, and data will be saved to binary file <code>/tmp/first.hsql.data</code>.
<li>See this <a href="http://www.hsqldb.org/doc/1.8/guide/apb.html">example .java file</a> on using a HSQLDB database from Java code. The JDBC connection string (starting with <code>jdbc:hsqldb:file:</code> in the <i>sqltool.rc</i> file above) should be specified in the first argument of <i>DriverManager.getConnection</i>.
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-43700809094238867902020-12-02T14:55:00.007+01:002020-12-02T14:56:13.301+01:00How to use Docker on Linux amd64 without installing it<p>This blog post explains how to download and use Docker on Linux amd64 without installing it, and how to clean up afterwards. It complements the <a href="https://docs.docker.com/engine/install/binaries/">official instructions</a>, and contains some copy-pasteable commands.</p>
<p>You will need to start <i>dockerd</i> as root, so we assume that you can run commands as root with <i>sudo</i>.</p>
<p>Visit <a href="https://download.docker.com/linux/static/stable/x86_64/">https://download.docker.com/linux/static/stable/x86_64/</a>, and choose a Docker version. For example, we will use <i>17.12.1-ce</i> . Download it by running the following command:</p>
<pre>$ (V=17.12.1-ce; mkdir "$HOME/Downloads/docker-$V" &&
cd "$HOME/Downloads/docker-$V" &&
wget -qO- "https://download.docker.com/linux/static/stable/x86_64/docker-$V.tgz" |
tar xzv)
</pre>
<p>If you are using Ubuntu 14.04, you need to install a few small packages:</p>
<pre>$ sudo apt-get install cgroup-lite aufs-tools # Only on Ubuntu 14.04.</pre>
<p>(<i>cgroup-lite</i> fixes the <i>Error starting daemon: Devices cgroup isn't mounted</i> error, and <i>aufs-tools</i> fixes the <i>Couldn't run auplink before unmount</i> warning. Optionally, see <a href="https://stackoverflow.com/a/33977992">this answer</a> on fixing the <i>Module br_netfilter not found</i> warning.)</p>
<p>In a new terminal window, start <i>dockerd</i>, and keep it running:</p>
<pre>$ sudo env PATH="$HOME/Downloads/docker-17.12.1-ce/docker:$PATH" dockerd</pre>
<p>Run this to fix permission issues connecting to <code>/var/run/docker.sock</code>:</p>
<pre>$ sudo chgrp sudo /var/run/docker.sock</pre>
<p>In each terminal window where you want to run <i>docker</i> commands, run this first:</p>
<pre>$ export PATH="$HOME/Downloads/docker-17.12.1-ce/docker:$PATH"</pre>
<p>Run this to test that everything is working (it will take a few seconds for the first time):</p>
<pre>$ docker run --rm -it hello-world</pre>
<p>Now you can run <i>docker</i> commands as usual.</p>
<p>To stop <i>dockerd</i>, press Ctrl-<i>C</i> in the terminal window it is running in, or run this:</p>
<pre>$ sudo fuser -k -TERM /var/run/docker.sock</pre>
<p>If there are some containers running, <i>dockerd</i> tries to kill them, and waits for 15 seconds. To avoid that delay, you can manually kill them (either before or after stopping dockerd):</p>
<pre>$ (P=; test -e /var/run/docker.sock &&
sudo chgrp sudo /var/run/docker.sock && P="$(docker ps -aq)";
test "$P" && docker kill $P)</pre>
<p>To delete Docker (including images and configuration), first stop <i>dockerd</i>, then run:</p>
<pre>$ sudo rm -rf /etc/docker /var/lib/docker /var/run/docker /var/run/docker.*</pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-54497122999561301222020-04-08T02:17:00.000+02:002020-04-08T02:28:38.179+02:00How to cross-compile to various EXE files using Digital Mars C Compiler on Linux<p>This blog post explains how to use Digital Mars C Compiler 8.57 on Linux with Wine to cross-compile to EXE files of 16-bit DOS, 32-bit DOS and 32-bit Windows (Windows 95 -- Windows 10). The actual program compiled is quite dumb (it doesn't matter), the focus is on installation and command-line flags.
<p>Please note that Digital Mars C Compiler hasn't been ported to Linux, so we need to run it in emulation: we will be running the Win32 version in Wine. (Another option would be running the DOS version in DOSBox.) Digital Mars C Compiler is now open source, you may want to contribute porting it to Linux <a href="https://github.com/DigitalMars/Compiler/issues/8">here</a>.
<p>Digital Mars C Compiler is very small: version 8.57 is less than 13 MiB, including C and C++ compiler, linker, DOS extender, <code>#include</code> files and libc for 16-bit DOS, 32-bit DOS and 32-bit Windows. If compressed with 7z (LZMA2), it's about 2.28 MiB. It's also impressive that all of it (except for the DOS extender) was written by a single person. See also <a href="https://news.ycombinator.com/item?id=17129678">discussion 1</a> and <a href="https://news.ycombinator.com/item?id=17139305">duscussion 2</a> on Hacker News, with replies from the author of the compiler.
<p>All commands below are to be run in a Linux terminal window without the leading <code>$</code>.
<p>Check that Wine is installed:
<pre>$ wine --version
wine-4.0.3 (Debian 4.0.3-1)</pre>
<p>If you get an error message instead of a version number, then install Wine first using your package manager.
<p>Download and extract the compiler:
<pre>$ mkdir digitalmarsc
$ R=http://ftp.digitalmars.com/Digital_Mars_C++/Patch/
$ wget -O digitalmarsc/dm857c.zip "$R"/dm857c.zip
$ (cd digitalmarsc && unzip dm857c.zip && rm -f dm857c.zip)
$ wget -O digitalmarsc/dm850dos.zip "$R"/dm850dos.zip
$ (cd digitalmarsc && unzip -n dm850dos.zip && rm -f dm850dos.zip)
$ wget -O digitalmarsc/dm831x.zip "$R"/dm831x.zip
$ (cd digitalmarsc && unzip -n dm831x.zip && rm -f dm831x.zip)
$ KR=https://github.com/Olde-Skuul/KitchenSink/raw/master/sdks/dos/x32/
$ wget -O digitalmarsc/dm/lib/cx.obj "$KR"/cx.obj
$ wget -O digitalmarsc/dm/lib/x32.lib "$KR"/x32.lib
$ wget -O digitalmarsc/dm/lib/zlx.lod "$KR"/zlx.lod
</pre>
<p>Please note that <a href="http://ftp.digitalmars.com/Digital_Mars_C++/Patch/cd857.zip">cd857.zip</a> contains additional files and it overlaps with <i>dm850dos.zip</i> (e.g. <i>dm/lib/sds.lib</i>).
<p>Create the test program source code:
<pre>$ cat >myprog.c <<'END'
#include <stdio.h> /* Code below works without it. */
const int answer = (int)0x44434241; /* dd 'ABCD' */
int mul(int a, int b) {
return a * b + answer;
}
int main(int argc, char **argv) {
(void)argv;
return mul(argc, argc);
}
END</pre>
<p>There is no need to set up environment variables, because Digital Mars C Compiler can its library files nearby its own directory. But we make the shorthand <code>wine dmc</code> work for convenience:
<pre>$ export WINEPATH="$(winepath -w digitalmarsc/dm/bin)"</pre>
<p>Do this to prevent Wine from displaying GUI windows:
<pre>$ unset DISPLAY</pre>
<p>Compile to all targets:
<pre>$ wine dmc -ml -v0 -odcprog.dosl.exe myprog.c
$ wine dmc -ms -v0 -odcprog.doss.exe myprog.c
$ wine dmc -mx x32.lib -v0 -odcprog.dos32.exe myprog.c
$ wine dmc -mn -v0 -odcprog.win32.exe myprog.c
$ wine dmc -c -ml -v0 -odcprog.dosl.obj myprog.c
$ wine dmc -c -ms -v0 -odcprog.doss.obj myprog.c
$ wine dmc -c -mx -v0 -odcprog.dos32.obj myprog.c
$ wine dmc -c -mn -v0 -odcprog.win32.obj myprog.c
$ wine dmc -c -mf -v0 -odcprog.os232.obj myprog.c</pre>
<p>More information about running Digital Mars C Compiler: <a href="https://digitalmars.com/ctg/sc.html">https://digitalmars.com/ctg/sc.html</a> . Please note that the <i>dmc</i> and <i>sc</i> commands were equivalent, but only <i>dmc</i> works in recent versions.
<p>Check the file type of the created executable (*.exe) files:
<pre>$ file dcprog.*.exe
dcprog.dos32.exe: MS-DOS executable, MZ for MS-DOS
dcprog.dosl.exe: MS-DOS executable
dcprog.doss.exe: MS-DOS executable
dcprog.win32.exe: PE32 executable (console) Intel 80386, for MS Windows</pre>
<p>Check the file type of the created object (relocatable, *.obj) files:
<pre>$ file dcprog.*.obj
dcprog.dos32.obj: 8086 relocatable (Microsoft), "myprog.c"
dcprog.dosl.obj: 8086 relocatable (Microsoft), "myprog.c"
dcprog.doss.obj: 8086 relocatable (Microsoft), "myprog.c"
dcprog.os232.obj: 8086 relocatable (Microsoft), "myprog.c"
dcprog.win32.obj: 8086 relocatable (Microsoft), "myprog.c"</pre>
<p>More information about the OMF OBJ file format (*.obj above): <a href="https://pierrelib.pagesperso-orange.fr/exec_formats/OMF_v1.1.pdf">https://pierrelib.pagesperso-orange.fr/exec_formats/OMF_v1.1.pdf</a>
<p>The *.obj files contain an memory model comment with code 0x9d which depends on the target and operating system:
dosl has <code>0lO</code>,
doss has <code>0sO</code>,
dos32 has <code>3fOpd</code>,
dosx has <code>7xO</code>,
os232 has <code>fnO</code>,
win32 has <code>7nO</code>.
<p>Alternatives for cross-compiling C code to some of these EXE targets on Linux:
<ul>
<li><a href="https://github.com/open-watcom/open-watcom-v2/">OpenWatcom</a> ((16-bit and 32-bit) * (DOS, Windows and OS/2) targets, see <a href="https://ptspts.blogspot.com/2020/04/openwatcom-exeprog.html">blog post</a> for details)
<li><a href="https://digitalmars.com/ctg/sc.html">Digital Mars C compiler</a> (on Linux needs Wine, 16-bit DOS, 32-bit DOS, 32-bit Windows targets, plus 32-bit OS/2 object target (no linking), see above for details)
<li><a href="https://packages.debian.org/stable/mingw-w64">mingw-w64</a> (win32 and win64 targets)
<li><a href="https://github.com/tkchia/gcc-ia16">gcc-ia16</a> (doss and dosl targets, i.e. 16-bit DOS EXE with various memory models)
<li><a href="https://github.com/andrewwutw/build-djgpp/releases">djgpp-linux32</a> (dosx target, needs cwsdpmi.exe or pmodstub.exe (PMOED/DJ) as separate downloads)
<li>You may be able to run C compilers released for Windows (e.g. the <a href="https://www.digitalmars.com/download/freecompiler.html">Digital Mars C compiler</a>) using <a href="https://www.winehq.org/">Wine</a>.
</ul>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-52444395053164880232020-04-07T23:26:00.000+02:002020-04-08T02:29:54.089+02:00How to cross-compile to various EXE files using OpenWatcom C compiler on Linux<p>This blog post explains how to use OpenWatcom 2.0 C compiler on Linux (i386 or amd64, any distribution) to cross-compile to EXE files of 16-bit DOS, 32-bit DOS, 16-bit Windows (Windows 3.1), 32-bit Windows (Windows 95 -- Windows 10), 16-bit OS/2 (1.x) and 32-bit OS/2 (2.x). The actual program compiled is quite dumb (it doesn't matter), the focus is on installation and command-line flags.
<p>All commands below are to be run in a Linux terminal window without the leading <code>$</code>.
<p>Download and extract the compiler:
<pre>$ mkdir open-watcom-2 open-watcom-2/tmp
$ R=https://github.com/open-watcom/open-watcom-v2/releases
$ wget -O open-watcom-2/tmp/open-watcom-2.zip \
"$R"/download/Current-build/open-watcom-2_0-c-linux-x86
$ (cd open-watcom-2/tmp && unzip open-watcom-2.zip)
$ (cd open-watcom-2/tmp && mv binl h lib286 lib386 ../)
$ (cd open-watcom-2/tmp && cp binw/dos32a.exe ../binl/)
$ (cd open-watcom-2/tmp && cp binw/dos4gw.exe ../binl/)
$ (cd open-watcom-2/binl && chmod +x owcc wcc wcc386 wlink)
$ rm -rf open-watcom-2/tmp</pre>
<p>Create the test program source code:
<pre>$ cat >myprog.c <<'END'
#include <stdio.h> /* Code below works without it. */
const int answer = (int)0x44434241; /* dd 'ABCD' */
int mul(int a, int b) {
return a * b + answer;
}
int main(int argc, char **argv) {
(void)argv;
return mul(argc, argc);
}
END</pre>
<p>Set up compilation environment:
<pre>$ export WATCOM="$PWD/open-watcom-2"
$ export PATH="$WATCOM/binl:$PATH" INCLUDE="$WATOM/h"</pre>
<p>Compile to all targets:
<pre>$ owcc -bdos -mcmodel=l -o owprog.dosl.exe myprog.c
$ owcc -bdos -mcmodel=s -o owprog.doss.exe myprog.c
$ owcc -bdos32a -o owprog.dos32.exe myprog.c
$ owcc -bdos4g -o owprog.dosx.exe myprog.c
$ owcc -bos2 -o owprog.os216.exe myprog.c
$ owcc -bos2v2 -o owprog.os232.exe myprog.c
$ owcc -bwindows -o owprog.win16.exe myprog.c
$ owcc -bnt -o owprog.win32.exe myprog.c
$ owcc -c -bdos -mcmodel=l -o owprog.dosl.obj myprog.c
$ owcc -c -bdos -mcmodel=s -o owprog.doss.obj myprog.c
$ owcc -c -bdos32a -o owprog.dos32.obj myprog.c
$ owcc -c -bdos4g -o owprog.dosx.obj myprog.c
$ owcc -c -bos2 -o owprog.os216.obj myprog.c
$ owcc -c -bos2v2 -o owprog.os232.obj myprog.c
$ owcc -c -bwindows -o owprog.win16.obj myprog.c
$ owcc -c -bnt -o owprog.win32.obj myprog.c</pre>
<p>More information about OpenWatcom and running it on Linux: <a href="https://wiki.archlinux.org/index.php/Open_Watcom">https://wiki.archlinux.org/index.php/Open_Watcom</a>
<p>Check the file type of the created executable (*.exe) files:
<pre>$ file owprog.*.exe
owprog.dos32.exe: MS-DOS executable, LE executable for MS-DOS, DOS/32A DOS extender (embedded)
owprog.dosl.exe: MS-DOS executable, MZ for MS-DOS
owprog.doss.exe: MS-DOS executable, MZ for MS-DOS
owprog.dosx.exe: MS-DOS executable, LE executable
owprog.os216.exe: MS-DOS executable, NE for OS/2 1.x (EXE)
owprog.os232.exe: MS-DOS executable, LX for OS/2 (console) i80386
owprog.win16.exe: MS-DOS executable, NE for MS Windows 3.x (EXE)
owprog.win32.exe: PE32 executable (console) Intel 80386, for MS Windows</pre>
<p>FYI the created <i>owprog.dosx.exe</i> file also needs the <i>dos4gw.exe</i> <a href="https://en.wikipedia.org/wiki/DOS_extender">DOS Extender</a> to run. You can copy the file from the <i>binl</i> directory.
<p>Check the file type of the created object (relocatable, *.obj) files:
<pre>$ file owprog.*.obj
owprog.dos32.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.dosl.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.doss.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.dosx.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.os216.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.os232.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.win16.obj: 8086 relocatable (Microsoft), ".../myprog.c"
owprog.win32.obj: 8086 relocatable (Microsoft), ".../myprog.c"</pre>
<p>More information about the OMF OBJ file format (*.obj above): <a href="https://pierrelib.pagesperso-orange.fr/exec_formats/OMF_v1.1.pdf">https://pierrelib.pagesperso-orange.fr/exec_formats/OMF_v1.1.pdf</a>
<p>The *.obj files contain an Intel-specific comment with code 0x9b which depends on the target and operating system:
dosl has <code>0lOed</code>,
doss has <code>0sOed</code>,
dos32 has <code>3fOpd</code>,
dosx has <code>3fOpd</code> (the .obj file is the same for dos32, dosx, os232, win32),
os216 has <code>0sOed</code> (the .obj file is the same for os216 and doss),
os232 has <code>3fOpd</code> (the .obj file is the same for dos32, dosx, os232, win32),
win16 has <code>0sOed</code> (the .obj file differs by only 2 bytes in doss and win16),
win32 has <code>3fOpd</code> (the .obj file is the same for dos32, dosx, os232, win32).
<p>Alternatives for cross-compiling C code to some of these EXE targets on Linux:
<ul>
<li><a href="https://github.com/open-watcom/open-watcom-v2/">OpenWatcom</a> ((16-bit and 32-bit) * (DOS, Windows and OS/2) targets, see above for details)
<li><a href="https://digitalmars.com/ctg/sc.html">Digital Mars C compiler</a> (16-bit DOS, 32-bit DOS, 32-bit Windows targets, plus 32-bit OS/2 object target (no linking), see <a href="https://ptspts.blogspot.com/2020/04/digitalmarsc-exeprog.html">blog post</a> for details)
<li><a href="https://packages.debian.org/stable/mingw-w64">mingw-w64</a> (win32 and win64 targets)
<li><a href="https://github.com/tkchia/gcc-ia16">gcc-ia16</a> (doss and dosl targets, i.e. 16-bit DOS EXE with various memory models)
<li><a href="https://github.com/andrewwutw/build-djgpp/releases">djgpp-linux32</a> (dosx target, needs cwsdpmi.exe or pmodstub.exe (PMOED/DJ) as separate downloads)
<li>You may be able to run C compilers released for Windows (e.g. the <a href="https://www.digitalmars.com/download/freecompiler.html">Digital Mars C compiler</a>) using <a href="https://www.winehq.org/">Wine</a>.
</ul>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-5661883202568327672019-02-10T18:24:00.000+01:002020-04-07T21:14:08.171+02:00Speed of in-memory algorithms in scripting languages<p>This blog post gives some examples how much faster in-memory algorithms are in scripting languages than in C.
<p>Before writing this blog post I had the general impression that the speed ratio between code in a scripting language and code in C for the same CPU-bound algorithm is between 5 and 20. I was very much surprised that for <a href="https://github.com/pts/muxzcat">LZMA2 decompression</a> I experienced a much larger ratio between Perl and C: 285.
<p>Then I looked at the <a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/c.html">C speeds</a> and <a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/perl.html">Perl speeds</a> on the Debian Computer Language Benchmarks Game, and I've found these ratios (in decreasing order) between Perl and C: 413, 79.7, 66.3, 62.2, 49.2, 20.8, 12.1, 10.2, 5.87, 1.91. So it turns out that <b>there is a huge fluctuation in the speed ratio, depending on the algorithm.</b>
<p>Takeaways:
<ul>
<li>One doesn't need to use 64-bit registers or vector (SIMD) instructions (e.g. AVX, SSE, MMX) or other special instructions in C code to get a huge speed ratio: for LZMA2 decompression, there can be a huge speed difference even if all variables are 32-bit unsigned integers.
<li>One doesn't need to use 64-bit code in C to get a huge speed ratio: for LZMA2 decompression, the benchmarked C code was running as 32-bit (i386, more specifically: i686) code.
<li>One doesn't have to use C compiler optimization flags for fast execution (e.g. <code>-O3</code> or <code>-O2</code>) to get a huge speed ratio: for LZMA2 decompression the size-optimized output of <code>gcc -Os</code> was that fast.
<li>Cache usage (e.g. L1 cache, L2 cache, L3 cache) can have a huge effect on speed of C code. The <a href="Linux i386 executable running the C code">https://github.com/pts/muxzcat/releases/download/v1/muxzcat</a> is 7376 bytes in total, thus the code fits into the fastest L1 cache of modern Intel processors (L1 cache size at lest 8 KiB, typically at least 32 KiB). The data itself doesn't fit to the cache though.
<li>I/O buffering and the associated memory copies can also affect execution speed. Typical size of <i>read(2)</i> calls is >60 KiB, and typical size of <i>write(2)</i> is even larger (2--3 times larger) for LZMA2 decompression, this is fast enough in both C and Perl code.
<li>Memory allocation can also affect execution speed. The C code for LZMA2 decompression doesn't do any memory allocation. The algorithm of the Perl code doesn't do any either (but the Perl interpreter may do some as part of its overhead), except for the occasional exponential doubling of the string capacity. (Preallocating these string buffers didn't make it any faster.)
<li>Even older C compilers (e.g. GCC 4.8 from 2014) can generate very optimized low-level i386 machine code.
<li>Some scripting languages are faster than others, e.g. Lua in <a href="http://luajit.org/">LuaJIT</a> and JavaScript in <a href="https://nodejs.org/">Node.js</a> are typically faster than the Python, Perl and Ruby interpreters written in C, and <a href="https://pypy.org/">PyPy</a> is faster than the Python interpreter written in C.
<li>Different integer sizes (e.g. 8-bit, 16-bit, 32-bit, 64-bit) can affect execution speed. Sometimes larger integers are faster (e.g. 32-bit is faster than 16-bit), because they are better aligned in memory, and fewer conversion instructions are necessary.
<li>Integer fixups can contribute to the slowness of scripting languages. For example the algorithm for LZMA2 decompression works with unsigned 32-bit integers, but Perl has only either signed 64-bit integers or signed 32-bit integers, so inputs of some operators (e.g. >>, <, ==, %, /) need to be bit-masked to get correct results. Out of these, / and % would be the slower to fix, but since LZMA2 decompression doesn't use these operators, < is the slowest: in total, the 32-bit Perl is 1.1017 times slower running the LZMA2 decompression than the 64-bit Perl, mostly because operator < and its possibly negative inputs need more complicated masking if Perl is doing 32-it arithmetic.
<li>Function calls can be very slow in scripting languages, while the C compiler can inline some of the smaller functions, avoiding most of the overhead. For LZMA2 decompression, manual inlining of the fixup for operator < on 32-bit Perls made the entire program about 1.3 times faster.
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-11782605641281134332019-02-10T14:46:00.002+01:002019-02-10T14:46:17.670+01:00Matching balanced parentheses with recursive Perl regular expressions<p>This blog post explains how to use recursive Perl regular expressions (regexp) to match substrings with balanced parentheses. Recursive regular expressions is also available <a href="https://stackoverflow.com/a/19487530">in Ruby</a> (with a different syntax) and <a href="https://stackoverflow.com/a/26386070">in the regex extension of Python</a> (but not in the built-in re module), but it's explicitly not available <a href="https://github.com/google/re2/wiki/Syntax">in RE2</a>.
<p>Let's suppose the input file <i>in.txt</i> contains lines like:
<pre>a = EQ(x + 6, 42);
a = EQ((x + 6) * 2, 42);
if (x + 6 == 42) { ... }
if (EQ(x + 6, 42)) { ... }
if (EQ((x + 6) * 2, 42)) { ... }</pre>
, and let's suppose we want to get all instances of <code>EQ</code> and <code>if</code>, with their arguments.
<p>A non-recursive regexp can get only the instances without nested parentheses:
<pre>$ <b><in.txt perl -0777 -wne '
while (m@(?>\b(if|while|EQ|NE|LT|LE|GT|GE)\s*\(([^()]*)\))@g)
{ print "$1($2)\n" }'</b>
EQ(x + 6, 42)
if(x + 6 == 42)
EQ(x + 6, 42)</pre>
<p>With a recursive regexp we can get all matches:
<pre>$ <b><in.txt perl -0777 -wne '
while (m@(?>\b(if|while|EQ|NE|LT|LE|GT|GE)\s*(\(((?:[^()]+|(?2))*)\)))@g)
{ print "$1$2\n" }'</b>
EQ(x + 6, 42)
EQ((x + 6) * 2, 42)
if(x + 6 == 42)
if(EQ(x + 6, 42))
if(EQ((x + 6) * 2, 42))</pre>
<p>Please note that the <code>EQ</code> inside the <code>if</code> was not matched, because with the global flag (<code>m@...@g</code>) Perl doesn't consider overlapping or enclosed matches.
<p>The recursive part of the regexp is the <code>(?2)</code>: it's a recursive reuse of paren group 2. For more information about recursive regexps, see <i>recursive subpattern</i> in <a href="http://perldoc.perl.org/perlre.html">perlre(1)</a>. The <code>(?>gt...)</code> construct is a performance optimization to prevent backtracking.
<p>It's also possible to match individual (comma-separated) arguments. For example, here is how to match both arguments of <code>EQ</code> separately, recursively:
<pre>$ <b><in.txt perl -0777 -wne '
while (
m@(>>\b(if|while|EQ|NE|LT|LE|GT|GE)\(((?:[^(),]+|(\(((?:[^()]+|(?3))*)\)))*),\s*((?2))\))@g)
{ print "$1($2, $5)\n" }'</b>
EQ(x + 6, 42)
EQ((x + 6) * 2, 42)
EQ(x + 6, 42)
EQ((x + 6) * 2, 42)</pre>
<p>The non-recursive version returns 2 arguments without nested parentheses:
<pre>$ <b><in.txt perl -0777 -wne '
while (m@(?>\b(if|while|EQ|NE|LT|LE|GT|GE)\s*\(([^(),]*),\s*([^(),]*))@g)
{ print "$1($2, $3)\n" }'</b>
EQ(x + 6, 42)
EQ(x + 6, 42)</pre>
<p>Here is how to match only a single argument (no comma) recursively:
<pre>$ <b><in.txt perl -0777 -wne '
while (
m@(>>\b(if|while|EQ|NE|LT|LE|GT|GE)\s*\(((?:[^(),]+|(\(((?:[^()]+|(?3))*)\)))*)\))@g)
{ print "$1($2)\n" }'</b>
if(x + 6 == 42)
if(EQ(x + 6, 42))
if(EQ((x + 6) * 2, 42))</pre>
<p>The non-recursive version returns 1 argument without nested parentheses:
<pre>$ <b><in.txt perl -0777 -wne '
while (m@(?>\b(if|while|EQ|NE|LT|LE|GT|GE)\s*\(([^(),]*)\))@g)
{ print "$1($2)\n" }'</b>
if(x + 6 == 42)</pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-57179458925559473422018-06-04T15:50:00.000+02:002018-06-06T15:59:32.561+02:00How to copy files securely between computers running Linux or Unix?<p>This blog post gives various recommendations on how to copy files securely between computers running Linux or Unix.
<p>All the recommendations below copy the file in an encrypted way, protecting
against eavesdropping and protecting partially against man-in-the-middle
attacks (i.e. a thrid party tricking the receiver to accept forged file
contents).
<p>If both computers run either of Chrome or Firefox, and it's convenient for you to use these
web browsers, visit any of the following sites to copy the file:
<a href="https://www.sharedrop.io/">sharedrop.io</a>,
<a href="https://www.reep.io/">reep.io</a>,
<a href="https://takeafile.com/">takeafile.com</a>,
<a href="https://send-anywhere.com/">send-anywhere.com</a>,
<a href="https://www.justbeamit.com/">justbeamit.com</a>.
These sites use WebRTC (thus the transfer is encrypted) to copy the file
directly from the sender to the receiver without uploading it
to a server, and they traverse NAT firewalls using STUN and ICE.
(Don't use sites based on WebTorrent (such as instant.io or file.pizza),
because the WebTorrent transfers are not end-to-end encrypted.)
<p>Otherise, if one of the computers is running the OpenSSH server (sshd), and the other
one is able to connect to it over the network, and you know a user's
password on the server (or SSH public keys are set up instead of a
password), then use scp or rsync.
Otherwise, if one of the computers is able to connect to the other over the
network, and the client computer (the one which initiates the TCP connection)
has the OpenSSH client (ssh) installed, you have root access on the server,
and you don't mind installing
software to the server temporarily, then follow the instructions in the
<a href="#dbscp">One-off SCP with Dropbear</a> section below.
<p>The rest of the setups are typically useful if one of the computers is recently installed (so it doesn't contain your SSH private keys yet), or you don't want any of them act as a server, or you don't have root access.
<p>Otherwise, if both computers are connected to the same local network (e.g.
same wifi network), and they are able to connect to each other, try ecplcnw
(available and documented here: <a href="https://github.com/pts/copystrap">https://github.com/pts/copystrap</a>).
<p>Otherwise, if both computers have web access, and you don't mind uploading
securely encrypted files to a shared hosting provider, use ecptrsh
(available and documented here: <a href="https://github.com/pts/copystrap">https://github.com/pts/copystrap</a>).
<p>Otherwise, if you have a USB pen drive, SD card, external hard disk or other
writable storage medium which you can physically take from one computer to
another, use ecplmdr
(available and documented here: <a href="https://github.com/pts/copystrap">https://github.com/pts/copystrap</a>).
<p>Otherwise I have no secure and convenient recommendation for you.
<h3>Other secure options for file copy</h3>
<ul>
<li><i>Direct connection between the computers using an Ethernet cable or serial cable.</i>
This can work, but it is not convenient, because it needs rare hardware and increasingly
rare ports on the laptops and extensive and error-prone manual setup.
<li><i>netcat for transfer + GPG for encryption.</i> Some more details
<a href="https://askubuntu.com/q/595849/3559">here</a>. This is similar to ecplcnw above,
but less convenient and less secure,
because user-invented passphrases tend to be weak, and strong
passphrases are long and cumbersome to type. Also it's a bit inconvenient to the get the
IP address in the command-lines right. Also the user has to remember some GPG quirks to
get the security right:
specifying <tt>--force-mdc</tt> and checking the return value of <tt>gpg -d</tt>.
<li><i>USB pen drive + GPG for encryption.</i> This is similar to ecplmdr above, but less
convenient and less secure,
because user-invented passphrases tend to be weak, and strong
passphrases are long and cumbersome to type. Also the user has to remember some GPG quirks to
get the security right:
specifying <tt>--force-mdc</tt> and checking the return value of <tt>gpg -d</tt>.
<li><i>Using a QR code and scanning it with the webcam: qrencode + zbarcam + GPG for encryption.</i>
This works for files smaller than about 10 KiB, because the resolution of the webcam in many laptops
is not good enough to scan large QR codes.
Without GPG this is not secure, in case someone is taking a video recording of the computer screen.
Also the user has to remember some GPG quirks to
get the security right:
specifying <tt>--force-mdc</tt> and checking the return value of <tt>gpg -d</tt>.
<li><i>Setting up the secret key in
your <a href="https://en.wikipedia.org/wiki/YubiKey">YubiKey</a> on one computer,
copying the public key from it onto the second computer, and connecting via ssh to
the second computer.</i>
This works if you already have a YubiKey, the first computer is
nearby, and it's convenient for you to set up and dump keys on your YubiKey.
How to retrieve the SSH public key from the YubiKey: use <tt>ssh-add -L | grep cardno:</tt>
Because of the many skilled manual steps involved, this solution is less convenient than the
recommendations above.
<li><i>Setting up the secret key in
your <a href="https://en.wikipedia.org/wiki/YubiKey">YubiKey</a> on one computer, adding metadata,
then using the list command in gpg --card-edit to get the metadata.</i>
This can be used to copy a few hundred bytes if both computers are nearby
(i.e. you can connect the same YubiKey to both). This is similar to using an USB pen drive
to copy files, but perhaps a bit more secure. (It's more secure only if an attacker stealing your
YubiKey can't extract the metadata without knowing the passphrase. This has to be checked.)
</ul>
<h3>Requirements</h3>
<p>Security requirements:
<ul>
<li>t encrypts the data end-to-end, only the receiver is able to decrypt it.
<li>The receiver is able to detect if the data is indeed what the sender
has sent (e.g. it was not tampered with and it was not replaced by the
data provided by the attacker).
</ul>
<p>Convenience requirements:
<ul>
<li>It works on the command-line.
<li>It works as a regular user (non-root) on the both computers.
<li>It works without software installation on the both computers.
<li>It works without creating any file other than the output data file in the
receiver. (We can relax this: a few small temporary files are OK, if they
get removed automatically in the end.)
<li>It works with very little typing (at most 20 characters of key typing in
total). Copy-pasting is OK, but not between the sender and the receiver.
<li>There is a mode which works on the local network without a public or
local service running and without extra hardware (network cables or USB pen
drives).
<li>There is a mode which works without a local network and without extra
hardware; it is allowed to use a public service.
<li>There is a mode which works without any network (local network or
internet); it is allowed to use a USB pen drive.
</ul>
<h3><a name="dbscp">One-off SCP with Dropbear</a></h3>
<p>If one of the computers (let's call it client) has the OpenSSH client (ssh)
installed, and is able to connect to the other computer (let's call it the
server), you have root access on the server,
and the server doesn't have a working OpenSSH server (sshd)
installed, and you don't mind intalling software to the server temporarily,
you can follow these steps to copy files securely.
<p>On the server, install Dropbear. For example, on Debian 9 or later, run this
as root (without the leading <tt>#</tt>):
<pre># apt-get install dropbear-bin</pre>
<p>On the server, install the scp command-line tool, part of OpenSSH. For
example, on Debian 9 or later:
<pre># apt-get install openssh-client</pre>
<p>On the server, generate an SSH host key, and start the server:
<pre># dropbearkey -t rsa -s 4096 -f dbhostkey
# /usr/sbin/dropbear -r dbhostkey -F -E -m -w -j -k -p 64358 -P dbtmp.pid
</pre>
<p>The last command (dropbear) makes the Dropbear SSH server keep running and
serving incoming connections until you press Ctrl-<i>C</i> in the terminal
window. This is normal.
<p>When <tt>dropbearkey</tt> above prints <tt>Fingerprint: md5</tt>, remember the value,
because you will have to compare it with the value printed by the client.
<p>On the client, initiate the copy with the following command (without the
leading <tt>$</tt>):
<pre>$ SSH_AUTH_SOCK= scp -o Port=64358 -o HostName=... -o User=... \
-F /dev/null -o UserKnownHostsFile=/dev/null \
-o HostKeyAlgorithms=ssh-rsa -o FingerprintHash=md5 SOURCE DESTINATION
</pre>
<p>In the command above:
<ul>
<li>Specify HostName=... as the host name of the server.
<li>Specify User=... as the non-root user name to be used on the server. scp
will ask that user's password on the client.
<li>SOURCE and DESTINATION can be a filename on the client, or, if prefixed by
<tt>r:</tt>, then it's a filename inside the home directory of the user on the
server.
<li>If scp complains about FingerprintHash, then drop the <tt>-o
FingerprintHash=md5</tt>, and try again.
<li>When the client prints <tt>RSA key fingerprint is MD5:...</tt>, compare the <tt>...</tt>
value with the value printed by dropbearkey on the server. If they don't
match perfectly, stop. If you continue even then, then you may be a victim
of a man-in-the-middle attack, and your copy is not secure.
</ul>
<p>You may run multiple copies with scp between the client and the server.
<p>As an alternative to scp, you can also use rsync to do the copies (if rsync
is installed to both the client and the server). The command to be run on
the client looks like this:
<pre>$ SSH_AUTH_SOCK= rsync -e 'ssh -o Port=64358 \
-o HostName=... -o User=... -F /dev/null \
-o UserKnownHostsFile=/dev/null -o HostKeyAlgorithms=ssh-rsa \
-o FingerprintHash=md5' --progress -avz SOURCE DESTINATION
</pre>
<p>Abort Dropbear on the server by pressing Ctrl-<C>.
<p>Having run the copies, remove unnecessary packages from the server. For
example (do it carefully, don't remove anything you need), on Debian 9:
<pre># sudo apt-get purge dropbear-bin libtommath1 libtomcrypt1
# sudo apt-get purge openssh-client
</pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-3514998230819654622018-04-28T01:16:00.000+02:002018-04-28T01:16:05.536+02:00How to force OpenSSH to log in with a specific password or public key<p>This blog post explains how to force the OpenSSH client to log in with a specific password or public key. This is useful if some of the SSH client config files (<code>/etc/ssh/ssh_config</code>, <code>/etc/ssh/ssh_known_hosts</code>, <code>/etc/ssh/ssh_known_hosts2</code>, <code>~/.ssh/config</code>, <code>~/.ssh/known_hosts</code>) or the ssh-agent are in a broken state, and you want to try whether login works independently of these client-side issues.
<p>Run this command to log in, substituting the <code>"${...}"</code> values:
<pre>SSH_AUTH_SOCK= /usr/bin/ssh -F /dev/null \
-o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null \
-o StrictHostKeyChecking=no \
-p "${PORT}" -i "${KEYFILE}" -- "${USERNAME}"@"${HOST}"</pre>
<p>Usage notes:
<ul>
<li>To use the default port (22), drop the <code>-p "${PORT}"</code>.
<li>To use password login instead of public key login, drop the <code>-i "${KEYFILE}"</code>.
<li>If you don't know where your public key file is, try <code>-i ~/.ssh/id_rsa</code>
<li>To use the same username as your local client username, drop the <code>"${USERNAME}"@</code>.
</ul>
<p>How it works:
<ul>
<li><code>SSH_AUTH_SOCK=</code> disables the ssh-agent for this connection.
<li>Spelling out <code>/usr/bin/ssh</code> makes sure that shell aliases, shell functions and strange directories in <code>$PATH</code> have no effect on which SSH client is used.
<li><code>-o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null</code> makes existing host keys in <code>known_hosts</code> files to be ignored, thus the connection will be established even if old or incorrect host keys are saved there. Please note that this also makes it impossible to detect a man-in-the-middle attack, so attackers may be able to steal your password if you use a password to log in; also attackers can steal the contents of your session (commands and their results).
<li><code>-o StrictHostKeyChecking=no</code> suppresses the prompt to add the host key to the <code>known_hosts</code> files.
</ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-20732307002763897352018-04-24T17:40:00.001+02:002018-04-24T17:41:12.798+02:00A quest to find a fast enclosure for multiple SATA 3.5" hard drives<p>This blog post documents the quest I'm undertaking to find a fast enclosure for multiple SATA 3.5" hard drives, supporting both USB 3 and eSATA, and the ability to read from both hard drives at the same time with at least 275 MB/s total speed. So far I haven't found a fast enough enclosure, so the quest is till ongoing. I'll keep updating the blog post with speed benchmark results.
<p>The maximum sequential read speed my drives support are 112 MB/s and 170 MB/s. (There are much faster drives on the market, e.g. Seagate IronWolf NAS 10 TB can read 250 MB/s in the first 1 TB of the disk.)
<p>I've decided not to order the <i>IcyBox IB-RD3662U3S,</i> because my online research indicates it would be too slow. It uses the chipset <a href="http://www.jmicron.com/PDF/brief/jmb352.pdf">JMicron JMB 352</a> (produced in 2014), which doesn't support UASP (thus it's slow and it uses too much CPU) and maximum SATA speed is 3 Gbit/s.
<p>I've ordered the <i>StarTech S3520BU33ER</i> instead, which uses the chipset <a href="http://www.jmicron.com/PDF/brief/jms562.pdf">JMicron JMS 562</a> (also produced in 2014), which supports UASP and maximum SATA speed is 6 Gbit/s. I'll run the benchmarks after it arrives.
<p>I've also found <i>OWC 0GB Mercury Elite Pro Dual RAID USB 3.1 / eSATA Enclosure Kit,</i> which is potentially even faster. It supports USB 3.1, eSATA, UASP, and claims to be very fast: more than 400 MB/s over both USB and eSATA. It also uses the same chipset: <a href="http://www.jmicron.com/PDF/brief/jms562.pdf">JMicron JMS 562</a>. It's avaialable <a href="https://www.amazon.com/OWC-Mercury-Elite-eSATA-Enclosure/dp/B06XRK93R9">from amazon.com</a> and <a href="https://eshop.macsales.com/item/OWC/MED3ER0GB/">from the manufacturer's webshop</a> (with expensive international delivery).
<p>Depending on the computer it can be much faster to connect the 2 hard drives within separate single-drive enclosures, using separate USB 3 ports or using an unpowered hub. I'm not pursuing this option right now, because I have other uses for my USB ports, and I want low CPU usage (eSATA uses less than USB 3).
<p>For a home media server, it may be cheaper to buy a NAS, e.g. QNAP TS-251+ with Ethernet and HDMI ports, DLNA with full HD video transcoding and other media server features, with maximum transfer speed of 224 MB/s. (Other kinds of QNAP NASes don't seem the be any faster.) However, with a NAS I wouldn't get the flexibility and configurability of stock Debian operating system running on a stock amd64 CPU with 4 GiB of RAM on <a href="http://pcengines.ch/apu2c4.htm">this machine</a>.
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-3210948352101665172018-04-21T15:56:00.002+02:002020-04-07T21:20:27.496+02:00How to update the BIOS on a Lenovo T400 laptop<p>This blog post explains how to update the BIOS to version 3.24 (released on 2012-12-16, latest release as of 2018-04-21) on a Lenovo T400 laptop.
<p><b>This instructions below don't seem to work, I get an <i>Operating system not found</i> error when booting from the pen drive.</b> Unfortunately I don't know how to fix that.
<p>You will need a working and charged battery pack for the BIOS update, so install the battery pack first and start charging it.
<p>If you are running Windows XP, Windows Vista or Windows 7 on the laptop, download the BIOS Update Utility from <a href="https://support.lenovo.com/ch/en/downloads/migr-70350">here</a> (choose 32-bit or the 64-bit version depending on your Windows type, or try both versions if you don't know), and run it, and you are done.
<p>Otherwise, if you are able to burn a CD or DVD (either on the Lenovo T400 laptop or on another computer), and you have a working DVD reader in the Lenovo T400, then download the installer DVD .iso from <a href="https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/7uuj49uc.iso">here</a>, burn it to a DVD, insert the DVD to the Lenovo T400, reboot the Lenovo T400, press the blue ThinkVantage button (near the top left corner of the keyboard), press <i>F12</i> to select a boot device, select the DVD, boot from it.
<p>Otherwise, if you have a USB pen drive of at least 34 MB in size whose contents can be overwritten, and you have a Linux system running (either on the Lenovo T400 laptop or on another computer), then connect the pen drive, figure out the device name using <code>sudo fdisk -l</code> (typically it will be <code>/dev/sdb</code> or <code>/dev/sdc</code>, but be extra careful, otherwise you will overwrite the contents of some other drive), run this command to download: <code>wget https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/7uuj49uc.iso</code>; run this command to copy the bootable BIOS update utility to the pen drive: <code>sudo dd if=7uuj49uc.iso of=/dev/sdB bs=49152 skip=1; sync</code> (replacing <code>/dev/sdB</code> with the device of the pen drive). Insert the pen drive to one of the USB slots of the Lenovo T400, reboot the Lenovo T400, press the blue ThinkVantage button (near the top left corner of the keyboard), press <i>F12</i> to select a boot device, select USB HDD, boot from it.
<p>After booting into the BIOS update utility, follow the instructions to update the system software. (Don't reboot or turn off until asked.) The next reboot will take longer, the Lenovo logo will appear and disappear 3 times. After that you are done.
<p>Now if you enter the BIOS setup at boot time (by pressing the blue ThinkVantage button), you will see version 3.24 (7UET94WW) 2012-10-17.Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-8821846818714029393.post-35111859844991858182018-04-09T17:50:00.000+02:002018-04-09T17:50:17.433+02:00How to change which characters are selected by double-clicking in xterm<p>Various terminal emulators on Linux (e.g. xterm, gnome-terminal, rxvt) have word selection: when you double-click a character, it selects the entire word containing the character. This blog post explains how to customize which characters are part of a word in xterm.
<p>The various defaults are for ASCII characters (in addition to digits and the letters a-z and A-Z):
<ul>
<li>gnome-terminal: <code># % & + , - . / = ? @ \ _ ~</code>
<li>rxvt: <code>! # $ % + - . / : _</code>
<li>xterm default: <code>_</code>
<li>xterm in Ubuntu: <code>! # % & + , - . / : = ? @ _ ~</code>
</ul>
<p>It's possible to customize which characters are part of a word in xterm by specifying the <i>charClass</i> resource. The values <code>:48</code> mean: consider these characters part of a word. Other numbers are character ranges, for example <code>43-47</code> mean the ASCII characters 43 (+), 44 (,), 45 (-), 46 (.) and 47 (/).
<p>Here is how to trigger various default behaviors from the command-line:
<ul>
<li>gnome-terminal: <code>xterm -xrm '*.VT100.charClass: 35:48,37:48,38:48,43-47:48,61:48,63-64:48,92:48,95:58,126:48'</code>
<li>rxvt: <code>xterm -xrm '*.VT100.charClass: 33:48,35-37:48,43:48,45-47:48,58:48,95:58'</code>
<li>xterm default: <code>xterm -xrm '*.VT100.charClass: 95:48'</code>
<li>xterm in Ubuntu: <code>xterm -xrm '*.VT100.charClass: 33:48,35:48,37-38:48,43-47:48,58:48,61:48,63-64:48,95:48,126:48'</code>
</ul>
<p>To save the setting permanently, add a line like this to your <i>~/.Xresources</i> file (create it if it doesn't exist):
<pre>! Here is a pattern that is useful for double-clicking on a URL (default xterm in Ubuntu):
XTerm.VT100.charClass: 33:48,35:48,37-38:48,43-47:48,58:48,61:48,63-64:48,95:48,126:48</pre>
<p>Make sure above that the line containing <i>charClass</i> doesn't start with <code>!</code>, because that would be a comment.
<p>The change takes affect automatically the next time you log in. To make it take effect earlier (for all xterms you start), run: <code>xrdb -merge ~/.Xresources</code>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-26400047676788977742017-12-06T17:00:00.000+01:002017-12-06T17:00:11.322+01:00How to restrict an SSH user to file transfers<p>This blog post explains how a user on a Unix server can be restricted to file transfers only over SSH. The restriction is implemented by specifying a login shell which imposes a whitelist of allowed commands (e.g. <i>rsync</i>, <i>sftp-server</i>, <i>scp</i>, <i>mkdir</i>), and Unix permissions are used to restrict which files can be read and/or written by these commands.
<h3>Implementation using a custom login shell</h3>
<p>First install Python 2 (as <i>/usr/bin/python</i>), then create a custom login shell binary, and save it to e.g. <code>/usr/local/bin/transfer_shell</code>. The contents of <code>/usr/local/bin/transfer_shell</code> should be:
<pre class="syntax"><span></span><span class="ch">#! /usr/bin/python</span>
<span class="c1"># by pts@fazekas.hu at Wed Dec 6 15:46:18 CET 2017</span>
<span class="sd">"""Login shell in Python 2 for SSH service restricted to data copying.</span>
<span class="sd">Use normal Unix permissions to restrict what files can be accessed.</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">stat</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="vm">__file__</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">W_OK</span><span class="p">)</span> <span class="ow">or</span> <span class="n">os</span><span class="o">.</span><span class="n">access</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="vm">__file__</span><span class="p">),</span> <span class="n">os</span><span class="o">.</span><span class="n">W_OK</span><span class="p">):</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: copy shell not safe</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s1">'SSH_ORIGINAL_COMMAND'</span><span class="p">,</span> <span class="s1">''</span><span class="p">):</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: bad command= config</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1">#cmd = os.getenv('SSH_ORIGINAL_COMMAND', '').split()</span>
<span class="c1">#print >>sys.stderr, sys.argv</span>
<span class="n">cs</span> <span class="o">=</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'-c'</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">])</span> <span class="ow">or</span> <span class="s1">''</span>
<span class="k">if</span> <span class="n">cs</span> <span class="o">==</span> <span class="s1">'/bin/sh .ssh/rc'</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="n">cs</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="c1"># cmd0 will be '' for interactive shells, thus it will be disallowed.</span>
<span class="n">cmd0</span> <span class="o">=</span> <span class="p">(</span><span class="n">cmd</span> <span class="ow">or</span> <span class="p">(</span><span class="s1">''</span><span class="p">,))[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1">#print >>sys.stderr, sorted(os.environ)</span>
<span class="k">if</span> <span class="n">cmd0</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'ls'</span><span class="p">,</span> <span class="s1">'pwd'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">,</span> <span class="s1">'cat'</span><span class="p">,</span> <span class="s1">'echo'</span><span class="p">,</span> <span class="s1">'cp'</span><span class="p">,</span> <span class="s1">'mv'</span><span class="p">,</span> <span class="s1">'rm'</span><span class="p">,</span>
<span class="s1">'mkdir'</span><span class="p">,</span> <span class="s1">'rmdir'</span><span class="p">,</span>
<span class="s1">'rsync'</span><span class="p">,</span> <span class="s1">'scp'</span><span class="p">,</span> <span class="s1">'/usr/lib/openssh/sftp-server'</span><span class="p">):</span>
<span class="c1"># In case of sftp, we can't write to stderr.</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: command not allowed: </span><span class="si">%s</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">cmd0</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">is_scp_unsafe</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
<span class="n">has_tf</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">arg</span> <span class="o">=</span> <span class="n">cmd</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">if</span> <span class="n">arg</span> <span class="o">==</span> <span class="s1">'--'</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">arg</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'-'</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">elif</span> <span class="n">arg</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'-t'</span><span class="p">,</span> <span class="s1">'-f'</span><span class="p">):</span> <span class="c1"># Flags indicating remote operation.</span>
<span class="n">has_tf</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">elif</span> <span class="n">arg</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'-v'</span><span class="p">,</span> <span class="s1">'-r'</span><span class="p">,</span> <span class="s1">'-p'</span><span class="p">,</span> <span class="s1">'-d'</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">True</span>
<span class="k">return</span> <span class="ow">not</span> <span class="n">has_tf</span>
<span class="k">if</span> <span class="p">((</span><span class="n">cmd0</span> <span class="o">==</span> <span class="s1">'rsync'</span> <span class="ow">and</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span> <span class="ow">or</span> <span class="n">cmd</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'--server'</span><span class="p">))</span> <span class="ow">or</span>
<span class="n">cmd0</span> <span class="o">==</span> <span class="s1">'scp'</span> <span class="ow">and</span> <span class="n">is_scp_unsafe</span><span class="p">(</span><span class="n">cmd</span><span class="p">)):</span>
<span class="c1"># This is to disallow arbitrary command execution with rsync -e and</span>
<span class="c1"># scp -S.</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: command-line not allowed: </span><span class="si">%s</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">cs</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'PATH'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'/bin:/usr/bin'</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'DISPLAY'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span> <span class="c1"># Disable X11.</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'XDG_SESSION_COOKIE'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'XAUTHORITY'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="s1">'data'</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: data dir not found</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># This is insecure: os.execl('/bin/sh', 'sh', '-c', cmd)</span>
<span class="n">os</span><span class="o">.</span><span class="n">execvp</span><span class="p">(</span><span class="n">cmd0</span><span class="p">,</span> <span class="n">cmd</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'error: command not found: </span><span class="si">%s</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">cmd0</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span></pre>
<p>Run these commands as root (without the leading <code>#</code>) to set the permissions <i>transfer_shell</i>:
<pre># chown root.root /usr/local/bin/transfer_shell
# chmod 755 /usr/local/bin/transfer_shell</pre>
<p>To set up restrictions for a new user
<ol>
<li>Create the Unix user if not already created.
<li>Set up Unix groups and permissions on the system so the user doesn't have access to more files than he should have.
<li>Optionally, set up SSH public keys in <code>~/.ssh/authorized_keys</code> for the user. No need to specify <code>command="..."</code> or other restrictions in that line.
<li>To change the login shell of the user, run this command as root (substituting <code><i>USER</i></code> with the login name of the user): <code>chsh -s /usr/local/bin/transfer_shell <i>USER</i></code>
<li>Create a symlink named <i>data</i> in the home directory of the user. It should point to the default directory for file transfers.
<li>It's strongly recommended that you make the home directory and its contents unwritable by the user. Example command (run it as root, substitute <code><i>USER</i></code>): <code>chown root.root ~<i>USER</i> ~<i>USER</i>/.ssh ~<i>USER</i>/.ssh/authorized_keys</code>
</ol>
<h3>Alternatives considered</h3>
<ul>
<li>Using a restrictive login shell and setting Unix file permissions. (This is implemented above, and also in <a href="https://github.com/scponly/scponly">scponly</a> and <a href="http://pizzashack.org/rssh/">rssh</a>/) The disadvantage is that by accident the Unix permissions may be set up incorrectly (i.e. they are too permissive), and the user has access to too many files. Another disadvantage is that the custom login shell implementation may be vulnerable or hard to audit (example exploits for running arbitrary commands with <i>rsync</i> and <i>scp</i>: <a href="https://www.exploit-db.com/exploits/24795/">https://www.exploit-db.com/exploits/24795/</a>).
<li>Using a restrictive <code>command="..."</code> in <code>~/.ssh/authorized_keys</code>. This is insecure, because OpenSSH sshd still runs <code>~/.bashrc</code> and <code>~/.ssh/rc</code> as shell scripts, and a malicious user could upload their own version of these files, or trigger some command execution in <code>/etc/bash.bashrc</code>. Any of these could lead to the user being able to execute arbitrary shell commands, which we don't want for this user.
<li>Running a restrictive, custom SSH server implementation on a different port (while OpenSSH sshd is still running on port 22). This comes with its own risk of possible security bugs, and needs to be upgraded regularly. Also it can be complex to understand and set up correctly.
<li>See some more alternatives here: <a href="https://serverfault.com/questions/83856/allow-scp-but-not-actual-login-using-ssh">https://serverfault.com/questions/83856/allow-scp-but-not-actual-login-using-ssh</a>.
</ul>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-34498609259476289242017-10-06T15:55:00.002+02:002017-10-06T15:55:57.143+02:00Comparison of encrypted Git remote (remote repository) implementations<p>This blog post is a comparison of encrypted Git remote implementations. A Git remote is a combination of storage space on a remote server, remote server software and local software working together. An encrypted Git remote is a Git remote which makes sure that the storage space on the remote server contains the Git objects encrypted. It is useful if the Git repository contains sensitive information (e.g. passwords, bank account details), and the remote server is not trusted to keep such information hidden from unauthorized readers.
<p>See the recent Hacker News dicsussion <a href="https://news.ycombinator.com/item?id=15401211">Keybase launches encrypted Git</a> for the encrypted, hosted cloud Git remote provided by Keybase.
<h3>Comparison</h3>
<ul>
<li>name of the Git remote software
<ul>
<li>grg: git-remote-gcrypt
<li>git-gpg: git-gpg
<li>keybase: git-remote-keybase, the encrypted, hosted cloud Git remote provided by Keybase
</ul>
<li>does it support collaboration (users with different keys pull and push)?
<ul>
<li>grg: yes
<li>git-gpg: yes
<li>keybase: yes
</ul>
<li>does it encrypt the local .git repository directory?
<ul>
<li>grg: no
<li>git-gpg: no
<li>keybase: no
</ul>
<li>does it encrypt any files in the local working tree?
<ul>
<li>grg: no
<li>git-gpg: no
<li>keybase: no
</ul>
<li>does it encrypt the remote repository users push to?
<ul>
<li>grg: yes, it encrypts locally before push
<li>git-gpg: yes, it encrypts locally before push
<li>keybase: yes, it encrypts locally before push
</ul>
<li>by looking at the remote files, can anyone learn the total the number of Git objects?
<ul>
<li>grg: no
<li>git-gpg: yes
<li>keybase: probably yes
</ul>
<li>can root on the remote server learn the list of contributors (users who do git pull and/or git push)?
<ul>
<li>grg: yes, by making sshd log which SSH public key was used
<li>git-gpg: yes, by making sshd log which SSH public key was used
<li>keybase: yes
</ul>
<li>by looking at the remote files, can anyone learn the list of contributors (users who do git pull and/or git push)?
<ul>
<li>grg: no
<li>git-gpg: no
<li>keybase: probably yes
</ul>
<li>by looking at the remote files, can anyone learn when data was pushed?
<ul>
<li>grg: yes
<li>git-gpg: yes
<li>keybase: probably yes
</ul>
<li>does it support hosting of encrypted remotes on your own server?
<ul>
<li>grg: yes
<li>git-gpg: yes
<li>keybase: no, at least not by default, and not documented
</ul>
<li>supported remote repository types
<ul>
<li>grg: rsync, local directory, sftp, git repo (local or remote)
<li>git-gpg: rsync, local directory
<li>keybase: custom, data is stored on KBFS (Keybase filesystem, an encrypted
network filesystem)
</ul>
<li>required software on the remote server
<ul>
<li>grg: sshd, (rsync or sftp-server or git)
<li>git-gpg: sshd, rsync
<li>keybase: custom, the KBFS server, there are no official installation
instructions
</ul>
<li>required local software
<ul>
<li>grg: git, gpg, ssh, (rsync or sftp), git-remote-gcrypt
<li>git-gpg: git, gpg, ssh, rsync, Python (2.6 or 2.7), git-gpg
<li>keybase: binaries provided by Keybase: keybase, git-remote-keybase,
kbfsfuse (only for remote repository creation)
</ul>
<li>product URL with installation instructions
<ul>
<li>grg: https://git.spwhitton.name/git-remote-gcrypt/tree/README.rst
<li>git-gpg: https://github.com/glassroom/git-gpg
<li>keybase: https://keybase.io/blog/encrypted-git-for-everyone
</ul>
<li>source code URL
<ul>
<li>grg: https://git.spwhitton.name/git-remote-gcrypt/tree/git-remote-gcrypt
<li>git-gpg: https://github.com/glassroom/git-gpg/blob/master/git-gpg
<li>keybase: https://github.com/keybase/kbfs/blob/master/kbfsgit/git-remote-keybase/main.go
</ul>
<li>implementation language
<ul>
<li>grg: Unix shell (e.g. Bash), single file
<li>git-gpg: Python 2.6 and 2.7, single file
<li>keybase: Go
</ul>
<li>source code size, number of bytes, including comments
<ul>
<li>grg: 21 448 bytes
<li>git-gpg: 19 702 bytes
<li>keybase: 5 617 305 bytes (including <code>client/go/libkb/**/*.go</code> and
<code>kbfs/{env,kbfsgit,libfs,libgit,libkbfs}/**/*.go</code>)
</ul>
<li>is the source code easy to understand?
<ul>
<li>grg: yes, but some developers reported it's less easy than git-gpg
<li>git-gpg: yes
<li>keybase: no, because it's huge; individual pieces are simple
</ul>
<li>encryption tool used
<ul>
<li>grg: gpg (works with old versions, e.g. 1.4.10 from 2008)
<li>git-gpg: gpg (works with old versions, e.g. 1.4.10 from 2008)
<li>keybase: custom, written in Go
</ul>
<li>is it implemented as a Git remote helper?
<ul>
<li>grg: yes, <code>git push</code> etc. works
<li>git-gpg: no, it works as <code>git gpg push</code> instead of <code>git
push</code> etc.
<li>keybase: yes, <code>git push</code> etc. works
</ul>
<li>how much extra disk space does it use locally, per repository?
<ul>
<li>grg: less than 1000 bytes
<li>git-gpg: stores 2 extra copies of the .git repository locally, one of them
containing only loose objects (thus mostly uncompressed)
<li>keybase: less than 1000 bytes
</ul>
<li>how much disk space does it use remotely, per repository?
<ul>
<li>grg: one encrypted packfile for each push, encryption has a small
(constant) overhead, occasionally runs <code>git repack</code> (locally, and
uploads the result), and right after repacking it stores only 1 packfile
(plus a small metadata file) per repository
<li>git-gpg: one encrypted file for each object, encryption has a small
(constant) overhead, no packfiles (thus the remote repository will be
large and contain a lot of files, because of the lack of diff
compression supported by the packfiles)
<li>probably one encrypted file for each object
</ul>
</ul>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-15702729480059741442017-09-02T15:39:00.003+02:002018-01-04T15:14:04.238+01:00How to run Windows XP on Linux using QEMU and KVM<p>This blog post is a tutorial explaining how to run Windows XP as a guest operating system using QEMU and KVM on a Linux host. It should take less then 16 minutes, including installation.
<p>Requirements: You need a recent Linux system (Ubuntu 14.04 LTS will work) with a GUI, 620 MB of free disk space and 550 MB of free memory. If you don't want to browse the web from Windows XP, then 300 MB of free memory is enough.
<p>Software used:
<ul>
<li>The latest version of Hiren's BootCD (version 15.2) was released on 2012-11-09. It contains a live (no need to install) mini Windows XP system with a web browser (Opera). (Additionally, it contains hundreds of system rescue, data recovery, antivirus, backup, password recovery, hard disk diagnostics and system diagnostics tools. To see many of them with screenshot, look at this <a href="http://us.informatiweb.net/tutorials/it/1-articles/60--hiren-boot-cd-mini-windows-xp.html">article about Hiren's BootCD</a>, or click on the <i>See CD Contents</i> link on the <a href="http://www.hirensbootcd.org/download/">official Hiren's BootCD download page</a>.)
<li>QEMU. It's a fully system emulator, which can emulate multiple architectures, and it can run multiple operating systems as a guest.
<li>KVM. It's a fast virtualization (emulation) of guest operating systems on Linux. It's used by QEMU, and it lets QEMU execute the CPU-intensive operations on guest systems quickly, with only 10% or less overhead. (I/O-intensive operations can be much slower.)
</ul>
<p>Log in to the GUI, open a terminal window, and run the following command (without the leading <code>$</code>, copy-paste it as a single, big, multiline paste):
<pre>$ python -c'import os, struct, sys, zlib
def work(f): # Extracts the .iso from the .zip on the fly.
while 1:
data, i, c, s = f.read(4), 0, 0, 0
if data[:3] in ("PK\1", "PK\5", "PK\6"): return f.read()
assert data[:4] == "PK\3\4", repr(data); data = f.read(26)
_, _, mth, _, _, _, cs, us, fnl, efl = struct.unpack("<HHHHHlLLHH", data)
fn = f.read(fnl); assert len(fn) == fnl
ef = f.read(efl); assert len(ef) == efl
if fn.endswith(".iso"): uf = open("hirens.iso", "wb")
else: mth = -1
if mth == 8: zd = zlib.decompressobj(-15)
while i < cs:
j = min(65536, cs - i); data = f.read(j); assert len(data) == j; i += j
if mth == 8: data = zd.decompress(data)
if mth != -1: uf.write(data)
if mth == 8: uf.write(zd.flush())
work(os.popen("wget -nv -O- "
"http://www.hirensbootcd.org/files/Hirens.BootCD.15.2.zip"))'</pre>
<p>The command above downloads the Hiren's BootCD image and extracts it to the file <i>hirens.iso</i>. (Alternatively, you could download from your browser and extract the <i>.iso</i> manually. That would use more temporary disk space.)
<p>Install QEMU. If you have a Debian or Ubuntu system, do it so by running the command (without the leading <code>$</code>):
<pre>$ sudo apt-get install qemu-system-x86</pre>
<p>On other Linux systems, use your package manager to install QEMU with KVM support.
<p>The only step in this tutorial which needs root access (and thus the root password) is the QEMU installation above.
<p>Run the following command in your terminal window (without the leading <code>$</code>, copy-paste it):
<pre>$ SDL_VIDEO_X11_DGAMOUSE=0 qemu-system-i386 -m 512 -machine pc-1.0,accel=kvm \
-cdrom hirens.iso -localtime -net nic -net user -smb "$HOME"</pre>
<p>This command will start a virtual machine running Hiren's Boot CD, and it will display it in a window (of size 800x600). The command will not exit until you close the window (and thus abort the virtual machine).
<p>The virtual machine will use 512 MB of memory (as specified by <code>-mem 512</code> above. It's possible for the mini Windows XP to use less memory, e.g. you if you specify <code>-mem 256</code> instead, then it will still work, but web browsing (with Opera) won't work, and you will have to click OK on the <i>Your system is low on vitual memory.</i> dialog later.
<p>In a few seconds, the boot menu of Hiren's BootCD is displayed in the QEMU window:
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjveQ3iK5jKrVgNtIJRze6d1Jgv3HQuzcRHIRAb4LzjbiIIMggNtc-AliRHiEXuACNF3Ba2WQ6blR4rfQ8-iA__EEWgE-2BDnFBAUPAqBfqnXNr-8A5gwk35QVF9FpkYd_wgWrndzZbLnEX/s1600/ss1.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjveQ3iK5jKrVgNtIJRze6d1Jgv3HQuzcRHIRAb4LzjbiIIMggNtc-AliRHiEXuACNF3Ba2WQ6blR4rfQ8-iA__EEWgE-2BDnFBAUPAqBfqnXNr-8A5gwk35QVF9FpkYd_wgWrndzZbLnEX/s1600/ss1.png" data-original-width="722" data-original-height="428" /></a>
<p>Press the down arrow key and press <i>Enter</i> to choose <i>Mini Windows Xp</i>. Then wait about 1 minute for Windows XP to start. It will look like this:
<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnQyHZXovi0sVbP01nQSxbI4PvLZ3FHDnDRgP2_6YwZT9BsaKXckbdI_sKTi_yxxdbcNd760tK10i_ic-rMCkPocZH1PnPCcelmXQq_fsVxiY_bcDuENbXeONlUPmhKTDOU00dwBwXYPm_/s1600/ss2.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnQyHZXovi0sVbP01nQSxbI4PvLZ3FHDnDRgP2_6YwZT9BsaKXckbdI_sKTi_yxxdbcNd760tK10i_ic-rMCkPocZH1PnPCcelmXQq_fsVxiY_bcDuENbXeONlUPmhKTDOU00dwBwXYPm_/s1600/ss2.jpg" data-original-width="802" data-original-height="628" /></a>
<p>To use the mouse within the QEMU window, click on the window. To release your mouse (to be used in other windows), press <i>Ctrl</i> and <i>Alt</i> at the same time.
<p>Networking (such as web and file sharing) is not enabled by default. To enable it, click on the <i>Network Setup</i> icon in the QEMU window desktop, and wait about 20 seconds. The IP address of the guest Windows XP is 10.0.2.15, and the IP address of host Linux system is 10.0.2.2. Because of the user mode networking emulation provided by QEMU, external TCP connections can also be made from Windows XP (e.g. you can browse the web). Please note that <i>ping</i> won't work (because QEMU doesn't emulate that).
<p>To browse the web, click on the <i>Internet</i> icon in the QEMU Windows desktop. It will start the Opera browser. Web browsing will be quite slow, so better try some fast sites such as <i>google.com</i> or <i>whatismyip.com</i>.
<p>To use the command line, click on the <i>Command prompt</i> icon in the QEMU Windows desktop. There is a useful command to type to that window: <i>net use s: \\10.0.2.4\qemu</i> (press <i>Enter</i> after typing it). That will make your Linux home folder available as drive S: in Windows XP, for reading and writing. (You can change which folder to make available by specifying it after <code>-smb</code> when starting QEMU.)
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfzDSLGKazgLvd_KAMbQU_fbYG8KzA_nJcDoHF8SqWynmtYYV41YGC352jW_2-NUXiiUrqcaQXB9BkVksQNYXJF8jjX5Iovjyd2BS54kIg9ps3BkvPD2D1G8RoFvZZAXXn7nHPCXyj11nn/s1600/ss3.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfzDSLGKazgLvd_KAMbQU_fbYG8KzA_nJcDoHF8SqWynmtYYV41YGC352jW_2-NUXiiUrqcaQXB9BkVksQNYXJF8jjX5Iovjyd2BS54kIg9ps3BkvPD2D1G8RoFvZZAXXn7nHPCXyj11nn/s1600/ss3.jpg" data-original-width="802" data-original-height="628" /></a>
<p>Copy-pasting between Linux and Windows XP clipboards doesn't work.
<p>You can make the QEMU window larger by changing <i>Start menu / Settings / Control Panel / Display / Settings / Screen resoluton</i> to <i>1024 by 768 pixels</i>. The <i>1024x768</i> shortcut on the QEMU Windows desktop doesn't work,
<p>Because of efficient CPU virtualization by KVM, an idle Windows XP in a QEMU window doesn't use more than 10% CPU on the host Linux system.
<p>Hiren's boot CD contains hundrends of Windows apps. Only a fraction of the apps are available from the Windows XP start menu. To see all apps, click on the <i>HBCD Menu</i> icon in the QEMU Windows desktop, and then click on the <i>Browser Folder</i> button.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-37471156214331721752017-02-28T18:11:00.000+01:002017-03-05T20:47:12.589+01:00How to avoid unnecessary copies when appending to a C++ vector<p>This blog post explains how to avoid unnecessary copies when appending to a C++ <i>std::vector</i>, and recommends the <a href="https://github.com/pts/fast_vector_append">fast_vector_append helper library</a>, which eliminates most copies automatically.
<p><b>TL;DR</b> If you are using C++11, and your element classes have an efficient <a href="http://en.cppreference.com/w/cpp/language/move_constructor">move constructor</a> defined, then just use <a href="http://en.cppreference.com/w/cpp/container/vector/push_back">push_back</a> to append, it won't do any unnecessary copies. In addition to that, if you are constructing the to-be-appended element, you can use <a href="http://en.cppreference.com/w/cpp/container/vector/emplace_back">emplace_back</a> to append, which even avoids the (otherwise fast) call to the move constructor.
<p>Copying is slow and needs a lot of (temporary memory) if the object contains lots of data. Such an object is a long <i>std::string</i>: the entire array of characters get copied to a new array. This hurts performance if the copy is unnecessary, e.g. if only a temporary copy is made. For example:
<pre class="syntax"><span></span><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">create_long_string</span><span class="p">(</span><span class="kt">int</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">v</span><span class="p">;</span>
<span class="p">{</span>
<span class="c1">// Case A.</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s1</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s2</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s3</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span>
<span class="c1">// Case B.</span>
<span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">s1</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="n">s1</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Case C.</span>
<span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="s">"foo"</span><span class="p">);</span>
<span class="c1">// Case D, from C++11.</span>
<span class="n">v</span><span class="p">.</span><span class="n">emplace_back</span><span class="p">(</span><span class="s">"foo"</span><span class="p">);</span>
<span class="c1">// Case E.</span>
<span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">create_long_string</span><span class="p">(</span><span class="mi">4</span><span class="p">));</span>
<span class="c1">// Case F.</span>
<span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">());</span> <span class="n">v</span><span class="p">.</span><span class="n">back</span><span class="p">().</span><span class="n">swap</span><span class="p">(</span><span class="n">s2</span><span class="p">);</span>
<span class="c1">// Case G, from C++11.</span>
<span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">s3</span><span class="p">));</span></pre>
<p>In Case A, <a href="https://en.wikipedia.org/wiki/Return_value_optimization">return value optimization</a> prevents the unnecessary copying: the string built in the function body of <i>create_long_string</i> is placed directly to <i>s1</i>.
<p>In Case B, a copy has to be made (there is no way around it), because <i>v</i> is still valid after <i>s1</i> is destroyed, thus it cannot reuse the data in <i>s1</i>.
<p>Case C could work without a copy, but in C++98 an unnecessary copy is made. So first <code>std::string("foo")</code> is called (which makes a copy of the data), and then the copy constructor of <i>std::string</i> is called to create a new string (with a 2nd copy of the data), which gets added to <i>v</i>.
<p>Case D avoids the 2nd (unnecessary) copy, but it works only from C++11. In earlier versions of C++ (such as C++98), <i>std::vector</i> doesn't have the <i>emplace_back</i> method.
<p>In Case E, there is an unnecessary copy in C++98: <i>create_long_string</i> creates an <i>std::string</i>, and it gets copied to a new <i>std::string</i> within <i>v</i>. It would be better if <i>create_long_string</i> could create the <i>std::string</i> to its final location.
<p>Case F shows the workaround in C++98 of adding <i>s2</I> to an <i>std::vector</i> without a copy. It's a workaround because it's a bit ugly and it still involves some copying. Fortunately this copying is fast: it copies only the empty string. As a side effect, the value of <i>s2</i> is lost, it will then be the empty string.
<p>Case G shows the C++11 way of adding <i>s3</I> to an <i>std::vector</i> without a copy. It doesn't work in C++98 (there is no <i>std::move</i> in C++98). The <code>std::move(s3)</code> visibly documents that the old value of <i>s3</i> is lost.
<p>C++11 (the version of C++ after C++98) introduces rvalue references, move constructors and move semantics to avoid unnecessary copies. This will fix both Case C and Case E. For this to work, new code needs to be added to the element class (in our case <i>std::string</i>) and to the container class (in our case <i>std::vector</i>) as well. Fortunately, the callers (including our code above and the body of <i>create_long_string</i>) can be kept unchanged. The following code has been added to the C++ standard library (STL) in C++11:
<pre class="syntax"><span></span><span class="k">class</span> <span class="nc">string</span> <span class="p">{</span>
<span class="p">...</span>
<span class="c1">// Copy constructor. C++98, C++11.</span>
<span class="n">string</span><span class="p">(</span><span class="k">const</span> <span class="n">string</span><span class="o">&</span> <span class="n">old</span><span class="p">)</span> <span class="p">{</span> <span class="p">...</span> <span class="p">}</span>
<span class="c1">// Move constructor. Not in C++98, added in C++11.</span>
<span class="n">string</span><span class="p">(</span><span class="n">string</span><span class="o">&&</span> <span class="n">old</span><span class="p">)</span> <span class="p">{</span> <span class="p">...</span> <span class="p">}</span> <span class="p">...</span> <span class="p">}</span>
<span class="p">};</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="p">,</span> <span class="p">...</span><span class="o">></span>
<span class="k">class</span> <span class="nc">vector</span> <span class="p">{</span>
<span class="p">...</span>
<span class="c1">// Takes a const reference. C++98, C++11.</span>
<span class="kt">void</span> <span class="n">push_back</span><span class="p">(</span><span class="k">const</span> <span class="n">T</span><span class="o">&</span> <span class="n">t</span><span class="p">);</span>
<span class="c1">// Takes an rvalue reference. Not in C++98, added in C++11.</span>
<span class="kt">void</span> <span class="nf">push_back</span><span class="p">(</span><span class="n">T</span><span class="o">&&</span> <span class="n">t</span><span class="p">);</span>
<span class="p">};</span></pre>
<p>As soon as both of these are added, when <code>v.push_back(...)</code> will attempt to call the 2nd method (which takes the rvalue reference), which will call to move constructor of <i>std::string</i> instead of the copy constructor. This gives us the benefit of no copying, because typically the move constructor is fast, because it doesn't copy data. In general, the move constructor creates the new object with the data of the <i>old</i> object, and it can leave the old old object in an arbitrary but valid state. For <i>std::string</i>, it just copies the pointer to the data (which is fast, because it doesn't copy the data itself), and sets the pointer in the old <i>std::string</i> to <i>nullptr</i>. Thus Case C and Case E become fast in C++11. Case B is not affected (it still copies), and that's good, because we want to print <i>s1</i> to <i>cout</i> below, so we want that data there. This happens automatically, because in the call <code>v.push_back(s1)</code>, <i>s1</i> is not an rvalue reference, thus the cost-reference <i>push_back</i> will be called, which does a copy. For more details about the magic to select the proper <i>push_back</i>, see <a href="http://www.cprogramming.com/c++11/rvalue-references-and-move-semantics-in-c++11.html">this tutorial</a> or <a href="http://thbecker.net/articles/rvalue_references/section_01.html">this tutorial</a>.
<h3>Guidelines to avoid unnecessary copies</h3>
<p>Define your (element) classes like this:
<ul>
<li>Define the default constructor (<code>C() { ... }</code>).
<li>Define the destructor (<code>~C() { ... }</code>).
<li>Define the copy constructor (<code>C(const C& c) { ... }</code>).
<li>It's a good practice to define operator=, but not needed here.
<li>For C++11 classes, define a move constructor (e.g. <code>C(C&& c) { ... }</code>).
<li>For C++11 classes, don't define a member <i>swap</i> method. If you must define
it, then also define a method <code>void shrink_to_fit() { ... }</code>. It
doesn't matter what the method does, you can just declare it. The
<a href="https://github.com/pts/fast_vector_append">fast_vector_append library</a>
detects <i>shrink_to_fit</i>, and will use the move constructor instead of the <i>swap</i>
method, the former being slightly faster, although neither copies the data.
<li>For C++98 classes, don't define a move constructor. In fact, C++98
doesn't support move constructors.
<li>For C++98 classes, define a member <i>swap</i> method.
</ul>
<p>To append a new element to an <i>std::vector</i> without unnecessary copying, as fast as possible, follow this advice from top to bottom:
<ul>
<li>If it's C++11 mode, and the object is being constructed (not returned
by a function!), use emplace_back without the element class name.
<li>If it's C++11 mode, and the class has a move constructor, use <i>push_back</i>.
<li>If it's C++11 mode, and the class has the member <i>swap</i> method, use:
<code>{ C c(42); v.resize(v.size() + 1); v.back().swap(c); }</code>
<li>If the class has the member <i>swap</i> method, use:
<code>{ C c(42); v.push_back(C()); v.back().swap(c); }</code>
<li>Use <i>push_back</i>. (This is the only case with a slow copy.)
</ul>
<h3>Automating the avoidance of unnecessary copies when appending to a vector</h3>
<p>It would be awesome if the compiler could guess the programmer's intentions, e.g. it would pick <i>emplace_back</i> if it is faster than <i>push_back</i>, and it will avoid the copy even in C++98 code, e.g. it will use <i>swap</i> if it's available, but the move constructor isn't. This is important because sometimes it's inconvenient to modify old parts of a codebase defining the element class, and it already has <i>swap</i>.
<p>For automation, use <code>fast_vector_append(v, ...)</code> in the <a href="https://github.com/pts/fast_vector_append">fast_vector_append library</a> to append elements to an <i>std::vector</i>. It works in both C++98 and C++11, but it can avoid more copies in C++11. The example above looks like:
<pre class="syntax"><span></span><span class="cp">#include</span> <span class="cpf">"fast_vector_append.h"</span><span class="cp"></span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">create_long_string</span><span class="p">(</span><span class="kt">int</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">></span> <span class="n">v</span><span class="p">;</span>
<span class="p">{</span>
<span class="c1">// Case A. No copy.</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s1</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s2</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s3</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span>
<span class="c1">// Case B. Copied.</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">s1</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="n">s1</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Case C. Not copied.</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="s">"foo"</span><span class="p">);</span>
<span class="c1">// Case D. Not copied.</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="s">"foo"</span><span class="p">);</span>
<span class="c1">// Case E. Copied in C++98.</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">4</span><span class="p">));</span>
<span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">s4</span> <span class="o">=</span> <span class="n">create_long_string</span><span class="p">(</span><span class="mi">4</span><span class="p">);</span>
<span class="c1">// Case E2. Not copied.</span>
<span class="n">fast_vector_append_move</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">s4</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Case F. Not copied.</span>
<span class="n">fast_vector_append_move</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">s2</span><span class="p">);</span>
<span class="c1">// Case G. Not copied.</span>
<span class="n">fast_vector_append_move</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">s3</span><span class="p">);</span>
<span class="c1">// Case H. Copied in C++98.</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">(</span><span class="s">"foo"</span><span class="p">));</span></pre>
<h3>Autodetection of class features with SFINAE</h3>
<p>The library <i>fast_vector_append</i> does some interesting <a href="https://en.wikipedia.org/wiki/Substitution_failure_is_not_an_error">SFINAE</a> tricks to autodetect the features of the element class, so that it will be able to use the fastest way of appending supported by the class.
<p>For example, this is how it detects whether to use the member <i>swap</i> method:
<pre class="syntax"><span></span><span class="c1">// Use swap iff: has swap, does't have std::get, doesn't have shrink_to_fit,</span>
<span class="c1">// doesn't have emplace, doesn't have remove_suffix. By doing so we match</span>
<span class="c1">// all C++11, C++14 and C++17 STL templates except for std::optional and</span>
<span class="c1">// std::any. Not matching a few of them is not a problem because then member</span>
<span class="c1">// .swap will be used on them, and that's good enough.</span>
<span class="c1">//</span>
<span class="c1">// Based on HAS_MEM_FUNC in http://stackoverflow.com/a/264088/97248 . </span>
<span class="c1">// Based on decltype(...) in http://stackoverflow.com/a/6324863/97248 .</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">T</span><span class="o">></span>
<span class="k">struct</span> <span class="n">__aph_use_swap</span> <span class="p">{</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">U</span><span class="p">,</span> <span class="n">U</span><span class="o">></span> <span class="k">struct</span> <span class="n">type_check</span><span class="p">;</span>
<span class="c1">// This also checks the return type of swap (void). The checks with</span>
<span class="c1">// decltype below don't check the return type.</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">B</span><span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_swap</span><span class="p">(</span><span class="n">type_check</span><span class="o"><</span><span class="kt">void</span><span class="p">(</span><span class="n">B</span><span class="o">::*</span><span class="p">)(</span><span class="n">B</span><span class="o">&</span><span class="p">),</span> <span class="o">&</span><span class="n">B</span><span class="o">::</span><span class="n">swap</span><span class="o">>*</span><span class="p">))[</span><span class="mi">2</span><span class="p">];</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_swap</span><span class="p">(...))[</span><span class="mi">1</span><span class="p">];</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">B</span><span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_get</span><span class="p">(</span><span class="k">decltype</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">get</span><span class="o"><</span><span class="mi">0</span><span class="o">></span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">B</span><span class="o">*</span><span class="p">)</span><span class="mi">0</span><span class="p">),</span> <span class="mi">0</span><span class="p">)))[</span><span class="mi">1</span><span class="p">];</span> <br><span class="c1">// ^^^ C++11 only: std::pair, std::tuple, std::variant, std::array.</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_get</span><span class="p">(...))[</span><span class="mi">2</span><span class="p">];</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">B</span><span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_s2f</span><span class="p">(</span><span class="k">decltype</span><span class="p">(((</span><span class="n">B</span><span class="o">*</span><span class="p">)</span><span class="mi">0</span><span class="p">)</span><span class="o">-></span><span class="n">shrink_to_fit</span><span class="p">(),</span> <span class="mi">0</span><span class="p">)))[</span><span class="mi">1</span><span class="p">];</span> <br><span class="c1">// ^^^ C++11 only: std::vector, std::deque, std::string, std::basic_string.</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_s2f</span><span class="p">(...))[</span><span class="mi">2</span><span class="p">];</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">B</span><span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_empl</span><span class="p">(</span><span class="k">decltype</span><span class="p">(((</span><span class="n">B</span><span class="o">*</span><span class="p">)</span><span class="mi">0</span><span class="p">)</span><span class="o">-></span><span class="n">emplace</span><span class="p">(),</span> <span class="mi">0</span><span class="p">)))[</span><span class="mi">1</span><span class="p">];</span> <br><span class="c1">// ^^^ C++11 only: std::vector, std::deque, std::set, std::multiset, std::map, std::multimap, std::unordered_multiset, std::unordered_map, std::unordered_multimap, std::stack, std::queue, std::priority_queue.</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_empl</span><span class="p">(...))[</span><span class="mi">2</span><span class="p">];</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="n">B</span><span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_rsuf</span><span class="p">(</span><span class="k">decltype</span><span class="p">(((</span><span class="n">B</span><span class="o">*</span><span class="p">)</span><span class="mi">0</span><span class="p">)</span><span class="o">-></span><span class="n">remove_suffix</span><span class="p">(</span><span class="mi">0</span><span class="p">),</span> <span class="mi">0</span><span class="p">)))[</span><span class="mi">1</span><span class="p">];</span> <br><span class="c1">// ^^^ C++17 only: std::string_view, std::basic_string_view.</span>
<span class="k">template</span> <span class="o"><</span><span class="k">typename</span> <span class="o">></span> <span class="k">static</span> <span class="kt">char</span> <span class="p">(</span><span class="o">&</span><span class="n">chk_rsuf</span><span class="p">(...))[</span><span class="mi">2</span><span class="p">];</span>
<span class="k">static</span> <span class="kt">bool</span> <span class="k">const</span> <span class="n">value</span> <span class="o">=</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">chk_swap</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2</span> <span class="o">&&</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">chk_get</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2</span> <br><span class="o">&&</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">chk_s2f</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2</span> <span class="o">&&</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">chk_empl</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2</span> <br><span class="o">&&</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">chk_rsuf</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> <span class="o">==</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">};</span></pre>
<p>The autodetection is used like this, to select one of the 2 implementations (either with <code>v.back().swap(t)</code> or <code>v.push_back(std::move(t))</code>):
<pre class="syntax"><span></span><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="p">,</span> <span class="k">typename</span> <span class="n">T</span><span class="o">></span> <span class="k">static</span> <span class="kr">inline</span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="o">::</span><span class="n">value_type</span><span class="p">,</span> <span class="n">T</span><span class="o">>::</span><span class="n">value</span> <span class="o">&&</span>
<span class="n">__aph_use_swap</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="o">::</span><span class="n">value_type</span><span class="o">>::</span><span class="n">value</span><span class="p">,</span> <span class="kt">void</span><span class="o">>::</span><span class="n">type</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">V</span><span class="o">&</span> <span class="n">v</span><span class="p">,</span> <span class="n">T</span><span class="o">&&</span> <span class="n">t</span><span class="p">)</span> <span class="p">{</span> <span class="n">v</span><span class="p">.</span><span class="n">resize</span><span class="p">(</span><span class="n">v</span><span class="p">.</span><span class="n">size</span><span class="p">()</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span> <span class="n">v</span><span class="p">.</span><span class="n">back</span><span class="p">().</span><span class="n">swap</span><span class="p">(</span><span class="n">t</span><span class="p">);</span> <span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="p">,</span> <span class="k">typename</span> <span class="n">T</span><span class="o">></span> <span class="k">static</span> <span class="kr">inline</span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="o">::</span><span class="n">value_type</span><span class="p">,</span> <span class="n">T</span><span class="o">>::</span><span class="n">value</span> <span class="o">&&</span>
<span class="o">!</span><span class="n">__aph_use_swap</span><span class="o"><</span><span class="k">typename</span> <span class="n">V</span><span class="o">::</span><span class="n">value_type</span><span class="o">>::</span><span class="n">value</span><span class="p">,</span> <span class="kt">void</span><span class="o">>::</span><span class="n">type</span>
<span class="n">fast_vector_append</span><span class="p">(</span><span class="n">V</span><span class="o">&</span> <span class="n">v</span><span class="p">,</span> <span class="n">T</span><span class="o">&&</span> <span class="n">t</span><span class="p">)</span> <span class="p">{</span> <span class="n">v</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">move</span><span class="p">(</span><span class="n">t</span><span class="p">));</span> <span class="p">}</span></pre>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-45404798690521507342017-02-24T19:13:00.004+01:002017-08-18T14:04:33.698+02:00How to back up your WhatsApp chats and photos safely on Android<p>This blog post explains how to make backups of your WhatsApp chats and photos safely on your Android device, and how to restore your backups. By safely we mean that you won't lose data unless you remove some backup files manually.
<p>WhatsApp saves all chats to the <i>WhatsApp/Databases</i> folder on the phone's storage (sdcard), and it saves all photos and other media files to the <i>WhatsApp/Media</i> folder. (In fact, from the chats only the file <i>WhatsApp/Databases/msgstore.db.cryptNNN</i> may be needed, where NNN is an integer, currently 12.) If you make a copy of these folders, and copy them back to a new or reinstalled Android device before installing WhatsApp, then this effectively restores the backup, WhatsApp will recognize and use these files the first time it's installed (you need to tap on the <i>Restore</i> button within WhatsApp). See this <a href="https://www.whatsapp.com/faq/en/android/20887921">FAQ entry</a> for more information on restoring WhatsApp backups on Android.
<p>WhatsApp supports creating backups to Google Drive (automatically, every day), and restoring those backups when the app is (re)installed. This sounds like convenient and safe, but it's not safe, you can still lose your chat history and photos (see below how). So if you care about your WhatsApp chat history and photos, and you need an automated backup for them, here is my recommendation: use the <i>FolderSync Lite</i> free Android app to make automatic backups to the cloud (it supports Google Drive, DropBox and more than 20 other cloud providers). To restore the backup, you can use FolderSync Lite, or you can download the files and copy them to your Android device manually.
<p>Here is how to set up FolderSync Lite on Android for automatic backups of WhatsApp chats, photos and other media:
<ol>
<li>Create a Google account, open <a href="http://drive.google.com/">Google Drive</a>, create a folder named <i>FolderSyncBackup-WhatsApp</i>, and within it create subfolders <i>Databases</i> and <i>Media</i> (both case sensitive). It can also be done similarly on Dropbox instead, but this tutorial focuses on Google Drive.
<li>Install FolderSync Lite to your Android device.
<li>Add your Google account to FolderSync lite.
<li>Create a folderpair for backing up chats:
<ul>
<li>Account: your Google account
<li>Unique name: wad
<li>Sync type: To remote folder
<li>Remote folder: /FolderSyncBackup-WhatsApp/Databases/
<li>Local folder: .../WhatsApp/Databases/
<li>Use scheduled sync: yes
<li>Sync itnerval: Daily
<li>Copy files to time-stamped folder: no
<li>Sync subfolders: yes
<li>Sync hidden files: yes
<li>Delete source files after sync: no
<li>Retry sync if failed: yes
<li>Only resync source files if modified (ignore target deletion): yes
<li>Sync deletions: no
<li>Overwrite old files: Always
<li>If conflicting modifications: Skip file
<li>Use WiFi: yes
<li>(Many settings below are fine, skipped here.)
</ul>
<li>Save the folderpair, and do the first sync manually.
<li>Create a folderpair for backing up media files, including photos:
<ul>
<li>Account: your Google account
<li>Unique name: wam
<li>Sync type: To remote folder
<li>Remote folder: /FolderSyncBackup-WhatsApp/Media/
<li>Local folder: .../WhatsApp/Media/
<li>(Subsequent settings are the same as above.)
</ul>
<li>Save the folderpair, and do the first sync manually.
<li>Optionally, you can turn of WhatsApp's automatic backup to Google Drive in the WhatsApp app's chat settings.
<li>To remove the WhatsApp's automatic backup files from Google Drive, go to <a href="http://drive.google.com/">Google Drive</a>, click on the gear icon (Settings), click on Settings, click on Manage Apps, find and click on the Options off WhatsApp Messenger, click on <i>Delete app data</i>, and then click on <i>Disconnect from Drive</i>.
</ol>
<p>If FolderSync starts failing consistently with the error message <i>IllegalStateException: Expected BEGIN_OBJECT but was STRING</i>, you can fix it by unlinking and reauthenticating the sync account on the Android device. To do so, open the <i>FolderSync</i> app on the device, tap <i>Accounts</i>, tap on your GoogleDrive account, tap on the black <i>UNLINK ACCOUNT</i> button, tap on the black <i>RE-AUTHENTICATE ACCOUNT</i> button, tap on <i>Save</i>, go back, tap on <i>Folderpairs</i>, and tap on the black <i>Sync</i> buttons with an error next to them.
<p>WhatsApp saves all chat history so far to a new file every day (file name pattern: <i>WhatsApp/Databases/msgstore-????-??-??.*.db.crypt*</i>). These files will accumulate and fill up your Google Drive quote in a year or two, so you may want to remove old files. You can do it manually on the <a href="http://drive.google.com/">Google Drive</a> web UI, just visit the <i>FolderSyncBackup-WhatsApp/Databases</i> folder, select old files (by date), and remove them. Alternatively, you can automate removal using an Apps Script. Here is how:
<ol>
<li>Visit <a href="http://script.google.com/">http://script.google.com/</a>
<li>You see a script editor window. Remove the existing code (<code>function myFunction() { ... }</code>).
<li>In the <i>File / Rename...</i> menu, rename the project to <i>script to remove old WhatsApp backups</i>
<li>Copy-paste the following code to the script editor window:
<pre class="syntax" style="background:#fff"><span></span><span class="kd">var</span> <span class="nx">FOLDER_NAME</span> <span class="o">=</span> <span class="s1">'FolderSyncBackup-WhatsApp'</span><span class="p">;</span>
<span class="c1">// Please note that msgstore.db.crypt12 will always be kept in addition to the dated files.</span>
<span class="kd">var</span> <span class="nx">MINIMUM_DBS_TO_KEEP</span> <span class="o">=</span> <span class="mi">8</span><span class="p">;</span> <span class="c1">// A value smaller than 8 doesn't make sense, WhatsApp keeps that much, and these files would be reuploaded by FolderSync.</span>
<span class="kd">var</span> <span class="nx">USE_TRASH</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">DO_DRY_RUN</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">getSingleFolder</span><span class="p">(</span><span class="nx">foldersIter</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">folder</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="nx">foldersIter</span><span class="p">.</span><span class="nx">hasNext</span><span class="p">())</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">folder</span> <span class="o">!=</span> <span class="kc">null</span><span class="p">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">"Multiple folders found."</span><span class="p">);</span>
<span class="nx">folder</span> <span class="o">=</span> <span class="nx">foldersIter</span><span class="p">.</span><span class="nx">next</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">folder</span> <span class="o">==</span> <span class="kc">null</span><span class="p">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">"Folder not found."</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">folder</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">getAll</span><span class="p">(</span><span class="nx">iter</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">while</span> <span class="p">(</span><span class="nx">iter</span><span class="p">.</span><span class="nx">hasNext</span><span class="p">())</span> <span class="p">{</span>
<span class="nx">result</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">iter</span><span class="p">.</span><span class="nx">next</span><span class="p">());</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">compareByName</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">an</span> <span class="o">=</span> <span class="nx">a</span><span class="p">.</span><span class="nx">getName</span><span class="p">(),</span> <span class="nx">bn</span> <span class="o">=</span> <span class="nx">b</span><span class="p">.</span><span class="nx">getName</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">an</span> <span class="o"><</span> <span class="nx">bn</span> <span class="o">?</span> <span class="o">-</span><span class="mi">1</span> <span class="o">:</span> <span class="nx">an</span> <span class="o">==</span> <span class="nx">bn</span> <span class="o">?</span> <span class="mi">0</span> <span class="o">:</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">removeOldWhatsAppBackups</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'<removeOldWhatsAppBackups>'</span><span class="p">);</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Config FOLDER_NAME='</span> <span class="o">+</span> <span class="nx">FOLDER_NAME</span><span class="p">);</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Config MINIMUM_DBS_TO_KEEP='</span> <span class="o">+</span> <span class="nx">MINIMUM_DBS_TO_KEEP</span><span class="p">);</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Config USE_TRASH='</span> <span class="o">+</span> <span class="nx">USE_TRASH</span><span class="p">);</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Config DO_DRY_RUN='</span> <span class="o">+</span> <span class="nx">DO_DRY_RUN</span><span class="p">);</span>
<span class="c1">// var folders = DriveApp.getFolders(); // Also lists subfolders.</span>
<span class="kd">var</span> <span class="nx">folder</span> <span class="o">=</span> <span class="nx">getSingleFolder</span><span class="p">(</span><span class="nx">getSingleFolder</span><span class="p">(</span><span class="nx">DriveApp</span><span class="p">.</span><span class="nx">getRootFolder</span><span class="p">().</span><span class="nx">getFoldersByName</span><span class="p">(</span><span class="nx">FOLDER_NAME</span><span class="p">)).</span><span class="nx">getFoldersByName</span><span class="p">(</span><span class="s1">'Databases'</span><span class="p">));</span>
<span class="kd">var</span> <span class="nx">files</span> <span class="o">=</span> <span class="nx">getAll</span><span class="p">(</span><span class="nx">folder</span><span class="p">.</span><span class="nx">getFiles</span><span class="p">());</span>
<span class="kd">var</span> <span class="nx">sortedFiles</span> <span class="o">=</span> <span class="p">[];</span>
<span class="kd">var</span> <span class="nx">i</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">files</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="o">++</span><span class="nx">i</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">file</span> <span class="o">=</span> <span class="nx">files</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="kd">var</span> <span class="nx">name</span> <span class="o">=</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getName</span><span class="p">();</span>
<span class="c1">// Logger.log(name + ': ' + file.getDateCreated()); // No last-modification time :-(.</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">name</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="sr">/^msgstore-\d\d\d\d-\d\d-\d\d[.]/</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">sortedFiles</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">file</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nx">sortedFiles</span><span class="p">.</span><span class="nx">sort</span><span class="p">(</span><span class="nx">compareByName</span><span class="p">);</span> <span class="c1">// The name reflects the date. Earlier file first.</span>
<span class="kd">var</span> <span class="nx">toKeep</span> <span class="o">=</span> <span class="nx">MINIMUM_DBS_TO_KEEP</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">toKeep</span> <span class="o"><</span> <span class="mi">1</span><span class="p">)</span> <span class="nx">toKeep</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">sortedFiles</span><span class="p">.</span><span class="nx">length</span> <span class="o">-</span> <span class="nx">toKeep</span><span class="p">;</span> <span class="o">++</span><span class="nx">i</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">file</span> <span class="o">=</span> <span class="nx">sortedFiles</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="c1">// Logger.log('?? ' + file.getName() + ': ' + file.getSize());</span>
<span class="c1">// A 100-byte decrease in file size is tolerable, can be a compression artifact.</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">file</span><span class="p">.</span><span class="nx">getSize</span><span class="p">()</span> <span class="o">-</span> <span class="mi">100</span> <span class="o"><</span> <span class="nx">sortedFiles</span><span class="p">[</span><span class="nx">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">].</span><span class="nx">getSize</span><span class="p">())</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">DO_DRY_RUN</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">USE_TRASH</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">file</span><span class="p">.</span><span class="nx">setTrashed</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">DriveApp</span><span class="p">.</span><span class="nx">removeFile</span><span class="p">(</span><span class="nx">file</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Utilities.sleep(100); // Prevent exceeding rate limit (currently 10 requests per second). Is it still in effect?</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'-- Removing: '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getName</span><span class="p">()</span> <span class="o">+</span> <span class="s1">': '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getSize</span><span class="p">());</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Keeping: '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getName</span><span class="p">()</span> <span class="o">+</span> <span class="s1">': '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getSize</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">sortedFiles</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="o">++</span><span class="nx">i</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">file</span> <span class="o">=</span> <span class="nx">sortedFiles</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Keeping recent: '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getName</span><span class="p">()</span> <span class="o">+</span> <span class="s1">': '</span> <span class="o">+</span> <span class="nx">file</span><span class="p">.</span><span class="nx">getSize</span><span class="p">());</span>
<span class="p">}</span>
<span class="nx">Logger</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'</removeOldWhatsAppBackups>'</span><span class="p">);</span>
<span class="p">}</span></pre>
<li>Click on the <i>File / Save</i> menu to save the script.
<li>Open the <i>Resources / All your triggers</i> menu window, choose <i>Add a new trigger</i>, add this: <i>removeOldWhatsAppBackups</i>, <i>Time-driven</i>, <i>Day timer</i>, <i>5am to 6am</i>, also add a notification to yourself in e-mail, sent at 6am.
<li>Click on <i>Save</i> to save the triggers.
<li>You may wonder if this Apps Script deletes chat backups in a safe way. You have to decide this for yourself. It keeps the 8 last backups, and it also keeps all backups after which the backup file size has decreased. This latter condition prevents the data loss case described below (in the next section).
</ol>
<p>That's it, automatic and safe backup of WhatsApp chats, photos and other media files to Google Drive is now set up using <i>FolderSync Lite</i>.
<h3>Is WhatsApp's built-in backup to Google Drive safe?</h3>
<p>No it isn't, you can lose all your data in some cases. The data loss happened to a friend in Feb 2017 in the following way:
<ul>
<li>The Android phone was lost, thus the <i>WhatsApp</i> folder on the phone wasn't available.
<li>There was a working and recent backup on Google Drive.
<li>When WhatsApp was installed to a new phone, it started restoring the backup from Google Drive.
<li>Shortly after the start of the restore, the internet connection broke and the restore was aborted. Only a little part of the messages were restored.
<li>The owner of the phone didn't notice that many chats are missing, and started using WhatsApp.
<li>Within 24 hours, WhatsApp created a new backup to Google Drive, overwriting the old, full data with the new partial data. At this point the majority of the old chats got lost.
<li>A couple of days later the owner of the phone noticed, but it was too late.
</ul>
<p>If WhatsApp's built-in backup to Google Drive had been written in a way that it never overwrites old backups, it would have been possible to reinstall WhatsApp and restore all the chats without data loss. Unfortunately the users of WhatsApp have no control on how WhatsApp does backups. But they can use an alternative backup method which never loses data (see the method with <i>FolderSync Lite</i> above).
<p>Another way to safely restore WhatsApp's built-in backup from Google Drive would be downloading it from Google Drive first, and keeping a a copy until WhatsApp has successfully restored everything on the new Android device. Unfortunately this is impossible, because it's impossible for the user to download their own backup from Google Drive (!), because it is saved to a hidden app folder, which only the WhatsApp app can read and write, and the user is unable to access it (apart from deleting it). This <a href="http://stackoverflow.com/q/22832104/97248">StackOverflow question</a> has an answer which describes a cumbersome and fragile way for downloading, but this can easily change in the future, so I don't recommend it for general use. I recommend the method with <i>FolderSync Lite</i> above instead, which makes it easy for the user to see and download their WhatsApp backup from Google Drive.Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-8821846818714029393.post-58726142685225407512017-02-07T17:44:00.000+01:002017-02-23T14:22:43.219+01:00Checking whether a Unix filename is safe in C<p>This blog post explains how to check whether a filename is safe for overwriting the file contents on a Unix system, and it also contains C code to do the checks.
<p>The use case is that an archive (e.g. ZIP) extractor is extracting an untrusted archive and creating and overwriting files. The archive may be created by someone malicious, trying to trick the extractor to overwrite sensitive system files such as <code>/etc/passwd</code>. Of course this would only work if the process running the extractor has the permission to modify system files (which it normally doesn't have, because it's not running as root). The most basic protection the extractor can provide is refusing to write to a file with an absolute name (i.e. starting with <code>/</code>), so that even if it's running as root, it would do no harm if it's not running in the root directory.
<p>However, checking whether the first character is a <code>/</code> isn't good enough, because the attacker may specify a file with name <code>../.././../../../.././../tmp/../etc/././passwd</code>, which is equivalent to <code>/etc/passwd</code> if the current directory isn't deep enough. Enforcing the following conditions makes sure that such attacks are not possible:
<ul>
<li>The filename isn't be empty.
<li>The filename doesn't start or end with a <code>/</code>.
<li>The strings <code>.</code> and <code>..</code> are not present as a pathname component (i.e. as an item when the filename is split on <code>/</code>).
<li>The filename doesn't contain a double slash: <code>//</code>.
</ul>
<p>Here is how to check all these in C:
<pre class="syntax"><span></span><span class="kt">int</span> <span class="nf">is_filename_safe</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">p</span><span class="p">)</span> <span class="p">{</span>
<span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">q</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(;;)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="n">q</span> <span class="o">=</span> <span class="n">p</span><span class="p">;</span> <span class="o">*</span><span class="n">p</span> <span class="o">!=</span> <span class="sc">'\0'</span> <span class="o">&&</span> <span class="o">*</span><span class="n">p</span> <span class="o">!=</span> <span class="sc">'/'</span><span class="p">;</span> <span class="o">++</span><span class="n">p</span><span class="p">)</span> <span class="p">{}</span>
<span class="cm">/* In the first iteration: An empty filename is unsafe. */</span>
<span class="cm">/* In the first iteration: A leading '/' is unsafe. */</span>
<span class="cm">/* In subsequent iterations: A trailing '/' is unsafe. */</span>
<span class="cm">/* In subsequent iterations: A "//" is unsafe. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">p</span> <span class="o">==</span> <span class="n">q</span> <span class="o">||</span>
<span class="cm">/* A pathname component "." is unsafe. */</span>
<span class="cm">/* A pathname component ".." is unsafe. */</span>
<span class="p">(</span><span class="o">*</span><span class="n">q</span> <span class="o">==</span> <span class="sc">'.'</span> <span class="o">&&</span> <span class="p">(</span><span class="n">q</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">==</span> <span class="n">p</span> <span class="o">||</span> <span class="p">(</span><span class="n">q</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="sc">'.'</span> <span class="o">&&</span> <span class="n">q</span> <span class="o">+</span> <span class="mi">2</span> <span class="o">==</span> <span class="n">p</span><span class="p">))))</span> <span class="p">{</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="cm">/* Unsafe. */</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="o">++</span> <span class="o">==</span> <span class="sc">'\0'</span><span class="p">)</span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span> <span class="cm">/* Safe. */</span>
<span class="p">}</span>
<span class="p">}</span></pre>
<p>An even better solution is running the extractor in an isolated sandbox (jail), thus protecting against malicious input and all kinds of software bugs.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-60191256773888931132016-11-08T16:54:00.002+01:002018-05-24T13:39:06.163+02:00How to fix Python SSL errors when downloading https pages<p>This blog post explains how to fix Python SSL errors when downloading web pages using the https:// protocol in Python (e.g. by using the <i>urllib</i>, <i>urllib2</i>, <i>httplib</i> or <i>requests</i>. This blog post has been written because many other online sources haven't given direct and useful advice on how to fix the errors below.
<h3>How to fix SSL23_GET_SERVER_HELLO unknown protocol</h3>
This error looks like (possibly with a line number different from 504):
<pre> self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol</pre>
<p>The fix is to upgrade your Python:
<ul>
<li>If you are using Python 2, upgrade to at least 2.7.7. It's recommended to upgrade to the latest (currently 2.7.11) though, or at least 2.7.9 (which has backported the <i>ssl</i> module (including the <i>ssl.SSLContext</i> customizations from 3.4.x). I have tested that the error above disappears when upgrading from 2.7.6 to 2.7.7. If you can't easily upgrade the Python 2 on the target system, you may want to try <a href="https://github.com/pts/staticpython/tree/master/release">StaticPython</a> on Linux (the <i>stacklessco2.7-static</i> and <i>stacklessxx2.7-static</i> binaries have OpenSSL and recent enough Python) or <a href="http://www.egenix.com/products/python/PyRun/">PyRun</a> on Linux, macOS, FreeBSD and other Unix systems.
<li>If you are using Python 3, upgrade to at least 3.4.3. It's recommended to upgrade to the latest (3.5.2 or later) though.
<li>If you are unable to upgrade from Python 2.6.x or 2.7.x, try this workaround, it works in some cases (e.g. on Ubuntu 10.04) and on some websites:
<pre>import ssl
from functools import partial
ssl.wrap_socket = partial(ssl.wrap_socket, ssl_version=ssl.PROTOCOL_TLSv1)</pre>
<p>There is a similar workaround for <code>ssl.sslwrap_simple</code> which also affects <code>socket.ssl</code>.
<li>If you are unable to upgrade from Python 2.6.x, 2.7.x, 3.2.x or 3.3.x, use <a href="https://pypi.python.org/pypi/backports.ssl">backports.ssl</a>.
<lI>If you are unable to upgrade from Python 1.x – 2.5.x or 3.0.x – 3.1.x, then probably there is no easy fix for you.
</ul>
<p>Typically it's not necessary to upgrade your OpenSSL library just to fix this error, ancient versions such as OpenSSL 0.9.8k (released on 2009-03-25) also work if Python is upgraded. The latest release from the 0.9.8 series (currently 0.9.8zh) or from the 1.0 series or from the 1.1 series should all work. But if you have an easy option to upgrade, then upgrade to at least the latest LTS (long-term-support) version (currently 1.0.2j).
<h3>How to fix SSL CERTIFICATE_VERIFY_FAILED</h3>
This error looks like (possibly with a line number different from 509):
<pre> self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)</pre>
<p>Server certificate verification by default has been introduced to Python recently (in 2.7.9). This protects against man-in-the-middle attacks, and it makes the client sure that the server is indeed who it claims to be.
<p>As a quick (and insecure) fix, you can turn certificate verification off, by at least one of these:
<ul>
<li>Set the <i>PYTHONHTTPSVERIFY</i> environment variable to 0 before the <i>ssl</i> module is loaded, e.g. run <code>export PYTHONHTTPSVERIFY=0</code> before you start the Python script.
<li>(alternatively) Add this to your code before doing the https:// request (it affects all requests from then on):
<pre>import os, ssl
if (not os.environ.get('PYTHONHTTPSVERIFY', '') and
getattr(ssl, '_create_unverified_context', None)):
ssl._create_default_https_context = ssl._create_unverified_context</pre>
</ul>
<p>The proper, secure fix though is to install the latest root certificates to your computer to a directory where the OpenSSL library used by Python finds it. Your operating system may be able to do it conveniently for you, for example on Ubuntu 14.04, running this usually fixes it: <code>sudo apt-get update && sudo apt-get install ca-certificates</code>).Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-8821846818714029393.post-54397814669494334082016-07-15T17:10:00.000+02:002016-07-15T17:10:59.242+02:00How to compile Lepton (JPEG lossless recompressor by Dropbox) without autotools<p>This blog post explains how to compile <a href="https://github.com/dropbox/lepton">Lepton</a>, the recently released JPEG lossless recompressor by Dropbox without autotools on Linux.
<p>You'll need a fairly recent C++ compiler with the development libraries installed. <i>g++-4.4</i> is too old, <i>g++-4.8</i> is good enough.
<p>Compile with the following command (without the leading <code>$</code>):
<pre>
$ git clone https://github.com/dropbox/lepton
$ cd lepton
$ # Optional: git reset --hard e3f6c46502d958aaba17fbe7c218f8ce2b8b3f48
$ g++ -std=c++0x -W -Wall -Wextra -Wno-unused-parameter -Wno-write-strings \
-msse4.1 \
-o lepton \
-fno-exceptions -fno-rtti \
-s -O2 \
-DGIT_REVISION=\"fake\" \
-I./src/vp8/decoder \
-I./src/vp8/encoder \
-I./src/vp8/model \
-I./src/vp8/util \
dependencies/md5/md5.c \
src/io/MemMgrAllocator.cc \
src/io/MemReadWriter.cc \
src/io/Seccomp.cc \
src/io/Zlib0.cc \
src/io/ZlibCompression.cc \
src/io/ioutil.cc \
src/lepton/bitops.cc \
src/lepton/fork_serve.cc \
src/lepton/idct.cc \
src/lepton/jpgcoder.cc \
src/lepton/lepton_codec.cc \
src/lepton/recoder.cc \
src/lepton/simple_decoder.cc \
src/lepton/simple_encoder.cc \
src/lepton/socket_serve.cc \
src/lepton/thread_handoff.cc \
src/lepton/uncompressed_components.cc \
src/lepton/validation.cc \
src/lepton/vp8_decoder.cc \
src/lepton/vp8_encoder.cc \
src/vp8/decoder/boolreader.cc \
src/vp8/decoder/decoder.cc \
src/vp8/encoder/boolwriter.cc \
src/vp8/encoder/encoder.cc \
src/vp8/model/JpegArithmeticCoder.cc \
src/vp8/model/model.cc \
src/vp8/model/numeric.cc \
src/vp8/util/billing.cc \
src/vp8/util/debug.cc \
src/vp8/util/generic_worker.cc \
src/vp8/util/memory.cc \
-lz \
-lpthread \
;
$ ./lepton --help
</pre>
<p>It works both on i386 (force it with <code>-m32</code>) and amd64 (force it with <code>-m64</code>) architectures. Other architectures are not supported, because Lepton uses the SSE4.1 CPU instructions.
<p>It also works with <code>-std=c++11</code>, but it doesn't work with <code>-std=c++98</code>.
<p>Lepton also has a copy of <i>zlib</i> embedded, you can compile it with <code>gcc</code> instead of <code>g++</code>, and use it instead of <code>-lz</code>.
<p>These instructions were tested and found working with the following version of Lepton:
<pre>
$ git rev-parse HEAD
# e3f6c46502d958aaba17fbe7c218f8ce2b8b3f48
</pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-31561473756034154192016-06-20T23:40:00.001+02:002016-06-20T23:40:13.766+02:00How to back up original photo files as is in Google<p>This blog post explains how to back up photos on Google Photos and Google Drive as is, keeping the original images files, bit-by-bit identical, without any scaling or reencoding.
<p>TL;DR If you want to keep the original image files, upload the photos to Google Drive, which keeps the original files (bit-by-bit identical as uploaded), and their size counts against your Google storage quota (see <a href="https://www.google.com/settings/storage">your usage</a>). Don't upload any image file to Google Photos.
<p>TL;DR If you want unlimited image uploads with the option of downloading the original image file (bit-by-bit identical), consider options other than Google Photos (such as Flickr and Deviantart).
<p>On Google Photos you can upload some images for free (i.e. those images don't count against your Google storage quota). This is the most important advantage for uploading to Google Photos (rather than Google Drive). But there are some important caveats:
<ul>
<li>You need to decide before uploading if you want free (it's called <i>high quality</i>) or not (<i>original</i>). Select it in the <a href="https://photos.google.com/settings">Google Photos settings</a>. This setting won't effect photos uploaded from your Android devices by the photo backup app.
<li>If you decide non-free (<i>original</i>), future uploads will be counted against your quota, no matter the size, the file format or the quality. That is, even small, low-quality JPEGs will count against your quota.
<li>If you choose free and you upload a PNG file of at most 16 megapixels, the original file is kept, and you'd be able to download it later.
<li>If you choose free and you upload a PNG file of more than 16 megapixels, then it will be scaled down and reencoded.
<li>If you choose free and you upload a JPEG file, then the photo gets scaled down to 16 megapixels (no change if already small enough), and then reencoded with a quality loss (which is small enough so that most humans don't notice), removes or rearranges some metadata (e.g. EXIF), and only the scaled and reencoded JPEG file is available for download.
<li>Google Photos does deduplication of your images. This has an unintended consequence. If you upload a photo 3 times to 3 different album, and you move the photo to the trash, it will be removed from all 3 albums. There is no way to move some photos in an album to the trash without affecting other albums.
<li>Google Photos does deduplication even across qualities. Thus if you upload an image as <i>original</i> first, and the upload it again as <i>high quality</i>, the <i>high quality</i> version will be ignored, and the <i>original</i> version will be present in both albums. It's also true the other way round: if you upload <i>high quality</i> first, then subsequent <i>original</i> uploads of the same image will be ignored.
<li>Even with non-free (<i>original</i>), Google doesn't remember the original file name, as uploaded: it converts e.g. the <code>.JPG</code> extension to <code>.jpg</code> (lower case).
<li>Immediately after the upload, the image info page shows incorrect information, and the <i>Download</i> link serves an incorrect (lower resolution) version of the image. For example, when I uploaded a new 1.1 MB JPEG file in <i>high quality</i> mode, the image info was showing <i>1.1 MB</i>, but when downloading it, it became a 340 kB JPEG file. After reloading the image page, the image info was showing <i>550 KB</i>, and downloading it yielded a file of that size. This makes experimenting with image upload sizes confusing.
<li>This is as of 2016-06-20, the behavior of Google Photos may change in the future.
</ul>
<p>Because of these caveats and unexpected behavior, to avoid quality loss, my recommendation is not to use Google Photos for backing up JPEG image files.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-9860507844114949132016-05-30T15:01:00.000+02:002016-05-30T15:02:45.272+02:00How to install Hungarian spell checking and hyphenation for LibreOffice and OpenOffice on Linux<p>This blog post explains how to install Hungarian spell checking and hyphenation for LibreOffice and OpenOffice on Linux. The instructions were tested on Ubuntu Trusty, but they should work well on other Linux distributions as well with small modifications.
<ol>
<li>The installation consists of downloading the right files, copying them to the right location, and restarting the LibreOffice and/or OpenOffice.
<li>Install LibreOffice or OpenOffice with your favorite package manager if not already installed.
<li>Download the files from <a href="http://magyarispell.sf.net/">http://magyarispell.sf.net/</a> (no need to click now) by running this commands (without the leading <code>$</code>) in a terminal window:
<pre>$ wget -O /tmp/hu_HU-1.6.1.tar.gz https://sourceforge.net/projects/magyarispell/files/Magyar%20Ispell/1.6.1/hu_HU-1.6.1.tar.gz/download #
$ wget -O /tmp/huhyphn_v20110815_LibO.tar.gz https://sourceforge.net/projects/magyarispell/files/OOo%20Huhyphn/v20110815-0.1/huhyphn_v20110815_LibO.tar.gz/download #
</pre>
<p>The commands are long, please make sure to copy-paste the entire line (both ending with <code>#</code>),
<p>Be patient, you may have to wait for 30 seconds for each download.
<li>Run these commands to copy the files to the right location:
<pre>$ (cd /tmp && tar xzvf ~/Downloads/hu_HU-1.6.1.tar.gz)
$ (cd /tmp/hu_HU-1.6.1 && sudo cp hu_HU.aff hu_HU.dic /usr/share/hunspell/)
$ (cd /tmp/ && tar xzvf ~/Downloads/huhyphn_v20110815_LibO.tar.gz)
$ (cd huhyphn_v20110815_LibO && sudo cp hyph_hu_HU.dic /usr/share/hyphen/)</pre>
<li>If it's already running, exit from LibreOffice and OpenOffice.
<li>Start LibreOffice or OpenOffice, type <i>asztall</i>, select it, change the language to <i>Hungarian</i> in <i>Format / Character</i>. Now the text should be underlined with read, and when you right-click, the suggestion <i>asztal</i> should be offered.
</ol>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-39560713429886306272016-04-06T17:00:00.000+02:002016-04-06T17:00:01.585+02:00How to disable (reject) any root password on Debian and Ubuntu<p>This blog post explains how to disable (reject) any root password on Debian and Ubuntu, thus rejecting login attempts as root. Becoming root with <i>sudo</i> (by typing the calling user's password) or <i>ssh</i> (using a public key) remains possible.
<p>TL;DR Run as root: <code>passwd -d -l root</code>
<h3>How to become root if password-based root logins are (or will be) disabled?</h3>
<p>Before disabling password-based root logins, make sure you have other ways to become root. One possible way is running <i>sudo</i> (without arguments) from a non-root user. To make this work, first you have to install <i>sudo</i> by running (without the leading <code>#</code>) as root:
<pre># apt-get install sudo</pre>
<p>as root. (Ubuntu systems come with <i>sudo</i> preinstalled, Debian systems don't have it by default.) Then run as root, replacing <i>MyUser</i> with your non-root login name:
<pre># adduser MyUser sudo</pre>
<p>After running this, running <i>sudo</i> as that user will ask for the user's password (not the root password), and when typed correctly, you will get a root shell, and will be able to run commands as root. (Type <i>exit</i> to exit from the root shell.)
<p>An alternative to <i>sudo</i> for becoming root without a password is running <i>ssh root@localhost</i>. For that you need a properly configured <i>sshd</i> (with <code>PermitRootLogin without-password</code> or <code>PermitRootLogin yes</code> in <i>/etc/ssh/sshd_config</i>), creating an SSH key pair and appending the public key to <i>/root/.ssh/authorized_keys</i>. If you need help setting this up or using it, then please ask a Unix or Linux guru friend.
<h3>How to disable password-based root logins</h3>
<p>To disable (reject) any root password on Debian and Ubuntu, run this (without the leading <code>#</code>) as root:
<pre># passwd -d -l root</pre>
<p>This effectively changes the 2nd field line starting with <code>root:</code> in <i>/etc/shadow</i> to <code>!</code>, thus the line will start with <code>root:!:</code>, making <i>login</i>, <i>su</i>, <i>ssh</i> (when using password authentication, i.e. no public key) reject login attempts as root. Typically the password wouldn't even be asked for, but if it is, any password would be rejected. An alternative to the command above is editing the <i>/etc/shadow</i> file manually (as root), and adding the <code>!</code>. Also the <code>-d</code> flag is not necessary, without it the password hash is still kept in <i>/etc/shadow</i> (but a <code>!</code> is prepended to disable it).
<p>Ubuntu comes with this default (<code>root:!:</code> in <i>/etc/shadow</i>), Debian doesn't.
<p>If you want to disable the root password in <i>ssh</i> only (and allow password-based root logins in <i>login</i> and <i>su</i>), then instead of running the command above, add (or change) the line
<pre>PermitRootLogin without-password</pre>
<p>to <i>/etc/ssh/sshd_config</i> (as root), and then run (as root):
<pre># /etc/init.d/ssh restart</pre>
<p>Please note that there are ways to permit a root login without a password (or with an empty password), but this is very bad security practice, so this blog post doesn't explain how to do it.
<h3>How to enable password-based root logins</h3>
<p>To enable password-based root logins again, run this as root:
<pre># passwd root</pre>
<p>It will ask you to specify the new password for root.
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-42462343397239648482016-03-01T23:09:00.000+01:002020-08-25T10:26:03.090+02:00keybase.txt<p>Please disregard this post, it's a cryptographic proof checked by <a href="https://keybase.io/pts">keybase.io/pts</a>.
<pre>
==================================================================
https://keybase.io/pts
--------------------------------------------------------------------
I hereby claim:
* I am an admin of https://ptspts.blogspot.com
* I am pts (https://keybase.io/pts) on keybase.
* I have a public key with fingerprint 537D 2E8D 6FAB 3265 9A1F 8767 33BB 974C 2FE0 93F2
To do so, I am signing this object:
{
"body": {
"key": {
"eldest_kid": "01110f1fe32d1d6263e9674e11d7249ac66f46d9f8c54c896c16205dcf68203493b90a",
"fingerprint": "537d2e8d6fab32659a1f876733bb974c2fe093f2",
"host": "keybase.io",
"key_id": "33bb974c2fe093f2",
"kid": "01110f1fe32d1d6263e9674e11d7249ac66f46d9f8c54c896c16205dcf68203493b90a",
"uid": "51f5f6cc0a558175f85f29dc164fe900",
"username": "pts"
},
"service": {
"hostname": "ptspts.blogspot.com",
"protocol": "https:"
},
"type": "web_service_binding",
"version": 1
},
"ctime": 1456178821,
"expire_in": 157680000,
"prev": "b14bb983d61e5b2097e313d7b246fa7e17030b598d6ae6a7a1545f36b5ef75c2",
"seqno": 27,
"tag": "signature"
}
which yields the signature:
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1
owGtUj1rVEEUfSYaMSCYxk6L6dRlMx9vvhZsLLQRtBAbi2U+7mwe2bx5vveyMYRt
JYWFIGil4t9QG20sUu4fEG20CtrYWDizKDaWDgPDvXPOmTP33sdnV4uVDfbm2c3N
j0++nzj6aou7Rw8PD5CNfh+NDtA2LA+Yeuj68Xbl0QhhQggOJACjnnhBBQMtZAmE
eElLbZwQoRReB+V46ZQWjgiKuXdBKIpZqZnV2KABClU9gbZpq7pPspxJT0F5EYxl
VHBtSFBSSMas1bJ0NADWLNBE3IpdZiRz1nQwrGLKpWC8tPcP/H/2vbuU4yTwIJzD
hnNFJA+KB6p9IpUBNMYZ2EFbmx1I6Kbv0HyAUmJWOcg1zZ/4e5n20E7jpGtiP3Rx
J7GbNvbRxWkCbPV9042yQL/fZMYe2PFvrbGtap8qmRgzaLsq1mhEEtL1VRYnJRdE
KkXJAMGDpmphXGUEl0LhtPI7MEuSlpTW3tOKeUGAW4q1BEaYl5aWqScSiMQMW65T
iwwIIw3hJQ9MWA5BcpcL3cH9OqIRlcmomSTRrprUpt9tAc3XD6+fLDZWirVTK3nG
ivUz5/4MXnereH7xx+3zcfH05+LT26ub7+68evR+/q14celaWDv+fGXR3Fg9Xny5
fPrlh9cXfgE=
=N5V9
-----END PGP MESSAGE-----
And finally, I am proving ownership of this host by posting or
appending to this document.
View my publicly-auditable identity here: https://keybase.io/pts
==================================================================
</pre>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-8821846818714029393.post-464722488367749922016-02-22T23:02:00.000+01:002016-02-22T23:02:39.367+01:00micropython for Linux i386, statically linked<p>This blog post is to announce statically linked binaries for Linux i386 of <a href="https://micropython.org">MicroPython</a>.
<p><i>MicroPython</i> (Python for microcontrollers) is an open source reimplementation (see <a href="https://github.com/micropython/micropython">sources on GitHub</a>) of the Python 3.4 language for microcontrollers with very little RAM (as low as 60 kB). The CPython interpreter is not used at all, <i>MicroPython</i> has a completely separate implementation in C, supporting the full Python 3.4 language syntax, but with a much smaller standard library (i.e. much fewer modules and classes, and existing classes have fewer methods). Unicode strings (i.e. the <i>str</i> class) are supported though.
<p><i>MicroPython</i> can be cross-compiled to many different platforms, including multiple microcontrollers (including the <a href="https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot">ESP8266</a> ($5) and the <a href="https://micropython.org/store">pyboard</a> ($40)) and to Unix systems (including Linux). The <code>micropython</code> binary seems to be 17.56 times smaller than the <code>python</code> binary for Linux i386 (both binaries were statically linked against uClibc using <a href="pts-xstatic">https://github.com/pts/pts-clang-xstatic/blob/master/README.pts-xstatic.txt</a>, and optionally compressed with <a href="http://upx.sf.net/">UPX</a>). The detailed file sizes are:
<ul>
<li><a href="https://raw.githubusercontent.com/pts/staticpython/master/release/python2.7-static">python2.7</a> (from <a href="https://github.com/pts/staticpython">StaticPython</a>, CPython 2.7): 6053147 bytes
<li>python2.7.upx: 3381556 bytes
<li><a href="https://github.com/pts/pts-build-micropython-xstatic/releases/download/v1/micropython">micropython</a> (1.6): 344788 bytes
<li><a href="https://github.com/pts/pts-build-micropython-xstatic/releases/download/v1/micropython.upx">micropython.upx</a> (1.6): 161568 bytes
</ul>
<p>The <a href="https://github.com/pts/pts-build-micropython-xstatic/blob/master/build.sh">script for recompiling MicroPython</a> for Linux i386, statically linked is also open source.
<p>Please note that neither <i>StaticPython</i> nor <i>MicroPython</i> open any external files (such as <i>.so</i> or <i>.py</i> or <i>.zip</i>) when starting up, all the Python interpreter and the Python standard library (and the libc as well) are statically linked in to the binary executable.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8821846818714029393.post-5751995694084114872016-01-06T21:32:00.000+01:002016-01-06T21:34:36.695+01:00How to extract comments from a JPEG file<p>This blog post explains how to extract comments from a JPEG file. Each JPEG file consists of segments. Each segment describes parts of the image data or metadata. The comments are in segments with marker COM (0xfe), there can be any number of them, anywhere (usually before the SOS segment) in the file.
<p>Use the <i>rdjpgcom</i> command-line tool to extract comments. The tool is part of <a href="http://libjpeg.sourceforge.net/">libjpeg</a>, and on Ubuntu and Debian systems it can be installed with (don't type the leading <code>$</code>):
<pre>$ sudo apt-get install libjpeg-progs</pre>
<p>Once installed, use it like this to print all comments in the JPEG file, with a terminating newline added to each:
<pre>$ rdjpgcom <i>file.jpg</i></pre>
<p>If the file doesn't have any comment, the output of <i>rdjpgcom</i> is empty. Here is how to add comments:
<pre>$ wrjpgcom -c 'COMfoo' com0.jpg >com1.jpg
$ wrjpgcom -c 'COMbar' com1.jpg >com2.jpg</pre>
After adding the comments, it will look like this:
<pre>$ rdjpgcom com2.jpg
COMfoo
COMbar</pre>
<p>If you also want to see the unprintable characters (unsafe on a terminal), pass the <i>-raw</i> flag:
<pre>$ rdjpgcom -raw <i>file.jpg</i></pre>
<p>If you need a library, use the following C code, which is a minimalistic reimplementation of <i>rdjpgcom -a</i>:
<pre class="syntax"><span class="cm">/*</span>
<span class="cm"> * getjpegcom.c: Get comments from JPEG files.</span>
<span class="cm"> * by pts@fazekas.hu at Wed Jan 6 20:48:07 CET 2016</span>
<span class="cm"> *</span>
<span class="cm"> * $ gcc -W -Wall -Wextra -Werror -s -O2 -o getjpegcom getjpegcom.c &&</span>
<span class="cm"> * $ ./getjpegcom <file.jpg</span>
<span class="cm"> */</span>
<span class="cp">#include <stdio.h></span>
<span class="cp">#if defined(MSDOS) || defined(WIN32)</span>
<span class="cp">#include <fcntl.h> </span><span class="cm">/* setmode. */</span><span class="cp"></span>
<span class="cp">#endif</span>
<span class="cm">/* Get and print all comments in a JPEG file. Comments are written to of,</span>
<span class="cm"> * with a newline appended as terminator.</span>
<span class="cm"> *</span>
<span class="cm"> * Returns 0 on success, or a negative number on error.</span>
<span class="cm"> */</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">get_jpeg_comments</span><span class="p">(</span><span class="kt">FILE</span> <span class="o">*</span><span class="n">f</span><span class="p">,</span> <span class="kt">FILE</span> <span class="o">*</span><span class="n">of</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">c</span><span class="p">,</span> <span class="n">m</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="n">ss</span><span class="p">;</span>
<span class="cp">#if defined(MSDOS) || defined(WIN32)</span>
<span class="n">setmode</span><span class="p">(</span><span class="n">fileno</span><span class="p">(</span><span class="n">f</span><span class="p">),</span> <span class="n">O_BINARY</span><span class="p">);</span>
<span class="n">setmode</span><span class="p">(</span><span class="n">fileno</span><span class="p">(</span><span class="n">of</span><span class="p">),</span> <span class="n">O_BINARY</span><span class="p">);</span>
<span class="cp">#endif</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ferror</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="k">return</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="cm">/* A typical JPEG file has markers in these order:</span>
<span class="cm"> * d8 e0_JFIF e1 e1 e2 db db fe fe c0 c4 c4 c4 c4 da d9.</span>
<span class="cm"> * The first fe marker (COM, comment) was near offset 30000.</span>
<span class="cm"> * A typical JPEG file after filtering through jpegtran:</span>
<span class="cm"> * d8 e0_JFIF fe fe db db c0 c4 c4 c4 c4 da d9.</span>
<span class="cm"> * The first fe marker (COM, comment) was at offset 20.</span>
<span class="cm"> */</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated (empty). */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">!=</span> <span class="mh">0xff</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">3</span><span class="p">;</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">!=</span> <span class="mh">0xd8</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">3</span><span class="p">;</span> <span class="cm">/* Not a JPEG file, SOI expected. */</span>
<span class="k">for</span> <span class="p">(;;)</span> <span class="p">{</span>
<span class="cm">/* printf("@%ld\n", ftell(f)); */</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">!=</span> <span class="mh">0xff</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">3</span><span class="p">;</span> <span class="cm">/* Not a JPEG file, marker expected. */</span>
<span class="k">if</span> <span class="p">((</span><span class="n">m</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="k">while</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xff</span><span class="p">)</span> <span class="p">{</span> <span class="cm">/* Padding. */</span>
<span class="k">if</span> <span class="p">((</span><span class="n">m</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xd8</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">4</span><span class="p">;</span> <span class="cm">/* SOI unexpected. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xd9</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span> <span class="cm">/* EOI. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xda</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span> <span class="cm">/* SOS. Would need special escaping to process. */</span>
<span class="cm">/* printf("MARKER 0x%02x\n", m); */</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="n">ss</span> <span class="o">=</span> <span class="p">(</span><span class="n">c</span> <span class="o">+</span> <span class="mi">0U</span><span class="p">)</span> <span class="o"><<</span> <span class="mi">8</span><span class="p">;</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="n">ss</span> <span class="o">+=</span> <span class="n">c</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ss</span> <span class="o"><</span> <span class="mi">2</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">5</span><span class="p">;</span> <span class="cm">/* Segment too short. */</span>
<span class="k">for</span> <span class="p">(</span><span class="n">ss</span> <span class="o">-=</span> <span class="mi">2</span><span class="p">;</span> <span class="n">ss</span> <span class="o">></span> <span class="mi">0</span><span class="p">;</span> <span class="o">--</span><span class="n">ss</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">((</span><span class="n">c</span> <span class="o">=</span> <span class="n">getc</span><span class="p">(</span><span class="n">f</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="o">-</span><span class="mi">2</span><span class="p">;</span> <span class="cm">/* Truncated. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xfe</span><span class="p">)</span> <span class="n">putc</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">of</span><span class="p">);</span> <span class="cm">/* Emit comment char. */</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">m</span> <span class="o">==</span> <span class="mh">0xfe</span><span class="p">)</span> <span class="n">putc</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">,</span> <span class="n">of</span><span class="p">);</span> <span class="cm">/* End of comment. */</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
<span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="n">argc</span><span class="p">;</span> <span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="n">argv</span><span class="p">;</span>
<span class="k">return</span> <span class="o">-</span><span class="n">get_jpeg_comments</span><span class="p">(</span><span class="n">stdin</span><span class="p">,</span> <span class="n">stdout</span><span class="p">);</span>
<span class="p">}</span></pre>
<p>Here is how to compile and run it:
<pre>$ gcc -W -Wall -Wextra -Werror -s -O2 -o getjpegcom getjpegcom.c
$ ./getjpegcom com2.jpg
COMfoo
COMbar</pre>Unknownnoreply@blogger.com0