2018-06-04

How to copy files securely between computers running Linux or Unix?

This blog post gives various recommendations on how to copy files securely between computers running Linux or Unix.

All the recommendations below copy the file in an encrypted way, protecting against eavesdropping and protecting partially against man-in-the-middle attacks (i.e. a thrid party tricking the receiver to accept forged file contents).

If both computers run either of Chrome or Firefox, and it's convenient for you to use these web browsers, visit any of the following sites to copy the file: sharedrop.io, reep.io, takeafile.com, send-anywhere.com, justbeamit.com. These sites use WebRTC (thus the transfer is encrypted) to copy the file directly from the sender to the receiver without uploading it to a server, and they traverse NAT firewalls using STUN and ICE. (Don't use sites based on WebTorrent (such as instant.io or file.pizza), because the WebTorrent transfers are not end-to-end encrypted.)

Otherise, if one of the computers is running the OpenSSH server (sshd), and the other one is able to connect to it over the network, and you know a user's password on the server (or SSH public keys are set up instead of a password), then use scp or rsync. Otherwise, if one of the computers is able to connect to the other over the network, and the client computer (the one which initiates the TCP connection) has the OpenSSH client (ssh) installed, you have root access on the server, and you don't mind installing software to the server temporarily, then follow the instructions in the One-off SCP with Dropbear section below.

The rest of the setups are typically useful if one of the computers is recently installed (so it doesn't contain your SSH private keys yet), or you don't want any of them act as a server, or you don't have root access.

Otherwise, if both computers are connected to the same local network (e.g. same wifi network), and they are able to connect to each other, try ecplcnw (available and documented here: https://github.com/pts/copystrap).

Otherwise, if both computers have web access, and you don't mind uploading securely encrypted files to a shared hosting provider, use ecptrsh (available and documented here: https://github.com/pts/copystrap).

Otherwise, if you have a USB pen drive, SD card, external hard disk or other writable storage medium which you can physically take from one computer to another, use ecplmdr (available and documented here: https://github.com/pts/copystrap).

Otherwise I have no secure and convenient recommendation for you.

Other secure options for file copy

  • Direct connection between the computers using an Ethernet cable or serial cable. This can work, but it is not convenient, because it needs rare hardware and increasingly rare ports on the laptops and extensive and error-prone manual setup.
  • netcat for transfer + GPG for encryption. Some more details here. This is similar to ecplcnw above, but less convenient and less secure, because user-invented passphrases tend to be weak, and strong passphrases are long and cumbersome to type. Also it's a bit inconvenient to the get the IP address in the command-lines right. Also the user has to remember some GPG quirks to get the security right: specifying --force-mdc and checking the return value of gpg -d.
  • USB pen drive + GPG for encryption. This is similar to ecplmdr above, but less convenient and less secure, because user-invented passphrases tend to be weak, and strong passphrases are long and cumbersome to type. Also the user has to remember some GPG quirks to get the security right: specifying --force-mdc and checking the return value of gpg -d.
  • Using a QR code and scanning it with the webcam: qrencode + zbarcam + GPG for encryption. This works for files smaller than about 10 KiB, because the resolution of the webcam in many laptops is not good enough to scan large QR codes. Without GPG this is not secure, in case someone is taking a video recording of the computer screen. Also the user has to remember some GPG quirks to get the security right: specifying --force-mdc and checking the return value of gpg -d.
  • Setting up the secret key in your YubiKey on one computer, copying the public key from it onto the second computer, and connecting via ssh to the second computer. This works if you already have a YubiKey, the first computer is nearby, and it's convenient for you to set up and dump keys on your YubiKey. How to retrieve the SSH public key from the YubiKey: use ssh-add -L | grep cardno: Because of the many skilled manual steps involved, this solution is less convenient than the recommendations above.
  • Setting up the secret key in your YubiKey on one computer, adding metadata, then using the list command in gpg --card-edit to get the metadata. This can be used to copy a few hundred bytes if both computers are nearby (i.e. you can connect the same YubiKey to both). This is similar to using an USB pen drive to copy files, but perhaps a bit more secure. (It's more secure only if an attacker stealing your YubiKey can't extract the metadata without knowing the passphrase. This has to be checked.)

Requirements

Security requirements:

  • t encrypts the data end-to-end, only the receiver is able to decrypt it.
  • The receiver is able to detect if the data is indeed what the sender has sent (e.g. it was not tampered with and it was not replaced by the data provided by the attacker).

Convenience requirements:

  • It works on the command-line.
  • It works as a regular user (non-root) on the both computers.
  • It works without software installation on the both computers.
  • It works without creating any file other than the output data file in the receiver. (We can relax this: a few small temporary files are OK, if they get removed automatically in the end.)
  • It works with very little typing (at most 20 characters of key typing in total). Copy-pasting is OK, but not between the sender and the receiver.
  • There is a mode which works on the local network without a public or local service running and without extra hardware (network cables or USB pen drives).
  • There is a mode which works without a local network and without extra hardware; it is allowed to use a public service.
  • There is a mode which works without any network (local network or internet); it is allowed to use a USB pen drive.

One-off SCP with Dropbear

If one of the computers (let's call it client) has the OpenSSH client (ssh) installed, and is able to connect to the other computer (let's call it the server), you have root access on the server, and the server doesn't have a working OpenSSH server (sshd) installed, and you don't mind intalling software to the server temporarily, you can follow these steps to copy files securely.

On the server, install Dropbear. For example, on Debian 9 or later, run this as root (without the leading #):

# apt-get install dropbear-bin

On the server, install the scp command-line tool, part of OpenSSH. For example, on Debian 9 or later:

# apt-get install openssh-client

On the server, generate an SSH host key, and start the server:

# dropbearkey -t rsa -s 4096 -f dbhostkey
# /usr/sbin/dropbear -r dbhostkey -F -E -m -w -j -k -p 64358 -P dbtmp.pid

The last command (dropbear) makes the Dropbear SSH server keep running and serving incoming connections until you press Ctrl-C in the terminal window. This is normal.

When dropbearkey above prints Fingerprint: md5, remember the value, because you will have to compare it with the value printed by the client.

On the client, initiate the copy with the following command (without the leading $):

$ SSH_AUTH_SOCK= scp -o Port=64358 -o HostName=... -o User=... \
    -F /dev/null -o UserKnownHostsFile=/dev/null \
    -o HostKeyAlgorithms=ssh-rsa -o FingerprintHash=md5 SOURCE DESTINATION

In the command above:

  • Specify HostName=... as the host name of the server.
  • Specify User=... as the non-root user name to be used on the server. scp will ask that user's password on the client.
  • SOURCE and DESTINATION can be a filename on the client, or, if prefixed by r:, then it's a filename inside the home directory of the user on the server.
  • If scp complains about FingerprintHash, then drop the -o FingerprintHash=md5, and try again.
  • When the client prints RSA key fingerprint is MD5:..., compare the ... value with the value printed by dropbearkey on the server. If they don't match perfectly, stop. If you continue even then, then you may be a victim of a man-in-the-middle attack, and your copy is not secure.

You may run multiple copies with scp between the client and the server.

As an alternative to scp, you can also use rsync to do the copies (if rsync is installed to both the client and the server). The command to be run on the client looks like this:

$ SSH_AUTH_SOCK= rsync -e 'ssh -o Port=64358 \
    -o HostName=... -o User=... -F /dev/null \  
    -o UserKnownHostsFile=/dev/null -o HostKeyAlgorithms=ssh-rsa \
    -o FingerprintHash=md5' --progress -avz SOURCE DESTINATION

Abort Dropbear on the server by pressing Ctrl-.

Having run the copies, remove unnecessary packages from the server. For example (do it carefully, don't remove anything you need), on Debian 9:

# sudo apt-get purge dropbear-bin libtommath1 libtomcrypt1
# sudo apt-get purge openssh-client

2018-04-28

How to force OpenSSH to log in with a specific password or public key

This blog post explains how to force the OpenSSH client to log in with a specific password or public key. This is useful if some of the SSH client config files (/etc/ssh/ssh_config, /etc/ssh/ssh_known_hosts, /etc/ssh/ssh_known_hosts2, ~/.ssh/config, ~/.ssh/known_hosts) or the ssh-agent are in a broken state, and you want to try whether login works independently of these client-side issues.

Run this command to log in, substituting the "${...}" values:

SSH_AUTH_SOCK= /usr/bin/ssh -F /dev/null \
    -o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null \
    -o StrictHostKeyChecking=no \
    -p "${PORT}" -i "${KEYFILE}" -- "${USERNAME}"@"${HOST}"

Usage notes:

  • To use the default port (22), drop the -p "${PORT}".
  • To use password login instead of public key login, drop the -i "${KEYFILE}".
  • If you don't know where your public key file is, try -i ~/.ssh/id_rsa
  • To use the same username as your local client username, drop the "${USERNAME}"@.

How it works:

  • SSH_AUTH_SOCK= disables the ssh-agent for this connection.
  • Spelling out /usr/bin/ssh makes sure that shell aliases, shell functions and strange directories in $PATH have no effect on which SSH client is used.
  • -o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null makes existing host keys in known_hosts files to be ignored, thus the connection will be established even if old or incorrect host keys are saved there. Please note that this also makes it impossible to detect a man-in-the-middle attack, so attackers may be able to steal your password if you use a password to log in; also attackers can steal the contents of your session (commands and their results).
  • -o StrictHostKeyChecking=no suppresses the prompt to add the host key to the known_hosts files.

2018-04-24

A quest to find a fast enclosure for multiple SATA 3.5" hard drives

This blog post documents the quest I'm undertaking to find a fast enclosure for multiple SATA 3.5" hard drives, supporting both USB 3 and eSATA, and the ability to read from both hard drives at the same time with at least 275 MB/s total speed. So far I haven't found a fast enough enclosure, so the quest is till ongoing. I'll keep updating the blog post with speed benchmark results.

The maximum sequential read speed my drives support are 112 MB/s and 170 MB/s. (There are much faster drives on the market, e.g. Seagate IronWolf NAS 10 TB can read 250 MB/s in the first 1 TB of the disk.)

I've decided not to order the IcyBox IB-RD3662U3S, because my online research indicates it would be too slow. It uses the chipset JMicron JMB 352 (produced in 2014), which doesn't support UASP (thus it's slow and it uses too much CPU) and maximum SATA speed is 3 Gbit/s.

I've ordered the StarTech S3520BU33ER instead, which uses the chipset JMicron JMS 562 (also produced in 2014), which supports UASP and maximum SATA speed is 6 Gbit/s. I'll run the benchmarks after it arrives.

I've also found OWC 0GB Mercury Elite Pro Dual RAID USB 3.1 / eSATA Enclosure Kit, which is potentially even faster. It supports USB 3.1, eSATA, UASP, and claims to be very fast: more than 400 MB/s over both USB and eSATA. It also uses the same chipset: JMicron JMS 562. It's avaialable from amazon.com and from the manufacturer's webshop (with expensive international delivery).

Depending on the computer it can be much faster to connect the 2 hard drives within separate single-drive enclosures, using separate USB 3 ports or using an unpowered hub. I'm not pursuing this option right now, because I have other uses for my USB ports, and I want low CPU usage (eSATA uses less than USB 3).

For a home media server, it may be cheaper to buy a NAS, e.g. QNAP TS-251+ with Ethernet and HDMI ports, DLNA with full HD video transcoding and other media server features, with maximum transfer speed of 224 MB/s. (Other kinds of QNAP NASes don't seem the be any faster.) However, with a NAS I wouldn't get the flexibility and configurability of stock Debian operating system running on a stock amd64 CPU with 4 GiB of RAM on this machine.

2018-04-21

How to update the BIOS on a Lenovo T400 laptop

This blog post explains how to update the BIOS to version 3.24 (released on 2012-12-16, latest release as of 2018-04-21) on a Lenovo T400 laptop.

This instructions below don't seem to work, I get an Operating system not found error when booting from the pen drive. Unfortunately I don't know how to fix that.

You will need a working and charged battery pack for the BIOS update, so install the battery pack first and start charging it.

If you are running Windows XP, Windows Vista or Windows 7 on the laptop, download the BIOS Update Utility from here (choose 32-bit or the 64-bit version depending on your Windows type, or try both versions if you don't know), and run it, and you are done.

Otherwise, if you are able to burn a CD or DVD (either on the Lenovo T400 laptop or on another computer), and you have a working DVD reader in the Lenovo T400, then download the installer DVD .iso from here, burn it to a DVD, insert the DVD to the Lenovo T400, reboot the Lenovo T400, press the blue ThinkVantage button (near the top left corner of the keyboard), press F12 to select a boot device, select the DVD, boot from it.

Otherwise, if you have a USB pen drive of at least 34 MB in size whose contents can be overwritten, and you have a Linux system running (either on the Lenovo T400 laptop or on another computer), then connect the pen drive, figure out the device name using sudo fdisk -l (typically it will be /dev/sdb or /dev/sdc, but be extra careful, otherwise you will overwrite the contents of some other drive), run this command to download: wget https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles/7uuj49uc.iso; run this command to copy the bootable BIOS update utility to the pen drive: sudo dd if=7uuj49uc.iso of=/dev/sdB bs=49152 skip=1; sync (replacing /dev/sdB with the device of the pen drive). Insert the pen drive to one of the USB slots of the Lenovo T400, reboot the Lenovo T400, press the blue ThinkVantage button (near the top left corner of the keyboard), press F12 to select a boot device, select USB HDD, boot from it.

After booting into the BIOS update utility, follow the instructions to update the system software. (Don't reboot or turn off until asked.) The next reboot will take longer, the Lenovo logo will appear and disappear 3 times. After that you are done.

Now if you enter the BIOS setup at boot time (by pressing the blue ThinkVantage button), you will see version 3.24 (7UET94WW) 2012-10-17.

2018-04-09

How to change which characters are selected by double-clicking in xterm

Various terminal emulators on Linux (e.g. xterm, gnome-terminal, rxvt) have word selection: when you double-click a character, it selects the entire word containing the character. This blog post explains how to customize which characters are part of a word in xterm.

The various defaults are for ASCII characters (in addition to digits and the letters a-z and A-Z):

  • gnome-terminal: # % & + , - . / = ? @ \ _ ~
  • rxvt: ! # $ % + - . / : _
  • xterm default: _
  • xterm in Ubuntu: ! # % & + , - . / : = ? @ _ ~

It's possible to customize which characters are part of a word in xterm by specifying the charClass resource. The values :48 mean: consider these characters part of a word. Other numbers are character ranges, for example 43-47 mean the ASCII characters 43 (+), 44 (,), 45 (-), 46 (.) and 47 (/).

Here is how to trigger various default behaviors from the command-line:

  • gnome-terminal: xterm -xrm '*.VT100.charClass: 35:48,37:48,38:48,43-47:48,61:48,63-64:48,92:48,95:58,126:48'
  • rxvt: xterm -xrm '*.VT100.charClass: 33:48,35-37:48,43:48,45-47:48,58:48,95:58'
  • xterm default: xterm -xrm '*.VT100.charClass: 95:48'
  • xterm in Ubuntu: xterm -xrm '*.VT100.charClass: 33:48,35:48,37-38:48,43-47:48,58:48,61:48,63-64:48,95:48,126:48'

To save the setting permanently, add a line like this to your ~/.Xresources file (create it if it doesn't exist):

! Here is a pattern that is useful for double-clicking on a URL (default xterm in Ubuntu):
XTerm.VT100.charClass: 33:48,35:48,37-38:48,43-47:48,58:48,61:48,63-64:48,95:48,126:48

Make sure above that the line containing charClass doesn't start with !, because that would be a comment.

The change takes affect automatically the next time you log in. To make it take effect earlier (for all xterms you start), run: xrdb -merge ~/.Xresources