How to automatically synchronize the shell history between terminal windows

This blog post explains how to automatically synchronize the Unix shell command-line history between multiple shell instances running on the same computer (but possibly different terminal windows), without requiring any manual action (such as running history -w or exiting from the shell) from the user, and also how to make commands be saved to the history file right at the point when the command is executed. The solutions described here work with bash 4.1 (tested with 4.1.5) or newer and zsh (tested with 4.3.4 and 4.3.10), run on Ubuntu Lucid. No attempt was made to synchronize the zsh history and the bash history.

Instructions for zsh

Put (append) this command to your ~/.zshrc and reopen your terminal windows so that the changes take effect:

setopt hist_ignore_dups share_history inc_append_history extended_history

The change takes effect in terminal windows and incoming SSH connections you open from now on. Close your old windows if necessary to avoid confusion. You can also run the source ~/.zshrc to make the change take effect.

Optionally, set your HISTSIZE and SAVEHIST to large enough values, e.g. put HISTSIZE=500000 SAVEHIST=5000000 in your ~/.zshrc.

Once the change took effect, you can start running a command in one terminal window, then just press Enter in another terminal window, then press Up in the other terminal window to get the command running in the other terminal window. Your shell history lines are now synchronized. Synchronization happens whenever you press Enter in a terminal window.

Instructions for bash

Since bash doesn't have the history sharing feature by default, you need to use a helper script for bash. Download it using the following command (without the first $):

$ wget -O ~/merge_history.bash http://raw.github.com/pts/pts-merge-history-bash/master/merge_history.bash

Run

$ touch ~/.merged_bash_history

to signify to the helper script that you want to use the new history feature.

Put (append) this to your ~/.bashrc so the helper script gets loaded at shell startup:

source "$HOME"/merge_history.bash

Optionally, set HISTSIZE and HISTFILESIZE to large enough values, e.g. append the following to your ~/.bashrc:

HISTSIZE=500000 HISTFILESIZE=5000000

The change takes effect in terminal windows and incoming SSH connections you open from now on. Close your old windows if necessary to avoid confusion. You can also run the source command above to make the change take effect.

Once the change took effect, you can start running a command in one terminal window, then just press Enter in another terminal window, then press Up in the other terminal window to get the command running in the other terminal window. Your shell history lines are now synchronized. Synchronization happens whenever you press Enter in a terminal window.

If you want your old shell history to be reused, please copy it any time, and then press Enter:

cp ~/.bash_history ~/.merged_bash_history

Explanation of the bash solution

Bash doesn't have support for manual or automatic synchronization of history files. By default, bash reads the history file ~/.bash_history (can be changed by setting the HISTFILE bash special variable) at startup, adds newly executed commands to its in-memory history buffer, and simply writes (or appends) the contents of the buffer to the history file upon clean exit. With this strategy, commands can be lost for good if the shell is killed with kill -9, or multiple shells are exiting (the last one wins, lines from all previous ones get irrecoverably lost). It's possible to affect this behavior with shopt -s histappend to force appending instead of overwriting, but that's still much less synchronization with command sorting by timestamp. The commands history -w, history -a history -c and history -r can be useful for managing the bash history files, but they may still cause data loss (see above).

To solve these problems, the helper script merge_history.bash above uses the following techniques:

• It uses HISTTIMEFORMAT bash special variable to make sure timestamps are saved into the history file.
• It contains a Perl script to merge history files (possibly created by concurrently running bash instances), sorting the results by timestamp and removing duplicates.
• Before displaying the prompt (using the PROMPT_COMMAND bash special variable) loads the merged history file from disk.
• When the user presses Enter to run a command, before the command is executed, it saves the in-memory history buffer to a file, and merges that file using the Perl script to the merged history file. It uses trap ... DEBUG (and various black magic trickery) to install its hook before the command is executed. Other solutions for bash history synchronization don't use this technique, so they suffer from the shortcoming of saving the most recently executed command too late (only when it finishes).

merge_history.bash was inspired by the bash preexec emulation script.

Get the most recent version of merge_history.bash from http://raw.github.com/pts/pts-merge-history-bash/master/merge_history.bash . For your studying convenience, here is an outdated copy:

MRG_DONEI=":$SHELLOPTS:"
if test "${MRG_DONEI#*:history:}" != "$PTS_DONEI" &&
   (test "${BASH_VERSION#[5-9].}" != "$BASH_VERSION" ||
    test "${BASH_VERSION#4.[1-9].}" != "$BASH_VERSION") &&
   test "$HOME" &&
   test -f "$HOME/.merged_bash_history"; then

# Merge the timestamped .bash_history files specified in $@ , remove
# duplicates, print the results to stdout.
function _mrg_merge_ts_history() {
  PERL_BADLANG=x perl -wne '
    use integer;
    use strict;
    use vars qw($prefix @lines);
    if (/^#(\d+)\n/) {
      $prefix = sprintf("%030d ", $1);
    } else {
      chomp; $prefix = sprintf("%030d ", time) if !defined $prefix;
      push @lines, "$prefix$_\n"; undef $prefix;
    }
    END {
      my $prev = "";
      for (sort @lines) {
        s@^(\d+) @@; my $ts = $1 + 0; my $cur = "#$ts\n$_";
        print $cur if $cur ne $prev;
        $prev = $cur;
      }
    }
  ' -- "$@"
}

# Read history from $HISTFILE_MRG, 
function _rdh() {
  test "$HISTFILE_MRG" || return
  local HISTFILE="$HISTFILE_MRG"
  # Make `history -w' prefix "$TIMESTAMP\n" to $HISTFILE
  local HISTTIMEFORMAT=' '
  history -c  # Clear the in-memory history.
  history -r  # Append the contents of $HISTFILE to the in-memory history.
}

export -n HISTTIMEFORMAT HISTFILE HISTFILE_MRG
unset HISTFILE  # No history file by default, equivalent to HISTFILE="".
unset HISTTIMEFORMAT
HISTFILE_MRG="$HOME/.merged_bash_history"
history -c  # Discard the current history, whatever it was.

function hook_at_debug() {
  test "$COMP_LINE" && return  # Within the completer.
  trap '' DEBUG  # Uninstall debug trap.
  test "$HISTFILE_MRG" || return
  if : >>"$HISTFILE_MRG"; then
    # Make `history -w' prefix "$TIMESTAMP\n" to $HISTFILE
    local HISTTIMEFORMAT=' '
    # TODO(pts): Don't save if nothing changed (i.e. `history -1` prints
    # the same sequence number as before).
    local TMPDIR="${TMPDIR:-/tmp}"
    local HISTFILE="$TMPDIR/whistory.$UID.$$"
    local MHISTFILE="$TMPDIR/mhistory.$UID.$$"
    history -w  # Write to /tmp/whistory.$$ .
    _mrg_merge_ts_history "$HISTFILE_MRG" "$HISTFILE" >"$MHISTFILE"
    command mv -f -- "$MHISTFILE" "$HISTFILE_MRG"
  fi
}

# Set these both so hook_at_debug gets called in a subshell.
set -o functrace > /dev/null 2>&1
shopt -s extdebug > /dev/null 2>&1

# As a side effect, we install our own debug hook. We wouldn't have to do
# that if bash had support for `preexec' (executed just after a command has
# been read and is about to be executed). in zsh.
PROMPT_COMMAND="trap '' DEBUG; _rdh; trap hook_at_debug DEBUG; $PROMPT_COMMAND"

fi  # End of the file's if guard.
unset MRG_DONEI