Linux notes

Home

linux-logo.png

Table of Contents

Desktop Environment

MATE

This is a nice lightweight environment once you have adjusted the shortcuts to be more familiar. In addition to the Control Center, there is mate-tweak which provides a GUI for additional customisation.

Panel

The following command should replace the current panel with a new copy of it: mate-panel --replace &. This may help if your panel disappears for some reason.

Linux and Ubuntu notes

Setting up a server

  • See these notes on how to set up a University of Melbourne Researcher Desktop.

The Melbourne Researcher Desktop

  1. Create a Linux Researcher Desktop via the Research Computing Portal and make a note of the IP address of the server.
  2. Add the host to your SSH configuration, ~/.ssh/config, using the example below. SSH into the server with ssh <hostalias>.
  3. Run sudo apt update and sudo apt upgrade and install the following packages: htop screen tree.
  4. Install Miniconda using the instructions on the Anaconda website. My conda notes might be useful at this point. (Basically, this is running their shell script then running the initialisation command.)
  5. Configure your file manager to connect to the server. For GNOME files you can ask it to connect to sftp://hostalias. In ~/.bashrc add prompt configuration below.
Host myDemoHost
        Hostname vm-XXX-XX-XXX-XX.desktop.cloud.unimelb.edu.au
        User azarebski

In this example XXX-XX-XXX-XX is the IP address of the host.

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\h\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\[\033[1;31m\](Hostalias)\[\033[0m\]\$ '

ImageMagick and convert

Help! Permissions problem

Follow these steps:

  1. Locate the policy.xml file. For Ubuntu, the file is usually found at /etc/ImageMagick-6/policy.xml.
  2. Open the policy.xml file with a text editor using root/administrative privileges.
  3. Locate the following line in the policy.xml file:
<policymap>
  ...
  <policies>
    <policy domain="coder" rights="none" pattern="PDF" />
    ...
  </policies>
  ...
</policymap>
  1. Modify the line to allow ImageMagick to read and write PDF files:
<policy domain="coder" rights="read|write" pattern="PDF" />
  1. Save and close the policy.xml file.
  2. Now, you should be able to execute the convert command
convert -density 300 fig01.pdf fig01.png

Dconf Editor

There is a GUI called Dconf Editor which makes editing your configuration much easier than editing files.

Updating Ubuntu

The do-release-upgrade command can be used to run an update. First run do-release-upgrade -c to check if there is an available upgrade. When asking to upgrade you might get an error message saying that there are packages that need to be upgraded first. Comments on the internet make this sound scary, but when I encountered this I just needed to remove an old program I didn't need which refused to upgrade and then everything worked fine.

System information

Neofetch

neofetch

Neofetch is a CLI system information tool written in BASH. Neofetch displays information about your system next to an image, your OS logo, or any ASCII file of your choice.

Environment variables

There are the env and printenv functions to print all the environment variables known to a shell. To get the hostname there is the hostname function (this information does not seem to appear as an environment variable).

USB

Mounting

The following steps seem to work…

  1. Find the drive with fdisk -l (for example /dev/sdc1)
  2. Mount the drive with mount (mount /dev/sdc1 /media/usb-drive)
  3. Use the drive
  4. Dismount with umount (umount /media/usb-drive)

Depending on how the drive is formatted you may need to install packages to work with exFAT.

Formatting a USB

There is a wikihow which describes how to format a USB from the command line:

https://www.wikihow.com/Format-a-USB-Flash-Drive-in-Ubuntu

Compression with zip

To zip the directory foo into a file called foo.zip run the following

zip -r foo.zip foo

To unpack a zipped up tarball use the following

tar -zxvf foobar.tar.gz

Graphics (PNG, SVG, …)

Shrinking images

The following will shrink foo.png to \(20\%\) of its original size. This leads to a smaller image of a smaller size.

convert -resize 20% foo.png

PDF to PNG

This will convert the input PDF to a PNG with 300 DPI.

convert -density 300 input.pdf output.png

You may get complaints about the colour space, in which case try

convert -density 300 input.pdf -colorspace RGB output.png

PDF to PNG with a white background

convert -density 300 -background white -alpha remove -alpha off input.pdf output.png

Dithering

This is the process of compressing an image to reduce its size. With a little bit of care this can produce appealing images that are much smaller than the original. Some of the logos on the home page are dithered using ditherit with the following settings.

Variable Value
Palette Blue & Yellow
Width 400px
Algorithm FloydSteinberg

In some cases these settings lead to larger files than the original, in that case, try tweak the width parameter and you can get vastly improved performance without visible issues.

Inverting colours

The following can be used to invert the colours of a PNG with ImageMagick

convert input.png -channel RGB -negate output.png

SVG to PDF/PNG

See my inkscape notes for details on how to do this.

Resizing a PNG

  1. Open the image in GIMP
  2. Image > Scale Image
  3. Set the interpolation to cubic
  4. Click Scale
  5. Save or export in the desired format

Removing background from PNG

  1. Open the image in GIMP as layers
  2. Right click and use Colours then Colours to Alpha and it should remove the background.
  3. Save this as a new image.

Working with PDF

LibreOffice Draw is surprisingly good for editing and annotating PDFs!

pdftk is a great tool for working with the pages of a PDF. The man page has some useful examples.

Splitting a PDF into pages

pdftk pdftk input.pdf burst

Extracting a subset of the pages

Extract pages 1–12 from input.pdf and write them to output.pdf

pdftk input.pdf cat 1-12 output output.pdf

Join multiple documents

pdftk in1.pdf in2.pdf cat output out1.pdf

Extracting figures

If you want to extract a figure from a PDF of a paper, see my Inkscape notes here.

Fonts

To install the Monaco font there is a useful stackexchange post linking to a script. For the Source Code Pro font, I have a script in my dotfiles repository. If the method above does not work, you can download the TTF file into ~/.fonts/ and then run sudo fc-cache -fv and it should work after that.

You can download all of the TFF files and then drag them into the Font Viewer application that comes with Ubuntu. Then there is a button to click which should install the font.

Filesystem Hierarchy Standard

According to Wikipedia "[t]he Filesystem Hierarchy Standard (FHS) is a reference describing the conventions used for the layout of a UNIX system." The standard is maintained by the Linux foundation and the main directories to know about are is well documented. If you are unsure of where something should be located, this would be a good place to go for guidance.

Filesystem permissions

A file can be read from, r, written to, w, or executed, x. The permission to carry out these actions is specified for the owner of the file, members of the group, and other users. For example, rwxr-x--- means the owner has read write and execute, other group members can read and execute and others cannot read, write or execute. The chmod functions is used to change the permissions.

chmod 750 foo.sh  # makes foo.sh rwxr-x---
RWX Code
rwx 7
rw- 6
r-x 5
r-- 4
-wx 3
-w- 2
--x 1
--- 0

Useful permission codes

Code Permissions Meaning
750 rwxr-x--- Only you edit
444 r--r--r-- Read only

screen

The screen program helps you manage long-running tasks, it is particularly helpful if you need to be able to disconnect from the task and come back later without interrupting the program in the mean time.

  • Start a session with screen.
    • use screen -S <session_id> to name a session
  • Detach from a session with C-a d.
  • Kill a session with C-a k.
  • Get help on available keybindings with C-a ?.
  • screen -ls lists sessions.
  • Re-attach to a session with screen -r (screen -r <session_id> if there is ambiguity.)

Dealing with large amounts of output in screen

You can copy text from a screen session to a file with the following steps:

  1. From inside the screen session enter copy mode by pressing C-a [.
  2. In copy mode you can navigate using vim keybindings; move the cursor to the beginning of the text you want to copy.
  3. Press the SPC to moving into selection mode and select the text.
  4. Press the SPC again to copy the selected text to screen's buffer.
  5. Now you can paste the copied text into a file. First, exit copy mode with Esc.
  6. While still in the screen session, open a file with vim and paste the copied text by pressing C-a ].

The Oxford VPN

To run the VPN GUI in Ubuntu

/opt/cisco/anyconnect/bin/vpnui

Then you need to connect to vpn.ox.ac.uk with the zool[0-9]{4} username.

ssh

Adding the following to your ~/.ssh/config file will make ssh-ing a bit nicer.

Host zool
    HostName <IP>
    User <USERNAME>

Then you only need to do ssh foobar.

Finding your IP address

This information is available from Settings > Network and then click on the button (cog) next to the wired connection.

Alternatively, you can use ifconfig at the command line which will print out a bunch of network information and you should be able to spot a sensible IP address among it.

scp

Here is an example of a script I used to copy files from my desktop at work to my laptop.

LOCAL_DIR=out/result-v2
REMOTE_DIR=/home/aez/projects/example/$LOCAL_DIR
REMOTE_HOST=zool

scp $REMOTE_HOST:$REMOTE_DIR/*{html,png} $LOCAL_DIR/

Note that this makes use of the zool alias from above.

Symlinks

ln -s /path/to/file /path/to/symlink

Note that these paths need to be full paths otherwise the file wont get found.

Finding files with find

find example 1

To find all the PNG files down from the current directory.

find . -regex ".*png$"

find example 2

To (recursively) find all files with "figure-1" appearing the their name use the following.

find . -name \*figure-1\* -print

Searching files with grep

grep [OPTION]... PATTERNS [FILE]...

grep example 1

Search files in the current directory with filenames matching "*l*notes.org" which contain the string "foo", and show the line number.

grep -r -n --include="*l*notes.org" "foo" .

finding and grepping example

The following recursively searches down from public-site/org for files with names matching the pattern .*book.* and then pipes these into a grep command looking for lines matching the pattern "rnest".

find public-site/org/ -regex ".*book.*" -print0 | xargs -0 grep "rnest"

Emacs

There is a very nice interface to grep provided by rgrep in Emacs.

Watching files with inotifywait

The package inotify-tools provides the inotifywait command which will run a command after an event relating to a file occurs.

Example

You want to compile a PDF every time the file demo.tex gets modified. The following shell script — modified from this stackoverflow answer — will achieve this using inotifywait.

WATCH_FILE=demo.tex

inotifywait -q -m -e close_write $WATCH_FILE |
    while read -r filename event; do
        pdflatex $WATCH_FILE
    done

Monitor configuration

The xrandr command allows you to specify the configuration of the monitor/s. There is a GUI for xranrd called arandr which may substantially simplify the process of configuring the monitor. Once you have a configuration you are happy with, you can use arandr to write it to file. By default these configurations will be stored in the ~/.screenlayout directory.

When I first installed Ubuntu, the windows were all far too small. Running xrandr -s 1440x900 was much more suitable for a 14 inch laptop. Update: all of a sudden the display on my second monitor went strange. I tried use xrandr to set the display to 1440x900 but it didn't appear as one of the options. The solution was to run the following command.

$ cvt 1440 900
# 1440x900 59.89 Hz (CVT 1.30MA) hsync: 55.93 kHz; pclk: 106.50 MHz
Modeline "1440x900_60.00"  106.50  1440 1528 1672 1904  900 903 909 934 -hsync +vsync

The output of this is then what you are meant to use to fix the display by running the following

$ xrandr --newmode "1440x900_60.00"  106.50  1440 1528 1672 1904  900 903 909 934 -hsync +vsync
$ xrandr --addmode DP-1 "1440x900_60.00"
$ xrandr -s 1440x900

They say for this to persist, you need to add those lines to a file at /etc/profile.d/external_monitor_resolution.sh but I haven't tried this yet.

Keyboard configuration

The caps lock key is solid real-estate but is wildly underutilised, and control is more useful than alt a lot of the time. Adding the following to a file called Xmodmap and then run it with xmodmap <path/to/Xmodmap>. It will switch escape and caps lock and the right alt key will become another control key.

clear lock
keycode 9 = Caps_Lock
keycode 66 = Escape

remove Control = Control_R
remove Mod1 = Alt_R
keycode 105 = Alt_R
keycode 108 = Control_R
add Control = Control_R
add Mod1 = Alt_R

Network connection Wi-Fi

Use nmcli device wifi list to get a list of all the available wifi networks. Note the SSID of the network you want to connect to.

nmcli device wifi connect <SSID> password <password>

Disk usage: du

The following will print, in human readable form, the size of all sub-directories of foobar recursively up to 2 levels deep.

du -h --max-depth=2 foobar

Although, unless you are limited to a terminal, something like disk usage analyzer GUI is a much better solution.

Disk usage: Disk Usage Analyzer

The Disk Usage Analyzer provides an interactive tree map of disk usage which makes it easy to find large files on your machine.

Monitoring processes with htop

The htop command provides an interactive process viewer. Make sure that you keep the F6 and F9 keys unbound if you want to use this because they are used to sort the processes and kill them respectively. When killing processes, you probably want to use signal number 9.

Viewing hierarchy tree

Definitely want to use this with colour!

tree -C <filepath>

Be careful though, if you use this without thinking you can get a ridiculous amount of material printed out. There is a command line option for ignoring particular directories.

rsync

The rsync utility provides a way to synchronise the files from one place to another. There is a useful introduction to rsync here (and a simpler one here.) Commands will look like the following:

rsync [OPTIONS] SRC DEST

My use case for this is keeping the same set of PDFs on multiple machines. The following command will make a copy of all the files on zool in the library to my local library.

rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh \
      --recursive --times --perms --links \
      zool:/home/aez/Documents/library/* /home/aez/Documents/library/

Here we have used zool which is a host described in ~/.ssh/config . I have a script to synchronise my libraries across machines.

Dry runs

If you want to test out a command first, there is the -n or --dry-run flags which should be used with -v to actually get the output verbosely printed. For example

rsync -anv dir1/ dir2

will print out a list of the files in dir1 that would be copied to dir2 but will not actually change any files until the n flag is removed. The a flag is for "archive" which is a detailed version of r (for recursive copying).

Do not overwrite

There is the flag --ignore-existing which tells rsync not to send files that already exist on the other side. This is useful if you do not want to accidentally overwrite something. To get a report of which files have been skipped you can also use --info=skip.

Getting help

Manuals

The manuals provide a built-in source of information. A nice feature of the man command is that it also allows you to search for a function based on the words in the description with man -k <command> and you can print just a short description of a command with man -f <command>.

Log files

Sometimes you can find error messages in the following log file. Note that this is in /var because the file has a variable size as it is logged to.

less /var/log/syslog

Start up and configuration

To avoid confusion between the .bashrc and .bash_profile you can just add the following to .bash_profile and then you'll have the same script evaluated regardless.

if [ -f $HOME/.bashrc ]; then
    source $HOME/.bashrc
fi

By default, tab completion is case sensitive, to fix this add the following line to the file /etc/inputrc.

set completion-ignore-case on

Accounts

Adding new users

An example command to add a user is the following:

sudo useradd -c "Jerry Smith" -d /home/jsmith -m -s /bin/bash jsmith
  • -c is to create a comment which is used for the users full name.
  • -d is their home directory.
  • -m says make the home directory if it does not exist.
  • -s to specify what shell they should use.
  • jsmith is the username (which should be alphanumeric and must start with a lowercase letter).

Setting a user's password

  1. Login as root with sudo -i.
  2. Set the password for jsmith with the following sudo passwd jsmith.
  3. Logout as root with exit.
  4. Check that jsmith can log in with su - jsmith.

Setting a user's groups

  • Check which groups jsmith is part of with groups jsmith.
  • Add jsmith to the fleeb group with usermod -aG fleeb jsmith.
  • Remove jsmith from the fleeb group with gpasswd -d jsmith fleeb.

Removing a user

As root run userdel -r jsmith to remove jsmith and their home directory.

Bluetooth

If you find that audio comes out stuttered, try restarting Bluetooth with the following command:

sudo service bluetooth restart

You may have to repeat this process a few times, but it seems to work eventually. Alternatively, you may need to add the following line to /etc/modprobe.d/alsa-base.conf before restarting the service for this to work:

options snd-hda-intel model=generic

Colour selection and hexcodes

The program Gpick displays the hexcode of the pixel under the cursor.

Nautilus - A file manager for GNOME

To open nautilus without the desktop fuss use nautilus --no-desktop

Using nautilus to browse files on a remote machine

  1. Open nautilus
  2. "Other Locations" > "Connect to Server"
  3. SSH not FTP

Packages

Removing a PPA

  1. Open Software & Updates.
  2. Navigate to the Other Software tab.
  3. Select and Remove the desired PPAs.

Installingfrom a .deb file

Note that the <path/to.deb> needs to actually be a path, e.g. ./foo.deb not foo.deb.

sudo apt install <path/to.deb>

Uninstalling packages

remove package with remove

sudo apt remove <package-name>

remove package configuration with purge

sudo apt purge <package-name>

AppImage

This is a new way to distribute packages: download the file, make the file exectuble, and run it. Done! If you are concerned about what the program might do, there are sandboxes available, for example Firejail. Inkscape is available as an appimage.

youtube-dl

There are instructions on how to install this program here. Use youtube-dl --help to get help on usage.

Download audio as MP3

youtube-dl --extract-audio --audio-format "mp3" <VIDEO/URL>

TODO Browsers

TODO Firefox

Scripting

On my machine at uni this is /bin/sh which points to /bin/dash.

There is a commandline tool shellcheck which performs static analysis on a shell script and provides advice to improve it.

Bioinformatics one-liners

Extract the trees from a NEXUS file and put them in a Newick file

This one-liner carries out the following pipeline:

  1. grep the lines of the nexus file containing trees
  2. Take the first 5000 of them
  3. Use awk to remove the text on the line before the tree
  4. Write the results to file.
grep "^tree STATE_" your_file.nexus | head -n 5000 | awk -F"= " '{print $2}' > first_5000_trees.nwk

Control flow

To run comands conditional upon the presence or absence of a directory, consider the following example

if [ -d "foobar" ]; then
    echo "There is a directory called foobar"
    exit 0
fi

if [ ! -d "foobar" ]; then
    echo "Could not find directory: foobar"
    exit 1
fi

Running things in parallel by backgrounding

Example 1

This solution came from the very clever Claire Miller for how one can do map-reduce style computations correctly via bash. Note that this actually needs bash, it does not work with sh alone.

# waiter.sh
echo "Starting"

pids=()

for i in $(seq 1 5)
do
  sleep 4 && echo "Done part $i" & pids+=($!)
done

wait "${pids[@]}"
echo "Finished :)"

which should produce the following output.

$ bash waiter.sh
Starting
Done part 3
Done part 1
Done part 2
Finished :)

Example 2

A slightly more complicated example demonstrating how to chunk processes into subsets to run in parallel before moving onto the next chunk.

#!/usr/bin/env bash

chunk_size=5                    # the number of processes in each chunk
limit=20                        # the total number of processes
ix=0

outer_limit="$(($limit-$chunk_size))"
while [ $ix -le $outer_limit ]
do
    pids=()
    chunk_limit="$(($ix+$chunk_size))"
    while [ $ix -lt $chunk_limit ]
    do
        padded_num=$(printf "%03d" $ix)
        echo $padded_num && sleep 1s & pids+=($!)
        ((ix++))
    done
    wait "${pids[@]}"
    echo "---"
done

which produces

 $ ./demo.sh
000
001
002
003
004
---
005
006
007
008
009
---
010
011
012
013
014
---
015
016
017
018
019
---

Arguments

As an example, here is a simple shell script

#!/bin/sh
# demo.sh
echo "foo: $2"
echo "bar: $1"

Running it we get the following

$ ./demo.sh hello world
foo: world
bar: hello

Functions

Converting ODG to PDF

Here is an example of a function which converts an ODG to a PDF

ConvertODG2PDF () {
    libreoffice --headless --convert-to pdf $1
}

ConvertODG2PDF awesome-figure.odg

Compiling and converting a standalone latex equation

compileStandAlone () {
    pdflatex $1.tex
    convert -flatten -density 300 $1.pdf -colorspace RGB $1.png
    rm $1.aux
    rm $1.log
    rm $1.pdf
}

Loops

Here is a silly little example of how to loop over an explicity list of strings.

for month in jan feb mar;
do
    cat foo-$month.csv
done

Alternatively, one can construct the list programmatically for example with for x in * which would be used to iterate over all files in the current directory or for x in $* which would iterate over all the command line arguments.

Here is another example of iterating over some integers and print 1, 2, 3 to standard out.

for ix in `seq 1 3`;
do
    echo $ix
done

Quiptic compiler

The following helps you to quickly download a bunch of crosswords and compile them into a single PDF. We use the sleep command to be kind to the Guardian servers.

for i in $(seq 1140 1165)
do
    sleep 2                     # be nice!
    google-chrome --headless --disable-gpu --print-to-pdf https://www.theguardian.com/crosswords/quiptic/$i/print
    mv output.pdf quiptic-$i.pdf
done

echo "==========================================================="
echo ""
echo "Guardian - Quiptics"
echo "-------------------"
echo ""
echo "Use the following command to compile the quiptics into a"
echo "single pdf after you have removed any broken ones by hand:"
echo ""
echo "pdftk quiptic-* cat output quiptics.pdf"
echo ""
echo "==========================================================="

Author: Alexander E. Zarebski

Created: 2024-10-19 Sat 10:50

Validate