Linux notes
Table of Contents
- Desktop Environment
- Linux and Ubuntu notes
- Setting up a server
- The Melbourne Researcher Desktop
- ImageMagick and
convert
- Dconf Editor
- Updating Ubuntu
- System information
- USB
- Compression with zip
- Graphics (PNG, SVG, …)
- Working with PDF
- Fonts
- Filesystem Hierarchy Standard
- Filesystem permissions
screen
- The Oxford VPN
- Finding your IP address
scp
- Symlinks
- Finding files with
find
- Searching files with
grep
- Watching files with
inotifywait
- Monitor configuration
- Keyboard configuration
- Network connection Wi-Fi
- Disk usage:
du
- Disk usage: Disk Usage Analyzer
- Monitoring processes with
htop
- Viewing hierarchy
tree
rsync
- Getting help
- Start up and configuration
- Accounts
- Bluetooth
- Colour selection and hexcodes
- Nautilus - A file manager for GNOME
- Packages
- Scripting
Desktop Environment
MATE
This is a nice lightweight environment once you have adjusted the shortcuts to
be more familiar. In addition to the Control Center, there is mate-tweak
which provides a GUI for additional customisation.
Panel
The following command should replace the current panel with a new copy of it:
mate-panel --replace &
. This may help if your panel disappears for some
reason.
Linux and Ubuntu notes
Setting up a server
- See these notes on how to set up a University of Melbourne Researcher Desktop.
The Melbourne Researcher Desktop
- Create a Linux Researcher Desktop via the Research Computing Portal and make a note of the IP address of the server.
- Add the host to your SSH configuration,
~/.ssh/config
, using the example below. SSH into the server withssh <hostalias>
. - Run
sudo apt update
andsudo apt upgrade
and install the following packages:htop screen tree
. - Install Miniconda using the instructions on the Anaconda website. My conda notes might be useful at this point. (Basically, this is running their shell script then running the initialisation command.)
- Configure your file manager to connect to the server. For GNOME
files you can ask it to connect to
sftp://hostalias
. In~/.bashrc
add prompt configuration below.
Host myDemoHost Hostname vm-XXX-XX-XXX-XX.desktop.cloud.unimelb.edu.au User azarebski
In this example XXX-XX-XXX-XX
is the IP address of the host.
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\h\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\[\033[1;31m\](Hostalias)\[\033[0m\]\$ '
ImageMagick and convert
Help! Permissions problem
Follow these steps:
- Locate the
policy.xml
file. For Ubuntu, the file is usually found at/etc/ImageMagick-6/policy.xml
. - Open the
policy.xml
file with a text editor using root/administrative privileges. - Locate the following line in the
policy.xml
file:
<policymap> ... <policies> <policy domain="coder" rights="none" pattern="PDF" /> ... </policies> ... </policymap>
- Modify the line to allow ImageMagick to read and write PDF files:
<policy domain="coder" rights="read|write" pattern="PDF" />
- Save and close the
policy.xml
file. - Now, you should be able to execute the
convert
command
convert -density 300 fig01.pdf fig01.png
Dconf Editor
There is a GUI called Dconf Editor which makes editing your configuration much easier than editing files.
Updating Ubuntu
The do-release-upgrade
command can be used to run an update. First run
do-release-upgrade -c
to check if there is an available upgrade. When asking
to upgrade you might get an error message saying that there are packages that
need to be upgraded first. Comments on the internet make this sound scary, but
when I encountered this I just needed to remove an old program I didn't need
which refused to upgrade and then everything worked fine.
System information
Neofetch
neofetch
Neofetch is a CLI system information tool written in BASH. Neofetch displays information about your system next to an image, your OS logo, or any ASCII file of your choice.
Environment variables
There are the env
and printenv
functions to print all the environment
variables known to a shell. To get the hostname there is the hostname
function
(this information does not seem to appear as an environment variable).
USB
Mounting
The following steps seem to work…
- Find the drive with
fdisk -l
(for example/dev/sdc1
) - Mount the drive with
mount
(mount /dev/sdc1 /media/usb-drive
) - Use the drive
- Dismount with
umount
(umount /media/usb-drive
)
Depending on how the drive is formatted you may need to install packages to work with exFAT.
Formatting a USB
There is a wikihow which describes how to format a USB from the command line:
Compression with zip
To zip the directory foo
into a file called foo.zip
run the following
zip -r foo.zip foo
To unpack a zipped up tarball use the following
tar -zxvf foobar.tar.gz
Graphics (PNG, SVG, …)
Shrinking images
The following will shrink foo.png
to \(20\%\) of its original size. This leads
to a smaller image of a smaller size.
convert -resize 20% foo.png
PDF to PNG
This will convert the input PDF to a PNG with 300 DPI.
convert -density 300 input.pdf output.png
You may get complaints about the colour space, in which case try
convert -density 300 input.pdf -colorspace RGB output.png
PDF to PNG with a white background
convert -density 300 -background white -alpha remove -alpha off input.pdf output.png
Dithering
This is the process of compressing an image to reduce its size. With a little bit of care this can produce appealing images that are much smaller than the original. Some of the logos on the home page are dithered using ditherit with the following settings.
Variable | Value |
---|---|
Palette | Blue & Yellow |
Width | 400px |
Algorithm | FloydSteinberg |
In some cases these settings lead to larger files than the original, in that case, try tweak the width parameter and you can get vastly improved performance without visible issues.
Inverting colours
The following can be used to invert the colours of a PNG with ImageMagick
convert input.png -channel RGB -negate output.png
SVG to PDF/PNG
See my inkscape notes for details on how to do this.
Resizing a PNG
- Open the image in GIMP
Image > Scale Image
- Set the interpolation to cubic
- Click
Scale
- Save or export in the desired format
Removing background from PNG
- Open the image in GIMP as layers
- Right click and use Colours then Colours to Alpha and it should remove the background.
- Save this as a new image.
Working with PDF
LibreOffice Draw is surprisingly good for editing and annotating PDFs!
pdftk
is a great tool for working with the pages of a PDF. The man page has
some useful examples.
Splitting a PDF into pages
pdftk pdftk input.pdf burst
Extracting a subset of the pages
Extract pages 1–12 from input.pdf
and write them to output.pdf
pdftk input.pdf cat 1-12 output output.pdf
Join multiple documents
pdftk in1.pdf in2.pdf cat output out1.pdf
Extracting figures
If you want to extract a figure from a PDF of a paper, see my Inkscape notes here.
Fonts
To install the Monaco font there is a useful stackexchange post linking to a
script. For the Source Code Pro font, I have a script in my dotfiles repository.
If the method above does not work, you can download the TTF file into
~/.fonts/
and then run sudo fc-cache -fv
and it should work after that.
You can download all of the TFF files and then drag them into the Font Viewer application that comes with Ubuntu. Then there is a button to click which should install the font.
Filesystem Hierarchy Standard
According to Wikipedia "[t]he Filesystem Hierarchy Standard (FHS) is a reference describing the conventions used for the layout of a UNIX system." The standard is maintained by the Linux foundation and the main directories to know about are is well documented. If you are unsure of where something should be located, this would be a good place to go for guidance.
Filesystem permissions
A file can be read from, r
, written to, w
, or executed, x
. The permission
to carry out these actions is specified for the owner of the file, members of
the group, and other users. For example, rwxr-x---
means the owner has read
write and execute, other group members can read and execute and others cannot
read, write or execute. The chmod
functions is used to change the permissions.
chmod 750 foo.sh # makes foo.sh rwxr-x---
RWX | Code |
---|---|
rwx |
7 |
rw- |
6 |
r-x |
5 |
r-- |
4 |
-wx |
3 |
-w- |
2 |
--x |
1 |
--- |
0 |
Useful permission codes
Code | Permissions | Meaning |
---|---|---|
750 |
rwxr-x--- |
Only you edit |
444 |
r--r--r-- |
Read only |
screen
The screen
program helps you manage long-running tasks, it is
particularly helpful if you need to be able to disconnect from the
task and come back later without interrupting the program in the mean
time.
- Start a session with
screen
.- use
screen -S <session_id>
to name a session
- use
- Detach from a session with
C-a d
. - Kill a session with
C-a k
. - Get help on available keybindings with
C-a ?
. screen -ls
lists sessions.- Re-attach to a session with
screen -r
(screen -r <session_id>
if there is ambiguity.)
Dealing with large amounts of output in screen
You can copy text from a screen
session to a file with the following
steps:
- From inside the screen session enter copy mode by pressing
C-a [
. - In copy mode you can navigate using vim keybindings; move the cursor to the beginning of the text you want to copy.
- Press the
SPC
to moving into selection mode and select the text. - Press the
SPC
again to copy the selected text toscreen
's buffer. - Now you can paste the copied text into a file. First, exit copy
mode with
Esc
. - While still in the
screen
session, open a file with vim and paste the copied text by pressingC-a ]
.
The Oxford VPN
To run the VPN GUI in Ubuntu
/opt/cisco/anyconnect/bin/vpnui
Then you need to connect to vpn.ox.ac.uk
with the zool[0-9]{4}
username.
ssh
Adding the following to your ~/.ssh/config
file will make ssh-ing a bit nicer.
Host zool HostName <IP> User <USERNAME>
Then you only need to do ssh foobar
.
Finding your IP address
This information is available from Settings > Network and then click on the button (cog) next to the wired connection.
Alternatively, you can use ifconfig
at the command line which will print out a
bunch of network information and you should be able to spot a sensible IP
address among it.
scp
Here is an example of a script I used to copy files from my desktop at work to my laptop.
LOCAL_DIR=out/result-v2 REMOTE_DIR=/home/aez/projects/example/$LOCAL_DIR REMOTE_HOST=zool scp $REMOTE_HOST:$REMOTE_DIR/*{html,png} $LOCAL_DIR/
Note that this makes use of the zool
alias from above.
Symlinks
ln -s /path/to/file /path/to/symlink
Note that these paths need to be full paths otherwise the file wont get found.
Finding files with find
find
example 1
To find all the PNG files down from the current directory.
find . -regex ".*png$"
find
example 2
To (recursively) find all files with "figure-1" appearing the their name use the following.
find . -name \*figure-1\* -print
Searching files with grep
grep [OPTION]... PATTERNS [FILE]...
grep
example 1
Search files in the current directory with filenames matching "*l*notes.org"
which contain the string "foo"
, and show the line number.
grep -r -n --include="*l*notes.org" "foo" .
finding and grepping example
The following recursively searches down from public-site/org
for files with
names matching the pattern .*book.*
and then pipes these into a grep command
looking for lines matching the pattern "rnest"
.
find public-site/org/ -regex ".*book.*" -print0 | xargs -0 grep "rnest"
Emacs
There is a very nice interface to grep
provided by rgrep
in Emacs.
Watching files with inotifywait
The package inotify-tools
provides the inotifywait
command which will run a
command after an event relating to a file occurs.
Example
You want to compile a PDF every time the file demo.tex
gets modified. The
following shell script — modified from this stackoverflow answer — will
achieve this using inotifywait
.
WATCH_FILE=demo.tex inotifywait -q -m -e close_write $WATCH_FILE | while read -r filename event; do pdflatex $WATCH_FILE done
Monitor configuration
The xrandr
command allows you to specify the configuration of the monitor/s.
There is a GUI for xranrd
called arandr
which may substantially simplify the
process of configuring the monitor. Once you have a configuration you are happy
with, you can use arandr
to write it to file. By default these configurations
will be stored in the ~/.screenlayout
directory.
When I first installed Ubuntu, the windows were all far too small. Running
xrandr -s 1440x900
was much more suitable for a 14 inch laptop. Update: all
of a sudden the display on my second monitor went strange. I tried use xrandr
to set the display to 1440x900
but it didn't appear as one of the options. The
solution was to run the following command.
$ cvt 1440 900 # 1440x900 59.89 Hz (CVT 1.30MA) hsync: 55.93 kHz; pclk: 106.50 MHz Modeline "1440x900_60.00" 106.50 1440 1528 1672 1904 900 903 909 934 -hsync +vsync
The output of this is then what you are meant to use to fix the display by running the following
$ xrandr --newmode "1440x900_60.00" 106.50 1440 1528 1672 1904 900 903 909 934 -hsync +vsync $ xrandr --addmode DP-1 "1440x900_60.00" $ xrandr -s 1440x900
They say for this to persist, you need to add those lines to a file at
/etc/profile.d/external_monitor_resolution.sh
but I haven't tried this yet.
Keyboard configuration
The caps lock key is solid real-estate but is wildly underutilised, and control is more useful than alt a lot of the time.
Adding the following to a file called Xmodmap
and then run it with xmodmap <path/to/Xmodmap>
.
It will switch escape and caps lock and the right alt key will become another control key.
clear lock keycode 9 = Caps_Lock keycode 66 = Escape remove Control = Control_R remove Mod1 = Alt_R keycode 105 = Alt_R keycode 108 = Control_R add Control = Control_R add Mod1 = Alt_R
Network connection Wi-Fi
Use nmcli device wifi list
to get a list of all the available wifi networks.
Note the SSID of the network you want to connect to.
nmcli device wifi connect <SSID> password <password>
Disk usage: du
The following will print, in human readable form, the size of all
sub-directories of foobar
recursively up to 2 levels deep.
du -h --max-depth=2 foobar
Although, unless you are limited to a terminal, something like disk usage analyzer GUI is a much better solution.
Disk usage: Disk Usage Analyzer
The Disk Usage Analyzer provides an interactive tree map of disk usage which makes it easy to find large files on your machine.
Monitoring processes with htop
The htop
command provides an interactive process viewer. Make sure that you
keep the F6
and F9
keys unbound if you want to use this because they are
used to sort the processes and kill them respectively. When killing processes,
you probably want to use signal number 9.
Viewing hierarchy tree
Definitely want to use this with colour!
tree -C <filepath>
Be careful though, if you use this without thinking you can get a ridiculous amount of material printed out. There is a command line option for ignoring particular directories.
rsync
The rsync
utility provides a way to synchronise the files from one place to
another. There is a useful introduction to rsync here (and a simpler one here.)
Commands will look like the following:
rsync [OPTIONS] SRC DEST
My use case for this is keeping the same set of PDFs on multiple machines. The
following command will make a copy of all the files on zool
in the library to
my local library.
rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh \ --recursive --times --perms --links \ zool:/home/aez/Documents/library/* /home/aez/Documents/library/
Here we have used zool
which is a host described in ~/.ssh/config
.
I have a script to synchronise my libraries across machines.
Dry runs
If you want to test out a command first, there is the -n
or --dry-run
flags
which should be used with -v
to actually get the output verbosely printed. For
example
rsync -anv dir1/ dir2
will print out a list of the files in dir1
that would be copied to dir2
but
will not actually change any files until the n
flag is removed. The a
flag
is for "archive" which is a detailed version of r
(for recursive copying).
Do not overwrite
There is the flag --ignore-existing
which tells rsync not to send files that
already exist on the other side. This is useful if you do not want to
accidentally overwrite something. To get a report of which files have been
skipped you can also use --info=skip
.
Getting help
Manuals
The manuals provide a built-in source of information. A nice feature of the
man
command is that it also allows you to search for a function based on the
words in the description with man -k <command>
and you can print just a short
description of a command with man -f <command>
.
Log files
Sometimes you can find error messages in the following log file. Note that this
is in /var
because the file has a variable size as it is logged to.
less /var/log/syslog
Start up and configuration
To avoid confusion between the .bashrc
and .bash_profile
you can just add
the following to .bash_profile
and then you'll have the same script evaluated
regardless.
if [ -f $HOME/.bashrc ]; then source $HOME/.bashrc fi
By default, tab completion is case sensitive, to fix this add the following line
to the file /etc/inputrc
.
set completion-ignore-case on
Accounts
Adding new users
An example command to add a user is the following:
sudo useradd -c "Jerry Smith" -d /home/jsmith -m -s /bin/bash jsmith
-c
is to create a comment which is used for the users full name.-d
is their home directory.-m
says make the home directory if it does not exist.-s
to specify what shell they should use.jsmith
is the username (which should be alphanumeric and must start with a lowercase letter).
Setting a user's password
- Login as root with
sudo -i
. - Set the password for
jsmith
with the followingsudo passwd jsmith
. - Logout as root with
exit
. - Check that
jsmith
can log in withsu - jsmith
.
Setting a user's groups
- Check which groups
jsmith
is part of withgroups jsmith
. - Add
jsmith
to thefleeb
group withusermod -aG fleeb jsmith
. - Remove
jsmith
from thefleeb
group withgpasswd -d jsmith fleeb
.
Removing a user
As root run userdel -r jsmith
to remove jsmith
and their home directory.
Bluetooth
If you find that audio comes out stuttered, try restarting Bluetooth with the following command:
sudo service bluetooth restart
You may have to repeat this process a few times, but it seems to work
eventually. Alternatively, you may need to add the following line to
/etc/modprobe.d/alsa-base.conf
before restarting the service for this to work:
options snd-hda-intel model=generic
Colour selection and hexcodes
The program Gpick displays the hexcode of the pixel under the cursor.
Nautilus - A file manager for GNOME
To open nautilus without the desktop fuss use nautilus --no-desktop
Using nautilus to browse files on a remote machine
- Open nautilus
- "Other Locations" > "Connect to Server"
- SSH not FTP
Packages
Removing a PPA
- Open Software & Updates.
- Navigate to the Other Software tab.
- Select and Remove the desired PPAs.
Installingfrom a .deb
file
Note that the <path/to.deb>
needs to actually be a path, e.g. ./foo.deb
not
foo.deb
.
sudo apt install <path/to.deb>
Uninstalling packages
remove package with remove
sudo apt remove <package-name>
remove package configuration with purge
sudo apt purge <package-name>
AppImage
This is a new way to distribute packages: download the file, make the file exectuble, and run it. Done! If you are concerned about what the program might do, there are sandboxes available, for example Firejail. Inkscape is available as an appimage.
youtube-dl
There are instructions on how to install this program here. Use youtube-dl
--help
to get help on usage.
Download audio as MP3
youtube-dl --extract-audio --audio-format "mp3" <VIDEO/URL>
TODO Browsers
TODO Firefox
- There is an interesting explainer on the Firefox Reader View.
- There are profiles as a way to toggle different configurations of Firefox.
Scripting
On my machine at uni this is /bin/sh
which points to /bin/dash
.
There is a commandline tool shellcheck
which performs static analysis on a
shell script and provides advice to improve it.
Bioinformatics one-liners
Extract the trees from a NEXUS file and put them in a Newick file
This one-liner carries out the following pipeline:
grep
the lines of the nexus file containing trees- Take the first 5000 of them
- Use
awk
to remove the text on the line before the tree - Write the results to file.
grep "^tree STATE_" your_file.nexus | head -n 5000 | awk -F"= " '{print $2}' > first_5000_trees.nwk
Control flow
To run comands conditional upon the presence or absence of a directory, consider the following example
if [ -d "foobar" ]; then echo "There is a directory called foobar" exit 0 fi if [ ! -d "foobar" ]; then echo "Could not find directory: foobar" exit 1 fi
Running things in parallel by backgrounding
Example 1
This solution came from the very clever Claire Miller for how one can do
map-reduce style computations correctly via bash. Note that this actually needs
bash
, it does not work with sh
alone.
# waiter.sh echo "Starting" pids=() for i in $(seq 1 5) do sleep 4 && echo "Done part $i" & pids+=($!) done wait "${pids[@]}" echo "Finished :)"
which should produce the following output.
$ bash waiter.sh Starting Done part 3 Done part 1 Done part 2 Finished :)
Example 2
A slightly more complicated example demonstrating how to chunk processes into subsets to run in parallel before moving onto the next chunk.
#!/usr/bin/env bash chunk_size=5 # the number of processes in each chunk limit=20 # the total number of processes ix=0 outer_limit="$(($limit-$chunk_size))" while [ $ix -le $outer_limit ] do pids=() chunk_limit="$(($ix+$chunk_size))" while [ $ix -lt $chunk_limit ] do padded_num=$(printf "%03d" $ix) echo $padded_num && sleep 1s & pids+=($!) ((ix++)) done wait "${pids[@]}" echo "---" done
which produces
$ ./demo.sh 000 001 002 003 004 --- 005 006 007 008 009 --- 010 011 012 013 014 --- 015 016 017 018 019 ---
Arguments
As an example, here is a simple shell script
#!/bin/sh # demo.sh echo "foo: $2" echo "bar: $1"
Running it we get the following
$ ./demo.sh hello world foo: world bar: hello
Functions
Converting ODG to PDF
Here is an example of a function which converts an ODG to a PDF
ConvertODG2PDF () { libreoffice --headless --convert-to pdf $1 } ConvertODG2PDF awesome-figure.odg
Compiling and converting a standalone latex equation
compileStandAlone () { pdflatex $1.tex convert -flatten -density 300 $1.pdf -colorspace RGB $1.png rm $1.aux rm $1.log rm $1.pdf }
Loops
Here is a silly little example of how to loop over an explicity list of strings.
for month in jan feb mar; do cat foo-$month.csv done
Alternatively, one can construct the list programmatically for example with for
x in *
which would be used to iterate over all files in the current directory
or for x in $*
which would iterate over all the command line arguments.
Here is another example of iterating over some integers and print 1, 2, 3 to standard out.
for ix in `seq 1 3`; do echo $ix done
Quiptic compiler
The following helps you to quickly download a bunch of crosswords and compile
them into a single PDF. We use the sleep
command to be kind to the Guardian
servers.
for i in $(seq 1140 1165) do sleep 2 # be nice! google-chrome --headless --disable-gpu --print-to-pdf https://www.theguardian.com/crosswords/quiptic/$i/print mv output.pdf quiptic-$i.pdf done echo "===========================================================" echo "" echo "Guardian - Quiptics" echo "-------------------" echo "" echo "Use the following command to compile the quiptics into a" echo "single pdf after you have removed any broken ones by hand:" echo "" echo "pdftk quiptic-* cat output quiptics.pdf" echo "" echo "==========================================================="