Linux
=====
###### tags: `chapter`, `IT`, `Linux`, `by-wasmer`, `public`, `from-2020-10`, `evolving`
[up <i class="fa fa-arrow-up"></i> IT](https://iffmd.fz-juelich.de/WxOZ75GTTHu2MNmnw2u5pA#IT)
# Sections
[TOC]
# General Linux
## Shell
- [explainshell.com](https://explainshell.com): copy-paste a shell command and get the detailed explanation.
## Shell startup files, config files
This assumes bash as shell.
For the following to make sense, one first must learn about shell types. There are four types: non-/interactive and non-/login shell. Here are some examples:
- *interactive non-login* shell: opening a graphical console (ie from desktop environment) on your own system.
- *interactive login* shell:
- Switching with `Ctrl+Alt+Fx` to one of the virtual consoles without GUI.
- Connecting to another computer via ssh.
- *non-interactive* shells: Programs usually run in *non-interactive* shells.
Some references:
- [SE - difference between interactive shells, login shells, non-login shell](https://unix.stackexchange.com/a/50667)
- [SE - difference between interactive shells, login shells, non-login shell 2](https://unix.stackexchange.com/a/46856)
- [SE - commands to check your current shell type](https://unix.stackexchange.com/a/26782)
Now, about shell (bash) startup files.
[Bash manual > Bash Startup Files](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html):
- When Bash is invoked as *interactive non-login* shell, the **bash rc file** `~/.bashrc` is read ('rc' stands for 'run commands', a common filename suffix in Unix for startup config files).
- When Bash is invoked as *interactive login* shell, it reads these files in order, if they exist:
- **system profile files**
- `/etc/environment`: system-wide env vars. set as `FOO=bar`.
- `/etc/profile`: loaded when a bash *login shell* is invoked. set variables with `export` command.
- **user profile file**. bash looks for `~/.bash_profile`, `~/.bash_login`, and `~/.profile`. The first found **in that order** is read, the subsequent **are ignored**. Also, the `~/.bashrc` is ignored by default. However, you can extend the startup sequence by adding any additional file that you want to be read in your first user profile file. For example, the sequence (`~/.bash_profile`, `~/.profile`, `~/.bashrc`) can be accomplished by appending these lines to `~/.bash_profile`: Note that the command `. ~/.bashrc` [is equivalent](https://stackoverflow.com/a/2518150/8116031) to `source ~/.bashrc`.
```shell
if [ -f ~/.profile ]; then . ~/.profile; fi
if [ -f ~/.bashrc ]; then . ~/.bashrc; fi
```
- On logout: `~/.bash_logout`.
**What should I put in which startup file?**
- The bash rc file is for anything you'd want in an *interactive* shell. Command prompt, EDITOR variable, bash aliases, etc. This file must not output anything.
- User profile files hold the stuff NOT specifically related to bash, such as environment variables (`PATH` and friends). Also anything that should be available to graphical applications OR to `sh` (for running `.sh` scripts).
Some setup suggestions:
- To load / update from files manually, call e.g. `source ~/.bash_login ; source ~/.bashrc`.
- [SE - load `~/.profile` and `~/.bashrc` via `~/.bash_profile`](https://superuser.com/a/789465)
- [SE - read bashrc as well in login shells automatically](https://stackoverflow.com/a/415444/8116031)
- [SE - historical origin for the different user profile file names](https://unix.stackexchange.com/a/624051)
- another possibility is to use shell functions to isolate interactive login shell commands, and call them when needed.
For wasmer's setup see [section wasmer config files](https://iffmd.fz-juelich.de/Jot20oPCQ_it2m1j25gz6A?both#wasmer-config-files).
References:
- [linuxize.com > environment variables](https://linuxize.com/post/how-to-set-and-list-environment-variables-in-linux/)
- [ubuntu.com > environment variables](https://help.ubuntu.com/community/EnvironmentVariables)
- [cyberciti.biz > Linux environment variables](https://www.cyberciti.biz/faq/set-environment-variable-linux/)
- [serverlab.ca > Linux environment variables](https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-set-environment-variables-in-linux/)
## Handling archives
Compress files into tarball:
```
tar -zcvf archive.tar.gz file.txt folder1 folder2
```
Unpack tarball `tar.gz`, `tar.bz2`, ... file:
```
tar xf archive.tar.gz
```
Options:
- `-x`: extract
- `-f`: file
## Setting environment variables
Setting a variable:
- `FOO=bar` sets a *shell* variable = *local* variable. New shells / child processes won't inherit.
- `export FOO=bar` sets an *environment* variable = *global* variable. New shells / child processes will inherit. In older shells this might have to be split into two commands: `FOO=bar; export FOO`. See a list of all exported variables with `export -p`.
- Note that no space allowed between key and value assignment.
- `readonly PI=3.14; export PI` sets `PI` as read-only env var.
Variables and quotes:
- If the value you're assigning to the variable doesn't contain any characters that are special to the shell, you don't need any quotes. In the above example, `FOO=bar`, `FOO='bar' and FOO="bar"` are identical.
- Double-quotes group substrings, but allows whatever shell you use to do variable substitution.
- Single-quoting groups substrings and prevents substitution.
Printing a variable value:
- `printenv FOO`
- `echo $FOO` or `echo "$FOO"`
Printing a list of variables:
- `printenv` / `env` lists all env variables
- `set` lists all env, shell, user-defined vars and functions
Unsetting a variable:
- `unset FOO` unsets a shell as well as an environment variable (i.e., unset it *globally* in the latter case)
Persisting an environment variable:
- Per user: Put one of the above commands into one of the shell startup **user profile files** (see [section Shell startup files / config files](https://iffmd.fz-juelich.de/Jot20oPCQ_it2m1j25gz6A?both#Shell-startup-files-config-files)).
- Per system: Put one of the above commands into one of the shell startup **system profile files** (see [section Shell startup files / config files](https://iffmd.fz-juelich.de/Jot20oPCQ_it2m1j25gz6A?both#Shell-startup-files-config-files)). Or better, create a `/etc/profile.d` directory for it (see [here](https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-set-environment-variables-in-linux/)).
Other useful commands:
- bash -c "echo $PI" starts a new bash shell and prints an env var. Good for testing if an env var got inherited.
- `env VAR_NAME=VALUE CMD1 ARG1` runs a Linux command with a modified environment (see [here](https://www.cyberciti.biz/faq/set-environment-variable-linux/)).
# File management
## Filesystems
Intent: list **where** (directories, local and remote), **what** (usage for what service), **how** (usage example scenario / how-to).
### Local PGI-1
| Filesystem | Type | Path | Backup | Quota space | Quota files | Use cases |
| ---------- | ----- | --------------- | ------ | ----------- | ----------- | ---------------- |
| `$HOME` | NFS | `/Users/$USER/` | yes | 50 GB | 2e5 | code |
| | NFS | `/Data/` | yes | ? | ? | data |
| | local | `/Local/` | no | 300 GB | none | data, programs |
| | local | `/opt/` | no | 75 GB | none | programs & cache |
Notes on table:
- local quota determined via `df -h /path`. probably local machine hardware - dependent.
More detailed notes:
* `$HOME` and `/Data` are mounted at the NFS server (NFS = distributed file system), thereby account-bound, available via all Linux machines, and included in backup. `/opt` and `/Local` are on the local hard disk, thereby machine-bound (as are the postgres SQL database with db at `/var`), and not included in backup:
```shell
mountpoint /Data; mountpoint "$HOME"; mountpoint /Local; mountpoint /opt
/Data is a mountpoint
/Users/wasmer is a mountpoint
/Local is a mountpoint
/opt is not a mountpoint
```
* `/Data/ias-1/$USER`: quota: no. backup: yes.
* Usages:
* `aiida` manual backup: if *not* installed on NFS storage (check via `verdi profile show`), save SQL dump & compressed form of file repo.
* `/opt`: quota: no. backup: no.
* `$HOME` = `/User/$USER` home. quota: yes. backup: yes.
* Disk quota: `quota -us`
* `/Local`: quota: no. backup: no.
* Usages:
* user-specific program installations.
### remote
#### [RWTH Cluster](https://iffmd.fz-juelich.de/1dPKGzXWSTScfdTPjBJi0w)
## Copy/move stuff
Copy a large directory, preserve everything:
```shell
rsync --info=progress2 -aHAX /src /dst
```
Result: `/dst/src/`.
Notes:
- `-aHAX` [reference](https://serverfault.com/a/43019), probably more thorough than `-zavh` [rsync copy Example2)](https://www.linuxtechi.com/rsync-command-examples-linux/) (see also here for table of `rsync` options).
- `--info=progress2` [reference](https://www.cyberciti.biz/faq/show-progress-during-file-transfer/) show progress bar during sync, as opposed to list of files `-v --progress`. Problem is, progress bar may is not very accurate (eg can be at 99% 80% of the time for large file count in path).
- check if sync was successful: use 'count files' command (search here 'count files') on `src/` and `dst/` and compare
- to *move* instead of *copy*, add `-remove-source-files`. It will only remove src files successfully synched.
## Disk usage
### How to get basic disk usage info
On your `iffX workstation`
```shell
# show your quota
quota -s
# show disk usage of all mounts.
df -h
# show disk usage of mount where path is located
df -h /mypath
# calc size of some location.
du -sh /mypath
du -hc --max-depth=0 /mypath
# show all subdir sizes sorted
du -sh /mypath/* | sort -hr
# show all subdir sizes sorted include, total
du -sh --max-depth=1 /mypath | sort -hr
# count files/dirs in a directory, simple:
find /mypath -type f | wc -l
# count files/dirs in a directory, detailed:
rsync --stats --dry-run -ax /path/to/dir /tmp
# count files, custom script
python ~/count_files.py --help
```
Notes:
- `df` is the 'disk-free' command.
- `du` is the 'disk-usage' command.
- Common options.
- `-h` = human-readable, eg MB not bytes.
- `-s` = summary. In case of `du`, equivalent to `--max-depth=0`.
- [reference](https://stackoverflow.com/a/34941137/8116031) for the 'count files' command (better than [these standard suggestions](https://www.computerhope.com/unix/tree.htm)) (see [here](https://iffmd.fz-juelich.de/6Bhk38pRTHao3FkGiK81YQ?both#Update-Dec2020-another-condaiff-conflict) for an example & evaluation of different file counting methods).
- The custom `count_files.py` mentioned above is [here](https://iffgit.fz-juelich.de/wasmer/home/-/blob/2b46b9e0a5c969859a6f52c0af73b7ea9c4859e2/count_files.py).
### How to resolve a filecount quota conflict
On `iff` workstations, the `quota -s` filecount quota on `$HOME` was 200k as of 2021-09. Some tools can make your surpass this limit pretty quickly.
See also [How to solve filecount quota conflict: Log](https://iffmd.fz-juelich.de/e4pn-y4mQDGpL9GeFINl7A#How-to-solve-filecount-quota-conflict-Log).
General solutions:
- Install new tools not in `$HOME`, but [somewhere else](https://iffmd.fz-juelich.de/Jot20oPCQ_it2m1j25gz6A?both#Filesystems): `/opt`, `/Local` or `/Data`.
- If those tools create config/cache in `$HOME`, nevertheless, try to move that permanently out of `$HOME` as well. Example see below.
- If config file locations of a tool cannot be moved out of `$HOME`, consider moving it to another location and creating a symbolic link (`ln -s`) at its original `$HOME` path pointing to that external locatio. Example see below.
- If cache file locations of a tool cannot be moved out of `$HOME`, periodically run a manual filecounter ([example](https://iffgit.fz-juelich.de/wasmer/home/-/blob/master/count_files.py)) to find the problem spots and remove the cache files if possible.
#### Example: move stuff out of `$HOME`: `conda`
How to resolve a filecount quota conflict example. Move stuff out of `$HOME`: `conda`. Date 2022.
The tools `conda` and `yarn` can cause the problem that they make you hit the current filecount quota 200k pretty quickly. The culprits are, in the standard installation (by admin):
```
~/.conda/envs/
~/.conda/pkgs/
~/.cache/yarn/
```
The first two can quickly jump to 200k files, the latter quickly to 10ks of files. The latter grows even if `yarn` is not installed, by e.g. npm packages installed via jupyter.
The solution in all three cases is to move these folders to either `/opt` or `/Local`, and tell `conda` resp. `yarn` about it via the config files `~/.condarc` or `~/.yarnrc`. See [this example](https://iffgit.fz-juelich.de/wasmer/home) for the corresponding lines.
Then [move the directories](https://iffmd.fz-juelich.de/Jot20oPCQ_it2m1j25gz6A?both#Copymove-stuff) to their specified new location.
conda and jupyter (latter perhaps after restart) should work, and new packages should be installed/cached in the new locations, the old ones should stay empty.
#### Example: move stuff out of `$HOME`: `vscode-server`
How to resolve a filecount quota conflict example. Move stuff out of `$HOME`: `vscode-server`. Date 2022.
for remote work, remember that vscode remote lets the theme/ui extensions run
locally, and the workspace extensions remotely on the remote vscode server. so
if one wants to use locally installed workspace extensions for remote
development, one must install them on the remote. when remote development is
active, the extensions window is separated into local and remote, with the local
workspace extensions deactivated and a button to install them on the remote.
There, they get installed in the =~/.vscode-server/=, or the
=~/.vscode-insiders-server/= folder under =/extensions/=. On =iff734=, I moved
these folders to =/Local= to bypass the $HOME quota, and then symlinked them
back into their original location, like this:
```shell
# for vscode
mkdir -p /Local/wasmer/.vscode-server/extensions
rsync -az --delete /Users/wasmer/.vscode-server/extensions/ /Local/wasmer/.vscode-server/extensions
rm -rf ~/.vscode-server/extensions/
ln -s /Local/wasmer/.vscode-server/extensions ~/.vscode-server/extensions
# for vscode-insiders
mkdir -p /Local/wasmer/.vscode-server-insiders/extensions
rsync -az --delete /Users/wasmer/.vscode-server-insiders/extensions/ /Local/wasmer/.vscode-server-insiders/extensions
rm -rf ~/.vscode-server-insiders/extensions/
ln -s /Local/wasmer/.vscode-server-insiders/extensions ~/.vscode-server-insiders/extensions
```
That worked. vscode doesn't complain that this folder is a symlink, treats it
like a normal folder.
# Networking
Moved to [Networking](https://iffmd.fz-juelich.de/-sZTHayUSFaI7dPdbwEuPg).
# How to use `screen` sessions
## Basics
`screen` is a small tool which allows you to run processes in a separate shell environment which you can detach from your normal shell. This is useful for long-running processes, like long calculations or SSH connections which should not be interrupted. A more powerful alternative is `tmux`, but for simple stuff `screen` does fine.
References:
- [linuxize.com - How To Use Linux Screen](https://linuxize.com/post/how-to-use-linux-screen/)
Shortkey notation:
- C-a = Ctrl+a
- S-x = Shift+x
- M-x = Alt+x
- Note: C-a is the 'leader key' for in-screen commands, like e.g. SPC in Spacemacs. Almost all commands work with holding Ctrl down or releasing it equally. E.g. `C-a d` == `C-a C-d`.
Starting sessions.
```shell
# start unnamed screen session
$ screen
# start named screen session
$ screen -S session_name
```
Deattach/reattach to sessions:
- detach: C-a C-d.
```shell
# reattach, one session running:
$ screen -r
# reattach, multiple sessions running:
# first display all session IDs:
$ screen -ls
# then, reattach:
$ screen -r 10835
```
Window management: inside a session, windows can be created.
- `C-a c` create window
- `C-a k` kill window
- `C-a "` list windows
- `C-a [0-9]` switch to numbered window
## Kill screen windows / sessions
References:
- [screen manual](https://www.gnu.org/software/screen/manual/screen.html)
- https://stackoverflow.com/a/1509764
- https://stackoverflow.com/a/6301852
- https://askubuntu.com/a/356026
Note: several solutions are offered, seems to depend on system which one will work. I will denote below if a command does not work on `iff` workstations.
### Kill screen session windows
Kill current window from inside session:
- `C-a k` (note: killing the last remaining window will exit the entire screen).
Kill a window from outside session:
- `screen -X -S [session # you want to kill] kill`
### Kill screen sessions
Kill current screen session from inside:
- `C-a \`
- `exit`
Kill a screen session from outside (eg when it hangs):
- `screen -X -S [session # you want to kill] quit`
- `screen -wipe`: man: "Remove sessions with the ‘-wipe’ option."
- `screen -X quit`: user: "on OS X, kills all active sessions"
# Linux operating systems
### Get OS info
```bash
# get OS info: CentOS
cat /etc/os-release
# get OS info: Debian/Ubuntu
cat /etc/lsb-release
# get Linux kernel info
uname -a
```
On `iff` workstations, should give you
- "Rocky Linux", as of 2022 autumn.
- "CentOS Linux 8", as of 2021-07.
Graphical alternatives:
- Gnome: Activities > About.
## CentOS operating system
This is the default operating system on `iff` workstations as of 2021-09.
## CentOS package manager `yum`
- `yum list installed | grep sql`: list installed sql-related packages
## Dealing with RPM packages
[Unpack RPM package](https://superuser.com/a/210055) (e.g. for build from source; will unpack into current directory):
```shell
rpm2cpio path/package.rpm | cpio -idv
```
RPM packages can only be installed via `sudo`. Ask an admin for installation if needed.
## CentOS Gnome file manager *Files*
- `C-l` in address bar toggles graphical <--> keyboard editable mode
# Linux desktop environments
## Get desktop environment info
Gnome:
- `echo $DESKTOP_SESSION` or `echo $XDG_CURRENT_DESKTOP
` should give you `gnome`.
- Graphical alternative: Activities > About. Includes version info.
## Gnome desktop environment
This is the default desktop environment on `iff` workstations as of 2021-09.
### Adding applications to the desktop
Applications are programs listed in the desktop **Activities** menu. Sometimes also called app launcher or similar.
Gnome keeps Applications entries, one file per program, in these locations:
```
/usr/share/applications/*.desktop # system
~/.local/share/applications/*.desktop # user
```
Templates for a new application entry can be found [here](https://unix.stackexchange.com/a/70461) and [here](https://unix.stackexchange.com/a/536489).
# How to use `cron` jobs
- [opensource.com > How to use cron on Linux](https://opensource.com/article/21/7/cron-linux). Concise basics intro.
# wasmer's local setup
# wasmer config files
- [[wasmer_home_dir]](https://iffgit.fz-juelich.de/wasmer/home) with `~/.bash*` config files
## wasmer file management
Date: 2020-11-27.
Work:
- `~/src/` (git libs like `jukkr`, `masci-tools` and so on)
- `~/src/wasmer/` my private git projects
- `~/work/`: not (as yet) git-tracked work stuff
- `~/Nextcloud`: meetings notes
AiiDA:
- env/lib: `/opt/conda/envs/aiida-kkr*/`
- profile `wasmer` repo: `~/.aiida/repository/wasmer`
- plugins:
- `~/Desktop/Coding/Python/conda-envs/current/src/masci-tools`: located here cause unfortunately created environment from `aiida_kkr.yml` env file there, so those sources got put there by conda. 'solved' by symlinking `~/src/masci-tools` to there...
- `~/Desktop/Coding/Python/conda-envs/current/src/aiida-kkr`: as above.
- `~/Desktop/Coding/Python/conda-envs/current/src/aiida-jutools`: as above.
User-installed software / outsourced config/cache dirs as of 2021-09:
```bash
$ l2 /opt/
drwxr-xr-x 3 wasmer pgi-1 18 2020-12-20 09:31:59 cache
drwxr-xr-x 4 wasmer pgi-1 30 2020-12-19 20:40:51 conda
drwxr-xr-x 8 wasmer pgi-1 115 2020-10-23 16:58:13 jdk-11.0.8
drwxr-xr-x 9 wasmer pgi-1 170 2021-09-10 13:48:28 pycharm-2021.2.1
$ l2 /Local/wasmer
drwxr-xr-x 3 wasmer pgi-1 4.0K 2020-12-17 08:44:41 .aiida
drwxr-xr-x 3 wasmer pgi-1 4.0K 2021-06-29 05:37:05 src
drwxr-xr-x 3 wasmer pgi-1 4.0K 2021-05-26 12:46:53 tmp
drwxr-xr-x 12 wasmer pgi-1 4.0K 2021-06-21 13:48:17 .vscode-extensions-failed-rsync
drwxr-xr-x 3 wasmer pgi-1 4.0K 2021-06-21 13:41:40 .vscode-server
drwxr-xr-x 3 wasmer pgi-1 4.0K 2021-06-21 13:34:22 .vscode-server-insiders
```
- [emacs27.1](https://superuser.com/a/638016): `~/emacs/`, `~/bin/`. But check if in meantime, `~/bin/emacs` is newer version.
Data:
```bash
l2 /Data/ias-1/wasmer/
drwxr-xr-x 3 wasmer pgi-1 27 2021-01-03 12:28:29 backup
drwxr-xr-x 4 wasmer pgi-1 46 2021-03-14 12:10:14 claix18
drwxr-xr-x 4 wasmer pgi-1 65 2020-12-21 16:35:05 downloaded
drwxr-xr-x 4 wasmer pgi-1 56 2021-03-29 17:03:03 export
drwxr-xr-x 3 wasmer pgi-1 30 2021-01-18 13:49:32 iffslurm
drwxr-xr-x 3 wasmer pgi-1 28 2021-07-15 12:57:35 src
```
- My system/admin installation locations @workstation
- `/usr/local/`: anaconda3, JabRef, jdk1.8.0_201 (OpenJDK), ...