Skip to content

Commit

Permalink
Fix markdown lint errors
Browse files Browse the repository at this point in the history
  • Loading branch information
richelbilderbeek committed May 14, 2024
1 parent 3b3662f commit f1991f7
Show file tree
Hide file tree
Showing 130 changed files with 6,858 additions and 6,864 deletions.
7 changes: 7 additions & 0 deletions docs/cluster_guides/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ As PI, you and your academic institution are ultimately responsible for your dat
While UPPMAX systems may have backup, these are not designed to act as the sole repository of primary data, e.g. raw data or originals.

Check failure on line 7 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 135]

## What does "backup" mean for my data?

The type of backup that is generally available for project storage at UPPMAX is incremental backup with 30 day retention. This means that any file that was deleted more than 30 days ago is irretrievably gone. Changes in a file are kept for 30 days, so we can potentially retrieve an old version up to a month after you edited it.

Check failure on line 11 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 329]

The backup service tries to backup all changes as often as they occur, but rapid changes will not register. Due to the large amounts of files in the file systems, a single backup session may take upwards of a week or more. This means that if you create a file and delete it the next day, it will probably not be backed up.

Check failure on line 13 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 322]
Expand All @@ -18,6 +19,7 @@ To ensure timely backups, it is very important to reduce the workload of the bac
- It is especially important that temporary files and files that are changed often are placed in nobackup directories.

Check failure on line 19 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Unordered list style [Expected: asterisk; Actual: dash]

Check failure on line 19 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Line length [Expected: 80; Actual: 118]

## Which directories are backed up?

Backup is done on:

- Home directories (on Rackham these also have snapshots)

Check failure on line 25 in docs/cluster_guides/backup.md

View workflow job for this annotation

GitHub Actions / check_markdown

Unordered list style [Expected: asterisk; Actual: dash]
Expand All @@ -28,23 +30,28 @@ Backup is done on:
- SNIC projects (named snicYYYY-X-ZZZZ)

# What should I put in directories with backup?

- In short, irreplaceable data should be placed there. This includes especially raw sequencing data and any other data that cannot be recreated by any effort. Scripts and other files that are needed to reproduce or repeat the analyses should also be placed on backup.

## What should I not put in directories with backup?

- Directories where you are actively working, especially if you are creating or modifying many files.
The backup mechanisms cannot keep up with large amounts of files changing on a rapid basis.

## How robust is uppmax storage?

- All UPPMAX storage systems use RAID technology to make storage more robust through redundancy.
- This means that two or more disks must fail in the same "RAID volume" before there is a risk of data loss.

- However, this technology does not protect against user error (e.g. "rm -rf * in your project directory) or in case of a significant disaster (e.g. fire in computer hall).
- Off-site backup is crucial.

## How can I access my backups?

You must contact UPPMAX support and ask for help. Provide as much information as possible, especially directory and file names.

## What about "snapshots"?

- In addition to the regular backup service, the home directories on Rackham have a feature called "snapshots".
- Snapshot makes a frozen "picture" of some file structure as it looks at the time the snapshot was taken.
- This allows you to restore a particular file as it was at some time point.
Expand Down
10 changes: 5 additions & 5 deletions docs/cluster_guides/bianca.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ suitable for working with sensitive data.
- [Submitting jobs, using Slurm](slurm.md)
- [Starting an interactive node](start_interactive_node_on_bianca.md)
- [File transfer](transfer_bianca.md)
- [File transfer using a graphical program](bianca_file_transfer_using_gui.md)
- [File transfer using a graphical program](bianca_file_transfer_using_gui.md)
- [The module system](bianca_modules.md)
- [IDEs](ides_on_bianca.md)
- [Jupyter](../software/jupyter.md)
- [RStudio](rstudio_on_bianca.md)
- :no_entry: [VSCode](vscode_on_bianca.md)
- [VSCodium](vscodium_on_bianca.md)
- [Jupyter](../software/jupyter.md)
- [RStudio](rstudio_on_bianca.md)
- :no_entry: [VSCode](vscode_on_bianca.md)
- [VSCodium](vscodium_on_bianca.md)
- [Workshops and courses](../workshops_courses/workshops_courses.md)
- NAISS-sens
- Best practices
Expand Down
8 changes: 4 additions & 4 deletions docs/cluster_guides/bianca_file_transfer_using_filezilla.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,17 +49,17 @@ In the 'Site Manager' dialog, click 'New site'

![](./img/filezilla_site_manager.png)

## 5. Create a name for the site, e.g. `bianca-sens123456`.
## 5. Create a name for the site, e.g. `bianca-sens123456`

In the 'New Site' dialog, create a name for the site, e.g. `bianca-sens123456`.

## 6. Configure site

In the 'New Site' dialog, use all standards, except:

* Set protocol to 'SFTP - SSH File Transfer Protocol'
* Set host to `bianca-sftp.uppmax.uu.se`
* Set user to `[username]-[project]`, e.g. `richel-sens123456`
* Set protocol to 'SFTP - SSH File Transfer Protocol'
* Set host to `bianca-sftp.uppmax.uu.se`
* Set user to `[username]-[project]`, e.g. `richel-sens123456`

???- tip "How does that look like?"

Expand Down
4 changes: 2 additions & 2 deletions docs/cluster_guides/bianca_file_transfer_using_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ Here, we show how to transfer files using a graphical tool called FileZilla.

In general, one can pick any graphical tools with these constraints:

* the tool must support SFTP
* the tool must not store a password
* the tool must support SFTP
* the tool must not store a password

Whatever tool one picks, it must do secure file transfer.
For secure file transfer, Bianca supports the SFTP protocol.
Expand Down
7 changes: 4 additions & 3 deletions docs/cluster_guides/bianca_file_transfer_using_winscp.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ Start WinSCP.
In WinSCP, click on 'Create new site'.

For that site, use all standards, except:
* Set file protocol to 'SFTP'
* Set host name to `bianca-sftp.uppmax.uu.se`
* Set user name to `[username]-[project]`, e.g. `richel-sens123456`

* Set file protocol to 'SFTP'
* Set host name to `bianca-sftp.uppmax.uu.se`
* Set user name to `[username]-[project]`, e.g. `richel-sens123456`

???- tip "How does that look like?"

Expand Down
1 change: 0 additions & 1 deletion docs/cluster_guides/bianca_modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,3 @@ do their work independently and undisturbed.

Because this is the same for nearly all UPPMAX clusters,
there is a general page on modules [here](modules.md)

43 changes: 21 additions & 22 deletions docs/cluster_guides/compiling_parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ This is a short tutorial about how to use the queuing system, and how to compile

For serial programs, see a short version of this page at [Compiling source code](compiling_serial.md).

## Compiling and running parallel programs on UPPMAX clusters.
## Compiling and running parallel programs on UPPMAX clusters

### Introduction

These notes show by brief examples how to compile and run serial and parallel programs on the clusters at UPPMAX.

All programs are of the trivial "hello, world" type. The point is to demonstrate how to compile and execute the programs, not how to write parallel programs!


## Running serial programs on execution nodes

Jobs are submitted to execution nodes through the resource manager.
Expand Down Expand Up @@ -57,7 +57,7 @@ The last line in the script is the command used to start the program.
Submit the job to the batch queue:

```console
$ sbatch hello.sh
sbatch hello.sh
```

The program's output to stdout is saved in the file named at the -o flag.
Expand Down Expand Up @@ -98,7 +98,6 @@ Before compiling a program for MPI we must choose, in addition to the compiler,

Check this compatibility page for a more complete picture of compatible versions.


### C programs

Enter the following mpi program in c and save in the file hello.c
Expand All @@ -123,27 +122,27 @@ Before compiling a program for MPI we must choose which version of MPI. At UPPMA
To load the openmpi module, enter the command below or choose other versions according to the lists above.
```console
$ module load gcc/10.3.0 openmpi/3.1.6
module load gcc/10.3.0 openmpi/3.1.6
```

To check that the openmpi modules is loaded, use the command:

```console
$ module list
module list
```

The command to compile a c program for mpi is mpicc. Which compiler is used when this command is issued depends on what compiler module was loaded before openmpi

To compile, enter the command:

```console
$ mpicc -o hello-mpi hello-mpi.c
mpicc -o hello-mpi hello-mpi.c
```

You should add optimization and other flags to the mpicc command, just as you would to the compiler used. So if the gcc compiler is used and you wish to compile an mpi program written in C with good, fast optimization you should use a command similar to the following:

```console
$ mpicc -fast -o hello-mpi hello-mpi.c
mpicc -fast -o hello-mpi hello-mpi.c
```

To run the mpi program hello using the batch system, we make a batch script with name ``hello-mpi.sh``
Expand Down Expand Up @@ -171,7 +170,7 @@ The last word on the last line is the program name hello.
Submit the job to the batch queue:

```console
$ sbatch hello-mpi.sh
sbatch hello-mpi.sh
```

The program's output to stdout is saved in the file named at the -o flag.
Expand Down Expand Up @@ -233,8 +232,8 @@ end program testampi
The program can be compiled by this procedure, using mpif90:

```console
$ module load intel/20.4 openmpi/3.1.6
$ mpif90 -Ofast -o testampi testampi.f90
module load intel/20.4 openmpi/3.1.6
mpif90 -Ofast -o testampi testampi.f90
```

The program can be run by creating a submit script sub.sh:
Expand Down Expand Up @@ -309,13 +308,13 @@ OpenMP uses threads that use shared memory. OpenMP is supported by both the gcc
Depending on your preferences load the chosen compiler:

```console
$ module load gcc/12.1.0
module load gcc/12.1.0
```

or

```console
$ module load intel/20.4
module load intel/20.4
```

### C programs
Expand All @@ -342,13 +341,13 @@ int main()
To compile, enter the command (note the -fopenmp or -qopenmp flag depending on compiler):

```console
$ gcc -fopenmp -o hello_omp hello_omp.c
gcc -fopenmp -o hello_omp hello_omp.c
```

or

```console
$ icc qfopenmp -o hello_omp hello_omp.c
icc qfopenmp -o hello_omp hello_omp.c
```

Also here you should add optimization flags such as -fast as appropriate.
Expand Down Expand Up @@ -382,7 +381,7 @@ The last line in the script is the command used to start the program.
Submit the job to the batch queue:

```console
$ sbatch hello.sh
sbatch hello.sh
```

The program's output to stdout is saved in the file named at the -o flag.
Expand Down Expand Up @@ -427,13 +426,13 @@ END
With gcc compiler:

```console
$ gfortran hello_omp.f90 -o hello_omp -fopenmp
gfortran hello_omp.f90 -o hello_omp -fopenmp
```

and with Intel compiler:

```console
$ ifort hello_omp.f90 -o hello_omp -qopenmp
ifort hello_omp.f90 -o hello_omp -qopenmp
```

Run with:
Expand All @@ -448,7 +447,6 @@ $ ./hello_omp
Number of threads = 4
```


A batch file would look similar to the C version, above.

## Pthreads
Expand Down Expand Up @@ -493,8 +491,8 @@ int main()
To compile, enter the commands
```console
$ module load gcc/10.2.0
$ gcc -pthread -o hello_pthread hello_pthread.c
module load gcc/10.2.0
gcc -pthread -o hello_pthread hello_pthread.c
```

To run the pthread program hello using the batch system, enter the following shell script in the file hello.sh:
Expand All @@ -521,8 +519,9 @@ The last line in the script is the command used to start the program.
Submit the job to the batch queue:

```console
$ sbatch hello.sh
sbatch hello.sh
```

The program's output to stdout is saved in the file named at the -o flag.
A test run of the above program yelds the following output file:

Expand Down
Loading

0 comments on commit f1991f7

Please sign in to comment.