Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
This course has been designed to help users access the research/teaching and Viking Linux servers. The aim is to provide you with some basic Linux commands to get you going.  You can access these machines

...

using Windows, Mac

...

width50

...

...

Have you

...

found any of the following when

...

doing research on your

...

own computer?

Panel
borderColorred
borderWidth3
  • Your workload is taking a long time to run

...

  • (>6hrs)

  • It uses all your

...

  • machine's resources (

...

  • compute cores or memory)

  • You need

...

  • lots of memory

  • You need

...

  • GPUs

  • You are either using or producing a lot of data

  • You think you could cut your job into smaller chunks and process them at the same time
  • You know you want to analyse larger datasets in the future.

We have a few different machines to use when

...

you have these problems: individual large machines known as the research and teaching servers, and the Viking compute cluster, a connected group of hundreds of machines. Here we will give you a very brief introduction on how to access these machines.

The research and teaching servers

These servers are also known as the Linux Managed Service or LMS for short. There are currently four research servers (research0, research1, research2 and research3) and two teaching servers (teaching0 and teaching1). Detailed information on the server specifications can be found here.  These machines are Desktops, similar to what you may have at home or in your office, but with a large number of

...

compute cores and memory.  This means that work that your local machine is struggling with may easily be run on one of these machines.

...

You can log on to these machines from anywhere on campus, or off campus if you use the Virtual Private Network (VPN) or SSH gateway service. Some caveats:

  • They are a shared machine which means a number of users may be logged on at the same time
  • They run Linux so you

...

  • need a little bit of Linux command line knowledge to get started

...

  • If you are

...

  • an undergraduate you will only have access to the teaching servers

...

  •  
  • They get rebooted on the first Tuesday of every month, so any jobs running then will be killed


Exercise 1 - Logging into the research or teaching servers.

There are different ways to login to the LMS depending on what operating system you are running.  We will break down the different options here.

Expand


Section
bordertrue

Before You Login

If you have not changed your IT Services password since August 2013 then you must do so before you will be able to login. All user password changes are manage via the My IT Account  web page. Click on the Password Management (IDM) link in the Manage Your Password field to change your password. You may be given the option to 'synchronise' your password

...

; please use this option if you do not want to change your password. The password change (or synchronisation) may take a few minutes before it is visible to the servers.

Accessing research and teaching servers

...

off

...

campus

To access the research

...

and teaching servers off campus you can either use the Virtual Private Network - VPN or the SSH gateway service (registration required). The instructions below should work if you log on through the VPN; the SSH service works slightly differently.


Section
bordertrue


Column
width50%

Access from a Windows desktop

Command-line access using PuTTY

PuTTY is available on all IT Services Managed Windows systems

...

. It is pre-installed on Classroom PCs

...

; on Office PCs you can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the start menu.

On unmanaged PCs you can download the installer from the PuTTY Website.

When you run PuTTY for the first time, enter the following settings to log on to the research0 server (

...

or replace research0 with teaching0, research1 etc):

  1. Add the name "research0.york.ac.uk" to the 'Host Name' field
  2. Check the 'Connection Type' to SSH
  3. Type the name "research0"  in 'Saved Sessions'
  4. Click 'Save'


Column
width50%

Image Modified




Section
bordertrue


Column
width50%
  1. Expand the 'SSH' tab from the 'Connection' list in the 'Category'

...

  1. box
  2. Choose 'X11' from 'SSH' list
  3. Check 'Enable X11 Forwarding'


Column
width50%

Image Modified




Section
bordertrue


Column
width50%
  1. Choose 'Session' from the 'Category'

...

  1. box
  2. Click 'Save'


Column
width50%

Image Modified



Connecting to

...

research0
  1. Start PuTTY
  2. Select 'research0' from the 'Saved Sessions'
  3. Click 'Open'
  4. The first time you connect you will get a security alert showing the fingerprint of the server, labeled as 'ssh-rsa' or 'ssh-ed25519'. Check the fingerprint below for the label shown and click "Yes" to proceed if it matches.


Code Block
languagebash
ssh-ed25519 255 SHA256:J4Bqti3rhC4qhnU0YLPX7jScNOlm+rXcHUBzl9AMqNE research0
ssh-ed25519 255 SHA256:lRu6KP3DviVUg1fxr39b1bzCRv5lsIlZrORA6qjVRTM research1
ssh-ed25519 255 SHA256:cMHW+V9JGelv0/16Add02UqgHcq209ATn2AdFIG2DR4 research2
ssh-ed25519 255 SHA256:nOrLWS02W5Dk8SsspFutMd/z1Jh2mq7rzo27vyYwl4w research3
ssh-ed25519 255 SHA256:wKe0Eohia/DfsPmYcg8R/huSpldKlgSQFSAdEaGJ7zg teaching0
ssh-ed25519 255 SHA256:lpdXExChAcX3xaEkknD8YBIGz1cQaFKQmAcJ710DYaI teaching1

ssh-rsa 2048 5c:43:e5:e6:57:e0:4d:9f:f8:b5:ca:52:2f:30:39:

...

ef research0

...


ssh-rsa 2048 bb:1f:6e:58:fa:d7:23:0d:ae:b2:b2:e8:62:a0:e7:5c 

...

research1
ssh-rsa 2048 13:42:77:57:ad:33:67:12:a4:8f:d3:26:24:37:2c:e0

...

 research2

...


ssh-rsa 2048 99:da:16:61:09:e0:19:1f:53:0e:2b:e9:2a:22:50:99 

...

research3
ssh-rsa 2048 ac:7e:1e:2a:05:d8:a1:3b:cf:b5:77:48:d0:bb:8a:

...

22 teaching0

...


ssh-rsa 2048 bb:de:33:ff:07:23:6b:0a:73:ad:1f:8a:57:b2:c7:77

...

 teaching1

...



ssh-ed25519

...

 5a:c9:c1:76:16:00:42:45:f9:e5:bd:63:5d:87:db:8a 

...

research0
ssh-ed25519

...

 f4:51:59:b7:b5:74:1c:14:a7:2c:78:1c:11:1f:72:cc 

...

research1
ssh-ed25519

...

 76:93:b3:2d:e0:73:cb:25:a5:9e:c7:bd:ce:76:8f:a2 

...

research2
ssh-ed25519 

...

64:b2:f0:11:93:fc:d7:ea:12:bb:90:bc:7d:06:75:cc 

...

research3
ssh-ed25519 

...

f4:ff:10:59:1e:5f:21:10:14:59:6e:04:16:63:7d:95

...

 teaching0

...


ssh-ed25519 

...

bc:73:f8:4f:40:7a:6b:41:73:90:f6:77:2b:4f:aa:d5

...

 teaching1

Image Modified

Section

Graphical login using x2go

Note
The previous methods using NX Client and eXceed are now deprecated.


Section
bordertrue


Column
width50%
x2go client is available on all IT Services Managed Windows systems

...

. It is pre-installed on Classroom PCs

...

; users on Office PCs can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the

...

Start Menu and there are pre-defined menu entries for research0 and teaching0.


Column
width50%
Image Modified






Section
bordertrue


Column
width50%

On unmanaged PCs you can download the installer from the x2go Website. You will need to configure the settings as follows

...

for research0 (or replace research0 with the name of the server you want to use):


  1. If the "New session" panel does not appear, select the menu item "Session | New session..."
  2. Enter "research0" in the 'Session name' field at the top
  3. Enter "research0.york.ac.uk" to the 'Server: Host

...

  1. :' field
  2. Change the 'Session Type' to XFCE

...

  1. Click "OK" (all other settings can stay on default).

Image Removed



Column
width50%
Image Added





Section
bordertrue


Column
width50%

If you wish to change any settings:


  1. Cancel any login dialogs
  2. Click the pull-down on the corner of the panel and select "Session preferences..."


Column
width50%
Image Modified



Section
bordertrue

Access from a Mac

Go to

...

Panel
titleLogging into research0

$ ssh research0.york.ac.uk

...

the Finder on your Mac, find Applications, open the Utilities folder in Applications and then start the Terminal app from the Utilities folder. (You may wish to add Terminal to your Dock.) Then type the following, using your university username (abc123) instead ofusername. You don't need to type the $; this is an example of a prompt, which tells us the terminal is ready for us to type something. You should see something similar when you open Terminal, though it may be a bit longer (it may show your username for example). Just type anything from after the $.

Panel
titleLogging into research0

$ ssh username@research0.york.ac.uk

For a graphical login, install and configure the x2go client software as described in the unmanaged Windows section above.

...

teaching0 and the other servers can be accessed in the same manner.

Access from a UNIX server or desktop

To login from a terminal window, type the following from your local

...

machine with your university username (abc123):

...

Panel
titleLogging into research0

...

$ ssh username@research0.york.ac.uk

 If you require X forwarding, type:

Panel
titleLogging research0 with X forwarding

$ ssh -X username@research0.york.ac.uk

You will be prompted for your IT Services password.

Accessing teaching0 is done in the same manner.

...




Using the research/teaching servers with the Linux command line

...

interface

Once you have successfully logged into the

...

research or teaching servers, they may look very different to what you are

...

used to, particularly if you are

...

used to using

...

Windows.

...

Please do not let this put you off.

...

The research computing team have successfully managed to help many people use these computers who have never used the Linux command line before.

...

It takes a bit of getting

...

used to but the more you use it the easier and quicker it will become over time.

The command line, or shell, has been the major interface for the Unix/Linux operating system since it was first conceived in the late 1960s. The shell allows interaction with the operating system through a text based interface

...

  • An easy to use command line interface (CLI)
  • Combine existing tools to create new tools
  • Input/output redirection
  • Wildcard characters for filename abbreviation
  • Variables and options for environment customisation
  • Ability to write shell functions and programs
  • Command-line editing
  • Command history (quick access to previous commands)
  • Arithmetic
  • Command abbreviations

...

, rather than the graphical interface you are used to. While the graphical interface is easy to learn, and usually makes simple things easy to do, it can be hard to do complex things like operate on large numbers of files, or make different tools work together. The shell can be hard to learn, but is much more powerful and flexible than most graphical interfaces, so can be very useful for research, where we often want to try new things on large data sets.

In this tutorial, we will only scratch the surface of the shell's features, just to get you started, but we will note some further features at the end of the tutorial that you may want to look into.

The user starts the shell by logging into the computer with a userid and password:


Code Block
languagebash
titleLogging in
******************************************************************************
***                   THE UNIVERSITY OF YORK IT SERVICES                   ***
***                                                                        ***
***                       THIS IS A PRIVATE COMPUTER                       ***
***                UNAUTHORISED ACCESS STRICTLY PROHIBITED                 ***
******************************************************************************

login: user001
password: 
Last login: Mon Sep  8 14:12:44 2014 from gallifrey.york.ac.uk
-bash-4.1$


The last line is a command prompt and it is the means by which the computer is telling you that it is ready to accept a command from you. If you do not see the prompt, the computer is probability still executing the last command you have typed. The user types commands which take the form:

...


program [ options ] [ arguments ]

...


Roughly speaking, program is the name of the program we want to run, arguments are objects we want to process (typically data files or folders), and options modify how the program will run. Options to a command are usually proceeded by a '-' or '- -'

...

to differentiate them from

...

arguments. The following

...

exercise demonstrates using the echo program with a series of arguments and the ls program with or without options.

Exercise 2 - Running commands in the Linux shell


Expand

When you see the prompt type the following command

...

(you can also copy and paste the command into your terminal).

Code Block
languagebash
echo I love York

What happens?  The terminal should write out "

...

I love York". 

...

echo

...

is a small program that takes

...

a series of

...

arguments and repeats them to the user.

Now try typing the following

...

commands.

Code Block
languagebash
ls
ls -l

What do you see? Something like this?

Code Block
languagebash
title

...

Example of command execution

...

-bash-4.1$ ls
bin  Chemistry  Desktop  examples  Experiments  intel  jobs  logs  tmp
-bash-4.1$ ls -l
total 296
drwxr-xr-x 2 abs4 csrv        4096 Jun 24 09:39 bin
drwxr-xr-x 3 abs4 csrv        4096 Jun  6 09:23 Chemistry
drwxr-sr-x 2 abs4 elecclust   4096 Mar 11 10:53 Desktop
drwxr-xr-x 3 abs4 csrv        4096 Jun 30 12:21 examples
drwxr-xr-x 5 abs4 csrv        4096 May 23 11:34 Experiments
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 12:26 intel
drwxr-sr-x 3 abs4 elecclust   4096 Aug 15 12:49 jobs
drwxr-xr-x 2 abs4 csrv      266240 Aug 15 13:48 logs
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 14:50 tmp
-bash-4.1$ 

The ls command lists all of the files in your current directory. A directory is equivalent to a folder.  The ls -l command gives you a long list, showing more information about each file or folder such as who is the owner of the file, who can access it, when it was last accessed. You can use most

...

Linux programs in a number of ways by adding extra

...

options.  Here for ls -l we added the -l

...

option. If you need to know more about a

...

program you can use either of the following:


Code Block
languagebash
ls --help

...

man ls

Filesystem organisation

...


ls can also take arguments as input. For example, to see the contents of a directory in your current directory, pass the name of the directory to ls as an argument. You can combine options and arguments, like the second command below:

Code Block
languagebash
ls Desktop
ls -l Desktop

Filesystem organisation

The file system is the component of the operating system that organises data into files. These files are organised into directories (just like directories in Windows Explorer or the Mac Finder).

When you have logged in you will be placed in a directory which is called your home

...

directory. To find the name of the directory use the program pwd (print working directory). 

...

Now try running the following commands. Here we have shown the prompt (-bash-4.1$) followed by the command to type on one line, then the output of the command on the following line (e.g after the -bash-4.1$, the first command is pwd, and the output is /home/userfs/e/ejb573; do not type the prompt as well, only the command). Your output will be slightly different as it will display the path to your home directory. 

Code Block
languagebash
titleLocating your home directory and files
-bash-4.1$ pwd
/

...

home/

...

userfs/

...

e/

...

ejb573
-bash-4.1$ cd /

...

home
-bash-4.1$ pwd
/

...

home
-bash-4.1$ cd
-bash-4.1$ pwd
/

...

home/

...

userfs/

...

e/

...

ejb573
-bash-4.1$ cd ..
-bash-4.1$ pwd
/

...

home/

...

userfs/

...

e
-bash-4.1$ cd 

...


-bash-4.1$ pwd
/

...

home/userfs/

...

e/

...

ejb573
-bash-4.1$ 


The output of the pwd command, /

...

home/

...

userfs/

...

e/

...

ejb573, is called a pathname, and this specifies the location of

...

user ejb573's home directory. The first '/' in the pathname is called the root directory

...

, the top-level directory in the hierarchy. Names following the '/' are directory names. Directories within directories are called sub-directories.

...

Path names can also specify the location

...

of files.

...

The last part of a pathname (after the last /) is typically the name of a

...

file or directory.

The cd

...

program lets you change your working directory to another

...

directory in the file system. cd with no arguments places you back in your home directory. The special

...

argument '..'

...

means the directory above your current directory (known as the parent directory).

...

Listing files and directories

...


Creating, moving and copying files and directories

You can create new files with the touch program and new directories with mkdir. You can move or copy files or directories to other locations with the mv (move) and cp (copy) programs.

First, let's create a new file and directory.

Code Block
languagebash
title

...

Create a new directory

...

-bash-4.1$ touch 

...

afile
-bash-4.1$ touch bfile
-bash-4.1$ mkdir new-dir
-bash-4.1$ ls

...


...

afile bfile  Chemistry  examples     intel  logs   

...

 

...

 tmp
bin    Desktop    Experiments  jobs 

...

 

...

 new-dir

 You should now see the files afile and bfile and the directory new-dir.  Let's experiment with afile and new-dir:

Code Block
languagebash
titleCreate a new directory
-bash-4.1$ ls
afile  bin       

...

 

...

Desktop  

...

 

...

Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ mv 

...

afile new-dir
-bash-4.1$ cp bfile new-dir
-bash-4.1$ ls
bfile  Chemistry  examples     intel 

...

 

...

logs 

...

 

...

 

...

 

...

 

...

tmp
bin 

...

   Desktop    Experiments 

...

 

...

jobs 

...

 

...

 new-dir
-bash-4.1$ ls new-dir
afile  bfile
-bash-4.1$ mv new-dir/afile .
-bash-4.1$ ls

...


...

afile  bin        Desktop   Experiments 

...

 jobs  new-dir
bfile  Chemistry 

...

 examples  intel        logs  

...

tmp
-bash-4.1$ 

This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. The '.' argument in "mv new-dir/afile ." means the current working directory, so this command moves 'afile' to your working directory.


Copying a directory is a little more complicated as the directory may contain files and directories. We use cp's '-r' option (recursive) to do this.

Code Block
languagebash
titleCopying a directory
-bash-4.1$ ls
afile  bin       

...

 Desktop   Experiments  jobs  tmp
bfile  Chemistry  

...

examples  intel        

...

logs
-bash-4.1$ ls tmp
icc-start  ifort-start  logs       mpi-stop 

...

 start  test
icc-stop   ifort-stop   mpi-start 

...

 new-dir   stop
-bash-4.1$ cp tmp/test .
cp: omitting directory `tmp/test'
-bash-4.1$ cp -r tmp/test .

...


-bash-4.1$ ls 
afile  bin   

...

     Desktop   Experiments  jobs 

...

 test
bfile  Chemistry  examples  intel    

...

    logs  

...

tmp
-bash-4.1$ ls test
test.c  test.cpp  test.f  test.f90  test.x
-bash-4.1$ 

In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory; we have to use the '-r' (recursive) option to tell cp to copy all files and directories within the directory.

Deleting files and directories

You can use the rm (remove) program to delete files. Please be careful with these commands, because Linux has no undo!

Code Block
languagebash
rm filename

To remove a directory and all of its contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.

Code Block
languagebash
rm -ri dirname



Displaying and editing the contents of files

There are a variety of different tools to help you display and edit the contents of your files.  We will provide some examples below but you may find other ones which you prefer to use in the future. 

Exercise 3 - Displaying the contents of files

Expand

File contents can be displayed with several different Linux programs. cat (concatenate files) will print your file to the screen, but this is not a good idea when the files are large. Instead, we can use less to view the file one screen at a time. Try them both on an existing file on the research/teaching servers (here we are using a file we already have called snark2).

Code Block
languagebash
titlecat and more
 -bash-4.1$ cat snark2

The Hunting of the Snark
By Lewis Carroll
Fit the First
            The Landing

"Just the place 

...

for 

...

a Snark!" 

...

the 

...

ls without any options or arguments lists the name of the files and directories in the current working directory. In this example above it is hard to see which names refer to files or directories. We will show you how to do this later on. The next example displays the directory in the long format using the '-l' option, much more information is displayed about the directories and files. The '-a' option shows all files, filenames starting with '.' are usually hidden from display. We can combine options to give more detail.

ls can take arguments as well. When specifying an argument ls displays the information for that file or directory.

Code Block
languagebash
titleDisplaying file or directory information
-bash-4.1$ ls Experiments
architest.dtr                   OLD               simple_verbs.dtr~
architest.dtr~                  OLD CART          simple verbs_to_Dunstan.txt
exploded.csv                    ordered           simple verbs_to_Dunstan.xlsx
Latest CART                     phon.csv          simple_verbs_to.txt
mian.rp                         phonsorted        simple_verbs.txt
NotesAboutInfixPredictions.pdf  simple_verbs.dtr  simpverbsort
-bash-4.1$ ls -l Experiments
total 368
-rw-r--r-- 1 abs4 csrv    919 May 23 11:08 architest.dtr
-rw-r--r-- 1 abs4 csrv    909 May 23 11:08 architest.dtr~
-rw-r--r-- 1 abs4 csrv   3613 May 23 11:08 exploded.csv
drwxr-xr-x 2 abs4 csrv   4096 May 23 11:34 Latest CART
-rw-r--r-- 1 abs4 csrv   4019 May 23 11:28 mian.rp
-rw-r--r-- 1 abs4 csrv 193602 May 23 11:08 NotesAboutInfixPredictions.pdf
drwxr-xr-x 2 abs4 csrv   4096 May 23 11:08 OLD
drwxr-xr-x 2 abs4 csrv   4096 May 23 11:08 OLD CART
-rw-r--r-- 1 abs4 csrv   3613 May 23 11:08 ordered
-rw-r--r-- 1 abs4 csrv   6217 May 23 11:08 phon.csv
-rw-r--r-- 1 abs4 csrv   6217 May 23 11:08 phonsorted
-rw-r--r-- 1 abs4 csrv  17663 May 23 11:08 simple_verbs.dtr
-rw-r--r-- 1 abs4 csrv  17647 May 23 11:08 simple_verbs.dtr~
-rw-r--r-- 1 abs4 csrv   8058 May 23 11:08 simple verbs_to_Dunstan.txt
-rw-r--r-- 1 abs4 csrv  30416 May 23 11:08 simple verbs_to_Dunstan.xlsx
-rw-r--r-- 1 abs4 csrv   4696 May 23 11:08 simple_verbs_to.txt
-rw-r--r-- 1 abs4 csrv  17525 May 23 11:08 simple_verbs.txt
-rw-r--r-- 1 abs4 csrv  17647 May 23 11:08 simpverbsort
-bash-4.1$ ls -ld Experiments
drwxr-xr-x 5 abs4 csrv 4096 May 23 11:34 Experiments
-bash-4.1$ ls -l afile 
-rw-r--r-- 1 abs4 csrv 0 Sep  8 16:26 afile
-bash-4.1$ ls /usr
appl          cmsmigratetest  lfa       phpweb        src       vleexam
archive       cvs             lib       puppet        systems   vle-sysadmin
backups       datasets        lib64     puppetdev     tmp       webmisc
bin           etc             libexec   researchcomp  transfer  work
central       games           local     rlink         userfs    yorkroot
cert          idm             logfiles  sbin          UserFS    yorkweb
cmsmedia      include         mirror    scratch       vle       yorkwebtest
cmsmediatest  java            misc      secbuffer     vle-arch
cmsmigrate    kerberos        opapp     share         vle-eldt
-bash-4.1$ 

Using a directory name as an option causes ls to list the contents of the directory. To list the attributes of the directory use the '-d' option. You can use a pathname as the argument.

Creating, moving and copying files and directories

You can create directories, move or copy files or directories to other locations in the filesystem using the mkdir (make directory) mv (move) and cp (copy) commands.

Code Block
languagebash
titleCreate a new directory
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ mv afile new-dir
-bash-4.1$ cp bfile new-dir
-bash-4.1$ ls
bfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir
-bash-4.1$ ls new-dir
afile  bfile
-bash-4.1$ mv new-dir/afile .
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ 

This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. Note the use of the '.' file to reference the current working directory. We can use full or partial pathnames to reference other parts of the file system.

Copying a directory is a little more complicated and the directory may contain files and directories. We use the '-r' command to cp to do this.

Code Block
languagebash
titleCopying a directory
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  tmp
bfile  Chemistry  examples  intel        logs
-bash-4.1$ ls tmp
icc-start  ifort-start  logs       mpi-stop  start  test
icc-stop   ifort-stop   mpi-start  new-dir   stop
-bash-4.1$ cp tmp/test .
cp: omitting directory `tmp/test'
-bash-4.1$ cp -r tmp/test .
-bash-4.1$ ls 
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ ls test
test.c  test.cpp  test.f  test.f90  test.x
-bash-4.1$ 

In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory. we have to use the '-r' (recursive) option to tell cp to copy all files and directory within the directory.

Deleting files and directories

The rm (remove) command is used to delete files.

Code Block
languagebash
titleDeleting files and directories
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ rm bfile
-bash-4.1$ ls
afile  Chemistry  examples     intel  logs  tmp
bin    Desktop    Experiments  jobs   test
-bash-4.1$

To deleting directories use the rmdir (remove directory) command.

Code Block
languagebash
titleDeleting directories and their contents
-bash-4.1$ rmdir dira
rmdir: failed to remove `dira': Directory not empty
-bash-4.1$ rm -r dira
-bash-4.1$ ls
afile  Chemistry  dirb      Experiments  jobs  test
bin    Desktop    examples  intel        logs  tmp
-bash-4.1$ rm -ri dirb
rm: descend into directory `dirb'? y
rm: descend into directory `dirb/dirb'? y
rm: remove regular empty file `dirb/dirb/afile'? y
rm: remove directory `dirb/dirb'? y
rm: remove regular empty file `dirb/afile'? y
rm: remove directory `dirb'? y
-bash-4.1$ 

rmdir will only remove empty directories. To remove a directory and all it's contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.

Editing and displaying the contents of files

Text Editors

  • vi and vim
  • emacs
  • nano

Displaying the contents of files

The commands cat (concatenate files) and more displays the contents of file.

...

languagebash
titlecat and more

...

Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.

"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."

-bash-4.1$ less snark
The Hunting of the Snark
By Lewis Carroll
            Fit the First
            The Landing
"Just the place for a Snark!" the Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.
"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."
The crew was complete: it included a Boots—
   A maker of Bonnets and Hoods—
A Barrister, brought to arrange their disputes—
   And a Broker, to value their goods.
A Billiard-marker, whose skill was immense,



To move through the file with less, or to quit back to the command line, use the following keys:

  • SPACE - display the next screen of text
  • q - quit displaying the file
  • b - skip backwards through he file
  • /pattern - search for text in the file


There are many text editors available on Linux. The easiest to use is probably nano:

Code Block
languagebash
nano filename

To edit text within nano, just type; to move around the file, use the arrow keys; to exit nano, type Control+X (^X is shown in the shortcuts at the bottom of the screen); to save the file, type Control+O (^O) and hit Enter when prompted for a filename. You might explore the other shortcuts shown at the bottom to try out some other features.

You might also try vi or emacs, which are harder to learn and use but very powerful once you get used to them. If you are using X forwarding (see above), you might also try gedit, which is a graphical text editor.





Copying files and directories remotely

You may need to copy files from your machine at home to one of the research/teaching servers.  There are a number of ways to do this.

Exercise 4 - Copying files from your machine to the research machines

Expand

There are different exercises on copying files dependent on what operating system you use on your local machine.

Expand
titleCopying Files To/From a Windows Desktop

Copying Files To/From a Windows Desktop

WinSCP is an open source free SFTP client, SCP client, FTPS client and FTP client for Windows. Its main function is file transfer between a local and a remote computer. WinSCP is available on IT Services supported desktops or can be downloaded from http://winscp.net/eng/index.php.

Section
bordertrue


Column
width50%

Run WinSCP  from the Start menu or double clicking on the icon on the desktop;

Image Added


A login window will appear. Fill in the hostname and your username:


Column
width50%

Image Added


Running WinSCP

Section
bordertrue


Column
width50%

You can click the "Save" button to save the session details for future use:


Column
width50%

Image Added



Section
bordertrue


Column
width50%

Return to the login window and click the "Login" button. Some hosts may present you with an information window:


Column
width50%

Image Added



Section
bordertrue


Column
width50%

You will then be prompted for your password:


Column
width50%

Image Added



Section
bordertrue


Column
width50%

The file manager window will be displayed:

The drag-and-drop interface is a similar to Windows file manager and its use should be intuitive.


Column
width50%

Image Added




Expand
titleCopying Files To/From a Linux/MacOS Desktop

Copying Files To/From a Linux/MacOS desktop

There are a number of ways to copying files and directories using Linux command line.


you can copy your data from any Linux device to Viking using the following commands 

  • scp
  • rsync

Here are a couple of examples.  

Section
bordertrue

scp 

This is recommended for a small number of files.

You wish to copy your data from local machine to your home area on the research machines.  Run the following commands on your local machine in the terminal. 

Code Block
languagebash
-bash-4.1$ touch cfile
-bash-4.1$ touch dfile
-bash-4.1$ mkdir new-dir2
-bash-4.1$ mv dfile new-dir2
-bash-4.1$ ls
cfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir2

Here we have created some test files and directories on your local machine.  We will now copy these to your home directory on research0.  You will need to enter your University of York password and replace abc123 with your University of York username.

Code Block
languagebash
-bash-4.1$ #For an individual file
-bash-4.1$ scp cfile abc123@research0.york.ac.uk:~/ 

-bash-4.1$ #For a folder with lots of files
-bash-4.1$ scp -r new-dir2 abc123@research0.york.ac.uk:~/

Now login to research0 you should now see the following files and folders in your home area.

Code Block
languagebash
-bash-4.1$ ls
cfile new-dir2

The scp command securely copies these files and folders to the research machine.  The -r flag means scp will recursively copy everything within the folder new-dir2.  The :~/ at the end of the command is the location of where to send the files on the research machine.  In this case it is your home directory. If we wanted to send a copy of dfile directly to new-dir2 folder use the following command.

Code Block
languagebash
-bash-4.1$ #For an individual file
-bash-4.1$ scp dfile abc123@research0.york.ac.uk:~/new-dir2/

This can save you an extra step of moving files on the research machines. Note there was already a copy of dfile in new-dir2 on research0 - we copied it there when we uploaded new-dir2. scp will silently write over any existing file without warning you, so be very careful.


What if you want to copy files from research0 to your local machine?  Run the following commands on your local machine.

Code Block
languagebash
-bash-4.1$ #For an individual file
-bash-4.1$ scp abc123@research0.york.ac.uk:~/cfile . 

-bash-4.1$ #For a folder with lots of files
-bash-4.1$ scp -r abc123@research0.york.ac.uk:~/new-dir2 .
-bash-4.1$ ls

After running ls, do you see the files on your local machine?


There are many options you can use with scp.  To view these options either run

Code Block
languagebash
man scp 

on the device you are using scp on or have a look at this scp wiki page.



RSYNC

Section
bordertrue

Rsync is another command that will let you copy files and folders to the research machines.  If you have a large number of files it is always best to use Rsync.  The Rsync command will compare the files on your computer to the files already on the server, so if the network is interrupted you can run the command again and it will pick up where you left off.  It will only copy files that do not exist on the other server or files that have been changed on your computer.


To copy a folder new-dir2 from your local machine to your home directory on research0, run the following command on your local machine.

Code Block
languagebash
rsync -avz new-dir2 abc123@research0.york.ac.uk:~/

The flags on rsync mean different things.  This would recursively transfer all files from the directory new-dir2 on your local machine onto the home directory on the research machine.  The files are transferred in "archive" mode, which  ensures that information about the files such as timestamps and permissions are preserved in the transfer.  Additionally, the files will be compressed during the transfer so the transfer will be faster.

There are many options you can use with rsync.  To view these options either run

Code Block
languagebash
man rsync 

on the device you are using rsync on or consult the rsync webpage.



Finding and running programs on the research/teaching machines

There are many programs available on the research/teaching machines.  Some programs can be used all the time (like ls, scp and rsync). Some programs need to be loaded using the module system. In this next section we will try both options and show you the best way to find the program you need for your teaching or research.  If you find the software you need is not installed, please email itsupport@york.ac.uk and we will aim to install the software for you.

Exercise 5 - Running applications on the research/teaching machines.


Expand

Running applications on the command line.

You will need to login to the research machines with X forwarding enabled.  If you followed the steps in Exercise 1 you will have already set this up.  From linux or MacOS ssh with the -X flag

Code Block
languagebash
ssh -X abc123@research0.york.ac.uk:~/

Once logged in (either with putty or ssh) type matlab on the command line and wait for the gui to pop up. 

Code Block
languagebash
matlab

There are a number of programs installed this way. If you have ever used linux on the classroom PCs all the programs available there are also available here.

Working with modules

Modules allow us to support many permutations of application versions, built with different compiler versions and technologies. We can thus support new application versions, or optimised builds, without disruption to users who still require earlier editions.  There are many programs installed for use on these machines.  To see what programs are available type 'module avail' in the terminal. Do you see a list of available software?


Code Block
languagebash
ejb573@research0:~$ module avail
--------------------- /usr/lib/Modules/3.2.10/modulefiles ----------------------
dot         module-git  module-info modules     null        use.own
----------------------- /usr/lib/Modules/uoymodulefiles ------------------------
its-hdd      its-net      Ubuntu/16.04 Ubuntu/18.04
--------------- /opt/york/phys/pkg/Modules/modulefiles/compilers ---------------
clang/3.8.0(default)  gcc/5.4.0(default)    gcc/7.4.0
clang/4.0.0           gcc/7 

...

 

...

 

...

 

...

 

...

 

...

 

...

 

...

 

...

 

...

   

...

 

...

 

...

 

...

 intel/17.0.0(default)
gcc/4.8.3             gcc/7.1.0             miniconda/3.7
gcc/4.9.0             gcc/7.3.0             Tcl/8.6.1
------------------ /opt/york/phys/pkg/Modules/modulefiles/mpi ------------------
mpich/3.2      mpich2/1.5     openmpi/1.10.7 openmpi/2.1.1
--------------- /opt/york/phys/pkg/Modules/modulefiles/libraries ---------------
cube/4.3.5           otf2/2.0     

...

 

...

 

...

 

...

 

...

 

...

 

...

 

...

 slepc/3.6.3
CUDA/10.1.160   

...

 

...

 

...

 

...

 

...

 petsc/3.4.5        

...

 

...

The cat command displays all the test in the users file on the screen. This can prove difficult to read if there are large amounts of text. The more command paginates the text and displays portions of it on the screen. The user can use  character command to move through the file:

  • SPACE - display the next screen of text
  • q - quit displaying the file
  • b - skip backwards through he file
  • /pattern - search for text in the file

Files and directory permissions

Groups are provided to manage sets of users and control access to fie and directories. All users belong to a default group and may be a member of other groups.

...

languagebash
titleGroup membership

...

 slepc/3.7.4
CUDA/7.5.18          petsc/3.5.4          slepc/3.9.1
CUDA/8.0.61.p02      petsc/3.6.4          slepc-complex/3.11.2
fftw/3.3.6(default)  petsc/3.7.6          slepc-complex/3.5.4
fftw/3.3.8           petsc/3.8.4          slepc-complex/3.7.4
ghost/1.0            petsc/3.9.0          spglib/1.10.4
hdf5/1.8.16          petsc-complex/3.11.4 spglib/1.9.10
hdf5/1.8.18          petsc-complex/3.5.4  sundials/2.7.0
lapack/3.7.0         petsc-complex/3.7.7  szip/2.1.1
lz4/1.9.1          

...

 

...

 scalasca/2.3.1 

...

 

...

 

...

 

...

 

...

 

...

 Tk/8.6.1
MESASDK/22Aug2018  

...

 

...

 

...

scorep/3.1           uda/1.0.0
netcdf/4.4.1.1  

...

 

...

 

...

 

...

 

...

 slepc/3.4.4        

...

 

...

 zlib/1.2.11
opari2/2.0.2         

...

The groups command displays which groups you are a member of. Each file and directory you create will be owned by you and be potentially accessible to a group. In the above example the file 'afile' is owned by 'abs4' and is accessible to the 'csrv' group.

There is a 'special' group sometimes called world, or other,  which contains all users of the system.

In the above example the first column of the directory listing shows the permissions of the files. These permissions control who is allowed acces the files and directories. There are three categories of user who can has potential rights to access the files - owner, group, world. The access rights to the files are displayed in the form of a sequence of letters like 'drwxr-xr-x'. The meaning is:

  • d - if present this is a directory, otherwise it is a file
  • the following 3 letters are in three groups and state the access permissions for the owner, group, world users
    • w - the file can be written to
    • r - the file can be read
    • x - if a file it can be executed, if a directory it can be accessed
  • example - drwxr-x--- 5 abs4 csrv        4096 Sep  8 17:01 tmp
    • this is a directory
    • the owner, abs4, can read, write and access the directory
    • members of the group, elecclust, can read and access the directory, they can not creat files in the directory
    • all other users do not have any access to the directory

To change file permissions use the chmod command.

Code Block
languagebash
titleChanging file permissions
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark-bash-4.1$ chmod go-rwx snark
-bash-4.1$ ls -l snark
-rw------- 1 abs4 csrv 25678 Sep  9 10:24 snark

The chmod command has the form:

chmod <mode> <file>

mode takes the form of:

[ugoa...][[+-=][perms...]...]

...

Code Block
languagebash
titleSome examples
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod a+rx snark
-bash-4.1$ ls -l snark
-rwxr-xr-x 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod go-rwx snark
-bash-4.1$ ls -l snark
-rwx------ 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u-w snark
-bash-4.1$ ls -l snark
-r-x------ 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod a=r snark
-bash-4.1$ ls -l snark
-r--r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u=w snark
-bash-4.1$ ls -l snark
--w-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u+r snark
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ 

...

languagebash
titleChanging group membership of a file

...

slepc/3.5.4

If you need to search for an application run module avail application-name on command line.  How many versions of python are there available on the research/teaching machines?

Expand
titleClick here to find out after you have ran the command yourself.


Code Block
languagebash
ejb573@research0:~$ module avail python
------------------------- /opt/yarcc/Modules/compilers -------------------------
python/2.7.11   python/3.4.0    python/3.6.0    python/3.6.5-bm
python/2.7.14   python/3.5.1    python/3.6.4
python/2.7.6    python/3.5.3    python/3.6.5

There are 12 versions available using the module command.

Now let us load a module.   On the command line run the following command

Code Block
languagebash
ejb573@research0:~$ python --version
Python 2.7.17

The default, system version Python version is 2.7.17.  We actually want a newer version of python to run our code.  Let us load a new version of Python with the module system

Code Block
languagebash
ejb573@research0:~$ module load python/3.6.5
ejb573@research0:~$ python --version
Python 3.6.5

We have now loaded a new version of python.  To check this you ran python --version.  You can also use module add python/3.6.5 to add a module.

To remove the module run

Code Block
languagebash
ejb573@research0:~$ module unload python/3.6.5
ejb573@research0:~$ python --version
Python 2.7.17


Extra Information

This has been a very basic introduction to the command line, just to get you started. You may also want to look up the following features:

  • Using pipes (|) to pass the output of one tool as input to the next, allowing you to make new tools by combining existing ones
  • Redirecting input and output to files with >
  • Using wildcard characters such as * to refer to many files or directories at once
  • Writing scripts: saving a series of commands to a text file and then running the file as a program
  • Running jobs in the background with &, nohup and nice
  • Variables and options for environment customisation
  • Command-line editing
  • Command history (quick access to previous commands)

If you wish to learn more about how to use Linux command line there are a number of online resources.  We do recommend The Unix Shell - Software Carpentry, where all the exercises and answers are available online. We also have our Online training resources page which provides various options depending on where you wish to get started.



What shall I do if I need more computational power?

The Viking Cluster

If you are finding that your code is still taking a long time to finish or you wish to scale your work, the Viking cluster may be what you need. 

Viking is a large Linux compute cluster aimed at users who require a platform for development and the execution of small or large compute jobs. 

See our Viking Quick Start Guide to get started.  There will also be a similar online course 'introduction to Viking' available by November 2021.


What is a cluster?

Column
width50%

A cluster consists of many (hundreds or thousands) rack mounted computers called nodes. It would be similar to having hundreds of desktop computers sitting in the same room and able to talk to each other.  Clusters are often accessed via login nodes, which can send jobs to the other nodes in the cluster. Your commands will not be run immediately, but will be sent to a queue, and run when there is space on the cluster to run them.

The Viking cluster is Linux based and can be accessed in a similar manner to the research servers, but instead of accessing, say, research0.york.ac.uk, you would access viking.york.ac.uk. We have extensive wiki pages on how to get started on Viking. Please see here for further details and if you ever need help email itsupport@york.ac.uk where one the research computing team can assist you. 


Column
width50%

Image Added


Info

Viking is a multidisciplinary facility, supporting a broad spectrum of research needs, free at the point of use to all University of York researchers. Viking is as much a facility for learning and exploring possibilities as it is a facility for running well-established high-performance computing workloads. In this light, we encourage users from all Faculties, backgrounds and levels of ability to consider Viking when thinking about how computing might support their research.

To access Viking you need a user account, and your user account needs to be associated with a project code. Project codes are typically given to research groups; ask your PI for their code, and if they don't have one, ask them to fill in the Project Application Form. Once you have a project code, you can apply for a user account by filling in the User Application Form. It can take 24 hrs for new accounts to be processed. The Viking Wiki has more help on How to access Viking.