Page tree
Go to start of banner

An Introduction to Research Computing at York

You are viewing an old version of this page. View the current version.

Version 60

This course has been designed to help users access the research/teaching and Viking Linux servers. The aim is to provide you with some basic Linux commands to get you going.  You can access these machines from Windows, Mac and Linux OS.

Introduction

There are a number of different computing facilities available at the University of York.  Have you found any of the following when doing research on your own computer?

• It uses all your machine's resources (compute cores or memory)

• You need lots of memory

• You need GPUs

• You are either using or producing a lot of data

• You think you could cut your job into smaller chunks and process them at the same time
• You know you want to analyse larger datasets in the future.

We have a few different machines to use when you have these problems: individual large machines known as the research and teaching servers, and the Viking compute cluster, a connected group of hundreds of machines. Here will will give you a very brief introduction on how to access these machines.

The research and teaching servers

These servers are also known as the Linux Managed Service or LMS for short. There are currently four research servers (research0, research1, research2 and research3) and two teaching servers (teaching0 and teaching1). Detailed information on the server specifications can be found here.  These machines are Desktops, similar to what you may have at home or in your office, but with a large number of compute cores and memory.  This means that work that your local machine is struggling with may easily be run on one of these machines. You can log on to these machines from anywhere on campus, or off campus if you use the Virtual Private Network (VPN) or SSH gateway service. Some caveats:

• They are a shared machine which means a number of users may be logged on at the same time
• They run Linux so you need a little bit of Linux command line knowledge to get started
• They get rebooted on the first Tuesday of every month

Exercise 1 - Logging into the research or teaching servers.

There are different ways to login to the LMS depending on what operating system you are running.  We will break down the different options here.

Accessing research and teaching servers off campus

To access the research and teaching servers off campus you can either use the Virtual Private Network - VPN or the SSH gateway service (registration required). The instructions below should work if you log on through the VPN; the SSH service works slightly differently.

Access from a Windows desktop

Command-line access using PuTTY

PuTTY is available on all IT Services Managed Windows systems. It is pre-installed on Classroom PCs; on Office PCs you can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the start menu.

When you run PuTTY for the first time, enter the following settings to log on to the research0 server (or replace research0 with teaching0, research1 etc):

1. Add the name "research0.york.ac.uk" to the 'Host Name' field
2. Check the 'Connection Type' to SSH
3. Type the name "research0"  in 'Saved Sessions'
4. Click 'Save'

1. Expand the 'SSH' tab from the 'Connection' list in the 'Category' box
2. Choose 'X11' from 'SSH' list
3. Check 'Enable X11 Forwarding'

1. Choose 'Session' from the 'Category' box
2. Click 'Save'

Connecting to research0
1. Start PuTTY
2. Select 'research0' from the 'Saved Sessions'
3. Click 'Open'
4. The first time you connect you will get a security alert showing the fingerprint of the server, labeled as 'ssh-rsa' or 'ssh-ed25519'. Check the fingerprint below for the label shown and click "Yes" to proceed if it matches.

```ssh-rsa 2048 5c:43:e5:e6:57:e0:4d:9f:f8:b5:ca:52:2f:30:39:ef research0
ssh-rsa 2048 bb:1f:6e:58:fa:d7:23:0d:ae:b2:b2:e8:62:a0:e7:5c research1
ssh-rsa 2048 99:da:16:61:09:e0:19:1f:53:0e:2b:e9:2a:22:50:99 research3
ssh-rsa 2048 ac:7e:1e:2a:05:d8:a1:3b:cf:b5:77:48:d0:bb:8a:22 teaching0

ssh-ed25519 5a:c9:c1:76:16:00:42:45:f9:e5:bd:63:5d:87:db:8a research0
ssh-ed25519 f4:51:59:b7:b5:74:1c:14:a7:2c:78:1c:11:1f:72:cc research1
ssh-ed25519 76:93:b3:2d:e0:73:cb:25:a5:9e:c7:bd:ce:76:8f:a2 research2
ssh-ed25519 64:b2:f0:11:93:fc:d7:ea:12:bb:90:bc:7d:06:75:cc research3
ssh-ed25519 f4:ff:10:59:1e:5f:21:10:14:59:6e:04:16:63:7d:95 teaching0
ssh-ed25519 bc:73:f8:4f:40:7a:6b:41:73:90:f6:77:2b:4f:aa:d5 teaching1```

The previous methods using NX Client and eXceed are now deprecated.
x2go client is available on all IT Services Managed Windows systems. It is pre-installed on Classroom PCs; users on Office PCs can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the Start Menu and there are pre-defined menu entries for research0 and teaching0.

On unmanaged PCs you can download the installer from the x2go Website. You will need to configure the settings as follows for research0 (or replace research0 with the name of the server you want to use):

1. If the "New session" panel does not appear, select the menu item "Session | New session..."
2. Enter "research0" in the 'Session name' field at the top
3. Enter "research0.york.ac.uk" to the 'Server: Host:' field
4. Change the 'Session Type' to XFCE
5. Click "OK" (all other settings can stay on default).

If you wish to change any settings:

2. Click the pull-down on the corner of the panel and select "Session preferences..."

Access from a Mac

Go to 'Finder | Applications | Utilities' and run the Terminal application on your local machine. Then type the following, using your university username (abc123):

Logging into research0

$ssh username@research0.york.ac.uk For a graphical login, install and configure the x2go client software as described in the unmanaged Windows section above. teaching0 and the other servers can be accessed in the same manner. Access from a UNIX server or desktop To login from a terminal window, type the following from your local machine with your university username (abc123): Logging into research0$ssh username@research0.york.ac.uk

If you require X forwarding, type:

Logging research0 with X forwarding

$ssh -X username@research0.york.ac.uk You will be prompted for your IT Services password. Accessing teaching0 is done in the same manner. Using the research/teaching servers with the Linux command line interface Once you have successfully logged into the research or teaching servers, they may look very different to what you are used too, particularly if you are used to using Windows. Please do not let this put you off. The research computing team have successfully managed to help many people use these computers who have never used the Linux command line before. It takes a bit of getting used to but the more you use it the easier and quicker it will become over time. The command line, or shell, has been the major interface for the Unix/Linux operating system since it was first conceived in the late 1960s. The shell allows interaction with the operating system through a text based interface, rather than the graphical interface you are used to. While the graphical interface is easy to learn, and usually makes simple things easy to do, it can be hard to do complex things like operate on large numbers of files, or make different tools work together. The shell can be hard to learn, but is much more powerful and flexible than most graphical interfaces, so can be very useful for research, where we often want to try new things on large data sets. In this tutorial, we will only scratch the surface of the shell's features, just to get you started, but we will note some further features at the end of the tutorial that you may want to look into. The user starts the shell by logging into the computer with a userid and password: Logging in ```****************************************************************************** *** THE UNIVERSITY OF YORK IT SERVICES *** *** *** *** THIS IS A PRIVATE COMPUTER *** *** UNAUTHORISED ACCESS STRICTLY PROHIBITED *** ****************************************************************************** login: user001 password: Last login: Mon Sep 8 14:12:44 2014 from gallifrey.york.ac.uk -bash-4.1$

```

The last line is a command prompt and it is the means by which the computer is telling you that it is ready to accept a command from you. If you do not see the prompt, the computer is probability still executing the last command you have typed. The user types commands which take the form:

program [ options ] [ arguments ]

Roughly speaking, program is the name of the program we want to run, arguments are objects we want to process (typically data files or folders), and options modify how the program will run. Options to a command are usually proceeded by a '-' or '- -' to differentiate them from arguments. The following exercise demonstrates using the echo program with a series of arguments and the ls program with or without options.

Exercise 2 - Running commands in the Linux shell

When you see the prompt type the following command (you can also copy and paste the command into your terminal).

`echo I love York`

What happens?  The terminal should write out "i love York".  echo is a small program that takes a series of arguments and repeats them to the user.

Now try typing the following commands.

```ls
ls -l```

What do you see? Something like this?

Exmaple of command execution
```-bash-4.1$ls bin Chemistry Desktop examples Experiments intel jobs logs tmp -bash-4.1$ ls -l
total 296
drwxr-xr-x 2 abs4 csrv        4096 Jun 24 09:39 bin
drwxr-xr-x 3 abs4 csrv        4096 Jun  6 09:23 Chemistry
drwxr-sr-x 2 abs4 elecclust   4096 Mar 11 10:53 Desktop
drwxr-xr-x 3 abs4 csrv        4096 Jun 30 12:21 examples
drwxr-xr-x 5 abs4 csrv        4096 May 23 11:34 Experiments
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 12:26 intel
drwxr-sr-x 3 abs4 elecclust   4096 Aug 15 12:49 jobs
drwxr-xr-x 2 abs4 csrv      266240 Aug 15 13:48 logs
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 14:50 tmp
-bash-4.1$``` The ls command lists all of the files in your current directory. A directory is equivalent to a folder. The ls -l command gives you a long list, showing more information about each file or folder such as who is the owner of the file, who can access it, when it was last accessed. You can use most Linux programs in a number of ways by adding extra options. Here for ls -l we added the -l option. If you need to know more about a program you can use either of the following: ```ls --help man ls``` ls can also take arguments as input. For example, to see the contents of a directory in your current directory, pass the name of the directory to ls as an argument. You can combine options and arguments, like the second command below: ```ls Desktop ls -l Desktop``` Filesystem organisation The file system is the component of the operating system that organises data into files. These files are organised into directories (similar to folders in Windows OS). When you have logged in you will be placed in a directory which is called your home directory. To find the name of the directory use the program pwd (print working directory). Now try running the following commands. Here we have shown the prompt (-bash-4.1$) followed by the command to type on one line, then the output of the command on the following line (e.g after the -bash-4.1$, the first command is pwd, and the output is /home/userfs/e/ejb573; do not type the prompt as well, only the command). Your output will be slightly different as it will display the path to your home directory. Locating your home directory and files ```-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$cd /home -bash-4.1$ pwd
/home
-bash-4.1$cd -bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$cd .. -bash-4.1$ pwd
/home/userfs/e
-bash-4.1$cd -bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$``` The output of the pwd command, /home/userfs/e/ejb573, is called a pathname, and this specifies the location of user ejb573's home directory. The first '/' in the pathname is called the root directory, the top-level directory in the hierarchy. Names following the '/' are directory names. Directories within directories are called sub-directories. Path names can also specify the location of files. The last part of a pathname (after the last /) is typically the name of a file or directory. The cd program lets you change your working directory to another directory in the file system. cd with no arguments places you back in your home directory. The special argument '..' means the directory above your current directory (known as the parent directory). Creating, moving and copying files and directories You can create new files with the touch program and new directories with mkdir. You can move or copy files or directories to other locations with the mv (move) and cp (copy) programs. First, let's create a new file and directory. Create a new directory ```-bash-4.1$ touch afile
-bash-4.1$touch bfile -bash-4.1$ mkdir new-dir
-bash-4.1$ls afile bfile Chemistry examples intel logs tmp bin Desktop Experiments jobs new-dir ``` You should now see the files afile and bfile and the directory new-dir. Let's experiment with afile and new-dir: Create a new directory ```-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$mv afile new-dir -bash-4.1$ cp bfile new-dir
-bash-4.1$ls bfile Chemistry examples intel logs tmp bin Desktop Experiments jobs new-dir -bash-4.1$ ls new-dir
afile  bfile
-bash-4.1$mv new-dir/afile . -bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$``` This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. The '.' argument in "mv new-dir/afile ." means the current working directory, so this command moves 'afile' to your working directory. Copying a directory is a little more complicated as the directory may contain files and directories. We use cp's '-r' option (recursive) to do this. Copying a directory ```-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  tmp
bfile  Chemistry  examples  intel        logs
-bash-4.1$ls tmp icc-start ifort-start logs mpi-stop start test icc-stop ifort-stop mpi-start new-dir stop -bash-4.1$ cp tmp/test .
cp: omitting directory `tmp/test'
-bash-4.1$cp -r tmp/test . -bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ls test test.c test.cpp test.f test.f90 test.x -bash-4.1$

```

In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory; we have to use the '-r' (recursive) option to tell cp to copy all files and directories within the directory.

Deleting files and directories

You can use the rm (remove) program is used to delete files. Please be careful with these commands, because Linux has no undo!

`rm filename`

To remove a directory and all of its contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.

`rm -r dirname`

Editing and displaying the contents of files

There are a variety of different tools to help you display the contents of your files.  We will provide some examples below but you may find other ones which you prefer to use in the future.

Text Editors

Classic editors

• vi and vim
• emacs
• nano
• geddit

Modern editors

• Visual Studio Code

Exercise 3 - Displaying the contents of files

There are different ways to display the contents of files.  On the command line there built in programs to quickly check and read your files.  The commands cat (concatenate files) and more displays the contents of file. The more command can also quickly display content too. Try them both on a file on the research/teaching servers.

cat and more
``` -bash-4.1$cat snark2 The Hunting of the Snark By Lewis Carroll Fit the First The Landing "Just the place for a Snark!" the Bellman cried, As he landed his crew with care; Supporting each man on the top of the tide By a finger entwined in his hair. "Just the place for a Snark! I have said it twice: That alone should encourage the crew. Just the place for a Snark! I have said it thrice: What I tell you three times is true." -bash-4.1$ more snark
The Hunting of the Snark
By Lewis Carroll
Fit the First
The Landing
"Just the place for a Snark!" the Bellman cried,
As he landed his crew with care;
Supporting each man on the top of the tide
By a finger entwined in his hair.
"Just the place for a Snark! I have said it twice:
That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
What I tell you three times is true."
The crew was complete: it included a Boots—
A maker of Bonnets and Hoods—
A Barrister, brought to arrange their disputes—
And a Broker, to value their goods.
A Billiard-marker, whose skill was immense,
--More--(2%)

```

The cat command displays all the test in the users file on the screen. This can prove difficult to read if there are large amounts of text. The more command paginates the text and displays portions of it on the screen. The user can use  character command to move through the file:

• SPACE - display the next screen of text
• q - quit displaying the file
• b - skip backwards through he file
• /pattern - search for text in the file

If you wish to open a file with a text editor you can typically open them on the command line with the following commands

```geddit filename &

emacs filename &
```

The &  allows you to run the program in the background and free up your terminal.

The shell provides the user with the following features:

• An easy to use command line interface (CLI)
• Combine existing tools to create new tools
• Input/output redirection
• Wildcard characters for filename abbreviation
• Variables and options for environment customisation
• Ability to write shell functions and programs
• Command-line editing