Exercise 5 - Running applications on the research/teaching machines.
There are a number of different computing facilities available at the University of York. Have you found any of the following when doing research on your own computer?
We have a few different machines to use when you have these problems: individual large machines known as the research and teaching servers, and the Viking compute cluster, a connected group of hundreds of machines. Here will will give you a very brief introduction on how to access these machines.
The research and teaching servers
These servers are also known as the Linux Managed Service or LMS for short. There are currently four research servers (research0, research1, research2 and research3) and two teaching servers (teaching0 and teaching1). Detailed information on the server specifications can be found here. These machines are Desktops, similar to what you may have at home or in your office, but with a large number of compute cores and memory. This means that work that your local machine is struggling with may easily be run on one of these machines. You can log on to these machines from anywhere on campus, or off campus if you use the Virtual Private Network (VPN) or SSH gateway service. Some caveats:
- They are a shared machine which means a number of users may be logged on at the same time
- They run Linux so you need a little bit of Linux command line knowledge to get started
- If you are a undergraduate you will only have access to the teaching servers
- They get rebooted on the first Tuesday of every month
Exercise 1 - Logging into the research or teaching servers.
There are different ways to login to the LMS depending on what operating system you are running. We will break down the different options here.
Before You Login
If you have not changed your IT Services password since August 2013 then you must do so before you will be able to login. All user password changes are manage via the My IT Account web page. Click on the Password Management (IDM) link in the Manage Your Password field to change your password. You may be given the option to 'synchronise' your password; please use this option if you do not want to change your password. The password change (or synchronisation) may take a few minutes before it is visible to the servers.
Accessing research and teaching servers off campus
To access the research and teaching servers off campus you can either use the Virtual Private Network - VPN or the SSH gateway service (registration required). The instructions below should work if you log on through the VPN; the SSH service works slightly differently.
Access from a Windows desktop
Command-line access using PuTTY
PuTTY is available on all IT Services Managed Windows systems. It is pre-installed on Classroom PCs; on Office PCs you can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the start menu.
On unmanaged PCs you can download the installer from the PuTTY Website.
When you run PuTTY for the first time, enter the following settings to log on to the research0 server (or replace research0 with teaching0, research1 etc):
- Add the name "research0.york.ac.uk" to the 'Host Name' field
- Check the 'Connection Type' to SSH
- Type the name "research0" in 'Saved Sessions'
- Click 'Save'
- Expand the 'SSH' tab from the 'Connection' list in the 'Category' box
- Choose 'X11' from 'SSH' list
- Check 'Enable X11 Forwarding'
- Choose 'Session' from the 'Category' box
- Click 'Save'
Connecting to research0
- Start PuTTY
- Select 'research0' from the 'Saved Sessions'
- Click 'Open'
- The first time you connect you will get a security alert showing the fingerprint of the server, labeled as 'ssh-rsa' or 'ssh-ed25519'. Check the fingerprint below for the label shown and click "Yes" to proceed if it matches.
Graphical login using x2go
On unmanaged PCs you can download the installer from the x2go Website. You will need to configure the settings as follows for research0 (or replace research0 with the name of the server you want to use):
- If the "New session" panel does not appear, select the menu item "Session | New session..."
- Enter "research0" in the 'Session name' field at the top
- Enter "research0.york.ac.uk" to the 'Server: Host:' field
- Change the 'Session Type' to XFCE
- Click "OK" (all other settings can stay on default).
If you wish to change any settings:
- Cancel any login dialogs
- Click the pull-down on the corner of the panel and select "Session preferences..."
Access from a Mac
Go to 'Finder | Applications | Utilities' and run the Terminal application on your local machine. Then type the following, using your university username (abc123):
For a graphical login, install and configure the x2go client software as described in the unmanaged Windows section above.
teaching0 and the other servers can be accessed in the same manner.
Access from a UNIX server or desktop
To login from a terminal window, type the following from your local machine with your university username (abc123):
If you require X forwarding, type:
You will be prompted for your IT Services password.
Accessing teaching0 is done in the same manner.
Using the research/teaching servers with the Linux command line interface
Once you have successfully logged into the research or teaching servers, they may look very different to what you are used too, particularly if you are used to using Windows. Please do not let this put you off. The research computing team have successfully managed to help many people use these computers who have never used the Linux command line before. It takes a bit of getting used to but the more you use it the easier and quicker it will become over time.
The command line, or shell, has been the major interface for the Unix/Linux operating system since it was first conceived in the late 1960s. The shell allows interaction with the operating system through a text based interface, rather than the graphical interface you are used to. While the graphical interface is easy to learn, and usually makes simple things easy to do, it can be hard to do complex things like operate on large numbers of files, or make different tools work together. The shell can be hard to learn, but is much more powerful and flexible than most graphical interfaces, so can be very useful for research, where we often want to try new things on large data sets.
In this tutorial, we will only scratch the surface of the shell's features, just to get you started, but we will note some further features at the end of the tutorial that you may want to look into.
The user starts the shell by logging into the computer with a userid and password:
The last line is a command prompt and it is the means by which the computer is telling you that it is ready to accept a command from you. If you do not see the prompt, the computer is probability still executing the last command you have typed. The user types commands which take the form:
Roughly speaking, program is the name of the program we want to run, arguments are objects we want to process (typically data files or folders), and options modify how the program will run. Options to a command are usually proceeded by a '-' or '- -' to differentiate them from arguments. The following exercise demonstrates using the echo program with a series of arguments and the ls program with or without options.
Exercise 2 - Running commands in the Linux shell
When you see the prompt type the following command (you can also copy and paste the command into your terminal).
What happens? The terminal should write out "i love York". echo is a small program that takes a series of arguments and repeats them to the user.
Now try typing the following commands.
What do you see? Something like this?
The ls command lists all of the files in your current directory. A directory is equivalent to a folder. The ls -l command gives you a long list, showing more information about each file or folder such as who is the owner of the file, who can access it, when it was last accessed. You can use most Linux programs in a number of ways by adding extra options. Here for ls -l we added the -l option. If you need to know more about a program you can use either of the following:
ls can also take arguments as input. For example, to see the contents of a directory in your current directory, pass the name of the directory to ls as an argument. You can combine options and arguments, like the second command below:
The file system is the component of the operating system that organises data into files. These files are organised into directories (similar to folders in Windows OS).
When you have logged in you will be placed in a directory which is called your home directory. To find the name of the directory use the program pwd (print working directory).
Now try running the following commands. Here we have shown the prompt (-bash-4.1$) followed by the command to type on one line, then the output of the command on the following line (e.g after the -bash-4.1$, the first command is pwd, and the output is /home/userfs/e/ejb573; do not type the prompt as well, only the command). Your output will be slightly different as it will display the path to your home directory.
The output of the pwd command, /home/userfs/e/ejb573, is called a pathname, and this specifies the location of user ejb573's home directory. The first '/' in the pathname is called the root directory, the top-level directory in the hierarchy. Names following the '/' are directory names. Directories within directories are called sub-directories. Path names can also specify the location of files. The last part of a pathname (after the last /) is typically the name of a file or directory.
The cd program lets you change your working directory to another directory in the file system. cd with no arguments places you back in your home directory. The special argument '..' means the directory above your current directory (known as the parent directory).
Creating, moving and copying files and directories
You can create new files with the touch program and new directories with mkdir. You can move or copy files or directories to other locations with the mv (move) and cp (copy) programs.
First, let's create a new file and directory.
You should now see the files afile and bfile and the directory new-dir. Let's experiment with afile and new-dir:
This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. The '.' argument in "mv new-dir/afile ." means the current working directory, so this command moves 'afile' to your working directory.
Copying a directory is a little more complicated as the directory may contain files and directories. We use cp's '-r' option (recursive) to do this.
In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory; we have to use the '-r' (recursive) option to tell cp to copy all files and directories within the directory.
Deleting files and directories
You can use the rm (remove) program is used to delete files. Please be careful with these commands, because Linux has no undo!
To remove a directory and all of its contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.
Displaying and editing the contents of files
There are a variety of different tools to help you display and edit the contents of your files. We will provide some examples below but you may find other ones which you prefer to use in the future.
Exercise 3 - Displaying the contents of files
File contents can be displayed with several different Linux programs. cat (concatenate files) will print your file to the screen, but this is not a good idea when the files are large. Instead, we can use less to view the file one screen at a time. Try them both on an existing file on the research/teaching servers (here we are using a file we already have called snark2).
To move through the file with less, or to quit back to the command line, use the following keys:
- SPACE - display the next screen of text
- q - quit displaying the file
- b - skip backwards through he file
- /pattern - search for text in the file
There are many text editors available on Linux. The easiest to use is probably nano:
To edit text within nano, just type; to move around the file, use the arrow keys; to exit nano, type Control+X (^X is shown in the shortcuts at the bottom of the screen); to save the file, type Control+O (^O) and hit Enter when prompted for a filename. You might explore the other shortcuts shown at the bottom to try out some other features.
You might also try vi or emacs, which are harder to learn and use but very powerful once you get used to them. If you are using X forwarding (see above), you might also try gedit, which is a graphical text editor.
Being a good citizen: check for others using the machine before running large jobs:
Running long jobs in the background?
Ctrl+Z / bg?
(Or just say, if you need to run long-running jobs, use Viking…?)
Copying files and directories remotely
You may need to copy files from your machine at home to one of the research/teaching servers. There are a number of ways to do this
Exercise 4 - Copying files from your machine to the research machines
There are different exersises on copying files dependent on what operating system you use on your local machine.
Copying Files To/From a Windows Desktop
WinSCP is an open source free SFTP client, SCP client, FTPS client and FTP client for Windows. Its main function is file transfer between a local and a remote computer. WinSCP is available on IT Services supported desktops or can be downloaded from http://winscp.net/eng/index.php.
Run WinSCP from the Start menu or double clicking on the icon on the desktop;
A login window will appear Fill in the hostname and your username:
You can click the "Save" button to save the session details for future use:
Return to the login window and click the "Login" button Some host may present you with an information window:
You will then be prompted for your password:
The file manager window will be displayed:
The drag-and-drop interface is a similar to Windows file manager and its use should be intuitive.
Copying Files To/From a Linux/MacOS desktop
There are a number of ways to copying files and directories using Linux command line.
you can copy your data from any Linux device to Viking using the following commands
Here are a couple of examples.
Recommended for a small number of files
You wish to copy your data from local machine to your home area on the research machines. Run the following commands on your local machine in the terminal.
Here we have created some test files and directories on your local machine. We will now copy these to research0. You will need to enter your university of york password and replace abc123 with your university of York username.
Now login to research0 you should now see the following files and folders in your home area.
The scp command securely copies these files and folders to the research machine. The -r flag means scp will recursively copy everything within the folder new-dir2. If you moved dfile to new-dir2 that has also been copied to the research machine. The :~/ at the end of the command is the location of where to send the files on the research machine. In this case it is your home area. If we wanted to send a copy of dfile directly to new-dir folder use the following command.
This can save you an extra step of moving files on the research machines.
What if you want to copy files from research0 to your local machine? Run the following commands on your local machine.
After running ls, do you see the files on your local machine?
There are many options you can use with scp. To view these options either run
on the device you are using scp on or have a look at this scp wiki page
Rsync is another command that will let you copy files and folders to the research machines. If you have a large number of files it is always best to use Rsync. The Rsync command will compare what is already in place, therefore if the network is interrupted you can run the command again and it will pick up where you left off. It will only copy files that do not exists on the other server or files that have been changed.
You wish to copy your data from your local machine to the research machine and place it in your home directory run the following command. This should be run on your local machine
Here is an example
The flags on rsync mean different things. This would recursively transfer all files from the directory new-dir2 on your local machine onto the home directory on the research machine. The files are transferred in "archive" mode, which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.
There are many options you can use with rsync. To view these options either run
on the device you are using rsync on or consult the rsync webpage
Finding and running programs on the research/teaching machines.
There are many programs available on the research/teaching machines. Some programs can be accessible on the research/teaching machines command line. Some programs can only be accessible using the module system. In this next section we will try both options and show you the best way to find the program you need for your teaching or research. If you find the software you need is not installed , please email firstname.lastname@example.org and we will aim to install the software for you.
Exercise 5 - Running applications on the research/teaching machines.
Running applications on the command line.
You will need to login to the research machines with X forwarding enabled. If you followed the steps in exercise 1 you will have already set this up. From linux or MacOS ssh with the -X flag
Once logged in (either with putty or ssh) type matlab on the command line and wait for the gui to pop up.
There are a number of programs installed this way. If you have ever used linux on the classroom PCs all the programs available there are also available here.
Using the module system
Working with modules
Modules allow us to support many permutations of application versions, built with different compiler versions and technologies. We can thus support new application versions, or optimised builds, without disruption to users who still require earlier editions. There are many programs installed for use on these machines. To see what programs are available type module avail in the terminal. Do you see a list of available software?
If you need to search for an application run module avail application-name on command line. How many versions of python are there available on the research/teaching machines?
There are 12 versions available using the module command.
Now let us load a module. On the command line run the following command
The version locally installed is Python version 2.7.17. We actually want a newer version of python to run our code. Let us load a new version of Python with the module system
We have now loaded a new version of python. To check this you ran python --version. You can also use module add python/3.6.5 to
This has been a very basic introduction to the command line, just to get you started. You may also want to look up the following features:
- Using pipes (|) to pass the output of one tool as input to the next, allowing you to make new tools by combining existing ones
- Redirecting input and output to files with >
- Using wildcard characters such as * to refer to many files or directories at once
- Writing scripts: saving a series of commands to a text file and then running the file as a program
- Variables and options for environment customisation
- Command-line editing
- Command history (quick access to previous commands)
The Viking Cluster
Viking is a large Linux compute cluster aimed at users who require a platform for development and the execution of small or large compute jobs.
Viking is a multidisciplinary facility, supporting a broad spectrum of research needs, free at the point of use to all University of York researchers. Viking is as much a facility for learning and exploring possibilities as it is a facility for running well-established high-performance computing workloads. In this light, we encourage users from all Faculties, backgrounds and levels of ability to consider Viking when thinking about how computing might support their research.
If you have found that your code is still running slow on the research machines then Viking might be the best place to move your workloads. To access Viking you do need an account which involves obtaining a project code and then you can apply for a user account. You will not be able to access Viking without following these steps. It can take 24 hrs for new accounts to be processed. Follow this link to apply for an account.
What is a cluster?
A cluster consists of many (100's - 1,000's) rack mounted computers called nodes. It would be having 100s of desktops computers sitting in the same room and able to talk to each other. Clusters are often accessed via login nodes.
The Viking cluster is linux based and can be accessed in a similar manner as the research servers. To access via putty replace any instance of the name researchX with Viking. Same with ssh when using Linux.