Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 48 Next »

This course has been designed to help users access the research/teaching and Viking Linux servers. The aim is to provide you with some basic Linux commands to get you going.  You can access these machines from Windows, Mac and Linux OS.

Introduction

There are a number of different computing facilities available at the University of York.  Ask yourselves if you have found the following when conducting your research on your local machine?

  • Your workload is taking a long time to run on either your local machine or the Linux managed Service. (>6hrs)

  • It uses all your machines resources (CPU/Memory)

  • You need access to lots of memory

  • You need access to GPUs

  • You are either using or producing a lot of data

  • You think you could cut your job into smaller chunks and process them at the same time
  • You know you want to analyse larger datasets in the future.

We have a few different machines to use when have found you are in the above position.  For this tutorial we will focus on the Linux machines such as research/teaching servers and Viking.  Here will will give you a very brief introduction on how to access these machines.

The research and teaching servers

These servers are also known as the Linux Managed Service or LMS for short. Detailed information on the server specifications can be found here.  These machines are Desktops, similar to what you may have at home or in your office but with a large number of CPUs and memory.  This means that work that your local machine is struggling with may easily be run on one of these machines. Some caveats

  • They are a shared machine which means a number of users may be logged on at the same time
  • They get rebooted on the first Tuesday of every month
  • They run Linux so you may have to learn a little bit of Linux command line to get started. 
  • If you are a undergraduate you will only have access to the teaching servers. 


Exercise 1 - Logging into the research or teaching servers.

There are different ways to login to the LMS depending on what operating system you are running.  We will break down the different options here.

Before You Login

If you have not changed your IT Services password since August 2013 then you must do so before you will be able to login. All user password changes are manage via the My IT Account  web page. Click on the Password Management (IDM) link in the Manage Your Password field to change your password. You may be given the option to 'synchronise' your password, please use this if you do not want to change your password. The password change (or synchronisation) may take a few minutes before it is visible to the servers.

Accessing research and teaching servers from off-campus

To access the research servers from Off-Campus you can either use the Virtual Private Network - VPN or the SSH gateway service (registration required).

Access from a Windows desktop

Command-line access using PuTTY

PuTTY is available on all IT Services Managed Windows systems: it is pre-installed on Classroom PCs, on Office PCs you can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the start menu.

On unmanaged PCs you can download the installer from the PuTTY Website

When you run PuTTY for the first time, enter the following settings (teaching0 uses the same settings):

  1. Add the name "research0.york.ac.uk" to the 'Host Name' field
  2. Check the 'Connection Type' to SSH
  3. Type the name "research0"  in 'Saved Sessions'
  4. Click 'Save'



  1. Expand the 'SSH' tab from the 'Category' list
  2. Choose 'X11' from 'SSH' list
  3. Check 'Enable X11 Forwarding'



  1. Choose 'Session' from 'Category' list
  2. Click 'Save'


Connecting to Research0
  1. Start PuTTY
  2. Select 'research0' from the 'Saved Sessions'
  3. Click 'Open'
  4. The first time you connect you will get a security alert showing the fingerprint of the server, labeled as 'ssh-rsa' or 'ssh-ed25519'. Check the fingerprint below for the label shown and click "Yes" to proceed if it matches.


ssh-rsa 2048 5c:43:e5:e6:57:e0:4d:9f:f8:b5:ca:52:2f:30:39:ef research0
ssh-rsa 2048 bb:1f:6e:58:fa:d7:23:0d:ae:b2:b2:e8:62:a0:e7:5c research1
ssh-rsa 2048 13:42:77:57:ad:33:67:12:a4:8f:d3:26:24:37:2c:e0 research2
ssh-rsa 2048 99:da:16:61:09:e0:19:1f:53:0e:2b:e9:2a:22:50:99 research3
ssh-rsa 2048 ac:7e:1e:2a:05:d8:a1:3b:cf:b5:77:48:d0:bb:8a:22 teaching0
ssh-rsa 2048 bb:de:33:ff:07:23:6b:0a:73:ad:1f:8a:57:b2:c7:77 teaching1

ssh-ed25519 5a:c9:c1:76:16:00:42:45:f9:e5:bd:63:5d:87:db:8a research0
ssh-ed25519 f4:51:59:b7:b5:74:1c:14:a7:2c:78:1c:11:1f:72:cc research1
ssh-ed25519 76:93:b3:2d:e0:73:cb:25:a5:9e:c7:bd:ce:76:8f:a2 research2
ssh-ed25519 64:b2:f0:11:93:fc:d7:ea:12:bb:90:bc:7d:06:75:cc research3
ssh-ed25519 f4:ff:10:59:1e:5f:21:10:14:59:6e:04:16:63:7d:95 teaching0
ssh-ed25519 bc:73:f8:4f:40:7a:6b:41:73:90:f6:77:2b:4f:aa:d5 teaching1



Graphical login using x2go

The previous methods using NX Client and eXceed are now deprecated.
x2go client is available on all IT Services Managed Windows systems : it is pre-installed on Classroom PCs, users on Office PCs can install it from Run Advertised Programs / Software Center. It appears under "Internet Tools" on the start menu - and there are pre-defined menu entries for research0 and teaching0






On unmanaged PCs you can download the installer from the x2go Website You will need to configure the settings as follows (teaching0 uses the same settings):


  1. If the "New session" panel does not appear, select the menu item "Session | New session..."
  2. Enter "research0" in the 'Session name' field at the top
  3. Enter "research0.york.ac.uk" to the 'Server: Host Name' field
  4. Change the 'Session Type' to XFCE
  5. All other settings can stay on default.






If you wish to change any settings:


  1. Cancel any login dialogs
  2. Click the pull-down on the corner of the panel and select "Session preferences..."

Access from a Mac

Go to 'Finder | Applications | Utilities' and run the Terminal application. Then type the following:

Logging into research0

For a graphical login, install and configure the x2go client software as described in the unmanaged Windows section above.

Teaching0 can be accessed in the same manner.

Access from a UNIX server or desktop

To login from a terminal window, type the following from your local device with your university username (abc123):

Logging into research0

$ssh username@research0.york.ac.uk

 If you require X forwarding, type:

Logging research0 with X forwarding

$ ssh -X username@research0.york.ac.uk

You will be prompted for your IT Services password.

Accessing teaching0 is done in the same manner.





Navigating the LMS and basic Linux command line to get you started

Once you have successfully logged into the LMS it may look very different to what you are use too, particularly if you are use to using windows.  Please do not let this put you off.  The research computing team have successfully managed to help many people use these computers who have never used Linux command line before.  It takes a bit of getting use to but the more you use it the easier and quicker it will become over time.


The shell has been the major interface for the Unix/Linux operating system since it was first conceived. The shell allows interaction with the operating system through a text based interface. The shell provides the user with the following features:


  • An easy to use command line interface (CLI)
  • Combine existing tools to create new tools
  • Input/output redirection
  • Wildcard characters for filename abbreviation
  • Variables and options for environment customisation
  • Ability to write shell functions and programs
  • Command-line editing
  • Command history (quick access to previous commands)
  • Arithmetic
  • Command abbreviations


The user starts the shell by logging into the computer with a userid and password


Logging in
******************************************************************************
***                   THE UNIVERSITY OF YORK IT SERVICES                   ***
***                                                                        ***
***                       THIS IS A PRIVATE COMPUTER                       ***
***                UNAUTHORISED ACCESS STRICTLY PROHIBITED                 ***
******************************************************************************

login: user001
password: 
Last login: Mon Sep  8 14:12:44 2014 from gallifrey.york.ac.uk
-bash-4.1$



The last line is a command prompt and it is the means by which the computer is telling you that it is ready to accept a command from you. If you do not see the prompt, the computer is probability still executing the last command you have typed. The user types commands which take the form:


command [ options ] [ arguments ]


Options to a command are usually proceeded by a '-' or '- -', this differentiates them from the arguments. The following example shows the echo command which prints the arguments and the ls command which take options arguments. The ls command display the users file. There will be more explanation of files and the ls command later.


Exercise 2 - Running commands in the Linux shell


When you see the prompt type the following command.  You can also copy and paste the command into your terminal

echo I love York

What happens?  The terminal should write out "i love York".  The echo command is a small program that takes an input called a string, a series of characters and repeats them to the user.

Now try typing the following command.

ls
ls -l

What do you see? Something like this?

Exmaple of command execution
-bash-4.1$ echo Hello World
Hello World
-bash-4.1$ ls
bin  Chemistry  Desktop  examples  Experiments  intel  jobs  logs  tmp
-bash-4.1$ ls -l
total 296
drwxr-xr-x 2 abs4 csrv        4096 Jun 24 09:39 bin
drwxr-xr-x 3 abs4 csrv        4096 Jun  6 09:23 Chemistry
drwxr-sr-x 2 abs4 elecclust   4096 Mar 11 10:53 Desktop
drwxr-xr-x 3 abs4 csrv        4096 Jun 30 12:21 examples
drwxr-xr-x 5 abs4 csrv        4096 May 23 11:34 Experiments
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 12:26 intel
drwxr-sr-x 3 abs4 elecclust   4096 Aug 15 12:49 jobs
drwxr-xr-x 2 abs4 csrv      266240 Aug 15 13:48 logs
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 14:50 tmp
-bash-4.1$ 

The ls command lists all of the files in your current directory. A directory is equivalent to a folder.  The ls -l command gives you more information about each file or folder such as who is the owner of the file, who can access it, when it was last accessed. You can use most linux commands in a number of ways by adding extra flags.  Here for ls -l we added the -l flag. If you need to know more about a command you can use either of the following


ls --help

or

man ls





Filesystem organisation


The file system is the component of the operating system that organises data into files. These files are organised into directories (similar to folders in Windows OS).


When you have logged in you will be placed in a directory which is called your home-directory. To find the name of the directory use the pwd (print working directory). 


Now try running the following commands (e.g after the -bash-4.1$ so the first command is pwd, the output is  /home/userfs/e/ejb573) on in the terminal.  Your output will be slightly different as it will display your username. 

Locating your home directory and files
-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ cd /home
-bash-4.1$ pwd
/home
-bash-4.1$ cd
-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ cd ..
-bash-4.1$ pwd
/home/userfs/e
-bash-4.1$ cd 
-bash-4.1$ pwd
/home/userfs/e/ejb573
-bash-4.1$ 


The output of the pwd command, /home/userfs/e/ejb573, is called a pathname, and this specifies the location of the users home directory. The first '/' in the pathname is the root directory. names following the '/' are directory names. Directories within directories are called sub-directories. Path names can also specify the location within the filesystem of files. Only the last name of a pathaname can be a file or directory.


The cd command lets you change your working directory to another location in the file system. cd with no arguments places you back in your home directory. The special directory '..' references the directory above your current directory (known as the parent directory).


Creating, moving and copying files and directories


You can create directories, move or copy files or directories to other locations in the filesystem using the mkdir (make directory) mv (move) and cp (copy) commands.


Create a new directory
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ mv afile new-dir
-bash-4.1$ cp bfile new-dir
-bash-4.1$ ls
bfile  Chemistry  examples     intel  logs     tmp
bin    Desktop    Experiments  jobs   new-dir
-bash-4.1$ ls new-dir
afile  bfile
-bash-4.1$ mv new-dir/afile .
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  new-dir
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ 


This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. Note the use of the '.' file to reference the current working directory. We can use full or partial pathnames to reference other parts of the file system.


Copying a directory is a little more complicated and the directory may contain files and directories. We use the '-r' command to cp to do this.


Copying a directory
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  tmp
bfile  Chemistry  examples  intel        logs
-bash-4.1$ ls tmp
icc-start  ifort-start  logs       mpi-stop  start  test
icc-stop   ifort-stop   mpi-start  new-dir   stop
-bash-4.1$ cp tmp/test .
cp: omitting directory `tmp/test'
-bash-4.1$ cp -r tmp/test .
-bash-4.1$ ls 
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ ls test
test.c  test.cpp  test.f  test.f90  test.x
-bash-4.1$ 


In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory. we have to use the '-r' (recursive) option to tell cp to copy all files and directory within the directory.


Deleting files and directories


The rm (remove) command is used to delete files.


Deleting files and directories
-bash-4.1$ ls
afile  bin        Desktop   Experiments  jobs  test
bfile  Chemistry  examples  intel        logs  tmp
-bash-4.1$ rm bfile
-bash-4.1$ ls
afile  Chemistry  examples     intel  logs  tmp
bin    Desktop    Experiments  jobs   test
-bash-4.1$


To deleting directories use the rmdir (remove directory) command.


Deleting directories and their contents
-bash-4.1$ rmdir dira
rmdir: failed to remove `dira': Directory not empty
-bash-4.1$ rm -r dira
-bash-4.1$ ls
afile  Chemistry  dirb      Experiments  jobs  test
bin    Desktop    examples  intel        logs  tmp
-bash-4.1$ rm -ri dirb
rm: descend into directory `dirb'? y
rm: descend into directory `dirb/dirb'? y
rm: remove regular empty file `dirb/dirb/afile'? y
rm: remove directory `dirb/dirb'? y
rm: remove regular empty file `dirb/afile'? y
rm: remove directory `dirb'? y
-bash-4.1$ 


rmdir will only remove empty directories. To remove a directory and all it's contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.


Editing and displaying the contents of files


Text Editors


  • vi and vim
  • emacs
  • nano


Displaying the contents of files


The commands cat (concatenate files) and more displays the contents of file.


cat and more
 -bash-4.1$ cat snark2

The Hunting of the Snark
By Lewis Carroll
Fit the First
            The Landing

"Just the place for a Snark!" the Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.

"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."

-bash-4.1$ more snark
The Hunting of the Snark
By Lewis Carroll
            Fit the First
            The Landing
"Just the place for a Snark!" the Bellman cried,
   As he landed his crew with care;
Supporting each man on the top of the tide
   By a finger entwined in his hair.
"Just the place for a Snark! I have said it twice:
   That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
   What I tell you three times is true."
The crew was complete: it included a Boots—
   A maker of Bonnets and Hoods—
A Barrister, brought to arrange their disputes—
   And a Broker, to value their goods.
A Billiard-marker, whose skill was immense,
--More--(2%)


The cat command displays all the test in the users file on the screen. This can prove difficult to read if there are large amounts of text. The more command paginates the text and displays portions of it on the screen. The user can use  character command to move through the file:


  • SPACE - display the next screen of text
  • q - quit displaying the file
  • b - skip backwards through he file
  • /pattern - search for text in the file


Files and directory permissions


Groups are provided to manage sets of users and control access to fie and directories. All users belong to a default group and may be a member of other groups.


Group membership
-bash-4.1$ groups
csrv pfs17 pfs34 csys cvssys itsilg rhpcs sshfix git5 git6 git7 elecclust yc-gauss yc-install yc-colum
-bash-4.1$ ls -l
total 332
-rw-r--r-- 1 abs4 csrv           0 Sep  8 16:26 afile
drwxr-xr-x 2 abs4 csrv        4096 Jun 24 09:39 bin
drwxr-xr-x 3 abs4 csrv        4096 Jun  6 09:23 Chemistry
drwxr-sr-x 2 abs4 elecclust   4096 Mar 11 10:53 Desktop
drwxr-xr-x 3 abs4 csrv        4096 Jun 30 12:21 examples
drwxr-xr-x 5 abs4 csrv        4096 May 23 11:34 Experiments
drwxr-xr-x 3 abs4 csrv        4096 Aug 14 12:26 intel
drwxr-sr-x 3 abs4 elecclust   4096 Aug 15 12:49 jobs
drwxr-xr-x 2 abs4 csrv      266240 Aug 15 13:48 logs
-rw-r--r-- 1 abs4 csrv       25678 Sep  9 10:24 snark
-rw-r--r-- 1 abs4 csrv         433 Sep  9 10:27 snark2
drwxr-xr-x 2 abs4 csrv        4096 Sep  8 17:08 test
drwxr-x--- 5 abs4 csrv        4096 Sep  8 17:01 tmp
-bash-4.1$ 


The groups command displays which groups you are a member of. Each file and directory you create will be owned by you and be potentially accessible to a group. In the above example the file 'afile' is owned by 'abs4' and is accessible to the 'csrv' group.


There is a 'special' group sometimes called world, or other,  which contains all users of the system.


In the above example the first column of the directory listing shows the permissions of the files. These permissions control who is allowed acces the files and directories. There are three categories of user who can has potential rights to access the files - owner, group, world. The access rights to the files are displayed in the form of a sequence of letters like 'drwxr-xr-x'. The meaning is:


  • d - if present this is a directory, otherwise it is a file
  • the following 3 letters are in three groups and state the access permissions for the owner, group, world users
    • w - the file can be written to
    • r - the file can be read
    • x - if a file it can be executed, if a directory it can be accessed
  • example - drwxr-x--- 5 abs4 csrv        4096 Sep  8 17:01 tmp
    • this is a directory
    • the owner, abs4, can read, write and access the directory
    • members of the group, elecclust, can read and access the directory, they can not creat files in the directory
    • all other users do not have any access to the directory


To change file permissions use the chmod command.


Changing file permissions
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark-bash-4.1$ chmod go-rwx snark
-bash-4.1$ ls -l snark
-rw------- 1 abs4 csrv 25678 Sep  9 10:24 snark


The chmod command has the form:


chmod <mode> <file>


mode takes the form of:


[ugoa...][[+-=][perms...]...]


  • u = user (owner)
  • g = group
  • o = other (world)
  • a = all (user, group, other)

  • + add permission
  • - remove permission
  • = explicitly set permission

  • w = can be written to
  • r = can be read
  • x = can be executed if a file, if a directory it can be accessed


Some examples
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod a+rx snark
-bash-4.1$ ls -l snark
-rwxr-xr-x 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod go-rwx snark
-bash-4.1$ ls -l snark
-rwx------ 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u-w snark
-bash-4.1$ ls -l snark
-r-x------ 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod a=r snark
-bash-4.1$ ls -l snark
-r--r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u=w snark
-bash-4.1$ ls -l snark
--w-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ chmod u+r snark
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ 


To change the group of a file use the  command chgrp <groupname> <filename>.


Changing group membership of a file
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 csrv 25678 Sep  9 10:24 snark
-bash-4.1$ groups
csrv pfs17 pfs34 csys cvssys itsilg rhpcs sshfix git5 git6 git7 elecclust yc-gauss yc-install yc-colum
-bash-4.1$ chgrp rhpcs snark
-bash-4.1$ ls -l snark
-rw-r--r-- 1 abs4 rhpcs 25678 Sep  9 10:24 snark
-bash-4.1$ 









  • No labels