test
Introduction
There are a number of different computing facilities available at the University of York. Ask yourselves if you have found the following when conducting your research on your local machine?
Your workload is taking a long time to run on either your local machine or the Linux managed Service. (>6hrs)
It uses all your machines resources (CPU/Memory)
You need access to lots of memory
You need access to GPUs
You are either using or producing a lot of data
- You think you could cut your job into smaller chunks and process them at the same time
- You know you want to analyse larger datasets in the future.
We have a few different machines to use when have found you are in the above position. For this tutorial we will focus on the Linux machines such as research/teaching servers and Viking. Here will will give you a very brief introduction on how to access these machines.
The research and teaching servers
These servers are also known as the Linux Managed Service or LMS for short. Detailed information on the server specifications can be found here. These machines are Desktops, similar to what you may have at home or in your office but with a large number of CPUs and memory. This means that work that your local machine is struggling with may easily be run on one of these machines. Some caveats
- They are a shared machine which means a number of users may be logged on at the same time
- They get rebooted on the first Tuesday of every month
- They run Linux so you may have to learn a little bit of Linux command line to get started.
- If you are a undergraduate you will only have access to the teaching servers.
Exercise 1 - Logging into the research or teaching servers.
There are different ways to login to the LMS depending on what operating system you are running. We will break down the different options here.
Navigating the LMS and basic Linux command line to get you started
Once you have successfully logged into the LMS it may look very different to what you are use too, particularly if you are use to using windows. Please do not let this put you off. The research computing team have successfully managed to help many people use these computers who have never used Linux command line before. It takes a bit of getting use to but the more you use it the easier and quicker it will become over time.
The shell has been the major interface for the Unix/Linux operating system since it was first conceived. The shell allows interaction with the operating system through a text based interface. The shell provides the user with the following features:
- An easy to use command line interface (CLI)
- Combine existing tools to create new tools
- Input/output redirection
- Wildcard characters for filename abbreviation
- Variables and options for environment customisation
- Ability to write shell functions and programs
- Command-line editing
- Command history (quick access to previous commands)
- Arithmetic
- Command abbreviations
The user starts the shell by logging into the computer with a userid and password
****************************************************************************** *** THE UNIVERSITY OF YORK IT SERVICES *** *** *** *** THIS IS A PRIVATE COMPUTER *** *** UNAUTHORISED ACCESS STRICTLY PROHIBITED *** ****************************************************************************** login: user001 password: Last login: Mon Sep 8 14:12:44 2014 from gallifrey.york.ac.uk -bash-4.1$
The last line is a command prompt and it is the means by which the computer is telling you that it is ready to accept a command from you. If you do not see the prompt, the computer is probability still executing the last command you have typed. The user types commands which take the form:
command [ options ] [ arguments ]
Options to a command are usually proceeded by a '-' or '- -', this differentiates them from the arguments. The following example shows the echo command which prints the arguments and the ls command which take options arguments. The ls command display the users file. There will be more explanation of files and the ls command later.
Exercise 2 - Running commands in the Linux shell
Filesystem organisation
The file system is the component of the operating system that organises data into files. These files are organised into directories.
When you have logged in you will be placed in a directory which is called your home-directory. To find the name of the directory use the pwd (print working directory).
-bash-4.1$ pwd /usr/researchcomp/elecclust/abs4 -bash-4.1$ cd /usr -bash-4.1$ pwd /usr -bash-4.1$ cd -bash-4.1$ pwd /usr/researchcomp/elecclust/abs4 -bash-4.1$ cd .. -bash-4.1$ pwd /usr/researchcomp/elecclust -bash-4.1$ cd . -bash-4.1$ pwd /usr/researchcomp/elecclust -bash-4.1$
The output of the pwd command, /usr/researchcomp/elecclust/abs4, is called a pathname, and this specifies the location of the users home directory. The first '/' in the pathname is the root directory. names following the '/' are directory names. Directories within directories are called sub-directories. Pathanmes can also specify the location within the filesystem of files. Only the last name of a pathaname can be a file or directory.
The cd command lets you change your working directory to another location in the file system. cd with no arguments places you back in your home directory. The special directory '..' references the directory above your current directory (known as the parent directory). The is another speicial direcory '.' which references the current directory. These two directories can be viewd as links.
Listing files and directories
To list the files in a directory use the ls (list) command.
-bash-4.1$ ls afile bin Chemistry Desktop examples Experiments intel jobs logs tmp -bash-4.1$ ls -l total 296 -rw-r--r-- 1 abs4 csrv 0 Sep 8 16:26 afile drwxr-xr-x 2 abs4 csrv 4096 Jun 24 09:39 bin drwxr-xr-x 3 abs4 csrv 4096 Jun 6 09:23 Chemistry drwxr-sr-x 2 abs4 elecclust 4096 Mar 11 10:53 Desktop drwxr-xr-x 3 abs4 csrv 4096 Jun 30 12:21 examples drwxr-xr-x 5 abs4 csrv 4096 May 23 11:34 Experiments drwxr-xr-x 3 abs4 csrv 4096 Aug 14 12:26 intel drwxr-sr-x 3 abs4 elecclust 4096 Aug 15 12:49 jobs drwxr-xr-x 2 abs4 csrv 266240 Aug 15 13:48 logs drwxr-xr-x 3 abs4 csrv 4096 Aug 14 14:50 tmp -bash-4.1$ ls -a . Experiments intel .profile .. .felix jobs .pulse afile .gconf .lesshst .pulse-cookie .bash_history .gconfd .local .python_history bin .gnome2 logs .qmon_preferences Chemistry .gnote .matlab .Rhistory .config .gnupg .mcrCache8.3 .ssh .dbus .gstreamer-0.10 .modulerc .subversion Desktop .gvfs .nautilus tmp .emacs.d .history .ngspice_history .Xauthority examples .ICEauthority .nx -bash-4.1$ ls -al total 440 drwx------ 30 abs4 elecclust 4096 Sep 8 16:26 . drwxrws--- 14 jaw500 elecclust 4096 Sep 8 16:25 .. -rw-r--r-- 1 abs4 csrv 0 Sep 8 16:26 afile -rw------- 1 abs4 elecclust 16495 Sep 8 15:40 .bash_history drwxr-xr-x 2 abs4 csrv 4096 Jun 24 09:39 bin drwxr-xr-x 3 abs4 csrv 4096 Jun 6 09:23 Chemistry drwxr-sr-x 4 abs4 elecclust 4096 Mar 11 10:53 .config drwx--S--- 3 abs4 elecclust 4096 Mar 11 10:51 .dbus drwxr-sr-x 2 abs4 elecclust 4096 Mar 11 10:53 Desktop drwxr-xr-x 3 abs4 csrv 4096 May 23 14:52 .emacs.d drwxr-xr-x 3 abs4 csrv 4096 Jun 30 12:21 examples drwxr-xr-x 5 abs4 csrv 4096 May 23 11:34 Experiments drwxr-xr-x 2 abs4 csrv 4096 Jul 1 12:00 .felix drwx--S--- 4 abs4 elecclust 4096 May 2 16:09 .gconf drwx--S--- 2 abs4 elecclust 4096 May 2 16:34 .gconfd drwx--S--- 4 abs4 elecclust 4096 Mar 11 10:53 .gnome2 drwxr-sr-x 3 abs4 elecclust 4096 Mar 11 10:53 .gnote drwx--S--- 2 abs4 elecclust 4096 Mar 11 10:52 .gnupg drwxr-sr-x 2 abs4 elecclust 4096 Mar 11 10:53 .gstreamer-0.10 drwx--S--- 2 abs4 elecclust 4096 Mar 11 10:52 .gvfs -rw------- 1 abs4 csrv 978 Jun 6 09:32 .history -rw------- 1 abs4 elecclust 314 Mar 11 10:52 .ICEauthority drwxr-xr-x 3 abs4 csrv 4096 Aug 14 12:26 intel drwxr-sr-x 3 abs4 elecclust 4096 Aug 15 12:49 jobs -rw------- 1 abs4 csrv 46 Jun 6 09:31 .lesshst drwxr-sr-x 3 abs4 elecclust 4096 Mar 11 10:52 .local drwxr-xr-x 2 abs4 csrv 266240 Aug 15 13:48 logs drwxr-xr-x 3 abs4 csrv 4096 May 2 16:06 .matlab drwxr-xr-x 9 abs4 csrv 4096 Jul 3 11:54 .mcrCache8.3 -rw-r--r-- 1 abs4 csrv 32 Sep 5 08:05 .modulerc drwxr-sr-x 2 abs4 elecclust 4096 Mar 11 10:53 .nautilus -rw------- 1 abs4 elecclust 0 Jan 13 2014 .ngspice_history drwx--S--- 6 abs4 elecclust 4096 Apr 25 13:36 .nx -rw-r--r-- 1 abs4 elecclust 145 May 19 11:59 .profile drwx------ 2 abs4 csrv 4096 Mar 11 10:54 .pulse -rw------- 1 abs4 elecclust 256 Mar 11 10:54 .pulse-cookie -rw------- 1 abs4 csrv 49 Jun 3 13:42 .python_history -rw-r--r-- 1 abs4 csrv 342 Jun 16 12:57 .qmon_preferences -rw------- 1 abs4 csrv 40 May 23 11:09 .Rhistory drwxr-sr-x 2 abs4 elecclust 4096 Jun 5 12:53 .ssh drwxr-xr-x 2 abs4 csrv 4096 May 2 16:06 .subversion drwxr-xr-x 3 abs4 csrv 4096 Aug 14 14:50 tmp -rw------- 1 abs4 csrv 488 Sep 8 15:48 .Xauthority -bash-4.1$
ls without any options or arguments lists the name of the files and directories in the current working directory. In this example above it is hard to see which names refer to files or directories. We will show you how to do this later on. The next example displays the directory in the long format using the '-l' option, much more information is displayed about the directories and files. The '-a' option shows all files, filenames starting with '.' are usually hidden from display. We can combine options to give more detail.
ls can take arguments as well. When specifying an argument ls displays the information for that file or directory.
-bash-4.1$ ls Experiments architest.dtr OLD simple_verbs.dtr~ architest.dtr~ OLD CART simple verbs_to_Dunstan.txt exploded.csv ordered simple verbs_to_Dunstan.xlsx Latest CART phon.csv simple_verbs_to.txt mian.rp phonsorted simple_verbs.txt NotesAboutInfixPredictions.pdf simple_verbs.dtr simpverbsort -bash-4.1$ ls -l Experiments total 368 -rw-r--r-- 1 abs4 csrv 919 May 23 11:08 architest.dtr -rw-r--r-- 1 abs4 csrv 909 May 23 11:08 architest.dtr~ -rw-r--r-- 1 abs4 csrv 3613 May 23 11:08 exploded.csv drwxr-xr-x 2 abs4 csrv 4096 May 23 11:34 Latest CART -rw-r--r-- 1 abs4 csrv 4019 May 23 11:28 mian.rp -rw-r--r-- 1 abs4 csrv 193602 May 23 11:08 NotesAboutInfixPredictions.pdf drwxr-xr-x 2 abs4 csrv 4096 May 23 11:08 OLD drwxr-xr-x 2 abs4 csrv 4096 May 23 11:08 OLD CART -rw-r--r-- 1 abs4 csrv 3613 May 23 11:08 ordered -rw-r--r-- 1 abs4 csrv 6217 May 23 11:08 phon.csv -rw-r--r-- 1 abs4 csrv 6217 May 23 11:08 phonsorted -rw-r--r-- 1 abs4 csrv 17663 May 23 11:08 simple_verbs.dtr -rw-r--r-- 1 abs4 csrv 17647 May 23 11:08 simple_verbs.dtr~ -rw-r--r-- 1 abs4 csrv 8058 May 23 11:08 simple verbs_to_Dunstan.txt -rw-r--r-- 1 abs4 csrv 30416 May 23 11:08 simple verbs_to_Dunstan.xlsx -rw-r--r-- 1 abs4 csrv 4696 May 23 11:08 simple_verbs_to.txt -rw-r--r-- 1 abs4 csrv 17525 May 23 11:08 simple_verbs.txt -rw-r--r-- 1 abs4 csrv 17647 May 23 11:08 simpverbsort -bash-4.1$ ls -ld Experiments drwxr-xr-x 5 abs4 csrv 4096 May 23 11:34 Experiments -bash-4.1$ ls -l afile -rw-r--r-- 1 abs4 csrv 0 Sep 8 16:26 afile -bash-4.1$ ls /usr appl cmsmigratetest lfa phpweb src vleexam archive cvs lib puppet systems vle-sysadmin backups datasets lib64 puppetdev tmp webmisc bin etc libexec researchcomp transfer work central games local rlink userfs yorkroot cert idm logfiles sbin UserFS yorkweb cmsmedia include mirror scratch vle yorkwebtest cmsmediatest java misc secbuffer vle-arch cmsmigrate kerberos opapp share vle-eldt -bash-4.1$
Using a directory name as an option causes ls to list the contents of the directory. To list the attributes of the directory use the '-d' option. You can use a pathname as the argument.
Creating, moving and copying files and directories
You can create directories, move or copy files or directories to other locations in the filesystem using the mkdir (make directory) mv (move) and cp (copy) commands.
-bash-4.1$ ls afile bin Desktop Experiments jobs new-dir bfile Chemistry examples intel logs tmp -bash-4.1$ mv afile new-dir -bash-4.1$ cp bfile new-dir -bash-4.1$ ls bfile Chemistry examples intel logs tmp bin Desktop Experiments jobs new-dir -bash-4.1$ ls new-dir afile bfile -bash-4.1$ mv new-dir/afile . -bash-4.1$ ls afile bin Desktop Experiments jobs new-dir bfile Chemistry examples intel logs tmp -bash-4.1$
This example creates a new directory, 'new-dir', We then move the file 'afile' to it and create a copy of 'bfile'. We then move the file 'afile' back to our current working directory. Note the use of the '.' file to reference the current working directory. We can use full or partial pathnames to reference other parts of the file system.
Copying a directory is a little more complicated and the directory may contain files and directories. We use the '-r' command to cp to do this.
-bash-4.1$ ls afile bin Desktop Experiments jobs tmp bfile Chemistry examples intel logs -bash-4.1$ ls tmp icc-start ifort-start logs mpi-stop start test icc-stop ifort-stop mpi-start new-dir stop -bash-4.1$ cp tmp/test . cp: omitting directory `tmp/test' -bash-4.1$ cp -r tmp/test . -bash-4.1$ ls afile bin Desktop Experiments jobs test bfile Chemistry examples intel logs tmp -bash-4.1$ ls test test.c test.cpp test.f test.f90 test.x -bash-4.1$
In this example we wish to copy the contents of the directory 'tmp/test' into the current directory. cp will not copy a directory. we have to use the '-r' (recursive) option to tell cp to copy all files and directory within the directory.
Deleting files and directories
The rm (remove) command is used to delete files.
-bash-4.1$ ls afile bin Desktop Experiments jobs test bfile Chemistry examples intel logs tmp -bash-4.1$ rm bfile -bash-4.1$ ls afile Chemistry examples intel logs tmp bin Desktop Experiments jobs test -bash-4.1$
To deleting directories use the rmdir (remove directory) command.
-bash-4.1$ rmdir dira rmdir: failed to remove `dira': Directory not empty -bash-4.1$ rm -r dira -bash-4.1$ ls afile Chemistry dirb Experiments jobs test bin Desktop examples intel logs tmp -bash-4.1$ rm -ri dirb rm: descend into directory `dirb'? y rm: descend into directory `dirb/dirb'? y rm: remove regular empty file `dirb/dirb/afile'? y rm: remove directory `dirb/dirb'? y rm: remove regular empty file `dirb/afile'? y rm: remove directory `dirb'? y -bash-4.1$
rmdir will only remove empty directories. To remove a directory and all it's contents use the rm -r (recursive) option to the rm command. To be safe and check the files before you remove them use -ri (recursive and interactive) options.
Editing and displaying the contents of files
Text Editors
- vi and vim
- emacs
- nano
Displaying the contents of files
The commands cat (concatenate files) and more displays the contents of file.
-bash-4.1$ cat snark2 The Hunting of the Snark By Lewis Carroll Fit the First The Landing "Just the place for a Snark!" the Bellman cried, As he landed his crew with care; Supporting each man on the top of the tide By a finger entwined in his hair. "Just the place for a Snark! I have said it twice: That alone should encourage the crew. Just the place for a Snark! I have said it thrice: What I tell you three times is true." -bash-4.1$ more snark The Hunting of the Snark By Lewis Carroll Fit the First The Landing "Just the place for a Snark!" the Bellman cried, As he landed his crew with care; Supporting each man on the top of the tide By a finger entwined in his hair. "Just the place for a Snark! I have said it twice: That alone should encourage the crew. Just the place for a Snark! I have said it thrice: What I tell you three times is true." The crew was complete: it included a Boots— A maker of Bonnets and Hoods— A Barrister, brought to arrange their disputes— And a Broker, to value their goods. A Billiard-marker, whose skill was immense, --More--(2%)
The cat command displays all the test in the users file on the screen. This can prove difficult to read if there are large amounts of text. The more command paginates the text and displays portions of it on the screen. The user can use character command to move through the file:
- SPACE - display the next screen of text
- q - quit displaying the file
- b - skip backwards through he file
- /pattern - search for text in the file
Files and directory permissions
Groups are provided to manage sets of users and control access to fie and directories. All users belong to a default group and may be a member of other groups.
-bash-4.1$ groups csrv pfs17 pfs34 csys cvssys itsilg rhpcs sshfix git5 git6 git7 elecclust yc-gauss yc-install yc-colum -bash-4.1$ ls -l total 332 -rw-r--r-- 1 abs4 csrv 0 Sep 8 16:26 afile drwxr-xr-x 2 abs4 csrv 4096 Jun 24 09:39 bin drwxr-xr-x 3 abs4 csrv 4096 Jun 6 09:23 Chemistry drwxr-sr-x 2 abs4 elecclust 4096 Mar 11 10:53 Desktop drwxr-xr-x 3 abs4 csrv 4096 Jun 30 12:21 examples drwxr-xr-x 5 abs4 csrv 4096 May 23 11:34 Experiments drwxr-xr-x 3 abs4 csrv 4096 Aug 14 12:26 intel drwxr-sr-x 3 abs4 elecclust 4096 Aug 15 12:49 jobs drwxr-xr-x 2 abs4 csrv 266240 Aug 15 13:48 logs -rw-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -rw-r--r-- 1 abs4 csrv 433 Sep 9 10:27 snark2 drwxr-xr-x 2 abs4 csrv 4096 Sep 8 17:08 test drwxr-x--- 5 abs4 csrv 4096 Sep 8 17:01 tmp -bash-4.1$
The groups command displays which groups you are a member of. Each file and directory you create will be owned by you and be potentially accessible to a group. In the above example the file 'afile' is owned by 'abs4' and is accessible to the 'csrv' group.
There is a 'special' group sometimes called world, or other, which contains all users of the system.
In the above example the first column of the directory listing shows the permissions of the files. These permissions control who is allowed acces the files and directories. There are three categories of user who can has potential rights to access the files - owner, group, world. The access rights to the files are displayed in the form of a sequence of letters like 'drwxr-xr-x'. The meaning is:
- d - if present this is a directory, otherwise it is a file
- the following 3 letters are in three groups and state the access permissions for the owner, group, world users
- w - the file can be written to
- r - the file can be read
- x - if a file it can be executed, if a directory it can be accessed
- example - drwxr-x--- 5 abs4 csrv 4096 Sep 8 17:01 tmp
- this is a directory
- the owner, abs4, can read, write and access the directory
- members of the group, elecclust, can read and access the directory, they can not creat files in the directory
- all other users do not have any access to the directory
To change file permissions use the chmod command.
-bash-4.1$ ls -l snark -rw-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark-bash-4.1$ chmod go-rwx snark -bash-4.1$ ls -l snark -rw------- 1 abs4 csrv 25678 Sep 9 10:24 snark
The chmod command has the form:
chmod <mode> <file>
mode takes the form of:
[ugoa...][[+-=][perms...]...]
- u = user (owner)
- g = group
- o = other (world)
- a = all (user, group, other)
- + add permission
- - remove permission
- = explicitly set permission
- w = can be written to
- r = can be read
- x = can be executed if a file, if a directory it can be accessed
-bash-4.1$ ls -l snark -rw-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod a+rx snark -bash-4.1$ ls -l snark -rwxr-xr-x 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod go-rwx snark -bash-4.1$ ls -l snark -rwx------ 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod u-w snark -bash-4.1$ ls -l snark -r-x------ 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod a=r snark -bash-4.1$ ls -l snark -r--r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod u=w snark -bash-4.1$ ls -l snark --w-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ chmod u+r snark -bash-4.1$ ls -l snark -rw-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$
To change the group of a file use the command chgrp <groupname> <filename>.
-bash-4.1$ ls -l snark -rw-r--r-- 1 abs4 csrv 25678 Sep 9 10:24 snark -bash-4.1$ groups csrv pfs17 pfs34 csys cvssys itsilg rhpcs sshfix git5 git6 git7 elecclust yc-gauss yc-install yc-colum -bash-4.1$ chgrp rhpcs snark -bash-4.1$ ls -l snark -rw-r--r-- 1 abs4 rhpcs 25678 Sep 9 10:24 snark -bash-4.1$