6. Introduction to Linux Command Line


If you are new to using a Linux command line, this overview will get you started. The command line lets you do things without the graphical interface that you might be used to doing by double clicking with your mouse via a graphical interface such as a File Browser on your laptop.

Some examples are navigating the file system, creating new files, opening files, unzipping compressed files, and seeing the content of folders (called “directories” on a Linux file system), creating new directories, moving files around and a whole slew of other useful things.

Linux Shell

A shell is a program that takes input from the keyboard typed by a user on the command line and gives them to the operating system (in our case Ubuntu 18.04.5) to execute. There are many shells such as bash, zsh, tcsh, etc. When you login to the Yens via ssh command you are in the interactive login shell.

See what shell you are running:

echo $0

When I run the above command, I see:

-bash

which means my login shell is bash but you might see another shell by default.

Home Directory

Before we explore common and useful Linux commands, let’s talk about the file system on the Yens. The Linux file system has a tree structure with the so called root directory (/) being, well, the root, of the tree structure. The absolulte paths mean that you specify the full path to a directory or a file starting with the root directory.

Every user on the Yens has a home directory. This is where you are when you login to the system. Check the absolute path with:

pwd

This will print your working directory (where <SUNetID> is your SUNet ID):

/home/users/<SUNetID>

To see this schematically, here is a visualization of the home directory on the file system:

The squares with ... in them indicate more directories that are not shown in the graph.

The path to your home directory is stored in $HOME environment variable. To see it, run:

echo $HOME

The echo command prints out the environment variable $HOME which stores the path to your home directory (where <SUNetID> is your SUNet ID):

/home/users/<SUNetID>

The home directory is not for storing large files or outputting large files while working on a project. It is a good place to store small files like scripts and text files. Your home directory storage space is capped at 25 G.

To see how much space you have used in your home directory, run:

gsbquota

You should see your home directory usage:

/home/users/<SUNetID>: currently using X% (XG) of 25G available

where X% and XG will be actual percent used and gigabytes used, respectively.

Project Directory

If you are a GSB researcher that is interested in starting a new project on the Yens, please complete and submit DARC’s new project request form. This form allows you to estimate disk usage, and specify any collaborators that should be added to the shared access list. IFS project access is granted by workgroups.

The project directories live on IFS and currently do not have space quotas. However, we ask that you be responsible and delete what you no longer need, intermediate files, etc because the sytem is currently 90% full.

Schematically, we can visualize the path to the project directory as follows:

The absolute path to your project space is:

/ifs/projects/<your-project-dir>

where <your-project-dir> is the name of your project directory (created for you by the DARC team after the request for a new project space form is filled out).

Common Linux Commands

Let’s say we want to navigate to our project directory. To “change directories”, use cd command:

cd /ifs/projects/<your-project-dir>

Every directory has a link to its parent directory. To navigate from a directory one level up, use:

cd ..

To go two levels up, use:

cd ../..

If you want to return to the directory where you previously were, use:

cd -

To come back to your home directory, type cd without any options:

cd

Another shortcut for home is ~. Again, you can nagivate to home from anywhere on the file system with:

cd ~

Or you can use the environment variable $HOME to return to the home directory:

cd $HOME

Experiment with cd command by navigating to different places on the file system then returning home by using the above commands.

Creating new files / directories

To create a new directory inside your home directory:

cd

mkdir test

By typing cd first, you will return to your home directory. The mkdir command makes a new empty subdirectory called “test” inside your home directory.

To create an empty text file, you can use the touch command:

touch test.txt

Listing directory contents

To list what files are inside current working directory, use:

ls

You should see test.txt file and test subdirectory listed if you ran this from your home directory.

test  test.txt

It is useful to list files in reverse chronological order with more details than ls gives you. To list files in “long format”, use:

ls -ltr

You should see:

total 4.5K
drwxr-xr-x 2 nrapstin operator 0 Nov 17 09:01 test
-rw-r--r-- 1 nrapstin operator 0 Nov 17 09:01 test.txt

The long format lists details about the directories and files. The first line before the long listing lists a total sum for all the file sizes (4.5K in this example).

The long format in ls command

  • The first letter indicates if the object is a direcotry (d), a regular file (-), a symbolic link (l) or other type of file.
  • The next three letters list permissions for the user who owns that file or directory: rwx for read, write and execute permissions. For example, the test subdirectory has user permissions as rwx which means the user who owns that file listed in the third column has full permissions to read, write and execute this directory.
  • The next three letters list group permissions: whether the group members have read, write or execute permissions (rwx) for the file or directory. For example, the test subdirectory has group permissions as r-x which means the group members can read and execute in that directory but they are not allowed to write there (only the owner user has write permissions).
  • The last three letters list others permissions: whether all other users have read, write or execute permissions. For example, the test subdirectory has other permissions as r-x (read and execute).
  • After the permissions the next column lists the number of hard links. So, the text file test.txt has one hard link and the directory test has two links inside (two subdirectories inside test directory). Empty directories create 2 subdirectories inside upon creation of the directory - one for itself . and one for its parent ...
  • The third and fourth columns list the user and group who own the directory or file. You should see your SUNetID and your group.
  • The next column lists file size in bytes. You should see 0 bytes for empty files and directories.
  • For each object the time of last modification is listed. Finally, the last columns lists the file or directory name.

Getting help

All Linux commands have manual pages that could be accessed with man, help, or info commands:

man ls

The manual explains the command and lists any optional arguments. Press Space bar to scroll through the manual and q to exit.

ls --help

Note that help is a bash command so if you are not using bash as your shell, it will not work for you.

You can also use info to view text-only information about the Linux command (with hyperlinks):

info ls

Moving files

You can copy and move files around the file system (where you have access permissions).

You can copy a file with cp command. For example, let’s make a copy of the test.txt file inside the test directory.

cp test.txt test

ls -ltr test

When we list files inside test directory with ls -ltr test, the newly copied file test.txt is listed there.

Note that if you already have a file called test.txt inside a directory where you are copying the file with the same name, the file will be overwritten (with no warning).

We can also move files and directories with mv command.

mv test.txt test
ls -ltr

We have moved the file test.txt inside test directory.

You can match several files to copy or move with the wild character (*) which means match anything. For example, if you have several text files all ending in .txt format then we can move all of them together into the test direcotry with:

mv *.txt test

Similarly, you can move a directory within another directory:

mv my_dir /path-to-move-to

This works for cp as well but cp requires a recursive flag r to copy directories:

cp -r my_dir /path-to-copy-to

Deleting files / directories

To delete a file or directories, use rm command.

rm test/text.txt

that will remove text.txt file inside test directory.

If you want to remove a directory, supply a recursive flag -r. You can also match names to remove all files that match a certain criteria with * character.

Downloading files from the web

Sometimes you want to download a file from the web. Instead of locally downloading then transferring the file to the Yens, we can download the file directly to the Yen. First, navigate where you want the file to be downloaded with cd command. Then use wget to download the file. Usually you have to copy the link to the file/folder on the web that you are trying to download.

For example, let’s say we want to install Rmpi R package from source. We would navigate to the CRAN website and copy the web link to the package source file. Then, run

wget https://cran.r-project.org/src/contrib/Rmpi_0.6-9.1.tar.gz

which will download the package source tarball to your Yen directory. If everything worked, you should see:

--2021-03-09 17:54:27--  https://cran.r-project.org/src/contrib/Rmpi_0.6-9.1.tar.gz
Resolving cran.r-project.org (cran.r-project.org)... 137.208.57.37
Connecting to cran.r-project.org (cran.r-project.org)|137.208.57.37|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 105893 (103K) [application/x-gzip]
Saving to: ‘Rmpi_0.6-9.1.tar.gz’

Rmpi_0.6-9.1.tar.gz           100%[===============================================>] 103.41K   209KB/s    in 0.5s

2021-03-09 17:54:28 (209 KB/s) - ‘Rmpi_0.6-9.1.tar.gz’ saved [105893/105893]

When you list the files in the current directory you will see Rmpi_0.6-9.1.tar.gz present.

Compressing and uncompressing files / directories

When you download the files or transfer them, the files could be compressed with zip and will have .zip extension. Transferring compressed files is recommended for faster transfer speeds.

To unzip the folder, run:

unzip test.zip

tar is a Unix utility for collecting files together into one archive file (commonly called a tarball). The name tar comes from “tape archive” and was used to archive a series of file objects together (as one collection or archive).

To untar the tarball (x to extract, v for verbose and f is the name of the tarball), run:

tar -xvf my_tar.tar

Commonly, the tarball may also be compressed with gunzip (having an extension .tar.gz or .tgz) or bzip2 (with .bz2 extension). Then to untar, add the unzip option as well (-z). Let’s practice with the downloaded Rmpi tarball:

tar -zxvf Rmpi_0.6-9.1.tar.gz

Uncompressing the tarball creates a new direcotry Rmpi with the source code for this R package.

Finally, let’s clean up the home directory by removing all the files and directories we made in this tutorial. Using -f option will not prompt for confirmation and will force delete all of the files, so use it with caution. This is a good thing to do when you are working on real projects too. Keep your working space tidy!

rm *.gz

rm -rf Rmpi test

If you want to learn more about Linux command line, Software Carpentry is an excellent resource.