Compute Questions

Software Questions

Data Questions

How Do I Run a Job that Continues after I Log Out?

Generally speaking, you can use the screen command, described in more detail below, to continue having your scripts run on the Yens after logging out, but a very important first question to ask is:

“Am I accessing anything stored in AFS (/afs/*** or /afs-home/***) space?”

If the answer to that question is “Yes”, then you will likely run into authentication issues. Although AFS space is frequently used within the computing infrastructure of the rest of the University, the biggest issue is that it requires Kerberos authentication tokens in order to access files and they time out after 25 hours. See here for UIT’s explanation. If you have jobs that access AFS space and expect to run for longer than that period of time, you will run into a slew of cryptic errors that are very likely related to your token expiring.

On the Yen servers, the best storage space for your scripts and data is ZFS.

screen

screen allows you to set up multiple virtual sessions within your terminal session. This can be useful for running multiple processes at the same time, but it can also be used to leave processes running after you log out of any Yens server.

To set up a screen, log onto the Yens through a terminal window and then type:

screen -S <sessionname>

This will take you to a new terminal with a blank view and a command prompt at the top of the page. Here, you can start issuing commands as usual and in particular, you can start a script that you expect to run for a long time.

Once your script is going, you can exit out of this screen session with the keyboard shortcut ctrl + a + d, which will return you to your original terminal. At this point, you can list out any screen sessions that you are running on the particular Yen server you are logged onto by typing:

screen -ls

If you exit out of the server here, you can be assured that the scripts in your screen sessions will continue to run, granted they are accessing ZFS space.

To return to any existing screen session, you simply need to type:

screen -r <sessionname>

If you are actively printing to the console in your script, you may notice that you cannot scroll up through the log of print statements when you are in a screen. In order to activate the “scroll mode”, you can use the keyboard shortcut ctrl + a and then hit esc. In this mode, you can peruse through your print statements with the up and down arrow keys. Once you are finished scrolling, you can exit out of the mode by hitting the q or esc keys.

You may run into a situation where you are disconnected or timed out from the Yens while you are attached to a screen session. When you log back onto the Yens and try to reconnect to that session again, you may find that you cannot re-attach to the screen, since it’s already “Attached”. In order to return to that session, you can just use this command:

screen -d -r <sessionname>

This detaches the previous connection to that screen session and then returns you to the screen.

Finally, once you are finished with a screen session, all you have to do is type:

exit

Still want AFS?

If you still require the usage of AFS space in your scripts, then you will need to take extra measures to ensure that your script is not disrupted due to expiring authentication tokens.

In this case, you can use krenew to keep your AFS authentication alive for up to one week. However, if you need to run jobs longer than this, you should probably use IFS space.

krenew prefaces any command you would normally run. As an example, if you were running a MATLAB script, you would run:

krenew -t -- matlab -nodisplay -r my_prog