Keep processes running in the background and even after SSH session disconnects

Problem

I started running a computation-intensive python script on an SSH session. But the session kept disconnecting, and I couldn't wait so long to the end. I lost all my results all the time.

Solutions

The solutions are all about running the script in the background without interrupting. And nohup, disown, screen, tmux, reptyr, jobs, bg and fg are involved.

1. If you don't disconnect the session, jobs, bg and fg work fine

• Ctrl+z to pause the process
• bg to run it in the background
• fg to bring it back if you want

You can keep switching as you want. jobs is to show the running processes in the current session.

2. If you need to disconnect the session, disown works

disown -h

disown -h disown all jobs running. So it'll keep running after disconnection.

Warning: it'll still stop working if it prints out something. > can be used to stopping print

3. If you need to disconnect the session but you need the output, nohup works

nohup python3 code.py &

The & makes it run in the background so we can keep interacting with the terminal. And now you are free to close the session.

If you want to bring it back to the foreground, use fg. If you want to bring it back to background, use Ctrl+z. But it's stopped after Ctrl+z. bg is to make it run again. The output will be saved in a file.

4. If you want to bring it back after reconnecting, you need to create a screen or tmux first (Best solution)

For nohub, once you log off the ssh session, it's hard to bring it back again. So you have to wait until it ends.

screen works better to solve this problem.  You don't even need nohup here.

• ssh into your remote session. Type screen. Then start the process you want.

• Press Ctrl+A then Ctrl+D. This will "detach" your screen session but leave your processes running. You can now log out of the remote session.

• If you want to come back later, log on again screen -rand type screen -r This will "resume" your screen session, and you can see the output of your process.

screen -r is to list all screens so that we can identify the status of each screen.

5. If you forgot screen or you want to bring it back after disown, reptyr works

• start a screen
• reptyr the process to it