SystemRescueCd is a Linux system rescue disk available as a bootable CD-ROM or USB stick for administrating or repairing your system and data after a crash. It aims to provide an easy way to carry out admin tasks on your computer, such as creating and editing the partitions of the hard disk. It comes with a lot of linux software such as system tools (parted, partimage, fstools, …) and basic tools (editors, midnight commander, network tools). It requires no installation since you just have to boot on the CD-ROM. It can be used to perform admin tasks on both linux servers, linux desktops or windows boxes. The kernel supports most of the important file systems (ext2/ext3/ext4, reiserfs, reiser4, btrfs, xfs, jfs, vfat, ntfs, iso9660), as well as network filesystems (samba and nfs).

Download the Linux Rescue Disk Here

http://www.sysresccd.org/Main_Page

Booting from SystemRescueCd

A successfully boot of SystemRescueCd presents the first screen with SystemRescueCd written in ASCII art.

Press F2/F3/F4/F5/F6 and read advanced boot instructions.

Press Enter to boot with the default options.

There are two parts in the boot command: <boot-image> <boot-options>. For example you may want to boot with rescue64 as boot-image and docache setkmap=uk as boot-options. Use spaces between options. Additional options are at Booting the CD-ROM
Main boot images

There are four main boot images with SystemRescueCd. The differences are detailed in the kernel page

* rescuecd The default for 32bit systems, with Framebuffer disabled.
* altker32 This is an alternative kernel for 32bit systems. Boot this kernel if you have problems with rescuecd. altker32 was named vmlinuz2

* rescue64 Default 64 bit kernel. Use it if you want to chroot to a 64bit linux system installed on your hard disk, or to run 64 bit programs. This kernel is able to boot SystemRescueCd from the cdrom with 32bit programs, and requires a processor with 64bit instructions (amd64 / em64t).
* altker64 This is an alternative kernel for 64bit systems. Boot with this kernel in case you have problems with rescue64.

Main boot options

Here are the most important boot options:

* docache: copy all the files it needs to RAM . This permits the sysrescuecd to be ejected and insert another disc in the drive. The system runs faster. Requires at least 256MB of memory.
* setkmap=cc: During the boot process, the system asks for the kind of keyboard. Use this option to avoid that question. Replace ‘cc’ with the keyboard you have: ‘us’ for USA, ‘uk’ for british, ‘de’ for german, …
* root=/dev/idxn: the root=<device> option boots an existing linux system. For example, if you have a linux Gentoo installed on /dev/sda6 , type rescuecd root=/dev/sda6 and Gentoo Linux will be started instead of the system on the CD. Use a 64bit kernel if your system has 64bit programs. For instance, you can boot a 64bit linux system installed on /dev/sda6 with rescue64 root=/dev/sda6. From SystemRescueCd-1.0.4, this option works with LVM disks, so you can use something like rescuecd root=/dev/VolGroup00/LogVol00. root=auto will scan the block devices of the computer to find a linux system. The first linux system found on the disks will be started. This allows starting the system from the CD in the event there is a problem with your boot loader or with your installed nkernel . See for more details.
* ide=nodma or all-generic-ide: use these options if there is a problem related to the hard disk, for instance if the kernel boot process hangs on a driver related to the storage.
* doxdetect or forcevesa: use these options if you cannot get the graphical environment to work when you type startx in the shell prompt.
* acpi-off / noapic / irqpool: use these options if you have any problem when the kernel boots: if it hangs on a driver or if it crashes, …

Additional Programs

* Some programs included are normally booted from their own floppy. Press F2 to display the list of the these floppy disk images. For instance memtest runs an extensive memory test. ntpass allows you to change the password of any windows user accounts including the administrator account.

Working in the console mode

Mount partitions in order to troubleshoot a Linux or a Windows system installed on your disk. You can mount linux filesystems (ext2fs, ext3fs, reiserfs, reiser4, jfs, xfs) You can backup/restore data or operating system files.

Midnight Commander (type mc )is able to copy/move/delete/edit files and directories. The vim and qemacs editors can be used to edit files.

Six virtual consoles are available. Press Alt+F1 for the first virtual console, Alt+F2 for the second one, …

Working in the graphical environment

If you need graphical tools (such as GParted) start the graphical environment by typing wizard . There are two graphical environments: Xorg and Xvesa. You should try Xorg first. If Xorg fails to start, run wizard again and choose Xvesa which should always work. The graphical environment allows you to work with GParted (partition manager), to use graphical editors (Geany or GVim), to browse the web with Firefox and use terminals like xfce-terminal or mrxvt.

Setting up your network

With SystemRescueCd you can use the network. It’s useful to make a backup over the network, download files, work remotely using ssh, telnet or access files that are shared on a Unix server (with NFS)

The most convenient way to configure your network is to type net-setup at shell prompt. You can also use the following command lines to configure a network interface by hand:

If your system has supported hardware, the network interface card (NIC) was auto-detected, and the driver loaded. The interface needs to be assigned an IP address and a default gateway. To use use dynamic configuration, dhcpcd eth0. Use ifconfig -a to display the IP address the DHCP server leased to the interface. To assign a specific static IP address, enter ifconfig eth0 192.168.10.17 . Next the default route is configured. For example, for an interface at address 192.168.10.17 connected to a gateway at 192.168.10.2 enter: route add default gw 192.168.10.2.

If you use Linux, you already have access to extremely powerful tools for creating custom backup solutions. The solutions in this article can help you perform simple to more advanced and secure network backups using open source tools that are part of nearly every Linux distribution.

Simple backups

This article follows a step-by-step approach that is quite straightforward once you follow the basic steps.

Let’s begin with a simple, yet powerful archive mechanism on our way to a more advanced distributed backup solution. Let’s examine a handy script called arc, which will allow us to create backup snapshots from a Linux shell prompt.

Listing 1. The arc shell script

#!/bin/sh
tar czvf $1.$(date +%Y%m%d-%H%M%S).tgz $1
exit $?

The arc script accepts a single file or directory name as a parameter and creates a compressed archive file with the current date embedded into the resulting archive file’s name. For example, if you have a directory called beoserver, you can invoke the arc script, passing it the beoserver directory name to create a compressed archive such as: beoserver.20040321-014844.tgz

The use of the date command to embed a date and timestamp helps to organize your archived files. The date format is Year, Month, Day, Hour, Minutes, and Seconds — although the use of the seconds field is perhaps a bit much. View the man page for the date command (man date) to learn about other options. Also, in Listing 1, we pass the -v (verbose) option to tar. This causes tar to display all of the files it’s archiving. Remove the -v option if you’d like the backup to proceed silently.

Listing 2. Archiving the beoserver directory

$ ls
arc beoserver
$ ./arc beoserver
beoserver/
beoserver/bookl.dat
beoserver/beoserver_ab_off
beoserver/beoserver_ab_on
$ ls
arc beoserver beoserver.20040321-014844.tgz

Advanced backups

This simple backup example is useful; however, it still includes a manual backup process. The industry’s best practices recommend backing up often, onto multiple media, and to separate geographic locations. The central idea is to avoid relying entirely on any single storage media or single location.

We’ll tackle this challenge in our next example, where we’ll examine a fictitious distributed network, illustrated in Figure 1, which shows a system administrator with access to two remote servers and an offsite data storage server.

Figure 1. Distributed network

The backup files on Server #1 and #2 will be securely transmitted to the offsite storage server, and the entire distributed backup process will occur on a regular basis without human intervention. We’ll use a set of standard tools that are part of the Open Secure Shell tool suite (OpenSSH), as well as the tape archiver (tar), and the cron task scheduling service. Our overall plan will be to use cron for scheduling, shell programming and the tar application during the backup process, OpenSSH secure shell (ssh) encryption for remote access, and authentication, and secure shell copy (scp) to automate file transfers. Be sure to review each tool’s man page for additional information.

Secure remote access using public/private keys

In the context of digital security, a key is a piece of data which is used to encrypt or decrypt other pieces of data. The public and private key scheme is interesting because data encrypted with a public key can only be decrypted with the associated private key. You may freely distribute a public key so that others can encrypt the messages they send you. One of the reasons that public/private key schemes have revolutionized digital security is because the sender and receiver don’t have to share a common password. Among other things, public/private key cryptography has made e-commerce and other secure transactions possible. In this article, we’ll create and use public and private keys to create a highly secure distributed backup solution.

Each machine involved in the backup process must be running the OpenSSH secure shell service (sshd) with port 22 accessible through any intermediate firewall. If you access remote servers, then there is a good chance you’re already using secure shell.

Our goal will be to provide machines with secure access without requiring the need to manually provide passwords. Some people think that the easiest way to do this is to set up password-less access: do not do this. It is not secure. Instead, the approach we’ll use in this article will take perhaps an hour of your time, set up a system which gives all the convenience of “passphraseless” accounts — but is recognized as being highly secure.

Let’s begin by ensuring that OpenSSH is installed and proceed to check its version number. At the time this article was written, the latest OpenSSH release was version 3.8, released on February 24, 2004. You should consider using a recent and stable release, and at the very least use a release which is newer than version 2.x. Visit the OpenSSH Security page for details regarding older version-specific vulnerabilities (see the link in Resources later in this article). At this point in time, OpenSSH is quite stable and has proven to be immune to many of the vulnerabilities which have been reported for other SSH tools.

At a shell prompt, type ssh with the capital V option to check the version number:

$ ssh -V
OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0×0090701f

If ssh returns a version number greater than 2.x, the machine is in relatively good shape. However, it is recommended that you use the latest stable releases of all software, and this is especially important for security-related software.

Our first step is to log in to the offsite storage server machine using the account, which will have the privilege of being able to access servers 1 and 2 (see Figure 1).

$ ssh accountname@somedomain.com

Once logged on to the offsite storage machine, use the ssh-keygen program to create a public/private key pair using the -t dsa option. The -t option is required, and is used to specify the type of encryption key we’re interested in generating. We’ll use the Digital Signature Algorithm (DSA), which will enable us to use the newer SSH2 protocol. See the ssh-keygen man page for more details.

During the execution of ssh-keygen, you’ll be prompted for the location where the ssh keys will be stored before you’re asked for a passphrase. Simply press enter when asked where to save the key and the ssh-keygen program will create a hidden directory called .ssh (if one doesn’t already exist) along with two files, a public and private key file.

An interesting feature of ssh-keygen is that it will allow you to simply press enter when prompted for a passphrase. If you don’t supply a passphrase, then ssh-keygen will generate keys which are not encrypted! As you can imagine, this isn’t a good idea. When asked for a passphrase, make sure to enter a reasonably long string message which contains alphanumeric characters rather than a simple password string.

Listing 3. Always choose a good passphrase

[offsite]:$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/accountname/.ssh/id_dsa):
Enter passphrase (empty for no passphrase): (enter passphrase)
Enter same passphrase again: (enter passphrase)
Your identification has been saved in /home/accountname/.ssh/id_dsa.
Your public key has been saved in /home/accountname/.ssh/id_dsa.pub.
The key fingerprint is:
7e:5e:b2:f2:d4:54:58:6a:fa:6b:52:9c:da:a8:53:1b accountname@offsite

Because the .ssh directory which ssh-keygen creates is a hidden “dot” directory, pass the -a option to the ls command to view the newly created directory:

[offsite]$ ls -a
. .. .bash_logout .bash_profile .bashrc .emacs .gtkrc .ssh

Enter the hidden .ssh directory and list the contents:

[offsite]$ cd .ssh
[offsite]$ ls -lrt
id_dsa id_dsa.pub

We now have a private key (id_dsa) and a public key (id_dsa.pub) in the hidden .ssh directory. You can examine the contents of each key file using a text editor such as vi or emacs, or simply by using the less or cat commands. You’ll notice that the contents consist of alphanumeric characters encoded in base64.

Next, we need to copy and install the public key on servers 1 and 2. Do not use ftp. Rather, use the secure copy program to transmit the public keys onto each of the remote machines:

Listing 4. Installing the public keys on the remote servers

[offsite]$ scp .ssh/id_dsa.pub accountname@server1.com:offsite.pub
accountname@server1.com’s password: (enter password, not new
passphrase!)
id_dsa.pub 100% |*****************************| 614 00:00

[offsite]$ scp .ssh/id_dsa.pub accountname@server2.com:offsite.pub
accountname@server2.com’s password: (enter password, not new
passphrase!)
id_dsa.pub 100% |*****************************| 614 00:00

After we install the new public keys, we’ll be able to sign on to each machine using the passphrase we specified when creating the private and public keys. For now, log in to each machine and append the contents of the offsite.pub file to a file called authorized_keys, which is stored in each remote machine’s .ssh directory. We can use a text editor or simply use the cat command to append the offsite.pub file’s contents onto the authorized_keys file:

Listing 5. Add offsite.pub to your list of authorized keys

[offsite]$ ssh accountname@server1.com
accountname@server1.com’s password: (enter password, not new
passphrase!)
[server1]$ cat offsite.pub >> ./ssh/authorized_keys

The next step involves employing a bit of extra security. First, we change the access rights for the .ssh directory so that only the owner has read, write, and execute privileges. Next, we’ll make sure that the authorized_keys file can only be accessed by the owner. And finally, we’ll remove the previously uploaded offsite.pub key file, since it’s no longer required. It’s important to ensure that access permissions are properly set because the OpenSSH server may refuse to use keys which have non-secure access rights.

Listing 6. Changing permissions with chmod

[server1]$ chmod 700 .ssh
[server1]$ chmod 600 ./ssh/authorized_keys
[server1]$ rm offsite.pub
[server1]$ exit

After completing the same process on server2, we are ready to return to the offsite storage machine to test the new passphrase type access. >From the offsite server you could type the following:

[offsite]$ ssh -v accountname@server1.com

Use the -v, or verbose flag option, to display debugging information while verifying that your account is now able to access the remote server using the new passphrase rather than the original password. The debug output displays important information which you might not otherwise see, in addition to offering a high level view of how the authentication process works. You won’t need to specify the -v flag on subsequent connections; but it is quite useful to do so while testing a connection.

Automating machine access using ssh-agent

The ssh-agent program acts like a gatekeeper, securely providing access to security keys as needed. Once ssh-agent is started, it sits in the background and makes itself available to other OpenSSH applications such as ssh and scp programs. This allows the ssh program to request an already decrypted key, rather than asking you for the private key’s secret passphrase each time it’s required.

Let’s take a closer look at ssh-agent. When ssh-agent runs it outputs shell commands:

Listing 7. ssh-agent in action

[offsite]$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XX1O24LS/agent.14179; export SSH_AUTH_SOCK;
SSH_AGENT_PID=14180; export SSH_AGENT_PID;
echo Agent pid 14180;

We can instruct the shell to execute the output commands which ssh-agent displays using the shell’s eval command:

[offsite]$ eval `ssh-agent`
Agent pid 14198

The eval command tells the shell to evaluate (execute) the commands generated by the ssh-agent program. Make sure that you specify the back-quote character (`) and not a single quote! Once executed, the eval `ssh-agent` statement will return the agent’s process identifier. Behind the scenes, the SSH_AUTH_SOCK and SSH_AGENT_PID shell variables have been exported and are now available. You can view their values by displaying them to the shell console:

[offsite]$ echo $SSH_AUTH_SOCK
/tmp/ssh-XX7bhIwq/agent.14197

The $SSH_AUTH_SOCK (short for SSH Authentication Socket) is the location of a local socket which applications can use to speak to ssh-agent. To ensure that the SSH_AUTH_SOCK and SSH_AGENT_PID variables are always registered, enter the eval `ssh-agent` statement into your ~/.bash_profile.

ssh-agent has now become a background process which is visible using the top and ps commands.

Now we’re ready to share our passphrase with ssh-agent. To do so, we must use a program called ssh-add, which adds (sends) our passphrase to the running ssh-agent program.

Listing 8. ssh-add for hassle-free login

[offsite]$ ssh-add
Enter passphrase for /home/accountname/.ssh/id_dsa: (enter passphrase)
Identity added: /home/accountname/.ssh/id_dsa
(/home/accountname/.ssh/id_dsa)

Now when we access server1, we’re not prompted for a passphrase:

[offsite]$ ssh accountname@server1.com
[server1]$ exit

If you’re not convinced, try removing (kill -9) the ssh-agent process and reconnecting to server1. This time, you’ll notice that server1 will request the passphrase for the private key stored in the id_dsa file in the .ssh directory:

[offsite]$ kill -9 $SSH_AGENT_PID
[offsite]$ ssh accountname@server1.com
Enter passphrase for key ‘/home/accountname/.ssh/id_dsa’:

Simplifying key access using keychain

So far, we’ve learned about several OpenSSH programs (ssh, scp, ssh-agent and ssh-add), and we’ve created and installed private and public keys to enable a secure and automated login process. You may have realized that most of our setup work only has to be done once. For example, the process of creating the keys, installing them, and getting ssh-agent to execute via a .bash_profile only has to be done once per machine. That’s the really good news.

The less than ideal news is that ssh-add must be invoked each time we sign on to the offsite machine and ssh-agent isn’t immediately compatible with the cron scheduling process which we’ll need to automate our backups. The reason that cron processes can’t communicate with ssh-agent is that cron jobs are executed as child processes by cron and thus do not inherit the $SSH_AUTH_SOCK shell variable.

Fortunately, there is a solution which not only eliminates limitations associated with ssh-agent and ssh-add, but also allows us to use cron to automate all sorts of processes requiring secure passwordless access to other machines. In his 2001 three-part developerWorks series, OpenSSH key management (see Resources for a link), Daniel Robbins presented a shell script called keychain, which is a front-end to ssh-add and ssh-agent and which simplifies the entire passwordless process. Over time, the keychain script has undergone a number of improvements and is now maintained by Aron Griffis, with a recent 2.3.2-1 release posted on June 17, 2004.

The keychain shell script is a bit too large to list in this article because the well-written script includes lots of error checking, ample documentation, and a generous serving of cross-platform code. However, keychain can be quickly downloaded from the project’s Web site (see Resources for a link).

Once you download and install keychain, using it is remarkably easy. Simply log in to each machine and add the following two lines to each .bash_profile:

keychain id_dsa
. ~/.keychain/$HOSTNAME-sh

The first time you log back in to each machine, keychain will prompt you for the passphrase. However, keychain won’t ask you to reenter the passphrase on subsequent login attempts unless the machine has been restarted. Best of all, cron tasks are now able to use OpenSSH commands to securely access remote machines without requiring the interactive use of passphrases. Now we have the best of both worlds, added security and ease of use.

Listing 9. Initializing keychain on each machine

KeyChain 2.3.2; http://www.gentoo.org/projects/keychain
Copyright 2002-2004 Gentoo Technologies, Inc.; Distributed under the
GPL

* Initializing /home/accountname/.keychain/localhost.localdomain-sh
file…
* Initializing /home/accountname/.keychain/localhost.localdomain-csh
file…
* Starting ssh-agent
* Adding 1 key(s)…
Enter passphrase for /home/accountname/.ssh/id_dsa: (enter passphrase)

Scripting a backup process

Our next task is to create the shell scripts, which will perform the necessary backup operations. The goal is to perform a complete database backup of servers 1 and 2. In our example, each server is running the MySQL database server and we’ll use the mysqldump command-line utility to export a few database tables to an SQL import file.

Listing 10. The dbbackup.sh shell script for server 1

#!/bin/sh

# change into the backup_agent directory where data files are stored.
cd /home/backup_agent

# use mysqldump utility to export the sites database tables
mysqldump -u sitedb -pG0oDP@sswrd –add-drop-table sitedb –tables
tbl_ccode tbl_machine tbl_session tbl_stats > userdb.sql

# compress and archive
tar czf userdb.tgz userdb.sql

On server 2, we’ll place a similar script which backs up the unique tables present in the site’s database. Each script is flagged as executable using:

[server1]:$ chmod +x dbbackup.sh

With a dbbackup.sh file on servers 1 and 2, we return to the offsite data server, where we’ll create a shell script to invoke each remote dbbackup.sh script prior to initiating a transfer of the compressed (.tgz) data files.

Listing 11. backup_remote_servers.sh shell script for use on the offsite data server

#!/bin/sh

# use ssh to remotely execute the dbbackup.sh script on server 1
/usr/bin/ssh backup_agent@server1.com “/home/backup_agent/dbbackup.sh”

# use scp to securely copy the newly archived userdb.tgz file
# from server 1. Note the use of the date command to timestamp
# the file on the offsite data server.
/usr/bin/scp backup_agent@server1.com:/home/backup_agent/userdb.tgz
/home/backups/userdb-$(date +%Y%m%d-%H%M%S).tgz

# execute dbbackup.sh on server 2
/usr/bin/ssh backup_agent@server2.com “/home/backup_agent/dbbackup.sh”

# use scp to transfer transdb.tgz to offsite server.
/usr/bin/scp backup_agent@server2.com:/home/backup_agent/transdb.tgz
/home/backups/transdb-$(date +%Y%m%d-%H%M%S).tgz

The backup_remote_servers.sh shell script uses the ssh command to execute a script on the remote servers. Because we’ve set up passwordless access, the ssh command is able to execute commands on servers 1 and 2 remotely from the offsite server. The entire authentication process is now handled automatically, thanks to keychain.

Scheduling

Our next and final task involves scheduling the execution of the backup_remote_servers.sh shell script on the offsite data storage server. We’ll add two entries to the cron scheduling server to request execution of the backup script twice per day, at 3:34 am and again at 8:34 pm. On the offsite server invoke the crontab program with the edit (-e) option.

[offsite]:$ crontab -e

The crontab invokes the default editor, as specified using the VISUAL or EDITOR shell environment variables. Next, type two entries and save and close the file.

Listing 12. Crontab entries on the offsite server

34 3 * * * /home/backups/remote_db_backup.sh
34 20 * * * /home/backups/remote_db_backup.sh

A crontab line contains two main sections, a time schedule section followed by a command section. The time schedule is divided into fields for specifying when a command should be executed:

Listing 13. Crontab format

+—- minute
| +—– hour
| | +—— day of the month
| | | +—— month
| | | | +—- day of the week
| | | | | +– command to execute
| | | | | |
34 3 * * * /home/backups/remote_db_backup.sh