Saturday, March 31, 2012

Introduction to Linux Threads – Part I

A thread of execution is often regarded as the smallest unit of processing that a scheduler works on.
A process can have multiple threads of execution which are executed asynchronously.
This asynchronous execution brings in the capability of each thread handling a particular work or service independently. Hence multiple threads running in a process handle their services which overall constitutes the complete capability of the process.

In this article we will touch base on the fundamentals of threads and build the basic understanding required to learn the practical aspects of Linux threads.

Why Threads are Required?

Thursday, March 29, 2012

Complete Kickstart: How to Save Time Installing Linux

The Requirements
Our requirements were that kickstart, once launched and after making a menu selection to choose a particular kickstart configuration, needed to be completely unattended. We also needed to install some local tools and make configuration changes to the installed boxes before they would be ready for use. The Anaconda installer menu must provide us with options to install multiple versions of this kickstart or to boot from hard drive. If no menu selection is made after a short timeout, the Anaconda installer is configured to boot from the hard drive.
Here’s what you need in order to perform a kickstart:
  1. 1. A Web server and/or FTP Server for delivery of the RPMs that are to be installed.
  2. 1. A DHCP server for IP address assignments and to launch PXE Boot.
  3. 1. A TFTP server for download of PXE Boot components to the machines being kickstarted.
  4. 1. An PXE Boot capable network card.
  5. 1. The BIOSes on the computers to be kickstarted must be configured to allow a network boot.
Each of the required servers can be located on a different system, or they can be combined onto a single computer.
In addition, Cisco Core routers require special configuration to transport UDP PXE Boot packets across subnet boundaries. Our environment requires the use of a serial console during Kickstart for menu selection. This gives us the ability to select from two or more different kickstart installations.
Initial Decisions and Sizing
We chose to use HTTP for file delivery, but due to the possibility that some need might arise in the future for an FTP kickstart, we decided to configure our kickstart server directory structure so that both FTP and HTTP can be used. We also chose to house the HTTP, TFTP, and DHCP servers on a single computer.
For our environment, we had no reason not to have all of the servers on one box, and the number of simultaneous kickstarts we expect to experience is well within the capability of the hardware and network infrastructure we have available. When sizing a prospective kickstart server the limiting factors are most likely to be the hard drive data transfer rates and the network. Experience has shown up to 20 or so systems can be kickstarted simultaneously in about an hour with a very modest Pentium 4, a single IDE hard drive, and a 100Mb connection.
Using a 3.0GHz Intel Core-Duo with 4GB of RAM and dual 120GB hard drives in RAID 1 configuration on a Gigabit Ethernet connection should allow us to support multiple simultaneous kickstarts in numbers far larger than we currently expect. The only reason we used this particular hardware is that it is what we had available.
Kickstart Sequence of Events
A network-based kickstart can be initiated by an PXE Boot capable network card. The PXE Boot first requests an IP address from a DHCP server. It also obtains the location of a PXE Boot file from the DHCP server. PXELINUX is a bootloader for Linux using the PXE network booting protocol. The PXE Boot file is loaded from the TFTP server along with the contents of a file which defines the location and name of the installation kernel and initrd.img file as well as some parameters for the boot kernel and a menu for the Anaconda installer. This configuration file for Anaconda also contains the location of the kickstart configuration file to be used during the installation.
The PXE Boot file then loads the boot kernel and initrd image still using TFTP. After booting, Anaconda is started and Anaconda loads the menu and displays a window with a timer with several menu options. The Menu and time-out can be skipped if you do not need to make any choices here.
After choosing the desired kickstart installation, Anaconda locates the kickstart configuration file from the HTTP server and reads it. The kickstart configuration file has a default name of ks.cfg, but can be named anything. We use several for our different configurations, so provide unique names for each. If all of the data required to perform a complete installation is included in the kickstart configuration file, the installation completes without further intervention from the administrator. The RPM files used during the installation are downloaded from the HTTP server as they are needed.
The kickstart configuration file can also contain bash script commands that can be run both before and after the rest of the installation. We make extensive use of the post-installation bash scripts to perform installations of locally required RPM packages and tarballs as well as to make configuration changes before the first reboot.
Hardware configuration
In order to boot from the network it is necessary not only to have a network card that is capable of a network boot, but also to configure the BIOS to boot appropriately. You have a couple of options. The first is to always attempt to boot from the network as the first choice, then CD/DVD, and then from the hard drive. The second is to boot from the CD/DVD first, then the hard drive, and finally from the network. Choose the option that best fits your needs.
When booting from the hard drive prior to booting from the network, an additional step requiring some manual intervention would be required to force a boot from the network. It is necessary to overwrite the boot record to prevent booting from the hard drive. This can be done with a small script or from the command line using the dd command but it is another point of intervention.
We chose to configure BIOS to boot first from the network. We then set a short timeout for Anaconda so that the default is to boot to the hard drive if no other action is taken.
DHCP Configuration
The /etc/dhcpd.conf file must be configured correctly to provide an IP address for each client host as well as information necessary to initiate a PXE Boot sequence for each kickstart client host. DHCP determines the host name using the MAC address of the NIC making the request. Although the IP address can be specified in the dhcpd.conf file, we use DNS to maintain the addresses and DHCP does the lookup and then passes the address to the host.
DHCP can also serve a range of addresses rather than a specific address for each host, but that is outside the scope of this article.
#######################################################################
allow booting;allow bootp;dns-update-style ad-hoc;
option domain-name "cisco.com";
option domain-name-servers 109.99.6.247;
max-lease-time 604800;
default-lease-time 604800;
deny unknown-clients;

# The next-server line is required even though we point to ourselves.
# Resolves some issues relating to pxeboot across subnets.
next-server 109.99.101.74;

# 109.99.222 Subnets:
subnet 109.99.222.0   netmask 255.255.255.0
  { authoritative ; option routers 109.99.222.1 ; }

# Red Hat Enterprise Linux 5.1 Kickstart boxes
group {
filename "RHEL/pxelinux.0";
host ems-lnc100.cisco.com
  { hardware ethernet 00:15:17:1D:42:88 ; fixed-address ems-lnc100.cisco.com ;}
host ems-lnx118.cisco.com
  { hardware ethernet 00:04:23:B7:9A:15 ; fixed-address ems-lnx118.cisco.com ;}
host ems-lnx145.cisco.com
  { hardware ethernet 00:04:23:B5:6B:A9 ; fixed-address ems-lnx145.cisco.com ;}
}
#######################################################################
Listing 1: The very basic dhcpd.conf required to support kickstarts.
The filename RHEL/pxelinux.0; statement in the group stanza directs the PXE Boot to load the pxelinux.0 boot file from the specified directory, RHEL. The full path for this directory in our setup is /opt/tftpboot/RHEL where /opt/tftpboot is a symbolic link to /tftpboot. The TFTP root, /tftpboot, is defined in /etc/xinetd.d/tftp.
In each host stanza we specify the MAC address of the NIC in the respective hosts and the hostname. DHCP queries DNS for the IP address and passes it back to the host along with the router and DNS server information.
We discovered during configuration of our server for the kickstart role that the next-server line is required in dhcpd.conf to resolve some PXE Boot issues even though the next-server is really the same server in our case. You should use this statement no matter which box hosts the PXE Boot server, even if it is the same as the DHCP server. It took us a couple days to figure this out and it is one of the things we could not find documented anywhere.
The allow booting and allow bootp statements are both required for kickstarts to function.
All of the options pertaining to PXE Boot can be placed in the group or individual host stanzas as well as in the global section of the DHCP configuration. This allows you as much granularity as you need to have multiple servers and kickstart configurations as well as to ensure that only specific hosts or groups of hosts can be kickstarted.
The PXE Boot files
Three PXE Boot files are required to perform a network boot. The first is pxelinux.0, the network boot loader. The second is the network boot kernel, vmlinuz, and the third is the initial RAM disk image, initrd.img.
We placed pxelinux.0 in /opt/tftpboot/RHEL/ as this is the location we specified in dhcpd.conf. We also have discovered that this is the only place from which it works.
The kernel and RAM disk image files are placed in a distribution or release unique location such as /opt/tftpboot/RHEL/RHEL-server. We also have an RHEL workstation based release we use and place its files in /opt/tftpboot/RHEL/RHEL-workstation. This allows us to keep them separate and helps us to know which is which. We have seen configurations in which files for different distributions and releases are all located in a single directory and named differently. Our method works better for us because we like the additional organization it imposes.
For the most part one set of PXE Boot files is pretty much like another. Most Red Hat Enterprise based distributions currently provide a set of these files. Most of these files should work with most distributions. However we did find that the Red Hat Enterprise Linux 5.1 files are specific to that distribution and that PXE Boot files from other distributions such as CentOS do not work with RHEL 5.1.
TFTP Configuration
The TFTP configuration file, /etc/xinetd.d/tftp, should look like the sample configuration below. We changed disable = yes to disable = no and server_args = -s -c -v -v -v /tftpboot to server_args = -s -c -v -v -v /opt/tftpboot.
#######################################################################
# default: off
service tftp
{
 disable = no
 socket_type  = dgram
 protocol  = udp
 wait   = yes
 user   = root
 server   = /usr/sbin/in.tftpd
 server_args  = -s -c -v -v -v /opt/tftpboot
 per_source  = 11
 cps   = 100 2
 flags   = IPv4
}
#######################################################################
Listing 2. The TFTP configuration file required only minor changes.
Creating the PXE Boot configuration file
Each host that is to be kickstarted requires a unique configuration file which is located in the /opt/tftpboot/RHEL/pxelinux.cfg directory. This file is used to specify the locations of specific files such as the kernel and the initrd image file. These files are named with the hexadecimal representation of the IP address of the computer to be kickstarted.
You’ll find an online IP to Hex converter at http://tinyurl.com/4lzthf and another tool, written in Perl, is available at http://tinyurl.com/4pz6g3. Usage is very straightforward for each of these tools.
For example, the IP address 192.168.0.55 converts to C0A80037 in hex, so in this case the name of the configuration file for the host with ip address 192.168.0.55 is C0A80037.
Loading the PXE Boot configuration file
The PXE Boot configuration files contain information that allows PXEBoot to locate the kernel and initrd image files for the kickstart process. They also specify the serial console parameters and provide a menu for selection of the desired kickstart. The kernel and initrd images are not the files that will be installed on the kickstarted machine, but are used only as the running operating system during the kickstart itself.
The PXE Boot process tries to load the correct file for the computer by using an interesting algorithm. First it tries to load a file with a name based on the MAC address of the system, then with names based on the hexadecimal IP address, removing one hex digit for each failure. The sequence would look like this:
/opt/tftpboot/RHEL/pxelinux.cfg/01-22-33-44-aa-cc-ee
/opt/tftpboot/RHEL/pxelinux.cfg/C0A80037
/opt/tftpboot/RHEL/pxelinux.cfg/C0A8003
/opt/tftpboot/RHEL/pxelinux.cfg/C0A800
/opt/tftpboot/RHEL/pxelinux.cfg/C0A80
/opt/tftpboot/RHEL/pxelinux.cfg/C0A8
/opt/tftpboot/RHEL/pxelinux.cfg/C0A
/opt/tftpboot/RHEL/pxelinux.cfg/C0
/opt/tftpboot/RHEL/pxelinux.cfg/C
/opt/tftpboot/RHEL/pxelinux.cfg/default
The contents of our files is identical for each of the installations this process is designed for, so only a single master file is located at /opt/tftpboot/RHEL/pxelinux.cfg. Then we use a soft link with the hexadecimal IP address as its name to point to a master file. We can do this because all of our Intel boxes have the same kickstart choices available. You could also use individual files if that suits your needs better.
The contents of our master file are shown below:
#######################################################################

# RHEL5 Kickstart configuration file.
#
# NOTE: The workstation and server versions of the RHEL 5.1 images require
# different initrd.img files.

default 1
prompt 1
timeout 200
display msgs/Main.msg
F1 msgs/Main.msg
F2 msgs/general.msg
F3 msgs/expert.msg
F4 msgs/param.msg
F5 msgs/rescue.msg

#F1 Main.msg

# Hard drive
label 1
    localboot 1

# RHEL5.1-MAX
label 2
    kernel RHEL-workstation/vmlinuz
    append ksdevice=eth0 initrd=RHEL-workstation/initrd.img \
    console=ttyS0,9600n8 ramdisk_size=6804 \
    ks=http://emstools2b.cisco.com/pub/kickstart/rhel-AP-Max-ks.cfg

# RHEL5.1-MIN
label 3
    kernel RHEL-server/vmlinuz
    append ksdevice=eth0 \
    initrd=RHEL-server/initrd.img console=ttyS0,9600n8 \
    ramdisk_size=6804
    ks=http://emstools2b.cisco.com/pub/kickstart/rhel-AP-Min-ks.cfg
#######################################################################
Listing 3. PXE Boot configuration files
In Listing 3 the PXE Boot configuration files contain data to create a menu for Anaconda and information that allows the PXE Boot process to locate the files needed to boot. Theappend lines have been split for formatting purposes, but should be on a single line when used.
Note that there are multiple stanzas in the file. One for each possible kickstart installation that is defined. Each stanza specifies different files for the vmlinuz, initrd.img and the location and name of the kickstart file to be used. Console parameters are also specified in the PXE Boot configuration file because we use the console to make the menu choice for the desired kickstart and to monitor the installation.
We also added the statement ksdevice=eth0 to the append line. This prevents manual intervention to choose the install NIC when more than one NIC is present. This information was also very hard to find.
This file also contains the definitions of the various menu options we want the Anaconda installer to provide, as well as the Function Key definitions for various help options. The menu options are created by Anaconda using the labels in each stanza. So the menu choices we have are 1, 2 and 3. Note that option 1 is local boot from the hard drive and that the “default 1″ line specifies that the system will boot to the hard drive after the timeout. The “timeout 200″ line specifies the length of the timeout in tenths of a second. This is a strange unit, but the value of 200 results in a timeout of twenty seconds.
The data to generate and display the menu itself is located in the file /tftpboot/RHEL/msgs/Main.msg. Separating the files that specify the options from the file that displays the available options allows us to define hidden options should we need to do that.
#######################################################################

       09Welcome to 0cThe Cisco Linux09 Installer!07
0a
                            |       |
                        . | | | . | | | .
                            '       '
                            C I S C O
07

Enter number of the Linux distribution you wish to install:

1. Cisco CEL 4
2. Red Hat Enterprise Linux MIN (Test)
3. Red Hat Enterprise Linux MAX (DEV test)

05[F1-Main] [F2-General] [F3-Expert] [F4-Kernel] [F5-Rescue]07

#######################################################################
Listing 4: The file /tftpboot/RHEL/msgs/Main.msg
You’ll see that Listing 4 contains the menu for the Anaconda installer. We have added our own options to the menu.
Cisco Core router configuration
The DHCP and TFTP protocols both use UDP rather then TCP packets. Most UDP packets are not forwarded across subnet boundaries and we have many different subnets in our network. Many Cisco routers with current versions of IOS have the ability to configure helper addresses for UDP packets. This enables the router to forward UDP packets to the DHCP and TFTP servers or to specific subnet(s).
Based on our experience, you should only configure this on the core router closest to your server.
#######################################################################
ip forward-protocol udp
!
interface ethernet 1
 ip helper-address 10.44.23.7
interface ethernet 2
 ip helper-address 192.168.1.19
#######################################################################
Listing 5. Sample Configuration
You’ll find a sample configuration from the Cisco IOS IP Configuration Guide, Release 12.2, in Listing 5 that provides an example of the commands required to set up a helper address.
If a protocol is not specified on your router, the helper address will forward all UDP packets to your kickstart server. If this is not what you want, be sure to specify only those protocols that need to be forwarded. This is another piece of information that was very hard to locate. Refer to the Cisco IOS IP Configuration Guide, Release 12.2, for details of this and related commands.
This will not be an issue if your DHCP and TFTP servers are located in the same subnet as all of the hosts you wish to kickstart.
Web Server (Apache) configuration
We chose Apache for our web server because it is supplied by all Red Hat distributions and because we use it on other internal servers so are familiar with its operation. Once you have Apache installed and running, nothing else needs to be done to the configuration to make it work for kickstarts. All you have to do is place the files in a location that is served by Apache.
Because we wanted our server to be as flexible as possible, we decided to plan for the eventuality that we would eventually support both FTP and HTTP kickstarts even though we are only using HTTP at this time. Therefore we chose a directory structure starting at /var/ftp/pub and created a symbolic link to this location from /var/www/html.
ln -s /var/ftp/pub /var/www/html/pub
Making the RPMs Available
While we wanted to make the ISO images of RHEL 5.1 available for download so that users can burn their own installation DVDs, it is also necessary to make the RPMs located in the ISO images available for the kickstarts. In order to accomplish this without having to store the files on the hard drive twice, we chose to keep only the ISO images on the hard drive and mount them using the loopback device to make the individual files in the ISO available to the kickstart.
To accomplish this, the following directories were created.
/var/ftp/pub/rhel/5.1/isos/i386/
   /var/ftp/pub/rhel/5.1/os
The iso images for RHEL 5.1 client and server were copied to the /var/ftp/pub/rhel/5.1/isos/i386 directory. The following entries were added to /etc/fstab to mount the ISOs automatically at boot time.
#######################################################################
/var/ftp/pub/rhel/5.1/isos/i386/rhel-5.1-client-i386-dvd.iso \
        /var/ftp/pub/rhel/5.1/os/i386/workstation iso9660 \
        loop=/dev/loop1,ro  0 0

/var/ftp/pub/rhel/5.1/isos/i386/rhel-5.1-server-i386-dvd.iso \
        /var/ftp/pub/rhel/5.1/os/i386/server \
        iso9660 loop=/dev/loop2,ro  0 0
#######################################################################
Listing 6: /etc/fstab
The entries in Listing 6 for /etc/fstab mount the ISO images so that the files in the images can be available for the kickstarts.
Note that we chose to use the loop1 and loop2 devices instead of the loop0 device so that the loop0 device would be available to anyone wanting to use a loopback.
The Kickstart Configuration File
The kickstart configuration file, by default named ks.cfg, is used by Anaconda to define the parameters of the installation. This file provides the answers to all of the questions and entries to all of the fields required by the installation process. Only by having answers to each and every question can the kickstart be fully automated. If any of the required fields does not have an entry the installation halts and waits for input.
Creating the Starting ks.cfg File
We initially created the kickstart file using the kickstart GUI configurator. Using this configurator allowed selection of the major software groups to be installed. There are other ways to obtain a kickstart configuration file to use as a starting point. Each time Red Hat Linux is installed, a kickstart configuration is stored at /root/anaconda-ks.cfg. This file can be used to exactly recreate the installation as it was performed. You could generate a kickstart file by performing a manual installation with the exact configuration you want and then use the anaconda-ks.cfg file generated as the starting point.
We renamed our kickstart files from anaconda-ks.cfg to something more meaningful, rhel-AP-Max-ks.cfg, and rhel-AP-Min-ks.cfg. This enables us to know from the names which type of installation the file is for, and also to keep multiple files in the same directory.
The kickstart configuration files have several sections. Each section has statements pertaining to a specific portion of the installation. Some sections are optional.
We did not use the %pre section which allows running scripts before the installation begins, so we will start with the command section. Most of this should be relatively self-explanatory, but if you need more information on any portion, the Red Hat Enterprise Linux Installation Guide (see Resources) contains an excellent description of each section of a kickstart file and describes each of the possible statements and commands that can be used. For the sake of brevity we will only discuss certain key portions of our kickstart file.
#######################################################################
# This is an installation not an upgrade
install
# The location of the RPM files
url --url http://emstools2b.cisco.com/pub/rhel/server
key 9a09007d99b6cd00
lang en_US
# Use text mode install
text
keyboard us
xconfig --defaultdesktop kde --resolution 640x480 --depth 8
network --device eth0 --bootproto dhcp --onboot=on
rootpw --iscrypted $1$tihTg7ne$hohhkj87hGGddg9B4WkXV1
authconfig --useshadow --enablemd5
selinux --disabled
timezone America/New_York
firewall --disabled
firstboot --disable
# Reboot after installation
reboot
bootloader --location=mbr --append="console=ttyS0,9600n8"
clearpart --all --initlabel

# define partitions
part /boot --fstype ext3 --size=512
part /opt --fstype ext3 --size=10000 --grow
part /usr --fstype ext3 --size=10000
part /tmp --fstype ext3 --size=7500
part /var --fstype ext3 --size=7500
part /home --fstype ext3 --size=2500
part swap --size=2048
part / --fstype ext3 --size=2048
part /usr/local --fstype ext3 --size=2000

#######################################################################
Listing 7: The command section of our kickstart file.
We added a key line to this section. This is what Red Hat calls the Installation number and is required to enable all Linux functionality. Just what would not be enabled is not specified. If the key line is not included in the kickstart files, the installation stops and waits for input which is not what we want.
We specified a text mode install because, as mentioned before, we need to use the console for installation and a graphical installation would not work for us.
We specified our console parameters in the “bootloader” line to ensure that they matched those of our console servers.
Due to issues we had with creating LVMs using kickstart, we only created EXT3 partitions, not Logical Volumes. We intend to revisit this and determine whether Logical Volumes can be used. It may be that, because our procedures are to simply re-kickstart systems that have any significant issues, such an effort would be more trouble than it is worth.
The %packages section of our kickstart file defines which groups are installed. These are the names preceeded by an “@” sign. Individual RPM packages can also be specified just by placing the appropriate package name on a line by itself in this section.
You can specify RPM packages that are not to be installed even if they are part of a group that you otherwise need to install. These RPMs are specified on a line by themselves but are preceeded by a “-” sign.
#######################################################################
%packages
@engineering-and-scientific
@mysql
@development-libs
@editors
@system-tools
@gnome-software-development
@text-internet
@gnome-desktop
@core
@base
@ftp-server
@network-server
@legacy-software-development
@java-development
@printing
@kde-desktop
@mail-server
@server-cfg
@sql-server
@admin-tools
@development-tools
@graphical-internet
festival
audit
kexec-tools
bridge-utils
device-mapper-multipath
dnsmasq
imake
-sysreport
mc
festival
audit
libgnome-java
libgtk-java
libgconf-java
kexec-tools
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
-compiz-kde
-knetworkmanager
-amarok
#######################################################################
Listing 8: The %packages section of the kickstart file defines the groups and packages to install.
Using Post-installation scripts
We invested a great deal of effort developing the post-install scripts defined in the %post section of the kickstart configuration file. These scripts allow us to perform installation and configuration of RPMs and tarballs that are not part of the Red Hat installation.
The important thing to remember about the post-installation scripts is that they are executed using the bash command interpreter in a chroot’ed environment that behaves as it will when rebooted after the installation. This allows virtually any action that you could possibly work into a script to be performed during the final stages of installation.
#######################################################################
%post
# Install the yum repository configuration files
cd /tmp
wget http://emstools2b.cisco.com/pub/local/lab-repos.tar
cd /
tar -xvf /tmp/lab-repos.tar

# Set an ID to be used for other scripts
touch /LINUX_RHEL_MINIMAL_INSTALL

# Install Kshell as a preference of some developers.
yum -y install ksh

# Configure some local NFS mount points
service portmap start
mount  emsnfs:/export/linux/post   /mnt
cat /mnt/auto_localnfs >> /etc/auto.misc
cat /mnt/auto_misc >> /etc/auto.misc

# Get the command to create the motd and create it for the first time.
cp /mnt/createMOTDLinux /etc/init.d/create_motd
mv /etc/motd /etc/motd.orig
/etc/init.d/create_motd > /etc/motd

umount /mnt

# Create symlinks for mount points
# the links to /localnfs are to work around the issue with Linux
# mount points not being browsable as they are in Unix
mkdir /localnfs
ln -s /misc/apps       /localnfs/apps
ln -s /misc/rtp-chaos  /localnfs/rtp-chaos
ln -s /misc/black-hole /localnfs/black-hole
ln -s /misc/tools      /localnfs/tools
ln -s /misc/tftpboot   /localnfs/tftpboot
mkdir /opt/scratch
ln -s /opt/scratch /scratch

# Create ssh authorized keys
# Make the directory
mkdir /root/.ssh

# Create the keys file
cat  << xxEOFxx >> /root/.ssh/authorized_keys
sh-dss AAAAB3NzaC1kc3MAAACBAKyW6vv6uHKGKL54765VBHKJHhbfvfhJ/rkspGK2pmAM7awj7EwB
/wUBZUucmQSYnyaOlbvS6NkdE+sUC/asU/mEZjzoQgP+kdahxfJvWATaJweVFjRdHrIZxPB4nlO+MEBb
cPmUP7cLLQ1KGbfUakr35qzb9RjpBPDcBSDW2GZRAAAAFQCD/qw8FCSfEyWAmtkXDioJBWUCOwAAAIAm
4czfxx+Srm7FxGDTsiL52ojKzZCzddTi6YclknBXYpa3jhjhDfgkbGfHc746cVXm3hJ9ZgA3RQpMypKn
WS6EHimjkjEeqfw/viqPR1NCvj1xVs9XDjRtCelwsxUNj31Y2RHCsusa6DDwG765bnlk/BO4lUGRQpNy
QAAjKyDhPwAAAIAPJQcSf0tc4OrqNxy/gjkhkhgghfTRerthkljhGuyKarrmWan9ZkkFJQYnp09GNasZ
zI7Zwau3oqfutPTWJFehBskFKvRpSjYd59vKjWpDyCE5xHYxZfDORTj4pzjRSyiXDP/viA5DBCUWieM4
zGWa1RKVdskjPFS56y5GAkEwcA== root@emsjumpsssh-dss AAAAB3NzaC1kc3MAAACBAOqZBr62GU
La+NwGUatvO7OVXqGDn4qXvR2GAUputz9uyYmcWTvoHG0D3eAQ2flqhpyhJQo63GyntUtmGkXIHFuM3z
4qDt19qcpFRj10ZzRbZGhf+QbJwkxA9fpOy/BmoYykW5l36Db/Dvlzk4zNgJAmGXb2rNv8RSqYC6kCZf
aNAAAAFQDTb8EsksyknY++4zXC3TPNrH/+MwAAAIEAz3OCUfZXo+e/lJ/KSFj1un378KGGo9qfGSpVMV
Tva/z58KAZ174tJpgnfA8+fQOwq/ip8s9UyHA2qR+BICjjZo1KatevFN7l4rpNSqdLivEasrGBu6fRTP
/kQ6vt+OLIAQyr8t9RqpZKUVdd9odFA9NLiuOhG//eh2cDSXmjFnkAAACAbgzdEMcCMeMT/XPJrkZ/md
TX/EJ6VNQEuTP3fhrjKKjccYobXPOQhvliIhPGFbtrRZlYRPPFAkAAse0qRPOsy8XHKD18WnQr5JNJx+
C5PYMkb8APY55Ydwwrt4EFeqnFpF3RXFhPY1eiZNAI33GopEGVTiLTO4ZW9mYC8EI7e28= root@emstools
xxEOFxx

# Copy the logbanner and change sshd_config
cat  << xxEOFxx >> /etc/LogBanner
                               WARNING!!!
                   READ THIS BEFORE ATTEMPTING TO LOGON                      

     This System is for the use of authorized users only.  Individuals
     using this computer without authority, or in excess of their
     authority, are subject to having all of their activities on this
     system monitored and recorded by system personnel.  In the course
     of monitoring individuals improperly using this system, or in the
     course of system maintenance, the activities of authorized users
     may also be monitored.  Anyone using this system expressly
     consents to such monitoring and is advised that if such
     monitoring reveals possible criminal activity, system personnel
     may provide the evidence of such monitoring to law enforcement
     officials.                                                              

     Cisco Acceptable Use Policy:                                            

http://wwwin.cisco.com/infosec/policies/acceptable_use.shtml

xxEOFxx

echo "Banner /etc/LogBanner" >> /etc/ssh/sshd_config

#######################################################################
Listing 9: Post-installation Script Example
You’ll see that Listing 9 contains a partial listing of our post-installation script which installs both RPMs and tarballs designed for our unique lab environment as well as performing other necessary tasks.
As you can see, our post-installation is quite extensive. In addition to performing installations of several software packages we require, it also sets up a login banner, creates the /root/.ssh directory and copies some public keys there. We have only shown one of these keys to save space.
Notice that we can also start services as in the line service portmap start and access files on NFS mounts during this last portion of the kickstart. Post-installation provides a very flexible environment for performing a great many automated tasks.
Perform the kickstart
Performing the kickstart is very easy because we have done all of the hard work in setting up the network kickstart. We have four basic steps to perform.
  1. Add the computer to DNS.
  2. Add the appropriate information to the dchpd.conf file.
  3. Boot the computer.
  4. Select the desired kickstart from the menu.
The automated kickstart does the rest. The first two steps only need to be performed the first time a computer is kickstarted; after that the DNS and DHCP information will already be there.
Troubleshooting a failing Kickstart
The most common problems with network kickstarts the way we have set it up are network failures, MAC addresses that are incorrectly entered in the dhcpd.conf file, using the MAC address for the wrong NIC. These problems will present themselves on the console with messages from PXE Boot on the NIC unable to obtain an IP address.
An incorrectly named hexadecimal IP address file for a system or a problem with the TFTP server will allow the NIC to obtain the network data, but fails to load the PXE Boot configuration file for the system. Be sure your TFTP server is configured correctly using the tftp file in the /etc/xinetd.d directory.

How to Setup VirtualBox Guest Additions and Network

In the previous article Install & Create Virtual Machine, we explained how to install VirtualBox and create a Guest machine in it.
This article explains how to setup network for a guest machine and installing Guest Additions in a guest machine.

Virtual Networking

Once a guest machine is installed, the first step that everyone like to do is to configure the network for that machine. VirtualBox provides a facility to add up-to 4 Network adapter (Ethernet card) to each guest machine through the GUI.

Networking Modes

The following are the different types of networking modes available to be configured for each network adapter
  • Network Address Translation (NAT)
  • Bridged Networking
  • Internal networking
  • Host-only networking

Network Address Translation

When a guest machine is set-up in this mode, then outside world will never be able to communicate with the Guest machine directly. When the guest machine send an IP packet to some remote machine, VirtualBox’s NAT Engine will receive the packet and extracts the TCP/IP segments, and change the IP address to the IP address of the HOST machine. So the outside world only see the IP address of the Host machine. When an reply is sent, then it will be received by the HOST machine and resend to the Guest Machine.

Bridged Networking

It is similar to connecting a physical machine in a network. Outside world can directly communicate with the Guest machine. When a guest is configured for Bridged Networking, then VirtualBox uses a device driver on the host system, which can intercept and inject data through the physical network adapter present in the HOST machine.

Internal networking

We can create more than one Guest machine, using VirtualBox. So if we want all the guest machine to communicate internally, then we can use this Internal Networking mode. Though we can use Bridged Networking also for this, Internal Networking is more secure.
In Bridged Networking, all the packets are sent/received from the physical network adapter in the HOST machine. If someone attaches a packet sniffer tool in the HOST, then all the communication will be captured. In case of Internal Networking, the communication between the Guest machines will be secure.

Host-Only Networking

It is a hybrid of Bridged and Internal networking. In this mode, the Guest machine can talk to each other and also with the Host machine. It will look like, all machine ( including Host and Guest Machines ), are connected to a physical network switch. The guest machine cannot communicate with the outside world too, since they are not connected via a physical interface.
So based on the need, we can actually choose between the networking modes. Now we will see how we can configure the ‘Networking’ in VirtualBox UI.

Steps to Configure Network

1) Launch ‘VirtualBox’
2) Select the ‘Guest Machine’ and click ‘Settings’.
3) Choose ‘Network’. It will have the details similar to

4) Check the ‘Enable Network Adapter’
5) Attached to, will have all the above mentioned mode. Choose the one you need. Here I choose ‘Bridged Adapter’
6) If your host network has more than 2 physical Ethernet card, then choose one among them, which will be used as the interface to send/receive packets by the Guest Machine
7) The advance menu, will have the Adapter type and MAC address for the virtual interface. Change it if you need.
8). Finally, you can ‘tick’ the Cable Connected check box. This is like physically connecting the network cable to a machine.


9) Click ‘OK’ and ‘Start’ the guest machine
10) Now, configure the network as usual using ifconfig or ‘network-admin’ command and restart the networking services.
11) We are done with setting up a networking for a Guest machine successfully.

Guest Additions

Guest Additions can be installed in a Virtual Machine, after installing the Operating System in it. It consists of device drivers and other applications which can optimize the performance and usability of a Virtual Machine.
The Guest Additions can provide the following features.,
  • Mouse pointer integration
  • Time synchronization
  • Shared folders
  • Seamless windows
  • Shared clipboard

Guest Addition Installation

Here we will see how to install guest additions in Debian Lenny, and for all flavors mostly this will be applicable.
By default under /usr/share/virtualbox/, there will be a file named, VBoxGuestAdditions.iso. We need to map this iso to the Guest machine. To do that,
1) Launch ‘VirtualBox’
2) Select the ‘Guest Machine’ & click ‘Settings’
3) Select ‘Storage’
4) On the ‘Storage Tree’, under ‘IDE Controller’, select the ‘Empty’ CD as shown in the following image

5) Now browse through the ‘/usr/share/virtualbox/’ directory, and choose the ‘VBoxGuestAdditions.iso’ file as shown below

6) Click ok
7) Start the Guest Machine
Once the Guest machine is started, you may notice some 2 mouse pointers. One for Host machine and One for Guest Machine. This often causes confusion, and we can avoid once we install the Guest Additions.
8). Login to the guest machine as ‘root’.
9) Mount the CD, with the help of ‘mount’ command.
# mount /dev/hdc /mnt
# cd /mnt
10) You will see a list of files as shown below
# ls -1
AUTORUN.INF
VBoxWindowsAdditions.exe
VBoxWindowsAdditions-x86.exe
VBoxWindowsAdditions-amd64.exe
autorun.sh
VBoxLinuxAdditions.run
VBoxSolarisAdditions.pkg
64Bit
32Bit
11) Install the following dependencies
apt-get install make gcc linux-headers-$(uname -r)
12) Execute
sh ./VBoxLinuxAdditions.run
13) Once it is installed successfully, reboot the machine.
14) Now when the machine is staring, Guest addition service will start automatically. You will notice only 1 mouse pointer now, when the mouse pointer is within the Guest machine, it operates on the guest machine, once it is moved out of the Guest machine window, it operates on the host machine.
VirtualBox also sync the time between the host and guest machine. By default, for every 10 seconds, the guest time will be reset to the host time.
You can control various timing related parameters using ‘VBoxService’ command on the guest machine.
The following illustrates the time reset by VirtualBox.
Guest-Machine:~# date -s "Tue Mar 18 11:41:43 IST 2012"
Sun Mar 18 11:41:43 IST 2012

Guest-Machine:~# date
Sun Mar 18 11:41:48 IST 2012

Guest-Machine:~# date
Sun Mar 18 11:41:53 IST 2012

Guest-Machine:~# date
Sun Mar 18 11:47:48 IST 2012
For more details with regard to Time Synchronization, please read Time Sync

Shared Clipboard

You can share the Clipboard ( A kind of buffer used for cut, copy, paste ), between the Guest machine and the host machine. It will work only if the Guest Additions is installed in the Guest machine. You can change the setting for clipboard sharing as shown in the figure.

Once it is done, you can copy a document in the Guest machine, and paste the document in a file under the host machine and vice versa.
VBoxClient is the command under the Guest machine which can be used to start the VirtualBox X Window System guest services like clipboard, display, seamless windows. It will be started automatically (if guest additions are installed) when the Guest machine is booted.

Wednesday, March 28, 2012

Three Sysadmin Rules You Can’t (And Shouldn’t) Break

When I drafted this article, I really came-up with 7 sysadmin habits. But, out of those 7 habits, three really stood out for me. While habits are good, sometimes rules might even be better, especially in the sysadmin world, when handling a production environment.

Rule #1: Backup Everything ( and validate the backup regularly ) 

 Experienced sysadmin knows that production system will crash someday, no matter how proactive we are. The best way to be prepared for that situation is to have a valid backup. If you don’t have a backup of your critical systems, you should start planning for it immediately. While planning for a backup, keep the following factors in your mind: What software (or custom script?) you would use to take a backup? Do you have enough disk space to keep the backup? How often would you rotate the backups? Apart from full-backup, do you also need regular incremental-backup? How would you execute your backup? i.e Using crontab or some other schedulers? If you don’t have a backup of your critical systems, stop reading this article and get back to work. Start planning for your backup immediately. A while back in one of the research conducted by some group (don’t remember who did that), I remember they mentioned that only 70% of the production applications are getting backed-up. Out of those, 30% of the backups are invalid or corrupted. Assume that Sam takes backup of the critical applications regularly, but doesn’t validate his backup. However, Jack doesn’t even bother to take any backup of his critical applications. It might sound like Sam who has a backup is in much better shape than Jack who doesn’t even have a backup. In my opinion, both Sam and Jack are in the same situation, as Sam never validated his backup to make sure it can be restored when there is a disater. If you are a sysadmin and don’t want to follow this golden rule#1 (or like to break this rule), you should seriously consider quitting sysadmin job and become a developer.

Rule #2: Master the Command Line ( and avoid the UI if possible )

There is not a single task on a Unix / Linux server, that you cannot perform from command line. While there are some user interface available to make some of the sysadmin task easy, you really don’t need them and should be using command line all the time.
So, if you are a Linux sysadmin, you should master the command line.
On any system, if you want to be very fluent and productive, you should master the command line. The main difference between a Windows sysadmin and Linux sysadmin is — GUI Vs Command line. Windows sysadmin are not very comfortable with command line. Linux sysadmin should be very comfortable with command line.
Even when you have a UI to do certain task, you should still prefer command line, as you would understand how a particular service works, if you do it from the command line. In lot of production server environment, sysadmin’s typically uninstall all GUI related services and tools.
If you are Unix / Linux sysadmin and don’t want to follow this rule, probably there is a deep desire inside you to become a Windows sysadmin.

Rule #3: Automate Everything ( and become lazy )

Lazy sysadmin is the best sysadmin.
There is not even a single sysadmin that I know of, who likes to break this rule. That might have something to do with the lazy part.
Take few minutes to think and list out all the routine tasks that you might do daily, weekly or monthly. Once you have that list, figure out how you can automate those. The best sysadmin typically doesn’t like to be busy. He would rather be relaxed and let the system do the job for him.

How to Backup Linux? 15 rsync Command Examples

rsync stands for remote sync.
rsync is used to perform the backup operation in UNIX / Linux.
rsync utility is used to synchronize the files and directories from one location to another in an effective way. Backup location could be on local server or on remote server.

Important features of rsync

  • Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security: rsync allows encryption of data using ssh protocol during transfer.
  • Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
  • Privileges: No special privileges are required to install and execute rsync

Syntax

$ rsync options source destination 
Source and destination could be either local or remote. In case of remote, specify the login name, remote server name and location.

Example 1. Synchronize Two Directories in a Local Server

To sync two directories in a local computer, use the following rsync -zvr command.
$ rsync -zvr /var/opt/installation/inventory/ /root/temp building file list ... done sva.xml svB.xml . sent 26385 bytes  received 1098 bytes  54966.00 bytes/sec total size is 44867  speedup is 1.63 $ 
In the above rsync example:
  • -z is to enable compression
  • -v verbose
  • -r indicates recursive
Now let us see the timestamp on one of the files that was copied from source to destination. As you see below, rsync didn’t preserve timestamps during sync.
$ ls -l /var/opt/installation/inventory/sva.xml /root/temp/sva.xml -r--r--r-- 1 bin  bin  949 Jun 18  2009 /var/opt/installation/inventory/sva.xml -r--r--r-- 1 root bin  949 Sep  2  2009 /root/temp/sva.xml 

Example 2. Preserve timestamps during Sync using rsync -a

rsync option -a indicates archive mode. -a option does the following,
  • Recursive mode
  • Preserves symbolic links
  • Preserves permissions
  • Preserves timestamp
  • Preserves owner and group
Now, executing the same command provided in example 1 (But with the rsync option -a) as shown below:
$ rsync -azv /var/opt/installation/inventory/ /root/temp/ building file list ... done ./ sva.xml svB.xml . sent 26499 bytes  received 1104 bytes  55206.00 bytes/sec total size is 44867  speedup is 1.63 $ 
As you see below, rsync preserved timestamps during sync.
$ ls -l /var/opt/installation/inventory/sva.xml /root/temp/sva.xml -r--r--r-- 1 root  bin  949 Jun 18  2009 /var/opt/installation/inventory/sva.xml -r--r--r-- 1 root  bin  949 Jun 18  2009 /root/temp/sva.xml 

Example 3. Synchronize Only One File

To copy only one file, specify the file name to rsync command, as shown below.
$ rsync -v /var/lib/rpm/Pubkeys /root/temp/ Pubkeys  sent 42 bytes  received 12380 bytes  3549.14 bytes/sec total size is 12288  speedup is 0.99 

Example 4. Synchronize Files From Local to Remote

rsync allows you to synchronize files/directories between the local and remote system.
$ rsync -avz /root/temp/ thegeekstuff@192.168.200.10:/home/thegeekstuff/temp/ Password: building file list ... done ./ rpm/ rpm/Basenames rpm/Conflictname  sent 15810261 bytes  received 412 bytes  2432411.23 bytes/sec total size is 45305958  speedup is 2.87 
While doing synchronization with the remote server, you need to specify username and ip-address of the remote server. You should also specify the destination directory on the remote server. The format is username@machinename:path
As you see above, it asks for password while doing rsync from local to remote server.
Sometimes you don’t want to enter the password while backing up files from local to remote server. For example, If you have a backup shell script, that copies files from local to remote server using rsync, you need the ability to rsync without having to enter the password.
To do that, setup ssh password less login as we explained earlier.

Example 5. Synchronize Files From Remote to Local

When you want to synchronize files from remote to local, specify remote path in source and local path in target as shown below.
$ rsync -avz thegeekstuff@192.168.200.10:/var/lib/rpm /root/temp Password: receiving file list ... done rpm/ rpm/Basenames . sent 406 bytes  received 15810230 bytes  2432405.54 bytes/sec total size is 45305958  speedup is 2.87 

Example 6. Remote shell for Synchronization

rsync allows you to specify the remote shell which you want to use. You can use rsync ssh to enable the secured remote connection.
Use rsync -e ssh to specify which remote shell to use. In this case, rsync will use ssh.
$ rsync -avz -e ssh thegeekstuff@192.168.200.10:/var/lib/rpm /root/temp Password: receiving file list ... done rpm/ rpm/Basenames  sent 406 bytes  received 15810230 bytes  2432405.54 bytes/sec total size is 45305958  speedup is 2.87 

Example 7. Do Not Overwrite the Modified Files at the Destination

In a typical sync situation, if a file is modified at the destination, we might not want to overwrite the file with the old file from the source.
Use rsync -u option to do exactly that. (i.e do not overwrite a file at the destination, if it is modified). In the following example, the file called Basenames is already modified at the destination. So, it will not be overwritten with rsync -u.
$ ls -l /root/temp/Basenames total 39088 -rwxr-xr-x 1 root root        4096 Sep  2 11:35 Basenames  $ rsync -avzu thegeekstuff@192.168.200.10:/var/lib/rpm /root/temp Password: receiving file list ... done rpm/  sent 122 bytes  received 505 bytes  114.00 bytes/sec total size is 45305958  speedup is 72258.31  $ ls -lrt total 39088 -rwxr-xr-x 1 root root        4096 Sep  2 11:35 Basenames 

Example 8. Synchronize only the Directory Tree Structure (not the files)

Use rsync -d option to synchronize only directory tree from source to the destination. The below example, synchronize only directory tree in recursive manner, not the files in the directories.
$ rsync -v -d thegeekstuff@192.168.200.10:/var/lib/ . Password: receiving file list ... done logrotate.status CAM/ YaST2/ acpi/  sent 240 bytes  received 1830 bytes  318.46 bytes/sec total size is 956  speedup is 0.46 

Example 9. View the rsync Progress during Transfer

When you use rsync for backup, you might want to know the progress of the backup. i.e how many files are copies, at what rate it is copying the file, etc.
rsync –progress option displays detailed progress of rsync execution as shown below.
$ rsync -avz --progress thegeekstuff@192.168.200.10:/var/lib/rpm/ /root/temp/ Password: receiving file list ... 19 files to consider ./ Basenames      5357568 100%   14.98MB/s    0:00:00 (xfer#1, to-check=17/19) Conflictname        12288 100%   35.09kB/s    0:00:00 (xfer#2, to-check=16/19) . . . sent 406 bytes  received 15810211 bytes  2108082.27 bytes/sec total size is 45305958  speedup is 2.87 
You can also use rsnapshot utility (that uses rsync) to backup local linux server, or backup remote linux server.

Example 10. Delete the Files Created at the Target

If a file is not present at the source, but present at the target, you might want to delete the file at the target during rsync.
In that case, use –delete option as shown below. rsync delete option deletes files that are not there in source directory.
# Source and target are in sync. Now creating new file at the target. $ > new-file.txt  $ rsync -avz --delete thegeekstuff@192.168.200.10:/var/lib/rpm/ . Password: receiving file list ... done deleting new-file.txt ./  sent 26 bytes  received 390 bytes  48.94 bytes/sec total size is 45305958  speedup is 108908.55 
Target has the new file called new-file.txt, when synchronize with the source with –delete option, it removed the file new-file.txt

Example 11. Do not Create New File at the Target

If you like, you can update (Sync) only the existing files at the target. In case source has new files, which is not there at the target, you can avoid creating these new files at the target. If you want this feature, use –existing option with rsync command.
First, add a new-file.txt at the source.
[/var/lib/rpm ]$ > new-file.txt
Next, execute the rsync from the target.
$ rsync -avz --existing root@192.168.1.2:/var/lib/rpm/ . root@192.168.1.2's password: receiving file list ... done ./  sent 26 bytes  received 419 bytes  46.84 bytes/sec total size is 88551424  speedup is 198991.96 
If you see the above output, it didn’t receive the new file new-file.txt

Example 12. View the Changes Between Source and Destination

This option is useful to view the difference in the files or directories between source and destination.
At the source:
$ ls -l /var/lib/rpm -rw-r--r-- 1 root root  5357568 2010-06-24 08:57 Basenames -rw-r--r-- 1 root root    12288 2008-05-28 22:03 Conflictname -rw-r--r-- 1 root root  1179648 2010-06-24 08:57 Dirnames 
At the destination:
$ ls -l /root/temp -rw-r--r-- 1 root root    12288 May 28  2008 Conflictname -rw-r--r-- 1 bin  bin   1179648 Jun 24 05:27 Dirnames -rw-r--r-- 1 root root        0 Sep  3 06:39 Basenames 
In the above example, between the source and destination, there are two differences. First, owner and group of the file Dirname differs. Next, size differs for the file Basenames.
Now let us see how rsync displays this difference. -i option displays the item changes.
$ rsync -avzi thegeekstuff@192.168.200.10:/var/lib/rpm/ /root/temp/ Password: receiving file list ... done >f.st.... Basenames .f....og. Dirnames  sent 48 bytes  received 2182544 bytes  291012.27 bytes/sec total size is 45305958  speedup is 20.76 
In the output it displays some 9 letters in front of the file name or directory name indicating the changes.
In our example, the letters in front of the Basenames (and Dirnames) says the following:
> specifies that a file is being transferred to the local host. f represents that it is a file. s represents size changes are there. t represents timestamp changes are there. o owner changed g group changed. 

Example 13. Include and Exclude Pattern during File Transfer

rsync allows you to give the pattern you want to include and exclude files or directories while doing synchronization.
$ rsync -avz --include 'P*' --exclude '*' thegeekstuff@192.168.200.10:/var/lib/rpm/ /root/temp/ Password: receiving file list ... done ./ Packages Providename Provideversion Pubkeys  sent 129 bytes  received 10286798 bytes  2285983.78 bytes/sec total size is 32768000  speedup is 3.19 
In the above example, it includes only the files or directories starting with ‘P’ (using rsync include) and excludes all other files. (using rsync exclude ‘*’ )

Example 14. Do Not Transfer Large Files

You can tell rsync not to transfer files that are greater than a specific size using rsync –max-size option.
$ rsync -avz --max-size='100K' thegeekstuff@192.168.200.10:/var/lib/rpm/ /root/temp/ Password: receiving file list ... done ./ Conflictname Group Installtid Name Sha1header Sigmd5 Triggername  sent 252 bytes  received 123081 bytes  18974.31 bytes/sec total size is 45305958  speedup is 367.35 
max-size=100K makes rsync to transfer only the files that are less than or equal to 100K. You can indicate M for megabytes and G for gigabytes.

Example 15. Transfer the Whole File

One of the main feature of rsync is that it transfers only the changed block to the destination, instead of sending the whole file.
If network bandwidth is not an issue for you (but CPU is), you can transfer the whole file, using rsync -W option. This will speed-up the rsync process, as it doesn’t have to perform the checksum at the source and destination.
#  rsync -avzW  thegeekstuff@192.168.200.10:/var/lib/rpm/ /root/temp Password: receiving file list ... done ./ Basenames Conflictname Dirnames Filemd5s Group Installtid Name  sent 406 bytes  received 15810211 bytes  2874657.64 bytes/sec total size is 45305958  speedup is 2.87 

6 Examples to Backup Linux Using dd Command (Including Disk to Disk)

Data loss will be costly. At the very least, critical data loss will have a financial impact on companies of all sizes. In some cases, it can cost your job. I’ve seen cases where sysadmins learned this in the hard way.

There are several ways to backup a Linux system, including rsync and rsnapshot that we discussed a while back.

This article provides 6 practical examples on using dd command to backup the Linux system. dd is a powerful UNIX utility, which is used by the Linux kernel makefiles to make boot images. It can also be used to copy data. Only superuser can execute dd command.

Warning: While using dd command, if you are not careful, and if you don’t know what you are doing, you will lose your data!
Example 1. Backup Entire Harddisk

To backup an entire copy of a hard disk to another hard disk connected to the same system, execute the dd command as shown below. In this dd command example, the UNIX device name of the source hard disk is /dev/hda, and device name of the target hard disk is /dev/hdb.

# dd if=/dev/sda of=/dev/sdb

“if” represents inputfile, and “of” represents output file. So the exact copy of /dev/sda will be available in /dev/sdb.
If there are any errors, the above command will fail. If you give the parameter “conv=noerror” then it will continue to copy if there are read errors.
Input file and output file should be mentioned very carefully, if you mention source device in the target and vice versa, you might loss all your data.

In the copy of hard drive to hard drive using dd command given below, sync option allows you to copy everything using synchronized I/O.

# dd if=/dev/sda of=/dev/sdb conv=noerror,sync

Example 2. Create an Image of a Hard Disk

Instead of taking a backup of the hard disk, you can create an image file of the hard disk and save it in other storage devices.There are many advantages to backing up your data to a disk image, one being the ease of use. This method is typically faster than other types of backups, enabling you to quickly restore data following an unexpected catastrophe.
# dd if=/dev/hda of=~/hdadisk.img

The above creates the image of a harddisk /dev/hda. Refer our earlier article How to view initrd.image for more details.
Example 3. Restore using Hard Disk Image

To restore a hard disk with the image file of an another hard disk, use the following dd command example.

# dd if=hdadisk.img of=/dev/hdb

The image file hdadisk.img file, is the image of a /dev/hda, so the above command will restore the image of /dev/hda to /dev/hdb.
Example 4. Creating a Floppy Image

Using dd command, you can create a copy of the floppy image very quickly. In input file, give the floppy device location, and in the output file, give the name of your floppy image file as shown below.

# dd if=/dev/fd0 of=myfloppy.img

Example 5. Backup a Partition

You can use the device name of a partition in the input file, and in the output either you can specify your target path or image file as shown in the dd command example below.

# dd if=/dev/hda1 of=~/partition1.img

Example 6. CDROM Backup

dd command allows you to create an iso file from a source file. So we can insert the CD and enter dd command to create an iso file of a CD content.

# dd if=/dev/cdrom of=tgsservice.iso bs=2048

dd command reads one block of input and process it and writes it into an output file. You can specify the block size for input and output file. In the above dd command example, the parameter “bs” specifies the block size for the both the input and output file. So dd uses 2048bytes as a block size in the above command.

Note: If CD is auto mounted, before creating an iso image using dd command, its always good if you unmount the CD device to avoid any unnecessary access to the CD ROM.

How to Setup Chroot SFTP in Linux (Allow Only SFTP, not SSH)

If you want to setup an account on your system that will be used only to transfer files (and not to ssh to the system), you should setup SFTP Chroot Jail as explained in this article.

In a typical sftp scenario (when chroot sftp is not setup), if you use sftp, you can see root’s file as shown below.

If you want to give sftp access on your system to outside vendors to transfer files, you should not use standard sftp. Instead, you should setup Chroot SFTP Jail as explained below.

Non-Chroot SFTP Environment

In the following example (a typical sftp environment), john can sftp to the system, and view /etc folder and download the files from there.

# sftp john@thegeekstuff.com john@thegeekstuff's password: sftp> pwd Remote working directory: /home/john  sftp> ls projects  john.txt documents   sftp> cd /etc sftp> ls -l passwd -rw-r--r--    0 0        0            3750 Dec 29 23:09 passwd  sftp> get passwd Fetching /etc/passwd to passwd /etc/passwd     100% 3750     3.7KB/s   00:00

Chroot SFTP Environment

In the following example, john can sftp to the system, and view only the directory that you’ve designated for john to perform sftp (i.e /incoming).

When john tries to perform ‘cd /etc’, it will give an error message. Since SFTP is setup in an chroot environment, john cannot view any other files in the system.

# sftp john@thegeekstuff.com john@thegeekstuff's password: sftp> pwd Remote working directory: /home/john  sftp> ls sftp> cd /etc Couldn't canonicalise: No such file or directory

Now that you know what Chroot SFTP environment is, let us see how to set this up.

1. Create a New Group

Create a group called sftpusers. Only users who belong to this group will be automatically restricted to the SFTP chroot environment on this system.

# groupadd sftpusers

2. Create Users (or Modify Existing User)

Let us say you want to create an user guestuser who should be allowed only to perform SFTP in a chroot environment, and should not be allowed to perform SSH.

The following command creates guestuser, assigns this user to sftpusers group, make /incoming as the home directory, set /sbin/nologin as shell (which will not allow the user to ssh and get shell access).

# useradd -g sftpusers -d /incoming -s /sbin/nologin guestuser # passwd guestuser

Verify that the user got created properly.

# grep guestuser /etc/passwd guestuser:x:500:500::/incoming:/sbin/nologin

If you want to modify an existing user and make him an sftp user only and put him in the chroot sftp jail, do the following:

# usermod -g sftpusers -d /incoming -s /sbin/nologin john

On a related note, if you have to transfer files from windows to Linux, use any one of the sftp client mentioned in this top 7 sftp client list.

3. Setup sftp-server Subsystem in sshd_config

You should instruct sshd to use the internal-sftp for sftp (instead of the default sftp-server).

Modify the the /etc/ssh/sshd_config file and comment out the following line:

#Subsystem       sftp    /usr/libexec/openssh/sftp-server

Next, add the following line to the /etc/ssh/sshd_config file

Subsystem       sftp    internal-sftp
# grep sftp /etc/ssh/sshd_config #Subsystem      sftp    /usr/libexec/openssh/sftp-server Subsystem       sftp    internal-sftp

4. Specify Chroot Directory for a Group

You want to put only certain users (i.e users who belongs to sftpusers group) in the chroot jail environment. Add the following lines at the end of /etc/ssh/sshd_config

# tail /etc/ssh/sshd_config Match Group sftpusers         ChrootDirectory /sftp/%u         ForceCommand internal-sftp

In the above:

  • Match Group sftpusers – This indicates that the following lines will be matched only for users who belong to group sftpusers
  • ChrootDirectory /sftp/%u – This is the path that will be used for chroot after the user is authenticated. %u indicates the user. So, for john, this will be /sftp/john.
  • ForceCommand internal-sftp – This forces the execution of the internal-sftp and ignores any command that are mentioned in the ~/.ssh/rc file.

5. Create sftp Home Directory

Since we’ve specified /sftp as ChrootDirectory above, create this directory (which iw equivalent of your typical /home directory).

# mkdir /sftp

Now, under /sftp, create the individual directories for the users who are part of the sftpusers group. i.e the users who will be allowed only to perform sftp and will be in chroot environment.

# mkdir /sftp/guestuser

So, /sftp/guestuser is equivalent to / for the guestuser. When guestuser sftp to the system, and performs “cd /”, they’ll be seeing only the content of the directories under “/sftp/guestuser” (and not the real / of the system). This is the power of the chroot.

So, under this directory /sftp/guestuser, create any subdirectory that you like user to see. For example, create a incoming directory where users can sftp their files.

# mkdir /sftp/guestuser/incoming

6. Setup Appropriate Permission

For chroot to work properly, you need to make sure appropriate permissions are setup properly on the directory you just created above.

Set the owenership to the user, and group to the sftpusers group as shown below.

# chown guestuser:sftpusers /sftp/guestuser/incoming

The permission will look like the following for the incoming directory.

# ls -ld /sftp/guestuser/incoming drwxr-xr-x 2 guestuser sftpusers 4096 Dec 28 23:49 /sftp/guestuser/incoming

The permission will look like the following for the /sftp/guestuser directory

# ls -ld /sftp/guestuser drwxr-xr-x 3 root root 4096 Dec 28 23:49 /sftp/guestuser  # ls -ld /sftp drwxr-xr-x 3 root root 4096 Dec 28 23:49 /sftp

7. Restart sshd and Test Chroot SFTP

Restart sshd:

# service sshd restart

Test chroot sftp environment. As you see below, when gusetuser does sftp, and does “cd /”, they’ll only see incoming directory.

# sftp guestuser@thegeekstuff.com guestuser@thegeekstuff's password:  sftp> pwd Remote working directory: /incoming  sftp> cd / sftp> ls incoming

When guestuser transfers any files to the /incoming directory from the sftp, they’ll be really located under /sftp/guestuser/incoming directory on the system.