Tag Archives: ReaR

ReaR MAC Address Mix-Up

Relax and Recover (ReaR) is a great tool for facilitating Linux bare-metal recovery. In works really well, however, there is a bug in the 1.14 release (it seems the same issue is present in 1.15, but I didn’t test yet) that effects restores for many bonded Ethernet interface configurations.

Problem

After the recovery of an Exadata compute node using ReaR I saw the following message during the boot process:

Bringing up interface bond1:  Device eth2 has different MAC address than expected, ignoring.

Examination of /etc/sysconfig/network-scripts/ifcfg-eth2 and dmesg showed that sure enough the MAC address in /etc/sysconfig/network-scripts/ifcfg-eth2 did not match that of the device… Hmmm.

A quick look in the restore log file revealed:

2013-09-02 20:20:47 Including finalize/GNU/Linux/30_create_mac_mapping.sh
2013-09-02 20:20:48 Including finalize/GNU/Linux/41_migrate_udev_rules.sh
2013-09-02 20:20:48 Including finalize/GNU/Linux/42_migrate_network_configuration_files.sh
2013-09-02 20:20:48 SED_SCRIPT: ';s/<original mac address removed>/<new mac address removed>/g;s/<ORIGINAL MAC ADDRESS REMOVED>/<NEW MAC ADDRESS REMOVED>/g'

Just to be clear the parts in < and > have been removed by me and represent place-holders for the MAC addresses. First in lower case and then upper case.

Cause

The cause of the issue is in 30_create_mac_mapping.sh, which is responsible for creating a MAC address mapping file (if needed). The MAC address mapping file ($CONFIG_DIR/mappings/mac) is created to handle situations where the restore is to a different host, or at least one where the MAC addresses for the network cards are different to those on the system that was backed up.

The 30_create_mac_mapping.sh script is really very simple and compares the MAC address in the restored ifcfg-<interface> files (if there are any) with the MAC addresses in /sys/class/net/<interface>/address. If there is a difference then it writes the old and current MAC addresses to the MAC address mapping file ($CONFIG_DIR/mappings/mac[1]) for later use by 42_migrate_network_configuration_files.sh. All good, right? Well, a problem can come about when dealing with bonded interfaces. Specifically, when in active-backup mode with fail_over_mac set to none (0 has the same meaning). In this configuration /sys/class/net/<interface>/address reports the same value for all slaves of a bond.

So, given that when booting into recovery mode ReaR attempts to create the network configuration at the time the ReaR boot ISO was created (via script 60-network-devices.sh), if you had bonded interfaces at the point you created the ReaR boot ISO, then you’ll have them in recovery mode, which means that when 30_create_mac_mapping.sh runs it will write MAC addresses to $CONFIG_DIR/mappings/mac, and then shortly after 42_migrate_network_configuration_files.sh will run and update the ifcfg-<interface> files, setting the MAC address for all interfaces in a given bond to the same value, which is not correct.

Solution

After initially thinking that I was going to need to come up with a patch for ReaR that would handle bonded interfaces appropriately, and I thinking that’s more complicate than it might sound, I had another look at 30_create_mac_mapping.sh and realised that if I create an empty $CONFIG_DIR/mappings/mac file (valid in my case as the MAC addresses have not changed when doing a straight restore to the same host) then ReaR will not create a new file, or add records to the existing file, and there will be no attempt to update the MAC addresses in the ifcfg-<interface> scripts.

The above worked and so a fix for the MAC address mix-up with bonded interfaces, when restoring to a host with the same MAC addresses, is to run the following command[1] before running “rear recover”:

# touch /etc/rear/mappings/mac

Footnotes
[1] – $CONFIG_DIR is a variable used in Relax and Recover, but note that your $CONFIG_DIR might not be /etc/rear

Relax and Recover

Relax and Recovery (ReaR) is great. It does exactly what it needs to do with the minimum of fuss.

What?

I’ll start by explaining what ReaR does and does not do. The “does not” part is important as many people I’ve spoken with about ReaR have been confused about exactly what it does.

The main purpose of ReaR is to create a bootable image, based on what is currently installed on a Linux host, that can be used to partition disks and retrieve a backup of the system. There are options for where to create the bootable image and what to do with it after it has been created.

The bootable image can be a USB device, an ISO file or a number of other options.

If you create a bootable image on a USB device then you may also wish to create a backup of your system on the same device, which ReaR will support.

When creating a bootable image as an ISO file you have a multitude of options for what do to with the file in order to get it off the box so that it can be used for recovery. The two options I have used are rsync and TSM.

The misconception I mentioned earlier is the belief that ReaR will backup your system. It can do that, but it is not a given and depends on your configuration acheter du cialis 5.

If ReaR isn’t backing up the system, what will?

ReaR will work with tar, rsync and a number of 3rd party commercial backup solutions. Again, I have used rsync and TSM.

Why?

You might be asking yourself why this is important/useful.

Imagine your beloved machine has suffered a death by file system corruption. You have a backup of the system, but what next? You need a way of getting the backup data back on disk. I know a number of places that do not restore full systems, but rather rebuild them if something goes horribly wrong with the operating system. I can see the value in the approach, however, it relies on you knowing what the state of the system was. For example:

  1. What configuration changes have been made since the installation?
  2. What additional software has been installed?
  3. What scripts have been put in place?

The list goes on.

It is totally possible to manage all that, but you have to be proactive. It’s a problem that won’t solve itself.

If you don’t have details of all the customisations then wouldn’t it be really nice to be able to run a command to pull back the contents of all your local file systems as they were at the point of the last backup, allowing you to simply reboot and be back in business? That’s what ReaR will do for you 🙂

Example Procedure

The following is an example of the produce to protect a system with ReaR and TSM during some operating system patching activities (assumes TSM is already installed):

  1. Install ReaR (rpms are available here).
  2. Configure ReaR to use TSM and to create an ISO file by updating /etc/rear/local.conf with a line of OUTPUT=ISO and another with BACKUP=TSM.
  3. Run “rear -v mkrescue” to create the bootable ISO and send it to TSM (mkbackup would have the same effect in this case as TSM will be handling the file system backups independently – I feel mkrescue makes it clearer what you’re doing).
  4. Perform a incremental backup of your file systems with TSM using “dsmc inc …”.
  5. Do your patching activity.

If all goes well then you don’t need to boot from the ReaR ISO and restore you operating system. But, let’s say it didn’t go well. Your system will no longer boot and there’s no immediately obvious way forward. You decide to restore. The procedure is:

  1. Restore the ReaR ISO to a location that will allow you to present it to the server. This is most likely to be your desktop so you can present the ISO file as a virtual CD-ROM over the ILOM interface.
  2. Present the ISO to the host to be recovered.
  3. Boot the host from the ISO – It is highly likely that you’ll need to change the boot order or get a pop-up menu to select the ISO as the boot media.
  4. Select “Recover <hostname>” at the grub prompt.
  5. Log in as root (password not required).
  6. Run “rear -v recover” and answer the interactive prompts.

Issues

Since starting to use ReaR I have encountered two problems:

  1. When recovering a host that used an ext4 file system for /boot I found myself facing at message of “Error 16: Inconsistent filesystem structure.” from grub. After a bit of digging around and trying to understand what the issue was I ended up modifying the /var/lib/rear/layout/disklayout.conf ReaR file to change the file system type for /boot from ext4 to ext2. I initially tried ext3, but as the system did not use ext3 for any of the file systems the module was not available.
  2. The version of ReaR that I was using had a bug (tracked on GitHub) that affected systems that do not have a separate /boot partition. There is a patch for the bug available, but if like me you’re happy to have a manual workaround, you need to perform the following actions after the restore completes:
# chroot /mnt/local
# PATH=/bin:/sbin:/usr/bin
# grub-install <disk path>
# exit
# reboot

Finally, it’s worth mentioning that ReaR is written in shell and is open source.