Tag Archives: Exadata

Clobbering grub.conf is Bad

I’m sharing this in the hope of saving someone from an unwelcome surprise.

Background

I recent upgraded an Exadata system from 11.2.3.2.1 to 11.2.3.3.1. Apart from what turns out to be a known bug[1] that resulted in the patching of the InfiniBand switches “failing”, it all seemed to go without a snag. That’s until I decided to do some node failure testing…

Having forced a node eviction I got on with something else while the evicted compute node (database server to non-Exadata folk) booted. After what seemed like a reasonable amount of time, and at an appropriate break in my other work, I attempted to connect to the previously evicted host. No joy. I connected to the ILOM console to see what was going on only to find something like this:

grub_prompt

My first though was, “Is there any conceivable way that causing a node eviction through generation of very heavy swap activity could be responsible for this?”

Investigation

Attempting to boot the host from the grub prompt using the kernel installed as part of 11.2.3.3.1 worked without any issues. Once the host was up I looked at /boot/grub/grub.conf. It was empty (zero bytes). I checked all compute nodes and found the same. Obviously this must have happened after the last reboot of the hosts otherwise they would have failed to boot beyond the grub prompt.

I raised an SR as this seemed like a big deal to me. Not that it wasn’t recoverable, but because it is a gremlin lying in wait to bite when least welcome. It’s easy to imagine a situation where having upgraded to 11.2.3.3.1 the environment is put back to use and at some point later a node is evicted or rebooted for another reason and it doesn’t boot back into the OS. That would suck. I expect my hosts to be in a state that allows them to boot cleanly; if that’s not the case then I want to know about it and be prepared.

The initial response in the SR was that I should run the upgrade again without the “Relax and Recover[2] entry in grub.conf stating, “… as there is some suspicion this might be related.”

Having restored the pre-upgrade backup on one of the compute nodes, I ran the upgrade again and started to investigate in detail. Rather than give a blow-by-blow account of that investigation, I’ll cut straight to the final conclusion.

Culprit

misceachboot (part of the Exadata “validations” framework and as the name suggests it runs every time an Exadata compute node boots) clobbers /boot/grub/grub.conf shortly after host startup if an entry for “Oracle Linux Server (2.6.18-308.24.1.0.1.el5)” is found in grub.conf. This is consistently repeatable with the simple test of copying the backup of grub.conf created by dbnodeupdate.sh over the empty grub.conf and rebooting.

As I’ve stated in the SR with Oracle, this appears to be something that would affect all Exadata systems that are upgraded from 11.2.3.2.1 to 11.2.3.3.1. Oracle Support have created bug 19428028, which is not visible at the time I write this.

If someone else had run into this problem then I’d like her/him to share it publicly so that I didn’t get caught out by it. Hence this blog post.

I would be very interested to hear from other Exadata users that have upgraded to 11.2.3.3.1, particularly if it was from 11.2.3.2.1, and whether or not they have seen the same problem.

Update (20th August 2014)

Having found more time to investigate I believe I’ve found the exact cause of the issue…

In the comments of this post I previously stated:

… the “Oracle Linux Server (2.6.18-308.24.1.0.1.el5)” entry is definitely the trigger.

Well, that’s not true!

Also, having taken the time to identify exactly what part of the image_functions code truncates grub.conf, it is now possible to be confident that the other 11.2.3.2.1 systems I have access to will not be affected.

So anyway, here’s the important points:

Point 1

Within image_functions a function named “image_functions_remove_from_grub” is defined that includes the following:

perl -00 -ne '/vmlinuz-$ENV{EXA_REMOVE_KERNEL_FROM_GRUB} / or print $_' -i $grub_conf

The “-00” part is particularly relevant. This invokes “paragraph mode”, which defines a paragraph as being the characters between two non-consecutive newlines.

The Perl command has the effect of removing any paragraph from $grub_conf (defined as /boot/grub/grub.conf) that contains the string “vmlinuz-$ENV{EXA_REMOVE_KERNEL_FROM_GRUB} “, where EXA_REMOVE_KERNEL_FROM_GRUB is a shell variable.

Point 2

For a reason I have yet to identify the grub.conf files on the compute nodes of the particular Exadata system that had been upgraded to 11.2.3.2.1 had spaces appended to the end of lines. The number of spaces appended was not consistent across nodes and I’ll continue to try to identify what was responsible. Anyway, this resulted in each break between entries in grub.conf not being simply a newline character, but rather a line with a number of spaces before the newline character.

End Result

I probably don’t need to explain, but in case it isn’t obvious: the combination of point 1 and 2 above means that the entire contents of grub.conf is seen as a single paragraph by the Perl command and as that paragraph contains the kernel referenced by the shell variable $EXA_REMOVE_KERNEL_FROM_GRUB it is removed from grub.conf resulting in an empty file.

Other Points

The incorrect assertion that it was the 2.6.18-308.24.1.0.1.el5 kernel entry that triggered the problem is the result of misceachboot repeatedly attempting to remove kernel 2.6.18-308.24.1.0.1.el5. If I’ve followed the logic in misceachboot correctly then the attempt to remove kernel 2.6.18-308.24.1.0.1.el5 happens on each boot because the rpm for that kernel is still “installed” even though the kernel files have been removed from /boot (and the entry removed from grub.conf). This is done because of a dependency between fuse-2.7.4-8.0.5.el5.x86_64 and kernel 2.6.18-308.24.1.0.1.el5. Whereas the attempt to remove kernel 2.6.32-400.21.1.el5uek is only performed once (assuming it is successful) after the first reboot post upgrade to 11.2.3.2.1 during which the rpm is completely removed along with the kernel files (and the entry removed from grub.conf).

Footnotes

1 – The bug is known and documented in MOS ID 1614149.1, however, there is no mention of it in MOS ID 1667414.1, which being entitled “Exadata 11.2.3.3.1 release and patch (17636228 )” and having a section of “Known Issues” seems like a reasonable place to expect it to be referenced. I have suggested to Oracle Support that an update to 1667414.1 referencing 1614149.1 would be a good idea. That was on 4th August 2014 and so far there’s been no update.
2 – Relax and Recover (rear) is a very useful bare-metal recovery tool for Linux.

Kdump to NFS in UEK (Solution)

I’ve previously written about a problem I encountered when kdump is configured to write to an NFS location with UEK (in Exadata software version 11.2.3.2.1). I’m please to report that the root cause of the problem has been identified and there is a very simple workaround.

There were some frustrating times working this particular SR, the most notable being a response that was effectively, “It works for me (and so I’ll just put the status to Customer Working).”

After a bit more to-ing and fro-ing it emerged that the environment where Oracle had demonstrated kdump could write to NFS had the NFS server on the same subnet as the host where kdump was being tested. After a quick test of my own, using a second compute node in the Exadata system as the NFS server, I confirmed that kdump was able to write to an NFS location on the same subnet in my environment as well.

Soon after reporting the above test in the SR I was pointed to MOS note 1533611.1, which unfortunately is not publicly available (yet) and so I cannot read it… The crux of the issue is that the network interface configuration files have BOOTPROTO=none and kdump is not handling this appropriately, which results in an incomplete network configuration for bond1 when switching to the dump kernel during a crash.

The fix: Change BOOTPROTO=none to BOOTPROTO=static

The part of all this that I find interesting is that the documentation for <a href="https://access prix cialis en france.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/s1-networkscripts-interfaces.html”>RHEL 5 and RHEL 6, and even RHEL 7, all list the following options for BOOTPROTO:

  • none
  • bootp
  • dhcp

“static” does not appear to be a formally valid value. In an attempt to find more information about the behaviour I looked for more details and only got as far as Red Hat BZ#805803 and BZ#802928, neither of which I can access directly, but I can see a summary here and here respectively.

In conclusion, it appears that the issue is actually a kdump bug. More specifically a mkdumprd bug. Thankfully the workaround is simple, it just took a long time to get to it.

Kdump to NFS Broken in UEK

There were problems that affected UEK and NFS when 11.2.3.2.1 was initially released (as covered by Andy Colvin). As mentioned in the <a href="http://blog.oracle-ninja cialis 5mg pas cher.com/2013/03/exadata-11-2-3-2-1-nfs-issues-ksplice-support-for-exadata/#comment-1067″>comments of Andy’s post: Oracle released an updated ISO with fixes for this problem (patch 16432033).

There were also problems with kdump not functioning after 11.2.3.2.1 as listed in “Issue 1.17” of Oracle MOS “Exadata 11.2.3.2.1 release and patch (14522699) for Exadata 11.1.3.3, 11.2.1.2.x, 11.2.2.2.x, 11.2.2.3.x, 11.2.2.4.x, 11.2.3.1.x, 11.2.3.2.x (Doc ID 1485475.1)”.

After updating to 11.2.3.2.1 using the updated ISO (with NFS fixes) and installing the later version of kexec-tools as per the fix for “Issue 1.17” I was not able to get kdump to write to NFS during a crash.

I had previously tested kdump to NFS when running Exadata software version 11.2.2.4.2, so I knew that worked. 11.2.2.4.2 uses the RHEL kernel (“non-UEK” as Oracle would have you say) so I decided to test 11.2.3.2.1 after switching to the RHEL kernel, which also involves reverting the kexec-tools to the earlier version. That confirmed the issue was only evident when using UEK. Time for an SR…

Bug 17730336 – UNABLE TO KDUMP OVER NFS WITH KEXEC-TOOLS-2.0.3-3.0.4.EL5.X86_64.RPM INSTALLED

The bug listed above has been raised, but no fix has been supplied yet.

Not being able to direct kdump to NFS is probably not going to be your biggest worry, but definitely something to be aware of if you’re running UEK.

Note that I have only tested with 2.6.32-400.21.1el5uek. Oracle have confirmed they have reproduced the issue with 2.6.39-400.126.1.el5uek. Other UEK kernels may or may not be affected.