A RHEL6 server kernel got updated but GRUB entries were not, causing an older kernel to be loaded on boot.

Background

Our hosting provider’s SLA include installing official updates* to our RHEL6 servers. As we do use some 3rd party repositories, we constantly monitor for available updates.

* excluding any software from 3rd party repositories

Most of the time we react faster to available updates than our hosting provider and tend to install even the official updates ourselves.

Last thursday our hosting provider installed a new kernel version 2.6.32-696.28.1.el6.x86_64. I mention this as I do not know what RHEL-specific infrastructure software they use when installing updates. Common people like us just use yum update from the command line…

What

Today I wanted to get rid of icinga’s notifications about server(s) requiring a reboot due updated kernel.

I did the usual routine for every server:

  • Remove server from the load balancer pool
  • Run yum update to install other (possible) updates
  • Reboot the server
  • Restore server to the load balancer pool

and felt good about myself as this was one task off from my TODO list for today.

Until I noticed that icinga still complains about the kernel.

How

Simple.

There are three kernels installed in /boot:

  • 2.6.32-696.20.1
  • 2.6.32-696.23.1
  • 2.6.32-696.28.1

The /boot/grub/grub.conf only contains references to the first two, excluding the 2.6.32-696.28.1.

Why?

I have no idea.

I’m not going to manually regenerate the grub.conf, instead I’m just going to wait for the next kernel update to see if things will fix themselves out.

I might change my mind though, who knows.

Update 2018-05-29

Probably this has something to do with this issue. Unfortunately I cannot access the full post.

I could work around the issue by running

$ sudo yum reinstall kernel-2.6.32-696.30.1.el6.x86_64

(Yes, our service provider seems to have installed yet another kernel version through satellite).

I hope that RedHat fixes the actual issue soon.