Kernel Bug Causes Crash

[mkandoll]

I am using Virtuozzo 4.0 on Linux with roughly 60 hardware nodes. I need to enable serial console on ttyS1, but when enabled the hardware nodes encounter kernel panic quite often. About 20 minutes after enabling serial console on all the hardware nodes I was forced to disable it to prevent further crashes.

Searching google the only reference I could find was at:
https://bugzilla.redhat.com/show_bug.cgi?id=545382

This appears to be a bug in the upstream redhat kernel which has not been patched by redhat, at least in 2.6.18.

I checked vzup2date -m batch show and see a couple of updates are listed for the kernel, but none reference this particular bug.
Is there another kernel available to run Virtuozzo hardware nodes beyond the one provided in vzup2date?

The entry I had to comment out of /etc/inittab:
#T0:23:respawn:/sbin/agetty -L ttyS1 57600 vt100

An excerpt of the crash from /var/log/messages:
Feb 22 15:37:14 -server- kernel: Unable to handle kernel NULL pointer dereference at 0000000000000050 RIP:
Feb 22 15:37:14 -server- kernel: [<ffffffff801bf06f>] uart_put_char+0x42/0x64
Feb 22 15:37:14 -server- kernel: PGD 2a8e66067 PUD 3ad723067 PMD 0
Feb 22 15:37:14 -server- kernel: Oops: 0002 [1] SMP
Feb 22 15:37:14 -server- kernel: last sysfs file:
Feb 22 15:37:14 -server- kernel: CPU: 7
Feb 22 15:37:14 -server- kernel: Modules linked in: ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U) slm_dmprst(PU) ip_vzredir(PU) vzredir(PU) vzcompat(PU) vzrst(U) vzsnap(PU) vzcpt(U) vzfs(PU) vzevent(U) exportfs(U) vzlinkdev(PU) vzethdev(U) vzlist(PU) vznet(PU) vzstat(PU) vzmon(U) ip_vznetstat(PU) vznetstat(PU) ipt_recent(U) xt_tcpudp(U) slm_kill(PU) slm_nofork(PU) slm_core(PU) slm_skill(PU) slm_if(PU) vztable(PU) vzdquota(U) vzdev(U) iptable_nat(U) ip_nat(U) xt_length(U) ipt_ttl(U) xt_tcpmss(U) ipt_TCPMSS(U) iptable_mangle(U) ipt_tos(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) autofs4(U) nfs(U) lockd(U) nfs_acl(U) sunrpc(U) 8021q(U) ipt_LOG(U) xt_limit(U) ipt_REJECT(U) xt_state(U) ip_conntrack(U) nfnetlink(U) xt_multiport(U) iptable_filter(U) ip_tables(U) x_tables(U) dm_mirror(U) dm_log(U) dm_multipath(U) scsi_dh(U) dm_mod(U) video(U) backlight(U) sbs(U) power_meter(U) hwmon(U) i2c_ec(U) i2c_core(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) sg(U) i5000
Feb 22 15:37:14 -server- kernel: edac(U) bnx2(U) edac_mc(U) shpchp(U) pcspkr(U) serio_raw(U) ide_cd(U) cdrom(U) ata_piix(U) libata(U) megaraid_sas(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
Feb 22 15:37:14 -server- kernel: Pid: 25, comm: events/7 Tainted: P 2.6.18-028stab092.1 #1 028stab092
Feb 22 15:37:14 -server- kernel: RIP: 0060:[<ffffffff801bf06f>] [<ffffffff801bf06f>] uart_put_char+0x42/0x64
Feb 22 15:37:14 -server- kernel: RSP: 0000:ffff81043de35cc0 EFLAGS: 00010006
Feb 22 15:37:14 -server- kernel: RAX: 0000000000000000 RBX: ffff810175eaff48 RCX: 0000000000000000
Feb 22 15:37:14 -server- kernel: RDX: 0000000000000050 RSI: 0000000000000286 RDI: ffffffff805710c0
Feb 22 15:37:14 -server- kernel: RBP: ffffffff805710c0 R08: 0000000000000002 R09: 000000000000003d
Feb 22 15:37:14 -server- kernel: R10: ffff81043dd11dc8 R11: ffffffff801c2c41 R12: 000000000000020d
Feb 22 15:37:14 -server- kernel: R13: 000000000000010a R14: ffff8104281e5128 R15: ffff8101d2a569c0
Feb 22 15:37:14 -server- kernel: FS: 0000000000000000(0000) GS:ffff81043dcc8940(0000) knlGS:0000000000000000
Feb 22 15:37:14 -server- kernel: CS: 0060 DS: 0018 ES: 0018 CR0: 000000008005003b
Feb 22 15:37:14 -server- kernel: CR2: 0000000000000050 CR3: 00000003da00b000 CR4: 00000000000006e0
Feb 22 15:37:14 -server- kernel: Process events/7 (pid: 25, veid=0, threadinfo ffff81043de34000, task ffff81043de30b60)
Feb 22 15:37:14 -server- kernel: Stack: ffff8101d2a56800 ffff8101d2a5680a 0000000000000246 ffffffff80050ac1
Feb 22 15:37:14 -server- kernel: ffff8101d2a56d4c 000000000000000d ffff8101d2a56800 ffffffff8001492b
Feb 22 15:37:14 -server- kernel: 0000000000000001 0000000100000002 ffff8104281e5042 ffff8104281e5143
Feb 22 15:37:14 -server- kernel: Call Trace:
Feb 22 15:37:14 -server- kernel: [<ffffffff80050ac1>] opost+0x99/0x1b9
Feb 22 15:37:14 -server- kernel: [<ffffffff8001492b>] n_tty_receive_buf+0x994/0xdc6
Feb 22 15:37:14 -server- kernel: [<ffffffff8008bd6c>] vcpu_put+0x8e/0x16f
Feb 22 15:37:14 -server- kernel: [<ffffffff800644fd>] thread_return+0x6a/0x177
Feb 22 15:37:14 -server- kernel: [<ffffffff801a6b36>] flush_to_ldisc+0x127/0x187
Feb 22 15:37:14 -server- kernel: [<ffffffff801a6a0f>] flush_to_ldisc+0x0/0x187
Feb 22 15:37:14 -server- kernel: [<ffffffff80050039>] run_workqueue+0x99/0xf7
Feb 22 15:37:14 -server- kernel: [<ffffffff8004c7cb>] worker_thread+0x0/0x122
Feb 22 15:37:14 -server- kernel: [<ffffffff8004c8bb>] worker_thread+0xf0/0x122
Feb 22 15:37:14 -server- kernel: [<ffffffff8008b605>] default_wake_function+0x0/0xe
Feb 22 15:37:14 -server- kernel: [<ffffffff80033d25>] kthread+0xfe/0x132
Feb 22 15:37:14 -server- kernel: [<ffffffff80061001>] child_rip+0xa/0x11
Feb 22 15:37:14 -server- kernel: [<ffffffff80033c27>] kthread+0x0/0x132
Feb 22 15:37:14 -server- kernel: [<ffffffff80060ff7>] child_rip+0x0/0x11
Feb 22 15:37:14 -server- kernel:
Feb 22 15:37:14 -server- kernel:
Feb 22 15:37:14 -server- kernel: Code: 44 88 24 10 8b 43 08 ff c0 25 ff 0f 00 00 89 43 08 5b 48 89
Feb 22 15:37:14 -server- kernel: RIP [<ffffffff801bf06f>] uart_put_char+0x42/0x64
Feb 22 15:37:14 -server- kernel: RSP <ffff81043de35cc0>
Feb 22 15:37:14 -server- kernel: CR2: 0000000000000050
Feb 22 15:37:14 -server- kernel: Kernel panic - not syncing: Fatal exception
Feb 22 20:52:14 -server- syslogd 1.4.1: restart.

[root@-server- ~]# uname -a
Linux -server- 2.6.18-028stab092.1 #1 SMP Wed Jul 20 19:47:12 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@-server- ~]# vzctl --version
vzctl v.4.0.0-226

[root@-server- ~]# rpm -qa | grep -i vz
vzrpm-4.2.1-4.2.36.swsoft
vzrpm41s9x64-4.1.1-177.16.2.swsoft
vzquota-4.0.0-12
vzlicutils-4.0.0-68.swsoft
vzkernel-2.6.18-028stab057.4
vzagent-opcontrol-4.0.0-351.61.swsoft
vzpkgenv44x64-4.0.0-41
vztt-4.0.0-133
vzrpm43x64-4.3.3-7_nonptl.12.swsoft
vzmodules-2.6.18-028stab053.10
vzagent-compat-opcontrol-4.0.0-76.swsoft
vzreport-4.0.0-11.swsoft
vzstat-4.0.0-31.swsoft
vzpkgenvdebx64-4.0.0-35.swsoft
vzsnmp-proxy-4.0.0-40.swsoft
vzmigrate-4.0.0-140
vzabackup-4.0.0-50.swsoft
vzpkgenv41s9x64-4.0.0-41
vzctl-lib-4.0.0-226
vzmodules-2.6.18-028stab092.1
vzrpm43nonptl-4.3.3-7_nonptl.2.swsoft
vzrpmx64-4.2.1-4.2.32.swsoft
vzkernel-install-4.0.1-5rh.swsoft
vzlinmigrate-4.0.0-16.swsoft
vzagent-compat-lib-4.0.0-76.swsoft
vzkernel-2.6.18-028stab053.14
vzyum-plugins-2.4.2-35.swsoft
deb2vzdeb-4.0.0-8.swsoft
vzvpn-4.0.0-3.swsoft
vzagent-distribution-x86_64-4.0.0-351.42.swsoft
vznetcfg-lib-4.0.0-115
vzkernel-2.6.18-028stab060.8
vzpkgenv43x64-4.0.0-41
vzpkgenv41x64-4.0.0-41
vzctl-4.0.0-226
vzup2date-4.0.0-116
vz-redhat-compat-4.0.0-2.swsoft
vzmodules-2.6.18-028stab053.14
vzrpm43-4.3.3-7_nonptl.11.swsoft
vzrpm44x64-4.4.1-22.10.swsoft
vzrpm41-4.1.1-222.2.swsoft
vzmodules-2.6.18-028stab053.4
vzhwcalc-4.0.0-3.swsoft
vzsvetools-4.0.0-71.swsoft
vzmodules-2.6.18-028stab057.4
vzfsutil-4.0.0-35.swsoft
vzagent-distribution-4.0.0-351.42.swsoft
vznetcfg-4.0.0-115
vzmodules-2.6.18-028stab060.8
vzpkgenv44-4.0.0-41
vzpkgenv41-4.0.0-41
vztt-build-4.0.0-133
libvzsnap-4.0.0-15.swsoft
vzrpm44-4.4.1-22.10.swsoft
vzrpm41s9-4.1.1-177.16.2.swsoft
vzkernel-2.6.18-028stab053.10
vzpkgtools-4.0.0-21.swsoft
vzpkgenv43-4.0.0-41
vzpkgenv41s9-4.0.0-41
vztt-lib-4.0.0-133
vzkernel-2.6.18-028stab092.1
vzrpm41x64-4.1.1-222.2.swsoft
vzkernel-2.6.18-028stab053.4
vzprocps-3.2.3-8.20.swsoft
vzbackup-4.0.0-62.swsoft
vzpkgenvdeb-4.0.0-35.swsoft
rpm2vzrpm-4.2.1-4.2.43.swsoft
vzdump-1.0-2
vzlinmigrate-lib-4.0.0-42.14.swsoft
vzcache-4.0.0-23.6.swsoft

[MakcyD]

Hello,

Only kernels listed in vzup2date are working on PVC hardware node.
Latest update available (CU-2.6.18-028stab098.1) is rebased on 2.6.18-275.18.1.el5, but as you stated, it does not include the fix for this issue.

This issue is not likely to be fixed by Parallels team unless the patch is implemented upstream.

[mkandoll]

I was afraid of that.

This bug essentially prevents the use of the bmc for serial over lan capabilities and will require more physical hands on the machines.

Do Virtuozzo 4.6 and 4.7 have support for kernels other than 2.6.18?

[MakcyD]

PVC 4.7 is rebased on 2.6.32 kernels, so upgrading to PVCfL 4.7 should help.
By the way, upgrading among 4.x versions is free of charge, so if the host OS is CentOS\RHEL 5.x upgrading should not make any problems.

[mkandoll]

Is there any documented procedure for upgrade from Virtuozzo 4.0 to 4.7?

[KristianM]

Hi,

no idea why, but it's a doc file:
http://download.parallels.com/pvc/47...nuxUpgrade.doc

Regards,
Kristian

[MakcyD]

Hi, there was quite a mess with documentation links at the moment of release, but now all docs are collected here:
parallels.com > Products > Parallels Virtuozzo Containers > Deployment Resources

All docs and upgrade guide in particular are in PDF.