我正在redhat6.3-6.5上运行hadoop2.2，我的所有机器都在一段时间后崩溃了。 /var/log/messages 重复显示：

Aug 11 06:30:42 jn4_73_128 kernel: BUG: soft lockup - CPU#1 stuck for 67s! [jsvc:11508]
Aug 11 06:30:42 jn4_73_128 kernel: Modules linked in: bridge stp llc iptable_filter ip_tables mptctl mptbase xfs exportfs power_meter microcode dcdbas serio_raw iTCO_w
dt iTCO_vendor_support i7core_edac edac_core sg bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif wmi mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_m
od [last unloaded: scsi_wait_scan]
Aug 11 06:30:42 jn4_73_128 kernel: CPU 1 
Aug 11 06:30:42 jn4_73_128 kernel: Modules linked in: bridge stp llc iptable_filter ip_tables mptctl mptbase xfs exportfs power_meter microcode dcdbas serio_raw iTCO_w
dt iTCO_vendor_support i7core_edac edac_core sg bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif wmi mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_m
od [last unloaded: scsi_wait_scan]
Aug 11 06:30:42 jn4_73_128 kernel: 
Aug 11 06:30:42 jn4_73_128 kernel: Pid: 11508, comm: jsvc Tainted: G        W  ---------------    2.6.32-279.el6.x86_64 #1 Dell Inc. PowerEdge R510/084YMW
Aug 11 06:30:42 jn4_73_128 kernel: RIP: 0010:[<ffffffff8104d088>]  [<ffffffff8104d088>] wait_for_rqlock+0x28/0x40
Aug 11 06:30:42 jn4_73_128 kernel: RSP: 0018:ffff8807786c3ee8  EFLAGS: 00000202
Aug 11 06:30:42 jn4_73_128 kernel: RAX: 00000000f6e9f6e1 RBX: ffff8807786c3ee8 RCX: ffff880028216680
Aug 11 06:30:42 jn4_73_128 kernel: RDX: 00000000fffff6e9 RSI: ffff88061cd29370 RDI: 0000000000000286
Aug 11 06:30:42 jn4_73_128 kernel: RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000001
Aug 11 06:30:42 jn4_73_128 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286
Aug 11 06:30:42 jn4_73_128 kernel: R13: ffff8807786c3eb8 R14: ffffffff810e0f6e R15: ffff8807786c3e48
Aug 11 06:30:42 jn4_73_128 kernel: FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
Aug 11 06:30:42 jn4_73_128 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 11 06:30:42 jn4_73_128 kernel: CR2: 0000000000e5bd70 CR3: 0000000001a85000 CR4: 00000000000006e0
Aug 11 06:30:42 jn4_73_128 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 11 06:30:42 jn4_73_128 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 11 06:30:42 jn4_73_128 kernel: Process jsvc (pid: 11508, threadinfo ffff8807786c2000, task ffff880c1def3500)
Aug 11 06:30:42 jn4_73_128 kernel: Stack:
Aug 11 06:30:42 jn4_73_128 kernel: ffff8807786c3f68 ffffffff8107091b 0000000000000000 ffff8807786c3f28
Aug 11 06:30:42 jn4_73_128 kernel: <d> ffff880701735260 ffff880c1def39c8 ffff880c1def39c8 0000000000000000
Aug 11 06:30:42 jn4_73_128 kernel: <d> ffff8807786c3f28 ffff8807786c3f28 ffff8807786c3f78 00007f092d0ad700
Aug 11 06:30:42 jn4_73_128 kernel: Call Trace:
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff8107091b>] ? do_exit+0x5ab/0x870
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff81070ce7>] ? sys_exit+0x17/0x20
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
Aug 11 06:30:42 jn4_73_128 kernel: Code: ff ff 90 55 48 89 e5 0f 1f 44 00 00 48 c7 c0 80 66 01 00 65 48 8b 0c 25 b0 e0 00 00 0f ae f0 48 01 c1 eb 09 0f 1f 80 00 00 00 00 <f3> 90 8b 01 89 c2 c1 fa 10 66 39 c2 75 f2 c9 c3 0f 1f 84 00 00 
Aug 11 06:30:42 jn4_73_128 kernel: Call Trace:
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff8107091b>] ? do_exit+0x5ab/0x870
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff81070ce7>] ? sys_exit+0x17/0x20
Aug 11 06:30:42 jn4_73_128 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
</em>

最后坠毁了

crash /usr/lib/debug/lib/modules/2.6.32-431.5.1.el6.x86_64/vmlinux  /opt/crash/127.0.0.1-2014-08-10-09\:47\:38/vmcore

crash 6.1.0-5.el6
Copyright (C) 2002-2012  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

please wait... (determining panic task)         
WARNING: active task ffff881071850040 on cpu 12 not found in PID hash

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.5.1.el6.x86_64/vmlinux
    DUMPFILE: /opt/crash/127.0.0.1-2014-08-10-09:47:38/vmcore  [PARTIAL DUMP]
        CPUS: 24
        DATE: Sun Aug 10 09:47:32 2014
      UPTIME: 7 days, 16:00:19
LOAD AVERAGE: 11.01, 3.11, 1.08
       TASKS: 724
    NODENAME: master1.otocyon.com
     RELEASE: 2.6.32-431.5.1.el6.x86_64
     VERSION: #1 SMP Fri Jan 10 14:46:43 EST 2014
     MACHINE: x86_64  (1895 Mhz)
      MEMORY: 64 GB
       PANIC: "Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0"
         PID: 23976
     COMMAND: "sh"
        TASK: ffff881071850aa0  [THREAD_INFO: ffff880a05c80000]
         CPU: 0
       STATE: TASK_INTERRUPTIBLE (PANIC)

crash> bt
PID: 23976  TASK: ffff881071850aa0  CPU: 0   COMMAND: "sh"
 #0 [ffff880028207b50] machine_kexec at ffffffff81038f3b
 #1 [ffff880028207bb0] crash_kexec at ffffffff810c5d82
 #2 [ffff880028207c80] panic at ffffffff8152751a
 #3 [ffff880028207d00] watchdog_overflow_callback at ffffffff810e696d
 #4 [ffff880028207d20] __perf_event_overflow at ffffffff8111c847
 #5 [ffff880028207da0] perf_event_overflow at ffffffff8111ce14
 #6 [ffff880028207db0] intel_pmu_handle_irq at ffffffff81022d87
 #7 [ffff880028207e90] perf_event_nmi_handler at ffffffff8152bd69
 #8 [ffff880028207ea0] notifier_call_chain at ffffffff8152d825
 #9 [ffff880028207ee0] atomic_notifier_call_chain at ffffffff8152d88a

# 10 [ffff880028207ef0] notify_die at ffffffff810a153e

# 11 [ffff880028207f20] do_nmi at ffffffff8152b4eb

# 12 [ffff880028207f50] nmi at ffffffff8152adb0

    [exception RIP: task_rq_unlock_wait+44]
    RIP: ffffffff810534fc  RSP: ffff880a05c81dc8  RFLAGS: 00000016
    RAX: 000000000ec70ebe  RBX: ffff881071850040  RCX: ffff8800282d6840
    RDX: 0000000000000ec7  RSI: 0000000000000000  RDI: ffff881071850040
    RBP: ffff880a05c81dc8   R8: dead000000200200   R9: dead000000200200
    R10: ffff8810734a42d0  R11: 0000000000000246  R12: 00000000000114b8
    R13: ffff8810734a4180  R14: ffff881071fd3440  R15: ffff881071fd3c48
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---

# 13 [ffff880a05c81dc8] task_rq_unlock_wait at ffffffff810534fc

# 14 [ffff880a05c81dd0] release_task at ffffffff81075454

# 15 [ffff880a05c81e10] wait_consider_task at ffffffff81075fb6

# 16 [ffff880a05c81e80] do_wait at ffffffff810763e6

# 17 [ffff880a05c81ee0] sys_wait4 at ffffffff810765d3

# 18 [ffff880a05c81f80] system_call_fastpath at ffffffff8100b072

    RIP: 0000003e1a2ac8be  RSP: 00007fffa58c6330  RFLAGS: 00010207
    RAX: 000000000000003d  RBX: ffffffff8100b072  RCX: 0000003e1a232be0
    RDX: 0000000000000000  RSI: 00007fffa58c62ec  RDI: ffffffffffffffff
    RBP: 00000000ffffffff   R8: 000000000203b8d0   R9: 000000000203d590
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000001  R15: 0000000000005d00
    ORIG_RAX: 000000000000003d  CS: 0033  SS: 002b

它发生在不同厂商的机器上，我尝试从redhat更新到最新的内核。有同样经历的人能帮上忙吗？