一次由隱藏大頁配置引發的數據庫OOM故障分析
一、事故發生
在週日清晨，收到緊急短信告警，數據庫實例發生異常重啓。首先登錄數據庫服務器，查看日誌記錄
2025-12-21T06:54:57.259156+08:00 77 [Note] [MY-010914] [Server] Aborted connection 77 to db: 'unconnected' user: 'root' host: '172.17.139.203' (Got an error reading communication packets).
2025-12-21T06:55:33.224314Z mysqld_safe Number of processes running now: 0
2025-12-21T06:55:33.248143Z mysqld_safe mysqld restarted
2025-12-21T06:55:34.053462+08:00 0 [Warning] [MY-011069] [Server] The syntax '--replica-parallel-type' is deprecated and will be removed in a future release.
2025-12-21T06:55:34.053569+08:00 0 [Warning] [MY-011068] [Server] The syntax '--ssl=off' is deprecated and will be removed in a future release. Please use --tls-version='' instead.
通過該日誌內容初步判斷重啓原因是發生了OOM異常，直接觀察系統日誌/var/log/messages，確認存在oom異常信息。
[root@gdb-adm ~]# grep -inr /var/log/messages
5:Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
11:Dec 21 06:55:33 gdb kernel: [419827.630530] oom_kill_process+0x24f/0x270
12:Dec 21 06:55:33 gdb kernel: [419827.630532] ? oom_badness+0x25/0x140
68:Dec 21 06:55:33 gdb kernel: [419827.630752] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
148:Dec 21 06:55:33 gdb kernel: [419827.631062] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-2036.slice/session-6188.scope,task=mysqld,pid=2567710,uid=2032
二、問題分析
1、內存設置檢查
服務器物理內存376G，而innodb_buffer_pool_size設置為200G，佔比為53%，符合預期。
free -h
total used free shared buff/cache available
Mem: 376Gi 267Gi 26Gi 5.0Mi 82Gi 53Gi
2、jemolloc判斷
作為GreatSQL數據庫或者開源MySQL數據庫，出現OOM的情況，很大可能是由於使用默認的glibc內存分配管理，內存使用後釋放不完全引起內存泄漏導致，通過命令lsof -p PID| grep jem 觀察內存分配管理方式
[root@gdb ~]# lsof -p 25424 | grep jem
mysqld 25424 mysql mem REG 8,2 2136088 2355262 /data/svr/greatsql/lib/mysql/libjemalloc.so.1
從返回可以看出配置正常，基本上可以排除此原因。
3、OOM日誌詳細分析
1）完整OOM日誌
Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 06:55:33 gdb kernel: [419827.630499] CPU: 14 PID: 9458 Comm: crontab-1 Kdump: loaded Not tainted 4.19.90-2107.6.0.0227.28.oe1.bclinux.x86_64 #1
Dec 21 06:55:33 gdb kernel: [419827.630500] Hardware name: FiberHome FitServer/FiberHome Boards, BIOS 3.4.V7 02/01/2023
Dec 21 06:55:33 gdb kernel: [419827.630507] Call Trace:
Dec 21 06:55:33 gdb kernel: [419827.630519] dump_stack+0x66/0x8b
Dec 21 06:55:33 gdb kernel: [419827.630527] dump_header+0x4a/0x1fc
Dec 21 06:55:33 gdb kernel: [419827.630530] oom_kill_process+0x24f/0x270
Dec 21 06:55:33 gdb kernel: [419827.630532] ? oom_badness+0x25/0x140
Dec 21 06:55:33 gdb kernel: [419827.630533] out_of_memory+0x11f/0x540
Dec 21 06:55:33 gdb kernel: [419827.630536] __alloc_pages_slowpath+0x9f5/0xde0
Dec 21 06:55:33 gdb kernel: [419827.630543] __alloc_pages_nodemask+0x2a8/0x2d0
Dec 21 06:55:33 gdb kernel: [419827.630549] filemap_fault+0x35e/0x8a0
Dec 21 06:55:33 gdb kernel: [419827.630555] ? alloc_set_pte+0x244/0x450
Dec 21 06:55:33 gdb kernel: [419827.630558] ? filemap_map_pages+0x28f/0x480
Dec 21 06:55:33 gdb kernel: [419827.630584] ext4_filemap_fault+0x2c/0x40 [ext4]
Dec 21 06:55:33 gdb kernel: [419827.630588] __do_fault+0x33/0x110
Dec 21 06:55:33 gdb kernel: [419827.630592] do_fault+0x12e/0x490
Dec 21 06:55:33 gdb kernel: [419827.630595] ? __handle_mm_fault+0x2a/0x690
Dec 21 06:55:33 gdb kernel: [419827.630597] __handle_mm_fault+0x613/0x690
Dec 21 06:55:33 gdb kernel: [419827.630601] handle_mm_fault+0xc4/0x200
Dec 21 06:55:33 gdb kernel: [419827.630604] __do_page_fault+0x2ba/0x4d0
Dec 21 06:55:33 gdb kernel: [419827.630609] ? __audit_syscall_exit+0x238/0x2c0
Dec 21 06:55:33 gdb kernel: [419827.630611] do_page_fault+0x31/0x130
Dec 21 06:55:33 gdb kernel: [419827.630616] ? page_fault+0x8/0x30
Dec 21 06:55:33 gdb kernel: [419827.630620] page_fault+0x1e/0x30
Dec 21 06:55:33 gdb kernel: [419827.630623] Mem-Info:
Dec 21 06:55:33 gdb kernel: [419827.630635] active_anon:50985791 inactive_anon:354 isolated_anon:0#012 active_file:677 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:105 writeback:123 unstable:0#012 slab_reclaimable:20583 slab_unreclaimable:49628#012 m
apped:319 shmem:1323 pagetables:106803 bounce:0#012 free:5313776 free_pcp:5715 free_cma:0
Dec 21 06:55:33 gdb kernel: [419827.630638] Node 0 active_anon:100766572kB inactive_anon:556kB active_file:1384kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:76kB dirty:32kB writeback:0kB shmem:2276kB shmem_thp: 0kB shmem_pmdm
apped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 06:55:33 gdb kernel: [419827.630645] Node 1 active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1200kB dirty:388kB writeback:492kB shmem:3016kB shmem_thp: 0kB shme
m_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 06:55:33 gdb kernel: [419827.630650] Node 0 DMA free:15892kB min:824kB low:1028kB high:1232kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocked:0kB kernel_stack:0k
B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630654] lowmem_reserve[]: 0 1347 191666 191666 191666
Dec 21 06:55:33 gdb kernel: [419827.630661] Node 0 DMA32 free:833940kB min:72972kB low:91212kB high:109452kB active_anon:559420kB inactive_anon:8kB active_file:68kB inactive_file:0kB unevictable:0kB writepending:32kB present:1733384kB managed:1405672kB mlocked:
0kB kernel_stack:52kB pagetables:1084kB bounce:0kB free_pcp:400kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630666] lowmem_reserve[]: 0 0 190319 190319 190319
Dec 21 06:55:33 gdb kernel: [419827.630672] Node 0 Normal free:10117540kB min:10117912kB low:12647388kB high:15176864kB active_anon:100207152kB inactive_anon:548kB active_file:808kB inactive_file:0kB unevictable:0kB writepending:0kB present:198180864kB managed:
194894048kB mlocked:0kB kernel_stack:13504kB pagetables:215840kB bounce:0kB free_pcp:536kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630679] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630683] Node 1 Normal free:10287732kB min:10288284kB low:12860352kB high:15432420kB active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB writepending:880kB present:201326592kB mana
ged:198175752kB mlocked:0kB kernel_stack:11836kB pagetables:210288kB bounce:0kB free_pcp:21924kB local_pcp:332kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630686] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630688] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Dec 21 06:55:33 gdb kernel: [419827.630694] Node 0 DMA32: 240*4kB (UME) 178*8kB (UME) 140*16kB (UME) 66*32kB (UME) 70*64kB (UME) 53*128kB (UME) 38*256kB (UME) 18*512kB (UE) 3*1024kB (U) 2*2048kB (UE) 193*4096kB (M) = 834640kB
Dec 21 06:55:33 gdb kernel: [419827.630702] Node 0 Normal: 3557*4kB (UE) 1963*8kB (UME) 651*16kB (UME) 1139*32kB (UME) 855*64kB (UME) 572*128kB (UME) 308*256kB (UE) 129*512kB (UME) 50*1024kB (UME) 27*2048kB (UME) 2359*4096kB (UME) = 10118588kB
Dec 21 06:55:33 gdb kernel: [419827.630712] Node 1 Normal: 3636*4kB (UME) 1848*8kB (UME) 2744*16kB (UME) 2139*32kB (UME) 1580*64kB (UME) 1073*128kB (UME) 613*256kB (UME) 280*512kB (UE) 130*1024kB (UE) 81*2048kB (UE) 2273*4096kB (UME) = 10289648kB
Dec 21 06:55:33 gdb kernel: [419827.630731] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630737] Node 0 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
Dec 21 06:55:33 gdb kernel: [419827.630738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630741] Node 1 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
Dec 21 06:55:33 gdb kernel: [419827.630742] 3360 total pagecache pages
Dec 21 06:55:33 gdb kernel: [419827.630744] 0 pages in swap cache
Dec 21 06:55:33 gdb kernel: [419827.630746] Swap cache stats: add 0, delete 0, find 0/0
Dec 21 06:55:33 gdb kernel: [419827.630746] Free swap = 0kB
Dec 21 06:55:33 gdb kernel: [419827.630747] Total swap = 0kB
Dec 21 06:55:33 gdb kernel: [419827.630748] 100314204 pages RAM
Dec 21 06:55:33 gdb kernel: [419827.630749] 0 pages HighMem/MovableOnly
Dec 21 06:55:33 gdb kernel: [419827.630749] 1691363 pages reserved
Dec 21 06:55:33 gdb kernel: [419827.630750] 0 pages hwpoisoned
Dec 21 06:55:33 gdb kernel: [419827.630750] Tasks state (memory values in pages):
Dec 21 06:55:33 gdb kernel: [419827.630752] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Dec 21 06:55:33 gdb kernel: [419827.630790] [ 926] 0 926 72470 811 507904 0 -250 systemd-journal
Dec 21 06:55:33 gdb kernel: [419827.630794] [ 960] 0 960 8269 1075 77824 0 -1000 systemd-udevd
Dec 21 06:55:33 gdb kernel: [419827.630798] [ 1623] 0 1623 729 28 32768 0 0 mdadm
Dec 21 06:55:33 gdb kernel: [419827.630800] [ 1672] 0 1672 23007 217 49152 0 -1000 auditd
Dec 21 06:55:33 gdb kernel: [419827.630803] [ 1674] 0 1674 1568 90 36864 0 0 sedispatch
Dec 21 06:55:33 gdb kernel: [419827.630806] [ 1712] 0 1712 78709 787 98304 0 0 ModemManager
Dec 21 06:55:33 gdb kernel: [419827.630808] [ 1714] 0 1714 571 16 32768 0 0 acpid
Dec 21 06:55:33 gdb kernel: [419827.630811] [ 1719] 81 1719 2891 845 49152 0 -900 dbus-daemon
Dec 21 06:55:33 gdb kernel: [419827.630813] [ 1727] 992 1727 599 38 32768 0 0 lsmd
Dec 21 06:55:33 gdb kernel: [419827.630815] [ 1730] 0 1730 619 33 32768 0 0 mcelog
Dec 21 06:55:33 gdb kernel: [419827.630817] [ 1735] 999 1735 743772 1030 229376 0 0 polkitd
Dec 21 06:55:33 gdb kernel: [419827.630820] [ 1736] 0 1736 77985 204 90112 0 0 rngd
Dec 21 06:55:33 gdb kernel: [419827.630827] [ 1739] 0 1739 2711 421 49152 0 0 smartd
Dec 21 06:55:33 gdb kernel: [419827.630829] [ 1741] 0 1741 20070 151 40960 0 -500 irqbalance
Dec 21 06:55:33 gdb kernel: [419827.630831] [ 1743] 0 1743 4492 227 61440 0 0 systemd-machine
Dec 21 06:55:33 gdb kernel: [419827.630837] [ 1753] 0 1753 114058 472 110592 0 0 abrtd
Dec 21 06:55:33 gdb kernel: [419827.630842] [ 1794] 0 1794 4780 468 65536 0 0 systemd-logind
Dec 21 06:55:33 gdb kernel: [419827.630844] [ 1830] 0 1830 263593 479 929792 0 0 abrt-dump-journ
Dec 21 06:55:33 gdb kernel: [419827.630846] [ 1831] 0 1831 261511 460 925696 0 0 abrt-dump-journ
Dec 21 06:55:33 gdb kernel: [419827.630850] [ 2802] 0 2802 199635 606 299008 0 0 esfdaemon
Dec 21 06:55:33 gdb kernel: [419827.630852] [ 2803] 0 2803 72799 12101 200704 0 0 bare-agent
Dec 21 06:55:33 gdb kernel: [419827.630855] [ 2805] 0 2805 59117 340 86016 0 0 cupsd
Dec 21 06:55:33 gdb kernel: [419827.630856] [ 2810] 0 2810 251667 734 1376256 0 0 rsyslogd
Dec 21 06:55:33 gdb kernel: [419827.630863] [ 2814] 0 2814 3350 227 53248 0 -1000 sshd
Dec 21 06:55:33 gdb kernel: [419827.630865] [ 2815] 0 2815 117707 3324 143360 0 0 tuned
Dec 21 06:55:33 gdb kernel: [419827.630869] [ 2828] 0 2828 65710 188 73728 0 0 gssproxy
Dec 21 06:55:33 gdb kernel: [419827.630872] [ 2848] 0 2848 53496 92 45056 0 0 init.ohasd
Dec 21 06:55:33 gdb kernel: [419827.630874] [ 2890] 0 2890 906 48 32768 0 0 atd
Dec 21 06:55:33 gdb kernel: [419827.630875] [ 2896] 0 2896 53748 118 49152 0 0 crond
Dec 21 06:55:33 gdb kernel: [419827.630878] [ 3692] 0 3692 3539 148 49152 0 0 xinetd
Dec 21 06:55:33 gdb kernel: [419827.630880] [ 3978] 0 3978 10985 242 61440 0 0 master
Dec 21 06:55:33 gdb kernel: [419827.630884] [ 4004] 89 4004 11331 527 69632 0 0 qmgr
Dec 21 06:55:33 gdb kernel: [419827.630888] [ 4093] 0 4093 43766 216 221184 0 0 sddog
Dec 21 06:55:33 gdb kernel: [419827.630890] [ 4112] 0 4112 285705 537 577536 0 0 sdmonitor
Dec 21 06:55:33 gdb kernel: [419827.630891] [ 4233] 0 4233 134053 596 466944 0 0 sdcc
Dec 21 06:55:33 gdb kernel: [419827.630895] [ 4259] 0 4259 168947 8371 667648 0 0 sdec
Dec 21 06:55:33 gdb kernel: [419827.630897] [ 4284] 0 4284 286675 1588 778240 0 0 sdexam
Dec 21 06:55:33 gdb kernel: [419827.630899] [ 4310] 0 4310 492216 50216 1331200 0 0 sdsvrd
Dec 21 06:55:33 gdb kernel: [419827.630906] [ 4330] 0 4330 29248 278 278528 0 0 udcenter
Dec 21 06:55:33 gdb kernel: [419827.630908] [ 8353] 0 8353 2184 321 45056 0 0 dhclient
Dec 21 06:55:33 gdb kernel: [419827.630910] [ 9243] 1086 9243 5274 639 73728 0 0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630915] [ 9245] 1086 9245 6383 1015 73728 0 0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630918] [ 9348] 1086 9348 470112 50291 761856 0 0 java
Dec 21 06:55:33 gdb kernel: [419827.630920] [ 9426] 0 9426 2184 323 45056 0 0 dhclient
Dec 21 06:55:33 gdb kernel: [419827.630922] [ 9852] 0 9852 53214 26 36864 0 0 agetty
Dec 21 06:55:33 gdb kernel: [419827.630926] [ 11463] 1002 11463 5276 639 73728 0 0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630936] [ 11465] 1002 11465 6383 1016 73728 0 0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630942] [ 11611] 1002 11611 14284908 1404 602112 0 0 agent60
Dec 21 06:55:33 gdb kernel: [419827.630945] [ 137615] 0 137615 136163 3215 147456 0 0 lvmdbusd
Dec 21 06:55:33 gdb kernel: [419827.630950] [ 796407] 2036 796407 5301 649 73728 0 0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630952] [ 796409] 2036 796409 43812 1109 94208 0 0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630954] [ 817343] 2032 817343 53508 130 53248 0 0 mysqld_safe
Dec 21 06:55:33 gdb kernel: [419827.630956] [2270020] 2032 2270020 2778466 1788 1466368 0 0 dbinit
Dec 21 06:55:33 gdb kernel: [419827.630958] [2567710] 2032 2567710 77307141 50817311 424357888 0 0 mysqld
Dec 21 06:55:33 gdb kernel: [419827.630960] [3453494] 998 3453494 1173 50 36864 0 0 chronyd
Dec 21 06:55:33 gdb kernel: [419827.630963] [3621338] 89 3621338 11065 249 65536 0 0 pickup
Dec 21 06:55:33 gdb kernel: [419827.630981] [3662845] 0 3662845 5297 648 73728 0 0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630983] [3662881] 0 3662881 44244 1356 98304 0 0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630985] [3662906] 89 3662906 11068 242 65536 0 0 trivial-rewrite
Dec 21 06:55:33 gdb kernel: [419827.630987] [3663080] 0 3663080 10991 235 65536 0 0 local
Dec 21 06:55:33 gdb kernel: [419827.630988] [3663097] 89 3663097 11131 254 65536 0 0 smtp
Dec 21 06:55:33 gdb kernel: [419827.630990] [3663098] 0 3663098 10991 235 65536 0 0 local
Dec 21 06:55:33 gdb kernel: [419827.630992] [3663108] 89 3663108 11073 242 65536 0 0 bounce
Dec 21 06:55:33 gdb kernel: [419827.630994] [3663141] 0 3663141 10991 235 65536 0 0 local
Dec 21 06:55:33 gdb kernel: [419827.630997] [3663177] 89 3663177 11066 242 69632 0 0 flush
Dec 21 06:55:33 gdb kernel: [419827.631003] [3663193] 89 3663193 11066 242 69632 0 0 flush
Dec 21 06:55:33 gdb kernel: [419827.631005] [3663201] 89 3663201 11066 242 69632 0 0 flush
Dec 21 06:55:33 gdb kernel: [419827.631007] [3663207] 0 3663207 53463 54 45056 0 0 sh
Dec 21 06:55:33 gdb kernel: [419827.631011] [3663208] 0 3663208 884643 7048 589824 0 0 promtail
Dec 21 06:55:33 gdb kernel: [419827.631019] [3663317] 89 3663317 11131 254 65536 0 0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631023] [3663318] 89 3663318 11131 254 65536 0 0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631025] [3663319] 89 3663319 11131 254 65536 0 0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631026] [3663320] 89 3663320 11131 254 65536 0 0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631028] [3663321] 89 3663321 11064 242 65536 0 0 error
Dec 21 06:55:33 gdb kernel: [419827.631030] [3663322] 89 3663322 11064 242 65536 0 0 error
Dec 21 06:55:33 gdb kernel: [419827.631032] [3663388] 0 3663388 53093 15 40960 0 0 sleep
Dec 21 06:55:33 gdb kernel: [419827.631048] [3663946] 0 3663946 4458 86 61440 0 0 systemd-cgroups
Dec 21 06:55:33 gdb kernel: [419827.631060] [3663947] 0 3663947 4071 84 57344 0 0 systemd-cgroups
Dec 21 06:55:33 gdb kernel: [419827.631062] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-2036.slice/session-6188.scope,task=mysqld,pid=2567710,uid=2032
Dec 21 06:55:33 gdb kernel: [419827.631071] Out of memory: Kill process 2567710 (mysqld) score 516 or sacrifice child
Dec 21 06:55:33 gdb kernel: [419827.632542] Killed process 2567710 (mysqld) total-vm:309228564kB, anon-rss:203269244kB, file-rss:0kB, shmem-rss:0kB
2）發生現象
Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 06:55:33 gdb kernel: [419827.632542] Killed process 2567710 (mysqld) total-vm:309228564kB, anon-rss:203269244kB, file-rss:0kB, shmem-rss:0kB
上述關鍵信息為進程crontab-1申請新的內存引起oom-killer，而被kill進程為mysqld佔用內存大小203269244kB
3) NUMA佔用分析
Dec 21 06:55:33 gdb kernel: [419827.630672] Node 0 Normal free:10117540kB min:10117912kB low:12647388kB high:15176864kB active_anon:100207152kB inactive_anon:548kB active_file:808kB inactive_file:0kB unevictable:0kB writepending:0kB present:198180864kB managed:
194894048kB mlocked:0kB kernel_stack:13504kB pagetables:215840kB bounce:0kB free_pcp:536kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630679] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630683] Node 1 Normal free:10287732kB min:10288284kB low:12860352kB high:15432420kB active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB writepending:880kB present:201326592kB mana
ged:198175752kB mlocked:0kB kernel_stack:11836kB pagetables:210288kB bounce:0kB free_pcp:21924kB local_pcp:332kB free_cma:0kB
從上述日誌，可以看出兩個numa node的剩餘free內存均低於了min的要求內存。
4) 內存佔用統計
根據OOM記錄的日誌信息，內存大概有如下分配(注意，系統日誌中rss列的單位為頁，默認4k大小)
| 進程 | 佔用內存 |
|---|---|
| mysqld | 193G |
| 其他進程 | 641M |
| NUMA剩餘 | 19.5G |
上述內存遠低於操作系統內存376G，缺失近163G
5) 大頁分析
繼續查看系統日誌
Dec 21 06:55:33 gdb kernel: [419827.630731] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630737] Node 0 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
Dec 21 06:55:33 gdb kernel: [419827.630738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630741] Node 1 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
解析為
| 頁類型 | 總頁數量 | 空閒頁 | |
|---|---|---|---|
| numanode0 | 2M | 40960 | 40960 |
| numanode0 | 1G | ||
| numanode1 | 2M | 40960 | 40960 |
| numanode1 | 1G |
可見大頁佔用了2M x 40960 x 2=160G內存，並且沒有被使用，剛好和內存統計相近
4、大頁配置查看
1) 檢查透明大頁配置
cat /sys/kernel/mm/transparent_hugepage/enabled，確認是關閉狀態
[root@gdb ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
2) 檢查傳統大頁配置
sysctl -p | grep vm ，可見並沒有相關配置
[root@gdb ~]# sysctl -p | grep vm
vm.zone_reclaim_mode=0
vm.swappiness=1
vm.min_free_kbytes=20480000
3) 大頁特性對比
| 特性維度 | 傳統大頁 | 透明大頁 |
|---|---|---|
| 檢查方式 | /etc/sysctl.conf 中的 vm.nr_hugepages |
/sys/kernel/mm/transparent_hugepage/enabled |
| 管理機制 | 靜態預分配。在系統啓動或配置後，內核立即從物理內存中劃出指定數量的大頁。這部分內存被“鎖定”，專用於大頁，不能被挪作他用（如進程的普通小頁）。 | 動態分配。內核在運行時根據內存訪問模式（如連續的512個4K頁被頻繁訪問），自動將小頁合併成一個大頁，或者在不再需要時拆分回小頁。這是一個“按需”的過程。 |
| 配置方式 | 1. 臨時：sysctl -w vm.nr_hugepages=N 2. 永久：在 /etc/sysctl.conf 中添加 vm.nr_hugepages=N，重啓或執行 sysctl -p 生效。 |
1. 臨時：echo > /sys/kernel/mm/transparent_hugepage/enabled 2. 永久：通過內核啓動參數 vi /etc/default/grub 在 GRUB_CMDLINE_LINUX變量中添加transparent_hugepage=always，重新生成GRUB配置grub2-mkconfig -o /boot/grub2/grub.cfg |
| 內存使用 | 專用且獨佔。分配後即使不使用，也會一直佔用物理內存，可能導致內存浪費。 | 共享池。使用普通的內存頁池，只在需要時才轉換，內存利用率更高。 |
| 性能特點 | 性能穩定可預測。應用程序（如Oracle DB, Redis）通過mmap()或shmget()顯式請求大頁時，能100%保證使用大頁，無缺頁中斷或合併操作開銷，性能最優、最穩定。 |
性能有波動風險。雖然大多數情況下能提升性能（減少TLB Miss），但在內存壓力大或碎片化時，內核的合併/拆分操作（khugepaged進程）會帶來不可預測的延遲尖峯，對延遲敏感型應用不利。 |
根據故障現象及大頁特點，猜測應該是由於配置了傳統大頁，鎖定了160G內存無法被其他進程使用，但是配置文件中並沒有該配置，現象很奇怪
4) 深度搜索
使用命令grep -R "nr_hugepages" /etc進行大範圍深度搜索，發現了問題所在
[root@gdb ~]# grep -R "nr_hugepages" /etc
/etc/sysctl.conf.bak-2025-07-13:vm.nr_hugepages=81920
可以看到配置文件在7月13日進行了備份調整，備份前確實是有傳統大頁配置，並且配置值和目前系統日誌中記錄值相同。
5) 配置變更測試
通過測試發現，即使配置文件中去傳統大頁設置，但是依然是存在大頁設置的
[root@qdb -]# cat /etc/sysctl.conf | grep h
kernel.shmall = 41943040
kernel.shmmax = 171798691840
kernel.shmmni=4096
#vm.hugetlb_shm_group=54321
#vm.nr_hugepages = 40960
[root@qdb -]# sysctl -p | grep h
kernel.shmall = 41943040
kernel.shmmax = 171798691840
kernel.shmmi=4096
[root@qdb -]# cat /proc/sys/vm/nr_hugepages
40960
調整配置後如果不重啓操作系統，需要手動釋放該部分內存
[root@gdb ~]# echo 0 > /proc/sys/vm/nr_hugepages
[root@gdb ~]# cat /proc/sys/vm/nr_hugepages
0
三、原因總結改進
1) 根本原因
大量 HugePages 被預留但數據庫未實際使用，導致普通內存不足，引發 OOM
2) 不正常的默認大頁配置
在操作系統默認情況下，未配置nr_hugepages，因此最初分析時未考慮傳統大頁方向。後經數據對比，發現傳統大頁存在內存佔用異常現象。經後續核實，由於該服務器為利舊使用，殘留了Oracle相關配置，導致該隱藏問題未被及時發現，又是一個國產化過程的小坑。
3) 後續改進
在基於現有服務器初始化步驟中，增加傳統大頁的檢查設置步驟
sed -i '/huge/d' /etc/sysctl.conf
sysctl -p | grep huge
echo 0 > /proc/sys/vm/nr_hugepages
