20190712龙湖地产vm内存溢出问题

现象：
vm挂起，数据库夯住
–alertlog:
on Jul 08 22:08:21 2019
System state dump requested by (instance=1, osid=28546 (PMON)), summary=[abnormal instance termination].
Mon Jul 08 22:08:27 2019
PMON (ospid: 28546): terminating the instance due to error 471
System State dumped to trace file /data/app/oracle/diag/rdbms/lfpcm/LFPCM/trace/LFPCM_diag_28557_20190708220825.trc
Dumping diagnostic data in directory=[cdmp_20190708220820], requested by (instance=1, osid=28546 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 28546
–messages:
ul 8 22:03:26 c4tschengbendb01 kernel: systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jul 8 22:03:27 c4tschengbendb01 kernel: systemd cpuset=/ mems_allowed=0
Jul 8 22:03:27 c4tschengbendb01 kernel: CPU: 3 PID: 1 Comm: systemd Not tainted 3.10.0-957.12.2.el7.x86_64 #1
Jul 8 22:03:27 c4tschengbendb01 kernel: Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 3288b3c 04/01/2014
Jul 8 22:03:27 c4tschengbendb01 kernel: Call Trace:
Jul 8 22:03:27 c4tschengbendb01 kernel: [] dump_stack+0x19/0x1b
Jul 8 22:03:27 c4tschengbendb01 kernel: [] dump_header+0x90/0x229
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ? ktime_get_ts64+0x52/0xf0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ? delayacct_end+0x8f/0xb0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] oom_kill_process+0x254/0x3d0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ? oom_unkillable_task+0xcd/0x120
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ? find_lock_task_mm+0x56/0xc0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] out_of_memory+0x4b6/0x4f0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] __alloc_pages_slowpath+0x5d6/0x724
Jul 8 22:03:27 c4tschengbendb01 kernel: [] __alloc_pages_nodemask+0x404/0x420
Jul 8 22:03:27 c4tschengbendb01 kernel: [] alloc_pages_current+0x98/0x110
Jul 8 22:03:27 c4tschengbendb01 kernel: [] __page_cache_alloc+0x97/0xb0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] filemap_fault+0x298/0x490
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ext4_filemap_fault+0x36/0x50 [ext4]
Jul 8 22:03:27 c4tschengbendb01 kernel: [] __do_fault.isra.59+0x8a/0x100
Jul 8 22:03:27 c4tschengbendb01 kernel: [] do_read_fault.isra.61+0x4c/0x1b0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] handle_pte_fault+0x2f4/0xd10
Jul 8 22:03:27 c4tschengbendb01 kernel: [] ? ep_send_events_proc+0x160/0x1b0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] handle_mm_fault+0x39d/0x9b0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] __do_page_fault+0x203/0x4f0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] trace_do_page_fault+0x56/0x150
Jul 8 22:03:27 c4tschengbendb01 kernel: [] do_async_page_fault+0x22/0xf0
Jul 8 22:03:27 c4tschengbendb01 kernel: [] async_page_fault+0x28/0x30
Jul 8 22:03:27 c4tschengbendb01 kernel: Mem-Info:
Jul 8 22:03:27 c4tschengbendb01 kernel: active_anon:801812 inactive_anon:678796 isolated_anon:0#012 active_file:11091 inactive_file:12150 isolated_file:74#012 unevictable:0 dirty:11 writeback:0 unstable:0#012 slab_reclaimable:44693 slab_unreclaimable:5563#012 mapped:1123518 shmem:1326631 pagetables:10409 bounce:0#012 free:29748 free_pcp:0 free_cma:0
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 DMA free:15908kB min:48kB low:60kB high:72kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 8 22:03:27 c4tschengbendb01 kernel: lowmem_reserve[]: 0 2830 15866 15866
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 DMA32 free:61180kB min:9124kB low:11404kB high:13684kB active_anon:200532kB inactive_anon:163596kB active_file:4268kB inactive_file:4792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2898784kB mlocked:0kB dirty:4kB writeback:0kB mapped:252012kB shmem:294428kB slab_reclaimable:17264kB slab_unreclaimable:1868kB kernel_stack:560kB pagetables:3208kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:14170 all_unreclaimable? yes
Jul 8 22:03:27 c4tschengbendb01 kernel: lowmem_reserve[]: 0 0 13035 13035
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 Normal free:41904kB min:42024kB low:52528kB high:63036kB active_anon:3006716kB inactive_anon:2551588kB active_file:40096kB inactive_file:43808kB unevictable:0kB isolated(anon):0kB isolated(file):296kB present:13631488kB managed:13351716kB mlocked:0kB dirty:40kB writeback:0kB mapped:4242060kB shmem:5012096kB slab_reclaimable:161508kB slab_unreclaimable:20384kB kernel_stack:6816kB pagetables:38428kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:135225 all_unreclaimable? yes
Jul 8 22:03:27 c4tschengbendb01 kernel: lowmem_reserve[]: 0 0 0 0
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 DMA: 14kB (U) 08kB 016kB 132kB (U) 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15908kB
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 DMA32: 1324kB (UEM) 818kB (UEM) 33116kB (UEM) 18532kB (UEM) 9764kB (UE) 71128kB (UEM) 76256kB (UEM) 14512kB (UEM) 71024kB (UEM) 02048kB 04096kB = 61480kB
Jul 8 22:03:27 c4tschengbendb01 kernel: Node 0 Normal: 10244kB (UEM) 7838kB (UEM) 154516kB (UEM) 22032kB (UEM) 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 42120kB
Jul 8 22:03:28 c4tschengbendb01 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 8 22:03:28 c4tschengbendb01 kernel: Node 0 hugepages_total=4765 hugepages_free=4765 hugepages_surp=0 hugepages_size=2048kB
Jul 8 22:03:28 c4tschengbendb01 kernel: 1350124 total pagecache pages
Jul 8 22:03:28 c4tschengbendb01 kernel: 0 pages in swap cache
Jul 8 22:03:28 c4tschengbendb01 kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 8 22:03:28 c4tschengbendb01 kernel: Free swap = 0kB
Jul 8 22:03:28 c4tschengbendb01 kernel: Total swap = 0kB
Jul 8 22:03:28 c4tschengbendb01 kernel: 4194174 pages RAM
Jul 8 22:03:28 c4tschengbendb01 kernel: 0 pages HighMem/MovableOnly
Jul 8 22:03:28 c4tschengbendb01 kernel: 127572 pages reserved