overcommit_memory目录overcommit_memoryOvercommit AccountingCommitLimit and Committed_ASovercommit_kbytes or overcommit_ratio

103 阅读 0 评论 68 点赞

我是靠谱客的博主朴素小天鹅，最近开发中收集的这篇文章主要介绍overcommit_memory目录overcommit_memoryOvercommit AccountingCommitLimit and Committed_ASovercommit_kbytes or overcommit_ratio，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

overcommit_memory

This value contains a flag that enables memory overcommitment.

When this flag is 0, the kernel attempts to estimate the amount of free memory left when userspace requests more memory.

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

When this flag is 2, the kernel uses a “never overcommit” policy that attempts to prevent any overcommit of memory. Note that user_reserve_kbytes affects this policy.

This feature can be very useful because there are a lot of programs that malloc() huge amounts of memory “just-in-case” and don’t use much of it.

The default value is 0.

See Overcommit Accounting and mm/util.c::__vm_enough_memory() for more information.

更多详情，需要去转看Overcommit Accounting。

补充：默认情况下，当malloc一次申请的虚拟内存大于128K则会转成mmap，此时不会在堆内存中通过brk指针申请虚拟内存，而是通过mmap在内存映射区划分内存。

1.先看malloc API文档中的MMAP_THRESHOLD：

Normally, malloc() allocates memory from the heap, and adjusts
the size of the heap as required, using sbrk(2).
When allocating
blocks of memory larger than MMAP_THRESHOLD bytes, the glibc
malloc() implementation allocates the memory as a private
anonymous mapping using mmap(2).
MMAP_THRESHOLD is 128 kB by
default, but is adjustable using mallopt(3).  Prior to Linux 4.7
allocations performed using mmap(2) were unaffected by the
RLIMIT_DATA resource limit; since Linux 4.7, this limit is also
enforced for allocations performed using mmap(2).

2.转到mallopt API文档中查看MALLOC_MMAP_THRESHOLD_：

MALLOC_MMAP_THRESHOLD_
Controls the same parameter as mallopt() M_MMAP_THRESHOLD.

3.再转看mallopt API文档中的M_MMAP_THRESHOLD：

M_MMAP_THRESHOLD
For allocations greater than or equal to the limit
specified (in bytes) by M_MMAP_THRESHOLD that can't be
satisfied from the free list, the memory-allocation
functions employ mmap(2) instead of increasing the program
break using sbrk(2).
Allocating memory using mmap(2) has the significant
advantage that the allocated memory blocks can always be
independently released back to the system.
(By contrast,
the heap can be trimmed only if memory is freed at the top
end.)
On the other hand, there are some disadvantages to
the use of mmap(2): deallocated space is not placed on the
free list for reuse by later allocations; memory may be
wasted because mmap(2) allocations must be page-aligned;
and the kernel must perform the expensive task of zeroing
out memory allocated via mmap(2).
Balancing these factors
leads to a default setting of 128*1024 for the
M_MMAP_THRESHOLD parameter.
The lower limit for this parameter is 0.
The upper limit
is DEFAULT_MMAP_THRESHOLD_MAX: 512*1024 on 32-bit systems
or 4*1024*1024*sizeof(long) on 64-bit systems.
Note: Nowadays, glibc uses a dynamic mmap threshold by
default.
The initial value of the threshold is 128*1024,
but when blocks larger than the current threshold and less
than or equal to DEFAULT_MMAP_THRESHOLD_MAX are freed, the
threshold is adjusted upward to the size of the freed
block.
When dynamic mmap thresholding is in effect, the
threshold for trimming the heap is also dynamically
adjusted to be twice the dynamic mmap threshold.
Dynamic
adjustment of the mmap threshold is disabled if any of the
M_TRIM_THRESHOLD, M_TOP_PAD, M_MMAP_THRESHOLD, or
M_MMAP_MAX parameters is set.

Overcommit Accounting

The Linux kernel supports the following overcommit handling modes

Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.

Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages.

Don’t overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

The overcommit policy is set via the sysctl vm.overcommit_memory.

The overcommit amount can be set via vm.overcommit_ratio (percentage) or vm.overcommit_kbytes (absolute value).

The current overcommit limit and amount committed are viewable in /proc/meminfo as CommitLimit and Committed_AS respectively.

最后一句提到了/proc/meminfo中的CommitLimit and Committed_AS，转去看看。

CommitLimit and Committed_AS

地址：proc(5) - Linux manual page (man7.org)

CommitLimit %lu (since Linux 2.6.10)
This is the total amount of memory currently
available to be allocated on the system, expressed
in kilobytes.
This limit is adhered to only if
strict overcommit accounting is enabled (mode 2 in
/proc/sys/vm/overcommit_memory).  The limit is
calculated according to the formula described under
/proc/sys/vm/overcommit_memory.
For further
details, see the kernel source file
Documentation/vm/overcommit-accounting.rst.

Committed_AS %lu
The amount of memory presently allocated on the
system.
The committed memory is a sum of all of
the memory which has been allocated by processes,
even if it has not been "used" by them as of yet.
A process which allocates 1 GB of memory (using
malloc(3) or similar), but touches only 300 MB of
that memory will show up as using only 300 MB of
memory even if it has the address space allocated
for the entire 1 GB.
This 1 GB is memory which has been "committed" to
by the VM and can be used at any time by the
allocating application.
With strict overcommit
enabled on the system (mode 2 in
/proc/sys/vm/overcommit_memory), allocations which
would exceed the CommitLimit will not be permitted.
This is useful if one needs to guarantee that
processes will not fail due to lack of memory once
that memory has been successfully allocated.

mode 2才生效，即overcommit_memory=2时才生效，转回去详细看看mode 2。

mode 2即不允许over commit，在进程的虚拟地址空间允许commit的范围不能超过swap+物理内存的配置量（默认50%）。

swap这个好理解，free、/proc/swaps、swapon -s都能查看。

物理内存的的配置量，这个在哪里？overcommit_kbytes或者overcommit_ratio，都在/proc/sys/vm路径下。

overcommit_kbytes or overcommit_ratio

路径：/proc/sys/vm

overcommit_kbytes

地址：Documentation for /proc/sys/vm/ — The Linux Kernel documentation

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this amount of physical RAM. See below.

Note: overcommit_kbytes is the counterpart of overcommit_ratio. Only one of them may be specified at a time. Setting one disables the other (which then appears as 0 when read).

overcommit_ratio

地址：Documentation for /proc/sys/vm/ — The Linux Kernel documentation

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this percentage of physical RAM. See above.

当overcommit_memory=2（即不允许over commit）时，overcommit_kbytes或overcommit_ratio两个选项用于配置允许commit的虚拟地址大小：

即允许commit的虚拟地址不能超过swap+overcommit_kbytes配置量
或允许commit的虚拟地址不能超过swap+overcommit_ratio* physical RAM

看上文红色加粗部分，overcommit_kbytes和overcommit_ratio是二选一，你配置其中一个的时候另一个自动被禁用。当cat /proc/sys/vm/overcommit_kbytes或/proc/sys/vm/overcommit_ratio值为0的时候就表示该选项是被禁用的那个。

其实这个点是作为Linux OOM的延伸，追下来你会发现oom其实是allocate page中的一个子流程，也就是说oom是在分配物理内存的过程中才会触发。

至于malloc、mmap这些申请虚拟内存是不会导致oom的（要等到实际用到物理内存的时候才会通过缺页异常来分配物理内存，这时候才会导致oom），你能不能在进程的虚拟地址空间overcommit，取决于配置项overcommit_memory、swap、overcommit_kbytes、overcommit_ratio*physical RAM等配置项。

Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.