389-ds-base

#49430 autotune and variable memory usage
Closed: wontfix by spichugi. Opened by orion.

Closed: wontfix
Reopen Issue

Issue Description

Depending on what a server is doing and the time of slapd startup, memory usage may be hightly variable. Sometimes I see a normal startup:

[26/Oct/2017:15:27:28.372140197 -0600] - NOTICE - ldbm_back_start - found 1016260k physical memory
[26/Oct/2017:15:27:28.374384257 -0600] - NOTICE - ldbm_back_start - found 434068k available
[26/Oct/2017:15:27:28.376550360 -0600] - NOTICE - ldbm_back_start - cache autosizing: db cache: 40650k
[26/Oct/2017:15:27:28.378634702 -0600] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 65536k
[26/Oct/2017:15:27:28.381687360 -0600] - NOTICE - ldbm_back_start - cache autosizing: changelog entry cache (2 total): 65536k
[26/Oct/2017:15:27:28.384736764 -0600] - NOTICE - ldbm_back_start - total cache size: 201072967 B;

Sometimes it gives me a scary warning:

[27/Sep/2017:03:19:35.961741759 -0600] - NOTICE - ldbm_back_start - found 1016260k physical memory
[27/Sep/2017:03:19:35.963918365 -0600] - NOTICE - ldbm_back_start - found 135152k available
[27/Sep/2017:03:19:35.965958987 -0600] - NOTICE - ldbm_back_start - cache autosizing: db cache: 40650k
[27/Sep/2017:03:19:35.968044011 -0600] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 65536k
[27/Sep/2017:03:19:35.970705299 -0600] - NOTICE - ldbm_back_start - cache autosizing: changelog entry cache (2 total): 65536k
[27/Sep/2017:03:19:35.973146106 -0600] - NOTICE - ldbm_back_start - total cache size: 201072967 B; 
[27/Sep/2017:03:19:35.975140790 -0600] - WARN - ldbm_back_start - It is highly likely your memory configuration of all backends will EXCEED your systems memory.
[27/Sep/2017:03:19:35.976919526 -0600] - WARN - ldbm_back_start - In a future release this WILL prevent server start up. You MUST alter your configuration.
[27/Sep/2017:03:19:35.982099452 -0600] - WARN - ldbm_back_start - Total entry cache size: 69197824 B; dbcache size: 33300807 B; available memory size: 138395648 B; 
[27/Sep/2017:03:19:35.990053652 -0600] - WARN - ldbm_back_start - This can be corrected by altering the values of nsslapd-cache-autosize, nsslapd-cache-autosize-split and nsslapd-dncachememsize
[27/Sep/2017:03:20:11.589774337 -0600] - WARN - ldbm_back_start - Your autosized cache values have been reduced. Likely your nsslapd-cache-autosize percentage is too high.
[27/Sep/2017:03:20:11.591916360 -0600] - WARN - ldbm_back_start - This can be corrected by altering the values of nsslapd-cache-autosize, nsslapd-cache-autosize-split and nsslapd-dncachememsize
[27/Sep/2017:03:20:11.593938158 -0600] - NOTICE - ldbm_back_start - found 1016260k physical memory
[27/Sep/2017:03:20:11.595959366 -0600] - NOTICE - ldbm_back_start - found 94660k available
[27/Sep/2017:03:20:11.598015707 -0600] - NOTICE - ldbm_back_start - cache autosizing: db cache: 18932k
[27/Sep/2017:03:20:11.599951982 -0600] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 65536k
[27/Sep/2017:03:20:11.602758519 -0600] - NOTICE - ldbm_back_start - cache autosizing: changelog entry cache (2 total): 65536k
[27/Sep/2017:03:20:11.605412847 -0600] - NOTICE - ldbm_back_start - total cache size: 183281254 B; 
[27/Sep/2017:03:20:11.607372676 -0600] - WARN - ldbm_back_start - It is highly likely your memory configuration of all backends will EXCEED your systems memory.
[27/Sep/2017:03:20:11.609462861 -0600] - WARN - ldbm_back_start - In a future release this WILL prevent server start up. You MUST alter your configuration.
[27/Sep/2017:03:20:11.611527832 -0600] - WARN - ldbm_back_start - Total entry cache size: 48465920 B; dbcache size: 15509094 B; available memory size: 96931840 B; 
[27/Sep/2017:03:20:11.613544324 -0600] - WARN - ldbm_back_start - This can be corrected by altering the values of nsslapd-cache-autosize, nsslapd-cache-autosize-split and nsslapd-dncachememsize
[27/Sep/2017:03:20:29.440380605 -0600] - WARN - ldbm_back_start - Your autosized cache values have been reduced. Likely your nsslapd-cache-autosize percentage is too high.
[27/Sep/2017:03:20:29.442455125 -0600] - WARN - ldbm_back_start - This can be corrected by altering the values of nsslapd-cache-autosize, nsslapd-cache-autosize-split and nsslapd-dncachememsize
[27/Sep/2017:03:20:29.444490702 -0600] - NOTICE - ldbm_back_start - found 1016260k physical memory
[27/Sep/2017:03:20:29.446610331 -0600] - NOTICE - ldbm_back_start - found 33988k available
[27/Sep/2017:03:20:29.448609827 -0600] - NOTICE - ldbm_back_start - cache autosizing: db cache: 6797k
[27/Sep/2017:03:20:29.450711026 -0600] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 65536k
[27/Sep/2017:03:20:29.455403900 -0600] - NOTICE - ldbm_back_start - cache autosizing: changelog entry cache (2 total): 65536k
[27/Sep/2017:03:20:29.459965748 -0600] - NOTICE - ldbm_back_start - total cache size: 173340753 B; 
[27/Sep/2017:03:20:29.462008298 -0600] - WARN - ldbm_back_start - It is highly likely your memory configuration of all backends will EXCEED your systems memory.
[27/Sep/2017:03:20:29.463958945 -0600] - WARN - ldbm_back_start - In a future release this WILL prevent server start up. You MUST alter your configuration.
[27/Sep/2017:03:20:29.466686635 -0600] - WARN - ldbm_back_start - Total entry cache size: 17401856 B; dbcache size: 5568593 B; available memory size: 34803712 B; 
[27/Sep/2017:03:20:29.468622640 -0600] - WARN - ldbm_back_start - This can be corrected by altering the values of nsslapd-cache-autosize, nsslapd-cache-autosize-split and nsslapd-dncachememsize
[27/Sep/2017:03:21:03.583518319 -0600] - NOTICE - ldbm_back_start - found 1016260k physical memory
[27/Sep/2017:03:21:03.586099101 -0600] - NOTICE - ldbm_back_start - found 409328k available
[27/Sep/2017:03:21:03.588307525 -0600] - NOTICE - ldbm_back_start - cache autosizing: db cache: 40650k
[27/Sep/2017:03:21:03.590554184 -0600] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (2 total): 65536k
[27/Sep/2017:03:21:03.594709706 -0600] - NOTICE - ldbm_back_start - cache autosizing: changelog entry cache (2 total): 65536k
[27/Sep/2017:03:21:03.598672169 -0600] - NOTICE - ldbm_back_start - total cache size: 201072967 B;

I'm curious to know if the disk cache is excluded from the memory calculations. I'm also concerned that at some point slapd will fail to start as the warning suggests. Is this truly the case?

Package Version and Platform

389-ds-base-1.3.6.1-21.el7_4.x86_64

Metadata Update from @firstyear:
- Issue assigned to firstyear

Each startup can cause this calculation to differ based on the ram consumed by other processes. We use memavailable in our calculation to avoid the vfs cache from our calculation. IE we only check kernel + application usage, and the max on the system, to calculate how much is there. Then we use 10% of that because we know that in some cases our memory usage can grow (but we are working on reducing this footprint).

So yes, each start can result in a different configuration like this. We have some minimums in place to keep from low memory use scenarios. In general, I think because your machine has 1GB of ram, some factors can influence this calculation a lot more, but provided you have enough swap it will work just fine if you come near that watermark.

So I don't think this is something to worry about, and in the future this will improve as we make the server more memory efficient,

Does that help?

Metadata Update from @firstyear:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None

Metadata Update from @firstyear:
- Issue set to the milestone: 1.4 backlog

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2489

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.