Here is an example:

idlistscanlimit: 4000 (default)
[11/Sep/2017:12:03:33.524447131 +1000] conn=2 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(&(objectClass=inetOrgPerson)(objectClass=person)(objectClass=posixAccount)(uid>=test0053715))" attrs="uid"
[11/Sep/2017:12:03:54.236090743 +1000] conn=2 op=1 RESULT err=0 tag=101 nentries=46286 etime=20.1288196523 notes=A
idlistscanlimit: 99999999999
[11/Sep/2017:12:01:34.797114055 +1000] conn=4 op=1 SRCH base="dc=example,dc=com" scope=2 filter="(&(objectClass=inetOrgPerson)(objectClass=person)(objectClass=posixAccount)(uid>=test0053715))" attrs="uid"
[11/Sep/2017:12:01:45.866876775 +1000] conn=4 op=1 RESULT err=0 tag=101 nentries=46286 etime=11.0069939325

So despite being over 40,000 entries, using the indexes is still faster.

In fact, it looks like the idlistscan limit is an attempt to optimise the "(&(uid=x)(objectClass=y))" case to improve this search, but the recent changes to filter_optimise in https://pagure.io/389-ds-base/issue/49372 solve this much more completely.

I think we should remove the idlistscanlimit, as no other database (IE postgresql) has such an odd concept of bypassing indexes. We can gain more from improve filter optimise than trying to hand tune these numbers for specific cases,

Following this, this would actually be a case to raise the filter test threshold, because this would allow better shortcutting of genuinely small sets that want to avoid large indexes. IE if we set test thrushold to say 32, but then if the set is larger than this, it's probably better to actually use the index. We could do some testing of the numbers here to be sure, but 32 - 64 could be good guesses.

Another point is that to maek filter optimisation easier, perhaps we should change their structure from linked list to something easier to sort / re-arrange?

https://pagure.io/389-ds-base/issue/47474

Relies on https://pagure.io/389-ds-base/issue/49372

I don't think we should change the default.

There are some scenarios wher it might be better to always return an idl. It is always a decision what is more costly: to handle and manage a very large idlist (it also has a lot of database accesses) or to handle all entries (which might be already in the entry cache).
The scanlimit is configurable and GSS knows to suggest proper settings.
If we change index lookup and idl handling (in other tickets) we need to document the effects it might have an the use of scan limits.
If we are really confident that always returning and idl would be the right thing, then we should remove the scan limit at all.

In general I think for changes like this we should have a set of reliable performance benchmarks first.

@lkrispen I have been performing a number of tests with the filter optimisation patch enabled, and in all cases the idlistscanlimit of 999999999 exceeded the performance of the current default. I will prepare a set of numbers for you.

As much as I love and respect GSS, they do NOT know how to suggest a proper setting here. This is a deeply confusing setting that relies on intimate knowledge of our internal search procedure that most users don't have.

I agree that we should remove the scan limit completely, but I would like to do this first to give us a clean backout strategy if it does fail on us,

waiting for the set of numbers

Yep! I need to make the numbers for you! I've been quite busy as you might know, and I want to work on this with @vashirov to make our performance test framework? If you are going to do something, may as well do it correctly.

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2435

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.