[Freeipa-devel] Paging in Web UI

[Petr Vobornik]

On 08/28/2012 04:17 PM, John Dennis wrote:
> On 08/28/2012 09:30 AM, Petr Vobornik wrote:
>> I would like to point out a problem in Web UI related to paging and
>> suggest a possible solution:
>>
>> Current implementation of paging in Web UI is not serving it purpose
>> which should be to do operations faster and be able to find all users.
>>
>> How current implementation should work:
>>    1) get all primary keys (pkeys) of certain object type (users, ...)
>>    2) load first 20 of them
>>    3) from 1) we know record count so we can choose other page (different
>> subset of 20)
>>
>> How it actually works:
>>    * get up to 2000 keys, with more items a LDAP limit is reached and
>> result is truncated
>>    * we don't know how many are there, paging doesn't work as it should
>>
>> Problem:
>> Let's say that paging is working (no LDAP limit) and consider this: Our
>> page size is 20 records. With 400 users it is 20 pages. With 2000 users
>> (current search limit) it is 200 pages and so on. Personally I wouldn't
>> like to go through 20 pages. 200 or more pages sounds like a punishment.
>> There is also a question if we even need to do it. I don't see a benefit
>> of paging here. For smaller setup an increase of page size may help
>> admin to see more users on a page more quickly (rolling mouse wheel is
>> faster than click on next page button).
>>
>> For large setups, size of the pkey list must also be taken on mind. For
>> each record there is a pkey and dn (dn is not required but it would need
>> some hacking in python code to remove it). At the moment the list size
>> raises drastically because JSON response includes indenting by spaces
>> and is not compressed (easy to fix, but still there). With 10 000 or
>> more users this pkey and dn list is pretty big (slow load on slower
>> networks -vpns). This list is also loaded on each page change
>> (record-wise, can be improved - cached).
>>
>> IMO with hundreds or thousands of users the most common use case is
>> search by login or name or something. Paging is not required this case.
>> It may be required if the result count is greater than size limit. In
>> such case an option to temporary increase size limit or enable paging
>> would be beneficial. Apart from this case, paging should be off. It will
>> speed up page load because it executes only one request.
>>
>> Possible solution:
>> 1) I suggest to introduce configuration options for paging. They can be
>> global (default for all search pages) or individual for pages or Web
>> UI's users. Per-user configuration can be stored in browser (to not
>> pollute LDAP with application stuff). Global configuration on server in
>> config plugin (minor polluting).
>>
>> Those options should be (per page and global):
>>    * enable paging
>>    * page size
>>
>> Note: when paging is disabled page size is basically --sizelimit option.
>>
>> 2) On search pages we should have controls to enable/disable paging and
>> to change sizelimit for this particular moment.
>> 3) Compress JSON responses (on httpd level)
>>
>> This way admin can configure default's for his deployment and user's can
>> adjust for certain situations.
>>
>> Btw always including member_xx attributes in find or show commands is
>> not good for search pages either but that's another topic.
>>
>> Comments are welcome.
>
> Your possible solution does not address how many results are fetched
> (unless I misunderstood).
If paging is enabled it doesn't, but it expects, that admin will disable 
it for larger setups. For smaller setups it isn't of much an issue. If 
paging is disabled, the limit is server 'search size limit' or 
--sizelimit option supplied by Web UI.
>
> I'm not sure how per-user preferences are handled in browsers, but don't
> forget we now have session support in the server. Server session data is
> available for use.

I was thinking about using browser local storage (basically key-value 
DB). It has a benefit over session, that it survives a browser restart 
but it should contain only non-sensitive data (other users may see it).
>
> If you don't want to fetch every record to implement paging smartly in
> conjunction with it's performance issues why not do the query on the
> server, cache the results in the session, and have the RPC return the
> total number of results plus a subset of the result. Another RPC could
> retrieve the next/previous subset of results from the cached result in
> the session.

I think most software do paging like this. I don't know the reasons for 
not doing it that way first time. My solution was counting with that we 
still don't want to do it. Endi do you know the reasons? Earlier 
sessions didn't exists, but it is doable without them too.
>
> I don't think there any need in JSON formatted data for pretty printing
> with indentation. Is it an accident or oversight we're pretty printing
> the JSON data in an RPC. For large data sets compression would be a win,
> but most of our RPC communication is not large. Compression has an
> overhead. With small data you'll use more cycles compressing and
> uncompressing than you would sending verbose but small data blobs.
> Compression should be an RPC option specified by either side of the
> connection and the receiver should be prepared to conditionally
> uncompress based on a flag value in the RPC. If you use server session
> data to support paging you may not need to introduce compression since
> you would only be passing one page of data at a time.
>
> User specified page size (without limitation) is an absolute necessity.
> I am frequently annoyed by web sites which either do not allow me to
> specify page size or constrain it to ridiculous hard coded limits such
> as 10, 20 or 30.
>

+1

-- 
Petr Vobornik