[Freeipa-devel] Designing better API compatibility

[Nathaniel McCallum]

On Fri, 2015-03-20 at 09:58 -0400, Simo Sorce wrote:
> On Fri, 2015-03-20 at 14:38 +0100, Martin Kosek wrote:
> > On 03/20/2015 02:19 PM, Simo Sorce wrote:
> > > On Fri, 2015-03-20 at 14:13 +0100, Martin Kosek wrote:
> > > > Hi guys,
> > > > 
> > > > I would like to resurrect the discussion we had during 
> > > > DevConf.cz time, about
> > > > API compatibility in the FreeIPA server.
> > > > 
> > > > So right now, we maintain the backward compatibility, old 
> > > > clients can talk to
> > > > newer servers. Forward compatibility is not maintained. 
> > > > Unfortunately, this is
> > > > not very helpful in real deployments, where the server will 
> > > > often be some
> > > > RHEL/CentOS system and the client may be the newest Fedora - 
> > > > with newer API
> > > > than the server. This is the toughest part we need to solve.
> > > > 
> > > > There 3 main areas we wanted to attack with respect to 
> > > > compatibility:
> > > > 
> > > > 1) API publishing and/or API browser
> > > > This is mostly documentation/interactive browser to see the 
> > > > supported API of
> > > > the server. It should not be difficult, it would just consume 
> > > > the metadata
> > > > already generated by the server.
> > > > 
> > > > Ticket: https://fedorahosted.org/freeipa/ticket/3129
> > > > 
> > > > 2) Forward compatibility of the direct API consumers
> > > > Until now, to keep newer clients working against older server, 
> > > > we are using the
> > > > following trick in the ipa-client-install:
> > > > 
> > > > https://git.fedorahosted.org/cgit/freeipa.git/tree/ipa-client/ipa-install/ipa-client-install#
> > > > n1649
> > > > 
> > > > It mostly works, one just needs to know the minimal version 
> > > > that needs to be
> > > > supported. It would be more user friendly, however, if this 
> > > > check is done on
> > > > the server automatically, without user having to research it. 
> > > > This applies both
> > > > for ipalib python lib consumer and for direct JSON-RPC 
> > > > consumers.
> > > > 
> > > > Ticket: https://fedorahosted.org/freeipa/ticket/4739
> > > > 
> > > > 3) Forward compatibility of the "ipa" client tool
> > > > There are different approaches how to fix this, the generally 
> > > > accepted idea was
> > > > to implement very thin client, which would download and cache 
> > > > metadata from the
> > > > server on the client and generate the CLI from it. We would 
> > > > only need to have
> > > > separate client only plugins, basically implementing 
> > > > interactive_promt_callback
> > > > from existing server side plugins.
> > > > 
> > > > Tickets: https://fedorahosted.org/freeipa/ticket/4739,
> > > > https://fedorahosted.org/freeipa/ticket/4768
> > > > 
> > > > 
> > > > Now, question is what we can do in 4.2. I do not think we can 
> > > > manage to rewrite
> > > > "ipa" command in the thin client, but we should do at least 
> > > > some portion of 1)
> > > > and 3).
> > > > 
> > > > I could not decipher that from our Devconf.cz notes. To me, 
> > > > the simplest way of
> > > > fixing forward compatibility seems to be following steps 
> > > > (besides not making
> > > > API backwards incompatible - i.e. what we do already):
> > > > 
> > > > - keep sending API version from client to server
> > > > - server should not refuse newer API versions
> > > > - only raise error when an unknown option or unknown command 
> > > > is used
> > > 
> > > This is not sufficient, older 3.3 and 4.x servers can't be 
> > > changed and we MUST be compatible with those.
> > > Basically the plan MUST work with already released servers, this 
> > > is a constraint that cannot be releaxed, please work within this 
> > > limitations.
> > 
> > Correct. I see 2 approaches here:
> > 
> > a) Thin client, which simply downloads metadata from the (old) 
> > server and won't 
> > use unsupported commands/parameters
> > b) Not-so-thin client that knows the minimal API versions of  
> > commands/parameters (can be annotated in the code), that would 
> > ping the server 
> > first to identify it's version, validate that the chosen set of  
> > commands/parameters is supported on that server and then send the 
> > commands with 
> > that version.
> 
> If we have a recognizable error the client can take an optimistic 
> approach, send the command normally, if it gets an error that the 
> server does not understand it, it checks the version in the reply 
> and falls back to an older "baseline" version of the command (if 
> possible) or bails out with an error.

My understanding was that:

1. We already publish all the information necessary to implement a 
thin client, and have for some time.

2. Thus, the thin client would work on both new and old versions since 
it just simply translates from user input into JSON/XML.

3. Only plugins with specific client behavior would need to be ported 
to the thin client. A prime example of this is otptoken-add-yubikey.

My preference is solidly for implementing the thin client first. Once 
we have decoupled the client from the current plugin framework, server-
side changes can be made in isolation. This decoupling is the move 
that is essentially necessary to provide proper API versioning. And if 
this can't land for 4.2, land it in the next release. I'd rather do 
API-stability correctly and a release later than rushed with 
compromises. We have to live with this forever.

Nathaniel