[Freeipa-devel] Client-side command in the IPA framework

Tue Mar 4 22:21:58 UTC 2014

On 03/04/2014 02:27 PM, Nathaniel McCallum wrote:
> On Tue, 2014-03-04 at 14:11 -0500, Dmitri Pal wrote:
>> On 03/04/2014 02:03 PM, Nathaniel McCallum wrote:
>>> On Mon, 2014-03-03 at 20:12 -0500, Dmitri Pal wrote:
>>>> On 03/01/2014 10:07 PM, Adam Young wrote:
>>>>> On 02/28/2014 10:21 AM, Petr Viktorin wrote:
>>>>>> On 02/28/2014 04:15 PM, Alexander Bokovoy wrote:
>>>>>>> On Fri, 28 Feb 2014, Nathaniel McCallum wrote:
>>>>>>>> On Fri, 2014-02-28 at 16:43 +0200, Alexander Bokovoy wrote:
>>>>>>>>> On Fri, 28 Feb 2014, Nathaniel McCallum wrote:
>>>>>>>>>> On Fri, 2014-02-28 at 10:47 +0100, Petr Vobornik wrote:
>>>>>>>>>>> On 28.2.2014 04:02, Rob Crittenden wrote:
>>>>>>>>>>>> Alexander Bokovoy wrote:
>>>>>>>>>>>>> On Thu, 27 Feb 2014, Nathaniel McCallum wrote:
>>>>>>>>>>>>>> So the recent discussion on importing tokens led me to write a
>>>>>>>>> script to
>>>>>>>>>>>>>> parse RFC 6030 xml files into IPA token data. This all works
>>>>>>>>> well. But
>>>>>>>>>>>>>> now I need to integrate it into the IPA framework.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This command will parse one or more xml files, creating a set
>>>>>>>>> of tokens
>>>>>>>>>>>>>> to be added. Given that we already have otptoken-add on the
>>>>>>>>> server-side,
>>>>>>>>>>>>>> it seems to me that all work needs to be done on the
>>>>>>>>> client-side. How do
>>>>>>>>>>>>>> I create a new client-side command that calls existing
>>>>>>>>> server-side API?
>>>>>>>>>>>>> subclass from frontend.Local, override run() or forward()
>>>>>>>>> method and
>>>>>>>>>>>>> perform batch
>>>>>>>>>>>>> operation of otptoken_add from there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> See cli.help, for example.
>>>>>>>>>>>> If you do an override, do forward() for cli-specific work.
>>>>>>>>>>>>
>>>>>>>>>>>> But you should do as little as possible for reasons you already
>>>>>>>>> stated:
>>>>>>>>>>>> the UI. Anything you do in forward Petr will need to implement
>>>>>>>>> in the UI.
>>>>>>>>>>>> Unfortunately we don't yet have a nice way to handle files.
>>>>>>>>> We have
>>>>>>>>>>>> tickets open at https://fedorahosted.org/freeipa/ticket/1225 and
>>>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2933
>>>>>>>>>>>>
>>>>>>>>>>>> If this file is something that would be pasted into a big text
>>>>>>>>> field
>>>>>>>>>>>> then you can probably handle it in a similarly clumsy way that
>>>>>>>>> we do
>>>>>>>>>>>> CSRs in the cert plugin.
>>>>>>>>>>>>
>>>>>>>>>>>> rob
>>>>>>>>>>> +1 for parsing it on server. Otherwise every client, not just CLI
>>>>>>>>> or Web
>>>>>>>>>>> UI, would have to reimplement the same logic - having it on server
>>>>>>>>> will
>>>>>>>>>>> support better integration with third party products.
>>>>>>>>>>>
>>>>>>>>>>> Parsing on client would be understandable if there was some middle
>>>>>>>>> step
>>>>>>>>>>> which would require some action from user, i.e, pick only some
>>>>>>>>> tokens to
>>>>>>>>>>> import.
>>>>>>>>>> If we parse on the server side, how do we handle the long-running
>>>>>>>>>> operation? Think of the case of importing hundreds or thousands of
>>>>>>>>>> tokens...
>>>>>>>>> Why then to do it as a IPA CLI command at all?
>>>>>>>>> This is an administrative task which can be done with a separate
>>>>>>>>> ipa-otp-import command, designated to run on IPA masters.
>>>>>>>> Agreed.
>>>>>>>>
>>>>>>>> 1. Is there a framework for this? Or should it just be an independent
>>>>>>>> script?
>>>>>>> We don't really have a framework for administrative tools. You may
>>>>>>> start
>>>>>>> with install/tools/ipa-adtrust-install, it is main part is relatively
>>>>>>> independent of the task (which is in
>>>>>>> ipaserver/install/adtrustinstance.py)
>>>>>>>
>>>>>> The framework is there, new tools use it, and there's a ticket to
>>>>>> convert old ones: https://fedorahosted.org/freeipa/ticket/2652 (it's
>>>>>> low priority in Future Releases, so not much progress is there...)
>>>>>> Also see http://www.freeipa.org/page/V3/Logging_and_output
>>>>>>
>>>>>>
>>>>> The RESTful approach would be:
>>>>>
>>>>> 1. Upload a file to a specific URL (not JSON RPC)
>>>>> 2.  Receive back a 202 Accepted  HTTP Request, to include an URL to
>>>>> poll for status
>>>>>
>>>>> Not certain the right response from the URL in step 2 would be, but I
>>>>> am assuming it would be 200 with the body of the message stating:
>>>>> processing or completed.
>>>>>
>>>>> It would be really nice if the Batch command could be handled this way
>>>>> as well.  The response back could be the partial responses until
>>>>> processing is complete.
>>>>>
>>>>> It might also be nice to supply an email address for notification of
>>>>> completed processing instead of polling, if it is going to be a really
>>>>> long running task.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Freeipa-devel mailing list
>>>>> Freeipa-devel at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-devel
>>>> Yes I think that:
>>>> 1) We should not limit it to server side operation only
>>>> 2) Upload the whole file and then process it.
>>>> 3) We should already have code to upload files, we did it for
>>>> entitlements and were supposed to use for certs.
>>>> 4) Make sure we have a generic upload mechanism that reads a chunk of a
>>>> configurable size and asks for more (pagination by 65K might be a good
>>>> default).
>>>>
>>>> Regarding token files specifically: they can be big but not super huge.
>>>> 10-20K tokens makes sense but probably not more. More than that would be
>>>> a real corner case becuase it is hard to deploy that amount of tokens at
>>>> the same time. It can take months and you do not want token file to
>>>> contain many tokens that would sit on the shelf. Tokens expire so it is
>>>> inefficient to buy huge chunks and let them sit unused.
>>>>
>>>> UI you allow uploading file too and then would process it locally.
>>>> The processing of the file should generate a log or report. It would be
>>>> nice to get indication from the server that it is still working so may
>>>> be upload protocol should be something like:
>>>>
>>>> client: Initialize the transfer
>>>> server: ready
>>>> client: here is the chunk of data
>>>> server: ack
>>>> ...
>>>> client: here is the last chunk of data
>>>> server: ack, (forks the file processing method that updates shared
>>>> status data) come back in x seconds
>>>> client: how are things?
>>>> server: working, here is current status, come back in x seconds
>>>> ...
>>>> client: how are things?
>>>> server: done, here is current status, have errors in a file
>>>> client: start download
>>>> server: here is the chunk
>>>> ...
>>>>
>>>> I think we can short socket the command for now to fail if it is not
>>>> local on the server and then build the upload mechanism but separate
>>>> command as proposed in this thread would lock us in a local approach
>>>> forever.
>>> The problem is that we have no infrastructure for any of this. It would
>>> all have to be built just for the import command. It is also a fairly
>>> rare, admin-only operation. For this reason, I am leaning towards
>>> implementing it as a simple script to be executed on the FreeIPA master.
>>> The main drawback of this approach is that you don't get import
>>> functionality in the UI.
>>>
>>> I also disagree that doing it this way now necessitates we do it this
>>> way forever. In fact, it is more likely that if we design a fully
>>> featured server implementation now, we'll get details wrong and be stuck
>>> with it. Doing it on the master using the existing API frees us to add a
>>> server-side API in the future.
>>>
>>> Nathaniel
>>>
>> We already have use cases for upload but we defer it again and again for
>> a better time.
>> We need to upload and download files for other use cases:
>>
>> Download:
>> a) Keytabs (we have a ticket to do it in the UI)
>> b) Reports
>> c) Import logs (for tokens or users)
>> ...
>>
>> Upload:
>> a) Token file
>> b) SSH public keys
>> c) Register user certificate in IPA
>> ...
>>
>> We need this mechanisms anyways.
>> We have been deferring this for too long.
> I agree.
>
>> I would rather defer support of the hardware tokens till next version
>> and do an upload in the cli and UI than cut the corner and create yet
>> one off mechanism.
>>
>> IMO we are going the wrong way about it.
> So the problem isn't hardware tokens, the problem is that users can
> modify tokens they shouldn't (these are most commonly hardware, but not
> necessarily so). Support for this can be added trivially once I get some
> of the more pressing issues off my plate. I don't see any reason to
> delay this. Let's not conflate issues.
>
> Imports are a different story. We could land an independent import
> script (IIS) in 4.0. This functionality could be moved to a framework
> script (FS) later. If backwards compatibility for this is a concern, the
> IIS could just point to the FS. If we desire not to support the IIS at
> all, we could ship it as contrib or distribute it independently.
>
> But once we have infrastructure for uploads/downloads/long-running task
> state, the code from the IIS should move to the server side with little
> modification. So it isn't lost effort.
>
> Having *some* form of import is highly desirable for 4.0.
>
> Nathaniel
>
I bet it will be a complete rewrite but I would not argue.

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.

-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/