[Freeipa-devel] Dojo and Web UI in 3.2

Petr Vobornik pvoborni at redhat.com
Wed Nov 7 15:46:12 UTC 2012


We discussed some things on IRC. I wrote some comments below to allow 
others to chime in.

Adam also created a public etherpad where we wrote most of the issues 
and possible solutions in some frameworks. I also wrote there short 
reviews of various JavaScript frameworks.

https://etherpad.openstack.org/webui-idm

On 11/01/2012 03:01 PM, Adam Young wrote:
> On 11/01/2012 09:25 AM, Petr Vobornik wrote:
>> On 10/31/2012 11:13 PM, Dmitri Pal wrote:
>>> On 10/30/2012 01:20 PM, Petr Vobornik wrote:
>>>> On 10/30/2012 06:48 AM, Endi Sukma Dewata wrote:
>>>>> On 10/29/2012 4:27 AM, Petr Vobornik wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I would like to make a bigger change in Web UI. Basically I think
>>>>>> Web UI
>>>>>> would benefit from using a Dojo toolkit, a JS framework. I would
>>>>>> like to
>>>>>> know if I can proceed with incorporating it. I think this is the best
>>>>>> time (end of 3.0/3.1 and a beginning of 3.2) to do such change.
>>>>>
>>>>> I don't know about the timing, but I agree we need a better framework.
>>>>> This was also discussed some time ago with Adam.
>
>
> Looked into Dojo.  My take was that the support of JQuery was more
> important than what Dojo provided.  I still feel that is the case.    If
> you are going to adopt a framework,  choose one that is built on JQuery.

I strongly disagree.

First jquery is mostly DOM manipulation, AJAX and some effects library. 
Other libraries can use it but there is no need to be build upon it. 
None or soft dependency on jquery is much better.

>
> We also wanted to make the Entity definition as declarative as possible,

I don't want to change it. I like this approach even thought it brings 
some problems.

> and we didn't want to be driven by another developers view of M-V-C.  I
> still feel this is most important.

Do we or don't we? Endi wrote that you agreed that we need a better 
framework? Should the framework fix only issues other than MVC or MVC too?

Most of the framework which might help us (persistance, structure) has 
some view of MVC. My conclusion about them is at the end of the mail and 
in the etherpad.

>
> There are certainly things I would want to tighten up in the IPA UI.  I
> am not sure that the construction policy (no new keywords) was the
> absolute best choice.

IMO 'new' keyword usage is not bad if proper convention is used: all 
functions which are supposed to be used as constructors must start with 
capital letter.

>
>
>>>>>
>>>>>> Why Dojo?
>>>>>> ---------
>>>>>> It may solve several existing issues in Web UI architecture/code
>>>>>> structure.
>>>>>>
>>>>>> TLDR version:
>>>>>> * incorporates a lot of functionality we don't have to write
>>>>>> * is considered to be a framework for enterprise class web
>>>>>> applications
>>>>>> * reuse an establish library which may be known to more developers
>>>>>> then
>>>>>> custom build in-house solution
>>>>>> * makes code more maintainable
>>>>>> * opportunity to separate Web UI framework and IPA specific parts
>>>>>>
>>>>>> Some functionality of Dojo overlaps with jquery which is not good,
>>>>>> but
>>>>>> the reason to use Dojo are the features jquery lacks.
>>>>>> I also look on other frameworks or independent libraries: AngularJS,
>>>>>> backbone.js, closure js framework and several single-purpose
>>>>>> libraries
>>>>>> (crossroads, hasher, underscore, amd-utils,. IMO Dojo is the best fit
>>>>>> for us.
>>>>>
>>>>> I have not used Dojo or any of the above frameworks so I can't really
>>>>> say anything good/bad about them, but backbone.js also seems to be a
>>>>> popular lightweight alternative if we aren't planning to use the
>>>>> Dijit.
>>>>> We should consider whether a framework will work nicely with others or
>>>>> whether it will require us to use it for everything.
>>>>
>>>> I fear that with backbone.js we would have to rewrite the whole UI to
>>>> not bend the framework much. From looking at examples IDK how would we
>>>> incorporate it. I think it would be a nice candidate if we would
>>>> rewrite UI from scratch.
>>>>
>>>> I like Dojo because we can make the changes gradual. In theory we can
>>>> just utilize the things from dojo which we want and leave the rest
>>>> untouched.
>>>>
>>>>>
>>>>> If possible, we should try to address the issues below independently
>>>>> from each other (even if we end up using Dojo for everything) while
>>>>> we're still learning about it. Some of the issues are actually general
>>>>> design issues, so it won't be solved simply by converting the
>>>>> framework.
>>>>> Try to avoid refactoring the code and converting the framework at the
>>>>> same time because the patch could become very big and hard to review.
>>>>
>>>> I agree. Originally I was just looking if there is some small library
>>>> which has a reasonable implementation for bindable object store. To
>>>> keep with saying: "best code is the one you don't have to write". I
>>>> ended with Dojo because it can do a lot of stuff we might need in
>>>> time. Of course all of dojo functionality can be replaced with other
>>>> libraries but I think it's better to use one (and because of AMD
>>>> modules, we can create small custom build) and not to worry about
>>>> making them all work together.
>>>>
>>>>>
>>>>>>   From Dojo I would like to use:
>>>>>>
>>>>>> ### 1) Build system and AMD modules
>>>>>> Web UI doesn't use any JavaScript optimizer therefore we need to keep
>>>>>> low file count and minimum size of files. We share one file for
>>>>>> multiple
>>>>>> components to keep the file count low. It leads to very long files.
>>>>>> It's
>>>>>> harder to maintain than separate files for each component. AMD
>>>>>> modules
>>>>>> and a builder can solve it.
>>>>>>
>>>>>> AMD modules [1] encourages to use one file per component. Then a dojo
>>>>>> builder can be used to modify the declaration of a module for file
>>>>>> concatenation then it can be sent to JavaScript minifier/compiler.
>
> I like the concept of modules, but make sure whatever you use works well
> with the debugger.  Keep development as straightforward as possible.

I Agree.

> I
> would take a performance hit at application load time if it means that
> the user sees the same thing as the developer:  it will pay off in the
> long haul.

Please elaborate.

IMO fast load times are one of the things which makes admins happy. I'm 
not convinced that admin needs to view the code. It's just nice to have. 
Most of people won't debug the code. Most (maybe all) Web UI bug reports 
contain just steps to reproduce, error message or screenshots. I don't 
remember stack-trace or similar code-related report.
>
>>>>>>
>>>>>> Dojo builder [2] is quite powerfull, it can define 'layers' which
>>>>>> serves
>>>>>> as a part of application - basically a sets of component so an
>>>>>> application with ie. 150 files can be compile to one or more (if
>>>>>> needed)
>>>>>> .js. Dojo loader takes care of the loading.
>>>>>>
>>>>>> AFAIU build system can be run at rpm build time using rhino (version
>>>>>> 1.7R3-4 needed). Uglify.js can be used as a compiler, it's JS library
>>>>>> and can be run in rhino too.  The builder and uglify.js would be
>>>>>> needed
>>>>>> to budle in our source tarball but they won't be in output rpms so
>>>>>> they
>>>>>> shouldn't break Fedora packaging rules. Both are BSD licensed.
>>>>>> Builder
>>>>>> would need a patch to support Uglify (it's commited in dojo upstream
>>>>>> trac but I want to incorporated it in latest stable version).
>
> Just want to reiterate the power of being able to dop development
> withouth having to do a build.  DO NOT MAKE A DELIBERATE BUILD STEP FOR
> JAVASCRIPT!

Build is just for production. Development will remain build-less.
Currently I'm the person which does the most UI debugging so it's 
natural that I don't want to make it harder for me :).

>>>>>>
>>>>>> [1] http://dojotoolkit.org/documentation/tutorials/1.8/modules/
>>>>>> [2] http://dojotoolkit.org/documentation/tutorials/1.8/build/
>>>>>
>>>>> I suppose now if we want to test the static page without web server we
>>>>> will need to build the pages first?
>>>>
>>>> It's not necessary. For developing purposes it is better to use
>>>> uncompiled files. Loader can load each separate file (if it's AMD
>>>> module) if it isn't in bundle.
>>>>
>>>>>
>>>>> If we use Dojo do we need to include Dojo js files into IPA source, or
>>>>> can we use the ones from Dojo rpm and only include it at build time?
>>>>
>>>> I'm for including it because we can create a build with the parts we
>>>> actually use. Dojo builder isn't in the rpm.
>>>>
>>>> AFAIK the  dojo rpm contains already built files. It has a core
>>>> dojo.js file which contains basic modules to work but the rest of
>>>> modules are separated. IDK if it can be used in a build process. I
>>>> have to test it.
>>>>
>>>>>
>>>>> There's also RequireJS (http://requirejs.org/).
>>>>
>>>> require.js is just AMD loader but it uses r.js as optimizer. r.js
>>>> seems more lightweight than dojo builder. I would stick with dojo
>>>> loader and builder if we use other dojo features.
>>>>
>>>>>
>>>>>> ### 2) Object store/model
>>>>>> At the beginning Web UI was quite simple - a command filled widgets,
>>>>>> widgets got edited, update command was constructed from widgets,
>>>>>> it was
>>>>>> sent to  server and at the end widgets were updated with new values.
>>>>>>
>>>>>> Now various parts of displayed page (action lists, panels, status
>>>>>> widgets, facet header) needs to use loaded values too or influence
>>>>>> others. This issue was often discussed in patch reviews. As a
>>>>>> solution,
>>>>>> some model object with bidirectional binding capabilities would be
>>>>>> greatly beneficial.
>>>>>>
>>>>>> Dojo provides a extensible object store [1]. I would like to use
>>>>>> 'memory
>>>>>> store' and 'observable' and extend it with 'dirty-check'
>>>>>> capabilities.
>>>>>>
>>>>>> [1]
>>>>>> http://dojotoolkit.org/documentation/tutorials/1.8/intro_dojo_store/
>>>>>
>>>>> Backbone also has similar mechanisms for events and sync.
>>>>>
>>>>>> ### 3) Class model
>>>>>> Yes, in IPA Web UI we are using some class model, but it has flaws,
>>>>>> mainly in initialization of a object (some initialization code of
>>>>>> parent
>>>>>> class may be called before inheritance is finished).
>>>>>>
>>>>>> Dojo has a class system [1] which is easy to use and support
>>>>>> inheritance, mixins and overrided function calls.
>>>>>>
>>>>>> [1] http://dojotoolkit.org/documentation/tutorials/1.8/declare/
>>>>>
>>>>> At first glance Backbone seems to be better in this area, but Dojo
>>>>> might
>>>>> also be sufficient for our purposes.
>>>>
>>>> Backbone seems to have advantage in calling super class' constructor.
>>>> It is not recommended in Dojo.
>>>>
>>>>>
>>>>> There's going to be some challenges converting the current class
>>>>> framework. Like you said, the current initialization isn't well
>>>>> structured, so we might need to fix that first before converting the
>>>>> class framework. And then we use 'that' instead of 'this', and
>>>>> sometimes
>>>>> both, and they get confusing, so we'd have to be very careful when
>>>>> converting them. Also, the current way to call a method in super class
>>>>> needs to be fixed.
>>>>
>>>> We can come with some strategy to keep patch reviews simple.
>>>> IE:
>>>> * move initialization stuff to init method (or more fine-grained) and
>>>> use builder to create and instance (in some parts its already done),
>>>> this way we eliminate statements in class declaration (except method
>>>> declaration and overriding)
>>>> * change to the classes: change that to this/self(to keep context in
>>>> closures)
>>>> * eliminate method overrides with inherited() calls.
>>>> * put init calls to constructor
>>>> * change builder again
>>>>
>>>>>
>>>>>> ### 4) Localization
>>>>>> Web UI and pages in config, migrate and error folder are not
>>>>>> localized
>>>>>> until successful login. It's because Web UI gets localized strings by
>>>>>> API call which needs prior authentication. We should use other
>>>>>> method to
>>>>>> provide localized config, error and login pages.
> The localization information is not necessarily accessable to the
> browser.  The only thing we can depend on is that the server gets it,
> and then sends back the strings.  That was the rationale for the current
> mechanism.
>>>>>>
>>>>>> Dojo has a easy to use localization solution [1]. Basically
>>>>>> strings are
>>>>>> defined in .js files as modules, translations in subfolders. Dojo
>>>>>> loader
>>>>>> and i18n framework take care of loading of proper string depending on
>>>>>> user's local. It doesn't need communication with JSON-RPC so can be
>>>>>> used
>>>>>> in config and login pages.
>>>>>>
>>>>>> We would  have to make a script to translate the format to a format
>>>>>> usable by transifex.
>>>>>>
>>>>>> [1] http://dojotoolkit.org/documentation/tutorials/1.8/i18n/
>>>>>
>>>>> A possible alternative is jquery-i18n-properties
>>>>> (http://code.google.com/p/jquery-i18n-properties/).
>>>>>
>>>>>> ### 5) Navigation
>>>>>> Our navigation code is limited to IPA entities. Introducing
>>>>>> non-entity
>>>>>> page like log (#3040) or help is problematic. Routing code should be
>>>>>> improved. Separate entity initialization and menu definition. As the
>>>>>> re-factoring will be at some point done, we might consider to use
>>>>>> dojo.router and dojo.hash instead of $.bbq. This is really
>>>>>> optional and
>>>>>> not strictly related to Dojo, but I wanted to mention it.
>>>>>>
>>>>>> Better handling of pkeys may fix `#2741 [ipa webui] Intermittent
>>>>>> errors
>>>>>> - 'cn' is required; limits exceeded for this query`
>>>>>
>>>>> Backbone also has router and history.
>>>>>
>>>>>> ### 6) Easier unit tests
>>>>>> By utilizing AMD modules, Localization, Model binding we might be
>>>>>> able
>>>>>> to write unit tests of widgets easier. Now we have to simulate
>>>>>> entity,
>>>>>> facets, fields, strings loaded from JSON-RPC...
>>>>>
>>>>> Yes, that would be great.
> I'd like better inversion of control in the JS code in general.  But you
> don't need to move to Dojo to do that.  As I recall, Dojo "wiring up"
> was about a 1/2 way solution (but it has been a long time since I looked)

AMD is not IoC, but it share some ideas. The only framework with DI (of 
the reviewed) is Angular.js.

>>>>>
>>>>>> ### 7) Separation of framework and IPA
>>>>>> We often talk about separating Web UI framework and IPA related
>>>>>> stuff.
>>>>>> We can do it along with this refactoring.
>>>>>> https://fedorahosted.org/freeipa/ticket/3030
>>>>>
>>>>> Yes, but it might need to be fixed separately.
>>>>
>>>> I don't want to this all stuff in one patch :).
>>>>>
>>>>>> What I don't want to do
>>>>>> -----------------------
>>>>>> * rewrite whole UI, changes should be gradual
>>>>>> * replace jQuery. Most of our widgets are using jquery, I don't
>>>>>> want to
>>>>>> change this code, so I don't want to replace it with Dojo widget
>>>>>> system
>>>>>> - too big effort. So in the result the HTML output should not be
>>>>>> changed
>>>>>> much - not to affect automated tests.
>>>>>>
>>>>>> ### Steps how could be Dojo incorporated
>>>>>> It should be done gradually.
>>>>>> 1) make the build system working. Without it a lot of .js files
>>>>>> would be
>>>>>> transmitted to browser which is really bad.
>>>>>
>>>>> This is probably the easiest part. The next steps could be
>>>>> difficult to
>>>>> untangle. So be ready for a long haul. :)
>>>>
>>>> The harder part is not to populate git and rpm with a lot of
>>>> third-party stuff. We need to convince dojo builder to use uglify.js
>>>> under rhino to avoid using of closure compiler or shrink-safe which
>>>> are not and probably will not be bundled in fedora and RHEL and
>>>> definitely we don't want to bundle them.
>>>>
>>>> I'm thinking about making a script which would create WebUI-devoloper
>>>> enviroment:
>>>>   * checkout dojo/dojo and dojo/util git repository to 1.8.1 tag
>>>>   * create a symbolic links to the tag
>>>>   * apply custom patches on dojo/util (there will definitely be one)
>>>>   * include in FreeIPA git only builded dojo.js (in the case we have
>>>> separate files for dojo.js and freeipa.js (just a suggestion of name))
>>>>   * will investigate if we can build the builder and include it in git
>>>> for rpm builds.
>>>>
>>>> We can of course include into our git a whole 1.8.1 dojo/dojo and
>>>> dojo/util but I would rather avoid it because it's a LOT of files.
>>>>
>>>>>
>>>>>> 2) separate components to individual files and transform then into
>>>>>> AMD
>>>>>> modules. We might change namespaces and dependencies to separate
>>>>>> Web UI
>>>>>> framework and IPA related stuff.
>>>>>> 3) change class system of entities and facets
>>>>>> 4) implement a 'model'
>>>>>>>>>>>> At this point it depends on priorities and new tickets with higher
>>>>>> priority may appear.
>>>>>
>>>>
>>>>
>>> Still sounds scary... What is the general recommendation? Are the
>>> arguments for or against Dojo convincing?
>>>
>> I proposed it, so the question is mainly for Endi or Adam, if he is
>> interested (added Cc).
>>
>> Just to add:
>> The question is whether we want to use third party code to solve some
>> issues or we want to write everything ourselves. Everything I wrote
>> above can be solved by us but it will probably take much more time.
>> Dojo itself won't fix our problems it just provides means to fix
>> them.  As Enti wrote, for the problems we have, it can be replaced by
>> multiple libraries combined(require.js, r.js, backbone.js,
>> jquery-i18n-properties).
>>
>> I don't want to repeat myself much so adding just some dojo-related
>> pros and cons:
>>
>> Cons:
>>  * third party code - we need to watch updates, sec issues
>>  * more complex build of UI which eventually needs more unit tests
>> (basically I want some build with or without Dojo)
>>
>> Pros:
>>  * third party code - don't have to reinvent the wheel
>>  * establish library (good docs, some books were writen)
>>  * using good modular structure, we can pick the parts we actually
>> want, when we want (loader, builder, class system, object store,
>> localization, modular system)
>>
>
>
> My take away is the a Framework is the right direction, but Dojo is not
> the right Framework.
Please elaborate.

> I would look to see if someone has built the
> equivalent on top of JQuery.  If not,  pull in the various pieces from
> JQuery based projects and move forward with that.
I disagree. As I said jquery is good at what it does. Problems we want 
to solve don't have anything to do with jquery. JQ is functional library 
which calls usually bind to some page element. That is not desired when 
dealing with architectural problems. Various plugins might also have 
problems with long-term support.

>  I think you are right
> to focus on unit tests and modularity.  But I think at this point, you
> would be better off with an incremental approach.

I still think Dojo is the best fit for us.

Conclusion from comparing various frameworks (narrowed to Ember.js, 
AngularJS, backbone.js, JavaScriptMVC, Dojo):
If we don't want to adopt other view of MVC we can ditch Ember.js, 
JavaScriptMVC and Angular JS, without their MVC there is no point to use 
them. If we would re-write the UI from scratch, which we don't want to, 
I would probably choose Ember.js because it uses nice structured way and 
has view composition, we would loose our declarative approach though.
We are left with backbone and dojo. Backbone is very lightweight, it 
can't do anything more than classes, events, router, views and model 
(REST persistence). All of it and *more* can be done with Dojo. Its MVC 
approach is very similar to ours. Model implementation is not only REST 
which is good for us as we use JSON-RPC. Because it's completely AMD, we 
can build a custom dojo build with only the components we care about.

Modularity:
I think we agreed on that we want to change UI to AMD modules. It 
basically means we go with Dojo + uglify.js or we choose require.js, 
r.js, uglify.js and some other library to solve non-module stuff.

-- 
Petr Vobornik




More information about the Freeipa-devel mailing list