[katello-devel] Bundler vs rpm-gems - in more detail

Tue Aug 28 11:24:55 UTC 2012

On 28/08/12 12:13 PM, Petr Chalupa wrote:
>
>
> On 28.08.12 11:08, Dmitri Dolguikh wrote:
>> On 28/08/12 08:21 AM, Petr Chalupa wrote:
>>>
>>>
>>> On 23.08.12 14:42, Dmitri Dolguikh wrote:
>>>> On 23/08/12 12:56 PM, Petr Chalupa wrote:
>>>>> My original email raised a lot of questions, I should wrote it with
>>>>> more details. I apology for that. I'll try again and elaborate.
>>>>>
>>>>> = Goals
>>>>>
>>>>> 1) To be able to implement the solution in a week or so.
>>>>> 2) Not to affect deployment, keep it same way as now.
>>>>> 3) When you are installing development environment on any rpm-based
>>>>> system rpm-gems are used even if they are older then gems in our
>>>>> gem-repo. (this is borken right now)
>>>>> 4) When you are installing development environment in any 
>>>>> non-rpm-base
>>>>> system you get installed gems as close as possible to what would you
>>>>> get on fedora 16. (you cannot install with bundle install only right
>>>>> now)
>>>>>   * (optionally including the security patches, lets discuss this
>>>>> later)
>>>>>
>>>>> = What was discussed, why I think it wont work
>>>>>
>>>>> == Branch for each platform
>>>>>
>>>>> * There would be little difference between the branches (only Gemfile
>>>>> or Gemfile.lock). Rest of the code will be same.
>>>>> * It would be too much work to maintain it.
>>>>>
>>>>> == Gemfile.lock per platform and environment
>>>>>
>>>>> * We would have to have 8 Gemfile.locks and maintain them all (4
>>>>> platforms multiplied by two environments)
>>>>> * Too much work unless its automated.
>>>>> * Automation would take time to setup.
>>>>>
>>>>> == Aeolus solution, bundler_ext [2]
>>>>>
>>>>> * Dependencies resolution is unaffected so 'bundle install' cannot be
>>>>> used to install development dependencies, it would ignore rpm-gems 
>>>>> and
>>>>> install gems from gem-repo.
>>>>> * If bundler_ext is used for setup development environment, control
>>>>> over which versions are required is lost (requiring is not done by
>>>>> bundler)
>>>>> * It affects only how requirements are loaded [1], it's useful only
>>>>> for deployment.
>>>>> * (We could use it in deployment to get rid of bundler, but that is
>>>>> out of this discussion's scope see Goal.2)
>>>>>
>>>>> == bundle install --without default
>>>>>
>>>>> This installs only development gems, but:
>>>>>
>>>>> * If some of the development gems has a production gem as dependency
>>>>> (eg. ActiveSupport), the gem is reinstalled by bundler from a 
>>>>> gem-repo
>>>>> over the older rpm-gem.
>>>>> * If only development gems are installed, bundler cannot be used to
>>>>> require gems. It will raise LoadError because from bundler's point of
>>>>> view production gems are not installed.
>>>>> * It would have to be combined with bundler_ext to find production
>>>>> rpm-gems, so you would loose control over which version are loaded
>>>>> again
>>>>> * If you forget to add '--without default' when you run 'bundle
>>>>> install' unwanted versions are installed.
>>>>>
>>>>> == vendor/cache
>>>>>
>>>>> Move all gems to src/vendor/cache (directly into katello/master or as
>>>>> a git sub-module) and remove sources form Gemfile completely. 'bundle
>>>>> install' installs gems form vendor/cache automatically if there 
>>>>> are no
>>>>> sources in Gemfile.
>>>>>
>>>>> * to have .gem files directly in katello/master is ugly
>>>>> * submodules are real pain
>>>> no need to use sub-modules
>>>>> * git is not that good with binary data
>>>>>
>>>> gems are mostly ruby, very not a lot of binary data.
>>>
>>> gems would be stored in vendor/cache as .gem files, one file per gem
>>> which are actually tar.gz (binary). That is what I meant sorry about
>>> the confusion.
>>>
>>>>> = My proposal
>>>>>
>>>>> == Bundler patch
>>>>>
>>>>> * it would be loaded in Gemfile, so every tool using bundler will
>>>>> get it
>>>> I don't think a single Gemfile is going to solve the issue: we'll need
>>>> to relax version constraints in Gemfile, and add a bunch of 
>>>> ruby-version
>>>> specific gems...
>>>>>
>>>>> * it would ensure that if rpm-gem is present it's used even if there
>>>>> is newer version available in gem-repo
>>>>> * ensure that locally installed gems are not confused with rpm-gems
>>>> You didn't explain how you are planning to differentiate between 
>>>> the two
>>>> - system-wide bundler-/ruby-gem-installed gems and rpm-installed 
>>>> gems...
>>>
>>> I don't think this is a problem, see [1] how I did it.
>>
>> This is not a good approach - you introduced platform-specific code into
>> main Katello code base.
>>
>>>>> * patch is optional, can be turned on/off by ENV variable or by 
>>>>> options
>>>>> * (I am also considering to add optional check that all production
>>>>> gems are installed as rpms)
>>>>>
>>>>> Deployment is unaffected, all dependencies are satisfied by rpm-gems.
>>>>>
>>>>> When you run bundle install to get development environment on any
>>>>> rpm-based machine, all your rpms have priority and they are used. All
>>>>> missing development gems are installed from our gem-repo.
>>>> I think this is the crux of the issue. Once we have a custom gem
>>>> repository, the above step has very little practical value. A ruby gem
>>>> is a ruby gem no matter what place in the system it's installed in.
>>>
>>> IMO This has a big practical value. When you are fixing a bug you want
>>> to be sure that you have same rpm-gems installed and required as in
>>> production where the bug was discovered.
>>
>> Do you understand how ruby loads gem dependencies? Can you explain the
>> difference between rpm-installed gem, and bundler-installed gem (for
>> system-wide installs)?
>>
>
> Yeah, I understand, We just don't understand each other :) How is this 
> relevant to what I said before? I would like to give you an answer but 
> I don't see the connection, please explain where are you heading with 
> this.

Ultimately, one way, or another, we are solving the problem of loading 
gem dependencies. I'd like you to demonstrate the difference between 
loading of a regular ruby gem and rpm-installed ruby gem by ruby 
interpreter.

>
>> -d
>>
>>>
>>>
>>>>>
>>>>> On non-rpm-based machine, 'bundle install' installs all gems
>>>>> production and development from our gem-repo.
>>>>>
>>>>> == Gem Repository
>>>>>
>>>>> Bundler patch will work only if all development and production gems
>>>>> are in the gem-repo. To get it fixed quickly 'bundle package' can be
>>>>> used to get all the gems. Missing gems can be then uploaded to
>>>>> katello-thirdparty (source for our gem-repo)
>>>>>
>>>>> = Summary
>>>>>
>>>>> I think that Bundler patch and updating gem-repo to contain all gems
>>>>> is the only one which can be done fast and will work on any 
>>>>> machine in
>>>>> deployment and development. I tried to be thorough but if I missed
>>>>> something or if you have better ideas please let me know.
>>>> The issue with dependency version variability needs solving. The patch
>>>> itself has little practical value, bundle install from custom 
>>>> repository
>>>> is sufficient. The irony is that you are trying to solve the problem
>>>> Bundler has successfully solved awhile ago. I'm baffled as to why we
>>>> continue to insist on solving the problems the "katello" way.
>>>>
>>>> -d
>>>>
>>>>>
>>>>> Petr
>>>>>
>>>>> [1]
>>>>> https://github.com/aeolusproject/conductor/blob/master/src/config/application.rb#L21 
>>>>>
>>>>>
>>>>>
>>>>> [2] https://github.com/aeolus-incubator/bundler_ext
>>>>>
>>>>> _______________________________________________
>>>>> katello-devel mailing list
>>>>> katello-devel at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>>
>>>>
>>>> _______________________________________________
>>>> katello-devel mailing list
>>>> katello-devel at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/katello-devel
>>>
>>> [1] https://github.com/Katello/katello/pull/526
>>
>>