[Pulp-list] Long upgrade times from 2.6 -> 2.8
Brian Bouterse
bbouters at redhat.com
Fri Jul 1 15:50:04 UTC 2016
With 2.8 specifically there is a *lot* of work done by the migrations
and the runtime should scale with the content size. In other words I
don't think it's doing useless work, there is just a lot of it.
I just did a quick audit of the two major offending migrations and they
are implemented sanely so I don't think there is a quick fix to improve
the runtime.
I believe to get a shorter runtime we would have to parallelize the
migrations.
Regarding things going wrong, we do assume that migrations should be
re-entrant meaning that if for some reason something strange occurs it
should always be safe to run pulp-manage-db and it will pick up where it
left off. Just a behavioral FYI.
These migration with 2.8 should not be norm, so I don't expect this to
be very common in the future until maybe a major release (3.0).
Given all ^, what do you think we should do?
-Brian
On 07/01/2016 11:23 AM, Eric Helms wrote:
> I think there are a couple of considerations.
>
> 1) The first issue is that a 6-18 hour upgrade window is not something
> users expect and we've not been warning them of such so they can plan an
> outage accordingly. Lengthy upgrades also have that tendency to make
> users feel something is wrong or increase the risk that something can go
> wrong in between.
> 2) The fundamental question of - is this a bug or does this make
> perfect sense and how it has to work?
> 3) Applying the upgrade on an existing 2.6 if it changed nothing of the
> environment could work, the tough part is having to distribute that
> backwards. Pulp would have to distribute it back to 2.6, and Katello
> would have to push out patches to our 2.4 release channel.
>
> Eric
>
> On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse <bbouters at redhat.com
> <mailto:bbouters at redhat.com>> wrote:
>
> I'm trying to understand if the pain point is related to downtime or
> total runtime.
>
> For instance, what if these migration could be run as a
> pre-migration step, ahead of time while Pulp was still online? The
> upgrade would still take just as long but you could use your (in
> this case) 2.6 install normally while the migrations are applying.
> Once they are done then the actual upgrade of the codebase could be
> very short.
>
> -Brian
>
> On 07/01/2016 09:20 AM, Eric Helms wrote:
>
>
>
> On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS)
> <AshbyJ at imsweb.com <mailto:AshbyJ at imsweb.com>
> <mailto:AshbyJ at imsweb.com <mailto:AshbyJ at imsweb.com>>> wrote:
>
> FWIW I just upgraded from 2.7 -> 2.8 and it was approx. 1-2 hr
> upgrade to get through the migrations in pulp-manage-db.____
>
> __ __
>
> 290 GB /var/lib/pulp____
>
> 16 GB MongoDB____
>
> __ __
>
> *From:*pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> <mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>>
> [mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>
> <mailto:pulp-list-bounces at redhat.com
> <mailto:pulp-list-bounces at redhat.com>>] *On Behalf Of *Michael
> Hrivnak
> *Sent:* Friday, July 01, 2016 8:31 AM
> *To:* Eric Helms <ehelms at redhat.com
> <mailto:ehelms at redhat.com> <mailto:ehelms at redhat.com
> <mailto:ehelms at redhat.com>>>
> *Cc:* pulp-list <pulp-list at redhat.com
> <mailto:pulp-list at redhat.com> <mailto:pulp-list at redhat.com
> <mailto:pulp-list at redhat.com>>>
> *Subject:* Re: [Pulp-list] Long upgrade times from 2.6 ->
> 2.8____
>
> __ __
>
> Did you get any feedback on whether one particular migration
> seemed
> to be running for a lot of that time?
>
>
> For the 1.5TB/100GB MongoDB scenario here is what I am able to glean
> from user logs (which I can share privately with anyone debugging):
>
> ~5 hours: Applying pulp_puppet.plugins.migrations version 4
> ~10 hours: Applying pulp_rpm.plugins.migrations version 28
>
> Use reports "lots of stating, unlinking, and linking of all the
> symlinks
> in /var/lib/pulb" if that helps.
>
> Another user reports ~6 hours on 176G of data.
>
> Eric
>
>
> ____
>
> __ __
>
> Michael____
>
> __ __
>
> On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms
> <ehelms at redhat.com <mailto:ehelms at redhat.com>
> <mailto:ehelms at redhat.com <mailto:ehelms at redhat.com>>>
> wrote:____
>
> Howdy,____
>
> __ __
>
> We (Katello) have had users reporting incredibly long
> upgrade
> times when upgrading from 2.6 to 2.8. This occurs during the
> pulp-manage-db step that is run as the beginning of our
> installers upgrade process. Based on the numbers below, does
> this make sense at all?____
>
> __ __
>
> Some numbers:____
>
> __ __
>
> 18 hour upgrade____
>
> 1.5 TB /var/lib/pulp____
>
> 100GB MongoDB____
>
> __ __
>
> __ __
>
> Thanks,____
>
> Eric____
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> <mailto:Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>>
> https://www.redhat.com/mailman/listinfo/pulp-list____
>
> __ __
>
>
>
> ------------------------------------------------------------------------
>
> Information in this e-mail may be confidential. It is
> intended only
> for the addressee(s) identified above. If you are not the
> addressee(s), or an employee or agent of the addressee(s),
> please
> note that any dissemination, distribution, or copying of this
> communication is strictly prohibited. If you have received this
> e-mail in error, please notify the sender of the error.
>
>
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
More information about the Pulp-list
mailing list