[Pulp-dev] Database support in Pulp 3

[Brian Bouterse]

I believe we have reached a point where Pulp (core and its plugins) can no
longer support MariaDB due to technical problems. I've been an advocate for
Pulp to support MariaDB because it's what our users want. The community
survey has 16 respondents (IIRC) and 30% of them said they wanted to use
MariaDB. Also, anecdotally at conference booths, users want choice in their
database. To not give them that, we need good, technical reasons. I didn't
feel we had enough of them before, but now I feel we're at a point where we
aren't able to solve these issues; it's becoming a choice of giving users
features they want versus db portability.

Here are the main reasons I think about (mostly what others have also
stated):

* utf8 issues - 3-byte utf-8 issues   <---- is this fixable with just a
schema change?
* full text search - pulp_ansible recently implemented full text search. To
my knowledge it's not possible with MySQL. full text search is amazing, and
it's driving pulp_ansible to drop postgreSQL. I bet all plugins will want
this. With MariaDB, there is no search.
* mixed plugin support concerns - With some plugins dropping support for
MariaDB, the pulp community will fracture based on what DB you choose to
use because not all plugins can run in all places.
* specific field support - plugin writers have expressed the desire to
store json data when their plugins natively contain json data and perhaps
you want to filter on it for example. Having mariaDB support prevents us
from using the JSONField.
* performance concerns - The performance testing showed that Pulp does run
significantly slower

If we drop MariaDB we should publish a blog post and drop it with RC4. To
remove MariaDB testing from Travis I propose we remove it from the
plugin_template and use the Travis CI tool from @dkliban to push that
config out to all repositories.

I'll be offline this week. I wanted to get this reply out there in the hope
that you all can make and enact the final decision.

Thanks,
Brian


On Thu, Jul 11, 2019 at 4:02 PM Daniel Alley <dalley at redhat.com> wrote:

> One more note:  Not all MySQL / MariaDB installations support
> transactions, which we use heavily (and rely on?)
>
>
> https://docs.djangoproject.com/en/2.2/topics/db/transactions/#transactions-in-mysql
>
> On Thu, Jul 11, 2019 at 3:55 PM David Davis <daviddavis at redhat.com> wrote:
>
>> Two plugins are currently running into issues trying to support
>> mariadb/mysql. The pulp_ansible plugin is interested in adding full text
>> search and JSONFields. Meanwhile, the pulp_python plugin is trying to store
>> emojis in text which mariadb/mysql doesn't handle well since it uses 3-byte
>> utf-8 by default[0]. Given such cases, I wonder if we'd be better served by
>> dropping mariadb/mysql support and going with Postgresql only. Gitlab
>> recently came to a similar conclusion[1].
>>
>> I personally am hesitant to give up being database agnostic but we
>> already don't support sqlite. Also, I see some advantages like utilizing
>> several Postgresql-only features like extra field types, full text search,
>> etc. Also, supporting multiple databases means we'll likely have to write
>> db specific code in some places or have plugins that only work with certain
>> database types.
>>
>> Thoughts?
>>
>> [0]
>> https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
>>      https://code.djangoproject.com/ticket/18392
>> [1] https://about.gitlab.com/2019/06/27/removing-mysql-support/
>>
>> David
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190714/c43a7352/attachment.htm>