[Pulp-list] pulp 3.3.1 server error

Wed May 13 16:40:11 UTC 2020

Could you create a clone of your production system and try upgrading the
cloned environment? I suspect that some problems may have been created from
the extra migration being created in between all the upgrade attempts on
your test system.

Could you provide the SQL schema (without data) from the database of the
system that is experiencing the problem? We want to compare to a clean
installed pulp 3.3.1 schema.

Could you provide the SQL schema (without data) from the database of the
3.2 production system?

On Wed, May 13, 2020 at 11:52 AM Bin Li (BLOOMBERG/ 120 PARK) <
bli111 at bloomberg.net> wrote:

> Here is what I have
> # django-admin showmigrations
> admin
> [X] 0001_initial
> [X] 0002_logentry_remove_auto_add
> [X] 0003_logentry_add_action_flag_choices
> auth
> [X] 0001_initial
> [X] 0002_alter_permission_name_max_length
> [X] 0003_alter_user_email_max_length
> [X] 0004_alter_user_username_opts
> [X] 0005_alter_user_last_login_null
> [X] 0006_require_contenttypes_0002
> [X] 0007_alter_validators_add_error_messages
> [X] 0008_alter_user_username_max_length
> [X] 0009_alter_user_last_name_max_length
> [X] 0010_alter_group_name_max_length
> [X] 0011_update_proxy_permissions
> contenttypes
> [X] 0001_initial
> [X] 0002_remove_content_type_name
> core
> [X] 0001_initial
> [X] 0002_increase_artifact_size_field
> [X] 0003_remove_upload_completed
> [X] 0004_add_duplicated_reserved_resources
> [X] 0005_progressreport_code
> [X] 0006_repository_plugin_managed
> [X] 0007_delete_progress_proxies
> [X] 0008_published_metadata_as_content
> [X] 0009_remove_task_non_fatal_errors
> [X] 0010_pulp_fields
> [X] 0011_relative_path
> [X] 0012_auto_20191104_2000
> [X] 0013_repository_pulp_type
> [X] 0014_remove_repository_plugin_managed
> [X] 0015_auto_20191112_1426
> [X] 0016_charfield_to_textfield
> [X] 0017_remove_task_parent
> [X] 0018_auto_20191127_2350
> [X] 0019_add_signing_service_model
> [X] 0020_change_publishedartifact_constraints
> [X] 0021_add_signing_service_foreign_key
> [X] 0022_rename_last_version
> [X] 0023_change_exporter_models
> [X] 0024_use_local_storage_for_uploads
> [X] 0025_task_parent_task
> [X] 0026_task_group
> [X] 0027_export_backend
> [X] 0028_import_importer_pulpimporter_pulpimporterrepository
> file
> [X] 0001_initial
> [X] 0002_file_related_names
> [X] 0003_auto_20191014_1721
> [X] 0004_filefilesystemexporter
> [X] 0005_filerepository
> [X] 0006_delete_filefilesystemexporter
> [X] 0007_filefilesystemexporter
> rpm
> [X] 0001_initial
> [X] 0002_updaterecord_reboot_suggested
> [X] 0003_DATA_incorrect_json
> [X] 0004_add_metadata_signing_service_fk
> [X] 0005_optimize_sync
> [X] 0006_opensuse_support
> [X] 0007_checksum_types
> [X] 0008_advisory_pkg_sumtype_as_int
> [X] 0009_revision_null
> sessions
> [X] 0001_initial
>
>
> From: ipanova at redhat.com At: 05/13/20 11:41:13
> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
> Cc: ttereshc at redhat.com, pulp-list at redhat.com
> Subject: Re: [Pulp-list] pulp 3.3.1 server error
>
> Can you confirm that migration 0009_revision_null was applied?
>
> <https://github.com/pulp/pulp_rpm/pull/1694/files#diff-baf8ad85df1ed2543d715f2a86de6cb4>
>
> You can see the migrations with this command 'django-admin showmigrations'
>
> --------
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, May 13, 2020 at 4:09 PM Bin Li (BLOOMBERG/ 120 PARK) <
> bli111 at bloomberg.net> wrote:
>
>> Hi Ina,
>>
>> The error was not from migration. I see the error when I try to create a
>> new repo. My pulp_installer is on the latest version. Rerun the installer
>> doesn't solve the issue. What should I do to fix this? We cannot reset
>> database from scratch since we already have a lot data.
>>
>> From: ipanova at redhat.com At: 05/13/20 09:43:45
>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>> Cc: ttereshc at redhat.com, pulp-list at redhat.com
>> Subject: Re: [Pulp-list] pulp 3.3.1 server error
>>
>> This issue was fixed in the 3.3.1 release [0][1]
>>
>> [0] https://pulp-rpm.readthedocs.io/en/latest/changes.html#bugfixes
>> [1] https://pulp.plan.io/issues/6662
>>
>>
>> --------
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Tue, May 12, 2020 at 11:30 PM Bin Li (BLOOMBERG/ 120 PARK) <
>> bli111 at bloomberg.net> wrote:
>>
>>> I removed the 0009_auto.. and rerun the installer. The upgrade was
>>> successful.
>>> Below is error log.
>>>
>>> May 12 17:25:05 myhost gunicorn[55426]: pulp: django.request:ERROR:
>>> Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/
>>> May 12 17:25:05 myhost gunicorn[55426]: Traceback (most recent call
>>> last):
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 84, in _execute
>>> May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql,
>>> params)
>>> May 12 17:25:05 myhost gunicorn[55426]:
>>> psycopg2.errors.NotNullViolation: null value in column
>>> "last_sync_revision_number" violates not-null constraint
>>> May 12 17:25:05 myhost gunicorn[55426]: DETAIL: Failing row contains
>>> (0053634d-d0ca-46de-81d4-b8acf19d592e, f, null, null, 0, null, {}).
>>> May 12 17:25:05 myhost gunicorn[55426]: The above exception was the
>>> direct cause of the following exception:
>>> May 12 17:25:05 myhost gunicorn[55426]: Traceback (most recent call
>>> last):
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/exception.py",
>>> line 34, in inner
>>> May 12 17:25:05 myhost gunicorn[55426]: response = get_response(request)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/base.py",
>>> line 115, in _get_response
>>> May 12 17:25:05 myhost gunicorn[55426]: response =
>>> self.process_exception_by_middleware(e, request)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/core/handlers/base.py",
>>> line 113, in _get_response
>>> May 12 17:25:05 myhost gunicorn[55426]: response =
>>> wrapped_callback(request, *callback_args, **callback_kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/views/decorators/csrf.py",
>>> line 54, in wrapped_view
>>> May 12 17:25:05 myhost gunicorn[55426]: return view_func(*args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/viewsets.py",
>>> line 114, in view
>>> May 12 17:25:05 myhost gunicorn[55426]: return self.dispatch(request,
>>> *args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py",
>>> line 505, in dispatch
>>> May 12 17:25:05 myhost gunicorn[55426]: response =
>>> self.handle_exception(exc)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py",
>>> line 465, in handle_exception
>>> May 12 17:25:05 myhost gunicorn[55426]:
>>> self.raise_uncaught_exception(exc)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py",
>>> line 476, in raise_uncaught_exception
>>> May 12 17:25:05 myhost gunicorn[55426]: raise exc
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/views.py",
>>> line 502, in dispatch
>>> May 12 17:25:05 myhost gunicorn[55426]: response = handler(request,
>>> *args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/mixins.py",
>>> line 19, in create
>>> May 12 17:25:05 myhost gunicorn[55426]: self.perform_create(serializer)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/mixins.py",
>>> line 24, in perform_create
>>> May 12 17:25:05 myhost gunicorn[55426]: serializer.save()
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/serializers.py",
>>> line 213, in save
>>> May 12 17:25:05 myhost gunicorn[55426]: self.instance =
>>> self.create(validated_data)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rest_framework/serializers.py",
>>> line 932, in create
>>> May 12 17:25:05 myhost gunicorn[55426]: instance =
>>> ModelClass._default_manager.create(**validated_data)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py",
>>> line 82, in manager_method
>>> May 12 17:25:05 myhost gunicorn[55426]: return
>>> getattr(self.get_queryset(), name)(*args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py",
>>> line 422, in create
>>> May 12 17:25:05 myhost gunicorn[55426]: obj.save(force_insert=True,
>>> using=self.db)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py",
>>> line 62, in save
>>> May 12 17:25:05 myhost gunicorn[55426]: super().save(*args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/base.py",
>>> line 107, in save
>>> May 12 17:25:05 myhost gunicorn[55426]: return super().save(*args,
>>> **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py",
>>> line 741, in save
>>> May 12 17:25:05 myhost gunicorn[55426]: force_update=force_update,
>>> update_fields=update_fields)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py",
>>> line 779, in save_base
>>> May 12 17:25:05 myhost gunicorn[55426]: force_update, using,
>>> update_fields,
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py",
>>> line 870, in _save_table
>>> May 12 17:25:05 myhost gunicorn[55426]: result =
>>> self._do_insert(cls._base_manager, using, fields, update_pk, raw)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/base.py",
>>> line 908, in _do_insert
>>> May 12 17:25:05 myhost gunicorn[55426]: using=using, raw=raw)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py",
>>> line 82, in manager_method
>>> May 12 17:25:05 myhost gunicorn[55426]: return
>>> getattr(self.get_queryset(), name)(*args, **kwargs)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py",
>>> line 1186, in _insert
>>> May 12 17:25:05 myhost gunicorn[55426]: return
>>> query.get_compiler(using=using).execute_sql(return_id)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/sql/compiler.py",
>>> line 1368, in execute_sql
>>> May 12 17:25:05 myhost gunicorn[55426]: cursor.execute(sql, params)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 67, in execute
>>> May 12 17:25:05 myhost gunicorn[55426]: return
>>> self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 76, in _execute_with_wrappers
>>> May 12 17:25:05 myhost gunicorn[55426]: return executor(sql, params,
>>> many, context)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 84, in _execute
>>> May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql,
>>> params)
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/utils.py",
>>> line 89, in __exit__
>>> May 12 17:25:05 myhost gunicorn[55426]: raise
>>> dj_exc_value.with_traceback(traceback) from exc_value
>>> May 12 17:25:05 myhost gunicorn[55426]: File
>>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/backends/utils.py",
>>> line 84, in _execute
>>> May 12 17:25:05 myhost gunicorn[55426]: return self.cursor.execute(sql,
>>> params)
>>> May 12 17:25:05 myhost gunicorn[55426]: django.db.utils.IntegrityError:
>>> null value in column "last_sync_revision_number" violates not-null
>>> constraint
>>> May 12 17:25:05 myhost gunicorn[55426]: DETAIL: Failing row contains
>>> (0053634d-d0ca-46de-81d4-b8acf19d592e, f, null, null, 0, null, {}).
>>> May 12 17:25:05 myhost gunicorn[55426]: 127.0.0.1 - admin
>>> [12/May/2020:21:25:05 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/
>>> HTTP/1.1" 500 27 "-" "HTTPie/0.9.4"
>>>
>>>
>>> From: ttereshc at redhat.com At: 05/12/20 16:19:31
>>> To: Bin Li (BLOOMBERG/ 120 PARK ) <bli111 at bloomberg.net>
>>> Cc: pulp-list at redhat.com
>>> Subject: Re: [Pulp-list] pulp 3.3.1 server error
>>>
>>> Was the upgrade successful after you removed the 0009_auto...?  Did you
>>> run the installer again?
>>>
>>> If no, please do so.
>>> If yes, could you share logs? There will be some traceback showing why
>>> 500 error happened.
>>>
>>> Thank you,
>>> Tanya
>>>
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200513/6c4b6149/attachment.htm>