[Rdo-list] First Mitaka test day summary

John Trowbridge trown at redhat.com
Thu Jan 28 15:46:59 UTC 2016



On 01/28/2016 10:43 AM, Sasha Chuzhoy wrote:
> With nested virtualization we don't need "--libvirt-type qemu".
> With that said, we noticed controller failures (one controller is down) on that setup.
> Thanks.
> 

Hmm, is nested virtualization documented anywhere? It is not how we do
CI, and in my experience it does not work well on RHEL/CentOS. I have
heard that improvements have been made on the latest Fedoras, but I have
never had much success with nested virt.

I think for virtual setups in particular, it would be good to mirror
exactly what is used in CI. This is the benefit of virtual after all.

> Best regards,
> Sasha Chuzhoy.
> 
> ----- Original Message -----
>> From: "John Trowbridge" <trown at redhat.com>
>> To: "Udi Kalifon" <ukalifon at redhat.com>
>> Cc: "rdo-list" <rdo-list at redhat.com>
>> Sent: Thursday, January 28, 2016 10:15:42 AM
>> Subject: Re: [Rdo-list] First Mitaka test day summary
>>
>>
>>
>> On 01/28/2016 10:08 AM, Udi Kalifon wrote:
>>> The deploy command was:
>>> openstack overcloud deploy --templates --control-scale 3
>>> --compute-scale 1 --ntp-server clock.redhat.com -e
>>> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
>>>
>>
>> For virtual deployments, it is needed to pass also '--libvirt-type qemu'.
>>
>> It is probably worth trying a redeploy by deleting the overcloud and
>> adding that argument.
>>
>>> To try and investigate this subject further, I enabled nested
>>> virtualization on the host, and changed the libvirt type to qemu.
>>> Unfortunately that also required turning off all the VMs on teh host
>>> for the kvm module to be reloaded - and now I have problems with
>>> keystone that doesn't want to come back up. I mailed the list about
>>> this problem in a separate thread.
>>>
>>> Thanks.
>>> Udi.
>>>
>>> On Thu, Jan 28, 2016 at 2:01 PM, John Trowbridge <trown at redhat.com> wrote:
>>>>
>>>>
>>>> On 01/28/2016 04:35 AM, Udi Kalifon wrote:
>>>>> I can't launch instances in Mitaka on a virtual environment. I made
>>>>> sure that nested virtualization is enabled on the host. If I do nova
>>>>> show on the failed instance I see this:
>>>>>
>>>>>  {"message": "No valid host was found. There are not enough hosts
>>>>> available.", "code": 500, "details": "  File
>>>>> \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line
>>>>> 372, in build_instances
>>>>>      context, request_spec, filter_properties)
>>>>>    File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\",
>>>>> line 416, in _schedule_instances
>>>>>      hosts = self.scheduler_client.select_destinations(context,
>>>>> spec_obj)
>>>>>    File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\",
>>>>> line 372, in wrapped
>>>>>      return func(*args, **kwargs)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\",
>>>>> line 51, in select_destinations
>>>>>      return self.queryclient.select_destinations(context, spec_obj)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\",
>>>>> line 37, in __run_method
>>>>>      return getattr(self.instance, __name)(*args, **kwargs)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\",
>>>>> line 32, in select_destinations
>>>>>      return self.scheduler_rpcapi.select_destinations(context, spec_obj)
>>>>>    File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\",
>>>>> line 121, in select_destinations
>>>>>      return cctxt.call(ctxt, 'select_destinations', **msg_args)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\",
>>>>> line 158, in call
>>>>>      retry=self.retry)
>>>>>    File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\",
>>>>> line 90, in _send
>>>>>      timeout=timeout, retry=retry)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\",
>>>>> line 466, in send
>>>>>      retry=retry)
>>>>>    File
>>>>>    \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\",
>>>>> line 457, in _send
>>>>>      raise result
>>>>>  ", "created": "2016-01-28T09:20:47Z"}
>>>>>
>>>>>
>>>>> On the controller, I see a lot of db-related errors in
>>>>> nova-scheduler.log:
>>>>>
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db [-]
>>>>> Unexpected error while reporting service status
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> Traceback (most recent call last):
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py",
>>>>> line 88, in _report_state
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> service.service_ref.save()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 221, in wrapper
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn(self, *args, **kwargs)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282,
>>>>> in save
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._check_minimum_version()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258,
>>>>> in _check_minimum_version
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> minver = self.get_minimum_version(self._context, self.binary)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 179, in wrapper
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> result = fn(cls, context, *args, **kwargs)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330,
>>>>> in get_minimum_version
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in
>>>>> service_get_minimum_version
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line
>>>>> 468, in service_get_minimum_version
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> filter(models.Service.forced_down == false()).\
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2503, in scalar
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = self.one()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2472, in one
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = list(self)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2515, in __iter__
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self._execute_and_instances(context)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2528, in _execute_and_instances
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> close_with_result=True)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2519, in _connection_from_session
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db     **kw)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 882, in connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> execution_options=execution_options)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 887, in _connection_for_bind
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> engine, execution_options)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 334, in _connection_for_bind
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> conn = bind.contextual_connect()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2036, in contextual_connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> **kwargs)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 92, in __init__
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.dispatch.engine_connect(self, self.__branch)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line
>>>>> 258, in __call__
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> fn(*args, **kw)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line
>>>>> 80, in _connect_ping_listener
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> connection.scalar(select([1]))
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 844, in scalar
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self.execute(object, *multiparams, **params).scalar()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 914, in execute
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return meth(self, multiparams, params)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line
>>>>> 323, in _execute_on_connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return connection._execute_clauseelement(self, multiparams, params)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 1010, in _execute_clauseelement
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> compiled_sql, distilled_params
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 1078, in _execute_context
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db     None,
>>>>> None)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 1335, in _handle_dbapi_exception
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> util.raise_from_cause(newraise, exc_info)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line
>>>>> 199, in raise_from_cause
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> reraise(type(exception), exception, tb=exc_tb)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 1071, in _execute_context
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> conn = self._revalidate_connection()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 393, in _revalidate_connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.__connection = self.engine.raw_connection(_connection=self)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2099, in raw_connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.pool.unique_connection, _connection)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2075, in _wrap_pool_connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> util.reraise(*sys.exc_info())
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2069, in _wrap_pool_connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in
>>>>> unique_connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return _ConnectionFairy._checkout(self)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 708, in
>>>>> _checkout
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> fairy = _ConnectionRecord.checkout(pool)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in
>>>>> checkout
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> rec.checkin()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
>>>>> line 60, in __exit__
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> compat.reraise(exc_type, exc_value, exc_tb)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in
>>>>> checkout
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> dbapi_connection = rec.get_connection()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 590, in
>>>>> get_connection
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.connection = self.__connect()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in
>>>>> __connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> connection = self.__pool._invoke_creator(self)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
>>>>> line 97, in connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return dialect.connect(*cargs, **cparams)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py",
>>>>> line 377, in connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self.dbapi.connect(*cargs, **cparams)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in
>>>>> Connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> return Connection(*args, **kwargs)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657,
>>>>> in __init__
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.connect()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 850,
>>>>> in connect
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._get_server_information()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1061,
>>>>> in _get_server_information
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> packet = self._read_packet()
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 895,
>>>>> in _read_packet
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> packet_header = self._read_bytes(4)
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 922,
>>>>> in _read_bytes
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> 2013, "Lost connection to MySQL server during query")
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost
>>>>> connection to MySQL server during query') [SQL: u'SELECT 1']
>>>>> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db [-]
>>>>> Unexpected error while reporting service status
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> Traceback (most recent call last):
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py",
>>>>> line 88, in _report_state
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> service.service_ref.save()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 221, in wrapper
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn(self, *args, **kwargs)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282,
>>>>> in save
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._check_minimum_version()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258,
>>>>> in _check_minimum_version
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> minver = self.get_minimum_version(self._context, self.binary)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 179, in wrapper
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> result = fn(cls, context, *args, **kwargs)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330,
>>>>> in get_minimum_version
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in
>>>>> service_get_minimum_version
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line
>>>>> 468, in service_get_minimum_version
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> filter(models.Service.forced_down == false()).\
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2503, in scalar
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = self.one()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2472, in one
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = list(self)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2515, in __iter__
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self._execute_and_instances(context)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2528, in _execute_and_instances
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> close_with_result=True)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2519, in _connection_from_session
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db     **kw)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 882, in connection
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> execution_options=execution_options)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 887, in _connection_for_bind
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> engine, execution_options)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 334, in _connection_for_bind
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> conn = bind.contextual_connect()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2034, in contextual_connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._wrap_pool_connect(self.pool.connect, None),
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2073, in _wrap_pool_connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db     e,
>>>>> dialect, self)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 1399, in _handle_dbapi_exception_noconnection
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> util.raise_from_cause(newraise, exc_info)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line
>>>>> 199, in raise_from_cause
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> reraise(type(exception), exception, tb=exc_tb)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2069, in _wrap_pool_connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in
>>>>> connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return _ConnectionFairy._checkout(self)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 708, in
>>>>> _checkout
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> fairy = _ConnectionRecord.checkout(pool)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in
>>>>> checkout
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> rec.checkin()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
>>>>> line 60, in __exit__
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> compat.reraise(exc_type, exc_value, exc_tb)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in
>>>>> checkout
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> dbapi_connection = rec.get_connection()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in
>>>>> get_connection
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.connection = self.__connect()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in
>>>>> __connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> connection = self.__pool._invoke_creator(self)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
>>>>> line 97, in connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return dialect.connect(*cargs, **cparams)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py",
>>>>> line 377, in connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self.dbapi.connect(*cargs, **cparams)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in
>>>>> Connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> return Connection(*args, **kwargs)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657,
>>>>> in __init__
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> self.connect()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 850,
>>>>> in connect
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._get_server_information()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1061,
>>>>> in _get_server_information
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> packet = self._read_packet()
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 895,
>>>>> in _read_packet
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> packet_header = self._read_bytes(4)
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 922,
>>>>> in _read_bytes
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> 2013, "Lost connection to MySQL server during query")
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost
>>>>> connection to MySQL server during query')
>>>>> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db [-]
>>>>> Unexpected error while reporting service status
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> Traceback (most recent call last):
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py",
>>>>> line 88, in _report_state
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> service.service_ref.save()
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 221, in wrapper
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn(self, *args, **kwargs)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282,
>>>>> in save
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._check_minimum_version()
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258,
>>>>> in _check_minimum_version
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> minver = self.get_minimum_version(self._context, self.binary)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
>>>>> 179, in wrapper
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> result = fn(cls, context, *args, **kwargs)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330,
>>>>> in get_minimum_version
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in
>>>>> service_get_minimum_version
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> use_slave=use_slave)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line
>>>>> 468, in service_get_minimum_version
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> filter(models.Service.forced_down == false()).\
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2503, in scalar
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = self.one()
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2472, in one
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> ret = list(self)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2515, in __iter__
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> return self._execute_and_instances(context)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2528, in _execute_and_instances
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> close_with_result=True)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
>>>>> 2519, in _connection_from_session
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db     **kw)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 882, in connection
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> execution_options=execution_options)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 887, in _connection_for_bind
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> engine, execution_options)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
>>>>> 334, in _connection_for_bind
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> conn = bind.contextual_connect()
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2034, in contextual_connect
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> self._wrap_pool_connect(self.pool.connect, None),
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>>>>> 2069, in _wrap_pool_connect
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> return fn()
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in
>>>>> connect
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> return _ConnectionFairy._checkout(self)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 729, in
>>>>> _checkout
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> fairy)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line
>>>>> 258, in __call__
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> fn(*args, **kw)
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db   File
>>>>> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line
>>>>> 352, in checkout
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db     if
>>>>> connection_record.info['pid'] != pid:
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db KeyError:
>>>>> 'pid'
>>>>> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool [-]
>>>>> Exception during reset or similar
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool Traceback
>>>>> (most recent call last):
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 631, in
>>>>> _finalize_fairy
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> fairy._reset(pool)
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 765, in
>>>>> _reset
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> pool._dialect.do_rollback(self)
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
>>>>> line 2519, in do_rollback
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> dbapi_connection.rollback()
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 723,
>>>>> in rollback
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 956,
>>>>> in _execute_command
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool     raise
>>>>> err.InterfaceError("(0, '')")
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> InterfaceError: (0, '')
>>>>> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool [-]
>>>>> Exception closing connection <pymysql.connections.Connection object at
>>>>> 0x510f550>
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool Traceback
>>>>> (most recent call last):
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 290, in
>>>>> _close_connection
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> self._dialect.do_close(connection)
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py",
>>>>> line 418, in do_close
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> dbapi_connection.close()
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool   File
>>>>> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 662,
>>>>> in close
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool     raise
>>>>> err.Error("Already closed")
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool Error:
>>>>> Already closed
>>>>> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool
>>>>> 2016-01-28 08:25:27.009 1121 INFO nova.servicegroup.drivers.db [-]
>>>>> Recovered from being unable to report status.
>>>>> 2016-01-28 08:27:21.165 31979 WARNING
>>>>> oslo_reports.guru_meditation_report [-] Guru mediation now registers
>>>>> SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1
>>>>> will no longer be registered in a future release, so please use
>>>>> SIGUSR2 to generate reports.
>>>>> 2016-01-28 08:27:21.544 31979 WARNING oslo_config.cfg
>>>>> [req-d3d0b357-c23b-4e8b-b322-32f5b225dc19 - - - - -] Option
>>>>> "use_local" from group "conductor" is deprecated for removal.  Its
>>>>> value may be silently ignored in the future.
>>>>> 2016-01-28 08:27:21.606 31979 INFO oslo_service.periodic_task
>>>>> [req-d3d0b357-c23b-4e8b-b322-32f5b225dc19 - - - - -] Skipping periodic
>>>>> task _periodic_update_dns because its interval is negative
>>>>> 2016-01-28 08:27:21.635 31979 INFO nova.service [-] Starting scheduler
>>>>> node (version 13.0.0-dev80.el7.centos)
>>>>>
>>>>> Was anyone successful in launching instances in Mitaka? I installed
>>>>> with the director, it's an HA deployment without network isolation in
>>>>> a virtualized environment.
>>>>>
>>>>
>>>> CI is passing on that configuration, and I have had success with it
>>>> manually. Could you post your deploy command?
>>>>
>>>>> Thanks a lot,
>>>>> Udi.
>>>>>
>>>>> On Wed, Jan 27, 2016 at 11:17 PM, John Trowbridge <trown at redhat.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 01/27/2016 01:25 PM, Udi Kalifon wrote:
>>>>>>> Hello.
>>>>>>>
>>>>>>> The good news is that I succeeded to deploy :). I haven't yet tried to
>>>>>>> test the overcloud for any sanity, but in past test days I was never
>>>>>>> able to report any success - so maybe it's a sign that things are
>>>>>>> stabilizing.
>>>>>>
>>>>>> That's awesome!
>>>>>>
>>>>>>>
>>>>>>> I deployed with rdo-manager on a virtual setup according to the
>>>>>>> instructions in https://www.rdoproject.org/rdo-manager/. I wasn't able
>>>>>>> to deploy with network isolation, because I assume that my templates
>>>>>>> from 7.x require changes, but I haven't seen any documentation on
>>>>>>> what's changed. If you can point me in the right direction to get
>>>>>>> network isolation working for this environment I will test it
>>>>>>> tomorrow.
>>>>>>>
>>>>>>> Some of the problems I hit today:
>>>>>>>
>>>>>>> 1) The link to the quickstart guide from the testday page
>>>>>>> https://www.rdoproject.org/testday/mitaka/milestone2/ points to a very
>>>>>>> old github page. The correct link should be the one I already
>>>>>>> mentioned: https://www.rdoproject.org/rdo-manager/
>>>>>>
>>>>>> I fixed that link, thanks!
>>>>>>
>>>>>>>
>>>>>>> 2) The prerequisites to installing ansible are not documented. On a
>>>>>>> fresh CentOS 7 I had to install python-virtualenv, git, and gcc. I
>>>>>>> then ran "easy_install pip" and "pip install
>>>>>>> git+https://github.com/ansible/ansible.git@v2.0.0-0.6.rc1#egg=ansible"
>>>>>>> to be able to run the playbook which installs the virtual environment.
>>>>>>
>>>>>> The quickstart.sh will do all of that for you, but if you wanted to
>>>>>> submit a patch for more detailed instructions for manually setting up
>>>>>> the virtualenv, that would great. For tripleo-quickstart, gerrit is
>>>>>> setup and follows the same gerrit workflow as everything else.
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Udi.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Rdo-list mailing list
>>>>>>> Rdo-list at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>>
>>>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rdo-list mailing list
>>>>>> Rdo-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>
>>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>




More information about the rdo-list mailing list