[rhos-list] Fwd: Re: Lbass seems not to be functioning in RDO grizzly

Paul Robert Marino prmarino1 at gmail.com
Sun Nov 17 22:04:32 UTC 2013 

Well the good news is I looked at it with fresh eyes this morning and
I found the primary problem.

the issue was a little goofy it seams service_plugins was specified
twice in the config the first time with
quantum.plugins.services.agent_loadbalancer.plugin.LoadBalancerPlugin
and the second time latter in the configuration where it was blank. I
removed the second blank version of the line then restarted quantum
server and the lbaas service and its working now.

This seems to be what caused the problem, but it brings to light an
other one in my mind.
This kind of goes to a deeper issue though, I've noticed that whenever
there is a bad config option or just an odd state in the system like a
service being restarted and trying to execute commands it doesn't need
too OpenStack tends to blame the amqp code in the traces rather than
giving a more useful message. these messages arent particularly
helpful with diagnosing the real error and in some cases distract from
finding them.




On Sun, Nov 17, 2013 at 9:37 AM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
> sorry for the delayed response I have a rather odd work schedule
>
> The box I'm currently testing on is Scientific Linux 6.4
> the quantum version is 2013.1.4-3
>
> further version numbers for possibly relevant RPM's below
>
> "
>  rpm -qa|grep -Pi '(quantum|haproxy|kernel|qpid)'
> qpid-cpp-server-cluster-0.14-22.el6_3.x86_64
> qpid-tools-0.14-6.el6_3.noarch
> openstack-quantum-openvswitch-2013.1.4-3.el6.noarch
> kernel-2.6.32-358.118.1.openstack.el6.x86_64
> python-quantumclient-2.2.1-2.el6.noarch
> dracut-kernel-004-303.el6.noarch
> python-qpid-0.14-11.el6_3.noarch
> qpid-cpp-client-ssl-0.14-22.el6_3.x86_64
> qpid-qmf-0.14-14.el6_3.x86_64
> python-quantum-2013.1.4-3.el6.noarch
> haproxy-1.4.22-5.el6_4.x86_64
> kernel-2.6.32-358.114.1.openstack.el6.gre.2.x86_64
> qpid-cpp-server-ssl-0.14-22.el6_3.x86_64
> openstack-quantum-2013.1.4-3.el6.noarch
> kernel-firmware-2.6.32-358.123.2.openstack.el6.noarch
> kernel-2.6.32-358.123.2.openstack.el6.x86_64
> qpid-cpp-server-0.14-22.el6_3.x86_64
> python-qpid-qmf-0.14-14.el6_3.x86_64
> openstack-quantum-linuxbridge-2013.1.4-3.el6.noarch
> qpid-cpp-client-0.14-22.el6_3.x86_64
> "
>
> On Fri, Nov 15, 2013 at 2:50 PM, Ryan O'Hara <rohara at redhat.com> wrote:
>>
>> Please let me know what operating system you are using and the version
>> your openstack-quantum package. Thanks.
>>
>> Ryan
>>
>>> -------- Original Message --------
>>> Subject: Re: [rhos-list] Lbass seems not to be functioning in RDO grizzly
>>> Date: Thu, 14 Nov 2013 16:07:55 -0500
>>> From: Paul Robert Marino <prmarino1 at gmail.com>
>>> To: rhos-list at redhat.com <rhos-list at redhat.com>
>>>
>>> correction I have found something in the lbaas log
>>>
>>> every few seconds im seeing this which indicates an issue with qpid
>>> "
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.ops] SENT[2b0a710]:
>>> ConnectionHeartbeat()
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.raw] SENT[2b0a710]:
>>> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.raw] READ[2b0a710]:
>>> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.ops] RCVD[2b0a710]:
>>> ConnectionHeartbeat()
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.ops] SENT[2b0d6c8]:
>>> ConnectionHeartbeat()
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.raw] SENT[2b0d6c8]:
>>> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.raw] READ[2b0d6c8]:
>>> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
>>> 2013-11-14 15:59:45    DEBUG [qpid.messaging.io.ops] RCVD[2b0d6c8]:
>>> ConnectionHeartbeat()
>>> 2013-11-14 15:59:45    ERROR [quantum.openstack.common.rpc.amqp] Timed
>>> out waiting for RPC response.
>>> Traceback (most recent call last):
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
>>> line 495, in __iter__
>>>     data = self._dataqueue.get(timeout=self._timeout)
>>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298,
>>> in get
>>>     return waiter.wait()
>>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129,
>>> in wait
>>>     return get_hub().switch()
>>>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line
>>> 177, in switch
>>>     return self.greenlet.switch()
>>> Empty
>>> 2013-11-14 15:59:45    ERROR
>>> [quantum.plugins.services.agent_loadbalancer.agent.manager] Unable to
>>> retrieve ready devices
>>> Traceback (most recent call last):
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/plugins/services/agent_loadbalancer/agent/manager.py",
>>> line 168, in sync_state
>>>     ready_logical_devices = set(self.plugin_rpc.get_ready_devices())
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/plugins/services/agent_loadbalancer/agent/api.py",
>>> line 36, in get_ready_devices
>>>     topic=self.topic
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py",
>>> line 80, in call
>>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py",
>>> line 140, in call
>>>     return _get_impl().call(CONF, context, topic, msg, timeout)
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
>>> line 630, in call
>>>     rpc_amqp.get_connection_pool(conf, Connection))
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
>>> line 614, in call
>>>     rv = list(rv)
>>>   File
>>> "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
>>> line 500, in __iter__
>>>     raise rpc_common.Timeout()
>>> Timeout: Timeout while waiting on RPC response.
>>> 2013-11-14 15:59:45    DEBUG [quantum.openstack.common.periodic_task]
>>> Skipping LbaasAgentManager.collect_stats, 1 ticks left until next run
>>>
>>> "
>>> whats strange about it is QPID seems to be working fine for all the
>>> other quantum agents.
>>>
>>> here is my qpid configuration in the quantum.conf note any fields
>>> encpasulated int <> have been scrubbed
>>>
>>> "
>>> fake_rabbit = False
>>> rpc_backend = quantum.openstack.common.rpc.impl_qpid
>>> qpid_hostname = <qpid.host.name>
>>> qpid_port = 5672
>>> qpid_hosts = <qpid.host.name>:5672
>>> qpid_username = <quantumuser>
>>> qpid_password = <quantumpasswd>
>>> qpid_sasl_mechanisms = PLAIN
>>> qpid_reconnect = True
>>> qpid_reconnect_timeout = 30
>>> qpid_reconnect_limit = 0
>>> qpid_reconnect_interval_min = 0
>>> qpid_reconnect_interval_max = 0
>>> qpid_reconnect_interval = 20
>>> qpid_heartbeat = 30
>>> qpid_protocol = tcp
>>> qpid_tcp_nodelay = True
>>> default_publisher_id = <network.host.name>
>>> "
>>>
>>>
>>> On Thu, Nov 14, 2013 at 3:37 PM, Paul Robert Marino
>>> <prmarino1 at gmail.com> wrote:
>>> > hello every one
>>> > Im tinkering with RDO grizzly and trying to get the lbaas agent working.
>>> > im using openvswitch with gre tunnels which is working well.
>>> >
>>> > in /etc/quantum/lbaas_agent.ini I set the following
>>> >
>>> > interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
>>> > ovs_use_veth = True
>>> > device_driver =
>>> > quantum.plugins.services.agent_loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver
>>> > use_namespaces = True
>>> >
>>> > in /etc/quantum/quantum.conf i set in the [DEFAULT] section
>>> >
>>> > service_plugins =
>>> > quantum.plugins.services.agent_loadbalancer.plugin.LoadBalancerPlugin
>>> >
>>> >
>>> > I restarted  quantum server and started the lbass agent.
>>> >
>>> > haproxy is installed.
>>> >
>>> > when I run quantum lb-pool-list
>>> >
>>> > i get this error
>>> > "
>>> > 404 Not Found
>>> >
>> x> > The resource could not be found.
>>> >
>>> > "
>>> >
>>> > the weird thing is when I look at the logs I done see any errors of
>>> > any kind not even in debug logging, and there is no indication on the
>>> > secure log that a sudo involving haproxy was ever attempted.
>>> >
>>> > also i dont see it listed if i do a quantum  ext-list
>>> >
>>> > The only thing that I can see that may indicate the cause is durring
>>> > the startup of quantum server I see the message
>>> > "
>>> > WARNING [quantum.api.extensions] Extension lbaas not supported by any
>>> > of loaded plugins
>>> > "
>>> >
>>> > this said I havent seen any indication on how to fix it on any of the
>>> > openstack docs or any of the blog sites where people have configured
>>> > it in the past. according to every thing Ive read it should be
>>> > working.
>>> >
>>> > does any one have any ideas on what may be the problem?
>>>
>>> _______________________________________________
>>> rhos-list mailing list
>>> rhos-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rhos-list
>>>
>>>
>>
>> _______________________________________________
>> rhos-list mailing list
>> rhos-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rhos-list

More information about the rhos-list mailing list