[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [rdo-list] RDO TripleO Mitaka Non-HA Overcloud Failing



Hello,

On Wed, 2016-08-03 at 19:40 +0100, Gunjan, Milind [CTO] wrote:
> Hi All,
>
> I am currently working on Tripleo Mitaka Openstack deployment on
> baremetal servers:
> Undercloud – 1 baremetal server with 2 NIC (1 for provisioning and
> 2nd for external network connectivity)
> Controller – 1 baremetal server ( 6 NICs with each openstack VLANs on
> separate NIC)
> Compute – 1 baremetal server
>
> I  followed Graeme's instructions here : https://www.redhat.com/archi
> ves/rdo-list/2016-June/msg00049.html to set up Undercloud .
> Undercloud deployment was successful and all the images required for
> overcloud deployment was properly built as per the instruction. I
> would like to mention that I used libvirt tools to modify the root
> password on overcloud-full.qcow2 and we also modified the grub file
> to include “net.ifnames=0 biosdevname=0” to restore old interface
> naming.

I don't think there is a problem doing this but the new naming
convention does allow you to specifically target nics during deployment
with reliability.

> I was able to successfully introspect 2 serves to be used for
> controller and compute nodes. Also , we added the serial device
> discovered during introspection as root device:
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/root_device='{"serial":
> "618e728372833010c79bead9066f0f9e"}'
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/root_device='{"serial":
> "618e7283728347101f2107b511603adc"}'

Sure, we use wwn values. I hacked the following which is a bit nasty
but hey, it works. There are probably easier ways to do it but...

https://github.com/cbrown2/openstack-scripts/blob/master/root_device_co
nfig.sh

> Next, we added compute and control tag to respective introspected
> node with local boot option:
>
> ironic node-update 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 add
> properties/capabilities='profile:control,boot_option:local'
> ironic node-update afcfbee3-3108-48da-a6da-aba8f422642c add
> properties/capabilities='profile:compute,boot_option:local'

I would just add this parameter to the instackenv.json file - one less
thing to have to run.

>
> We used multiple NIC templates for control and compute node which has
> been attached along with network-environment.yaml file. Default
> network isolation template file has been used.
>
>
> Deployment script looks like this :
> #!/bin/bash
> DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
> template_base_dir="$DIR"
> ntpserver=<sprint.ntp.server.ip> #Sprint LAB
> openstack overcloud deploy --templates \
> -e /usr/share/openstack-tripleo-heat-templates/environments/network-
> isolation.yaml \
> -e ${template_base_dir}/environments/network-environment.yaml \
> --control-flavor control --compute-flavor compute  \
> --control-scale 1 --compute-scale 1 \
> --ntp-server $ntpserver \
> --neutron-network-type vxlan --neutron-tunnel-types vxlan --debug

I'm not sure why you set ntp and templates variable but meh. I'd really
be inclined to drop network isolation and see if it still deploys.

> Heat stack deployment goes on more really long time (more than 4
> hours) and  gets stuck at postdeployment configurations. Please find
> below the capture during install :

You can change the deploy timeout with:

--timeout 120

for 2 hours for example.

>
> Every 2.0s: ironic node-list && nova list && heat stack-list && heat
> resource-list -n5 overcloud | grep -vi complete
>                   Wed Aug  3 17:33:37 2016
>
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> | UUID                                 | Name | Instance
> UUID                        | Power State | Provisioning State |
> Maintenance |
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> | 604f7dfc-38af-4fe0-8986-4c8ac5f956e2 | None | 9e7aae15-cabc-4489-
> a1b2-778915a78df2 | power on    | active             | False       |
> | afcfbee3-3108-48da-a6da-aba8f422642c | None | c1ab52a9-461a-4a11-
> a13e-e57ff0a3ae2a | power on    | active             | False       |
> +--------------------------------------+------+--------------------
> ------------------+-------------+--------------------+-------------+
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> | ID                                   | Name                    |
> Status | Task State | Power State | Networks               |
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> | 9e7aae15-cabc-4489-a1b2-778915a78df2 | overcloud-controller-0  |
> ACTIVE | -          | Running     | ctlplane=192.168.149.9 |
> | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a | overcloud-novacompute-0 |
> ACTIVE | -          | Running     | ctlplane=192.168.149.8 |
> +--------------------------------------+-------------------------+---
> -----+------------+-------------+------------------------+
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> | id                                   | stack_name | stack_status  |
> creation_time       | updated_time |
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> | 26ee0150-4cfa-4268-9107-8bfbf6712913 | overcloud  | CREATE_FAILED |
> 2016-08-03T08:11:34 | None         |
> +--------------------------------------+------------+---------------
> +---------------------+--------------+
> +---------------------------------------------+--------------------
> ---------------------------+-----------------------------------------
> -------------------------------
> ---------+--------------------+---------------------+--------------
> -------------------------------------------------------------------
> ------------------------------+
> | resource_name                               |
> physical_resource_id                          | resource_type
>          | resource_status    | updated_time        |
> stack_name
>                                          |
> +---------------------------------------------+--------------------
> ---------------------------+-----------------------------------------
> -------------------------------
> ---------+--------------------+---------------------+--------------
> -------------------------------------------------------------------
> ------------------------------+
> | ComputeNodesPostDeployment                  | 3797aec6-e543-4dda-
> 9cd1-c7261e827a64          | OS::TripleO::ComputePostDeployment
>          | CREATE_FAILED      | 2016-08-03T08:11:35 |
> overcloud
>                                          |
> | ControllerNodesPostDeployment               | 6ad9f88c-5c55-4125-
> 97f1-eb0e33329d16          | OS::TripleO::ControllerPostDeployment
>          | CREATE_FAILED      | 2016-08-03T08:11:35 |
> overcloud
>                                          |
> | ComputePuppetDeployment                     | 8b199f85-e4f9-48ad-
> 9aee-b1cdf4900b9f          | OS::Heat::StructuredDeployments
>          | CREATE_FAILED      | 2016-08-03T08:29:19 | overcloud-
> ComputeNodesPostDeployment-
> 6vxfu2g2qucy
>     |
> | ControllerOvercloudServicesDeployment_Step4 | 15509f59-ff28-43af-
> 95dd-6247a6a32c2d          | OS::Heat::StructuredDeployments
>          | CREATE_FAILED      | 2016-08-03T08:29:20 | overcloud-
> ControllerNodesPostDeployment-35y7uafngfwj
>                                     |
> | 0                                           | 7cd0aa3d-742f-4e78-
> 99ca-b2a575913f8e          | OS::Heat::StructuredDeployment
>          | CREATE_IN_PROGRESS | 2016-08-03T08:30:04 | overcloud-
> ComputeNodesPostDeployment-6vxfu2g2qucy-ComputePuppetDeployment-
> cpahcct3tfw3                        |
> | 0                                           | 5e9308f7-c3a9-4a94-
> a017-e1acb694c036          | OS::Heat::StructuredDeployment
>
>
> [stack mitaka-uc ~]$ openstack software deployment show 5e9308f7-
> c3a9-4a94-a017-e1acb694c036
> +---------------+--------------------------------------+
> | Field         | Value                                |
> +---------------+--------------------------------------+
> | id            | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
> | server_id     | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
> | config_id     | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
> | creation_time | 2016-08-03T08:32:10                  |
> | updated_time  |                                      |
> | status        | IN_PROGRESS                          |
> | status_reason | Deploy data available                |
> | input_values  | {}                                   |
> | action        | CREATE                               |
> +---------------+--------------------------------------+
>
> [stack mitaka-uc ~]$ openstack software deployment show --long
> 5e9308f7-c3a9-4a94-a017-e1acb694c036
> +---------------+--------------------------------------+
> | Field         | Value                                |
> +---------------+--------------------------------------+
> | id            | 5e9308f7-c3a9-4a94-a017-e1acb694c036 |
> | server_id     | 9e7aae15-cabc-4489-a1b2-778915a78df2 |
> | config_id     | 86d49e66-2f25-4cb1-b623-5ae87b01bb64 |
> | creation_time | 2016-08-03T08:32:10                  |
> | updated_time  |                                      |
> | status        | IN_PROGRESS                          |
> | status_reason | Deploy data available                |
> | input_values  | {}                                   |
> | action        | CREATE                               |
> | output_values | None                                 |
> +---------------+--------------------------------------+
>
> [stack mitaka-uc ~]$ openstack stack resource list 3797aec6-e543-
> 4dda-9cd1-c7261e827a64
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
> | resource_name           | physical_resource_id                 |
> resource_type                                   | resource_status |
> updated_time        |
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
> | ComputeArtifactsConfig  | a33cd04d-61ab-4429-8565-182409c2b97f |
> file:///usr/share/openstack-tripleo-heat-       | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> |                         |                                      |
> templates/puppet/deploy-artifacts.yaml          |
> |                     |
> | ComputePuppetConfig     | 5bb712b0-5358-46c7-a444-f9adedfedd50 |
> OS::Heat::SoftwareConfig                        | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> | ComputePuppetDeployment | 8b199f85-e4f9-48ad-9aee-b1cdf4900b9f |
> OS::Heat::StructuredDeployments                 | CREATE_FAILED   |
> 2016-08-03T08:29:19 |
> | ComputeArtifactsDeploy  | 1d13bf34-fc66-4bf1-a3b7-1dd815f58f5a |
> OS::Heat::StructuredDeployments                 | CREATE_COMPLETE |
> 2016-08-03T08:29:19 |
> | ExtraConfig             |                                      |
> OS::TripleO::NodeExtraConfigPost                | INIT_COMPLETE   |
> 2016-08-03T08:29:19 |
> +-------------------------+--------------------------------------+---
> ----------------------------------------------+-----------------+--
> -------------------+
>
> [stack mitaka-uc ~]$ openstack stack resource list 8b199f85-e4f9-
> 48ad-9aee-b1cdf4900b9f
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> | resource_name | physical_resource_id                 |
> resource_type                  | resource_status    |
> updated_time        |
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> | 0             | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
> OS::Heat::StructuredDeployment | CREATE_IN_PROGRESS | 2016-08-
> 03T08:30:04 |
> +---------------+--------------------------------------+-------------
> -------------------+--------------------+---------------------+
> [stack mitaka-uc ~]$ openstack software deployment show 7cd0aa3d-
> 742f-4e78-99ca-b2a575913f8e
> +---------------+--------------------------------------+
> | Field         | Value                                |
> +---------------+--------------------------------------+
> | id            | 7cd0aa3d-742f-4e78-99ca-b2a575913f8e |
> | server_id     | c1ab52a9-461a-4a11-a13e-e57ff0a3ae2a |
> | config_id     | 24e5c0db-f84f-4a94-8f8e-8e38e73ccc86 |
> | creation_time | 2016-08-03T08:30:05                  |
> | updated_time  |                                      |
> | status        | IN_PROGRESS                          |
> | status_reason | Deploy data available                |
> | input_values  | {}                                   |
> | action        | CREATE                               |
> +---------------+--------------------------------------+
>
<snip>

I'd be inclined to ssh into the nodes themselves and check the logs.
Also watch:

http://www.anstack.com/blog/2016/07/22/tripleo-deep-dive-session-3.html

for further assistance with debugging from the experts. It just looks
like post deploy puppet config has gone wrong so I don't think you are
far off.

heat --help

will give the various options for drilling down to debug the failure on
the various levels.

>
> Please let me know if there is any other logs which I can provide
> that can help in troubleshooting.
>
>
> Thanks a lot in Advance for your help and support.
>
> Best Regards,
> Milind Gunjan
>
>
>
> This e-mail may contain Sprint proprietary information intended for
> the sole use of the recipient(s). Any use by others is prohibited. If
> you are not the intended recipient, please contact the sender and
> delete all copies of the message.
--
Regards,

Christopher Brown
OpenStack Engineer
OCF plc

Tel: +44 (0)114 257 2200
Web: www.ocf.co.uk
Blog: blog.ocf.co.uk
Twitter: @ocfplc


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]