[rhos-list] Quantum: Packet Fragmentation Issue?

Wed Jun 26 08:55:47 UTC 2013

Dear Rhys,

Thanks for the quick turnaround. Yes, we're using be2net for eth4. We'll give enable-vlan-splinters=true a try.

Regards,

Balazs Fulop   
Morgan Stanley | Enterprise Infrastructure   
Lechner Odon fasor 8 | Floor 07   
Budapest, 1095   
Phone: +36 1 881-3941   
Balazs.Fulop at morganstanley.com   

Be carbon conscious. Please consider our environment before printing this email.    

-----Original Message-----
From: Rhys Oxenham [mailto:roxenham at redhat.com] 
Sent: Wednesday, June 26, 2013 10:21 AM
To: Fulop, Balazs (Enterprise Infrastructure)
Cc: Thomas Graf; Chris Wright; Szombath, Lajos (Enterprise Infrastructure); rhos-list at redhat.com
Subject: Re: [rhos-list] Quantum: Packet Fragmentation Issue?

Hi Balazs,

On 26 Jun 2013, at 08:30, "Fulop, Balazs" <Balazs.Fulop at morganstanley.com> wrote:

> Dear Thomas,
> 
>>> Any chance you could capture the traffic on a switch between the hosts
>>> to see if the packet is corrupted on the sending or receiving side?
> 
> Given I don't maintain the switch this will be tricky but I'll try.
> 
>>> benet is currently not approved for use with OVS in RDO
> 
> I'm not sure I follow. Can you please elaborate?

be2net is the driver that supports the Emulex OneConnect 10G card. It is on the approved list for use with Open vSwitch and RDO/RHOS. 

Can you confirm you're using this driver?

> 
>>> As suggested earlier, use of VLAN splinters could help to work around
>>> this issue.
> 
> What are VLAN splinters? Could you please give us some documentation pointers? The following returns 404 not found:
> http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=FAQ;hb=HEADcan
> 

This link works if you delete the 'can' at the end. 

Cheers
Rhys. 

> Regards,
> 
> Balazs Fulop   
> Morgan Stanley | Enterprise Infrastructure   
> Lechner Odon fasor 8 | Floor 07   
> Budapest, 1095   
> Phone: +36 1 881-3941   
> Balazs.Fulop at morganstanley.com   
> 
> 
> Be carbon conscious. Please consider our environment before printing this email.    
> 
> 
> -----Original Message-----
> From: Thomas Graf [mailto:tgraf at redhat.com] 
> Sent: Wednesday, June 26, 2013 1:28 AM
> To: Fulop, Balazs (Enterprise Infrastructure)
> Cc: gkotton at redhat.com; Johnray Fuller; rhos-list at redhat.com; Chris Wright; Robert Kukura; jmh at redhat.com
> Subject: Re: [rhos-list] Quantum: Packet Fragmentation Issue?
> 
> On 06/25/2013 09:42 PM, Fulop, Balazs wrote:
>> Dear All,
>> 
>> Thank you for looking into this. I tried option a) but unfortunately the packets still appear to get scrambled (tcpdump effectively shows the same and ssh will never connect). The network card used on both hosts in this demo cluster is:
>> Emulex Corporation OneConnect 10Gb NIC
>> 
>> If you have any further ideas on a possible resolution / workaround, please kindly let me know.
> 
> Looking at your packet capture below it seems like a VLAN tag has been
> inserted but has been corrupted [0x3edd 0x3c4e].
> 
> Any chance you could capture the traffic on a switch between the hosts
> to see if the packet is corrupted on the sending or receiving side?
> 
> benet is currently not approved for use with OVS in RDO, you can find
> the latest list here:
> https://access.redhat.com/site/articles/289823
> 
> As suggested earlier, use of VLAN splinters could help to work around
> this issue.
> 
> Best,
> Thomas
> 
>> 
>> Regards,
>> 
>> Balazs Fulop
>> Morgan Stanley | Enterprise Infrastructure
>> Lechner Odon fasor 8 | Floor 07
>> Budapest, 1095
>> Phone: +36 1 881-3941
>> Balazs.Fulop at morganstanley.com
>> 
>> 
>> Be carbon conscious. Please consider our environment before printing this email.
>> 
>> 
>> -----Original Message-----
>> From: Thomas Graf [mailto:tgraf at redhat.com]
>> Sent: Tuesday, June 25, 2013 9:18 PM
>> To: gkotton at redhat.com
>> Cc: Johnray Fuller; rhos-list at redhat.com; Fulop, Balazs (Enterprise Infrastructure); Chris Wright; Robert Kukura; jmh at redhat.com
>> Subject: Re: [rhos-list] Quantum: Packet Fragmentation Issue?
>> 
>> On 06/25/2013 09:07 PM, Gary Kotton wrote:
>>> On 06/25/2013 08:54 PM, Johnray Fuller wrote:
>>>> Hello,
>>>> 
>>>> 
>>>> I appear to have an issue with packet fragmentation
>>>> 
>>>> When we try to ssh from one VM to another where the VMs run on
>>>> different hosts on the source host the physical link (eth4) shows:
>>>> 
>>>> 13:24:19.520812 fa:16:3e:dd:3c:4e (oui Unknown) > fa:16:3e:6c:eb:80
>>>> (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 2, p 0,
>>>> ethertype IPv4, (tos 0x0, ttl 64, id 31721, offset 0, flags [DF],
>>>> proto TCP (6), length 60)
>>>> 10.0.0.11.60025 > 10.0.0.12.ssh: Flags [S], cksum 0xdb91 (correct),
>>>> seq 3087184310, win 14600, options [mss 1460,sackOK,TS val 11061855
>>>> ecr 0,nop,wscale 6], length 0
>>>> 
>>>> While on the receiving end we see:
>>>> 
>>>> 13:24:20.555105 3e:6c:eb:80:fa:16 (oui Unknown) > 3c:4e:fa:16:fa:16
>>>> (oui Unknown), ethertype Unknown (0x3edd), length 78:
>>>> 0x0000:  3c4e 0800 4500 003c 7be9 4000 4006 aabc <N..E..<{. at .@...
>>>> 0x0010:  0a00 000b 0a00 000c ea79 0016 b802 b1b6  .........y......
>>>> 0x0020:  0000 0000 a002 3908 db91 0000 0204 05b4  ......9.........
>>>> 0x0030:  0402 080a 00a8 ca5f 0000 0000 0103 0306  ......._........
>>> 
>>> This looks like it could be related to VLAN splinters when using
>>> openvswitch. Are you using Openvswitch? Maybe
>>> http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=FAQ;hb=HEADcan
>>> help.
>> 
>> This is not related to VLAN splinters but VLANs usage in general.
>> 
>> We have seen this before and it usually is caused by an intermediate
>> device on the host having the same MTU as the interface inside the VM.
>> Typically both have 1500. The VM outputs 1500 sized frames, OVS adds
>> a VLAN header and that exceeds the MTU of any device on the host.
>> 
>> Fix is to either
>> 
>> a) decrease the MTU inside the VM by 4 if VLAN tagged packets are to
>>     go out to a physical ethernet
>> 
>> b) increase MTU of all intermediate interfaces on the host by at least
>>     4 to avoid fragmentation.
>> 
>> c) increase MTU of all soft devices on the host and enable jumbo frames
>>     on the physical ethernet device.
>> 
>> I would choose b) for Neutron if tunneling is being used. If external
>> VLANs are in play option c) is nice with a fallback to frags if jumbo
>> frames are unsupported.
>> 
>> 
>> 
>>> 
>>>> It seems that encapsulation causes the packet to break. Does anyone
>>>> have any ideas on how to troubleshoot this?
>>>> 
>>>> These VMs are on different hosts.
>>>> 
>>>> We tried increasing the mtu on both hosts' eth4, but still no joy.
>>>> 
>>>> We found the following, https://review.openstack.org/#/c/31518/ ,
>>>> which might be related, but this patch was abandoned.
>>>> 
>>>> Any assistance would be  greatly appreciated.
>>>> 
>>>> J
>> 
>> 
>> 
>> --------------------------------------------------------------------------------
>> 
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
> 
> 
> 
> --------------------------------------------------------------------------------
> 
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
> 
> _______________________________________________
> rhos-list mailing list
> rhos-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rhos-list

--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.