Support and Troubleshooting from the Ubuntu Server Guide by OnWorks

5.9. Support and Troubleshooting

DPDK is a fast evolving project. In any case of a search for support and further guides it is highly recommended to first check if they apply to the current version.

• DPDK Mailing Lists38

• For OpenVswitch-DPDK OpenStack Mailing Lists39

• Known issues in DPDK Launchpad Area40

• Join the IRC channels #DPDK or #openvswitch on freenode.

Issues are often due to missing small details in the general setup. Later on, these missing details cause problems which can be hard to track down to their root cause. A common case seems to be the "could not open network device dpdk0 (No such device)" issue. This occurs rather late when setting up a port in Open vSwitch with DPDK. But the root cause most of the time is very early in the setup and initialization. Here an example how a proper initialization of a device looks - this can be found in the syslog/journal when starting Open vSwitch with DPDK enabled.

ovs-ctl[3560]: EAL: PCI device 0000:04:00.1 on NUMA socket 0 ovs-ctl[3560]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd ovs-ctl[3560]: EAL: PCI memory mapped at 0x7f2140000000 ovs-ctl[3560]: EAL: PCI memory mapped at 0x7f2140200000

38 http://dpdk.org/ml

39 http://openvswitch.org/mlists

40 https://bugs.launchpad.net/ubuntu/+source/dpdk

If this is missing, either by ignored cards, failed initialization or other reasons, later on there will be no DPDK device to refer to. Unfortunately the logging is spread across syslog/journal and the openvswitch log. To allow some cross checking here an example what can be found in these logs, relative to the entered command.

#Note: This log was taken with dpdk 2.2 and openvswitch 2.5 Captions:

CMD: that you enter

SYSLOG: (Inlcuding EAL and OVS Messages) OVS-LOG: (Openvswitch messages)

#PREPARATION

Bind an interface to DPDK UIO drivers, make Hugepages available, enable DPDK on OVS

CMD: sudo service openvswitch-switch restart

SYSLOG:

2016-01-22T08:58:31.372Z|00003|daemon_unix(monitor)|INFO|pid 3329 died, killed (Terminated), exiting

2016-01-22T08:58:33.377Z|00002|vlog|INFO|opened log file /var/log/openvswitch/ovs- vswitchd.log

connecting...

OVS-LOG:

systemd[1]: Stopping Open vSwitch... systemd[1]: Stopped Open vSwitch.

systemd[1]: Stopping Open vSwitch Internal Unit... ovs-ctl[3541]: * Killing ovs-vswitchd (3329)

ovs-ctl[3541]: * Killing ovsdb-server (3318) systemd[1]: Stopped Open vSwitch Internal Unit. systemd[1]: Starting Open vSwitch Internal Unit... ovs-ctl[3560]: * Starting ovsdb-server

ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1

ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs- version=2.5.0 "external-ids:system-id=\"e7c5ba80-bb14-45c1-b8eb-628f3ad03903\"" "system- type=\"Ubuntu\"" "system-version=\"16.04-xenial\""

ovs-ctl[3560]: * Configuring Open vSwitch system IDs

ovs-ctl[3560]: 2016-01-22T08:58:31Z|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/openvswitch

ovs-vswitchd: ovs|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/ openvswitch

ovs-ctl[3560]: EAL: Detected lcore 0 as core 0 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 1 as core 1 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 2 as core 2 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 3 as core 3 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 4 as core 4 on socket 0

ovs-ctl[3560]: EAL: Detected lcore 5 as core 5 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 6 as core 0 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 7 as core 1 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 8 as core 2 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 9 as core 3 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 10 as core 4 on socket 0 ovs-ctl[3560]: EAL: Detected lcore 11 as core 5 on socket 0

ovs-ctl[3560]: EAL: Support maximum 128 logical core(s) by configuration. ovs-ctl[3560]: EAL: Detected 12 lcore(s)

ovs-ctl[3560]: EAL: VFIO modules not all loaded, skip VFIO support... ovs-ctl[3560]: EAL: Setting up physically contiguous memory...

ovs-ctl[3560]: EAL: Ask a virtual area of 0x100000000 bytes

ovs-ctl[3560]: EAL: Virtual area found at 0x7f2040000000 (size = 0x100000000) ovs-ctl[3560]: EAL: Requesting 4 pages of size 1024MB from socket 0

ovs-ctl[3560]: EAL: TSC frequency is ~2397202 KHz

ovs-vswitchd[3592]: EAL: TSC frequency is ~2397202 KHz

ovs-vswitchd[3592]: EAL: Master lcore 0 is ready (tid=fc6cbb00;cpuset=[0]) ovs-vswitchd[3592]: EAL: PCI device 0000:04:00.0 on NUMA socket 0

ovs-vswitchd[3592]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd

ovs-vswitchd[3592]: EAL: Not managed by a supported kernel driver, skipped ovs-vswitchd[3592]: EAL: PCI device 0000:04:00.1 on NUMA socket 0

ovs-vswitchd[3592]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd ovs-vswitchd[3592]: EAL: PCI memory mapped at 0x7f2140000000 ovs-vswitchd[3592]: EAL: PCI memory mapped at 0x7f2140200000

ovs-ctl[3560]: EAL: Master lcore 0 is ready (tid=fc6cbb00;cpuset=[0]) ovs-ctl[3560]: EAL: PCI device 0000:04:00.0 on NUMA socket 0

ovs-ctl[3560]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd

ovs-ctl[3560]: EAL: Not managed by a supported kernel driver, skipped ovs-ctl[3560]: EAL: PCI device 0000:04:00.1 on NUMA socket 0

ovs-ctl[3560]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd ovs-ctl[3560]: EAL: PCI memory mapped at 0x7f2140000000 ovs-ctl[3560]: EAL: PCI memory mapped at 0x7f2140200000

ovs-vswitchd[3592]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3

ovs-vswitchd[3592]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528 ovs-ctl[3560]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3

ovs-ctl[3560]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528 ovs-ctl[3560]: Zone 0: name:<RG_MP_log_history>, phys:0x83fffdec0, len:0x2080,

virt:0x7f213fffdec0, socket_id:0, flags:0

ovs-ctl[3560]: Zone 1: name:<MP_log_history>, phys:0x83fd73d40, len:0x28a0c0, virt:0x7f213fd73d40, socket_id:0, flags:0

ovs-ctl[3560]: Zone 2: name:<rte_eth_dev_data>, phys:0x83fd43380, len:0x2f700, virt:0x7f213fd43380, socket_id:0, flags:0

ovs-ctl[3560]: * Starting ovs-vswitchd

ovs-ctl[3560]: * Enabling remote OVSDB managers systemd[1]: Started Open vSwitch Internal Unit. systemd[1]: Starting Open vSwitch... systemd[1]: Started Open vSwitch.

CMD: sudo ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev

SYSLOG:

2016-01-22T08:58:56.344Z|00008|memory|INFO|37256 kB peak resident set size after 24.5 seconds

2016-01-22T08:58:56.346Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation

2016-01-22T08:58:56.346Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3

2016-01-22T08:58:56.346Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids

2016-01-22T08:58:56.346Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state

2016-01-22T08:58:56.346Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone

2016-01-22T08:58:56.346Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark

2016-01-22T08:58:56.346Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label

2016-01-22T08:58:56.360Z|00016|bridge|INFO|bridge ovsdpdkbr0: added interface ovsdpdkbr0 on port 65534

2016-01-22T08:58:56.361Z|00017|bridge|INFO|bridge ovsdpdkbr0: using datapath ID 00005a4a1ed0a14d

2016-01-22T08:58:56.361Z|00018|connmgr|INFO|ovsdpdkbr0: added service controller "punix:/ var/run/openvswitch/ovsdpdkbr0.mgmt"

OVS-LOG:

ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev

systemd-udevd[3607]: Could not generate persistent MAC address for ovs-netdev: No such file or directory

kernel: [50165.886554] device ovs-netdev entered promiscuous mode kernel: [50165.901261] device ovsdpdkbr0 entered promiscuous mode

CMD: sudo ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk

SYSLOG:

2016-01-22T08:59:06.369Z|00019|memory|INFO|peak resident set size grew 155% in last 10.0 seconds, from 37256 kB to 95008 kB

2016-01-22T08:59:31.520Z|00022|dpdk|INFO|Port 0: 8c:dc:d4:b3:6d:e9

2016-01-22T08:59:31.522Z|00024|bridge|INFO|bridge ovsdpdkbr0: added interface dpdk0 on port 1

2016-01-22T08:59:31.522Z|00025|bridge|INFO|bridge ovsdpdkbr0: using datapath ID 00008cdcd4b36de9

2016-01-22T08:59:31.523Z|00002|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'

OVS-LOG:

ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk

ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a79ebc0 hw_ring=0x7f211a7a6c00 dma_addr=0x81a7a6c00