ERROR: Unable to read status (-110) with Marvell 88e6190x on x86

[salekseev]

This is more of fishing for help. I have a device made by Silicom as a uCPE router for ATT (ATT-V150) that is Intel C3000 (x86) based and includes a Marvell 88E6190X switch. Attaching system architecture diagram:

Appliance and switch work with commercial vyatta software which inits the switch in userspace, something like:

# cat /run/dataplane/platform.conf
[Dataplane]
backplane_port1=0:5:0.0
backplane_port2=0:5:0.1
fal_plugin = /usr/lib/libfal-mvl.so.1
[Marvell1]
onie_bus = /dev/i2c-1
onie_location=54
onie_size=400
backplane_port1=0:5:0.0,10
backplane_port2=0:5:0.1,9
mdio_pci=0:5:0.0
sw_dev_addr=0,16
sw_dev_num=1
sw_port=0,0,0,b,bmc0
sw_port=1,0,4,p,p4
sw_port=2,0,3,p,p3
sw_port=3,0,1,s,p1
sw_port=4,0,2,s,p2
sw_port=5,0,6,p,p6
sw_port=6,0,5,p,p5
sw_port=7,0,8,p,p8
sw_port=8,0,7,p,p7
poe_mask=0xc0
[hardware-features]
security.vpn.ipsec=1
[hardware-interface-features]
firewall.in.switch_vif=1
firewall.local.switch_vif=1
firewall.out.switch_vif=1
hardware-switching=1
[all-interface-features]
poe=p7,p8
[portmonitor]
hw_rx_sess_count=1
hw_tx_sess_count=1
hw_sess_count=2
hw_rx_src_count=8
hw_tx_src_count=8

Jul 29 19:46:10 vyatta dataplane[2585]: DATAPLANE: Dataplane version 3.7.86.61 - Copyright (c) 2021, AT&T Intellectual Property. All rights reserved.
Jul 29 19:46:10 vyatta dataplane[2585]: FAL: Initializing plugin: /usr/lib/libfal-mvl.so.1
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: fal_plugin_init_log:1259
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Parsing platform config file /run/dataplane/platform.conf
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Bkplane pci(0:5:0.0)->mvl port 10
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Bkplane pci(0:5:0.1)->mvl port 9
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: MDIO pci(0:5:0.0)->mvl
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Set (10/1)as backplane
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Set (9/2)as backplane
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 1 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 2 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 3 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 4 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 5 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 6 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 7 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 8 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 9 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 10 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_sem_create Semaphore id 11 created
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Global 1 Control2 updated for swid 0, 7c0 -> 7c0
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Marvell SDK Loaded
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: 2 backplane ports are setup
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: mvl_setup_bkplane_ports
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Forced link down for bkplane port 9
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Backplane tuning is 0x1724 for port 9
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Backplane speed is 2.5G for port 9
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Forced link down for bkplane port 10
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Backplane tuning is 0x1724 for port 10
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Backplane speed is 2.5G for port 10
Jul 29 19:46:10 vyatta dataplane[2585]: FAL_MVL: Onie eeprom init
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: get_bkplane_dpdk_portid Bkplane port 9 set to auto negotiate
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: setup_mgmt_bkplane Set bkplane 9 as CPU port and recv TO_CPU traffic
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port for net_sw_portsw0p7
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing sw_port_vdev for net_sw_portsw0p7
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port net_sw_portsw0p7
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Swport created mvl_port:8, dpdk_port:3, bkplane:2
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Port 8 disabled and .1Q secure mode set
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: get_bkplane_dpdk_portid Bkplane port 9 set to auto negotiate
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port for net_sw_portsw0p8
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing sw_port_vdev for net_sw_portsw0p8
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port net_sw_portsw0p8
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Swport created mvl_port:7, dpdk_port:4, bkplane:2
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Port 7 disabled and .1Q secure mode set
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: get_bkplane_dpdk_portid Bkplane port 9 set to auto negotiate
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port for net_sw_portsw0p5
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing sw_port_vdev for net_sw_portsw0p5
Jul 29 19:46:11 vyatta dataplane[2585]: SW_PORT: Initializing switch port net_sw_portsw0p5
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Swport created mvl_port:6, dpdk_port:5, bkplane:2
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: Port 6 disabled and .1Q secure mode set
Jul 29 19:46:11 vyatta dataplane[2585]: FAL_MVL: get_bkplane_dpdk_portid Bkplane port 9 set to auto negotiate

But I've been struggling to get it discovered with Linux DSA and MV88E6XXX driver. As well as getting mdio-tools to talk to it via MDIO bus:

# mdio
fixed-0
ixgbe-mdio-0000:05:00.0

# mdio ixgbe*
ERROR: Unable to read status (-110)

# mdio ixgbe* mvls

Any advice would be greatly appreciated.

[wkz]

mdio ixgbe* will try to probe every C22 address on the bus and list the results. This assumes that the underlying bus controller will return 0xffff for missing devices. Some controllers will instead detect the absence of a device, and will return an error instead. This might be what you're running into.

Try to address a single register and see if that works better for you. E.g:

# mdio ixgbe* mvls <SWADDR> raw 0:3
0x0a00 (hopefully)

[sl0wfi]

I have the same problem. I have tried guessing at addresses but everything comes back 0xffff.

Can anyone suggest the addresses to use for Marvell LinkStreet switch chips? Do these need to be initializes with iic or GPIO before they will communicate?

[wkz]

1. Check the hardware datasheet (not functional spec) for the pins marked ADDR[N]n. These are usually shared with the LED matrix signals.
2. Check the strapping of these signals on your board's schematic to figure out the address.

There's always the option of just trying every address, which will be in the inclusive range 0..31

You do not need any initialization via any other interface, but it is not uncommon that the chip's RESETn input is connected to a GPIO under the CPU's control. Check your schematic and see if you need to "manually" release the chip out of reset.

[jasonhensler]

Forgive me if this have been mentioned already, but the switch address is mentioned as 0,16 in the config file above. Guessing this is not hex, on my ia3000, I'm trying ./mdio/mdio ixgbe* mvls 0x10 raw 0:3
and getting 0x0a01 back.

[wkz]

Congrats! That's the expected value. The least significant nibble holds the chip revision, which in your case is 1, hence the difference from my example, which assumed revision 0.

[sl0wfi]

@jasonhensler Do you know any software or capability to configure the ports on the 88e6190X in the ia3000? (Sorry if this is getting off topic from MDIO-tools specifically.)

[salekseev]

@sl0wfi I suspect normal Linux Distributed Switch could be used if Device Tree is provided somehow like https://gist.github.com/ZoranStojsavljevic/423b96e2ca3bd581f7ce417cb410c465. Problem is x86 normally does not have Device Trees but one could be injected by the boot loader like grub (https://www.gnu.org/software/grub/manual/grub/html_node/devicetree.html#devicetree) if flattened DTB (https://devicetree-specification.readthedocs.io/en/stable/flattened-format.html) could be created. There are some limited examples on x86 like https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/platform/ce4100/falconfalls.dts, it'll probably need to be compiled to a binary format afterwards with something like https://github.com/dgibson/dtc.

I have not tested any of that btw.

[jasonhensler]

@salekseev I suspect that we will face issues with the x550/ixgbe driver not implementing mdio natively or would that not be required if we had proper DTS file?

Unfortunately for us, att vyatta seems to be using dpdk for the switch setup, skipping an /dev/mdio device completely. However the Firebox M270 m270 and netgate xg-7100 have used Intel phys to a 88e6190x so it should be possible. Any pointer on this would be very welcomed.

[salekseev]

@wkz could you give some guidance on how to find meaningful registers for 88e6190x now that we figured out its address on the platform like: mdio ixgbe* mvls 0x10 raw 0:3? My biggest challenge is that in default configuration (without drivers loaded) only 1 switchport is active out of 8 external, and internally I think only one of the KR CPU ports.

Writing to some of those registers should allow us to set ports configuration even without DSA support loaded, right? Also, mvls utility only works with DSA?

# for i in $(seq 0 15); do mdio ixgbe* mvls 0x10 raw 0:$i; done
0x0e07
0xc000
0x0000
0x0a01
0x007f
0x0000
0x07fe
0x0001
0x2c80
0x0001
0x8000
0x0001
0x0000
0x0000
0x0000
0x9100

[wkz]

You really need the functional specification for your device, which I have, but can't share with you because the world is broken.

Some say there are ways of finding datasheets for older devices online, like the "marvell 6097 datasheet", using popular search engines. That might be worth a shot 🤷‍♂️

Armed with that, and the knowledge that while the port registers used to (e.g. on 6097) start at offset 0x10, they start at offset 0x0 in the 6190, you should be able to make sense of most of the dump above. Focus on registers 0, 1, and 4.

The U-Boot driver might be a good source of info. It is much more to the point than the Linux driver, since it only does the most essential setup.

Re: mvls: Yes, it operates on data supplied by the mv88e6xxx (DSA) driver via devlink memory regions. Most of the code for parsing the port registers should be pretty easy to glue together with data from mdio though.

[salekseev]

@wkz I have a copy of the data sheet, including Link Street® 88E6390X/88E6390/88E6290/88E6190X/88E6190 Switch Functional Specification (it's ridiculous that it's being distributed under NDA) but I'm struggling to translate pins etc to registers access. Sorry, I'm no hardware engineer. Do you think you could give me an example based on either 6097 (offset 0x10) or 6190 (offset 0x0) to configure a switch port. To enable it and to set port vlan to 1 for example? I appreciate your help and babysitting btw.

[wkz]

For a builtin copper port, you probably have to:

• Enable the port (set the two LSBs in register 4)
• Clear any power-down state on the corresponding PHY (via registers 24/25 in Global2)

For (R)GMII or SERDES interfaces, you might also have to:

• Initialize the SERDES in the correct mode
• Apply a fixed speed configuration to the port in register 1

Again, the U-Boot sources are probably the most distilled code you'll find which does this.

If you want to setup VLANs, you'll need to load and entry into the VTU, update the port's PVID register. At this point I would probably spend the time on figuring out how to supply a DTB to your kernel so that you can make use of the mv88e6xxx driver instead. Then you can just attach your ports to a VLAN filtering bridge and then the driver will configure your switch to match the bridge's setup.

[erikarn]

well, i have mdio working on freebsd ixgbe, and i'm debugging this switch code right now. :-) I may have some questions!

[erikarn]

ok so first question - the internal PHYs. Do you know what they are?

The PHYs I'm seeing are:

# etherswitchcfg phy 2.2
        reg 2.0x02=0x0141
# etherswitchcfg phy 2.3
        reg 2.0x03=0x0c00
# 

which isn't (yet) handled as a specific marvell phy, but that ID looks like it's an old gige phy used by the older intel gige NICs.
Do you know what PHYs are used inside this chip and what their device OUIs are? I just want to know that I'm seeing the right one.

[erikarn]

heh you were right, all but one PHY had the poewr down bit set in CR (reg 0) in each phy. Oops. :-)