Glossary

DSA = distributed switch architecture (is this a Marvell original?)
TBI = ten bit interface, appears to be a chip management interface, perhaps an alternative to the MII family

MDIO is the standard for managing Ethernet PHY devices, using a simple two-wire interface consisting of bi-directional data and clock. Ethernet drivers use MDIO signals to read chip information and configure the device (MAC <-> PHY). But what's the difference between clause 22 and clause 45? This refers to the clause of the spec IEEE RFC802.3 and 22 was the original, and defines the format of the MDIO data frame. Clause 45 was added later to expand the capability of the interface. MDIO is one component of the MII set of interfaces. The MDC/MDIO pair is known as MIIM for MII Management. The interface has been compared to I2C, as it's multi-drop.

Media Independent Interface (MII) is the bus connecting MAC to PHY. MII includes a data path (with 4 lines each for TX/RX) and a management path (MDIO). RMII is the reduced (simplified) version, with RGMII being its Gigabit counterpart. MII is still preferred in a real-time network since RMII is a wrapper on MII and adds latency and variation. Looks like RMII is just for 10/100, not GigE.

RGMII is a 12-pin interface (whereas GMII is 24) as it has a clock, enable, and 4x data for each direction.

RMII requires a 50 MHz clock where MII requires a 25 MHz clock for faster data transfer and RMII has a reduced signal line count. Using RMII should reduce board space, but could increase cost (oscillator, supporting chip).

The T in 10baseT stands for twisted-pair.

SGMII is a serial variant, reducing to one TX pin each direction. Both SGMII and 1000Base-x are dual 1000Mbps SERDES pairs.

What if I have to fake out the MDIO pins?

Let's say you have a test board with an FPGA on which I/O pins are limited so the ethernet management MDIO pins were left out. This means we have to fake out the MDIO responses in the drivers for both u-boot and linux. The IF will operate fine despite this.

U-Boot Ethernet Magic (ARM)

Locate your board device config file (e.g. your *evm.h, such as TI's davinci_dvevm.h) Add a #define as needed to identify the Ethernet hack in the code.

Go to the vendor's ethernet driver file (e.g. cpu/arm926ehs/davinci/ether.c). In eth_phy_detect() fake out an active phy and also fake the ack in davinci_eth_phy_read(). For example, in the second instance, OR in MDIO_USERACCESS0_ACK. Force the status in gen_get_link_status() and also gen_auto_negotiate(). In davinci_emac_initialize() hardcode the phy ID.

Linux Ethernet Magic (ARM)

Go to the vendor's EMAC file (e.g. linux-*/drivers/net/davinci_emac.c). Add a #define as needed to identify the hack, return a hardcoded value from emac_mii_read() and force device selection to 0 in emac_dev_open()

What's the deal with Ethernet DMA on the iMX6?

When working on the drivers, keep an eye out for a nasty documentation disaster: Freescale doesn't let you know that the DMA requires two TxBDs to work. In the second, you don't need to set the ready bit, but you must set the SW ownership bit. Otherwise the first BD won't even transmit. Weird. And shame on those morons at Freescale for wasting my time.

Ethernet on the TriCore

It appears that only DMA mode is available on the TriCore, there is no other buffering provided besides RAM. This probably makes sense with the large size of Eth packets, and one wonders if this is true of all SoCs with an Eth peripheral.

Note that the MAC module will not come out of reset unless it is receiving a good clock from the PHY (or switch) on GREFCLK .

Linux Generic PHY Driver

fec_enet_mii_init
 ->mdiobus_register
  ->mdious_scan
   ->get_phy_device
    ->get_phy_id
    ->phy_device_create
   ->phy_device_register

How do you set the platform_device dev_id? Can't find where that struct is populated. In fec_enet_mii_probe there's logic to cycle through the array of 32 PHYs, but unless the dev_id is set it won't work. It's also strange that it checks for phy_id 0 instead of -1. phy_id is also used in different places as the PHY address (0-32) or the PHY vendor ID coming out of the on-chip registers. Files of interest include drivers/net/phy/phy.c, drivers/net/phy/phy_device.c, drivers/net/fec.c

Layer 2 Framing

The FCS/CRC at the end of the frame is typically added automatically by the NIC/adapter going out and checked and stripped by the receiving NIC/adapater so you won't see it in Wireshark without adjusting some settings. If the FCS check fails, switches and NICs will drop the packets and they won't be passed along or up the stack.

Links

The only good description I've found so far about the difference between Ethernet loopback tests. http://www.h3c.com/portal/Technical_Support___Documents/Technical_Documents/Switches/H3C_S5120_Series_Switches/Configuration/Operation_Manual/H3C_S5120-SI_CG-Release_1101-6W105/201108/723571_1285_0.htm#_Ref262480841

Linux kernel header Eth defines: https://docs.huihoo.com/doxygen/linux/kernel/3.7/uapi_2linux_2if__ether_8h.html

SGMII and SFP

SFP is kind of a form factor but it has different connections it can support. Typically optical or copper RJ-45, but can also be some custom connection like an SMA to SFP converter for example.

Note that people don't refer to an optical translator as a PHY, so optical links in the SFP case are more like a MAC-MAC or "direct" connection. An optical not-PHY takes an electrical level and just puts it on basically an LED, so it's an electric-light transducer.

SGMII can be auto-neg or fixed-link. Note that this is a different level of auto-neg than the Eth PHY auto-neg. For a Microchip switch with SGMII, but the driver doesn't recognize the phy-mode of "sgmii", this is what has worked for me:

  port@6 {
                reg = <6>;
                phy-mode = "rgmii-id";
                label = "lan6";
                mode = <0>;
                fixed-link {
                    speed = <1000>;
                    full-duplex;
                };
            };

Debugging

At least one switch datasheet I read, for the SGMII MAC to PHY connection, said AC coupling is required for the RX and TX lines so I wonder if that is the proper setting for an oscilloscope when probing those as well.

Linux Device Tree Bindings for Eth Drivers

There a lot of different terms, and some are frequently present while others aren't. Here is some documentation: https://elixir.bootlin.com/linux/v4.1/source/Documentation/devicetree/bindings/net

The mode here refers to direct-connect, as in is not fiber, does not go through SFP.

More on phy-mode bindings: https://www.thegoodpenguin.co.uk/blog/linux-ethernet-phy-mode-bindings-explained/

What is reg? reg represents the index of the gpio chip select associated to this SPI device or also in the DSA bindings, Describes the port address in the switch
What is mode? selects what type of HW connection we are dealing with
What is phy-handle? phy-handle: phandle, specifies a reference to a node representing a PHY device; this property is described in ePAPR and so preferred; replaces both phy and phy-device

SGMII example

There are 3 SGMII operating modes: 0 for direct connect, 1 for fiber, and 2 for copper 10/100/1000. The driver can detect whether fiber or 10/100/1000 SFP is used, so the default mode will be 1 and later changes to 2. If direct connection is used then the driver needs to be told to use mode 0.

Resets

Something to pay attention to when designing a new board. Some switch chips may have undetermined behavior if they are not issued a reset on boot. For example, the Microchip KSZ9477 datasheet asks for a reset on boot. In their own eval board, they built in a reset circuit using a voltage supervisor chip and a 200ms delay. In this way, they could issue the bootup reset in hardware and the driver (FW/SW) did not have to worry about it. However, another option is to give the driver information about the reset GPIO (assuming it's connected to the processor) using the device tree source. In this way the driver can issue a reset on boot, as long as the code is there to do it.

ksz9477@0 {
        compatible = "microchip,ksz9477";
        reg = <0>;

        spi-max-frequency = <20000000>;
        spi-cpha;
        spi-cpol;

    	pinctrl-0 = <&pinctrl_spi_irqn>;

	interrupt-parent = <&pioB>;
        interrupts = <28 IRQ_TYPE_LEVEL_LOW>;
        interrupt-controller;
        #interrupt-cells = <2>;

        reset-names = "phy";
        reset-gpios = <&pioC 31 GPIO_ACTIVE_LOW>;
        reset-assert-us = <1000>;
        reset-deassert-us = <2000>;

PHY Address

It's important that the PHY address boot pins be stable on boot, so install all suggested pull-up resistors or what have you.

The Microchip KSZ9477

A seven port switch. Note that there are multiple potential drivers for this device. The EVB-KSZ9477 repo driver (DSA family), the initial default selection, does not have SGMII port support included.

There is also a non-DSA driver option that does include SGMII support, and it can be built as a loadable module and loaded after boot. However, I've seen it cause a kernel crash on boot when it was enabled as a built-in module and I don't know why.

Microchip also has an "experimental DSA driver" with their recently added SGMII support from the EVB-LAN9646 repo. This can be applied in patch form. To use the DSA SGMII support, if in kernel linux-linux4microchip-2023.04 for example, you first update ksz_common.c and ksz_common.h files to the 2023.10 version and then apply the patch at
https://github.com/Microchip-Ethernet/EVB-LAN9646/blob/main/buildroot-external-microchip/patches/linux-linux4microchip-2023.10/012-ksz_dsa.patch
to the build/linux-custom/drivers/net/dsa/microchip/ folder.

Switches vs Routers

Switches are Layer 2 devices, they do not operate at Layer 3, so switches need to use MAC addresses, not IP addresses. Linux iproute and iptables tools are layer 3, using IP headers, and do not manipulate Eth frame MAC headers.

Routers are Layer 3 devices, so they have a sort of broader horizon.

Using iptables, arptables, ebtables, etc

This is a good diagram of the routing paths: https://serverfault.com/questions/680825/when-is-route-lookup-performed-when-a-packet-flows-through-iptables

Parts

The TI TLK105 is similar enough to the Micrel KSZ9031 (not the 21) that the Uboot phy driver can be shared. It looks like many of the basic PHY registers are common or standardized, such as BMSR, which is part of why this works. Uboot has #defines for these.

RJ-45 Connectors

Note that there are simple connectors that don't have integrated LEDs that connect the 8-pins directly to the pads that stick out the underside of the board where the connector is mounted. But other connectors have built-in components (like LEDs and transformers) that make things less straightforward. In that case, you can't ohm-check the pads directly as a debugging tactic. See the connector datasheets for more information.