Novena DVT Issue Log

Hardware Bringup Notes

• The DDR3 timing has shifted with the new PCB vendor. The hard-coded timings for Micron memories probably needs to be recalibrated.
• R20S/C16S needs tuning, for sure. The default values break Gbit ethernet. Simply removing R20S does restore bandwidth, but the matter warrants further investigation.
• The heartbeat LED D14D is installed 90 degrees rotated. It still lights, it just fires downwards instead of to the side.
• Y10U/Y11U are missing per factory notes. They must be hand-installed for USB to work.
• C11M, C30M, C18M, C52C C45M capacitor is 1210 instead of 1206. Factory has been notified of the mis-ordering of the part.
• R57B 100 ohm -> move to DNP because there is on-chip termination for the FPGA

• For PVT, add 300 ohm, 1% resistor pull-up to 2.5V to fix duty cycle asymmetry on RGMII_REF_CLK

• For PVT, add 0/5in delay line to Gbit RXC, and 0/5/10 in selectable delays (via 0-ohm jumper) for Gbit TXC. Note that PHY has tuning mechanism to add delays, and RXC delay was fully verified to be compliant, so the 5in delay is just for margin. For TXC I believe delay is also compensated in PHY but it's not possible to measure the internal node, and therefore selectable paths are provided.

Gigabit Ethernet Debug Notes

Gigabit ethernet has been inconsistent in performance. Multiple factors have been affecting this. Here are the findings.

You learn something new every day

Fun fact: RGMII, for versions prior to 2.0, require a 1.6ns skewed clock -- done by adding about 10 inches or so of trace on FR-4 to the clock line versus the data lines. Apparently, someone in HP thought this was a good idea, to the point where HP decided to put notes about patents and IP ownership inside the spec.

Turns out it's a terrible idea. Nobody wants to route 10 inches of extra trace on a clock line. Fortunately, the Micrel PHY chip offers a skew adjust bank of registers to enable you to route clocks with no skew on the PCB.

xobs wrote a tool to manipulate MII registers directly, https://github.com/xobs/mii-talk

These parameters tune out the skew fairly well:

miitalk -w 11=0x8104 -w 12=0xf0f0
miitalk -w 11=0x8105 -w 12=0x0
miitalk -w 11=0x8106 -w 12=0xffff

We're using the extended control register set for MII, which means we can't just read and write them directly. You set bit 12 of register 11 to indicate a write, and then enter the extended register number in the lower 12 bits. Then, you put the write data into register 12, hence the odd form for miitalk arguments.

• MII extended register number 0x104 is the RGMII clock and control pad skew. Basically, we apply max skew to the clock.
• MII extended register number 0x105 is the RGMII RX data pad skew. We maximally unskew it.
• MII extended register number 0x106 is undocumented in the data sheet, but tests show that it controls RGMII TX data pad skew. A value of 0xFFFF worked nicely.

For the PVT version of the board, an extra 5in trace length (~0.7ns path length) will be added to give more margin, as currently the cal values get us to near the minimum value required (min requirement 1ns, measured 1.3ns), and the spec calls for up to 2.6ns.

Another issue discovered is that IOMUXC_SW_PAD_CTL_GRP_DDR_TYPE_RGMII was set for a 1.2V I/O, when it should be set for 2.5V I/O. This is fixed using the manual command

devmem2 0x20e0790 w 0xc0000

iperf testing shows we're limited now by the 100Mbit hub. Previously bandwidths would drop to <10Mbits/s without proper cal on some boards.

root@novena:~/tests/mii-talk# iperf -c xx.xx.xx.xx -d                              
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to xx.xx.xx.xx, TCP port 5001
TCP window size: 78.5 KByte (default)
------------------------------------------------------------
[  4] local xx.xx.xx.xx port 43017 connected with xx.xx.xx.xx port 5001
[  5] local xx.xx.xx.xx port 5001 connected with xx.xx.xx.xx port 15521
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec    108 MBytes  90.7 Mbits/sec
[  5]  0.0-10.0 sec    110 MBytes  91.8 Mbits/sec

Tuning TX to PHY

The PHY default values are overdriving the outputs. Drive clock at 60 ohms, data at 48 ohms.

Therefore:

devmem2 0x20e036c w 0x1b020  # set RGMII_TXC
devmem2 0x20e0388 w 0x1b028  # set TX_CTL
devmem2 0x20e0370 w 0x1b028  # set RGMII_TD0
devmem2 0x20e0374 w 0x1b028  # TD1, etc.
devmem2 0x20e0378 w 0x1b028
devmem2 0x20e037c w 0x1b028

Due to asymmetry in the refclk, it may be necessary to juice up TD0-TD3 to a value of 0x1b0b0, but this causes a lot of overshoot. Let's try 0x1b028 and instead fix the asymmetry issue.

Note that DVT has TXC and TD* exactly lined up when measured at the PHY, so adding a few inches of trace length on TXC isn't a bad idea. On PVT there will be an option to put a 0-ohm jumper to select 0 in, 5in, and 10in delays.

Refclk asymmetry can be largely corrected by adding a 300 ohm pull-up to 2.5V on the receiving end as a terminator. A stronger pull-up (100 ohms) causes too much upward shift of the low value; a weaker one (1k) has virtually no impact.

TL;DR version

#!/bin/sh

echo "Temp fix to patch up gigabit ethernet timings. Will be rolled into a later kernel release."
echo "Assumes you have checked out and built  https://github.com/xobs/mii-talk in ~/tests/mii-talk"

/home/root/tests/mii-talk/miitalk -w 11=0x8104 -w 12=0xf0f0
/home/root/tests/mii-talk/miitalk -w 11=0x8105 -w 12=0x0
/home/root/tests/mii-talk/miitalk -w 11=0x8106 -w 12=0xffff

devmem2 0x20e0790 w 0xc0000

devmem2 0x20e036c w 0x1b020  # set RGMII_TXC
devmem2 0x20e0388 w 0x1b028  # set TX_CTL
devmem2 0x20e0370 w 0x1b028  # set RGMII_TD0
devmem2 0x20e0374 w 0x1b028  # TD1, etc.
devmem2 0x20e0378 w 0x1b028
devmem2 0x20e037c w 0x1b028

Quick Tests

• Memory tester: http://memtester.sourcearchive.com/downloads/4.2.2-1/memtester_4.2.2.orig.tar.gz
• wget large file from bunniefoo on all active interfaces: curl -o /dev/null http://bunniefoo.com/bunnie/test.bin
• mass storage on both USB ports

Helpful: stty rows 72 columns 132

Distribution Log

SN001: bunnie

• first time pass

SN002: xobs

• first time pass

SN003: bn.d

• ASIX chip has bad solder joints; now fixed, runs at 8428kiB
• Gbit: 8416kiB (limited by hub)
• 2GB single-rank Crucial SO-DIMM
• USB ok

SN004: ra.e

• Gbit: 8600kiB (limited by hub)
• ASIX: 8400kiB
• 4GB dual-rank Crucial SO-DIMM
• USB ok

SN005: te.r

• 4GB dual-rank PNY SO-DIMM
• Gbit: 3700kiB
• ASIX: 8319kiB
• USB ok