Novena DVT Issue Log
Contents
Hardware Bringup Notes
- The DDR3 timing has shifted with the new PCB vendor. The hard-coded timings for Micron memories probably needs to be recalibrated.
- R20S/C16S needs tuning, for sure. The default values break Gbit ethernet. Simply removing R20S does restore bandwidth, but the matter warrants further investigation.
- The heartbeat LED D14D is installed 90 degrees rotated. It still lights, it just fires downwards instead of to the side.
- Y10U/Y11U are missing per factory notes. They must be hand-installed for USB to work.
- C11M, C30M, C18M, C52C C45M capacitor is 1210 instead of 1206. Factory has been notified of the mis-ordering of the part.
- R57B 100 ohm -> move to DNP because there is on-chip termination for the FPGA
- For PVT, add 0/5in delay line to Gbit RXC, and 0/5/10 in selectable delays (via 0-ohm jumper) for Gbit TXC. Note that PHY has tuning mechanism to add delays, and RXC delay was fully verified to be compliant, so the 5in delay is just for margin. For TXC I believe delay is also compensated in PHY but it's not possible to measure the internal node, and therefore selectable paths are provided.
Gigabit Ethernet Debug Notes
Gigabit ethernet has been inconsistent in performance. Multiple factors have been affecting this. Here are the findings.
You learn something new every day
Fun fact: RGMII, for versions prior to 2.0, require a 1.6ns skewed clock -- done by adding about 10 inches or so of trace on FR-4 to the clock line versus the data lines. Apparently, someone in HP thought this was a good idea, to the point where HP decided to put notes about patents and IP ownership inside the spec.
Turns out it's a terrible idea. Nobody wants to route 10 inches of extra trace on a clock line. Fortunately, the Micrel PHY chip offers a skew adjust bank of registers to enable you to route clocks with no skew on the PCB.
xobs wrote a tool to manipulate MII registers directly, https://github.com/xobs/mii-talk
These parameters tune out the skew fairly well:
miitalk -w 11=0x8104 -w 12=0xf0f0 miitalk -w 11=0x8105 -w 12=0x0 miitalk -w 11=0x8106 -w 12=0xffff
We're using the extended control register set for MII, which means we can't just read and write them directly. You set bit 12 of register 11 to indicate a write, and then enter the extended register number in the lower 12 bits. Then, you put the write data into register 12, hence the odd form for miitalk arguments.
- MII extended register number 0x104 is the RGMII clock and control pad skew. Basically, we apply max skew to the clock.
- MII extended register number 0x105 is the RGMII RX data pad skew. We maximally unskew it.
- MII extended register number 0x106 is undocumented in the data sheet, but tests show that it controls RGMII TX data pad skew. A value of 0xFFFF worked nicely.
For the PVT version of the board, an extra 5in trace length (~0.7ns path length) will be added to give more margin, as currently the cal values get us to near the minimum value required (min requirement 1ns, measured 1.3ns), and the spec calls for up to 2.6ns.
Another issue discovered is that IOMUXC_SW_PAD_CTL_GRP_DDR_TYPE_RGMII was set for a 1.2V I/O, when it should be set for 2.5V I/O. This is fixed using the manual command
devmem2 0x20e0790 w 0xc0000
iperf testing shows we're limited now by the 100Mbit hub. Previously bandwidths would drop to <10Mbits/s without proper cal on some boards.
root@novena:~/tests/mii-talk# iperf -c xx.xx.xx.xx -d ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to xx.xx.xx.xx, TCP port 5001 TCP window size: 78.5 KByte (default) ------------------------------------------------------------ [ 4] local xx.xx.xx.xx port 43017 connected with xx.xx.xx.xx port 5001 [ 5] local xx.xx.xx.xx port 5001 connected with xx.xx.xx.xx port 15521 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 108 MBytes 90.7 Mbits/sec [ 5] 0.0-10.0 sec 110 MBytes 91.8 Mbits/sec
TL;DR version
#!/bin/sh echo "Temp fix to patch up gigabit ethernet timings. Will be rolled into a later kernel release." echo "Assumes you have checked out and built https://github.com/xobs/mii-talk in ~/tests/mii-talk" /home/root/tests/mii-talk/miitalk -w 11=0x8104 -w 12=0xf0f0 /home/root/tests/mii-talk/miitalk -w 11=0x8105 -w 12=0x0 /home/root/tests/mii-talk/miitalk -w 11=0x8106 -w 12=0xffff devmem2 0x20e0790 w 0xc0000
Tuning TX to PHY
The PHY default values are overdriving the outputs. Drive clock at 60 ohms, data at 48 ohms.
Therefore:
devmem2 0x20e036c w 0x1b020 # set RGMII_TXC devmem2 0x20e0388 w 0x1b028 # set TX_CTL devmem2 0x20e0370 w 0x1b028 # set RGMII_TD0 devmem2 0x20e0374 w 0x1b028 # TD1, etc. devmem2 0x20e0378 w 0x1b028 devmem2 0x20e037c w 0x1b028
Due to asymmetry in the refclk, it may be necessary to juice up TD0-TD3 to a value of 0x1b0b0, but this causes a lot of overshoot. Let's try 0x1b028 and instead fix the asymmetry issue.
Note that DVT has TXC and TD* exactly lined up when measured at the PHY, so adding a few inches of trace length on TXC isn't a bad idea. On PVT there will be an option to put a 0-ohm jumper to select 0 in, 5in, and 10in delays.
Quick Tests
- Memory tester: http://memtester.sourcearchive.com/downloads/4.2.2-1/memtester_4.2.2.orig.tar.gz
- wget large file from bunniefoo on all active interfaces: curl -o /dev/null http://bunniefoo.com/bunnie/test.bin
- mass storage on both USB ports
Helpful: stty rows 72 columns 132
Distribution Log
SN001: bunnie
- first time pass
SN002: xobs
- first time pass
SN003: bn.d
- ASIX chip has bad solder joints; now fixed, runs at 8428kiB
- Gbit: 8416kiB (limited by hub)
- 2GB single-rank Crucial SO-DIMM
- USB ok
SN004: ra.e
- Gbit: 8600kiB (limited by hub)
- ASIX: 8400kiB
- 4GB dual-rank Crucial SO-DIMM
- USB ok
SN005: te.r
- 4GB dual-rank PNY SO-DIMM
- Gbit: 3700kiB
- ASIX: 8319kiB
- USB ok
