Novena ddr3 notes
Contents
Software support
In order to tune DDR3, a RAM-only version of u-boot was created. This u-boot is totally stripped down to basically an interactive shell with few features. A set of calibration access routines were added:
tmd - memory display for tuning tmdel - delay calibration tmrr - read data and checksum from write tmtest - simple RAM read/write test tmw - memory write (fill) for tuning tmw2 - memory write (fill) for tuning, ignores trailing data tmwcal - write calibration tmww - write random data to RAM
Routines like tmrr and tmww just generate traffic to trigger the scope.
tmwcal and tmdel perform the DDR3 calibration routines, per Freescale application notes.
Currently, all the other shell routines break because global/static variable storage is broken. u-boot really tries hard to relocate itself to a new location in RAM, which I don't need it to do. Right now I just jump out of board init and call the interactive shell directly, skipping the board_init_f function which does the relocation. The only thing I'm really missing is hush functionality (which makes up-arrow work) but in the end I decided it wasn't worth the debugging to get that piece working.
To build this special u-boot, do the following.
Check out the tree:
git clone git@github.com:sutajiokousagi/u-boot-imx6.git
Set up the environment (as necessary per toolchain, this is for the OE self-built chain):
source /usr/local/oecore-x86_64/environment-setup-armv7a-angstrom-linux-gnueabi
build it:
make clean; make novena-ramtune_config; make
This should result in a u-boot.imx that is about 80k in size. It's targeted to load into OCRAM at address 0x009073e0. You can grow the image up to about 160k or so in size and it will still work.
burn it:
sudo dd if=u-boot.imx of=/dev/sdX seek=2
This locates the u-boot onto the SD card into the location expected by the initial bootstrapping ROM.
Bringup notes
First, it's important to note that it's impossible to write-level calibrate any memory made by Micron. This is because Micron only mirrors the calibration clock to DQ0 of each memory. However, other manufacturers, such as Kingston and Samsung, mirror the clock to all lanes, allowing for such calibration.
That being said, it was possible to collect calibration runs from several devices.
2-rank calibrations
SN001 samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):
Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x000C0009 MMDC_MPWLDECTRL1 after write level cal: 0x003A002B MMDC_MPWLDECTRL0 after write level cal: 0x00460052 MMDC_MPWLDECTRL1 after write level cal: 0x004D0101 Novena U-Boot > tmwcal Start write leveling calibration . Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x000C000A MMDC_MPWLDECTRL1 after write level cal: 0x0039002B MMDC_MPWLDECTRL0 after write level cal: 0x00460052 MMDC_MPWLDECTRL1 after write level cal: 0x004D0072 Novena U-Boot > Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x4418040F MPDGCTRL1 PHY0 (0x021b0840) = 0x040F0457 MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440 MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440 Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x40404040 MPRDDLCTL PHY1 (0x021b4848) = 0x40404040 Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x2D2E432F MPWRDLCTL PHY1 (0x021b4850) = 0x4736483E Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02CF0001 MPDGHWST1 PHY0 (0x021b0880) = 0x02D80001 MPDGHWST2 PHY0 (0x021b0884) = 0x03170001 MPDGHWST3 PHY0 (0x021b0888) = 0x02CF0001 MPDGHWST0 PHY1 (0x021b487c) = 0x03000040 MPDGHWST1 PHY1 (0x021b4880) = 0x03400040 MPDGHWST2 PHY1 (0x021b4884) = 0x03000040 MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0
SN004 Samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):
Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x00160014 MMDC_MPWLDECTRL1 after write level cal: 0x003F0037 MMDC_MPWLDECTRL0 after write level cal: 0x004E0057 MMDC_MPWLDECTRL1 after write level cal: 0x01040108 Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x4425041D MPDGCTRL1 PHY0 (0x021b0840) = 0x04180455 MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440 MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440 Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x40404040 MPRDDLCTL PHY1 (0x021b4848) = 0x40404040 Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x2E324134 MPWRDLCTL PHY1 (0x021b4850) = 0x4737493F Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02DD0001 MPDGHWST1 PHY0 (0x021b0880) = 0x02E50001 MPDGHWST2 PHY0 (0x021b0884) = 0x03150001 MPDGHWST3 PHY0 (0x021b0888) = 0x02D80001 MPDGHWST0 PHY1 (0x021b487c) = 0x03000040 MPDGHWST1 PHY1 (0x021b4880) = 0x03400040 MPDGHWST2 PHY1 (0x021b4884) = 0x03000040 MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0
Kingston 2-rank KVR16S11/4 cal:
Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x000F001B MMDC_MPWLDECTRL1 after write level cal: 0x017E003B MMDC_MPWLDECTRL0 after write level cal: 0x00490135 MMDC_MPWLDECTRL1 after write level cal: 0x0061010F Novena U-Boot > tmwcal Start write leveling calibration . Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x000F001A MMDC_MPWLDECTRL1 after write level cal: 0x017E003B MMDC_MPWLDECTRL0 after write level cal: 0x00490135 MMDC_MPWLDECTRL1 after write level cal: 0x0062010E Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440 MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440 MPDGCTRL0 PHY1 (0x021b483c) = 0x4516054A MPDGCTRL1 PHY1 (0x021b4840) = 0x0523047D Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x40404040 MPRDDLCTL PHY1 (0x021b4848) = 0x40404040 Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x43414443 MPWRDLCTL PHY1 (0x021b4850) = 0x4633473E Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x03000040 MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040 MPDGHWST2 PHY0 (0x021b0884) = 0x03000040 MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0 MPDGHWST0 PHY1 (0x021b487c) = 0x038A0001 MPDGHWST1 PHY1 (0x021b4880) = 0x03560001 MPDGHWST2 PHY1 (0x021b4884) = 0x033D0001 MPDGHWST3 PHY1 (0x021b4888) = 0x03630001
Centon MICU58PIA / CAP1066SO2048.01 (generic POS):
Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x00120022 MMDC_MPWLDECTRL1 after write level cal: 0x017F0045 MMDC_MPWLDECTRL0 after write level cal: 0x0050013F MMDC_MPWLDECTRL1 after write level cal: 0x00690118 Novena U-Boot > tmwcal Start write leveling calibration . Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x00120023 MMDC_MPWLDECTRL1 after write level cal: 0x017E0045 MMDC_MPWLDECTRL0 after write level cal: 0x00500141 MMDC_MPWLDECTRL1 after write level cal: 0x006A0118 Novena U-Boot > tmdel Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440 MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440 MPDGCTRL0 PHY1 (0x021b483c) = 0x45170555 MPDGCTRL1 PHY1 (0x021b4840) = 0x051F0479 Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x40404040 MPRDDLCTL PHY1 (0x021b4848) = 0x40404040 Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x43424544 MPWRDLCTL PHY1 (0x021b4850) = 0x36313339 Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x03000040 MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040 MPDGHWST2 PHY0 (0x021b0884) = 0x03000040 MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0 MPDGHWST0 PHY1 (0x021b487c) = 0x03950001 MPDGHWST1 PHY1 (0x021b4880) = 0x03570001 MPDGHWST2 PHY1 (0x021b4884) = 0x03390001 MPDGHWST3 PHY1 (0x021b4888) = 0x035F0001
SN001 Kingston 2-rank KVR16S11/4 cal, with new uboot (1/13):
write starting at 10000000 checksum: 7ffd001e read starting at 10000000 computed: 7ffd001e, readback: 7fe961b9 Start write leveling calibration .Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x002A002E MMDC_MPWLDECTRL1 after write level cal: 0x0015004E MMDC_MPWLDECTRL0 after write level cal: 0x00640150 MMDC_MPWLDECTRL1 after write level cal: 0x00680126 init cs0: 1 cs1: 1 cal cs0: 1 cs1: 0 db size: 2 Starting DQS gating calibration... .errorcount: 0 DQS gating calibration completed, hit enter to proceed. Starting read calibration... intdel0: 40404040 / intdel1: 40404040 .errorcount: 0 Read calibration completed, hit enter to continue Starting write calibration... intdel0: 40404040 / intdel1: 40404040 .errorcount: 0 Write calibration completed, hit enter to continue MMDC registers updated from calibration Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x437E041A MPDGCTRL1 PHY0 (0x021b0840) = 0x03500445 MPDGCTRL0 PHY1 (0x021b483c) = 0x4473053B MPDGCTRL1 PHY1 (0x021b4840) = 0x050D046A Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x45414642 MPRDDLCTL PHY1 (0x021b4848) = 0x4548424C Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x45414545 MPWRDLCTL PHY1 (0x021b4850) = 0x45324641 Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02DA0001 MPDGHWST1 PHY0 (0x021b0880) = 0x02BE0001 MPDGHWST2 PHY0 (0x021b0884) = 0x03050001 MPDGHWST3 PHY0 (0x021b0888) = 0x02900001 MPDGHWST0 PHY1 (0x021b487c) = 0x037B0001 MPDGHWST1 PHY1 (0x021b4880) = 0x03330001 MPDGHWST2 PHY1 (0x021b4884) = 0x032A0001 MPDGHWST3 PHY1 (0x021b4888) = 0x034D0001 errorcount: 0 init cs0: 1 cs1: 1 cal cs0: 1 cs1: 0 db size: 2 Starting DQS gating calibration... .errorcount: 0 DQS gating calibration completed, hit enter to proceed. Starting read calibration... intdel0: 40404040 / intdel1: 40404040 .errorcount: 0 Read calibration completed, hit enter to continue Starting write calibration... intdel0: 40404040 / intdel1: 40404040 .errorcount: 0 Write calibration completed, hit enter to continue MMDC registers updated from calibration Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x4401041B MPDGCTRL1 PHY0 (0x021b0840) = 0x03530443 MPDGCTRL0 PHY1 (0x021b483c) = 0x4475053F MPDGCTRL1 PHY1 (0x021b4840) = 0x050E046A Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x45424642 MPRDDLCTL PHY1 (0x021b4848) = 0x4548424C Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x45414545 MPWRDLCTL PHY1 (0x021b4850) = 0x45324641 Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02DB0001 MPDGHWST1 PHY0 (0x021b0880) = 0x02C10001 MPDGHWST2 PHY0 (0x021b0884) = 0x03030001 MPDGHWST3 PHY0 (0x021b0888) = 0x02930001 MPDGHWST0 PHY1 (0x021b487c) = 0x037F0001 MPDGHWST1 PHY1 (0x021b4880) = 0x03350001 MPDGHWST2 PHY1 (0x021b4884) = 0x032A0001 MPDGHWST3 PHY1 (0x021b4888) = 0x034E0001 errorcount: 0 write starting at 10000000 checksum: 7ffd001e read starting at 10000000 computed: 7ffd001e, readback: 7ffd001e ### main_loop entered: bootdelay=1
1-rank calibrations
Kingston 1-rank (1GB magic DIMM) from HP craptop HP594907-HR1-ELFEU / 1GB 1Rx8 PC3-10600S-9-10-B1:
Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x004B0055 MMDC_MPWLDECTRL1 after write level cal: 0x00700069 MMDC_MPWLDECTRL0 after write level cal: 0x0067010F MMDC_MPWLDECTRL1 after write level cal: 0x0114011D Novena U-Boot > tmwcal Start write leveling calibration . Write leveling calibration completed, errcount: 0 MMDC_MPWLDECTRL0 after write level cal: 0x004B0055 MMDC_MPWLDECTRL1 after write level cal: 0x00700069 MMDC_MPWLDECTRL0 after write level cal: 0x0067010F MMDC_MPWLDECTRL1 after write level cal: 0x0114011F Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x437D0413 MPDGCTRL1 PHY0 (0x021b0840) = 0x040F0416 MPDGCTRL0 PHY1 (0x021b483c) = 0x44570455 MPDGCTRL1 PHY1 (0x021b4840) = 0x045F0426 Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x4C44434A MPRDDLCTL PHY1 (0x021b4848) = 0x4B494151 Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x34344139 MPWRDLCTL PHY1 (0x021b4850) = 0x3E2F4638 Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02D30001 MPDGHWST1 PHY0 (0x021b0880) = 0x02BD0001 MPDGHWST2 PHY0 (0x021b0884) = 0x02D60001 MPDGHWST3 PHY0 (0x021b0888) = 0x02CF0001 MPDGHWST0 PHY1 (0x021b487c) = 0x03150001 MPDGHWST1 PHY1 (0x021b4880) = 0x03170001 MPDGHWST2 PHY1 (0x021b4884) = 0x02E60001 MPDGHWST3 PHY1 (0x021b4888) = 0x031F0001 errorcount: 0
With new codebase integrated into u-boot (jan 13):
MMDC_MPWLDECTRL0 after write level cal: 0x003F0046 MMDC_MPWLDECTRL1 after write level cal: 0x00660059 MMDC_MPWLDECTRL0 after write level cal: 0x005F0107 MMDC_MPWLDECTRL1 after write level cal: 0x00640115 Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x436D037D MPDGCTRL1 PHY0 (0x021b0840) = 0x037F040D MPDGCTRL0 PHY1 (0x021b483c) = 0x44500447 MPDGCTRL1 PHY1 (0x021b4840) = 0x04570421 Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x48454244 MPRDDLCTL PHY1 (0x021b4848) = 0x474C434F Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x32304137 MPWRDLCTL PHY1 (0x021b4850) = 0x452C4637 Status registers, upper and lower bounds, for read DQS gating. MPDGHWST0 PHY0 (0x021b087c) = 0x02BD0001 MPDGHWST1 PHY0 (0x021b0880) = 0x02AD0001 MPDGHWST2 PHY0 (0x021b0884) = 0x02CD0001 MPDGHWST3 PHY0 (0x021b0888) = 0x02BF0001 MPDGHWST0 PHY1 (0x021b487c) = 0x03070001 MPDGHWST1 PHY1 (0x021b4880) = 0x03100001 MPDGHWST2 PHY1 (0x021b4884) = 0x02E10001 MPDGHWST3 PHY1 (0x021b4888) = 0x03170001
Errcount is zero, and the trivial write/read test seems to work with these parameters.
Note above that the errorcount is 0 on this run.
Micron CT25664BC1067.M8FMR:
Can't do write leveling calibration, use from kingston above.
Read DQS Gating calibration MPDGCTRL0 PHY0 (0x021b083c) = 0x440D0422 MPDGCTRL1 PHY0 (0x021b0840) = 0x0418041F MPDGCTRL0 PHY1 (0x021b483c) = 0x4459045E MPDGCTRL1 PHY1 (0x021b4840) = 0x0467042E Read calibration MPRDDLCTL PHY0 (0x021b0848) = 0x4A3F454B MPRDDLCTL PHY1 (0x021b4848) = 0x464A424F Write calibration MPWRDLCTL PHY0 (0x021b0850) = 0x3636443F MPWRDLCTL PHY1 (0x021b4850) = 0x42334A3D
Other Notes
Measurements performed using TDS5104B 1GHz oscilloscope with dual active P6245 probes:
- probe 3 has about a -5% full scale offset (1.5V reads around 1.45V)
- probe 4 is pretty much dead on
drive strength on A5 looks fine with 60 ohm setting (0x20)
setting of 0x38 improves margin by about 90mV. Tiny overshoot seen. slight improvement in slew rate
conclusion: set to 48 ohm, 0x28 --> looks best here, absolutely no overshoot, signal amplitude matched to clock
dqs7 measured on the dimm side of the termination resistor:
- dimm->cpu rank 0, DQS7 0.36 - 1.19V swing, 830mv total. seems over-driven compared to clock, some reflection seen. in-phase with clock.
- dimm->cpu rank 1 same effect
- cpu->dimm rank 0, DQ7 0.45 - 1.1V swing, 650mV total. a bit cleaner. negative overshoot down to 0.22v seen on leading edge. 180 degrees out of phase with respect to clock.
- cpu->dimm rank 1, same effect
DQS4 is seen as problematic in read/write tests, as evidenced by data patterns:
Novena U-Boot > tmw.l 0xa0000000 0x0000ffff 0x400 Novena U-Boot > tmd.b 0xa0000000 a0000000: ff ff 00 00 3f ff 00 00 ff ff 00 00 d0 ff 00 00 ....?........... a0000010: ff ff 00 00 ba ff 00 00 ff ff 00 00 f6 ff 00 00 ................
significantly, DQS4 is the very last DQS in the fly-by topology chain!
Handy Resources
Board layout files for SO-DIMMs. Looks like 90% of all SO-DIMMs use the reference design with very, very minor tweaks. There are basically two types of SO-DIMM layouts, one for single-rank, and one for dual-rank. The exception is Samsung, their low power DDR line has a novel layout. http://www.jedec.org/standards-documents/focus/memory-module-designs-dimms/DDR3/204-pin%20Unbuffered%20SODIMMs
