Novena ddr3 notes

Software support

In order to tune DDR3, a RAM-only version of u-boot was created. This u-boot is totally stripped down to basically an interactive shell with few features. A set of calibration access routines were added:

tmd     - memory display for tuning
tmdel   - delay calibration
tmrr    - read data and checksum from write
tmtest  - simple RAM read/write test
tmw     - memory write (fill) for tuning
tmw2    - memory write (fill) for tuning, ignores trailing data
tmwcal  - write calibration
tmww    - write random data to RAM

Routines like tmrr and tmww just generate traffic to trigger the scope.

tmwcal and tmdel perform the DDR3 calibration routines, per Freescale application notes.

Currently, all the other shell routines break because global/static variable storage is broken. u-boot really tries hard to relocate itself to a new location in RAM, which I don't need it to do. Right now I just jump out of board init and call the interactive shell directly, skipping the board_init_f function which does the relocation. The only thing I'm really missing is hush functionality (which makes up-arrow work) but in the end I decided it wasn't worth the debugging to get that piece working.

To build this special u-boot, do the following.

Check out the tree:

git clone git@github.com:sutajiokousagi/u-boot-imx6.git

Set up the environment (as necessary per toolchain, this is for the OE self-built chain):

source /usr/local/oecore-x86_64/environment-setup-armv7a-angstrom-linux-gnueabi

build it:

make clean; make novena-ramtune_config; make

This should result in a u-boot.imx that is about 80k in size. It's targeted to load into OCRAM at address 0x009073e0. You can grow the image up to about 160k or so in size and it will still work.

burn it:

sudo dd if=u-boot.imx of=/dev/sdX seek=2

This locates the u-boot onto the SD card into the location expected by the initial bootstrapping ROM.

Bringup notes

First, it's important to note that it's impossible to write-level calibrate any memory made by Micron. This is because Micron only mirrors the calibration clock to DQ0 of each memory. However, other manufacturers, such as Kingston and Samsung, mirror the clock to all lanes, allowing for such calibration.

That being said, it was possible to collect calibration runs from several devices.

2-rank calibrations

SN001 samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000C0009
MMDC_MPWLDECTRL1 after write level cal: 0x003A002B
MMDC_MPWLDECTRL0 after write level cal: 0x00460052
MMDC_MPWLDECTRL1 after write level cal: 0x004D0101
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000C000A
MMDC_MPWLDECTRL1 after write level cal: 0x0039002B
MMDC_MPWLDECTRL0 after write level cal: 0x00460052
MMDC_MPWLDECTRL1 after write level cal: 0x004D0072
Novena U-Boot > 

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x4418040F
MPDGCTRL1 PHY0 (0x021b0840) = 0x040F0457
MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440
MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x2D2E432F
MPWRDLCTL PHY1 (0x021b4850) = 0x4736483E

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02CF0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02D80001
MPDGHWST2 PHY0 (0x021b0884) = 0x03170001
MPDGHWST3 PHY0 (0x021b0888) = 0x02CF0001
MPDGHWST0 PHY1 (0x021b487c) = 0x03000040
MPDGHWST1 PHY1 (0x021b4880) = 0x03400040
MPDGHWST2 PHY1 (0x021b4884) = 0x03000040
MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0

SN004 Samsung M473B5273DH0-YK0 cal (note custom layout to Samsung):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00160014
MMDC_MPWLDECTRL1 after write level cal: 0x003F0037
MMDC_MPWLDECTRL0 after write level cal: 0x004E0057
MMDC_MPWLDECTRL1 after write level cal: 0x01040108

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x4425041D
MPDGCTRL1 PHY0 (0x021b0840) = 0x04180455
MPDGCTRL0 PHY1 (0x021b483c) = 0x45001440
MPDGCTRL1 PHY1 (0x021b4840) = 0x0E000440

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x2E324134
MPWRDLCTL PHY1 (0x021b4850) = 0x4737493F

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02DD0001
MPDGHWST1 PHY0 (0x021b0880) = 0x02E50001
MPDGHWST2 PHY0 (0x021b0884) = 0x03150001
MPDGHWST3 PHY0 (0x021b0888) = 0x02D80001
MPDGHWST0 PHY1 (0x021b487c) = 0x03000040
MPDGHWST1 PHY1 (0x021b4880) = 0x03400040
MPDGHWST2 PHY1 (0x021b4884) = 0x03000040
MPDGHWST3 PHY1 (0x021b4888) = 0x07C007C0

Kingston 2-rank KVR16S11/4 cal:

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000F001B
MMDC_MPWLDECTRL1 after write level cal: 0x017E003B
MMDC_MPWLDECTRL0 after write level cal: 0x00490135
MMDC_MPWLDECTRL1 after write level cal: 0x0061010F
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x000F001A
MMDC_MPWLDECTRL1 after write level cal: 0x017E003B
MMDC_MPWLDECTRL0 after write level cal: 0x00490135
MMDC_MPWLDECTRL1 after write level cal: 0x0062010E

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440
MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440
MPDGCTRL0 PHY1 (0x021b483c) = 0x4516054A
MPDGCTRL1 PHY1 (0x021b4840) = 0x0523047D

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x43414443
MPWRDLCTL PHY1 (0x021b4850) = 0x4633473E

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x03000040
MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040
MPDGHWST2 PHY0 (0x021b0884) = 0x03000040
MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0
MPDGHWST0 PHY1 (0x021b487c) = 0x038A0001
MPDGHWST1 PHY1 (0x021b4880) = 0x03560001
MPDGHWST2 PHY1 (0x021b4884) = 0x033D0001
MPDGHWST3 PHY1 (0x021b4888) = 0x03630001

Centon MICU58PIA / CAP1066SO2048.01 (generic POS):

Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00120022
MMDC_MPWLDECTRL1 after write level cal: 0x017F0045
MMDC_MPWLDECTRL0 after write level cal: 0x0050013F
MMDC_MPWLDECTRL1 after write level cal: 0x00690118
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x00120023
MMDC_MPWLDECTRL1 after write level cal: 0x017E0045
MMDC_MPWLDECTRL0 after write level cal: 0x00500141
MMDC_MPWLDECTRL1 after write level cal: 0x006A0118
Novena U-Boot > tmdel


Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x44001440
MPDGCTRL1 PHY0 (0x021b0840) = 0x0E000440
MPDGCTRL0 PHY1 (0x021b483c) = 0x45170555
MPDGCTRL1 PHY1 (0x021b4840) = 0x051F0479

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x43424544
MPWRDLCTL PHY1 (0x021b4850) = 0x36313339

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x03000040
MPDGHWST1 PHY0 (0x021b0880) = 0x02C00040
MPDGHWST2 PHY0 (0x021b0884) = 0x03000040
MPDGHWST3 PHY0 (0x021b0888) = 0x07C007C0
MPDGHWST0 PHY1 (0x021b487c) = 0x03950001
MPDGHWST1 PHY1 (0x021b4880) = 0x03570001
MPDGHWST2 PHY1 (0x021b4884) = 0x03390001
MPDGHWST3 PHY1 (0x021b4888) = 0x035F0001

1-rank calibrations

Kingston 1-rank (1GB magic DIMM) from HP craptop HP594907-HR1-ELFEU / 1GB 1Rx8 PC3-10600S-9-10-B1:

Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x0032003E
MMDC_MPWLDECTRL1 after write level cal: 0x0057004E
MMDC_MPWLDECTRL0 after write level cal: 0x0050006D
MMDC_MPWLDECTRL1 after write level cal: 0x00670108
Novena U-Boot > tmwcal
Start write leveling calibration 
.
Write leveling calibration completed, errcount: 0
MMDC_MPWLDECTRL0 after write level cal: 0x0032003E
MMDC_MPWLDECTRL1 after write level cal: 0x0057004E
MMDC_MPWLDECTRL0 after write level cal: 0x004E006D
MMDC_MPWLDECTRL1 after write level cal: 0x00670108
Novena U-Boot > tmwcal

Read DQS Gating calibration
MPDGCTRL0 PHY0 (0x021b083c) = 0x43400401
MPDGCTRL1 PHY0 (0x021b0840) = 0x04190440
MPDGCTRL0 PHY1 (0x021b483c) = 0x44770300
MPDGCTRL1 PHY1 (0x021b4840) = 0x04700F3F

Read calibration
MPRDDLCTL PHY0 (0x021b0848) = 0x40404040
MPRDDLCTL PHY1 (0x021b4848) = 0x40404040

Write calibration
MPWRDLCTL PHY0 (0x021b0850) = 0x41424644
MPWRDLCTL PHY1 (0x021b4850) = 0x45384A42

Status registers, upper and lower bounds, for read DQS gating. 
MPDGHWST0 PHY0 (0x021b087c) = 0x02C10141
MPDGHWST1 PHY0 (0x021b0880) = 0x02800141
MPDGHWST2 PHY0 (0x021b0884) = 0x03000141
MPDGHWST3 PHY0 (0x021b0888) = 0x02D90001
MPDGHWST0 PHY1 (0x021b487c) = 0x02400181
MPDGHWST1 PHY1 (0x021b4880) = 0x033701AB
MPDGHWST2 PHY1 (0x021b4884) = 0x007F0001
MPDGHWST3 PHY1 (0x021b4888) = 0x033001AA
errorcount: 2

Note above that the errorcount is 2 on this run, i.e., the DQ gating calibration fully succeeded.

Other Notes

Measurements performed using TDS5104B 1GHz oscilloscope with dual active P6245 probes:

• probe 3 has about a -5% full scale offset (1.5V reads around 1.45V)
• probe 4 is pretty much dead on

drive strength on A5 looks fine with 60 ohm setting (0x20)

setting of 0x38 improves margin by about 90mV. Tiny overshoot seen. slight improvement in slew rate

conclusion: set to 48 ohm, 0x28 --> looks best here, absolutely no overshoot, signal amplitude matched to clock

dqs7 measured on the dimm side of the termination resistor:

• dimm->cpu rank 0, DQS7 0.36 - 1.19V swing, 830mv total. seems over-driven compared to clock, some reflection seen. in-phase with clock.
• dimm->cpu rank 1 same effect
• cpu->dimm rank 0, DQ7 0.45 - 1.1V swing, 650mV total. a bit cleaner. negative overshoot down to 0.22v seen on leading edge. 180 degrees out of phase with respect to clock.
• cpu->dimm rank 1, same effect

DQS4 is seen as problematic in read/write tests, as evidenced by data patterns:

Novena U-Boot > tmw.l 0xa0000000 0x0000ffff 0x400
Novena U-Boot > tmd.b 0xa0000000
a0000000: ff ff 00 00 3f ff 00 00 ff ff 00 00 d0 ff 00 00    ....?...........
a0000010: ff ff 00 00 ba ff 00 00 ff ff 00 00 f6 ff 00 00    ................

significantly, DQS4 is the very last DQS in the fly-by topology chain!

Handy Resources

Board layout files for SO-DIMMs. Looks like 90% of all SO-DIMMs use the reference design with very, very minor tweaks. There are basically two types of SO-DIMM layouts, one for single-rank, and one for dual-rank. The exception is Samsung, their low power DDR line has a novel layout. http://www.jedec.org/standards-documents/focus/memory-module-designs-dimms/DDR3/204-pin%20Unbuffered%20SODIMMs