sdram read results are occasionally wrong about beaglewire HOT 10 CLOSED

I'm still figuring out the tools and am not certain how these things work... I ran:

icetime \
	-p pinmap.pcf \
	-P tq144:4k \
	-C /usr/share/icestorm/chipdb-8k.txt \
	-t \
	sdram.asc

and it results in:

Resolvable net names on path 
     0.896 ns ..  7.470 ns gpmc_controller.addr[2]
     8.028 ns ..  8.897 ns $abc$10549$n1381_1
     9.362 ns .. 14.355 ns $abc$10549$n161
    15.016 ns .. 17.859 ns $abc$10549$n162
                  lcout -> mem[4][2]

Total number of logic levels: 4
Total path delay: 17.86 ns (55.99 MHz)

So I think that means that the controller didn't make timing for the 200 MHz clock?

from beaglewire.

Re-running with -d hx8k, the worst case is not as bad, but still not 200 MHz:

Resolvable net names on path:
     0.640 ns ..  5.101 ns gpmc_controller.addr[2]
     5.479 ns ..  6.068 ns $abc$10549$n1381_1
     6.384 ns ..  9.772 ns $abc$10549$n161
    10.220 ns .. 12.149 ns $abc$10549$n162
                  lcout -> mem[4][2]

Total number of logic levels: 4
Total path delay: 12.15 ns (82.31 MHz)

from beaglewire.

I've modified the SDRAM interface to reduce the number of GPMC cycles that are required osresearch@257be49 and added a bandwidth test. osresearch@1b06055

The correctness results still aren't there (a few percent error, especially when crossing into a different SDRAM page), but the preliminary bandwidth numbers are around 1.3 MB/s.

The page crossing seems to be partly the culprit -- doing a 0x400 sized random read/write cycles produces at 3% error (all on the same page), while a 0x800 sized random walk generates a 50% error rate on the benchmark.

from beaglewire.

I've tried to re-work the SDRAM state machine to work on the clock edge and have managed to reduce the second read error rate to almost zero, which indicates to me that there might be some issues with race conditions.

But in general the first read result on different pages seems problematic.

from beaglewire.

sdram_single_test() is an attempt to eliminate write errors from the measurement and only look at read errors. It chooses a random location and writes a random value to it, then reads it a few thousand times. It keeps a histogram of the bytes that come back and they do show a consistency - most of the time the right value is read and the errors are almost always the same byte or three (i.e. it is not a uniformly random distribution of read errors):

single: 000bf520=fc errors 13693 1.31%
34 13693
fc 1034883
single: 0009f68a=c2 errors 13813 1.32%
5c 6325
c2 1034763
c4 7488
single: 000d465d=cd errors 13875 1.32%
bd 847
c3 7463
ca 5565
cd 1034701
single: 0006d249=dd errors 7463 0.71%
7b 7463
dd 1041113
single: 000feff8=c1 errors 13670 1.30%
4f 7426
9c 696
c1 1034906
fb 5548
single: 000f8ae0=f2 errors 7472 0.71%
d4 7472
f2 1041104
single: 00081bc7=73 errors 7342 0.70%
73 1041234
ce 7342
single: 000f7a3b=73 errors 13198 1.26%
73 1035378
bc 5844
fc 7354

It is probably worth looking at the timing distribution of the errors; is it dependent on the refresh cycle, for instance?

Since the results seem incredibly sensitive to small changes in the verilog, here is the bitstream that I'm using: sdram.bin.gz

from beaglewire.

The code is now driving the SDRAM at clk/16 so that I can watch it on my scope. The good news is that this version of the read routine gets 0% errors for linear and random reads:

always @ (posedge sd_clk)
begin
        if (do_read) begin
                rd_data <= data_in_from_buffer;
                pmod1[0] <= data_in_from_buffer != wr_data_r;
        end
end

However this version that does not do the extra debug output to the pmod1 port gets around 10% read errors:

always @ (posedge sd_clk)
begin
        if (do_read) begin
                rd_data <= data_in_from_buffer;
        end
end

Both versions are generating around 107-114 MHz max path delays, which is well below the 6.25 MHz sd_clk timing. I'm wondering if there is something weird in the icestorm/yosys/arachne output here. This is with all three tools built from today's checkout.

from beaglewire.

I'm leaning towards something in the toolchain. This version generates SDRAM read errors with a wire that nothing even uses:

// comment this out for 0% errors, include the wire for 5% errors
wire totally_bogus_unused_wire;

always @ (posedge sd_clk)
begin
        if (do_read) begin
                rd_data <= data_in_from_buffer;
                pmod1[0] <= data_in_from_buffer != wr_data_r;
        end
end

from beaglewire.

I spent some quality time with a colleague who actually knows Verilog and they were able to help me build a simple test case for the SDRAM controller using Lattice's reference code and icestorm. We were able to verify that we can read/write the RAM without errors, which indicates that the bug is likely in the GPMC clock-crossing code, so we hacked that up and have something that works (slowly).

I'll clean it up and try to apply it to the existing SDRAM controller so that we can hopefully close out these issues.

from beaglewire.

@osresearch @pmezydlo Any update on this?
Have you uploaded anything on any repo related to the fixture of this bug?

from beaglewire.

Now we are using litedram + serv for sdram, it passed the litex mem test.
https://twitter.com/QwertyEmbedded/status/1418128122042486787

regards,
omkar bhilare
[email protected]

from beaglewire.