DDR3 Test doesn't do anything

[tudang]

Hi Jamey,

I'm able to build the tests/ddr3 but I observe that the test doesn't do anything. The test is wrapped in an IF statement which never fires.

https://github.com/cambridgehackers/connectal/blob/master/tests/ddr3/testddr3.cpp#L56
I've removed the ``if'' to see how it works. The program just hangs forever.

I would like to read and write to DDR3 but it's difficult to follow the test. It's great if you could help me to rewrite the test or an example to use DDR3 with Connectal.

[jameyhicks]

Hi,

If it hangs forever, something is wrong. That being said, I never actually finished this test.

Looking at the test, it does seem to be more complicated than it needs to be. Let's see what I can do.

Also, the modules and interfaces used in the test are documented here: http://www.connectal.org/doc/current/ref/bsv/memreadengine.html

[jameyhicks]

heh, now I understand what some of that code was for.

[jameyhicks]

Pushed a simpler version of the test. Building it for vc707 now. (vc707g2)

[jameyhicks]

I had to add some more cross clock domain constraints to pcie-clocks.xdc.

After running the example, I discovered it did not read from system memory, which it should have done. I used pcieflat to look at PCIE transactions.

So then I looked at the Makefile, which had the read client commented out. Fixed that and rebuilding...

[tudang]

I've built the previous commit on nfsume and got some timing violation errors. I include the log here in case you're interested.

*** timing violation ***
Slack (VIOLATED) :        -3.378ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            host_pcieHostTop_pciehost/traceif/fromPcieTraceBram_cbram_bram/RAM_reg_4/REGCEAREGCE
                            (rising edge-triggered cell RAMB36E1 clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -2.928ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_block0_status_reg_replica/C
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_aw_channel_0/axi_mc_cmd_translator_0/axi_mc_incr_cmd_0/axaddr_incr_reg[9]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -4.924ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_block1_status_reg/C
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_dramWriteGearbox_elem_2_reg[110]/CE
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -6.616ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_elem1_status_0_reg/C
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_aw_channel_0/axi_mc_cmd_translator_0/axi_mc_incr_cmd_0/axaddr_incr_reg[9]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -0.364ns  (arrival time - required time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_elem_0_reg[112]/C
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_w_channel_0/mc_app_wdf_data_reg_reg[112]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -2.613ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_awfifo/dGDeqPtr_reg[1]/CLR
                            (recovery check against rising-edge clock clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -1.705ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/rst200/reset_hold_reg[8]/CLR
                            (recovery check against rising-edge clock sys_clk  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -3.273ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_ddr3_infrastructure/rstdiv0_sync_r1_reg/C
                            (rising edge-triggered cell FDPE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_dramWriteGearbox_sInReset_pre_isInReset_reg/PRE
                            (recovery check against rising-edge clock userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -1.441ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

[jameyhicks]

I pushed a change to pcie-clocks.xdc to fix those timing violation.

After building c60f0d5 it ran to completion but the data read back from DRAM did not match what I thought I wrote. Please give it a try -- I may not have time to look at it again until tomorrow.

[tudang]

The timing issue has gone. I got the following output when run the program.

src dram[0000]=00000000
src dram[0004]=00000001
src dram[0008]=00000002
...
src dram[03f8]=000000fe
src dram[03fc]=000000ff

The writing task may be blocked.

[jameyhicks]

Please run pcieflat to see if it is doing any DMA.

[tudang]

I guess it does. Here are the first few lines of the output of running pcieflat

             ts     delta   response                     XXX          tlp          address  off   be       tag     clid  nosnp  laddr        data
                                pkttype format               foo (be hit eof sof)            (1st last)        req     stat  bcnt    length
TXqq  450636380          0 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287000 be(1st: f last:f) tag:00 reqid:0000 length:32 
RXpp  450636550        170 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:00 8300 00ff 0 00 080 00  16 00000000 
RXcc  450636551          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000004000000030000000200000001 
RXcc  450636552          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000008000000070000000600000005 
RXcc  450636553          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000000c0000000b0000000a00000009 
RXcc  450636554          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000000f0000000e0000000d 
RXpp  450636555          1 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:00 8300 00ff 0 00 040 40  16 10000000 
RXcc  450636556          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000014000000130000001200000011 
RXcc  450636557          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000018000000170000001600000015 
RXcc  450636558          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000001c0000001b0000001a00000019 
RXcc  450636559          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000001f0000001e0000001d 
TXqq  450636601         42 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287080 be(1st: f last:f) tag:01 reqid:0000 length:32 
...

[jameyhicks]

That is good -- at least it did some reads.

Usually it's the end of the trace that is more interesting.

[jameyhicks]

Have you used Vivado Integrated Logic Analyzer? This might be a good time for it.

If you add mkProbe in your BSV code, its signals will have (* mark_debug="true" *) so that they show up in the netlist after synthesis.

I am interested in how many times rl_req_aw and rl_wdata fire. I would reduce 1024 in Ddr3Test.bsv to 256 or 128.

[tudang]

This is the end of the trace

TXqq  450637930         43 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287380 be(1st: f last:f) tag:07 reqid:0000 length:32 
RXpp  450638097        167 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:07 8300 00ff 0 00 080 00  32 e0000000 
RXcc  450638098          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000e4000000e3000000e2000000e1 
RXcc  450638099          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000e8000000e7000000e6000000e5 
RXcc  450638100          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000ec000000eb000000ea000000e9 
RXcc  450638101          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f0000000ef000000ee000000ed 
RXcc  450638102          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f4000000f3000000f2000000f1 
RXcc  450638103          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f8000000f7000000f6000000f5 
RXcc  450638104          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000fc000000fb000000fa000000f9 
RXcc  450638105          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:00000000000000ff000000fe000000fd 
TXqq 3035184128 2584546023 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 00000002747a7000 be(1st: f last:f) tag:08 reqid:0000 length:32 
RXpp 3035184301        173 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:08 8300 00ff 0 00 080 00  32 00000000 
RXcc 3035184302          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000004000000030000000200000001 
RXcc 3035184303          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000008000000070000000600000005 
RXcc 3035184304          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000000c0000000b0000000a00000009 
RXcc 3035184305          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000100000000f0000000e0000000d 
RXcc 3035184306          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000014000000130000001200000011 
RXcc 3035184307          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000018000000170000001600000015 
RXcc 3035184308          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000001c0000001b0000001a00000019 
RXcc 3035184309          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000001f0000001e0000001d 
{'DmaRReq': 9, 'DmaRRsp': 14, 'DmaRCon': 72}
95

[pietrobressana]

Hi Jamey,

I'm working on Tu's DDR3 project, in order to add DDR3 capability to our Paxos implementation.

Can you please answer the following questions?

• Is Xilinx MIG supported into NetFPGA SUME framework? Is there any project that enables DDR3 capability on SUME?
• Is connectal's DDR3 test application working with SUME?
• Which DDR3 controller do you use in connectal? Do you use Xilinx MIG, or a custom one? Is connectal's controller ready to be used with SUME?

Thanks

[jameyhicks]

Hi Pietro,

I have not used DDR3 on the SUME board. All the projects using connectal that use DDR3 use the Xilinx MIG. Some of these projects use the DDR3 code in the Bluespec distribution, which was generated by Xilinx MIG. For example: bluespec/lib/board_support/bluenoc/xilinx/VC709/verilog/ddr3_v2_1/ddr3_v2_1

Because connectal is open source, I cannot really check in the output of the MIG, so my approach is to generate the cores as needed.

This test application has never been fully debugged, but I think we're close to having it working now. I will spend some more time on it today.

[tudang]

Thank Jamey,

As soon as you update the test, I could run it on the NetFPGA SUME.

[pietrobressana]

Hi Jamey,

While building the DDR3 test application with "make build.nfsume" command, the following error message is generated:

Verilog file created: /home/netfpga/github/connectal/tests/ddr3/nfsume/verilog/mkPcieTop.v
Error: XCI file /home/netfpga/github/connectal/out/nfsume/axiddr3/axiddr3.xci not found
make[1]: *** [fpgamake.mk] Error 255
make: *** [build.nfsume] Error 2

Do you know how can I solve this issue?

Thanks

[jameyhicks]

make ip.nfsume

[jameyhicks]

there is a way to make that dependence automatic, so I will fix the makefile.

[pietrobressana]

Thank you!

[jameyhicks]

Commit 9b40d04 generates IP cores if the project has synth-ip.tcl.

Commit f2717e9 fixes the problem where transferLen was set to zero because I divided by the wrong number.

But it turns out the address may be a byte address, so b0bcc05 removes the division and makes transferLen a parameter set by software. Building and testing that one now.

[jameyhicks]

That version runs but there is still a problem -- it looks to me like the sync fifos are not behaving properly. Either the constraints in pcie-clocks.xdc are not correct or Vivado still does not like either the sync FIFO or the gearbox.

When I look a the probes with Integrated Logic Analyzer, the sequence of addresses and data are not what they should be.

[tudang]

I saw the same behaviour for NetFPGA SUME. Another cause could be the addresses are not parsed correctly.

[jameyhicks]

I'm tracing the addresses and the output of awfifo is skipping addresses I expect to see.

[pietrobressana]

Hi,

I would like to debug the project with ILA core, as you suggested to Tu.

The BSV code of the project already includes "mkprobe", but no ILA probe is included into the generated bitfile.

How can I generate the probes?

Thank you!

[jameyhicks]

I tried to have a fully script solution to incorporate the ILA but so far have not succeeded.

Here is the process for nfsume:
make build.nfsume

When it has written top-post-link.dcp I start vivado from another shell:
vivado nfsume/Impl/TopDown/top-post-link.dcp

From vivado, I manually instantiate the ILA. The problem that prevents scripting is any constant values attached to the probes because Vivado cannot figure out the clock. I either remove these nets or attach the same clock as used by the other signals of a bus.

After the ILA is instantiated:

opt_design
place_design
phys_opt_design
route_design
write_bitstream -force debug.bit
write_debug_probes -force debug.ltx
report_timing_summary -file debug_timing_summary.txt

Now open the hardware manager and program the board. I usually do this from a machine separate from the test machine, especially on a test like this that has a chance of locking up the machine.

Once you have the trigger set up and you're ready to capture data, on the test machine:

pciescanportal;
NOPROGRAM=1 ubuntu.exe

If you do not define NOPROGRAM, it will reprogram the FPGA.

[jameyhicks]

In commit c30c882 I replaced mkSyncFIFO with mkDualClockFifo, my own import BVI wrapper for Xilinx's FIFO18. That improved the situation. I reconfigured the gearboxes so both ends were on the same clock domain and introduced mkDualClockFifo to cross the clock domain. Now the test is reading back from DRAM the same values it wrote to DRAM.

I updated the test to count and display mismatches in commit 84d0e03

Please give the new version a try.

[pietrobressana]

The new version seems to work fine.

Now we can add DDR3 capability to our project.

Thanks!

[pietrobressana]

Hi,

I would like to re-build the DDR3 example in order to make some changes into the provided *.cpp code.
However, after downloading the last version of connectal repo, the building procedure fails due to a number of warnings.
The error messages are related to the BSV hardware specification.

I have attached here only a small subset of the warnings:

Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0036)
Rule "tile_0_handle_startWriteDram_request" will appear to fire before
"tile_0_handle_startReadDram_request" when both fire in the same clock
cycle, affecting:
calls to
tile_0_lDdr3Test_transferLen.write vs. tile_0_lDdr3Test_transferLen.write
Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0036)
Rule "tile_0_lDdr3Test_we_rv_writeChannels_store_cmd" will appear to fire
before "tile_0_lDdr3Test_we_rv_writeChannels_load_ctxt_b" when both fire in
the same clock cycle, affecting:
calls to
tile_0_lDdr3Test_we_rv_writeChannels_clientBursts.write vs. tile_0_lDdr3Test_we_rv_writeChannels_clientBursts.write
tile_0_lDdr3Test_we_rv_writeChannels_clientStart.write vs. tile_0_lDdr3Test_we_rv_writeChannels_clientStart.write
Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117)
Rule tile_0_handle_startReadDram_request' shadows the effects of tile_0_handle_startWriteDram_request' when they execute in the same clock
cycle. Affected method calls:
tile_0_lDdr3Test_transferLen.write
To silence this warning, use the -no-warn-action-shadowing' flag. Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117) Ruletile_0_lDdr3Test_we_rv_writeChannels_rlWriteDone' shadows the effects
of tile_0_lDdr3Test_we_rv_writeChannels_store_cmd' when they execute in the same clock cycle. Affected method calls: tile_0_lDdr3Test_we_rv_writeChannels_clientInFlight.write To silence this warning, use the-no-warn-action-shadowing' flag.
Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117)
Rule tile_0_lDdr3Test_we_rv_writeChannels_load_ctxt_b' shadows the effects oftile_0_lDdr3Test_we_rv_writeChannels_store_cmd' when they execute in the
same clock cycle. Affected method calls:
tile_0_lDdr3Test_we_rv_writeChannels_clientBursts.write,
tile_0_lDdr3Test_we_rv_writeChannels_clientStart.write
To silence this warning, use the -no-warn-action-shadowing' flag. Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117) Ruletile_0_lDdr3Test_we_rv_rl_writeDone' shadows the effects of
tile_0_lDdr3Test_we_rv_rl_arbitration' when they execute in the same clock cycle. Affected method calls: tile_0_lDdr3Test_we_rv_reqNotDone.write To silence this warning, use the-no-warn-action-shadowing' flag.
Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117)
Rule tile_0_lDdr3Test_rl_read_start' shadows the effects of tile_0_lDdr3Test_rl_req_ar' when they execute in the same clock cycle.
Affected method calls:
tile_0_lDdr3Test_dramReadOffset.write
To silence this warning, use the -no-warn-action-shadowing' flag. Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117) Ruletile_0_lDdr3Test_rl_write_start' shadows the effects of
tile_0_lDdr3Test_rl_req_aw' when they execute in the same clock cycle. Affected method calls: tile_0_lDdr3Test_dramWriteOffset.write To silence this warning, use the-no-warn-action-shadowing' flag.

Pietro

[pietrobressana]

The error that causes the build to stop is below. The other issues are warnings. I'm not sure if they can be safely ignored.

Warning: "/home/netfpga/gitHub/connectal/bsv/PcieTop.bsv", line 76, column 8: (G0117)
Rule tile_0_lDdr3Test_rl_write_start' shadows the effects of tile_0_lDdr3Test_rl_req_aw' when they execute in the same clock cycle.
Affected method calls:
tile_0_lDdr3Test_dramWriteOffset.write
To silence this warning, use the `-no-warn-action-shadowing' flag.
Schedule dump file created: /home/netfpga/gitHub/connectal/tests/ddr3/nfsume/obj/mkPcieTop.sched
Verilog file created: /home/netfpga/gitHub/connectal/tests/ddr3/nfsume/verilog/mkPcieTop.v
Error: XCI file /home/netfpga/gitHub/connectal/out/nfsume/axiddr3/axiddr3.xci not found
make[1]: *** [fpgamake.mk] Error 255
make: *** [build.nfsume] Error 2

[pietrobressana]

There is another issue. The BOARD variable in the makefile doesn't seem to get set correctly. You can see in the first invocation that the grep is looking for "".json.

$ make clean
grep: /home/netfpga/gitHub/connectal/boardinfo/.json: No such file or directory
make: *** No rule to make target clean'. Stop. netfpga@node98:~/gitHub/connectal/tests/ddr3$ BOARD=nfsume make clean make: *** No rule to make targetclean'. Stop.

[hanw]

instead of doing make clean, you can just delete the nfsume directory.

[hanw]

This error: XCI file /home/netfpga/gitHub/connectal/out/nfsume/axiddr3/axiddr3.xci not found
can be resolved by running this command

make ip.nfsume

[robertsoule]

@hanw, the issue is not that the clean target doesn't exist. It is that for any makefile target, the BOARD variable isn't getting set correctly. So, replace 'make clean' with 'make build' or any other command, and you'll get the same error related to grep.

[robertsoule]

@hanw @jameyhicks : we tried to run make ip.nfsume, but the target is not in the Makefile. Perhaps it was removed when the Makefile was refactored a few days ago?

[hanw]

can you try make build.nfsume ? BOARD is set by the suffix.

[robertsoule]

@hanw, sorry, you're right. Perhaps the reason we didn't see BOARD set is because we didn't have a target for ip.nfsume. So, the two issues are from the same problem.

[jameyhicks]

I removed ip.nfsume in a recent refactor. Unfortunately, I attached the command to build the XCI files to the wrong rule. Commit c7d5b82 fixes that problem.