Verilator generating incorrect instructions

[alistair23]

I'm seeing failures when running Tock on the latest-ish (c83a36c) Verilator build.

When setting the PMP configuration there is a code segment like this:

2001571c <_ZN5riscv3csr3CSR13pmpconfig_get17h60efc285dd598fa6E>:
2001571c:       45bd                    li      a1,15
2001571e:       06a5ea63                bltu    a1,a0,20015792 <.LBB486_17+0x6>
20015722:       050a                    slli    a0,a0,0x2
20015724:       200165b7                lui     a1,0x20016
20015728:       47458593                addi    a1,a1,1140 # 20016474 <.LJTI486_0>
2001572c:       952e                    add     a0,a0,a1
2001572e:       4108                    lw      a0,0(a0)
20015730:       8502                    jr      a0

Looking at the Verilator instruction log, I can see that the first 31 times this is run it looks like this:

573302	    256699	2001571c	45bd	c.li	x11,15	 x11=0x0000000f
573306	    256701	2001571e	06a5ea63	bltu	x11,x10,20015792	 x11:0x0000000f x10:0x00000000
573308	    256702	20015722	050a	c.slli	x10,0x2	 x10:0x00000000 x10=0x00000000
573310	    256703	20015724	200165b7	lui	x11,0x20016	 x11=0x20016000
573312	    256704	20015728	47458593	addi	x11,x11,1140	 x11:0x20016000 x11=0x20016474
573314	    256705	2001572c	952e	c.add	x10,x11	 x10:0x00000000 x11:0x20016474 x10=0x20016474
573326	    256711	2001572e	4108	c.lw	x10,0(x10)	 x10:0x20016474 x10=0x20015732 PA:0x20016474 store:0x00000000

Then on the 32nd time I see this

578718	    259407	2001571c	45bd	c.li	x11,15	 x11=0x0000000f
578722	    259409	2001571e	06a5ea63	bltu	x11,x10,20015792	 x11:0x0000000f x10:0x00000003
578724	    259410	20015722	050a	c.slli	x10,0x2	 x10:0x00000003 x10=0x0000000c
578726	    259411	20015724	200165b7	lui	x11,0x20016	 x11=0x20016000
578728	    259412	20015728	47458593	addi	x11,x11,1140	 x11:0x20016000 x11=0x20016474
578730	    259413	2001572c	952e	c.add	x10,x11	 x10:0x0000000c x11:0x20016474 x10=0x20016480
578734	    259415	2001572e	4108	c.lw	x0,0(x10)	 x10:0x20016480  x0=0x00000000 PA:0x20016480 store:0x00000000

Notice that the instruction decoding at address 0x2001572e changes from c.lw x10,0(x10) to c.lw x0,0(x10) which generates a fault and the next instruction is the trap handler

578752	    259424	20000400	1000006f	jal	x0,20000500	  x0=0x00000000

It doesn't look like the instruction in memory has changed, it's always 0x4108, just the decoding has changed. The failure seems to be related to enabling ePMP and the MML bit as well, although the area in memory is accessible

[alistair23]

I see this for a range of other instructions as well, for example

200195e4:       00048a63                beqz    s1,200195f8 <main+0x1d64>

becomes

664080	    302088	200195e4	00048a63	beq	x0,x0,200195f8	  x0:0x00000000  x0:0x00000000

When run on Verilator

and

20025298:       00008067                ret

becomes

666606     303351  20025298    00008067    jalr    x0,0(x0)      x0:0x00000000  x0=0x00000000

but only if I make PMP changes, otherwise it's

666612	    303354	20025298	00008067	jalr	x0,0(x1)	  x1:0x200195e0  x0=0x00000000

[tjaychen]

@arunthomas not sure how we should tag this issue.

[alistair23]

It seems related to ePMP. It seems to happen after changing some ePMP addresses, even if the instructions are still in the allowed read/write/exec region by the new PMP configs.

I also see similar failures on the FPGA, so it doesn't seem to be completely isolated to Verilator.

[arunthomas]

@cfrantz was planning to look into the Tock ePMP issues

[msfschaffner]

Is this bug still present?

[msfschaffner]

CC @GregAC

[GregAC]

I believe this is a flaw in the way our instruction tracer works. It uses the RVFI interface to observe retired instructions and when one sees an exception we prevent it from writing to the register file, which on the RVFI interface just means setting the write register to 0.

Hence in this case with the faulting load it reports as a load to x0 rather than x10. So no instruction memory changes or other faults in instruction memory read happening here, just an awkward instruction trace. I'll file an issue in Ibex to note we should improve it but it's not a high priority

[GregAC]

Ibex issue here: lowRISC/ibex#2099 and closing this issue