Added support for Mellanox Innova-2 SmartNIC #1667

doihead · 2023-11-10T19:50:42Z

Adds support for the Mellanox Innova-2 SmartNIC, an FPGA accelerated 25 Gb NIC which has become EOL and is now readily available on Ebay for ~140 USD. These cards contain an XCKU15P, a Xilinx UltraScale+ FPGA with 1143450 logic cells, around half of an Alveo U200 and 8 gigabytes of DDR4-2666.

Related PRs / Issues

None

UI / API Impact

Adds example mellanox config to example config_build_recipies.yaml

Verilog / AGFI Compatibility

None

Contributor Checklist

Is this PR's title suitable for inclusion in the changelog and have you added a changelog:<topic> label?
Did you add Scaladoc/docstring/doxygen to every public function/method?
Did you add at least one test demonstrating the PR?
Did you delete any extraneous prints/debugging code?
Did you state the UI / API impact?
Did you specify the Verilog / AGFI compatibility impact?

If applicable, did you regenerate and publicly share default AGFIs?

If applicable, did you apply the ci:fpga-deploy label?

If applicable, did you apply the Please Backport label?

Reviewer Checklist (only modified by reviewer)

Note: to run CI on PRs from forks, comment @Mergifyio copy main and manage the change from the new PR.

Is the title suitable for inclusion in the changelog and does the PR have a changelog:<topic> label?
Did you mark the proper release milestone?
Did you check whether all relevant Contributor checkboxes have been checked?

abejgonzalez

This is great! I'm surprised it was so easy to port!

abejgonzalez · 2023-11-15T02:52:44Z

deploy/sample-backup-configs/sample_config_build_recipes.yaml

+    TARGET_PROJECT: firesim
+    DESIGN: FireSim
+    TARGET_CONFIG: FireSimRocketConfig
+    PLATFORM_CONFIG: BaseXilinxAlveoConfig


In #1669 this PLATFORM_CONFIG is not shared between FPGA types. I think that is a right approach. Can you create a copy of it and rename it to something like BaseMellanoxInnova2Config?

abejgonzalez · 2023-11-15T02:53:34Z

deploy/sample-backup-configs/sample_config_build_recipes.yaml

+    deploy_quintuplet: null
+    platform_config_args:
+        fpga_frequency: 60
+        build_strategy: TIMING


This is unsupported right? I would just put UNSUPPORTED for now to make it clear this does nothing (other cfgs should do the same thing - i.e the U250 ones but we haven't done that).

abejgonzalez · 2023-11-15T02:54:17Z

platforms/mellanox_innova_2/cl_firesim/design/axi_tieoff_master.v

Is this a copy of the other AXI tieoff? Dedup?

This isn't quite an exact copy, it's got a changed AXI bus frequency since my implementation of the memory controller has different memory timings.

abejgonzalez · 2023-11-15T02:56:33Z

platforms/mellanox_innova_2/cl_firesim/scripts/implementation_easy.tcl

+
+file copy -force ${root_dir}/vivado_proj/firesim.runs/${impl_run}/design_1_wrapper.bit ${root_dir}/vivado_proj/firesim.bit
+
+write_cfgmem -force -format mcs -interface SPIx4 -size 1024 -loadbit "up 0x01002000 ${root_dir}/vivado_proj/firesim.bit" -verbose  ${root_dir}/vivado_proj/firesim.mcs


Looks like some of this doesn't match across implementation files. Can you just delete this file (there should only be the implementation_<version>.tcl files anyways.

abejgonzalez · 2023-11-15T02:56:46Z

platforms/mellanox_innova_2/cl_firesim/scripts/implementation_ml_2021.1.tcl

Ditto on deleting this.

abejgonzalez · 2023-11-15T03:00:51Z

platforms/mellanox_innova_2/cl_firesim/scripts/synthesis.tcl

Can you simlink all files that are shared with the U250 flow?

abejgonzalez · 2023-11-15T03:01:33Z

sim/make/driver.mk

+# Compile Driver
+$(mellanox_innova_2): export CXXFLAGS := $(CXXFLAGS) $(common_cxx_flags) $(DRIVER_CXXOPTS) \
+              -idirafter ${CONDA_PREFIX}/include -idirafter /usr/include
+$(mellanox_innova_2): export LDFLAGS := $(LDFLAGS) $(common_ld_flags) -Wl,-rpath='$$$$ORIGIN' \
+              -L${CONDA_PREFIX}/lib -Wl,-rpath-link=/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu
+
+$(mellanox_innova_2): $(header) $(DRIVER_CC) $(DRIVER_H) $(midas_cc) $(midas_h)
+	mkdir -p $(OUTPUT_DIR)/build
+	cp $(header) $(OUTPUT_DIR)/build/
+	$(MAKE) -C $(simif_dir) driver MAIN=$(PLATFORM) PLATFORM=$(PLATFORM) \
+		DRIVER_NAME=$(DESIGN) \
+		GEN_FILE_BASENAME=$(BASE_FILE_NAME) \
+		GEN_DIR=$(OUTPUT_DIR)/build \
+		OUT_DIR=$(OUTPUT_DIR) \
+		DRIVER="$(DRIVER_CC)" \
+		TOP_DIR=$(chipyard_dir)


Tabbing is different from the rest of the file.

abejgonzalez · 2023-11-15T03:03:45Z

platforms/mellanox_innova_2/cl_firesim/scripts/implementation_2021.1.tcl

+
+file copy -force ${root_dir}/vivado_proj/firesim.runs/${impl_run}/design_1_wrapper.bit ${root_dir}/vivado_proj/firesim.bit
+
+write_cfgmem -force -format mcs -interface SPIx8 -size 128 -loadbit "up 0x0 ${root_dir}/vivado_proj/firesim.bit" -verbose  ${root_dir}/vivado_proj/firesim.bin


Call this MCS (just to make sure this should be format mcs not bin right)?

abejgonzalez · 2023-11-15T03:04:09Z

deploy/buildtools/bitbuilder.py

+            if self.BOARD_NAME == "mellanox_innova_2":
+                local(f"cp {local_cl_dir}/vivado_proj/firesim*.mcs {tar_staging_path}")


Why is this needed? Shouldn't the MCS file be called firesim.mcs?

This is needed because the mellanox uses two QSPI chips in parallel so you end up with a firesim_primary.mcs and a firesim_secondary.mcs. The VCU118 looks like it has a similar situation, but that code still expects that the first MCS file is firesim.mcs, hence the bit of hack

abejgonzalez · 2023-11-15T03:05:25Z

Also if you could add a description to the PR that would be great (you can see examples in other PRs).

abejgonzalez · 2023-11-17T18:31:01Z

Also if this can be rebased on the most recent main that would be great (there are a lot of flow improvements to the Alveo flow that apply here).

…ganization

…duplicated AXI Tieoff since mellanox is now using same DRAM Frequency as Alveo

doihead added 2 commits November 10, 2023 01:14

Initial Commit of Mellanox Innova-2 Flex board support files

8f568ed

Added example mellanox config to the sample build recipies file

04be08e

doihead marked this pull request as draft November 10, 2023 19:55

doihead marked this pull request as ready for review November 10, 2023 19:56

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC~~ Added support for Mellanox Innova-2 SmartNIC changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC changelog:added~~ Added support for Mellanox Innova-2 SmartNIC label:changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC label:changelog:added~~ Added support for Mellanox Innova-2 SmartNIC :label changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC :label changelog:added~~ Added support for Mellanox Innova-2 SmartNIC :changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC :changelog:added~~ Added support for Mellanox Innova-2 SmartNIC changelog:<added> Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC changelog:<added>~~ Added support for Mellanox Innova-2 SmartNIC changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC changelog:added~~ Added support for Mellanox Innova-2 SmartNIC :changelog:added Nov 10, 2023

doihead changed the title ~~Added support for Mellanox Innova-2 SmartNIC :changelog:added~~ Added support for Mellanox Innova-2 SmartNIC Nov 10, 2023

abejgonzalez requested changes Nov 15, 2023

View reviewed changes

abejgonzalez added the changelog:added Put PR title in 'Added' section of changelog label Nov 15, 2023

abejgonzalez assigned doihead Nov 15, 2023

abejgonzalez requested a review from sagark November 15, 2023 03:04

abejgonzalez added 12 commits December 4, 2023 02:57

Fix no keys in share_with_accounts

a633d47

Initial support for 4 DRAM channels on U250 + Reorg.

eeb7233

Fix platform configs

798a75a

Reorg even more | Move FireSim out of BD

f2ce661

Small fixes

bd1c1b4

Small fixes

704cc5e

More fixes

c6741b6

Fix U200

dd5b9a7

Fix U280

9421b22

Dedup post_impl.tcl

922ef2c

Remove extra create_generated_clock

2a41896

Fix proc spacing

3b2bfc1

abejgonzalez and others added 12 commits December 4, 2023 02:57

Fix circuit paths | Remove extra firesim_wrapper for U250

9ee5802

Add host_clock definition to U2*0

6c630fb

Fix VCU118 XDCs

d7c96e4

Statically hint URAMs for L2 | Small cleanup

1d9272c

Update .gitignore

fc0aec4

Separate local v.s. AWS CI

17f5318

Rename local -> non-aws

b8af8ee

Update local bitstream(s) [ci skip]

68f0233

Move BRAMQueue to targetutils

c618feb

Rebased Mellanox Configuration on latest main to reflect new Alveo or…

48efab0

…ganization

Fixed git issues and added in symlinks

4aaa3b6

Tightened up memory timings to run at 2400MT instead of 1833MT and de…

57f95c7

…duplicated AXI Tieoff since mellanox is now using same DRAM Frequency as Alveo

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for Mellanox Innova-2 SmartNIC #1667

Added support for Mellanox Innova-2 SmartNIC #1667

doihead commented Nov 10, 2023 •

edited

abejgonzalez left a comment

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

doihead Dec 4, 2023 •

edited

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

abejgonzalez Nov 15, 2023

doihead Dec 4, 2023

abejgonzalez commented Nov 15, 2023

abejgonzalez commented Nov 17, 2023


		file copy -force ${root_dir}/vivado_proj/firesim.runs/${impl_run}/design_1_wrapper.bit ${root_dir}/vivado_proj/firesim.bit

		write_cfgmem -force -format mcs -interface SPIx4 -size 1024 -loadbit "up 0x01002000 ${root_dir}/vivado_proj/firesim.bit" -verbose ${root_dir}/vivado_proj/firesim.mcs

		if self.BOARD_NAME == "mellanox_innova_2":
		local(f"cp {local_cl_dir}/vivado_proj/firesim*.mcs {tar_staging_path}")

Added support for Mellanox Innova-2 SmartNIC #1667

Are you sure you want to change the base?

Added support for Mellanox Innova-2 SmartNIC #1667

Conversation

doihead commented Nov 10, 2023 • edited

Related PRs / Issues

UI / API Impact

Verilog / AGFI Compatibility

Contributor Checklist

Reviewer Checklist (only modified by reviewer)

abejgonzalez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doihead Dec 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abejgonzalez commented Nov 15, 2023

abejgonzalez commented Nov 17, 2023

doihead commented Nov 10, 2023 •

edited

doihead Dec 4, 2023 •

edited