Use hierarchical namespaces in name resolution #1415

sezna · 2024-04-23T16:53:55Z

This PR replaces #1364

This PR introduces a feature called hierarchical namespaces, a module-like system that maintains backward compatibility. It is the foundational groundwork that will pave the way for:

Naked namespaces (files without namespace declarations)
Package publishing and sharing
Explicit item imports
Exports and re-exports
and the rest of the stuff here

Key changes include:

VecIdent type introduction: This abstraction over Vec<Ident> represents the new namespaces in the AST. Alternate names could be DotIdent or Path, but those names were already loaded with meaning in our codebase. I'm open to feedback on nomenclature here.
NamespaceId usage: This replaces our previous flat map-style Rc<str> -> Res item tracking.
NamespaceTree: This tracks the namespace hierarchy separately in low-cost data structure, and allows us to keep our basic tys and terms maps relatively untouched.
Name resolution algorithm update: The algorithm in resolve.rs received a heavy facelift. Not only did I try to simplify the existing code, but I also added the required functionality to check candidate partially-opened or re-opened namespaces.

This PR maintains 100% backward compatibility with the existing resolution engine, preserving features like re-opening namespaces, shadowing semantics, merging namespaces via aliases, and absolute-path opens. It also introduces the ability to reference a function in a different namespace if its parent namespace is opened.

Additional notes:

Katas used a form of shadowing that wasn't tested in our compiler. This PR adds a test for that.
Pursuing Use debug printing for expect_test in the parser and other places. #1321 could help differentiate user-facing and compiler-engineer-facing string representations of structs, which are increasingly conflated.
There's potential for refactoring ast, fir, and hir to use shared generic types, reducing duplicate Ident/etc types.
Completions functionality remains the same, but it's not yet aware of the ability to partially open namespaces (e.g. open Microsoft.Quantum. This will be addressed in future updates.
There is a performance regression, but we are performing a full tree-traversal namespace resolution algorithm in a place where we used to just do a constant table lookup. I have optimized this as much as I believe is reasonable for now, bringing the regression down from ~40%, where it was originally, to around 5-10% based on the benchmark run.

Also closes #1336

compiler/qsc_data_structures/src/index_map.rs

language_service/src/completion.rs

compiler/qsc_frontend/src/resolve.rs

minestarks

Looked at all the files. Will revisit once you've had a chance to look at the comments.

github-actions · 2024-05-09T18:24:38Z

Change in memory usage detected by benchmark.

Memory Report for `aee80eb`

Test	This Branch	On Main	Difference
compile core + standard lib	16584781 bytes	15917505 bytes	667276 bytes

github-actions · 2024-05-09T18:35:32Z

Benchmark for `aee80eb`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	342.9±2.46µs	348.3±2.06µs	+1.57%
Array literal evaluation	196.2±14.32µs	198.9±1.64µs	+1.38%
Array update evaluation	425.0±3.32µs	431.6±2.07µs	+1.55%
Core + Standard library compilation	16.9±0.20ms	18.7±0.59ms	+10.65%
Deutsch-Jozsa evaluation	5.1±0.05ms	5.1±0.15ms	0.00%
Large file parity evaluation	34.5±0.13ms	34.5±0.15ms	0.00%
Large input file compilation	11.4±0.22ms	12.0±0.14ms	+5.26%
Large input file compilation (interpreter)	45.2±1.24ms	46.5±1.21ms	+2.88%
Large nested iteration	33.2±0.15ms	34.2±0.68ms	+3.01%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1548.3±33.11µs	1534.3±33.32µs	-0.90%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.7±0.10ms	7.6±0.09ms	-1.30%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1418.5±42.46µs	1405.5±49.23µs	-0.92%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.0±0.17ms	27.5±0.82ms	+1.85%
Teleport evaluation	88.4±3.68µs	88.6±4.22µs	+0.23%

github-actions · 2024-05-09T21:05:31Z

Change in memory usage detected by benchmark.

Memory Report for `a5efff9`

Test	This Branch	On Main	Difference
compile core + standard lib	16584781 bytes	15917505 bytes	667276 bytes

github-actions · 2024-05-09T21:16:17Z

Benchmark for `a5efff9`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	340.0±1.32µs	365.5±2.78µs	+7.50%
Array literal evaluation	180.0±2.43µs	179.9±1.51µs	-0.06%
Array update evaluation	420.8±0.79µs	448.1±2.11µs	+6.49%
Core + Standard library compilation	16.9±0.14ms	18.0±0.10ms	+6.51%
Deutsch-Jozsa evaluation	5.1±0.04ms	5.2±0.08ms	+1.96%
Large file parity evaluation	33.9±0.11ms	34.0±0.70ms	+0.29%
Large input file compilation	11.8±0.45ms	12.1±0.08ms	+2.54%
Large input file compilation (interpreter)	46.8±1.70ms	45.5±0.47ms	-2.78%
Large nested iteration	33.2±0.22ms	36.3±1.00ms	+9.34%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1555.0±50.74µs	1529.1±34.31µs	-1.67%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.7±0.09ms	7.6±0.08ms	-1.30%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1411.7±29.34µs	1395.9±26.73µs	-1.12%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.1±0.18ms	27.4±0.15ms	+1.11%
Teleport evaluation	88.9±3.45µs	89.5±3.77µs	+0.67%

github-actions · 2024-05-10T01:03:00Z

Change in memory usage detected by benchmark.

Memory Report for `a368e9e`

Test	This Branch	On Main	Difference
compile core + standard lib	16584781 bytes	15917505 bytes	667276 bytes

github-actions · 2024-05-10T01:14:17Z

Benchmark for `a368e9e`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	347.0±5.12µs	338.6±1.55µs	-2.42%
Array literal evaluation	199.0±1.44µs	179.9±0.87µs	-9.60%
Array update evaluation	433.0±8.53µs	421.5±2.03µs	-2.66%
Core + Standard library compilation	18.5±0.70ms	18.2±0.38ms	-1.62%
Deutsch-Jozsa evaluation	5.0±0.05ms	5.1±0.05ms	+2.00%
Large file parity evaluation	33.8±0.12ms	34.0±0.40ms	+0.59%
Large input file compilation	11.8±0.28ms	12.4±0.25ms	+5.08%
Large input file compilation (interpreter)	48.9±1.15ms	49.3±0.98ms	+0.82%
Large nested iteration	33.5±0.20ms	32.9±0.22ms	-1.79%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1551.9±27.87µs	1529.4±30.38µs	-1.45%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.7±0.16ms	7.6±0.08ms	-1.30%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1434.5±66.77µs	1409.0±68.83µs	-1.78%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.7±0.40ms	27.5±0.19ms	-0.72%
Teleport evaluation	87.3±3.73µs	89.3±4.61µs	+2.29%

minestarks

Thank you!

github-actions · 2024-05-13T18:32:46Z

Change in memory usage detected by benchmark.

Memory Report for `828a194`

Test	This Branch	On Main	Difference
compile core + standard lib	16583445 bytes	15916185 bytes	667260 bytes

github-actions · 2024-05-13T18:43:20Z

Benchmark for `828a194`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	339.5±1.94µs	347.4±2.53µs	+2.33%
Array literal evaluation	182.3±3.22µs	202.7±4.20µs	+11.19%
Array update evaluation	423.8±1.30µs	430.2±6.69µs	+1.51%
Core + Standard library compilation	16.7±0.05ms	17.8±0.07ms	+6.59%
Deutsch-Jozsa evaluation	5.0±0.05ms	5.1±0.31ms	+2.00%
Large file parity evaluation	33.6±0.10ms	33.7±0.54ms	+0.30%
Large input file compilation	11.2±0.07ms	12.0±0.25ms	+7.14%
Large input file compilation (interpreter)	43.3±0.80ms	44.8±0.55ms	+3.46%
Large nested iteration	33.3±0.38ms	34.2±1.13ms	+2.70%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1535.3±32.03µs	1526.3±41.48µs	-0.59%
Perform Runtime Capabilities Analysis (RCA) on large file sample	7.6±0.07ms	7.5±0.05ms	-1.32%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1405.6±31.23µs	1399.9±79.49µs	-0.41%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.0±0.50ms	27.4±0.37ms	+1.48%
Teleport evaluation	87.5±3.43µs	87.8±3.96µs	+0.34%

compiler/qsc_data_structures/src/namespaces.rs

github-actions · 2024-05-17T17:07:06Z

Change in memory usage detected by benchmark.

Memory Report for `9cf3387`

Test	This Branch	On Main	Difference
compile core + standard lib	16582941 bytes	15915681 bytes	667260 bytes

github-actions · 2024-05-17T17:17:41Z

Benchmark for `9cf3387`

Click to view benchmark

Test	Base	PR	%
Array append evaluation	329.9±5.57µs	329.4±2.21µs	-0.15%
Array literal evaluation	182.5±11.38µs	175.0±4.60µs	-4.11%
Array update evaluation	409.2±3.49µs	412.7±1.80µs	+0.86%
Core + Standard library compilation	17.6±0.44ms	19.5±0.81ms	+10.80%
Deutsch-Jozsa evaluation	5.0±0.05ms	5.1±0.05ms	+2.00%
Large file parity evaluation	34.1±0.07ms	34.2±0.30ms	+0.29%
Large input file compilation	12.1±0.29ms	12.6±0.84ms	+4.13%
Large input file compilation (interpreter)	47.0±1.79ms	48.8±0.98ms	+3.83%
Large nested iteration	32.2±0.20ms	32.7±0.21ms	+1.55%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample	1577.7±41.28µs	1564.3±40.03µs	-0.85%
Perform Runtime Capabilities Analysis (RCA) on large file sample	8.2±0.12ms	7.7±0.09ms	-6.10%
Perform Runtime Capabilities Analysis (RCA) on teleport sample	1463.1±148.53µs	1446.0±96.06µs	-1.17%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries	27.3±0.23ms	27.6±0.23ms	+1.10%
Teleport evaluation	89.1±6.53µs	89.9±4.89µs	+0.90%

sezna added 30 commits March 14, 2024 15:46

wip

016153c

wip

b8a801e

wip

a83a1e9

wip

5d6a33a

wip

e5ec193

it builds??

a18ef5d

wip

a041e58

wip: add nested namespace structure

4a644ef

Switch to using new namespace API

ac8745d

doc comment

bf6561b

start working on converting HIR to new structure

c571f2e

wip: refactor core namespaces into namespace ids

21fac31

wip

f52ed17

wip

e6c15ba

wip

2b1e51d

Refactor new namespace work into qsc_data_structures

422542b

add more namespace ids

0b4069a

Merge branch 'main' of github.com:microsoft/qsharp into alex/hns

b2c0f2f

wip: continue refactor of namespaceids

38327a9

update methods for namespace tree node

261aa8a

finish lining up types for completion lists

3e6d39d

fmT

71e3ddd

introduce vecident to qsc_fir

dd1e312

lower VecIdent for fir and hir

51eefa1

remove todos from name locator and hover in lang service

2187506

include prelude namespace IDs in default opens

50ddf45

refactor opens to use string identifiers for namespaces instead of ids

ee532c0

remove todos in completions

8713435

implement get in name resolution

0bcd526

continue work on resolve.rs

a78fe73