Async diagnostics analyzer work queue

[neoGeneva]

Async Worker Queue Changes

Included in this pull requests is some changes to how the worker queue works for the CSharpDiagnosticWorkerWithAnalyzers. The motivation for this work is a couple of performance related issues with the current queue, that requires a new approach.

The main issues I aim to resolve are:

1. The current queue can block on a single file, and wont utilize its additional threads. So if a file changes, the current queue will dequeue that item, and then any files added while that one is processing wont dequeue until that initial file has processed, and this can be a significant issue if the file has a long processing time. For example, if in VSCode I make a change to a large file, then change tabs to a smaller one I wont see changes in the small one for a long time.
2. The current queue will continue processing a file even its changes are now out of date. When a couple of changes to a file are made in rapid succession it has to wait for the first changes to finish processing (for the same reason as above) before it can process the new changes. For example, if a file takes 20 seconds to process, and I make two changes in a short time, it'll be 40 seconds before I see the result of the second change.
3. Background work can execute before foreground work. Because the foreground and background threads run in parallel it means foreground work is competing for CPU with background work, this means if there's a lot of work in both queues then some foreground work will execute after background work.
4. There's also a couple of other minor performance issues here. Like it consumes a small amount of CPU while not processing any items, it can generate a lot of unneeded tasks, there's an unneeded delay between batches.

My fixes for these issues are:

1. The queue no longer works in batches, but dequeues a single file at a time. Then in the CSharpDiagnosticWorkerWithAnalyzers there are N threads (75% of cores by default) that loop dequeuing items from the queue. This means items will be processed as soon as there is capacity.
2. The queue now cancels running work if a item is re-enqueued because each active work item now has an associated CancellationTokenSource.
3. Foreground work is new always dequeued before any background work, so it'll always take priority.
4. Items are now dequeued asynchronously as soon as they're available so there's no need to spin or delay while waiting for new tasks.

Additional Notes:

• I removed the throttling concept from the queue, I don't think it's needed with the amount of threads being used to manage CPU load.
• Part of the motivation of this is some additional work I've been working on that emits critical diagnostics earlier, but issue number 2 above was a blocking issue.
• I ran some tests looking at start up time and CPU, both metrics remain unchanged, which is good as things change is about smarter processing of items rather than reducing CPU.

[JoeRobich]

@neoGeneva Thanks for this enhancement. Just a few nits and suggestions. We should be able to get this in for the next release.

[neoGeneva]

Hey @JoeRobich, thanks for reviewing these changes! I've been working on a separate version of this pull request locally, but it's been a while and I want to make sure that any bug fixes I might have are included this pull too, so once I've done that (and the changes you've requested here), I'll update my branch and let you know, probably later this week.

[JoeRobich]

Hi @neoGeneva, I am hoping to get a new release together next week and wanted to include these changes. Would it be ok if I updated this PR with my suggestions and take it as is?

[neoGeneva]

Hey @JoeRobich, I started to have a look but haven't had a chance to reply sorry! I didn't have any changes from my other branches, but I did notice that the wrong cancellation token is used in a couple of places after a merge from main (should be using the combined one.)

[JoeRobich]

I did notice that the wrong cancellation token is used in a couple of places after a merge from main

Oops. I get that fixed up.

Thanks @neoGeneva!

[JoeRobich]

Thanks @neoGeneva!