64 bit key allocation

[borland]

For Octopus internal developers: There is an RFC for this here, which has received positive feedback and a green light to move ahead

Background

Nevermore's Key Allocation logic uses 32 bit integers. This means if we ever tried to allocate an ID greater than 2,147,483,647, an OverflowException will be thrown from within Nevermore and the operation will fail.

Because the nevermore key allocator allocates ID's in blocks of (by default) 20, this could happen before the actual table itself gets 2 billion rows. Alternatively, some tables are transactional in nature, where rows may be inserted/deleted rapidly, causing the ID to reach 2 billion even if there are only a small number of rows at any given time.

Best practice for integer-based database ID's is to use 64-bit rather than 32-bit integers. This corresponds to the bigint type in SQL server

Results

This PR updates the KeyAllocator and IKeyAllocator to work in terms of 64 bit values (long in C#), enabling the use of bigint in the underlying KeyAllocation SQL database table.

The C# code has been deliberately written to work with either the prior 32 bit KeyAllocation table, or an updated 64 bit table. This means users of Nevermore can upgrade to this version without requiring any database/schema migrations. They can then (if they would like to) alter their KeyAllocation table to be 64 bit at a future moment of their choosing.

Public API Breaking Change

While this is deliberately not breaking at the database level, the IKeyAllocator interface, KeyAllocator class, and StringPrimaryKeyHandler class have all changed from int to long. These are public types and if you use them you will need to update your code accordingly. It should be a small and simple change.

How to enable 64 bit key allocation

Once upgrading to this version of Nevermore, you will then need to create a database migration to change your KeyAllocation table, and the GetNextKeyBlock stored procedure.

It will likely look something like this:

UPDATE [dbo].[KeyAllocation] ALTER COLUMN [Allocated] bigint not null;

IF EXISTS (SELECT NULL FROM sys.procedures WHERE name = 'GetNextKeyBlock' AND SCHEMA_NAME(schema_id) = 'dbo')
	DROP PROCEDURE dbo.GetNextKeyBlock
GO

CREATE PROCEDURE dbo.GetNextKeyBlock
(
	@collectionName nvarchar(50),
	@blockSize int
)
AS
BEGIN
	SET NOCOUNT ON
	DECLARE @result bigint

	UPDATE KeyAllocation
		SET @result = Allocated = (Allocated + @blockSize)
		WHERE CollectionName = @collectionName

	if (@@ROWCOUNT = 0)
	begin
		INSERT INTO KeyAllocation (CollectionName, Allocated) values (@collectionName, @blockSize)
		SELECT CAST(@blockSize as bigint) -- type must be the same as @result
	end

	SELECT @result
END

[nick-roscarel-octo]

I'm a bit confused about the extra rules here. Was this intentional? (ditto for Nevermore.sln.DotSettings)

[borland]

The DotSettings file is owned by Rider.
Every so often, a new version of Rider invents some new settings and it decides it wants to write them into these DotSettings files.
I thought that was what was happening here, so I didn't really pay it much attention.

On second look though, I see it has "UserSettings" which means it isn't that. I'll back these out of the branch, thanks

[nick-roscarel-octo]

LGTM otherwise, nice stuff!

[nick-roscarel-octo]

nit: is there any particular reason to go for casting rather than changing the type of @blockSize itself?

[borland]

yeah, blocksize is an input parameter to the stored procedure. Changing it to bigint would likely require callers in C# to change their code from int to long and be a breaking change.

We can do breaking changes if we need to, but in that case the block size doesn't need to exceed 2 billion; 32 bit int is perfectly good there, so it's less work for everyone to leave that be. This cast is the tradeoff 😄