Fix Performance Issue for filtering on varchar columns in MS_SQL database

[abhishekkumams]

Why make this change?

• Closes MSSQL API slow with large tables due to incorrect varchar/nvarchar in sp_executesql #1973
  • Slow response of DAB when filtering on varchar columns in MS_SQL compared to other DBs.
  • Same query in mssql taking around ~1sec, compared to <50ms in other DBs for large number of rows, eg. 20 million
  • This was happening because the query generated by DAB when runs on Sql Server for varchar columns, it automatically assumes the filter value of type nvarchar, due to which it does implicit cast leading to slowness.
  • To fix this we needed to identify if the filter is for varchar column, we explitily specify it to be varchar for faster response.

What is this change?

• We are modifying the DbCommand and adding extra information that is sent to SqlClient.

• We introduced a new property called SqlDbType, By Default for String DbType, SqlDbType was considered as nvarchar. now we are updating this property to the correct sql type.

• This allows db to directly execute the command without type casting due to type mismatch.

• Carved out the code which creates DbCommand in QueryExecuter, and added an override method in MsSqlQueryExecuter. here we set the property SqlDbType for the Parameters.

  • Link to Discussion which talks about this highlighting the issue : SWA hosted DAB API response time (performance) #1964 (comment)
  • Added a new property SqlDbType in Column class, currently we used to have only DotNet DbType which doesn't differenciate between varchar and Nvarchar. So, adding this SqlDbType gave us the freedom to modify query based on the sql db type.

How was this tested?

• Since, this is a performance issue and only a new column got introduced. The main aim here was to make sure excising functionality doesn't break.
• Added few more extra tests.

The performance difference is more visible when there are large number of rows. So, I added 20 million records and ran perf test.
Below are the performance results with avg response time:

	varchar(before)	nvarchar(before
GraphQL	~734ms	~23ms
REST	~655ms	~24ms

	varchar(after)	nvarchar(after)
GraphQL	~18ms	~23ms
REST	~18ms	~25ms

BEFORE CHANGE

GRAPHQL

VARCHAR

NVARCHAR

REST

VARCHAR

NVARCHAR

AFTER CHANGE

GRAPHQL

VARCHAR

NVARCHAR

REST

VARCHAR

NVARCHAR

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[seantleonard]

Based on your screenshots, it seems like "varchar" variant had the perf boost:

	varchar(before)	nvarchar(before
GraphQL	734ms	19ms
REST	655ms	24ms

	varchar(after)	nvarchar(after)
GraphQL	18ms	23ms
REST	18ms	25ms

Question about:

CAST(@param1 AS VARCHAR(MAX))

why cast instead of setting the sqldbtype on the dbconnectionparameter that is created?

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[seantleonard]

last question about tests.

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[abhishekkumams]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[seantleonard]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[aaronburtle]

Looks good, the performance increases here are awesome! Just had a couple clarifying questions, a suggestion for some potential comment.

updated