Add `Sql.Regexp` #4392

jogibear9988 · 2024-01-26T00:39:25Z

supersedes #1077

Fixes #698

jogibear9988 · 2024-01-26T00:41:59Z

in #1077 you wrote I should create an extension lib.

I could do this, but could I also add this expression https://github.com/linq2db/linq2db/pull/4392/files#diff-2a6a9163861fbdc4015c6f32a3373c85469cb401552aea366b2d2c07ede63fb5R1198
in an external lib?

I would like to have the support for Regex in Query.
And should this be an extra lin in linq2db, or should it be my own?

viceroypenguin · 2024-01-26T00:41:29Z

Source/LinqToDB/Sql/Sql.Strings.cs

+		[Expression(PN.PostgreSQL, "{0} ~ {1}", ServerSideOnly = true, IsPredicate = true)]
+		[Expression(PN.Oracle, "REGEXP_LIKE({0}, {1})", ServerSideOnly = true, IsPredicate = true)]
+		[Expression(PN.SapHana, "{0} REGEXP {1}", ServerSideOnly = true, IsPredicate = true)]
+		[Expression(PN.SqlServer, "dbo.regex({0}, {1})", ServerSideOnly = true, IsPredicate = true)] // supported via https://github.com/infiniteloopltd/SQLServerRegex


I would remove this line; sql server should throw an exception, and if consumers want to use regex, they can add the library specifically. Having this will create a false expectation for the consumer, if they look at the failed query and see dbo.regex() in the querystring, and wonder why they don't have the regex method.

I tend to agree, I wouldn't commit core linq2db functions to custom libraries.
What if MS adds Regex support in the future (quite possible) and it's not exactly the same dialect / not 100% compatible with this lib?

If that lib is popular enough to be included in main linq2db lib, I'd put it on a specific static class: InfiniteLoop.Regex(..).

viceroypenguin · 2024-01-26T00:42:01Z

Source/LinqToDB/Sql/Sql.Strings.cs

+		[Expression(PN.Oracle, "REGEXP_LIKE({0}, {1})", ServerSideOnly = true, IsPredicate = true)]
+		[Expression(PN.SapHana, "{0} REGEXP {1}", ServerSideOnly = true, IsPredicate = true)]
+		[Expression(PN.SqlServer, "dbo.regex({0}, {1})", ServerSideOnly = true, IsPredicate = true)] // supported via https://github.com/infiniteloopltd/SQLServerRegex
+		public static bool RegExpIsMatch(string text, string expression)


Add xmldocs for this method.

sdanyliv · 2024-01-26T08:50:21Z

I'm still against using new Regex() in LINQ queries. Don't. Each database engine has it's own regular expression dialect non comparable with .NET Regex . We discussed that several times!

Introduce only Sql.Regexp() function instead RegExpIsMatch and do not create Expression mapping

jogibear9988 · 2024-01-26T09:36:54Z

It's okay for me, if I could support it in a extension, so I could add my own "Expressions" so it works on my system with std regex

viceroypenguin · 2024-01-26T12:50:15Z

I'm still against using new Regex() in LINQ queries. Don't. Each database engine has it's own regular expression dialect non comparable with .NET Regex . We discussed that several times!

Introduce only Sql.Regexp() function instead RegExpIsMatch and do not create Expression mapping

@jogibear9988 this is a really good point in the context of source-generated regular expressions, which may not operate as expected in this context. Like sdanyliv said, it will create the expectation that the db will operate on the same regex rules that .net does, which is not true for any db engine.

jogibear9988 · 2024-01-26T13:25:39Z

Yes, I wanted to now, is there a way for me, to add the line I did in Expressions.cs later in my usercode?

So that in my code we still could use the "Regexp" object?

We mostly use simple Regexes, wich would worknearly in all DBs.
(and in sqlserver and sqlite they would in any case, cause there C# code is called)

jods4 · 2024-01-26T21:15:20Z

Source/LinqToDB/DataProvider/SqlServer/SqlServerConfiguration.cs

@@ -10,5 +10,13 @@ public static bool GenerateScopeIdentity
 			get => SqlServerOptions.Default.GenerateScopeIdentity;
 			set => SqlServerOptions.Default = SqlServerOptions.Default with { GenerateScopeIdentity = value };
 		}
+
+		/// <summary>
+		/// You can specify a UserDefinedFunction wich is called to run a regex against your data.


typo in "wich".

jods4 · 2024-01-26T21:23:43Z

Source/LinqToDB/DataProvider/SqlServer/SqlServerConfiguration.cs

+		/// Look at https://github.com/infiniteloopltd/SQLServerRegex for a sample
+		/// in this case the value would be "dbo.regex({0}, {1})"
+		/// </summary>
+		public static string? RegexUserDefinedFunctionName { get; set; }


Do we really want to have such specific options? It's basically exactly the same level of abstraction as [Expression] but for a single specific function.
We could imagine similar options for many other SQL libraries -- and I really don't think we want to do that.
Moreover, I'd rather reserve the "official" linq2db regex function to a possible future "native" regex function in SQL Server (should it ever happen).

I'd rather have:

Docs/fully working example code to create such a static function with [Expression("dbo.regex({0}, {1})")].

Shipping such functions in different namespaces that can be imported (using LinqToDB.InfiniteLoop) and/or different libraries.

I'd move the complete code to an external library, if someone could tell me, how I could accomplish the add of the C# regex class to Expressions.cs, so I could use that in my queries

@jogibear9988 This PR is currently doing a (string, string) extension, isn't it?
Do you mean you'd like to be able to handle Regex.IsMatch(..)?

yes, I'd like to use regex.ismatch in my queries

I'm gonna be honest, that's terrifying.

But as I also saind, if we don't want to do this change, it is okay.

I'm only looking for a way, how I can extend Expressions.cs default... from outside.
Should we add a method? Problem is, that once initialized, adding something to it, does not have an effect. So it needs refactoring

I think your approche is to limited.

Can you elaborate? Seems much more generic and flexible to me. In any case, it covers dbo.regex({1}, {2}) that you're wanting in this PR.

Here's why I think your approach is limited and worst:

You introduce a new option RegexUserDefinedFunctionName that will only work for Regex.IsMatch. No other function is supported. Right now there's PR Support for NodaTime #3624 that's adding a lot of ugly code to support NodaTime that would benefit from a generic way to do this.

RegexUserDefinedFunctionName is static. What if I want a different config for two different DB instances? How could we easily run tests in parallel?

IMHO this design does not scale to more functions/libraries and is not easily maintainable long-term.
It moves inside linq2db core something that could trivially be done in user-land.

I'm only looking for a way, how I can extend Expressions.cs default... from outside.
Should we add a method? Problem is, that once initialized, adding something to it, does not have an effect. So it needs refactoring

I think this is the way to go and it's basically my proposal above.
Yes, it's a non-trivial refactoring but I think it is the best option long-term. Only issue right now is that there's the large @sdanyliv parser refactoring going on. We need to merge this before doing other large changes.

okay for me

BTW will gladly help with this. It's been on my mind for a while.

jogibear9988 · 2024-01-27T15:03:29Z

efcore seem to support this for postgres:

https://www.npgsql.org/efcore/mapping/translations.html#string-functions

jogibear9988 · 2024-01-27T15:04:29Z

and also in sqlite they seem to support them:

dotnet/efcore#28078

jogibear9988 · 2024-01-27T15:07:10Z

They also seem to automaticly map a regex function for sqlite (if you'd like to)

dotnet/efcore#18845

sdanyliv · 2024-01-28T08:22:25Z

It is Roji's decision because he created PostgreSQL provider, and probably ported to SQLite. And I think he is wrong here.

jogibear9988 · 2024-01-28T08:41:21Z

I don't think so ;-) , but I'll wait till we have a possibility to add my own mappings for Standard methods.
I also need it for the Guid.ToString call in my other pull request.

jogibear9988 · 2024-01-29T08:51:58Z

So, I now removed the functionality from Linq2db and added a sample in the Tests.
So maybe we can merge this, it test if regex registering is possible and usable.
I this it's usefull as a information.

viceroypenguin · 2024-01-29T13:10:13Z

@jogibear9988 looks like you remove both the expression and the Sql.Regexp method. You will need to restore the Sql. method.

jogibear9988 · 2024-01-29T14:09:09Z

@jogibear9988 looks like you remove both the expression and the Sql.Regexp method. You will need to restore the Sql. method.

No, I moved everything to an extra file. Cause for me it only makes sense if SqlServer is also supported. So I only added it to the tests as a sample. If someone likes to use it, he could copy the file from the tests. Or maybe I create a extra Nuget for Linq2db -> For Example Linq2DB.Regex, wich then maybe also contains the assembly for sqlserver

jogibear9988 · 2024-01-29T14:09:49Z

So we can merge this, as a sample, or close the pull req.
As I can now add the support from outside it's okay

viceroypenguin · 2024-01-29T14:10:49Z

Given that the code doesn't build without changes to the Sql. class, I'll go ahead and close it.

jogibear9988 · 2024-01-29T14:12:30Z

@viceroypenguin now I understand, it should build. This was wrong, I'll fix it

MaceWindu · 2024-01-30T16:58:36Z