@DbArray not allowing null to be persisted, stores empty list instead.

[DwayneSmurdon]

Hello!

We are using ebean 13.1.0 on Postgres 14, unlike in some (all) prior versions, ebean seems to persist an empty list / array of cardinality of 0 in my postgres db, even if I explicitly set it to null.

This significantly breaks are code, because now a call to myModel.update() changes @DBArray fields to empty lists, marks the field as "dirty" (technically it now is dirty) and updates the db with an empty list.

The weird thing is, I can't track/trap the code that is changing the value. In Intellij IDEA I can watch the value change from null to empty list in the debugger.

myModel.col1 = null; # a watch on this field will show value of col1 as null [expected]
myModel.col2 = null; # a watch on this field will show value of col2 as null but automagically now col1 = {} (empty list) [very very unexpected and very annoying]`

The fact I set col2 = null is irrelevant, I could have set it to anything and still col1 gets automatically set to empty list.

One concrete scenario that breaks our application completely is where we don't even select 9 of 10 @DBArray fields from the db. We only want to select a single @DBArray field, transform the values, and update the db. In prior ebean versions, the non-selected field values were null (even if the db has values) and thus update() ignored them, as desired.

Is this a bug? Can someone help me learn how I could have found the code automagically changing my fields in Intellij IDEA debugger? Stepping over code line by line didn't lead me to the code responsible.

[rbygrave]

I think we are looking at #1867 ... which effectively introduced this behavior in ebean-agent 12.1.5

So more specifically, have you just upgraded from an Ebean version that was older than 12.1.5? (that's around Nov 2019)

[rbygrave]

The original discussion is at: https://groups.google.com/g/ebean/c/rKQp17GTCJs ... which lead to the change in #1867

@DwayneSmurdon Are we able to confirm this is #1867 and not some other issue?

[ajcamilo]

as I said, it was an inconsistency issue. I understand if this is reverted and is easily fixed in my code.

[rbygrave]

Yeah, I think I just completely missed the database null side of this issue back then (bit of a bad miss on my part). As I see it at the moment, if it's actually a database non-null column then the behavior is correct (and net positive) - so revert is an option but the other option that I'm leaning towards would be to detect if it's database nullable and then turn off the initialisation for that case (which looks very doable and I think would be intuitive for users).

Design Background:
For @OneToOne and @ManyToOne we really want this "initialisation" for 2 reasons - 1) Partial objects and 2) Change detection for ManyToMany and orphanRemoval=true. It's also good for these 2 collection types to never be null when accessed and this is what the enhancement gives us for these two types of collections.

For @DbArray we have the case where we may desire database null (rather than database not null and empty []). So for that case when we want database null we actually want a null java collection.

---

Next steps:

• get some failing tests (and ideally confirm we are indeed fixing the right issue)
• chose the desired fix. I'm leaning towards detecting if it is database nullable column and turning off "DbArray initialisation" for that case - e.g. "initialisation" only enabled for @DbArray(nullable=false) or equivalent.

[rbygrave]

The fact I set col2 = null is irrelevant, I could have set it to anything and still col1 gets automatically set to empty list.

Noting that this is the behavior of @OneToMany and @ManyToMany. It is there for good reasons with those @OneToMany and @ManyToMany cases but I can see what you mean wrt @DbArray so we will have to review #1867 etc.

One concrete scenario that breaks our application completely

I don't actually understand what you are saying there. Can you explain that in code form or have another go at explaining that? Maybe you are talking about multiple @DbArray fields and perhaps their value is initially null when they are loaded? If you explain this more fully that would be good to understand what the behavior is.

Is this a bug?

Well it would be good to know what version of ebean this app has been upgraded from. If it is prior to 12.1.5 etc. If it is later than 12.1.5 I think we'd really like to see a test case to show the "before" behavior - perhaps fork the example minimal project - https://github.com/ebean-orm-examples/example-minimal

[rbygrave]

Is this a bug

Still want to know the prior version this app was upgraded from. That said, reviewing #1867 it does look like we should be able to create a failing test case for the case of nullable @DbArray. So yes I am expecting this to be a bug for that case.

If the behavior is what I think (we get that failing test case with null value for @DbArray) then I think that points us to the possible fix which is that we should only use this initialisation mechanism when @DbArray is known to be non-null (via kotlin non-nullable type, via explicit @NotNull ... and perhaps we add a nullable attribute to @DbArray (to then be similar to @Column(nullable))?

Important not to get ahead of ourselves here though so lets confirm the version that this app upgraded from (that didn't have the issue) and look to get a failing test (probably loading a bean with a @DbArray that has a null value, doing something, then update()). @DwayneSmurdon can you provide that version and the failing test case?

Thanks, Rob.

[DwayneSmurdon]

Sorry, I was away for almost two weeks. I will have too look at exact versions. It's a bit difficult to know because: we updated our stack of Java, Play, Play-Ebean, and thus ebean was transitive, and didn't notice this problem right away.

I've tried downgrading Ebean, but at some point I recall that version of Ebean was not compatible with Java 17.

I think this combo is still failing:

libraryDependencies += "io.ebean" % "ebean-agent" % "11.38.1"
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.8.15")
addSbtPlugin("com.typesafe.play" % "sbt-play-ebean" % "6.2.0-RC7" exclude("io.ebean", "ebean"))

But this could be a state where 11.38.1 isn't Java 17 compatible and we stopped here, because downgrading Java is painful at this point and I was pretty sure there was a bug.

However, at some point I was having trouble stopping sbt from overriding the ebean version. Our project has several modules, I think we had to change config in a lot of places.

I'm unable to test right now, or I would.

I read #1867 and it seems to be describe what I am seeing. The timing of that change could very well fit our timeframe.

[rbygrave]

I read #1867 and it seems to be describe what I am seeing. The timing of that change could very well fit our timeframe.

Ok good.

I'm unable to test right now, or I would.

Not sure what you mean here specifically but wrt testing Ebean behavior we always encourage and have the option to create a minimal example that only has Ebean and related dependencies like jdbc drivers etc and we provide minimal examples for maven and gradle:

• https://github.com/ebean-orm-examples/example-minimal
• https://github.com/ebean-orm/examples

This takes out framework dependencies like Play, Spring, Dropwizard etc and this is the best way to communicate behavior (expected vs unexpected) and makes it easier for other people to investigate, compare across Ebean versions and potentially turn use cases into tests in Ebean itself.

[rbygrave]

Created this as an Issue: #2855

[rbygrave]

With Ebean version 13.12.0 this option is now by default on - https://github.com/ebean-orm/ebean/releases/tag/ebean-parent-13.12.0