Too easy create incoherent result without enough restriction

[FrankYFTang]

Currently, the API allow "very flexible" setting to each fields, which mean it could easily produce incoherent results- is this what we really want to produce?

For example:

const d = { years: 1, months: 2, weeks: 3, days: 3,  hours: 4, minutes: 5, seconds: 6, milliseconds: 7};
const df = new Intl.DurationFormat("en", {style:"short", years:"long"})
df.format(d)

will produce
"1 year, 2 mths, 3 wks, 3 days, 4 hr, 5 min, 6 sec, and 7 ms"
this approach also provide a lot of knot and bolt for the caller to tune,

But it also means our test need to cover a huge number of testing configuration:

with "style" and a total 10 styles and 10 displays for each field, the total combination to be test are

(4+1) x (5+1)^10 x (2+1)^10 = 1.7852336e+13 way too many possible combinations to test (yes, I know some combination are not allowed, but that mean we need to test those combination should throw )

4+1 : style could be undefined, "long", "short", "narrow", "digital" => 5 possible values to test
5+1: style for each field could be undefined, "long", "short", "narrow", "numeric" and "2-digits" => 6 possible values to test
2+1: display for each field could be undefined, "auto", "always" => 3 possible values to test

I am not even counting the 11 possible values of fractionalDigits undefined, 0, .... 9 => 11 possible settings.

[FrankYFTang]

@sffc @anba @ryzokuken @gibson042 @romulocintra

[sffc]

"1 year, 2 mths, 3 wks, 3 days, 4 hr, 5 min, 6 sec, and 7 ms"

That does not look incoherent to me

There are a lot of cobinations, but, unlike the DateTimeFormat components bag, I'm not aware of any combinations that are incoherent. Can you provide an example?

[FrankYFTang]

Can you provide an example?
How about

d8> d = { years: 1, months: 2, weeks: 3, days: 3,  hours: 4, minutes: 5, seconds: 6, milliseconds: 7};
d8> df = new Intl.DurationFormat("en", {style:"narrow", years:"long", weeks: "long"})
d8> df.format(d) 
"1 year, 2m, 3 weeks, 3d, 4h, 5m, 6s, and 7ms"

or

d8> df = new Intl.DurationFormat("en", {style:"narrow", years:"long", weeks: "long", hours: "long", milliseconds: "long"})
d8>  df.format(d) 
"1 year, 2m, 3 weeks, 3d, 4 hours, 5m, 6s, and 7 milliseconds"

[FrankYFTang]

Actaully, if we switch to pass ListFormat with {type: "unit", style} then the output will be
'1 year 2m 3 weeks 3d 4h 5m 6s 7ms'
and
'1 year 2m 3 weeks 3d 4 hours 5m 6s 7 milliseconds'

[sffc]

I think those are not incoherent, but I get your point.

[ryzokuken]

Yeah, I don't believe the results are completely incoherent either, but I understand your concern. That said, fine-tuning the result in this way happens due to explicit programmer choice, and the default "blanket" styles (just to call them that) are fairly coherent in this way.

I feel that it's fine to allow this degree of freedom to the programmer if they believe a certain kind of result makes sense for their interface and as we've realized after months if not over a year of circling around, there's no perfect solution to this API design.

[FrankYFTang]

ok, If I follow your argument of " it's fine to allow this degree of freedom to the programmer if they believe a certain kind of result makes sense" then would that also apply to locales so we should allow programmer to set the locale of the years to English, months to French, weeks to Thai, and days to Chinese for the sake of "allow this degree of freedom to the programmer if they believe a certain kind of result makes sense" ?

I am not supporting such move. I just want to use that to point out that most arguments you can think about to counter that move could also use to counter the current flexiblity to let the programmer to mixed styles together. The key reason I bring this up is because I try to wrinte down a test plan to test the implementation and then I realize the size of the testing matrix is just way too big for me to handle.

[ryzokuken]

I don't agree that mixed locales and mixed unit styles are equally "incoherent" or even are so in the same way.

That said, we have a more pressing issue here: we've tried multiple other designs for this API over and over again, and nothing worked. This is the only workable design we could come up with. Could we perhaps try to innovate on the side of test262 instead?

[sffc]

Discussion: https://github.com/tc39/ecma402/blob/master/meetings/notes-2022-09-01.md#too-easy-create-incoherent-result-without-enough-restriction-115

Conclusion: @ryzokuken to drive a relationship witih @ptomato to devise a Test262 strategy for this and come back with concrete recommendations that could reduce the testing complexity.

[sffc]

TG2 discussion: https://github.com/tc39/ecma402/blob/master/meetings/notes-2022-10-06.md#too-easy-create-incoherent-result-without-enough-restriction

Conclusion: We are happy with @ptomato's proposed testing strategy. @romulocintra and @ryzokuken will write the tests.

[FrankYFTang]

The last conclusion of what discussed in https://github.com/tc39/ecma402/blob/master/meetings/notes-2022-10-06.md#too-easy-create-incoherent-result-without-enough-restriction
is "RCA and USA to write the tests." in Oct 6, 2022.

now is more than 8 months after Oct 6, 2022,

I look at https://github.com/tc39/test262/tree/main/test/intl402/DurationFormat/prototype/format

there are zero coverage of testing how the following options will impact the output of Intl.DurationFormat.prototype.(format|formatToParts)

"years"
"yearsDisplay"
"months"
"monthsDisplay"
"weeks"
"weeksDisplay"
"days"
"daysDisplay"
"hours"
"hoursDisplay"
"minutes"
"minutesDisplay"
"seconds"
"secondsDisplay"
"milliseconds"
"millisecondsDisplay"
"microseconds"
"microsecondsDisplay"
"nanoseconds"
"nanosecondsDisplay"
"fractionalDigits"
"numberingSystem"

There are test to test the impact the output of Intl.DurationFormat.prototype.(format|formatToParts) from "style" and some test to valid these option reading will or will not throw error, but no validation of the formtted result.

Any ETA for the availability?

[sffc]

Here's a potential testing strategy. Take the product of the following:

1. 10-20 Duration object inputs
2. All four base styles
3. Combinations of 0, 1, and 2 units
4. For each unit, all permutations of options for that unit; for example:
   • undefined/undefined
   • undefined/narrow
   • undefined/short
   • undefined/long
   • auto/undefined
   • auto/narrow
   • auto/short
   • auto/long
   • always/undefined
   • always/narrow
   • always/short
   • always/long

The thing to test on the output is which fields end up being present in the output, which we can determine from formatToParts.

We can run over all this in a loop of 20*4*(1 + 9 + 36)*(12ish) = 44160 cases.

[FrankYFTang]

We can run over all this in a loop of 204(1 + 9 + 36)*(12ish) = 44160 cases.

what is (1 + 9 + 36) ? I do not understand what is that.

[FrankYFTang]

also, {hours, minutes, seconds} has 5 not 3 styles
{milliseconds, microseconds, and nanoseconds} has 4 styles
and the above strategy not including the testing for fractionalDigits yet, right?

[sffc]

We can run over all this in a loop of 20_4_(1 + 9 + 36)*(12ish) = 44160 cases.

what is (1 + 9 + 36) ? I do not understand what is that.

There are 9 10 units (year, month, week, day, hour, minute, second, millisecond, microsecond, nanosecond) so it should be (1 + 10 + 45) which is (10c0 + 10c1 + 10c2)

also, {hours, minutes, seconds} has 5 not 3 styles {milliseconds, microseconds, and nanoseconds} has 4 styles

Yes which is what I meant with "12ish"; it should be more for those units

and the above strategy not including the testing for fractionalDigits yet, right?

Yes, true, we should include fractionalDigits as an extra axis. I'd say just three values (undefined, 0, 3) should be enough.

With these combinations we're probably going to be above 100k cases, but I think that's fine. These can be performed in a tight loop, and the expectations can be derived directly from the test inputs.

Note also that one possible expectation is RangeError, because not all combinations here are valid.

[FrankYFTang]

oh, I see, be aware with such strategy, option bags such as the following will NOT be part of that testing

{years: "short", months: "long", days: "narrow"} => becasue this is a combination of 3 unit styles, more than 2.
{hours: "short", minutes: "2-digit", seconds: "numeric"} => becasue this is a combination of 3 unit styles, more than 2.

[sffc]

That's fair. Combinations of 3 might be safer. Just throwing out ideas on how to make the tests run faster.