Date Taken sort order and related issues

[paperboyo]

Grid currently allows only for two sort orders. The default one everywhere is Date Uploaded. Upon entering the Collection view, the sort order switches automatically to Recently Added First (no way of reversing, but one can switch to Date Uploaded). We know users want to be able to sort images in the browser by Date Taken. This will allow them to see images grouped by their “natural” order, instead of shoots being seemingly randomly mixed up by the order they were ingested.

Something to bear in mind is: the quality of Date Taken information really matters here. We have observe all these (and other) problems for supplied images:

• no Date Taken information at all (~20% of our corpus lacks it; 17% for the Nov’2022)
• wrong Date Taken (major suppliers still provide wrong Date Taken eg. for archival “File photos”)
• partial Date Taken information (eg. timezone missing etc)
• mangled Date Taken (a cursory look at us trying to deal with all the weird issues should give pause for thought)

Here are some thoughts, some more related to the sort order, some less. Some need to be sorted out before we build it, some not. But all matter:

1. Grid converts all Date Taken information into UTC internally. This is wrong. While reconciling into a common date for display makes sense, loosing time zone completely is wrong. We should never loose useful information like time zone.
2. Because of this UTC model, Grid cannot distinguish between partial dates (eg. an archival photo given only a year) and a photo genuinely taken on the 1st of January that very year. While they may be equivalent for display in the Taken order (if we decide so), they are not the same. This is wrong.
3. Editing Date Taken requires users to enter the full date (users cannot enter partial dates). While we can fix the code about partials where we read them from embedded metadata, here, in editing, we effectively are forcing users to enter misleading and incorrect data, when they only know a partial date. This is wrong and needs fixing. Users should be allowed to enter partial date information.
4. Grid currently uses local clock to determine what is “now”. We had reports from users not seeing imagery, because they had their clock incorrectly set 2 days prior to the current date. Haha.
5. Everyone agrees there will be more sort orders in the future. Everyone agrees we need a dropdown as there won’t be space for so many radio buttons. Whether the dropdown should only list values for sorting and the ordering device should be separate or should the dropdown contain two entries per value (with no separate ordering device) is a matter for discussion, both approaches having pros and cons. We agreed to go with two entries per value (Taken date (newest first) and Taken date (oldest first) or similar), because Date Taken will be just one new sort order in the near term, so we won’t risk overloading the dropdown. It can be argued it is more explicit (is “oldest first“ Z>A or A>Z?), the downside that we have to acknowledge is bigger cognitive load when changing (and – choosing) the dropdown options and more clicks to change the order.
6. Because of the reasons below and because of user habits, Date Taken shouldn’t be the default sort order. We may decide to switch to it automatically when user explicitly engages searching by Date Taken, though? (similarly to how order changes in Collection context and how, one day, sort order will switch to Date Modified when user searches by Date Modified)
7. We have to decide how “sticky” the sort order choice should be.
   a) Should clicking on Grid logo (Home button) reset sort order to default Date Uploaded? (yes?)
   b) Should it “survive” other navigation? (yes?)
   c) since it will travel in shared links, it has a potential of changing another user’s sort order (this already applies to eg. Free to use)
   d) are there users who would want to switch to Date Taken sort order as default permanently? (we do have a Show paid “permission” to change default cost view for some users, if so – those two may be a beginning of User Preferences)
8. The biggest decision we have to make is what to do when user engages Date Taken sort order with images without Date Taken. Because partials should probably be sorted at the beginning of the second/minute/hour/day/month/year etc. There are arguments for hiding them: this provides some way of weeding out images with worse metadata. But we have to remember it won’t weed out wrong Date Taken. And I am, personally, terrified of hiding images from users. Consider some of these scenarios:
   a) user clicks on a link to a collection/label with &orderBy=-dateTaken, they see what they were sent, they navigate away, with their sort order changed to Date Taken, they click on clear search X (not the Home button) and they are not seeing ~20% of imagery
   b) user switches Recently Added First to Taken Date (newest fist) within a collection of 67 images and suddenly sees only 45 of them
   c) a News user who was insistent on Date Taken order and on hiding images without Date Taken and is extremely happy to be seeing images they trawl every day in order is so happy they never switch back to Date Uploaded. They miss 20% of images. Their editor one day comes back to them shouting, because they missed an image from a supplier with so-so metadata hygiene. Of course no editors ever shout in real life, but I wouldn’t want to cause any stress to anyone. Even if they ask for it.
9. We can alleviate pt. 9 concerns in at least two ways:
   a) never hide images without Date Taken (users can always engage -has:dateTaken, which is known to three people on earth, probably) and display them using a fallback of Upload Date
   b) do as above, but display a contextual UI device allowing users to quickly hide images without Date Taken (maybe also partials?)
   c) initially hide images without Date Taken when user chooses Date Taken sort order, but clearly display how many images were hidden because they have no Date Taken and allow users to show them (in their fallback position). I think the “new images since” ticker is the closest UI device we have to provide such a contextual UI device.
10. We may want to signal to the user sort order has changed and is set to non-default state. When they enter collection view (already true; I bet nobody notices). When the click shared link with Date Taken sort order. When they engage search by Date Taken and sort order switches automagically. Maybe just some sort of border around Sort dropdown?
11. I think we may want to switch to Date Taken being shown on image thumbnails when searching by Date Taken and when sorting by it. Hover gives all dates, but maybe users should be also able to choose what’s displayed on thumbnails too (by just clicking?)?

OK. This is a wall of text. Apologies. I think there is a task to see how other tools deal with partial/missing dates. Hiding images terrifies me (even when signalled). Some user wishes should not be granted lightly or we will be chased out by the same users who expressed them with forks. I am a user too. I know.

Will be adding here, please do add your thoughts too! Date Taken sorting order is exciting! I’ve wanted it badly for at least last five years!

[pradasa]

Thanks for this Mateusz! I wasn't aware of some of the challenges we had with time, like UTC and machines being 2 days ahead...

Regarding your points:

1. Agree, we should indicate time zone to be accurate and avoid confusion. Why do we convert to UTC internally?
2. Agree
3. Agree

I want to spend some time looking at points 5-11 and see what other people do in those cases. ecommerce sorting might be helpful for ideas on how to resolve these challenges. I have seen John Lewis do a toggle to show/hide "items out of stock" for x search. This could be something to explore regarding image results with missing metadata.

I'll have a look today and share things on Louise's Miro.

[paperboyo]

Thanks! Worth noting that points 1–3 are only indirectly related to Date Taken sorting. AFAIU, UTC is crucial to be able to do any kind of comparisons between dates (they come in all sorts of formats). All engineers I have ever spoken to about that have very strong feelings about the benefits of UTC. My points in 1–3 is not to get rid of it, only to highlight challenges of loosing time zone information and forcing users to invent the data they don’t know (partial dates).

Time zone usefulness isn’t that clear, but it is information we are currently loosing in the main Grid model (it obviously stays inside the pictures it was sent in). Consider this scenario:

• image is sent from Australia with exifsub: "Date/Time Original": "2023:01:26 09:38:21", "Sub-Sec Time Original": "27" and "Time Zone Original": "+10:00"
• we create our own UTC composite (still in fileMetadata): "Date/Time Original Composite": "2023-01-25T23:38:21.270Z" (time zone is lost, baked in)
• a user edits the Date Taken (there is no time zone there)
• one day, Grid will be able to write metadata back to files, if it will use this Date Taken, it would lose the original time zone

In this scenario, albeit futuristic (we don’t write metadata back yet), the information still resides in the original image. But not so for Date Taken added entirely in Grid. I think editing Date Taken applies user’s local browser time zone automatically?

For partial dates, xmp.photoshopDateCreated allows for partial dates:

We convert them to UTC (adding 1st of Jan to year-only dates). We will have the same problem when exporting them. But, again, it’s worse for images where year-only Date Taken was added directly in Grid: there we force users to enter the full date. And we won’t have data to tell us users just lied under pressure to go back on (nothing stops them from entering middle of the year or December).

Again, this is food for thought. And only tangentially related to sorting. I may have made (my usual) mistake of mentioning these issues (1–3) here. I think it’s better to keep this discussion focused on the challenges of sorting by Date Taken: what to do with images missing Date Taken (because for the reasons above, partials will all just show under their “fake” placement, mostly 1st of Jan, or midnight etc). With partials, our hands are tight, unless we arbitrarily exclude all images with “starting” Date Takens from the display (nope, nope, nope).

[andrew-nowak]

one more prior art on partial dates, with some details of how they store internally: https://www.wikidata.org/wiki/Special:ListDatatypes#time

[davmacbea]

1. Grid converts all Date Taken into UTC. This is wrong

Agreed - we should hold the time zone information and not force to UTC.

2. Grid cannot distinguish between partial dates 

Agreed. We should hold the dates in normal date-time format along side a precision indicator. We should agree a convention for setting a (falsely) precise date-time for imprecise date-times. That convention should use the beginning of the time window. So “sometime in 1975” becomes “1975-01-01T00:00:00”. You could argue that the mid-point of the time window is better (so midday on 2 July), but I think that would just end up confusing people. Something looks a bit weird about midnight on 1 January, whereas midday on the 2 July looks normal.

Precision indicator could be something like [century | decade | year | month | day | precise]

3. in editing, we effectively are forcing users to enter misleading

Agreed. Does this mean we need widget variants that ask for differing levels of precisions? Does that mean the widget needs a precision picker?

4. Grid currently uses local clock to determine what is “now”.

Probably should only use local settings to determine user time zone and use server time to define “now”.

5. Describing sort orders and allowing easy picking of them does need some thought. My feeling is that fuller descriptions will lowers cognitive load (terse things like “A>Z” and “Z>A” take my brain ages to process).

6. Default ordering does need some thought. I agree that if a user filters by a date then they are obviously interested in that attribute, but perhaps sometimes that is when they would least want to see the images sorted in the order of that date as they have already “dealt with” the date bit by filtering. Needs more discussion.

7. On ordering stickiness

We could remember a user's last sort order and that would survive navigation (as it stands we'll need to remember two user choices - the normal view and the collection view). Louise made the valid point that remembering sort order is not the norm on shopping sites, so transient (non-sticking) ordering may be more intuitive to users. This however may be a dark pattern on those shopping sites that forces users back to an ordering that benefits the shop rather than the user. We will need to override normal ordering when a user opens an URL to a search that someone has sent them. In this case that should be sorted as the sender intended. On navigation the user’s previous sort order to should be reinstated.

8-9. What to do when date info is missing

I also don’t think we should not hide images due to a sort.  Sorting is not filtering. We could fall back from Date Taken to Upload Date, but I think that might be confusing. The images with missing metadata could be displayed at the end of the result set (perhaps ordered by Upload Date). Perhaps we could have a visual break in the results and below this break we show the thumbnails with missing metadata (like a horizontal line that says “below here the photos don’t have Date Taken set”)

10. Agree that it would be good to have a visual indicator when there has been a non-user-invoked order change. Maybe a quick visual treatment like a glowing border around the sort control that then fades. Might have accessibility challenges for screen readers.

11. There’s so little space to display data on the thumbnail view and giving the user controls to pick what is shown could get quite messy. I know it’s called Grid, but have we ever considered a thumbnail list view, where each image has a thumbnail-height row the whole width of the search area? Could be good for when the user wants lots of metadata on their results page.

[pradasa]

On the topic of sorting and filtering, Matteusz reminded me about some explorations we did ages ago that might come in handy for (10) the visual indicator.

The idea was to indicate to users that they are seeing a filtered view while filters themselves are not visible without clicking a button in this example.

Regarding (7) ordering stickiness, I think that regardless of whether we keep it through a session or reset every time they load a new page, it might be worth looking at if the label should reflect the selected sorting method.

I do agree that in our case it might be worth keeping the sorting selection given that everyone will be signed in and each user might have their own preference. I don't know enough to suggest a solution so I'll do some digging on this.

[paperboyo]

Thanks a bunch!

Ad. 3. This is current widget (after extra click on calendar). Not sure if native calendar allows for setting partials (or how would it behave when asked to display one), but cheapest would be to make the editing field accept partial input. Eg. in Adobe app, one can just enter 1978. One for UX wizards, I guess?

Ad. 5. the problem of spelling out two sort options per value will only rear its head when we realise these all need (data already there, bar one) their respective sort orders (how amazing that would be; how sad to display them in upload order only!):

Ad. 6, 7. Agree could be dark patter + lack of ambition. But more discussion and playing around defo needed! Changing one thing affecting other things can be frustrating/unexpected, unless… it’s entirely expected (going to Collection changes sort to Last added to Collection already, maybe going to My deleted should change the sort order to time deleted, My leases – time leased etc). But, again, worth checking how it feels. Agreed too that pt. 10 could help inform users sort order is not default/has changed. In general, to have a UI device to indicate to users any search/filtering options vary from “home” would be great, I think (checkboxes are already blue when ticked)

Ad. 8–9.

The images with missing metadata could be displayed at the end of the result set

It dawned on me, after I wrote my ramblings, that this is the best option. Obviously. I think a dividing line will be easily missed (sometimes it’s thousands before the ones with missing date would start). But I think pt. 11 could help (ie. if Taken: would say empty or missing or even just stay empty that would give users an idea they reached the missing ones). This is the kiddo way of indicating that I invented for more informative tooltips:

Whether dates on thumbnails should silently follow sort order (my initial hunch is yes?) and/or if they should be switchable by showing a dropdown by clicking on them or by going to some Preferences page/dialog, I don’t have strong feelings. But having them hard-coded to Uploaded for all other sort orders would be sad, I think.

Lastly, and most interestingly, regarding

I know it’s called Grid, but have we ever considered a thumbnail list view, where each image has a thumbnail-height row the whole width of the search area? Could be good for when the user wants lots of metadata on their results page.

Most obviously! Grid is just like any other (hopefully, good) browser/viewer combo. Information density should be adjustable. It’s a fluid spectrum: one image>two for comparison>many images>a spreadsheet-like list with metadata as columns (sortable). Paradigm since day one 😉. Not before React, I don’t think?

Agreed on all other points! Thanks so much for taking the time. No idea how underlying changes to date modelling could be tackled in an incremental manner, but it’s good to know where we wanna go. (cc @andrew-nowak who had opinions on dates)

[louisegoodspeed]

@pradasa @drewcochrane @louisegoodspeed @NickyPhelanBBC met yesterday and discussed points 6/7 around the selecting sort by persisting across searches. We all felt this was something that would need more research as all points made above are valid. We were discussing the following scenarios and wondering how what is technically possible would affect what was best...

1. The Default is Date Upload (New to Old)
2. User selects = Date Taken (new to old)

When they start a new search in a the same session, if the date taken (new to old) persists.

• What happens if the user returns for a new session, would we be able to/want to keep the users preference for all future searches?
• Or would a user just expect that sort by to persist for the one session and revert back for a new session to the default?
• And technically would it have to go back to default or is there some way it could persist?
• Would it have to be part of their settings in their profile to allow it to persist?

I think this captures our discussion fully UX folks, but let me know if I missed anything....

[paperboyo]

Grid persists some stuff per browser (local storage), eg. the state of panel locking. This will persists until user changes browser/machine or clears browser site data. This is okayish for keeping unimportant small tweaks.

I don’t think we have any user preferences that would travel with their login, certainly possible technically and would be expected would the preferences be extensive (imagine setting up 4 metadata templates visible only to you [not currently possible], you wouldn’t want them to ever disappear if you change the machine). We also don’t have any preferences that would apply to groups of users (again, thinking templates: different default sets for archivists and newsy folk).

Then there is a question: what’s a “session” above? Logging in the next day is. But clicking Home to clear everything to default also is a kind of a session: currently it resets the search makeup completely (bar the bugs – change date range, click Home, range resets, but not its UI: ). That’s a very useful thing and a pattern all over the interwebz. Should clicking Home reset the sort order too (probably?).

(pt. 10 and Ana’s comment) Indicating in UI that sort order (and any other search device) is straying from default should help users too. Chips help here, but eg. a blue border around date range UI when it’s not set to “anytime” would be clearer, I think.