surveycto: add new cursor #524

josephjclark · 2024-04-23T15:49:48Z

Summary

This PR adds:

A format function in the common cursor
Simplified timestamp handling in surveyCTO
A custom cursor exported by surveycto

Issues

Closes #520

Simplified Date handling

The date option in fetchSubmissions can now accept a surveyCTO date, an epoch or unix timestamp, or basically anything you can pass into new Date(). This has been documented lightly in a way that encourages the CTO MMM dd, yyy h:mm:ss a format, but doesn't prescribe it.

The "magic" is handled by a little function which takes a date value and guarantees to return a string formatted in the surveyCTO way.

Why settle on this format? Because it's quite human readable, so when queries get printed in the log, you can see the actual parameters without having to do the maths on the timestamp.

I also think it's important for openfn users who aren't surveyCTO experts to be able to use dates in the way they expect. Just pass a date into the date property - we'll figure out the formatting. Life's too short to worry about this stuff.

Documentation has been updated and simplified to this effect.

The actual support for this is implemented in fetchSubmissions and the new cursor function.

common.cursor.format

The new format function in common allows you to intercept the cursor value just before it gets written to state, so that you can specifier your own formats (or even values). It's more than a formatter tbh but its usage feels quite intuitive as a format (as opposed to parse or something).

See the docs PR for usage: OpenFn/docs#479

surveycto.cursor

surveyCTO takes advantage of the new cursor functionality.

It exports its own cursor function with a formatter baked in. So when you pass a date into cursor, what will actually be written to state is a surveyCTO style string.

You can pass in today or Apr 18, 2024 6:26:21 AM or 2024-04-23T15:49:31.764Z and it'll all just work. The result is compatible with filterSubmissions.

josephjclark · 2024-04-23T15:50:55Z

packages/surveycto/src/Adaptor.js

 */

 /**
 * Fetch form submissions
 * @example <caption>Fetch all form submissions</caption>
 * fetchSubmissions('test');
- * @example <caption> With `MMM dd, yyyy h:mm:ss a` date format</caption>


These examples have been trimmed down so that each one only shows one thing. Ie, the CSV example doesn't include a date.

josephjclark · 2024-04-23T15:54:28Z

packages/surveycto/src/Utils.js

+    'MMM dd, yyy h:mm:ss a'
+  );
+
+  console.log(`Converted timestamp "${date}" to "${formatted}" (UTC)`);


I added this as an afterthought because it seems quite important 🤔

On the other hand it's noisy and maybe a bit defensive. Opinions welcome!

josephjclark · 2024-04-25T08:33:55Z

Hi @mtuchi I think this will be interesting to you - do you have time to take a look today?

mtuchi · 2024-04-25T14:39:42Z

I ran pnpm i & pnpm build i am seeing new changes in ast.json file

modified:   packages/asana/ast.json
modified:   packages/common/ast.json
modified:   packages/googlesheets/ast.json
modified:   packages/http/ast.json
modified:   packages/kobotoolbox/ast.json
modified:   packages/msgraph/ast.json
modified:   packages/mssql/ast.json
modified:   packages/surveycto/ast.json

josephjclark · 2024-04-25T14:40:29Z

Thank you @mtuchi , that's fine, we still need to do the release engineering stuff. I'm more interested in whether this behaviour is as useful as I hope it is!

mtuchi · 2024-04-25T14:45:28Z

The cursor is working but i am seeing these double logs with the cursor

Compiled all expressions in workflow
Converted timestamp "Thu Apr 25 2024 00:00:00 GMT+0300 (East Africa Time)" to "Apr 24, 2024 9:00:00 PM" (UTC)
Setting cursor "today" to: Apr 24, 2024 9:00:00 PM
Converted timestamp "Apr 24, 2024 9:00:00 PM" to "Apr 24, 2024 6:00:00 PM" (UTC)
Fetching 'test' submissions for: Apr 24, 2024 6:00:00 PM

When i run the following code

cursor('today');
fetchSubmissions('test', { date: $.cursor });

My gut tells me the cursor should also tell me the date in unix timestamp. I think it's useful when i need to debug. But i don't think it's very important since the cursor date format is in human readable and i can easily relate to that

mtuchi · 2024-04-25T14:49:05Z

I like that it converts the time to UTC first

fetchSubmissions('test', { date: 'Apr 24, 2024 9:00:00 PM' });

Will log

Converted timestamp "Apr 24, 2024 9:00:00 PM" to "Apr 24, 2024 6:00:00 PM" (UTC)
Fetching 'test' submissions for: Apr 24, 2024 6:00:00 PM

mtuchi · 2024-04-25T14:51:28Z

What's your thought on supporting something like this ?

cursor('today', {format: 'unix'});
fetchSubmissions('test', { date: $.cursor });

josephjclark · 2024-04-25T15:00:22Z

Hmm, yeah the logging is tricky because it's hard to set context. I'll think about it because the logs are definitely too verbose right now.

What would be your expected behavour for this?

cursor('today', {format: 'unix'});

To write state.cursor as 1714057191?

What would the utility of that be?

mtuchi · 2024-04-25T16:41:38Z

I guess the use case is setting the state.cursor to unix timestamp because i think we still want to support unix timestamp right ?
Eg: date:'1714057191' Should still work right ?

fetchSubmissions('test', { date: '1714057191' });

//Logs
Converted timestamp "1714057191" to "Apr 25, 2024 2:59:51 PM" (UTC)
Fetching 'test' submissions for: Apr 25, 2024 2:59:51 PM

mtuchi · 2024-04-25T16:47:15Z

I am not sure if a user specify a unix timestamp, we should convert it to UTC?
Right now any date input gates converted to UTC. Eg

fetchSubmissions('test', { date: '1714057191' });

//Logs
Converted timestamp "1714057191" to "Apr 25, 2024 2:59:51 PM" (UTC)
Fetching 'test' submissions for: Apr 25, 2024 2:59:51 PM

I guess it doesn't hurt making sure the timestamp is in UTC 🤷🏽 .

Also if i am in UTC timezone i don't think we need to convert the time to UTC right ?

josephjclark · 2024-04-25T17:18:53Z

Users can still pass unix timestamps in. We'll still convert them for strings, which gives clearer logs.

We could choose to leave unix timestamps alone but the idea of all of this for the cursor to take any reasonable date format and "normalise" it to a UTC string. That's the whole idea.

So this works fine: fetchSubmissions('test', { date: '1714057191' }); (wait, is the timestamp a number or a string?)

Don't get hunt up on "converting to UTC", the point is more that we convert to String. Unix and Epoch timestamps are always in UTC, so we're basically just stringifying them for readability.

mtuchi · 2024-04-25T18:05:39Z

Okay got you 👍🏽

Regarding

So this works fine: fetchSubmissions('test', { date: '1714057191' }); (wait, is the timestamp a number or a string?)

I have tried both number and string and it worked

fetchSubmissions('test', { date: 1714057191 });
fetchSubmissions('test', { date: '1714057191' });
// Both log 👇🏽 
Converted timestamp "1714057191" to "Apr 25, 2024 2:59:51 PM" (UTC)
Fetching 'test' submissions for: Apr 25, 2024 2:59:51 PM

josephjclark · 2024-05-08T09:34:38Z

@mtuchi I think this is ready for release and we're both happy with it?

Would you mind giving it an approval before I merge?

Sadly because I've touched common all the adaptors have been bumped, even thouhh they're unaffected (only adaptors exporting cursor are affected). I've raised an issue to look into this but tbh I think it's a neccessary evil

mtuchi · 2024-05-08T10:31:03Z

@josephjclark you can go ahead build & push tags then squash merge this pr

josephjclark added 3 commits April 23, 2024 12:35

common: support custom formatter on cursor

88f99a8

surveycto: add new cursor function

b67f81b

surveycto: docs

64c8670

josephjclark changed the title ~~cursorcto: add new cursor~~ surveycto: add new cursor Apr 23, 2024

josephjclark commented Apr 23, 2024

View reviewed changes

josephjclark added 3 commits April 23, 2024 16:52

surveycto: add logging

a9327ef

remove comment

49e72a7

format

8f425ad

josephjclark commented Apr 23, 2024

View reviewed changes

josephjclark requested a review from mtuchi April 23, 2024 15:54

josephjclark mentioned this pull request Apr 23, 2024

Explain cursor.format OpenFn/docs#479

Open

This comment was marked as outdated.

Sign in to view

josephjclark added 2 commits April 23, 2024 17:22

surveycto: fix jsdoc

ab5de66

surveycto: linter fixes

ee7da7d

josephjclark added 4 commits May 8, 2024 10:14

Merge branch 'main' into cursor-surveycto

52699be

surveyCTO: tweak docs

9e913fd

version bumps

2df5888

lockfile

e892ccb

mtuchi approved these changes May 8, 2024

View reviewed changes

josephjclark merged commit dadff02 into main May 8, 2024
1 check passed

josephjclark deleted the cursor-surveycto branch May 8, 2024 10:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

surveycto: add new cursor #524

surveycto: add new cursor #524

josephjclark commented Apr 23, 2024 •

edited

josephjclark Apr 23, 2024

josephjclark Apr 23, 2024

This comment was marked as outdated.

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024 •

edited

mtuchi commented Apr 25, 2024

mtuchi commented Apr 25, 2024 •

edited

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented May 8, 2024

mtuchi commented May 8, 2024

surveycto: add new cursor #524

surveycto: add new cursor #524

Conversation

josephjclark commented Apr 23, 2024 • edited

Summary

Issues

Simplified Date handling

common.cursor.format

surveycto.cursor

josephjclark Apr 23, 2024

Choose a reason for hiding this comment

josephjclark Apr 23, 2024

Choose a reason for hiding this comment

This comment was marked as outdated.

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024 • edited

mtuchi commented Apr 25, 2024

mtuchi commented Apr 25, 2024 • edited

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented Apr 25, 2024

mtuchi commented Apr 25, 2024

josephjclark commented May 8, 2024

mtuchi commented May 8, 2024

josephjclark commented Apr 23, 2024 •

edited

mtuchi commented Apr 25, 2024 •

edited

mtuchi commented Apr 25, 2024 •

edited