✨ (api) ability to run bank sync via the API #2683

MatissJanis · 2024-04-29T19:57:29Z

Allow running 3rd party (gocardless, simplefin) bank-sync via the API package.

This will allow people to run the bank-sync operation in the background on a schedule (i.e. with cron jobs).

Usage:

const api = require('@actual-app/api');

(async () => {
  await api.init({
    // Budget data will be cached locally here, in subdirectories for each file.
    dataDir: '/path/to/budget/file',
    // This is the URL of your running server
    serverURL: 'https://xxxxxxx.fly.dev',
    // This is the password you use to log into the server
    password: 'xxxxxx',
  });

  // This is the ID from Settings → Show advanced settings → Sync ID
  await api.downloadBudget('xxxxxx-xxxx-xxxxx-xxxxx');

  // Run bank sync for a single account
  await api.runBankSync({ accountId: 'xxxxxx-xxxxx-xxx-xxxxx-xxxxxxxx' });

  // Run bank sync for all accounts
  await api.runBankSync();

  await api.shutdown();
})();

Docs: actualbudget/docs#350

netlify · 2024-04-29T19:57:46Z

✅ Deploy Preview for actualbudget ready!

Name	Link
🔨 Latest commit	`69aa671`
🔍 Latest deploy log	https://app.netlify.com/sites/actualbudget/deploys/6644df46114a1a00083ba3a2
😎 Deploy Preview	https://deploy-preview-2683.demo.actualbudget.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

github-actions · 2024-04-29T19:58:38Z

Bundle Stats — desktop-client

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
9	4.72 MB → 4.72 MB (-6 B)	-0.00%

Changeset

File	Δ	Size
`home/runner/work/actual/actual/packages/loot-core/src/client/actions/account.ts`	📉 -6 B (-0.14%)	4.11 kB → 4.1 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

No assets were bigger

Smaller

Asset	File Size	% Changed
static/js/index.js	3 MB → 3 MB (-6 B)	-0.00%

Unchanged

Asset	File Size	% Changed
static/js/indexeddb-main-thread-worker-e59fee74.js	13.5 kB	0%
static/js/resize-observer.js	18.37 kB	0%
static/js/BackgroundImage.js	122.29 kB	0%
static/js/narrow.js	59.81 kB	0%
static/js/usePreviewTransactions.js	790 B	0%
static/js/AppliedFilters.js	20.54 kB	0%
static/js/wide.js	261.78 kB	0%
static/js/ReportRouter.js	1.23 MB	0%

github-actions · 2024-04-29T19:58:39Z

Bundle Stats — loot-core

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
1	1.2 MB → 1.2 MB (+431 B)	+0.03%

Changeset

File	Δ	Size
`packages/loot-core/src/shared/errors.ts`	📈 +122 B (+3.23%)	3.69 kB → 3.81 kB
`packages/api/methods.ts`	📈 +391 B (+2.98%)	12.81 kB → 13.19 kB
`packages/loot-core/src/server/api.ts`	📈 +286 B (+1.53%)	18.28 kB → 18.56 kB
`packages/loot-core/src/server/main.ts`	📈 +43 B (+0.07%)	59.95 kB → 59.99 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
kcab.worker.js	1.2 MB → 1.2 MB (+431 B)	+0.03%

Smaller

No assets were smaller

Unchanged

No assets were unchanged

matt-fidd · 2024-05-01T12:50:38Z

When calling runBankSync with no arguments, I'm receiving the below error.

/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43402
            handlers['accounts-bank-sync'] = async function ({ id }) {
                                                                                             ^

TypeError: Cannot destructure property 'id' of 'undefined' as it is undefined.
    at handlers.accounts-bank-sync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43402:64)
    at handlers.api/bank-sync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34823:72)
    at runHandler (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:44781:33)
    at Object.send (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:44463:94)
    at send (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/methods.js:31:21)
    at Object.runBankSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/methods.js:58:12)
    at /data/home/matt/cron/actual/bankSync.js:25:12

Node.js v18.19.1

MatissJanis · 2024-05-01T16:52:47Z

Thanks for testing @matt-fidd ! I must have accidentally removed something when I cleaned up the implementation.

Should be patched now. Give it another go.

matt-fidd · 2024-05-01T17:14:38Z

Thanks @MatissJanis!

Looks better now!

It appears to hit a problem halfway through my sync where it tries to re sync with the server in the middle of polling gocardless and falls over.

Bank Sync operation for account: ########
  Pulling transactions from GoCardless
  Syncing since 2024-05-01T17:10:28.143Z-0000-b80ed5e91c4802ff 7 (attempt: 0)
  SyncError: SyncError: encrypt-failure
      at Module.encode (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47024:35)
      at _fullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47705:77)
      at async /data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47611:32 {
    reason: 'encrypt-failure',
    meta: { isMissingKey: true }
  }

From then on, any API interactions are met with the below error.

node bankSync.js
- Connecting
-- Success
- Downloading
Loaded spreadsheet from cache (3136 items)
Syncing since 2024-05-01T17:10:28.143Z-0000-b80ed5e91c4802ff 7 (attempt: 0)
SyncError: SyncError: encrypt-failure
    at Module.encode (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47024:35)
    at _fullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47705:77)
    at async /data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47611:32
    at async initialFullSync (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:47597:32)
    at async handlers.sync-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:43980:32)
    at async handlers.api/download-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34777:36) {
  reason: 'encrypt-failure',
  meta: { isMissingKey: true }
}
/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34779
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getSyncError)(result.error, localBudget.id));
                              ^

Error: We had an unknown problem opening ?My-Finances-5cca4be?.
    at handlers.api/download-budget (/data/home/matt/cron/actual/node_modules/@actual-app/api/dist/app/bundle.api.js:34779:31)

Node.js v18.19.1

and to be able to continue, I have to wipe the cache directory and resync from blank

matt-fidd · 2024-05-01T22:21:56Z

Some debugging later, the error is thrown here as the keys object is empty.

With the debug lines below added

export function getKey(keyId) {
  if (keyId == null || keys[keyId] == null) {
+    console.log(keyId);
+    console.log(keys);
+    console.log(keys[keyId]);
    throw new Error('missing-key');
  }
  return keys[keyId];
}

The following is printed (master key has been redacted but it is set!)

Loaded spreadsheet from cache (3136 items)                                                                                                                                                                                                                                  
Syncing since 2024-05-01T21:31:22.845Z-0073-9f76373186bb94dd 13 (attempt: 0) 
                                                                                                                                                                                               
***REDACTED KEY***                                                                                                                                                                                                                                       
{}                                                                                                                                                                                                                                                                          
undefined              
                                                                                                                                                                                                                                                     
SyncError: SyncError: encrypt-failure
...

MatissJanis · 2024-05-02T17:08:38Z

Looks like the e2e encryption is having problems there for you. Did you set the encryption password in the code?

https://actualbudget.org/docs/api/#connecting-to-a-remote-server

await api.downloadBudget('1cfdbb80-6274-49bf-b0c2-737235a4c81f', {
    password: 'password1',
});

matt-fidd · 2024-05-03T08:47:13Z

I did set the encryption password and it manages to connect and sync a couple of accounts, and then it gets caught up somewhere and empties the keys object in /loot-core/src/server/encryption.ts. I've checked and neither the unloadKey or unloadAllKeys functions have been called.

matt-fidd · 2024-05-03T14:10:11Z

Done some more digging, I can't force it to happen with any other API functions. If I just sync one bank account over and over it also does not happen, only when calling api.runBankSync();

MatissJanis · 2024-05-03T16:29:59Z

Thanks for the investigation @matt-fidd . I've split off the patch of this issue into a separate PR: #2698

matt-fidd · 2024-05-03T18:32:20Z

Amazing, thank you @MatissJanis!

youngcw · 2024-05-13T14:40:20Z

@MatissJanis Is there much more that needs added here or is the ready for review?

MatissJanis · 2024-05-13T16:30:19Z

@youngcw this Pr depends on #2698. Once that one is merged - I'll open this one back up.

MatissJanis · 2024-05-15T16:21:22Z

Out for review again.

Would appreciate another round of testing from you @matt-fidd ! 🙇

matt-fidd · 2024-05-15T16:44:46Z

All looking good!!

✨ (api) ability to run bank sync via the API

33c9816

trafico-bot bot added the 🔍 Ready for Review Pull Request is not reviewed yet label Apr 29, 2024

Release notes

238eac9

MatissJanis mentioned this pull request Apr 29, 2024

✨ (api) ability to run bank sync via the API actualbudget/docs#350

Merged

MatissJanis marked this pull request as ready for review April 29, 2024 20:03

Patch bug in multi-account sync

f685771

MatissJanis mentioned this pull request May 3, 2024

🐛 (api) fix performing api operations for local e2e encrypted files #2698

Merged

MatissJanis changed the title ~~✨ (api) ability to run bank sync via the API~~ [WIP] ✨ (api) ability to run bank sync via the API May 3, 2024

MatissJanis added 🚧 WIP Still work-in-progress, please don't review and don't merge and removed 🔍 Ready for Review Pull Request is not reviewed yet labels May 3, 2024

Merge branch 'master' into matiss/api-bank-sync

69aa671

MatissJanis changed the title ~~[WIP] ✨ (api) ability to run bank sync via the API~~ ✨ (api) ability to run bank sync via the API May 15, 2024

trafico-bot bot added 🔍 Ready for Review Pull Request is not reviewed yet and removed 🚧 WIP Still work-in-progress, please don't review and don't merge labels May 15, 2024

youngcw approved these changes May 15, 2024

View reviewed changes

trafico-bot bot added ✅ Approved Pull Request has been approved and can be merged and removed 🔍 Ready for Review Pull Request is not reviewed yet labels May 15, 2024

MatissJanis merged commit f835280 into master May 16, 2024
19 checks passed

MatissJanis deleted the matiss/api-bank-sync branch May 16, 2024 07:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ (api) ability to run bank sync via the API #2683

✨ (api) ability to run bank sync via the API #2683

MatissJanis commented Apr 29, 2024 •

edited

netlify bot commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024 •

edited

matt-fidd commented May 1, 2024 •

edited

MatissJanis commented May 1, 2024

matt-fidd commented May 1, 2024 •

edited

matt-fidd commented May 1, 2024 •

edited

MatissJanis commented May 2, 2024

matt-fidd commented May 3, 2024

matt-fidd commented May 3, 2024

MatissJanis commented May 3, 2024

matt-fidd commented May 3, 2024

youngcw commented May 13, 2024

MatissJanis commented May 13, 2024

MatissJanis commented May 15, 2024

matt-fidd commented May 15, 2024

✨ (api) ability to run bank sync via the API #2683

✨ (api) ability to run bank sync via the API #2683

Conversation

MatissJanis commented Apr 29, 2024 • edited

netlify bot commented Apr 29, 2024 • edited

✅ Deploy Preview for actualbudget ready!

github-actions bot commented Apr 29, 2024 • edited

Bundle Stats — desktop-client

github-actions bot commented Apr 29, 2024 • edited

Bundle Stats — loot-core

matt-fidd commented May 1, 2024 • edited

MatissJanis commented May 1, 2024

matt-fidd commented May 1, 2024 • edited

matt-fidd commented May 1, 2024 • edited

MatissJanis commented May 2, 2024

matt-fidd commented May 3, 2024

matt-fidd commented May 3, 2024

MatissJanis commented May 3, 2024

matt-fidd commented May 3, 2024

youngcw commented May 13, 2024

MatissJanis commented May 13, 2024

MatissJanis commented May 15, 2024

matt-fidd commented May 15, 2024

MatissJanis commented Apr 29, 2024 •

edited

netlify bot commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024 •

edited

matt-fidd commented May 1, 2024 •

edited

matt-fidd commented May 1, 2024 •

edited

matt-fidd commented May 1, 2024 •

edited