Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Async Methods to server #884

Open
wants to merge 54 commits into
base: dev
Choose a base branch
from

Conversation

bratelefant
Copy link

This is a first try on async methods as a PR draft. Updated eslint and mocha testing as well; core.js and cursor.js were a starting point. Work is progressing on server.js.

Related to #865

dr-dimitru and others added 25 commits June 29, 2022 16:08
📦 v2.2.0

__Breaking Changes__

- ⚠️ Changes in `namingFunction`, — now naming function acts the same on the Client and Server, upon insert, load, and write. Test your implementation with changed logic. Output of Server function supersedes Client's function output

__Changes__

- 📔 Merge veliovgroup#843 and fix veliovgroup#820, thanks to @Prinzhorn
- 📔 Documentation refactoring focused on examples and its simplifications
- 👨‍💻 Support nested custom path returned from `namingFunction`
- 👨‍💻 Fix `namingFunction` behavior on Client and Server in upload, load, and write methods, closing veliovgroup#842; Thanks to @chrschae
- 👷‍♂️ Now library exports its helpers `import { FilesCollection, helpers };`
- 👷‍♂️ Add `.meteorignore` to minimize package's footprint
📦 v2.2.1

-  👨‍💻 Fix veliovgroup#842, a newly detected bug by @chrschae; Fixing case when `namingFunction` returns new nested path cause exception in `.write()` and `.load()` methods
📦 v2.3.0

__New features:__

- ✨ `opts.sanitize` method, read more in [*Constructor* docs](https://github.com/veliovgroup/Meteor-Files/blob/master/docs/constructor.md); Thanks to @xet7 and @mfilser

__Other Changes:__

- 👷‍♂️ Minor codebase enhancements and cleanups
v2.3.1

__Changes:__

- 👨‍💻 Improve `createIndex` helper
- 👨‍💻 Improve error output when FileSystem destination not writable; Related to veliovgroup#857, thanks to @Leekao
- 🐞 Fix custom `allowedOrigins` option for CORS; Closing veliovgroup#850, thanks to @djlogan2;

__Notes:__

- 👨‍🔬 Tested with latest release of `meteor@2.8.0`
📔 Improve AWS S3 documentation
📦 v2.3.2

- 👨‍💻 Potential fix for veliovgroup#857 (windows)
📦 v2.3.3

__Major changes:__

- no

__Changes:__

- 👨‍🔧 Fixed veliovgroup#870, thanks to @Gobliins
- 🤝 Compatibility with `meteor@2.11.0`
@bratelefant
Copy link
Author

Hi guys, please comment on this first attempt. In order to better understand what's going on I decided to add some tests for the server part, FilesCollection, core and cursor. Unfortunately this adds overall dependencies, especially in package.js; if anyone knows how to limit these dependencies to dev / testing, pleas let me know. Also tweaked the .eslintrc a little bit. Core and cursor now additionally have async methods. Work on server.js is in progress.

I suggest that the async versions of all the methods do not use any callbacks but throw errors instead and return promises resolving the results, cf. FilesCollection.addFileAsync for example.

Which callbacks need async versions? E.g. for my project, I need an async version of protected(), since I do policy checks against the db here, which will be async as well of course. I guess an async version onAfterUpload could be helpful, since the aws sdk v3 for example also uses async functions to send putObject requests etc.

@harryadel
Copy link
Contributor

@bratelefant I use the exclamation mark in packages to get over such restrictions

ostrio:files@3.0.0-beta.3!

@dallman2
Copy link

dallman2 commented Feb 5, 2024

Do we also need changes in ostrio:cookies, or how did you guys work around this for testing on meteor 3.0 beta?

In general, my solution was to packages that were causing issues and modify their package.js manually. Most packages were compatible with Meteor 3.0, but had an API constraint of 2.x. In this case, I would check if the files package has a 3.0 compatible version and explicitly require it in this package.

@harryadel
Copy link
Contributor

Now, as I'm attempting to upgrade our application to 3.0, I get these errors

                                                                                                                                      
   While selecting package versions:                                                                                                  
   error: Conflict: Constraint webapp@1.13.2 is not satisfied by webapp 2.0.0-beta300.0.                                              
   Constraints on package "webapp":                                                                                                   
   * webapp@~2.0.0-beta300.0 <- top level                                                                                             
   * webapp@2.0.0-beta300.0 <- meteor-base 1.5.2-beta300.0                                                                            
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base      
   3.0.0-beta300.0 <- accounts-password 3.0.0-beta300.0                                                                               
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password  
   3.0.0-beta300.0                                                                                                                    
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- webapp 2.0.0-beta300.0 <- autoupdate 2.0.0-beta300.0                    
   * webapp@2.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base 3.0.0-beta300.0 <-                  
   accounts-password 3.0.0-beta300.0                                                                                                  
   * webapp@2.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password 3.0.0-beta300.0                 
   * webapp@2.0.0-beta300.0 <- autoupdate 2.0.0-beta300.0                                                                             
   * webapp@2.0.0-beta300.0 <- mongo 2.0.0-beta300.0                                                                                  
   * webapp@1.13.2 <- ostrio:files 3.0.0-beta.3                                                                                       
   * webapp@1.3.10 <- ostrio:cookies 2.7.2 <- ostrio:files 3.0.0-beta.3                                                               


   Conflict: Constraint mongo@1.16.1 is not satisfied by mongo 2.0.0-beta300.0.                                                       
   Constraints on package "mongo":                                                                                                    
   * mongo@2.0.0-beta300.0 <- top level                                                                                               
   * mongo@~2.0.0-beta300.0 <- top level                                                                                              
   * mongo@2.0.0-beta300.0 <- reactive-dict 1.3.2-beta300.0                                                                           
   * mongo@2.0.0-beta300.0 <- accounts-base 3.0.0-beta300.0 <- accounts-password 3.0.0-beta300.0                                      
   * mongo@1.16.1 <- ostrio:files 3.0.0-beta.3                                                                                        
   * mongo@1.10.1 || 1.12.0 || 2.0.0-beta300.0 <- aldeed:collection2 4.0.0                                                            
   * mongo@1.0.8 || 2.0.0-beta300.0 <- trusted-care:nested-collection-helpers-enhanced 0.0.3                                          
   * mongo@1.16.8 || 2.0.0-beta300.0 <- dburles:mongo-collection-instances 1.0.0-beta300.1                                            
   * mongo@1.16.8 || 2.0.0-beta300.0 <- lai:collection-extensions 1.0.0-beta300.1 <- dburles:mongo-collection-instances               
   1.0.0-beta300.1                                                                                                                    
                                                                                                                                      
   Conflict: Constraint ddp-client@2.6.1 is not satisfied by ddp-client 3.0.0-beta300.0.                                              
   Constraints on package "ddp-client":
   * ddp-client@~3.0.0-beta300.0 <- top level
   * ddp-client@3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base 3.0.0-beta300.0 <- accounts-password 3.0.0-beta300.0
   * ddp-client@3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password 3.0.0-beta300.0
   * ddp-client@3.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base 3.0.0-beta300.0 <-
   accounts-password 3.0.0-beta300.0
   * ddp-client@3.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password 3.0.0-beta300.0
   * ddp-client@2.6.1 <- ostrio:files 3.0.0-beta.3



  Conflict: Constraint webapp@1.3.10 is not satisfied by webapp 2.0.0-beta300.0.
   Constraints on package "webapp":
   * webapp@~2.0.0-beta300.0 <- top level
   * webapp@2.0.0-beta300.0 <- meteor-base 1.5.2-beta300.0
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base
   3.0.0-beta300.0 <- accounts-password 3.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password
   3.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- routepolicy 1.1.2-beta300.0 <- webapp 2.0.0-beta300.0 <- autoupdate 2.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-base 3.0.0-beta300.0 <-
   accounts-password 3.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- ddp-server 3.0.0-beta300.0 <- ddp 1.4.2-beta300.0 <- accounts-password 3.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- autoupdate 2.0.0-beta300.0
   * webapp@2.0.0-beta300.0 <- mongo 2.0.0-beta300.0
   * webapp@1.13.2 <- ostrio:files 3.0.0-beta.3
   * webapp@1.3.10 <- ostrio:cookies 2.7.2 <- ostrio:files 3.0.0-beta.3
   
=> Your application has errors. Waiting for file change.

Does this pertain to the problem raised by @jankapunkt ?

https://forums.meteor.com/t/solved-how-to-publish-packages-with-specific-meteor-3-0-version-to-resolve-top-level-conflicts/61091/3

@dallman2
Copy link

dallman2 commented Feb 6, 2024

@harryadel you have several packages that are relying on old packages, even though they have a new 'beta' version. For me, the easiest solution was to clone the packages that relied on old versions and manually update their dependencies. Eventually, most packages will update and everything will be easy again. Until then, manual updates were the easiest way I found to actually be able to test meteor 3.

@harryadel
Copy link
Contributor

harryadel commented Feb 6, 2024

I know Daniel that I can fork the package locally and do my own thing but I don't think this is the issue here, please examine the error log I pasted carefully, specifically these lines:

   * mongo@1.16.1 <- ostrio:files 3.0.0-beta.3                                                                                        

  * ddp-client@2.6.1 <- ostrio:files 3.0.0-beta.3

   * webapp@1.13.2 <- ostrio:files 3.0.0-beta.3           
   * webapp@1.3.10 <- ostrio:cookies 2.7.2 <- ostrio:files 3.0.0-beta.3

ostrio:files is still trying to fetch pre 3.0 versions in mongo, ddp-client and webapp while ostrio:cookies is also trying to fetch an older version of webapp. So, that's why the issue might be that @dr-dimitru hasn't added the --release option Jan referred to before and the latest beta is still versioning from 2.14.

@dallman2
Copy link

dallman2 commented Feb 6, 2024

Ahhh I see. My mistake, I was having flashbacks to my migration a few months ago.

From the looks of the files changed in the pull request, version 2.14 is still a valid API level to use, along with the 3.0 beta API level. Explicitly stating that the package should use the new versions of the packages you're having issues with, or just removing the 2.14 API level altogether could both be solutions.

@harryadel
Copy link
Contributor

Ahhh I see. My mistake, I was having flashbacks to my migration a few months ago.

meteor-horror

More like a PTSD 😅

@dallman2
Copy link

dallman2 commented Feb 6, 2024

Got a chance to look into this more today, maybe try clearing the contents of the .versions file and re-running the project. I have had issues in the past where a dependancy resolution gets messed up because of a lingering out of date version in that file.

Edit: did not work locally :(, trying other solutions

@dr-dimitru
Copy link
Member

@bratelefant @harryadel @dallman2 @jankapunkt ostrio:files@3.0.0-beta.4 is out!

Now with api.versionsFrom(['2.14', '3.0-beta.0']) and ostrio:cookies@2.8.0 so you shouldn't experience dependency issues, let me know!

@dallman2
Copy link

dallman2 commented Feb 9, 2024

@dr-dimitru beta.4 successfully build in my project! Also compatible with the path we discussed in the cookies repo

@harryadel
Copy link
Contributor

@dr-dimitru Thank you will give it a try and report back 👍

@harryadel
Copy link
Contributor

harryadel commented Feb 26, 2024

Aside from the issue listed here, the packages are working wonderfully for us. Thank you @bratelefant, amazing work 👏 👏 👏

@dr-dimitru time for new releases?

EDIT: Do any of you guys use Cordova? I run into a couple of issues when async changes paired with Cordova.

@harryadel
Copy link
Contributor

When I tried to update the cookies package it led to a conflict since ostrio:files is reliant on 2.8

Screenshot_2024-03-24_16-26-47

@dr-dimitru

@dallman2
Copy link

EDIT: Do any of you guys use Cordova? I run into a couple of issues when async changes paired with Cordova.

We don't, unsure of others in the thread. What issues are you having?

@@ -22,6 +29,7 @@ import nodePath from 'path';
const bound = Meteor.bindEnvironment(callback => callback());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of sheer curiosity, is there still a need for Meteor.bindEnvironment? Meteor 3.0 no longer uses fibers/futures.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harryadel good catch, it should be safe to remove it now.
@bratelefant wdyt?

@dr-dimitru
Copy link
Member

dr-dimitru commented Mar 25, 2024

@harryadel please give a freshly published v3.0.0-beta.5 a try. Issue from your screenshot should be solved now. LMK

@harryadel
Copy link
Contributor

harryadel commented Mar 26, 2024

@dallman2 Still trying to pint point the root cause, though I figured to share my problems in case someone else ran into them.

Thank you Dimitry @dr-dimitru

@dallman2
Copy link

Will try this update tomorrow and let you know how it goes (I'm on Pacific coast time :P). I've been deep into css modules all day so I'm going to have to shift a bit to deal with the files collection.

@dallman2
Copy link

No issues for me using beta.5. That is to say, my app starts and runs. Have not done totally comprehensive testing.

@harryadel
Copy link
Contributor

@bratelefant @dr-dimitru Maybe it's time to release a new beta that relies on the latest RC instead of beta?

@jankapunkt
Copy link
Collaborator

A beta release would be awesome so I can do some integration tests with our projects and give you early feedback

@dr-dimitru
Copy link
Member

@harryadel @bratelefant @jankapunkt 3.0.0-beta.6 is out with support for meteor@3.0.0-rc.0

@harryadel
Copy link
Contributor

Thanks for acting so quickly @dr-dimitru 👏

@harryadel
Copy link
Contributor

harryadel commented May 21, 2024

Can you make onBeforeUpload on the client side async please?
https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

cc @bratelefant @dr-dimitru

@make-github-pseudonymous-again
Copy link
Contributor

Can you make onBeforeUpload on the client side async please? https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

cc @bratelefant @dr-dimitru

I think in general all hooks should be async. Any of them might require interacting with the database which is now something that is async by default.

@make-github-pseudonymous-again
Copy link
Contributor

@bratelefant My only pain point so far with the current state of the PR is the lack of documentation: took me some time to figure out that the callback version of Meteor.write was gone. The error you get by trying to use the old signature on the new one is just a timeout, if you are lucky to have configured one (tests). This PR could have been done without this change, right? If we are aiming at a new version of ostrio:files without any callback-style functions, then we should also change the client signatures (insert, etc.).

@bratelefant
Copy link
Author

Can you make onBeforeUpload on the client side async please? https://github.com/bratelefant/Meteor-Files/blob/69cddf91c6432ac872d70218949d9537a143d667/upload.js#L688

That'd be definitely one major point to add to this pr. Did all changes with focus on the server side, client only methods did not get touched.

onBeforeUpload is an isomorphic method that runs on both server and client so sync operations would only work on client side and fail on serverside on 3.0 but async operation works on both. So, I suggest to make all the callers of isomorphic methods support async functions.

Yes, in our discussion here at some point we decided to make this an update with breaking changes by removing all sync methods on the server side.

@make-github-pseudonymous-again I tried to keep documention in the server side method comments aligned with the code changes, but that was definitely not checked agains tests / code.

I would be really grateful if someone could jump on the bandwagon and contribute to the PR, I am having a pretty 'crunchy' time at work atm

Comment on lines +1076 to +1081
let _id;
try {
_id = await this.collection.insertAsync(helpers.clone(result));
try {
await this._preCollection.updateAsync({_id: opts.fileId}, {$set: {isFinished: true}});
if (_id) result._id = _id;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it conceivable that this id-setting logic sometimes fail? I have an existing test that yields fileRef._id = undefined in the on('end') handler of FileCollection#insert (client-side call)`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onUploaded is event worse: the fileRef is simply undefined.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants