Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: util.shell(), util.sleep() and util.spawn() do not work in preflight #5425

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Chriscbr
Copy link
Contributor

@Chriscbr Chriscbr commented Jan 5, 2024

Currently, util.shell does not behave as expected when called in preflight code because the underlying implementation returns a promise. There were a couple possible solutions for this:

  • We could restrict the function as inflight-only, but this wouldn't address use cases for performing shell operations in preflight.
  • We could change the implementation to use synchronous Node.js APIs, but this would harm performance of inflight code.

Neither of these address the issue fully. Instead, I've opted to allow unphased functions in the SDK to have both preflight and inflight implementations. The compiler is then able to use its existing phase information to emit the name of either the preflight or inflight method implementation when producing JavaScript code.

In the process of writing this, I also:

  • added phase-specific implementations for the fs module
  • added a preflight implementation of util.sleep that relies on the fact that we currently execute preflight code inside a worker thread

I'm not sure whether the second change is one that we want to keep but it's cool that it works.

Fixes #5418
Fixes #5420

Checklist

  • Title matches Winglang's style guide
  • Description explains motivation and solution
  • Tests added (always)
  • Docs updated (only required for features)
  • Added pr/e2e-full label if this feature requires end-to-end testing

By submitting this pull request, I confirm that my contribution is made under the terms of the Wing Cloud Contribution License.

@Chriscbr Chriscbr requested a review from a team as a code owner January 5, 2024 03:00
@monadabot
Copy link
Contributor

monadabot commented Jan 5, 2024

Thanks for opening this pull request! 🎉
Please consult the contributing guidelines for details on how to contribute to this project.
If you need any assistence, don't hesitate to ping the relevant owner over Discord.

Topic Owner
Wing SDK and utility APIs @chriscbr
Wing Console @ainvoner, @skyrpex, @polamoros
JSON, structs, primitives and collections @hasanaburayyan
Platforms and plugins @hasanaburayyan
Frontend resources (website, react, etc) @tsuf239
Language design @eladb
VSCode extension and language server @markmcculloh
Compiler architecture, inflights, lifting @yoav-steinberg
Wing Testing Framework @tsuf239
Wing CLI @markmcculloh
Build system, dev environment, releases @markmcculloh
Library Ecosystem @chriscbr
Documentation @hasanaburayyan
SDK test suite @tsuf239
Examples @skorfmann
Wing Playground @eladcon

@monadabot
Copy link
Contributor

monadabot commented Jan 5, 2024

Console preview environment is available at https://wing-console-pr-5425.fly.dev 🚀

Last Updated (UTC) 2024-01-05 03:51

@monadabot
Copy link
Contributor

monadabot commented Jan 5, 2024

Benchmarks

Comparison to Baseline ⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜
Benchmark Before After Change
version 73ms±1.28 73ms±0.79 +0ms (+0.49%)⬜
jsii_big.test.w -t sim 3412ms±15.93 3407ms±9.25 -6ms (-0.17%)⬜
jsii_big.test.w -t tf-aws 3493ms±5.69 3501ms±8.72 +8ms (+0.22%)⬜
functions_1.test.w -t sim 636ms±4.88 631ms±6.1 -6ms (-0.9%)⬜
functions_1.test.w -t tf-aws 1836ms±71.01 1879ms±86.33 +43ms (+2.36%)⬜
hello_world.test.w -t sim 629ms±6.33 630ms±4.3 +2ms (+0.29%)⬜
hello_world.test.w -t tf-aws 4821ms±17.62 4850ms±24.93 +30ms (+0.62%)⬜
functions_10.test.w -t sim 688ms±4.44 691ms±4.01 +3ms (+0.45%)⬜
functions_10.test.w -t tf-aws 4155ms±17.5 4177ms±13.91 +22ms (+0.52%)⬜
jsii_small.test.w -t sim 594ms±3.56 601ms±4.03 +7ms (+1.21%)⬜
jsii_small.test.w -t tf-aws 718ms±4.81 712ms±5.11 -6ms (-0.78%)⬜
empty.test.w -t sim 592ms±7.32 593ms±4.26 +1ms (+0.1%)⬜
empty.test.w -t tf-aws 706ms±4.74 709ms±4.7 +3ms (+0.42%)⬜

⬜ Within 1.5 standard deviations
🟩 Faster, Above 1.5 standard deviations
🟥 Slower, Above 1.5 standard deviations

Benchmarks may vary outside of normal expectations, especially when running in GitHub Actions CI.

Results
name mean min max moe sd
version 73ms 72ms 75ms 1ms 1ms
jsii_big.test.w -t sim 3407ms 3382ms 3425ms 9ms 13ms
jsii_big.test.w -t tf-aws 3501ms 3480ms 3518ms 9ms 12ms
functions_1.test.w -t sim 631ms 620ms 647ms 6ms 9ms
functions_1.test.w -t tf-aws 1879ms 1739ms 2100ms 86ms 121ms
hello_world.test.w -t sim 630ms 621ms 639ms 4ms 6ms
hello_world.test.w -t tf-aws 4850ms 4790ms 4898ms 25ms 35ms
functions_10.test.w -t sim 691ms 682ms 697ms 4ms 6ms
functions_10.test.w -t tf-aws 4177ms 4151ms 4209ms 14ms 19ms
jsii_small.test.w -t sim 601ms 592ms 609ms 4ms 6ms
jsii_small.test.w -t tf-aws 712ms 697ms 723ms 5ms 7ms
empty.test.w -t sim 593ms 585ms 602ms 4ms 6ms
empty.test.w -t tf-aws 709ms 698ms 718ms 5ms 7ms
Last Updated (UTC) 2024-01-18 22:03

@monadabot monadabot added the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Jan 5, 2024
libs/wingsdk/src/util/util.ts Show resolved Hide resolved
@@ -135,32 +141,37 @@ export class Util {
* @param opts `ShellOptions`, such as the working directory and environment variables.
* @returns The standard output of the shell command.
* @throws An error if the shell command execution fails or returns a non-zero exit code.
* @inflightImpl __shell__inflight
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this reminded me of python so it looked kinda weird 🌚 Doesn't really matter but maybe a more direct convention?

Suggested change
* @inflightImpl __shell__inflight
* @inflightImpl shell$$inflight

I assume a leading underscore was needed because it's marked as @internal, but I would also suggest just making it private instead. Thanks to the magic of JS, it's still accessible so it should still work.

@@ -801,6 +801,8 @@ pub struct FunctionSignature {
/// - `$args$`: the arguments passed to this function call
/// - `$args_text$`: the original source text of the arguments passed to this function call, escaped
pub js_override: Option<String>,
/// During jsify, calls to this function will be replaced with this other function name
pub inflight_impl: Option<String>,
Copy link
Contributor

@eladb eladb Jan 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in winglang this would just be two functions with the same name, correct?

class Foo {
  bar() { log("preflight");
  inflight bar() { log("inflight"); }
}

In that vane, this makes me wonder if it makes more sense to model and implement this as two function members with the same name and have the compiler select the phase-specific version (might break like a million assumptions).

Then, in the JSII side, we can decide that foo$inflight is imported as the inflight version of foo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're onto something...

Copy link
Contributor

@eladb eladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. See my thoughts about a possible alternative approach.

@eladb eladb removed the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Jan 18, 2024
@eladb
Copy link
Contributor

eladb commented Jan 18, 2024

@Chriscbr let's get this one merged :-)

Comment on lines +804 to +805
/// During jsify, calls to this function will be replaced with this other function name
pub inflight_impl: Option<String>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to try avoiding this new machinery by reusing the _toInflightType methods we already support

Copy link

github-actions bot commented Feb 8, 2024

Hi,

This PR has not seen activity in 20 days. Therefore, we are marking the PR as stale for now. It will be closed after 7 days.
If you need help with the PR, do not hesitate to reach out in the winglang community discord at winglang.discord.com.
Feel free to re-open this PR when it is still relevant and ready to be worked on again.
Thanks!

@github-actions github-actions bot added the Stale label Feb 8, 2024
@garysassano garysassano removed the Stale label Feb 8, 2024
Comment on lines +275 to +282
public static sleep(delay: Duration): void {
Atomics.wait(
new Int32Array(new SharedArrayBuffer(4)),
0,
0,
delay.seconds * 1000
);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With #6157, this approach will no longer work (preflight runs in a child process instead of a worker).
I think the only "reasonable" way left would be doing it via a shell.

@garysassano garysassano changed the title fix: util.shell, util.sleep do not work in preflight fix: util.shell(), util.sleep() and util.spawn() do not work in preflight Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

util shell does not work in preflight util.sleep is only available inflight
5 participants