Emily API Local Environment #150

AshtonStephens · 2024-05-14T01:05:58Z

Emily API Local Environment

Description

This PR adds a local Emily development environment.

Add Emily integration test environment - necessary
Add Docker scripts to modify local environment - necessary
Change CDK Lambda to use x86 when running locally on an x86 machine - necessary
Optimizes Makefile caching - really nice

Note that this may still not work for some machines with weird docker setups.

Makefile Optimization

I included Makefile caching because running cargo-lambda takes a while (around 1 minute) when cargo believes that the sources for emily/lambda or the dependencies of the sources have been changed. We've been using Makefiles incorrectly so far; each target is loosely meant to be a file, and in cases where the target isn't a file you're meant to make it a dependency of .PHONEY.

For example, this command doesn't make the file install so it's departing from the "file as target" paradigm.

install: 
	pnpm install
.PHONEY: install

Makefiles define a target to be out of date when the target corresponds to a file that has a "last modified" timestamp earlier than the targets it's dependent on OR if any of the dependencies are out of date. Note that the definition of out of date is:

Recursive - so for caching to work all dependencies all the way down need to cache properly
Entirely dependent on file timestamps - so all dependencies should be tied to a file
An approach to this that takes advantage of the caching for installation needs to associate the installation with a file that can be updated when there's a dependency change. Here's the updated installation task from this PR:

INSTALL_TARGET := pnpm-lock.yaml
$(INSTALL_TARGET): $(wildcard package* */package* */*/package*)
	pnpm install
	touch pnpm-lock.yaml

Here we're using the last updated timestamp of the pnpm-lock.yaml to indicate if the installation is out of date or needs to be rerun because the package.json files have been modified. I've kept the install target because when the repository is first pulled in the "last modified" timestamp for pnpm-lock.yaml and the package.json files will have the same timestamp so make will refuse to reinstall because the target is up to date, the install command forgoes that.

Other changes

Comments added in the files

Testing

Ran the integration environment from a fresh repository by running the following:

make install && make emily-integration-test

and then hit it with

curl http://127.0.0.1:3000/withdrawals | jq

AshtonStephens · 2024-05-16T13:32:38Z

emily/api-definition/package.json

@@ -3,7 +3,7 @@
  "scripts": {
    "build-openapi": "smithy build",
    "clean-openapi": "smithy clean",
-    "build-rs": "openapi-generator-cli generate -i ../../.generated-sources/smithy/openapi/openapi/Emily.openapi.json -g rust -c rust-config.json -o ../../.generated-sources/emily",
+    "build-rs": "npx @openapitools/openapi-generator-cli generate -i ../../.generated-sources/smithy/openapi/openapi/Emily.openapi.json -g rust -c rust-config.json -o ../../.generated-sources/emily",


This is necessary to prevent people from needing to install the openapi-generator-cli locally.

@hstove FYI, feel a bit bad about reversing this change especially since I approved it.

devenv/aws-setup/initialize.py

AshtonStephens · 2024-05-16T13:57:00Z

devenv/aws-setup/initialize.py

+    template = json.load(fp)
+
+# Create convenience mapping.
+template_resource_ids_for_type = defaultdict(list)


The resource ids used as a key for a resource in the template is actually a unique token that is associated with the resource for within the template, but there's a separate AWS resource id, which is why this refers to template_resource_id and not resource_id.

template_resource_token could be a better name, but it is an identifier.

AshtonStephens · 2024-05-16T13:59:02Z

emily/cdk/lib/emily-stack-utils.ts

            throw new Error('Must define AWS account on either "AWS_ACCOUNT" or "CDK_DEFAULT_ACCOUNT" env variables.');
        }
+        // If the account is undefined and we're letting it slide set the account to 12 "x"s.


The local environment needs this to be set, but it doesn't matter what it is. Someone who hasn't set up AWS locally shouldn't need to setup AWS CLI to run the local environment if it isn't strictly necessary.

netrome

I don't fully follow everything that is happening here, but generally it seems to look alright. The only showstopper for me is calling sudo from within the makefile. Would it be possible to skip that?

devenv/aws-setup/initialize.py

netrome · 2024-05-16T17:12:41Z

docker-compose.emily.yml

+      - AWS_ACCESS_KEY_ID=xxxxxxxxxxxx
+      - AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxx


I assume these placeholders doesn't do anything. Can we omit them completely?

Nope, they need to be set unfortunately. (boto3 vomits otherwise)

Maybe I could set them in the script...

Actually ignore me, you can set it in the client.

Makefile

djordon

Looks good, I did have one question though.

djordon · 2024-05-16T18:56:32Z

emily/cdk/package.json

@@ -5,16 +5,15 @@
    "emily": "bin/emily.js"
  },
  "scripts": {
-    "build": "(cd ../api-definition && npm run build-openapi) && tsc",
+    "synth": "tsc && npx aws-cdk synth -q --no-staging",


So this command generates the CloudFormation files now? Maybe this is obvious, but why is build no longer necessary?

The original "build" did something a little misleading.

It would:

make the openapi template from the api-description sibling directory

Compile the typescript to javascript

That just builds the sources, but it doesn't build any cdk artifact. Normally you'd run npm run build and then run some command like cdk deploy or cdk synth that utilize the generated javascript.cdk deploy synthesizes the template and then deploys the artifacts to AWS, and the cdk synth command just makes the template.

I've changed this so that it'll take the typescript sources and turn them into a the cdk template right off the bat and assumes that the openapi template has already been generated. This is a better separation of responsibilities given the new structure of the Makefile. In the build command before there was no usable output (no template) and it modified sources even when it didn't need to be rebuilt.

I also changed the name to synth because it better matches the cdk idioms.

I see. Thank you for the explanation, I did not know that.

So going forward, when we do make build it would not build the cdk template files anymore. That would get done if we run emily integration tests, setup a dev env (but with local configuration), or when/if we add some commands to deploy.

djordon

Looks good!

docker-compose.emily.yml

.generated-sources/emily/build.rs

netrome

I'd love to do a workshop/lab to go over everything here. I can spin things up locally now, but if I try to curl Emily I only get server errors.

curl http://localhost:3000/deposits | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    36  100    36    0     0      2      0  0:00:18  0:00:12  0:00:06     9
{
  "message": "Internal server error"
}

with the logs

Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT environment variable. The current timeout is 10.0 (seconds).

I'm probably just missing something here. Looking forward to learning more about this!

netrome · 2024-05-21T18:57:25Z

devenv/aws-setup/initialize.py

+def main():
+    """
+    Main function to read a CDK template, create missing DynamoDB tables,
+    replace local Lambda paths, and write the modified template.
+    """
+    ddb_client = boto3.client(
+        'dynamodb',
+        endpoint_url=DYNAMODB_ENDPOINT,
+        region_name="us-west-2",
+        aws_access_key_id="xxxxxxxx",
+        aws_secret_access_key="xxxxxxxx",
+    )
+    template = read_template(INPUT_CDK_TEMPLATE)
+    create_missing_tables(template, ddb_client)
+    modified_template = replace_local_lambda_path(template, LOCAL_LAMBDA_PATH)
+    write_template(modified_template, OUTPUT_CDK_TEMPLATE)


Beautiful. Thank you for refactoring this 🙏 It's much easier for me to understand what's happening now.

Makefile

AshtonStephens · 2024-05-21T20:12:36Z

I'd love to do a workshop/lab to go over everything here. I can spin things up locally now, but if I try to curl Emily I only get server errors.
curl http://localhost:3000/deposits | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    36  100    36    0     0      2      0  0:00:18  0:00:12  0:00:06     9
{
  "message": "Internal server error"
}
with the logs
Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT environment variable. The current timeout is 10.0 (seconds).
I'm probably just missing something here. Looking forward to learning more about this!

This is what I talked about on the call earlier; there's an issue on Linux computers that I wasn't able to resolve, and I've had very poor luck debugging it. There aren't any tests in CI and most of us are using MacOS which makes me consider this temporarily okay, but I need to find a resolution. I plan to track that in #74

The SAM docker container which I've named apigateway is running the sam cli which simulates a lambda by spinning the code up in a docker container that emulates an AWS lambda. Normally you'd run sam from your local command line but for a full dev environment it's better to have it as a container.

By default the SAM cli will create the lambda instance as a child container of the one running the cli, but since we have the cli running in its own docker container it needs to make a sibling container as opposed to a child container. There are a number of differences between MacOS and Linux when it comes to docker, and I think I've caught and handled most of them, but for whatever reason when the sam cli launches the sibling lambda container on MacOS the cli can connect to it, but it cannot connect to it on Linux.

If the sam cli is running outside of a container on Linux it will work, so I believe it's safe to say it's an issue with the way the network is set up, but I've tried a lot of things and nothing seems to work:

External docker network for the compose.
User permissions on the bootstrap binary.
Variations of host IP config options in the sam local-api start command.

netrome · 2024-05-22T06:59:26Z

I'd love to do a workshop/lab to go over everything here. I can spin things up locally now, but if I try to curl Emily I only get server errors.
curl http://localhost:3000/deposits | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    36  100    36    0     0      2      0  0:00:18  0:00:12  0:00:06     9
{
  "message": "Internal server error"
}
with the logs
Timed out while attempting to establish a connection to the container. You can increase this timeout by setting the SAM_CLI_CONTAINER_CONNECTION_TIMEOUT environment variable. The current timeout is 10.0 (seconds).
I'm probably just missing something here. Looking forward to learning more about this!
This is what I talked about on the call earlier; there's an issue on Linux computers that I wasn't able to resolve, and I've had very poor luck debugging it. There aren't any tests in CI and most of us are using MacOS which makes me consider this temporarily okay, but I need to find a resolution. I plan to track that in #74

The SAM docker container which I've named apigateway is running the sam cli which simulates a lambda by spinning the code up in a docker container that emulates an AWS lambda. Normally you'd run sam from your local command line but for a full dev environment it's better to have it as a container.

By default the SAM cli will create the lambda instance as a child container of the one running the cli, but since we have the cli running in its own docker container it needs to make a sibling container as opposed to a child container. There are a number of differences between MacOS and Linux when it comes to docker, and I think I've caught and handled most of them, but for whatever reason when the sam cli launches the sibling lambda container on MacOS the cli can connect to it, but it cannot connect to it on Linux.

If the sam cli is running outside of a container on Linux it will work, so I believe it's safe to say it's an issue with the way the network is set up, but I've tried a lot of things and nothing seems to work:
* External docker network for the compose.

* User permissions on the bootstrap binary.

* Variations of host IP config options in the `sam local-api start` command.

Ah I see. Thank you for the explanation, and nice that you created a ticket to track this. Then it also sounds like I could work around this for the time being by using the sam CLI on my machine instead of in docker.

radicleart · 2024-05-22T08:13:48Z

If the sam cli is running outside of a container on Linux it will work, so I believe it's safe to say it's an issue with the way the network is set up, but I've tried a lot of things and nothing seems to work:

Is the problem calling host services? If so running the container with network host flag on linux might work?

docker run --network host <your-docker-image>

// or in compose
network_mode: host

djordon · 2024-05-22T12:13:07Z

If the sam cli is running outside of a container on Linux it will work, so I believe it's safe to say it's an issue with the way the network is set up, but I've tried a lot of things and nothing seems to work:

Is the problem calling host services? If so running the container with network host flag on linux might work?
docker run --network host <your-docker-image>

// or in compose
network_mode: host

I was going to suggest trying this:

    extra_hosts:
      - "host.docker.internal:host-gateway"

But I'm not sure about the differences between the two.

AshtonStephens · 2024-05-22T18:19:32Z

Note: @netrome @djordon I made two changes after merging with the latest change from the blocklist client:

Makefile change to use Makefile caching on the blocklist client.
GitHub action change to only run the required Make steps.

This PR ended up changing more than I wanted because it introduced a new make paradigm and that meant that the new changes that used the old paradigm needed to be altered.

AshtonStephens · 2024-05-22T18:25:40Z

Makefile

+AUTOGENERATED_BLOCKLIST_CLIENT_CLIENT := .generated-sources/blocklist-api/src/lib.rs
+BLOCKLIST_OPENAPI_PATH=./.generated-sources/blocklist-openapi-gen
+BLOCKLIST_OPENAPI_SPEC=$(BLOCKLIST_OPENAPI_PATH)/blocklist-client-openapi.json


AshtonStephens

Highlighed the new things.

AshtonStephens · 2024-05-22T18:26:44Z

Makefile

-BLOCKLIST_OPENAPI_PATH=./.generated-sources/blocklist-openapi-gen
+# Generate the client code using the OpenAPI spec
+$(AUTOGENERATED_BLOCKLIST_CLIENT_CLIENT): $(BLOCKLIST_OPENAPI_SPEC)
+	pnpm --prefix $(BLOCKLIST_OPENAPI_PATH) run build

 # Generate the OpenAPI spec for Blocklist Client
-generate-openapi-spec: install
+$(BLOCKLIST_OPENAPI_SPEC): $(INSTALL_TARGET) $(filter-out $(BLOCKLIST_OPENAPI_SPEC), $(wildcard $(subst dir, $(BLOCKLIST_OPENAPI_PATH), $(THREE_DIRS_DEEP))))
 	cd ./.generated-sources/blocklist-openapi-gen && cargo build
-
-# Generate the client code using the OpenAPI spec
-blocklist-api-source: generate-openapi-spec
-	pnpm --prefix $(BLOCKLIST_OPENAPI_PATH) run build


AshtonStephens · 2024-05-22T18:27:05Z

.github/workflows/unit-tests.yaml

-      - name: Generate OpenAPI spec for blocklist client
-        run: make generate-openapi-spec
+      - name: Run Basic tests
+        run: make test

-      - name: Generate blocklist API client
-        run: make blocklist-api-source
-
-      - name: Run tests
+      - name: Run Integration tests


djordon

I think our CI is doing cargo test twice now but besides that it looks good.

djordon · 2024-05-22T19:14:50Z

Makefile

+ifeq ($(findstring Linux, $(shell uname)), Linux)
+_CONTAINER_HOST := localhost
+else
+_CONTAINER_HOST := host.docker.internal


Oh interesting, I had thought the host.docker.internal path would work on both macOS and linux. Good to know.

djordon · 2024-05-22T19:16:57Z

.github/workflows/unit-tests.yaml

+      - name: Run Basic tests
+        run: make test


Right now the make integration-tests command runs the basic tests as well. We can either remove this or update the stanza in the yaml or update the command. I'm okay with either.

I'll remove the test command then push it.

AshtonStephens · 2024-05-22T19:21:14Z

I see the suggestions on how to fix the issues on Linux computers and will use them in #74, right now just trying to get this through.

AshtonStephens added the emily API that communicates with Signers to trigger sBTC operations. label May 14, 2024

AshtonStephens self-assigned this May 14, 2024

AshtonStephens linked an issue May 14, 2024 that may be closed by this pull request

Deposit API Local environment for ARM #73

Closed

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch 6 times, most recently from 03d3506 to 52a52e1 Compare May 16, 2024 12:35

AshtonStephens requested review from djordon and netrome May 16, 2024 13:23

AshtonStephens commented May 16, 2024

View reviewed changes

AshtonStephens marked this pull request as ready for review May 16, 2024 13:38

AshtonStephens commented May 16, 2024

View reviewed changes

devenv/aws-setup/initialize.py Show resolved Hide resolved

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 52a52e1 to d9e07a8 Compare May 16, 2024 13:53

AshtonStephens commented May 16, 2024

View reviewed changes

AshtonStephens changed the title ~~Emily API Local Environment - DRAFT~~ Emily API Local Environment May 16, 2024

netrome requested changes May 16, 2024

View reviewed changes

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from d9e07a8 to 183211a Compare May 16, 2024 19:04

djordon reviewed May 16, 2024

View reviewed changes

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 183211a to 4958fe1 Compare May 16, 2024 21:23

djordon approved these changes May 17, 2024

View reviewed changes

djordon reviewed May 17, 2024

View reviewed changes

docker-compose.emily.yml Show resolved Hide resolved

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 4958fe1 to 7f40341 Compare May 17, 2024 21:48

netrome reviewed May 20, 2024

View reviewed changes

.generated-sources/emily/build.rs Outdated Show resolved Hide resolved

AshtonStephens requested a review from netrome May 21, 2024 13:56

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 805497a to 3339bc2 Compare May 21, 2024 18:45

netrome approved these changes May 21, 2024

View reviewed changes

AshtonStephens added 7 commits May 22, 2024 09:33

feat: Update emily client build.rs to only add clippy directives.

b914879

feat: Update aws-cdk version

ad61662

feat: Emily API local environment

cb0408d

feat: Optimize Makefile and include Emily dev environment tasks

a0de5d4

feat: Change lambda architecture for local stage on x machines

95810ff

feat: Add fallback to missing environment variables for local testing

461b768

Change container host when on a linux computer for emily env

910db38

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 3339bc2 to 910db38 Compare May 22, 2024 18:10

AshtonStephens commented May 22, 2024

View reviewed changes

djordon approved these changes May 22, 2024

View reviewed changes

feat: Replace redundant GitHub Workflow step with greater make command.

4fa8075

AshtonStephens force-pushed the 73-deposit-api-local-environment-for-arm branch from 2e3ad3d to 4fa8075 Compare May 22, 2024 19:20

AshtonStephens merged commit 572f9d5 into main May 22, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emily API Local Environment #150

Emily API Local Environment #150

AshtonStephens commented May 14, 2024 •

edited

AshtonStephens May 16, 2024

AshtonStephens May 16, 2024

AshtonStephens May 16, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

netrome left a comment

netrome May 16, 2024

AshtonStephens May 16, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

netrome May 16, 2024

djordon left a comment

djordon May 16, 2024

AshtonStephens May 16, 2024

djordon May 17, 2024 •

edited

djordon left a comment

netrome left a comment

netrome May 21, 2024

AshtonStephens commented May 21, 2024

netrome commented May 22, 2024

radicleart commented May 22, 2024 •

edited

djordon commented May 22, 2024

AshtonStephens commented May 22, 2024

AshtonStephens May 22, 2024

AshtonStephens left a comment

AshtonStephens May 22, 2024

AshtonStephens May 22, 2024

djordon left a comment

djordon May 22, 2024

djordon May 22, 2024

AshtonStephens May 22, 2024

AshtonStephens commented May 22, 2024

		- AWS_ACCESS_KEY_ID=xxxxxxxxxxxx
		- AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxx

Emily API Local Environment #150

Emily API Local Environment #150

Conversation

AshtonStephens commented May 14, 2024 • edited

Emily API Local Environment

Description

Makefile Optimization

Other changes

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AshtonStephens May 16, 2024 • edited

Choose a reason for hiding this comment

AshtonStephens May 16, 2024 • edited

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AshtonStephens May 16, 2024 • edited

Choose a reason for hiding this comment

AshtonStephens May 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djordon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djordon May 17, 2024 • edited

Choose a reason for hiding this comment

djordon left a comment

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AshtonStephens commented May 21, 2024

netrome commented May 22, 2024

radicleart commented May 22, 2024 • edited

djordon commented May 22, 2024

AshtonStephens commented May 22, 2024

Choose a reason for hiding this comment

AshtonStephens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djordon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AshtonStephens commented May 22, 2024

AshtonStephens commented May 14, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

AshtonStephens May 16, 2024 •

edited

djordon May 17, 2024 •

edited

radicleart commented May 22, 2024 •

edited