Allow loading of values from .env file repo root

[jongio]

Some projects are going to want to put env vars that are shared in all envs and shipped in the repo, even as a placeholder, like this:

SOMEVALUE=

Right now we don't allow this because the user has to first create an env, then open the .env file, then copy and paste the values there.

I would like our code that loads env vars from the current env to also check and load a .env file from the root of the repo, if it exists.

Values in env/.env values would override values in root/.env, so load root/.env first and then env/.env

If we have .env file in root of repo

SOMEVALUE=FOO

And this in the env/.env file

SOMEVALUE=BAR

Then SOMEVALUE would be set to BAR

If we only have, root/.env:

SOMEVALUE=FOO

Then SOMEVALUE would be set to FOO.

[ellismg]

Spoke with @jongio and we are going change this slightly to say "a .env file next to azure.yaml" instead of "the root of a repo" but otherwise this makes sense. I have a small refactoring already of Environment as part of the ACA PR. We can do this next.

[savannahostrowski]

@ellismg @rajeshkamal5050 I think that supporting something like this would alleviate some of the headaches we're seeing around azd + environment variables (notably things like passing API keys down into Bicep). If we can load the .env in the root then a workflow like this:

1. Add API keys to the .azure/env name/.env
2. Define them in main.parameters.json
3. Define them as params at the top of main.bicep
4. Pass them into the container app module for my API
5. Define them at the top of api.bicep
6. Use them as env variables in the container app

could end up looking more like this with the simplified init flow:

1. Add API keys to the .azure/env name/.env
2. Defining them in main.parameters.json
3. Defining them as params at the top of main.bicep

[weikanglim]

@savannahostrowski The behavior you're describing would be resolved with #2739

Do we want also want to add a "global .env" as suggested by the issue? Given how remote environments have been implemented, I think the .env file needs to be in under .azure and not in the same directory as azure.yaml.

[weikanglim]

We can also take the stance that .env next to azure.yaml is a "local override", but wouldn't be completely consistent with the environment file design.

Another thing to note: resharing env files for multiple purposes can lead to env var management complexity.

[CMeeg]

I was looking for more discussions on the subject of environment variables and stumbled across this issue so I'm looking forward to seeing what changes you might make in this area.

I went through a few iterations of how to deal with env vars in my azd template, and although I'm pretty happy with the end solution I know that it doesn't address some areas (like secure varaibles e.g. from a key vault) so I think I will revisit it at some point.

My template includes a Next.js app and the main challenge I wanted to solve in my template was keeping the management of environment variables at development time exactly as per Next.js's docs, but have them flow through into azd and the IAC scripts as seamlessly as possible i.e. not having to define the same environment variable in several different places. I also needed to expose the output from the IAC scripts in the azd env files (under the .azure folder) to my app so they were available during the build/package step. The exact same process had to also work in a CI/CD envronment also.

I was able to use azd hooks and some custom scripts to achieve what I wanted and I'm happy with that approach.

The remote environments feature was released pretty much as I was completing development of my template so I need to dig into that, but reading through the comments on this ticket it sounds like the expectation for an azd template would be that the default location for managing environment variables would be inside the .azure folder.

My issue with that approach would be that you would have to change the "normal" development flow of your app to work with azd, but it feels like it should be the other way around - you should continue to develop your app as you normally would and azd should work on top of that.

For example, Next.js expects your .env* files to be situated at the root of your repo, but if I'm managing my environment variables under the .azure folder I would have to remember to edit the file under .azure and then copy the changes over to the repo root.

Am I right in my understanding that the recommendation from azd will become to manage environment variables for your apps/services inside the .azure folder?

[weikanglim]

Current date, azd's env file and the app env file are indeed separate. azd's env file holds Azure infrastructure config or output & potential input, and the app env file could contain settings for your app. This kind of design would allow separation of concerns, where your Azure environments can be managed independently than the different variations of your app.

Such usage could include:

1. You have a staging environment in Azure. What if you wanted to deploy to staging using the prod version of your app settings?
2. You want your local app to reference Azure staging instead of dev.

In the current system, suppose that if you wanted to point to Azure resources in your app, you could:

1. Define the outputs as NEXT_<VARIABLE> or WEB_<VARIABLE>.
2. Write a script that feeds azd's env file into the app env file. azd could offer a gesture in the future like: azd env export --prefix <NEXT> --to <.ENV file>
3. This still means running individual commands ad-hoc. For example:
   • Run azd provision
   • (Optional) Run azd env export --prefix <NEXT> --to <ENV file> if you'd like the app to connect remotely to Azure instead of local emulators

I do think that there can be a more streamlined experience for hosting simple static web apps. An interesting idea here is: what if azd allowed you to have an option in azure.yaml to do that we did previously, but in an automated fashion:

infra:
   env-export:
      - dev: env.development
      - staging: .env.staging
      - production: .env.production

The schema is slightly more verbose if we wanted to support prefix(es) so that azd doesn't end up overwriting your app's .env file unintentionally.

[savannahostrowski]

@CMeeg Thanks for the feedback.

My issue with that approach would be that you would have to change the "normal" development flow of your app to work with azd, but it feels like it should be the other way around - you should continue to develop your app as you normally would and azd should work on top of that.

I love that you called this out. This is something we should have top of mind when we are designing improvements here. We do strive to map to the developer's local dev paradigm where possible and play nice with other tools and conventions in their ecosystem.

@weikanglim I do sort of like your idea here. I wish we could just honour what the developer has in their project root and use that where needed. Copying things between .envs is sort of confusing for me personally and feels like azd clutter on top of what the developer has already configured.

[CMeeg]

Thanks for clarifying @weikanglim and for the feedback @savannahostrowski. It sounds like maybe I did misinterpret the intended direction that this issue is going in. I was just concerned that I would have to change how I'm developing my app to fit with azd, but it doesn't sound like that will be the case.

It sounds like how I've been treating the env files and trying to keep that separation of concerns in my template does gel with the current state of play then so that is good to know.

The flow I've ended up with is:

1. Local development uses .env files as recommended by application/framework (Next.js in my case)
2. When running azd provision a preprovision hook merges the application .env files (there can be many) into a single json file
3. The main.bicep script loads the json file so that the merged values from the application .env files can be used in the infrastructure definitions or to pass them through as environment variables to the Container App if they are needed at runtime
4. Outputs from main.bicep pass any infrastructure values out into the azd environment .env file (as normal with azd provision)
5. A postprovision hook runs to merge the azd env file with the local env file (if one exists)
6. When running azd deploy the merged env file from step 5 is copied into and used by the build "step" of the Dockerfile so that the application build process can also use the infrastructure outputs, but doesn't need to be "aware" of azd i.e. as far as the application is aware it's just a local env file, but it just happens to now also contain the values from the azd env file

When running in a pipeline I include a task that can load environment variables specific to the target environment from either GitHub environment variables or an AZDO variable group (depending on where you're running your pipeline) into a local env file so that the above flow can then run in the pipeline exactly as it would locally.

The main upside of this approach is that I can continue to manage my environment variables as I normally would, and they become automatically "available" to use in the infrastructure through the azd preprovision hook, and the output from azd is then also automatically "available" to my app through the azd postprovision hook. I still have to make code changes to use the environment variables of course, but at least I only have to manage them in one place.

The main downside I think is that a json file is probably not as "correct" as using a Bicep parameters file - this is something I am going to explore.

If I understand your ideas correctly the potential additions to azd could effectively replace step 5 above, so merge/replace a local env file with the contents of an azd env file (or certain keys based on known prefix(es)). This could be useful instead of me having to write my own scripts to do that.

The use cases around connecting my local app to different environment's resources in Azure I'm less sure of. Personally I am uneasy about having app settings from other environments like staging or production in my development environment, but I can see why this could be useful if you have provisioned your own dev environment in Azure and you want to run against those provisioned Azure resources locally rather than having to azd deploy. Especially, like you say, to avoid emulators or in some cases use Azure resources that don't have emulators.

I think what I was hoping for when I first saw this issue was that azd would at some point support the flow of environment variables that I have now in my template, but through built-in commands or behaviour. The amount of variance in how env files work in each application/service/framework that azd wants to support must make this really hard to make a general solution for though so to be honest I'm really happy that the hooks provided by azd gives me the flexibility to implement what I have in the first place!

[rajeshkamal5050]

@savannahostrowski @weikanglim Not a must-fix for 1.5.0. Adding it to Ge bucket. Please pull it in when appropriate.