You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running a command, for example astro dev pytest the astro cli invokes a docker build command using the project's Dockerfile as a base.
To speed up these commands, we install dbt into its own stage. Our Dockerfile looks like this:
FROM quay.io/astronomer/astro-runtime:9.5.0 as dbt_build
ADD ./dbt_requirements.txt ./dbt_requirements.txt
RUN python -m venv --copies dbt_venv && source dbt_venv/bin/activate && \
pip install --no-cache-dir -r dbt_requirements.txt && deactivate
FROM quay.io/astronomer/astro-runtime:9.5.0
COPY --chown=astro:0 --from=dbt_build /usr/local/airflow/dbt_venv dbt_venv
COPY --chown=astro:0 docker/aws_config /home/astro/.aws/config
ENV PATH="${AIRFLOW_HOME}/dbt_venv/bin:${PATH}"
dbt_requirements.txt contents is:
dbt-snowflake==1.6.4
The cli injects docker commands after each FROM quay.io/astronomer/astro-runtime:9.5.0 call, looking at the logs they are:
COPY packages.txt .
RUN if [[ -s packages.txt ]]; then apt-get update && cat packages.txt | tr '\r\n' '\n' | sed -e 's/#.*//' | xargs apt-get install -y --no-install-recommends && apt-get clean && rm -rf /var/lib/apt/lists/*; fi
COPY requirements.txt .
RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!"; exit 1; fi; pip install --no-cache-dir --root-user-action=ignore -r requirements.txt
COPY --chown=astro:0 . .
The last command COPY --chown=astro:0 . . is really slowing down any astro dev command because it causes the dbt_build stage venv creation step cache to invalidate if any file in the project folder changes.
This could be resolved by omitting the final COPY --chown=astro:0 . . command from build stages that are not part of the final build image.
What CLI Version did you experience this bug?
Astro CLI Version: 1.19.1
This CLI bug is related to which Astronomer Platform?
[ x ] Astro
[ x ] Software
[ x ] None/Unknown
What Operating System is the above CLI installed on?
MacOs & Linux
🪜 Steps To Reproduce
Use the Dockerfile pasted above
Run astro dev pytest
Run astro dev pytest again, observe building the Docker container build is fully cached
Change any file except dbt_requirements.txt, requirements.txt and packages.txt, observe the Docker container build cache is invalidated
The text was updated successfully, but these errors were encountered:
Describe the bug
When running a command, for example
astro dev pytest
the astro cli invokes a docker build command using the project'sDockerfile
as a base.To speed up these commands, we install dbt into its own stage. Our
Dockerfile
looks like this:dbt_requirements.txt
contents is:The cli injects docker commands after each
FROM quay.io/astronomer/astro-runtime:9.5.0
call, looking at the logs they are:The last command
COPY --chown=astro:0 . .
is really slowing down anyastro dev
command because it causes thedbt_build
stage venv creation step cache to invalidate if any file in the project folder changes.This could be resolved by omitting the final
COPY --chown=astro:0 . .
command from build stages that are not part of the final build image.What CLI Version did you experience this bug?
Astro CLI Version: 1.19.1
This CLI bug is related to which Astronomer Platform?
What Operating System is the above CLI installed on?
MacOs & Linux
🪜 Steps To Reproduce
Dockerfile
pasted aboveastro dev pytest
astro dev pytest
again, observe building the Docker container build is fully cacheddbt_requirements.txt
,requirements.txt
andpackages.txt
, observe the Docker container build cache is invalidatedThe text was updated successfully, but these errors were encountered: