Comments (3)
we noticed that if the source code were to change (like a simple test.txt file being added, or an index.js file being manipulated), our RUN yarn install would no longer be cached.
This is expected with your Dockerfile above that uses COPY .
. This will copy different files, resulting in different end build result. Different files would also be visible to the yarn install
command so it would need to run again.
Cache mismatches are common if you use COPY .
, for example git clone
does not create a .git
directory in a deterministic way. If you reduce the files used by your build to only files that you actually need, you reduce the chance of cache invalidation.
If you have a reproducible case/steps where you think cache is not matched properly you can post that and we can look at the specifics.
from buildkit.
This is expected with your Dockerfile above that uses
COPY .
. This will copy different files, resulting in different end build result. Different files would also be visible to theyarn install
command so it would need to run again.
I don't see how the COPY .
that is happening after the RUN yarn install
step invalidates the previous step?
yarn install
is only interacting with with the package.json and yarn.lock file, and running this Dockerfile locally with Docker does not produce the same cache invalidation issues as it does with buildkit.
I tested with pruning all my Docker cache, running it once, adding a test.txt file and running build again, and as shown here, the yarn install
is still cached.
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 261B 0.0s
=> [internal] load metadata for docker.io/library/node:18-alpine 0.5s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [build 1/6] FROM docker.io/library/node:18-alpine@sha256:4837c2ac8998cf172f5892fb45f229c328e4824c43c8506f8ba9c7996d702430 0.0s
=> [stage-1 1/2] WORKDIR /app 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 446B 0.0s
=> CACHED [build 2/6] WORKDIR /app 0.0s
=> CACHED [build 3/6] COPY package.json yarn.lock .npmrc /app 0.0s
=> CACHED [build 4/6] RUN yarn install --network-timeout 100000 0.0s
=> [build 5/6] COPY . /app 0.0s
=> [build 6/6] RUN yarn build 1.2s
=> CACHED [stage-1 2/2] COPY --from=build /app/dist /app/dist 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:4b77c51068465e05ea438134e137de21bbb3c69f287d78e9ffdd81c767df62cf
I'll create a reproducible case early week, just wanted to get initial thoughts out quickly 😊
from buildkit.
I haven't had the time to setup a reproducible case, as the setup has quite some configuration, so it would take a while to give an exact replica.
We did come up with something that seems to work for local cache, each image being built now stores it's cache in a separate subdirectory, meaning that no builds will ever write to the same cache directory unless it's the same image.
This seems to work, and we have had much better success with the cache not being broken.
Here is a code snippet added to the buildkit.sh
for anyone else interested:
cache_subpath="${UNIQUE_IMAGE_ID}"
buildkit_cache="${CACHE_PATH}/buildkit/${cache_subpath}"
mkdir -p $buildkit_cache
cache="--export-cache type=local,dest=$buildkit_cache,mode=max --import-cache type=local,src=$buildkit_cache"
$BUILDCTL --addr=$(cat $tmp/addr) build ${BUILD_ARGS} --frontend dockerfile.v0 --local context=. --local dockerfile=. --opt filename=./${DOCKERFILE} \
--output type=local,dest=output $cache
from buildkit.
Related Issues (20)
- `FROM --platform=$BUILDPLATFORM a as b` not overriding `TARGETPLATFORM` for multi-stage builds HOT 7
- `checksum`ing "local files" for remote builder contexts HOT 2
- Docker buildkit stuck with high CPU and unresponsive HOT 7
- COPY argument `--chmod` not working on directories
- dockerfile:1.7-labs: COPY fails where there exists broken symbolic link and --exclude is specified HOT 5
- ResourceExhausted: grpc: received message larger than max (_ vs. 4194304) HOT 1
- COPY creates file with incorrect contents HOT 2
- Th
- Getting error when trying to build multiarch images with an argoworkflow - failed to solve: process "/dev/.buildkit_qemu_emulator HOT 2
- Network Issue with using RUN in dockerfile in Windows Builds HOT 11
- Debug upload to gha cache HOT 1
- Buildkit export getting delayed on exporting layers stage HOT 2
- docs: proposal to raise awareness about an unexpected behavior of COPY --link
- [Feature Request] Add --cache-repo option to cache layers
- The heredoc syntax is not being recognized. HOT 8
- dockerfile: improve caching source metadata resolve requests
- Option to disable OpenTelemetry
- Option to disable submodules / recursive clone
- Rename JSONArgsRecommended to ExecArgsRecommended
- Verify linting rules descriptions and links HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buildkit.