Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

grillo-delmal · 2023-11-05T14:56:58Z

When you try to build an app with dub and one of their dependencies is located in a read-only directory, the whole process fails with the following message (when running with --vverbose):

Error <target file>: Read-only file system
Full exception: std.file.FileException@std/file.d(4332): <target file>: Read-only file system
----------------
??:? @trusted void std.file.copyImpl(scope const(char)[], scope const(char)[], scope const(char)*, scope const(char)*, std.typecons.Flag!("preserveAttributes").Flag) [0x7efe6d875c59]
source/dub/compilers/buildsettings.d:4251 [0x516daf]
source/dub/internal/vibecompat/core/file.d:78 [0x4d4b12]
source/dub/internal/vibecompat/core/file.d:193 [0x51722e]
source/dub/generators/build.d:509 [0x67f30c]
source/dub/generators/build.d:299 [0x67e01d]
source/dub/generators/build.d:234 [0x679e50]
source/dub/generators/build.d:174 [0x67aadc]
source/dub/generators/build.d:128 [0x67a3cf]
source/dub/generators/build.d:180 [0x678fa6]
source/dub/generators/generator.d:185 [0x4b798d]
source/dub/commandline.d:1463 [0x7b4891]
source/dub/commandline.d [0x7b4d5d]
source/dub/commandline.d:509 [0x6f9b64]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x7efe6d520b8c]
??:? _d_run_main2 [0x7efe6d5209a2]
??:? _d_run_main [0x7efe6d5207ec]
??:? [0x7efe6d17b149]
??:? __libc_start_main [0x7efe6d17b20a]
??:? [0x47d914]

This patch adds code to catch that exception and stop the dependency from being copied to the target directory, letting the build process to continue using the dependency results directly from the tmp-dir.

This might help to solve #1473

github-actions · 2023-11-05T14:57:11Z

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5331080 bin/dub
 rough build time=64s

Full build output

DUB version 1.34.0, built on Oct 15 2023
LDC - the LLVM D compiler (1.35.0):
  based on DMD v2.105.2 and LLVM 16.0.6
  built with LDC - the LLVM D compiler (1.35.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.35.0-beta.1+commit.4.g9fb664fe: building configuration [application]
source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp`
source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp`
source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which wouldn't be `@safe` because of:
/opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5331080 bin/dub
STAT:rough build time=64s

WebFreak001 · 2023-12-04T23:57:21Z

I don't think this will use the dependency from tmp-dir, right? It rather looks like it would just reuse the already present binary, if any, in that case and cause build failure due to file system issues to use old binaries, which is a no-go.

If the DUB packages are in a read-only filesystem, I think you need to rather specify --cache=local|system|user to the correct value. Ideally this should be possible to be settable using the dub settings.json, but isn't yet

grillo-delmal · 2023-12-19T15:11:55Z

I think you need to rather specify --cache=local|system|user to the correct value.

Even if you set the --cache parameter, running dub build will still fail if the dependency library sources are located in a read-only folder. This is because the process tries to move the resulting binaries from the build cache into the source folder during the build process (and thus why the exception is triggered). I have tested this on the use cases described in this repository: https://github.com/grillo-delmal/dub-system-wide-lib-tests .

What I'm trying to patch here is that, when the process correctly builds a dependency and fails to move the resulting binaries to the source code path, stop the process from crashing and continue the build process, skipping the part where the resulting dependency binaries are moved to the source directory.

In the piece of code I modified, the settings.tempBuild variable is only true if you are using dub run instead of dub build. The only thing that gets skipped by applying the patch is a call to updateCacheDatabase, which I assume its there to update the new built dependency path after it was moved correctly.

In this context, the only thing that I'm assuming is that if you don't call the updateCacheDatabase method, the build process will search for the built binary in the library build cache. I assume this considering that dub run works fine even if it doesn't need to run this code block. Please correct me if I'm wrong :)

Do you mean that, in the case where you have an older binary available in the source directory, the cache database will point to that file even if a new one was just built and is available in the build cache? Wouldn't this still be a problem if you try to run dub run without any changes?

edit: Trying to make sense x100. Sometimes even I can't understand what I write

Leave results in cache if can't write target dir

6e6bf4a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

grillo-delmal commented Nov 5, 2023

github-actions bot commented Nov 5, 2023 •

edited

WebFreak001 commented Dec 4, 2023

grillo-delmal commented Dec 19, 2023 •

edited

Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

Are you sure you want to change the base?

Leave dependencies results in tmp dir if they can't get writen to their target directory #2713

Conversation

grillo-delmal commented Nov 5, 2023

github-actions bot commented Nov 5, 2023 • edited

WebFreak001 commented Dec 4, 2023

grillo-delmal commented Dec 19, 2023 • edited

github-actions bot commented Nov 5, 2023 •

edited

grillo-delmal commented Dec 19, 2023 •

edited