ef migrations bundle causing runtime errors since upgrade to .Net8

[MarkTallentire]

Hello.

Very strange issue we are currently experiencing after updating our project to .Net 8.

As part of our pipelines we run ef migrations bundle with the following commands

 - task: DotNetCoreCLI@2
        displayName: Bundle Migrations (Teams)
        inputs:
          command: custom
          custom: ef
          arguments: >
            migrations bundle --self-contained -r linux-x64 --output $(Build.ArtifactStagingDirectory)/efbundle-teams --context TeamsDbContext  -p <redacted> -s .<redacted>--force
      - task: DotNetCoreCLI@2
        displayName: Bundle Migrations (Validations)
        inputs:
          command: custom
          custom: ef
          arguments: >
            migrations bundle --self-contained -r linux-x64 --output $(Build.ArtifactStagingDirectory)/efbundle-validations --context ValidationDbContext  -p <redacted> --force
        env:
          connectionstrings__teams: $(CONNECTIONSTRING__TEAMS)
          devops: "true"

This worked perfectly fine when we were running a combination of .net6 and .net7 but in .net8 it seems to be messing with the build created in the previous step of our pipeline and causing the following ReflectionTypeLoadException error which is only observed during runtime:

Unhandled exception. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Keycloak.Net, Version=16.4.0.673, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

If I remove the migrations steps from my pipeline everything builds and runs as expected. I have tried adding --no-build and messing with the other commands like runtime etc to no avail. Adding the -v flag shows everything running fine

Output from -v:

Using project '/azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/Fake.Company.Landlord.Data.csproj'.
Using startup project '/azp/_work/1/s/vNext/src/Fake.Company.Api/Fake.Company.Api.csproj'.
Writing '/azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/obj/Fake.Company.Landlord.Data.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=/tmp/tmpjg5zDL.tmp /verbosity:quiet /nologo /azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/Fake.Company.Landlord.Data.csproj
Writing '/azp/_work/1/s/vNext/src/Fake.Company.Api/obj/Fake.Company.Api.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=/tmp/tmp7jDOEN.tmp;Configuration=Release /verbosity:quiet /nologo /azp/_work/1/s/vNext/src/Fake.Company.Api/Fake.Company.Api.csproj
dotnet exec --depsfile /azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0/Fake.Company.Api.deps.json --additionalprobingpath /root/.nuget/packages --runtimeconfig /azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0/Fake.Company.Api.runtimeconfig.json /root/.dotnet/tools/.store/dotnet-ef/8.0.4/dotnet-ef/8.0.4/tools/net8.0/any/tools/netcoreapp2.0/any/ef.dll migrations bundle --self-contained -r linux-x64 --output /azp/_work/1/a/efbundle-teams --context TeamsDbContext --force --assembly /azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0/Fake.Company.Landlord.Data.dll --project /azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/Fake.Company.Landlord.Data.csproj --startup-assembly /azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0/Fake.Company.Api.dll --startup-project /azp/_work/1/s/vNext/src/Fake.Company.Api/Fake.Company.Api.csproj --project-dir /azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/ --root-namespace Fake.Company.Landlord.Data --language C# --framework net8.0 --configuration Release --nullable --working-dir /azp/_work/1/s --verbose
Using assembly 'Fake.Company.Landlord.Data'.
Using startup assembly 'Fake.Company.Api'.
Using application base '/azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0'.
Using working directory '/azp/_work/1/s/vNext/src/Fake.Company.Api'.
Using root namespace 'Fake.Company.Landlord.Data'.
Using project directory '/azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/'.
Remaining arguments: .
Finding DbContext classes...
Finding IDesignTimeDbContextFactory implementations...
Finding application service provider in assembly 'Fake.Company.Api'...
Finding Microsoft.Extensions.Hosting service provider...
Using environment 'Development'.
Using application service provider from Microsoft.Extensions.Hosting.
Found DbContext 'TeamsDbContext'.
Found DbContext 'ValidationDbContext'.
Finding DbContext classes in the project...
Using context 'TeamsDbContext'.
Building bundle...
dotnet publish --runtime linux-x64 --output /tmp/0q0zp54i.adh/publish --self-contained --configuration Release
MSBuild version 17.8.5+b5265ef37 for .NET
  Determining projects to restore...
  Restored /azp/_work/1/s/vNext/src/EdiFabric.Templates/EdiFabric.Templates.csproj (in 357 ms).
  Restored /azp/_work/1/s/Keycloak.Net/Keycloak.Net.csproj (in 504 ms).
  Restored /azp/_work/1/s/FakeCompany.Util/FakeCompany.Util.csproj (in 508 ms).
  Restored /azp/_work/1/s/HCToolkit/HCToolkit.csproj (in 721 ms).
  Restored /azp/_work/1/s/vNext/src/Fake.Company/Fake.Company.csproj (in 735 ms).
  Restored /azp/_work/1/s/vNext/src/Common/Common.csproj (in 727 ms).
  Restored /azp/_work/1/s/vNext/src/Fake.Company.EDI/Fake.Company.EDI.csproj (in 739 ms).
  Restored /azp/_work/1/s/vNext/src/Fake.Company.Tenant.Data/Fake.Company.Tenant.Data.csproj (in 749 ms).
  Restored /azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/Fake.Company.Landlord.Data.csproj (in 779 ms).
  Restored /azp/_work/1/s/vNext/src/Fake.Company.Api/Fake.Company.Api.csproj (in 764 ms).
  Restored /tmp/0q0zp54i.adh/efbundle-teams.csproj (in 797 ms).
  FakeCompany.Util -> /azp/_work/1/s/FakeCompany.Util/bin/Release/netstandard2.0/FakeCompany.Util.dll
  EdiFabric.Templates -> /azp/_work/1/s/vNext/src/EdiFabric.Templates/bin/Release/net8.0/EdiFabric.Templates.dll
  Common -> /azp/_work/1/s/vNext/src/Common/bin/Release/net8.0/Common.dll
  HCToolkit -> /azp/_work/1/s/HCToolkit/bin/Release/net8.0/HCToolkit.dll
  Keycloak.Net -> /azp/_work/1/s/Keycloak.Net/bin/Release/net6.0/Keycloak.Net.dll
  Fake.Company -> /azp/_work/1/s/vNext/src/Fake.Company/bin/Release/net8.0/Fake.Company.dll
  Fake.Company.Landlord.Data -> /azp/_work/1/s/vNext/src/Fake.Company.Landlord.Data/bin/Release/net8.0/Fake.Company.Landlord.Data.dll
  Fake.Company.EDI -> /azp/_work/1/s/vNext/src/Fake.Company.EDI/bin/Release/net8.0/Fake.Company.EDI.dll
  Fake.Company.Tenant.Data -> /azp/_work/1/s/vNext/src/Fake.Company.Tenant.Data/bin/Release/net8.0/Fake.Company.Tenant.Data.dll
  Fake.Company.Api -> /azp/_work/1/s/vNext/src/Fake.Company.Api/bin/Release/net8.0/linux-x64/Fake.Company.Api.dll
  efbundle-teams -> /tmp/0q0zp54i.adh/bin/Release/net8.0/linux-x64/efbundle-teams.dll
  efbundle-teams -> /tmp/0q0zp54i.adh/publish/
Done. Migrations Bundle: /azp/_work/1/a/efbundle-teams
Don't forget to copy appsettings.json alongside your bundle if you need it to apply migrations.

Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
Finishing: Bundle Migrations (Teams)

Pipeline in full:

name: 16.4.0.$(rev:r)

trigger:
  batch: true
  branches:
    include:
      - master
      - develop
  paths:
    exclude:
      - /vNext/web
      - /build

pool: containeragents

parameters:
  - name: configuration
    displayName: Build Configuration
    type: string
    default: Release
    values:
      - Debug
      - Release

  - name: runTests
    displayName: Run Tests?
    type: boolean
    default: true

variables:
  - name: projectsToPackage
    value: "**/<redacted>"
  - name: BuildConfiguration
    value: ${{ parameters.configuration }}
  - group: Shared

jobs:
  - job: Build
    steps:
      - checkout: self
        clean: true

      - task: MicrosoftSecurityDevOps@1
        displayName: "Microsoft Security DevOps"

      - task: DotNetCoreCLI@2
        displayName: Restore Packages
        inputs:
          command: restore
          projects: |
            $(projectsToPackage)
            **//<redacted>
            **//<redacted>
          verbosityRestore: Minimal
          arguments: >
            --configuration ${{ parameters.configuration }}
            
      - task: DotNetCoreCLI@2
        displayName: Build
        inputs:
          command: build
          projects: |
            $(projectsToPackage)
            **//<redacted>
            **//<redacted>
          arguments: >
            --configuration ${{ parameters.configuration }}
            --no-restore
            /p:VersionPrefix=$(Build.BuildNumber)
            /nodeReuse:false

      - ${{ if eq('true', parameters.runTests) }}:
          - task: DotNetCoreCLI@2
            displayName: Test
            inputs:
              command: test
              projects: |
                **//<redacted>
                **//<redacted>
                **//<redacted>
              arguments: >
                --configuration ${{ parameters.configuration }}
                --no-build
                --no-restore
                --filter Category!=Integration
                /p:ParallelizeTestCollections=true

      - task: DotNetCoreCLI@2
        displayName: Bundle Migrations (Teams)
        inputs:
          command: custom
          custom: ef
          arguments: >
            migrations bundle --self-contained -r linux-x64 --output $(Build.ArtifactStagingDirectory)/efbundle-teams --context TeamsDbContext  -p ./<redacted> -s ./<redacted> --force

      - task: DotNetCoreCLI@2
        displayName: Bundle Migrations (Validations)
        inputs:
          command: custom
          custom: ef
          arguments: >
            migrations bundle --self-contained -r linux-x64 --output $(Build.ArtifactStagingDirectory)/efbundle-validations --context ValidationDbContext  -p /<redacted> --force
        env:
          connectionstrings__teams: $(CONNECTIONSTRING__TEAMS)
          devops: "true"
        
      - task: DotNetCoreCLI@2
        displayName: Create artifacts
        inputs:
          command: publish
          publishWebProjects: False
          projects: "$(projectsToPackage)"
          arguments: >
            --no-restore
            --no-build
            --configuration ${{ parameters.configuration }}
            --output $(Build.ArtifactStagingDirectory)
          zipAfterPublish: True

      - task: CopyFiles@2
        displayName: Copy DockerFile
        inputs:
          Contents: "/<redacted>"
          TargetFolder: $(Build.ArtifactStagingDirectory)

      - task: PublishBuildArtifacts@1
        displayName: Publish artifacts
        inputs:
          ArtifactName: drop

[MarkTallentire]

After playing around a bit more the workaround here is to complete the publish step prior to the migration bundle step. I still think this is unwanted behaviour because the bundle shouldn't be messing with the built files at all.