You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While trying to upgrade to Spinnaker 1.33.0 recently, we noticed that Orca and Clouddriver were unable to start due to Liquibase checksum validation throwing exceptions.
Cloud Provider(s):
Kubernetes
Environment:
Clouddriver (5.83.0 and 5.84.0) and Orca (8.48.0) with PostgreSQL 16.2 storage.
Other Spinnaker services are running with other persistence backends (Redis, S3, etc.), so no idea whether they are affected too.
Feature Area:
PostgreSQL persistence for Clouddriver and Orca.
Description:
We are in the middle of upgrading our Spinnaker clusters and of configuring SQL persistence where possible (using PostgreSQL).
Before we upgraded to Spinnaker 1.33.0, while running Spinnaker 1.13.2, we already had configured the SQL backend for Clouddriver fully, and were running a dual repostitory setup for Orca (with Redis as primary still). When we upgraded to 1.33.0, we started noticing changeset checksum validation errors from Liquibase causing the services to fail to start.
We narrowed it down for Clouddriver to the 5.83.0 releae, which seems that this also matches the timing of the Liquibase upgrade from 3.10.3 to 4.24.0 (see spinnaker/kork#1117). Perhaps partially related, it seems that 4 of the Orca changesets got updated in spinnaker/orca#4601 for managing afterColumn in PostgreSQL. I could not find a similar change for Clouddriver though.
Steps to Reproduce:
For Clouddriver we narrow the issue down to the 5.83.0 release.
With the following Clouddriver SQL and Redis configuration:
By running Clouddriver 5.82.2, the DB should be initialized with tables and the migrations are EXECUTED.
By then running Clouddriver 5.83.0, the service fails to start due to unexpected checksums.
Additional Details:
Clouddriver 5.83.0 logs
2024-04-11 08:54:23.959 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dockerRegistryImageLookupController': Unsatisfied dependency expressed through field 'cacheView'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheView' defined in class path resource [com/netflix/spinnaker/clouddriver/cache/CacheConfig.class]: Unsatisfied dependency expressed through method 'cacheView' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'catsModule' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'catsModule' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sqlAgentProvider' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'sqlAgentProvider' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sqlTableMetricsAgent' defined in class path resource [com/netflix/spinnaker/config/SqlCacheConfiguration.class]: Unsatisfied dependency expressed through method 'sqlTableMetricsAgent' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/netflix/spinnaker/kork/sql/config/DefaultSqlConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.ValidationFailedException: Validation Failed:
3 changesets check sum
db/changelog/20180919-initial-schema.yml::mysql-change-state-stauts-to-enum-type::robzienert was: 8:f0bfebd55de9168e38a8ef9c7217c610 but is now: 8:d41d8cd98f00b204e9800998ecf8427e
db/changelog/20180919-initial-schema.yml::mysql-revert-change-state-stauts-to-enum-type::afeldman was: 8:d6f5eedc195011826620cc0355e8352d but is now: 8:d41d8cd98f00b204e9800998ecf8427e
db/changelog/20190913-task-sagaids.yml::mysql-update-state-enum-values::robzienert was: 8:9601af668599fbc12e338b9b84c66f56 but is now: 8:d41d8cd98f00b204e9800998ecf8427e
Orca 8.48.0 logs
14:09:39.886 ERROR 1 --- [ main] o.s.boot.SpringApplication : [] Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'redisOrcaQueueConfiguration': Unsatisfied dependency expressed through method 'redisQueueObjectMapper' parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'taskResolver' defined in com.netflix.spinnaker.orca.config.OrcaConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.spinnaker.orca.TaskResolver]: Factory method 'taskResolver' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dependsOnExecutionTask' defined in URL [jar:file:/opt/orca/lib/orca-core-8.48.0.jar!/com/netflix/spinnaker/orca/pipeline/tasks/DependsOnExecutionTask.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dualExecutionRepository' defined in URL [jar:file:/opt/orca/lib/orca-core-8.48.0.jar!/com/netflix/spinnaker/orca/pipeline/persistence/DualExecutionRepository.class]: Unsatisfied dependency expressed through constructor parameter 4; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sqlExecutionRepository' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Unsatisfied dependency expressed through method 'sqlExecutionRepository' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/netflix/spinnaker/config/SqlConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.ValidationFailedException: Validation Failed:
6 changesets check sum
db/changelog/20180515-execution-canceled-column.yml::add-canceled-column::robzienert was: 8:4c214f01b403163ac014e78ef3603c71 but is now: 8:7758552b0792c8636c7c9b7c231a2a53
db/changelog/20180510-add-legacy-id-fields.yml::add-legacy-id-fields::cthielen was: 8:f97b552b426f284932461b028e9d6d9c but is now: 8:f598d61450cc5c4d37255ffa0242a7ba
db/changelog/20180521-status-enum.yml::modify-status-column-enum::afeldman was: 8:a927aca379a0eecf2a6eecdee281be9f but is now: 8:d41d8cd98f00b204e9800998ecf8427e
db/changelog/20180724-partitions.yml::partition-updated-executions::rzienert was: 8:26bce67df849ed409cdd5dffc335a95b but is now: 8:1f52146b80fa28e7adca3abf66b7b035
db/changelog/20181016-add-start-time.yml::20181016-add-start-time::robzienert was: 8:576429ab611ed541e5f99ab282de4177 but is now: 8:5858aae541ec6b018626e6c9f3312ca9
db/changelog/20200327-deleted-executions-table.yml::create-deleted-executions-table::mvulfson was: 8:10687930e669629d8c370cd5a2348585 but is now: 8:11b5df8d131f259ddf2292832cd12219
The text was updated successfully, but these errors were encountered:
Issue Summary:
While trying to upgrade to Spinnaker 1.33.0 recently, we noticed that Orca and Clouddriver were unable to start due to Liquibase checksum validation throwing exceptions.
Cloud Provider(s):
Kubernetes
Environment:
Clouddriver (5.83.0 and 5.84.0) and Orca (8.48.0) with PostgreSQL 16.2 storage.
Other Spinnaker services are running with other persistence backends (Redis, S3, etc.), so no idea whether they are affected too.
Feature Area:
PostgreSQL persistence for Clouddriver and Orca.
Description:
We are in the middle of upgrading our Spinnaker clusters and of configuring SQL persistence where possible (using PostgreSQL).
Before we upgraded to Spinnaker 1.33.0, while running Spinnaker 1.13.2, we already had configured the SQL backend for Clouddriver fully, and were running a dual repostitory setup for Orca (with Redis as primary still). When we upgraded to 1.33.0, we started noticing changeset checksum validation errors from Liquibase causing the services to fail to start.
We narrowed it down for Clouddriver to the 5.83.0 releae, which seems that this also matches the timing of the Liquibase upgrade from 3.10.3 to 4.24.0 (see spinnaker/kork#1117). Perhaps partially related, it seems that 4 of the Orca changesets got updated in spinnaker/orca#4601 for managing
afterColumn
in PostgreSQL. I could not find a similar change for Clouddriver though.Steps to Reproduce:
For Clouddriver we narrow the issue down to the 5.83.0 release.
With the following Clouddriver SQL and Redis configuration:
By running Clouddriver 5.82.2, the DB should be initialized with tables and the migrations are
EXECUTED
.By then running Clouddriver 5.83.0, the service fails to start due to unexpected checksums.
Additional Details:
Clouddriver 5.83.0 logs
Orca 8.48.0 logs
The text was updated successfully, but these errors were encountered: