Releases: nasa/cumulus
v11.1.2
Release v11.1.2
Please note changes in 11.1.2 may not yet be released in future versions, as
this is a backport and patch release on the 11.1.x series of releases. Updates that
are included in the future will have a corresponding CHANGELOG entry in future
releases.
MIGRATION NOTES
-
The changes introduced in CUMULUS-2955 should result in removal of
files_granule_cumulus_id_index
from thefiles
table (added in the v11.1.1
release). The success of this operation is dependent on system ingest loadIn rare cases where data-persistence deployment fails because the
postgres-db-migration
times out, it may be required to manually remove the
index and then redeploy:
DROP INDEX IF EXISTS files_granule_cumulus_id_index;
Changed
- CUMULUS-2955
- Updates
20220126172008_files_granule_id_index
to not create an index on
granule_cumulus_id
on the files table. - Adds
20220609024044_remove_files_granule_id_index
migration to revert
changes from20220126172008_files_granule_id_index
on any deployed stacks
that might have the index to ensure consistency in deployed stacks
- Updates
v12.0.0
Breaking Changes
- CUMULUS-2903
- The minimum supported version for all published Cumulus Core npm packages is now Node 14.19.1
- Tasks using the
cumuluss/cumulus-ecs-task
Docker image must be updated to
cumuluss/cumulus-ecs-task:1.8.0
. This can be done by updating theimage
property of any tasks defined using thecumulus_ecs_service
Terraform
module.
Changed
- CUMULUS-2932
- Updates
SyncGranule
task to includedisableOrDefaultAcl
function that uses
the configuration ACL parameter to set ACL to private by default or disable ACL. - Updates
@cumulus/sync-granule
download()
function to take in ACL parameter - Updates
@cumulus/ingest
proceed()
function to take in ACL parameter - Updates
@cumulus/ingest
addLock()
function to take in an optional ACL parameter - Updates
SyncGranule
example worfklow config
example/cumulus-tf/sync_granule_workflow.asl.json
to includeACL
parameter.
- Updates
v11.1.1
Release v11.1.1
Added
Changed
-
CUMULUS-2885
- Updated @cumulus/aws-client to use new AWS SDK v3 packages for S3 requests:
- @aws-sdk/client-s3
- @aws-sdk/lib-storage
- @aws-sdk/s3-request-presigner
- Updated code for compatibility with updated @cumulus/aws-client and AWS SDK v3 S3 packages:
- @cumulus/api
- @cumulus/async-operations
- @cumulus/cmrjs
- @cumulus/common
- @cumulus/collection-config-store
- @cumulus/ingest
- @cumulus/launchpad-auth
- @cumulus/sftp-client
- @cumulus/tf-inventory
- lambdas/data-migration2
- tasks/add-missing-file-checksums
- tasks/hyrax-metadata-updates
- tasks/lzards-backup
- tasks/sync-granule
- Updated @cumulus/aws-client to use new AWS SDK v3 packages for S3 requests:
-
CUMULUS-2920
- Update npm version for Core build to 8.6
-
CUMULUS-2922
- Added @cumulus/example-lib package to example project to allow unit tests example/script/lib dependency.
- Updates Mutex unit test to address changes made in #2902
-
CUMULUS-2924
- Update acquireTimeoutMillis to 400 seconds for the db-provision-lambda module to address potential timeout issues on RDS database start
-
CUMULUS-2925
- Updates CI to utilize audit-ci v6.2.0
- Updates CI to utilize a on-container filesystem when building Core in 'uncached' mode
- Updates CI to selectively bootstrap Core modules in the cleanup job phase
v10.0.2
Release v10.0.2
Please note changes in 10.0.2 may not yet be released in future versions, as
this is a backport and patch release on the 10.0.x series of releases. Updates that
are included in the future will have a corresponding CHANGELOG entry in future
releases.
Changed
- CUMULUS-2837
- Update process-s3-dead-letter-archive to unpack SQS events in addition to
Cumulus Messages - Update process-s3-dead-letter-archive to look up execution status using
getCumulusMessageFromExecutionEvent (common method with sfEventSqsToDbRecords) - Move methods in api/lib/cwSfExecutionEventUtils to
@cumulus/message/StepFunctions
- Update process-s3-dead-letter-archive to unpack SQS events in addition to
v11.1.0
Release v11.1.0
MIGRATION NOTES
-
11.1.0 is an amendment release and supersedes 11.0.0. However, follow the migration steps for 11.0.0 when updating from prior versions.
-
CUMULUS-2905
- Updates migration script with new
migrateAndOverwrite
and
migrateOnlyFiles
options.
- Updates migration script with new
Added
- CUMULUS-2860
- Added an optional configuration parameter
skipMetadataValidation
tohyrax-metadata-updates
task
- Added an optional configuration parameter
- CUMULUS-2870
- Added
last_modified_date
as output to all tasks in Terraformingest
module.
- Added
- CUMULUS-NONE
- Added documentation on choosing and configuring RDS at
deployment/choosing_configuring_rds
.
- Added documentation on choosing and configuring RDS at
Changed
- CUMULUS-2703
- Updated
ORCA Backup
reconciliation report to reportcumulusFilesCount
andorcaFilesCount
- Updated
- CUMULUS-2849
- Updated
@cumulus/aws-client
to use new AWS SDK v3 packages for DynamoDB requests:@aws-sdk/client-dynamodb
@aws-sdk/lib-dynamodb
@aws-sdk/util-dynamodb
- Updated code for compatibility with AWS SDK v3 Dynamo packages
@cumulus/api
@cumulus/errors
@cumulus/tf-inventory
lambdas/data-migration2
packages/api/ecs/async-operation
- Updated
- CUMULUS-2864
- Updated
@cumulus/cmr-client/ingestUMMGranule
and@cumulus/cmr-client/ingestConcept
functions to not perform separate validation request
- Updated
- CUMULUS-2870
- Updated
hello_world_service
module to pass inlastModified
parameter in command list to trigger a Terraform state change when thehello_world_task
is modified.
- Updated
Fixed
- CUMULUS-2849
- Fixed AWS service client memoization logic in
@cumulus/aws-client
- Fixed AWS service client memoization logic in
v11.0.0
Release v11.0.0
Please note - this release has been superseded by release v11.1.0.
Users upgrading to V11 should upgrade instead to at least v11.1.0, but should follow the migration instructions noted here.
v9.9->v11.0 MIGRATION NOTES
Release v11.0 is a maintenance release series, replacing v9.9. If you are
upgrading to or past v11 from v9.9.x to this release, please pay attention to the following
migration notes from prior releases:
Migration steps
After deploying the data-persistence
module, but before deploying the main cumulus
module
- Due to a bug in the PUT
/rules/<name>
endpoint, the rule records in PostgreSQL may be
out of sync with records in DynamoDB. In order to bring the records into sync, re-deploy and re-run the
data-migration1
Lambda with a payload of
{"forceRulesMigration": true}
:
aws lambda invoke --function-name $PREFIX-data-migration1 \
--payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE
As part of the cumulus
deployment
- Please read the documentation on the updates to the granule files schema for our Cumulus workflow tasks and how to upgrade your deployment for compatibility.
- (Optional) Update the
task-config
for all workflows that use thesync-granule
task to includeworkflowStartTime
set to
{$.cumulus_meta.workflow_start_time}
. See here for an example.
After the cumulus
deployment
As part of the work on the RDS Phase 2 feature, it was decided to re-add the
granule file type
property on the file table (detailed reasoning
https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=219186829). This
change was implemented as part of CUMULUS-2672/CUMULUS-2673, however granule
records ingested prior to v11 will not have the file.type property stored in the
PostGreSQL database, and on installation of v11 API calls to get granule.files
will not return this value. We anticipate most users are impacted by this issue.
Users that are impacted by these changes should re-run the granule migration
lambda to only migrate granule file records:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateOnlyFiles": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
You should note that this will only move files for granule records in
PostgreSQL. If you have not completed the phase 1 data migration or
have granule records in dynamo that are not in PostgreSQL, the migration will
report failure for both the DynamoDB granule and all the associated files and the file
records will not be updated.
If you prefer to do a full granule and file migration, you may instead
opt to run the migration with the migrateAndOverwrite
option instead, this will re-run a
full granule/files migration and overwrite all values in the PostgreSQL database from
what is in DynamoDB for both granules and associated files:
PAYLOAD=$(echo '{"migrationsList": ["granules"], "granuleMigrationParams": {"migrateAndOverwrite": "true"}}' | base64)
aws lambda invoke --function-name $PREFIX-postgres-migration-async-operation \
--payload $PAYLOAD $OUTFILE
Please note: Since this data migration is copying all of your granule data
from DynamoDB to PostgreSQL, it can take multiple hours (or even days) to run,
depending on how much data you have and how much parallelism you configure the
migration to use. In general, the more parallelism you configure the migration
to use, the faster it will go, but the higher load it will put on your
PostgreSQL database. Excessive database load can cause database outages and
result in data loss/recovery scenarios. Thus, the parallelism settings for the
migration are intentionally set by default to conservative values but are
configurable. If this impacts only some of your data products you may want
to consider using other granuleMigrationParams
.
Please see the second data migration
docs
for more on this tool if you are unfamiliar with the various options.
Notable changes
- CUMULUS-2703
ORCA Backup
is now a supportedreportType
for thePOST /reconciliationReports
endpoint
Added
- CUMULUS-2311 - RDS Migration Epic Phase 2
- CUMULUS-2208
- Added
@cumulus/message/utils.parseException
to parse exception objects - Added helpers to
@cumulus/message/Granules
:getGranuleProductVolume
getGranuleTimeToPreprocess
getGranuleTimeToArchive
generateGranuleApiRecord
- Added
@cumulus/message/PDRs/generatePdrApiRecordFromMessage
to generate PDR from Cumulus workflow message - Added helpers to
@cumulus/es-client/indexer
:deleteAsyncOperation
to delete async operation records from ElasticsearchupdateAsyncOperation
to update an async operation record in Elasticsearch
- Added granules
PUT
endpoint to Cumulus API for updating a granule.
Requests to this endpoint should be submitted without anaction
attribute in the request body. - Added
@cumulus/api-client/granules.updateGranule
to update granule via the API
- Added
- CUMULUS-2303
- Add translatePostgresProviderToApiProvider method to
@cumulus/db/translate/providers
- Add translatePostgresProviderToApiProvider method to
- CUMULUS-2306
- Updated API execution GET endpoint to read individual execution records
from PostgreSQL database instead of DynamoDB - Updated API execution-status endpoint to read execution records from
PostgreSQL database instead of DynamoDB
- Updated API execution GET endpoint to read individual execution records
- CUMULUS-2302
- Added translatePostgresCollectionToApiCollection method to
@cumulus/db/translate/collections
- Added
searchWithUpdatedAtRange
method to
@cumulus/db/models/collections
- Added translatePostgresCollectionToApiCollection method to
- CUMULUS-2301
- Created API asyncOperations POST endpoint to create async operations.
- CUMULUS-2307
- Updated API PDR GET endpoint to read individual PDR records from
PostgreSQL database instead of DynamoDB - Added
deletePdr
to@cumulus/api-client/pdrs
- Updated API PDR GET endpoint to read individual PDR records from
- CUMULUS-2782
- Update API granules endpoint
move
action to update granules in the index
and utilize postgres as the authoritative datastore
- Update API granules endpoint
- CUMULUS-2769
- Update collection PUT endpoint to require existance of postgresql record
and to ignore lack of dynamoDbRecord on update
- Update collection PUT endpoint to require existance of postgresql record
- CUMULUS-2767
- Update provider PUT endpoint to require existence of PostgreSQL record
and to ignore lack of DynamoDB record on update
- Update provider PUT endpoint to require existence of PostgreSQL record
- CUMULUS-2759
- Updates collection/provider/rules/granules creation (post) endpoints to
primarily check for existence/collision in PostgreSQL database instead of DynamoDB
- Updates collection/provider/rules/granules creation (post) endpoints to
- CUMULUS-2714
- Added
@cumulus/db/base.deleteExcluding
method to allow for deletion of a
record set with an exclusion list of cumulus_ids
- Added
- CUMULUS-2317
- Added
@cumulus/db/getFilesAndGranuleInfoQuery()
to build a query for searching file
records in PostgreSQL and return specified granule information for each file - Added
@cumulus/db/QuerySearchClient
library to handle sequentially fetching and paging
through results for an arbitrary PostgreSQL query - Added
insert
method to all@cumulus/db
models to handle inserting multiple records into
the database at once - Added
@cumulus/db/translatePostgresGranuleResultToApiGranule
helper to
translate custom PostgreSQL granule result to API granule
- Added
- CUMULUS-2672
- Added migration to add
type
text column to Postgres databasefiles
table
- Added migration to add
- CUMULUS-2634
- Added new functions for upserting data to Elasticsearch:
@cumulus/es-client/indexer.upsertExecution
to upsert an execution@cumulus/es-client/indexer.upsertPdr
to upsert a PDR@cumulus/es-client/indexer.upsertGranule
to upsert a granule
- Added new functions for upserting data to Elasticsearch:
- CUMULUS-2510
- Added
execution_sns_topic_arn
environment variable to
sf_event_sqs_to_db_records
lambda TF definition. - Added to
sf_event_sqs_to_db_records_lambda
IAM policy to include
permissions for SNS publish forreport_executions_topic
- Added
collection_sns_topic_arn
environment variable to
PrivateApiLambda
andApiEndpoints
lambdas. - Added
updateCollection
to@cumulus/api-client
. - Added to
ecs_cluster
IAM policy to include permissions for SNS publish
forreport_executions_sns_topic_arn
,report_pdrs_sns_topic_arn
,
report_granules_sns_topic_arn
- Added variables for report topic ARNs to
process_dead_letter_archive.tf
- Added variable for granule report topic ARN to
bulk_operation.tf
- Added
pdr_sns_topic_arn
environment variable to
sf_event_sqs_to_db_records
lambda TF definition. - Added the new function
publishSnsMessageByDataType
in@cumulus/api
to
publish SNS messages to the report topics to PDRs, Collections, and
Executions. - Added the following functions in
publishSnsMessageUtils
to handle
publishing SNS messages for specific data and event types:publishCollectionUpdateSnsMessage
publishCollectionCreateSnsMessage
publishCollectionDeleteSnsMessage
publishGranuleUpdateSnsMessage
publishGranuleDeleteSnsMessage
publishGranuleCreateSnsMessage
publishExecutionSnsMessage
publishPdrSnsMessage
- `pub...
- Added
- CUMULUS-2208
v9.9.4
Release v9.9.4
Changed
- CVE-2022-2477
- Update node-forge to 1.3.0 in
@cumulus/common
to address CVE-2022-2477
- Update node-forge to 1.3.0 in
- CUMULUS-2868
- Added
iam:PassRole
permission tostep_policy
intf-modules/ingest/iam.tf
- Added
v10.1.2
Release 10.1.2
Added
- CUMULUS-2859
- Update
postgres-db-migration
lambda timeout to default 900 seconds - Add
db_migration_lambda_timeout
variable todata-persistence
module to
allow this timeout to be user configurable
- Update
- CUMULUS-2868
- Added
iam:PassRole
permission tostep_policy
intf-modules/ingest/iam.tf
- Added
v10.1.1
Release v10.1.1
Migration steps
Due to a bug in the PUT /rules/ endpoint, the rule records in PostgreSQL may be out of sync with records in DynamoDB. In order to bring the records into sync, re-run the previously deployed data-migration1 Lambda with a payload of {"forceRulesMigration": true}:
aws lambda invoke --function-name $PREFIX-data-migration1 \
--payload $(echo '{"forceRulesMigration": true}' | base64) $OUTFILE
Added
- CUMULUS-2846
- Added @cumulus/db/translate/rule.translateApiRuleToPostgresRuleRaw to translate API rule to PostgreSQL rules and keep undefined fields
Changed
- CUMULUS-NONE
- Adds logging to ecs/async-operation Docker conatiner that launches async tasks on ECS. Sets default async_operation_image_version to 39.
- CUMULUS-2845
- Updated rules model to decouple createRuleTrigger from create.
- Updated rules POST endpoint to call rulesModel.createRuleTrigger directly to create rule trigger.
- Updated rules PUT endpoints to call rulesModel.createRuleTrigger if update fails and reversion needs to occur.
- CUMULUS-2846
- Updated version of localstack/localstack used in local unit testing to 0.11.5
Fixed
- Upgraded lodash to version 4.17.21 to fix vulnerability
- CUMULUS-2845
- Fixed bug in POST /rules endpoint causing rule records to be created inconsistently in DynamoDB and PostgreSQL
- CUMULUS-2846
- Fixed logic for PUT /rules/ endpoint causing rules to be saved inconsistently between DynamoDB and PostgreSQL
- CUMULUS-2854
- Fixed queue granules behavior where the task was not accounting for granules that already had createdAt set. Workflows downstream in this scenario should no longer fail to write their granules due to order-of-db-writes constraints in the database update logic.
v9.4.2
Release v9.4.2
CUMULUS-2725
- Updated providers endpoint to return encrypted password
- Updated providers model to try decrypting credentials before encryption to
allow for better handling of updating providers
CUMULUS-2841
- Add integration test to validate PDR node provider that requires password
credentials succeeds on ingest