CUMULUS-3433 - Update to nodeJs v20

.nvmrc

@@ -1 +1 @@
-16.19.0
+20.12.2

CHANGELOG.md

@@ -8,10 +8,27 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Migration Notes
 
+#### CUMULUS-3433 Update to node.js v20
+
+The following applies only to users with a custom value configured for
+`async_operation_image`:
+
+- As part of the node v20 update process, a new version (49) of the Core
+  async-operation container was published - [cumuluss/async
+  operation](https://hub.docker.com/layers/cumuluss/async-operation)  The
+  default value for `async_operation_image` has been updated in the `cumulus`
+  module, however if you are using an internal image repository such as ECR,
+  please make sure to update your deployment configuration with the newly
+  provided image.
+
+  Users making use of a custom image configuration should note the base image
+  for Core async operations must support node v20.x.
+
 #### CUMULUS-3449 Please follow instructions before upgrading Cumulus.
 
-- The updates in CUMULUS-3449 requires manual update to postgres database in production environment. Please follow
-  [Update Cumulus_id Type and Indexes](https://nasa.github.io/cumulus/docs/next/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449)
+- The updates in CUMULUS-3449 requires manual update to postgres database in
+  production environment. Please follow [Update Cumulus_id Type and
+  Indexes](https://nasa.github.io/cumulus/docs/next/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449)
 
 #### CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded
 
@@ -39,9 +56,11 @@ The Lambda will trigger an Async Operation and return an `id` such as:
 "taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}
 ```
 
-which you can then query the Async Operations [API Endpoint](https://nasa.github.io/cumulus-api/#retrieve-async-operation) for
-the output or status of your request. If you want to directly observe the progress of the migration as it runs, you can view
-the CloudWatch logs for your async operations (e.g. `PREFIX-AsyncOperationEcsLogs`).
+which you can then query the Async Operations [API
+Endpoint](https://nasa.github.io/cumulus-api/#retrieve-async-operation) for the
+output or status of your request. If you want to directly observe the progress
+of the migration as it runs, you can view the CloudWatch logs for your async
+operations (e.g. `PREFIX-AsyncOperationEcsLogs`).
 
 ### Breaking Changes
 
@@ -91,9 +110,21 @@ the CloudWatch logs for your async operations (e.g. `PREFIX-AsyncOperationEcsLog
   - `tf-modules/monitoring` module now deploys Glue table for querying dead-letter-archive messages.
 - **CUMULUS-3616**
   - Added user guide on querying dead-letter-archive messages using AWS Athena.
+- **CUMULUS-3433**
+  - Added `importGot` helper method to import `got` as an ESM module in
+    CommmonJS typescript/webpack clients.
 
 ### Changed
-
+- **CUMULUS-3433**
+  - Updated all node.js lambda dependencies to node 20.x/20.12.2
+  - Modified `@cumulus/ingest` unit test HTTPs server to accept localhost POST
+    requests, and removed nock dependency from tests involving `fs.Readstream`
+    and `got` due to a likely incompatibility with changes in node v18, `got`,
+    fs.Readstream and nock when used in combination in units
+    (https://github.com/sindresorhus/got/issues/2341)
+  - Updated `got` dependency in `@cumulus/ingest` to use `@cumulus/common`
+    dynamic import helper / `got` > v10 in CommonJS.
+  - Updated all Core lambdas to use [cumulus-message-adapter-js](https://github.com/nasa/cumulus-message-adapter-js) v2.2.0
 - **CUMULUS-3629**
   - dla guarantees de-nested SQS message bodies, preferring outermost metadata as found.
   - dla uses execution Name as filename and ensures no ':' or '/' characters in name

bamboo/Dockerfile

@@ -1,6 +1,6 @@
 ## Dockerfile to create integration/unit test environment
-FROM node:16.19.0-buster
-RUN apt update && npm config set unsafe-perm true &&\
+FROM node:20.12.2-buster
+RUN apt update &&\
     apt install -y netcat zip jq rsync zip
 RUN apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
 libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
@@ -15,13 +15,8 @@ WORKDIR "/opt/Python-$PYTHON_VERSION.$PYTHON_VERSION_PATCH"
 RUN ./configure --enable-optimizations && make -j4 && make altinstall
 
 WORKDIR /
-
-
-
-
-
 RUN update-alternatives --install /usr/bin/python python /usr/local/bin/python3.10 1 && \
     update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
 RUN python -m pip install -U pip
-RUN npm install -g npm@8.6 && npm config --global set script-shell /bin/bash
+RUN npm config --global set script-shell /bin/bash
 RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip && ./aws/install

bamboo/bootstrap-integration-tests.sh

@@ -5,8 +5,6 @@ set -ex
 . ./bamboo/abort-if-not-pr-or-redeployment.sh
 . ./bamboo/abort-if-skip-integration-tests.sh
 
-npm config set unsafe-perm true
-
 if [[ $USE_TERRAFORM_ZIPS == true ]]; then
   ## Configure TF deployment to use deployed packages for the version being built
   echo "***Deploying stack with deployment packages"

bamboo/deploy-dev-integration-test-stack.sh

@@ -6,8 +6,6 @@ set -ex
 . ./bamboo/abort-if-not-pr-or-redeployment.sh
 . ./bamboo/abort-if-skip-integration-tests.sh
 
-npm config set unsafe-perm true
-
 if [[ $USE_TERRAFORM_ZIPS == true ]]; then
   ## If this flag is set, we want to use the output of the 'publish' stage
   ## to deploy in the integration test stage, so abort

bamboo/docker-compose-local.yml

@@ -14,15 +14,15 @@ services:
       - ../:/source/cumulus
       - ../packages/test-data:/tmp/cumulus_unit_test_data
     ports:
-      - "20:20"
-      - "21:21"
-      - 47400-47470:47400-47470
-      - 3030:3030
-      - 2222:2222
-      - 4550-4599:4550-4599
-      - 5432:5432
-      - 8080:8080
-      - 9200:9200
+      - "127.0.0.1:20:20"
+      - "127.0.0.1:21:21"
+      - 127.0.0.1:47400-47470:47400-47470
+      - 127.0.0.1:3030:3030
+      - 127.0.0.1:2222:2222
+      - 127.0.0.1:4550-4599:4550-4599
+      - 127.0.0.1:5432:5432
+      - 127.0.0.1:8080:8080
+      - 127.0.0.1:9200:9200
   localstack:
     image: localstack/localstack:3.0.0
   elasticsearch:

bamboo/generate-ts-build-cache.sh

@@ -13,8 +13,6 @@ npm install @octokit/graphql@2.1.1 simple-git@3.7.0
 . ./bamboo/abort-if-not-pr.sh
 
 
-npm config set unsafe-perm true
-
 set -o pipefail
 
 CURRENT_WORKING_DIR=$NONCACHE_WORKING_DIR

bamboo/httpd.conf

@@ -2,20 +2,20 @@
 # This is the main Apache HTTP server configuration file.  It contains the
 # configuration directives that give the server its instructions.
 # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
-# In particular, see 
+# In particular, see
 # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
 # for a discussion of each configuration directive.
 #
 # Do NOT simply read the instructions in here without understanding
 # what they do.  They're here only as hints or reminders.  If you are unsure
-# consult the online docs. You have been warned.  
+# consult the online docs. You have been warned.
 #
 # Configuration and logfile names: If the filenames you specify for many
 # of the server's control files begin with "/" (or "drive:/" for Win32), the
 # server will use that explicit path.  If the filenames do *not* begin
 # with "/", the value of ServerRoot is prepended -- so "logs/access_log"
 # with ServerRoot set to "/usr/local/apache2" will be interpreted by the
-# server as "/usr/local/apache2/logs/access_log", whereas "/logs/access_log" 
+# server as "/usr/local/apache2/logs/access_log", whereas "/logs/access_log"
 # will be interpreted as '/logs/access_log'.
 
 #
@@ -45,11 +45,11 @@ ServerRoot "/usr/local/apache2"
 # ports, instead of the default. See also the <VirtualHost>
 # directive.
 #
-# Change this to Listen on specific IP addresses as shown below to 
+# Change this to Listen on specific IP addresses as shown below to
 # prevent Apache from glomming onto all bound IP addresses.
 #
 #Listen 12.34.56.78:80
-Listen 3030 
+Listen 3030
 
 #
 # Dynamic Shared Object (DSO) Support
@@ -171,7 +171,7 @@ LoadModule version_module modules/mod_version.so
 LoadModule unixd_module modules/mod_unixd.so
 #LoadModule heartbeat_module modules/mod_heartbeat.so
 #LoadModule heartmonitor_module modules/mod_heartmonitor.so
-#LoadModule dav_module modules/mod_dav.so
+LoadModule dav_module modules/mod_dav.so
 LoadModule status_module modules/mod_status.so
 LoadModule autoindex_module modules/mod_autoindex.so
 #LoadModule asis_module modules/mod_asis.so
@@ -189,16 +189,16 @@ LoadModule autoindex_module modules/mod_autoindex.so
 #LoadModule negotiation_module modules/mod_negotiation.so
 LoadModule dir_module modules/mod_dir.so
 #LoadModule imagemap_module modules/mod_imagemap.so
-#LoadModule actions_module modules/mod_actions.so
+LoadModule actions_module modules/mod_actions.so
 #LoadModule speling_module modules/mod_speling.so
 #LoadModule userdir_module modules/mod_userdir.so
 LoadModule alias_module modules/mod_alias.so
-#LoadModule rewrite_module modules/mod_rewrite.so
+LoadModule rewrite_module modules/mod_rewrite.so
 
 <IfModule unixd_module>
 #
 # If you wish httpd to run as a different user or group, you must run
-# httpd as root initially and it will switch.  
+# httpd as root initially and it will switch.
 #
 # User/Group: The name (or #number) of the user/group to run httpd as.
 # It is usually good practice to create a dedicated user and group for
@@ -239,7 +239,7 @@ ServerAdmin you@example.com
 
 #
 # Deny access to the entirety of your server's filesystem. You must
-# explicitly permit access to web content directories in other 
+# explicitly permit access to web content directories in other
 # <Directory> blocks below.
 #
 <Directory />
@@ -261,6 +261,7 @@ ServerAdmin you@example.com
 #
 DocumentRoot "/usr/local/apache2/htdocs"
 <Directory "/usr/local/apache2/htdocs">
+
     #
     # Possible values for the Options directive are "None", "All",
     # or any combination of:
@@ -297,8 +298,8 @@ DocumentRoot "/usr/local/apache2/htdocs"
 </IfModule>
 
 #
-# The following lines prevent .htaccess and .htpasswd files from being 
-# viewed by Web clients. 
+# The following lines prevent .htaccess and .htpasswd files from being
+# viewed by Web clients.
 #
 <Files ".ht*">
     Require all denied
@@ -351,8 +352,8 @@ LogLevel warn
 
 <IfModule alias_module>
     #
-    # Redirect: Allows you to tell clients about documents that used to 
-    # exist in your server's namespace, but do not anymore. The client 
+    # Redirect: Allows you to tell clients about documents that used to
+    # exist in your server's namespace, but do not anymore. The client
     # will make a new request for the document at its new location.
     # Example:
     # Redirect permanent /foo http://www.example.com/bar
@@ -369,7 +370,7 @@ LogLevel warn
     # the filesystem path.
 
     #
-    # ScriptAlias: This controls which directories contain server scripts. 
+    # ScriptAlias: This controls which directories contain server scripts.
     # ScriptAliases are essentially the same as Aliases, except that
     # documents in the target directory are treated as applications and
     # run by the server when requested rather than as documents sent to the
@@ -388,6 +389,15 @@ LogLevel warn
     #Scriptsock cgisock
 </IfModule>
 
+
+### Configuration to allow PUT endpoint
+
+RewriteEngine On
+RewriteCond %{REQUEST_METHOD} ^POST$
+RewriteCond %{REQUEST_URI} ^/post_test.*$
+RewriteRule .* - [R=200,L,E=no-gzip:1]
+
+
 #
 # "/usr/local/apache2/cgi-bin" should be changed to whatever your ScriptAliased
 # CGI directory exists, if you have that configured.
@@ -481,10 +491,10 @@ LogLevel warn
 #MaxRanges unlimited
 
 #
-# EnableMMAP and EnableSendfile: On systems that support it, 
+# EnableMMAP and EnableSendfile: On systems that support it,
 # memory-mapping or the sendfile syscall may be used to deliver
 # files.  This usually improves server performance, but must
-# be turned off when serving from networked-mounted 
+# be turned off when serving from networked-mounted
 # filesystems or if support for these functions is otherwise
 # broken on your system.
 # Defaults: EnableMMAP On, EnableSendfile Off
@@ -494,9 +504,9 @@ LogLevel warn
 
 # Supplemental configuration
 #
-# The configuration files in the conf/extra/ directory can be 
-# included to add extra features or to modify the default configuration of 
-# the server, or you may simply copy their contents here and change as 
+# The configuration files in the conf/extra/ directory can be
+# included to add extra features or to modify the default configuration of
+# the server, or you may simply copy their contents here and change as
 # necessary.
 
 # Server-pool management (MPM specific)

example/cumulus-tf/lambdas.tf

@@ -4,7 +4,7 @@ resource "aws_lambda_function" "async_operation_fail" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/asyncOperations/lambda.zip")
   handler          = "index.fail"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   tags = local.tags
 
@@ -23,7 +23,7 @@ resource "aws_lambda_function" "async_operation_success" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/asyncOperations/lambda.zip")
   handler          = "index.success"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   tags = local.tags
 
@@ -42,7 +42,7 @@ resource "aws_lambda_function" "sns_s3_executions_test" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/snsS3Test/lambda.zip")
   handler          = "index.handleExecutions"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   environment {
     variables = {
@@ -68,7 +68,7 @@ resource "aws_lambda_function" "sns_s3_granules_test" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/snsS3Test/lambda.zip")
   handler          = "index.handleGranules"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   environment {
     variables = {
@@ -94,7 +94,7 @@ resource "aws_lambda_function" "sns_s3_pdrs_test" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/snsS3Test/lambda.zip")
   handler          = "index.handlePdrs"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   environment {
     variables = {
@@ -120,7 +120,7 @@ resource "aws_lambda_function" "sns_s3_collections_test" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/snsS3Test/lambda.zip")
   handler          = "index.handleCollections"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   environment {
     variables = {
@@ -146,7 +146,7 @@ resource "aws_lambda_function" "ftpPopulateTestLambda" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/ftpPopulateTestLambda/dist/lambda.zip")
   handler          = "index.handler"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
   timeout          = 150
   memory_size      = 512
 
@@ -186,7 +186,7 @@ resource "aws_lambda_function" "lzards_api_client_test" {
   source_code_hash = filebase64sha256("${path.module}/../lambdas/lzardsClientTest/dist/webpack/lambda.zip")
   handler          = "index.handler"
   role             = module.cumulus.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
   timeout          = 600
   memory_size      = 512
 

example/cumulus-tf/modules/s3_access_test/main.tf

@@ -14,7 +14,7 @@ resource "aws_lambda_function" "s3_acccess_test" {
   source_code_hash = filebase64sha256("${path.module}/../../../lambdas/s3AccessTest/lambda.zip")
   handler          = "index.handler"
   role             = var.lambda_processing_role_arn
-  runtime          = "nodejs16.x"
+  runtime          = "nodejs20.x"
 
   tags = var.tags
 }

example/cumulus-tf/variables.tf

@@ -350,7 +350,7 @@ variable "rds_admin_access_secret_arn" {
 variable "async_operation_image_version" {
   description = "docker image version to use for Cumulus async operations tasks"
   type = string
-  default = "48"
+  default = "49"
 }
 
 variable "cumulus_process_activity_version" {