New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update aws sdk and image uploader #7555
base: main
Are you sure you want to change the base?
Conversation
This is just to trigger a build with AWS credentials |
12d3565
to
01be580
Compare
Ready for review |
- fix js code - add test images
Addresses the feedback, tested and verified to work locally. @nick-mon1 |
- fix bug in task - log changes to files
…update-aws-sdk-and-image-uploader
This pull request also contains the fix for this Security Issue https://github.com/GSA/digitalgov.gov/security/dependabot/43
|
1535878
to
c6988a7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated my .env
file with the new key names.
Following the testing steps in description:
How To Test
- Add an image(s) and static file to /content/uploads
- Run npm gulp upload
3.Upload to S3 after processing
I get the following error:
➜ npx gulp upload
[15:25:36] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:25:36] Starting 'upload'...
[15:25:36] Starting 'fileTidy'...
[15:25:36] Finished 'fileTidy' after 383 μs
[15:25:36] Starting 'cleanInbox'...
[15:25:36] Finished 'cleanInbox' after 4.53 ms
[15:25:36] Starting 'writeDataFile'...
[15:25:36] Finished 'writeDataFile' after 6.52 ms
[15:25:36] Starting 'processImages'...
[15:25:36] Finished 'processImages' after 228 μs
[15:25:36] Starting 'removeProcessedImage'...
Removing processed images
No images to process
Error processing file: [undefined]. ENOENT: no such file or directory, scandir './content/uploads/_working-images/to-process'
[15:25:36] Finished 'removeProcessedImage' after 882 μs
[15:25:36] Starting 'uploadImage'...
[15:25:36] Finished 'uploadImage' after 3.62 ms
[15:25:36] Starting 'uploadFile'...
[15:25:36] Finished 'uploadFile' after 1.08 ms
[15:25:36] Starting 'cleanup'...
cleanup
[15:25:36] Finished 'cleanup' after 990 μs
[15:25:36] Finished 'upload' after 20 ms
The image needs to be in /content/uploads/_inbox
. That runs successfully:
➜ npx gulp upload
[15:29:23] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:29:23] Starting 'upload'...
[15:29:23] Starting 'fileTidy'...
[15:29:23] Finished 'fileTidy' after 380 μs
[15:29:23] Starting 'cleanInbox'...
[15:29:23] Finished 'cleanInbox' after 5.78 ms
[15:29:23] Starting 'writeDataFile'...
[15:29:23] Finished 'writeDataFile' after 9.38 ms
[15:29:23] Starting 'processImages'...
[15:29:23] Finished 'processImages' after 234 μs
[15:29:23] Starting 'removeProcessedImage'...
Removing processed images
[15:29:23] Finished 'removeProcessedImage' after 2.7 ms
[15:29:23] Starting 'uploadImage'...
file is written
[15:29:24] [create] test-dg-s3-upload-pr-7498.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w200.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w400.png
[15:29:25] [create] test-dg-s3-upload-pr-7498_w800.png
[15:29:25] Finished 'uploadImage' after 1.42 s
[15:29:25] Starting 'uploadFile'...
[15:29:25] Finished 'uploadFile' after 3.25 ms
[15:29:25] Starting 'cleanup'...
cleanup
[15:29:25] Finished 'cleanup' after 3.8 ms
[15:29:25] Finished 'upload' after 1.45 s
Unfortunately, I still don't see my files in the S3 bucket.
This is the image I'm testing - test-dg-s3-upload-pr-7498.png
.
I'll dive into this tomorrow. Thanks @mejiaj |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ I was able to successfully upload an image and see it on digitalgov
bucket. But no go for uploading files, see my suggestion for getting around the InvalidBucketName: Bucket name shouldn't contain '/', received 'digitalgov/static'
error.
function cleanup() { | ||
console.log("cleanup"); | ||
return del([ | ||
"content/uploads/_working-images/**", | ||
"content/uploads/_working-files/**", | ||
]); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When uploading an image, I don't see the content/uploads/_working-images
removed. I updated the code to remove the directory instead of the files.
function cleanup() { | |
console.log("cleanup"); | |
return del([ | |
"content/uploads/_working-images/**", | |
"content/uploads/_working-files/**", | |
]); | |
} | |
function cleanup() { | |
console.log("cleanup"); | |
return del([ | |
"content/uploads/_working-images", | |
"content/uploads/_working-files", | |
]); | |
} |
function uploadFile() { | ||
publisher.config.params.Bucket = "digitalgov/static"; | ||
return gulp.src("content/uploads/_working-files/to-process/*") | ||
.pipe(publisher.publish()) | ||
.pipe(awspublish.reporter({ | ||
states: ['create', 'update', 'delete'] | ||
})) | ||
.pipe(vinylPaths(del)) | ||
.on('error', function(err) { | ||
console.error('File upload failed:', err); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still get InvalidBucketName: Bucket name shouldn't contain '/', received 'digitalgov/static'
. The solution looks to be renaming the file to include static/filename.pptx
before uploading.
We should remove line 31 and try using gulp-rename to include the path in the filename.
Summary
Replaces outdate
gulp-s3-upload
package withawspublish
package and closes #7554.Solution
How To Test
/content/uploads/_inbox/
npx gulp upload
Dev Checklist