Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update aws sdk and image uploader #7555

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

RileySeaburg
Copy link
Member

@RileySeaburg RileySeaburg commented May 10, 2024

Summary

Replaces outdate gulp-s3-upload package with awspublish package and closes #7554.

Solution

How To Test

  1. Add an image(s) and static file to /content/uploads/_inbox/
  2. Run npx gulp upload
  3. Upload to S3 after processing

Dev Checklist

  • PR has correct labels
  • A11y testing (voice over testing, meets WCAG, run axe tools)
  • Code is formatted properly

@RileySeaburg RileySeaburg self-assigned this May 10, 2024
@RileySeaburg
Copy link
Member Author

This is just to trigger a build with AWS credentials

@RileySeaburg RileySeaburg force-pushed the rs-update-aws-sdk-and-image-uploader branch from 12d3565 to 01be580 Compare May 10, 2024 21:04
@RileySeaburg
Copy link
Member Author

Ready for review

@nick-mon1 nick-mon1 self-requested a review May 14, 2024 14:51
config/gulp/file-upload.js Outdated Show resolved Hide resolved
config/gulp/file-upload.js Outdated Show resolved Hide resolved
config/gulp/file-upload.js Show resolved Hide resolved
- fix js code
- add test images
@RileySeaburg
Copy link
Member Author

Addresses the feedback, tested and verified to work locally. @nick-mon1

@RileySeaburg RileySeaburg reopened this May 15, 2024
@RileySeaburg RileySeaburg added Dev: frontend ideas and issues related to the presentation layer of the site Dev: Workflow labels May 15, 2024
@nick-mon1 nick-mon1 self-requested a review May 17, 2024 19:44
config/gulp/file-upload.js Outdated Show resolved Hide resolved
data/images/clock-iconcop.yml Outdated Show resolved Hide resolved
data/images/free.yml Outdated Show resolved Hide resolved
data/images/pexels-photo-2829031.yml Outdated Show resolved Hide resolved
config/gulp/file-upload.js Show resolved Hide resolved
@RileySeaburg
Copy link
Member Author

This pull request also contains the fix for this Security Issue

https://github.com/GSA/digitalgov.gov/security/dependabot/43

$ npm list lodash.trim
digitalgov.gov@2.0.0 /home/riley/programming/digitalgov.gov
└── (empty)

@RileySeaburg RileySeaburg force-pushed the rs-update-aws-sdk-and-image-uploader branch from 1535878 to c6988a7 Compare May 17, 2024 21:51
Copy link
Contributor

@mejiaj mejiaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RileySeaburg

I've updated my .env file with the new key names.

Following the testing steps in description:

How To Test

  1. Add an image(s) and static file to /content/uploads
  2. Run npm gulp upload
    3.Upload to S3 after processing

I get the following error:

➜ npx gulp upload
[15:25:36] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:25:36] Starting 'upload'...
[15:25:36] Starting 'fileTidy'...
[15:25:36] Finished 'fileTidy' after 383 μs
[15:25:36] Starting 'cleanInbox'...
[15:25:36] Finished 'cleanInbox' after 4.53 ms
[15:25:36] Starting 'writeDataFile'...
[15:25:36] Finished 'writeDataFile' after 6.52 ms
[15:25:36] Starting 'processImages'...
[15:25:36] Finished 'processImages' after 228 μs
[15:25:36] Starting 'removeProcessedImage'...
Removing processed images
No images to process
Error processing file: [undefined]. ENOENT: no such file or directory, scandir './content/uploads/_working-images/to-process'
[15:25:36] Finished 'removeProcessedImage' after 882 μs
[15:25:36] Starting 'uploadImage'...
[15:25:36] Finished 'uploadImage' after 3.62 ms
[15:25:36] Starting 'uploadFile'...
[15:25:36] Finished 'uploadFile' after 1.08 ms
[15:25:36] Starting 'cleanup'...
cleanup
[15:25:36] Finished 'cleanup' after 990 μs
[15:25:36] Finished 'upload' after 20 ms

The image needs to be in /content/uploads/_inbox. That runs successfully:

➜ npx gulp upload
[15:29:23] Using gulpfile ~/web/digitalgov.gov/gulpfile.js
[15:29:23] Starting 'upload'...
[15:29:23] Starting 'fileTidy'...
[15:29:23] Finished 'fileTidy' after 380 μs
[15:29:23] Starting 'cleanInbox'...
[15:29:23] Finished 'cleanInbox' after 5.78 ms
[15:29:23] Starting 'writeDataFile'...
[15:29:23] Finished 'writeDataFile' after 9.38 ms
[15:29:23] Starting 'processImages'...
[15:29:23] Finished 'processImages' after 234 μs
[15:29:23] Starting 'removeProcessedImage'...
Removing processed images
[15:29:23] Finished 'removeProcessedImage' after 2.7 ms
[15:29:23] Starting 'uploadImage'...
file is written
[15:29:24] [create] test-dg-s3-upload-pr-7498.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w200.png
[15:29:24] [create] test-dg-s3-upload-pr-7498_w400.png
[15:29:25] [create] test-dg-s3-upload-pr-7498_w800.png
[15:29:25] Finished 'uploadImage' after 1.42 s
[15:29:25] Starting 'uploadFile'...
[15:29:25] Finished 'uploadFile' after 3.25 ms
[15:29:25] Starting 'cleanup'...
cleanup
[15:29:25] Finished 'cleanup' after 3.8 ms
[15:29:25] Finished 'upload' after 1.45 s

Unfortunately, I still don't see my files in the S3 bucket.

This is the image I'm testing - test-dg-s3-upload-pr-7498.png.
test-dg-s3-upload-pr-7498

@RileySeaburg
Copy link
Member Author

I'll dive into this tomorrow. Thanks @mejiaj

@nick-mon1 nick-mon1 self-requested a review May 24, 2024 15:55
Copy link
Contributor

@nick-mon1 nick-mon1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ I was able to successfully upload an image and see it on digitalgov bucket. But no go for uploading files, see my suggestion for getting around the InvalidBucketName: Bucket name shouldn't contain '/', received 'digitalgov/static' error.

Comment on lines 43 to 49
function cleanup() {
console.log("cleanup");
return del([
"content/uploads/_working-images/**",
"content/uploads/_working-files/**",
]);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When uploading an image, I don't see the content/uploads/_working-images removed. I updated the code to remove the directory instead of the files.

Suggested change
function cleanup() {
console.log("cleanup");
return del([
"content/uploads/_working-images/**",
"content/uploads/_working-files/**",
]);
}
function cleanup() {
console.log("cleanup");
return del([
"content/uploads/_working-images",
"content/uploads/_working-files",
]);
}

Comment on lines +30 to +40
function uploadFile() {
publisher.config.params.Bucket = "digitalgov/static";
return gulp.src("content/uploads/_working-files/to-process/*")
.pipe(publisher.publish())
.pipe(awspublish.reporter({
states: ['create', 'update', 'delete']
}))
.pipe(vinylPaths(del))
.on('error', function(err) {
console.error('File upload failed:', err);
});
Copy link
Contributor

@nick-mon1 nick-mon1 May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still get InvalidBucketName: Bucket name shouldn't contain '/', received 'digitalgov/static'. The solution looks to be renaming the file to include static/filename.pptx before uploading.

We should remove line 31 and try using gulp-rename to include the path in the filename.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dev: frontend ideas and issues related to the presentation layer of the site Dev: Workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update gulp-s3-upload to newer package
4 participants