Add outputDir support for screenshots and videos

[vigneshshanmugam]

Summary

Synthetics agent by default stores the screenshots and other attachments in the synthetics specific project folder path which is under .synthetics and does not expose any API for getting these screenshots and videos if captured via PW outDir folder. We would like to add support for the outputDIr option and also help users gather these written paths via the test results.

Proposal

outputDir Option

Users should be able to specific the outputDir path to the Synthetics runner via the config option/CLI flag etc to store all the screenshots/videos. If not specified, the runner will default to the previous folder .synthetics and store them under each test for backwards compatability

export default {
  outputDir: "./test-results"
}

Attachments Results

The Stored results from the test run, be it screenshots, videos, trace should be added to the Test results exporter interface where the users can retrive these information on demand. Our reporters should also use the TestResult/JourneyEnd interface and act on these screenshots instead of directly fetching it from .synthetics folder.

onJourneyEnd(result => {
  result.attachments.forEach(attachment => {
     console.log(a.name) // name of the attachment
     console.log(a. contentType) // Content type of this attachment - ex:  'application/json' or 'image/png'.
     console.log(a.path) // path on the filesystem to the attached file.
     console.log(a.body) // attachment body used instead of a file.
 })
});

API is inspired from Playwright Attachments API - https://playwright.dev/docs/api/class-testresult#test-result-attachments

With this, we should be able to support the usecases such as

• Extracting the screenshots to do visual screenshot diffing Update base reporter to emit screenshot as .jpg instead of json format #805
• Extracting videos from the journey run feat: added buildkite reporter #701
• Long term, we can manually add support for adding attachments to existing test reporters for visualization.