Proposal for Unified Pluggable Scanner API #164
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
|
|
||
| # Proposal: Unified API for Pluggable Image Scanners | ||
|
|
||
| Authors: Prahalad Deshpande @prahaladd | ||
| Proposed Reviewers: Steven Zou @szou, Daniel Pacak @danielpacak, Zach Hill @zhill | ||
|
|
||
|
|
||
| - [Proposal: Unified API for Pluggable Image Scanners](#proposal-unified-api-for-pluggable-image-scanners) | ||
| - [Discussions](#discussions) | ||
| - [Abstract](#abstract) | ||
| - [Use Cases](#use-cases) | ||
| - [Unified API Requirements](#unified-api-requirements) | ||
| - [Unified API Design](#unified-api-design) | ||
| - [Supporting multiple scan types](#supporting-multiple-scan-types) | ||
| - [Scan capability advertisement by scanner implementations](#scan-capability-advertisement-by-scanner-implementations) | ||
| - [Single endpoint to trigger scans](#single-endpoint-to-trigger-scans) | ||
| - [Scan API Endpoint Multiplexing](#scan-api-endpoint-multiplexing) | ||
| - [Scan API response handling](#scan-api-response-handling) | ||
| - [Error Handling](#error-handling) | ||
| - [Advantages of proposed design](#advantages-of-proposed-design) | ||
| - [References](#references) | ||
|
|
||
| ## Discussions | ||
|
|
||
| [Initial Meeting of the Harbor Scanner Workgroup on 19th March 2021](https://drive.google.com/file/d/1TXjsASDPEBL30yFAk0Wnmg5sB3Ka4yw2/view?usp=sharing) | ||
| ## Abstract | ||
|
|
||
| Maintaining the security and compliance of container images stored within the Harbor registry entails more than just software package and OS vulnerability scanning results such as enumerating the Bill of Materials present in an image or validating the contents of the container file system. | ||
| Such a functionality would be exposed by one or more scanners that are specialized to perform the respective type of scan and return the results. | ||
| However, with multiple types of scan reports being generated the number of APIs that would need to be implemented by the third-party scanners and their corresponding registration and management within Harbor would soon become an error prone and tedious process. | ||
| This proposal details a specification that provides a unified API surface for different kinds of scanners that want to integrate with Harbor while also reducing the registration and management overhead within Harbor | ||
|
|
||
| ## Use Cases | ||
|
|
||
| Ensuring security and compliance of the container images stored within a Harbor registry may involve more than just OS and software package vulnerability management. Below is a (non-exhaustive) list of use cases that also lie in the area of security and compliance management | ||
|
|
||
| - Maintaining a detailed Bill of Materials (BoM) of all the software packages being used. | ||
| - Maintaining a list of all the licenses associated with the software packages being used. | ||
| - Running pre-checks on the contents and permissions of files and directories on a container image before rolling out the image to production. | ||
| - Scanning Docker images for sensitive information (such as passwords or secret keys leaked through environment variables) | ||
|
|
||
| The above use cases can be addressed either by a single sophisticated scanner that can provide information required above (and additional data) or by a collection of scanners each specialized for a specific type of scan. | ||
|
|
||
| Irrespective of the number of scanners involved in the scanning process, Harbor must provide a mechanism for the scanners to register themselves and advertise their scanning capabilities, submit scan jobs to these scanners and then collate the resulting scan data (and errors) to provide a unified output that can be presented to the UI or the caller. | ||
|
|
||
| ## Unified API Requirements | ||
|
|
||
| * Expose a small canonical set of public REST endpoints ensuring standard behavior by all implementations and simplified management. | ||
| * De-couple Harbor core and pluggable scanner API evolution from the evolution of the actual scanner implementation. | ||
| * Responses from the API should leverage existing scanner implementations. | ||
| * The data format returned by the scanners should leverage or be backward compatible with the existing data format. | ||
| * Provide mechanisms for API clients to specify parameters that provide fine grain control over the scanning process | ||
| * Extensible in supporting new types of scanning functionalities over periods of time | ||
| * Well defined error handling mechanisms. | ||
|
|
||
| ## Unified API Design | ||
|
|
||
| The approach for designing a unified API surface is detailed below. | ||
|
|
||
| ### Supporting multiple scan types | ||
|
|
||
| There would be one MIME type for the result data of each scan type. A few MIME types are proposed below: | ||
|
|
||
| | Mime Type | Type | Version | Status | | ||
| | --- | --- | --- | --- | | ||
| | application/vnd.security.vulnerability.report; version=1.1 | OS/Package vulnerability data | 1.1 | Released in Harbor 2.3.0 | | ||
| | application/vnd.security.bom.report; version=1.0 | Bill of Materials (BoM) | 1.0 | Proposed | | ||
|
There was a problem hiding this comment. There are several near-standard formats that would make sense for SBoM instead of a new proprietary one: SPDX and CycloneDX are the two with the most traction. Those will help with interoperability, and may include enough information for the FS verification type as well. SPDX in particular supports files and digests as long as you are ok with very large manifests. |
||
| | application/vnd.security.filesystem.report; version=1.0 | File system content validation | 1.0 | Proposed | | ||
|
|
||
|
|
||
| ### Scan capability advertisement by scanner implementations | ||
|
|
||
| Every scanner advertises the supported scanning capabilities using the existing mechanism of publishing scanner metadata. | ||
| The existing `v1.1` endpoint `/metadata` would be used to advertise this information. | ||
|
|
||
| ### Single endpoint to trigger scans | ||
| There will be a single `/scan` endpoint that would be used to trigger scans for multiple types of reports. The `/scan` endpoint will be enhanced to allow clients to specify the types of scan reports to be generated and also a mechanism to control the scanning process. An example request is shown below | ||
| ```shell | ||
| curl -XPOST http://scanner-adapter:8080/api/v2/scan -H 'Content-Type: application/vnd.scanner.adapter.scan.request+json; version=2.0' -d { | ||
| "registry": { | ||
| "url": "harbor-harbor-registry:5000", | ||
| "authorization": "Bearer: JWTTOKENGOESHERE" | ||
| }, | ||
| "artifact": { | ||
| "repository": "library/mongo", | ||
| "digest": "sha256:917f5b7f4bef1b35ee90f03033f33a81002511c1e0767fd44276d4bd9cd2fa8e" | ||
| } | ||
| "scanTypes": { | ||
|
There was a problem hiding this comment. If the scanType property is a subset of produced MIME types advertised by the /metadata endpoint maybe we can think about different / consistent naming here. More importantly, I'd use an array of objects instead of a map with MIME types as a key to enable further extensibility: {
"scanTypes": [
{
"mimeType": "application/vnd.security.vulnerability.report; version=1.1",
"params": {}
},
{
"mimeType": "application/vnd.security.bom.report; version=1.0",
"params": {
"allowGNU": false,
"format": "spdx"
},
"propertyAddedInTheFuture": {}
}
]
}I'd also justify why combining all mime types in a single request is better than multiple scan requests per mime type. There was a problem hiding this comment. I agree with the MIME type naming part. perhaps we can all agree upon some well defined names. These could also be specified within the OCI spec that @steven-zou started. Additionally, I am in fact using an array of objects to specify scan types. The value of the |
||
| "application/vnd.security.vulnerability.report; version=1.1" : {}, | ||
| "application/vnd.security.bom.report; version=1.0" : {"allowGNU" : "false", "format": "spdx"} | ||
| } | ||
| } | ||
| ``` | ||
| In the above request the `scanTypes` key refers to an object that contains key-value pairs with the MIME type as a key. | ||
| - Each type of scan is identified by it's MIME type - in the above request we are requesting for vulnerability scan and BoM scan from a scanner. | ||
| - Each type of scan can be further controlled or additional params passes in using the key value pairs. For e.g. in the above sample request, the BoM scanner can be controlled using the keys allowGNU and the format of representing the BoM information (in this case spdx). | ||
| - Scanner can be requested to perform only a specific type of scan by including only the specific corresponding MIME type in the request along with any control parameters. | ||
|
|
||
| The response from the scanner are packaged in an uber-container structure again keyed by the MIME type. The object against each MIME type would contain the following | ||
| - Internal scan tracking identifier for the composite scan. | ||
| - The scan id and status for each MIME type. The status would be either one of started, completed, running, aborted, failed | ||
|
There was a problem hiding this comment. I don't think we need this complexity. It's non backward compatible and requires designing and implementing a workflow to rescan for a particular MIME types in case of partial errors. I believe that most of the time a scanner adapter will focus on one MIME types (e.g. existing vulnerability scanners). For scanners that support multiple MIME types it's probably acceptable to retry all scan types. Beyond that, scanner adapters will have to maintain composite identifiers to implement state machine for each scan type. We'll have to upgrade all existing adapters as well. There was a problem hiding this comment. Will give in some more thought to this aspect. Lets discuss this in a meeting. |
||
|
|
||
|
|
||
| A sample response for the above request is provided | ||
|
|
||
| ```shell | ||
| Content-Type: application/vnd.scan.report; version=2.0; | ||
| Status: 200 OK | ||
| { | ||
| "scanId": "1", | ||
| "application/vnd.security.vulnerability.report; version=1.1" : { | ||
| "scanId" : "1", | ||
| "scanProgress" : "started", | ||
| }, | ||
| "application/vnd.security.bom.report; version=1.0" : { | ||
| "scanId": "2", | ||
| "scanProgress": "started" | ||
| } | ||
| } | ||
| ``` | ||
| ### Scan API Endpoint Multiplexing | ||
|
|
||
| To satisfy the API request and response patterns mentioned above, the scanner API endpoint middleware for scan would perform the following actions: | ||
|
|
||
| * Extract the MIME type from the request | ||
| * Validate the MIME types present in the request. | ||
| * Check the MIME type is in the list of known MIME types | ||
| * Check that there is a scanner available for specified MIME type | ||
| * For each MIME type in the scan request payload, in a non-blocking manner: | ||
| * Send the scan request to the registered scanners for the MIME type | ||
| * Collect the scan result (or error) for the MIME type. | ||
| * Create an uber-container data structure which will contain the scan results keyed by the corresponding MIME types and send the result to client. | ||
|
|
||
| ### Scan API response handling | ||
| The scanning process would be executing multiple types of scans asynchronously and hence a sophisticated response handling mechanism would required. | ||
|
There was a problem hiding this comment. In the example above there's one scan request with multiple scanTypes indexed by mime type. On the other hand we'd be tracking multiple scan identifiers embedded in the scan response. Why not sending multiple scan requests and tracking them one by one? Or if we stick to one request with multiple scanTypes we could track all mime types as a whole There was a problem hiding this comment. Th reason I maintained per scan type based ID is because of the fact that some scans may take longer than expected to complete (with either success or failure). In such cases given that not all scan types finish at the same time we may want the client to start reporting status for each type of scan job separately thereby making it more response as opposed to waiting for all scans to complete. With such a workflow the scan ID would be needed to be per scan type since we would use the existing reporting mechanisms that fetch scan status based on scan identifiers. |
||
|
|
||
| For a given job, the multiplexer component would maintain a mapping between the composite scan Id and the granular status of each of the component scans as and when received from the scanner. | ||
|
There was a problem hiding this comment. For some scanners keeping such granularly and multiple scan identifiers might be overly complex. Especially for existing scanners that support only one mime type. There was a problem hiding this comment. I will give in some more thoughts to this aspect. Lets discuss it in our meeting |
||
|
|
||
| A client can request for the either of the composite scan status or the status of the individual scans. This mechanism allows to offload the responsbility of the timing of the scan result to the client implementation. It also greatly simplifies error handling since clients can opt to handle errors at the composite scan level or the granular scan level. | ||
|
|
||
| If a client submits a composite scan job containing multiple scan types, then it can choose between polling for the completion of the complete composite job and then retrieving the results or polling separately for the individual scans and retrieving results as and when they are ready. | ||
|
|
||
|
|
||
| ### Error Handling | ||
|
|
||
| The above API invocation pattern requires an enhanced error handling mechanism since multiple scan jobs of different types would be executing and some of them could potentially error out. There are three important aspects to be considered in error handling | ||
| - how does the scanner report an error to Harbor core? | ||
| - how is the error reported to the caller? | ||
| - what is the impact of an error on the overall scan job? | ||
|
|
||
| A clean, simple and efficient mechanism to handle errors would be as follows | ||
|
There was a problem hiding this comment. We could return an array of errors. Each error might have a mimeType or scanType property to indicate which one is failing There was a problem hiding this comment. That is what exactly I have mentioned below. We will have an array of objects (one per MIME type) and the corresponding results/error data. Did you imply that some other detail was missing? |
||
|
|
||
| - Scanners use the existing mechanism to report their scan errors to Harbor | ||
| - If a scanner reports an error when scanning for a particular MIME type, then the uber-container response to the client will contain the serialized error against the MIME type. | ||
| - Each scan is mutually exclusive of the other and the failure of one does not affect the other. The response of a composite scan job submission would contain the error details for each MIME type and it is the responsibility of the client to handle the error appropriately. | ||
|
|
||
|
|
||
| ## Advantages of proposed design | ||
| 1. Prevents combinatorial explosion of REST endpoints for each type of scan (and/combinations) to be supported. | ||
| 2. Adopts a "what you send is what you receive" model. | ||
| 3. Allows clients to control what data they want from the scanner adapter - when the same scanner supports multiple types of scan functionalities | ||
| 4. Provides a mechanism to fine tune scanning operation. | ||
| 5. Incremental changes on the existing API model exposed by Harbor. | ||
|
There was a problem hiding this comment. It seems that complex structure of scan identifiers per mime type would required lots of changes in Harbor core / jobservice There was a problem hiding this comment. Lets discuss this. will also like to know what @steven-zou thinks about this part. |
||
| 6. Provides complete control to clients with respect to retrieval of results, handling of job status and errors. | ||
| 7. Decouples Harbor pluggable scanner API evolution from the actual pluggable scanner capability by providing a "wrapper" over already existing MIME types. For e.g, we do not need re-define a new MIME type for vulnerability scan. Also consider a use case where-in the scanner adapter supports only the 1.0 version of the vulnerability report and is also a BoM scanner. In such cases all that the client needs to specify is the appropriate MIME type. Hence scanner adapters can adopt API specs at their pace and independent of Harbor releases. | ||
|
|
||
| ## References | ||
|
|
||
| - [Pluggable Scanner API Specification v1.1](https://github.com/goharbor/pluggable-scanner-spec) | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if this title suggests the ultimate goal of this proposal.
IIRC we wanted to leverage API v1.1 which is already defined as a minimal set of API endpoints and leverages MIME types. IMO the main focus should be around new MIME types and new scanners / security tools to onboard incrementally
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@danielpacak The reason I thought about this title is because of the fact that we are presenting unified endpoints for scanning different types of data. The details about the MIME types being supported are still subject to discussion with respect to the names to be used etc. The importance of these MIME types has been highlighted in the section that describes how the client can retrieve the various types of results and also in the section Supporting Multiple Mime Types.
We can elaborate on these MIME types if required