Generic REST resources for Chef - demonstrated for NetApp ONTAP clusters

Project to add a new REST resource type to Chef, allowing development and maintenance for REST API-based devices as a low-code task.

Simply specify REST URL patterns and mappings between JSON data and Chef properties to write REST platform support within the hour.

This enables devices like storage systems, network equipment or anything with a REST API being managed in a DevOps manner, putting it under desired state and control it via CI/CD pipelines.

Recipe:

ontap_cluster "ontap1" do
  location    "Seattle"
  contact     "noreply@chef.io"
  dns_domains %w[lab.chef.io]
  timezone    "US/Pacific"
end

Implementation:

resource_name :ontap_cluster

 # ... provides, properties etc as with any Chef Custom Resource ...

rest_api_collection "/api/cluster"
rest_api_document   "/api/cluster"

rest_property_map ({
  name:         "name",
  contact:      "contact",
  dns_domains:  "dns_domains",
  location:     "location",
  name_servers: "name_servers",
  timezone:     "timezone.name"
})

Note that this project currently still requires wrapping the new resource in Ruby statements and put into libraries/.

See Resource implementation for details and Resource documentation for usage.

Motivation

In the past, vendors supported and maintained resource cookbooks to configure devices that were not traditionally in scope for Chef - from Cisco Nexus switches to NetApp storage systems.

Most of these projects have been abandoned, as Chef focused on traditional OS and agent-based deployment.

A likely cause for this is the high maintenance effort to put in for keeping implementation up to date, both due to needing to manage the transmission of commands AND the device/API-specific implemention.

Since Chef 16.6, Target Mode enabled users to retrieve remote system inventory ("Remote Ohai") and execute remote commands. All this is based on the Train transport framwork, effectively removing transmission management requirements.

As REST APIs follow similar paradigms as configuration management and provide structured data, implementing support for them mainly consists of mapping REST URLs/properties to Chef resources/properties.

Providing a base implementation for REST and some easy to use DSL methods, enables high-speed development of resources and makes maintenance easy even for the long-term. Generation of Chef resources from OpenAPI specifications is a possibility.

This approach allows all common DevOps methods to be used with REST-capable devices - NetOps, StorageOps, DatacenterOps, ...

Implementation Parts

On a high level, this project contains three components:

1. Chef Shim
   Code which extends Chef to support REST resources: rest_resource and rest_resource_dsl
2. NetApp Platform Support Pack
   Example on how to a) refine the base REST resource by API-specifics (error handling, pagination mainly) and b) write API-specific resources mainly by using new REST DSL.
   
3. Demo Cookbook
   To set up a NetApp ONTAP cluster and create shares for users to consume.

All communication is handled via the train-rest transport, which I developed in early 2020 for a PoC to remotely manage server management controllers based on the Redfish standard.

Chef Shim

This part creates a new resource type RestResource which is derived from the base Chef Resource. It is intended to be abstract and only subclassed by PSPs (see next section). It provides additional DSL methods for specifying REST API URLs, mapping between Chef and REST and some control over attributes sent. Also, it includes a default implementation of :configure and :delete actions.

Also, the new resource type provides helpers to interact with the API - abstracting away the calls to the underlying train-rest transport, if needed.

As it is intended to be subclassed, it offers methods to transform REST responses (rest_postprocess) and do API-specific error handling (rest_errorhandler). You can also use super inside of the :configure/:delete actions to add more steps in your usable PSP resource.

Platform Support Pack (PSP) Concept

A PSP is a collection of resources for Chef Target Mode which provides:

• OS/Platform detection
• Remote system inventory
• Custom Resources
  • Abstract Platform Resource
  • Primary Resources
• Custom InSpec resources (currently not possible with Chef <17.7)
• optionally
  • DSL helper functions
  • Train-specific components (e.g. specialized REST API authenticators)

For more background on the PSP concept see my explanation on aws-blog.de (EN)

Demo Cookbook

For using the demo cookbook, you will need either a physical NetApp ONTAP system, a virtualized NetApp ONTAP Select or the ONTAP simulator. The last one is only available for NetApp partners and customers on their support website.

The cookbook assumes a base setup has been completed. That includes only the features available in the web-based setup wizard: Discovery of nodes, setting of the node/cluster IPs and adding of licenses. Without these steps, a cluster would not be reachable and resources won't work.

Managing licenses sadly isn't supported in a desired state way as there is no way to check if a license key has already been applied.

Setup Steps

• Set up a NetApp lab, guides can be found e.g on Flackbox
• If using the simulator, assign the spare disks manually to a node
• Get Chef 17.7 or an older version with Ohai PR-1698 and Chef PR-12151 applied
• Configure ~/.chef/credentials or /etc/chef/credentials for your cluster, using target_protocol = "rest" and username, password fields like in my project presentation video
• Git clone this repository
• Adjust settings in recipe/* to match your cluster and setup
• Execute chef-client -z -t YOURTARGET -r "recipe[rest-for-chef::RECIPE]"
• You can also bootstrap the client to a Chef Server, see my Blog on TM with a Chef Server for details

In-Depth Documentation

• Terms/Glossary
• Program Flow
• REST Resource DSL Extensions
• Custom Authentication
• What's Next

Requirements

• Chef 17.7 (Transport type specification in credentials)
• Ohai 17.7 (Remote Ohai fix)
• Train-REST 0.4.0 (Authentication fixes, Session management)
• physical or simulated NetApp ONTAP system for running the demo cookbook (set up via System Manager, including licenses)