fog design document

Synopsis

This is the work in progress design document See #3508 that will eventually replace the contents of CONTRIBUTING.md

The reason that this isn't in CONTRIBUTING is that this is going to a bit experimental and is focussed on implementing new providers. The fog project needs other people to curate issues and help with documentation - i.e. more than just adding new providers.

A brief history of fog according to tokengeek

Those who cannot remember the past are condemned to repeat it - George Santayana

It may be useful to understand how fog started and why some of the things are the way they are.

When Wes started fog it was a Ruby library for AWS (which didn't offer an SDK at the time). The abstractions that were implemented were portable and abstract enough to be expanded to other providers like Rackspace.

The design was to have abstract resources (models) at one level and also offering access to underlying API actions (requests). Because there wasn't an AWS SDK the requests were implemented within fog. Rather than an impenetrable wall, you could still access particular API calls even if there was not an abstraction for it.

A DSL was created to define services and requests. The implementation of this was based on a service/request being in one file but part of one Ruby class.

It was also envisioned to offer a REPL environment for cloud resources. Start a console and start controlling resources. To make things easier behind the scenes authentication was added from details in ~/.fog

Testing cloud resources was also something new. Shindo was added as an "integration" layer since unit testing was not really suitable for the interface against real third party APIs. One feature of this is that some providers offer "mock" functionality which emulates API endpoints.

So if you had no access to a cloud, you could switch on mocks, enter a REPL and explore APIs.

fog's HTTP system was extracted as Excon to allow it to be reused. This allowed other projects to use it and also allowed HTTP abstraction and testing tools to target Excon.

With fog's success and Wes' openness to accept contributions, the number of providers grew fast. People could add new services and providers easily.

Soon there were more providers than can be easily managed. The abstraction concept was lost and the number of files (caused by the DSL) grew making fog a monolithic gem.

Numerous pain points started to emerge. A number of issues were created discussing it.

In May 2014 a number of key contributors met in Atlanta for a low key summit. We discussed issues and solutions for two days. We came up with a general direction.

One the things we did start on was extracting out providers into standalone gems. I started this due to a number of pain points felt using fog in production systems at Brightbox. So fog-brightbox was extracted as a pilot.

However when we all returned, day jobs, family and other responsibilities prevented us doing too much. People changed jobs, moved cities and responsibilities changed.

More modules were extracted since managing providers separately is a much more scalable approach. However a number of file layout, require issues and aiming for backwards compatibility created performance problems.

So we've been sitting on a number of issues and need to make some drastic steps. Without the background context these might seem extreme or obvious.

That takes us to the roadmap and also this design document.

Mission statement

The fog gem is an abstraction layer over a number of modular cloud services. It hides the differences between providers to make using them easy and portable.

Main issues to resolve

All modular providers - the performance issues means we have stopped directly extracting code but all providers should be implemented as modules/plugins.
- No provider specific code to be in fog or fog-core
Reduced footprint - Very few use cases need multiple providers at once.
- The fog gem will discard all/most providers as dependencies.
- App developers should specifically depend on their desired provider.
- Upstream projects can query for available providers installed on a system.
- Third party SDKs may be used to DRY out implementations as long as abstraction remains.
REPL implementation details leaked - when using fog as a library, global configuration and authentication can be accessed by services or rely on ENV settings intended for starting a console.
- Service abstraction code moved to fog-core
- Configuration object abstracted so provider specific keys
- .fog based configuration becomes deprecated for REPL usage
Idiomatic provider gems - the layout of gems is based on a direct rip from fog which means the file layout is shared. This causes load clashes and packaging issues.
- Gems should be named fog-provider and the structure should be based on lib/fog-provider
Correct namespacing - The namespace Fog::Service::Provider was used for certain services but not others. This was applied inconsistently. As providers are split out there is a potential for clashes in namespaces as well.
- Providers should be namespaced as Fog::Provider::*
Responses are based on Excon - many requests return Excon errors or inconsistently
- Fog::Response to be returned object for all requests
- Not HTTP specific (since some providers are local)
- Providers map API responses into generic responses, collections and Hash like objects.
- Mapping between a Fog::Response and a Fog::Model becomes a fog-core concern.
Mocks as implemented are completely different code paths - When you make a mocked request, if implemented a custom built state machine emulates result of the API calls.
- Mocks (as internal state machine) become per service option implemented as a module
- Providers do not have to create Mock versions of all classes.
- Providers may implement mocking using other means such as Excon or Webmock.
Testing with Shindo is unfamiliar
- Providers test their own code, Shindo tests disappear from fog
- All tests should be done in Minispec
- Services with interface specifications must be tested.

DRAFT - Creating a providers implementation

(Note: It's Midnight on Good Friday as the first version is being written so it is very rough and won't be revisited for a few days.)

Install fog-generator (when it exists)
fog-generator provider example

Should have a sample gem:

lib/fog-example/...
spec/...
LICENCE.md
README.md
etc.

Create a request: fog-generator request example get_value

# lib/fog-example/requests/get_value.rb
module Fog
  class Example
    module Requests
      def get_value(*args)
         # call API 
         Fog::Response.new(...)
      end
    end
  end
end

Create a service: fog-generator service example compute

# lib/fog-example/compute.rb
module Fog
  class Example
    class Compute
      # makes "public", still defined above but method not "request"
      request :get_value

      model Server
    end
  end
end

Implement abstraction

module Fog
  class Example
    class Compute
      class Server < Fog::Compute::Server
        attribute :name

        def get(*args)
          load(get_value.data)
        end
      end
    end
  end
end

Update the entry point to the gem

# lib/fog-example.rb
require "fog/core"
require "compute"

module Fog
  class Example < Fog::Provider
    service Compute # DSL picks up Fog::Example::Compute and infers all details
  end
end

Usage

require "fog-example"

# We need configuration, it won't come from .fog by default
config = Fog::Config.from_fog_file # Oh it's still here if you want it
service = Fog::Compute.new(config)
# Wait, provider to use is part of a new config... interesting
config2 = Fog::Config.new(provider: :example, auth_method: :oauth2, api_version: "1.0")
service2 = Fog::Compute.new(config2)
# Last minute change needed?
service3 = Fog::Compute.new(config2, api_version: "2.0", auth_secret: "wibble")

# As a method
service.requests.get_value # => <Fog::Response success=true data=>{} ...>

# As a request
service.get_value

# Using models
service.servers # => <Fog::Collection ...>
service.servers.get("id") # => <Fog::Example::Server>

Provide feedback

Saved searches