Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define the countertop API #82

Closed
slifty opened this issue Jul 27, 2020 · 2 comments · Fixed by #83
Closed

Define the countertop API #82

slifty opened this issue Jul 27, 2020 · 2 comments · Fixed by #83
Assignees
Labels
discussion The conversation is the point

Comments

@slifty
Copy link
Member

slifty commented Jul 27, 2020

Task

Description

We have an issue already for creating the countertop (#23) and two issues that talk about the need for an API to communicate with the countertop (#65 and #66).

This issue is a blend of a discussion issue and a task; I would like to use it to define the initial developer API for the countertop. This is the API that developers will use to do things like:

  • Register, configure, and activate appliances.
  • Subscribe to data events.
  • Start and stop the countertop / processing pipeline.

Relevant Resources / Research

None

Related Issues

@slifty slifty added the discussion The conversation is the point label Jul 27, 2020
@slifty slifty self-assigned this Jul 27, 2020
@slifty
Copy link
Member Author

slifty commented Jul 27, 2020

Note: for the purposes of this phase I think we should not design around Docker. I would like to consider creating some Docker / Appliance utilities, including a root Docker image that all appliances buld from in their docker files, and a parallel DockerAppliance which would serve as a shim to interact with that docker image. This way our appliance ecosystem would support Docker, but would not prescribe it.

We talked a fair amount in #65 about the desire to not have appliances be dependencies of this repository, and moving to having instance repositories of TV kitchen that define an individual countertop topology. This means that the countertop API will not load appliance packages directly since it won't have access to the modules directly.

With that model I immediately see two ways an appliance could be registered in the countertop:

  1. Registering Appliance prototypes with the countertop, thus allowing the countertop to create instances as it needs. Instances would then be requested via API (with configurations). The countertop would manage creating the appliance instances.

  2. Registering instantiated / configured appliances could be passed to the countertop directly.

In theory both of these paths could work.

The benefits of registering instances

  1. It feels intuitive -- the user has complete control over the configuration and setup of the appliance and they register something directly.

  2. The countertop does not need to expose configuration options (which means that it does not need to be updated if the IAppliance configuration API changes).

The benefits of registering prototypes

  1. The countertop has complete control over how and when appliances are instantiated. For example, we talked about creating one appliance per stream / data pair -- this would mean that an individual appliance prototype / configuration pair might be instantiated multiple times in a given topology.

Note: I think we would need to create a clone method in IAppliance to support this otherwise.

  1. Implementation scripts would not need to know how to create and configure appliances; rather, they would just need to know how to interact with the countertop API. For example, this would mean the countertop would be the only thing that needs to know how to handle instantiation errors from appliances.

  2. Treating prototype registration as a separate step, would decouple the act of creating appliances from registering them. For instance maybe someone would want to create an implementation tool where the types of appliances that are possible in a topology are defined by a config file, but the definition of the topology itself is defined via a user interface, API, or command line tool.


Right now I am leaning towards the more flexible (but potentially more verbose) architecture of registering prototypes as a first step.

@slifty
Copy link
Member Author

slifty commented Jul 30, 2020

All right, how about this:

  • registerAppliance( prototype, configuration ) :: adds an appliance to the countertop, returns an ID. If the countertop has started already it will start the appliance immediately. If the countertop has not been started, the appliance will not be started. Multiple copies of a given appliance may be created to process multiple data streams.

  • deregisterAppliance( applianceID ) :: removes a given appliance from the countertop.

  • getAppliances() :: returns a list of appliance prototype / configuration pairs.

  • getWorkers() :: returns a list of active countertop workers (there is one worker per appliance + source pair).

  • start() :: starts the countertop (and all appliances registered).

  • stop() :: stops the countertop (and all appliances registered).

  • on() :: register event listeners (list of events TBD; possibly just the appliance events we already have).

This still allows us to create appliances on demand without relying on half baked non-features such as cloning, but feels a bit more intuitive because adding an appliance does not become a two step process.


EDIT: I had originally suggested addAppliance / removeAppliance, but I think maybe register / deregister might be better since add might imply instances.

slifty added a commit that referenced this issue Aug 12, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 12, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 17, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 17, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 17, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 17, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
slifty added a commit that referenced this issue Aug 17, 2020
This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
This was referenced Sep 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion The conversation is the point
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant