Skip to content

Latest commit

 

History

History
738 lines (622 loc) · 35.6 KB

README.md

File metadata and controls

738 lines (622 loc) · 35.6 KB

Table of Contents

Motivation

Around 18 months ago, while working on what is the largest Cycle.js codebase I know of (~20K lines of javascript), I realized how hard it was to actually make sense and maintain a large Cycle.js application. Focusing on those issues derived from Cycle.js usage and compounded by application size:

  • a large portion of the code was stream handling originating from the use of components and the necessity to wire them together. The domain logic, and as a result, the application logic was lost into a sea of streams' sometimes-cryptic operations.
  • extra confusion about inputs (sources), their meanings and usage, due to inputs addressing two separate concerns: component parameterization and interfacing with external systems. A bunch of constants was lifted into streams in miscellaneous places to serve as parameters for generic components, and that led to more stream arithmetic, noise, and in some occurrences bugs
  • modifying, fixing, and extending that code proved to be a gamble, with any bug fixing or debugging sessions counted in hours. To be fair, the complete absence of documentation (and tests) explained a lot of that. The absence of unit tests itself could be explained by, well, the pain that it is to write them with streams in the middle, which led to resorting to sometimes brittle, often slow, selenium-based end-to-end tests.
  • hard to figure out quickly, with certainty the exact workflow that the application was implementing (you know, multi-step processes where any step may fail and you need to backtrack), let alone add new logical branches (error recovery...)

And yet, while that application was large, it cannot really be said to be an exceptionally complex application. Rather it was the standard CRUD application which is 90% of business applications today. No fancy animations, adaptive UI as the only UX trick, otherwise mostly fields and forms, a remote database, and miscellaneous domain-driven workflows.

This was the motivation behind my dedicating my (limited) free time to add the missing capabilities to the framework. I singled out those four areas: componentization, visual debugging, testing, concurrency control. I am happy that, finally, the first step is in a sufficient state of progress that it can be shared.

That first step is a componentization model for Cycle.js, that builds upon the original idea of a component as a function and extends it further. Components are (mostly) what they used to be. Components can however now be parameterized through a dedicated settings argument, capturing the component's parameterization concern, and which is inherited down the component tree. Components, importantly, can be built through a series of component combinators that eliminate a lot of stream noisy, repetitive code. Those component combinators have been extracted and abstracted from the 20K lines of code, so they should cover a large number of cases that one encounters. The proposed component model could be seen in many ways as a generalization of that of React, extending it to handle concerns other than the view, which opens the door to using a JSX-like syntax if you so fancy. The component model also sets up the work for tracing and visualization tools for the second step, without any modification of Cycle.js internals.

This is really a working draft, akin to a proof of concept. Performance was not at all looked upon, combinators only work with RxJS, the version of Cycle used brings us back to the time when Cycle.js could still be considered a library (vs. a framework), build is not optimized, console.logs are all over the place, only tested on chrome evergreen, etc.

It works nicely though. It succeeds in providing a higher-level abstraction so you can focus on the interdependence of components that defines the user interface logic, rather than having to constantly fiddle with a large number of implementation details.

Each combinator features (and if not, will feature) a dedicated non-trivial example of use, and is documented and tested. A sample application is available to showcase how combinators work together with components to build a non-trivial application.

A series of articles covers the theoretical underpinning in more detail (read chronologically -- concepts are built progressively). A specific article shows the step-by-step building of the showcased sample application. A shorter introduction can be found in the README for the repository.

So what is a component combinator?

A component combinator is a parametrizable function which... combines components. To each combinator will correspond a different combining logic. For instance, the ListOf combinator will take two components, and will admit a stream source name as parameter, which emits arrays (for instance ListOf({list:..., as:...}, [C1, C2])). The combined component will activate the C1 component whenever the source emits an empty array, and a combined list of C2 components otherwise. So here the combining logic follows an iteration logic.

The simplest of those combinators is Combine (for instance Combine{{...}, [C0, [C1, C2]]}), whose DOM sink is the result of merging Cx DOM sinks into C0 DOM sink; and whose non-DOM sinks are the merge of the respective Cx non-DOM sink. Here the combining logic is a merge logic.

So the general principle is fairly simple and generic. Now let's see some practical examples of use.

Examples

Separating layout from feature

The following implementation corresponds to the layout specifications:

  • layout specifications (in order from top to bottom of appearance on screen)
    • header
    • feature: we will reuse here the component from the nested routing demo (introduced in a later section)
    • footer
      • made of three groups (with miscellaneous navigation links) and a header
    • sitemap

The application would be correspondingly broken down as follows:

function LayoutContainer(sources, settings) {
  return {
    [DOM_SINK]: $.of(div([
      div(".ui.fixed.inverted.menu", { "slot": "header", }, []),
      div(".ui.main.text.container", { "slot": "body", }, []),
      div(".ui.inverted.vertical.footer.segment", { "slot": "footer", }, []),
    ]))
  }
}

export const App = Combine({}, [LayoutContainer, [
  InSlot('body', [Feature]),
  InSlot('header', [Header]),
  InSlot('footer', [Footer]),
]]);

The Footer component itself is broken down as follows:

function FooterContainer(sources, settings) {
  return {
    [DOM_SINK]: $.of(
      div([
        div(".ui.center.aligned.container", [
          div(".ui.stackable.inverted.divided.grid", [
            div(".three.wide.column", { slot: 'group1' }, []),
            div(".three.wide.column", { slot: 'group2' }, []),
            div(".three.wide.column", { slot: 'group3' }, []),
            div(".seven.wide.column", { slot: 'footer_header' }, [])
          ]),
          div({ "slot": "sitemap" }, [])
        ]),
      ])
    )
  }
}

export const Footer = Combine({}, [FooterContainer, [
  InSlot('group1', [FooterGroup1]),
  InSlot('group2', [FooterGroup2]),
  InSlot('group3', [FooterGroup3]),
  InSlot('footer_header', [FooterHeader]),
  InSlot('sitemap', [Sitemap]),
]]);

By using a container component which specifies where to distribute the DOM content of children components, it is possible to separate layout concerns from feature concerns; and break down layout concern into smaller concerns in an organized, readable, and maintainable way. The breakdown can be realized so that future layout changes mostly impact the LayoutContainer component.

The content distribution mechanism we use is the slot mechanism made popular by web components. Container components declare slots, and children components fill those slots with content (the InSlot combinator is one way to associate a slot to a component).

slot demo with Combine combinator

Note that all component combinators use the same default for merging children components' sinks (whether DOM sinks or non-DOM sinks). Those defaults have been extracted for our large codebase and seem to cover the vast majority of the patterns which occurred in that codebase.

Two things can already be noted here: we haven't had to write any merging code by hand, and the structure of our application is more self-evident, i.e. simpler to read.

Let's now see another example, which addresses a very fundamental need for web applications.

Nested routing

The following implementation corresponds to:

  • Functional specifications
    • user visits '/' -> display home page
      • the home page allows navigating to different sections of the application
    • when the user visits a given section of the application
      • a breadcrumb shows the user where he stands in the sitemap
      • a series of clickable cards are displayed
        • when the user clicks on a given card, details about that card are displayed, and corresponding to a specific route for possible bookmarking
  • Technical specifications
    • HomePage takes the concern of implementing the home page logic.
    • Card is parameterized by its card content and is in charge of implementing the card logic
    • CardDetail is parameterized by its card content and is in charge of displaying the extra details of the card
export const App = InjectSourcesAndSettings({
  sourceFactory: injectRouteSource,
  settings: {
    sinkNames: [DOM_SINK, 'router'],
  }
}, [
  OnRoute({ route: '' }, [
    HomePage
  ]),
  OnRoute({ route: 'aspirational' }, [
    InjectSourcesAndSettings({ settings: { breadcrumbs: ['aspirational'] } }, [
      AspirationalPageHeader, [
        Card(BLACBIRD_CARD_INFO),
        OnRoute({ route: BLACK_BIRD_DETAIL_ROUTE }, [
          CardDetail(BLACBIRD_CARD_INFO)
        ]),
        Card(TECHX_CARD_INFO),
        OnRoute({ route: TECHX_CARD_DETAIL_ROUTE }, [
          CardDetail(TECHX_CARD_INFO)
        ]),
        Card(TYPOGRAPHICS_CARD_INFO),
        OnRoute({
          route: TYPOGRAPHICS_CARD_DETAIL_ROUTE,
        }, [
          CardDetail(TYPOGRAPHICS_CARD_INFO)
        ]),
      ]])
  ]),
]);

animated demo

The (gory) nested routing switching logic is hidden behind the OnRoute combinator. With that out of the way, the routing logic can be expressed very naturally (in a very similar way to React router's dynamic routing, in which the router is a component like any other). There is no pre-configuration of routes, outside of the application. Routes are directly and naturally included in their context.

Let's move on to cases exemplifying simple control flow logic (branching).

Login gateway

For instance, the specification for a login section of an application could go as such:

  • Functional specifications
    • if user is logged, show the main page
    • if user is not logged, show the login page, and redirect to index route when login is performed
  • Technical specifications
    • MainPage takes the concern of implementing the main page logic.
    • LoginPage is parameterized by a redirect route and is in charge of logging in the user
    • convertAuthToIsLoggedIn emits IS_NOT_LOGGED_IN or IS_LOGGED_IN according to whether the user is logged or not

login demo with Switch combinator

A tentaive code to implement those specification with our library would look like:

export const App = Switch({
  on: convertAuthToIsLoggedIn,
  as : 'switchedOn',
}, [
  Case({ when: IS_NOT_LOGGED_IN }, [
    LoginPage({ redirect: '/component-combinators/examples/SwitchLogin/index.html?_ijt=7a193qn02ufeu5it8ofa231v7e' })
  ]),
  Case({ when: IS_LOGGED_IN }, [
    MainPage
  ]),
]);

The same code could be written in a JSX-like dialect as:

export const App =
  <Switch on=convertAuthToIsLoggedIn as='switchedOn'>
      <Case when=IS_NOT_LOGGED_IN>
        <LoginPage redirect='/component-combinators/examples/SwitchLogin/index.html?_ijt=7a193qn02ufeu5it8ofa231v7e'/>
      </Case>
      <Case when=IS_LOGGED_IN>
        <MainPage />
      </Case>
  </Switch>

The same code could also be written in a dedicated DSL:

export const App = dsl`
  Switch On ${convertAuthToIsLoggedIn} (As switchedOn)
    When ${IS_NOT_LOGGED_IN} :
      LoginPage {redirect:'/component-combinators/examples/SwitchLogin/index.html?_ijt=7a193qn02ufeu5it8ofa231v7e'}
    When ${IS_LOGGED_IN} :
      MainPage
`

Syntax, whichever chosen (we will work only with the first one) is but a detail. What is important here is that:

  • the stream wiring concern has disappeared within the Switch combinator (i.e. has been abstracted out), while the user interface logic can be written in a way that is very close to its specification, hence easier to understand and check for correctness
  • The developer cannot make any mistake in the stream switching logic, nor does he have to check while debugging that the error does not come from an erroneous switch handling. Provided that the Switch combinator has been properly implemented and tested, the corresponding concern is out of the way.
  • A debugging developer can narrow down a cause of misbehavior for example by selectively modifying arguments, deleting branches of the component tree, stubbing components, etc. That is, reasoning, investigating can be made at a component level first, before, if necessary, going at the lower stream level.

Next, we have a look at complex control flow logic (branching, jumping, looping, etc.).

Multi-step workflow

The specification for a multi-step application process, as coming from the designer team, is as follows:

sequence

We have here a sequence of screens, with conditional transitioning logic according to the state of the application. That logic is later refined in parallel with the dev team to take the final control flow form:

complete control flow

We won't include code samples here for the sake of brevity. The previous control flow graph is specified in the form of a state machine and implemented with the EFSM component combinator.

demo

We refer however the curious reader to:

Let's continue with the combinator covering iteration logic.

Dynamically changing list of items

The following implementation corresponds to:

  • Functional specifications
    • display a list of cards reflecting input information from a card database
    • a pagination section allows displaying X cards at a time
  • Technical specifications
    • Card is parameterized by its card content and is in charge of implementing the card logic
    • Pagination is in charge of the page number change logic
export const App = InjectSources({
  fetchedCardsInfo$: fetchCardsInfo,
  fetchedPageNumber$: fetchPageNumber
}, [
  ForEach({
      from: 'fetchedCardsInfo$',
      as: 'items',
      sinkNames: [DOM_SINK],
    }, [AspirationalPageHeader, [
      ListOf({ list: 'items', as: 'cardInfo' }, [
        EmptyComponent, // Component activated in case list is empty
        Card, // // Component activated otherwise
      ])
    ]]
  ),
  ForEach({
    from: 'fetchedPageNumber$',
    as: 'pageNumber',
    sinkNames: [DOM_SINK, 'domainAction$']
  }, [
    Pagination
  ])
]);

ForEachList demo

The reactive update (on fetchedCardsInfo$) and iteration logic (on the array of items received from fetchedCardsInfo$) are taken care of with the ForEach and the ListOf combinators.

Same as always, there is no gory stream-merging code in the way of understanding the application logic. Furthermore, edge cases attached to the iteration logic are already conveniently taken care of.

Our library however seeks to cover most of the generic needs arising. However, in a real application, the existing combinator list might not cover the full spectrum of combining logics (being that essentially infinite). For those specific needs not covered, our library also includes a component combinator factory, the same one from which all existing combinators are actually derived. Let's see other combinators and our combinator factory at work through an example.

Composing an app from components

The original point of the combinator library is to compose an application by building it from components. In the previous sections, we have shown some combinators handling logic, and control flow. There are other combinators handling state injection and interface adaptation, so that inputs can be adapted to an existing component interface (imagine web components with a predefined event (sources) and property interface (settings)), fostering reuse. New component combinators can be created thanks to our generic component combinator factory m.

Taking a page from our showcased sample application, it looks like this (only tidbits, for the full code, see the example repo, for a step-by-step building of the application, refer to the corresponding article on my blog):

App = Combine({}, [
 SidePanel,
 MainPanel
]);

SidePanel =
 Combine({}, [Div('.app__l-side'), [
   Navigation({}, [
     NavigationSection({ title: 'Main' }, [
       NavigationItem({ project: { title: 'Dashboard', link: 'dashboard' } }, [])
     ]),
     NavigationSection({ title: 'Projects' }, [
       InSlot('navigation-item', [ListOfItemsComponent])
     ]),
     NavigationSection({ title: 'Admin' }, [
       NavigationItem({ project: { title: 'Manage Plugins', link: 'plugins' } }, [])
     ]),
   ])
 ]]);

Displaying the list of items requires to get the item from a source. That source is passed through InjectSources:

const ListOfItemsComponent =
  InjectSources({ projectNavigationItems$: getProjectNavigationItems$ }, [
    ForEach({ from: 'projectNavigationItems$', as: 'projectList' }, [
      ListOf({ list: 'projectList', as: 'project' }, [
        EmptyComponent,
        NavigationItem
      ])
    ])
  ]);
  
function getProjectNavigationItems$(sources, settings) {
   return sources.projects$
     .map(filter(project => !project.deleted))
     .map(map(project => ({
       title: project.title,
       link: ['projects', project._id].join('/')
     })))
     .distinctUntilChanged()
     // NOTE : this is a behaviour
     .shareReplay(1)
     ;
}

sample app demo with miscellaneous combinators

Note the use of the ad-hoc combinators Navigation, NavigationSection and NavigationItem. They are for instance defined as:

// Components
// Navigation(..., [NavigationSection(..., [NavigationItem(...,[])])])
function NavigationContainerComponent(sources, settings) {
  const { user$, projects$ } = sources;
  // combineLatest allows to construct a behaviour from other behaviours
  const state$ = $.combineLatest(user$, projects$, (user, projects) => ({ user, projects }))

  return {
    [DOM_SINK]: state$.map(state => {
      return div('.navigation', [
        renderTasksSummary(state),
        nav({ slot: 'navigation-section' }, [])
      ])
    })
  }
}

function Navigation(navigationSettings, componentArray) {
  return m({}, (navigationSettings, [NavigationContainerComponent, componentArray])
}

This is the first introduction of the m component combinator factory. We will not expand here on the specifications for m --- that is done in the corresponding documentation. It suffices to know that the first argument of m specifies the combining logic and that Combine is actually the partial application of m with an empty object (i.e. default combining logic is used). The Navigation component could for instance also be written as:

function Navigation(navigationSettings, componentArray) {
  return Combine(navigationSettings, [NavigationContainerComponent, componentArray])
}

Unless you have to implement some very specific combining logic that we have not met in our large codebase, you should not have to use m in another form than Combine. If that should happen, you will need to delve into the documentation where we detail the three strategies we use for combining components. If you come up with a useful component combinator that is not here, feel free to publish it in its own package.

While the full syntax and semantics of the component combinators have not been exposed, hopefully, the examples serve to portray the merits of using a component model, under which an application is written as a component tree, where components are glued with convenient component combinators covering frequently occurring patterns. I certainly think it is simpler to write, and more importantly, simpler to read, maintain and debug, which is paramount at scale.

Let's have a proper look at combinators' syntax and the available combinators extracted from the 20K-line Cycle.js codebase.

Combinators

Syntax

In general combinators follow a common syntax1:

  • Combinator :: Settings -> ComponentTree -> Component
    • Settings :: *
    • Component :: Sources -> Settings -> Sinks
    • ComponentTree :: ChildrenComponents | [ContainerComponent, ChildrenComponents]
    • ContainerComponent:: Component
    • ChildrenComponents :: Array<Component>

Combinator list

The proposed library has the following combinators:

Combinator Description
Combine The simplest combinator, which traverses a component tree, applying default merge functions to components' sinks along the way. Distinguishes between DOM sink and non-DOM sink, and implements a slot mechanism for merging DOM sinks
InSlot Assign DOM content to a slot (a la web component)
OnRoute Activate a component based on the route changes. Allows nested routing.
Switch Activate component(s) depending on the incoming value of a source
FSM Activate components based on inputs and current state of a state machine. Allows implementing a flow of screens and actions according to complex control flow rules.
ForEach Activate component for each incoming value of a source
ListOf Activate a list of a given component based on an array of items
Pipe Sequentially compose components
InjectSources Activate a component that will be injected extra sources
InjectSourcesAndSettings Activate a component that will receive extra sources and extra settings
m The core combinator from which all other combinators are derived. m (for merge) basically traverses a component tree, applying default or provided reducing functions along the way.

Documentation, demo, and tests for each combinator can be found in their respective repository.

Theoretical background

The theoretical underpinnings can be found as a series of articles on my blog:

Documentation

Documentation for component combinators and drivers can be found in the projects portion of my blog.

Installation

Packages

The following packages are available:

Package Description
@rxcc/components Contains the core component combinators
@rxcc/drivers Exposes a few useful drivers, in particular, drivers to handle command and queries on a domain, and read the DOM state
@rxcc/testing Mocks for the provided drivers, and the testing library used for testing the components combinators
@rxcc/contracts A bunch of predicates and utility functions to handle contract checking and assertions
@rxcc/utils Miscellaneous utility functions (debugging, component helpers, etc.)

Any of those can be installed with npm. For instance:

npm install @rxcc/components

Tests

Tests are performed with good old QUnit, i.e. in the browser. This allows debugging code in the browser, and also the possibility in a debugging session to actually display some components' output directly in the DOM (vs. looking at some virtual representation of the DOM). To run the available tests, in the root directory, type :

  • npm install
  • npm run build-node-test
  • have a look at /test/index.js to pick up which test you want to run (400+ tests available in total)
  • npm run test
  • then open with a local web server the index.html in test directory

Roadmaps

Roadmap v0.5

Details

The core target of this release will be to prepare the architecture for visual tracing, and specify the (visual) shape that this should take. A small proof of concept should be produced. A secondary target is to start a very basic UI component library, not going over the proof of concept level.

The current roadmap for the v0.5 stands as:

  • Core
    • robustness of settings:
      • in some cases, should be inherited down the tree, in other cases should be private (find a nice syntax). That creates complexity but reduces bug surface so worth it.
      • very important bug possibility which is facilitated as of now: a settings could be set at some location in the tree, and then read wrongly at another location of the tree because they have the same name. that prevents from using default values for settings, the default value could be wrongly overridden by settings inherited from up the tree.
        • as of now, expected settings SHOULD be mandatory and SHOULD be namespaced to avoid collisions
    • external system's state reading
      • document driver reads synchronously a value from the DOM. That is possible because the DOM is locally available. Value whether to uniformize read mechanism by having them all returning Observable. That should enable simpler tracing!!
    • see what can be done to have a better concurrency model (i.e. beyond FSM)
    • type contracts error handling for component's settings (which types of component combinator expects, types of settings, etc.)
    • error management:
      • error boundaries?
      • error logging (use chrome's console.context? replace string formatting for console?)
      • improve error reporting (human-readable message, add guards, include blame information in the form of erroneous arguments)
    • logging and visualization (!)
    • conversion to web components
  • Component combinator library
  • Component library
  • Drivers library
    • analyze benefits of immutability for store drivers
    • rename Action driver to Command driver, to make it obvious this is Command and Query separation
  • Demo
    • continue to complete demo from Angular2 book on GitHub site
    • Real world app?
  • Testing
    • Model-based testing for FSM, i.e. automatic test cases generation
    • study testing with pupeeteer.js (chrome headless browser) -- cypress looks quite good too
    • improve API for runTestScenario to make it less verbose
  • Combinators
    • Portal combinator (render DOM in a specific location)
    • Catch combinator? cf. Core -- error management
    • Switch combinator
      • cover the default: part of switch statement
    • State machine combinator FSM
      • convert FSM structure to graphml or dot or tgf format
      • automatic generation of graphical representation of the FSM
      • refactor the asynchronous FSM into synchronous EHFSM + async module
        • this adds the hierarchical part, improvement in core library are automatically translated in improvement to this library, and closed/open principle advantages
      • investigate prior art for reuse opportunities
    • Event combinator WithEvents to think about (specifications? cf. current mEventFactory)
    • State combinator WithState to think about (specifications? rationale?)
    • Action combinator ComputeActions to think about (specifications? rationale?)
  • Distribution
    • monorepo?
    • individual combinator packages?

Roadmap v0.4

Please note that the library is still wildly under development:

  • APIs might will go through breaking changes
  • you might encounter problems in production
  • performance has not been investigated as of yet

The current roadmap for the v0.4 stands as:

  • Core
    • component model
    • DOM merge with slot assignment (a la web component)
    • documentation for a-la-web-component slot mechanism
    • documentation combinators
    • nice blog site: GitHub pages?
      • select static site generator (Jekyll, Hexo, Hugo)
      • blog site architecture
      • theoretical underpinnings
    • implement relevant part of sample application taken from Angular2 book
  • Testing
    • Testing library runTestScenario
    • Mocks for DOM and document driver
    • Mock for domain query driver
  • Combinators
    • Generic combinator m
    • combinator Combine
    • Routing combinator onRoute
    • Switch combinator
      • Switch
      • Case
    • State machine combinator FSM
    • ForEach combinator ForEach
    • List combinator ListOf
    • Injection combinator
      • InjectSources
      • InjectSourcesAndSettings
    • Query driver
    • Action driver
    • sequential composition combinator (Pipe)

Demos

Example application

The example application is taken from the book Mastering Angular2 components. Cf. screenshot here.

  • sits in examples/AllInDemo directory
  • npm install
  • npm run wbuild
  • then open with a local web server the index.html in $HOMEDIR/examples/AllInDemo directory

State Machine

  • go to $HOMEDIR/examples/volunteerApplication
  • npm install
  • npm run wbuild
  • then open with a local web server the index.html in $HOMEDIR/examples/volunteerApplication directory

Switch

  • go to $HOMEDIR/examples/SwitchLogin
  • npm install
  • npm run wbuild
  • then open with a local web server the index.html in $HOMEDIR/examples/SwitchLogin directory

OnRoute

  • go to $HOMEDIR/examples/NestedRoutingDemo
  • npm install
  • npm run wbuild
  • then open with a local web server the index.html in $HOMEDIR/examples/NestedRoutingDemo directory

ForEach and List

  • go to $HOMEDIR/examples/ForEachListDemo
  • npm install
  • npm run wbuild
  • then open with a local web server the index.html in $HOMEDIR/examples/ForEachListDemo directory

Contribute

Contributions are welcome in the following areas:

  • DevOps
    • monorepos
    • whatever makes sense to make the repository more manageable
  • reducing build size
  • transpose code to latest version of Cycle

Known issues

That is a paragraph that I am sure will grow with time :-)

Footnotes

  1. The EFSM combinator as of v0.4 is an exception to that rule.