preloading + testing flaw

[kellyselden]

Took me a long time to narrow this down. If you are trying to test a lazy engine, component integration tests won't work unless you preload the assets, or an acceptance test happens to run first that already downloaded the assets.

This PR exposes the issue by removing the preloader and acceptance tests. Now the component test fails.

It seems there is a bug that component integration tests aren't triggering lazy engines to download. The preloader shouldn't be required for apps right?

[rwjblue]

I'd need to dig in to have a definitive answer, but I suspect the real issue is that we never actually do owner.lookup('engine:foo-bar') and handle the returned promise in ember-test-helpers. I'm not 100% sure where that should be done though.

[trentmwillis]

Here are my thoughts on the matter:

The short version: Engines should have their own test suite which always preloads assets. Consuming applications should not preload assets.

The long version: When Engines are lazy loaded, their assets get built into a separate bundle. Thus, unless you load that bundle in some fashion, those assets won't be present when tests run. This isn't a problem in Acceptance tests (which will trigger the bundle load), but it is a problem for Integration and Unit tests which (by default) won't trigger any bundle load.

The solution here is simple in theory, but the execution requires some nuance. The basic idea is that Engine assets should always be preloaded when tested by that Engine.

What I mean by this is that each Engine should have its own test suite which should always preload its assets. Engines shouldn't need to test the loading behavior of the Engine, since that is the purview of the ember-engines addon. This means the assets will already be loaded for any unit, integration, or acceptance tests by a given Engine.

However, when testing within the context of a host application, assets should not be preloaded. Since the host app should not contain unit or integration tests for the Engine (similarly to how they shouldn't contain unit/integration tests for addons) we don't need to worry about the assets being loaded. Additionally, for Acceptance tests you should reset the loaded asset state after each test (see here), that way you can verify that when you visit an Engine within your application, it loads correctly with any necessary dependencies.

All of that said, in order for this to be practical for in-repo-engines, we need to finish ember-cli/rfcs#60.

[kellyselden]

@trentmwillis You're so right that ember-cli/rfcs#60 needs to be finished. I felt so dirty writing my component tests for engine components in my host app's context.

[denzo]

@kellyselden @trentmwillis @rwjblue just wondering if there was any progress or if there's some sort of roadmap as it is hard to tell what stage the mentioned RFC is at.

Let us know if there's anything we could help with.