Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add import & export by file, admin api and methods exported by start() #395

Merged
merged 2 commits into from Mar 15, 2019

Conversation

Kouzukii
Copy link
Contributor

@Kouzukii Kouzukii commented Feb 27, 2019

This PR adds a state service which enables import & export of features and strategies as JSON and YAML via /api/admin/state/export and /api/admin/state/import, via configuration option importFile as well as by:

const unleash = require('unleash-server');

unleash.start({...})
  .then(({stateService}) => {
    const exportedData = stateService.export({includeStrategies: true, includeFeatureToggles: true});
    stateService.import({data: exportedData, userName: 'importer'});
  });

The option dropBeforeImport will delete all existing features/strategies before import

example npx usage
npx unleash-server --databaseUrl ... --importFile features.yml --dropBeforeImport

This allows you to quickly bring up an unleash server for client development with custom feature toggles, without using database dumps.

To-Do

  • Documentation

@coveralls
Copy link

coveralls commented Feb 27, 2019

Coverage Status

Coverage decreased (-0.5%) to 91.424% when pulling 50e6bd0 on Kouzukii:import-export into db3d88e on Unleash:master.

@ivarconr
Copy link
Member

Great idea! 🎉👍

The code need some improvement and we need to discuss a few concerns. Will come back to you soon when I have some time !⏰

lib/options.js Outdated Show resolved Hide resolved
@ivarconr
Copy link
Member

ivarconr commented Mar 4, 2019

So I finally have some time to follow up on this!

  1. Should it be possible to import/export just toggles and/or strategies individually?
  2. How do you think the import/export API endpoints should be used? In the PR you have connected these at the /api/admin/ route, which is intended to serve the Frontend. Do you think this should be a feature implemented here afterwards? Would it make more sense to expose this as a cli based API? How should the users of unleash protect such an API (most of the functionality exposed under /api/client is not particular harmful and would be hard to misuse to anything meaningful if the client key is compromised)
  3. You have not addressed the history-events. Does it make sense to keep those as-is and add on top? Shouldn't the whole database be flushed if the dropBeforeImport is set? What about metrics?

server,
eventBus,
import: importData.bind(null, config),
export: exportData.bind(null, config),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you think it is important to expose import/export outside of Unleash? How do you think this should be used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as an alternative option to the admin API which requires authentication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in our specific use case, we have
features.yaml which is an export from the production system
features.overrides.yaml git-tracked overrides
features.local.yaml non git-tracked overrides
the entrypoint merges all 3 and imports them into unleash.
maybe it should allow multiple --importFile's ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on a second thought, I don't think allowing multiple importFile's is a good idea. The logic for merging the export files is a bit displaced. This should be done by a user defined entrypoint.

lib/import-export.js Outdated Show resolved Hide resolved
@Kouzukii
Copy link
Contributor Author

Kouzukii commented Mar 7, 2019

  1. Should it be possible to import/export just toggles and/or strategies individually?
    as in export a single feature toggle?

You mean export a single featue toggle?
I don't see a use case for this. If developers want a selection of feature toggles they can export it and remove the unwanted entries from the JSON/YAML pretty quickly.

  1. How do you think the import/export API endpoints should be used? In the PR you have connected these at the /api/admin/ route, which is intended to serve the Frontend. Do you think this should be a feature implemented here afterwards? Would it make more sense to expose this as a cli based API? How should the users of unleash protect such an API (most of the functionality exposed under /api/client is not particular harmful and would be hard to misuse to anything meaningful if the client key is compromised)

I think this could be easily added to the feature overview in the frontend.
We use the API endpoints at the moment to copy production / staging feature toggles onto development environments in kubernetes.
I don't think clients should have the option to modify any feature toggle data, they should be read-only.

  1. You have not addressed the history-events. Does it make sense to keep those as-is and add on top? Shouldn't the whole database be flushed if the dropBeforeImport is set? What about metrics?

The drop functionality is supposed to be for disposeable environments where the history doesn't matter. If the user wants the history to be preserved, he should create a database dump.
Maybe we should drop the history though when dropBeforeImport is used.
Metrics is an interesting thought, it is probably worth adding an additional option flushMetrics or similar.

@ivarconr
Copy link
Member

ivarconr commented Mar 7, 2019

One thing that worries me about you suggested approach

All changes of feature toggles and strategies in the current unleash design are driven through the events and you are actually able to recreate a correct state just by replying all the events.

  1. Will you suggestion break with this guarantee?
  2. Would it be possible to base export/import on the events and just reply these on import? This will potentially also make import/export work for future needs and avoid if/else statements re-implementing logic.

What do you think?

@ivarconr
Copy link
Member

ivarconr commented Mar 7, 2019

You mean export a single feature toggle?

No, I mean export just the list of "feature toggles" or just all "strategies". I think there is cases where I would like to export all toggles from one env and apply them to another env.

Metrics is an interesting thought, it is probably worth adding an additional option flushMetrics or similar.

After thinking about it I don't think this is necessary:

  • If it is an exiting environment in use you probably want to keep the history
  • If it is a newly create environment it does not matter.

@Kouzukii
Copy link
Contributor Author

Kouzukii commented Mar 7, 2019

All changes of feature toggles and strategies in the current unleash design are driven through the events and you are actually able to recreate a correct state just by replying all the events.

  1. Will you suggestion break with this guarantee?

No, import() uses only events to create the desired state of the imported data.
However the dropBeforeImport option (currently) breaks this.

  1. Would it be possible to base export/import on the events and just replay these on import? This will potentially also make import/export work for future needs and avoid if/else statements re-implementing logic.

this would bloat exported data, the idea was to keep the export as minimal and humanly-readable as possible.
I was also thinking about how to reduce the reimplemented logic, but haven't reached a satisfying conclusion. Maybe some of the creation/update logic could be migrated to the feature/strategy-store.

No, I mean export just the list of "feature toggles" or just all "strategies". I think there is cases where I would like to export all toggles from one env and apply them to another env.

this is already possible
(e.g. /api/admin/export?features=1 will only export features)

async import(req, res) {
const userName = extractUser(req);
const { drop } = req.query;
const data = req.body;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should accept data sent as a file upload.

Copy link
Contributor Author

@Kouzukii Kouzukii Mar 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I'll add an optional file parameter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this would require multipart/form-data. Right now we're only using bodyparser.json() middleware.

Copy link
Member

@ivarconr ivarconr Mar 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

lib/server-impl.js Outdated Show resolved Hide resolved
@ivarconr
Copy link
Member

ivarconr commented Mar 9, 2019

the idea was to keep the export as minimal and humanly-readable as possible.

I agree, and a yaml file is perfect for this.

I was also thinking about how to reduce the reimplemented logic, but haven't reached a satisfying conclusion. Maybe some of the creation/update logic could be migrated to the feature/strategy-store.

I can look in to this (We should be able to merge this PR before we have a perfect solution, we can always refactor so no worries.)

However the dropBeforeImport option (currently) breaks this.

Could "flush database" be a separate concern? I think that in your case of disposable environment it would make sense to:

  1. flush the whole database
  2. migrate the database
  3. import toggles/strategies from importFromFile

Agree?

I think this can work for the import API also (/api/admin/import/). I mean if the user says dropBeforeImport: flush entire database, migrate, import.

But I have a few worries about flushing:

  • What happens if multiple unleash-instances stars at once? We might actually require a db-lock to avoid multiple competing instances.
  • What about runtime flushing. How do we guarantee that it will not hurt clients/users during the flush? We might want to hold back flushing the db runtime?

@Kouzukii
Copy link
Contributor Author

Kouzukii commented Mar 9, 2019

  1. flush the whole database
  2. migrate the database
  3. import toggles/strategies from importFromFile

Agree?

sounds good, I'll implement this.

What happens if multiple unleash-instances start at once? We might actually require a db-lock to avoid multiple competing instances.

this already is a concern with the current db-migrator on an empty database, no?

What about runtime flushing. How do we guarantee that it will not hurt clients/users during the flush? We might want to hold back flushing the db runtime?

if we implement a db lock, we could just return a 5xx error code when requesting /api/client/features, so clients will use their cached toggles.

@ivarconr
Copy link
Member

ivarconr commented Mar 9, 2019

this already is a concern with the current db-migrator on an empty database, no?

No, db-migrator uses transactions to avoid race conditions.

if we implement a db lock, we could just return a 5xx error code when requesting /api/client/features

Sounds like an ok solution

@ivarconr
Copy link
Member

ivarconr commented Mar 9, 2019

On second thoughts;

Importing at startup will not work well with the multiple instances case. It will end up importing and updating the same state multiple times. if one add dropBeforeImport all toggles will be droppet at startup importet for each instance. We must find a solution to avoid this behaviour!

Top of my head is that we could introduce a helper table for imports where the file content is hashed and stored. Import will just happen if that table does not contain the desired importFile. This will make it impossible to reimport the same content multiple times.

This might however be undesirable, if unleash has been changed and one would like to reset to previous state.

@ivarconr
Copy link
Member

@Kouzukii, after thinking about this a bit more I want Unleash to not take too much responsibility in to around guaranteeing non-duplicate imports across multiple instances of Unleash. My suggestion for the way forward is the following:

1. Unleash exposes stateService

Unleash will return a service capable to perform import and export. It will be up to the implementer to use Unleash as a library and take advantage of the import. Unleash will only accept json as import. Unleash will only perform the operation as requested, it will be up to the user to assure that it only happens once per instance, if this matters. Import will follow the import protocol described in 4).

unleash.start({...})
  .then(unleash => {
    const myData = unleash.stateService.export({strategies: true, featureToggles: true});
    unleash.stateService.import({data: myData, userName: 'importer'});
  });

Import functions will in addition follow the flush protocol, described in 3).

2. Unleash exposes import/export APIs

/api/admin/state/export

This endpoint will be used to export feature toggles and strategies. Should support json, yaml and file-downloads.

/api/admin/state/import

This endpoint should support json, yaml or file uploads of json/yml. It also take addition argument to flush database which should follow the flush protocol, described later.

3. Flush protocol

This is a special event stored in the event-store. It exits one for strategies and one for feature toggles. All flush events are stored just as any other unleash event is stored.

await eventStore.store({
   type: FLUSH_STRATEGIES,
   createdBy: userName
});

The result of FLUSH_STRATEGIES is that all strategies are deleted. This is handled by the strategy-store.js.

await eventStore.store({
   type: FLUSH_FEATURES,
   createdBy: userName
});

The result of FLUSH_FEATURES is that all feature toggles are deleted. This is handled by the feature-toggle-store.js. When fetching history for a specific feature toggle, unleash should only return everything after last FLUSH_FEATURES event.

4. Import protocol

When a toggle/strategy is imported it should be marked as just that. It should be up to the respective store to do conflict resolution (If toggle exists it should be updated with the new state. If toggle is archived it should be automatically revived as an effect of this change).

Example of an imported toggle event.

await eventStore.store({
   type: FEATURE_IMPORT,
   createdBy: userName,
   data: <toggleDef>
});

Example of an imported strategy event

await eventStore.store({
   type: STRATEGY_IMPORT,
   createdBy: userName,
   data: <strategyDef>
});

I hope this makes sense. I love to hear your reactions and suggestions on the purposed way forward.

@Kouzukii
Copy link
Contributor Author

@ivarconr this seems like a very elegant and future-proof solution.

@ivarconr
Copy link
Member

Cool, do you want to refactor your PR in this direction or do you want me to build on your great work and do it (I have some time later this week)?

@Kouzukii
Copy link
Contributor Author

I am already on it ;)

@ivarconr
Copy link
Member

Your are awesome ⚡

@Kouzukii
Copy link
Contributor Author

3. Flush protocol

Are you sure about this name?
I feel like an argument like --flushBeforeImport could be ambiguous to users.

@ivarconr
Copy link
Member

Are you sure about this name?
I feel like an argument like --flushBeforeImport could be ambiguous to users.

No, I'm open to keeping drop

if (!strategies && !featureToggles) {
strategies = true;
featureToggles = true;
}
Copy link
Member

@ivarconr ivarconr Mar 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be a bit better to just:

const strategies = req.query.strategies ? true : false;
const features = req.query.features ? true : false;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well you could do Boolean(req.query.strategies) as well
the point was, if no query arguments are passed, both features and strategies are exported.

const YAML = require('js-yaml');
const moment = require('moment');
const multer = require('multer');
const upload = multer({ limits: { fileSize: 5242880 } });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the limit i 5MB? Why did you choose this number?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the file is kept in memory, so I chose a number that wouldn't impact memory usage greatly.
Maybe this should be exposed as a config?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sounds reasonable. I would be very surprised if anyone need more than 5MB, but you never know.

const server = app.listen({ port: options.port, host: options.host }, () =>
logger.info(`Unleash started on port ${server.address().port}`)
);

return new Promise((resolve, reject) => {
server.on('listening', () => resolve({ app, server, eventBus }));
return await new Promise((resolve, reject) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why should Unleash return a promise if it awaits for it to complete?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure what happens when you return a promise in an async method.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the caller will have to wait of it to complete, if they care.

}

async importFile({ importFile, dropBeforeImport, userName }) {
let data = await new Promise((resolve, reject) =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

 let data = await new Promise((resolve, reject) =>
            fs.readFile(importFile, (err, v) =>
                err ? reject(err) : resolve(v)
            )
        );

can be simplified to:

let data = await fs.readFile(importFile);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it this possible without fs-extra?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no your right. require('fs').promises was added in node 10. I don't want to require new node version before next major of unleash.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are doing the await straight away there is no reason to no just use the sync api for reading the file.

A async aproach could be:

/// util functions:
function readFile(importFile) {
    return new Promise((resolve, reject) =>
        fs.readFile(importFile, (err, v) => (err ? reject(err) : resolve(v)))
    );
}

function yamlToJs(importFile, data) {
    return mime.lookup(importFile) === 'text/yaml' ? yaml.safeLoad(data) : data;
}

using the util functions the importFile could look like:

importFile({ importFile, dropBeforeImport, userName }) {
        return readFile(importFile)
            .then(yamlToJs.bind(this, importFile))
            .then(data => this.import({ data, dropBeforeImport, userName }));
    }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't like await huh 😄
but i like this approach, done.

async import({ data, userName, dropBeforeImport }) {
const { eventStore } = this.config.stores;

if (typeof data === 'string') {
Copy link
Member

@ivarconr ivarconr Mar 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be handled in importFile. import should assume and validate json data.

data = JSON.parse(data);
}

data = await joi.validate(data, dataSchema);
Copy link
Member

@ivarconr ivarconr Mar 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't mutate input arguments. You may break clients calling this method.

const importData = await joi.validate(data, dataSchema);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add this to eslintrc ;)

data: { name: 'all-features' },
});
}
for (const feature of data.features) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use forEach instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now using Promise.all(data.features.map(...))

}
}

async export({ strategies, featureToggles }) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better names for readability would be inlcudeStrategies and includeFeatureToggles

Copy link
Member

@ivarconr ivarconr Mar 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and should the default to true?

async export({ inlcudeStrategies = true, includeFeatureToggles = true })

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took this from your example ;)
But I'll change it.

if (strategies) {
result.strategies = (await strategyStore.getStrategies())
.filter(strat => strat.editable)
.map(strat => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better if the strategyStore could return all not-built-in-strategies. This could be a filter argument, alla: strategyStore.getStrategies({builtIn: false})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solved via strategyStore.getEditableStrategies()

lib/state-service.js Outdated Show resolved Hide resolved
lib/state-service.js Outdated Show resolved Hide resolved
Kouzukii added a commit to Kouzukii/unleash that referenced this pull request Mar 14, 2019
@@ -96,7 +100,6 @@ class FeatureToggleStore {
name: data.name,
description: data.description,
enabled: data.enabled ? 1 : 0,
archived: data.archived ? 1 : 0,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing archived from eventDataToRow() should not have negative side effects

Copy link
Contributor Author

@Kouzukii Kouzukii Mar 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does seem to affect e2e tests however, because they use this to create archived feature toggles

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

turns out this is covered by the featureSchema validation, so we can leave this unchanged

if (strategies) {
result.strategies = (await strategyStore.getStrategies())
.filter(strat => strat.editable)
.map(strat => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solved via strategyStore.getEditableStrategies()

Kouzukii added a commit to Kouzukii/unleash that referenced this pull request Mar 14, 2019
Kouzukii added a commit to Kouzukii/unleash that referenced this pull request Mar 14, 2019
.where({ name: rowData.name })
.update(rowData)
.then(result =>
result === 0 ? this.db(TABLE).insert(rowData) : result
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, we do this now and can improve with upsert in a later version

@Kouzukii
Copy link
Contributor Author

I'll add the documentation today and then we could merge it.

@ivarconr
Copy link
Member

ivarconr commented Mar 15, 2019

Cool. I've tested the functionality on our setup and I see that relying on validating import with the current feature-toggle schema has some weaknesses.

E.g. In our setup we have toggles with description as an empty string (validation was not that strict in early versions of unleash). I will try to sort these things out so that we don't end up in a situation where one is not able to import an export.

@ivarconr
Copy link
Member

ivarconr commented Mar 15, 2019

Cool. I've tested the functionality on our setup and I see that relying on validating import with the current feature-toggle schema has some weaknesses.

E.g. In our setup we have toggles with description as an empty string (validation was not that strict in early versions of unleash). I will try to sort these things out so that we don't end up in a situation where one is not able to import an export.

So I created a patch for the schema validation to be truly backward compatible, would you mind adding this patch to this PR?

https://gist.github.com/ivarconr/291d5e6f58fc0a02ba58ed6549192cc2#file-schema-patch

(git apply mypatch.patch)

@Kouzukii
Copy link
Contributor Author

So I created a patch for the schema validation to be truly backward compatible, would you mind adding this patch to this PR?

Done, as well as the documentation. Unless you see anything else I think it's ready for merge.

@ivarconr
Copy link
Member

Nice job @Kouzukii! I think this is ready too.

I have started on a UI for this as well.
https://github.com/Unleash/unleash-frontend/compare/feat/admin

@ivarconr ivarconr merged commit 1f55eae into Unleash:master Mar 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants