New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add import & export by file, admin api and methods exported by start() #395
Conversation
Great idea! 🎉👍 The code need some improvement and we need to discuss a few concerns. Will come back to you soon when I have some time !⏰ |
So I finally have some time to follow up on this!
|
lib/server-impl.js
Outdated
server, | ||
eventBus, | ||
import: importData.bind(null, config), | ||
export: exportData.bind(null, config), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you think it is important to expose import/export outside of Unleash? How do you think this should be used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as an alternative option to the admin API which requires authentication.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in our specific use case, we have
features.yaml
which is an export from the production system
features.overrides.yaml
git-tracked overrides
features.local.yaml
non git-tracked overrides
the entrypoint merges all 3 and imports them into unleash.
maybe it should allow multiple --importFile
's ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
on a second thought, I don't think allowing multiple importFile
's is a good idea. The logic for merging the export files is a bit displaced. This should be done by a user defined entrypoint.
You mean export a single featue toggle?
I think this could be easily added to the feature overview in the frontend.
The drop functionality is supposed to be for disposeable environments where the history doesn't matter. If the user wants the history to be preserved, he should create a database dump. |
One thing that worries me about you suggested approach All changes of feature toggles and strategies in the current unleash design are driven through the
What do you think? |
No, I mean export just the list of "feature toggles" or just all "strategies". I think there is cases where I would like to export all toggles from one env and apply them to another env.
After thinking about it I don't think this is necessary:
|
No,
this would bloat exported data, the idea was to keep the export as minimal and humanly-readable as possible.
this is already possible |
lib/routes/admin-api/import.js
Outdated
async import(req, res) { | ||
const userName = extractUser(req); | ||
const { drop } = req.query; | ||
const data = req.body; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should accept data sent as a file upload.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. I'll add an optional file
parameter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But this would require multipart/form-data. Right now we're only using bodyparser.json()
middleware.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
I agree, and a yaml file is perfect for this.
I can look in to this (We should be able to merge this PR before we have a perfect solution, we can always refactor so no worries.)
Could "flush database" be a separate concern? I think that in your case of disposable environment it would make sense to:
Agree? I think this can work for the import API also ( But I have a few worries about flushing:
|
sounds good, I'll implement this.
this already is a concern with the current db-migrator on an empty database, no?
if we implement a db lock, we could just return a 5xx error code when requesting |
No, db-migrator uses transactions to avoid race conditions.
Sounds like an ok solution |
On second thoughts; Importing at startup will not work well with the multiple instances case. It will end up importing and updating the same state multiple times. if one add dropBeforeImport all toggles will be droppet at startup importet for each instance. We must find a solution to avoid this behaviour! Top of my head is that we could introduce a helper table for imports where the file content is hashed and stored. Import will just happen if that table does not contain the desired importFile. This will make it impossible to reimport the same content multiple times. This might however be undesirable, if unleash has been changed and one would like to reset to previous state. |
@Kouzukii, after thinking about this a bit more I want Unleash to not take too much responsibility in to around guaranteeing non-duplicate imports across multiple instances of Unleash. My suggestion for the way forward is the following: 1. Unleash exposes stateServiceUnleash will return a service capable to perform import and export. It will be up to the implementer to use Unleash as a library and take advantage of the import. Unleash will only accept json as import. Unleash will only perform the operation as requested, it will be up to the user to assure that it only happens once per instance, if this matters. Import will follow the import protocol described in 4). unleash.start({...})
.then(unleash => {
const myData = unleash.stateService.export({strategies: true, featureToggles: true});
unleash.stateService.import({data: myData, userName: 'importer'});
}); Import functions will in addition follow the flush protocol, described in 3). 2. Unleash exposes import/export APIs
This endpoint will be used to export feature toggles and strategies. Should support json, yaml and file-downloads.
This endpoint should support json, yaml or file uploads of json/yml. It also take addition argument to flush database which should follow the flush protocol, described later. 3. Flush protocolThis is a special event stored in the event-store. It exits one for strategies and one for feature toggles. All flush events are stored just as any other unleash event is stored. await eventStore.store({
type: FLUSH_STRATEGIES,
createdBy: userName
}); The result of FLUSH_STRATEGIES is that all strategies are deleted. This is handled by the strategy-store.js. await eventStore.store({
type: FLUSH_FEATURES,
createdBy: userName
}); The result of 4. Import protocolWhen a toggle/strategy is imported it should be marked as just that. It should be up to the respective store to do conflict resolution (If toggle exists it should be updated with the new state. If toggle is archived it should be automatically revived as an effect of this change). Example of an imported toggle event. await eventStore.store({
type: FEATURE_IMPORT,
createdBy: userName,
data: <toggleDef>
}); Example of an imported strategy event await eventStore.store({
type: STRATEGY_IMPORT,
createdBy: userName,
data: <strategyDef>
}); I hope this makes sense. I love to hear your reactions and suggestions on the purposed way forward. |
@ivarconr this seems like a very elegant and future-proof solution. |
Cool, do you want to refactor your PR in this direction or do you want me to build on your great work and do it (I have some time later this week)? |
I am already on it ;) |
Your are awesome ⚡ |
Are you sure about this name? |
No, I'm open to keeping drop |
if (!strategies && !featureToggles) { | ||
strategies = true; | ||
featureToggles = true; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be a bit better to just:
const strategies = req.query.strategies ? true : false;
const features = req.query.features ? true : false;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well you could do Boolean(req.query.strategies)
as well
the point was, if no query arguments are passed, both features and strategies are exported.
const YAML = require('js-yaml'); | ||
const moment = require('moment'); | ||
const multer = require('multer'); | ||
const upload = multer({ limits: { fileSize: 5242880 } }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so the limit i 5MB? Why did you choose this number?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the file is kept in memory, so I chose a number that wouldn't impact memory usage greatly.
Maybe this should be exposed as a config?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, sounds reasonable. I would be very surprised if anyone need more than 5MB, but you never know.
lib/server-impl.js
Outdated
const server = app.listen({ port: options.port, host: options.host }, () => | ||
logger.info(`Unleash started on port ${server.address().port}`) | ||
); | ||
|
||
return new Promise((resolve, reject) => { | ||
server.on('listening', () => resolve({ app, server, eventBus })); | ||
return await new Promise((resolve, reject) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why should Unleash return a promise if it awaits for it to complete?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not quite sure what happens when you return a promise in an async method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the caller will have to wait of it to complete, if they care.
lib/state-service.js
Outdated
} | ||
|
||
async importFile({ importFile, dropBeforeImport, userName }) { | ||
let data = await new Promise((resolve, reject) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let data = await new Promise((resolve, reject) =>
fs.readFile(importFile, (err, v) =>
err ? reject(err) : resolve(v)
)
);
can be simplified to:
let data = await fs.readFile(importFile);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it this possible without fs-extra?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no your right. require('fs').promises
was added in node 10. I don't want to require new node version before next major of unleash.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you are doing the await straight away there is no reason to no just use the sync api for reading the file.
A async aproach could be:
/// util functions:
function readFile(importFile) {
return new Promise((resolve, reject) =>
fs.readFile(importFile, (err, v) => (err ? reject(err) : resolve(v)))
);
}
function yamlToJs(importFile, data) {
return mime.lookup(importFile) === 'text/yaml' ? yaml.safeLoad(data) : data;
}
using the util functions the importFile could look like:
importFile({ importFile, dropBeforeImport, userName }) {
return readFile(importFile)
.then(yamlToJs.bind(this, importFile))
.then(data => this.import({ data, dropBeforeImport, userName }));
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you don't like await huh 😄
but i like this approach, done.
lib/state-service.js
Outdated
async import({ data, userName, dropBeforeImport }) { | ||
const { eventStore } = this.config.stores; | ||
|
||
if (typeof data === 'string') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be handled in importFile. import
should assume and validate json data.
lib/state-service.js
Outdated
data = JSON.parse(data); | ||
} | ||
|
||
data = await joi.validate(data, dataSchema); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't mutate input arguments. You may break clients calling this method.
const importData = await joi.validate(data, dataSchema);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add this to eslintrc ;)
lib/state-service.js
Outdated
data: { name: 'all-features' }, | ||
}); | ||
} | ||
for (const feature of data.features) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use forEach
instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now using Promise.all(data.features.map(...))
lib/state-service.js
Outdated
} | ||
} | ||
|
||
async export({ strategies, featureToggles }) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
better names for readability would be inlcudeStrategies
and includeFeatureToggles
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and should the default to true?
async export({ inlcudeStrategies = true, includeFeatureToggles = true })
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took this from your example ;)
But I'll change it.
lib/state-service.js
Outdated
if (strategies) { | ||
result.strategies = (await strategyStore.getStrategies()) | ||
.filter(strat => strat.editable) | ||
.map(strat => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better if the strategyStore
could return all not-built-in-strategies. This could be a filter argument, alla: strategyStore.getStrategies({builtIn: false})
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
solved via strategyStore.getEditableStrategies()
lib/db/feature-toggle-store.js
Outdated
@@ -96,7 +100,6 @@ class FeatureToggleStore { | |||
name: data.name, | |||
description: data.description, | |||
enabled: data.enabled ? 1 : 0, | |||
archived: data.archived ? 1 : 0, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removing archived from eventDataToRow()
should not have negative side effects
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it does seem to affect e2e tests however, because they use this to create archived feature toggles
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
turns out this is covered by the featureSchema validation, so we can leave this unchanged
lib/state-service.js
Outdated
if (strategies) { | ||
result.strategies = (await strategyStore.getStrategies()) | ||
.filter(strat => strat.editable) | ||
.map(strat => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
solved via strategyStore.getEditableStrategies()
.where({ name: rowData.name }) | ||
.update(rowData) | ||
.then(result => | ||
result === 0 ? this.db(TABLE).insert(rowData) : result |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, we do this now and can improve with upsert in a later version
I'll add the documentation today and then we could merge it. |
Cool. I've tested the functionality on our setup and I see that relying on validating import with the current feature-toggle schema has some weaknesses. E.g. In our setup we have toggles with description as an empty string (validation was not that strict in early versions of unleash). I will try to sort these things out so that we don't end up in a situation where one is not able to import an export. |
So I created a patch for the schema validation to be truly backward compatible, would you mind adding this patch to this PR? https://gist.github.com/ivarconr/291d5e6f58fc0a02ba58ed6549192cc2#file-schema-patch ( |
Done, as well as the documentation. Unless you see anything else I think it's ready for merge. |
Nice job @Kouzukii! I think this is ready too. I have started on a UI for this as well. |
This PR adds a state service which enables import & export of features and strategies as JSON and YAML via /api/admin/state/export and /api/admin/state/import, via configuration option
importFile
as well as by:The option
dropBeforeImport
will delete all existing features/strategies before importexample npx usage
npx unleash-server --databaseUrl ... --importFile features.yml --dropBeforeImport
This allows you to quickly bring up an unleash server for client development with custom feature toggles, without using database dumps.
To-Do