Webpack require context support

[hanstf]

This is a great tools and we have been using it for a while.

Recently we would like to build our code with webpack and planning to use the require.context to build all of the seed files. It would be great if mongo-seeding able to read from the context as well. Or can take the inspiration of knex seeding's seedSource option where we can supply the method to get the content and the list of the folders.

[pkosiec]

Hey @hanstf 👋 This is purely webpack-related and I'm not sure how it could work if mongo-seeding doesn't use webpack for build, especially that you use already built library 🤔

Or can take the inspiration of knex seeding's seedSource option where we can supply the method to get the content and the list of the folders.

Well, this is effectively what the readCollectionsFromPath does. You can totally read the collections by yourself, even customizing the code from here: https://github.com/pkosiec/mongo-seeding/blob/main/core/src/populator/populator.ts#L50.
And then, you'll do import.

Does that make sense?

[hanstf]

Hi @pkosiec,

Thank you for your quick reply!

Let me give more context on this, currently our project is a monorepo applications, with some common libraries that compile with webpack and we configured it not to compile the node modules so mongo-seeding is not compiled and imported as it is from node modules. We also configured 2 entry points in webpack, 1 for the main node js server and the other one is to perform seed.

We are trying 2 ways to integrate with mongo-seeding in our seed entry point:
1st way:

• webpack will compile seed application and perform tree shaking of the imports to the common library so that it will generate a single file
• copy the .ts seed files into the dist folder as it is without any webpack compilation for each of the seed file
• readCollectionsFromPath will read from that seed file folder
• but since the readCollectionsFromPath will uses the import-fresh to do the dynamic import, and my seed file contains an import to our common library, it wont compile correctly unless I am doing a full copy of the whole project

we can set entry point for each seed file but it will be too tedious and will create a lot of duplicate codes

2nd way that we are using now is we are excerpting the populator code with require.context method from webpack to dynamically include a file into the compiled code

const seedFileNames = use webpack require.context() to get list of the seed file names
const nameToContentMap = use webpack require.context() to map a file name with its seed content;
const collectionsMetadata = [];

for (const seedFileName of seedFileNames) {
      const folderName = path.dirname(seedFileName);
      const splitFolderName = folderName.split('/');
      const lastFolderName = splitFolderName[splitFolderName.length - 1];
      const metadata = getCollectionMetadata(lastFolderName); // copied from populator class
      collectionsMetadata.push(metadata);
}

for (const collection of sortCollections(collectionsMetadata)) { // sort collections copied from populator class
      const content = nameToContentMap.get(collection.name);
      const collectionName = collection.name;
     // do something with the content and collectionName
}

So would be great if mongo-seeding provides an API instead of supplying a path, we can supply an array of folder and method to get the content/even the content itself. Similar to knex custom seed source, https://knexjs.org/guide/migrations.html#custom-seed-sources.

Sorry for this long explanation, you may ignore if this is not something related to mongo-seeding

[pkosiec]

Hi @hanstf,
Thank you very much for the explanation, and also sorry for a long delay. I think I get the problem, but I'm a bit struggling with how such API could look like in Mongo Seeding. Just to be clear - I absolutely want to solve the problem you described 👍

From what I understood, you'd still like to use most of the logic of readCollectionsFromPath but read the files in a different way. I thought in such cases where readCollectionsFromPath is not useful, users instead would provide the SeederCollection[] to the import method by their own.
I do understand that in your case this would be so similar to the readCollectionsFromPath that you'd like to use mongo-seeding logic related to getting collection metadata (order, name) based on directory. Is that correct?

I'm thinking about the following options:

1. exporting all necessary helpers needed to make sure reading your collections is a few lines of custom code
   • what that would be though?
2. adding readCollectionsFromSource(source:CollectionSource) method, where the CollectionSource would be similar to what you proposed, like

interface CollectionSource {
  getDirectories: () => Promise<CollectionDirectory>;
}

interface CollectionDirectory {
  name: string;
  getFiles: () => Promise<CollectionFile>;
}

interface CollectionFile {
  read: () => Promise<unknown>;
}

or

interface CollectionSource {
  getDirectories: () => Promise<string[]>;
  listFiles: (directory: string) => Promise<string[]>;
  readFile: () => Promise<unknown>;
}

What I dislike in this approach is that it is still somewhat tied to filesystem 🤔 Maybe there's a more elegant way to abstract it? 🤔 Of course the readCollectionsFromPath would use this readCollectionsFromSource method underneath.

3. or... maybe you need just the readFile abstraction? That could be configurable as a part of Mongo Seeding configuration.

What do you think?
Cheers!