Anyway simpleschema, collection2 and peerdb can work together

[emmanuelbuah]

First of all, I want to say kudos for putting in the work to get a reactive layer on top of mongo (server-side) to handle declarative updates. Both collection2 and simpleschema solve specific issues, same as peerdb. I personally see peerdb more of a better package to replace collection-hooks but I wonder if peerdb api can be made to work with collection2 and simpleschema. I really see 3 concerns here:

Data schema integrity and validation -> SimpleSchema
Collection data schema validation on insert, upsert and update -> Collection2
Provide Collection data hooks on insert, upsert, update and delete -> Collection-hooks || Peerdb

I will personally like to use Peerdb over collection-hooks because of its reactive nature and the ability to handle hooks even if the changes are made by another client. For me peerdb is a clear winner but I cant seem to feel like it will even be much better if it can work in tandem with SimpleSchema and Collection2.

My question is - how do I get the benefit of simpleschema + collection2 + peerdb. Is it be possible to integrate all 3 to work together. Will it be better to integrate peerdb definitions into simpleschema since both of them are in a way defining metadata information. Example, If would be great if we could do something like

//data validation provided by simpleschema 
PostSchema = new SimpleSchema({
    content: {
        type: String,
        optional: false,
    },
   author: {
     type:Object,
     optional: false
   },
   "author.username": {
      type: String
  },
   "author.displayName": {
      type: String
  },
   hits: {
        type: Number
    }
   //etc
});

//peerdb hook through SimpleSchema
PostSchema.References({
    fields: function() {
      return {
        author: Post.ReferenceField(Person, ['username', 'displayName']),
      };
    }
  });

PostSchema.Triggers({
    //trigger definition here ....
  });

Post = new Meteor.Collection('posts');
Post.attachSchema(PostSchema); //provided by collection2

In the approve sample code, simpleschema handles validation (i.e. make sure author's username and displayname is a string), collection2 handle running validation and peerdb extension to simpleschema + collection2 maintains data references and hooks. This is just a thought but I think something along these lines will provide the community with 80% of what its needed for data manipulation. I want to use peerdb now but I also don't want to loose my schema validation checks on my collection (as well as on the client through autoform)

[mitar]

So how I imagine this to work is that I extend definitions of fields also to include non-reference fields and then validate documents in objectify. Once we have that we can also add a way to export schema definition as simple-schema, so that you can use it in meteor-autoforum. The issue I have by using simple-schema internally as well is that PeerDB support references, so schema returned will have to be normalized for simple-schema output (so subdocuments will not be seen as references), while objectify might be able to recursively go and check things because it will know where are subdocuments in fact instances of another document.

[mitar]

So I think that PeerDB replaces collection2, but can use simple-schema for compatibility with other packages.

[emmanuelbuah]

@mitar Thanks for your response. You know best but I think integration with simpleschema will increase peerdb adaptation since its widely used for Meteor projects. Another place of concern (subjective) is the name and constructor api.

Name

When I saw peerdb, I thought it was a new reactive/ddp database like redis-livedata. Just like you said, its a better collection2 so might be better to give it a name relating to mongo/collections. Examples peer:meteor.collection or peer:collection. This will make it easy for people to identify its use at first glance.

Definition api

In relation to collection definition api, I love coffeescript but I think I'm right to say there are a lot more people pure javascript devs than coffeescrip. With that said, the current peerdb document definition api seems a bit ....verbose if written in javascript by hand. Below is a person document definition in pure javascript without coffeescript

var Person,
  __hasProp = {}.hasOwnProperty,
  __extends = function(child, parent) { for (var key in parent) { if (__hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor(); child.__super__ = parent.prototype; return child; };

Person = (function(_super) {
  __extends(Person, _super);

  function Person() {
    return Person.__super__.constructor.apply(this, arguments);
  }

  Person.Meta({
    name: 'Person'
  });

  return Person;

})(Document);

Compared to Person collection definition is Mongo.Collection or Collection2

  var Person = new Meteor.Collection('Person')

Is it possible to make the metadata definition api as simple (hide the inheritance complexity under a simple api). Example. Peerdb collection/document can be defined using the same kind of definition style as Meteor.Collection/Collections2 and simpleschema such as ...

//definition 

 var Person =  new Peer.Collection('Person',{
    idGeneration: 'STRING'      //options passed to Mongo.Collection
   } ,
  fields :{
      username: {
        type: String,
        optional: false,
      },
      ...
  });
  var Posts =  new Peer.Collection('Posts',{
    idGeneration: 'MONGO'      //options passed to Mongo.Collection
   } ,
  fields :{
      content: {
        type: String,
        optional: false,
      },
      author: {
       type:  Object
       reference: ReferenceField(Person, ['username', 'displayName']),  
       optional: false
     },
     subscribers: {
       type:  [Object]
       reference: [ReferenceField(Person)],
       optional: true
     }
  }
}

//fetch
Posts.find({}).fetch()
//schema 
Posts.schema() 

This is all subjective. I just wanted to offer my thoughts as a dev and supporter. Keep up the great work.

[Sivli-Embir]

I personally dislike SimpleSchema but I think @emmanuelbuah makes a lot of good points. I would agree the name could be better. Also while I love Coffeescript and I feel @emmanuelbuah is right people will want a choice between pure JS and Coffee.

[mitar]

Just for the record, name stays. :-) I have many projects starting with "peer" in this is one of them. :-) It is a family of related projects and while PeerDB currently resembles collection2, it has quite some other features in the future I am planning to implement. Like versioning. My goal with the project is to create an easy way to build collaborative tools on top of Meteor. And PeerDB is a layer for that. If others find some other uses to it, even better. But in my context PeerDB is a nice name for something which helps collaboration.

About JS integration: I would welcome a pull request with somebody making an alternative API which you can call with something like Document.create(...) or something, which is more suitable for JS programmers. And which would then internally extend CoffeeScript class and do the rest. Maybe it could be simply something similar to what you pass to Meta. So same function or something. (You need function to be able to define lazy references between documents. So a way how SimpleSchema works cannot really work once you have references.) So, some boiler-plate code for making it easy to use in JavaScript.

About SimpleSchema integration: I welcome also a pull request which would add to Meta a method which converts it into SimpleSchema compatible schema. So once you create a document, you can access fields for example with Post.Meta.fields, so there could be Post.Meta.simpleSchema(). This can then be used to pass to autoform and similar packages.

For that to work we first have to create also other fields, not just references. So the idea is to create similar classes for other types of fields, where then community can create even more specialized fields.

It would be interesting also to create fields which allow serialization/deserialization to internal MongoDB types from higher-level types (think IP field), but on the other hand Meteor does this through EJSON, so an idea would be that defining a PeerDB document would also based on fields define an EJSON type for direct serialization from JavaScript document. Just some ideas.

I am designing PeerDB to be pretty similar to how Django (or Mongoose) is doing things (class-based, having abstract class, using Meta and document managers). So if anyone like to implement parts of that (for example, field classes for various basic types found in MongoDB) in similar spirit, that would be a pull request I would be happy about. (So if you ever wonder how to design something that I would gladly merge, think from Django/Mongoose approach.)

[mitar]

I see above also another implicit suggestion, to allow other forms of ID generation. I think that could be made into another ticket and could become part of Meta definition if somebody wants to tackle. For now, you can always define class like:

class Post extends Document
  @Meta =>
     name: 'Post'
     collection: new Meteor.Collection 'Posts', idGeneration: 'MONGO'

I think it should work, but I have not tried it. Feel free to make a pull request with a test. (We already have 0.5 MB of tests, so why not more.)

[emmanuelbuah]

@mitar Yes. I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition. I noticed your PeerLibrary project a few days back. I think its a great project. I actually browsed some of the other relating libs. I will spend sometime to digest your feedback in relation to where help is needed. I might be able to take some of the tasks.

[mitar]

I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition.

I would not pass things directly, but provide configuration options for Meta.

[mitar]

I just published another package, a way to define publish middleware.

[testbird]

I can confirm that the name "peerdb" had me overlook this project, because it sounded like an alternative db backend, not like bringing a missing completion for meteor collections.

BTW: The corresponding collection2 issue is here: Meteor-Community-Packages/meteor-collection2#31 (simple-schema definitions will get references/relations support)

[testbird]

Thank you for your name explanations above mitar. I think it created the understanding in me that peerdb is not at the database level, but above meteor collections, it provides "peer collections" or "peer documents" :-)

[testbird]

Concerning the new "meteor-middleware" package you mentioned above, please allow me to comment that its naming may be non-self-descriptive for the community as well. (Making it unfortunately and unnecessarily hard to find, notice and grasp.)

If i understant it at correctly, the package improves what meteor users know as publish functions, and provides stackable publish hooks? May this be described as "peer publishing stack" or simmilar?

[mitar]

:-)

Ah, lovely naming discussions. :-)

[mitar]

I think the best way to find packages is to add links to them to wikis, reference them in mails, etc. :-)

[mitar]

So what would be a better name (package name) in your opinion?

[testbird]

Hmm, I think a good library name (same for functions etc.) is one that can give a new user an idea of what the lib does and handles (building on the conventions that already exist among potential users).

Note that my thinking of peer documents and a stackable publish hooks may totally describe wrong holes for the nails. I don't know if "peerdocs" or "peerpublish" -stack? is something that could be matching your thoughts of your packages closely enough.

[mitar]

So for me middleware is something intuitive because it is the way express works. And for me this is exactly what publish is in Meteor, a stack of things you want to send. Express sends HTML, Meteor sends documents.

[testbird]

Ok then, I understand where you're comming from. It easily happens to mix contexts, and confuse or misunderstand each other. I don't think midleware has a specific meaning in meteor yet, sure it describes other things in other contexts. From (my limited) meteor user side view your improving the publish workflow (possibly dramatically I can't tell yet).

(I kind of like many naming conventions in meteor also because they use words that have matching "real world" meaning like "publish".)

[derekrazo]

@testbird

I completely agree that the naming choices made for the peerdb, and meteor-middleware packages are non-descriptive. This indeed makes both packages unnecessarily hard to find, notice and grasp.

[derekrazo]

List of potential alternative names:

peerdocs
peerpublish
publishstack
hookstack
publishhookstack

@mitar any of these look better to you? :)

[testbird]

Back to topic, peoples' thoughts about the planned references/relations feature for simple-schema, collection2 and autoforms would be interesting. Meteor-Community-Packages/meteor-collection2#31

Maybe code could be shared to define/handle references/relations.
And code could be shared to keep fields in sync. Meteor-Community-Packages/meteor-collection2#31 (comment)
While the peer* packages can continue to provide its class based interface.

So there could be references with cached fields (with automatic hadling of syncs) as well as reference only fields (with automatic handling of publications). Forms to switch references, and forms that cascade down to edit referenced docs...

IMHO it would be great if collection2 and "peerdocs" could both provide read-time optimized, (de)referenced and cached, write-time sychronized embedded fields based on shared code.

[emmanuelbuah]

@testbird @mitar @derekrazo In relation to what makes a good name, I tend to use the rule "Don't make the user think". A good name does an extremely good job of convey what the entity it represents is used for. @mitar I really like the work you guys are doing with peer library and believe some of the projects you have created will be very useful to the the meteor community as a whole. To make them reusable in other projects other than peer library, I will encourage they be treated as such which includes naming and implementation(decoupled from the peer library ecosystem/no strict dependencies). I think you guys are on your way to achieving both.

As to peerdb, I still think it would benefit from a name change. I like @derekrazo suggestion of peerdocs(or peercollection for peerdb) and peerpublish(for meteor-middleware) is great. They are good name that convey the use of both libraries as well as giving credit to the peer community.

[mitar]

I will make an executive decision: the name stays. I understand your concerns and I really appreciate that you care about the package and that you gave me thoughtful feedback, but I think it is a good name: it is short, quite unique (only ~3000 hits on Google), searching for "peerdb" already gives the first hit this repository, and as mentioned previously, it aligns with other components which are of interest to me. And I think it would already be too much work for everybody to change it.

Let's focus discussion to other things please. And let's just make this project even more popular and then the name will not matter.

(BTW: One issue here is that Meteor currently searches only Meteor package names and not also descriptions, see meteor/meteor#2709, so this makes the package hard to find. Once that is addressed name will have less importance.)

[testbird]

All right.
As you say, it was just feedback about the current search and comprehension experience, and it remains your decisionion how to make the best out of the current pre 1.0 meteor package renaming mess. Concerning the google paths, nevertheless, I dont't think it would be much of a problem, if one happens to find the old git project, it should just properly point to the new project page.

---

Ok now, brainstorming to working with collection2.

Both projects provide their database object abstraction layer. Both want to support relations/references among collections. Peerdb already has code to support references and maintain "write-time embedded cache documents" of referenced documents.

Peerdb and collection2 follow different API styles.

What could be the benefits to factor out and sharing code to maintain "write-time embedded cache documents"? Or even moving it into collection2?

• Peerdb bulding on tried and tested collection2 objects?
• Autoform support to properly render all the kinds of forms required to manage relations (switching/adding/removing links, editing referenced documents (cascading) for peer documents.
• Peerdb focusing on providing a different API based on collection2 objects?
• Benefitting from many allready existing and tested field and validation features.
• Eventually also benefitting from support for traditional referencing scheme (read-time fetching of references)?

???

[testbird]

Couldn't you make peerdb work "on top" of collection2s? Meaning one could define and use several collection2s, and peerdb package could use those collections, plus implements automatic syncing of references? (even if, for now, it would require defining the fields a second time in collection2)

If that is possible simple-schema/collection2 folks may be able to quickly support relation definitions in their syntax, by making use of the working auto-sync implementation from peerdb.

And the good part beyond sharing code is, once this works, collection2 folks could start working on the autoform support for easy handling of relations, while peerdb could focus on its own syntax for schema definition, possibly translating to the new relation-aware simple-schema internally.

[aldeed]

Hey all, just now getting around to reading this thread. I hadn't looked at peerdb until now either. So...

• Haven't tried it yet, but peerdb seems to solve a number of issues. Nice job!
• With regard to simple-schema, the thing I hear most from those who use it is that they want to define the relationships in the schema object.
• In most projects, I think I would personally prefer to not have the ORM layer on top, but in a very large app with complex relationships, I think peerdb would be super helpful.
• I'm hoping to start c2 relationships development by creating a joins package within the next few weeks. It will be targeted at those simpler cases where you don't necessarily want the full peerdb Document approach. For starters, I only plan to support find/publish.
• Later, more advance features like the reactive cascading operations could possibly be added, and maybe some of the peerdb logic could be pulled out into a common package for us to share.
• I agree that we should try to let peerdb and collection2 work together as much as possible. I don't necessarily see how peerdb is a replacement since it doesn't validate against a schema, does it? It would replace autoValue functions maybe, but not anything else. Possibly I could make a peerdb extension package that allows you to attach a schema to the underlying collection, too, using collection2.
• I don't think it would be overly hard to make autoform work out of the box with peerdb, provided we figure out a way to attach a simple-schema to the underlying collection and expose it.

[testbird]

Hi Eric! Mitar also provides a package for publishing relations https://github.com/peerlibrary/meteor-related, its appoach is most straight forward from the packages I've seen. It simply allows to use publish functions within publish functions recursively. Interesting if you want to generate default publishers from schema definitions?

But IIUC, once relationships can be defined in simple-schema/collection2, there are two alternatives to impementing them. One is the once-at-write-time updated "embedded reference field cache" (peerdb) and the other is the at-each-read-read-time (in meteor plus at-publish&recomputation-time) "normalized" references fetching.
Currently I can think for an app to decide towards normalized fetching only if the amount of duplication would be unbearable or the apps write-load is actually higher then read-load.

Mitar, on the peerdb project page you mention a limitiation for peerdb that I think I have not fully understood, yet: It says [peerdb syncing embedded fields] "requires that those relations are the same for all users". Isn't it possible to let the publish function only publish different sets of (synced and not synced) fields to different users? Since as for the user references in peer documents are regular fields of regularly published documents? Now I noticed the project page says nothing about actually publishing peer documents, so may that where I'm thinking wrong here?

[mitar]

It simply allows to use publish functions within publish functions recursively.

I must say that I have not tested it with more levels. If you did so and it works for you, please tell me. Or contribute a test case. :-)

(Mostly because I believe that things probably get very underperforming at that stage and you should probably rethink your approach to data design.)

Isn't it possible to let the publish function only publish different sets of (synced and not synced) fields to different users?

You can. But that is not the task of PeerDB. That's why there is middleware, so that you can easily filter things for particular users, and reuse such filters.

What I am trying to say there that PeerDB works on data without an user context. So when you are syncing relations, they are done for all users. Of course you can filter things out, but they will require that you store things for all users.

So it is best used for data which are somehow static for all users, static to data itself. For example, author of a blog post is the same for everyone, is part of data itself. And is good candidate for PeerDB. But a post field which would contain likes of all user's friends of that blog post is a bad candidate. It is not the same for all users. In that case you probably want to embed data for all users (which can become quite large), or dynamically resolve things (using related).

So for me all those three packages nicely play with each other and address various use cases.

[aldeed]

Re meteor-related, I have been using publish-composite lately, but I think all the similar packages (there are 5 or 6 now) have good and bad points. I am planning to dig into them all more, figure out the differences and which is best, and then hopefully just use that package as a base and write the collection2 joins package on top of it.

As @mitar says, I think different relationships deserve different treatment. Sometimes a read-time join might be best while other times a write-time join might be best. My sense is that read-time joins are simpler to maintain and understand, and so I'd like for a collection2 rels package to focus on that but eventually provide a very simple "switch" you can flip to begin storing a property in denormalized and synced fashion (and ideally be able to "unflip" the switch too if you change your mind). This can get tricky and might be better left to a full scale solution like peerdb.

[mitar]

This can get tricky and might be better left to a full scale solution like peerdb.

I was thinking about a similar thing. To have some analysis of your relative queries and would then propose you to optimize data at write time.

[aldeed]

@mitar, better yet allow user to define some optimization parameters (optimize joins if property lookups per minute exceeds N or property lookup time takes > N ms), and then let it intelligently optimize. Performance costs might outweigh benefits, but maybe you could turn on for awhile until things are optimal, then turn off.

[testbird]

With meteor-related I actually hadn't thought of multi-level recursion. You're right of course, Mitar.

Thanks also for explaining on the peerdb "limitation" you mentioned. I think what you explained pretty much matches what I tried to describe with the duplication drawback of syncing embedded fields. I wouldn't think of it as a hard "limitation", but something to weigh compared to read-time relation fetching.

Concerning the feature switchablility: There are many important use-cases, where one would like to sync a field with its reference field only in one direction (e.g. central current state vs. data entry and history records), and I think only the write-time sync approach can nicely support these "directional" use-cases. (seems clearer to me now than at the time of my 'syncCachedField:' proposal Meteor-Community-Packages/meteor-collection2#31 (comment))

For weights I have now:

write-time syncing

• + allows to define directional updates
• * high write loads may bring performance or eventual consistency issues.
• - introduces duplication

publish + read-time relation fetching

• - introduces performance scaling issues (round trip delays)

In my case (many cases?) the duplication and unidirectional syncing is actually the desired solution, and the write-load can be considered low for these records.

Concerning the decisions about to "cascade changes or change references", I have doubts it can be solved satisfactory on the collection/schema definition level.
I tend to think this may ulitimately only be definable at the UI definition level, where the desired widgets or switches can be chosen based on the UI usage context.

[mitar]

high write loads may bring performance or eventual consistency issues.

That's why you do those updates in dedicated workers. PeerDB allows you to configure it so that you can use dedicated workers and split updates across multiple of them (each taking care of a subset of a database). This way your writes finish quickly, are just normal inserts. And then eventually PeerDB sync things as they change. In practice this works very well because you often do not need to update all references immediately after one change. (But probably it depends on your use case.)

introduces duplication

You mean data duplication? I think this is OK and something natural in MongoDB. Data denormalization is part of non-relational databases. This is why you can then have nice sharding. Ideally, you would have to do only one query to the database and get all data back in that one document you get back.

Concerning the decisions about to "cascade changes or change references", I have doubts it can be solved satisfactory on the collection/schema definition level.

I don't understand you here. What is "cascade changes or change references" for you? PeerDB nicely solves complicated references and cascades. It works very well. You can mix anything.

[testbird]

Jep, I was refering to data duplication, it is not a inherent consistency concern here, because it is a programmatically managed cache, but we still have to consider the storage and transmission (filtering) concequences.

To "cascade changes or change references" came up in Meteor-Community-Packages/meteor-collection2#31. As I understood it, it is this: When a change is made to a field A that relates to another records' field, should that change be made to the referenced field (cascade), should the target that A relates to be changed to a record that matches the new value, or should a new record be inserted with the new value in the referenced collection.

The write-time syncing approach may be able to steer clear from this, because all the syncing that ever happens is well defined, and the user continues to accesses the different collections explicitly, without necessarily thinking in any artifical ("ORM vietnam" like) meta model that spans over multiple lower-level collections, bringing with it a lot of implications.

[testbird]

Maybe a collection2 read-time reference fetching could steer clear from the problem if the (write-time) cascading is defined in the same explicit way as for write-time syncing (define them all as "two-way syncs" and flag them as read-time join), and using these definitions only as defaults for direct programmatically writes to the relating fields (i.e. use the convention to cascade by default). And leave all the other forms of handling relations to be explicitly defined in the UI (autoform), where the user will have to be explicit about what kind of form is appropriate for the context.

[testbird]

Forgot a very clear solution that combines read-time joins with write-time concistency (transactions and rollback): http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits Just store all links between documents as atomical documents in a separate collection (also very useful to track the status of usynchronous updating of the actually related documents). And the link records would also allow to be given their own names and properties.

[mitar]

OK. I think we should be wary of reimplementing whole relation database on top of MongoDB. :-) I think that MongoDB has its advantages and is useful when you can define your data structure so that it does not require much dynamic relations. In many cases you can do that. If you need more than that, then probably some SQL database is more suitable.

[testbird]

Thanks, you pretty much summed it up. As you didn't mentioned nothing concerning the thought that write-time syncs stay clear from the problems, I'd currently take away:

• Defining write-time syncs is much less likely to introduce implications or side-effects (Then defining some sort of mongo-joins that introduce loose ends, or implications and conflicts) and can cover more use cases.
• Nevertheless, a two-way sync may optionally be transformed into a corresponding read-time join (for use-cases where the duplication is an issue).

=> ?

1. Define smple-schema syntax for write-time syncs?
2. Wire up collection2 to use the peerdb code to implement write-time syncs and some other package to implement two-way-sync compatible read-time joins?

[mitar]

I still don't understand what is two-way sync. And also, it is question where you are looking here. At query time, or at subscription time. So in publish you do not really care about update queries. You just want to publish documents which client needs. And then you can reconstruct data on the client based on that information. PeerDB already provides that for the client side. So when you have references, referenced subdocuments become instances of those documents, with all methods and stuff like that.

[testbird]

Sorry Mitar, two-way sync I think you call "circular reference" (changing author X name in Person.documents will change all author fields in his Post.documents AND changing the author information in one Post.document will change the author name in his Person.document, and all his other Post.documents subsequently.)

I understand write-time synced (embedded) fields don't require any special publish magic, and (as they are embedded) you can get them all with a single database query.

Read-time joins require carefully dealing with publish to get the related documents to the client, and multiple database queries for mongo server publishing queries as well as client minimongo queries.

Further, if collection2 is going to automate the publishing of relations, this may only be possible in a generic way that publishes all related document fields for all users (the same amount of data then with peerdb, what you had thought to be a limitiation of peerdb actually).
Eric if you are still leaning towards beginning with the read-time joins for collection2, maybe consider using https://github.com/peerlibrary/meteor-middleware (stackable-publish-hooks) in the publishing, to allow for finer-grained app-specific customization?

[testbird]

I currently don't see an advantage for collection2 to implement publish features, if they have no advantage over using peerdb's write-time syncing.

If a more efficent publishing for read-time joins would require custom app-specific publishing anyway, I'd like to suggest the following:

Let collection2 use peerdb write-time sync logic to sync related fields, but add a special switch to the simple-schema definition for two-way sync relations. That switch then simply allows to disabe the syncing for this relation, meaning collection2 then assumes that proper app-specifc publish functions are in place and only stores the bare relation (id field) without embedding and syncing any referenced fields. And autoform then also knows from this switch that it needs to fetch the relations separately. (PS: This way collection2 would also allow the app developer to use any "publish with relations like" package he prefers.)

[mitar]

Sorry Mitar, two-way sync I think you call "circular reference" (changing author X name in Person.documents will change all author fields in his Post.documents AND changing the author information in one Post.document will change the author name in his Person.document, and all his other Post.documents subsequently.)

No, PeerDB does not work like this. Circular reference means that you can have for example a tree of comments, where you specify that field parent of document Comment is pointing to another (or even same) document Comment. In fact it is quite complicated to provide and API to define such circular references, because of how JavaScript works. I am not sure how you would be able to define something like this in collection2.

About your use case. How PeerDB operates is that you should not be changing fields in subdocuments automatically. You always change top-level documents and then they are synced into subdocuments. I have seen no practical reason (I even believe it would decrease code readability) to allow to change subdocuments, not to mention that you would then have to observe all subdocuments everywhere for possible changes. This is really not good. There are already many observes opened.

But in PeerDB it is easy to modify things, because each subdocument becomes its own JavaScript document. So post instanceof Post == true and post.author instanceof Person == true. So it is easy to add methods to Person document to modify it in place, if somebody would like such things. I prefer having clear separation between update queries and data. But one can easy define something like:

class Person
  setName: (name) =>
    @constructor.documents.update @_id, $set: name: name

And then call post.author.setName('foo').

I understand write-time synced (embedded) fields don't require any special publish magic, and (as they are embedded) you can get them all with a single database query.

Even more. You can make indexes on them and use them in queries. For example, imagine that you would like to find all posts belonging to an username. If you have username in the subdocument in all posts, then you can simply query over that. So it becomes easy to construct complicated MongoDB queries and they are computed in Mongo, not on the server side (MongoDB client).

Eric if you are still leaning towards beginning with the read-time joins for collection2

BTW, maybe you can already use collection2 with PeerDB. Try something like:

class Person
  @Meta
     name: 'Person'

Person.Meta.collection.attachSchema Schemas.Person

[mitar]

I think that calling then Person.documents.insert() and other queries should go through collection2 validation.

[PiTopRyan]

Hi mitar, I am trying to have Collection2 validate the data going into PeerDB. I tried your suggestion (but using Javascript instead) and I'm getting this error:

Error: Not all documents migrated to the latest schema version (1.0.0): 5sNEeobHuvMCcsgz6

I am trying quite a simple schema:

var Schemas = {};

Schemas.Person = new SimpleSchema({
    username: {
        type: String
    },
    display_name: {
        type: String
    }
});

Person.Meta.collection.attachSchema(Schemas.Person);

and the insert code is:

Person.documents.insert({
        username: 'ryan',
        display_name: 'ryan'
)};

I realise that this may be more of a question related to Collection2 than PeerDB, but I thought I'd run it by you in case it was obvious for you to see what's going wrong.

There are two issues I see:

1. I don't actually see any reason for the data not to be valid, it seems to me that the insert should be valid data.
2. Even though it isn't liking the data (for whatever reason), it is still being input into the database i.e. Collection2 doesn't seem to be rejecting the insert, the data is passing through and then it is throwing the error and crashing the system.

Any help would be much appreciated.

[mitar]

This error is from PeerDB migrations feature, not Collection2. It still gets inserted. Just migrations complain. Does the document in the database has _schema field? This is added by the PeerDB. Maybe Collection2 is removing it?

[testbird]

What if you add a field named _schema to the collection2 schema?

[mitar]

Maybe add it as an optional string field.

[testbird]

Another question would be the referencing peerdoc fields. Add them as object fields to collection2?

[PiTopRyan]

Ok, I now have this as the Collection2 Schema in /lib/schemas/person.js:

var Schemas = {};

Schemas.Person = new SimpleSchema({
    username: {
        type: String
    },
    display_name: {
        type: String
    },
    _schema: {
        type: String,
        optional: true
    }
});

Person.Meta.collection.attachSchema(Schemas.Person);

and this as the insert in /server/fixtures.js:

Person.documents.insert({
    username: "rjdunwoody91",
    display_name: "Ryan"
    }, function(error, result) {
        console.log(error);
});

This works fine, the item is inserted and no errors are shown. I originally forgot to add the "optional: true" field and that didn't work, so definitely required.

Thanks for your help!

[PiTopRyan]

Hi again,

I am running into this error:

Exception in template helper: TypeError: Cannot read property 'points_required' of undefined

<points_required> is a field in a peerdb collection called Level. The data is actually showing on the page, so it must be loading after the initial attempt to retrieve the info. I have actually used the debugger and it seems that this is the case. The data is published like this in the /server directory:

Meteor.publish("levelData", function () {
    return Level.documents.find();
});

And it is subscribed to in the Iron:Router Router.route method for that page. Like I said, the data is showing on the page so it's not a critical error, I just want to track down what the root cause is so I can avoid any problems in the future.

Any help would be much appreciated.

Kind regards
Ryan

[mitar]

What do you mean? You probably don't check in your template helper if document returned from the client-side copy of data has the document you are searching for. You render templates before all data is available.

[mitar]

I am trying to create some benchmarks and it would be useful to learn about db schemas people use PeerDB for. Care to share?

[lonniev]

Reading @PiTopRyan's Nov 2014 remarks, I infer that he was able to use SimpleSchema with PeerDB to offer at least simple schema input validations while using PeerDB collections. This is a valid inference?

@lonniev is very new to meteor and peerdb...