DBD v2.0 format discussion

[Marlamin]

Last month, DBD turned 2 years old. 🎉 It's been quite the ride so far! We're still in the middle of things and will hopefully soon (update: now merged) merge in the pre-wod branch @barncastle has been hard at work on as well as to continue working through the backlog of 6.x/7.x versions we're missing.

Adoption

Adoption still isn't at the point we initially wanted it to be. Why is this? Is it the lack of interest in WoW tool development or is the format too annoying to deal with? Not enough libraries out there to deal with DBD? What can we do to improve this?

Version 2 changes

Throughout this time I'm sure some of you have grown to miss/dislike some things in the current format and with Shadowlands Beta slowly coming to an end changes-wise, I thought this would be a good time to start discussing a possible v2 of the format. Hopefully we can get to a good proposal that we can spend some time on implementing when things calm down in a few months.

A good starting point might be some of the things that were left in the initial format discussion, such as adding enums and flags to the format. I currently deal with these by manually keeping them up to date in the WoW.tools repo, would be cool if we can somehow integrate these into the format if enough people agree. The same goes for flags. No idea how we should handle conditional enums/flags, though.

One thing I'd like us see add (and have discussed previously) is adding a file that combines the tablehash, filedataid and name of DB2s as a way to deal with unnamed DB2s. This idea is based on @MMOSimca's TableHash.cfg, but with the FileDataID added too for if the file is unnamed and to possibly help readers with loading them from CASC.

Also, there are still some open issues such as #40 and #51 that we should tackle here if they need format changes.

Implementation/migration

I'd suggest starting off with a new directory, maybe named "v2" instead of definitions. How to handle DBDv1 going forward is up for debate, but the easiest route when we do switch to primarily v2 will probably be to have some GitHub action run DBDefsConverter which will output v2 -> v1 or something.

Disagree with something? Did I miss something? Post below. Issue will remain open for quite some time until we come to a general consensus like last time.

[barncastle]

My first remark is that I'd like to get a definition for every build we possibly can even if it is just meta dumps before moving on to v2. This will 1. give us a good base for version 2 so we can focus on the spec more than the data and 2. should increase the rate of adoption. This should also be a very achievable considering we have the tools now.

Adoption:
This is quite a hard one to tackle but I'd say the biggest targets that need to be brought across are the server cores themselves. Most tools out there, especially in the modding community, are using Trinity, AzerothCore or (C)Mangos as their main source of reference and rightly so because they are the literal heart of the community. As such most column names, formats and readers are being built in-line with what these projects' implementations. If we could approach these projects and get their input and adoption this would trickle down and increase adoption rates. This also applies to all major/well known projects such as simc and Stoneharry's spell editor.

In regards to the modding community, the biggest issues are favouritism and project longevity. Most modding is happening on WotLK/Vanilla cores as either they are nostalgic favourites and/or their project has spanned several years. At the moment we aren't catering for them however we must do so if we want adoption to increase - ideally we should focus on getting end-of-expansion definitions 100% correct. The unfortunate side-effect is that most modders were raised using abandonware such as MyDBCEditor and WDBX and continue to propagate it to newcomers. As these tools are abandoned we can't target them directly but we could maybe maintain a set of profiles, DBD branded, that these tools support to introduce people into the DBD ecosphere until another tool (like DBMonster or DBClientFiles.Net) that supports DBD comes in to replace them?

In regards to parsers in other languages - this will happen organically once the format is adopted by more people, so I don't feel this is something we need to focus on just yet - something for v3 maybe?

Version 2 changes:

• My biggest OCD gripe at the moment is the lack of standardisation of the layout of the definitions themselves. This is only minor but I'd like to adhere to a standard that orders all definitions by newest to oldest and also uses build ranges as much as possible. These 60+ build lists are really off putting to say the least and merging branches has been a nightmare because everything keeps getting re-ordered. Ideally the automated dump would just increment the final build of the range as we're capturing every build that goes out.

• I'm completely onboard with having an index file for file identification.

• The big topic and focus of v2 (imo) is of course enums and flags. At the moment we're using comments as a quick fix for wow.tools however this isn't ideal. Looking at the original discussion the general consensus was to keep DBD as clean as possible so I'd like to suggest having an additional associated file (even <dbdname>.dat) that stores a range of additional information including this data. How this is formatted is up for debate I'd prefer to have a hex - name pair that we commonly use on the wiki ideally but open for anything else. I'm not sure if Blizz reuse enums or whatnot so versioning might be something we need to look into but we could just reuse the build/build-range/layout spec we've already adopted?

• I'd also like to throw in storing metadata-only data too such as the Flags for WDB3 and the default values for WDB6 common fields and potentially TACT keys (* see the end) since sections aren't always decrypted. The metadata data should really go in main dbd as this is a parsing requirement however TACT keys should probably go into the counterpart data file.

• Something @bloerwald brought up too is adding another keyword for what is (poorly) named secondary indices aka flags & 0x2 - the relationship column for pre-WDC which definitely needs recording!

• For ItemDisplayInfo.db2 wrong cardinality #40: I think we should approach this with symbolic notation as per point 1 here. Namely for the sake of being as true to format as possible and not misleading people with invalid cardinalities when the layouthash doesn't change.

• For Different relationship tables between versions for same columns #51: I think the best choice is to move foreign keys into the definitions themselves like we have for the bitwidths. It does increase the amount of text but it gives much more control over how we handle this data.

Implementation/migration
Personally I'd work across two branches instead of having two folders as this is the point of git. Also, if we went the route of having an additional data counterpart file we could even work on the same repo with minimal fuss.

* I've been thinking for a while that maybe we should create a gist or repo that replicate's the wow.tools tactkey page that doesn't hit the site.

[Shauren]

TrinityCore here

My 2c

Adoption:
I think this is going to be hard when we need to have the format ahead of time in the form of a c++ structure

Naming:
master branch (current client version) has used the renderservice names with some style changes (casing, _lang) since I got my hands on them
3.3.5 has also recently adopted them so it should not be an issue for future projects using it as resource

[Marlamin]

My first remark is that I'd like to get a definition for every build we possibly can even if it is just meta dumps before moving on to v2. This will 1. give us a good base for version 2 so we can focus on the spec more than the data and 2. should increase the rate of adoption. This should also be a very achievable considering we have the tools now.
For 6.0-7.0 we can semi-easily do this by adding more patterns to the dumper. For more versions older than that (though I think pre-wod covers us well there).

Version 2 changes:

• My biggest OCD gripe at the moment is the lack of standardisation of the layout of the definitions themselves. This is only minor but I'd like to adhere to a standard that orders all definitions by newest to oldest and also uses build ranges as much as possible. These 60+ build lists are really off putting to say the least and merging branches has been a nightmare because everything keeps getting re-ordered. Ideally the automated dump would just increment the final build of the range as we're capturing every build that goes out.

I hope to have time to merge the sorting stuff you did as well as update the tools to support sorting (and automatically sort each automatic merge). Update: done.

• The big topic and focus of v2 (imo) is of course enums and flags. At the moment we're using comments as a quick fix for wow.tools however this isn't ideal. Looking at the original discussion the general consensus was to keep DBD as clean as possible so I'd like to suggest having an additional associated file (even <dbdname>.dat) that stores a range of additional information including this data. How this is formatted is up for debate I'd prefer to have a hex - name pair that we commonly use on the wiki ideally but open for anything else. I'm not sure if Blizz reuse enums or whatnot so versioning might be something we need to look into but we could just reuse the build/build-range/layout spec we've already adopted?

• I'd also like to throw in storing metadata-only data too such as the Flags for WDB3 and the default values for WDB6 common fields and potentially TACT keys (* see the end) since sections aren't always decrypted. The metadata data should really go in main dbd as this is a parsing requirement however TACT keys should probably go into the counterpart data file.

Additional metadata file sounds okay to me (maybe we should call it .dbdmeta HEH). Not entirely sure how to deal with versioning, build/build-range/layout could work but we'd have to have the changes for those for v2 pretty cleared out.

• For Different relationship tables between versions for same columns #51: I think the best choice is to move foreign keys into the definitions themselves like we have for the bitwidths. It does increase the amount of text but it gives much more control over how we handle this data.

Sounds good to me.

Implementation/migration
Personally I'd work across two branches instead of having two folders as this is the point of git. Also, if we went the route of having an additional data counterpart file we could even work on the same repo with minimal fuss.

Yeah, that's better.

• I've been thinking for a while that maybe we should create a gist or repo that replicate's the wow.tools tactkey page that doesn't hit the site.

Yeah, it is getting a bit crazy now. 283 new keys were added in the last 5 days alone. Separate repo for this sounds good.
Update: Done. https://github.com/wowdev/TACTKeys

[billy1arm]

Mangos here - although a little late to the party.

I love the approach and agree that this should be as widely supported as possible, i have been working on a suite of DBC/DB2 handling apps which are multiple expansion compatible and I was originally using a format which I devised myself - Which isn't a million miles away from this format.

I'm happy to make the switch to using this (and V2 moving forward). Obviously our primary focus is Classic through to Mop ATM.

[username223]

Adoption still isn't at the point we initially wanted it to be. Why is this? Is it the lack of interest in WoW tool development or is the format too annoying to deal with? Not enough libraries out there to deal with DBD? What can we do to improve this?

I've started writing my own server for fun and wanted to auto generate a DBC library for Rust. I ended up using the xml definitions created by ChaosVex for the Ember project primarily because:

• They were already in an easy to parse format.
• They supported enums/flags which I felt was important in providing a good API for the library.

I actually looked at this project but decided against using it because:

• No enums/flags.
• No easy way to get the information. I'm on Linux so I can't compile and run the Visual Studio based C# project without some major hassle, and I wanted my library generator to be in Rust so I didn't attempt to get the Python lib working.

Both of these are actually IMO relatively easy to solve, although I probably don't have the same depth of knowledge about the subject that you do, so there might be something that nullifies my ideas.

In order to ease the access to the information I would have CI create a json files that are automatically updated with every commit to master. This makes the information available to basically every programming language without needing to parse the custom DBD format. It doesn't even really have to be a good format as long as it's predictable, so you could probably just dump whatever the existing python script creates.

For enums/flags I would suggest just adding to the v1 format in order to get the information out there and get experience with what works and what doesn't before v2.

I suggest a syntax akin to something like this:

ENUMS // Same for flags, just with FLAGS keyword
Race BUILD 1.12.1.5875 // Type name, "BUILD" keyword and build versions like normal
HUMAN 1 // Enumerator name and value, not sure if you want SCREAMING_SNAKE_CASE or PascalCase like for everything else
ORC 2
DWARF 3
NIGHT_ELF 4
// enum ends either on COLUMNS, FLAGS or on line containing BUILD

COLUMNS // Like normal
int ID
int<Map::ID> ContinentID
Race Race // First column is type, second is name like the others. Defaults to int if not enum for BUILD

BUILD 1.12.1.5875 // Like normal
$id$ID<32>
ContinentID<32> // Still normal

Race<32>  // Field name and size. 

I know that race is actually a foreign key into the race table, it's just an example since I can't come up with anything else.

Let me know what you think.

[Marlamin]

With WoW.tools having bit the dust when 10.1+ comes looking I'd like to get that manifest file going this year still so tools like wow.export and other tools will continue to have a way to map DB2s to FDIDs as well as having proper capitalized names without relying on the listfile being updated. Enums and flags (as discussed in the above post) are important to get documented/standardized too but with less haste.

Relevant part of OP:

One thing I'd like us see add (and have discussed previously) is adding a file that combines the tablehash, filedataid and name of DB2s as a way to deal with unnamed DB2s. This idea is based on @MMOSimca's TableHash.cfg, but with the FileDataID added too for if the file is unnamed and to possibly help readers with loading them from CASC.

Simca's current format looks like this:

Achievement: D2EE2CA7
Achievement_Category: 231B414D
AdventureJournal: 8EEF1A97
AdventureMapPOI: 6275EEF7
AlliedRace: A40F98B8

This does not list filedataIDs and we have to keep in mind that DB2 and DBCs for the same table have different FileDataIDs. Continuing on with the above idea gets us to something like this (option 1):

Achievement D2EE2CA7 841607,1260179
Achievement_Category 231B414D 841639,1324299
AdventureJournal 8EEF1A97 1115949

With tablename, tablehash and a comma separated list of FileDataIDs (dbc, db2 and whatever else they do in the future). With .scn files also being added for a currently unknown purpose, we can add another column for those after the table filedataids if they ever become relevant.

This format should also support unnamed DB2s, of which there are currently 5 in 10.0.2. Dumper can't dump these as they obviously don't have definitions (or they'd be named). We already dealt with these previously and named them UNK_.dbd in the definitions at the time.

There's only one problem I can think of with the above format and that is how to deal with "optionals" (such as unnamed DB2s or files with no known FDID if they were removed before FDIDs were added).

Something along the lines of this could work for that, with TABLE optional/defaulting to UNK_<TABLEHASH> and FDID being entirely optional. If tablehashes were to go away that could theoretically be optional too, but that is a worry for later.

Option 2:

TABLE Achievement 
HASH D2EE2CA7 
FDID 841607, 1260179

TABLE Achievement_Category 
HASH 231B414D 
FDID 841639, 1324299

TABLE AdventureJournal 
HASH 8EEF1A97 
FDID 1115949

FDID 3549053 
HASH 6F1A6B5A

FDID 4276966
HASH A5D81C93

FDID 4281310
HASH 94980739

FDID 4281311
HASH 2225AC8D

FDID 4281312
HASH 2E7A049C

This file (both options) would both be automatically updated by the dumper (which knows about the name, hash and filedataid) as well as manually editable for unknown DB2s. As for placement, I'm not entirely sure yet, but maybe in the root directory as "Manifest.dbdm" (DBD manifest) or something along the lines of that. Suggestions there are very welcome.

Also tagging @QartemisT and @ferronn-dev for feedback on this as they've both expressed interest in working on WoWDBDefs in the future.

[ferronn-dev]

Thoughts on using JSON or YAML to make it trivial to parse?

Regarding multiple fdids: can those be explicitly labeled as well?

[QartemisT]

I would +1 for YAML. :D

[Marlamin]

Thoughts on using JSON or YAML to make it trivial to parse?

No specific opinion here, but keep in mind it does introduce dependencies for everyone parsing it vs. keeping it self-contained like DBDefsLib is now. I would prefer JSON in that case due to native support in .NET (and JS obviously) without too much fuss, YAML would require a 3rd-party library which I'm not a fan of.

Regarding multiple fdids: can those be explicitly labeled as well?

Explicitly labeled as in? Edit: Nevermind, you meant separate DBC/DB2 filedataids. Got it!

[Marlamin]

Concept file: manifest.json (GitHub wanted a .txt, it is a .json).

Let me know if there's any comments/feedback.

[Marlamin]

Given no feedback on the above manifest.json format, that one has been pushed to master as well as a manually triggered GitHub action that dumps/merges DBDs as well as the manifests.