First .plan entry on Github. I've maintained a .plan file for a few months now but much of the information is private. I've decided to start a public .plan file to keep track of my progress, help stay motivated, and share the progress of my projects (currently, mainly, Slipstream) with the community.
Most of my .plan writing is stream of consciousness. However, when I am working on a particularly nebulous problem, it helps to write about it more coherently so that I can really wrap my head around it. Today is one of those days:
Today I am continuing work to reorganize our database's datascheme so that it is easier to perform bulk operations on. To do that, I needed to rip out our previous Object Document Mapping (ODM) layer (Typesaurus). Typesaurus hasn't provided much benefit and has in fact hidden some functionality of Firestore.
Firestore's default library is not much better however, so I had to write my own ORM, or ODM, or whatever, to make it easier to work with. Right now its called DocumentProxy, following the naming conventions of Firestore (DocumentReference, DocumentSnapshot, etc). It is not currently following the ActiveRecord design pattern, but I think it should eventually.
The feature that spurred this reorg / refactor is the ability to reward players with XP periodically, outside of battle. The challenges that I faced were:
-
The Firestore API does not support batch-updating across multiple subcollections.
You can only do batch updates to subcollections on one parent collection.
-
The Firestore client SDK does not provide a way to store data changes in memory.
You can make data changes to the DB, but then you can't modify the data mapping. So if you want to reserialize a record after you've changed it, you need to fetch it again from the DB. This is by design, and there are reasons for it, but Typesaurus did this and worked fine. So I'm creating a lighter-weight ODM layer similar to Typesaurus, but does not hide the underlying Firestore SDK data.
The reorg / refactor is working locally as far as I can tell. My biggest concerns are:
-
Data integrity
The Firestore client SDK's set and update methods by default will delete any keys that exist in the DB but aren't included in the payload. In most cases today, I think this is not what I want. But I may have missed or misunderstood the behavior in some parts. So I have to test and make sure that data doesn't randomly go missing.
-
Faults
The Firestore client SDK's default behavior when a transaction fails is to throw an error. In the Slipstream game and auth servers, these errors must be caught and recovered from. But I must make sure that I've caught all of these instances and that the recovery behavior is well defined.
Aside from the data access layer, I also have some concerns for the feature I'm working on (periodic XP rewards)
-
Recovery
Ensure that if an XP grant fails, the game server retries granting that same XP so that the player gets their reward eventually.
-
No "double-spend"
XP received in game should never be rewarded to a player more than once. However, I don't want this to come at the cost of not rewarding players for legitimate XP. So, if it comes to this vs "Recovery", I would favor Recovery.
-
No overlap of reward systems
There is already a system in the game-servers to reward players with XP after a battle ends. This is not done in bulk, but rather as separate transactions done 1-by-1 asynchronously. I must ensure that XP awarded out of battle does not get counted toward the in-battle reward system. And this must also be true if a reward attempt has failed and is pending retry.
Next steps:
- Continue testing data reorganization and integrity.
- Ensure out-of-battle rewards still function as expected
- Continue working on PendingProgressionRewards system
- Refactor PostBattleProgressionRewardsProcessor to work with the new batching API, and ensure it can recover from failures
Noticed an issue where RPCClient was not properly cleaning up its notification subscriptions when disposed. This caused the client to request all static data N times, where N was the number of times the user re-authenticated per session. It didn't seem to have any side effects, but its resolved.
Back to testing the integrity post-reorg. My plan is to deploy this and migrate data tonight, commandeer Eric and Alfredo to help do regression testing, then continue work on out-of-battle XP once the dust has settled.
I'm going to take a break from regression testing and focus on shoring up the new usage of Firestore client SDK. Will be testing:
- When set/update/add/delete methods throw, errors are caught elegantly and game server continues functioning
- Make sure Firestore client SDK has built in retries when a failure happens. This was happening before (as evidenced by my logs), but I'm not sure it was provided by Typesaurus or the underlying SDK. Hoping it's built into the SDK
Now how do I turn on those internal logs again...
--
Here's what I found: It's possible that the DB client has a built-in retry w/ debounce. The functions client does not. Previously, I manually added a debounce for every function call. I've now refactored ServiceFunctionClient to include the debounce by default, and hidden the internals of that in ServiceFunctionClientInternal. ServiceFunctionClient will be used for adding / remove / interacting with functions in the future. All internals are abstracted away.
Also, it looks like these are the services that get called from the client:
- checkIn (apps/game/src/App.ts)
- gameSessionUpdate (apps/game/src/game-lift/GameLiftBootstrapper.ts)
- grantCrewmatePostBattleWinnings (apps/game/src/battle/battle-results/PostBattleProgressionRewardsProcessor.ts)
- grantCrewmatePostCampaignWinnings (apps/game/src/campaigns/campaign-results/PostCampaignProgressionRewardsProcessor.ts)
6pm - Weekly Demos meeting:
TODO: ShipTechLevelChanged Notif should include latest gem economy changes so that in-ship gem display counters can decrement when tech is upgraded.
9:37pm
Regression test looks good. Good enough for deploy + data migration. Merging this work into main and deploying
- Deploy new code - Success
- Run migrations dry run (
npm run migrations:production:migrate:dry) - Success set:782 deleted:782 - Run migrations dry production (
npm run migrations:production:migrate:commit) - Success? set:782 deleted:782 - Redeploy auth and dev1 game servers
Everything seems to have gone fine. However, I've just learned that Retool cannot do "complex" queries like finding a record by a reference's value. So you can't do like, SELECT * FROM crew WHERE user = "users/abc123".
TODO: Redeploy to Gamelift
Notice a problem in the prod Auth server. Apparently Firestore uses Timestamp objects instead of Date objects. And elsewhere in the auth code I saw POJSOs that looked like { "_seconds": 999, "_nanoseconds": 999 }. Not sure in which situation these timestamps are numbers, string, Dates, Timestamps, or generic objects, but I wrote a helper to convert them. I should really narrow that down to avoid data corruption
Been quietly working on tests for the BackgroundProgressionRewardManager. I've got some basic tests there that don't test the worst edge cases, but give me reasonable confidence in it. Now I'm writing tests for the Firebase functions side of things - getPartialForCrewmateXpGrant now, and later serviceGrantMultipleCrewmatesXpHandler
Randomly thought of a way to express my thoughts on styling for comments:
- Single-line comments should not go past the 100 character column
- Multi-line comments should not go past the 80 character column
Examples:
80 char column \ / 100 char column
| |
# INCORRECT: | |
# This is a line that is longer than 80 chars but the resulting new line looks| |
# weird | |
| |
# Correct: | |
# This is a line that is longer than 80 chars but the resulting new line looks weird |
| |
# INCORRECT | |
# Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla euismod, nisl eget aliquam ultricies, nunc nisl ultricies nunc, quis aliquam ni
| |
# INCORRECT | |
# Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla euismod, nisl eget aliquam ultricies
| |
# Correct | |
# Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla euismod, nisl| |
# eget aliquam ultricies | |
| |
# INCORRECT | |
# Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla euismod, nisl eget aliquam |
# ultricies, nunc nisl ultricies nunc, quis aliquam ni | |
| |
# Correct | |
# Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla euismod, nisl| |
# eget aliquam ultricies, nunc nisl ultricies nunc, quis aliquam ni | |
This is such a stupid thing to be thinking about. I've just always felt like some comments (and code) are okay to go past the agreed upon column limit if the resulting line break actually makes the comment / code less readable. A few dangling words on a line of their own are distracting imo
Back to work.
3:24pm
Tests done. Now to test manually. But there's a 3:30 planning meeting coming up first
Planning meeting notes:
- Apple SignIn is the next big ticket item for me
Back to manually testing. So far I discovered one bug where XP rewards are successfully granted, but the game server fails to parse the response correctly. In that case, the rewards stay in the processing queue and get granted once again. This isn't an issue if the function server + game server agree on the successful response data's shape, but in the odd event that they don't, I want to handle this case.
11:10pm
Background XP rewards are officially working end-to-end locally. The client receives a notif any time XP goes up, and animated the hud accordingly. As well as renders level changes and the skill points available UI when a level up occurs. One bug - The dialog where one would spend skill points doesn't actually show any as available when it loads. How this is possible, I do not know...
Finished Background XP rewards end-to-end. Merged, deployed. Eric tested a bit, no major bugs. Today I will do a bit more cleanup, and do more testing around downtime / network issues. Then I would like to refactor PostBattleProgressionRewardsProcessor to work in batches. Oh, and I broke cheatGrantCrewmateXpHandler, or at least the way the client uses it. Will fix that.
This morning I made a diagram to help communicate the game's front-end architecture a bit more. 1. Because Alfredo will do some of the work for simulating a game-server locally (more on that later), and 2. Because in one of his PR's I noticed some concern-crossing. I was able to easily spot the concern-crossing and needed to explain how others would be able to do it. Hence this diagram:
Back to work.
1:45pm
TODO: Investigate speeding up CloudDBClient#GetDocById. Could append .json to URL to let DB do JSON conversion for me (so client doesn't have to)
Currently creating some automation for my test suite. Automatic user and service-account signin. Using a new tool, Insomniac - a replacement for Postman. Pretty good so far.
Using Insomniac, I set up an Environment (Slipstream Local) that automatically logs in as a user or service-account. I created to demo endpoints - serviceGetCrewByUserId and cheatGrantCrewmateXp. For former signs in as the service-account. The latter signs in as whatever user is in the environment field twitchUserName. Very cool
Back to fixing cheatGrantCrewmateXpHandler - I fixed that locally. It was a client-side issue. I should be able to connect to dev1 now and try again without needing to deploy new code to firebase prod or dev1.
Yup, that worked. Committing.
Demos meeting. Group testing background XP rewards. Looks good except there's a good amount of floating point precision error. Would be great to get that fixed but Eric's not too worried.
Found a bug where skill points that were available go missing (on the client). Database seems correct but the client is overwriting the value with something that's cached or something.
Also the score multiplier is wrong. It should be .05 of the score for the particular stat. The multiplier is BattleRewardsConstants.BattleScoreXpRatio. Fixed
--
Yesterday I started working on Apple SignIn. Managed to get a development build to work with Apple SignIn entitlement, added the open-source Unity Apple SignIn framework to the game, and was able to generate an Apple JWT. Now I've got to deliver that JWT to the auth server at a yet-to-be created endpoint. Create a generic account (with randomly generated name I guess)
Before continuing work on Apple SignIn today, I looked into upgrading our server stack to Node18. Firebase is dropping support for Node16 (Released on Oct 20, 2020. LTS version Apr 20, 2021), but Amazon Linux 2 on GameLift only supports Node16. It looks like Amazon Linux 2023 supports Node18 (Released Apr 20, 2022. Via the dnf package manager). So I believe I'll be able to upgrade Firebase and GameLift to Node 18. I guess Node 20 (Released Apr 19, 2023) is too much to ask for?
Back to Apple SignIn.
Apple SignIn is an absolute mess. I'm saving notes in another markdown file.
My understanding of Apple SignIn has improved. I've managed to get it working for new users. A barebones user gets created for them on initial login (just like Twitch auth). I've scoured the codebase to replace all usage of TwitchID and TwitchUsername where appropriate (Captains still require Twitch auth).
So next up on my agenda is:
- Implement refresh tokens for Apple SignIn (in progress)
- Implement Quick login before attempting a fresh signin
- Refactors. Client: OAuthSigninController should be renamed to AuthUIController, Twitch logic should be broken out into a separate module, same for AppleSignIn. AuthUIController should react to events from those child objects. Server: Similarly, Apple and Twitch should be broken out into modules if their concepts are similar enough (I think they are).
- Regression testing, specifically for Twitch / Twitch auth / expiration
We have a Dev stream coming up in about 5 hours. Eric will be showing off some of the new stuff in game.
Even without the dev stream, I think this will be 2 more days of work. I'll make some headway on items 1 and 2 today. But refactoring and regression testing will most likely be longer. It would be pretty awesome if I were done tomorrow!
One point of frustration that I've been battling with - my 2019 Macbook is becoming painfully slow. Specifically when I have 4 VSCode projects open (Client, Game Server, Auth Server, Firebase), Unity, XCode, iOS Simulator. We don't really have the budget for the laptop I'd like, and opting for a mid-tier upgrade seems like a waste. But if it impedes development enough, it might be worth it to upgrade.
Wishlist item:
MacBook Pro M3 Max
16 inch screen
16-core CPU
40-core GPU
48GB Unified Memory
1TB SSD Storage
Last refreshed October 2023
$4000
That reminds me - good time to back up my system. This laptop could blow up any day now.
Apple Sign In is merged into main and was deployed to dev1 / prod. I've tested the basic flow, it works fine. I just realized though that there can now potentially be two users with the same email address in the database. I'm wondering if that's okay or not. We don't use the email address as a unique identifier in any way, so maybe it's okay.
Time for our weekly demos meeting.
I spent the week working on small client bugs. Switching back to Auth to handle merging Twitch + Apple sign in data.
The idea is to favor Twitch auth information, but keep keychain data for both Auth methods so that the user can henceforth choose which auth method they want to use. Once a user signs in with Twitch, they inherit the Twitch DisplayName, Profile Pic, and Email Address. We'll use email address as an identifier to determine if its the same user.
Taking a break on the Apple Sign In account merging thing. I had trouble focusing on it last week, for personal reasons. This week I'm going to focus on App Store approvals and launch stuff (trailer, media, etc). Today, I'm going to get Game Lift up and running with the latest code, hopefully on the latest version of Linux. I am going to test upgrading to Amazon Linux 2023 and Node 18.
I looked into using Amazon Linux 2023. Ran into a problem: Amazon Linux 2023 requires GameLift Server SDK 5.0.0. We're using a bootleg version of the Server SDK that was ported to Node, and I believe it was targetting Server SDK version 3.2.1 (according to this line of code: https://github.com/j1mmie/GameLift-Nodejs-ServerSDK/blob/0c6d64a98d68bab65927d0b03a7237e631b7ba8a/src/Server/GameLiftServerAPI.ts#L14). To add to the confusion, we've been using Server SDK 4.0.2 in our "production" servers during alpha testing, and that's worked fine. So one question is, was the change from 3.2.1 to 4.0.2 so subtle that it didn't break much? (It broke some stuff, I think. That's why I forked dplusic's branch above). Maybe the change from 4.0.2 to 5.0.0 is equally subtle? I know Amazon GameLift now supports "GameLift Anywhere", which our Server SDK does not understand. But it might be possible to easily modify our bootleg fork to support 5.0.0 features.
Before I do any porting, I need to get production servers online for our launch. I'm going to switch back to Amazon Linux 2 and see if I can get Node 18 running on that. Seems unlikely though, in which case we'll have to switch back to Node 16. Another painful realization: Amazon Linux 2 End of Life is 2025-06-30 :( So we have < 1.5 years to come up with a long term solution. Agones + Kubernetes looking really compelling right about now. Supports node out of the box: https://agones.dev/site/docs/guides/client-sdks/nodejs/
Node 16 is not ideal because Firebase is phasing it out. Ideally all of the server projects use the same version of Node - I'd prefer not to support Node 16 for GameLift and Node 18 for Firebase. ugh.
4:30pm
Good news, I found a version of Amazon Linux 2 that is used by Elastic Beanstalk that supports Node 18.19.0. It was a bit of a hack to find it see discussion here, and I don't know if Amazon will be publically supporting these builds outside of Beanstalk. But it works, and providing a binary for Amazon Linux 2 is already my workflow. So... I guess let's use it? What's the worst that could happen.
I've tested Firebase, Auth, and Game in Node 18. Things are looking good. I'm now going to make sure the GameLift deploy scripts are in order, and attempt a deploy.
6:19pm
Production servers went live without much trouble. The game client is running into an error when trying to deserialize GameSessions - likely introduced when I got rid of Typesaurus. Two problems: 1. The Firebase functions getMySessions and getSessions are not serializing documents to the expected format. 2. GameSessionModel is using Dates, not Timestamps.
11:35pm
Fixed the API issues (both client and server have been updated to support eachother).
Tomorrow I need to:
- Simulate a long response time for "startGameSession" - UI should behave appropriately
- Add support for feature flags
- Add a boolean feature flag that allows anyone to launch a ship
- Add a string feature flag that tells the client which version of the API its using
I've added feature flags to the DB. Right now the client pulls them after signin. Also, they're not cached in any way. So my tasks are:
- Move feature flag retrieval to the title screen. This way, we can determine if the game is in maintenance mode and if the client has the right API version quickly.
- Implement the maintenance mode feature flag
- Implement the API version feature flag
- Add caching (possibly using https://firebase.google.com/docs/firestore/bundles, https://extensions.dev/extensions/firebase/firestore-bundle-builder)
2:13pm
Took a brief detour to enable automated DB backups in Firebase. These are my findings:
--
Two backup schedules have been created:
- Nightly, retained for 7 days
- Weekly on Monday, retained for 14 weeks
The schedules are retrievable via the command line:
➜ git:(main) ✗ gcloud alpha firestore backups schedules list --database='(default)'
---
createTime: '2024-01-25T22:08:30.917453Z'
name: projects/PROJECT_NAME/databases/(default)/backupSchedules/64379432-9965-4cd7-b4d1-25079051e7d6
retention: 8467200s
updateTime: '2024-01-25T22:08:30.917453Z'
weeklyRecurrence:
day: MONDAY
---
createTime: '2024-01-25T22:01:27.982275Z'
dailyRecurrence: {}
name: projects/PROJECT_NAME/databases/(default)/backupSchedules/4bdc6324-afbb-4d03-899e-921a03ee1d5b
retention: 604800s
updateTime: '2024-01-25T22:01:27.982275Z'
Backups can be listed with the following command, but none exist yet:
➜ git:(main) ✗ gcloud alpha firestore backups list
Listed 0 items.
You can restore backups to a different DB for debugging purposes. For example:
gcloud alpha firestore databases restore \
--source-backup=projects/PROJECT_NAME/locations/LOCATION/backups/BACKUP_ID
--destination-database='MY_RESTORATION_DATABASE'
^ If MY_RESTORATION_DATABASE already exists (I believe)
More info on how backups work here:
https://cloud.google.com/firestore/docs/backups
--
Creating a caching layer for static data that is in our database. I'm using Google Cloud Storage for this. So far PlatformConfig is working. TODO:
- Update StaticData Svc on the frontend to use this service
- Create a StaticDataManager that can be used by Functions, Auth, and Game that caches this data in memory
- Create a good workflow for publishing changes
I'm currently dealing with some extended Jury Duty so work has been a bit slow.
I'm working on converting the StaticData Svc to use these new cached JSON files. I realized that cache busting might still be a problem using Google Cloud Storage. So I'm going to look into Firebase Remote Config to see if I can store a version number for the JSON bundles. The idea is:
- Client checks Remote Config for JSON bundle version #
- Client determine whether it needs to download new static data
- Yes: Download
- No: Assume downloaded
When we want to publish new JSON bundles, we have a script that performs these operations:
- Get Remote Config for latest version number
- Increment version number
- Generate bundles
- Save in some folder like /json-bundles/{version}/{collection}.json
- Save the new Remote Config
The hope is that Remote Config is not as expensive as a Firestore request, and it the cache will be busted when the config is changed. Then we don't have to worry about caching policies with the particular bundle version - the version number will change and be guaranteed to be a new file.
Confirmed: Remote Config is free.
--
Hit a snag - Remote Config has some limitations. For one, there is throttling (per user, 5 reads / hour). Secondly, Remote Config data expires every 90 days. So instead, I'll publish the JSON bundle version number to the platformConfig collection. We'll incur an additional read DB read every so often from our users. But it saves us the dozens of reads per user we're doing currently and reduces dependencies / complexity.
--
Static data caching layer is going well. The client fetches platformConfig to determine the latest static data version. It then fetches a copy of the static data from Firebase Storage. It saves that copy to the disk so that it may pull from the disk if the version number hasn't changed.
Next up is to get the servers to use static data instead of live data. This is important so that changes to the static data before they're published don't accidentally get used by the server side.
I'm working on a static data manager for the auth and game servers. One potential issue is that the auth server doesn't know when new data has been published. We'll either need to manually restart it, or build that into the publishing process. Alternatively, we could refresh the static data every hour or so. But that kind of obfuscates the issue and could potentially lead to more confusion. I think we'll build it into our manual deploy plan to restart auth when necessary.
Learning: I discovered that the Firestore client DOES have built in retry capability in case a document read fails.
Learning: The Firebase Storage client SDK does not retry, at least in the event of a "not found" error. Might have to add my own exponential backoff to that.
6:36pm
I've updated the server StaticDataManager to pull all static data at startup. Next up, I should test to ensure that SectorCompletionLogic is still functioning as expected. The Captain should get the appropriate Economy rewards after completing a sector.
Otherwise, static data isn't used elsewhere much. I think I'll need to add PlatformConfigManager and StaticDataManager to the functions project, so that functions can used cached data as well. That means I'll need to move PlatformConfigManager and StaticDataManager to @slipstream/core. I should also make it so that it doesn't fetch static data until it's requested. I'll also need to think about caching static data in a Google Function, and busting that cache when new data is published. All of this is too much to think about - maybe I just punt it. Ugh
I've done the above. StaticDataManager and PlatformConfigManager are now in @slipstream/core. I've created a new class for functions/lambdas - LazyStaticCollection, which manages downloading cached data and storing in memory for the duration of the container's life. I've converted over all usage of static data in functions to lazy static collections.
Next I must do some manual testing and write more tests.
Features to test:
- GrantCrewmateXp (Post-battle XP grants)
- GrantMultipleCrewmatesXp (Out-of-battle XP grants)
- UnlockCrewmate
- UpgradeCrewmateStat
- Sector Cleared rewards
THEN:
-
Make clients refresh platformConfig on restart, and react to changes in
maintenanceModeandcurrentApiVersionaccordingly. ApparentlystaticDataBundleVersionalready triggers new static data bundle downloads. -
Fix multi-gem pickup bug
TODO: Research @nx/js:tsc generateExportsField
12:09pm
generateExportsField doesn't do anything
I reorganized the Nx project to have fewer dependencies (using nx's implicitDependencies feature). Looks like we shaved off a lot for functions deploys. Maybe that means faster deploys + warmup times?? The dependency graph, visualized, is MUCH cleaner. I'm just concerned that I've broken something that's not obvious.
4:33pm
Added Maintenance Mode and API Versioning to both client and servers, including Functions. Need a way for Admins to punch through the versioning. ALso need to make sure that services are automatically punched through.
11:12pm
Explore using Environment variables to clear platformConfig and staticLazyCollection caches, in functions. https://stackoverflow.com/questions/75839352/is-there-a-way-to-restart-running-instances-of-firebase-functions-gcloud-functi
^ Will look into this further later, since it's entirely server side and not a blocker for launch.
I will end tonight with looking into a feature bug: "1462 - Run fails on jumping to outpost with last fuel cell"
TODO: Figure out why nx-firebase insists on including test-helpers folder despite it being excluded in function tsconfig.app.json. This would remove dependencies @jest/globals and jest-mock.
Launch day is tomorrow. Here's the plan:
- Scale up EC2 Auth server from something like t2.small to m5.large
- Set Max Size of GameLift production fleet
Tutorial notes:
- Slug fight would be more interesting
- Repair shields or repair and recharge shields?
Dealing with an iOS crash in production. The user was able to send over an .ips file which contains a stacktrace. However, the information is unsymbolized:
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000001, 0x0000000107cd8144
Triggered by Thread: 0
Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 UnityFramework 0x107cd8144 0x107cc0000 + 98628
1 UnityFramework 0x10820f0dc 0x107cc0000 + 5566684
2 UnityFramework 0x108efec20 0x107cc0000 + 19131424
3 UnityFramework 0x108efea80 0x107cc0000 + 19131008
4 UnityFramework 0x108eff2f8 0x107cc0000 + 19133176
5 UnityFramework 0x108eff288 0x107cc0000 + 19133064
6 UnityFramework 0x108f5141c 0x107cc0000 + 19469340
7 UnityFramework 0x108452ebc 0x107cc0000 + 7941820
8 UnityFramework 0x1084c19dc 0x107cc0000 + 8395228
9 UnityFramework 0x1083780e8 0x107cc0000 + 7045352
10 UnityFramework 0x108378128 0x107cc0000 + 7045416
11 UnityFramework 0x108378384 0x107cc0000 + 7046020
12 UnityFramework 0x10895d2f0 0x107cc0000 + 13226736
13 UnityFramework 0x107cd44fc 0x107cc0000 + 83196
14 QuartzCore 0x1a4d3d010 CA::Display::DisplayLinkItem::dispatch_(CA::SignPost::Interval<(CA::SignPost::CAEventCode)835322056>&) + 48
15 QuartzCore 0x1a4d401f8 CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 864
16 QuartzCore 0x1a4d3fd04 CA::Display::DisplayLink::callback(_CADisplayTimer*, unsigned long long, unsigned long long, unsigned long long, bool, void*) + 844
17 QuartzCore 0x1a4dbefa8 CA::Display::DisplayLink::dispatch_deferred_display_links(unsigned int) + 348
18 UIKitCore 0x1a59d9a4c _UIUpdateSequenceRun + 84
19 UIKitCore 0x1a59d913c schedulerStepScheduledMainSection + 144
20 UIKitCore 0x1a59d91f8 runloopSourceCallback + 92
21 CoreFoundation 0x1a37390ac __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 28
22 CoreFoundation 0x1a3738328 __CFRunLoopDoSource0 + 176
23 CoreFoundation 0x1a3736adc __CFRunLoopDoSources0 + 244
24 CoreFoundation 0x1a3735818 __CFRunLoopRun + 828
25 CoreFoundation 0x1a37353f8 CFRunLoopRunSpecific + 608
26 GraphicsServices 0x1e6cc34f8 GSEventRunModal + 164
27 UIKitCore 0x1a5b5b8a0 -[UIApplication _run] + 888
28 UIKitCore 0x1a5b5aedc UIApplicationMain + 340
29 UnityFramework 0x107cd4158 0x107cc0000 + 82264
30 Slipstream 0x1020dbccc 0x1020d4000 + 31948
31 dyld 0x1c648adcc start + 2240
I was able to symbolize some of these memory markers by using Apple's atos command line tool. First, I needed to download the .dSYM file for this particular build. Unity Cloud Build provides those.
Then, I had to specify specific memory markers, as well as their binaries' loading points, to convert them into readable text. There's more info on that process here: https://developer.apple.com/documentation/xcode/adding-identifiable-symbol-names-to-a-crash-report#Symbolicate-the-crash-report-with-the-command-line
I followed the command line approach because XCode is a complicated nightmare.
Here are the commands I ran:
atos -arch arm64 -o symbols.dsym/UnityFramework.framework.dSYM/Contents/Resources/DWARF/UnityFramework -l 0x107cc0000 0x107cd8144 0x10820f0dc 0x108efec20 0x108efea80 0x108eff2f8 0x108eff288 0x108f5141c 0x108452ebc 0x1084c19dc 0x1083780e8 0x108378128 0x108378384 0x10895d2f0 0x107cd44fc
atos -arch arm64 -o symbols.dsym/UnityFramework.framework.dSYM/Contents/Resources/DWARF/UnityFramework -l 0x107cc0000 0x107cd4158
atos -arch arm64 -o symbols.dsym/Slipstream.app.dSYM/Contents/Resources/DWARF/Slipstream -l 0x1020dbccc 0x1020d4000
And here is the new stack trace that I cobbled together from the above output:
Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 UnityFramework 0x107cd8144 CrashedCheckBelowForHintsWhy (in UnityFramework) (CrashReporter.mm:116)
1 UnityFramework 0x10820f0dc UnhandledExceptionHandler_CUSTOM_iOSNativeUnhandledExceptionHandler(ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeStringPtrOpaque*, ScriptingBackendNativeStringPtrOpaque*) (in UnityFramework) (CoreBindings.gen.cpp:0)
2 UnityFramework 0x108efec20 il2cpp::vm::Runtime::InvokeWithThrow(MethodInfo const*, void*, void**) (in UnityFramework) (Runtime.cpp:0)
3 UnityFramework 0x108efea80 il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (in UnityFramework) (Runtime.cpp:574)
4 UnityFramework 0x108eff2f8 il2cpp::vm::Runtime::CallUnhandledExceptionDelegate(Il2CppDomain*, Il2CppDelegate*, Il2CppException*) (in UnityFramework) (Runtime.cpp:866)
5 UnityFramework 0x108eff288 il2cpp::vm::Runtime::UnhandledException(Il2CppException*) (in UnityFramework) (Runtime.cpp:700)
6 UnityFramework 0x108f5141c ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool) (.cold.1) (in UnityFramework) (ScriptingInvocation.cpp:313)
7 UnityFramework 0x108452ebc ScriptingInvocation::ScriptingInvocation(ScriptingMethodPtr) (in UnityFramework) (ScriptingInvocation.cpp:29)
8 UnityFramework 0x1084c19dc Scripting::UnityEngine::UnitySynchronizationContextProxy::ExecuteTasks(ScriptingExceptionPtr*) (in UnityFramework) (CoreScriptingClasses.cpp:2475)
9 UnityFramework 0x1083780e8 ExecutePlayerLoop(NativePlayerLoopSystem*) (in UnityFramework) (PlayerLoop.cpp:0)
10 UnityFramework 0x108378128 ExecutePlayerLoop(NativePlayerLoopSystem*) (in UnityFramework) (PlayerLoop.cpp:407)
11 UnityFramework 0x108378384 PlayerLoop() (in UnityFramework) (PlayerLoop.cpp:514)
12 UnityFramework 0x10895d2f0 UnityPlayerLoopImpl(bool) (in UnityFramework) (LibEntryPoint.mm:342)
13 UnityFramework 0x107cd44fc -[UnityAppController(Rendering) repaint] (in UnityFramework) (UnityAppController+Rendering.mm:57)
14 QuartzCore 0x1a4d3d010 CA::Display::DisplayLinkItem::dispatch_(CA::SignPost::Interval<(CA::SignPost::CAEventCode)835322056>&) + 48
15 QuartzCore 0x1a4d401f8 CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 864
16 QuartzCore 0x1a4d3fd04 CA::Display::DisplayLink::callback(_CADisplayTimer*, unsigned long long, unsigned long long, unsigned long long, bool, void*) + 844
17 QuartzCore 0x1a4dbefa8 CA::Display::DisplayLink::dispatch_deferred_display_links(unsigned int) + 348
18 UIKitCore 0x1a59d9a4c _UIUpdateSequenceRun + 84
19 UIKitCore 0x1a59d913c schedulerStepScheduledMainSection + 144
20 UIKitCore 0x1a59d91f8 runloopSourceCallback + 92
21 CoreFoundation 0x1a37390ac __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 28
22 CoreFoundation 0x1a3738328 __CFRunLoopDoSource0 + 176
23 CoreFoundation 0x1a3736adc __CFRunLoopDoSources0 + 244
24 CoreFoundation 0x1a3735818 __CFRunLoopRun + 828
25 CoreFoundation 0x1a37353f8 CFRunLoopRunSpecific + 608
26 GraphicsServices 0x1e6cc34f8 GSEventRunModal + 164
27 UIKitCore 0x1a5b5b8a0 -[UIApplication _run] + 888
28 UIKitCore 0x1a5b5aedc UIApplicationMain + 340
29 UnityFramework 0x107cd4158 -[UnityFramework runUIApplicationMainWithArgc:argv:] (in UnityFramework) (main.mm:96)
30 Slipstream 0x1020dbccc main (in Slipstream) (main.mm:28)
31 dyld 0x1c648adcc start + 2240
Unfortunately it doesn't tell me much, other than the fact that the crash might be occuring from our own scripts (ScriptingInvocation::Invoke). But that's pretty helpful, it means we might be able to fix it.
Eric also discovered that the app crashes at about the same time when there is no Internet connection available. I loaded the game in Unity to see what the behavior is when there's no Internet connection. There are two errors in the logs. The first is Rudderstack, the second is our own error while attempting to download PlatformConfig. Eric and I are both betting that the crash has something to do with Rudderstack, aka the bain of my existence.
I'm going to build the app locally to see if it crashed when there's no Internet connection on my iPhone.
Good news, it crashes. I have a repro case. Now to decipher the logs.
Memory optimization findings:
Use of await Awaiters.NextFrame should be avoided. GameFrame library should be removed. It apparently allocates memory on every use. Some systems, like MusicManager and RpcClient use it every frame
MusicManager was also allocating memory every frame when determining the play-time of a particular sound clip. That’s been fixed
RpcClient was using , which allocated memory every frame. I’ve forked that and made a version that does not do this. *My changes could have some unexpected implications. Also, my client would not be compatible with WebGL builds.
FPS display was allocating memory every frame when construction the strings to display. I’ve created a version that uses Images and Textures, not Textfields + Fonts, to display the FPS number, without allocating a string.
The Restarter script / scene may have been de-allocating scenes in an order in which the Persistent scene was referencing things from the Gameplay scene, even though the Gameplay scene was unloaded. The offenders I find were all audio related - MusicManager, SoundEffectsPlayer, and AudioListenerManager. I’ve refactored that to shut them down gracefully before unloading scenes. This could avoid some errors
I removed the Rudderstack SDK, which likely had plenty of memory issues. But in particular, I’ve bade a stream-based REST client that hopefully allocated less memory, as it streams JSON as it’s being constructed, directly to a socket, to the network. The system should have a buffer where this stream is processed instead of our application creating / destroying buffers for every tracking event.
Another learning: Image.useSpriteMesh consumes CPU and allocates memory each frame. It should be avoided, but I don’t have a better solution for displaying Sprites in the UI right now. When possible though, Sprites should be used instead of Images. So in UI, we basically have to use Images. But in the Gameplay scene, we should consider converting our various TV screen displays to be sprite-based rather than canvas-based. There will likely be other benefits to doing this in terms of performance (but not maintainability)
Learning: Deploy a single Firebase function:
firebase deploy --only functions:functions:myFunctionName --config firebase.json
Deploy a few functions (but not all)
firebase deploy --only functions:functions:myFunctionName1,functions:functions:myFunctionName2,functions:functions:myFunctionName3 --config firebase.json
Skins are getting close to launch.
- Skinned assets are collected in a "Skin" scriptable object.
- Skin AnimatorControllers can be generated via Animancer's "Remap Sprite Animation" tool
- Crew skin definitions are in the
crewSkinscollection - Unlocked skins are located in the
unlockedCrewSkinscollection - Seed data is in the file
apps/migrations/src/fixtures/crewSkins-v0.0.13.json - Client code is backward compatible with old DB (currently production)
- Client code is forward compatible with new DB (currently in local)
Skins are ready for testing in dev1.
I was a bit confused about how to test and roll out this release for two reasons:
This is obviously not ideal, but I've been putting off creating proper dev and staging Firebase + GameLift instances since I'm the only backend developer. But aside from that, it's also not ideal because skins required creating new static data in the prod DB. That static data isn't "published" just because it's in the DB. There's a publishing endpoint, which will cause all clients to download the newly published data. I didn't want to make clients download this data unnecessarily just because I don't have my dev environment shit together. So the solution was to upload the static data JSON file manually to our Google Cloud Storage bucket where those live, in the old static data bundle version's folder. I've essentially patched the static data bundle with new info. Only clients that request this new info will actually get it. The old clients (let's call them v1.01) won't know about it, so they won't request it.
Those are; the current version of release clients (v1.01), and the upcoming client that supports skins (v1.03). The solution was to add a "legacy client" system to the server code. I've hard-coded legacy client checksums into the game server code. When the server detects a client using these checksums, it marks it as isLegacy. I've also added the option to set specific Notifications in the game server to be legacyCompatible or not (the default is true). The CrewmateSkinChange notification is marked as legacyCompatible === false. What this means is that legacy v1.01 clients will not receive CrewmateSkinChanged notifications.
I've deployed this server code to dev1, and it now supports both v1.01 and v1.03 clients at the same time. A player on the v1.03 client can change their crewmate's skin, and the v1.01 will just ignore it. It'll render their archetype's default skin as it always has done.
Once we confirm this code is fit for production, I'll deploy this to GameLift. At that point, all players' clients will be marked as "legacy." While that's technically not true, it prepares us for the case when an Apple reviewer is ready to review v1.03. They'll be able to launch or connect to a ship, and even change their skins, without affecting v1.0.1 clients.
After the review is approved, we'll release v1.03 and (hopefully) have no more legacy clients.
At that point, we can remove legacy support from the server code. The specific legacy checksums and feature gates, not the framework itself. The framework we can reuse for future deploys.
Moving forward I think this is the proper strategy for supporting somewhat zero-downtime deploys. Update server code to accept live release clients and upcoming release clients at the same time, block legacy clients from receiving new features, test in dev1, deploy to prod, start app review process, release apps to prod, remove legacy blocks from server code, repeat.
This is just game-server code. We'll need similar behavior for Functions, Auth, and data. It's a lot to think about, but there are some general best pracitces:
- Adding database collections is fine
- Adding fields to data is fine
- Appending fields to the end of a MessagePacked message is fine
- Adding new Functions endpoints is fine
Anything other than the above probably needs some extra consideration. So the best strategy is to just design with those 4 rules in mind. Instead of renaming a field, add a new one (even if it's redundant for one or two releases). Instead of reorganizing a new MessagePacked object, make a v2 of it.
Discovered we might be duplicating our sprites in atlases.
https://thegamedev.guru/unity-addressables/spriteatlas-save-memory/
https://forum.unity.com/threads/addressables-and-sprite-atlas.1006754/
https://forum.unity.com/threads/sprite-atlas-assets-duplication-in-bundles.1207783/#post-9153119
[x] TODO: Test offline mode on server side Update: Seems good
[x] TODO: Move scene UI assets to separate Addressables
When a scene references an item in an addressable directly, the entire contents of the addressable are copied into the scene.
For example:
- Stars
- Icons
- UI Panels + Components
[ ] TODO: Route Functions egress through VPC so that we have a static outbound IP address. This will allow us to limit usage of our Steam Publisher Web API Key to a specific range.
More info: https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
The reason I want to do this is because security around the Steam Publisher Web API Keys is pretty lax. The API key is transmitted via GET parameters, there's no permissioning (it's a super user), no expiration date, etc. What a piece of shit.
Correction: API key can be transmitted via the non-standard x-webapi-key (all lowercase) header. Still a PoS.
For the morning:
[x] TODO: Re-sync static JSON (migrations + static JSON files) [x] TODO: Test Alfredo's Outfits branch and start code review
Review of Alfredo's branch:
- Might need loading animation when clicking on different outfits. Brief moment where character is invisible. Could be longer on other clients.
- Contrast not high enough on "Equipped" text. Suggestion: 1px drop shadow
- There's sometimes a weird jitter on the crewmate "running" preview animation when switching outfits, and als when switching crewmates.
- Alignment of datakey icon on Checkout dialog is not consistent with other uses
- Outfits briefly display old crewmate's outfits when changing crewmates. Would be better to clear them out immediately when the crewmate changes, then gradually load the new ones in
It seems that Sprite Atlases get duplicated into any scene that references an image that is included in an atlas. We need to restructure our addressables and atlases to reduce duplication (build size, memory footprint) while still remaining performant.
Ideas for addressable groups:
-
Ships
-
Audio
-
Crew
-
GameplayTextures
-
GameplayPrefabs
-
CaptainUITextures
-
CaptainUIPrefabs
-
GeneralUITextures
-
GeneralUIPrefabs
-
Dialogs
With the above arrangement, we'll also need separate Sprite Atlases:
GameplayAtlas CaptainUIAtlas GeneralUIAtlas
Might need some special group/atlas for textures that are shared between the game and UI.
Maybe new folder structure: Move textures out of Prefabs directory and into Textures directory. Atlases will correspond to entire directories to keep it simple. Possible same for Addressables
Woke up to a nice surprise. The build size for Windows went from 221mb to a piddly 82mb! I think we can do better, but that's great for now. There were only a few minor visual regressions (most likely due to some assets being loaded asynchronously now when they weren't before). Performance doesn't seem to have changed.
[x] TODO: Test new servers with old clients (v1.01 and v1.02)
The new builds (1.03 and 1.041) went out to users without a hitch yesterday. We did a dev stream an hour after hitting the deploy butons. I'm getting the hang of switching from totally-focused, hyper-technical, hyper-coordinated mode to casual dev-stream mode. The trick is to be incredibly overprepared in the days before. I've never worked this efficiently and intentionally before; I'm not sure if it's from wisdom that comes with age, or just because I really care about the product. Either way, I hope we as a team keep up this level of excellence, for our players', company's, and our own sakes.
1.03 is the iOS build that has a few minor UI glitches that we didn't catch in time for the App Store review process. Actually, we probably had plenty of time because the App Store review was delayed this weekend (perhaps due to Holi?). I think we can swap out builds while an app is pending review without increasing the review turnaround time, but we were too cautious to try. Maybe we'll try next time for a low-impact release.
1.041 has a few more fixes that were caught last minute. NPE avoidance, UI improvements, etc. It went live for Windows, Mac, and Android.
All this version-talk has made me realize that Eric and I have been versioning the software differently. I've been using Semantic Versioning internally, while he has been using a decimal system. I've asked him to change that. We'll see if it's possible. Since he had previously named the build 1.041, that is the equivalent of 1.41.0 in semantic versioning. We'll need to do something like 1.50.0 to get back on track with semantic versioning.
Up next: New currencies and Steam purchase-flow.
In preparation for the new currency systems, I'd like to normalize the schema of our records. This is a good time to do that, since we built the current-generation server to be backward compatible with previous-generation clients. To faciliate backward compatibility, we had some sparse / loose data schemas. Now most users should be on current-generation clients, so we can get rid of anything that the previous-generation depended on.
Without doing too much auditing here's a list of data we can normalize:
- users
- inventory
- credits (add, non-null)
- reSpecTokens (add, non-null)
- crew
- skin (remove)
- equipedSkin (make non-null)
Another thing - we should remove support for the previous-generation client. Currently this is done through protocol checksums. There is an exception programmed into the server to allowed previous-generation checksums. We can just remove that and redeploy to GameLift.
After that, I would like to finalize a procedure for converting Steam IAPs into Slipstream currency. I've already done the following:
- Set up a Steam Inventory + storefront
- Integrated a C# Steam SDK with our game to retrieve store items
- Created a proof-of-concept for purchasing items from Steam (both for sandbox and production data)
- Created and Function that:
- Authenticated the user
- Scans their inventory for consumable items
- Consumes those items
- Saves the equivalent currency / quantity consumed into the
Next step is to create a function that does steps 3 and 4 in one press. Then Alfredo can use that in his UI.
I'd also like to add more thorough checks and confirmations to ensure we don't:
- Accidentally consume a Steam credits-pack without saving those credits to the Slipstream DB
- Leave an exploit where hackers can convert credits-packs to credits without consuming an inventory item
I've moved the admin CLI tool (I wrote a while back) into the mono-repo. This allows the admin CLI to make use of the many library helpers I've written for our servers. I've also updated it to work for both local and prod environments. Example:
npm run admin-cli PROD get-user j1mmie
npm run admin-cli local find-orphaned-docs unlockedCrewSkins user
# Find all users with 16 crewUnlockKeys available to spend.
npm run admin-cli PROD query users inventory.crewUnlockKeys.available 16
Upcoming deploys:
[x] Auth server - New User creation code that includes new credit and respec token balances [x] unlockCrewmate Function - New non-nullable EquippedSkin
Follow-up migrations:
[x] Backfill User inventories to include credits and respec tokens [x] Backfill Crew to have EquippedSkin even if they haven't equipped one
Gamelift deploy:
[ ] Deploy new code that removes old client checksum from legacy list
Cheat notes for Firestore set vs update. Take from https://stackoverflow.com/a/46600599/430070
-
setwithoutmergewill overwrite a document or create it if it doesn't exist yet -
setwithmergewill update fields in the document or create it if it doesn't exists -
updatewill update fields but will fail if the document doesn't exist -
createwill create the document but fail if the document already exists
There's also a difference in the kind of data you provide to set and update.
For set you always have to provide document-shaped data:
set(
{a: {b: {c: true}}},
{merge: true}
)
With update you can also use field paths for updating nested values:
update({
'a.b.c': true
})
TODO: publishStaticData service endpoint needs to be redeployed so that it can be used by the slipadmin cli. After that, the retool static data dashboard will need to be updated so that it supports the data field (which is required by Google Cloud callables)
publishStaticData needs to be more fool proof. If new "static" collections are added, but publishStaticData is not deployed with that knowledge, and then publishStaticData is executed, it will create a new static data bundle without those static columns. It will then direct clients to load the incomplete bundle, probably causing the game to be unable to load.
Fortunately I avoided this because the static data in prod's database happens to not be changed. So the commit hash was the same (and that commit hash had a hacked static data bundle)
I've slowed down with work a bit lately. Got some bad news recently - my grandmother passed away somewhat suddenly. Weird situation. Had to do a lot to help my family out during this time. I flew home and it took a lot of my time and energy. I'm pretty much just getting back into the swing of things.
Steam microtransactions:
- I need some sort of Steam Svc that tracks if:
- Steam has been initialized
- ItemDefs have been loaded
- A transaction has finalized
- Then I need the CreditShop to hook into this somehow (ideally event driven) and do the appropriate actions when a purchase is complete
- Run
ConsumeInventoryItems UserWalletLogic.SyncUserWallet- Show a success / failure dialog
- Run
SteamUser.OnMicroTxnAuthorizationResponse += OnPurchaseFinished;
Working on Daily Bonus Run implemention. Mostly finished, actually. Couple loose ends for tomorrow (today) morning:
- Test many more times
- Merge Daily-Bonus-Run-Squashed into main
- Deploy new functions to prod:
- getDailyBonusReport
- serviceStartDailyBonusCampaign
- cheatResetDailyBonusCampaign
- Deploy new server code to dev1
Some time before the new client is released, we must:
- Test connecting old clients to dev1. Captains on old clients won't see a Daily Bonus button, but nothing should break. Nothing should break for crew.
-
Update server code to allow old client signatures (whatever the current-gen build has)Protocol signatures have actually not changed. That's because they don't take into account the structure of individual messages, and we've only changed the structure of messages. This could be a problem down the line, so I've created a task to add more granular checksum calculation. But for this deploy, it seems like the server is backward compatible with the last two client releases. - Deploy new fleet with next-gen server code. Must be backward compatible with current-gen clients.