Add Firestore indexes to local emulator

[nicholaschiang]

[REQUIRED] Environment info

firebase-tools:
7.15.0

Platform:
Ubuntu 18.04.4

[REQUIRED] Test case

I'm trying to test our Firestore indexes, but they seem to always be working (regardless of the indexing settings included in firestore.indexes.json). Below is the test case in question:

const FILTERS = {
    'grade': ['==', TUTOR.grade],
    'gender': ['==', TUTOR.gender],
    'location': ['==', TUTOR.location],
    'type': ['==', TUTOR.type],
    'payments.type': ['==', TUTOR.payments.type],
    'config.showProfile': ['==', true],
};
const SORTERS = [
    'avgRating',
    'numRatings',
];

describe('Tutorbook\'s Database Indexing', () => {
    const db = authedApp({
        uid: PUPIL.uid,
        email: PUPIL.email,
        access: PUPIL.access,
    });
    const filterCombos = combinations(Object.keys(FILTERS));
    return Promise.all(filterCombos.map(filters => it('lets users filter ' +
        'profiles by ' + filters.join(', '), () => {
            var query = db.collection('users')
                .where('access', 'array-contains-any', PUPIL.access);
            filters.map(filter => query =
                query.where(filter, FILTERS[filter][0], FILTERS[filter][1]));
            return Promise.all(SORTERS.map(sorter =>
                firebase.assertSucceeds(query.orderBy(sorter).get())));
        })));
});

[REQUIRED] Steps to reproduce

Just try testing any type of composite or collection group query that would (normally) require an index. It will work on the emulator suite regardless of the existence of the required index (but it won't work in production without the index).

[REQUIRED] Expected behavior

These tests should fail unless there are Firestore indexes (specified in firestore.indexes.json) that support the composite indexes tested.

[REQUIRED] Actual behavior

All the tests succeed (even though they shouldn't).

[nicholaschiang]

Just FYI the ultimate goal here is to test each possible combination of the different filters (as defined in FILTERS and SORTERS) available in our app's search screen.

[samtstern]

@nicholaschiang thanks for the feedback! Right now you're correct that the Firestore emulator will execute any query even if the index has not been "created".

@yuchenshi do you know if we will be able to support stricter indexing in the Firestore emulator?

[yuchenshi]

Strict indexing in the Firestore Emulator is not a planned feature (yet) but I'll definite discuss this with the team.

On the other hand, we do plan to support dumping all indexes automatically created (say during the test), which you can compare against firestore.indexes.json. I believe this will address OP's use case as well.

[nicholaschiang]

@yuchenshi That would be a useful (although not exactly what I'm asking for) feature; those "tests" would ensure that my firestore.indexes.json supported the "tested" queries. Just make sure that feature accounts for Firestore's ability to merge indexes and doesn't just create a composite index for each query type.

[nicholaschiang]

Oh, and @samtstern you should have this specification (that the Firestore emulator will execute any query even if there isn't an index to support it) added to the differences between emulator and production webpage.

[manwithsteelnerves]

Its unfortunate that firestore emulator doesn't support index rules.
@yuchenshi I see it useful feature considering the merge will be handled. Thats like a master piece!

What are the chances to consider this feature?

[yuchenshi]

We're considering both features of dumping and tracking index rules and we'd appreciate input on the use cases.

As a developer, how do you plan on using emulators to prototype, test, and gain more confidence about your index rules? Shoot us some workflows just like what @nicholaschiang did and we'll figure out the best way to make that happen.

[manwithsteelnerves]

Great!
Our use-case is pretty similar to the OP mentioned.
We have a REST Api's which has optional query parameters. To check if each request has required indexes already in place, we use jest framework to fire the requests with all possible combinations. If it throws errors that indexes are missing and we can create the required indexes.

So for 4 optional query params with multiple values allowed in each param, we need to test around more than 200 possibilities. And creating indexes for multiple combinations manually is error prone.

If there is a provision for firestore emulator to create the required indexes list, we can just run the all possible cases generated through code and use the indexes created from firestore emulator. This in-fact saves lot of time.

[manwithsteelnerves]

@yuchenshi Can you please let us know if anything planned related to this feature?

[sk-]

@yuchenshi we just got bitten by this.

We deployed a quick fix, which actually broke later in production due to the missing index. Note that that part of the code was tested, so it'd be great if the emulator had the same behavior as production.

[yuchenshi]

@manwithsteelnerves While we don't share timelines and cannot commit to any plans, but I do want to let you know that we hear you and real pain points (including the latest one shared by @sk- ) helps a lot driving our internal discussion. It's a technically challenging and involved feature for sure to get the index semantics match production, but that doesn't mean we're not considering / doing it.

[manwithsteelnerves]

As I see it may take more time, I ended up creating 139 manual indexes for my project for all possible test cases. Hopefully, there will be a better solution through emulator workflow!

[jpangburn]

Out of the two options discussed "match production" and "dumping all indexes automatically created [during some set of tests]" , if we could only have one then I prefer the second one. For me, the point of writing a bunch of automated tests is that so later when I add some feature or fix some bug, I don't break a bunch of other things. With that in mind, if the emulator could dump the list of indexes that I need that aren't already in my firestore.indexes.json- that seems pretty helpful.

If I ran a test set and nothing came out then I'd know that some tests I added (to verify some bug) weren't increasing the indexes I need. Or if I prototype some feature, it would be nice if the emulator made it clear "hey buddy, you're going to need another 100 indexes to support this" by dumping a list of those hundred indexes and I can realize I had a stupid idea before getting too far into it.

Matching production by giving me an error is kinda useful, but just giving the list of additional indexes needed is going to help me make progress faster I think. And @nicholaschiang 's comment is critical "Just make sure that feature accounts for Firestore's ability to merge indexes and doesn't just create a composite index for each query type." Might even be nice if it also said "hey, you've got this set of indexes [X,Y,Z] in your firestore.indexes.json that were not necessary during this test run".

[merlinnot]

I'd be in favor of the former (match production) because the latter works only if you have a single application talking to the database - what if you have dozens services in separate repositories talking to the same database?

[jpangburn]

the latter works only if you have a single application talking to the database

Wouldn't you just reset the emulator, run whatever tests you have on your dozens of services in separate repositories, and then end up with the final set of indexes that you need? I guess I don't see how it's any different than a single application. Just as a single application is using one Firebase project, presumably your dozens are services are as well?

I guess in the case were you had multiple firebase projects being served by one emulator, you'd want your indexes grouped by project. But otherwise I don't see a difference- though I could easily be missing something :-)

[merlinnot]

I was referring to

if the emulator could dump the list of indexes that I need that aren't already in my firestore.indexes.json

I assumed you'd like to run your tests and then deploy the indexes in the CI. That wouldn't work in my case, where I have dozens of separate repositories (services) and indexes are stored in a separate repository, shared by all of the services.

So, "if we could only have one", for my use case I'd prefer to load my manually crafted indexes (where you also have optimizations for de-duplication of queries) in all of the services and run test against it. If something fails, I'd add these indexes to my repository with indexes.

Hope that explains my reasoning better 🙂

[manwithsteelnerves]

I assumed you'd like to run your tests and then deploy the indexes in the CI. That wouldn't work in my case, where I have dozens of separate repositories (services) and indexes are stored in a separate repository, shared by all of the services.

To understand better,

If you have single firestore db used by multiple services, isn't it possible to run all the diff service tests and update the dumped index file from emulator?
Or
Run tests per project, merge each dumped indexe data, and push to your main firestore?

[merlinnot]

Not really, no. We have dozens of services interacting with the same database. Each service is developed in a separate repository, by independent teams, deployed individually. We also follow a true to the bone continuous delivery process, so we have multiple (tens/hundreds) of production deployments daily.

With that in mind, what you're asking is if for these hundreds deployments daily we can run tests of all our services, instead of just one service, for which the change is made. Let's explore it with an example:

Let's assume 50 services, 300 deployments daily (total across all services), 10 minutes of CI time to run tests for each service. That consumes 300*10=3000 build minutes daily. If we'd run that for all services each time, it's 50 times more, so the cost increase is very significant.

On top of that, what's even worse, some services already run tests on 32-CPU machines (fortunately the emulator supports multithreading). We obviously can't afford a test run that takes 5010 minutes, so we'd have to run it in parallel. But then we'd need a VM that has 5032=1600 CPU cores, and such a thing doesn't exist. We could of course run Firestore emulator on a separate VM and execute all tests against it, but the emulator is not very well optimized and it consumes very significant resources for tests of each service, e.g. on a 32 core machine we usually dedicate ~16 cores to the emulator.

For your second point, we could theoretically do that, but that sounds a little complex. I imagine that for each CI run we'd have to pull current Firestore indexes from a central repository, run tests for a service and then validate that all indexes exist. But then we'd be implementing ourselves what we defined here as the first option ("match production"): for the emulator itself to fail when indexes don't exist. And as it was previously mentioned in this very thread, it's not as easy as it might look like, specifically due to features such as index merging.

So for me it drills down to implementing "match production" in Firestore or implementing it myself, so I'd obviously prefer it to be implemented in the Emulator.

I also understand your point, I see usefulness of it for small applications, but I believe that with the growth of any application "match production" will eventually become a preference for everyone.

[manwithsteelnerves]

Ok, thats really big scale. Its great to see you are able to manage those many projects where I'm struggling to fit firestore in one because of its inconsistent latencies and limitations.

So, will dumping the missing indexes compared to production will definitely solve all problems right? It's great to see an index merged dump though. And these indexes will be updated to production manually.

This will work for us too as initially nothing will be on production for us and all dumped indexes will be shown by the emulator which covers our case too.

[jpangburn]

TL;DR - I agree with the argument for "match production", it would help very large projects a lot. Would still be nice if there was also a way to generate an optimized list of indexes after running a suite of tests, both for new projects and for large long running projects that may have built up a list of non-optimized indexes.

Loooong answer:
Yes, you clearly have a well oiled machine going there. The only reason I'm commenting is since they asked us for use cases, I'm trying to help hash it out so they'll have a clear picture (one way or the other) to implement this thing we all need :-)

I was coming at this from the view of having a project with a firestore.indexes.json file that you update with whatever indexes you need, and do a firestore deploy and it uploads your indexes, your rules, your functions, etc. But you aren't doing that @merlinnot , are you? Your devs are writing their own projects, making changes, the CI is running tests for that project, and if the tests pass then it deploys that project's artifacts to production Firestore. But the CI is not updating any rules or indexes, is it? This is where the pain is now- the tests can pass but then in production there's a missing index so there's a failure so then someone goes into the Firestore console (or you have your own client that assists you with your flow) and adds that index. This is why you said:

I imagine that for each CI run we'd have to pull current Firestore indexes from a central repository, run tests for a service and then validate that all indexes exist

So I think it helps your use case to be clear that when you say "match production", you do NOT mean that some central repository's firestore.indexes.json and firestore.rules is loaded to the emulator. You mean that the emulator downloads the indexes and rules from production, and proceeds to "match production" as fully as possible. You need it to behave this way because due to index merging, it's possible that you'll waste indexes by only loading a firestore.indexes.json for a single project (unless they all have completely separated collections and there's no collectionGroups across them). Is that correct?

That would be pretty useful in your large case because then the CI would fail, and it would fail specifically on the test that was lacking the hypothetical "new needed index". That would go right back to the developer that wrote it without needing to test the entire set of projects.

For new developing projects, it's nice to have the other way where you just get a list of needed indexes after running all your tests, update your firestore.indexes.json file, and then feel comfortable to deploy to production. Especially if you could get an optimized list that takes advantage of merging. But as @merlinnot points out, with the benefit of wisdom from hard knocks I'm sure, is that as projects grow they will begin to prefer "match production". I can't disagree with that. The pluses are large in the true CI case with many projects- you get failure right where it belongs. These large projects are likely where Firebase should optimize (at least for financial benefit). The downside is you can end up with a very non-optimized set of indexes because you added each as they were needed.

Hope this helps Firebase devs! Thanks @merlinnot for your very well thought out and detailed description of your use case.

[samtstern]

@jpangburn @merlinnot I just want to say this conversation is great and we're listening to both of you!

[merlinnot]

@jpangburn By "match production" I mean loading indexes manually myself, not downloading it automatically. I'd prefer that as we strive to keep our development environments fully offline, both for performance and ability to develop under any conditions - aircrafts or simply when one's internet breaks down. We'd probably end up fetching it from a central place and caching, but definitely loading ourselves to the emulator, the same as it's currently done with Firestore Rules. That also allows people to test changes to indexes, we'd definitely have tests in the "Firestore indexes" repository.

At the same time I fully agree that auto-generating indexes would be very helpful for small/medium size applications, would speed up development a lot.

So I guess we can agree that both features would be very useful. Maybe they could be somehow combined, e.g. when some indexes are missing, print helpful messages:

• "A new index is required for this query, add to your indexes.json file."
• "Extending an existing index is required for this query, add to your index."

We could also have some flag that would result in generating these indexes automatically instead of failing queries.

[seanburlington]

I just caught this issue in staging

It's great that the issues list here is so open with clear responses from the dev team - that really saves me time because I now know exactly what this issue is

It would be great if it was documented here https://firebase.google.com/docs/emulator-suite/connect_firestore

[samtstern]

@seanburlington that's a good idea I'll get it documented

[chrisjacobs91]

Chipping in on this. Am I right in thinking there is a delay in the emulator building composite indexes? (perhaps as they can't be specified in advance due to the emulator not supporting indexes)

When I change data and then perform a query that would usually require a composite index, I get out of date data back from Firestore.

[samtstern]

@chrisjacobs91 that should not be the case and if you can reproduce that please file a separate issue. The emulator, just like real Firestore, should always be strongly consistent.

[manwithsteelnerves]

@samtstern Any updates on this feature? Is it scheduled or not anytime soon?

[cupidchan]

Absolutely agree with you @zanona and having a staging will help with no question. However, the inconsistency between the local development and Staging is still here because of this issue. We just moved the problem backward to another less risky environment.

[go-sean-go]

One other point: Is it not still true that many Firebase settings are ONLY accessible via web console? If I'm wrong, please let me know; I know there have been some advances in the config file lately.

For me, the last time I reviewed the Firebase ecosystem, it was very unfriendly to a Staging environment specifically because it requires a separate project for each environment, and MANY settings are not configurable by config file (i.e. requires a web console to manually configure settings, and cannot easily see the diff between environments). (Maybe this has changed; if not, even one person working alone is bound to make mistakes when trying to keep the two environments in-sync--let alone a whole team with a complex release scheme.)

That is why the Emulator/local dev has an outsized importance in Firebase dev, in my experience--and what makes this issue so severe.

Editorializing a bit: I imagine most Firebase users are on small and/or fast-moving teams that don't need or want to worry about Nth-level details in the environment. Syncing anonymized data between environments is not easy, nor is it cheap.

There is of course lots of gray area between a team willing to take some risks vs. a structured team with multiple environments and processes, but I can only imagine the majority of Emulator users are operating without a structured Staging environment (this is 100% speculation).

Worth noting that Firebase does provide this document (which I hadn't seen before today; not sure how old it is): https://firebase.google.com/docs/projects/dev-workflows/overview-environments

For me it is unreasonably difficult in today's Firebase's setup for the reasons mentioned above, but I do hope achieving that as a "one-click solution" is their North star. Depends on their target market, I guess--but, if I cared to implement all that stuff myself, write my own scripts to manage it, etc...well, I probably wouldn't be using Firebase, right? I'd be dangerously close to just rolling my own env to contain costs.

Again - just adding this context here as I hope it helps the Firebase team evaluate the severity of this problem. For me it feels like a broken promise without at least a log warning, or similar. Hope that helps!

[mcfarljw]

I'm going to be the first to chime in for getting this supported in 2023. I'm think I'm running in to the same issue as I need a compound query for two properties deletedAt and uploadedAt for an endpoint that incrementally fetches data that may or may not be deleted.

const ref = firestore
  .collection('users')
  .doc(user.id)
  .collection('items')
  .where('uploadedAt', '>=', query.offset)

if (query.excludeDeleted)
  ref.where('deletedAt', '==', 0)

But I can't test this is working when developing because the emulator completely ignores ref.where('deletedAt', '==', 0) when declared with no error or warning and just returns everything after the uploadedAt offset.

[dfink91]

@mcfarljw

I'm going to be the first to chime in for getting this supported in 2023. I'm think I'm running in to the same issue as I need a compound query for two properties deletedAt and uploadedAt for an endpoint that incrementally fetches data that may or may not be deleted.

const ref = firestore
  .collection('users')
  .doc(user.id)
  .collection('items')
  .where('uploadedAt', '>=', query.offset)

if (query.excludeDeleted)
  ref.where('deletedAt', '==', 0)

But I can't test this is working when developing because the emulator completely ignores ref.where('deletedAt', '==', 0) when declared with no error or warning and just returns everything after the uploadedAt offset.

as far as I remember, you need to store the result of ref.where(....) somewhere as .where returns a new immutable query. Use that new reference to get the records.

That being said, your problem is not the same as the one discussed here. hope this helps you.

[mcfarljw]

as far as I remember, you need to store the result of ref.where(....) somewhere as .where returns a new immutable query. Use that new reference to get the records.

You are right, it looks like it returns the correct results when using the emulator if I chain them together. I always get this syntax mixed up. Here is my updated example that does query deletedAt correctly:

const ref = firestore
  .collection('users')
  .doc(user.id)
  .collection('items')
  .where('uploadedAt', '>=', query.offset)
  .where('deletedAt', '==', 0)

I'll still chime in with my support because even with the change it doesn't recommend indexes and is disconnected from production which makes using it to run tests limited in reliability in terms of catching serious index related issues which is my real pain point at the moment.

[risalfajar]

On the other hand, we do plan to support dumping all indexes automatically created (say during the test), which you can compare against firestore.indexes.json. I believe this will address OP's use case as well.

Tomorrow will be exactly 3 years after this statement. Is there still no progress? Anything? @yuchenshi @taeold

Currently firestore.indexes.json is just useless.

[yugisu-flux]

I believe you don't even need to implement the actual behavior behind indexes; please, just bring the validation part that looks at every db query and validates it against the indexes file. Faced this issue several times in production since I started using Emulators for development.

[jaredweinfurtner]

On the other hand, we do plan to support dumping all indexes automatically created (say during the test), which you can compare against firestore.indexes.json. I believe this will address OP's use case as well.

Tomorrow will be exactly 3 years after this statement. Is there still no progress? Anything? @yuchenshi @taeold

Currently firestore.indexes.json is just useless.

Not only useless, but deceptive.

[garrett-wombat]

Like @yugisu-flux said. I think its enough to add a warning in the emulator that you've performed a query requiring an index that doesn't exist. Just so the developer will know wheres going to be an issue in production.

[christhompsongoogle]

Hi all, I do understand the sentiment here and we haven't forgotten about this item.

If it's helpful in the short term, after performing a compound query you can get the index dump from the firestore emulator here:
http://127.0.0.1:8080/emulator/v1/projects/{projectId}:indexUsage?database=projects/{projectId}/databases/(default)

Example:

http://127.0.0.1:8080/emulator/v1/projects/demo-something:indexUsage?database=projects/demo-something/databases/(default)

{
  "reports": [{
    "index": {
      "name": "projects/demo-something/databases/(default)/collectionGroups/cities/indexes/_",
      "queryScope": "COLLECTION",
      "fields": [{
        "fieldPath": "state",
        "order": "ASCENDING"
      }, {
        "fieldPath": "population",
        "order": "ASCENDING"
      }, {
        "fieldPath": "__name__",
        "order": "ASCENDING"
      }]
    },
    "numQueries": "1"
  }]
}

Note that the emulator doesn't know about your application's indexes, it simply attempts to create these from a set of theoretically optimal indexes. If your indexes don't match exactly it's still possible that all of your queries are covered by other indexes and will not fail in production.

Edit: The queries need to be executed to be stored at runtime and it must be a query that isn't automatically indexed (like a single-field query with no ordering). See https://firebase.google.com/docs/firestore/query-data/indexing for more info.

Disclaimer: this is unlaunched and the functionality might change in backward-incompatible ways.

[JZubero]

Disclaimer: this is unlaunched and the functionality might change in backward-incompatible ways.

Not sure if I am getting it right. Is this supposed to work already or are you proposing this for an upcoming emulator update?

After trying with several emulator setups (where I know that firestore indexes exist) it keeps returning:

{
}

[dominicbartl]

@JZubero Did you actually run queries against the emulator which require an index? The emulator doesn't read your firestore.indexes.json it records the indexes based on your queries.

Also, make sure that you correctly replaced the project ID in the URL. It's in there twice. I got an {} response when requesting for a non-existing project.

The endpoint works for me with firebase-tools@11.30.0:

{
  "reports": [{
    "index": {
      "name": "projects/myproject-local/databases/(default)/collectionGroups/users/indexes/_",
      "queryScope": "COLLECTION",
      "fields": [{
        "fieldPath": "asd",
        "order": "ASCENDING"
      }, {
        "fieldPath": "test",
        "order": "ASCENDING"
      }, {
        "fieldPath": "__name__",
        "order": "ASCENDING"
      }]
    },
    "numQueries": "3"
  }]
}

[JZubero]

Did you actually run queries against the emulator which require an index?

Did the indexes get recorded at runtime only?

I made sure that projectId is valid.

[dominicbartl]

Did the indexes get recorded at runtime only?

Not quite sure what you mean, but you have to execute queries using the emulator for it to add it to its report. If you don't query anything, the emulator won't show any results and return {}. Start the emulator and your application, make sure you run the queries and then call the URL

[konti-kun]

Thanks for the great feature: index dump!

I tried it, but it seems that this index dump is still logged until it is restarted.
Is there any way to clear it like a REST API?

[christhompsongoogle]

I added some additional info - as dominicbartl noted some queries (like single field queries) do not require an index.

Is there any way to clear it like a REST API?

Currently the index usage is stored until the emulators are restarted, there's no REST API for clearing them out.

[ekalosak]

Is there a sanctioned place to vote on these features? Getting the unpublished auto-indexing emulator feature to a place where it can update the local firebase.index.json on the fly would be Really Cool, for what it's worth.

[mono0926]

@ekalosak

Is there a sanctioned place to vote on these features?

https://firebase.uservoice.com/forums/948424-general/suggestions/46593439-firestore-indexes-in-emulator

[risalfajar]

https://firebase.uservoice.com/forums/948424-general/suggestions/46593439-firestore-indexes-in-emulator

This is the first time I heard of this site, from where did you know it?

[mono0926]

@risalfajar

https://firebase.uservoice.com/forums/948424-general/suggestions/46593439-firestore-indexes-in-emulator

This is the first time I heard of this site, from where did you know it?

Here: https://firebase.google.com/community

Previously, it was private information within the Firebase Alpha Program Google Groups.

[lukepighetti]

This is absolutely critical. We currently have no way to warn developers that they are creating functionality that requires indexes in production. All we expect is a warning in the console just like we experience when developing on a managed Firestore instance.

[jaredweinfurtner]

it is now 2024 and still no solution. @christhompsongoogle, do you have any update?

[yuvalhermelin-fijoya]

@christhompsongoogle We would appreciate your response on this :) We know you and the whole firebase team are working hard and doing lots of amazing things, but a timeline for this feature would be nice! Thanks!

[omervexler-fijoya]

@christhompsongoogle I really ❤️ Google and firebase, would love to receive updates if there are any. Thanks!

[yuvalhermelin-fijoya]

@kroikie pinging you here too :) Thanks!

[christhompsongoogle]

Hi all, unfortunately I'm unable to provide any timeline for this feature. If this is critical to your application, the only way to know for sure that a query is covered is testing the indexes+queries against a production instance of Firestore, in a test project for example.

[aetherknight]

This feature (or rather, this feature but for datastore-mode) is necessary for us to be able to move off of the older datastore-emulator. Switching to the newer firestore emulator is also necessary for us to be able to move to the latest version of NDB (2.3.1).

We rely heavily on unit testing to ensure that all queries work with our indexes -- it is not practical to be able to test every possible query combination and every code path in a deployed environment after every code change. Our code base is large and complex, and many queries are fairly deep in our code base and cannot be directly or easily called. They often have edge cases that are easy to test with unit tests, but are not easily tested through integration testing in a dev or staging environment.