Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Redis Search silently fails to Sort when the index schema is too large #4494

Open
claridgicus opened this issue Feb 27, 2024 · 7 comments
Open

[BUG] Redis Search silently fails to Sort when the index schema is too large #4494

claridgicus opened this issue Feb 27, 2024 · 7 comments
Assignees
@raz-mon
Labels
bug stale

Comments

@claridgicus
Copy link

claridgicus commented Feb 27, 2024 •
edited

Describe the bug
I am running a redissearch instance to curate product information for many merchants
Merchants divvy up their products into user-friendly "categories" which typically contain their own unique sort order.
I have loaded in the merchants' products using prefix keys to partition them from each other
I create a "Category" index, which contains useful information for a customer to interact with, price, tags, title etc and also includes a reference to a nested JSON object "sortOrder"

My index looks a bit like this

'$.title': {
	type: SchemaFieldTypes.TEXT,
	SORTABLE: true,
	AS: 'title',
},

'$.tags': {
	type: SchemaFieldTypes.TAG,
	AS: 'tags',
},

"$['sortOrder']['example1']": {
        type: SchemaFieldTypes.NUMERIC,
        SORTABLE: true, 
        AS: 'example1' 
}  << I might have 2-600 of these for a given store

When an index exceeds 250 sort definitions (which if you ask redis to create an index it will successfully create one without error or warning) querying for the index with an appropriate SortBy will result in out-of-order objects.

To Reproduce
Steps to reproduce the behaviour:

  1. Load in a couple of thousand product objects
  2. Define an arbitrary numeric sort for them in a child property
  3. Create an index that has 250 unique sorts
  4. Query for products, with "SortBy" set to be one of your unique sorts

Expected behaviour
This should still work, albeit slowly - or the index should fail to create in the first instance

I can provide access to a redis instance over DM to someone to debug an actual dataset if required.
@raz-mon
Copy link
Collaborator

raz-mon commented Feb 28, 2024 •
edited

Hi @claridgicus, thanks for reaching out!
Which RediSearch version are you using?
Can you please share:

  • Your full index definition (FT.INFO <index-name>).
  • The queries you are dispatching
  • Their corresponding results showing the unordered values (a concise one would be good 😉).

Did you change any config params?
Sure, an rdb/aof file for us to reproduce with would be great.

@claridgicus
Copy link
Author

claridgicus commented Feb 29, 2024 •
edited

Hey @raz-mon

I'm using RedisLabs and I've tried two different nodes
No config changes

Save and BgSave are unavailable in Redislabs, but I can provide you creds if you can provide me a way to send them to you (to my non production node)

My 7.2 Test Node
Redis
7.2.3
Search and query
v 2.8.11

My Main Production Node
Redis
6.2.6
Search and query
v 2.6.15

I have attempted a version with the maxtextfields option - but that doesn't help me either (documentation on that feature could also be slightly improved)

here's my query - nothing too hairy
"idx:cherrie-baby:cp:" "(@vendor:Jellycat) (@implode:[1 2])" "SORTBY" "jellycats" "ASC" "LIMIT" "0" "20"

the outputted sort order of the query on a broken index is

[
  undefined, undefined, undefined,
  undefined, undefined, 89,
  undefined, undefined, 15,
  undefined, undefined, undefined,
  undefined, 153,       undefined,
  108,       undefined, undefined,
  undefined, undefined
]

if I give the undefined entries a number say '999' they will still appear out of order in the results

the output on a non broken index is

[
   1,  4,  5,  6,  7,  8,  9,
  10, 11, 12, 13, 14, 15, 16,
  17, 18, 19, 20, 21, 22
]

here's my big boy "index info"

I've tried removing "-", removing any entry in the sort that contains a number in it, creating a base index without the sort fields and altering to add all my sorts

How I currently have it working, whenever I exceed 250 sorts it breaks, so I'm degrading the feature set of my app to provide the sort functionality to the key groups my client requires - without going back to what I was doing, (one index per sort order - which results in a memory usage many times what I can justify for a client)

{
  indexName: 'idx:cherrie-baby:cp:',
  indexOptions: [],
  indexDefinition: [Object: null prototype] {
    key_type: 'JSON',
    prefixes: [ 'cherrie-baby:p:' ],
    default_score: '1'
  },
  attributes: [
    [Object: null prototype] {
      identifier: '$.title',
      attribute: 'title',
      type: 'TEXT',
      WEIGHT: '1',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.tags',
      attribute: 'tags',
      type: 'TAG',
      SEPARATOR: ''
    },
    [Object: null prototype] {
      identifier: '$.id',
      attribute: 'id',
      type: 'TEXT',
      WEIGHT: '1'
    },
    [Object: null prototype] {
      identifier: '$.created_date',
      attribute: 'created_date',
      type: 'TEXT',
      WEIGHT: '1',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.parents-frank-green',
      attribute: 'parents-frank-green',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycats',
      attribute: 'jellycats',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-dog',
      attribute: 'jellycat-dog',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-dresses',
      attribute: 'minti-dresses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.subo',
      attribute: 'subo',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.acorn-kids',
      attribute: 'acorn-kids',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-sale',
      attribute: 'girl-sale',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.aster-oak',
      attribute: 'aster-oak',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy-boys',
      attribute: 'wilson-frenchy-boys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby',
      attribute: 'huxbaby',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.big-kids',
      attribute: 'big-kids',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.speedy-monkey',
      attribute: 'speedy-monkey',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.olli-ella-strolleys-luggys',
      attribute: 'olli-ella-strolleys-luggys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-dresses',
      attribute: 'rock-your-baby-dresses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-clear',
      attribute: 'connetix-clear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.goldie-ace',
      attribute: 'goldie-ace',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.activity-craft',
      attribute: 'activity-craft',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-pastel',
      attribute: 'connetix-pastel',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.teethers-bibs-dummies',
      attribute: 'teethers-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.summertime',
      attribute: 'summertime',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-main',
      attribute: 'new-arrival-main',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-sale-tops',
      attribute: 'girl-sale-tops',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-shoes',
      attribute: 'baby-shoes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.original-bibs-dummies',
      attribute: 'original-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl-bottoms',
      attribute: 'huxbaby-girl-bottoms',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-girls',
      attribute: 'minti-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.easter',
      attribute: 'easter',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.the-somewhere-co',
      attribute: 'the-somewhere-co',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-boy-decor',
      attribute: 'baby-boy-decor',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrivals-clothing',
      attribute: 'new-arrivals-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.connetix-rainbow',
      attribute: 'connetix-rainbow',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.love-to-dream',
      attribute: 'love-to-dream',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.my-dream-light',
      attribute: 'my-dream-light',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.toy-sale-1',
      attribute: 'toy-sale-1',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ethicool',
      attribute: 'ethicool',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.penny-scallan',
      attribute: 'penny-scallan',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boxing-day',
      attribute: 'boxing-day',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.kinderfeets',
      attribute: 'kinderfeets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.heydoodle',
      attribute: 'heydoodle',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-mythical',
      attribute: 'jellycat-mythical',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.timber-tinkers',
      attribute: 'timber-tinkers',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.de-lux-bibs-dummies',
      attribute: 'de-lux-bibs-dummies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.milky-girls',
      attribute: 'milky-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.magnetic-play',
      attribute: 'magnetic-play',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.toddler-gifts',
      attribute: 'toddler-gifts',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.womens-shoes',
      attribute: 'womens-shoes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.salt-water-originals',
      attribute: 'salt-water-originals',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.construction',
      attribute: 'construction',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-boy',
      attribute: 'new-arrival-boy',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.write-to-me',
      attribute: 'write-to-me',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-girls',
      attribute: 'rock-your-baby-girls',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.alex-ant',
      attribute: 'alex-ant',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti',
      attribute: 'minti',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-amuseable-sports',
      attribute: 'jellycat-amuseable-sports',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-boy-sale-2022',
      attribute: 'baby-boy-sale-2022',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boxing-day-girl',
      attribute: 'boxing-day-girl',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy',
      attribute: 'wilson-frenchy',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ziggy-lou',
      attribute: 'ziggy-lou',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-swimwear',
      attribute: 'girl-swimwear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.boy-hair-accessories-hats',
      attribute: 'boy-hair-accessories-hats',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.new-arrival-boy-clothing',
      attribute: 'new-arrival-boy-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.snuggle-hunny-blankets',
      attribute: 'snuggle-hunny-blankets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-girl-sale-2022',
      attribute: 'baby-girl-sale-2022',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.minti-jumpers-and-jackets',
      attribute: 'minti-jumpers-and-jackets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bebe',
      attribute: 'bebe',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.girl-swimwear-sale',
      attribute: 'girl-swimwear-sale',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.crywolf',
      attribute: 'crywolf',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.jellycat-sea-life',
      attribute: 'jellycat-sea-life',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.all-4-ella',
      attribute: 'all-4-ella',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.for-parents',
      attribute: 'for-parents',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.rock-your-baby-swim',
      attribute: 'rock-your-baby-swim',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.nature',
      attribute: 'nature',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.eos-summer-baby',
      attribute: 'eos-summer-baby',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-socks-tights',
      attribute: 'baby-socks-tights',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.mystery-boxes',
      attribute: 'mystery-boxes',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.crywolf-accessories',
      attribute: 'crywolf-accessories',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.milky',
      attribute: 'milky',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.holdie-folk',
      attribute: 'holdie-folk',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bella-lace',
      attribute: 'bella-lace',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.winter',
      attribute: 'winter',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.frank-green',
      attribute: 'frank-green',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.puzzles',
      attribute: 'puzzles',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.bf40',
      attribute: 'bf40',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.medium-jellycat-bunnies',
      attribute: 'medium-jellycat-bunnies',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.wilson-frenchy-girlg',
      attribute: 'wilson-frenchy-girlg',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-glasses-sunglasses',
      attribute: 'baby-glasses-sunglasses',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.ob-designs',
      attribute: 'ob-designs',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-swimwear',
      attribute: 'huxbaby-swimwear',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.dinkum-doll',
      attribute: 'dinkum-doll',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl',
      attribute: 'huxbaby-girl',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.baby-girl-clothing',
      attribute: 'baby-girl-clothing',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.towels',
      attribute: 'towels',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.puffers',
      attribute: 'puffers',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.babiators',
      attribute: 'babiators',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.christmas-toys',
      attribute: 'christmas-toys',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.olli-ella-baskets',
      attribute: 'olli-ella-baskets',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    [Object: null prototype] {
      identifier: '$.collectionPosition.huxbaby-girl-tops',
      attribute: 'huxbaby-girl-tops',
      type: 'NUMERIC',
      SORTABLE: 'UNF'
    },
    ... 225 more items
  ],
  numDocs: '6949',
  maxDocId: '6949',
  numTerms: '12767',
  numRecords: '179692',
  invertedSzMb: '0.9440984725952148',
  vectorIndexSzMb: '0',
  totalInvertedIndexBlocks: '28787',
  offsetVectorsSzMb: '0.1320819854736328',
  docTableSizeMb: '0.7863950729370117',
  sortableValuesSizeMb: '10.280499458312988',
  keyTableSizeMb: '0.29492950439453125',
  recordsPerDocAvg: '0',
  bytesPerRecordAvg: '25.858684539794922',
  offsetsPerTermAvg: '5.509199142456055',
  offsetBitsPerRecordAvg: '0.7707521915435791',
  hashIndexingFailures: '8',
  indexing: '0',
  percentIndexed: '3131.8291015625',
  gcStats: {
    bytesCollected: undefined,
    totalMsRun: undefined,
    totalCycles: undefined,
    averageCycleTimeMs: undefined,
    lastRunTimeMs: undefined,
    gcNumericTreesMissed: undefined,
    gcBlocksDenied: undefined
  },
  cursorStats: {
    globalIdle: undefined,
    globalTotal: undefined,
    indexCapacity: undefined,
    idnexTotal: undefined
  },
  stopWords: 8
}

@claridgicus
Copy link
Author

I can probably spin up a local node and provide a dump if providing the creds is too hard.

@raz-mon
Copy link
Collaborator

raz-mon commented Mar 3, 2024

Hi @claridgicus,
Thanks for the info.

Let me make sure I understand you correctly - what you refer to as "sorts" are fields of your index that you may use to sort the results of a query, correct? And when you have more then 250 such fields, the sorting doesn't work (i.e., a "broken index"). Is this the case for all fields you try sorting by? Does the sorting succeed for the same field (that doesn't succeed for more than 250 sorts) and for the same data when you have less than 250 sorts? This is interesting.

Can you provide a dump.rdb file or a script that reproduces this issue? Whatever is more comfortable for you. If you go for the rdb, please state the used versions as well as the queries you run showing the unsuccessful sort.

Thanks

@claridgicus
Copy link
Author

claridgicus commented Mar 5, 2024 •
edited

@raz-mon I will try to get you a dump tomorrow.

A bit of clarification - I've debugged my way through my application to find the issue occurs when the index hits 256 sortable fields (exact number) regardless of type.

When the index exceeds the 256 - all the sorts stop working.

It does work when I alter the index below the 256 limit, so by just arbitrarily removing some sorts from my index it will resume working normally - so a "broken" index can be altered into a working index.

@claridgicus
Copy link
Author

Sorry I have had a mega couple of weeks, I will see about that dump this week

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stale label May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raz-mon raz-mon

Labels
bug stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@claridgicus @raz-mon