Gearset scanner

[giniedp]

With the latest game expansion and the introduction of new perks the gearset scanner is not able to match many items properly any more. Some improvements and band aids have been applied recently. Here i want to summarize a guide how the algorithm could be re-worked to catch the currently broken cases.

Text Recognition

• Start with a list of recognized text lines.

Recognize item name

• Find the item rarity
• Use text until rarity as item name

Recognize attribute perk

• Skip lines until we find a text like +26 Dexterity.
• Split attribute perks have two of such lines.

Attributes currently follow that format. An exception is the Magnify attribute perk, which has the following translation key and text

"ui_stat_magnify_tooltip": "+{amount1} Magnify (highest attribute: {attribute1})",

Assuming the text recognition catches the round brackets, we could

• strip all text around (...)
• and/or strip all text after (

Recognize perks names

• Read all lines that include a colon :
• Use text until colon as perkname

The perk description may also contain a colon and lead to false positives. Currently this is issued by the following known tooltip text

"perktooltips_stacks": "Perk Stacks: ",
"perktooltips_max_stacks": "Perk Max Stacks: ",

other languages

• de uses "Effektstapel: " and "Maximale Effektstapel: "
• es uses "Acumulaciones de ventaja: " and "Acumulaciones máx. de ventaja: "
• fr uses "Acumulaciones máx. de ventaja : " and "Cumuls d'atouts max : "

Since text recognition is not 100% on point, we could try following approach:

• lookup for translation of e.g. perktooltips_stacks
• find the word that is the last before the colon: e.g. Stacks:
• strip that from perk lines

Build Item Candidates

• Either user has to provide an itemType scope
• or we could try to recogniz itemType from input image
• preselect an itemPool from database that match the requested itemType

For each item from itemPool create a candidate item

Match Attribute perk

From recognition we are given the recognized attribute names and potentially values

• Find either a fixed perk slot or a bucket slot on item for attribute
• If fixed slot found
  • get items attribute perk from database
  • match against recognized attribute
  • track rating
• If bucket slot found
  • find best matching attribute perk from database
  • track rating

Fixed perk slots

If item has fixed perks then

• find best matching pair of perkname and slotPerk
• mark perkname as consumed
• mark slot as done
• track a rating coefficient, how well the perkname did match the slotPerk
• repeat until all fixed perks are done or no perkname left

Perk Bucket slots

If item has perk buckets, do for each bucket

• find best matching pair of perkname and perkBucket
• mark perkname as consumed
• mark perkBucket as done
• track a rating coefficient, how well the perkname did fit into the perkBucket
• repeat until all buckets are done or no perkname left

perk buckets point to a list of perks that can roll on that slot. However, users now are able to re-slot such slots at the klin. Maybe we should just ignore the bucket pool.

Build item name

Item names are a combination from PREFIX, name, SUFFIX. Only if item does not IgnoreNameChange

• find perk providing a prefix
• find perk providing a suffix
• build name
• match against recognized name and track rating

Collect results

The candidates should now have a rating value based on

• item name
• matching perk names
• matching number of perks?

Results then should be sorted by best match. The current system rejects results that were not able to 'consume' all perks. This sometimes leads to empty results. We should instead rely on the rating value.

Artifact Pitfall

The new artifacts all come with a special perk that actually encodes 2 perks

1. Magnify (this is a tooltip name)
2. some other perk. (this is the actual perk)

e.g. https://www.nw-buddy.de/perks/table/PerkID_Artifact_Set1_HeavyHead

A solution to this yet needs to be found.

[reaganchisholm]

Few thoughts after reading through:

1. I know we disabled the curves effect for now but do you think its possible to tweak the curves effect on the image processing to help us get more consistent results? Ideally we can eliminate the issue with Perk Stacks: and Perk Max Stacks: by "erasing" that text with curves but that does kind of break Recognize attribute perk aspect sometimes because they are a similar style in the image.

2. "Results then should be sorted by best match. The current system rejects results that were not able to 'consume' all perks. This sometimes leads to empty results. We should instead rely on the rating value." Definitely think this will be a better user experience, I think it would feel better to at least get an item returned and maybe missing a perk or two that you have to add yourself.

3. I think adding some error messages may help out a lot as well. I think right now users are getting confused because they get no feedback when an item fails. I also think we can potentially update the UI to be a bit more clear that the results can include multiple items in order of a rating. I didn't actually realize that functionality was there until I was working on the code. Did a really quick mockup of how to potentially improve the design to be more clear for the user. If you're interested in going down this path, I can take some more time to think it through some more. It comes down to if we want to accept that there will always be a margin of error with the importer and update the UI/UX to walk the user through those issues if they happen.

4. For Artifacts, could we return the item based on name match? again, this would sometimes leave the user having to add their final perk if they have it upgraded. Maybe that's more confusing because they uploaded the screenshot they may expect it to already have the perk... hmmm, similar discussion to above.

5. Should we add a more robust amount of sample items for tests? Ideally we can automate testing every weird quirk or item that we run into to prevent regressions while testing fixes but that will obviously bloat our test times. Maybe we have a smaller pool of samples to test that runs in CI but we have a larger full one we can run locally when working on the importer? Somewhat defeats the purpose of the tests if we have one we only runs sometimes but not sure another way around without making the tests long

Let me know your thoughts!

[giniedp]

1. I am infavor of everything that improves the recognition. The curves need more testing and tweaking. I had not tested and just used whatever was configured. Since the recognized text contained more errors than without the curves for me, i had disabled it. I would keep it as an optional mechanism that we can enable and tweak when needed once the core has been improved.

2. Agree. The UI definitely needs a rework. The horizontal swipe with a peek into next item is a good idea. https://swiperjs.com/ is already included as a dependency, maybe we can explore that more. Think we should aim for best recognition result, but we always have to expect some errors. Especially with content updates.

3. Not quite. If we can recognize that the user scanned an artifact, we simply limit the set of processed items. So we only look at all artifacts artifacts and still try to match the recognized perks.

4. We can use tests as we grow and as we encounter weird items. There should be some core samples to cover the basics at least. Having a growing set of samples for offline testing sounds good to me.

Another thought about recognition. Tesseract gives us all recognized characters with coordinates and dimensions from where that character was found in the image. Same goes for lines. We could do an extra UI step and present to the user what perk names have been detected as an overlay over the input image. The user then can simply reject the false positives and continue recognition. Ideally i dont want the user to be interrupted and just be presented with results. But that might be a good mechanic for 1. debugging and 2. in case a patch breaks a lot.