[Feature Request]: Dupe detection

[kphoenix137]

Feature Type

Other (please specify)

Describe

I think this is more a broad thing more than anything. I believe we should re-add dupe detection and deletion into the game. @StephenCWills previously brought up the fact that it's possible for items to be wrongly deleted due to duplicate items being obtainable in the same game session. Once that's completely sorted out, I'd like this to be revisited.

I believe we should check for dupes on the ground, in the inventory, and in the stash, and delete duplicate copies of items. Perhaps it would be mutually exclusive, or not, depending on potential issues with it.

In addition, but not less important, I think we should have a dupe list of items that can be found online. There are many websites that have downloads of ITM files that were commonly used with Diablo trainers, where players could import high end gear into the game. We could compile a list of the seeds, indexes, and creations flags of these items (specifically just the high end items like perfect stat items that end-game characters use), and automatically delete copies of these items. This brings up a very real problem though; It's entirely possible for people to legitimately find amazing items that share the exact same item data as these downloadable dupes. In response to that, I think we can implement adding a new flag to all items in DevilutionX, that can be used as an identifier that an item was specifically found in DevilutionX, and not vanilla. This could be used to protect items found in DevilutionX that match the item data from these dupe items in vanilla from being deleted. Of course, a player could bring these items into vanilla and back into DevilutionX, which would result in deletion, but I think it's enough of a safeguard for a very unlikely circumstance already.

On the note of adding a new creation flag, this can be dual purpose. For example, @qndel 's proposal for fixing the problem with unique drops involves a new creation flag. A single creation flag can be used for both these purposes, since it's a flag that's added in the version it's implemented in going forward.

For anyone who might say "that's going to be a LOT of work", I fully understand, and I'm willing to do a lot of the heavy lifting involved. I would just like the support of others in this endeavor.

[D1-Constantine]

That would be completely awesome

[StephenCWills]

@StephenCWills previously brought up the fact that it's possible for items to be wrongly deleted due to duplicate items being obtainable in the same game session. Once that's completely sorted out, I'd like this to be revisited.

The issue in question that needs to be fixed is #6261.

The other issue I found is documented here: diasurgical/devilution#64 (comment). However, it seems to be a lot less common.

The solution for both issues is pretty much the same.

[AJenbo]

An issue is that you can't ever really rule out legit duplicates. And what do you do if someone duplicate another players items? Whos should be deleted?

This brings up a very real problem though; It's entirely possible for people to legitimately find amazing items that share the exact same item data as these downloadable dupes. In response to that, I think we can implement adding a new flag to all items in DevilutionX, that can be used as an identifier that an item was specifically found in DevilutionX, and not vanilla. This could be used to protect items found in DevilutionX that match the item data from these dupe items in vanilla from being deleted. Of course, a player could bring these items into vanilla and back into DevilutionX, which would result in deletion, but I think it's enough of a safeguard for a very unlikely circumstance already.

You could avoid all of that by simply checking the seed which is what you would want for duplicate items any way, it's also way simpler then trying to check stats.

[julealgon]

In response to that, I think we can implement adding a new flag to all items in DevilutionX, that can be used as an identifier that an item was specifically found in DevilutionX, and not vanilla. This could be used to protect items found in DevilutionX that match the item data from these dupe items in vanilla from being deleted. Of course, a player could bring these items into vanilla and back into DevilutionX, which would result in deletion, but I think it's enough of a safeguard for a very unlikely circumstance already.

Still nothing would prevent someone from updating those tools to add the new parameter and create arbitrary fake DevilutionX items, correct?

Without a server-driven validation system, it will never be possible to completely prevent dupes. Are you OK with this being just a "half-measure", to make it "slightly more difficult" to hack the game but still completely possible?

[AJenbo]

Still nothing would prevent someone from updating those tools to add the new parameter and create arbitrary fake DevilutionX items, correct?

Abilty

[kphoenix137]

An issue is that you can't ever really rule out legit duplicates. And what do you do if someone duplicate another players items? Whos should be deleted?

True, although I think "legit duplicate" is a bit of an oxymoron. Well my suggestion is to make all duplication checks mutually exclusive, meaning that having an item on the ground and an item in the inventory that's a dupe of that item will not result in the destruction of either item. I think that would become way too messy and be exploitable for griefing. As for ground items, it would function the same way as vanilla, where the existing item will be deleted. Dropping a dupe would delete the dupe already on the ground. Picking up a dupe of an item you already have in your inventory would destroy the item already in the inventory.

This brings up a very real problem though; It's entirely possible for people to legitimately find amazing items that share the exact same item data as these downloadable dupes. In response to that, I think we can implement adding a new flag to all items in DevilutionX, that can be used as an identifier that an item was specifically found in DevilutionX, and not vanilla. This could be used to protect items found in DevilutionX that match the item data from these dupe items in vanilla from being deleted. Of course, a player could bring these items into vanilla and back into DevilutionX, which would result in deletion, but I think it's enough of a safeguard for a very unlikely circumstance already.

You could avoid all of that by simply checking the seed which is what you would want for duplicate items any way, it's also way simpler then trying to check stats.

Sorry, I didn't mean checking item stats. By item data I specifically mean seed, creation flags, and item index, which I believe is what was used to decide in vanilla if an item was a dupe of another. Items can share the same seed but be completely different due to something as little as an index change or a creation flag change. My suggestion for another creation flag is to make "new" items exempt from the vanilla cheat items blacklist database.

In response to that, I think we can implement adding a new flag to all items in DevilutionX, that can be used as an identifier that an item was specifically found in DevilutionX, and not vanilla. This could be used to protect items found in DevilutionX that match the item data from these dupe items in vanilla from being deleted. Of course, a player could bring these items into vanilla and back into DevilutionX, which would result in deletion, but I think it's enough of a safeguard for a very unlikely circumstance already.

Still nothing would prevent someone from updating those tools to add the new parameter and create arbitrary fake DevilutionX items, correct?

Without a server-driven validation system, it will never be possible to completely prevent dupes. Are you OK with this being just a "half-measure", to make it "slightly more difficult" to hack the game but still completely possible?

I think AJenbo's one word response was extremely on point. From what I've personally observed over these past couple months, the level of ability of the people cheating and distributing cheat items has been quite novice level. It appears to be users who use Cheat Engine in order to duplicate, modify item stats, and modify character stats. It also seems to be apparent that these users are also importing items from vanilla that were obtained using a Trainer, such as Doom-Gaze's trainer, which was one of the more popular ones. Websites still exist that have large zip archives filled with items for download. These are the items that I'm interested in blacklisting. I believe we can go after the Cheat Engine users with very little effort. I'm well aware going after cheaters with programming ability is a losing game, but I think those are in short supply.

[kphoenix137]

On a related note, I had another idea today. Cheat Engine has a distinct feature that allows users to lock a memory value or a range of memory values, which continuously overwrites those memory addresses with the data that existed there at the time of locking. From what I understand, this is what one of our more recent cheaters was using in order to distribute duplicate items. They would lock the item data in the inventory, and be able to pick up the item without actually removing that item from the grid, allowing them to endlessly duplicate an item in place (probably distributing elixirs to other players). This idea might be a bit more effort, but I think we can add some type of validation that does a check immediately after CheckInvCut() is called, to see if the exact item data still exists in the slot we just pulled it from. If it does we could execute an app_fatal(). Just an idea, maybe this isn't as easy as it sounds in my head. Any further input on this would be helpful.

[AJenbo]

Well my suggestion is to make all duplication checks mutually exclusive, meaning that having an item on the ground and an item in the inventory that's a dupe of that item will not result in the destruction of either item. I think that would become way too messy and be exploitable for griefing. As for ground items, it would function the same way as vanilla, where the existing item will be deleted. Dropping a dupe would delete the dupe already on the ground. Picking up a dupe of an item you already have in your inventory would destroy the item already in the inventory.

Ok, but that won't really work for what I have seen where players where careful to only drop on of each elixir at a time. It's basically vulnerable to the same issue as the original detection and people already know how to get around it, so much so that they will do it even when it's not there.

[kphoenix137]

Can you use some example scenarios? I just don't think I fully understand what you're saying

[AJenbo]

P1: have elixir
for (int i = 0; i < 50; i++) {
    P1: drop dup of item
    p2: pick up item
    p2: use item
}

dupe successful.

[kphoenix137]

P1: have elixir
for (int i = 0; i < 50; i++) {
    P1: drop dup of item
    p2: pick up item
    p2: use item
}

dupe successful.

under what context? There is no dupe to be able to drop if the dupe has already been deleted.

[StephenCWills]

On a related note, I had another idea today. Cheat Engine has a distinct feature that allows users to lock a memory value or a range of memory values, which continuously overwrites those memory addresses with the data that existed there at the time of locking. From what I understand, this is what one of our more recent cheaters was using in order to distribute duplicate items. They would lock the item data in the inventory, and be able to pick up the item without actually removing that item from the grid, allowing them to endlessly duplicate an item in place (probably distributing elixirs to other players). This idea might be a bit more effort, but I think we can add some type of validation that does a check immediately after CheckInvCut() is called, to see if the exact item data still exists in the slot we just pulled it from. If it does we could execute an app_fatal(). Just an idea, maybe this isn't as easy as it sounds in my head. Any further input on this would be helpful.

It would be helpful to better understand exactly how this locking feature works. If it's able to do something like making a region of memory read-only, then your idea could work okay. If instead it just periodically overwrites the data in that memory location on a timer, then it probably won't be quick enough for us to detect that it's happening.

Even if we could detect it, cheaters could just not lock the memory and instead manually put the item back in their inventory, which I assume wouldn't be that much more difficult.

Furthermore, dropping an item on the ground doesn't call CheckInvCut() so cheaters could just lock HoldItem to continuously drop dupes right from their cursor.

[kphoenix137]

On a related note, I had another idea today. Cheat Engine has a distinct feature that allows users to lock a memory value or a range of memory values, which continuously overwrites those memory addresses with the data that existed there at the time of locking. From what I understand, this is what one of our more recent cheaters was using in order to distribute duplicate items. They would lock the item data in the inventory, and be able to pick up the item without actually removing that item from the grid, allowing them to endlessly duplicate an item in place (probably distributing elixirs to other players). This idea might be a bit more effort, but I think we can add some type of validation that does a check immediately after CheckInvCut() is called, to see if the exact item data still exists in the slot we just pulled it from. If it does we could execute an app_fatal(). Just an idea, maybe this isn't as easy as it sounds in my head. Any further input on this would be helpful.

It would be helpful to better understand exactly how this locking feature works. If it's able to do something like making a region of memory read-only, then your idea could work okay. If instead it just periodically overwrites the data in that memory location on a timer, then it probably won't be quick enough for us to detect that it's happening.

Even if we could detect it, cheaters could just not lock the memory and instead manually put the item back in their inventory, which I assume wouldn't be that much more difficult.

Furthermore, dropping an item on the ground doesn't call CheckInvCut() so cheaters could just lock HoldItem to continuously drop dupes right from their cursor.

I'll look into this further. That's a good point with the HoldItem problem.

[AJenbo]

under what context? There is no dupe to be able to drop if the dupe has already been deleted.

I don't see that what you have mentioned would prevent someone from dropping a duplicate, but maybe it just isn't clear enough how it's supposed to work.

Also like Stephen said:

Even if we could detect it, cheaters could just not lock the memory and instead manually put the item back in their inventory, which I assume wouldn't be that much more difficult.