Automapping: Explicitly detect map edges

[eishiya]

In the comments for #3100, the possibility of adding a MatchType for explicitly matching tiles outside the map was discussed, but there was no consensus and the issue was closed without implementing anything like this. I still think some way to have rules that only match at map edges would be a useful feature.

Use-cases for matching (near) map borders:

• generating a decorative frame around a map of any size
• making sure the map is always impassable at the edges, e.g. by making sure the tiles near the edges are always water or cliffs
• automatically placing solid or one-way collision tiles under a platform depending on whether it's near the bottom of the map.

It is already possible to accomplish edge detection like this using a guide layer: a filled guide layer and rules with MatchOutsideMap and no overflow/wrap means that any location on the guide layer that is empty must be outside the map. However, this means the rules can't use OverflowBorder and WrapBorder, unless the layer is prepared such that it contains different tiles at the edges than the interior.
Theoretically Automapping can even be used to generate this guide layer, though not with "Automap While Drawing".

Still, it would be nice to not have to use additional guide layers to get at information that should be trivial to access: is this cell next to a particular map border?

Unfortunately, I don't have any good ideas on the UI for this. Some things I've considered:

• OutsideMap MatchType. This would match any cell that is outside the map on input layers, and any cell inside the map on inputnot layers. Infinite maps would never match this special tile (except on inputnot layers).
  • This is probably the "simplest" option and has the benefit of being fairly clear visually in the rule, but
  • It would require MatchOutsideMap to be true to work, since otherwise Tiled doesn't build match regions where any cells are outside the map. This means all the other rules have to be designed with that in mind.
• Map edge MatchType tiles for each possible edge and corner (even hex maps have only 4 sides and 4 corners). The edge "tiles" would match the relevant corners as well.
  • This would allow actually specifying the border itself in the rule, so it would work regardless of MatchOutsideMap.
  • It would be quite tedious for users though, they'd have to pick the correct tile instead of being able to just draw the shape they're looking for, which goes against how Automapping generally works.
  • Compact - you only need one tile to match a particular edge or corner, whereas the above method requires at least 2 for an edge and at least 3 for a corner.
• A combination of the above: Instead of OutsideMap, have MapEdge MatchType, which matches the edge itself rather than cells, and thus works regardless of MatchOutsideMap.
  • This would require more complex logic for Tiled, as these tiles would actually affect matching on cells other than where they are. Internally, perhaps they could be converted to something akin to map edge/corner "tiles".
  • The upshot is this should be quite intuitive for users.
• Two MatchType tiles, which behave as if they were matching a phantom guide layer like what I described above: MapBorder matches cells that touch the map border, and MapInterior matches cells that do not. These should be enough to determine where in the map you are, without needing to ever look outside the map.
  • This would work similarly to the map edge/corner match type tiles, but with fewer tiles for the user to worry about.
  • It would require a minimum of 2 cells to match an edge and 3 to match a corner, so compared to dedicated corner/edge tiles, just like OutsideMatch, except these tiles would be entirely within the map bounds.
• Using rectangle Objects to define the edges/corners. Like maps, rectangles have a well-defined interior and border. Where the rectangle touches or crosses a tile, check for for that edge of the map. The edges of the rectangle that do not touch/cross any input tiles are disregarded.
  • Might not be intuitive. Takes up a lot of extra space too, since the unimportant part of the rectangle has to be outside the rest of the rule.
  • It's difficult to move rules when they consist of Tiles and Objects, since TIled has no good tools to move data on both types of layers simultaneously.

Another important issue is that if matching the borders is done using regular input layers, it can be unclear how this interacts with other input tiles at the same location. Unlike other tiles, I feel like the map border stuff should AND with the other tiles, not OR like input layers currently are. So, perhaps it might be good to use a new layer "type" to define border locations, e.g. border layers just to make it completely unambiguous.

My current favourite option is the MatchType tiles that look at a phantom guide layer, and I am ambivalent as to whether they must be placed on a special layer. I'd love to hear if anyone has better ideas!

[MarkOates]

Some time ago I built out my own autotiling system (before trying out and leaning into Tiled's) I came to the conclusion that a lot of features are very context-depedent (like a regex), and I'll just keep finding new match cases that users would want over time.

As far as I got on the particular problem in this issue, I defined rule-specific logic for out-of-bounds tiles as either being "always positive" match, "always negative" match, or "match as if extruded". Again, this property was defined on the "Filter" (equivalent to a Tiled "rule", it's also similar to a Tiled map's MatchOutsideMap property.). As far as I know, Tiled rules are not first class objects that have properties, this may be part of the difficulty in finding a simple solution.

Regardless, I still ended up with a realization that these things are ultimately complex, and all cases can't be captured.

So, line of thinking on my autotile system was to allow custom-defined matching. For Tiled, this would mean adding a new MatchType that is "Custom", and allow a script of some sort to be used. Users could traverse the nearby tiles, define matching parameters and patterns. The script itself might be contained in a correlated property MatchTypeScript. The MatchTypeScript could be used alternatively in place of MatchType. This would catch all cases.

When looking through the Tiled source, it seemed like the roadblock in this would be injecting a script interpreter for this MatchType around here, and all the work involved in verifying and validating the script.

These were my thoughts on the topic at least.

[bjorn]

Again, this property was defined on the "Filter" (equivalent to a Tiled "rule", it's also similar to a Tiled map's MatchOutsideMap property.). As far as I know, Tiled rules are not first class objects that have properties, this may be part of the difficulty in finding a simple solution.

Well, in fact since Tiled 1.9, rules can have individual properties by placing rectangle objects over those rules and setting certain properties on them. However, MatchOutsideMap currently remains a per-map property, but if desired it could be made a per-rule option as well along with the OverflowBorder and WrapBorder properties. Neither really provides a solution to this issue, though.

I think another option which hasn't been mentioned here, would be the ability to specify that the outside of a map should be considered equal to a specific tile. Then, you can use that tile to match it, or in some cases, using this option will be enough to make your rules behave as desired on the edges. The main issue with this type of configuration is that there is currently no custom "tile reference" property, though this has already been asked about before (#3235).

[eishiya]

I think another option which hasn't been mentioned here, would be the ability to specify that the outside of a map should be considered equal to a specific tile.

I think this adds unnecessary complications when instead the outside of a map could match MatchType "OutsideMap" or something. I don't see much benefit in customising which tile the outside of the map matches, since it's so easy to add a specific special tile as an input option to rules. We already have options for treating the outside as empty (MatchOutsideMap on its own), as extruded (OverflowBorder) and as wrapped (WrapBorder), perhaps this could be a fourth option (maybe "MarkBorder"?).

Edit: Ideally, I'd like for "OutsideMap" to always match tiles outside the map regardless of the OverflowBorder, WrapBorder, etc options, so that I can use both in some cases. For example, rules that tidy up tree trunks might work with OverflowBorder and just pretend the trunk continues downwards past the map edge, but rules that tidy up tree crowns might want to look for tiles outside the map edge specifically and just pretend they crowns are "correct", while extruding wouldn't give correct matches since normal crowns look diffrent from that. But I understand this makes the implementation more difficult. I think making the MatchOutsideMap properties be per-rule would help, since it would allow keeping rules that need different border behaviours in a single map, so my example could have OverflowBorder on the rules map for my trunks, but "OutsideMap" for the tree crown rules.

[bjorn]

I think this adds unnecessary complications when instead the outside of a map could match MatchType "OutsideMap" or something.

While an "OutsideMap" matching tile might sound like a good idea from a user's perspective, it's a somewhat different story from the implementation side. Currently, all match types reflect some set of tiles that are either desired or not desired and the input matching function can just loop over those and compare them (so, in this process, the "match type" is no longer relevant at all - it is currently only inspected during the "rule compilation" step).

So, dragging "match type" into the matching code is somewhat involved and likely to affect performance, whereas adding a fallback tile to use for "outside the map" would be relatively straight-forward, heh.

[eishiya]

Makes sense D: Maybe it could use a fake "outside the map" tile from a secret Tileset that only Tiled knows about? This could behave like any other tile to the matching code. MatchType "OutsideMap" would match this tile, and the out-of-bounds area would be treated as populated with this tile when the relevant options are set.

Still not ideal though, as I think it would be best to be able to address the out-of-bounds region independently of the "tiles" that "populate" it, allowing mixing both out of bounds matching and the different MatchOutsideMap modes within a single rule.

[MarkOates]

How about a tile layer (similar to input_ and output_) where you could define overflow tiles that would implicitly be placed there for that rule in the event of an overflow. (Might call it overflow_?)

Imagine a twisty tree 6x8 derived from an input_ using an air tile and tree tile. As it approaches the overflow it would need a complex array of tiles to fill in the overflow shape in order to fit the match, and a simple OverflowBorder , WrapBorder, extrude, or map-or-rule-wide user-defined fill tile wouldn't fit the bill.

This could cause issues when multiple overflow rules could seemingly produce conflicting ideas of what tile exists in the overflow space, placing different "imaginary overflow tiles" to fit their match, creating a "Schrodinger's tile" situation.

[eishiya]

How would this proposed overflow_ data relate to the working map, which does not contain these tiles, and what benefit would this have over a single overflow tile? Part of the reason for this feature request is precisely because it's common for rules to define contexts that get cut off by the map border in maps, and some way is needed to identify those situations. Since those details simply aren't in the map, some way to identify parts of the region that might be cut off is sufficient.

Just for clarity, I want to illustrate how a single tile would work to define the out-of-bounds area, whether that's a MatchType "OutsideMap" tile, or a custom tile:
Let's say I have an input that matches this tree, and I want to match this tree even if any part of its crown is cut off because it's near the map edge. The bluish region here would be my "out of bounds" area defined by the single tile, any of these tiles would count as a match if they contain the appropriate tree crown tile, or are out of bounds:

This would mean either of these would match, in addition to many other possible matches, while still being more specific than matching just the two trunk tiles:

In practice, I'd probably mark up that entire rule with the out of bounds tile, so that it takes effect when even a single tile from the tree is in the map.

[bjorn]

Maybe it could use a fake "outside the map" tile from a secret Tileset that only Tiled knows about? This could behave like any other tile to the matching code. MatchType "OutsideMap" would match this tile, and the out-of-bounds area would be treated as populated with this tile when the relevant options are set.

Yeah, that's an interesting idea, that should make the comparison only a little bit more complicated given that we'd still need to also support the case where tiles outside the map are treated as empty.

To come back about a section from your first post:

Another important issue is that if matching the borders is done using regular input layers, it can be unclear how this interacts with other input tiles at the same location. Unlike other tiles, I feel like the map border stuff should AND with the other tiles, not OR like input layers currently are.

In your above example of the tree, it seems you're suggesting to match as usual, considering "outside map OR part of tree" a match. Or was the reference to "edge" matching something else than "outside of map" matching? In any case I hope we can treat the layers the same, with "input" layers doing OR and "inputnot" layers doing AND.

[eishiya]

In your above example of the tree, it seems you're suggesting to match as usual, considering "outside map OR part of tree" a match.

For that example rule, that is correct. For another rule, such as one defining a frame for a map or map-edge cleanup, I may have the outside-the-map tile as the only input at those locations.

As for that bit from the OP: I think there I was considering rules that I'd want to only match the border, while also checking for specific overflows (most useful with WrapBorder) - there are situations where one wants to look for those matches while also making sure the rule matches only at the border. But there are also situations like that tree example where one wants to match either at a border or not at a border.
Thinking about it now, just treating it as regular tiles and ORing would probably be fine, and is closer to what's desired in most cases. The effect of AND can be accomplished by checking the border of another (possibly non-existent/dummy) layer instead of the layer where we check for specific tiles.

As for making sure Empty matches also work, could it perhaps be made that when MatchOutsideMap is true but the other two options aren't, MatchType Empty is treated as matching Empty and the secret OutsideMap tile? I guess that wouldn't really help in the wrap and extrude cases though, since you'd still need the OutsideMap tile to match outside map and the overflow/wrap tiles...
Edit: Since borders are shared by all layers and there are no plans to support layer offset compensation, maybe any time Automapping encounters the MatchType "OutsideMap" tile, it should "move" it to a separate input layer that matches a dummy layer populated with empty inside the map and with the special tile outside the map? This would allow the regular input layers to match tiles as they always have, and the extra Tiled-generated layers would handle matching map bounds.