I can put both the problem that this fixes, and how it fixes it, in a nutshell:

The original regexes did not support block-attribute-list's that contain, themselves, an inline macro:
this is because the "meat" of each of these regex's matches is from a character-group that dismisses the possibility of a literal ']' (i.e., \]) lying WITHIN the attribute-list:
either this one:
[^,\\]]+
or this one:
[^\\],.#%]+

The fix for this is to make this character group be only the first of a pair of alternatives -- the latter of which ALLOWS a literal ']' but only if "looking ahead" from it reveals that it is NOT the end of that line of text -- and thus, that it is not the CLOSING ']' of the whole block-attribute-list:
either
(?:[^,\\]]|\\](?=[ \\t]*\\S))+
or
(?:[^\\],.#%]|\\](?=[ \\t]*\\S))+

And the reason that THREE regexes had to be changed, rather than one, to fix this problem is:

1. the regex on line 1713 doesn't match any text: it's just one large LOOK-AHEAD! (It leaves it to the regex on line 1716 to match the text.)
2. the regex on line 340 is still looking to match the same text that line 1716's regex matched already; because it is invoked by the "captures" dictionary that starts on line 1717.

(Although https://macromates.com/manual/en/language_grammars says that "captures" dictionaries can "currently" only assign a name to a captured group's matched text, that document covers only TextMate 1.5.1; and from the discussion in https://www.apeth.com/nonblog/stories/textmatebundle.html (see about halfway down), we may surmise that Texmate 2 allows what we see in our own grammar starting at line 1717 -- viz., a "captures" dictionary that contains captures that assign not (or not only) a name, but (also) a "patterns" array -- to look for yet more matches within the same, already-matched text!)

Hey!

Thank you for taking the time to explain this issue in detail. In a nutshell, I'm not convinced that the TextMate grammar is suitable for highlighting AsciiDoc text. The work on the AsciiDoc specification has revealed all the ambiguities and limitations of a grammar like TextMate.

I am not opposed to merging this improvement, but I believe it would be better to work on a semantic token provider that will allow for a much more precise and accurate syntax highlighting.

https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-token-provider

I've created a meta issue (pinned) because there are a lot of cases where the TextMate grammar produces wrong highlighting.

Ah! Quite interesting. I will try to be more persevering in my perusal of that page (it didn't initially connect with my brain too wll, so I had dropped it :-/

Since you relate it to "the work on the AsciiDoc specification", I wonder whether this (already marching forward?) work you propose depends on that spec-work? or is it independent?

I won't say that we have to wait for a complete specification to start working on a semantic token provider.

If you are interested, you can join: https://chat.asciidoc.org and/or take a look at https://github.com/opendevise/asciidoc-parsing-lab/tree/main