Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose unwrapped Rouge HTML formatter #4472

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

NyanKiyoshi
Copy link

This exposes the Rouge HTML formatter prior to any extension wrapping in order to allow further customizations when overriding the Asciidoctor::SyntaxHighlighter::RougeAdapter class.

This is a similar fix to an older issue reported at #3953.

Prior to these changes, executing for example this code (autolink-urls-in-comments.rb):

class ExtendedRougeSyntaxHighlighter < (Asciidoctor::SyntaxHighlighter.for 'rouge')
  register_for 'rouge'

  def create_formatter node, source, lang, opts
    formatter = super
    formatter.singleton_class.prepend (Module.new do
      def safe_span tok, safe_val
        if tok.token_chain[0].matches? ::Rouge::Token::Tokens::Comment
          safe_val = safe_val.gsub(/https?:\/\/\S+/, '<a href="\&">\&</a>')
        end
        super
      end
    end)
    formatter
  end
end

Could potentially cause to override classes that do not implement safe_span(), for example RougeExt::Formatters::HTMLTableLineNumberer as we expect Rouge::Formatters::HTML instead.

Those other classes (RougeExt::Formatters::*) may be useful for extensions to keep enabled but currently need to be disabled as they are shadowing Rouge::Formatters::HTML.

The autolink-urls-in-comments.rb example would only work when not passing the attributes %linenums, and highlight:

:source-highlighter: rouge

== This works

[,ruby]
----
get {
  render "Hello, World!"   # https://example.com
}
----

== This doesn't work

[%linenums,ruby]
----
get {
  render "Hello, World!"   # https://example.com
}
----

== This doesn't work

[,ruby,highlight=2]
----
get {
  render "Hello, World!"   # https://example.com
}
----
HTML Output (look for <a presence):
<div class="sect1">
    <h2 id="_this_works">This works</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><span class="n">get</span> <span class="p">{</span>
  <span class="n">render</span> <span class="s2">"Hello, World!"</span>   <span class="c1"># <a
            href="https://example.com">https://example.com</a></span>
<span class="p">}</span></code></pre>
            </div>
        </div>
    </div>
</div>
<div class="sect1">
    <h2 id="_this_doesnt_work">This doesn&#8217;t work</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><table class="linenotable"><tbody><tr><td class="linenos"><pre>1</pre></td><td
        class="code"><pre><span class="n">get</span> <span class="p">{</span>
</pre></td></tr><tr><td class="linenos"><pre>2</pre></td><td class="code"><pre>  <span class="n">render</span> <span
        class="s2">"Hello, World!"</span>   <span class="c1"># https://example.com</span>
</pre></td></tr><tr><td class="linenos"><pre>3</pre></td><td class="code"><pre><span class="p">}</span>
</pre></td></tr></tbody></table></code></pre>
            </div>
        </div>
    </div>
</div>
<div class="sect1">
    <h2 id="_this_doesnt_work_2">This doesn&#8217;t work</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><span class="n">get</span> <span class="p">{</span>
<span class="hll">  <span class="n">render</span> <span class="s2">"Hello, World!"</span>   <span class="c1"># https://example.com</span>
</span><span class="p">}</span></code></pre>
            </div>
        </div>
    </div>
</div>
HTML Output after the fix (this PR):
<div class="sect1">
    <h2 id="_this_works">This works</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><span class="n">get</span> <span class="p">{</span>
  <span class="n">render</span> <span class="s2">"Hello, World!"</span>   <span class="c1"># <a href="https://example.com">https://example.com</a></span>
<span class="p">}</span></code></pre>
            </div>
        </div>
    </div>
</div>
<div class="sect1">
    <h2 id="_this_doesnt_work">This doesn&#8217;t work</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><table class="linenotable"><tbody><tr><td class="linenos"><pre>1</pre></td><td class="code"><pre><span class="n">get</span> <span class="p">{</span>
</pre></td></tr><tr><td class="linenos"><pre>2</pre></td><td class="code"><pre>  <span class="n">render</span> <span class="s2">"Hello, World!"</span>   <span class="c1"># <a href="https://example.com">https://example.com</a></span>
</pre></td></tr><tr><td class="linenos"><pre>3</pre></td><td class="code"><pre><span class="p">}</span>
</pre></td></tr></tbody></table></code></pre>
            </div>
        </div>
    </div>
</div>
<div class="sect1">
    <h2 id="_this_doesnt_work_2">This doesn&#8217;t work</h2>
    <div class="sectionbody">
        <div class="listingblock">
            <div class="content">
<pre class="rouge highlight"><code data-lang="ruby"><span class="n">get</span> <span class="p">{</span>
<span class="hll">  <span class="n">render</span> <span class="s2">"Hello, World!"</span>   <span class="c1"># <a href="https://example.com">https://example.com</a></span>
</span><span class="p">}</span></code></pre>
            </div>
        </div>
    </div>
</div>

This exposes the Rouge HTML formatter prior to any extension wrapping in order to allow further customizations when overriding the `Asciidoctor::SyntaxHighlighter::RougeAdapter` class.

Prior to these changes, executing for example this code:

```ruby
class ExtendedRougeSyntaxHighlighter < (Asciidoctor::SyntaxHighlighter.for 'rouge')
  register_for 'rouge'

  def create_formatter node, source, lang, opts
    formatter = super
    formatter.singleton_class.prepend (Module.new do
      def safe_span tok, safe_val
        if tok.token_chain[0].matches? ::Rouge::Token::Tokens::Comment
          safe_val = safe_val.gsub(/https?:\/\/\S+/, '<a href="\&">\&</a>')
        end
        super
      end
    end)
    formatter
  end
end
```

Could potentially cause to override classes that do not implement `safe_span()`, for example `RougeExt::Formatters::HTMLTableLineNumberer` as we expect `Rouge::Formatters::HTML` instead.

Those other classes (`RougeExt::Formatters::*`) may be useful for extensions to keep enabled but currently need to be disabled as they are shadowing `Rouge::Formatters::HTML`.
@NyanKiyoshi NyanKiyoshi changed the title Fix/shadowed rouge html formatter Expose unwrapped Rouge HTML formatter Jul 22, 2023
@mojavelinux
Copy link
Member

I'm open to introducing another method that creates the base HTML formatter. However, in order to proceed, I would need to see the following changes:

  • rename create_html_formatter to create_base_formatter (HTML is implied here)
  • change the method create_base_formatter to accept the same arguments: node, source, lang, opts
  • add tests that use the new method to do what you intend to use it for

@NyanKiyoshi
Copy link
Author

Thank you for the review @mojavelinux!

I want to add three test cases:

  1. When providing no extra code-block attributes, the rouge extension works as expected (e.g. replacing a comment # example.com to # <a href=...>example.com</a>)
  2. When providing attribute %linenums it still behaves as expected (as we are changing the return value of create_formatter())
  3. When providing highlight=[...] (same as 2))

I was looking into those for this pull request following your request, but I'm not quite sure how to proceed. As an illustration, attached is an example test case and what it could look like:

diff --git a/test/syntax_highlighter_test.rb b/test/syntax_highlighter_test.rb
index 7a6fd2f2..e9f7a7df 100644
--- a/test/syntax_highlighter_test.rb
+++ b/test/syntax_highlighter_test.rb
@@ -1147,6 +1147,42 @@ def highlight?
       result = run_command(asciidoctor_cmd, '-r', (fixture_path 'include-asciidoctor.rb'), '-s', '-o', '-', '-a', 'source-highlighter=rouge', (fixture_path 'source-block.adoc'), use_bundler: true) {|out| out.read }
       assert_xpath '//pre[@class="rouge highlight"]', result, 1
     end
+
+    test 'should invoke overridden safe_span when linenums option is enabled' do
+      begin
+        # (Will need to move this outside the test so it can be reused for different test cases)
+        Class.new Asciidoctor::SyntaxHighlighter.for 'rouge' do
+          register_for 'rouge'
+          def create_base_formatter node, source, lang, opts
+            formatter = super
+            raise "Formatter doesn't have method 'safe_span'" unless formatter.respond_to?(:safe_span)
+            formatter.singleton_class.prepend (Module.new do
+              def safe_span tok, safe_val
+                if tok.token_chain[0].matches? ::Rouge::Token::Tokens::Comment
+                  safe_val = safe_val.gsub(/https?:\/\/\S+/, '<a href="\&">\&</a>')
+                end
+                super
+              end
+            end)
+            formatter
+          end
+        end
+
+        input = <<~'EOS'
+        [source,ruby]
+        ----
+        puts 'Hello, World!'  # https://example.com
+        ----
+        EOS
+
+        doc = document_from_string input, attributes: { 'source-highlighter' => 'rouge' }
+        output = doc.convert
+        assert_include '# <a href="https://example.com">https://example.com</a>', output
+      end
+    ensure
+      # Here we should unregister the class
+      # [...]
+    end
   end

   context 'Pygments', if: ENV['PYGMENTS_VERSION'] do

First problem: to be able to test extensions against rouge syntax highlight, I need to register a new class using Class.new Asciidoctor::SyntaxHighlighter.for 'rouge' [...] but I did not see any currently implemented ways to unregister the extension after the test case in order to not change the behavior unrelated tests.

I considered adding new methods, such as unregister(), and unregister_all() that are present in Asciidoctor::Extensions, but this would cause to deactivate ExtendedRougeSyntaxHighlighter which is not acceptable as it would change behaviors of other tests (on top of greatly being out of scope for this PR).

Other solution would be to allow to unregister by class object (e.g. unregister_class(my_class)) which seems like a potential good way but this would involve doing more changes that need proper tests (I do not have any experience in Ruby thus it would be quite time consuming as well).

Another way could have been to use directly ‎Asciidoctor::SyntaxHighlighter::Factory::registry‎ to unregister the class, but it's a private method thus it will not work (on top of being fairly dirty).

Easiest way would probably be to put the tested class into a ruby file, and start a child process so we do not need to unregister anything, e.g., asciidoctor -r my_extension.rb [...] like done in this test case: https://github.com/asciidoctor/asciidoctor/blob/f3800cc9c92faf8370041b2b27a61124318ed289/test/syntax_highlighter_test.rb#L1147. I'm personally thinking that's the best solution.

Second problem: the class from autolink-urls-in-comments.rb seems like a good fit for testing the proper behaviors against %linenums and highlight=[...] attributes as it's easy to test and easy to understand as it replaces comments, but copy-pasting it would not be unnecessary code duplication and thus not acceptable.

I would think it could be a good idea instead of adding the tests into test/syntax_highlighter_test.rb I could write tests for docs/modules/syntax-highlighting/examples/autolink-urls-in-comments.rb which would also allow to avoid the case where this example code would break in the future due to changes (as it's not covered by tests), while allowing us to properly tests the extension changes from this PR. What do you think about this? If you agree, where do you suggest to put those tests?

Thank you in advance for your inputs!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants