Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation no longer include JSON response body when using with Rack >= 2.1.0 #456

Open
sikachu opened this issue Jan 15, 2020 · 16 comments

Comments

@sikachu
Copy link

sikachu commented Jan 15, 2020

Hello,

This issue is pretty much for reporting the incompatibility to the gem author, and hopefully it will help anyone who runs into this problem to understand what's going on.

Basically, after we upgrade our dependencies to use Rack 2.1.1, we noticed that our generated documentation no longer show JSON response but instead showing [binary data] instead.

Digging in further, we found out that in rack/rack@8c62821, especially this change, MockResponse#body now creates a buffer and use << to join the content together. However, on line 195, the author uses String.new without specifying the encoding, resulting in Ruby creating a new String with ASCII-8BIT encoding by default.

As it turns out, rspec_api_documentation relies on string encoding to determine if it should include the response body in the documentation or not:

if response_body.encoding == Encoding::ASCII_8BIT
"[binary data]"
else
formatter = RspecApiDocumentation.configuration.response_body_formatter
return formatter.call(response_content_type, response_body)
end

Hence, the change in Rack broke this conditional.

I've reported this issue to Rack in rack/rack#1486, and hopefully we can solve this soon.

The solution right now for us is to lock Rack to ~> 2.0.8 for now.

Thank you very much.

@FrankFang
Copy link

Any other workaround?

FrankFang added a commit to jrg-team/rspec_api_documentation that referenced this issue Feb 25, 2020
@Tao-Galasse
Copy link

Tao-Galasse commented Mar 11, 2020

I ran into the same issue recently. I don't really like this solution, but I fixed it using mokey patching:
config/initializers/rspec_api_documentation.rb

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      last_response.body.encode("utf-8")
    end
  end
end

Indeed, it seems to be caused by an encoding issue where utf-8 become ascii-8bit

@eimermusic
Copy link

This seems to also be the result after Rails 6 upgrade.

@bitlather
Copy link

Just created a new rails 6 app and had this problem, Tao's response above fixed it (#456 (comment))

@cpursley
Copy link

cpursley commented May 19, 2020

Tao-Galasse's solution worked but I found that I needed to stick it right before the RspecApiDocumentation.configure block instead of the initializers directory.

@skibox
Copy link

skibox commented Jun 29, 2020

There's an additional issue with endpoints that use send_data:

     Failure/Error: last_response.body.encode("utf-8")
     
     Encoding::UndefinedConversionError:
       "\xD3" from ASCII-8BIT to UTF-8
     # ./config/initializers/rspec_api_documentation.rb:6:in `encode'
     # ./config/initializers/rspec_api_documentation.rb:6:in `response_body'

I've tweaked Tao's fix, it ain't perfect but it works for my projects:

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      if last_response.headers["Content-Type"].include?("json")
        last_response.body.encode("utf-8")
      else
        "[binary data]"
      end
    end
  end
end

FredaFei added a commit to FredaFei/rspec_api_documentation that referenced this issue Jul 4, 2020
yyzclyang added a commit to yyzclyang/rspec_api_documentation that referenced this issue Jul 26, 2020
@ShamoX
Copy link

ShamoX commented Aug 27, 2020

On my side, I fixed it by two mechanisms :

  1. @skibox's @Tao-Galasse solution remade mine
  2. Use response_body_formatter like it is intended to filter the different kind of output and make it OK for apitome

Resulting in this spec/support/rspec_api_documentation.rb file :

# frozen_string_literal: true

# Fix a bug that generate erroneous "[binary data]" not yet fixed on rspec_api_documentation:6.1.0
module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      body = last_response.body
      if body.empty? || last_response.headers['Content-Type'].include?('json')
        body.encode('utf-8')
      else
        '"[binary data]"'
      end
    rescue Encoding::UndefinedConversionError
      '"[binary data]"'
    end
  end
end

# configure rspec_api_documentation
RspecApiDocumentation.configure do |config|
# ..... some unrelated config here  

  # Change how the response body is formatted by default
  # Is proc that will be called with the response_content_type & response_body
  # by default response_content_type of `application/json` are pretty formated.
  config.response_body_formatter = lambda do |response_content_type, response_body|
    if response_content_type.include?('application/json')
      JSON.parse(response_body)
      return response_body
    elsif response_content_type.include?('text') || response_content_type.include?('txt')
      # quote it for JSON Parser in documentation reader like APITOME
      return "\"#{response_body}\""
    else
      return '"[binary data]"'
    end
  rescue JSON::ParseError
    '"[binary data]"'
  end

# ... Other unrelated config
end

@davidstosik
Copy link
Contributor

Just to make sure people notice, this issue would be fixed by #458 🙇‍♂️

@jakehow
Copy link
Member

jakehow commented Oct 2, 2020

For anyone following this, Ive merged #458 please let me know if this resolves the issue for you. cc @artofhuman

@Uysim
Copy link

Uysim commented Oct 6, 2020

Hope this can be fixed. We need to work with Rails 6

@jakehow
Copy link
Member

jakehow commented Oct 7, 2020

@Uysim please test and let everyone know

@incubus
Copy link

incubus commented Oct 27, 2020

Doesn't work with application/vnd.api+json content type. Probably it's better to use include?('json')?

@davidstosik
Copy link
Contributor

davidstosik commented Oct 27, 2020

@incubus Note the change in #458 does not, in itself alter the gem's behavior.
If you use the default response_body_formatter, then you will experience the same problem as before.

To solve your problem, please:

  1. Make sure you use the master branch (at least until there's a new release).
  2. Define a custom response_body_formatter in your configuration, like so:
RspecApiDocumentation.configure do |config|
  config.response_body_formatter = 
    Proc.new do |content_type, response_body|
      if content_type =~ /application\/.*json/
        JSON.pretty_generate(JSON.parse(response_body))
      else
        response_body
      end
    end
end

Note that the above, compared with the new default (see below) introduced in #458 would stop filtering out potentially binary data.

elsif content_type =~ /application\/.*json/

As the test response_body.encoding == Encoding::ASCII_8BIT has become unreliable to filter binary data, it will be under your responsibility to figure out, in your context, what you can check to determine whether you want to display a given response or not.

Here's a more complex example:

RspecApiDocumentation.configure do |config|
  config.response_body_formatter =
    Proc.new do |content_type, response_body|
      # http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature
      if response_body[0,8] == "\x89PNG\r\n\u001A\n"
        "<img src=\"data:image/png;base64,#{Base64.strict_encode64(response_body)}\" />"
      elsif content_type =~ /application\/.*json/
        JSON.pretty_generate(JSON.parse(response_body))
      elsif response_body.encoding == Encoding::ASCII_8BIT # note 1
        "[binary?]" # note 2
      else
        response_body
      end
    end
end

Notes:

  1. This is the old check, still unreliable, but because it happens after checking for JSON or PNG, then you would be able to display those two properly. You might lose the display of very plain text (the else part) if Rack returns everything as ASCII_8BIT (same problem as before).

  2. To alleviate the above, you might want to return a better string than just [binary?] if you aren't sure your test is accurate. You could for example return something like this:

    <details><summary>[binary?]</summary>
      #{response_body}
    </details>

This would look like this:

[binary?] ˇÿˇ‡��JFIF�����`�`��ˇ·�ÄExif��MM�*�����������������������������J�����������R�(����������ái���������Z�������`�������`������†�����������†���������������ˇ· !http://ns.adobe.com/xap/1.0/� �ˇÌ�8Photoshop 3.0�8BIM��������8BIM�%������‘�åŸè�≤�ÈÄ òϯB~ˇ‚�ËICC_PROFILE������ÿappl� ��mntrRGB XYZ �Ÿ����������acspAPPL����appl������������������ˆ÷������”-appl������������������������������������������������desc�������odscm���x���úcprt�������8wtpt���L����rXYZ���`����gXYZ���t����bXYZ���à����rTRC���ú����chad���¨���,bTRC���ú����gTRC���ú����desc��������Generic RGB Profile������������Generic RGB Profile��������������������������������������������������mluc����������� skSK���(���ÑdaDK���.���¨caES���$���⁄viVN���$���˛ptBR���&���"ukUA���*���HfrFU���(���rhuHU���(���özhTW�������¬nbNO���&���ÿcsCZ���"���˛heIL������� itIT���(���>roRO���$���fdeDE���,���äkoKR�������∂svSE���&���ÿzhCN�������ÃjaJP�������‚elGR���"���¸ptPO���&����nlNL���(���DesES���&����thTH���$���ltrTR���"���êfiFI���(���≤hrHR���(���⁄plPL���,����ruRU���"���.arEG���&���PenUS���&���v�V�a�e�o�b�e�c�n�˝� �R�G�B� �p�r�o�f�i�l�G�e�n�e�r�e�l� �R�G�B�-�b�e�s�k�r�i�v�e�l�s�e�P�e�r�f�i�l� �R�G�B� �g�e�n�Ë�r�i�c�C�•�u� �h�Ï�n�h� �R�G�B� �C�h�u�n�g�P�e�r�f�i�l� �R�G�B� �G�e�n�È�r�i�c�o���0�3�0�;�L�=�8�9� �?�@�>�D�0�9�;� �R�G�B�P�r�o�f�i�l� �g�È�n�È�r�i�q�u�e� �R�V�B�¡�l�t�a�l�·�n�o�s� �R�G�B� �p�r�o�f�i�lê�u(� �R�G�B� Çr_icœè�G�e�n�e�r�i�s�k� �R�G�B�-�p�r�o�f�i�l�O�b�e�c�n�˝� �R�G�B� �p�r�o�f�i�l�‰�Ë�’�‰�Ÿ�‹� �R�G�B� �€�‹�‹�Ÿ�P�r�o�f�i�l�o� �R�G�B� �g�e�n�e�r�i�c�o�P�r�o�f�i�l� �R�G�B� �g�e�n�e�r�i�c�A�l�l�g�e�m�e�i�n�e�s� �R�G�B�-�P�r�o�f�i�l«|º�� �R�G�B� ’�∏\” «|fnê�� �R�G�B� cœèeáNˆN�Ç,� �R�G�B� 0◊0Ì0’0°0§0Î�ì�µ�Ω�π�∫�Ã� �¿�¡�ø�∆�Ø�ª� �R�G�B�P�e�r�f�i�l� �R�G�B� �g�e�n�È�r�i�c�o�A�l�g�e�m�e�e�n� �R�G�B�-�p�r�o�f�i�e�l�B���#�D���%�L� �R�G�B� ���1�H�'�D���G�e�n�e�l� �R�G�B� �P�r�o�f�i�l�i�Y�l�e�i�n�e�n� �R�G�B�-�p�r�o�f�i�i�l�i�G�e�n�e�r�i� �k�i� �R�G�B� �p�r�o�f�i�l�U�n�i�w�e�r�s�a�l�n�y� �p�r�o�f�i�l� �R�G�B���1�I�8�9� �?�@�>�D�8�;�L� �R�G�B�E�D�A� �*�9�1�J�A� �R�G�B� �'�D�9�'�E�G�e�n�e�r�i�c� �R�G�B� �P�r�o�f�i�l�etext����Copyright 2007 Apple Inc., all rights reserved.�XYZ ������ÛR�������œXYZ ������tM��=Ó���–XYZ ������Zu��¨s���4XYZ ������(����ü��∏6curv���������Õ��sf32������ B���fiˇˇÛ&���í��˝ëˇˇ˚¢ˇˇ˝£���‹��¿lˇ¿���������"�������ˇƒ���������������������������� �ˇƒ�µ����������������}��������!1A��Qa�"q�2Åë°�#B±¡�R—$3brÇ �����%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyzÉÑÖÜáàâäíìîïñóòôö¢£§•¶ß®©™≤≥¥µ∂∑∏π∫¬√ƒ≈∆«»… “”‘’÷◊ÿŸ⁄·‚„‰ÂÊÁËÈÍÒÚÛÙıˆ˜¯˘˙ˇƒ���������������������������� �ˇƒ�µ����������������w�������!1��AQ�aq�"2Å��Bë°±¡ #3R�br— �$4·%Ò����&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyzÇÉÑÖÜáàâäíìîïñóòôö¢£§•¶ß®©™≤≥¥µ∂∑∏π∫¬√ƒ≈∆«»… “”‘’÷◊ÿŸ⁄‚„‰ÂÊÁËÈÍÚÛÙıˆ˜¯˘˙ˇ€�C�������0��0D000D\DDDD\t\\\\\tåttttttåååååååå®®®®®®ƒƒƒƒƒ‹‹‹‹‹‹‹‹‹‹ˇ€�C�"$$848`44`ÊúÄúÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊÊˇ›����ˇ⁄� ��������?�È(¢ä�ˇŸ

@incubus
Copy link

incubus commented Oct 27, 2020

@davidstosik it works, thanks!

@daviidy
Copy link

daviidy commented Apr 18, 2021

I ran into the same issue recently. I don't really like this solution, but I fixed it using mokey patching:
config/initializers/rspec_api_documentation.rb

module RspecApiDocumentation
  class RackTestClient < ClientBase
    def response_body
      last_response.body.encode("utf-8")
    end
  end
end

Indeed, it seems to be caused by an encoding issue where utf-8 become ascii-8bit

It worked for me. Thank you so much

@DakotaLMartinez
Copy link

frozen_string_literal: true

Fix a bug that generate erroneous "[binary data]" not yet fixed on rspec_api_documentation:6.1.0

module RspecApiDocumentation
class RackTestClient < ClientBase
def response_body
body = last_response.body
if body.empty? || last_response.headers['Content-Type'].include?('json')
body.encode('utf-8')
else
'"[binary data]"'
end
rescue Encoding::UndefinedConversionError
'"[binary data]"'
end
end
end

I did a slight variation on this one so I could get the responses to show up as pretty printed:

RspecApiDocumentation.configure do |config|
  config.response_body_formatter = lambda do |response_content_type, response_body|
    if response_content_type.include?('application/json')
      return JSON.pretty_generate(JSON.parse(response_body))
    elsif response_content_type.include?('text') || response_content_type.include?('txt')
      # quote it for JSON Parser in documentation reader like APITOME
      return "\"#{response_body}\""
    else
      return '"[binary data]"'
    end
  rescue JSON::ParseError
    '"[binary data]"'
  end
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests