Request page PDF generation broken

[gbp]

We're unable to generate PDFs of the request page when generating request zip downloads.

There seems to be several issues effecting this which have been hidden to us due to Alaveteli transparently falling back to a txt only request page.

The default command is wkhtmltopdf.

The first issue seemed to stem from the command output. Alaveteli assumes the command shouldn't output anything to stdout, if it does then it has failed.

alaveteli/app/controllers/request_controller.rb

Lines 680 to 686 in a787d6e

	if !output.nil?
	file_info = { :filename => 'correspondence.pdf',
	:data => File.open(tmp_output.path).read }
	done = true
	else
	logger.error("Could not convert info request #{info_request.id} to PDF with command '#{convert_command} #{tmp_input.path} #{tmp_output.path}'")
	end

This appears to be an upstream change in the command. We attempted to fix this by adding the -q quiet command flag to limit output: #6257

This isn't ideal and means re-users couldn't specify there preferred alternative to wkhtmltopdf.

Also we really we shouldn't assume success/failure based on if there was output or not. We have tarting looking at refactoring AlaveteliExternalCommand so we catch errors in a better way: #6432

During this latest refactoring, we actually got wkhtmltopdf command to run but it would still warn and error with output like:

Warning: Blocked access to file
Error: Failed to load about:blank, with network status code 301 and http status code 0 - Protocol "about" is unknown
Exit with code 1 due to network error: ProtocolUnknownError

The requests for about:blank come from how wkhtmltopdf now works. By default in v 0.12.6 it will block local file access and any requests to local files now gets redirected to about:blank. This is erroring due to the about protocol not being handled wkhtmltopdf/wkhtmltopdf#4460, there is a unmerged PR to address this wkhtmltopdf/wkhtmltopdf#5031.

To get around this people seem to be passing the --enable-local-file-access command flag but there could be security concerns with this approach. Personally I feel uneasy out using this flag.

As a test I enabled local file access but there are still more problems:

Error: Failed to load file://fonts.googleapis.com/css?family=Source+Sans+Pro:400,600,700,400italic, with network status code 302 and http status code 0 - Request for opening non-local file file://fonts.googleapis.com/css?family=Source+Sans+Pro:400,600,700,400italic
Warning: Failed to load file:///assets/application-d79707b9d116ae2b9ea132076a72b97273de3c5278c0829f75c535c31e9a7a0e.js (ignore)
Warning: Failed to load file:///assets/request-attachments-bbc9b56bbe5a258b5c8ed43a32d0de342c32aaa5d8b6273dcff9204156c64246.js (ignore)
Warning: Failed to load file:///assets/hamburger-69694b0ff70afebfeccc1586611fb7b186aa746730747ab1262304bda49f1386.png (ignore)
Warning: Failed to load file:///assets/status/status-sprite-4e22eaf9319519a3c9549d5406b3c474fbb62daa1f54137379c148fac6c8c8d9.png (ignore)
Warning: Failed to load file:///assets/correspondence-collapse/collapse-bbdf936773dca505bc2d42dce338f8b5201356e555b7bd96630bd78e50bfb477.png (ignore)
Warning: Failed to load file:///assets/delivery-status/unknown-555d738f8f55699a83ff304cc579e1196ff1b99a80dd7c29428a7828fb0963e8.png (ignore)
Warning: Failed to load file:///assets/loader-000-on-fff-16px-b428ce2459f5218683af04fee7b23fe7eb53d084928fc151fcad022c72943de7.gif (ignore)
Exit with code 1 due to network error: ProtocolInvalidOperationError

It looks like we need to ensure there won't be any local file access at all. There are some other flags (--no-images, --disable-javascript, --print-media-type, ...) which might help us generate PDFs.

[gbp]

On the Alaveteli staging deployment its possible to download a request page ZIP which contains a PDF. So why does this work and can we get fixed without overhauling the PDF generation?

On the server the:

$ /usr/bin/wkhtmltopdf-static --version
Name:
  wkhtmltopdf 0.11.0 rc1

Locally on my Mac:

> wkhtmltopdf --version
wkhtmltopdf 0.12.6 (with patched qt)

So the server version doesn't include the breaking change which blocks local filesystem access by default which got introduced with 0.12.6. So this will only be an issue once we upgrade the package /cc @sagepe

The staging site is current set the AtEU theme, I wonder if there is something in the WDTK theme which causes some output to stdout and prevents the PDF from being generated. This might get fixed by #6432 but first I'll attempt to replicate the issue by switching staging to the WDTK theme.

[gbp]

With staging running WDTK theme is still is able to generate PDFs.

I have attempted to use WDTK production console to get the HTML source:

c = RequestController.new
c.instance_variable_set :@info_request, InfoRequest.find_by(url_title: '...')
c.instance_variable_set :@render_to_file, true
c.request = ActionDispatch::Request.new("rack.input" => "")
c.instance_variable_set :@locales, { :current => AlaveteliLocalization.locale, :available => [] }
c.action_name = 'show'
File.write 'tmp/pdf_source.html', c.render_to_string(template: 'request/show')

Running this through wkhtmltopdf-static and we get an error.

$ wkhtmltopdf-static pdf_source.html out.pdf
Loading pages (1/6)
QSslSocket: cannot resolve CRYPTO_num_locks                  ] 10%
QSslSocket: cannot resolve CRYPTO_set_id_callback
QSslSocket: cannot resolve CRYPTO_set_locking_callback
QSslSocket: cannot resolve sk_free
QSslSocket: cannot resolve sk_num
QSslSocket: cannot resolve sk_pop_free
QSslSocket: cannot resolve sk_value
QSslSocket: cannot resolve SSL_library_init
QSslSocket: cannot resolve SSL_load_error_strings
QSslSocket: cannot resolve SSLv2_client_method
QSslSocket: cannot resolve SSLv3_client_method
QSslSocket: cannot resolve SSLv23_client_method
QSslSocket: cannot resolve SSLv2_server_method
QSslSocket: cannot resolve SSLv3_server_method
QSslSocket: cannot resolve SSLv23_server_method
QSslSocket: cannot resolve OPENSSL_add_all_algorithms_noconf
QSslSocket: cannot resolve OPENSSL_add_all_algorithms_conf
QSslSocket: cannot resolve SSLeay
QSslSocket: cannot call unresolved function CRYPTO_num_locks
QSslSocket: cannot call unresolved function CRYPTO_set_id_callback
QSslSocket: cannot call unresolved function CRYPTO_set_locking_callback
QSslSocket: cannot call unresolved function SSL_library_init
QSslSocket: cannot call unresolved function SSLv23_client_method
QSslSocket: cannot call unresolved function sk_num
QSslSocket: cannot call unresolved function SSLv23_client_method3%
QSslSocket: cannot call unresolved function SSL_library_init
Counting pages (2/6)
Resolving links (4/6)
Loading headers and footers (5/6)
Printing pages (6/6)
QPixmap: Cannot create a QPixmap when no GUI is being used   ] Page 1 of 3
QPixmap: Cannot create a QPixmap when no GUI is being used
QPainter::begin: Paint device returned engine == 0, type: 2
QPainter::setRenderHint: Painter must be active to set rendering hints
QPainter::translate: Painter not active
QPainter::setPen: Painter not active
QPainter::setBrush: Painter not active
QPainter::setWorldTransform: Painter not active
QPainter::setOpacity: Painter not active
QPainter::setFont: Painter not active
Segmentation fault

[gbp]

The QSslSocket lines can be resolved by removing the Google Analytics script tag. On staging Google Analytics isn't configured so the script tag isn't included.

[gbp]

The other QPixmap and QPainter lines seem can be resolved by removing opacity CSS properties. Unsure why staging doesn't have this problem.

[gbp]

The elements with the opacity seem to be located in the popup site banner.

[gbp]

On staging my user account had dismissed the popup site banner so the issue with opacity CSS isn't present.

When removing my dismissal AnnouncementDismissal.destroy_all I can now replicate the error on staging 🎉

This fix could be as simple as:

diff --git a/app/views/layouts/default.html.erb b/app/views/layouts/default.html.erb
index 04c9bbd12..057b8cd95 100644
--- a/app/views/layouts/default.html.erb
+++ b/app/views/layouts/default.html.erb
@@ -52,11 +52,13 @@
     <div class="entirebody">
 
     <a href="#content" class="show-with-keyboard-focus-only skip-to-link" tabindex="0">Skip to content</a>
+      <% unless @render_to_string %>
       <!-- begin popups -->
         <%= render 'general/site_wide_announcement' if site_wide_announcement.present? %>
         <div id="country-message">
         </div>
       <!-- end popups -->
+      <% end %>
 
       <%= render :partial => 'general/responsive_header' %>
 
@@ -81,6 +83,7 @@
       <%= render :partial => 'general/responsive_footer' %>
     </div>
 
+    <% unless @render_to_string %>
     <%= render partial: 'ga_code' unless @user&.is_admin? %>
 
     <%= javascript_include_tag "application" %>
@@ -109,6 +112,7 @@
       </script>
     <% end %>
     <%= content_for :javascript %>
+    <% end %>
 
     <%= render :partial => 'general/before_body_end' %>
 

[gbp]

Deployed, moved aside the previously cached PDFs and the confirmed the PDF generation now works 🎉

[gbp]

For macOS user we can install v0.11.0 rc1 of wkhtmltopdf from https://github.com/wkhtmltopdf/obsolete-downloads/blob/master/README.md

I'll open a ticket for supporting v0.12.x