All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.
- Add
x-prerender-render-idandx-prerender-render-atmeta tags to header
- Log error if Page.stopLoading fails
- If a request made during rendering gets a 5XX response the render process will be marked as dirty
- Ability to parse content from the shadow DOM
- Fire plugin event
tabNavigatedwhen tab navigation is finished. SaveerrorTextintab.prerender.navigateError - Mark page done if
navigateErrorexists
- Allow enabling of request and JS logging to console with the
PRERENDER_LOG_REQUESTSenvironment variable
- Fix plugin event logging after reject
- Logging if the plugin event lasts more than 10 seconds
- Save if the rendering was timed out in tab.prerender.timedout
- Add option to skip setting
customElements.forcePolyfill = true
- Undefined page outerHTML treated as an error
- Save errors to tab.prerender.errors
- Log initial request
- Ignore EventSource requests when counting the number of requests in flight
- Fix wait for connect call
- Retry CDP websocket connection.
- Upgraded
chrome-remote-interfacefrom0.28.xto0.30.0.
- We didn't wait some promises like
setUserAgentOverride,setBypassServiceWorkerandsetOverrideCertificateErrorsin the past. Under heavy load, we might send requests before these overrides are set properly.
prerenderReadyDelaynow can be configured viareq.prerender.prerenderReadyDelay
- fix:
firstPrerenderReadyTimeshould be evulated in each rendering process, not only the first rendering
- stop rendering after
prerenderReadyset to true.
- modified the
checkIfPageIsDoneLoadingfunction to stop checking the page status if we received a redirect.
- added
timeoutStatusCodetotab.prerenderand tothis.options. We are returning with this status code if a page won't load inpageLoadTimeout.
- added
timeSpentConnectingToBrowser,timeSpentOpeningTab,timeSpentLoadingUrl,timeSpentParsingPage,timeUntilErrortoreq.prerenderto allow for debugging of certain issues with the server spending too much time in different lifecycle sections - added a fix to setHeader warning by splitting headers on any line returns
- added ability to configure the chrome remote debugging port for running more than one instance of chrome on the same server
- added automatic closing of browser alert dialogs
- moved some code from responseReceived down to loadingFinished to help more accurately know when content is done downloading
- removed the deletion of some CSP headers that weren't really causing any issues
- added configurable options for pdf printing to let you override all options necessary using
this.options.pdfOptions - fixed timeouts on redirects
- added ability to override other express options on the
.listen()function by passing in an object now instead of just the port
- We were relying on
document.doctypeto return the full doctype string but that string changed in Chrome v72. We now parse the full doctype object directly in order to build the proper doctype and this change is backwards compatible with older Chrome versions.
- Added
domContentEventFiredso thatcheckIfPageIsDoneLoadingwill wait at least fordomContentEventFiredbefore also waiting for all network requests to finish. This should hopefully take care of any edge cases where a page is saved too early when Chrome doesn't send new network requests during the parsing of a large .js file.
- fixed issue with creating browser tabs in a new context (to clear cookies/local storage)
LOG_REQUESTSshows console logging from the webpage being loaded- fixed
this.options.followRedirecttypo to now bethis.options.followRedirects
- Updated Mocha to 5.2.0, Sinon to 6.1.4 and a few minor package numbers
- Added package-lock.json
- Removed a check for success in the response of
Target.disposeBrowserContextto fix an issue with Chrome 68 removing that response object.
- Removed the
Page.addScriptToEvaluateOnNewDocument({source: 'localStorage.clear()'})since it seemed to be causing an issue with Chrome in some cases. Going to look for a better fix here since our context should be clearing this already.
- For checking if a URL returns a redirect, we were checking to see if the request returning the redirect URL matched which failed in some cases where the encoding of the URL was different in the request. That code now checks the request ID to see if it matches the original request.
- Service worker enable/disable can be enabled/disabled on a per tab basis by setting
req.prerender.enableServiceWorkerin therequestReceivedevent.
- Added ability to turn on/off services workers.
- Added
this.isBrowserConnected = falseinsideserver.restartBrowser()so the prerender server won't try to render any new requests before the browser is actually restarted. Fixes a very small edge case at scale.
- Added
localStorage.clear()on a new page being loaded due to bug in BrowserContext local storage being cleared: https://bugs.chromium.org/p/chromium/issues/detail?id=754576
- Changed
document.getElementsByTagName('html')[0].outerHTMLtodocument.firstElementChild.outerHTMLwhen querying pagehtmlto improve performance.
- Make sure we only call
Buffer.byteLengthon a string to fix an error in newer versions of Node
- Changed
request.loaderIdtorequest.requestIdinrequestWillBeSentdue to issue with Chrome 64 changing loaderId format.
- Added ability for the prerender server to restart Chrome due to some connection issues we've been seeing after a server is running for a few hours.
- Chrome re-uses the original request ID on a redirect so we are saving off the fact that we saw a redirect to make sure we return a correct 301
- Changed dependencies from ^ to ~ to make the semver more specific
- Added removal of
<link rel="import" src="">tags after the page is done loading to theremoveScriptTagsplugin. Imported HTML can have script tags in it, and since it's already been rendered to the page we can safely remove it when running that plugin.
- Added
if (window.customElements) customElements.forcePolyfill = true,ShadyDOM = {force: true}, andShadyCSS = {shimcssproperties: true}to fix Polymer app rendering.
- Added back
res.setHeaderfor plugins to use
- Set
renderTypetohtmlfor non "/render" endpoint
- Added Headless Chrome as a rendering engine!
- Added new event types:
requestReceived,tabCreated,pageLoaded. - Added new Prerender server option:
chromeLocation - Added ability to request jpg and png screenshots
- Added ability to request pdf export of a page
- Added ability to request HAR file of page load times
- Removed PhantomJS and all references to it
- Removed old event types:
beforePhantomRequest,onPhantomPageCreate,afterPhantomRequest,beforeSend - Removed In Memory Cache (moved to new repo)
- Removed S3 HTML Cache (moved to new repo)
- Removed Prerender server options that are no longer needed:
workers,iterations,softIterations,cookiesEnabled,pageDoneCheckTimeout,resourceDownloadTimeout,jsTimeout,noJsExecutionTimeout,evaluateJavascriptCheckTimeout
See the Readme.me for in depth descriptions of all of the new changes!
- Whoops. Make sure
shouldEncodeURLBeforeBrowserFetchdefaults to true.
- Added
shouldEncodeURLBeforeBrowserFetchto allow projects that use prerender to determine whether they want to callencodeURIon the URL before fetching it in PhantomJS. Useful for some URLs that might have encoded slashes in them, since encoding them further would cause incorrect behavior.
- Fixed issue where PhantomJS crashed and then disposing caused bad phantomjs state
- Bumped all dependency versions to latest
- Added ability for cluster master to kill last known phantomjs pid for a worker if the worker dies (preventing orphaned phantomjs instances)
- Better terminating for cluster workers
- Added NUM_SOFT_ITERATIONS to try to kill phantomjs and reclaim memory when no requests are in flight. This should be set to a low number (1-10) so that PhantomJS can be restarted often when it isn't doing anything else. NUM_ITERATIONS should still be set to something like 40-50 to make sure to force kill PhantomJS even if a request is in flight.
- Added clearing of memory cache to prevent PhantomJS from returning a 304
- Fixed issue where prerender-status-code set to
200was causing the page to skip being cached - Fixed an issue where we weren't using the correct pid when trying to force kill PhantomJS.
- Moved clearing of memory cache and local storage to before the page loads instead of after. This will prevent edge cases that could cause a 304.
- Fixed issue where S3HtmlCache was calling next() before finishing saving to the cache
- Fixed issue where we were calling
hasOwnPropertyon aquerystringthat no longer had Object on it's prototype chain
- Fixed issue where a webpage calling window.close would cause Prerender to be unable to shutdown PhantomJS properly
- S3 cache plugin was incorrectly saving non-200 status code responses to the cache
- preserve phantom arguments when server is restarting
- use default when phantomArguments is empty
- prevent multiple phantomjs instances from being started with low number of iterations
- try to check to see if phantomjs has actually been disposed. if not, force kill it.
- added engines to package.json and fixed possible bug in checking options passed in
- prevent weird hangup on error setting a header with a newline
- make sure we catch any errors thrown from phridge and continue
- kill workers (and phantomjs) on SIGTERM
- Changed
PHANTOM_CLUSTER_NUM_WORKERStoPRERENDER_NUM_WORKERSin server.js - Changed
PHANTOM_WORKER_ITERATIONStoPRERENDER_NUM_ITERATIONSin server.js - Switched from
phantomjs-nodebridge tophridge - All Prerender plugins that access PhantomJS need to be rewritten to support new phridge integration. For example, change this:
req.prerender.page.set('onConsoleMessage', function(msg) {
console.log(msg);
});
to this:
req.prerender.page.run(function() {
this.onConsoleMessage = function(msg) {
console.log(msg);
};
});
Please see phridge for more info on how to interact with PhantomJS through phridge.
###Removed
- Removed
PHANTOM_CLUSTER_BASE_PORTsincephridgedoesn't start a webserve to talk to PhantomJS, so it's no longer needed. - Removed
PHANTOM_CLUSTER_MESSAGE_TIMEOUTsincephridgedoesn't start a webserve to talk to PhantomJS, so it's no longer needed.