@dafinel Can you try this directly at
https://mozilla.github.io/pdf.js/web/viewer.html#page=2
and let me know.

Looks like an issue of underlying component.

I have a somewhat similar issue. I think it's worth mentioning here instead of creating a new issue.

I have a few buttons on a page that load a different document in the same PDF viewer. The PDF are loaded as byte array too, as in the original issue. The code looks like this :

  @ViewChild('pdfViewer', { static: false }) public pdfViewer: PdfJsViewerComponent;
  @ViewChild('pdfViewerContainer', { static: false }) public pdfViewerContainer: ElementRef<HTMLElement>;

  displayPDF(pdfIndex: number): void {
    this.pdfIndex = pdfIndex;
    this.getPDF(pdfIndex).subscribe((file: any) => {
      this.pdfViewer.pdfSrc = file;
      this.pdfViewer.zoom = 'page-fit';
      this.pdfViewer.refresh();
      setTimeout(() => {
        this.pdfViewerContainer.nativeElement.scrollIntoView({ behavior: 'smooth', block: 'nearest' });
      }, 0);
    });
  }

Everything is loaded fine and work as intended, but when I click on the back button of my browser, it stays on the same page and simply goes back to the previous file that was loaded in the PDF viewer.
This is true on every browser I've tried, plus I get the issue mentioned by OP that PDF content is cleared when on Firefox.

Regards.

Same issue in Chrome, when i click browser Back button it stays on the same browser page.

Same Issue here. Seems to be caused by the iFrame the PDF viewer is in and strange Behavior of Firefox to include iFrame URL changes in the back/forward History.
A possible workaround is mentioned here: officert/vue-friendly-iframe#20 (comment)