{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":309258998,"defaultBranch":"main","name":"browsertrix-crawler","ownerLogin":"webrecorder","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2020-11-02T04:37:14.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/13686290?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1715532127.0","currentOid":""},"activityList":{"items":[{"before":"7493ceb55041b161b6e2dd3d9b5ca7000687286e","after":"642a727a59227f49daf3d2dd64a0fdb765c30a50","ref":"refs/heads/gh-pages","pushedAt":"2024-05-12T16:44:05.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"github-actions[bot]","name":null,"path":"/apps/github-actions","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/15368?s=80&v=4"},"commit":{"message":"Deployed c71274d with MkDocs version: 1.6.0","shortMessageHtmlLink":"Deployed <a class=\"commit-link\" data-hovercard-type=\"commit\" data-hovercard-url=\"https://github.com/webrecorder/browsertrix-crawler/commit/c71274d8415be669d3066a4b55206fec7437f57a/hovercard\" href=\"https://github.com/webrecorder/browsertrix-crawler/commit/c71274d8415be669d3066a4b55206fec7437f57a\"><tt>c71274d</tt></a> with MkDocs version: 1.6.0"}},{"before":"6f880951e923bffdbb5467ae3e6a9b625351bfbf","after":null,"ref":"refs/heads/store-region-env-var","pushedAt":"2024-05-12T16:42:07.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"}},{"before":"d2fbe7344f4595ebcab7aef2fd20157458c58aa9","after":"c71274d8415be669d3066a4b55206fec7437f57a","ref":"refs/heads/main","pushedAt":"2024-05-12T16:42:04.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"add STORE_REGION env var to be able to specify region (#565)\n\ndefaults to us-east-1 for minio compatibility\r\nfixes #515","shortMessageHtmlLink":"add STORE_REGION env var to be able to specify region (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2283143667\" data-permission-text=\"Title is private\" data-url=\"https://github.com/webrecorder/browsertrix-crawler/issues/565\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/webrecorder/browsertrix-crawler/pull/565/hovercard\" href=\"https://github.com/webrecorder/browsertrix-crawler/pull/565\">#565</a>)"}},{"before":"262856906fd09b9d7badddcc2d0598fd5608a26a","after":"6f880951e923bffdbb5467ae3e6a9b625351bfbf","ref":"refs/heads/store-region-env-var","pushedAt":"2024-05-09T09:13:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"update docs","shortMessageHtmlLink":"update docs"}},{"before":"3ca35259cf55bbb8370518d16b4e7c327f0149ae","after":null,"ref":"refs/heads/skip-empty-frames","pushedAt":"2024-05-09T09:05:39.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"}},{"before":"bd5368cbcae013fc6f29581ddb031a595a87eac4","after":"d2fbe7344f4595ebcab7aef2fd20157458c58aa9","ref":"refs/heads/main","pushedAt":"2024-05-09T09:05:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"Skip Checking Empty Frame + eval timeout (#564)\n\nDon't run frame.evaluate() on an empty frame, also add a timeout just in\r\ncase to frame.evaluate().","shortMessageHtmlLink":"Skip Checking Empty Frame + eval timeout (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2283078561\" data-permission-text=\"Title is private\" data-url=\"https://github.com/webrecorder/browsertrix-crawler/issues/564\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/webrecorder/browsertrix-crawler/pull/564/hovercard\" href=\"https://github.com/webrecorder/browsertrix-crawler/pull/564\">#564</a>)"}},{"before":"cba796178bb312f93429aa74035679aedf1bf567","after":"8e31c7fb8c6737a3c66b744ac6610b4722403ba2","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-08T19:44:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"One more round","shortMessageHtmlLink":"One more round"}},{"before":"46a66e09d36708888ba11cd9fad79aa47a2de153","after":"cba796178bb312f93429aa74035679aedf1bf567","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-08T18:45:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Console log errors in seed tests","shortMessageHtmlLink":"Console log errors in seed tests"}},{"before":"0e34e83224ac4b41544892fdf088cb810a78eea0","after":"46a66e09d36708888ba11cd9fad79aa47a2de153","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-08T17:53:37.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Make sure --failOnFailedSeed results in exit code of 1\n\nWhen pages are unreachable due to DNS not resolving, the crawler\nwas failing with exit code 17, which conflicts with expected and\ndocumented behavior.","shortMessageHtmlLink":"Make sure --failOnFailedSeed results in exit code of 1"}},{"before":"8c64cbd335195354229bd37ee156e916b91562b1","after":"0e34e83224ac4b41544892fdf088cb810a78eea0","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-08T17:46:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Make sure --failOnFailedSeed results in exit code of 1","shortMessageHtmlLink":"Make sure --failOnFailedSeed results in exit code of 1"}},{"before":"0d03ffe338a4221e429e9b5b1fadcf863a51b89f","after":"262856906fd09b9d7badddcc2d0598fd5608a26a","ref":"refs/heads/store-region-env-var","pushedAt":"2024-05-07T12:39:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"undo get-folder-size change, still incorrect types","shortMessageHtmlLink":"undo get-folder-size change, still incorrect types"}},{"before":null,"after":"0d03ffe338a4221e429e9b5b1fadcf863a51b89f","ref":"refs/heads/store-region-env-var","pushedAt":"2024-05-07T12:26:34.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"add STORE_REGION env var to be able to specify region, defaults to us-east-1 by default\nfixes #515\ndependency: update get-folder-size in storage.js to latest, fix type errors","shortMessageHtmlLink":"add STORE_REGION env var to be able to specify region, defaults to us…"}},{"before":"e62fe136bcf0484b007c452fe5025de604ca7f31","after":"2292c9cfc21baba72e399a9c20a7181296fa157a","ref":"refs/heads/bump-base-image-1.64.122","pushedAt":"2024-05-07T12:00:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"bump to 1.65.126\npuppeteer-core to 22.8.0","shortMessageHtmlLink":"bump to 1.65.126"}},{"before":"ddc3e104dbb783ddf56f6f9d00c49f1ffce42f4b","after":"bd5368cbcae013fc6f29581ddb031a595a87eac4","ref":"refs/heads/main","pushedAt":"2024-05-07T11:46:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"version: bump to 1.1.2","shortMessageHtmlLink":"version: bump to 1.1.2"}},{"before":"66395da4b208c75e76c56c8e2206e4cdf5cbb8dd","after":"3ca35259cf55bbb8370518d16b4e7c327f0149ae","ref":"refs/heads/skip-empty-frames","pushedAt":"2024-05-07T11:45:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"add timed run to all frame.evaluate() just in case","shortMessageHtmlLink":"add timed run to all frame.evaluate() just in case"}},{"before":null,"after":"66395da4b208c75e76c56c8e2206e4cdf5cbb8dd","ref":"refs/heads/skip-empty-frames","pushedAt":"2024-05-07T09:55:29.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"skip empty url frames: likely frame is not initialized yet and running frame.evaluate() never returns\nskip such frames altogether for now","shortMessageHtmlLink":"skip empty url frames: likely frame is not initialized yet and runnin…"}},{"before":"36f008451ca5a8a422581a69054bfca05a337062","after":"8c64cbd335195354229bd37ee156e916b91562b1","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-06T18:00:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Expand failOnFailedSeed help text","shortMessageHtmlLink":"Expand failOnFailedSeed help text"}},{"before":"29903e30556b4f43ab42b1b1385501e893caee76","after":"36f008451ca5a8a422581a69054bfca05a337062","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-06T17:52:29.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Fix --failOnFailedSeed regression\n\nRegardless of whether --failOnInvalidStatus is set, a 4xx/5xx\nresponse for a seed page should always fail the crawl if the\nuser has passed --failOnFailedSeed","shortMessageHtmlLink":"Fix --failOnFailedSeed regression"}},{"before":"81388e6c5556fdad6e4326d512aec0b2c04415a9","after":"29903e30556b4f43ab42b1b1385501e893caee76","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-06T17:35:36.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Always fail crawl if failOnFailedSeed set and seed is 4xx/5xx","shortMessageHtmlLink":"Always fail crawl if failOnFailedSeed set and seed is 4xx/5xx"}},{"before":null,"after":"81388e6c5556fdad6e4326d512aec0b2c04415a9","ref":"refs/heads/issue-563-fail-on-failed-seed-regression","pushedAt":"2024-05-06T17:33:25.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"Always fail crawl if failOnFailedSeed set and seed is 4xx/5xx","shortMessageHtmlLink":"Always fail crawl if failOnFailedSeed set and seed is 4xx/5xx"}},{"before":"98bc453c41156ad2434c7c94ff72cf0a3f7ba73e","after":null,"ref":"refs/heads/check-all-frameids-for-worker","pushedAt":"2024-05-06T15:04:34.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"}},{"before":"22b2136eb96a56239e1fcef7772ec728e7892954","after":"ddc3e104dbb783ddf56f6f9d00c49f1ffce42f4b","ref":"refs/heads/main","pushedAt":"2024-05-06T15:04:31.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"tw4l","name":"Tessa Walsh","path":"/tw4l","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6758804?s=80&v=4"},"commit":{"message":"improved handling of requests from workers: (#562)\n\nOn sites with regular workers, requests from workers were being skipped\r\nas there was no match for the worker frameId.\r\n\r\nAdd recorder.hasFrame() frameId to match not just service-worker\r\nframeIds but also other frame ids already tracked in the frameIdToExecId\r\nmap.","shortMessageHtmlLink":"improved handling of requests from workers: (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2280515018\" data-permission-text=\"Title is private\" data-url=\"https://github.com/webrecorder/browsertrix-crawler/issues/562\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/webrecorder/browsertrix-crawler/pull/562/hovercard\" href=\"https://github.com/webrecorder/browsertrix-crawler/pull/562\">#562</a>)"}},{"before":null,"after":"98bc453c41156ad2434c7c94ff72cf0a3f7ba73e","ref":"refs/heads/check-all-frameids-for-worker","pushedAt":"2024-05-03T15:57:23.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"worker fetch:\n- add recorder.hasFrame() frameId to match not just service-worker frameIds but all frameIds, store in frameIdToExecId on the worker data","shortMessageHtmlLink":"worker fetch:"}},{"before":"9a361fe500f0cb3b7ebf74e872d40473e04f4867","after":null,"ref":"refs/heads/more-profile-page-load-wait","pushedAt":"2024-05-02T15:55:26.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"}},{"before":"a61206fd73595d8ca3aaf99ce2125aad27493072","after":"22b2136eb96a56239e1fcef7772ec728e7892954","ref":"refs/heads/main","pushedAt":"2024-05-02T15:55:22.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"profiles: ensure initial page.load() is awaited (#561)\n\nrefactor to create a startLoad() method and await it, follow-up to #559","shortMessageHtmlLink":"profiles: ensure initial page.load() is awaited (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2275185059\" data-permission-text=\"Title is private\" data-url=\"https://github.com/webrecorder/browsertrix-crawler/issues/561\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/webrecorder/browsertrix-crawler/pull/561/hovercard\" href=\"https://github.com/webrecorder/browsertrix-crawler/pull/561\">#561</a>)"}},{"before":null,"after":"9a361fe500f0cb3b7ebf74e872d40473e04f4867","ref":"refs/heads/more-profile-page-load-wait","pushedAt":"2024-05-02T08:29:31.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"profiles: ensure initial page.load() is awaited, just await startLoad(), follow-up to #559","shortMessageHtmlLink":"profiles: ensure initial page.load() is awaited, just await startLoad…"}},{"before":"9afa9e8009772e7c73fca629f064c0b43d328400","after":null,"ref":"refs/heads/profile-catch-page-goto","pushedAt":"2024-04-25T07:35:00.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"}},{"before":"15d2b0975785dfaeda165252f53b834f1af2b455","after":"a61206fd73595d8ca3aaf99ce2125aad27493072","ref":"refs/heads/main","pushedAt":"2024-04-25T07:34:57.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"profiles: ensure all page.goto() promises have at least catch block or are awaited (#559)\n\nIn particular, an API call to /navigate starts, but doesn't wait for a\r\npage load to finish, since user can choose to close the profile browser\r\nat any time. This ensures that user operations don't cause the browser to crash if\r\npage.goto() is interrupted/fails (browser closed, profile is saved, etc...) while a page is still loading.\r\n\r\nbump to 1.1.1","shortMessageHtmlLink":"profiles: ensure all page.goto() promises have at least catch block o…"}},{"before":null,"after":"9afa9e8009772e7c73fca629f064c0b43d328400","ref":"refs/heads/profile-catch-page-goto","pushedAt":"2024-04-25T07:13:39.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"profiles: ensure all page.goto() promises have at least catch block/are awaited, to handle possibly interruptions\n(user closing browser, committing profile, etc..) without causing a crash","shortMessageHtmlLink":"profiles: ensure all page.goto() promises have at least catch block/a…"}},{"before":"423e5652124753a87bcd5e27fe4982f88d47c669","after":"ebe8b387473a4924e16ff86155288350be0aa596","ref":"refs/heads/add-warc-info","pushedAt":"2024-04-19T17:39:50.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"ikreymer","name":"Ilya Kreymer","path":"/ikreymer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1015759?s=80&v=4"},"commit":{"message":"Merge branch 'main' into add-warc-info","shortMessageHtmlLink":"Merge branch 'main' into add-warc-info"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAER8DMCgA","startCursor":null,"endCursor":null}},"title":"Activity · webrecorder/browsertrix-crawler"}