{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":452825945,"defaultBranch":"main","name":"multimodal","ownerLogin":"facebookresearch","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2022-01-27T20:01:00.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/16943930?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1717426984.0","currentOid":""},"activityList":{"items":[{"before":"dbeed9724bc9099be173c86871df7d41f3b7e58c","after":"e4d288b45b89cee462a21ab264405f3f368adc21","ref":"refs/heads/main","pushedAt":"2024-04-25T23:41:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"set bias=True for linear layer (#527)\n\nSummary:\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/527\n\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/526\n\n1. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear).\n2. add configuration of using MLP instead of attention pooler for vision adapter;\n\nReviewed By: Bellaktris\n\nDifferential Revision:\nD55897450\n\nPrivacy Context Container: 303860477774201\n\nfbshipit-source-id: 8e012b0c3d37566364f216dbfa8aec389142afe1","shortMessageHtmlLink":"set bias=True for linear layer (#527)"}},{"before":"c261d7199f6e0ba2af5f217b4a4f141921861f88","after":"dbeed9724bc9099be173c86871df7d41f3b7e58c","ref":"refs/heads/main","pushedAt":"2024-03-03T01:43:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"apply Black 2024 style in fbcode (4/16)\n\nSummary:\nFormats the covered files with pyfmt.\n\npaintitblack\n\nReviewed By: aleivag\n\nDifferential Revision: D54447727\n\nfbshipit-source-id: 8844b1caa08de94d04ac4df3c768dbf8c865fd2f","shortMessageHtmlLink":"apply Black 2024 style in fbcode (4/16)"}},{"before":"5a6a283825ab5a733da086e812eee35ce5136052","after":"c261d7199f6e0ba2af5f217b4a4f141921861f88","ref":"refs/heads/main","pushedAt":"2024-02-22T16:44:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"update mypy version (#524)\n\nSummary:\nOur CI is failing ([ex](https://github.com/facebookresearch/multimodal/actions/runs/7934357313/job/21665167793?pr=521)) due to a mismatch in the mypy version used with pytorch core. Update mypy version to fix it\n\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/524\n\nTest Plan:\nFresh install and run mypy\n\n```\nconda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia\npip install -e \".[dev]\"\nmypy\nSuccess: no issues found in 135 source files\n```\n\nCI is green now\n\nReviewed By: kartikayk\n\nDifferential Revision: D54050126\n\nPulled By: ebsmothers\n\nfbshipit-source-id: 922e0a555c14af6f2f27953603b8d280c513f5ff","shortMessageHtmlLink":"update mypy version (#524)"}},{"before":"2cbab1f089ae84b557bf19bd91ff03d4efd4b1d9","after":"5a6a283825ab5a733da086e812eee35ce5136052","ref":"refs/heads/main","pushedAt":"2024-02-22T16:38:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix docstring of CLIP model (#523)\n\nSummary:\nFix typo and missing description for `layers`\n\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/523\n\nTest Plan: Fixes #{issue number}\n\nReviewed By: kartikayk\n\nDifferential Revision: D54049570\n\nPulled By: ebsmothers\n\nfbshipit-source-id: ffe6f21cc0eb448cb7bb67d1d11f0ac765263c2f","shortMessageHtmlLink":"Fix docstring of CLIP model (#523)"}},{"before":"1cccc584d0c4b6c632ace3cf9426609b03aef276","after":"2cbab1f089ae84b557bf19bd91ff03d4efd4b1d9","ref":"refs/heads/main","pushedAt":"2024-02-16T20:12:12.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"create configuration file for MaMMUT training (#521)\n\nSummary:\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/521\n\nMostly based on original coca and https://github.com/lucidrains/MaMMUT-pytorch\n\nUpdate the logics of loading checkpoint for MaMMUT text decoder as well.\n\nDifferential Revision:\nD52891614\n\nPrivacy Context Container: 303860477774201\n\nfbshipit-source-id: 192a1826fd59a80bf99e8545408e19938069a599","shortMessageHtmlLink":"create configuration file for MaMMUT training (#521)"}},{"before":"6bf3779a064dc72cde48793521a5be151695fc62","after":"1cccc584d0c4b6c632ace3cf9426609b03aef276","ref":"refs/heads/main","pushedAt":"2024-02-15T20:49:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Replace torch.jit.instance with instance to assist torch.compile (#522)\n\nSummary:\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/522\n\ntorch.jit.instance is mainly needed to refine Generics, but in this case instance is sufficient, instance unlike torch.jit.instance is also playing nicely with torch.compile\n\nReviewed By: ebsmothers\n\nDifferential Revision: D53797660\n\nfbshipit-source-id: b8e4c5542be866bb09dd331ece4725ae8f57dfcf","shortMessageHtmlLink":"Replace torch.jit.instance with instance to assist torch.compile (#522)"}},{"before":"63c629a031675fe8604f88cbc2145aa00ae410f0","after":"6bf3779a064dc72cde48793521a5be151695fc62","ref":"refs/heads/main","pushedAt":"2024-01-20T04:30:19.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add optional field for multimodal pooled embeddings (#519)\n\nSummary:\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/519\n\nMaMMUT (in next diff) could output the multimodal pooled embeddings directly.\n\nReviewed By: ebsmothers, satyanshukla\n\nDifferential Revision: D52821534\n\nfbshipit-source-id: a1251365384f03dcdadab7d2984fdc5d277ca26b","shortMessageHtmlLink":"Add optional field for multimodal pooled embeddings (#519)"}},{"before":"fc92ceacbda410491d969d6271778721bec21063","after":"63c629a031675fe8604f88cbc2145aa00ae410f0","ref":"refs/heads/main","pushedAt":"2024-01-04T04:22:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fixes for CoCa cascaded attention poolers (#518)\n\nSummary:\nA couple fixes to CoCa's attention pooling as pointed out in https://github.com/facebookresearch/multimodal/issues/517. Specifically, we need to change the input dim for the contrastive pooler to match the output dim from the captioning pooler in the case of cascaded attention pooling. We should also set `n_queries=1` for the contrastive pooler so that the pooled embeddings can be directly fed into contrastive loss (after appropriate normalization).\n\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/518\n\nTest Plan:\n```\nfrom torchmultimodal.models.coca.coca_model import coca_vit_l_14\nmodel = coca_vit_l_14()\nbs, c, h, w, seq_len, vocab_size = 2, 3, 224, 224, 77, 49408\nimages = torch.randn(bs, c, h, w)\ntexts = torch.randint(0, vocab_size, (bs, seq_len))\nout = model(images, texts)\nprint(out.image_pooled_output.shape, out.multimodal_embeddings.shape)\n...\ntorch.Size([2, 1, 768]) torch.Size([2, 76, 49408])\n```\n\nAdd new unit test:\n\n```\npython -m pytest -v tests/models/coca/test_coca_model.py\n...\n===== 4 passed in 3.18s ======\n```\n\nReviewed By: pbontrager\n\nDifferential Revision: D52523771\n\nPulled By: ebsmothers\n\nfbshipit-source-id: 7c0197605e478ae6e3204f1ec0ab2e6adbf2377e","shortMessageHtmlLink":"Fixes for CoCa cascaded attention poolers (#518)"}},{"before":"3ef5f891c0d3b5a3379609531b5558f1ca5bc2a8","after":"2a2e742198d46a07fb1859d6694bd3ff0eb03f12","ref":"refs/heads/coca-fixes","pushedAt":"2024-01-03T22:19:56.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"Add test case for cascaded attention pooler","shortMessageHtmlLink":"Add test case for cascaded attention pooler"}},{"before":null,"after":"3ef5f891c0d3b5a3379609531b5558f1ca5bc2a8","ref":"refs/heads/coca-fixes","pushedAt":"2024-01-03T01:22:28.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"Fixes for CoCa cascaded attention poolers","shortMessageHtmlLink":"Fixes for CoCa cascaded attention poolers"}},{"before":"17e4099eddfc20f7d588a9774ddc87d6a3582b10","after":null,"ref":"refs/heads/gh/ebsmothers/25/head","pushedAt":"2023-12-17T15:21:54.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"6a6252d9de8c7462b1a3d31f4790c1110cc0c60b","after":null,"ref":"refs/heads/gh/ebsmothers/25/orig","pushedAt":"2023-12-17T15:21:54.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"af325a4d914434378120e898af3e7c7615a01c96","after":null,"ref":"refs/heads/gh/ebsmothers/25/base","pushedAt":"2023-12-17T15:21:53.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"79e7a030ec055589279364f3e5b7c59051557e06","after":null,"ref":"refs/heads/gh/ebsmothers/24/orig","pushedAt":"2023-12-17T15:21:53.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"ac69977ca49a053a382d05ef112eb15ea6eced0c","after":null,"ref":"refs/heads/gh/ebsmothers/24/head","pushedAt":"2023-12-17T15:21:53.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"6433f8fce39992a20eb629cff2203f27b35d6b0d","after":null,"ref":"refs/heads/gh/ebsmothers/24/base","pushedAt":"2023-12-17T15:21:52.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"fd42096e8b90b80569948cb44dfccf2bf1bdb773","after":"fc92ceacbda410491d969d6271778721bec21063","ref":"refs/heads/main","pushedAt":"2023-12-05T22:29:05.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"update readme (#512)\n\nSummary: Pull Request resolved: https://github.com/facebookresearch/multimodal/pull/512\n\nReviewed By: pbontrager\n\nDifferential Revision: D51864202\n\nPulled By: ebsmothers\n\nfbshipit-source-id: 8d364b6ccbfe558c365924035a5fa32ec74c2916","shortMessageHtmlLink":"update readme (#512)"}},{"before":"6433f8fce39992a20eb629cff2203f27b35d6b0d","after":"fd42096e8b90b80569948cb44dfccf2bf1bdb773","ref":"refs/heads/main","pushedAt":"2023-11-28T00:29:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Allow non-zero text pad token for Clip text transform (#516)\n\nSummary: Pull Request resolved: https://github.com/facebookresearch/multimodal/pull/516\n\nReviewed By: pbontrager\n\nDifferential Revision: D51514007\n\nfbshipit-source-id: a5b361dc40d6db9d2c50d111cae18f8d3327e79d","shortMessageHtmlLink":"Allow non-zero text pad token for Clip text transform (#516)"}},{"before":"fa81663ce6717f2f0fe150fec7bc077c5b52e9bb","after":null,"ref":"refs/heads/gh/ebsmothers/22/orig","pushedAt":"2023-11-19T15:23:11.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"b979282ab207f57221f41da248bac8ad178e6fb7","after":null,"ref":"refs/heads/gh/ebsmothers/22/head","pushedAt":"2023-11-19T15:23:11.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"eb775d18f0931ff83b3e777a6cec4e3fcac9e3db","after":null,"ref":"refs/heads/gh/ebsmothers/22/base","pushedAt":"2023-11-19T15:23:11.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"}},{"before":"3bc0375d2f0bf89fe3577fe50f55f09db43498ef","after":"6a6252d9de8c7462b1a3d31f4790c1110cc0c60b","ref":"refs/heads/gh/ebsmothers/25/orig","pushedAt":"2023-11-16T21:55:30.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add another shape comment\n\nghstack-source-id: b913cc8ba73477a7c44103a9e7818afaaa0a25ac\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/514","shortMessageHtmlLink":"add another shape comment"}},{"before":"7530e6a247221fb6720290047719102f6bde3420","after":"79e7a030ec055589279364f3e5b7c59051557e06","ref":"refs/heads/gh/ebsmothers/24/orig","pushedAt":"2023-11-16T21:55:30.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add shape comment\n\nghstack-source-id: c180752c77198d936cb3564efe54b5771264bdcd\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/513","shortMessageHtmlLink":"add shape comment"}},{"before":"e0108870a64302698dee3e4cb7d3eb506a9c139a","after":"17e4099eddfc20f7d588a9774ddc87d6a3582b10","ref":"refs/heads/gh/ebsmothers/25/head","pushedAt":"2023-11-16T21:55:28.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"Update on \"add another shape comment\"\n\n\n\n\n[ghstack-poisoned]","shortMessageHtmlLink":"Update on \"add another shape comment\""}},{"before":"e9d0d035168793d288016e176eb45501cdb7c927","after":"ac69977ca49a053a382d05ef112eb15ea6eced0c","ref":"refs/heads/gh/ebsmothers/24/head","pushedAt":"2023-11-16T21:55:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"Update on \"add shape comment\"\n\n\n\n\n[ghstack-poisoned]","shortMessageHtmlLink":"Update on \"add shape comment\""}},{"before":"e9d0d035168793d288016e176eb45501cdb7c927","after":"af325a4d914434378120e898af3e7c7615a01c96","ref":"refs/heads/gh/ebsmothers/25/base","pushedAt":"2023-11-16T21:55:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"Update base for Update on \"add another shape comment\"\n\n\n\n\n[ghstack-poisoned]","shortMessageHtmlLink":"Update base for Update on \"add another shape comment\""}},{"before":null,"after":"3bc0375d2f0bf89fe3577fe50f55f09db43498ef","ref":"refs/heads/gh/ebsmothers/25/orig","pushedAt":"2023-11-16T21:48:26.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add another shape comment\n\nghstack-source-id: 34b6b6024ea915e5a94a47a63fdd3a7d52b278ec\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/514","shortMessageHtmlLink":"add another shape comment"}},{"before":null,"after":"7530e6a247221fb6720290047719102f6bde3420","ref":"refs/heads/gh/ebsmothers/24/orig","pushedAt":"2023-11-16T21:48:26.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add shape comment\n\nghstack-source-id: 7373630cb100912924e6569d0405de6c8415cc9f\nPull Request resolved: https://github.com/facebookresearch/multimodal/pull/513","shortMessageHtmlLink":"add shape comment"}},{"before":null,"after":"e0108870a64302698dee3e4cb7d3eb506a9c139a","ref":"refs/heads/gh/ebsmothers/25/head","pushedAt":"2023-11-16T21:48:20.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add another shape comment\n\n[ghstack-poisoned]","shortMessageHtmlLink":"add another shape comment"}},{"before":null,"after":"e9d0d035168793d288016e176eb45501cdb7c927","ref":"refs/heads/gh/ebsmothers/25/base","pushedAt":"2023-11-16T21:48:20.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ebsmothers","name":null,"path":"/ebsmothers","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24319399?s=80&v=4"},"commit":{"message":"add shape comment\n\n[ghstack-poisoned]","shortMessageHtmlLink":"add shape comment"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEOqrMRQA","startCursor":null,"endCursor":null}},"title":"Activity ยท facebookresearch/multimodal"}