{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":388946490,"defaultBranch":"main","name":"velox","ownerLogin":"facebookincubator","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2021-07-23T23:00:23.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/19538647?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1715581312.0","currentOid":""},"activityList":{"items":[{"before":"a54929be5b4550bb2c0ac8fe9bcc9061ccda7200","after":"b29d93374d8e75a595c85550428eb008a371d456","ref":"refs/heads/main","pushedAt":"2024-05-13T23:31:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add overwrite flag to simple function registration API (#9158)\n\nSummary:\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9158\n\nThe `SimpleFunctionRegistry::registerFunction(aliases, constraints)`\nAPI used to always overwrite the function registry when the function\nname and signature already exists. This diff adds an overwrite flag to\nthis API to control the behavior of overwriting. The overwrite flag is\ntrue by default. The\n`SimpleFunctionRegistry::registerFunction(aliases, constraints, overwrite)`\nAPI returns a bool that is true only when all aliases are successfully\nregistered.\n\nReviewed By: bikramSingh91\n\nDifferential Revision: D55041377\n\nfbshipit-source-id: 76cea41f98de717dd8cea83df7771226d85d8b53","shortMessageHtmlLink":"Add overwrite flag to simple function registration API (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2195921868\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9158\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9158/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9158\">#9158</a>)"}},{"before":"f12c859020b2caa7d10d41d4557112934e5f34f0","after":"a54929be5b4550bb2c0ac8fe9bcc9061ccda7200","ref":"refs/heads/main","pushedAt":"2024-05-13T22:58:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Re-enable spill in aggregation variance test (#9791)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9791\n\nReviewed By: tanjialiang\n\nDifferential Revision: D57284297\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: ed3d3899d15e13c7d05f572056f56a42abfdf3e6","shortMessageHtmlLink":"Re-enable spill in aggregation variance test (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2293412173\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9791\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9791/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9791\">#9791</a>)"}},{"before":"02082e3a72bb2690db6f14a4b673e69f09c86348","after":"f12c859020b2caa7d10d41d4557112934e5f34f0","ref":"refs/heads/main","pushedAt":"2024-05-13T22:28:57.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add WS partial/full throttling signal to Koski (#9742)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9742\n\nReviewed By: Yuhta, DanielMunozT\n\nDifferential Revision: D57083302\n\nfbshipit-source-id: 9bc24383e3897dec23d0c3f30d8105895b55d752","shortMessageHtmlLink":"Add WS partial/full throttling signal to Koski (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2284432405\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9742\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9742/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9742\">#9742</a>)"}},{"before":"f6b07aaa53c91b0f77c844992b3913d08186cf8b","after":"02082e3a72bb2690db6f14a4b673e69f09c86348","ref":"refs/heads/main","pushedAt":"2024-05-13T07:28:27.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Handle buffered dwrf write exception to avoid server crash (#9785)\n\nSummary:\nCatch exception during BufferedWriter processing to avoid server crash as ~BufferedWriter expect either the object is aborted or all the data mutations have been flushed.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9785\n\nReviewed By: tanjialiang\n\nDifferential Revision: D57263986\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: cb1a39bd05751235be493dd0e0ce5a4e77cb0f2b","shortMessageHtmlLink":"Handle buffered dwrf write exception to avoid server crash (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2291791057\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9785\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9785/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9785\">#9785</a>)"}},{"before":null,"after":"8b9ceada89051ccfb4fee2134ccd83dea14a8f84","ref":"refs/heads/dependabot/github_actions/pypa/cibuildwheel-2.18.0","pushedAt":"2024-05-13T06:21:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump pypa/cibuildwheel from 2.12.1 to 2.18.0\n\nBumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.12.1 to 2.18.0.\n- [Release notes](https://github.com/pypa/cibuildwheel/releases)\n- [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md)\n- [Commits](https://github.com/pypa/cibuildwheel/compare/v2.12.1...v2.18.0)\n\n---\nupdated-dependencies:\n- dependency-name: pypa/cibuildwheel\n  dependency-type: direct:production\n  update-type: version-update:semver-minor\n...\n\nSigned-off-by: dependabot[bot] <support@github.com>","shortMessageHtmlLink":"Bump pypa/cibuildwheel from 2.12.1 to 2.18.0"}},{"before":"10a194ea93f6d79e2243307c38b1ee00e63394d5","after":"f6b07aaa53c91b0f77c844992b3913d08186cf8b","ref":"refs/heads/main","pushedAt":"2024-05-13T05:44:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Make documentations format aligned with convetion (#9783)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9783\n\nReviewed By: xiaoxmeng\n\nDifferential Revision: D57259323\n\nPulled By: tanjialiang\n\nfbshipit-source-id: 36f25a5a5ede98a8f13e7cf0b3c37f4b48aa7338","shortMessageHtmlLink":"Make documentations format aligned with convetion (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2291478402\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9783\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9783/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9783\">#9783</a>)"}},{"before":"dacc93e23768ded77ada750a460e009f5259adbc","after":"10a194ea93f6d79e2243307c38b1ee00e63394d5","ref":"refs/heads/main","pushedAt":"2024-05-13T04:25:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove dead code in TableWriteTest.cpp (#9774)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9774\n\nReviewed By: tanjialiang\n\nDifferential Revision: D57259265\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: 3d6f1e18c15b17b5709a086768a7a86e58ea9815","shortMessageHtmlLink":"Remove dead code in TableWriteTest.cpp (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2290571176\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9774\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9774/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9774\">#9774</a>)"}},{"before":"2c98308b4563d0c58ab016708b835bb7fce4a9ce","after":"dacc93e23768ded77ada750a460e009f5259adbc","ref":"refs/heads/main","pushedAt":"2024-05-13T01:04:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Driver should have slicing diabled in serial mode (#9782)\n\nSummary:\nCurrently driver sets the slicing time under serial mode, which is unwanted as in serial mode there is nothing to yield for. This PR sets slicing time to be 0 (meaning slicing/yield disabled) when driver is under serial mode.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9782\n\nReviewed By: xiaoxmeng\n\nDifferential Revision: D57258885\n\nPulled By: tanjialiang\n\nfbshipit-source-id: 5e0892bc7a689bedbd57536c5142c9cd694dab7f","shortMessageHtmlLink":"Driver should have slicing diabled in serial mode (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2291456805\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9782\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9782/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9782\">#9782</a>)"}},{"before":null,"after":"fbc792ef32526fbc3c6677624c990591fa67946a","ref":"refs/heads/tablewritertest_deadcode","pushedAt":"2024-05-11T01:11:14.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"aditi-pandit","name":"Aditi Pandit","path":"/aditi-pandit","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87670623?s=80&v=4"},"commit":{"message":"Remove dead code in TableWriteTest.cpp","shortMessageHtmlLink":"Remove dead code in TableWriteTest.cpp"}},{"before":"ed0ecdd99d5ce6684e8bc60ea96a656b778d5105","after":"2c98308b4563d0c58ab016708b835bb7fce4a9ce","ref":"refs/heads/main","pushedAt":"2024-05-11T00:59:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Clean up legacy code from Buffer::reallocate and reduce window where Buffer is invalid (#9755)\n\nSummary:\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9755\n\nWhen running Velox with the save input on expression setting enabled, I'm occasionally seeing\nSIGSEGVs writing out the Vectors when the exception was thrown while resizing the offset or length\nBuffer in a MapVector or an ArrayVector, e.g. because of an OOM.  When this happens in\nBuffer::reallocate it can leave the Buffer in an invalid state before the exception is caught (when the\nexception message is being generated) triggering the crash.\n\nWhile I doubt we can 100% guarantee VectorSaver won't trigger a crash we can at least fix this case.\n\nInstead of detaching buffer before reallocating it, if we detach it after we still avoid a double free and\nthe buffer is (almost) always valid. There are windows after the old buffer is freed before MemoryPool::reallocate returns, and while setting up newBuffer where buffer would still be invalid,\nbut these are much less likely to run into issues (these would mainly be bugs in the code).\n\nWhile digging through this I also noticed that the comments on MemoryPool::reallocate have gotten\nout of date and Buffer::reallocate is handling some unnecessary cases (newPtr can never be equal to\nold), so I've cleaned those up.\n\nReviewed By: xiaoxmeng\n\nDifferential Revision: D57139357\n\nfbshipit-source-id: b2a0391a77d6ccbdee9e96f3a85bde9ab815b2e9","shortMessageHtmlLink":"Clean up legacy code from Buffer::reallocate and reduce window where …"}},{"before":"35c4fa31f3d39b9daeac90139130a1d369efb6db","after":"ed0ecdd99d5ce6684e8bc60ea96a656b778d5105","ref":"refs/heads/main","pushedAt":"2024-05-11T00:43:19.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Split Arithmetic Functions Registration (#9693)\n\nSummary:\nCompiling a single ArithmeticFunctionsRegistration.cpp consumes a lot of memory.\nSplit the functions into three: MathematicalFunctions, MathematicalOperators, ProbabilityTrigonometricFunctions.\nThese categories are inspired by the Presto documentation and split the functions equally.\nI see a 1/3 memory reduction on my local setup.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9693\n\nReviewed By: bikramSingh91\n\nDifferential Revision: D57026005\n\nPulled By: kgpai\n\nfbshipit-source-id: 4de5bde811e8ec8691bb727ad19314375a1913da","shortMessageHtmlLink":"Split Arithmetic Functions Registration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2276167331\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9693\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9693/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9693\">#9693</a>)"}},{"before":"8184113040c742fbef3d6a3e9d6ffb7b554ef38d","after":"35c4fa31f3d39b9daeac90139130a1d369efb6db","ref":"refs/heads/main","pushedAt":"2024-05-11T00:38:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Selective Nimble reader skeleton (#9768)\n\nSummary:\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9768\n\nX-link: https://github.com/facebookincubator/nimble/pull/51\n\nThe first diff for Nimble selective reader.  Contains:\n- Reader class and factory implementation\n- All primitive type column readers (no dictionary output) and struct column reader\n- `ChuckedDecoder` to accept visitor and apply it on multiple chunks\n- Basic simple implementation of `readWithVisitor` for all testable encodings\n- Unit tests and randomized unit tests covering all primitive types and struct\n\nbypass-github-export-checks\n\nReviewed By: oerling\n\nDifferential Revision: D57162138\n\nfbshipit-source-id: 453b082d7fadeafac261fefa44fb38314f822326","shortMessageHtmlLink":"Selective Nimble reader skeleton (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2290121694\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9768\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9768/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9768\">#9768</a>)"}},{"before":"4e9f9cd30c48bdc0644e26b991e902bf78791ba6","after":"4b45f340699787fa669940780ec4b36ec0d44eab","ref":"refs/heads/gh-pages","pushedAt":"2024-05-10T22:38:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"github-actions[bot]","name":null,"path":"/apps/github-actions","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/15368?s=80&v=4"},"commit":{"message":"Update documentation","shortMessageHtmlLink":"Update documentation"}},{"before":"54a060df1f472a1c00a482d912e71fc71013f018","after":"8184113040c742fbef3d6a3e9d6ffb7b554ef38d","ref":"refs/heads/main","pushedAt":"2024-05-10T22:38:00.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add shuffle Spark function (#9415)\n\nSummary:\nDoc: https://spark.apache.org/docs/latest/api/sql/#shuffle\nCode: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L1257\n\nSpark `shuffle` uses a specific seed for random function, which is the sum\nof the `seed` provided by the user and the configuration `spark.partition_id`,\nthis is the different from presto's behavior. So we need to refactor existing\n`shuffle` to allow it to accept random / custom seed.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9415\n\nReviewed By: mbasmanova\n\nDifferential Revision: D57212806\n\nPulled By: kgpai\n\nfbshipit-source-id: 76fdeea9b30297fe9d0c684f02d5e74da2d1f313","shortMessageHtmlLink":"Add shuffle Spark function (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2233031964\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9415\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9415/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9415\">#9415</a>)"}},{"before":"bdbd2555cbb04249ef1db337a29c4e3811e072f0","after":"54a060df1f472a1c00a482d912e71fc71013f018","ref":"refs/heads/main","pushedAt":"2024-05-10T21:12:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Update aws-sdk-cpp version to 1.11.321(from 1.11.169) (#9756)\n\nSummary:\n`AdaptiveRetryStrategy` and `GetRetryCount()` API is not available in version 1.11.169. We need this version upgrade for `RetryStrategy` support of S3FileSytem.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9756\n\nReviewed By: bikramSingh91\n\nDifferential Revision: D57173421\n\nPulled By: kgpai\n\nfbshipit-source-id: 03081424123251ec23c9ddeb35c70028a0f571c3","shortMessageHtmlLink":"Update aws-sdk-cpp version to 1.11.321(from 1.11.169) (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2286725588\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9756\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9756/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9756\">#9756</a>)"}},{"before":"f1b6ccf0c6618d827f8912ad51ba88ec7b08b144","after":"bdbd2555cbb04249ef1db337a29c4e3811e072f0","ref":"refs/heads/main","pushedAt":"2024-05-10T19:19:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix name of runner used for fuzzer jobs. (#9753)\n\nSummary:\nFixes https://github.com/facebookincubator/velox/issues/9744\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9753\n\nReviewed By: spershin\n\nDifferential Revision: D57131704\n\nPulled By: kgpai\n\nfbshipit-source-id: 9cc91322f1ab6195a076fe1b350c4c02768bb921","shortMessageHtmlLink":"Fix name of runner used for fuzzer jobs. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2286563902\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9753\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9753/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9753\">#9753</a>)"}},{"before":"7b97c9ccec5fb80e017de34fb4edca9e0f4fcdba","after":"4e9f9cd30c48bdc0644e26b991e902bf78791ba6","ref":"refs/heads/gh-pages","pushedAt":"2024-05-10T16:09:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"github-actions[bot]","name":null,"path":"/apps/github-actions","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/15368?s=80&v=4"},"commit":{"message":"Update documentation","shortMessageHtmlLink":"Update documentation"}},{"before":"7d76b1d9ee814c8911d849697071acccf3e18b0c","after":"f1b6ccf0c6618d827f8912ad51ba88ec7b08b144","ref":"refs/heads/main","pushedAt":"2024-05-10T16:08:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Fix spill read runtime stats unit (#9765)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9765\n\nReviewed By: tanjialiang\n\nDifferential Revision: D57196250\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: 97531dee3b90216606a4859b5340b21f684eb2f2","shortMessageHtmlLink":"Fix spill read runtime stats unit (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2288939606\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9765\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9765/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9765\">#9765</a>)"}},{"before":"96500ece283c6868fff4e4702cd0991a9c2baa10","after":"b10d7866636ea6d9d722ebebae74af7a99433de5","ref":"refs/heads/bucketed_table","pushedAt":"2024-05-10T05:50:29.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"aditi-pandit","name":"Aditi Pandit","path":"/aditi-pandit","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87670623?s=80&v=4"},"commit":{"message":"Add support to write bucketed (but not partitioned) tables","shortMessageHtmlLink":"Add support to write bucketed (but not partitioned) tables"}},{"before":"63e7e11c1a453c17674d9e10e09f98f68d609dcf","after":"96500ece283c6868fff4e4702cd0991a9c2baa10","ref":"refs/heads/bucketed_table","pushedAt":"2024-05-10T04:19:12.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"aditi-pandit","name":"Aditi Pandit","path":"/aditi-pandit","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87670623?s=80&v=4"},"commit":{"message":"Add support to write bucketed (but not partitioned) tables","shortMessageHtmlLink":"Add support to write bucketed (but not partitioned) tables"}},{"before":"b62702720a253e93288b07f99aa28ba7a567d611","after":"63e7e11c1a453c17674d9e10e09f98f68d609dcf","ref":"refs/heads/bucketed_table","pushedAt":"2024-05-10T00:01:30.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"aditi-pandit","name":"Aditi Pandit","path":"/aditi-pandit","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87670623?s=80&v=4"},"commit":{"message":"Add support to write bucketed (but not partitioned) tables","shortMessageHtmlLink":"Add support to write bucketed (but not partitioned) tables"}},{"before":"75b597c14bf3665234725810046c62cfcffac990","after":"7b97c9ccec5fb80e017de34fb4edca9e0f4fcdba","ref":"refs/heads/gh-pages","pushedAt":"2024-05-09T23:26:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"github-actions[bot]","name":null,"path":"/apps/github-actions","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/15368?s=80&v=4"},"commit":{"message":"Update documentation","shortMessageHtmlLink":"Update documentation"}},{"before":"49c3ebb2028c8db380a5a414538baba38f14ef7b","after":"7d76b1d9ee814c8911d849697071acccf3e18b0c","ref":"refs/heads/main","pushedAt":"2024-05-09T23:25:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Propagate additional dwrf writer options (#9752)\n\nSummary:\nIn order to tune writer memory usage pattern, we need additional 2 options for writer\nhive.exec.orc.compress.size.min\nhive.exec.orc.linear.stripe.size.heuristics\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9752\n\nReviewed By: HuamengJiang\n\nDifferential Revision: D57131358\n\nPulled By: tanjialiang\n\nfbshipit-source-id: c755948f6263766a504a9427802271a90dc34074","shortMessageHtmlLink":"Propagate additional dwrf writer options (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2286558800\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9752\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9752/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9752\">#9752</a>)"}},{"before":"371a1541c5a80ea532a1087e1217bd92bf3dcd1d","after":"b62702720a253e93288b07f99aa28ba7a567d611","ref":"refs/heads/bucketed_table","pushedAt":"2024-05-09T22:02:24.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"aditi-pandit","name":"Aditi Pandit","path":"/aditi-pandit","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/87670623?s=80&v=4"},"commit":{"message":"Add support to write bucketed (but not partitioned) tables","shortMessageHtmlLink":"Add support to write bucketed (but not partitioned) tables"}},{"before":"3e98d408006f62b234dac34cafaf91e1e748f295","after":"49c3ebb2028c8db380a5a414538baba38f14ef7b","ref":"refs/heads/main","pushedAt":"2024-05-09T17:17:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Row index metadata column support for table scan (#9174)\n\nSummary:\nSpark support people query row_index metadata column of parquet file. Checking this spark part implement -https://github.com/apache/spark/commit/95aebcbf100de1dbedd32626ce67bd01014c973e\n\nhowever, velox doesn't support row_index metadata, below spark query would return null for row index column\n\n```\nselect a, _tmp_metadata_row_index from table;\n```\n\nrelated issue https://github.com/facebookincubator/velox/issues/9165\n\nThe PR introduces a new column handle type `kRowIndex` which can be used to indicate which column is row index column need be generated if you want to add a new column to the results containing the row numbers.  The new column contains row number of type `BIGINT` in the file starting from 0 before any filtering and mutation, and works on all file formats supported.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9174\n\nReviewed By: mbasmanova\n\nDifferential Revision: D56472291\n\nPulled By: Yuhta\n\nfbshipit-source-id: 848693a9ccc5ee5e3279f012d6198721b7691d6f","shortMessageHtmlLink":"Row index metadata column support for table scan (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2197266160\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9174\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9174/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9174\">#9174</a>)"}},{"before":"dceaff0f2dc9a6cf003c75e98b4584b4eeb31802","after":"3e98d408006f62b234dac34cafaf91e1e748f295","ref":"refs/heads/main","pushedAt":"2024-05-09T16:49:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Remove lint warnings in ParquetReaderTest (#9760)\n\nSummary: Pull Request resolved: https://github.com/facebookincubator/velox/pull/9760\n\nReviewed By: mbasmanova\n\nDifferential Revision: D57162713\n\nfbshipit-source-id: 5a19fdd0cc0917cfde5cd8fe7654bd79aac36523","shortMessageHtmlLink":"Remove lint warnings in ParquetReaderTest (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2287910121\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9760\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9760/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9760\">#9760</a>)"}},{"before":"eb356fddc6b8cf15124a95c6808cc0cd5e0dc412","after":"dceaff0f2dc9a6cf003c75e98b4584b4eeb31802","ref":"refs/heads/main","pushedAt":"2024-05-09T16:30:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Move E2EFilterTestBase into tests/utils to be reused by Nimble (#9749)\n\nSummary:\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9749\n\nIn order to reuse this base class in nimble, we need to move it to\n`tests/utils` which will be included in `VELOX_BUILD_MINIMAL_WITH_DWIO`.  Also\nfactor out `ColumnSelector::fromScanSpec` so that we can reuse this logic in\nnimble without depending on `HiveConnectorUtil`.\n\nReviewed By: pedroerp\n\nDifferential Revision: D57114400\n\nfbshipit-source-id: 406413afb924a85d1a6c93c59c8ca5d755f0b725","shortMessageHtmlLink":"Move E2EFilterTestBase into tests/utils to be reused by Nimble (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2286155942\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9749\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9749/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9749\">#9749</a>)"}},{"before":"c325d4fa64dca0ff053a500a30a9503c44c57489","after":"eb356fddc6b8cf15124a95c6808cc0cd5e0dc412","ref":"refs/heads/main","pushedAt":"2024-05-09T15:52:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Create row type for list children when there are >1 child fields (#9533)\n\nSummary:\nAccording to the spec of list backward compatibility [link](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists).\n`If the repeated field is a group with multiple fields, then its type is the element type and elements are required.`\n\nCurrently Parquet reader assumes list type only has one child, which would fail for this backward compatibility case. This change makes it that when there are multiple fields, creating a new row type instance which has all the fields as its children.\n\nThis diff is co-authored with qqibrow .\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9533\n\nReviewed By: mbasmanova\n\nDifferential Revision: D56783371\n\nPulled By: Yuhta\n\nfbshipit-source-id: 50a03e55e8b49ae6f82ca8ae4101b953b60ba5e8","shortMessageHtmlLink":"Create row type for list children when there are &gt;1 child fields (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2251260169\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9533\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9533/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9533\">#9533</a>)"}},{"before":null,"after":"c325d4fa64dca0ff053a500a30a9503c44c57489","ref":"refs/heads/codegen","pushedAt":"2024-05-09T07:25:57.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"duanmeng","name":"macduan","path":"/duanmeng","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5107360?s=80&v=4"},"commit":{"message":"Add async memory reclaim task to avoid recursive arbitration (#9734)\n\nSummary:\nHash join run spill in parallel by using async worker. The spill work might trigger memory allocation\nfrom non-spill memory pool such as lazy io triggered when materializing the column vector to write\nout. We do bypass memory arbitration if the memory allocation is for arbitration by thread-local context.\nHowever, async worker doesn't set it when running on background executor. This causes the recursive\narbitration which will deadlock.\nThis PR fixes the issue by providing createAsyncMemoryReclaimTask utility which helps setup memory\narbitration context and uses it hash join spill. Unit test is added for verification.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9734\n\nReviewed By: bikramSingh91, tanjialiang, tonyxug, oerling\n\nDifferential Revision: D57092581\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: 8c3ab0d838214cf8bb8b657be395c4690c954dd1","shortMessageHtmlLink":"Add async memory reclaim task to avoid recursive arbitration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2282522105\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9734\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9734/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9734\">#9734</a>)"}},{"before":"b8a26ce8ebd58d0cf5503f5559bb7a6b227eec63","after":"c325d4fa64dca0ff053a500a30a9503c44c57489","ref":"refs/heads/main","pushedAt":"2024-05-09T04:33:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"facebook-github-bot","name":"Facebook Community Bot","path":"/facebook-github-bot","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6422482?s=80&v=4"},"commit":{"message":"Add async memory reclaim task to avoid recursive arbitration (#9734)\n\nSummary:\nHash join run spill in parallel by using async worker. The spill work might trigger memory allocation\nfrom non-spill memory pool such as lazy io triggered when materializing the column vector to write\nout. We do bypass memory arbitration if the memory allocation is for arbitration by thread-local context.\nHowever, async worker doesn't set it when running on background executor. This causes the recursive\narbitration which will deadlock.\nThis PR fixes the issue by providing createAsyncMemoryReclaimTask utility which helps setup memory\narbitration context and uses it hash join spill. Unit test is added for verification.\n\nPull Request resolved: https://github.com/facebookincubator/velox/pull/9734\n\nReviewed By: bikramSingh91, tanjialiang, tonyxug, oerling\n\nDifferential Revision: D57092581\n\nPulled By: xiaoxmeng\n\nfbshipit-source-id: 8c3ab0d838214cf8bb8b657be395c4690c954dd1","shortMessageHtmlLink":"Add async memory reclaim task to avoid recursive arbitration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2282522105\" data-permission-text=\"Title is private\" data-url=\"https://github.com/facebookincubator/velox/issues/9734\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/facebookincubator/velox/pull/9734/hovercard\" href=\"https://github.com/facebookincubator/velox/pull/9734\">#9734</a>)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAESOku6AA","startCursor":null,"endCursor":null}},"title":"Activity · facebookincubator/velox"}