{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":69863074,"defaultBranch":"master","name":"arbor","ownerLogin":"arbor-sim","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2016-10-03T11:03:22.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/24656287?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1706526014.0","currentOid":""},"activityList":{"items":[{"before":"08a157fb52455c3a45b7ffae1bf6aabd33a43023","after":"9f20e8380e58ee2a61c243254d2d0c093ecf666d","ref":"refs/heads/master","pushedAt":"2024-04-03T15:34:19.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Remove `simulation::inject_events` (#2265)\n\nThis removes `simulation::inject_events` which is made obsolete by\r\n`recipe::event_generators`.\r\nThe method exposed internals by giving access to the `lid` of synapses\r\n(over labels) and broke\r\nabstractions (`gid` must be present on this MPI rank and all events must\r\nbe deliverable during\r\nthe current time (as per documentation)). It was used only in a few\r\ntests and examples and these\r\nuses were replaced by `event_generator`s.","shortMessageHtmlLink":"Remove simulation::inject_events (#2265)"}},{"before":"ed67763233d274a3d3209025285fb4b834df763d","after":"08a157fb52455c3a45b7ffae1bf6aabd33a43023","ref":"refs/heads/master","pushedAt":"2024-03-05T14:34:57.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Unify loading of Standard Morphology Files. (#2218)\n\nAll morphology loaders now return a `loaded_morphology` object\r\ncomprising\r\n- `morphology`: the morphology corresponding to the data on disk\r\n- `segment_tree`: the raw segment tree\r\n- `label_dict`: all labels defined in the file\r\n- `asc`/`swc`: `axon`, `soma`, `dend`, and `apic` based on tag 1, 2, 3,\r\n4.\r\n- `nml`: the union of the label dicts contained in the metadata,\r\nprefixed by `grp:`, `nmd:`, and `seg:`.\r\n- `metadata`: a set of items given by the fileformat\r\n - `swc`: nothing\r\n - `asc`: a list of spines and marker sets\r\n - `nml`: \r\n - `named_segments` a label dict\r\n - `segment_groups` a label dict `name` -> segment ids\r\n - `group_segments` a map from segment id to group name(s)\r\n - `segments` a label dict, one entry per id\r\n - `cell_id` id of the cell defining this morphology, if any\r\n - `id` id of the morphology\r\n\r\nNote that loading neuroml still goes via the `neuroml` object and the\r\n`cell_morphology`/`morphology`\r\naccessors return an optional `loaded_morphology`.\r\n\r\n## ⚠️ Breaking ⚠️ \r\n\r\n- Removed the `load_*_raw` functions, as `loaded_morphology` bundles the\r\nsegment tree.\r\n- The interface of `load_*` now has an extra indirection.\r\n- The interface of `neuroml` changed.\r\n- `label_dict::import` is now called `extend` for future safety (=C++\r\nmodules) and Python; calling things `import` (=a keyword) is not well\r\nreceived by `black` et al.\r\n\r\n## Issues\r\n\r\nCloses #1981","shortMessageHtmlLink":"Unify loading of Standard Morphology Files. (#2218)"}},{"before":"d84083da61facb6db658d7f858d0ed2e6227cf75","after":"ed67763233d274a3d3209025285fb4b834df763d","ref":"refs/heads/master","pushedAt":"2024-02-27T13:49:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Improve `partition_load_balance` (#2206)\n\nSpin-off from #2005. Make the primary load balancing cleaner and faster\r\nand more maintainable.\r\nThus:\r\n- remove all MPI calls, this is now purely local\r\n- remove temporary data structures and/or coral them into their own\r\nlittle scopes\r\n- simplify `super_cells` vs `regular_cells`\r\n- sort less\r\n- sparse connection tables\r\n\r\nPartially inspired by external feedback","shortMessageHtmlLink":"Improve partition_load_balance (#2206)"}},{"before":"22c771182d90742ae315ffda0f11ea04eb4a7c57","after":"d84083da61facb6db658d7f858d0ed2e6227cf75","ref":"refs/heads/master","pushedAt":"2024-02-26T10:01:46.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"update deprecated build instructions for cscs-ci (#2256)","shortMessageHtmlLink":"update deprecated build instructions for cscs-ci (#2256)"}},{"before":"f7205975947622724a3269147e383d0f2bea56ec","after":"22c771182d90742ae315ffda0f11ea04eb4a7c57","ref":"refs/heads/master","pushedAt":"2024-02-26T07:33:17.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"fix auto detect of sve length (#2257)\n\n- apply pertinent compiler flags when trying to compile the detection\r\ncode\r\n- improve compiler error message when sve length is not set","shortMessageHtmlLink":"fix auto detect of sve length (#2257)"}},{"before":"e7fb95bf8a1334138ad20fd56b794d01d6a58db6","after":"f7205975947622724a3269147e383d0f2bea56ec","ref":"refs/heads/master","pushedAt":"2024-02-22T15:40:03.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Polish CodeQL and do some chores (#2255)\n\nPolish CodeQL integration:\r\n- Fix C++ warnings.\r\n- _Ignore_ current Python warnings:\r\n - `tsplot` is not processing html\r\n - `json/.../serve.py` is not ours\r\n- Add a status badge to README\r\n- Bump GH Actions as required.\r\n- Black was unhappy about our new type stubs?!","shortMessageHtmlLink":"Polish CodeQL and do some chores (#2255)"}},{"before":"60445a43c524585cae9322759341bc615f9ad6a4","after":"e7fb95bf8a1334138ad20fd56b794d01d6a58db6","ref":"refs/heads/master","pushedAt":"2024-02-06T12:29:18.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Create codeql.yml (#2220)\n\nSee what happens when we enable the default CodeQL template.","shortMessageHtmlLink":"Create codeql.yml (#2220)"}},{"before":"3f0d076408c1ea4104a64de43181672166baf3db","after":"ed5cd735cce6ce5dd81bd3e2a9d50f8bfd735d8f","ref":"refs/heads/fix/windows-cmake","pushedAt":"2024-01-30T11:10:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"change to gitattributes approach","shortMessageHtmlLink":"change to gitattributes approach"}},{"before":null,"after":"3f0d076408c1ea4104a64de43181672166baf3db","ref":"refs/heads/fix/windows-cmake","pushedAt":"2024-01-29T11:00:14.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"Avoid `bash\\r` from shebang by calling `bash` directly on custom command","shortMessageHtmlLink":"Avoid bash\\r from shebang by calling bash directly on custom command"}},{"before":"4892d048a7930757205af47facc2729eaff154be","after":null,"ref":"refs/heads/single-cell-example-lineno-fix","pushedAt":"2024-01-29T10:28:44.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"}},{"before":"9f18a505adfcd55ada466917cd55debf01ce427c","after":"60445a43c524585cae9322759341bc615f9ad6a4","ref":"refs/heads/master","pushedAt":"2024-01-19T13:46:52.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Ux/units (#2233)\n\nThe core issue here is to add units to the user facing API. I decided on\r\nusing the LLNL/units\r\nlibrary, which offers conversion and checking at runtime. Runtime is a\r\nrequirement -- as much\r\nas I love static guarantees --, but keeping the interface uniform\r\nbetween Python and C++ is a\r\nmust.\r\n\r\nWhile setting this up, I noticed the severe lack of IDE/LSP support for\r\nArbor, so I added typing\r\nstubs using https://github.com/sizmailov/pybind11-stubgen. The\r\nconjunction of typing and units\r\nexposed misuse of pybind11 in several places, so next I had to massage\r\nthe ordering of bindings,\r\nadjust the specification of default arguments, and add the odd missing\r\nbinding.\r\n\r\nThe schedule/event generator interface was tightened up, hiding the\r\n`*_impl` structs and exposing\r\nonly the type erased `schedule` object. That in turn required\r\nde-generification of the Poisson\r\nschedule. Now, Mersenne twister is the only choice and I will remove\r\nthat later on for the CBRNG\r\nwe are already using elsewhere.\r\n\r\nCurrently, units are used for:\r\n- [X] simulation\r\n- [X] schedule/generator\r\n- [x] paintables\r\n- [X] placeables\r\n - [X] iclamp\r\n - [X] threshold\r\n- [X] connections\r\n- [X] gap junctions\r\n\r\nAdding units to mechanism interfaces is _interesting_ but requires more\r\nwork and thought, so\r\nI'll defer that to a later point in time. We'd need to adjust modcc to\r\nexpose and **check** units\r\nand devise a scheme to handle missing units.\r\n\r\nGeneric TODOs; some might spin off into separate issues.\r\n- [x] ~~rename py::iclamp OR cpp::i_clamp for consistency~~ covered by\r\n#2239\r\n- [x] use scale/base for iexpr paintables for consistency with\r\nscaled_mech\r\n- [x] ~~Use CBRNG for Poisson schedule~~ covered by #2243 \r\n- [ ] Automate stub generation. A wishlist item, requires installing\r\nextra software.\r\n- [x] Properly integrate units w/ spack. NB. Units doesn't have a\r\nspackage.\r\n\r\nCloses #1983 \r\nCloses #2032\r\n\r\n---------\r\n\r\nCo-authored-by: boeschf <48126478+boeschf@users.noreply.github.com>","shortMessageHtmlLink":"Ux/units (#2233)"}},{"before":"93fb291b44ef8477dbebe35091dee4d8e3ebf191","after":"9f18a505adfcd55ada466917cd55debf01ce427c","ref":"refs/heads/master","pushedAt":"2024-01-19T08:51:03.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Fix #2250 by using a consistent naming strategy. (#2251)\n\nUse the `module_name` string to name files written by `modcc` consistently.\r\n\r\nFixes #2250","shortMessageHtmlLink":"Fix #2250 by using a consistent naming strategy. (#2251)"}},{"before":"21f5029015f65e9e2fe4ea9b229b9924a469d305","after":"93fb291b44ef8477dbebe35091dee4d8e3ebf191","ref":"refs/heads/master","pushedAt":"2024-01-11T14:04:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"disable spack latest_release (#2246)\n\nThe spack CI test fails when a point release (`latest_release`) for\r\n`spack` is selected, but works for the `develop` branch. This is most\r\nlikely connected to the external `spack` buildcache, maintained by spack\r\ndevelopers (https://github.com/spack/github-actions-buildcache).\r\nTherefore, we disable the `latest_release` case until a workaround is\r\nimplemented or a compatible `spack` buildcache is available.","shortMessageHtmlLink":"disable spack latest_release (#2246)"}},{"before":"d4579b12992900fa920e029cabda7d9b0ef442c3","after":"21f5029015f65e9e2fe4ea9b229b9924a469d305","ref":"refs/heads/master","pushedAt":"2024-01-11T13:40:57.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"boeschf","name":null,"path":"/boeschf","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/48126478?s=80&v=4"},"commit":{"message":"fix gpu warp intrinsics (#2247)\n\n`reduce_by_key` depends on warp level intrinsics to transfer values\r\nbetween different threads (lanes) participating in the reduction. The\r\npertinent intrinsic is `__shfl_down_sync` which is accessed through\r\nArbor's wrapper function `shfl_down`. However, the contribution from\r\neach thread to the reduction was erroneously truncated to an integer\r\nvalue. This PR fixes the signature of the respective wrapper functions\r\nand modifies the unit test in order to check that floating point\r\nreductions are not truncated.\r\nWhile cleaning up the cuda code path, the workaround using two 32-bit\r\nshuffle instructions for 64 bit data types (doubles) was removed - this\r\nwas probably a leftover from cuda versions prior to 9.0.","shortMessageHtmlLink":"fix gpu warp intrinsics (#2247)"}},{"before":"5a965a3075dcf16f6cf5a1071085a7992b589919","after":"d4579b12992900fa920e029cabda7d9b0ef442c3","ref":"refs/heads/master","pushedAt":"2023-12-22T13:10:57.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"AdhocMan","name":"Simon Frasch","path":"/AdhocMan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26937327?s=80&v=4"},"commit":{"message":"Doc upgrades (#2229)\n\n- Polish & modernise two cells + gap junction example.\r\n- Investigate problems with mathjax\r\n- Update spike data format docs.\r\n- Switches theme to FURO, which is better maintained and document (like\r\n... at all)\r\n- with the new theme, we gain dark mode, yet we need to fix our graphics\r\nto make it a really nice experience.\r\n\r\nFixes #2227 \r\nFixes #2228\r\n\r\n---------\r\n\r\nCo-authored-by: Simon Frasch ","shortMessageHtmlLink":"Doc upgrades (#2229)"}},{"before":"ce6dd9d59dde8e8efaa35bac4f7d58c8469dbd8c","after":null,"ref":"refs/heads/Helveg-patch-2","pushedAt":"2023-12-08T11:12:42.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"}},{"before":"af0e286d92a308bde3e4608eedcf01de6944445f","after":"5a965a3075dcf16f6cf5a1071085a7992b589919","ref":"refs/heads/master","pushedAt":"2023-12-08T07:47:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Update `segment` to `msegment` as name present in arbor module (#2237)\n\nIt seems the `segment` class has been renamed to `msegment`, I updated\r\nthe morphology docs to reflect this (probably breaking a couple labels\r\nelsewhere?)","shortMessageHtmlLink":"Update segment to msegment as name present in arbor module (#2237)"}},{"before":"562ba9ebd14eba04bcdd6a48c37243b3461d681f","after":"af0e286d92a308bde3e4608eedcf01de6944445f","ref":"refs/heads/master","pushedAt":"2023-12-08T07:46:11.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Backport tinyopt pr#16 EOF comparison fix (#2242)\n\n* Use `int` for `istream::get()` result for correct EOF checks when char\r\nis unsigned.\r\n\r\nFixes #2059.","shortMessageHtmlLink":"Backport tinyopt pr#16 EOF comparison fix (#2242)"}},{"before":"1e4bd830e23c00e2da5825bd988d67b5de1c45e0","after":"562ba9ebd14eba04bcdd6a48c37243b3461d681f","ref":"refs/heads/master","pushedAt":"2023-12-08T07:45:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Fix a few integer type mismatches in for loops. (#2241)\n\n* Use size_t to iterate size_t-bounded loops in `internal_hash()`.\r\n* Use unsigned to iterate unsigned-bounded loops in `test_probe.cpp`.\r\n\r\nFixes #2240","shortMessageHtmlLink":"Fix a few integer type mismatches in for loops. (#2241)"}},{"before":"a1831e3a842ffd3689b8287f5101477f231ee5cc","after":null,"ref":"refs/heads/Helveg-patch-1","pushedAt":"2023-12-05T22:36:38.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"}},{"before":"f8853fc120efd1ccc9dca2b0d4340cd2cbe5967a","after":"1e4bd830e23c00e2da5825bd988d67b5de1c45e0","ref":"refs/heads/master","pushedAt":"2023-12-05T22:36:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"Updated cable_cell constructor parameter order (#2236)","shortMessageHtmlLink":"Updated cable_cell constructor parameter order (#2236)"}},{"before":"c391775009cf394b5aae0b9051c2ba3b4109fa55","after":"f8853fc120efd1ccc9dca2b0d4340cd2cbe5967a","ref":"refs/heads/master","pushedAt":"2023-12-05T14:30:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Fix missing plot code (#2234)\n\nIt was missing the plot instructions","shortMessageHtmlLink":"Fix missing plot code (#2234)"}},{"before":null,"after":"ce6dd9d59dde8e8efaa35bac4f7d58c8469dbd8c","ref":"refs/heads/Helveg-patch-2","pushedAt":"2023-12-05T14:24:50.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"Update `segment` to `msegment` as name present in arbor module","shortMessageHtmlLink":"Update segment to msegment as name present in arbor module"}},{"before":null,"after":"a1831e3a842ffd3689b8287f5101477f231ee5cc","ref":"refs/heads/Helveg-patch-1","pushedAt":"2023-12-05T14:07:00.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"Updated cable_cell constructor parameter order","shortMessageHtmlLink":"Updated cable_cell constructor parameter order"}},{"before":null,"after":"4892d048a7930757205af47facc2729eaff154be","ref":"refs/heads/single-cell-example-lineno-fix","pushedAt":"2023-12-05T11:57:49.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Helveg","name":"Robin De Schepper","path":"/Helveg","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/28923979?s=80&v=4"},"commit":{"message":"Fix missing plot code","shortMessageHtmlLink":"Fix missing plot code"}},{"before":"db868d31b1f9c8a6fdd7646942499cd5a835eb4f","after":"c391775009cf394b5aae0b9051c2ba3b4109fa55","ref":"refs/heads/master","pushedAt":"2023-11-29T12:19:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Switch `std::string` -> hashes for label resolution. (#2215)\n\nUse the simple but well-known FNV-1a hash function to map\r\n`cell_tag_type` aka `std::string` to an `uint64_t`\r\nfor label resolution. The former type has a size of 32B or more and the\r\nlatter 8B, thus cutting the storage and bandwidth\r\nrequirements by 3/4. \r\n\r\nThe hash function is implemented from the reference given on the\r\nauthors' website/wikipedia and is extremely\r\nsimple. If we ever experience issues, we might consider switching this\r\nto something of higher quality via an\r\nexternal library, candidates are `xxHASH` and `Murmur3`.\r\n\r\nhttps://github.com/Cyan4973/xxHash\r\n\r\nNote: This should further relieve the memory pressure on larger scale\r\nsimulation as formulated in #1969 and make\r\n#2005 less urgent.\r\n\r\nThere is no performance impact (at laptop scale), but the memory savings\r\nare worth it.","shortMessageHtmlLink":"Switch std::string -> hashes for label resolution. (#2215)"}},{"before":"29d84bd95329bebcc171bcb1feeca3625ddfb125","after":"db868d31b1f9c8a6fdd7646942499cd5a835eb4f","ref":"refs/heads/master","pushedAt":"2023-11-24T11:20:23.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"rename py_recipe to recipe. (#2232)\n\nSolve name mismatch Python ./. C++.\r\n\r\nCloses #1634","shortMessageHtmlLink":"rename py_recipe to recipe. (#2232)"}},{"before":"65290d65f915aac6778ed41abf297869c468cfc5","after":"29d84bd95329bebcc171bcb1feeca3625ddfb125","ref":"refs/heads/master","pushedAt":"2023-11-21T12:14:16.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"📊 Improve spike delivery (#2222)\n\n# Introduction \r\n\r\nIssue #502 claims that under certain circumstances, spikes walking can\r\nbecome a bottleneck.\r\nL2 cache misses are posited as the root cause. The setup required for\r\nthis is pretty extreme\r\nthough.\r\n\r\n# Baseline\r\nSelected to show the problem while being feasible to run on a laptop.\r\n\r\nInput\r\n```json\r\n{\r\n \"name\": \"test\",\r\n \"num-cells\": 256,\r\n \"duration\": 400,\r\n \"min-delay\": 10,\r\n \"fan-in\": 10000,\r\n \"realtime-ratio\": 0.01,\r\n \"spike-frequency\": 40,\r\n \"threads\": 4,\r\n \"ranks\": 8000\r\n}\r\n```\r\n\r\nTiming\r\n```\r\n❯ hyperfine 'bin/drybench ~/src/arbor/example/drybench/params.json'\r\nBenchmark 1: bin/drybench ~/src/arbor/example/drybench/params.json\r\n Time (mean ± σ): 4.334 s ± 0.146 s [User: 6.026 s, System: 0.418 s]\r\n Range (min … max): 4.148 s … 4.667 s 10 runs\r\n```\r\n\r\n# Changes\r\n\r\n- Store connection list as structure of arrays instead of array of\r\nstructures. This reduces cache misses during binary search for the\r\ncorrect source.\r\n- Use `lower bound` instead of `equal range`. Removes one binary search,\r\nwhich is cache-unfriendly.\r\n- Treat all spikes from the same source in one go. Keeps all values\r\naround instead of discarding and re-acquiring.\r\n- Swap member order in `spike_event` reduces size from 24 to 16 bytes.\r\n\r\n# Outcome\r\n\r\nWe get some minor reduction in runtime on my local machine\r\n```\r\n❯ hyperfine 'bin/drybench ../example/drybench/params.json'\r\nBenchmark 1: bin/drybench ../example/drybench/params.json\r\n Time (mean ± σ): 4.225 s ± 0.167 s [User: 5.939 s, System: 0.397 s]\r\n Range (min … max): 4.064 s … 4.632 s 10 runs\r\n```\r\n\r\n4.064s vs 4.148s, but this is still within one $\\sigma$.\r\n\r\n# Routes not taken\r\n\r\n## Using a faster `sort`\r\nTried `pdqsort`, but sorting isn't a real bottleneck and/or `pdq`\r\ndoesn't\r\nimprove on our problem.\r\n\r\n## Using a faster `lower_bound`\r\nSimilarly to `sort` no variation of `lower_bound` improves measurably.\r\nWe can conclude that the enqueuing is bound by the actual pushing of\r\nevents into cells' queues. As noted in #502, L2 cache misses are a\r\nprobable\r\nroot cause.\r\n\r\n## Building a temporary buffer of events\r\nTried to create a scratch space to dump all events into, keyed by their\r\nqueue index.\r\nTo avoid allocations, the scratch buffer \r\nThen, we sort this by index and build the queues in one go. Proved a\r\nsignificant\r\nslow-down.\r\n\r\n## SoA-Splitting Spikes\r\nDoesn't improve our main bottleneck and `spike` is pretty small (=not\r\nmuch\r\nwaste on a cacheline) already.\r\n\r\n## Multithreading the appends\r\nThat'll only worsen the amount of cache misses as L2 is shared between\r\nthreads.\r\nAlso if cache misses are our problem, this won't address the root cause.\r\n\r\n## Adding a pre-filter to spike processing\r\nReducing the incoming spikes by tracking all sources terminating at the\r\nlocal process\r\ndidn't yield an improvement while rejecting ~25% of all incoming event.\r\nInstead, a significant\r\nslow-down was observed.\r\n\r\n# Juwels Booster\r\n## Input Deck and Configuration\r\n- JuwelsBooster develbooster queue\r\n- CMake\r\n ```\r\ncmake .. -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DARB_USE_BUNDLED_LIBS=ON\r\n-DARB_VECTORIZE=ON -DARB_WITH_PYTHON=OFF -DARB_WITH_MPI=ON\r\n-DBUILD_SHARED_LIBS=OFF -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON\r\n-DARB_PROFILIING=ON -DCMAKE_INSTALL_PREFIX=../install -G Ninja\r\n-DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DARB_GPU=cuda\r\n-DCMAKE_BUILD_TYPE=release\r\n ```\r\n- Input\r\n ```json\r\n {\r\n \"name\": \"test\",\r\n \"num-cells\": 8000,\r\n \"duration\": 200,\r\n \"min-delay\": 10,\r\n \"fan-in\": 5000,\r\n \"realtime-ratio\": 0.01,\r\n \"spike-frequency\": 50,\r\n \"threads\": 4,\r\n \"ranks\": 10000\r\n }\r\n ```\r\n\r\n## Validation\r\n```\r\nbenchmark parameters:\r\n name: test\r\n cells per rank: 8000\r\n duration: 200 ms\r\n fan in: 5000 connections/cell\r\n min delay: 10 ms\r\n spike freq: 50 Hz\r\n cell overhead: 0.01 ms to advance 1 ms\r\nexpected:\r\n cell advance: 16 s\r\n spikes: 800000000\r\n events: 3204710400\r\n spikes: 2000 per interval\r\n events: 151969 per cell per interval\r\nHW resources:\r\n threads: 4\r\n ranks: 10000\r\n```\r\nand\r\n```\r\n808110000 spikes generated at rate of 10 spikes per cell\r\n```\r\n## Baseline\r\n### Summary\r\n```\r\n---- meters -------------------------------------------------------------------------------\r\nmeter time(s) memory(MB)\r\n-------------------------------------------------------------------------------------------\r\nmodel-init 113.175 288.951\r\nmodel-run 107.741 1306.015\r\nmeter-total 220.916 1594.966\r\n```\r\n\r\n### Profiler\r\n```\r\nREGION CALLS WALL THREAD %\r\nroot - 78.541 314.163 100.0\r\n communication - 36.991 147.964 47.1\r\n enqueue - 21.247 84.986 27.1\r\n sort 320000 19.952 79.807 25.4\r\n merge 320000 1.174 4.695 1.5\r\n setup 320000 0.121 0.483 0.2\r\n walkspikes 40 15.735 62.940 20.0\r\n exchange - 0.010 0.038 0.0\r\n sort 40 0.006 0.022 0.0\r\n gatherlocal 40 0.004 0.016 0.0\r\n gather 40 0.000 0.000 0.0\r\n remote 40 0.000 0.000 0.0\r\n post_process 40 0.000 0.000 0.0\r\n```\r\n\r\n### Perf\r\n```\r\n Performance counter stats for 'bin/drybench ../example/drybench/params.json':\r\n\r\n 245572.30 msec task-clock:u # 1.000 CPUs utilized\r\n 0 context-switches:u # 0.000 /sec\r\n 0 cpu-migrations:u # 0.000 /sec\r\n 29077724 page-faults:u # 118.408 K/sec\r\n 730063012768 cycles:u # 2.973 GHz (83.33%)\r\n 25160708320 stalled-cycles-frontend:u # 3.45% frontend cycles idle (83.33%)\r\n 332882140632 stalled-cycles-backend:u # 45.60% backend cycles idle (83.33%)\r\n 1080323993228 instructions:u # 1.48 insn per cycle\r\n # 0.31 stalled cycles per insn (83.33%)\r\n 243594581289 branches:u # 991.946 M/sec (83.33%)\r\n 3447791949 branch-misses:u # 1.42% of all branches (83.33%)\r\n\r\n 245.609154303 seconds time elapsed\r\n\r\n 218.071863000 seconds user\r\n 25.114123000 seconds sys\r\n```\r\n\r\n## Feature Branch\r\n### Summary\r\n```\r\n---- meters -------------------------------------------------------------------------------\r\nmeter time(s) memory(MB)\r\n-------------------------------------------------------------------------------------------\r\nmodel-init 112.901 939.580\r\nmodel-run 84.730 871.134\r\nmeter-total 197.631 1810.714\r\n```\r\n\r\n### Profiler\r\n```\r\nREGION CALLS WALL THREAD %\r\nroot - 71.717 286.869 100.0\r\n communication - 30.408 121.633 42.4\r\n enqueue - 20.006 80.023 27.9\r\n sort 320000 18.984 75.938 26.5\r\n merge 320000 0.937 3.746 1.3\r\n setup 320000 0.085 0.340 0.1\r\n walkspikes 40 10.401 41.605 14.5\r\n exchange - 0.001 0.005 0.0\r\n sort 40 0.001 0.004 0.0\r\n gatherlocal 40 0.000 0.001 0.0\r\n gather 40 0.000 0.000 0.0\r\n remote 40 0.000 0.000 0.0\r\n post_process 40 0.000 0.000 0.0\r\n spikeio 40 0.000 0.000 0.0\r\n```\r\n\r\n### Perf\r\n```\r\n Performance counter stats for 'bin/drybench /p/project/cslns/hater1/arbor/example/drybench/params.json':\r\n\r\n 221852.22 msec task-clock:u # 1.000 CPUs utilized\r\n 0 context-switches:u # 0.000 /sec\r\n 0 cpu-migrations:u # 0.000 /sec\r\n 28014394 page-faults:u # 126.275 K/sec\r\n 658832257282 cycles:u # 2.970 GHz (83.33%)\r\n 49927962696 stalled-cycles-frontend:u # 7.58% frontend cycles idle (83.33%)\r\n 285682904743 stalled-cycles-backend:u # 43.36% backend cycles idle (83.33%)\r\n 943000553536 instructions:u # 1.43 insn per cycle\r\n # 0.30 stalled cycles per insn (83.33%)\r\n 212435336483 branches:u # 957.553 M/sec (83.33%)\r\n 3372189010 branch-misses:u # 1.59% of all branches (83.33%)\r\n\r\n 221.907422663 seconds time elapsed\r\n\r\n 195.279197000 seconds user\r\n 24.508514000 seconds sys\r\n```\r\n\r\n## Sorting events by `time` only\r\n\r\nw/ pdqsort\r\n```\r\n---- meters -------------------------------------------------------------------------------\r\nmeter time(s) memory(MB)\r\n-------------------------------------------------------------------------------------------\r\nmodel-init 111.256 939.580\r\nmodel-run 79.065 871.150\r\nmeter-total 190.321 1810.730\r\n```\r\nwith `util::sort_by`\r\n```\r\n---- meters -------------------------------------------------------------------------------\r\nmeter time(s) memory(MB)\r\n-------------------------------------------------------------------------------------------\r\nmodel-init 111.666 939.581\r\nmodel-run 78.900 871.131\r\nmeter-total 190.565 1810.712\r\n```\r\n\r\n### Profiler\r\n```\r\nREGION CALLS WALL THREAD %\r\nroot - 68.611 274.442 100.0\r\n communication - 27.576 110.306 40.2\r\n enqueue - 17.728 70.912 25.8\r\n sort 320000 16.853 67.410 24.6\r\n merge 320000 0.808 3.231 1.2\r\n setup 320000 0.068 0.270 0.1\r\n walkspikes 40 9.848 39.391 14.4\r\n```\r\n\r\n# Conclusion \r\nWe find a 30% decrease in time spent on spike walking and a 10% decrease\r\nend-to-end. Note that\r\nthis case is extremely heavy on the communication part and advances\r\ncells at 100x _realtime_ ie\r\n1ms wallclock for 100ms biological time which is far beyond any cable\r\ncell we encounter in the wild.\r\n\r\nBy just sorting on the `time` field -- we don't care about the ordering\r\nbeyond that -- we find another\r\n5%.\r\n\r\n## Side note: Memory measurement\r\nThe memory measurement isn't trustworthy, the 290MB reported above for\r\nthe baseline turned into\r\n1300MB on repeating the benchmark.\r\n\r\n# Related Issues\r\n\r\nCloses #502","shortMessageHtmlLink":"📊 Improve spike delivery (#2222)"}},{"before":"9312a4a73be59cd8c7688ad4ed87368a98973092","after":"65290d65f915aac6778ed41abf297869c468cfc5","ref":"refs/heads/master","pushedAt":"2023-11-14T20:57:31.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"better spack workflow (#2231)\n\nPrevious spack workflow doesn't run through anymore. A possible reason\r\nmight be the spack cache which is cached with github's own cache\r\nfeature. This PR's intent:\r\n- simplify spack workflow\r\n- use external build-cache\r\n(https://github.com/spack/github-actions-buildcache)","shortMessageHtmlLink":"better spack workflow (#2231)"}},{"before":"3fb761ad45054c5c183861603ef800315b2f4379","after":"9312a4a73be59cd8c7688ad4ed87368a98973092","ref":"refs/heads/master","pushedAt":"2023-10-20T17:06:46.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"thorstenhater","name":"Thorsten Hater","path":"/thorstenhater","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/24411438?s=80&v=4"},"commit":{"message":"Use Labels for Probes instead of Numerical Ids (#2213)\n\n# Overview\r\n\r\nThis rectifies a longstanding exception from Arbor's scheme of `(gid,\r\n\"label\")` to address\r\npoint-like (or set-of-point-like) entities. Probes are up to now\r\naddressed as `(gid, lid)`, where\r\n`lid` is computed from the position in the return value of\r\n`recipe::get_probes`. This is not\r\nonly a more uniform interface, but also a safer one (imagine adding an\r\nadditional probe in the\r\nmiddle).\r\n\r\nThat is, from now now on, each probes needs to add a stringy tag, eg\r\n```cxx\r\n auto get_probes(cell_gid_type gid) const override { return {{cable_probe_voltage(...), \"Um\"}}; }\r\n```\r\nand we consume this as\r\n```cxx\r\n// ....\r\nauto sim = simulation(rec);\r\nsim.add_sampler({0, \"Um\"}, ...);\r\n```\r\n\r\nIn addition the optional `tag` field was removed. It has been hardly\r\never been used (no examples\r\nand no models in the wild I know of). It has also been hidden from\r\nPython. Thus, the two APIs are\r\nnow closer aligned.\r\n\r\n\r\n# ⚠️ Breaking\r\n\r\n- Each probes needs a string tag.\r\n- Numerical ids are no longer allowed.\r\n- (C++) `probe_info::tag` is gone\r\n\r\n# TODO\r\n- [x] Update docs.\r\n- [x] Check python examples.\r\n- [x] Check python tests.\r\n\r\n# Future tasks\r\nHere are some things I plan to do later on, but might overburden this\r\nPR. The idea is to streamline\r\nthe interface presented to the user and easing their cognitive load.\r\n\r\n- We should probe POINT processes by label, too.\r\n- We should streamline the sampler interface, currently there are 4\r\ncases\r\n 1. POINT: single item metadata of specialised struct\r\n 2. POINT cell: vector of these structs.\r\n 3. DENSITY/voltage/current/...: scalar float, `mlocation`\r\n4. DENSITY/voltage/current/... cell: vector of float, vector of `mcable`\r\n \r\nOf these we could at least merge 1 & 2, potentially 3 & 4, and maybe\r\n1-4.\r\n- Potentially adding pre-canned objects/hooks to dump data in certain\r\nformats (although I'd rather _not_ take additional dependencies, HDF5 or\r\notherwise).","shortMessageHtmlLink":"Use Labels for Probes instead of Numerical Ids (#2213)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEJtVDqAA","startCursor":null,"endCursor":null}},"title":"Activity · arbor-sim/arbor"}