Philosophy: What is RaptorJIT? #235
Comments
|
hi there! :) note that the bitops in 5.3 and lj differs in their behavior (cuz of optimizations). however anything is possible, but such a change have the possibility to break existing stuffs written for lj. i think u can find more about this in the issues of lj git, but this is only based on what i remember, so anything valuable can exists in lj mailing list just as well... btw i think this can/could be a good point, as ive already told to the lua folks that it would be the best (imho) if they would plan development together with the lj folks (mayb they do, just i dont see) instead of taking a very different path, as a huge selling point for the whole luaverse is that lj is the most powerful jit in the known universe (if im right), so this applies to rj as well. i already wrote this here and there around the lua community, that i came from python land, that was a huge love, but one of the main reasons was that 2.x 3.x shift totally broke its ecosystem and i needed stuffs from both worlds, however luaish stuffs are way much more compatible, but thats slowly bitten here and there on the road... i think rj should stay nearer to lj, and actually lj is what needs to get closer 1st to 5.2 and 5.3. maybe u could have a compile time switch for these to take the 5.3 or the lj approach, as these are not that big stuffs. on the other hand i think the planned/considered(?) elimination of asm optimizations from rj could (i guess) take away much horse power and give a chance for other jits to be faster than rj that are slower than lj, so the actual selling point could go, at least for rj, and it couldnt take the huge role to follow up the point of lua whlie keep this selling point for the luaverse. we are all here cuz we know how kawl the whole luaverse is, we are building our beloved and favourite toys here so i believe that its important for every1 around to not make chaos around. i think this whole issue is mainly around luajit, but i think this is a responsibility for every1 around to keep the lua ecosystem consistent, however i also dunno whats the best real world solution, the best i could do is to start to learn luajit right from the makefile, and try my best to be able to do whatever in case of need and possibly help make stuffs better around, but mainly i think its upon Mike, who is such a mysterious genius, so mayb there4 hard to understand (btw all my respect for him, but not only for him! :) ) and i think that i already told these actually to him as well, just like the lua core team but i dunno what they did for this, if anything. i think this could be handled by a round table conversation held by our great leaders :D the other thing i can think about is to follow up lj plans like hyperthreading, quadgc and whatever, so consume the current issues list to make it able to move forward, that is mayb a main reason for not moving forward with more stuffs around... bests! :) |
I thought the only major difference was that Lua 5.3's bit operations were infix, rather than LuaJIT's function-style.
I think the Lua ecosystem needs a way of declaring features akin to Rust's features. What about a
LuaJIT is good, but because it's fallen into maintenance mode, it's not doing very much. As a result, the JIT community seems to be catching up.
Mike's stated he has no intention of tracking 5.2 or 5.3, because it breaks the API. I think this is a useful point, but we need a way of advancing the language, or LJ will entirely stagnate.
This seems to be a fundamental misunderstanding of @lukego's goals. He wants to remove assembly from the interpreter, not the JIT. This makes the interpreter slower (though hopefully not by much), but the JIT will remain full speed.
Mike's gone, for the most part. This project exists because of that. While his genius is not in doubt, his friendliness could perhaps have used some improvements. Any help is welcome.
I don't think we should follow in Mike's footsteps purely for the sake of it; dogma is a dangerous thing. However, his proposal of a GC should be a good start, because there have been actual implementations based on it. |
lj have more optimizations and less comfort, but i dunno the exact details
i think i cant really understand this :D there can be a module in lua that implements bitops as lj does it, but i think its already given as lj bitops exists in a separated module written right by Mike; however on the contrary lj would need to give a way to extend its grammar with the new operators. farer from this i dunno what else it could be used for. for example there are a s***load of oop implementations but all of them differs, and thats the reason y there is more than 1 of them :D btw i dunno rust, if u give me a pointer to get a taste of this in rust, then thx for that :)
so i need to learn faster and do everything faster to be able to help around :D ive got such a huge knowledge, but im feelin nowhere to Mike, Luke and others around about these optimizations stuffs :/
thx, this was right the missing bit from my knowledge, i only knew about "wont fix" stuffs. btw this is relatively invisible from lua side and c can always have conditional parts via cpp (i hate those who call c++ to cpp, cuz i cant use the abbrevation safely for the c preprocessor :( (sad story)). so its nowhere to impossible, but cuz of the various hardcore optimizations it can b very hard, but a version dump or whatever can do the trick in worse case, but mayb compatibility cant b held forever for everything, mayb lj is already enough mature to break up with the path of puc lua ... i dunno, the whole case is kinda much of a hard nut...
ahh, ur right! :D as i read this i can remember now that this is really what he have said... i dunno the actual impact of this, but the best would b then to jit everything or every important bit, but thats a different beast, as lj and rj are tracing jits, and i believe (based on nothing :D ) that its reasonable, and that it will b only beneficial for long running apps to jit everything. otoh ive recently subscribed to snabb just for learning more and know better whats the actual purpose for rj, and ive seen there some dynasm stuffs, so i think rj folks must be fine with asm, and my strong belief is that once a thing was written right then it wont be an issue to exists as far as there arent breaking changes, but even then small modifications cant mean big pain if the base is already given. i think the only necessary thing is to make magic explained in place and whatever optimization that shouts "hey i could be written in that way instead" must be explained simply like easy/proper/common/whatever approach is not taken for whatever reason like its suboptimal, it wrecks something elsewhere, only a single hacky way could reach the actual goal, its incompatible with whatever platform, its futureproof cuz whatever plans/possibilities, it was measured to be superior... id really like to say that i would like to take good care of those bits, but my free time is much chaotic and my asm knowledge is still kinda weak, so i just cant promise anything, however i will do my best and wont forget about this. except if experts around can make me sure that it wont really hurt and whatever. :D
geniuses are always in lack of time, i think thats based on this, my point here is to appreciate his miraculous gifts instead of complaining about whatever around. i think weve seen the same around Torvalds. its hard to have more than 2 hands 1 body 1 life at once, and/or more than 24h a day or 7days in a week... (another sad story). if he took his time to breathe and having a life then he totally deserve it! :) if he is about to build a better flying saucer than aliens have, then i wish i could see it :D im sooo much deeply curious about what he is working on after such a great stuff :D
i believe he have good senses for what could be beneficial, so there4 i saied that. i think that Mike dare to take one step behind cuz there are already enough great minds around who are able to understand and touch the internals of lj, but they also lack in time, i hope this wont mean that lj will slowly lose its glory and/or rj wont be any worse after giving up some goodies for ease. i think that v8 have the 2nd place, but thats nowhere to the greatness of the taste of lua, and i think it also must have much fat on it, and its nowhere to give me as much good feelings as luaverse can, even if im a fullstack developer. :D i think moving slowly is much better than making suboptimal stuffs, but dont get me wrong, im here cuz i totally appreciate rj, im just worrying about the future while i dont have enough knowledge to know if my fears are valid or not, and in the meantime my sentences are also about getting info, and not only suggesting the right path, cuz im nowhere to really know whats that... btw Luke seems to be the right person to continue such a complex stuff, he is playing with it like if it would be a childtoy, and also, the force is with him! :D |
This is tricky. I can think of at least three different angles to think about this from and each one seems to suggest a different course of action :). Lua is a good language and seems to improve with every release. So all other things being equal it seems desirable to track the evolution of new Lua releases. So one could argue that it makes sense to move to Lua 5.3 and continue from there. Users want backwards compatibility. Language changes that break existing applications and require porting are a pain because they suck time away from users and can prevent people from adopting new releases at all. People want to be able to run Snabb, ljsyscall, LuaRadio, OpenResty, etc on RaptorJIT with the minimum of fuss. So one could argue that it makes sense to stick with the language we have now. Quasi-compatibility can also be a drag on the whole community. If application developers are trying to keep their code portable between Lua, LuaJIT, and RaptorJIT, then this creates a bunch of extra work and inhibits them from using all the best new features. That also creates a disincentive to evolve the languages at all since doing so mostly causes problems. So one could argue that it makes sense to break compatibility with Lua and LuaJIT once and for all so that each language can follow its own path. How to balance these forces? Here is one idea for the evening:
That's my $0.02 for this evening... |
|
Here is a thought experiment too: Where will Lua, LuaJIT, and RaptorJIT be language-wise in 10 years? Here's one possibility... Lua is on version 6.x. The language is better but it is deeply incompatible with Lua 5.1 and 5.3. If LuaJIT or RaptorJIT wants to support Lua 6.x they need to break all existing applications. LuaJIT is on v2.1. Fixes have been made, ports have improved, but features have not been added. RaptorJIT is backwards-compatible with v1.0 but has added a bunch of new features to make it a competitive alternative to C/C++/D/Rust. These are mostly features that were invented by our community based on experience but we also cherry-pick from Lua e.g. the bit-operations syntax from 5.3. ... Or how else might things play out? |
|
also: It would be very interesting to have an Issue about Lua 5.3 new features and which ones can/can't be adopted in a backwards-compatible way. I haven't looked at Lua 5.3 so much myself so maybe somebody who has could create that? |
|
they can said to be feature complete, so actually there is no need to rush anywhere, but practise is a different thing... :D i think the best would be to keep them near while moving forward. btw i had to realize in the meantime that i didnt count upon the libs/apps based on them, thats an even harder nut to keep them go together, however i think that most important things that stay with 5.1 are mostly doing so cuz of lj. at this point im wondering if Mike is planning to stay with 5.1 forever or better said cherry pick goodies from newer versions but mainly remain the same and slowly getting farer and farer from lua... this sounds too weird for me, even if there are optimizations that are intentionally different. so there4 i said that mayb its about cleaning up the current todo list (aka issues), as there are kinda much undone stuffs around, and mayb its not a good idea to start too much things at once, cuz thats asking for pain and chaos. furthermore quadgc and hyperthreading takes the least possible noise around, as they are critical like a brain surgery, and i strongly believe that they would carry more than enough breaking changes inside to not care about anything else in the meantime, and they have to be battle tested just as well to get mature too. however i dunno if anyone took these hardcore tasks, while i think Mike wanted to have these before he became busy and had to have one step back. im saying this cuz lj is mainly about performance in high level programming, as every little possibility to optimize have been taken, and i think thats totally right, but only takes the right guidance/reasoning/docs/comments/whatever around them, just like i sayed somewhere above. |
|
On Mon, 18 Feb 2019, 06:47 Luke Gorrie ***@***.*** wrote:
Here is a thought experiment too: Where will Lua, LuaJIT, and RaptorJIT be
language-wise in 10 years?
Here's one possibility...
Lua is on version 6.x. The language is better but it is deeply
incompatible with Lua 5.1 and 5.3. If LuaJIT or RaptorJIT wants to support
Lua 6.x they need to break all existing applications.
And likely the ecosystem (LuaRocks etc) has chosen to use it regardless,
because the Lua interpreter has been rewritten to be notably faster, and
it's still a useful scripting language.
LuaJIT is on v2.1. Fixes have been made, ports have improved, but features
have not been added.
This would kill LuaJIT for most situations; it'd be a dead implementation
for everybody who isn't CloudFlare.
RaptorJIT is backwards-compatible with v1.0 but has added a bunch of new
features to make it a competitive alternative to C/C++/D/Rust. These are
mostly features that were invented by our community based on experience but
we also cherry-pick from Lua e.g. the bit-operations syntax from 5.3.
This would kill us. With the ecosystem having moved onto 6.x, most packages
targeting 5.1 would be unmaintained. That means we can't draw on things
like LuaRocks as package managers, and would either have to port things
over, or try to create our own ecosystem. That would create a maintenance
burden we can't manage currently.
... Or how else might things play out?
RaptorJIT decides that to survive it needs to follow the ecosystem, but it
also values backwards compatibility to some extent. RaptorJIT can be built
as a library targeting 6.x, 6.(x - 1) and 6.(x - 2).
|
|
@szbnwer I am not enthusiastic about following the LuaJIT 2.2/3.0 roadmap. I don't know how Mike came up with this feature list but they are not the highest priorities for actually helping the users who I work with. The big problem is how much time and effort it takes to learn how to effectively optimize programs. This is due to the compiler being too opaque and unpredictable. This is why the work that I have done early on with RaptorJIT has focused on making the JIT more transparent and more predictable. To me the JIT is simply not complete until this work is done. Of course there are also interesting features on the LuaJIT 3.0 roadmap, other people have already done work on them, and we can eagerly merge those into RaptorJIT when they are complete (along with any other good features that people develop independently.) But I don't think RaptorJIT should give special treatment to feature ideas just because they were on Mike's TODO list when he stepped down. |
This is the world that I already live in, fwiw. I don't use libraries from Luarocks and I don't especially see them as suitable for use with LuaJIT/RaptorJIT. The C-API for example kills performance, and most C code can be rewritten in Lua on top of ljsyscall. So I see it as a burden to try and maintain compatibility with the Lua ecosystem rather than to forge ahead with the LuaJIT/RaptorJIT ecosystem that I have been living in for 6+ years already. |
|
Related link: LuaPower is a LuaRocks-like software ecosystem for LuaJIT. |
|
I have to acknowledge that I may be over-estimating the porting effort required to make applications support new Lua versions. The HoPL paper paints the picture that new versions are so incompatible that existing applications tend to stick with the older versions forever. This does not sound like fun to me -- I want applications like Snabb to be able to benefit from new RaptorJIT development as quickly as possible. That paper was written a long time ago though. If new Lua versions are designed such that the whole LuaRocks ecosystem can smoothly transition between versions then maybe things are not so incompatible as I imagine them to be. I would like to understand better what changes are coming in Lua 5.3+ and exactly how compatible/incompatible they are. Great if somebody can write a summary on a new issue. |
|
So what about all the projects caught in the middle, trying to support both
Lua and LuaJIT? Löve2D would be an excellent example.
Maintaining compatibility with Lua is a burden, but it should not be a
major one. If you break compatibility with Lua, you're screwing over both
LuaRocks and LuaPower (the latter of which seems to be a project by only a
few people). Maintaining a replacement ecosystem and convincing others to
use it is a much bigger burden.
Snabb has very different requirements to that of other Lua projects, and
even of other projects in general; Qt doesn't poke the GPU directly in the
name of speed. I wouldn't call your experiences representative of the Lua
landscape at large.
…On Mon, 18 Feb 2019, 08:10 Luke Gorrie ***@***.*** wrote:
Related link: LuaPower <https://luapower.com/> is a LuaRocks-like
software ecosystem for LuaJIT.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#235 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABbx294FMVKph0Jijl1RXv6eHaMmXyj3ks5vOl_agaJpZM4a_gE9>
.
|
I really hate the idea of projects being caught in the middle and I want to avoid that. So can you tell me: why can't Löve2D just pick Lua or LuaJIT and stick with that?
This depends on your perspective. For me it's Lua/LuaRocks that is the "other" "replacement" ecosystem. I am happy running code that is 100% written for LuaJIT/RaptorJIT, replacing all C code with Lua on ljsyscall, etc. How come I should switch from this beautiful native ecosystem to a collection of "caught in the middle" libraries instead? |
|
Also relevant history: Emacs/XEmacs schism. Emacs/XEmacs is worth considering as a cautionary tale for LuaJIT/RaptorJIT. Many years ago I spent/wasted a lot of energy on maintaining code that was "caught in the middle" between those two editors and it was a huge relief to ultimately be able to drop XEmacs support and focus on GNU. |
|
On Mon, 18 Feb 2019, 08:30 Luke Gorrie ***@***.*** wrote:
So what about all the projects caught in the middle, trying to support
both Lua and LuaJIT? Löve2D would be an excellent example.
I really hate the idea of projects being caught in the middle and I want
to avoid that. So can you tell me: why can't Löve2D just pick Lua or LuaJIT
and stick with that?
Why is PUC Lua written in strict ANSI C89 and tested on (at least) GCC,
Clang and MSVC++?
Portability.
Löve doesn't need to commit to a specific implementation if they support
all of them.
Maintaining a replacement ecosystem and convincing others to use it is a
much bigger burden.
This depends on your perspective. For me it's Lua/LuaRocks that is the
"other" "replacement" ecosystem. I am happy running code that is 100%
written for LuaJIT/RaptorJIT, replacing all C code with Lua on ljsyscall,
etc. How come I should switch from this beautiful native ecosystem to a
collection of "caught in the middle" libraries instead?
Actually a large portion of Lua code is native, with people only reaching
for the C API when they need speed.
For example, Luacheck is written in Lua.
Honestly Luke if you're so bothered about the C API, then just dump it.
… —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#235 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABbx207JQtUKlJZOBFYygQF-b03NELIKks5vOmSUgaJpZM4a_gE9>
.
|
I am only trying to tease out all the trade-offs through discussion here. I don't really have strong feelings about what the ultimate solution turns out to be. This idea is worth discussing on its own though -- even if only to flesh out the texture around this whole discussion. Side-issue created at #236. |
|
Clarification: I am only bothered by people being caught in the middle between implementations. I want people to be able to warmly embrace their favourite implementation and take full advantage of its capabilities. I would like to see people doing this with PUC Lua, with LuaJIT, and with RaptorJIT. The dystopian vision for me is a future where nobody uses Lua 5.3 features, nobody uses the FFI, nobody uses new RaptorJIT features, etc, because everybody feels compelled to target the lowest-common-denominator between all Lua-like languages. This feels a bit like trying to write practical C code or shell scripts that work equally well on Linux, *BSD, Solaris, and IRIX. I'm happy to have left those days behind me! |
|
(sry for the length, there were much topics around... :D ) @lukego hopefully 6.0 is only in the air, but not just around the next corner, based on https://www.lua.org/work/ and the mailing list. btw the puc folks are very strong after all those requests they get on a regular basis :D for the incompatibilities i could only say the obvious from my side: http://www.lua.org/manual/5.3/manual.html#8 - as im still only planning to make my app able to be switched to run on top of wichever engine where i even considered ravi and openresty lj, and i still dunno terra titan and whatever other implementations... why i actually care much about compatibility from my own perspective is that id like to be able to use my tools behind whatever libs i can find in the wild, to make them alive, be able to experiment with them and then collect the essence from a living codebase through my current+planned toolset. also cuz it would be nice to be able to use them together just the way they are... i actually wanna gravitate tools for sys hacking, web hacking, editing whatever type of file, having a flexible gui and what not on the same base. as a huge cancer in the world is that we have open source from kinda much anything we can wish for, but we have to following the development of everything that is interesting (and wish it to not run away every few days but have the next feature we want); learning every language that is used; reimplement all our toys between them; being in dilemma to choose this or that with one or the another wished feature; and write plugins for huge monsters and hope that they will be anywhere near to our actual intentions....... then also, what about speed, flexibility, compatibilty, wiring everything all together, like watching a js heavy website collect some non ascii data make interactions with out systems based on it and also be able to interact with the whole... do it in node? nope, npm installs are heavier than the sun, no1 will be able to ever read whatever they have, so as modify it, but send an another unread letter for Santa Claus in the form of an another git issue :D :( write everything in c and glue with bash? uhmm nearer, but not enough human friendly... cloud everythin? plzz NOO! :D so this is how i arrived to lua and luajit back in time, a nice language with much horse power and flexibility :) actually i only care about puc cuz there is so much info around that, and there are more stuffs built for that, and only after these cuz its extremely small, as sbc's iot devices and soc's are much interesting, not mentioning the old devices that are not enough for nothing in the present times, while 64k is enough to go to the moon :D btw i think that there is a strong possibility that 5.1< can become a minority and the 5.1 branch go on its own, and in this case it worth considering to stay nearer to lj than go for the current lua, and now im only about compatibility... dropping the c api would make it kinda hard for me to use my beloved gui toolkit ( http://tekui.neoscientists.org/screenshots.html ) with rj, even if id like to rewrite it at one point, but thats for the farer future, as i wanna drop the oop approach, make it work with ffi, and give it wayland support, and none of these gonna b made from one day to an another :/ the original developers are very busy, so its slowly getting nearer to be an abandonware, but Timm have said that it can be mainly called mature and feature complete, so there4 theres no much happening around it. the luakit browser is also based on the c api thats in my interest, but the gui is much more critical, even if id be happy without it, but only after im not relaying on it. :D btw i think the metatable stuffs are more of a waste, anything can be done without them (except for weak k&v and the gc metamethod, but those could still exists, and maybe userdata wouldnt like to not have it, but ive never used those to know) and in exchange a deeply entangled overhead could go and the actual happenings in the codebase would become way much more transparent, while its just a matter of an explicit function call instead of checking the existence of it all the time... otoh imho, lua ffi and pure lua syntax are the most important points for not burning all the bridges around. btw this "caught in the middle" approach is much friendlier if its like importing this or that file based on the current engine, so both worlds can be happy and used the right way with the most efficiency. and for that last message, i think linux is the best place to be, as m$ and appl€ are just like cancer, bsd have strong points, but closed source (the possibility and the fundamental license incompatibility with the bigger half of the open source world) makes it bad, and the rest arent really on the map. (except for android, but i wish for the day of its death by the rise of pure linux phones, like librem 5 and whatever). so what matters imo is the support for the different architectures, but thats really a thing that i dare to hope that they can come back to rj any day now when there will b any need, even if thats not a very simple field... @ZirconiumX |
Trouble I have here is what @agladysh mentioned over on #236: the performance of the C-API is qualitatively different on PUC Lua verses LuaJIT and RaptorJIT. On PUC Lua the C-API is an efficient way to call from Lua to C. On LuaJIT/RaptorJIT the C-API is a performance-killer that interacts badly with the JIT. So if you have performance-sensitive code calling from Lua to C then you need to make different optimizations for PUC Lua verses for LuaJIT. In other words, maintaining support for PUC Lua would be costing you a lot of performance on LuaJIT/RaptorJIT. This also means that lots of libraries like |
|
I personally would prefer RaptorJIT to ditch Lua compatibility if that means nice features for systems programming. As far as I can tell, Lua really wants to be in a relationship with C, while RaptorJIT merely tolerates the existence of C as a status quo. In ten years, ideally, I want to be able to recommend RaptorJIT for any project that would have used C/Rust/Go/... in place. I feel like there are many wild features waiting to distinguish RaptorJIT from the crowd, and maybe clinging to Lua will make implementing these harder. I am also not too worried about library compatibility, or libraries in general. For me personally its more important that a language is rich to write in, than rich in ecosystem. Feel free to disregard that though, as I am a compulsive yak shaver. ;-) |
There has been a discussion this morning on Slack about the following simple question.
Is RaptorJIT an implementation of Lua? Or is RaptorJIT an implementation of a language based on Lua 5.1?
Personally, I think until RaptorJIT gets enough attention to be considered an implementation in its own right and not just a LuaJIT fork, we should consider tracking Lua 5.3, to maximise code reuse - PUC Lua has some useful features we should consider, like native 64-bit integers and bitwise operation syntax.
The text was updated successfully, but these errors were encountered: