Guided tour to the CPython source code #6
Comments
|
Some of this is lightly touched in the devguide (e.g. http://cpython-devguide.readthedocs.io/en/latest/setup.html#directory-structure), but in no way thoroughly. |
|
Ah, thanks. I would like to go a step deeper for the directories containing most of the fundamental C code:
Other topics should probably include
|
|
@gvanrossum are you soliciting assistance on this from others that aren't core devs too? |
|
For documentation about defining a type, it depends. There's the C API tutorial and then there's xxsubtype.c. @lorenanicole Guido can say if I'm wrong, but there's absolutely no reason to restrict this to core devs. Honestly, it would be best to have someone who isn't a core dev involved so no bad assumptions of pre-existing knowledge muddles the tour. |
|
@lorenanicole: What Brett said. The core devs already know this and probably have huge blind spots. My own blind spots are even bigger. (I wrote some of the stuff Brett pointed to and I had forgotten about it. :-) I also believe that a lot of the existing docs in this area assume lots of other skills that aren't so easy to come by. While I don't think we should include a C tutorial, there are a lot of C patterns that aren't unique to Python but are still worth explaining in some detail, either because Python's version has certain important details, or just because they aren't that widely known. Some more random topics:
|
|
|
@gvanrossum Love this idea! It's similar to what I want to do with notebooks and Phillip Guo's grad school course. It also captures well the spirit of our PyCon conversation. All, The next paragraph that I'm writing is done with gentle kindness, respect, and thoughtfulness: I do want the contributors to the documentation developed here to primarily or at least 50% written by those that are not core developers. A few reasons:
Just open a PR with the folder name and outline that we would like to iterate on over time. Also, please let me know if you would like to be added as a maintainer on the repo. I'll be on vacation with limited internet access beginning Friday to the 2nd week of August. |
|
IMHO, I think that it would be great to frame out by core devs and then have non-core devs tear it apart, ask questions, restructure and edit. However, I do not do work like this every day. But @Bradamant3 does! :-) @Bradamant3, do you have thoughts on ways to approach this? Background on @Bradamant3 |
|
@jackiekazil and all -- I am honored to be mentioned here, and I'd be thrilled to be involved. I'm anything but a core dev -- I'm still pretty new to Python, still working my way through Learn Python the Hard Way, although I've taken a couple of handfuls of programming classes (no C, sadly) and write sample code for part of my living. (I'm a technical writer by trade, on the more technical side of things.) I'll need to spend some time over the next week looking through what's already in the repo and the other resources y'all have mentioned to see where I think I can best help out. I have much to learn, and if I can help others learn by documenting my own learning as I help out with this project, that would be the biggest thrill of all. |
|
Hi @Bradamant3, We would love for you to be part of this. There's a special place in my heart for technical writing since it's key to breaking down barriers for users to use your software and critical for onboarding developers 🌻 Please feel free to ask questions (large or small). Looking forward to you helping out. |
|
My bytecode chapter for the Architecture of Open Source Applications book might be a better example than my bytecode talk (although the content substantially overlaps): http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html One of the hardest parts of writing the chapter was getting clarity about the intended audience and the intended goal. I eventually settled on an audience of Python programmers who want to learn more about the language for its own sake. (I personally don't think knowing about the guts of the interpreter helps you write better Python, but I do think it helps you deepen your knowledge of computer science and is darn fun.) Given that, for this project, who's the intended audience, and what's the goal? What is the reader hoping to get out of the guides? Some possible goals I can imagine:
We don't need perfect agreement on the goals, but I think some clarity early on will pay off significantly later on! With more clarity on the goals, I'd be happy to contribute to this. I think the object model and garbage collection would be particularly rich topics. |
|
@akaptur and all: |
|
+10k as well to all the above :-) once a more definitive outline is settled I would love to step in and help. @jackiekazil + @willingc I'll defer to you two for directions. |
|
Nice to see @akaptur here. Hope to see you at PyBay. One of the things that I have wanted to do for over a year is to edit down @pgbovine's great 10 hour course http://www.pgbovine.net/cpython-internals.htm that walked through the Python2 codebase. I wanted to create a series of Jupyter notebooks that would correspond to each of his videos and link to source code. Here's a link to one of the notebooks that I started last year: https://github.com/willingc/pyladies-cpython/blob/master/Notes%20on%20Lecture%201.ipynb I'm wondering (if @pgbovine is cool with us referring to the content and @gvanrossum and @brettcannon see value) if we could use the @pgbovine course as a starting point and outline linking out to other resources and docs as appropriate. Perhaps creating a study group to do two lectures a month or something. This is everyone's group so no need to defer to me. If you have a good idea, please run with it. Personally, I want to see something that digs into the guts and I see some value in the video lecture combined with written word. Overall, I'm better on code guts than usability so @Bradamant3's talent and insights are very valued. Thoughts? P.S. @akaptur I think your bullet list covers the audience. |
|
Also leaving for Japan for a long overdue vacation. @jackiekazil, would you mind working with @estherbester and @audreyr to get yourself and others push access to this repo? I tried to add folks last night but needed org access not just repo admin access. |
|
I am wiling to contribute! Thanks |
|
My only worry with videos that are tied to something like the internals of CPython is they will become outdated and videos are nowhere near as easy to update as written documentation. It's one thing for a professor to record lectures he's going to give anyway, and it's another to get a volunteer to re-record an entire video for some key type just because we tweaked some detail in a release. Now that's not to say people couldn't do videos, but I think they should be entirely ancillary to anything written and simply something that could be pointed to instead of building something around the videos. |
|
Thanks for the kind words! Yes, you have my permission to use and remix the content as you wish. Please add a link back to the original source material webpage ( http://www.pgbovine.net/cpython-internals.htm ) in the appropriate place(s). Best wishes. |
|
Quick poll now that Brett has brought it up... Do people here prefer video
lectures or text? I personally strongly prefer text, but my ways of
learning were set long before online video was a thing.
|
|
@gvanrossum it depends. |
|
Video sounds quite exciting however I think that's a stretch goal? Let's start with text then we can regroup and see about video. |
|
Text is fine by me. Let's proceed forward with an outline. I'm really pleased to see the interest building 😄 If I have time when I return from Japan, I may do the Jupyter notebooks as a work related thing since there is no additional video production needed. Something that could be used with some Jupyter/JupyterHub work that I'm doing for education. |
|
OK, so with a general agreement to start out with a tutorial document, what's the next step? Outline? Basically this sounds like it's going to be a mix of "this is how Python works to help you debug problems", and "this is how Python is structured to help you find your way around". If we need a jumping-off point then simply starting with explaining what's in all the top-level directories of a source checkout is as good a place as any. That can feed into navigating around, e.g. you can always use a |
|
I just got off the phone with @emilyemorehouse who is also interested in this issue. I promised her one specific deliverable: an overview of what's in the most important top-level directories and the main entry points underneath there. I will spend some time today writing up what I can. While we're collecting useful links, maybe my old Python history blogs could be of some use. There's a wealth of technical information there: http://python-history.blogspot.com/. A few highlights:
|
|
@brettcannon I think that an outline is a great next step. What's the best way for us to do this? I've started on a set of docs based on some of @gvanrossum's direction that I've divided into a few sections -- resources (with tracking on what has been scoured for information), notes on meetings/project goals, notes from resources, and notes from digging into CPython itself. I've currently only gone through @akaptur's Bytecode talk but I've got a decent list of resources. My personal goal from this is to be able to gain a strong understanding of the codebase in order to work towards becoming a core contributor, so I'm certainly going to lean more towards documenting how Python and its code works. I'm more than happy to share all of my docs, both here and from a personal repo as not all notes will be entirely relevant. @jackiekazil, I can start adding if you give me access. |
|
@emilyemorehouse Without settling on an overall direction exactly, I don't know if we could do an outline, hence why I suggested we just start with an overview of the directories and see where it takes us. Part of the problem is I don't know what does or doesn't need to be covered for a new contributor as I have 13 years of knowledge which completely separates me from a beginner's perspective of what's not obvious. Otherwise I would just start from what it takes for someone to fix a bug and trying to cover each of the steps (navigating the code, how Python works to understand how to diagnose, and how best to write C code). |
|
As the newest newcomer, but with some background in helping get similar projects off the ground, may I make a suggestion? |
|
Ladies and gentlemen, I have something to share. I spent a few hours writing a rambling draft that walks you through what happens when Python starts up, from For now I wrote it using a new Dropbox product, "Paper". READ THIS BEFORE YOU CLICK ON THE LINK: Because Paper lets you comment and edit, it will also reveal your identity to others viewing the document, even if you're just viewing yourself, unless you use anonymous browsing. https://paper.dropbox.com/doc/Yet-another-guided-tour-of-CPython-XY7KgFGn88zMNivGJ4Jzv |
|
@gvanrossum Thanks for sharing this early. It's wonderful. I really like the conversational style. I love the look and readability in Paper too. Huge +1 to adding this to this repo in any form that you and others like. Thank you for doing this, Guido. |
|
Hello, I have received a notification from @lorenanicole about this thread because I am interested with this topic. In fact, I have presented my talk https://speakerdeck.com/matrixise/exploring-our-python-interpreter at EuroPython 2016, PythonFOSDEM 2016, PyCon.CA 2015 and PyCon.IE 2015. This talk has been shared on the core-mentorship mailing list. If you are on this mailing list, here is the link to my message: https://mail.python.org/mailman/private/core-mentorship/2015-November/003274.html I have already discussed with Victor Stinner (@Haypo) about an eventual book on the topic where I would like to present the internals parts of CPython, and he is interested. In fact, with my talk, I have observed that some people ask me how to start with the contribution to CPython (ok, I am not a very active contributor, but I try in function of my time) and they think that a good introduction or just a good "book" is a good starter. For my part, I wanted to start the project during August, just after EuroPython with some articles or maybe with a GitBook, trying to work on a Table of Content and after that the content. So, I am interested because I want to become a contributor of CPython, fix some issues in the interpreter, learn for me and explain to everybody. |
|
Just an FYI, @matrixise , the core-mentorship mailing list is private so only members of that list can see your link. |
|
maybe a good reason to subscribe to this mailing list (not you, I know you are already subscribed 😉) but you are right |
|
By the way, I'm looking for help on the topic of tools for spelunking C code. Emily started reading my guided tour and discovered that she could use a better editor to navigate such a large code base. My own setup is actually quite crusty (Emacs and |
|
I actually found a great plugin for Sublime using CTags and it's a huge improvement. Goto definitions actually work the way they should now. I think the bottom line is that if you're staying away from a full IDE, tags are the way to go. |
|
I don't know if we want to go down the route of IDE/editor-specific recommendations w/o being very selective of only editors core devs actively use in case someone asks for help (suggesting you find an editor/IDE that supports "goto definition" seems entirely reasonable to me, though). If we're talking about finding definitions of functions using just standard CLI tools, I just do, e.g. |
|
@brettcannon I definitely agree, staying editor/IDE agnostic is definitely important for use in the guide, this was more of a general question of "how do people do this, and how can I improve my methods". I don't necessarily see this information as the highest value to include in these efforts (yet), but knowing that it's a question that people may have is at least something to keep in mind later. On another note, I've pushed most of my notes to a github repo here as promised. It's very casual and some of it (probably) isn't relevant, but I've at least starting compiling a some lists of resources and content that we can pull from. |
|
@emilyemorehouse Good stuff in your repo. I return from vacation on Tuesday and would be happy to merge anything that you would like into this repo. @emilyemorehouse Good tip on sublime. Do you know if Atom supports CTags? I'm assuming it does too. One of the nice things about Atom, Sublime, or even Visual Studio Code (thanks @brettcannon for the tip at PyCon sprints) are the ability to easily visualize a large codebase. |
|
@willingc I believe that CTag support is native in Atom, it will detect any tag files and use it automatically. |
|
VS Code supports GTAGS (which can use CTAGS). I suspect any of the mature editors will support tags in some fashion and recommending one that uses them as a way to navigate source is fine. |
|
These days, is there a difference between |
|
@gvanrossum from what I can gather, yes, there's still a difference. |
|
Ah, thanks! Indeed, my Emacs does not like the ctags output.
|
|
Am wondering about the status on this. I was looking for documentation on the Cpython source code and found myself on this thread. Being a new contributor on Cpython, I am interested to help so I can also understand in the process of helping on this. So, what is the status? cc @gvanrossum @emilyemorehouse @willingc @brettcannon I see that the last discussion was 2 almost 3 years ago so i was wondering if this is still relevant. |
|
Hello @nanjekyejoannah, I propose to you to read the devguide on https://devguide.python.org and try to find the "easy" issues on the bug tracker. I could help you for these issues. You can also use the Zulip channel and discuss with us on #core/help. You have also the core-mentorship mailing Have a nice day and see you later. |
|
@matrixise thanks for this info. I also wanted to know the status on this. |
|
Hi @nanjekyejoannah! Welcome, and thanks for your contributions to CPython 🎉 I definitely think this issue is still relevant! I haven't had the time to formalize much documentation on this; I have numerous notes that I need to translate from my shorthand into something others can understand 😓 @gvanrossum and I's approach was originally going to be to write out information in Dropbox Paper (markdown support with collaboration/comments) and then figure out what tool/framework would be best to publish it in. We could also use markdown files in a GitHub repository. Personally, I was leaning towards a static site and also have the url I do have a couple of things that may be useful, though very rough:
I'd love to collaborate and synthesize the ideas in this thread, then divide and conquer. There are so many parts of CPython that a thorough guide will need the efforts of many people. |
|
@nanjekyejoannah Thank you for rebooting this thread. Adding @Mariatta @csabella who may have additional thoughts/content and @vstinner who has some handy docs. One of the things that I did last year was come up with some Minimalist docs for JupyterHub to simplify the content from the main JupyterHub docs. I started doing something similar for the devguide while on a flight, Minimalist CPython but got distracted with other projects. |
|
@willingc Thank you for pointing out this project! I haven't even tried to look at the C code yet, so, no I don't have any content. You wrote:
Even though I'm a core dev (😱), I would classify myself as part of the other 50% for this project. But, to be honest, I wouldn't even know where to begin helping out at this point. This post regarding the scope here makes a lot of sense. It seems like there's so much that can possibly be done, it's hard to know where to start, although @emilyemorehouse and @nanjekyejoannah are headed in a good direction. |
|
@emilyemorehouse Well, IMO this : https://paper.dropbox.com/doc/CPython-Guide-m7BQyPth6AIDUdZ6EmBNM draft has a draft scope or something pointing in the direction of what I think should be done. Point is we need to start somewhere. I suggest we build on from there taking advantage of collaboration from the efforts of many people. The next logical thing would be @emilyemorehouse moving that draft to like a Github repository/Dropbox Paper/static site etc. You can panel bit it to something we can start with. We can use any form that allows others to collaborate. I would lean more towards a github repository but may be the core devs know what has worked better in similar situations. I just think we need to find a way of starting. |
|
I saw my name mentioned here. There are multiple existing documentations:
You are free to create a new site if you prefer. My collection of links, "Documentations of CPython internals": |
|
Great suggestions @nanjekyejoannah. Do you and @emilyemorehouse want me to open a new repo here on the pyladies org for work to begin? The drafts could begin and once it's further along perhaps move to the python org. Thoughts? |
|
Actually, I added @nanjekyejoannah @emilyemorehouse @csabella @Mariatta as maintainers on this repo. Feel free to use it for drafts or whatever else related to this. |
I propose that we collaborate on creating a guided tour to the CPython source code. I think such a thing is vastly overdue, and I am not aware of existing resources (though I'll gladly take pointers that prove me wrong).
I've more or less promised @emilymorehouse to help her get started with something like this, and I'm happy to put time in. @willingc I hope you think this would be a useful document to maintain in this repo? I suppose it would go under the cpython tree.
The proposed document differs from the existing guides like the devguide -- it doesn't tell you anything about the patch submission flow, nor even how to get things to build -- it should focus on how to read the CPython source code (using your editor of choice).
Some topics could include:
The text was updated successfully, but these errors were encountered: