Season of Docs 2022 Case Study
Organization: SymPy
Project: Improving SymPy's Solvers Documentation
Organization Description: SymPy is a Python library for symbolic mathematics. It was started in 2005, and has grown into a full-featured computer algebra system (CAS), which is used by people from a wide variety of domains, including education, finance, engineering, and science. As a symbolic mathematics system, it is applicable to virtually every field of science.
Solving mathematical problems is a top interest of the SymPy community, and the existing documentation lacks basic information on that topic.
It is important for SymPy to address this deficiency because many visitors are likely unable to quickly learn how to use SymPy to solve their math problems.
Here is the full proposal.
Solvers were tied for the top result of "what areas of SymPy do you feel are not documented well enough?" in a survey of the community. According to the Google Search Console, for Google search terms that lead searchers to our documentation site, the second most popular term is “sympy solve”, exceeded by only “sympy”. Before this project, the vast majority of Google searches for “sympy solve” led visitors to the API reference page for solvers. This page is very technical and long, and provides little high-level guidance on how to solve various types of math problems. Further, the SymPy documentation has few examples of how to programmatically extract information from results.
Analysis of Google Search Console results revealed that the search terms involving solving could be grouped into categories such as equation and numerical, and those categorized ranked by number of Google search impressions, providing a way to prioritize solving guides. The project maintainers, and the invited technical writer, agreed this was an area of interest.
Our budget was straightforward and we kept to it:
| Budget item | Amount | Running Total | Notes/justifications |
|---|---|---|---|
| Technical writer develops user-facing solving page and sub-pages. | 15000.00 | 15000.00 |
TOTAL: 15000.00
We estimated the work based on the number of hours we expected creating each page to take. We underestimated the number of hours per page; we kept the total number of hours constant at 300 by scaling back the number of pages. There were no unexpected expenses, and we spent the entire grant award. We did not have other funds outside of Season of Docs that we were able to use.
The technical writer was Jeremy Monat (@bertiewooster). Project mentoring was done by Aaron Meurer (@asmeurer), who is a core contributor to SymPy and a project maintainer. Aaron approached Jeremy based on Jeremy's previous documentation work with SymPy: contributing to the documentation four times to SymPy, and leading the selection of a new Sphinx documentation theme. We met every two weeks on a video call to discuss the project progress.
April 2022 - May 2022: Planned, developed, and incorporated feedback on the solving landing (main) page.
June 2022: Completed solving landing (main) page. Drafted and got feedback on first sub-page, solve an equation algebraically.
July 2022: Completed solve an equation algebraically page. Drafted and got feedback on solving guidance, and solve
- a system of equations algebraically
- numerically
- a system of inequalities algebraically
August 2022: Drafted and got feedback on solving matrix equations and ODE (ordinary differential equations).
September 2022: Drafted and got feedback on the last two new pages, solving polynomials and Diophantine equations.
October 2021: Added links to the solving guide from existing modules. Combined multiple types of solving (for example, system of equations, system of linear equations, and system of nonlinear equations) into one row in table on main page, and into one sub-page with an example of each (linear and nonlinear). Updated several pages in response to review comments. Linked to solving guide from tutorial, solvers, and solveset pages.
November 2021: Completed Diophantine, matrix, and polynomial pages. Waiting on ODE (ordinary differential equations) pull request to be merged.
The up-to-date solving guides can be found on the development documentation website.
Things not completed The original target was 15 solving sub-pages, but it soon became apparent that
- that was too ambitious, both for the technical writer to create, and for the mentor and community to review
- some of the solving topics had a lot of overlap, so it made sense to put them on a single page, and simply describe all the grouped topics in a single row on the main page
- there were not that many independent solving topics that merited their own page
As a result, we revised the target to eight solving sub-pages, which we believe covers most of the topics that searchers seek based on Google Search Console data, plus one solving guidance page. We completed all of those pages.
In terms of helping SymPy users get to an easy-to-navigate page for solving, using Google Search Console we planned to measure the percent of searches for “sympy solve” which
- return the new solving page as the first result
- lead to the searcher clicking on the new solving page instead of another page on SymPy.
The canonical version of the solving guide has not yet appeared in Google Search Console results, probably due to it not being published until the SymPy 1.11 release on Aug 23, 2022. We have linked to it from the pages that "sympy solve" currently leads to, which should signal Google that the new solving guide is a relevant result to present to searchers, but the changes will not appear in canonical versions of those pages until the next SymPy release.
We also got dozens of constructive comments on the new solving pages from the SymPy community via the pull request process.
In terms of content, we tracked the number of sub-pages created from the list of the types of solving needed based on priority. We created eight sub-pages, starting the most important (solve an equation algebraically, solve a system of equations algebraically, solve numerically, and solving guidance), plus one solving guidance page.
Note: Not all content will be included in the canonical version of documentation until the next version of SymPy is released.
Many things went well: We developed a wide-ranging guide to solving different types of mathematical problems summarized on a home page and detailed on sub-pages, we came up with a template that worked well and we evolved over the project, and we got considerable feedback from the SymPy community. Even during the project, we linked to it in answer to user questions on the SymPy listserv.
The most underestimated thing was the amount of time it took to create and refine new pages given feedback from the community.
We consider the project very successful in terms of providing quality, example-rich, math-focused guides for users to solve problems and programmatically extract results. In terms of the search traffic metric, it is too early to tell because the pages will not be in the canonical version of documentation until the next version of SymPy is released, and even after that there will be lag as search engines index and start to surface the new pages.
The project resulted in a new section of SymPy's online documentation for solving math problems. Previously the solving documentation did not provide high-level guidance on how to solve common math problems, was highly technical (in terms of Python), was organized around SymPy commands rather than types of math problems, and was not in a central location (it was on the API pages for the specific functions). The new solving documentation includes:
- the main solving page which provides examples in standard mathematical (rather than programming) format so visitors know which page to visit to solve their problem
- the solving guidance page which provides tips on how to use SymPy relevant to many types of solving
- the sub-pages which provide alternatives to consider, guidance, ways to programmatically use the results, tradeoffs to consider, and the types of problems that cannot be solved
For other projects, I would recommend keeping in mind that there may be significant feedback from the community on pull requests. As a result, when estimating the pace of work, the amount of time required per pull request is quite a bit more than simply drafting the page, typically at least a month for a somewhat-lengthy page. For a project of this type, where pages require code examples, I would recommend aiming for around six pages--the nine pages this project ended up including was difficult to complete. Also, it is wise for the technical writer to have several pages up for review (as pull requests) simultaneously so they can make progress on several of them, rather than waiting to complete one page before starting another page. Realize that the amount of documentation created also taxes the mentor (who typically reviews it), so that can present another limit on the amount of content that can be added in a time period. And because pages will often be based on doctests (code snippets with commands and outputs that demonstrate how to use package functionality), use a system that works for you to develop them; I put doctests in Jupyter Notebook files in a public repo so I could save, revise, and copy them, and so anyone else could access them. In terms of tracking external metrics such as Google Search Console data, realize that your changes will not go live to the canonical documentation site (not the dev site) until a release of SymPy, so it may be months between when your content is merged and when it goes live.
If doing a project where there will be one main page and several similar sub-pages,
- Create a template for the sub-pages for consistency and speed of drafting
- Realize that there may be merge conflicts if sub-page pull requests affect the main page
- Because your pull requests may not be merged for a while, realize that you will need to go into your open pull requests to get content that you need in others, and that git is not particularly helpful for that
- Let your mentor know which pull requests are blockers, that is, which ones you need them to be merged before you can complete another one (typically because the second pull request needs to link to the first)
Regarding project management, I would recommend the technical writer contact SymPy's sponsoring organization, NumFOCUS, as soon as the project is approved, to get in place the contractor agreement to allow for timely transfer of funds from NumFOCUS (after they are received from Google) to the technical writer. Also, the technical writer should develop a metric to track their progress and check it against the time elapsed in the Season of Docs period. In SymPy 2022 Season of Docs progress tracker Google Sheet, I estimated that creating each new page was one unit of work, then estimated my progress on each page. For example, if I completed drafting the page, that might be 70% complete; if I had addressed all existing comments and anticipating few or none until the pull request would be merged, that might be 90% complete.