[solver] Proposal: add classic tableau option for dopri5

[Zymrael]

Hi @rtqichen! Thanks for all the work done on maintaining the repo. I've been working on other niche numerical methods in PyTorch recently, and in the process implemented several standard, general-purpose ODE solvers. While checking against torchdiffeq for implementation accuracy, I noticed you've opted for a dopri5 modified tableau (from here?).

This PR is simply a proposal for adding, as an option, the classic dopri5 tableau (as per Hairer I, II.5). Running DETEST yields interesting results, with classic_dopri5 being more accurate at low tols, although stricter tolerances make modified dopri5 a clear winner in terms of NFEs:

======= dopri5 | tol=1.000000e-03 =======
A1: NFE 68 | Time 0.004477977752685547 | Err 7.272080e-04
A2: NFE 44 | Time 0.003423452377319336 | Err 1.500953e-03
A3: NFE 140 | Time 0.00914144515991211 | Err 1.716421e-03
A4: NFE 32 | Time 0.0025844573974609375 | Err 2.036043e-02
A5: NFE 32 | Time 0.0023107528686523438 | Err 1.539949e-03
B1: NFE 320 | Time 0.027723073959350586 | Err 4.882935e-02
B2: NFE 158 | Time 0.010473251342773438 | Err 4.669148e-04
B3: NFE 86 | Time 0.007264614105224609 | Err 4.134018e-04
B4: NFE 176 | Time 0.01801609992980957 | Err 2.398165e-02
B5: NFE 152 | Time 0.013022661209106445 | Err 2.673915e-01
C1: NFE 86 | Time 0.005913734436035156 | Err 1.827017e-03
C2: NFE 422 | Time 0.027717113494873047 | Err 9.209132e-04
C3: NFE 194 | Time 0.012867450714111328 | Err 1.451195e-04
C4: NFE 194 | Time 0.013176441192626953 | Err 2.741384e-04
C5: NFE 44 | Time 0.007622957229614258 | Err 2.438725e-01
D1: NFE 284 | Time 0.027350902557373047 | Err 5.659548e-01
D2: NFE 236 | Time 0.0227053165435791 | Err 1.376881e+00
D3: NFE 248 | Time 0.024084806442260742 | Err 1.186594e+00
D4: NFE 338 | Time 0.03231334686279297 | Err 5.510144e-01
D5: NFE 554 | Time 0.05266284942626953 | Err 1.526093e+00
E1: NFE 92 | Time 0.009804010391235352 | Err 6.680222e-03
E2: NFE 404 | Time 0.03472018241882324 | Err 1.600952e-01
E3: NFE 218 | Time 0.020805835723876953 | Err 1.830766e-01
E4: NFE 62 | Time 0.0052242279052734375 | Err 4.148992e-03
E5: NFE 86 | Time 0.007855892181396484 | Err 8.007785e-05
Total NFE 4670 | Total Time 0.4032628536224365 | GeomAvg Error 1.255112e-02
======= dopri5 | tol=1.000000e-04 =======
A1: NFE 86 | Time 0.005663633346557617 | Err 6.257896e-05
A2: NFE 50 | Time 0.003863096237182617 | Err 1.412436e-04
A3: NFE 194 | Time 0.01262211799621582 | Err 1.799199e-03
A4: NFE 50 | Time 0.0038950443267822266 | Err 1.154925e-03
A5: NFE 38 | Time 0.0027451515197753906 | Err 2.127661e-03
B1: NFE 398 | Time 0.03472161293029785 | Err 3.453201e-03
B2: NFE 170 | Time 0.01134800910949707 | Err 1.129785e-04
B3: NFE 92 | Time 0.007935523986816406 | Err 2.060349e-05
B4: NFE 224 | Time 0.023092031478881836 | Err 1.171406e-02
B5: NFE 212 | Time 0.018027067184448242 | Err 2.497443e-03
C1: NFE 104 | Time 0.007101297378540039 | Err 1.622338e-05
C2: NFE 422 | Time 0.027708053588867188 | Err 2.998785e-05
C3: NFE 206 | Time 0.013611793518066406 | Err 5.843422e-05
C4: NFE 200 | Time 0.013532876968383789 | Err 6.165084e-05
C5: NFE 38 | Time 0.00679469108581543 | Err 2.190981e-02
D1: NFE 224 | Time 0.02194356918334961 | Err 4.141287e-01
D2: NFE 272 | Time 0.026378154754638672 | Err 2.012052e-01
D3: NFE 338 | Time 0.0327143669128418 | Err 8.307503e-02
D4: NFE 434 | Time 0.041992902755737305 | Err 8.628855e-02
D5: NFE 704 | Time 0.06785154342651367 | Err 1.296403e-01
E1: NFE 140 | Time 0.014842033386230469 | Err 6.369574e-04
E2: NFE 488 | Time 0.04215192794799805 | Err 3.490006e-03
E3: NFE 386 | Time 0.036216020584106445 | Err 1.138460e-02
E4: NFE 74 | Time 0.006269693374633789 | Err 3.648367e-04
E5: NFE 74 | Time 0.0067937374114990234 | Err 2.044865e-04
Total NFE 5618 | Total Time 0.48981595039367676 | GeomAvg Error 1.543488e-03
======= dopri5 | tol=1.000000e-05 =======
A1: NFE 116 | Time 0.007616996765136719 | Err 7.028651e-06
A2: NFE 68 | Time 0.005178689956665039 | Err 9.528487e-06
A3: NFE 332 | Time 0.02120065689086914 | Err 1.328810e-04
A4: NFE 74 | Time 0.005660057067871094 | Err 6.129209e-05
A5: NFE 68 | Time 0.004552125930786133 | Err 8.070523e-05
B1: NFE 542 | Time 0.047696590423583984 | Err 4.215027e-04
B2: NFE 194 | Time 0.012939929962158203 | Err 2.628725e-06
B3: NFE 122 | Time 0.010501861572265625 | Err 5.987609e-06
B4: NFE 350 | Time 0.03617119789123535 | Err 1.571189e-04
B5: NFE 296 | Time 0.025326013565063477 | Err 1.709396e-04
C1: NFE 134 | Time 0.009191751480102539 | Err 3.511705e-06
C2: NFE 428 | Time 0.0281832218170166 | Err 3.737126e-06
C3: NFE 218 | Time 0.014476776123046875 | Err 2.719570e-06
C4: NFE 218 | Time 0.014812707901000977 | Err 2.373542e-06
C5: NFE 50 | Time 0.008850336074829102 | Err 1.582316e-03
D1: NFE 290 | Time 0.02858114242553711 | Err 2.911460e-02
D2: NFE 368 | Time 0.035894155502319336 | Err 1.548090e-02
D3: NFE 458 | Time 0.044406890869140625 | Err 7.420157e-03
D4: NFE 590 | Time 0.05716371536254883 | Err 7.877367e-03
D5: NFE 872 | Time 0.08407020568847656 | Err 1.052913e-02
E1: NFE 218 | Time 0.02293992042541504 | Err 5.252812e-05
E2: NFE 686 | Time 0.059548377990722656 | Err 3.176970e-04
E3: NFE 536 | Time 0.05064678192138672 | Err 7.590794e-04
E4: NFE 92 | Time 0.007841825485229492 | Err 2.188373e-05
E5: NFE 98 | Time 0.008839845657348633 | Err 1.170171e-04
Total NFE 7418 | Total Time 0.6522917747497559 | GeomAvg Error 1.199603e-04
======= dopri5 | tol=1.000000e-06 =======
A1: NFE 152 | Time 0.010069131851196289 | Err 4.827294e-08
A2: NFE 92 | Time 0.006957054138183594 | Err 6.298700e-07
A3: NFE 464 | Time 0.029741287231445312 | Err 4.952653e-06
A4: NFE 86 | Time 0.00665283203125 | Err 1.167257e-05
A5: NFE 86 | Time 0.005841255187988281 | Err 2.249016e-05
B1: NFE 836 | Time 0.07345247268676758 | Err 1.821727e-05
B2: NFE 224 | Time 0.014986753463745117 | Err 1.058051e-06
B3: NFE 164 | Time 0.014186382293701172 | Err 1.045420e-06
B4: NFE 542 | Time 0.05601191520690918 | Err 2.967035e-05
B5: NFE 458 | Time 0.03908538818359375 | Err 3.317421e-05
C1: NFE 194 | Time 0.013300895690917969 | Err 4.386875e-07
C2: NFE 440 | Time 0.028952836990356445 | Err 8.595858e-07
C3: NFE 242 | Time 0.016213417053222656 | Err 3.355572e-07
C4: NFE 236 | Time 0.01613163948059082 | Err 4.573760e-07
C5: NFE 74 | Time 0.013030767440795898 | Err 1.062149e-04
D1: NFE 428 | Time 0.04241347312927246 | Err 1.257160e-03
D2: NFE 548 | Time 0.05366659164428711 | Err 7.037382e-04
D3: NFE 692 | Time 0.0673072338104248 | Err 2.068639e-04
D4: NFE 872 | Time 0.0846104621887207 | Err 7.570222e-05
D5: NFE 1268 | Time 0.12240004539489746 | Err 3.269919e-04
E1: NFE 332 | Time 0.03477048873901367 | Err 5.266044e-06
E2: NFE 1118 | Time 0.09650158882141113 | Err 2.004079e-05
E3: NFE 770 | Time 0.07302641868591309 | Err 6.289734e-05
E4: NFE 134 | Time 0.011199474334716797 | Err 2.371371e-06
E5: NFE 128 | Time 0.011538505554199219 | Err 8.093759e-06
Total NFE 10580 | Total Time 0.9420483112335205 | GeomAvg Error 9.501093e-06
======= classic_dopri5 | tol=1.000000e-03 =======
A1: NFE 74 | Time 0.0049016475677490234 | Err 5.004626e-04
A2: NFE 44 | Time 0.0033502578735351562 | Err 7.709617e-04
A3: NFE 140 | Time 0.009174346923828125 | Err 1.088762e-02
A4: NFE 32 | Time 0.002538919448852539 | Err 6.432366e-03
A5: NFE 32 | Time 0.0023818016052246094 | Err 1.177826e-02
B1: NFE 308 | Time 0.02685070037841797 | Err 9.802774e-02
B2: NFE 158 | Time 0.010577678680419922 | Err 1.145038e-04
B3: NFE 86 | Time 0.007359027862548828 | Err 9.037543e-05
B4: NFE 176 | Time 0.01804518699645996 | Err 8.083184e-02
B5: NFE 164 | Time 0.013936519622802734 | Err 1.195584e-01
C1: NFE 98 | Time 0.0066378116607666016 | Err 1.776586e-04
C2: NFE 422 | Time 0.02778148651123047 | Err 2.113468e-04
C3: NFE 188 | Time 0.012477397918701172 | Err 4.900898e-04
C4: NFE 194 | Time 0.01314854621887207 | Err 2.432564e-04
C5: NFE 38 | Time 0.006754875183105469 | Err 1.638266e-01
D1: NFE 212 | Time 0.02073192596435547 | Err 3.351970e-01
D2: NFE 218 | Time 0.021211862564086914 | Err 9.807782e-01
D3: NFE 266 | Time 0.025694608688354492 | Err 6.840998e-01
D4: NFE 338 | Time 0.03253650665283203 | Err 6.834353e-02
D5: NFE 494 | Time 0.04722785949707031 | Err 1.531686e+00
E1: NFE 98 | Time 0.010416507720947266 | Err 4.326926e-03
E2: NFE 452 | Time 0.03856348991394043 | Err 2.261163e-02
E3: NFE 236 | Time 0.02242445945739746 | Err 9.629339e-02
E4: NFE 74 | Time 0.006181001663208008 | Err 1.761328e-03
E5: NFE 74 | Time 0.006783008575439453 | Err 7.778303e-04
Total NFE 4616 | Total Time 0.3976874351501465 | GeomAvg Error 8.988490e-03
======= classic_dopri5 | tol=1.000000e-04 =======
A1: NFE 92 | Time 0.0060770511627197266 | Err 2.916632e-05
A2: NFE 56 | Time 0.004297018051147461 | Err 1.017095e-04
A3: NFE 218 | Time 0.014070987701416016 | Err 8.912369e-04
A4: NFE 50 | Time 0.0039031505584716797 | Err 8.106782e-04
A5: NFE 50 | Time 0.003450155258178711 | Err 2.529341e-04
B1: NFE 416 | Time 0.03628230094909668 | Err 6.827358e-03
B2: NFE 176 | Time 0.011710882186889648 | Err 9.780602e-05
B3: NFE 98 | Time 0.008459806442260742 | Err 1.333701e-05
B4: NFE 248 | Time 0.025453805923461914 | Err 6.364400e-03
B5: NFE 218 | Time 0.018602609634399414 | Err 4.537811e-04
C1: NFE 110 | Time 0.0074765682220458984 | Err 1.844306e-05
C2: NFE 416 | Time 0.027292728424072266 | Err 1.150706e-04
C3: NFE 206 | Time 0.013575077056884766 | Err 3.586540e-05
C4: NFE 200 | Time 0.013570547103881836 | Err 1.392522e-05
C5: NFE 38 | Time 0.006752490997314453 | Err 1.381693e-02
D1: NFE 242 | Time 0.023567676544189453 | Err 2.440569e-01
D2: NFE 278 | Time 0.027112483978271484 | Err 1.198586e-01
D3: NFE 356 | Time 0.03470158576965332 | Err 5.292218e-02
D4: NFE 458 | Time 0.04431796073913574 | Err 6.164090e-02
D5: NFE 722 | Time 0.06937789916992188 | Err 6.468093e-02
E1: NFE 146 | Time 0.015445232391357422 | Err 4.083396e-04
E2: NFE 512 | Time 0.04418444633483887 | Err 3.432746e-03
E3: NFE 398 | Time 0.03742623329162598 | Err 8.040900e-03
E4: NFE 74 | Time 0.00635528564453125 | Err 3.466123e-04
E5: NFE 74 | Time 0.0068056583404541016 | Err 2.973533e-04
Total NFE 5852 | Total Time 0.5102696418762207 | GeomAvg Error 1.015368e-03
======= classic_dopri5 | tol=1.000000e-05 =======
A1: NFE 116 | Time 0.007664918899536133 | Err 1.027875e-05
A2: NFE 68 | Time 0.005214214324951172 | Err 1.129971e-05
A3: NFE 314 | Time 0.02040719985961914 | Err 1.153991e-06
A4: NFE 74 | Time 0.0057201385498046875 | Err 8.822567e-05
A5: NFE 62 | Time 0.004254579544067383 | Err 1.474818e-04
B1: NFE 596 | Time 0.052278757095336914 | Err 2.192278e-04
B2: NFE 194 | Time 0.012949466705322266 | Err 5.179664e-06
B3: NFE 122 | Time 0.010641813278198242 | Err 5.022381e-06
B4: NFE 380 | Time 0.03914618492126465 | Err 9.838870e-05
B5: NFE 338 | Time 0.028708219528198242 | Err 1.600651e-04
C1: NFE 146 | Time 0.01000833511352539 | Err 1.944324e-06
C2: NFE 428 | Time 0.028006553649902344 | Err 5.088504e-06
C3: NFE 224 | Time 0.01486349105834961 | Err 6.746738e-06
C4: NFE 218 | Time 0.014754533767700195 | Err 5.464824e-06
C5: NFE 50 | Time 0.008871078491210938 | Err 9.863673e-04
D1: NFE 308 | Time 0.030380725860595703 | Err 1.727389e-02
D2: NFE 398 | Time 0.03881502151489258 | Err 9.095499e-03
D3: NFE 482 | Time 0.046936988830566406 | Err 4.932397e-03
D4: NFE 626 | Time 0.060646772384643555 | Err 4.744445e-03
D5: NFE 920 | Time 0.08882927894592285 | Err 4.288838e-03
E1: NFE 236 | Time 0.024759531021118164 | Err 3.355515e-05
E2: NFE 740 | Time 0.06419014930725098 | Err 2.474431e-04
E3: NFE 584 | Time 0.055007219314575195 | Err 4.727776e-04
E4: NFE 92 | Time 0.007829904556274414 | Err 1.907472e-05
E5: NFE 98 | Time 0.008855581283569336 | Err 6.119212e-05
Total NFE 7814 | Total Time 0.6897406578063965 | GeomAvg Error 8.798166e-05
======= classic_dopri5 | tol=1.000000e-06 =======
A1: NFE 164 | Time 0.010822534561157227 | Err 3.351860e-07
A2: NFE 98 | Time 0.007381439208984375 | Err 4.362106e-07
A3: NFE 488 | Time 0.03133702278137207 | Err 9.324239e-06
A4: NFE 98 | Time 0.007458209991455078 | Err 5.360427e-06
A5: NFE 86 | Time 0.0059070587158203125 | Err 9.322541e-06
B1: NFE 902 | Time 0.07929325103759766 | Err 2.872437e-05
B2: NFE 242 | Time 0.016164302825927734 | Err 6.248520e-07
B3: NFE 182 | Time 0.015682220458984375 | Err 2.257513e-07
B4: NFE 602 | Time 0.06194925308227539 | Err 1.843895e-05
B5: NFE 500 | Time 0.04262804985046387 | Err 1.895112e-05
C1: NFE 206 | Time 0.014014244079589844 | Err 1.402909e-07
C2: NFE 446 | Time 0.029422283172607422 | Err 3.637485e-07
C3: NFE 248 | Time 0.016567707061767578 | Err 1.835208e-07
C4: NFE 242 | Time 0.01653909683227539 | Err 3.069871e-07
C5: NFE 80 | Time 0.014023780822753906 | Err 7.181224e-05
D1: NFE 476 | Time 0.04698610305786133 | Err 6.669330e-04
D2: NFE 590 | Time 0.05748486518859863 | Err 4.043924e-04
D3: NFE 734 | Time 0.07121038436889648 | Err 1.116423e-04
D4: NFE 950 | Time 0.09186553955078125 | Err 6.592285e-05
D5: NFE 1370 | Time 0.13202929496765137 | Err 3.119618e-04
E1: NFE 374 | Time 0.039215087890625 | Err 3.344464e-06
E2: NFE 1172 | Time 0.1013031005859375 | Err 1.514140e-05
E3: NFE 836 | Time 0.07946562767028809 | Err 3.218069e-05
E4: NFE 134 | Time 0.011386632919311523 | Err 1.655161e-06
E5: NFE 134 | Time 0.012078046798706055 | Err 1.853281e-06
Total NFE 11354 | Total Time 1.0122151374816895 | GeomAvg Error 6.188659e-06

If nothing else, this would allow apples--to--apples benchmarking with Julia packages, since OrdinaryDiffEq.jl has the standard tableau as the standard option afaik. Classic dopri5 might also be useful to those mainly training Neural ODEs for ML tasks at tolerances ~10^{-3}. Not sure if of interest, but there you go!