Use cached index instead of GitHub API for recipe search #910

homebysix · 2023-10-29T23:42:05Z

This PR supersedes #862, and attempts to introduce a completely redesigned autopkg search capability based on parsing a precompiled search index rather than using GitHub's API to search for recipes dynamically. The index file is regenerated by GitHub Actions every 4 hours in this repo.

This PR also includes a revised print_gh_search_results() function that creates the table spacing dynamically given a list of dictionaries containing table data.

Example output

Basic search (cold cache)

% time ./Code/autopkg search coconutbattery
Refreshing local search index...

Name                                     Repo                  Path                                                                      
----                                     ----                  ----                                                                      
coconutBattery.jamf.recipe               grahampugh-recipes    Jamf_Recipes/coconutBattery.jamf.recipe                                   
coconutBattery-pkg-upload.jamf.recipe    grahampugh-recipes    Jamf_Package_Upload_Only_Recipes/coconutBattery-pkg-upload.jamf.recipe    
coconutBattery.munki.recipe              homebysix-recipes     coconutBattery/coconutBattery.munki.recipe                                
coconutBattery.pkg.recipe                homebysix-recipes     coconutBattery/coconutBattery.pkg.recipe                                  
coconutBattery.install.recipe            homebysix-recipes     coconutBattery/coconutBattery.install.recipe                              
coconutBattery.download.recipe           homebysix-recipes     coconutBattery/coconutBattery.download.recipe                             
coconutBattery.jss.recipe                jss-recipes           coconutBattery/coconutBattery.jss.recipe                                  
coconutBattery.filewave.recipe           peshay-recipes        coconut Battery/coconutBattery.filewave.recipe                            

To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search coconutbattery  0.42s user 0.15s system 8% cpu 6.370 total

Basic search (warm cache)

% time ./Code/autopkg search coconutbattery

Name                                     Repo                  Path                                                                      
----                                     ----                  ----                                                                      
coconutBattery-pkg-upload.jamf.recipe    grahampugh-recipes    Jamf_Package_Upload_Only_Recipes/coconutBattery-pkg-upload.jamf.recipe    
coconutBattery.jamf.recipe               grahampugh-recipes    Jamf_Recipes/coconutBattery.jamf.recipe                                   
coconutBattery.install.recipe            homebysix-recipes     coconutBattery/coconutBattery.install.recipe                              
coconutBattery.pkg.recipe                homebysix-recipes     coconutBattery/coconutBattery.pkg.recipe                                  
coconutBattery.munki.recipe              homebysix-recipes     coconutBattery/coconutBattery.munki.recipe                                
coconutBattery.download.recipe           homebysix-recipes     coconutBattery/coconutBattery.download.recipe                             
coconutBattery.jss.recipe                jss-recipes           coconutBattery/coconutBattery.jss.recipe                                  
coconutBattery.filewave.recipe           peshay-recipes        coconut Battery/coconutBattery.filewave.recipe                            

To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search coconutbattery  0.36s user 0.07s system 10% cpu 4.183 total

Search with no results

% time ./Code/autopkg search boguskeyword  
Nothing found.
To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search boguskeyword  0.36s user 0.07s system 9% cpu 4.373 total

Search with >100 results

% time ./Code/autopkg search munki       

Name                                                                     Repo                          Path                                                                                                                              
----                                                                     ----                          ----                                                                                                                              
macfuse.munki.recipe                                                     48kRAM-recipes                macFUSE/macfuse.munki.recipe                                                                                                      
yEd.munki.recipe                                                         48kRAM-recipes                yWorks/yEd.munki.recipe                                                                                                           
SweetHome3D.munki.recipe                                                 48kRAM-recipes                SweetHome3D/SweetHome3D.munki.recipe                                                                                              
EmacsForMacOSX.munki.recipe                                              48kRAM-recipes                Emacs/EmacsForMacOSX.munki.recipe                                                                                                 
osxfuse.munki.recipe                                                     48kRAM-recipes                OSXFUSE/osxfuse.munki.recipe                                                                                                      
kobo.munki.recipe                                                        aanklewicz-recipes            Kobo/kobo.munki.recipe                                                                                                            
GameSalad.munki.recipe                                                   aanklewicz-recipes            GameSalad/GameSalad.munki.recipe                                                                                                  
Download Full Installer.munki.recipe                                     aanklewicz-recipes            Download Full Installer/Download Full Installer.munki.recipe                                                                      
Prezi Video.munki.recipe                                                 aanklewicz-recipes            PreziVideo/Prezi Video.munki.recipe                                                                                               
SEB.munki.recipe                                                         aanklewicz-recipes            Safe Exam Browser/SEB.munki.recipe                                                                                                
SonicPi.munki.recipe                                                     aanklewicz-recipes            SonicPi/SonicPi.munki.recipe                                                                                                      
[...snip...]
Netiquette.munki.recipe                                                  zentral-recipes               Netiquette/Netiquette.munki.recipe                                                                                                
tailwindcss-cli.munki.recipe                                             zentral-recipes               tailwindcss/tailwindcss-cli.munki.recipe                                                                                          
JamfProSwitcher.munki.recipe                                             zentral-recipes               JamfProSwitcher/JamfProSwitcher.munki.recipe                                                                                      
NodeExporter.munki.recipe                                                znerol-recipes                Prometheus/NodeExporter.munki.recipe                                                                                              
TimeMachineExporter.munki.recipe                                         znerol-recipes                Prometheus/TimeMachineExporter.munki.recipe                                                                                       

To add a new recipe repo, use 'autopkg repo-add <repo name>'

WARNING: Search yielded more than 100 results. Please try a more specific search term.
./Code/autopkg search munki  0.37s user 0.09s system 10% cpu 4.467 total

Search with specific user/org

% time ./Code/autopkg search -u facebookarchive intellij
'autopkg search' no longer directly searches GitHub users or orgs other than the autopkg org.
However, this page may provide some useful results:
https://github.com/search?q=intellij+org%3Afacebookarchive+lang%3Axml+OR+lang%3Ayaml&type=code
./Code/autopkg search -u facebookarchive intellij  0.20s user 0.04s system 96% cpu 0.246 total

Search paths only

TODO

Search keyword with major version

% time ./Code/autopkg search FinalDraft12               

Name                                 Repo                 Path                                                
----                                 ----                 ----                                                
FinalDraft12.pkg.recipe              homebysix-recipes    FinalDraft/FinalDraft12.pkg.recipe                  
FinalDraft12.pkg.recipe.yaml         moofit-recipes       Final Draft 12/FinalDraft12.pkg.recipe.yaml         
FinalDraft12.download.recipe.yaml    moofit-recipes       Final Draft 12/FinalDraft12.download.recipe.yaml    
FinalDraft12.download.recipe         n8felton-recipes     FinalDraft/FinalDraft12.download.recipe             
FinalDraft12.munki.recipe            n8felton-recipes     FinalDraft/FinalDraft12.munki.recipe                

To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search FinalDraft12  0.36s user 0.07s system 10% cpu 4.340 total

Search keyword without major version

% time ./Code/autopkg search FinalDraft  

Name                                 Repo                 Path                                                
----                                 ----                 ----                                                
Final Draft 11.pkg.recipe            dataJAR-recipes      Final Draft 11/Final Draft 11.pkg.recipe            
Final Draft 11.download.recipe       dataJAR-recipes      Final Draft 11/Final Draft 11.download.recipe       
Final Draft 11.munki.recipe          dataJAR-recipes      Final Draft 11/Final Draft 11.munki.recipe          
FinalDraft12.pkg.recipe              homebysix-recipes    FinalDraft/FinalDraft12.pkg.recipe                  
FinalDraft12.download.recipe.yaml    moofit-recipes       Final Draft 12/FinalDraft12.download.recipe.yaml    
FinalDraft12.pkg.recipe.yaml         moofit-recipes       Final Draft 12/FinalDraft12.pkg.recipe.yaml         
FinalDraft12.munki.recipe            n8felton-recipes     FinalDraft/FinalDraft12.munki.recipe                
FinalDraft12.download.recipe         n8felton-recipes     FinalDraft/FinalDraft12.download.recipe             

To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search FinalDraft  0.36s user 0.08s system 10% cpu 4.283 total

Info search results find a single repo

% time ./Code/autopkg info "Final Draft 11.munki"
Looking for Final Draft 11.munki...
Did not find Final Draft 11.munki in recipe map
Rebuilding recipe map with current working directories...
Looking for Final Draft 11.munki...
Did not find Final Draft 11.munki in recipe map
Search GitHub AutoPkg repos for a Final Draft 11.munki recipe? [y/n]: y

Name                           Repo               Path                                          
----                           ----               ----                                          
Final Draft 11.munki.recipe    dataJAR-recipes    Final Draft 11/Final Draft 11.munki.recipe    


Add recipe repo 'dataJAR-recipes'? [y/n]: n
We are not currently searching on-disk for recipes; if you intended to run a specific file not in the recipe map, please pass in a real file path instead.
No valid recipe found for Final Draft 11.munki
./Code/autopkg info "Final Draft 11.munki"  0.50s user 0.10s system 5% cpu 10.619 total

Info search results find multiple repos

% time ./Code/autopkg info "Final Draft 12.download"
Looking for Final Draft 12.download...
Did not find Final Draft 12.download in recipe map
Rebuilding recipe map with current working directories...
Looking for Final Draft 12.download...
Did not find Final Draft 12.download in recipe map
Search GitHub AutoPkg repos for a Final Draft 12.download recipe? [y/n]: y

Name                                 Repo                Path                                                
----                                 ----                ----                                                
FinalDraft12.download.recipe.yaml    moofit-recipes      Final Draft 12/FinalDraft12.download.recipe.yaml    
FinalDraft12.download.recipe         n8felton-recipes    FinalDraft/FinalDraft12.download.recipe             

To add a new recipe repo, use 'autopkg repo-add <repo name>'

We are not currently searching on-disk for recipes; if you intended to run a specific file not in the recipe map, please pass in a real file path instead.
No valid recipe found for Final Draft 12.download
./Code/autopkg info "Final Draft 12.download"  0.45s user 0.08s system 9% cpu 5.644 total

Unit tests

% cd Code; /usr/local/autopkg/python -m unittest tests.test_searchcmd  
Nothing found.
To add a new recipe repo, use 'autopkg repo-add <repo name>'
.Nothing found.
To add a new recipe repo, use 'autopkg repo-add <repo name>'
.No search query specified!
.Nothing found.
To add a new recipe repo, use 'autopkg repo-add <repo name>'
.
----------------------------------------------------------------------
Ran 4 tests in 13.040s

OK

Search with no query

% time ./Code/autopkg search                            
No search query specified!
./Code/autopkg search  0.20s user 0.04s system 89% cpu 0.272 total

Path-only search

% time ./Code/autopkg search -p panic    

Name                                 Repo                   Path                                       
----                                 ----                   ----                                       
Transmit5.jamf-upload.recipe.yaml    davidbpirie-recipes    Panic/Transmit5.jamf-upload.recipe.yaml    
Coda.munki.recipe                    keeleysam-recipes      Panic/Coda.munki.recipe                    
Coda.download.recipe                 keeleysam-recipes      Panic/Coda.download.recipe                 
Coda2.munki.recipe                   keeleysam-recipes      Panic/Coda2.munki.recipe                   
Unison.munki.recipe                  keeleysam-recipes      Panic/Unison.munki.recipe                  
Coda2.download.recipe                keeleysam-recipes      Panic/Coda2.download.recipe                
Transmit.download.recipe             keeleysam-recipes      Panic/Transmit.download.recipe             
Unison.download.recipe               keeleysam-recipes      Panic/Unison.download.recipe               
Transmit.munki.recipe                keeleysam-recipes      Panic/Transmit.munki.recipe                
Coda2.munki.recipe                   recipes                Panic/Coda2.munki.recipe                   
Transmit.munki.recipe                recipes                Panic/Transmit.munki.recipe                
Transmit5.download.recipe            recipes                Panic/Transmit5.download.recipe            
Transmit.download.recipe             recipes                Panic/Transmit.download.recipe             
Transmit5.munki.recipe               recipes                Panic/Transmit5.munki.recipe               
Transmit.install.recipe              recipes                Panic/Transmit.install.recipe              
Coda2.download.recipe                recipes                Panic/Coda2.download.recipe                
Transmit5.install.recipe             recipes                Panic/Transmit5.install.recipe             
Coda2.install.recipe                 recipes                Panic/Coda2.install.recipe                 
Coda2.pkg.recipe                     recipes                Panic/Coda2.pkg.recipe                     
Transmit5.pkg.recipe                 recipes                Panic/Transmit5.pkg.recipe                 
Transmit.pkg.recipe                  recipes                Panic/Transmit.pkg.recipe                  

To add a new recipe repo, use 'autopkg repo-add <repo name>'
./Code/autopkg search -p panic  0.36s user 0.08s system 9% cpu 4.523 total

To do / help wanted

Consider these items before merging:

Build more test cases, including for new print print_gh_search_results() and get_search_results() functions
Write feature announcement to be used in release notes
Consider how to integrate new search capability into GitHubSession.search_for_name() function
Consider how to integrate new search capability into get_repository_from_identifier() function
Consider whether the scope and structure of the data in the recipe search index is appropriate for this feature launch
Add example output above

homebysix · 2024-02-12T21:40:33Z

@nmcspadden I'm inclined to merge this to dev now, and finish up some of the rough edges (specifically what to do about GitHubSession.search_for_name()) between now and the 3.0 release. Otherwise it's likely a worse merge conflict will form later.

nmcspadden · 2024-02-13T06:56:25Z

Can you do a quick test on the dev_new_recipe_logic branch and see if it still works? If not, I'll take it and resolve the merges.

homebysix added 11 commits October 26, 2023 09:05

Leverage cached search index instead of searching GitHub directly

3030da9

Provide URL to GitHub results for non-autopkg search

c5c031e

Deprecate --use-token option

3f2b929

Align capitalization on search-related warnings

0f0d319

GitHub user/org format validation

969cf3c

Log warnings as error

16708c6

Use log() instead of print() for output

ac59885

Merge branch 'dev' into dev_search_index2

d82b832

Make external user/org search context more specific

a435c48

Align test return codes with new search

55828d6

Add type hints to new functions

68b9f9b

homebysix mentioned this pull request Oct 29, 2023

Use cached index instead of GitHub API for recipe search #862

Closed

homebysix added 3 commits October 29, 2023 17:28

Account for default GH user 'autopkg'

53b3ac6

Support path-only searches

327e9e9

Handle missing GitHub token

74ab091

homebysix changed the title ~~WIP: Use cached index instead of GitHub API for recipe search~~ Use cached index instead of GitHub API for recipe search Feb 12, 2024

homebysix marked this pull request as ready for review February 12, 2024 21:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use cached index instead of GitHub API for recipe search #910

Use cached index instead of GitHub API for recipe search #910

homebysix commented Oct 29, 2023 •

edited

homebysix commented Feb 12, 2024

nmcspadden commented Feb 13, 2024

Use cached index instead of GitHub API for recipe search #910

Are you sure you want to change the base?

Use cached index instead of GitHub API for recipe search #910

Conversation

homebysix commented Oct 29, 2023 • edited

Example output

Basic search (cold cache)

Basic search (warm cache)

Search with no results

Search with >100 results

Search with specific user/org

Search paths only

Search keyword with major version

Search keyword without major version

Info search results find a single repo

Info search results find multiple repos

Unit tests

Search with no query

Path-only search

To do / help wanted

homebysix commented Feb 12, 2024

nmcspadden commented Feb 13, 2024

homebysix commented Oct 29, 2023 •

edited