Skip to content

jrdmb/cosmotalks-datasette

Repository files navigation

cosmotalks-datasette

This is a work-in-progress on a database collection of links to research or academic talks by cosmologists intended for others in the field. The data is currently gathered from 205 diverse sources (called 'series') that have published talks from conferences, seminars, workshops, course lectures, summer schools, colloquia, etc. Currently 7,011 online talks are listed, and more are forthcoming.

This collection was started out of personal interest as a service to the cosmology community because no one site was found that comprehensively compiles such data from a wide range of sources. There are many sources available so this represents only a fraction of the total that's out there; still, it's a starter attempt to extend beyond what's otherwise available in one place.

The basic data structure is a talk title, including presenter(s), the series site hosting the talk video (e.g., Cosmology Talks on YouTube), and a talk date (yyyy-mm-dd format). Partial dates may exclude dd or mm-dd but any date entry should include yyyy as a minimum.

There are two database application versions available for querying the data (it's nice to have options with different features!). The first version has the most up-to-date live data and is recommended for most purposes. The second app is more full-featured but by nature of the Google Cloud Run service on which it's hosted, the database is static and thus does not have the very latest data updates until the app is periodically rebuilt and uploaded to Cloud Run. New data is published to it approximately every 2 weeks. A list of talks that have been collected (and are present in the first version) but are pending a periodic update into the Cloud Run app is available here.

The following info below has details about the second version of the app, its code, and how it's built.

Datasette is the open-source tool used to publish the data. It was created by Simon Willison (github repo, @simonw on twitter). Many thanks to Simon for making this excellent tool available - it's a great match for this project and also especially for projects utilizing SQLite database files. This app can be self-hosted locally or remotely using Datasette, and it can also be hosted as a Docker container - see the Publishing Data section of the Datasette documentation.

Installation

This is an overview of the general procedure to build and run this locally or remotely, perhaps adding your own data to the .csv source files. The notes below are specific to a linux install. Simon's docs include instructions for a Datasette installation on a Mac.

SQLite3 is required and assumed to be installed.

Install Datasette

Install required Datasette plugins csvs-to-sqlite, datasette-render-markdown, and datasette-template-sql.

Build the cosmotalks.db database file from source .csv files (cosmotalks.csv and series.csv) by executing bash shell script build-db.sh. If you wish to add your own data, first edit files cosmotalks.csv and series.csv. Note the comments in the shell script.

An option for the db build: many of the videos are hosted on YouTube which has a feature providing a link to a snapshot image file of the video. An example of such an image is included at the bottom of this page*. (Thanks to Simon Willison for the tip about this feature.) To include references to these snapshots requires execution of PHP program populate-ytids-table.php during the build. If the snapshots are desired and your setup has the ability to execute PHP programs, the build script automatically runs the program. If you don't want the images or you don't have PHP available, delete the populate-ytids-table.php php program before executing the build.

Running the Cosmotalks app

After the build step, the basic command and optional parameters to execute the app are listed in the documentation here. Also refer to the Deploying Datasette section of the documentation. The command I use to start the app on a local linux server is:

nohup datasette cosmotalks.db -m metadata.json --template-dir=templates/ --plugins-dir=plugins/ --setting suggest_facets off --setting default_facet_size 15 2>&1 &

The app listens on port 8001 by default, a different port can be specified with the -p parameter.

The web app can then be accessed locally at http://127.0.0.1:8001. To run it behind a proxy for remote access, see this doc section. See this for other deployment options.

Cosmology Resource Materials page

Users of this database may also find useful the Cosmology Resource Materials page.


* Example of the optional YouTube image feature:

youtube snapshot image

About

Database app of links to 7,011 online talk videos by cosmology researchers presenting their work to others in the field. Data is from 205 publicly available sources. Database app is hosted on Google Cloud Run at:

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published