Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PMTiles S3 bucket support with authentication #1171

Open
4 tasks
nyurik opened this issue Feb 2, 2024 · 2 comments
Open
4 tasks

Implement PMTiles S3 bucket support with authentication #1171

nyurik opened this issue Feb 2, 2024 · 2 comments

Comments

@nyurik
Copy link
Member

nyurik commented Feb 2, 2024

pmtiles crate will soon support s3 buckets directly. Martin needs to enable support for S3-hosted pmtiles

TODO

  • create a new PmtS3Source similar to PmtHttpSource
  • establish config settings for pmtiles on s3, including authentication
  • establish CLI parameters for pmtiles on s3
  • update martin book with usage examples

Developers can already use by adding this at the end of the root Cargo.toml:

[patch.crates-io]
pmtiles = { git = "https://github.com/stadiamaps/pmtiles-rs", branch = "add-s3-backend" }

See also #1125
CC: @alexjball @lseelenbinder

@nyurik
Copy link
Member Author

nyurik commented Feb 2, 2024

for configuration, we currently have this pattern:

pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as a pmt source (filename without extension)
    - /path/to/pmt.pmtiles
    # A web server with a PMTiles file that supports range requests
    - https://example.org/path/tiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: /path/to/pmt.pmtiles
    # A named source to a web server with a PMTiles file that supports range requests
    pm-web2: https://example.org/path/tiles.pmtiles

We can add s3 support as either an s3 subsection under pmtiles, or by adding a new top level pmtiles-s3. I think the first option is better, but it might be trickier to implement.

pmtiles:
  s3:
    [...something...]

pmtiles-s3:
    [...something...]

Config should cover the following cases:

  • one s3 bucket with a single pmtiles file
  • one s3 bucket with multiple pmtiles files
    • file names are listed in the config
    • files are auto-discovered (using s3 enumeration - requires additional permission)
  • multiple s3 buckets

So it seems the usage is similar to postgres configuration, where we can have one or more pg connection (i.e. s3 authentication), and each connection can have one or more table (i.e. pmtiles files), and the table name can be declared in the config (with the corresponding source name), or they can be auto-discovered, in which case the source name is auto-generated, e.g. {filename} or {bucket}_{filename}.

@jleedev
Copy link
Collaborator

jleedev commented Feb 3, 2024

The protomaps/PMTiles supports specifying a bucket (and a path prefix), and allows fetching any pmtiles object by name, rather than having to specify them all ahead of time, or listing the objects. Would require some refactoring here, but it's a sensible way to avoid expensive reconfigurations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants