Replies: 2 comments 6 replies
-
|
Overall sounds reasonable to me.
How do you envision this might influence design/architecture? I don't know about "planet scale" but at least nation-level deployments should be reasonable? Further down the line it could be interesting to explore something like what OpenTripPlanner Android does in being able to shard over multiple servers (potentially hosted by various entities). To me Headway is the most interesting in something that will be hosted in a distributed fashion by various independent parties, so design shouldn't be restricted to what you foresee yourself being able to pull off. (In case I'm projecting project vision too much here set it straight)
That's the way to go. So the long-lived "server" image should ideally be able to just kick off without any preprocessing inside the container. All data should be from a mounted volume. So basically ETL.
Makes sense! And as long as this is made like that it should work just as fine with other orchestration systems like Openshift, Nomad, etc, without headway having to be conscious about or support them. BTW, questions like the above are motivations behind twelve-factor app design patterns with IMO are best practices for good reason so def worth a read if you aren't familiar :) |
Beta Was this translation helpful? Give feedback.
-
|
Cool, thank you for weighing in! The more I've thought about it, the more I think #5 might be the last blocker before a planet-scale deployment becomes reasonable within my budget. I just switched to GraphHopper which works at planet scale if you can pre-generate the graph on a machine with ~128gb of RAM. Photon supposedly works at planet scale on fairly meager hardware (32GB RAM). Tiles are static content so serving them at planet scale is a solved problem. I just need to replace Nominatim with another source of truth because it requires 64GB of memory at a minimum for the planet. As far as how it may impact design, the only thing I can think of is that a multi-day import running in a k8s init container is probably a bummer, so it might make sense to pre-process data offline and then upload it to the preferred S3-compatible bucket of the week. Then the init container would only have to download the data and load it into volumes. A friend said he's willing to let me use 128gb of RAM on his server for a few days a month, so getting the RAM consumption of the entire stack into the 32-64GB range for planet.osm.pbf would make it affordable-ish to host on DigitalOcean or similar. |
Beta Was this translation helpful? Give feedback.
-
|
The one thing I'm kind of stuck on is whether it even makes sense to offer a canonical instance that covers the entire planet. If the goal is enabling privacy why am I soliciting people to send me their data like that? I know self hosting isn't for everyone but still it feels wrong. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW I could spare ~60 GB RAM for jobs and will probably try running this either way so happy to coordinate that.
Unless you're really motivated in doing that (which it sounds like you aren't ATM), I wouldn't bother at this point (and fully agreed on the last point; the goal shouldn't be to replace existing data silos with new ones). If the code and UX is good and it's straightforward enough to set up, I'm sure others will step up - and if not you could return to consider it later. Not to mention that if it gets popular, you may feel forced to spend more time and money on keeping the lights on than you bargained for. If you're running a smaller one with some local relevant areas for personal use and it's not too much to expose it for others, I'm sure that will also be appreciated and serve as a useful demo/test site, without the same scale of time and resources. Also helpful as a reference for self-hosters. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, it's easy to get caught up in what I can do and not what I should do. I think going commercial is the ultimate fate of anyone who tries to host tiles for the whole planet and I'd like to avoid that. I think I have a good vision of what this should be and I need to stick with it! HN comment threads also have a way of derailing. |
Beta Was this translation helpful? Give feedback.
-
|
Spinning off a discussion here from the comments of #10 It seems like we could just add an entry point script to all of the dockerfiles and then mount host directories as volumes https://stackoverflow.com/a/32398921 This really doesn't look that bad and it could completely remove all of those needless copies, ephemeral containers and |
Beta Was this translation helpful? Give feedback.
-
|
Indeed! As I mentioned in #10, in most cases actually adding the user inside the container is not necessary; most processes will run just fine with an arbitrary uid and in those cases you won't need either an env var, nor adding an initscript just for that sake. (The benefit of skipping that is you can start the container directly with the unprivileged user and don't have to rely on the container itself dropping privileges properly) |
Beta Was this translation helpful? Give feedback.
-
|
I will prove out an OpenTripPlanner build target that binds a host directory instead of creating volumes and copying back and forth. If I can get that working with the host user's UID/GID then I'll go back and convert the rest of the build targets to work that way. There will be a lot of churn in the process, but I think it will be worth it. Also now that it's obvious other people actually care about this project I'll also try and up my code quality a bit. It currently has built-in-a-week energy and I think it would be nice to make it a bit more professional. |
Beta Was this translation helpful? Give feedback.
-
Just wanted to have a central place to discuss how to deal with the docker configuration. I know @3nprob seemed to have opinions about this.
First, my assumptions about what headway should be and how it should be used:
Rationale for 1 is that it will eventually need to use OpenTripPlanner for transit, which doesn't extend to planet scale if I remember correctly. It's also incredibly expensive to host a planet-scale geocoder and unless this project has a reliable source of funding that's not in the cards. Rationales for 2 and 3 should be clear, but feel free to object obviously!
With all that said, I'd like to gather feedback about what people think it should look like. Right now all the data is collected into the container images, which isn't great. A friend of mine suggested a kubernetes init image to preprocess the data, which makes a lot of sense to me. I think that one container could potentially handle everything from downloading the extract to generating the geocoder indexes. The container could also check the freshness of the pre-processed data and conditionally skip steps if the data is fresh enough.
If k8s can run that job during the deployment process then it could be very simple to deploy updates. Once the data is all generated it could be put on a volume that the server containers could mount as read-only. For the docker-compose case, the makefile could just run that container locally to generate the volume then the rest of the containers could be run by docker-compose as they are right now.
Does this make sense?
Beta Was this translation helpful? Give feedback.
All reactions