GTFS R standard

[polettif]

Continuing discussions from gtfs-router#74. Pinging @mpadge, @dhersz, @rafapereirabr, @tbuckl

• GTFS standards describe column types which @mpadge listed in an R file.
• For most types, R does not have an equivalent built-in classe (like time, date, language code, URL, etc.).
• A "GTFS R standard" could define how gtfs files are represented in R which would help with interoperability between packages.

My thoughts on this:

• The most basic standard is: Use only base R data types (logical, numeric, integer and character). IMO this means, dates and times are not converted.
• ENUM could be implemented as factors? I don't really like and use factors though to be honest, I never quite see the benefit while I'm always wary that I'm losing data (via NAs) somehow.
• The discussion in gtfs-router#74 already pointed in a difficult direction: How times and dates are handled in a function or package is mainly dependent on the use case. For pure calculations, integers are great but with human interaction they don't mean much. HMS is simply a wrapper for integers but there's added overhead which slows calculations down.
• I think completely separating interface and implementation for gtfs is quite difficult in R. @mpadge, you said:

There is no underlying difference in between data.table and other representations (like from tidytransit) - all are stored in memory in the same way, and lists of columns, so this should work pretty seamlessly (data.table would stay data.table, same for data.frame and whatever else).

I honestly don't know enough about this but when working with data.table I can't use the same syntax as I do with tibble and/or data.frame. Are data.table's simply lists of columns? Where does all the modify-by-reference magic happen then?

• Would tidytransit switching to raw data.frames benefit other packages?

[polettif]

The most basic standard is: Use only base R data types (logical, numeric, integer and character).

To add to that: It's important to separate GTFS data (that can be read or written) and additional "analysis data" that gets added to the feed. Additional data in tidytransit is added in a sublist gtfs_obj$. (e.g. set_servicepattern) or as additional columns (set_hms_times). For stops, the whole data.frame can be changed (gtfs_as_sf).

Maybe the standard should rather handle how data is added to a feed, with naming conventions for additional columns, common function prefixes, how to handle replacement functions (f(x) <-y), how additional data is stored and so on.

[dhersz]

Great question!

I have always find the gtfs_obj$. very ingenious, and I really like using this to add large amounts of analysis data to the feed. I also think that's a clever way of separating "analysis-only" data to "actual gtfs" data (correct me if I'm wrong, but anything inside gtfs_obj$. is not written to to disk when you use tidytransit::write_gtfs(), right?). Let's say we add a frequencies "file" to the GTFS object, I believe that the correct way to do so would be to create a frequencies dataframe inside gtfs_obj, and not inside gtfs_obj$.. Users that deal with "non-standard" GTFS files could also create/deal with dataframes inside the GTFS object, instead of dealing it from ., as they probably wish to keep their non-standard files after manipulating it.

As far as columns go, I have some questions. After using set_hms_times(), which time columns are written to disk with write_gtfs()? The original character columns or the new hms columns? Are character columns kept up to date with the hms columns, if the latter are altered by a tidytransit function?

Perhaps "analysis columns" should also be handled like "analysis dataframes", thus better located inside gtfs_obj$.? Not sure. I had never puth much thought into it, but this is definitely something we should take a good look at.

(edit: ok, placing analysis columns out of their original dataframe does not make much sense lol, a naming convention is probably much more useful)

[polettif]

Yes, dataframes in gtfs_obj$. are not written by write_gtfs(). The new hms times columns are also not written to file, the original character column is kept as is and used by write_gtfs.

Are character columns kept up to date with the hms columns, if the latter are altered by a tidytransit function?

You bring up a good point, they are not kept up to date. However, I think implementing this is very atypical for R (as it's rather an OOP approach). It's easy with set_*(gtfs_obj) functions but implementing a side effect with g$table$hms_col <- y sounds tricky.

Perhaps "analysis columns" should also be handled like "analysis dataframes", thus better located inside gtfs_obj$.?

Yeah, your edit sums up the problem with this approach quite nicely :)

On a general side note: RStudio seems to have issues with code completion with gtfs_obj$. or maybe that's just me.

[mpadge]

Thanks @polettif for starting the discussion, and I think and hope this'll be a great place for us all to work towards common standards. A couple of thoughts in response:

1. The idea of using factors as ENUM is interesting. I hadn't thought about that, because I really dislike factors. The thing about an ENUM is it's just a raw representation with no class at all, whereas factors in R are kind of the opposite, in that they take a base type and actually add additional stuff. So they're kind of the opposite of what ENUMs are meant to be. But they're still probably the closest R gets, so I remain completely undecided, and look forward to further discussion.
2. The comments about no underlying difference between data.table and data.frame basically mean that the two formats can be interchanged (as.data.<whatever>) without needing to copy the object. That in turns means it's no issue at all, because conversion costs nothing, so it doesn't really matter what class tables are in. And in turn my suggestion would be that a standard can suggest that tables may be stored in as any class at all, as long as:
   • transforming to other classes can be done without copying the object; and
   • There is either clear documentation for how to do this; or explicit functions to convert.

[polettif]

formats can be interchanged (as.data.<whatever>) without needing to copy the object.

Maybe I'm not reading this correctly but as.data.table creates a copy of the object. That's why I think it's not simply a matter of using different S3 classes that just use different print methods. I do like the approach of clearly documenting conversion functions though.

[dhersz]

Hmm, {data.table} has a couple of functions to convert objects by reference (setDT and setDF) that ensues that the objects are not copied, but I'm not sure if there is a {tidyverse} equivalent to deal with tibbles accordingly.

[polettif]

I don't think there's an equivalent in the tidyverse since there's no modifying by reference for tibbles.

[dhersz]

Hi, thanks @polettif for creating this discussion.

ENUM could be implemented as factors? I don't really like and use factors though to be honest, I never quite see the benefit while I'm always wary that I'm losing data (via NAs) somehow.

I don't really like factors either, so apparently that's something we all have in common. And to be honest, I don't think that factors would help us here. I'll give an example. Let's suppose we have an integer vector holding route_types, this is conceivably what it'd look like:

> route_type <- c(1L, 2L, 3L, 3L, 1L)
> route_type
[1] 1 2 3 3 1

It's pretty easy to subset this vector based on a specific route_type we want to look. Here, for example, the usual route_type[route_type == 1] returns a length-2 vector holding only 1s, and something very similar could be done to filter the dataframes we use to represent the GTFS.

Now let's say that we convert this to a factor, where each label is the "meaning" behind each value (I'm assuming this is how you had thought about using the factors). We could do something like this (please, if there's a more clever way of using factors let me know, I really don't have that "intimacy" with this type):

> fac <- factor(route_type, labels = c("subway", "rail", "bus"))
> fac
[1] subway rail   bus    bus    subway
Levels: subway rail bus

Now the we have to select values based on their labels, not on their "underlying levels" (e.g 1, 2 and 3):

> fac[fac == 1]
factor(0)
Levels: subway rail bus
> fac[fac == "subway"]
[1] subway subway
Levels: subway rail bus

I see some problems here:

1. If we do that we are choosing to distance a bit from the original GTFS specification, and we would have to handle this when writing the data to disk, as well as in other cases.
2. For each ENUM in the official specification we would have to come up with clever short labels to allow for easy data manipulation (while this is easy for the route_type column, this is much harder for the continuous_drop_off column, for example).
3. We would have to find a way to communicate to the users how we have labelled each ENUM column, which may be very hard as well.

Perhaps that's what @mpadge meant by "ENUM is just a raw representation with no class at all, whereas factors in R are kind of the opposite", but I believe we would be better off had we had some type of data structure that preserved how the data is shown/manipulated, but in which we could add some "underlying meaning/context" to each value. Factors (AFAIK) do the opposite, so I don't think they'd be useful for us here.

[mpadge]

Yep, all great points @dhersz, and I entirely agree. There are other alternatives, and in particular vctrs has some really good type-stable behaviour mechanisms for R, but ... same problem, really, in that this adds more overhead, whereas an ENIUM really is and should be just raw, immutable, and either matches or errors. If we were to implement some approximation to that behaviour, we - as developers - would need to ensure that it was all effectively hidden both from users, and others wishing to adopt or modify our functionality. Some way to approximate behaviour of ENUM objects would be great, but the end result should also work and be workable across all packages regardless of which mechanism is chosen. Tricky!

[dhersz]

Perhaps we could just keep them as integers, as you proposed in you gtfs-reference implementation, but keep track of what values are "allowed" for each ENUM. That does not fix the fact that we cannot emulate ENUMs in R, but at least we can eventually keep track of "invalid" values as per the official reference.

Right now you implement the fields specifications as character vectors, which are fairly fast and easy to read, but perhaps we could add more metadata to them (e.g. the allowed values of each column) if we handled them as lists or data.frames, what do you think? ({tidytransit} and {gtfstools} now use lists, but perhaps data.frames are more readable)

I'll give an example how that could work. This is how the stops file is now currently implemented:

"stops" <- c ("stop_id"                 = "id",         # required
              "stop_code"               = "text",       # optional
              "stop_name"               = "text",       # cond. required
              "stop_desc"               = "text",       # optional
              "stop_lat"                = "latitude",   # cond. required
              "stop_lon"                = "longitude",  # cond. required
              "zone_id"                 = "id",         # cond. required
              "stop_url"                = "url",        # optional
              "location_type"           = "enum",       # optional
              "parent_station"          = "id",         # cond. required
              "stop_timezone"           = "timezone",   # optional
              "wheelchair_boarding"     = "enum",       # optional
              "level_id"                = "id",         # optional
              "platform_code"           = "text")       # optional

Perhaps we could do something like below (please note that I'm not listing the conditionally required fields as so, just for demonstration purposes):

stops_file_spec <- rep("req", 14)
stops_field <- c(
  "stop_id", "stop_code", "stop_name", "stop_desc", "stop_lat", "stop_lon",
  "zone_id", "stop_url", "location_type", "parent_station", "stop_timezone",
  "wheelchair_boarding", "level_id", "platform_code"
)
stops_field_spec <- c("req", "opt", "req", "opt", "req", "req", rep("opt", 8))
stops_coltypes <- c("id", "text", "text", "text", "latitude", "longitude", "id", "url", "enum", "id", "timezone", "enum", "id", "text")
stops_allowed_values <- list(NA, NA, NA, NA, NA, NA, NA, NA, c(0, 1, 2, 3, 4), NA, NA, c(0, 1, 2), NA, NA)

stops <- list(stops_file_spec, stops_field, stops_field_spec, stops_coltypes, stops_allowed_values)

P.S. I just found out, while writing this comment, that normal data.frames cannot handle list columns, so I opted for storing the final stops object in a list to be as "language" agnostic as possible. But we could use a data.table or a tibble here to facilitate visualisation.

[dhersz]

Another thought that has recently come up to me:

• How should we deal with classes?

Right now each package deals with them differently. {gtfs2gps} keeps them as a list, {tidytransit} and {gtfsrouter} set them as both list and gtfs and {gtfstools} sets a dt_gtfs class.

Why is this important? Right now I check for the class of the GTFS object in all {gtfstools} functions to make sure that I'm dealing with a GTFS generated by the package. I initially thought about setting a gtfs class, but then problems could eventually come up when trying to apply {gtfstools} functions to an {tidytransit} GTFS object. I suspect that functions from {tidytransit} and {gtfsrouter} also check for similar conditions, and the GTFS objects created by them would eventually fail in unexpected ways if trying to apply each others functions. Correct me if I'm wrong, please.

[mpadge]

@dhersz My suspicion would be that you are absolutely right there, and that this is really quite a problem

[mpadge]

How about this for a suggestion, by way of solution: We work together by pulling code from our various repos to make a single unifying gtfsio package that does nothing but read-write. All of our packages can then remove their own idiosyncratic read-write routines, we all depend on that single package, and everything is auto-magically unified and interoperable. And my suggestion, and I'm sure that of @dhersz as well, would be to use data.table as the underlying class for each table. Extending from @dhersz/s comments above about class structures, I would suggest that the final result also needs to be a class-based object, so we can all implement basic checks and stop random other things being passed through to our various functions. I'd be happy to start it off (as an r-transit repo, of course), but equally happy if anybody else particularly wanted to. What do we all think?

[rafapereirabr]

I think this is a good way forward. A simple package with fast read and write functions would make things much easier, as long as the functions are flexible enough to allow one to read/write specific columns.

[dhersz]

I really like this idea too!

I don't see any problems on using data.table as the underlying class for each table here. In fact I ran some tests to check if that does make converting to tibble harder, but apparently it does not:

z <- data.frame(x = rnorm(1000000), y = rnorm (1000000), z = rnorm(1000000))
readr::write_csv(z, "../random_stuff/random_table.csv")
microbenchmark::microbenchmark(
    df_to_tbl = fread("../random_stuff/random_table.csv", data.table = FALSE) %>% as_tibble(),
    dt_to_tbl = fread("../random_stuff/random_table.csv") %>% as_tibble(),
    just_tbl  = readr::read_csv("../random_stuff/random_table.csv"),
    times = 100L
)
#> Unit: milliseconds
#>       expr      min       lq      mean   median        uq       max neval
#>  df_to_tbl 120.2706 131.7505  142.8350 137.2665  146.5591  255.9231   100
#>  dt_to_tbl 121.1661 132.4258  145.8947 139.4506  153.1208  257.5641   100
#>   just_tbl 936.5799 955.1880 1009.1044 976.6649 1026.5981 1654.8540   100

Still, I'd like to hear @polettif opinion on the subject.

I don't think I can create repos under r-transit to start off the package.

[mpadge]

flexible enough to allow one to read/write specific columns.

also important to read/write only specified tables, which is the task that @dhersz presented us with to start off this whole discussion.

[polettif]

@mpadge Sounds good to me 👍 Feel free to start a repo in r-transit. BTW, I guess we'll go back to fragmentation that @tbuckl already pointed out.

@dhersz Thanks. I think tidytransit will keep using tibbles, so that's good to know.

We should also make sure to clearly outline what each package can do and provide links among them so people can find what they need (see also ropensci-archive/gtfsr#60)