UTC-but-not-UTC confusion

[elazar]

I suspect this isn't an issue with rrule, but with my understanding of it and/or its timezone support. I've already read related sections in the documentation for rrule and luxon.

Code sample

const { RRule, RRuleSet } = require(`rrule`);

const dtstart = new Date();
console.log(`dtstart`, dtstart);

const firstHour = dtstart.getHours();
const secondHour = (firstHour + 2) % 24;

const getRRule = options => new RRule(Object.assign({}, options, {
    freq: RRule.DAILY,
    count: 1,
    dtstart,
    byminute: 0,
    bysecond: 0
}));

const showNextInstance = tzid => {
    const ruleSet = new RRuleSet();
    ruleSet.rrule(getRRule({ tzid, byhour: firstHour }));
    ruleSet.rrule(getRRule({ tzid, byhour: secondHour }));
    console.log(tzid, ruleSet.after(dtstart));
};

showNextInstance(undefined);
showNextInstance(`UTC`);
showNextInstance(`local`);
showNextInstance(`America/Chicago`);

Actual output

dtstart 2019-04-05T11:51:23.744Z
undefined 2019-04-06T06:00:00.744Z
UTC 2019-04-06T06:00:00.744Z
local 2019-04-06T06:00:00.744Z
America/Chicago 2019-04-06T06:00:00.744Z

Expected output

I would think the specification of tzid would have some effect on the output, but the same output is generated for each timezone.

I also can't see what the correlation is between the times provided in the rules and the output for the next event instance. I'd expect that the next instance would roughly correspond to one of the two hours represented by the supplied rules depending on how the specified dtstart and tzid values are interpreted or applied.

rrule version

2.6.0

Operating system

OS X 10.14.2, Node 8.11.4

Local timezone

$ date
Fri Apr  5 06:32:33 CDT 2019

[Noitidart]

I'm also terribly confused.

[Noitidart]

Ok I got it, there was this key paragraph in the readme:

You have to use these two helper fucntions for the local timezone. You would have to modify offset to get the offset for the X timezone if you want to modify this to support tzid.

/**
 * Mutates date. Add timezone to the date.
 *
 * @param {Date} [date]
 */
export function localizeDate(date = new Date()) {
  const offset = new Date().getTimezoneOffset() * 60 * 1000; // get offset for local timezone (can modify here to support tzid)
  date.setTime(date.getTime() - offset);
  return date;
}

/**
 * Mutates a date. De-timezones a date.
 *
 * @param {Date} [date]
 */
export function utcifyDate(date = new Date()) {
  const offset = new Date().getTimezoneOffset() * 60 * 1000; // get offset for local timezone (can modify here to support tzid)
  date.setTime(date.getTime() + offset);
  return date;
}

Then your code would change to:

const { RRule, RRuleSet } = require(`rrule`);

const dtstart = localizeDate(new Date());
console.log(`dtstart`, utcifyDate(new Date(dtstart))); // even though it is utc string, this is the local time

const firstHour = dtstart.getUTCHours();
const secondHour = (firstHour + 2) % 24;

const getRRule = options => new RRule(Object.assign({}, options, {
    freq: RRule.DAILY,
    count: 1,
    dtstart,
    byminute: 0,
    bysecond: 0
}));

const showNextInstance = tzid => {
    const ruleSet = new RRuleSet();
    ruleSet.rrule(getRRule({ byhour: firstHour })); // remove tzid as i dont support this
    ruleSet.rrule(getRRule({ byhour: secondHour })); // removed tzid as i dont support this
    console.log('local:', utcifyDate(ruleSet.after(dtstart));
};

showNextInstance(undefined);
showNextInstance(`UTC`);
showNextInstance(`local`);
showNextInstance(`America/Chicago`);

[davidgoli]

I suspect there's not going to be a good way around this in this library, built as it is on JS' confusing and irregular Date implementation. I'm presently working on a complete rewrite, but it will take some time to get it right.

[Noitidart]

Thanks @davidgoli your work is so super appreciated!!!

I think we need someone that understands to write an article to help others understand the irregular date implementation. I honestly still don't get it, I just tweaked things till it worked.

[ScarlettZebra]

It would be because the returned date that you're printing to the console is a JavaScript date that is being converted into a string, and JavaScript automatically prints Dates to console in UTC (See the 'Z' at the end of your timestamps that denotes zero offset). Since the next occurrences in each of your four cases are happening at the exact same time (although in different time zones), it's printing the exact same timestamp four times.

[ScarlettZebra]

Also note that the JavaScript Date object keeps track of time using the number of milliseconds that have elapsed since 00:00:00 UTC, Thursday, 1 January 1970 (Note the "UTC" there!), not by some kind of string interpretation of "12:34:56pm" or anything of the sort.

[jorroll]

@davidgoli

I'm presently working on a complete rewrite, but it will take some time to get it right.

You might want to check out rSchedule's source. All iteration is done with a special DateTime object (not from Luxon) that takes an input and converts it to a floating utc date that looks equivalent to the input. Iteration is done with this UTC datetime so DST isn't an issue. After an occurrence is found, before it is yielded to the user it is converted back to a normal date in the appropriate timezone.

For example: this date

import { DateTime as LuxonDateTime } from 'luxon';

const date = LuxonDateTime.fromObject({
  year: 2019,
  month: 1,
  day: 1,
  zone: 'America/Los_Angeles'
})

would be converted to this UTC date and the timezone would be saved

const fakeDate = Date.UTC(2019, 0, 1)
const fakeDateZone = 'America/Los_Angeles'

Note that this UTC date look similar to the Luxon date (they both look like 2019/1/1) but, since they are in different timezones, they don't actually represent the same time.

This being said, iterating with this special "UTC" date lets us iterate without worrying about daylight savings time. After the correct date is found, we can convert is back to a luxon date.

For example:

LuxonDateTime.fromObject({
  year: fakeDate.year,
  month: fakeDate.month,
  day: fakeDate.day,
  zone: fakeDateZone,
})

This keeps the iteration logic nice an UTC friendly while still supporting arbitrary timezones.

[davidgoli]

@thefliik I got the primitives down; it's weeks that are the real headache here. Though I see rschedule has also punted (so far) on byweekno & bysetpos support...

[jorroll]

Ah ya. byweekno is very annoying.

[shcoderAlex]

any news?

[tim-phillips]

I think it's easy to conflate issues with the Date object and problems calculating occurrences for RRULE, so I'm going to talk about them separately before combining the two. I had to write this out mostly so I could sort out my understanding of the problem and figured I'd post here in case it helps others.

JavaScript Date object

I don't agree that the JS Date object is irregular, but there are common misunderstandings with the object that do make it seem that way. It is confusing to say the least.

As @hlee5zebra mentioned, dates created with the Date constructor are always represented in UTC. They may be displayed in the local time, but behind the scenes they are in fact represented in Unix Epoch time. We can always run .getTime() or .toISOString() on the instantiated Date object to remind ourselves that we're working in UTC.

The implementation also correctly follows the convention of doing all processing in UTC and only converting to local time on display to the user. It just so happens that we, the developers, are the user; but then we have our own user in mind which muddies the water. When developing our apps, we must make sure to follow the convention and save dates as UTC in our backend (JS already processes dates in UTC so we're good there). If you store dates as UTC in your data store, they should arrive at the client as ISO strings, be parsed correctly by the Date object, and be displayed in the user's local time, all without doing any heavy lifting.

This write up has helped me understand the Date object and discover how much I can do with it: https://www.toptal.com/software/definitive-guide-to-datetime-manipulation

RRULE

Calculating occurrences for an RRULE gets tricky because there is a fundamental difference in user expectation and the processing of dates in UTC, noticeable when these two time representations cross a day barrier.

For example, if our user wants recurrence that occurs every Tuesday, they're going to expect all occurrences to fall on a Tuesday in their local time. However, all calculations are done in UTC, so an RRULE like:

DTSTART:2019-10-29T00:02:26Z
RRULE:FREQ=WEEKLY;BYWEEKDAY=TU

would return occurrences on Tuesdays in UTC, but for any user located significantly west of the prime meridian, occurrences would appear on Mondays in local time, which is obviously not the result anyone is hoping for.

Getting it right

All that said, what the developer must do to solve for this is force JavaScript to think that the local date is actually a UTC date, give that "fake" date to rrule.js, and have the library do its calculations in this local UTC time. When you get the occurrences from the .all() method, you then instantiate new Date objects using the "UTC" parts of those dates. This effectively makes the results match the expectation.

import { RRule } from 'rrule'

const dtstart = new Date('2019-10-29T00:02:26Z') // same as: `new Date(Date.UTC(2019, 9, 29, 0, 2, 26))`
console.log(dtstart.toISOString()) // 2019-10-29T00:02:26.000Z
console.log(dtstart) // Mon Oct 28 2019 17:02:26 GMT-0700 (Pacific Daylight Time)

const fakeDateStart = setPartsToUTCDate(dtstart)
console.log(fakeDateStart.toISOString()) // 2019-10-28T17:02:26.000Z
console.log(fakeDateStart) // Mon Oct 28 2019 10:02:26 GMT-0700 (Pacific Daylight Time)

const rule = new RRule({
  freq: RRule.WEEKLY,
  byweekday: [RRule.TU],
  count: 2,
  dtstart: fakeDateStart
})

const localUTCOccurrences = rule.all()
console.log(localUTCOccurrences.map(toISOString)) // ["2019-10-29T17:02:26.000Z", "2019-11-05T17:02:26.000Z"]
console.log(localUTCOccurrences) // [Tue Oct 29 2019 10:02:26 GMT-0700 (Pacific Daylight Time), Tue Nov 05 2019 09:02:26 GMT-0800 (Pacific Standard Time)]

const occurrences = localUTCOccurrences.map(setUTCPartsToDate)
console.log(occurrences.map(toISOString)) // ["2019-10-30T00:02:26.000Z", "2019-11-06T01:02:26.000Z"]
console.log(occurrences) // [Tue Oct 29 2019 17:02:26 GMT-0700 (Pacific Daylight Time), Tue Nov 05 2019 17:02:26 GMT-0800 (Pacific Standard Time)]

function setPartsToUTCDate(d) {
  return new Date(
    Date.UTC(
      d.getFullYear(),
      d.getMonth(),
      d.getDate(),
      d.getHours(),
      d.getMinutes(),
      d.getSeconds()
    )
  )
}

function setUTCPartsToDate(d) {
  return new Date(
    d.getUTCFullYear(),
    d.getUTCMonth(),
    d.getUTCDate(),
    d.getUTCHours(),
    d.getUTCMinutes(),
    d.getUTCSeconds()
  )
}

function toISOString(d) {
  return d.toISOString()
}

https://codesandbox.io/s/rrule-localutc-conversion-zxlki

Or in short:

import { RRule } from 'rrule'

const date = new Date('2019-10-29T00:02:26Z')

const rule = new RRule({
  freq: RRule.WEEKLY,
  byweekday: [RRule.TU],
  count: 2,
  dtstart: setPartsToUTCDate(date)
})

const occurrences = rule.all().map(setUTCPartsToDate)

https://codesandbox.io/s/rrule-localutc-conversion-in-short-ez7g0

As far as I can see, Chrome, Firefox, and Safari all display dates as strings in local time when logging to the console, and Node.js as ISO strings in UTC.

[jbrodie]

@tim-phillips I am also suffering for this right now and looking for some sort of monkey patch to get around it.

I have a recurring event at 8 PM EST that works fine, but 9 PM EST is pushing it back to the day before it should be.

Looking forward to see if you can find an answer to this.

[Noitidart]

@tim-phillips that is an excellent breakdown. I'm still going through it, but it looks like it will help me finally understand, thank you!

[tonylau]

@tim-phillips Thank you for your analysis.

However, can you explain the following? It seems your solution still can return Wednesday depending on your computer's local timezone.

[davidgoli]

@tonylau don't look at the stringified output in your console, which will always be converted into your local timezone. You should use the getUTC*() methods to get the parts of the date, which will be the same regardless of your local timezone.

[tonylau]

@davidgoli

I set my timezone to UTC+14 (Kiritimati Island/Line Islands Time) and got the following, even when I'm using getUTCDay(). The day is still returned as Wednesday when it should be Tuesday?

Sandbox: https://codesandbox.io/s/rrule-localutc-conversion-in-short-0uzt1

[elazar]

Potentially relevant to this discussion: https://github.com/tc39/proposal-temporal

[jessicaelee]

@tim-phillips thank you!!!!!!!!!! that was so helpful!!!!!!! Do you happen to know how to make the ics with the rrule? I am having the day issue again when I use the ics package in node and when I receive the calendar invites, they are a day off. (But I'm able to put everything in my db correctly because of your code!! thank you so much. )

[bouncingbumble]

@tim-phillips you absolute legend