Support ICS files

[xmha97]

Feature

Support ICS files.
https://en.wikipedia.org/wiki/ICalendar

Sample ICS file:

BEGIN:VCALENDAR
PRODID:-//Microsoft Corporation//Outlook MIMEDIR//EN
VERSION:1.0
BEGIN:VEVENT
DTSTART:19980114T210000Z
DTEND:19980114T230000Z
LOCATION:My office
CATEGORIES:Business
DESCRIPTION;ENCODING=QUOTED-PRINTABLE:This is a note associated with the meeting=0D=0A
SUMMARY:Meeting to discuss salaries
PRIORITY:3
END:VEVENT
END:VCALENDAR

See this repository for more calendars: https://github.com/xmha97/Calendars

[anufrievroman]

I looked into it, indeed the database could be in .ics and the file could be synced for example using vdirsync. I tried, and it seems to work, but the first problem I see that although technically ISC format is clear, actually everybody uses slightly different format.
Nextcloud, for example, creates me a separate ICS file for each event. Other platforms, as far as I see, can put many events into the same file, which becomes incompatible with Nextcloud...

Also, not all entries seem to be so universal. Nextcloud for example writes DTSTART;VALUE=DATE:20220129 while standard specification wants DTSTART:19980114T210000Z, not sure if that might be an issue.

Then, it's also not clear to me how to manage todo tasks. Should they be in the same file, in a different file, or each in a different file.

p.s. I am new to this system, so maybe I just don't get it...

[xmha97]

My events are in Google Calendar and I want to see them in this app.
Google Calendar publishes events in the following link format:
https://calendar.google.com/calendar/ical/en.usa%23holiday%40group.v.calendar.google.com/public/basic.ics
Alternative Link:
https://raw.githubusercontent.com/xmha97/Calendars/main/Holidays%20in%20United%20States.ics

[mpldr]

If I may add to this feature request the option to add these from the commandline. I am using aerc for emails and it would be great to just do something like piping it into calcure or providing it with a ---import-ics [file] flag to add the event.

[anufrievroman]

I started looking into this issue, it seems like the easiest and most supported way to go is ics.py library. However, in addition to the problems mentioned above, I discovered several more problems:

• this library does not support recurring event (while current format of calcure does)
• this library has bugs importing all-day events (which is the default calcure's type)
• ICS format has many fields, some of which might be valuable for your calendar client. However, that library and calcure cannot possibly support all those fields and the data in them might be lost.
• To preserve as many fields as possible in the files imported from other calendars, it looks like I'll have to rewrite a program quite substantially.

In conclusion, while last two points are doable, the first two points are somewhat critical, because if numerous devs of the ics library couldn't figure it out in many years, I probably won't be able to quickly fix it by myself. So I am still looking for some easy and compatible way to import/export the events to ics.

[gfarrell]

Khal uses the icalendar library with some helper functions which might provide some inspiration.

I do think that ics support is pretty vital given almost all other calendar systems support it (e.g. if I want to sync events to my phone and my nextcloud, at the moment this is all done with ics files and caldav).

I use khal currently, which is great, but I'm always keen to see nice TUI programmes for this sort of thing.

[anufrievroman]

Thank you, I'll look into the khal example!

[tkapias]

Hello,

I am just here to support the request and give a little more perspective.

Currently I use Khard, Khal and Todoman, using files in vdir folders.
Folders synchronized with vdirsyncer to a Caldav Baikal/Sabre server, itself synced on the phone with DAVx5 and used by a fork of Etar-Calendar, and Tasks.org.

Most of the people who want a CLI or TUI stack with open and proper protocols, are on these tools. One tool for each feature, all connected to each other. Perfect.
If you don't care about separating things people use Nextcloud, for more simplicity, but it's the same protocols.

This is one of the few combinations that works at the moment if you want to replace what a closed ecosystem or a GUI like Gnome/Evolution can do.

But we have 3 major problems that got stuck for years:

• Khal and Todoman are hardly TUI, just formatted CLI.
• Khal and Todoman don't work in symbiosis. Same thing for the address book with Khard.
• Todoman does not support basic VTODO features, like categories.

As a result, things that work on my phone or on Evolution are not available in my terminal.
Everyone regularly check if replacements are coming, but either the newcoming apps are incompatible with the existing stacks, they leave out essential features, or they have a closed ecosystem like emacs or todo.txt.

I was disappointed with Calcurse, which is useless, as it can only import one calendar and only supports a third of the basic VTODO fields.

For the moment my hopes are on Calcure, I will try to invest time here.

Honestly, if this project manages to be compatible with caldav or vdir, and propose a dozen VTODO fields. A lot of people will rush here.

[anufrievroman]

Thank you for the details. I agree it would be great but I still don't see any easy way to do it. As far as I can see from calcurse experience, they implemented that several years ago, but it is not finally very usable, like you said some fields are missing, and their issue section is filled with bags related to caldav.
On the surface, it does not look very difficult. For example, to sync tasks you just need to write a new loader and a new saver method in the 'repository.py', instead of load_tasks_from_csv(self) and save_tasks_from_csv(self).
In practice, I can only see it working if there is a good ics exporting/importing library for python, but with each library I tried I have some problems: ics.py mentioned above, icalendar does not even seem to be developed any longer, while formats change...
Besides problems with libraries, I have some other problems like which of calcure features go into which field and is there a field for that at all. But this we can try to resolve with time.

For anyone willing to contribute with caldav support:

It would be very helpful is somebody could write a simple working example of import/export from a folder of real ics files (not just a link to the docs please, I read the docs for all these libs.). Let's start with tasks. Please take several todo files (more than one), each must have a few mandatory fields like DESCRIPTION, STATUS, DUE plus a few fields that remain unused (like they are unsupported). The minimal working example should read these files, and save fields for each file into some variables, change something, and export back into original files. Fields originally present in the files, but not imported, should not be lost.

Thank you for your interest to the project :)

p.s. at the moment, I am not 100% convinced that full support of ics is possible, but I think at least the ability to only display events/tasks from ics must be implemented.

[tkapias]

Thank you, I will contribute.

I already have some informations that I wrote recently. I hope something help.

At the bottom, I added a small analysis of a folder with VEVENT and VTODOs. When I got the time I will make a better one from more sources and with other fields.

ANALYSIS OF SOME TOOLS THAT I USE OR USED

VDirSyncer :

• Linux CLI Python syncing tool.
• Alternatives : manual scripts ?
• Use CALDAV/CARDAV
• Limited parsing of ICALENDAR and VCARD with internal python VOBJECT
• Forked from icalendar
• Respect RFC 5545, but probably not limited by it at as it only parse UID and cut large sets of Sections to individual files.

Result : The easiest way to work with ICALENDAR on Linux without CALDAV implementation.

Baikal

• PHP server.
• Alternatives : Radicale, Nextcloud ?
• Use CALDAV/CARDAV
• Sync with Vdirsyncer
• Parse ICALENDAR and VCARD with VOBJECT
  • Current PHP library sabre/vobject 4.4.3
  • It's a pretty good reference and there is some php tools
  • It can use JCAL/JCARD and XCAL/XCARD too.
• I found 3 references to RFC 5545 from 2009, but none to RFC 7986 from 2016.
• I found no references to "X-" fields for HTML or "VVENUE" sections for location.

Result: Powerful and clean php library respecting standard RFC 5545 for ICALENDAR.

DAVx5

• Android syncing app.
• Alternatives : Using CALDAV directly from the Calendar or Todo apps.
• Use CALDAV/CARDAV/WEBDAV
• It process ICALENDAR with ical4android
  • Using the java library ical4j
• They are referencing a lot of RFC and new standard additions:
  • RFC5545 - Internet Calendaring and Scheduling Core Object Specification (iCalendar)
  • RFC5546 - iCalendar Transport-Independent Interoperability Protocol (iTIP)
  • RFC6047 - iCalendar Message-Based Interoperability Protocol (iMIP)
  • RFC6868 - Parameter Value Encoding in iCalendar and vCard
  • RFC7953 - Calendar Availability
  • RFC7808 - Time Zone Data Distribution Service
  • RFC7986 - New Properties for iCalendar
  • RFC7529 - Non-Gregorian Recurrence Rules in iCalendar
  • RFC9073 - Event Publishing Extensions to iCalendar
  • RFC9074 - "VALARM" Extensions for iCalendar

Result: Powerful, modern and extensive libraries but messy as a reference and in Java.

Tasks

• Android Todo app, fork of Astrid.
• Alternatives : Astrid and some others
• Use CALDAV and ICALENDAR
• Reference a compatibility with a lot of other solutions
• Support limited Markdown in Title and Descriptions
• Support Sub-tasks
• Support recurring tasks, but by changing dates instead of creating new instances as the spec suggests
• I don't see anything in the code than goes beyond the RFC 5545 for ICALENDAR.
• It shares some libraries with Davx5.

Result: Not really a library reference but a good choice of limited features and good ICALENDAR export example.

Nextcloud Calendar/Tasks

• JS Web app
• Use CALDAV
• Sync with Vdirsyncer
• Support Markdown in some fields
• Support Sub-tasks
• No support for recurring tasks
• Parse ICALENDAR with ical.js
• And 2 wrappers : calendar-js and cdav-library
• Respect RFC 5545, features are limited to it.
• I see some more libraries but for reading content from other services.

Result: Largely used today, compatibility need to be tested.

Gnome/Evolution

• Desktop Calendar application
• Sync/Import/Export CALDAV and ICALENDAR/VCARD.
• Calendars and tasks are managed with the C library Libical
• They are referencing some RFCs :
  • RFC5545 - Internet Calendaring and Scheduling Core Object Specification (iCalendar)
  • RFC5546 - iCalendar Transport-Independent Interoperability Protocol (iTIP)
  • RFC6047 - iCalendar Message-Based Interoperability Protocol (iMIP)
  • RFC7529 - Non-Gregorian Recurrence Rules in iCalendar
  • RFC7986 - New Properties for iCalendar
  • RFC9073 - Event Publishing Extensions to iCalendar
  • RFC9074 - "VALARM" Extensions for iCalendar
• As a combined tool it feature a lot of backends and support markdown, sub-tasks and recurring tasks.

Result : A good bench because it work with every ICALENDAR sources and probably most features.

RESULTS:

• For now stick with RFC5545, the most recent ones are not really used yet and provides feature way beyond TUI usages.
• Do not sync with CALDAV, but with a local vdir folder.
• Support Sub-tasks, they are widely used.
• Support Opt-in limited Markdown for at least the descriptions.
• Support recurring tasks, most tools will support it soon. The easiest is to copy/delete task instances following a pattern.
• If you keep implemented Non-Gregorian calendars, it's not an issue as it's still translated in universal time. But you may have to check RFC7529 - Non-Gregorian Recurrence Rules in iCalendar.
• One ICS file per event or task, even subtasks, VDirSyncer or Baikal separate them.
• Use timezone features.
• Use multiple calendars per account. CalDav, Vdir, Baikal and Davx5 support it.
• For features that you can not report in the ICS file, don't do it.
  • Tasks has file attachments and color info for categories that are not synced in the ICS and it's not a big deal.
  • calendars and todos are not meant to be teamwork oriented, but private time management tools with some import/export features.
  • You can have a Calcure local db where there is more features than in ICS files.
  • Or keep them in "X-" fields, but no other tools will probably use them and you can not be sure that the infomation will not be altered by another vendor "X-" fields usage.

---

ANALYSIS OF A CALENDAR EVENT ASSOCIATED WITH A TASK AND 2 SUB-TASKS

The tasks where created on android with Tasks. Then related to an event on the ETAR calendar app.
Then I synced it all by Davx5->Baikal->Vdirsyncer.

I get a folder for my calendar with 6 files:

• 2 metadata files, optional and related to the calendar itself:

  • color -> contain some RGB value "#F24D00"
  • displayname - > contain a name for this Calendar

• 4 .ics ICALENDAR files:
  The filenames are all UIDS but I put descriptive name for here.

  • event-1.ics
  • task-1.ics
  • sub-task-1.ics
  • sub-task-2.ics

• event-1.ics: A calendar event (VEVENT) with a reminder (VALARM)

BEGIN:VCALENDAR
VERSION:2.0
PRODID:DAVx5/4.2.2-ose ical4j/3.1.2 (org.lineageos.etar)
BEGIN:VEVENT
DTSTAMP:20220718T105218Z
UID:78e8251c-9de3-4349-b0e2-f3cadb2560f1
SEQUENCE:1
SUMMARY:Test
LOCATION:Paris\, France
DESCRIPTION:My description\n\n# Title\n\n- list\n- list
DTSTART;VALUE=DATE:20220725
DTEND;VALUE=DATE:20220726
CLASS:PRIVATE
STATUS:TENTATIVE
BEGIN:VALARM
TRIGGER:-PT10M
ACTION:DISPLAY
DESCRIPTION:Test
END:VALARM
END:VEVENT
END:VCALENDAR

DTSTAMP:(TEXT)indicates the date/time that the instance of the iCalendar object was created.
SEQUENCE:(INTEGER)sequence number of the calendar component within a sequence of revisions

SUMMARY:(TEXT)short summary or subject for the calendar component
LOCATION:(TEXT) intended venue for the activity defined by a calendar component
DESCRIPTION:(TEXT) more complete description of the calendar component

DTSTART;VALUE=DATE:(DATE-TIME by default, but can specifie another value type like here DATE) specifies when the calendar component begins
DTEND;VALUE=DATE:(DATE-TIME by default, but can specifie another value type like here DATE) specifies when the calendar component ends

CLASS:(TEXT) the access classification for a calendar component
STATUS:(TEXT) property is used by the "Organizer" to provide a confirmation of the event to the "Attendees".

TRIGGER:(DURATION by default, but can specifie another value type) specifies when an alarm will trigger
ACTION:(TEXT) action to be invoked when an alarm is triggered

• task-1.ics: A calendar task (VTODO) with a reminder (VALARM)

BEGIN:VCALENDAR
VERSION:2.0
PRODID:DAVx5/4.2.2-ose ical4j/3.1.2
BEGIN:VTODO
DTSTAMP:20220718T061550Z
UID:1678082195267907683
CREATED:20220718T061536Z
LAST-MODIFIED:20220718T061536Z
SUMMARY:(Sans titre)
GEO:48.85658;2.35183
DESCRIPTION:My description\n\n# Title\n\n- list\n- list
PRIORITY:1
STATUS:NEEDS-ACTION
CATEGORIES:TestTag1,TestTag2
X-APPLE-SORT-ORDER:678788985
DUE;VALUE=DATE:20220725
DTSTART;VALUE=DATE:20220718
BEGIN:VALARM
TRIGGER;RELATED=END:PT0S
ACTION:DISPLAY
DESCRIPTION:Default Tasks.org description
END:VALARM
END:VTODO
END:VCALENDAR

LAST-MODIFIED:(DATE-TIME) analogous to the modification date and time for a file in the file system

GEO:(two SEMICOLON separated FLOAT values) information related to the global position for the activity

PRIORITY:(INTEGER) relative priority for a calendar component

STATUS:(TEXT) can indicate that an action item needs action, is completed, is in process or being worked on, or has been cancelled

CATEGORIES:(TEXT) categories for a calendar component, more than one category can be specified separated by COMMA
TRIGGER;RELATED=END:(DURATION by default, but can specifie another value type) A trigger set relative to the end of the event or to-do

• sub-task-1.ics: A calendar sub-task (VTODO) related to another

BEGIN:VCALENDAR
VERSION:2.0
PRODID:DAVx5/4.2.2-ose ical4j/3.1.2
BEGIN:VTODO
DTSTAMP:20220718T061550Z
UID:849058032632992142
CREATED:20220718T061536Z
LAST-MODIFIED:20220718T061536Z
SUMMARY:SubTask1
PRIORITY:9
STATUS:NEEDS-ACTION
RELATED-TO;RELTYPE=PARENT:1678082195267907683
END:VTODO
END:VCALENDAR

RELATED-TO;RELTYPE=PARENT:(TEXT) represent a relationship or reference between one calendar component and another

SOME THINGS:

• I did not use recurrency in this example. But it is just a instance replication feature.
• This example use apparently only 20% of the fields available in the RFC. But reading the RFC is hard, I look for a better doc. I found this one for an older RFC : https://www.kanzaki.com/docs/ical/.
• TEXT fields are escaped so it's easy to use markdown without any addition
• I don't see any relation info beetween the VEVENT and VTODO even is Tasks created the VEVENT from it. It's simple like that, but it would be nice if a link existed, or you parse SUMMARYs and DTSTART (but they are optional...).

[tkapias]

And about the libraries, 2 of them were listed above in this thread, there is a third one, vobject.

• ics-py
  • Well documented
  • Maintained
  • But, too much issues like you said above
• icalendar
  • Partial documentation
  • Maintained
  • Use by Khal or Vdirsync projects, but with wrappers or forked because of limitations
  • The lack of some parts in the documentation make it hard to check what is missing actually
• vobject
  • Well documented
  • Not maintained for 4 years but still get PRs and issues. Apparently the maintainer is still in business in the calendars but stopped sharing updates.
  • Support VCARDs too, can be nice to parse birthdays
  • Support most feature lacking in the 2 other ones
  • A lot of ongoing issues and PRs that need to be assessed to see if it's usable.

Something like https://github.com/sabre-io/vobject but not in PHP would have been nice.

[anufrievroman]

Okay, I had some time to work on it. For now, I am trying to figure out limitations and possible issues with different format.
But so far, this script based on ics library works quite well with Nextcloud-generated ics files (everything in one file). I'll leave it here for future reference:

from ics import Calendar

event_file = "myevent.ics"
task_file = "mytasks.ics"

print("====== EVENTS ======")

with open(event_file, 'r') as f:
    ics_text = f.read()

c = Calendar(ics_text)
for event in c.events:
    print(event.name)
    print(event.begin)
    print(event.all_day)
    print(event.description)
    print(event.status)
    print(event.classification)

print("====== TASKS ======")

with open(task_file, 'r') as f:
    ics_text = f.read()

c = Calendar(ics_text)
for task in c.todos:
    print(task.name)

[anufrievroman]

It's been a while, but I started implementing it. So far, n 170671c I implemented read-only tasks and events from .ics files. You need to add path to your ics files (which is synced by vdirsyncer) to your config:

ics_task_files = /home/user/.config/calcure/mytasksfile.ics
ics_event_files = /home/user/.config/calcure/myeventsfile.ics

I tested with nextcloud, it works pretty well. It should be available to everyone in version 2.7. Then, we'll test read-only mode a bit and think how to better implement editing mode and maybe switch completely to .ics file and migrate everybody.

UPDATE:
Also, multiple files can be loaded, comma separated without spaces:

ics_task_files = /home/user/.config/calcure/mytasksfile.ics,/home/user/.config/calcure/anothertasksfile.ics

[anufrievroman]

Okay, I just published version 2.7 where viewing .ics files is working. At least my tests with Nextcloud and vdirsyncer work. Please upgrade pip install --upgrade calcure, try with your cloud, and let me know how it works for you.

For instructions, see this wiki page and vdirsyncer manuals.

Have fun :)

p.s. it turns out that different calendar services implement .ics filse quite differently, so some issues are to be expected. If you experience one, please share.

[thezeroalpha]

Thank you for implementing this! I use vdirsyncer with a Baikal server, events and tasks are stored in the same .ics file. calcure displays both perfectly.

Edit: I noticed the info about reading multiple files isn't in the wiki, perhaps it might be good to also add it there.

[euglevi]

First of all, great app! Would it be possible to add wildcards to the ics_event_files option to identify multiple ics files? This is because I use khal/vdirsyncer and I sync to a folder rather than to a file.

[anufrievroman]

Thank you!
Well, there is this problem with ics that different services either do separate files or all events in one file. I decided to go with the single file format because that's how you can import different calendars (like personal and work etc.), display them with different colors etc (it is coming in the next version). And in general, it seems the single file format is more common. So the option to have multiple files is really meant for different calendars rather than for different individual events.
There are other reasons, but long story short: It's difficult to support both types. Plus, I don't even have a clear idea how I'd implement wildcards from config...

However, I think the solution is simple, just add another record to your vdirsyncer config to also sync it as a single file (here is an example). So you'd have a single file for calcure and a folder for khal (can't khal read single files?). I know it's not optimal, but for now it's like that.

[euglevi]

The solution was indeed pretty simple. It implies duplicates in terms of syncing and ics files, but that is not a big worry.

One question though: is calcure able to read timezones in ics files or did I do something wrong? Because I cannot seem to get the proper timezone for some events. By investigating this a little bit further, it seems that Calcure gets the DTEND and the DTSTART but not the X-WR-TIMEZONE. Am I right?

[nfultz]

One nice-to-have would be specifying the ics location as a url instead of a file path - then fetching it as the program starts.

EDIT:

I've added a minimal working example in PR #42

[anufrievroman]

Thank you for PR, it works great! I'll test a bit more and publish it as a part of 2.8 version.

[nfultz]

Thank you for PR, it works great! I'll test a bit more and publish it as a part of 2.8 version.

You may want to update the docs for using google calendar also, in gcal go to settings | sharing | secret url and plug that in to the config file - note that the percent signs have to be escaped though.

[anufrievroman]

You may want to update the docs for using google calendar also, in gcal go to settings | sharing | secret url and plug that in to the config file - note that the percent signs have to be escaped though.

That's handy! I was actually thinking what's the easiest way to read from gcal, and it seems to be the one. I managed to escape % with raw=True when I import that particular line.

[anufrievroman]

Just pushed a new version to PyPi, in which, thank to @nfultz , displaying from URLs (for example Google calendar) also works. Here is the instruction how to set it up. Please upgrade and test.

In the next version, I'll rewrite it all a bit to get different colors for different calendars #40

[WhyNotHugo]

Hi! I maintain vdirsyncer and todoman. I like this TUI and would love to see some synergy. Mainly, the way information is laid out at-a-glance (as seen in the main screenshot of this repo) is something that I'm really missing.

I read through this long thread, and here are the main thoughts I have in mind:

• vdirsyncer can synchronize remote caldav collections into a local filesystem directory. I think it's a good companion app for this since you can deal with just reading icalendar from a filesystem and just rely on vdirsyncer for caldav/networking/synchronization.
• vdirsyncer uses the vdir convention to store entries locally. The TLDR is: each collection is synchronized to its own directory, and we save one event per file. Having a single event per files make transactional writes easier. However, both todoman and khal keep their own sqlite cache to avoid re-parsing hundreds or thousands of icalendar entries at startup each time.
• vdirsyncer can also synchronize from Google using their caldav implementation.
• vdirsyncer does not use the icalendar library because we don't want t parse icalendar files: if a server has a broken file saves, we want to be able to download this to disk (otherwise, how do you recover/fix it?).

Also, not all entries seem to be so universal. Nextcloud for example writes DTSTART;VALUE=DATE:20220129 while standard specification wants DTSTART:19980114T210000Z, not sure if that might be an issue.

You probably want to use a library which handles this for you. In todoman I use the icalendar library and icalendar.Todo class has a decode() method which returns either of the above as a date or datetime object respectively. Parsing icalendar is a lot of work that's nice to avoid if possible.

If you have specific questions or doubts, I'm happy to help. It's always nice to see more implementations using icalendar/vdir in general.

[anufrievroman]

Hi @WhyNotHugo , thank you for checking out this project!
Yeah, vdirsyncer is basically the way to go as far as syncing is concerned. I definitely don't plan supporting syncing with servers natively, so thank you for the awesome tool. I actually have lots of questions and thoughts, but I'll try to be concise:

1. Since I want to display events from both files and URLs, and they should be provided in the config (and for other reasons), I decided to use singlefile option rather than vdir convention. Here is the example the users are supposed to follow. So, is there anything inherently wrong with this approach and this config (except the necessity to reparse everything at startup)?
2. By the way, there is a problem I experienced that vdirsyncer creates multiple PRODID lines (py-ics library does not like it). People at py-ics assured me that this is against the law. So I am not sure if it is a bug or a feature. Anyway, I made a fix in calcure to skip those extra lines, but still...
3. I guess I mostly figured out parsing and displaying, but I still have doubts if I even should consider editing those files. Correct me if I am wrong, but it looks like different services have different fields in their ics and if we miss some of them (py-ics does not seem to support all fields), rewrite the file, and push it back to the server, the data will be lost, or maybe even the whole file is corrupted. So, potentially it could be a disaster. Am I overcomplicating this, or it's a real concern? I noticed that todoman takes care of this somehow. Is it via icalendar library, or there is something else? Sorry, I didn't fully understand the todoman's code base yet.

[WhyNotHugo]

1. singlefile should work fine. The only downside is that other tools (e.g.: todoman) support only vdir so I wouldn't be able to use the same collection for both. TBH, vdir makes writing easier, but makes reading a bit more work.

2. vdirsyncer doesn't tamper the ics files in any way. IIRC, PRODID is a bit of a special one, but I'm 99% sure we never insert it anywhere. Are you sure that it's not the caldav server that's adding this? Never mind, it looks like this does happens for singlefile due to how we concat multiple entries: vdirsyncer puts two PRODIDs in a singlefile and then pushes them back to server pimutils/vdirsyncer#763 I think this shouldn't be too hard to fix.

3. Yeah, I take care never to drop any unknown fields. Basically there's two types: icalendar.Todo which contains the data read from the file is semi-parsed form, but contains ALL data. OTOH, there'todoman.Todo, which is a wrapper for the above, which convenience methods like start() and is_complete(). However, I don't serialize todoman.Todo itself; I load the icalendar.Todo from disk, and update it with all the data from the todoman.Todo. All other fields are untouched. So basically, if you ics file had X-MAGIC: blue, that never makes it into the todoman.Todo instance.

This magic happens in VTodoWriter.serialize.

Sorry, I didn't fully understand the todoman's code base yet.

Yeah, it's become pretty complicated over the years, mostly to optimize parsing. Basically we have an sqlite database (the "cache"). It keeps the mtime (modification time) of each file and a bunch of fields like start time, due date, etc.

At startup, todoman reads the sqlite database, and traverses directories. If any new files are found, they're read and added to the database. If a file is an event (e.g.: don't care about it for todoman), it goes into another table in the DB, so we know not to bother reading it next time. If any files' last modified date has change, we re-read it and fill the sqlite db again. At this point, the cache is fully up to date (and this is a lot cheaper than parsing all files). todoman list and friends only read this sqlite database (which is updated at startup anyway).

Ironically, modifying an entry does re-read the original ics file as mentioned above: to keep all unknown fields intact.

The sqlite database is safe to delete at any time: it really is just a cache. The next run is just slower because it has to regenerate it.

[anufrievroman]

After all the discussions, I implemented bd8e9ae reading from ics folders as well. Now, you can use single ics file, a url, or a folder or url files, the program should understand what you mean and parse accordingly. Please pip install --upgrade calcure to version 2.8 and test.

[mathstuf]

Things look good overall except that Fastmail's editor creates TZUNTIL lines which dateutil doesn't support yet.

[anufrievroman]

@mathstuf does it crash the program? I think I can try to skip lines like that, as we already do with PRODID.

EDIT: I implemented a quick fix for that, in version 2.8.2 it should at least not crash on this line and also parse other files even if one file has something wrong in it.

[WhyNotHugo]

I've pointed the config to an existing collection:

 calcurse_events_file = /home/hugo/.local/share/calendars/502708f4-4b94-4963-b042-f7340c829bdf/

But calcure just renders an empty calendar. Any ideas? (I'm using main).

[anufrievroman]

@WhyNotHugo could you please check info.log file in your config folder? Does it mention anything relevant?

[WhyNotHugo]

info.log is empty.

[anufrievroman]

info.log is empty

Ah, wait, the calcurse_events_file is the path to the events from calcurse (another program).
The parameter to provide path to .ics files is ics_event_files =

[WhyNotHugo]

Oh, I didn't notice it said calcurse; I thought it said calcure.

The auto-generated config file has this entry, but not ics_event_files, so I completely missed that.

Works okay now.

[eguiraud]

All files beside problematic ones should be loading in version 2.8.2

Hello, I guess I have a problematic one :D calcure quickly flashes a [ERROR] Failed to parse /path/to/calendar.ics before displaying an empty calendar at the terminal. Unfortunately I can't share the ICS file as it is my personal calendar of the last 5 years or so.

Any way we can figure out a workaround?

EDIT:
info.log just repeats that [ERROR] message, no further details

[mathstuf]

Any way we can figure out a workaround?

I edited the venv to log the exception and then searched where that came from for my issue.

[eguiraud]

Ok, I just removed the except branch that swallowed the exception and logged an error so I could get a better look at the exception itself:

Traceback (most recent call last):
  File "/home/blue/calcure_dbg_env/bin/calcure", line 8, in <module>
    sys.exit(cli())
  File "/home/blue/calcure_dbg_env/lib/python3.10/site-packages/calcure/__main__.py", line 1040, in cli
    curses.wrapper(main)
  File "/usr/lib/python3.10/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/home/blue/calcure_dbg_env/lib/python3.10/site-packages/calcure/__main__.py", line 928, in main
    user_ics_events = event_loader_ics.load()
  File "/home/blue/calcure_dbg_env/lib/python3.10/site-packages/calcure/loaders.py", line 367, in load
    cal = ics.Calendar(ics_file)
  File "/home/blue/calcure_dbg_env/lib/python3.10/site-packages/ics/icalendar.py", line 69, in __init__
    self._populate(containers[0])  # Use first calendar
  File "/home/blue/calcure_dbg_env/lib/python3.10/site-packages/ics/component.py", line 54, in _populate
    raise ValueError(
ValueError: A VCALENDAR must have at most one PRODID

So it's still the problem with multiple PRODIDs. I'm on 2.8.2, I guess the fix you mention above is not working for this file @anufrievroman ?

[anufrievroman]

Hi @eguiraud , thank you for reporting and investigating!
Could you please at least share the header part of your .ics and maybe also search your file for any PRODID lines? The thing is, I assumed that these lines go one after another in the header only (in which case additional lines are ignored), but maybe in your file it is not the case. If so, it's easy to fix, I'll try.

NOTE TO MYSELF: I need to make errors more explicit, at least showing the error type...

[eguiraud]

This is the beginning of events_personal.ics for my personal Nextcloud calendar as produced by vdirsyncer sync:

BEGIN:VCALENDAR
PRODID:-//IDN nextcloud.com//Calendar app 2.0.3//EN
CALSCALE:GREGORIAN
VERSION:2.0
PRODID:-//IDN nextcloud.com//Calendar app 2.0.2//EN
PRODID:DAVx5/4.1.1-ose ical4j/3.1.2 (ws.xsoh.etar)
PRODID:-//IDN nextcloud.com//Calendar app 2.3.1//EN
PRODID:DAVx5/4.0-ose ical4j/3.1.0 (ws.xsoh.etar)
PRODID:DAVx5/4.2.0.3-ose ical4j/3.1.2 (ws.xsoh.etar)
PRODID:-//IDN nextcloud.com//Calendar app 2.1.3//EN
PRODID:-//IDN nextcloud.com//Calendar app 3.4.2//EN
PRODID:DAVx5/3.3.10-ose ical4j/3.0.24 (ws.xsoh.etar)
PRODID:-//IDN nextcloud.com//Calendar app 3.2.2//EN
PRODID:-//Microsoft Corporation//Outlook 16.0 MIMEDIR//EN
PRODID:-//Kopano//8.7.9//EN
PRODID:DAVx5/4.1-ose ical4j/3.1.1 (ws.xsoh.etar)
PRODID:+//IDN bitfire.at//DAVx5/2.6.4-ose ical4j/2.2.6
PRODID:DAVx5/4.2.0.3-ose ical4j/3.1.2
PRODID:+//IDN bitfire.at//DAVx5/2.6.5-ose ical4j/2.2.6
PRODID:-//github.com/rianjs/ical.net//NONSGML ical.net 4.0//EN
PRODID:DAVx5/3.3.9-ose ical4j/3.0.21 (ws.xsoh.etar)
PRODID:-//Acme Widgets\, Inc.//NONSGML ExportToCalendar//EN
PRODID:DAVx5/4.2.3.4-ose ical4j/3.2.5 (ws.xsoh.etar)
PRODID:DAVx5/3.3.12-ose ical4j/3.0.28 (ws.xsoh.etar)
PRODID:DAVx5/3.4-ose ical4j/3.1.0 (ws.xsoh.etar)
PRODID:DAVx5/4.2.6-ose ical4j/3.2.5 (ws.xsoh.etar)
PRODID:DAVx5/4.3-ose ical4j/3.2.7 (ws.xsoh.etar)
PRODID:DAVx5/4.2.3.2-ose ical4j/3.2.5 (ws.xsoh.etar)
PRODID:DAVx5/3.3-ose ical4j/3.0.19 (ws.xsoh.etar)
PRODID:DAVx5/3.3.5-ose ical4j/3.0.20 (ws.xsoh.etar)
PRODID:-//swiss.com//Your Flight//EN
PRODID:icalendar-ruby
PRODID:DAVx5/3.3.8-ose ical4j/3.0.21 (ws.xsoh.etar)
PRODID:DAVx5/4.1.1-ose ical4j/3.1.2
PRODID:DAVx5/3.2.1.3-ose ical4j/3.0.19 (ws.xsoh.etar)
PRODID:DAVx5/4.2.2-ose ical4j/3.1.2 (ws.xsoh.etar)
PRODID:+//IDN bitfire.at//DAVx5/3.0-ose2 ical4j/2.2.6
PRODID:DAVx5/3.3.4-ose ical4j/3.0.20 (ws.xsoh.etar)
PRODID:Data::ICal 0.23
PRODID:-//Nextcloud calendar v1.7.1
PRODID:-//zoom.us//iCalendar Event//EN
PRODID:+//IDN bitfire.at//DAVx5/3.1-ose ical4j/3.0.18
PRODID:-//CERN//INDICO//EN
BEGIN:VEVENT

Interestingly, if I export the calendar to a .ics directly from Nextcloud it looks very different, with only 2 PRODID lines (but anyway again more than 1 and non-consecutive).

[anufrievroman]

I see, I actually had exactly the same problem. This is something @WhyNotHugo might be interested as well.
Anyway, I made a fix for this issue. Please update to 2.8.3 pip install --upgrade calcure it should work now.

[eguiraud]

Thank you for the quick reaction! Unfortunately I can't use vdirsyncer+calcure because of pimutils/vdirsyncer#818 at the moment -- I can give this a go if/when that gets fixed :)

[thezeroalpha]

Thanks for your work on this. Most of it works very well, but I ran into an issue with an ICS folder synced via vdirsyncer containing metadata:

• vdirsyncer has the command metasync, which can sync calendar metadata, such as the color or display name. If you use the filesystem storage (i.e., a directory containing ICS files, "Vdir"), this metadata is stored in files such as color or displayname in that directory (see vdirsyncer docs here).
• When calcure is configured to read ICS files from a directory, it also reads these metadata files, which results in an error: "[ERROR] Failed to parse ... (1:9) Expecting <ALPHADIGIT_MINUS_PLUS> : #0E61B9FF" (with #0E61B9FF the color of the calendar stored in the color file).
• That seems to come from this code, which loads all files without any filtering. So either we could filter files there and (a) only add ICS files (which we determine either by extension or by file contents), or (b) exclude those specific known metadata files. Or we could use the color file to override color_ics_calendars, and perhaps similarly for other metadata files.

[anufrievroman]

Hi @thezeroalpha , thank you for reporting!
I think we definitely should check (in the code you highlighted) that the file has .ics extension. That would be most straightforward fix.

At the moment, I am on vacation, so if someone could add a line, something like:

If filename.endswith(".ics"):

That would be very helpful!

[anufrievroman]

Thank you again for the PR! The change was pushed into the repo, it's in the version 2.9.

[anufrievroman]

Hi guys, we are testing improved support for ics files with icalendar library, thanks to contribution by @jose1711
Could you please install the test branch and confirm that it works for you?

pipx install git+https://github.com/anufrievroman/calcure@icalendar

This version is supposed to parse ics files much faster than before.

[euglevi]

By a first look, it seems to work well and it is way much faster than before.

[euglevi]

Thanks again for the app! I really like it.

I want to point out an issue I am having with one of my calendars. This calendar originates in Outlook, then I import it in Google. As with my other calendars, I sync it with vdirsyncer and its calendar events appear regularly on khal. When I try to import it in calcure, the calendar does not show up. The same if I just use the link provided by Outlook for ics files. Is there some kind of basic incompatibility between the ics libraries/calcure and Outlook ics files?

[anufrievroman]

@euglevi is there anything mentioned in the info.log file? It's in the config folder. There's no inherent incompatibility, but sometimes there are lines in ics files that break things. Which version you are using?

[euglevi]

I am using the git version with the new ics library, but this problem was already there with the previous library. The info.log file is empty. Just to give you an example of the ics file calcure did not read which was related to a today's event:

BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
X-WR-CALNAME:Calendario
X-WR-TIMEZONE:UTC
BEGIN:VTIMEZONE
TZID:Europe/Amsterdam
X-LIC-LOCATION:Europe/Amsterdam
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Europe/Amsterdam:20231123T123000
DTEND;TZID=Europe/Amsterdam:20231123T130000
DTSTAMP:20231111T090419Z
UID:040000008200E00074C5B7101A82E00800000000F0C6281F7406DA01000000000000000
 010000000CEE3ED476AC3E441B654F3E3E70F4A2E
CLASS:PUBLIC
CREATED:20231111T090418Z
DESCRIPTION:Dear All\,\n\nwe are happy to invite you to the Research Semina
 r “Learning it the hard way: Conflicts\, economic sanctions and military ai
 d” by Prof. Edoardo Grillo on date 23/11/2023 from 12:30 to 13:30 in room B
 Z E3.12.\n\nFor online participation\, please register in advance at the fo
 llowing link: https://events.teams.microsoft.com/event/639dbdbd-cbfc-4cf7-9
 413-19f54ce87d83@92513267-03e3-401a-80d4-c58ed6674e3b\n\nBest Regards\n\n\n
LAST-MODIFIED:20231111T090419Z
LOCATION:
SEQUENCE:2
STATUS:CONFIRMED
SUMMARY:Research Seminar “Learning it the hard way: Conflicts\, economic sa
 nctions and military aid” by Prof. Edoardo Grillo 
TRANSP:OPAQUE
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-APPT-SEQUENCE:2
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-DISALLOW-COUNTER:FALSE
X-MICROSOFT-DONOTFORWARDMEETING:FALSE
X-MICROSOFT-REQUESTEDATTENDANCEMODE:DEFAULT
END:VEVENT
END:VCALENDAR

[jose1711]

See this line:

calcure/calcure/data.py

Line 149 in 87ca8c5

if 100 > len(item.name) > 0 and item.name != "\[":

It is a limit for a number of characters of event name.

len('Research Seminar “Learning it the hard way: Conflicts, economic sanctions and military aid” by Prof. Edoardo Grillo ') = 116

If you lift that limit you should be able to see it in calcure. Don't know why the limitation is there though.

[anufrievroman]

That's probably right! No idea why and when I put it there :D
It might be to prevent occasional paste of a buffer (with a huge number of characters including enter) that I experienced. Anyways, 100 is too small indeed, don't see a reason not to make it 1000. Thank you for investigating!

[anufrievroman]

@euglevi I fixed it in the main branch, could you try this version on your files?

pipx install git+https://github.com/anufrievroman/calcure

[euglevi]

I confirm that now it works perfectly!

[anufrievroman]

There was still a little issue apparently, but now it seems to be fixed. I think there still might be a problem that event with time but without timezone might have issues, although I was not able to produce such event.

[anufrievroman]

Okay, in the new version 3.0 the support of ics files is much improved, added time support, multiday support, and improved the speed, thanks to contributions from @jose1711

Please upgrade and test:

pipx upgrade calcure