import should automatically update existing events based on UID/SEQUENCE values

[dbarnett]

Right now if you import an ics file with an updated version of an event already on your target calendar, it will fail to import saying the event is a duplicate. Instead it should update the existing event (since UID indicates it's the same logical event and SEQUENCE indicates the version being imported is newer). See #730 for additional context.

There's already a workaround to do this manually but the process is clunky (by feeding the dumped rej.ics file back through import --use-legacy-import).

Note versions prior to #712 / 4.4.0 didn't have this issue, but had the inverse issue of updates being too aggressive and tending to generate duplicate events and notification spam.

[dbarnett]

On a technical level, this is happening because for some bizarre reason Google's import method that we switched to accepts a sequence number but doesn't do anything with it and instead just blows up saying events with the same UID are "duplicates".

It seems like we need more logic to sometimes use import and sometimes insert and I don't see any obvious way to detect which one without calling multiple methods on the same event.

[michaelmhoffman]

I believe reimporting ICS events using the Google Calendar web interface changes nothing if the UID is the same and the event still exists.

I have some code that generates ICS files and I set the UID to sha256("\0".join([dtstart, dtend, location, summary, description]).encode("utf-8")).hexdigest() to always have unique UIDs for any changes to the major fields, which avoids this issue.

[dbarnett]

Hmm, then it might be good to stick close to their existing behavior, but in the context of gcalcli the "duplicate" behavior surprised me at first. I was testing gcalcli import by trying to reimport an event I'd previously imported and deleted, and didn't understand at first why the import had no effect (because the event still "existed" but only as a deleted event).

Either way we could try to make the behavior pretty self-explanatory in the command output, and consider supporting a way to force it when needed.

[ntzb]

hi @dbarnett
I don't really understand the current status of "update" in relation to this issue.
before 4.5, when I tried to import events with updated date/time (after first importing an event for the first time), it created duplicate events.
with 4.5, as you mention, I see in the logs

Skipped duplicate event "<SUMMARY{}test>".
Added 0 events to calendar my@gmail.com
Dumped 1 failed events to /tmp/gcalcli.fyifm7ji/rej.ics.
Note: you can try reprocessing this file with `gcalcli import --use-legacy-import` to work around some failure causes (but with potentially noisier update notifications).

so the event is not being imported again, and suggests using --use-legacy-import.

you mention:

There's already a workaround to do this manually but the process is clunky (by feeding the dumped rej.ics file back through import --use-legacy-import).

so I assume --use-legacy-import uses pre 4.5 functionally? e.g. will it create duplicate events again?
so what can the solution be?
I'm running this automatically in a script, so can a workaround be:

1. detect the error
2. delete the original event
3. add the failed .ics file

@michaelmhoffman if the UID is not the same, how does it know which event to update?

[ntzb]

I just did a test of a workaround I'm trying to implement

1. identify Skipped duplicate event
2. delete the old event (if can be reasonably found)
3. re-insert the event.

after deleting the old event using gcalcli --calendar=my@gmail.com delete test --iamaexpert, I no longer see it in my calendar, but still trying to import the updated event I'm getting Skipped duplicate event even after using --nocache.

any ideas?

[dbarnett]

@ntzb yeah those are the clunky parts. From what I remember here the API has two options for IMPORT vs UPDATE and neither one works quite how you'd expect. It's probably possible (but not easy) to wrap the implementation it in some custom duplicate handling to figure out which events are logical duplicates of which other events and give an option to try to update them in-place or something.

I believe you can always get the same effect by reprocessing just the duplicates with --use-legacy-import and then deleting the originals, except that recreating and deleting events will spam any attendees. I personally would avoid trying to automate it with a script though for the same reasons it got tricky to implement this to seamlessly do the right thing. About deleting events, I'm not sure but I suspect what's happening for you is that a deleted event that's still in Trash still exists as a hidden event in the API, so you'll still see it as an ignored duplicate if you try to reprocess a deleted event without --use-legacy-import.

I'm not claiming any of that behavior is user friendly but that's as far as I got in implementing it nicely before running out of brain cells for it, and the awkwardness comes from how their API works, not just how gcalcli happens to be implemented.

[aurelf]

I was hit by this problem, and it seems that using --use-legacy-import has no effect.
My understanding is that I hit this error here:

gcalcli/gcalcli/gcal.py

Line 1682 in 2d25ec0

# TODO: #492 - Offer to force import dupe anyway?

The ics file contains a single event with UID and LAST-MODIFIED. I imagine it could be unambiguous to update it?

And there seem to be no way to :

1. get import to update the event
2. force import
   Only solution seem to be to remove the UID line in the ics file.

Temporary solutions:

• Return an error code on the command line, so I could know the meeting was not added (scripted, I may not see the message).
• Have an option to ask the user to force add the event (remove UID and try importing again).

Unrelated to the original issue, but related to the last comment (let me know if I should fill a new issue for that?): I have a script to comment out the participants to a meeting, so that they don't get a flood of emails when I delete an event in google calendar. It would be nice to have an option to selectively import participants in an event (so I wouldn't need an ugly script for that) ?

[dbarnett]

Check #836 about the invite emails. I actually have a PR up that I believe fixes the --use-legacy-imports flag being backwards (see latest comments there) but I'd really like if someone could look it over before I merge it and be available to test if it actually improves their failure case.