Lost calendar, address, tasks after server crash

[charleslcso]

So happen that our Proxmox LXC containers all shutdown unexpectedly. Bad stuff.

Then realise all users' Radicale data is gone: addresses, calendar events, tasks/reminders.

I'll need to backup Radicale regularly. Also, how should this kind of event be handled?

[pbiering]

non-transient data should never stored inside containers but outside on stable and backup-ed file system...

usually, all files in the collections-root main folder should be backed-up, simple full backup e.g.

tar czf /path/to/radicale-collection-root-backup-$(date +'%Y%m%d-%H%M%S').tar.gz /var/lib/radicale/collections/collection-root/

one can think about excluding the cache directories,

frequency strongly depending on the change rate

also one can use a more sophisticated tools like amanda or duplicity to run incremental backups and able to have a history in case a particular file only need to be restored

[pbiering]

this strongly depends how far the (hopefully journaling) file system was abstracted.

per se a container has no persistent file system at all internally, only in-memory

if a container requires persistent file system, this has to be mounted from outside into the container

if the "outside" crashes itself it's strongly depending of which kind it was...

• a virtual machine based on a disk image
  or
• a physical system having disks (RAID)

-> as long as the (virtual) disk is not damaged, the file system should be able to be recover

And radicale is not vanishing files on stop, can it be that you run radicale in a container without persistent storage, e.g. using only tempfs in memory instead of a persistent file system mounted into the container (from the host or via NFS, iSCSI or something else)?

[charleslcso]

Hi @pbiering

The container running radicale is using ext4 fs, from host. Yes, there are tmpfs in the container, but they are not used by radicale.

Using the default radicale parameters, collections are stored in /root/.var/lib/radicale/collections/collection-root//, which means /root is not in tmpfs.

[pbiering]

radicale is not deleting files on startup and ext4 filesystem usually has a journal -> the files should be even after a container or host crash visible on the host file system, except some cleanup magic by used infrastructure either on crash/shutdown or (re)start wipes such mapped directory. Backup should be also done on host level and pushed to or pulled from another host.

[charleslcso]

non-transient data should never stored inside containers but outside on stable and backup-ed file system...

usually, all files in the collections-root main folder should be backed-up, simple full backup e.g.

tar czf /path/to/radicale-collection-root-backup-$(date +'%Y%m%d-%H%M%S').tar.gz /var/lib/radicale/collections/collection-root/

one can think about excluding the cache directories,

frequency strongly depending on the change rate

also one can use a more sophisticated tools like amanda or duplicity to run incremental backups and able to have a history in case a particular file only need to be restored

Peter,

Shit, it happens again. I must restart the server, and now all my Reminders and Calendars are mostly gone! Except there are 1 calendar and 1 addressbook in Web UI.

All other calendars have vanished.

I made frequent backups on another server, and I restored it in /.var/lib/radicale/collections/collection-root/ (this is where radicale stores files in Debian), but still, in Web UI, nothing seems to have restored.

By checking the directory listing of the backup, content size wise it should be good.

What to do what to do?

[pbiering]

Have you verified permission of files after restore?

Check also whether the structure was proper restored
https://github.com/Kozea/Radicale/wiki/Collection-Storage

Also run storage verification (best in debug mode)
https://github.com/Kozea/Radicale/wiki/Server-Diagnostics---Troubleshooting

In addition I strongly recommend mounting a persistent storage from host or network into the container or install software itself on a host or inside a VM (with persistent disk image...) to avoid such issue again.

[charleslcso]

Have you verified permission of files after restore?

Check also whether the structure was proper restored https://github.com/Kozea/Radicale/wiki/Collection-Storage

Also run storage verification (best in debug mode) https://github.com/Kozea/Radicale/wiki/Server-Diagnostics---Troubleshooting

In addition I strongly recommend mounting a persistent storage from host or network into the container or install software itself on a host or inside a VM (with persistent disk image...) to avoid such issue again.

Radicale is installed in a VM with persistent disk image. I have no idea why it happens. It was a proper shutdown.

What's in the /root/.var/..../collection, with the dot in front of var?

[pbiering]

the . in front of a directory name is turning the directory into "hidden" - used in Unix since many many years. There are also such other directories in parallel in an user's home directory.

Running radicale interactive and especially as root user is not recommended for any productive usage.

For your skill set I strongly recommend using packaged (DEB/RPM) radicale installation for the operating system you are using.
They are also containing scripts for running as service and usually store data under /var/lib/.

If your current OS unfortunately not up-to-date versions either trigger maintainer to update or switch the OS e.g. to Enterprise Linux (e.g. Alma - usually 5 year lifecycle) or Fedora Linux (if 1,5 yearlifecycle is acceptable) as they have up-to-date support.