Catalog DVD taking very long time?

[xtiansimon]

I'm using gocatcli to catalog DVD backups of files. These are the full ~4g capacity.
My backups of Windows are not tar'd or zip'd.

Right now gocat is at 10hours and counting (after two cancelled archive attempts after 5 hours each).

> # gocatcli index /mnt/dvd 003-Backup-2024-12
 ⠇  indexing /mnt/dvd (10h53m54s)   

Do you know why this might be the case?
Are there settings in gocatcli which need to be configured to speed up DVD archive?
Or is this expected behavior, in which case any suggestions or tips how best to run the command?

[deadc0de6]

This all depends on the content of those 4g. At the two extremes you have a single file of 4g and about 4^9 files of 1 byte. It's hard for me to say without more info. Per default most of the costly indexing operations are disabled (archive, checksum). What you can still try to disable is mime type detection with -M --nomime.

Did the indexing finished at some point?

You could also add more granularity and try indexing different top folders independently to try to identify which part is the costly one.

You're mentioning My backups of Windows, are you actually indexing Windows backups? Because if you do, most of what you index is useless, you should maybe focus on user directories (documents, desktop, etc).

[EvgeniyBlinov]

My command just stuck too with no activity on disks

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,81    0,00    0,61    0,00    0,00   98,59

Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
dm-0              0,00         0,00         0,00         0,00          0          0          0
dm-1              0,00         0,00         0,00         0,00          0          0          0
dm-2              0,00         0,00         0,00         0,00          0          0          0
dm-3              0,00         0,00         0,00         0,00          0          0          0
dm-4              0,00         0,00         0,00         0,00          0          0          0
loop0             0,00         0,00         0,00         0,00          0          0          0
loop1             0,00         0,00         0,00         0,00          0          0          0
loop2             0,00         0,00         0,00         0,00          0          0          0
sda               0,00         0,00         0,00         0,00          0          0          0
sdb               0,00         0,00         0,00         0,00          0          0          0
sdc               0,00         0,00         0,00         0,00          0          0          0

gocatcli -c ./disks/Apacer_AS350_120GB/p3_LINUX.catalog index /mnt/sdb/sdb2
5h10m43s

[deadc0de6]

Indexing is done in two steps. The first step will create the tree by walking the filesystem and the second step will calculate the size of each nodes of the tree recursively. Since you're seeing no activity, you could well be in the second stage.

However after testing gocatcli on very large filesystems I noticed several elements I could improve:

• the tree is entirely constructed in memory (memory exhaustion, bugs crashing an already done indexing, etc)
• nodes are highly dependent (because it's a tree)
• the data model could be improved to speed up and allow intermediate saves
• using json is not optimal, an alternative would be toml or sqlite for example
• some improvements could be done in the data structure
• add more logs for better user feedbacks

I've started to work on some improvements in this PR and will keep you posted on the progress.