cron.php OCA\DAV\CardDAV\SyncJob causes a high load daily

[CamZie]

Steps to reproduce

1. Have a lot of users. E.g. 5000
2. Wait for the OCA\DAV\CardDAV\SyncJob cron job to run (happens once in 24h)

Expected behaviour

Job should run quickly and the load on the server should not go higher.

Actual behaviour

It runs for hours depending on the amount of users. For approximately 5000 users it needs 3 hours. It tries to parse all users and their addressbook cards in one pass, which causes a high load on the database and takes up more server resource than it should.

Server configuration

Operating system: Debian 9.1 (stretch)

Web server: Nginx 1.10.3

Database: MySQL 5.8

PHP version: 7

Nextcloud version: 12.0.2

Contacts version: 2.0.1

Updated from an older Nextcloud or fresh install: Updated from Nextcloud version 12

Signing status:

No errors have been found.

List of activated apps:

Enabled:
  - activity: 2.5.2
  - admin_audit: 1.2.0
  - calendar: 1.5.6
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - encryption: 1.6.0
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_videoplayer: 1.1.0
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - nextcloud_announcements: 1.1
  - notes: 2.3.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - tasks: 0.9.5
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "12.0.2.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "theme": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": true,
        "enable_avatars": false,
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "log_authfailip": true,
        "loglevel": 2,
        "maintenance": false,
        "customclient_desktop": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "share_folder": "\/Shared",
        "overwritewebroot": "\/",
        "overwriteprotocol": "https",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "singleuser": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "trashbin_retention_obligation": "auto,30",
        "activity_expire_days": 14,
        "logtimezone": "Europe\/Zurich",
        "log_type": "errorlog",
        "skeletondirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***"
    }
}

Are you using encryption: yes

Client configuration

Browser:
not relevant because it is a CLI/server side issue
Operating system:
not relevant because it is a CLI/server side issue

Logs

Web server error log

Web server error log

no errors

Nextcloud log (data/nextcloud.log)

Nextcloud log

no errors

Browser log

Browser log

no errors

[MorrisJobke]

cc @georgehrke

[georgehrke]

@CamZie Do you have a lot of federated shares?

[CamZie]

@georgehrke we do not have any federated shares.

[CamZie]

I would like to ask if there are any news for this problem?

[georgehrke]

I would like to ask if there are any news for this problem?

Not yet. I forgot to ask: Do you use any user-backend? If yes, how fast is the connection to the user-backend?

I'm asking because OCA\DAV\CardDAV\SyncJob does not touch any user address book. All it does is creating/updating VCards for each user in the system address book and if you use federated shares sync the system address book with other instances.

[CamZie]

Thank you for the information. We are not using any user-backend and simply have all the users in the MySQL database.

Is it possible that the cause of this problem is the OCA\DAV\CardDAV\SyncJob, which creates/updates the vCards for each users in one try?

[georgehrke]

Is it possible that the cause of this problem is the OCA\DAV\CardDAV\SyncJob, which creates/updates the vCards for each users in one try?

So you are not sure that the high load was caused by OCA\DAV\CardDAV\SyncJob, but it was just a wild guess?

Can you check in the oc_jobs table what other jobs are scheduled for the instance? Maybe the last_run column will give a hint what job took so much time.

[CamZie]

We were sure that this was caused by the OCA\DAV\CardDAV\SyncJob, because we also saw the following entry on the database:

id: 565626
class: OCA\DAV\CardDAV\SyncJob
argument: null
last_run: 1512512102
last_checked: 1512549901
reserved_at: 0
execution_duration: 9260

[MorrisJobke]

Looks like we need to optimise that job then. @rullzer @schiessle @blizzz Any idea for this?

[rullzer]

I'd have to run test. But I see why it fails.

The first part of the job is actually playing nice I'd say: https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L320-L325

Just for each user update that user which seems fine (and makes sense).

However the second part: https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L328

Creates a huge structure in RAM. We should probably try to add a generator or something here... I'll see if I can add 10k users easily to my dev setup and see it perform

[rullzer]

Mmmm so thinking more about this... why don't we just listen to the postDelete hook properly... and then drop the user from the sytemaddressbook...

[rullzer]

Aaaah because the mapping is reversed... (sorry for the live debug log :P)