Conversation
|
考虑到长耗时任务,失败任务id不是绝对递增的,不可以简单通过最小id过滤。 可以通过时间过滤,一会扫描近期增量数据,比如3天内。 |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces cursor-based pagination to improve the scanning efficiency of failed task logs by adding a lastJobLogId parameter to avoid repeatedly scanning the entire log table (which can contain millions of records with 30-day retention).
Key Changes:
- Added
lastJobLogIdparameter to filter queries, scanning only logs withid > lastJobLogId - Modified the monitoring thread to track the last processed log ID across iterations
- Whitespace cleanup in XML mapper file
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| xxl-job-admin/src/main/resources/mybatis-mapper/XxlJobLogMapper.xml | Added conditional id > lastJobLogId filter to findFailJobLogIds query for cursor-based pagination; minor whitespace cleanup |
| xxl-job-admin/src/main/java/com/xxl/job/admin/dao/XxlJobLogDao.java | Updated findFailJobLogIds method signature to accept lastJobLogId parameter |
| xxl-job-admin/src/main/java/com/xxl/job/admin/core/thread/JobFailMonitorHelper.java | Introduced lastFailLogId tracking variable and passed it to the DAO query for incremental scanning |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| try { | ||
|
|
||
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(1000); | ||
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000); |
There was a problem hiding this comment.
[nitpick] Missing space after comma in the method call. Should be findFailJobLogIds(lastFailLogId, 1000) instead of findFailJobLogIds(lastFailLogId,1000) for consistency with Java code style conventions.
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId,1000); | |
| List<Long> failLogIds = XxlJobAdminConfig.getAdminConfig().getXxlJobLogDao().findFailJobLogIds(lastFailLogId, 1000); |
What kind of change does this PR introduce? (check at least one)
The description of the PR:
提升失败任务日志的扫描效率。
Other information:
在实际业务重任务量很大,log 表只存 30 天也有几百万条记录,定时扫描导致 mysql cpu 一直占满