Conversation
There was a problem hiding this comment.
Code Review
This pull request implements CPU affinity support for taosd, allowing threads to be pinned to specific CPU cores categorized by management, write, and read tasks. It introduces new configuration parameters (enableCpuAffinity, managementCpuCores, readCpuRatio), a system table (ins_cpu_allocation), and the SHOW CPU_ALLOCATION command. The review identified critical issues regarding potential buffer overflows and array out-of-bounds access in the core allocation logic and string formatting, specifically when handling systems with high core counts or truncated strings.
There was a problem hiding this comment.
Pull request overview
This PR introduces a CPU-affinity feature for taosd threads (management/write/read), exposes the allocation via SHOW CPU_ALLOCATION / information_schema.ins_cpu_allocation, wires the new sys table through parser/executor/catalog paths, and adds CI/system tests plus documentation updates.
Changes:
- Add CPU-affinity allocation/init logic and apply affinity to multiple thread entrypoints.
- Add
SHOW CPU_ALLOCATIONstatement +ins_cpu_allocationsystem table retrieval path. - Add a new CPU-affinity test suite and register it in CI, plus update EN/ZH docs for SQL + config.
Reviewed changes
Copilot reviewed 49 out of 50 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Adds CPU-affinity test cases to CI task list. |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Tests SHOW CPU_ALLOCATION and ins_cpu_allocation output and disabled/enabled behavior. |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Tests readCpuRatio allocation math and core-id partitioning. |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Tests managementCpuCores behavior and visibility via SHOW DNODE VARIABLES. |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Tests master switch behavior and /proc thread affinity observation. |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Adds sequential assignment, restart, and basic ops smoke coverage. |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Utilities for CPU count, parsing core IDs, and reading /proc affinity. |
| test/cases/34-CpuAffinity/conftest.py | Attempts to mitigate persisted config affecting multi-class config permutations. |
| test/cases/34-CpuAffinity/init.py | Marks the directory as a package for local imports. |
| source/util/src/tworker.c | Applies affinity in worker thread functions; propagates threadCategory from worker configs. |
| source/util/src/tsched.c | Pins scheduler queue thread to management category. |
| source/util/src/tlog.c | Pins log threads to management category. |
| source/util/src/tcache.c | Pins cache refresh thread to management category. |
| source/os/src/osTimer.c | Pins timer/alarm threads to management category. |
| source/os/src/osSysinfo.c | Implements CPU allocation init + taosSetCpuAffinity() and status accessors. |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management category. |
| source/libs/transport/src/transSvr.c | Pins transport accept/worker threads to management category. |
| source/libs/transport/src/transCli.c | Pins client worker thread to management category. |
| source/libs/transport/src/thttp.c | Pins HTTP client send thread to management category. |
| source/libs/parser/src/parTranslater.c | Maps SHOW CPU_ALLOCATION to information_schema.ins_cpu_allocation. |
| source/libs/parser/src/parTokenizer.c | Adds CPU_ALLOCATION keyword token. |
| source/libs/parser/src/parAuthenticator.c | Gates SHOW CPU_ALLOCATION on sysinfo permission (like other sysinfo shows). |
| source/libs/parser/src/parAstParser.c | Adds meta-cache reservation for the new sys table on SHOW. |
| source/libs/parser/inc/sql.y | Adds grammar rule for SHOW CPU_ALLOCATION. |
| source/libs/nodes/src/nodesUtilFuncs.c | Adds node creation/destruction handling for new SHOW node type. |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieval to DND systable retrieve path. |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management category. |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write category. |
| source/dnode/vnode/src/sma/smaEnv.c | Pins RSMA executor threads to write category. |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management category. |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Calls taosInitCpuAllocation() during dnode env init. |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Categorizes vnode worker pools as read/write for affinity. |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins multiple dnode mgmt threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements ins_cpu_allocation systable block build/fill in DND retrieve handler. |
| source/common/src/tglobal.c | Adds server config options: enableCpuAffinity, managementCpuCores, readCpuRatio. |
| source/common/src/systable.c | Defines schema/meta entry for ins_cpu_allocation. |
| include/util/tworker.h | Adds threadCategory fields to worker pools/config structs. |
| include/os/osSysinfo.h | Adds CPU affinity enums/status structs and APIs. |
| include/common/tmsg.h | Adds new query node type for SHOW CPU_ALLOCATION. |
| include/common/tglobal.h | Exposes new CPU-affinity global config variables. |
| include/common/systable.h | Adds TSDB_INS_TABLE_CPU_ALLOCATION name constant. |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/01-components/08-config-scope.md | Lists new CPU-affinity configs (ZH). |
| docs/zh/14-reference/01-components/01-taosd.md | Documents new CPU-affinity configs and behavior (ZH). |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (EN). |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (EN). |
| docs/en/14-reference/01-components/08-config-scope.md | Lists new CPU-affinity configs (EN). |
| docs/en/14-reference/01-components/01-taosd.md | Documents new CPU-affinity configs and behavior (EN). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Adds a CPU-affinity feature to TDengine server: new config knobs compute a 3-way CPU core allocation (management/write/read), threads are pinned accordingly, and the allocation is exposed via SHOW CPU_ALLOCATION and information_schema.ins_cpu_allocation, with CI/system tests and docs updates.
Changes:
- Introduce CPU affinity allocation/init + per-thread category pinning across multiple subsystems.
- Add
SHOW CPU_ALLOCATIONandins_cpu_allocationsystem table wiring end-to-end (parser → executor → dnode retrieve → systable meta). - Add a new CI test suite (
34-CpuAffinity) and documentation for new SQL/config surface.
Reviewed changes
Copilot reviewed 49 out of 50 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Adds CPU-affinity tests to CI task list |
| test/cases/34-CpuAffinity/init.py | Initializes new test package |
| test/cases/34-CpuAffinity/conftest.py | Clears persisted local config between test classes |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Helper utilities for CPU/thread affinity inspection |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Tests SHOW CPU_ALLOCATION and ins_cpu_allocation correctness |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Tests readCpuRatio behavior and ID allocation |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Tests managementCpuCores behavior and visibility |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Tests enable/disable switch behavior and /proc validation |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Tests edge cases and basic ops/restart scenarios |
| source/util/src/tworker.c | Applies affinity when worker threads start; propagates category from configs |
| source/util/src/tsched.c | Pins scheduler queue thread to management cores |
| source/util/src/tlog.c | Pins log threads to management cores |
| source/util/src/tcache.c | Pins cache refresh thread to management cores |
| source/os/src/osTimer.c | Pins timer/alarm threads to management cores |
| source/os/src/osSysinfo.c | Implements CPU allocation computation + taosSetCpuAffinity() |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management cores |
| source/libs/transport/src/transSvr.c | Pins transport accept/worker threads to management cores |
| source/libs/transport/src/transCli.c | Pins client worker thread to management cores |
| source/libs/transport/src/thttp.c | Pins HTTP client send thread to management cores |
| source/libs/parser/src/parTranslater.c | Adds sys-table show adapter + rewrite handling for SHOW CPU_ALLOCATION |
| source/libs/parser/src/parTokenizer.c | Adds CPU_ALLOCATION keyword token |
| source/libs/parser/src/parAuthenticator.c | Allows SHOW CPU_ALLOCATION gated by SYSINFO |
| source/libs/parser/src/parAstParser.c | Collects sysmeta key for SHOW CPU_ALLOCATION |
| source/libs/parser/inc/sql.y | Adds grammar rule for SHOW CPU_ALLOCATION |
| source/libs/nodes/src/nodesUtilFuncs.c | Registers/destroys new SHOW node type |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieval to dnode |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management cores |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write cores |
| source/dnode/vnode/src/sma/smaEnv.c | Pins RSMA executor to write cores |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management cores |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Calls taosInitCpuAllocation() during dnode init |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Assigns worker pools to read/write categories |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management cores |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins various dnode management threads to management cores |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements ins_cpu_allocation data block construction/fill |
| source/common/src/tglobal.c | Adds config parameters: enableCpuAffinity/managementCpuCores/readCpuRatio |
| source/common/src/systable.c | Adds ins_cpu_allocation sys table schema/metadata |
| include/util/tworker.h | Adds threadCategory plumbing to worker pool/config structs |
| include/os/osSysinfo.h | Declares CPU affinity types/APIs (EThreadCategory, status structs) |
| include/common/tmsg.h | Adds new AST node type enum for SHOW CPU_ALLOCATION |
| include/common/tglobal.h | Declares new global config variables |
| include/common/systable.h | Adds TSDB_INS_TABLE_CPU_ALLOCATION constant |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (ZH) |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (ZH) |
| docs/zh/14-reference/01-components/08-config-scope.md | Adds config scope entries (ZH) |
| docs/zh/14-reference/01-components/01-taosd.md | Documents new taosd config knobs (ZH) |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (EN) |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (EN) |
| docs/en/14-reference/01-components/08-config-scope.md | Adds config scope entries (EN) |
| docs/en/14-reference/01-components/01-taosd.md | Documents new taosd config knobs (EN) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
This PR introduces a CPU affinity feature controlled by new server config parameters, exposes the allocation via a new SHOW CPU_ALLOCATION command / information_schema.ins_cpu_allocation table, and adds CI/system tests plus documentation updates to validate and describe the behavior.
Changes:
- Add CPU affinity configuration (
enableCpuAffinity,managementCpuCores,readCpuRatio) and initialize CPU-core allocation at dnode startup. - Add
SHOW CPU_ALLOCATIONandinformation_schema.ins_cpu_allocationplumbing (parser/auth/catalog/executor/dnode systable retrieval). - Add a new CI test suite (
34-CpuAffinity) and update docs for the new SQL and configuration options.
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
test/ci/cases.task |
Adds the new CPU affinity test suite to CI execution. |
test/cases/34-CpuAffinity/test_cpu_show_allocation.py |
New tests for SHOW CPU_ALLOCATION and information_schema.ins_cpu_allocation. |
test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py |
New tests validating readCpuRatio behavior and core splits. |
test/cases/34-CpuAffinity/test_cpu_management_cores.py |
New tests validating managementCpuCores behavior. |
test/cases/34-CpuAffinity/test_cpu_affinity_switch.py |
New tests validating enable/disable switch and (Linux) /proc affinity behavior. |
test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py |
New edge-case tests, including restart and basic ops with affinity enabled. |
test/cases/34-CpuAffinity/cpu_affinity_utils.py |
Adds Linux helpers for affinity inspection and parsing. |
test/cases/34-CpuAffinity/conftest.py |
Clears persisted config between test classes to avoid cross-class contamination. |
test/cases/34-CpuAffinity/__init__.py |
Marks the directory as a Python package. |
test/cases/21-MetaData/test_meta_ins_tables.py |
Updates expected system table counts due to the new info schema table. |
test/cases/21-MetaData/test_meta_information_schema.py |
Updates expected table counts due to the new info schema table. |
test/cases/10-Operators/04-Set/test_union_bugs.py |
Updates expected row counts impacted by the new system table. |
source/util/src/tworker.c |
Applies CPU affinity in worker thread entrypoints and propagates thread category in worker pool init. |
source/util/src/tsched.c |
Pins scheduler task queue thread to management category. |
source/util/src/tlog.c |
Pins log threads to management category. |
source/util/src/tcache.c |
Pins cache refresh thread to management category. |
source/os/src/osTimer.c |
Pins timer threads to management category. |
source/os/src/osSysinfo.c |
Implements CPU allocation computation, affinity application, and status export. |
source/libs/wal/src/walMgmt.c |
Pins WAL thread to management category. |
source/libs/transport/src/transSvr.c |
Pins transport accept/worker threads to management category. |
source/libs/transport/src/transCli.c |
Pins client worker thread to management category. |
source/libs/transport/src/thttp.c |
Pins HTTP client send thread to management category. |
source/libs/parser/src/parTranslater.c |
Maps SHOW CPU_ALLOCATION to the new info schema table and includes it in rewrite handling. |
source/libs/parser/src/parTokenizer.c |
Adds CPU_ALLOCATION keyword. |
source/libs/parser/src/parAuthenticator.c |
Enables auth handling for SHOW CPU_ALLOCATION under sysinfo gating. |
source/libs/parser/src/parAstParser.c |
Adds metadata-collection support for SHOW CPU_ALLOCATION. |
source/libs/parser/inc/sql.y |
Adds grammar rule for SHOW CPU_ALLOCATION. |
source/libs/nodes/src/nodesUtilFuncs.c |
Adds node creation/destruction cases for the new SHOW statement type. |
source/libs/executor/src/sysscanoperator.c |
Routes ins_cpu_allocation retrieval through the dnode systable path. |
source/libs/catalog/src/ctgCache.c |
Pins catalog update thread to management category. |
source/dnode/vnode/src/vnd/vnodeAsync.c |
Pins vnode async loop to write category. |
source/dnode/vnode/src/sma/smaEnv.c |
Pins rsma executor to write category. |
source/dnode/mnode/impl/src/mndMain.c |
Pins mnode timer threads to management category. |
source/dnode/mgmt/node_mgmt/src/dmEnv.c |
Initializes CPU allocation during dnode init. |
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c |
Propagates thread categories into vnode worker pool configurations. |
source/dnode/mgmt/mgmt_vnode/src/vmInt.c |
Pins vnode open/close/restore/timer threads to management category. |
source/dnode/mgmt/mgmt_dnode/src/dmWorker.c |
Pins various dnode mgmt threads to management category. |
source/dnode/mgmt/mgmt_dnode/src/dmHandle.c |
Implements systable block build/fill for ins_cpu_allocation retrieval. |
source/common/src/tglobal.c |
Adds new global config variables and wires them into config registration/load. |
source/common/src/systable.c |
Adds ins_cpu_allocation schema to information_schema. |
include/util/tworker.h |
Extends worker pool/cfg structs with threadCategory. |
include/os/osSysinfo.h |
Declares CPU affinity APIs/types and allocation status structures. |
include/common/tmsg.h |
Adds AST node type for SHOW CPU_ALLOCATION. |
include/common/tglobal.h |
Declares new global config variables. |
include/common/systable.h |
Adds TSDB_INS_TABLE_CPU_ALLOCATION systable name constant. |
docs/zh/14-reference/03-taos-sql/52-show.md |
Documents SHOW CPU_ALLOCATION. |
docs/zh/14-reference/03-taos-sql/50-meta.md |
Documents INS_CPU_ALLOCATION in information_schema. |
docs/zh/14-reference/01-components/08-config-scope.md |
Documents new config parameters in the config scope list. |
docs/zh/14-reference/01-components/01-taosd.md |
Documents CPU affinity configuration options for taosd. |
docs/en/14-reference/03-taos-sql/52-show.md |
Documents SHOW CPU_ALLOCATION. |
docs/en/14-reference/03-taos-sql/50-meta.md |
Documents INS_CPU_ALLOCATION in information_schema. |
docs/en/14-reference/01-components/08-config-scope.md |
Documents new config parameters in the config scope list. |
docs/en/14-reference/01-components/01-taosd.md |
Documents CPU affinity configuration options for taosd. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
This PR introduces a CPU-affinity feature controlled by new server configs, adds observability via SHOW CPU_ALLOCATION / information_schema.ins_cpu_allocation, pins various internal threads to the computed CPU sets, and adds CI/system tests plus documentation updates.
Changes:
- Add CPU affinity allocation initialization +
taosSetCpuAffinity()and apply it across multiple background/worker threads. - Add
SHOW CPU_ALLOCATIONandinformation_schema.ins_cpu_allocationsystem table plumbing (parser/auth/meta/sys-scan/dnode retrieval). - Add a new CPU-affinity test suite and update existing information_schema table-count expectations and docs (EN/ZH).
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Registers new CPU-affinity pytest cases in CI task list. |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Tests SHOW CPU_ALLOCATION and information_schema table behavior. |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Tests read/write split logic for multiple readCpuRatio values. |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Tests managementCpuCores allocation behavior and visibility. |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Tests master switch behavior and /proc-based affinity validation. |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Tests sequential assignment, sums, restart persistence, and basic ops. |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Utilities for CPU counting and /proc affinity inspection. |
| test/cases/34-CpuAffinity/conftest.py | Clears persisted config between classes to avoid cross-test pollution. |
| test/cases/34-CpuAffinity/init.py | Marks the CPU-affinity test package. |
| test/cases/21-MetaData/test_meta_ins_tables.py | Updates expected system-table counts. |
| test/cases/21-MetaData/test_meta_information_schema.py | Updates expected information_schema counts. |
| test/cases/10-Operators/04-Set/test_union_bugs.py | Updates expected union row-count due to new system table. |
| source/util/src/tworker.c | Applies affinity in worker thread entrypoints; propagates threadCategory. |
| source/util/src/tsched.c | Pins scheduler queue thread to management CPU set. |
| source/util/src/tlog.c | Pins log/rotate threads to management CPU set. |
| source/util/src/tcache.c | Pins cache refresh thread to management CPU set. |
| source/os/src/osTimer.c | Pins timer threads to management CPU set. |
| source/os/src/osSysinfo.c | Implements CPU allocation computation + taosSetCpuAffinity(); exposes status. |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management CPU set. |
| source/libs/transport/src/transSvr.c | Pins transport accept/worker threads to management CPU set. |
| source/libs/transport/src/transCli.c | Pins transport client worker thread to management CPU set. |
| source/libs/transport/src/thttp.c | Pins HTTP client send thread to management CPU set. |
| source/libs/parser/src/parTranslater.c | Maps SHOW CPU_ALLOCATION to information_schema system table. |
| source/libs/parser/src/parTokenizer.c | Adds CPU_ALLOCATION keyword token. |
| source/libs/parser/src/parAuthenticator.c | Enforces SYSINFO permission for SHOW CPU_ALLOCATION. |
| source/libs/parser/src/parAstParser.c | Collects meta for SHOW CPU_ALLOCATION. |
| source/libs/parser/inc/sql.y | Adds grammar for SHOW CPU_ALLOCATION. |
| source/libs/nodes/src/nodesUtilFuncs.c | Adds node creation/destruction handling for the new SHOW node type. |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieval to dnode sys-table retrieval. |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management CPU set. |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write CPU set. |
| source/dnode/vnode/src/sma/smaEnv.c | Pins RSMA executor to write CPU set. |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management CPU set. |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Initializes CPU allocation during dnode init. |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Assigns worker pools to read/write categories. |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management CPU set. |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins dnode management threads to management CPU set. |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements dnode-side retrieval for ins_cpu_allocation. |
| source/common/src/tglobal.c | Adds new server config items: enableCpuAffinity, managementCpuCores, readCpuRatio. |
| source/common/src/systable.c | Adds system table schema/meta for ins_cpu_allocation. |
| include/util/tworker.h | Adds threadCategory to worker pool/config structs. |
| include/os/osSysinfo.h | Adds CPU affinity API, enums, and allocation status types. |
| include/common/tmsg.h | Adds AST node type for SHOW CPU_ALLOCATION. |
| include/common/tglobal.h | Declares new global config variables. |
| include/common/systable.h | Adds TSDB_INS_TABLE_CPU_ALLOCATION name constant. |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/01-components/08-config-scope.md | Documents new CPU-affinity configs (ZH). |
| docs/zh/14-reference/01-components/01-taosd.md | Documents CPU-affinity configs (ZH). |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (EN). |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (EN). |
| docs/en/14-reference/01-components/08-config-scope.md | Documents new CPU-affinity configs (EN). |
| docs/en/14-reference/01-components/01-taosd.md | Documents CPU-affinity configs (EN). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
This PR introduces a CPU affinity feature for taosd, including configuration knobs, runtime CPU allocation visibility via SHOW CPU_ALLOCATION / information_schema.ins_cpu_allocation, and a dedicated CI test suite to validate allocation behavior.
Changes:
- Add CPU affinity configuration (
enableCpuAffinity,managementCpuCores,readCpuRatio) and initialize CPU allocation at dnode startup. - Pin key thread categories (management/read/write) by calling
taosSetCpuAffinity()from multiple thread entry points and worker pools. - Add
SHOW CPU_ALLOCATIONsupport plus a newinformation_schemasystem table (ins_cpu_allocation), and add/adjust tests & documentation accordingly.
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Adds CPU affinity test cases to CI task list. |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Adds tests for SHOW CPU_ALLOCATION and ins_cpu_allocation. |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Adds tests for read/write core split behavior across ratios. |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Adds tests for managementCpuCores behavior and visibility. |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Adds tests for enable/disable behavior and /proc affinity checks. |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Adds edge-case and restart/ops coverage tests for affinity. |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Adds helper utilities for CPU set parsing and /proc inspection. |
| test/cases/34-CpuAffinity/conftest.py | Cleans persisted config between classes to avoid cross-test contamination. |
| test/cases/34-CpuAffinity/init.py | Marks the CPU affinity test package. |
| test/cases/21-MetaData/test_meta_ins_tables.py | Updates expected system table counts. |
| test/cases/21-MetaData/test_meta_information_schema.py | Updates expected information_schema table list/counts. |
| test/cases/10-Operators/04-Set/test_union_bugs.py | Updates expected row count due to the new system table. |
| source/util/src/tworker.c | Adds worker-thread affinity setting + plumbs threadCategory through worker configs. |
| source/util/src/tsched.c | Pins scheduler task queue thread to management category. |
| source/util/src/tlog.c | Pins log threads to management category. |
| source/util/src/tcache.c | Pins cache refresh thread to management category. |
| source/os/src/osTimer.c | Pins timer/alarm threads to management category. |
| source/os/src/osSysinfo.c | Implements CPU allocation computation and taosSetCpuAffinity(). |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management category. |
| source/libs/transport/src/transSvr.c | Pins transport server threads to management category. |
| source/libs/transport/src/transCli.c | Pins transport client worker thread to management category. |
| source/libs/transport/src/thttp.c | Pins HTTP client thread to management category. |
| source/libs/parser/src/parTranslater.c | Maps SHOW CPU_ALLOCATION to ins_cpu_allocation and enables rewrite handling. |
| source/libs/parser/src/parTokenizer.c | Registers CPU_ALLOCATION as a keyword token. |
| source/libs/parser/src/parAuthenticator.c | Adds permission gating for SHOW CPU_ALLOCATION. |
| source/libs/parser/src/parAstParser.c | Ensures meta cache reservation for SHOW CPU_ALLOCATION. |
| source/libs/parser/inc/sql.y | Adds grammar for SHOW CPU_ALLOCATION. |
| source/libs/nodes/src/nodesUtilFuncs.c | Adds node create/destroy support for the new SHOW stmt type. |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieval to dnode sys-table path. |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management category. |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write category. |
| source/dnode/vnode/src/sma/smaEnv.c | Pins RSMA executor to write category. |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management category. |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Calls taosInitCpuAllocation() during dnode env init. |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Sets thread categories for vnode worker pools. |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins multiple dnode management threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements dnode-side ins_cpu_allocation block building/filling and retrieval dispatch. |
| source/common/src/tglobal.c | Adds global config vars + registers/loads CPU affinity config items. |
| source/common/src/systable.c | Adds ins_cpu_allocation schema and registers it in infosMeta. |
| include/util/tworker.h | Adds threadCategory fields to worker pool/config structs. |
| include/os/osSysinfo.h | Adds affinity-related enums/structs and new exported APIs. |
| include/common/tmsg.h | Adds AST node type for SHOW CPU_ALLOCATION. |
| include/common/tglobal.h | Exposes CPU affinity config globals. |
| include/common/systable.h | Adds system table name constant for ins_cpu_allocation. |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (zh). |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (zh). |
| docs/zh/14-reference/01-components/08-config-scope.md | Adds CPU affinity configs to scope list (zh). |
| docs/zh/14-reference/01-components/01-taosd.md | Documents CPU affinity configuration parameters (zh). |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (en). |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (en). |
| docs/en/14-reference/01-components/08-config-scope.md | Adds CPU affinity configs to scope list (en). |
| docs/en/14-reference/01-components/01-taosd.md | Documents CPU affinity configuration parameters (en). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces a CPU affinity feature for taosd: configurable core allocation across management/write/read thread categories, with visibility via a new SHOW CPU_ALLOCATION command and information_schema.ins_cpu_allocation system table, plus documentation and CI/system tests.
Changes:
- Add CPU affinity configuration (
enableCpuAffinity,managementCpuCores,readCpuRatio) and CPU-core allocation logic with per-category pinning. - Expose allocation via
SHOW CPU_ALLOCATIONandinformation_schema.ins_cpu_allocation(parser + systable + dnode retrieve + executor routing). - Add a new CPU affinity test suite and update existing information_schema table-count tests and docs (EN/ZH).
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Adds CPU affinity test cases into CI task list. |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Tests SHOW/INS table schema + enable/disable behavior. |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Tests read/write split logic across ratios. |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Tests management core reservation behavior. |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Tests master switch behavior and thread masks via /proc. |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Edge cases: sequential IDs, sum checks, restart, basic ops. |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Adds helpers for CPU counts and /proc affinity parsing. |
| test/cases/34-CpuAffinity/conftest.py | Clears persisted config between classes to avoid cross-test leakage. |
| test/cases/34-CpuAffinity/init.py | Marks CPU affinity test package. |
| test/cases/21-MetaData/test_meta_ins_tables.py | Updates expected information_schema table counts. |
| test/cases/21-MetaData/test_meta_information_schema.py | Updates expected table counts and sys table name list. |
| test/cases/10-Operators/04-Set/test_union_bugs.py | Updates expected union result size due to new sys table. |
| source/util/src/tworker.c | Applies affinity for worker threads and propagates threadCategory. |
| source/util/src/tsched.c | Pins scheduler thread to management category. |
| source/util/src/tlog.c | Pins log/rotate threads to management category. |
| source/util/src/tcache.c | Pins cache refresh thread to management category. |
| source/os/src/osTimer.c | Pins timer threads to management category. |
| source/os/src/osSysinfo.c | Implements CPU allocation + taosSetCpuAffinity() on Linux. |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management category. |
| source/libs/transport/src/transSvr.c | Pins transport accept/worker threads to management category. |
| source/libs/transport/src/transCli.c | Pins transport client worker thread to management category. |
| source/libs/transport/src/thttp.c | Pins HTTP thread to management category. |
| source/libs/parser/src/parTranslater.c | Maps SHOW CPU_ALLOCATION to information_schema table. |
| source/libs/parser/src/parTokenizer.c | Adds CPU_ALLOCATION keyword tokenization. |
| source/libs/parser/src/parAuthenticator.c | Adds permission gating for SHOW CPU_ALLOCATION. |
| source/libs/parser/src/parAstParser.c | Adds meta-cache reservation for ins_cpu_allocation. |
| source/libs/parser/inc/sql.y | Adds SHOW CPU_ALLOCATION grammar rule. |
| source/libs/nodes/src/nodesUtilFuncs.c | Adds node create/destroy handling for new SHOW stmt. |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieves via dnode sys-table path. |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management category. |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write category. |
| source/dnode/vnode/src/sma/smaEnv.c | Pins rsma executor to write category. |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management category. |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Initializes CPU allocation during dnode env init. |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Assigns thread categories for vnode worker pools. |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins various dnode mgmt threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements ins_cpu_allocation block build/fill on dnode. |
| source/common/src/tglobal.c | Adds CPU affinity config variables and config registration. |
| source/common/src/systable.c | Adds schema + metadata for ins_cpu_allocation. |
| include/util/tworker.h | Adds threadCategory fields to worker pool configs/structs. |
| include/os/osSysinfo.h | Adds CPU affinity APIs and data structures. |
| include/common/tmsg.h | Adds AST node type for SHOW CPU_ALLOCATION. |
| include/common/tglobal.h | Declares new global config variables. |
| include/common/systable.h | Adds ins_cpu_allocation table name constant. |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (ZH). |
| docs/zh/14-reference/01-components/08-config-scope.md | Lists CPU affinity config params (ZH). |
| docs/zh/14-reference/01-components/01-taosd.md | Documents CPU affinity config params (ZH). |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION (EN). |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION (EN). |
| docs/en/14-reference/01-components/08-config-scope.md | Lists CPU affinity config params (EN). |
| docs/en/14-reference/01-components/01-taosd.md | Documents CPU affinity config params (EN). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
This PR introduces CPU affinity support in taosd, including configuration knobs, thread pinning by category (management/write/read), and a new system table + SHOW CPU_ALLOCATION for introspection, along with CI tests and documentation updates.
Changes:
- Add CPU affinity allocation/init logic and apply affinity to multiple server threads and worker pools.
- Add
information_schema.ins_cpu_allocationandSHOW CPU_ALLOCATIONSQL support end-to-end (parser/planner/executor/dnode retrieval). - Add a new CI test suite (
34-CpuAffinity) and update existing information_schema table-count assertions + docs.
Reviewed changes
Copilot reviewed 53 out of 54 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ci/cases.task | Adds CPU affinity test suite execution to CI task list. |
| test/cases/34-CpuAffinity/test_cpu_show_allocation.py | Adds tests for SHOW CPU_ALLOCATION and ins_cpu_allocation consistency/validation. |
| test/cases/34-CpuAffinity/test_cpu_read_write_ratio.py | Adds tests for read/write core split behavior and custom allocations. |
| test/cases/34-CpuAffinity/test_cpu_management_cores.py | Adds tests for management core allocation and DNODE VARIABLES visibility. |
| test/cases/34-CpuAffinity/test_cpu_affinity_switch.py | Adds tests for enabling/disabling affinity and checking thread masks via /proc. |
| test/cases/34-CpuAffinity/test_cpu_affinity_edge_cases.py | Adds edge case tests (sequential assignment, restart persistence, basic ops). |
| test/cases/34-CpuAffinity/cpu_affinity_utils.py | Adds Linux /proc helpers for affinity inspection and core-id parsing. |
| test/cases/34-CpuAffinity/conftest.py | Adds per-test-class cleanup of persisted config to avoid cross-class leakage. |
| test/cases/34-CpuAffinity/init.py | Initializes the new test package. |
| test/cases/21-MetaData/test_meta_ins_tables.py | Updates expected information_schema system table counts. |
| test/cases/21-MetaData/test_meta_information_schema.py | Updates expected table counts/list to include ins_cpu_allocation. |
| test/cases/10-Operators/04-Set/test_union_bugs.py | Updates expected row count impacted by the new system table. |
| source/util/src/tworker.c | Applies CPU affinity in worker thread entrypoints and propagates threadCategory from cfg. |
| source/util/src/tsched.c | Pins scheduler task queue thread to management category. |
| source/util/src/tlog.c | Pins log threads to management category. |
| source/util/src/tcache.c | Pins cache refresh thread to management category. |
| source/os/src/osTimer.c | Pins timer/alarm threads to management category. |
| source/os/src/osSysinfo.c | Implements CPU allocation init and taosSetCpuAffinity() logic. |
| source/libs/wal/src/walMgmt.c | Pins WAL thread to management category. |
| source/libs/transport/src/transSvr.c | Pins transport accept/worker threads to management category. |
| source/libs/transport/src/transCli.c | Pins client transport worker thread to management category. |
| source/libs/transport/src/thttp.c | Pins HTTP client send thread to management category. |
| source/libs/planner/src/planPhysiCreater.c | Routes ins_cpu_allocation scans similarly to dnode variables. |
| source/libs/parser/src/parTranslater.c | Adds SHOW adapter + dnode sys-table routing and rewrite support for CPU allocation. |
| source/libs/parser/src/parTokenizer.c | Adds CPU_ALLOCATION keyword tokenization. |
| source/libs/parser/src/parAuthenticator.c | Gates SHOW CPU_ALLOCATION behind sysinfo permission like other sys tables. |
| source/libs/parser/src/parAstParser.c | Adds meta-key collection for SHOW CPU_ALLOCATION / ins_cpu_allocation. |
| source/libs/parser/inc/sql.y | Adds SHOW CPU_ALLOCATION grammar production. |
| source/libs/nodes/src/nodesUtilFuncs.c | Enables node creation/destruction for the new SHOW statement type. |
| source/libs/executor/src/sysscanoperator.c | Routes ins_cpu_allocation retrieval to DNODE systable path. |
| source/libs/catalog/src/ctgCache.c | Pins catalog update thread to management category. |
| source/dnode/vnode/src/vnd/vnodeAsync.c | Pins vnode async loop to write category. |
| source/dnode/vnode/src/sma/smaEnv.c | Pins rsma executor to write category. |
| source/dnode/mnode/impl/src/mndMain.c | Pins mnode timer threads to management category. |
| source/dnode/mgmt/node_mgmt/src/dmEnv.c | Initializes CPU allocation during dnode environment init. |
| source/dnode/mgmt/mgmt_vnode/src/vmWorker.c | Assigns worker pools/cfgs to read/write categories via threadCategory. |
| source/dnode/mgmt/mgmt_vnode/src/vmInt.c | Pins vnode open/close/restore/timer threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmWorker.c | Pins various dnode mgmt threads to management category. |
| source/dnode/mgmt/mgmt_dnode/src/dmHandle.c | Implements ins_cpu_allocation block build/fill and retrieve handling. |
| source/common/src/tglobal.c | Adds affinity-related config vars and registers them in server config. |
| source/common/src/systable.c | Adds ins_cpu_allocation system table schema/metadata. |
| include/util/tworker.h | Adds threadCategory fields to worker pool/cfg structs. |
| include/os/osSysinfo.h | Adds CPU affinity API/types (EThreadCategory, SCpuAllocStatus, etc.). |
| include/common/tmsg.h | Adds new query node type for SHOW CPU_ALLOCATION. |
| include/common/tglobal.h | Exposes new affinity-related global config variables. |
| include/common/systable.h | Adds TSDB_INS_TABLE_CPU_ALLOCATION name constant. |
| docs/zh/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION. |
| docs/zh/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION. |
| docs/zh/14-reference/01-components/08-config-scope.md | Adds CPU affinity config parameters to config-scope table. |
| docs/zh/14-reference/01-components/01-taosd.md | Documents CPU affinity configuration parameters in taosd reference. |
| docs/en/14-reference/03-taos-sql/52-show.md | Documents SHOW CPU_ALLOCATION. |
| docs/en/14-reference/03-taos-sql/50-meta.md | Documents INS_CPU_ALLOCATION. |
| docs/en/14-reference/01-components/08-config-scope.md | Adds CPU affinity config parameters to config-scope table. |
| docs/en/14-reference/01-components/01-taosd.md | Documents CPU affinity configuration parameters in taosd reference. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Verify sequential assignment | ||
| expected_mgmt = set(range(0, m)) | ||
| expected_write = set(range(m, m + w)) | ||
| expected_read = set(range(m + w, m + w + r)) | ||
|
|
| TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "managementCpuCores", tsManagementCpuCores, 1, 256, CFG_SCOPE_SERVER, CFG_DYN_SERVER_LAZY, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM)); | ||
| // Compute dynamic defaults for readCpuCores and otherCpuCores | ||
| { | ||
| int32_t total = (int32_t)tsNumOfCores; | ||
| int32_t remaining = total - tsManagementCpuCores; | ||
| if (remaining < 2) remaining = 2; | ||
| tsReadCpuCores = remaining / 2; | ||
| tsOtherCpuCores = remaining - tsReadCpuCores; | ||
| } | ||
| TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "readCpuCores", tsReadCpuCores, 1, 256, CFG_SCOPE_SERVER, CFG_DYN_SERVER_LAZY, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM)); | ||
| TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "otherCpuCores", tsOtherCpuCores, 1, 256, CFG_SCOPE_SERVER, CFG_DYN_SERVER_LAZY, CFG_CATEGORY_LOCAL, CFG_PRIV_SYSTEM)); |
| uError("failed to CPU affinity for category, categroy:%d", category); | ||
| return; |
| for row_idx in range(3): | ||
| category = tdSql.queryResult[row_idx][0] | ||
| cores = tdSql.queryResult[row_idx][1] | ||
| core_ids = tdSql.queryResult[row_idx][2] | ||
| enabled = tdSql.queryResult[row_idx][3] | ||
|
|
| # Management should have 1 core, core_ids="0" | ||
| assert alloc["management"]["cores"] == 1, \ | ||
| f"Expected management cores=1, got {alloc['management']['cores']}" | ||
| assert alloc["management"]["core_ids"] == "0", \ | ||
| f"Expected management core_ids='0', got {alloc['management']['core_ids']}" |
| for row_idx in range(3): | ||
| category = tdSql.queryResult[row_idx][0] | ||
| cores = tdSql.queryResult[row_idx][1] | ||
| core_ids = tdSql.queryResult[row_idx][2] | ||
| enabled = tdSql.queryResult[row_idx][3] | ||
|
|
| alloc = {} | ||
| for row_idx in range(3): | ||
| category = tdSql.queryResult[row_idx][0] | ||
| cores = tdSql.queryResult[row_idx][1] | ||
| core_ids = tdSql.queryResult[row_idx][2] | ||
| alloc[category] = {"cores": cores, "core_ids": core_ids} |
| if (sched_getaffinity(0, sizeof(cpu_set_t), cpuset) != 0) { | ||
| // Fallback: set all cores based on tsNumOfCores | ||
| for (int i = 0; i < (int)tsNumOfCores; i++) { | ||
| CPU_SET(i, cpuset); | ||
| } | ||
| return (int32_t)tsNumOfCores; |
| total_cores = 0 | ||
| for row_idx in range(3): | ||
| total_cores += tdSql.queryResult[row_idx][1] | ||
|
|
| # Verify sequential assignment: mgmt=[0], write=[1,2], read=[3] | ||
| assert all_core_ids["management"] == {0}, \ | ||
| f"management: expected {{0}}, got {all_core_ids['management']}" | ||
| assert all_core_ids["write"] == {1, 2}, \ | ||
| f"write: expected {{1, 2}}, got {all_core_ids['write']}" | ||
| assert all_core_ids["read"] == {3}, \ | ||
| f"read: expected {{3}}, got {all_core_ids['read']}" |
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.