Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Commit 47bd959

Browse files
gavofyorkggwpezeskimorKiChjang
authored
General Message Queue Pallet (#12485)
* The message queue * Make fully generic * Refactor * Docs * Refactor * Use iter not slice * Per-origin queues * Multi-queue processing * Introduce MaxReady * Remove MaxReady in favour of ready ring * Cleanups * ReadyRing and tests * Stale page reaping * from_components -> from_parts Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move WeightCounter to sp_weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add MockedWeightInfo Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Deploy to kitchensink Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use WeightCounter Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Small fixes and logging Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add service_page Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Typo Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move service_page below service_queue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add service_message Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use correct weight function Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Overweight execution * Refactor * Missing file * Fix WeightCounter usage in scheduler Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix peek_index Take into account that decoding from a mutable slice modifies it. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add tests and bench service_page_item Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add debug_info Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add no-progress check to service_queues Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add more benches Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bound from_message and try_append_message Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add PageReaped event Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename BookStateOf and BookStateFor Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update tests and remove logging Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove redundant per-message origins; add footprint() and sweep_queue() * Move testing stuff to mock.rs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add integration test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix no-progress check Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix debug_info Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup merge and tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix footprint tracking * Introduce * Formatting * OverweightEnqueued event, auto-servicing config item * Update tests and benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Provide change handler * Add missing BookStateFor::insert and call QueueChangeHandler Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update benchmarks and weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use weight metering functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * weightInfo::process_message_payload is gone Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add defensive_saturating_accrue Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename WeightCounter to WeightMeter Ctr+Shift+H should do the trick. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test on_initialize Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add module docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove origin from MaxMessageLen The message origin is not encoded into the heap and does therefore not influence the max message length anymore. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add BoundedVec::as_slice Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test Page::{from_message, try_append_message} Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs * Do nothing in sweep_queue if the queue does not exist ... otherwise it inserts default values into the storage. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test ring (un)knitting Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Upgrade stress-test Change the test to not assume that all queued messages will be processed in the next block but split it over multiple. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests... Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Rename BoundedVec::as_slice to as_bounded_slice Conflicts with deref().as_slice() otherwise. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix imports Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove ReadyRing struct Was used for testing only. Instead use 'fn assert_ring' which also check the service head and backlinks. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix stale page watermark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test feature and clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * QueueChanged handler is called correctly Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update benches Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Abstract testing functions Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Simplify tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make stuff compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Extend overweight execution benchmark Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove TODOs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test service queue with faulty MessageProcessor Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update pallet ui tests to 1.65 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * More docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review doc fixes Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add weight_limit to extrinsic weight of execute_overweight * Correctly return unused weight * Return actual weight consumed in do_execute_overweight * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Set version 7.0.0-dev Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make it compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Switch message_size to u64 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Switch message_count to u64 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make CI green Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Docs * Update tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue * Dont mention README.md in the Cargo.toml Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove reference to readme Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: parity-processbot <> Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
1 parent 90ab4fa commit 47bd959

20 files changed

Lines changed: 3883 additions & 15 deletions

File tree

Cargo.lock

Lines changed: 32 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ members = [
121121
"frame/offences/benchmarking",
122122
"frame/preimage",
123123
"frame/proxy",
124+
"frame/message-queue",
124125
"frame/nomination-pools",
125126
"frame/nomination-pools/fuzzer",
126127
"frame/nomination-pools/benchmarking",

bin/node/runtime/Cargo.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ pallet-indices = { version = "4.0.0-dev", default-features = false, path = "../.
7575
pallet-identity = { version = "4.0.0-dev", default-features = false, path = "../../../frame/identity" }
7676
pallet-lottery = { version = "4.0.0-dev", default-features = false, path = "../../../frame/lottery" }
7777
pallet-membership = { version = "4.0.0-dev", default-features = false, path = "../../../frame/membership" }
78+
pallet-message-queue = { version = "7.0.0-dev", default-features = false, path = "../../../frame/message-queue" }
7879
pallet-mmr = { version = "4.0.0-dev", default-features = false, path = "../../../frame/merkle-mountain-range" }
7980
pallet-multisig = { version = "4.0.0-dev", default-features = false, path = "../../../frame/multisig" }
8081
pallet-nomination-pools = { version = "1.0.0", default-features = false, path = "../../../frame/nomination-pools"}
@@ -150,6 +151,7 @@ std = [
150151
"sp-inherents/std",
151152
"pallet-lottery/std",
152153
"pallet-membership/std",
154+
"pallet-message-queue/std",
153155
"pallet-mmr/std",
154156
"pallet-multisig/std",
155157
"pallet-nomination-pools/std",
@@ -229,6 +231,7 @@ runtime-benchmarks = [
229231
"pallet-indices/runtime-benchmarks",
230232
"pallet-lottery/runtime-benchmarks",
231233
"pallet-membership/runtime-benchmarks",
234+
"pallet-message-queue/runtime-benchmarks",
232235
"pallet-mmr/runtime-benchmarks",
233236
"pallet-multisig/runtime-benchmarks",
234237
"pallet-nomination-pools-benchmarking/runtime-benchmarks",
@@ -282,6 +285,7 @@ try-runtime = [
282285
"pallet-identity/try-runtime",
283286
"pallet-lottery/try-runtime",
284287
"pallet-membership/try-runtime",
288+
"pallet-message-queue/try-runtime",
285289
"pallet-mmr/try-runtime",
286290
"pallet-multisig/try-runtime",
287291
"pallet-nomination-pools/try-runtime",

bin/node/runtime/src/lib.rs

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1135,6 +1135,25 @@ impl pallet_bounties::Config for Runtime {
11351135
type ChildBountyManager = ChildBounties;
11361136
}
11371137

1138+
parameter_types! {
1139+
/// Allocate at most 20% of each block for message processing.
1140+
///
1141+
/// Is set to 20% since the scheduler can already consume a maximum of 80%.
1142+
pub MessageQueueServiceWeight: Option<Weight> = Some(Perbill::from_percent(20) * RuntimeBlockWeights::get().max_block);
1143+
}
1144+
1145+
impl pallet_message_queue::Config for Runtime {
1146+
type RuntimeEvent = RuntimeEvent;
1147+
type WeightInfo = ();
1148+
/// NOTE: Always set this to `NoopMessageProcessor` for benchmarking.
1149+
type MessageProcessor = pallet_message_queue::mock_helpers::NoopMessageProcessor;
1150+
type Size = u32;
1151+
type QueueChangeHandler = ();
1152+
type HeapSize = ConstU32<{ 64 * 1024 }>;
1153+
type MaxStale = ConstU32<128>;
1154+
type ServiceWeight = MessageQueueServiceWeight;
1155+
}
1156+
11381157
parameter_types! {
11391158
pub const ChildBountyValueMinimum: Balance = 1 * DOLLARS;
11401159
}
@@ -1699,6 +1718,7 @@ construct_runtime!(
16991718
RankedPolls: pallet_referenda::<Instance2>,
17001719
RankedCollective: pallet_ranked_collective,
17011720
FastUnstake: pallet_fast_unstake,
1721+
MessageQueue: pallet_message_queue,
17021722
}
17031723
);
17041724

@@ -1793,6 +1813,7 @@ mod benches {
17931813
[pallet_indices, Indices]
17941814
[pallet_lottery, Lottery]
17951815
[pallet_membership, TechnicalMembership]
1816+
[pallet_message_queue, MessageQueue]
17961817
[pallet_mmr, Mmr]
17971818
[pallet_multisig, Multisig]
17981819
[pallet_nomination_pools, NominationPoolsBench::<Runtime>]

frame/message-queue/Cargo.toml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
[package]
2+
authors = ["Parity Technologies <admin@parity.io>"]
3+
edition = "2021"
4+
name = "pallet-message-queue"
5+
version = "7.0.0-dev"
6+
license = "Apache-2.0"
7+
homepage = "https://substrate.io"
8+
repository = "https://github.com/paritytech/substrate/"
9+
description = "FRAME pallet to queue and process messages"
10+
11+
[dependencies]
12+
codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false, features = ["derive"] }
13+
scale-info = { version = "2.1.2", default-features = false, features = ["derive"] }
14+
serde = { version = "1.0.137", optional = true, features = ["derive"] }
15+
log = { version = "0.4.17", default-features = false }
16+
17+
sp-core = { version = "7.0.0", default-features = false, path = "../../primitives/core" }
18+
sp-io = { version = "7.0.0", default-features = false, path = "../../primitives/io" }
19+
sp-runtime = { version = "7.0.0", default-features = false, path = "../../primitives/runtime" }
20+
sp-std = { version = "5.0.0", default-features = false, path = "../../primitives/std" }
21+
sp-arithmetic = { version = "6.0.0", default-features = false, path = "../../primitives/arithmetic" }
22+
sp-weights = { version = "4.0.0", default-features = false, path = "../../primitives/weights" }
23+
24+
frame-benchmarking = { version = "4.0.0-dev", default-features = false, optional = true, path = "../benchmarking" }
25+
frame-support = { version = "4.0.0-dev", default-features = false, path = "../support" }
26+
frame-system = { version = "4.0.0-dev", default-features = false, path = "../system" }
27+
28+
[dev-dependencies]
29+
sp-tracing = { version = "6.0.0", path = "../../primitives/tracing" }
30+
rand = "0.8.5"
31+
rand_distr = "0.4.3"
32+
33+
[features]
34+
default = ["std"]
35+
std = [
36+
"codec/std",
37+
"scale-info/std",
38+
"sp-core/std",
39+
"sp-io/std",
40+
"sp-runtime/std",
41+
"sp-std/std",
42+
"sp-arithmetic/std",
43+
"sp-weights/std",
44+
"frame-benchmarking?/std",
45+
"frame-support/std",
46+
"frame-system/std",
47+
]
48+
runtime-benchmarks = [
49+
"frame-benchmarking/runtime-benchmarks",
50+
"frame-support/runtime-benchmarks",
51+
"frame-system/runtime-benchmarks",
52+
]
53+
try-runtime = ["frame-support/try-runtime"]

0 commit comments

Comments
 (0)