bug: With 100GB data, Sum() or Avg() leads StoneDB memory overflow. #21
Description
Describe the problem
subquery subquery - aggregation overflow
problem
Aggregation overflow occurs under 100g data volume
run query 1
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1105 (HY000) at line 4 in file: '/data5/zsp/tpc-h/queries/query1. sql': The query includes syntax that is not supported by the storage engine. Stonedb: Aggregation overflow.
SQL statement
-- using default substitutions
select
l_ returnflag,
l_ linestatus,
sum(l_quantity) as sum_ qty,
sum(l_extendedprice) as sum_ base_ price,
sum(l_extendedprice * (1 - l_discount)) as sum_ disc_ price,
sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_ charge,
avg(l_quantity) as avg_ qty,
avg(l_extendedprice) as avg_ price,
avg(l_discount) as avg_ disc,
count(*) as count_ order
from
lineitem
where
-- l_ shipdate <= date '1998-12-01' - interval '90' day (3)
l_ shipdate <= date '1998-12-01' - interval '90' day
group by
l_ returnflag,
l_ linestatus
order by
l_ returnflag,
l_ linestatus;
-- limit -1;
Solution
- Reduce the amount of aggregate data processed by a single instance;
- Rewrite SQL statements;
- Research MariaDB and other column storage engines to process this part of the source code of aggregation, and rewrite the operation logic of stonedb.
conclusion
The amount of data is large, reaching the maximum value of the aggregation operation processed by a single instance, causing the aggregation overflow to be thrown.
Expected behavior
How To Reproduce
Environment
Additional context