feat(flinksql): collect comment, type attribute for entity by Cythia828 · Pull Request #319 · DTStack/dt-sql-parser

Cythia828 · 2024-06-20T08:58:15Z

Feature

收集字段时，收集字段的类型和字段注释
收集新建实体名时，收集实体的注释信息，如果有的话

Fix

修复 [Bug]: ctxToWord is a bad function #305

Description

关于用带_取属性名，antlr4ng目前解析出来属性名都是带_，是公有的。甚至以type命名前后都有下划线，应该是以防重名，无关私有公有
关于findAttributeChildren方法里的child类型，因为antlr4ng 的ParserRuleContext里，children并未重申类型，导致children类型为ParseTree[]，需要和三方提PR修改
关于table alias，需要作为实体去收集，当前PR不放相关内容
调整实体类型，将实体分为常规实体、方法实体、字段实体。值得一提的是，实体均带有isAlias属性和alias、reference、comment几个可选属性，alias为别名，reference为被引用的实体。

…f ctxToWord

src/grammar/flink/FlinkSqlParser.g4

src/parser/common/entityCollector.ts

HaydenOrz · 2024-07-01T02:58:46Z

src/parser/flink/flinkEntityCollector.ts

+        this.pushEntity(ctx, EntityContextType.DATABASE_CREATE, {
+            needCollectAttr: true,
+            attrList: [attrName.comment],
+            endContext: 'CreateDatabaseContext',


可以用 CreateDatabaseContext.constructor.name 代替吗

不可以哦，在这里CreateDatabaseContext.constructor.name 是Function

CreateDatabaseContext.name

src/parser/common/entityCollector.ts

src/grammar/flink/FlinkSqlParser.g4

Cythia828 · 2024-07-03T01:45:22Z

@liuxy0551 @HaydenOrz 关于你们的comment，已经提交了新的commit，可以再次review一下

src/parser/common/entityCollector.ts

HaydenOrz · 2024-07-04T02:33:50Z

src/parser/common/entityCollector.ts

+    return 'line' in value;
+}
+
+export function isColumnEntityContext(value: EntityContext): value is ColumnEntityContext {


src/parser/common/entityCollector.ts

HaydenOrz · 2024-07-04T02:34:38Z

src/parser/common/entityCollector.ts

+    return 'params' in value && 'returns' in value && 'relatedEntities' in value;
+}
+
+export function isWordRange(value: any): value is WordRange {


同上，且参数类型应该是 TextSlice | WordRange

src/parser/common/entityCollector.ts

HaydenOrz · 2024-07-04T02:42:54Z

src/parser/flink/flinkEntityCollector.ts

+        this.pushEntity(ctx, EntityContextType.DATABASE_CREATE, {
+            needCollectAttr: true,
+            attrList: [attrName.comment],
+            endContext: 'CreateDatabaseContext',


CreateDatabaseContext.name

Cythia828 · 2024-07-04T04:03:58Z

@HaydenOrz 优化的commit已提交，可再次review

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(DTStack#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(DTStack#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(DTStack#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <>

* feat: improve errorListener msg (#281) * feat: add mysql errorListener and commonErrorListener * feat: improve other sql error msg * feat: support i18n for error msg * feat: add all sql errorMsg unit test * feat: update locale file and change i18n funtion name * test: upate error unit test * feat(flinksql): collect comment, type attribute for entity (#319) * feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <> * fix: spell check (#337) Co-authored-by: liuyi <[email protected]> * ci: check-types and test unit update * feat: collect entity's attribute(#333) * feat(trinosql): collect trino sql's attribute(comment,alias,colType) * feat(hivesql): collect hive sql's attribute(comment,alias,colType) * feat(impalasql): collect attribute(comment, colType, alias) * feat(sparksql): collect entity's attribute (comment,alias, colType) * feat: update endContextList of collect attribute * feat(postgresql): collect hive sql's attribute(alias,colType) * feat: update interface of attrInfo and alter entitycollect ts file * feat(mysql): collect entity's attribute(comment,colType,alias) * ci: fix check-types problem --------- Co-authored-by: zhaoge <> * chore(release): 4.1.0-beta.0 * fix: #362 set hiveVar value (#369) * fix: #371 export EntityContext types (#372) * fix: minimum collect candidates boundary to fix parse performance (#378) * fix: minimum collect candidates boundary to fix parse performance * fix: fix check-types * fix: remove debugger code * fix(flink): fix flinksql syntax error about ROW and function using (#383) Co-authored-by: zhaoge <> * build: pnpm antlr4 --lang all --------- Co-authored-by: 霜序 <[email protected]> Co-authored-by: XCynthia <[email protected]> Co-authored-by: 琉易 <[email protected]> Co-authored-by: liuyi <[email protected]> Co-authored-by: zhaoge <> Co-authored-by: Hayden <[email protected]> Co-authored-by: JackWang032 <[email protected]>

* feat: improve errorListener msg (#281) * feat: add mysql errorListener and commonErrorListener * feat: improve other sql error msg * feat: support i18n for error msg * feat: add all sql errorMsg unit test * feat: update locale file and change i18n funtion name * test: upate error unit test * feat(flinksql): collect comment, type attribute for entity (#319) * feat(flinksql): collect comment, type attribute for entity * feat(flinksql): delete console log * fix(#305): delete function ctxToWord,using ctxToText instead of ctxToWord * feat: update attribute's type * feat(flinksql): update flinksql's entitycollect unit test * feat: optimize interface and update unit test * feat: update collect attr detail * feat: optimize interface and some function's arguments * feat: add comment and update params' name * feat: collect alias in select statement * feat: update collect attribute function and update unit test --------- Co-authored-by: zhaoge <> * fix: spell check (#337) Co-authored-by: liuyi <[email protected]> * ci: check-types and test unit update * feat: collect entity's attribute(#333) * feat(trinosql): collect trino sql's attribute(comment,alias,colType) * feat(hivesql): collect hive sql's attribute(comment,alias,colType) * feat(impalasql): collect attribute(comment, colType, alias) * feat(sparksql): collect entity's attribute (comment,alias, colType) * feat: update endContextList of collect attribute * feat(postgresql): collect hive sql's attribute(alias,colType) * feat: update interface of attrInfo and alter entitycollect ts file * feat(mysql): collect entity's attribute(comment,colType,alias) * ci: fix check-types problem --------- Co-authored-by: zhaoge <> * chore(release): 4.1.0-beta.0 * fix: #362 set hiveVar value (#369) * fix: #371 export EntityContext types (#372) * fix: minimum collect candidates boundary to fix parse performance (#378) * fix: minimum collect candidates boundary to fix parse performance * fix: fix check-types * fix: remove debugger code * fix(flink): fix flinksql syntax error about ROW and function using (#383) Co-authored-by: zhaoge <> * build: pnpm antlr4 --lang all * Feat/follow keywords (#407) * feat: provide follow keywords when get suggestions * chore: add watch script * refactor: optimize spark grammar (#360) * feat: support semantic context of isNewStatement (#361) * feat: support semantic context of isStatamentBeginning * docs: add docs for semantic context * feat: unify variables in lexer (#366) * feat: unify variables in lexer * fix: all sql use WHITE_SPACE * feat: complete after error syntax (#334) * refactor: split getMinimumParserInfo to slice input and parser again * test: complete after error syntax * feat: complete after error syntax * feat: use createParser to get parserIns and remove parserWithNewInput * feat(all sql): add all sql expression column (#358) * feat(impala): add impala expression column * feat(trino): add expression column * feat(hive): add hive expression column * feat(spark): add spark expression column * feat(mysql): add mysql expression column unit test * feat(flink): add flink expression column * feat(postgresql): add pg expression column * feat: #410 optimize processCandidates tokenIndexOffset (#411) * test: test suggestion wordRanges with range when processCandidates without tokenIndexOffset * feat: #410 optimize processCandidates tokenIndexOffset --------- Co-authored-by: 霜序 <[email protected]> Co-authored-by: XCynthia <[email protected]> Co-authored-by: 琉易 <[email protected]> Co-authored-by: liuyi <[email protected]> Co-authored-by: zhaoge <> Co-authored-by: Hayden <[email protected]> Co-authored-by: JackWang032 <[email protected]> Co-authored-by: JackWang032 <[email protected]>

zhaoge added 2 commits June 20, 2024 16:12

feat(flinksql): collect comment, type attribute for entity

36803b7

feat(flinksql): delete console log

cb9a37a

Cythia828 requested review from HaydenOrz, LuckyFBB, liuxy0551 and mumiao June 20, 2024 08:58

Cythia828 force-pushed the feat/collect_attribute branch from 2c3cbd4 to cb9a37a Compare June 28, 2024 08:36

zhaoge added 3 commits June 28, 2024 16:45

fix(DTStack#305): delete function ctxToWord,using ctxToText instead o…

e328d78

…f ctxToWord

feat: update attribute's type

2287ba4

feat(flinksql): update flinksql's entitycollect unit test

09ac711

Cythia828 mentioned this pull request Jun 28, 2024

[Bug]: ctxToWord is a bad function #305

Closed

Cythia828 changed the base branch from main to next July 1, 2024 01:47

liuxy0551 reviewed Jul 1, 2024

View reviewed changes

src/grammar/flink/FlinkSqlParser.g4 Outdated Show resolved Hide resolved

src/parser/common/entityCollector.ts Outdated Show resolved Hide resolved

HaydenOrz reviewed Jul 1, 2024

View reviewed changes

LuckyFBB reviewed Jul 1, 2024

View reviewed changes

src/grammar/flink/FlinkSqlParser.g4 Show resolved Hide resolved

zhaoge added 2 commits July 2, 2024 18:46

feat: optimize interface and update unit test

4b6728a

feat: update collect attr detail

550273a

HaydenOrz reviewed Jul 4, 2024

View reviewed changes

feat: optimize interface and some function's arguments

ac0bdb2

zhaoge added 3 commits July 4, 2024 16:06

feat: add comment and update params' name

22a9454

feat: collect alias in select statement

009effc

feat: update collect attribute function and update unit test

16425d7

HaydenOrz merged commit a94e7fc into DTStack:next Jul 5, 2024

Cythia828 deleted the feat/collect_attribute branch July 5, 2024 08:03

Conversation

Cythia828 commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Feature

Fix

Description

Uh oh!

Uh oh!

Uh oh!

HaydenOrz Jul 1, 2024

Choose a reason for hiding this comment

Uh oh!

Cythia828 Jul 3, 2024

Choose a reason for hiding this comment

Uh oh!

HaydenOrz Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cythia828 commented Jul 3, 2024

Uh oh!

Uh oh!

HaydenOrz Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HaydenOrz Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HaydenOrz Jul 4, 2024

Choose a reason for hiding this comment

Uh oh!

Cythia828 commented Jul 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Cythia828 commented Jun 20, 2024 •

edited

Loading