Skip to content
This repository was archived by the owner on Sep 9, 2025. It is now read-only.

Improve SQLite storage #293

Merged
hendrikvanantwerpen merged 6 commits into from
Jul 27, 2023
Merged

Improve SQLite storage #293

hendrikvanantwerpen merged 6 commits into from
Jul 27, 2023

Conversation

@hendrikvanantwerpen
Copy link
Contributor

@hendrikvanantwerpen hendrikvanantwerpen commented Jul 4, 2023 •
edited
Loading

This PR fixes several issues with the storage API.

Fixes #299.

Changes

  • Add methods to clear SQLiteReader instances, which makes it easier to reuse a reader without accumulating cached data.
  • Fix column name in SQL statement
  • Small tweaks to the storage API
  • Fix deserialization errors by using a different serialization method

Serialization format

The rmp-serde crate seems less stable than I'd hoped. I porpose to switch to postcard, which seems broadly used, has a documented stable wire format, and performs well in the benchmarks at https://github.com/djkoloski/rust_serialization_benchmark.

@hendrikvanantwerpen hendrikvanantwerpen self-assigned this Jul 4, 2023
@github-actions
Copy link

github-actions bot commented Jul 4, 2023

Performance Summary

Comparing base acb0fab with head f6ba30c on microsoft/[email protected]. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before 
--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.576^           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #         :          :                                       
     |           #         :          :                                       
     |           #        ::  :       :                                ::     
     |           #     :: ::  :       :     :                         ::: :   
     |         @@#     :  ::  :@      :  ::::         ::            : :::::   
     |    @@:  @ #   ::: @::  :@     ::  :: :         :             :::::::   
     |    @ :  @ #:: : : @::  :@  :::::  :: : :: ::::::   :    @: :::::::::@  
   0 +----------------------------------------------------------------------->Gi
     0                                                                   205.4
After 
--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    MB
972.9^            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                                                   @       
     |            #         :      :                                  @ :     
     |            #         :      :::                                @ :     
     |           :#      :  :      ::    :                           :@ :     
     |     @::   :#      :  :    : ::    :          ::               :@ :     
     |     @:    :#:::   :  :   :: ::  ::::         :                :@ :     
     |     @:    :#:: :: :  :   :: ::  : ::@       ::    :@      : : :@::     
     |     @: :  :#:: :: : :::@@:: :: :: ::@  :: : ::  :::@  ::::: : :@:::: : 
   0 +----------------------------------------------------------------------->Gi
     0                                                                   184.2

@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Performance Summary

Comparing base acb0fab with head 87e8c76 on microsoft/[email protected]. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before 
--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.142^            ##                                                          
     |            #                                                           
     |            #                                                           
     |            #                                                           
     |           @#                                                           
     |           @#                                                     :     
     |           @#                                                     :     
     |           @#         :            ::           :                 :     
     |       @@  @#      @@ :         ::::            :              @@ :     
     |       @   @#      @  :      :: : ::          :::              @  :     
     |      :@   @# :::  @  :   :: :  : ::     :::: :::    :      :::@ ::     
     |      :@   @# : :  @  :   : ::  : ::  :  :::  :::::  : :  : : :@ :::::  
   0 +----------------------------------------------------------------------->Gi
     0                                                                   185.1
After 
--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.539^           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #        ::         :                                        
     |           #        :          :                                        
     |           #       @: :        :    ::           :             ::::     
     |      :  ::#     ::@: :        :   ::            :             :: ::    
     |     ::  : #     : @: :        : ::::      :  ::::             :: :: :  
     |     ::  : #:@@: : @: :@      :: : :: ::: :::::: :  ::   ::: : :: ::::  
   0 +----------------------------------------------------------------------->Gi
     0                                                                   221.5

@hendrikvanantwerpen hendrikvanantwerpen marked this pull request as ready for review July 27, 2023 16:09
@hendrikvanantwerpen hendrikvanantwerpen requested a review from a team as a code owner July 27, 2023 16:09
@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Performance Summary

Comparing base e721b3a with head fef68c7 on microsoft/[email protected]. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before 
--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.154^            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                  ::                                       
     |            #                  :                                        
     |            #                  :                                        
     |       @@  @#      :: @@       :    :                                   
     |       @   @#      :  @        :    :           ::              :       
     |       @   @#  ::  :  @        :    :           :    ::         :       
     |    @@:@   @#:::   :  @     :  : @:::      : ::::    :    :: @@ :    :  
     |    @ :@   @#:::   :  @ ::  :  : @: :  ::  ::: ::   ::  :::  @  :::::: :
   0 +----------------------------------------------------------------------->Gi
     0                                                                   189.4
After 
--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.539^           ##                                                           
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                                                            
     |           #                     ::                                     
     |           #                     :                                      
     |           #        ::           :                             ::  ::   
     |          @#      : :  :       :::                             : : :    
     |          @#      :@:  :       : : ::::          ::           @: :::    
     |      :   @#      :@:  :::     : : :::           :            @: :::    
     |      ::  @# :::  :@:  ::    : : : :::     @@  :::  :: :@@::  @: ::: :  
   0 +----------------------------------------------------------------------->Gi
     0                                                                   196.1

@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Performance Summary

Comparing base e721b3a with head 0e55284 on microsoft/[email protected]. For details see workflow artifacts. Note that performance is tested on the last commits with changes in stack-graphs, not on every commit.

Before 
--------------------------------------------------------------------------------
Command:            base/target/release/tree-sitter-stack-graphs-typescript index -D base.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 base-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.154^            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                  ::                                       
     |            #                  :                                        
     |            #                  :                                        
     |       @@  @#      :: @@       :    :                                   
     |       @   @#      :  @        :    :           ::              :       
     |       @   @#  ::  :  @        :    :           :    ::         :       
     |    @@:@   @#:::   :  @     :  : @:::      : ::::    :    :: @@ :    :  
     |    @ :@   @#:::   :  @ ::  :  : @: :  ::  ::: ::   ::  :::  @  :::::: :
   0 +----------------------------------------------------------------------->Gi
     0                                                                   189.4
After 
--------------------------------------------------------------------------------
Command:            head/target/release/tree-sitter-stack-graphs-typescript index -D head.sqlite --max-file-time=30 --hide-error-details -- test/src/compiler
Massif arguments:   --massif-out-file=perf.out
ms_print arguments: --x=72 --y=12 head-perf-results/perf.out
--------------------------------------------------------------------------------


    GB
1.131^            #                                                           
     |            #                                                           
     |            #                                                           
     |            #                  :                                        
     |            #                  :                                   :    
     |            #                  :                                   :    
     |            #                  :                                  @:    
     |            #         ::       :    :                           : @:    
     |           @#         :        : :  :            :              : @:    
     |       @   @#     @@: : @      : ::::           ::   ::        :: @:    
     |      :@   @#:: : @ : : @ ::   : :: :  :  :::@@:::  ::     @   :::@::   
     |      :@   @#:::::@ : : @:: :::: :: : ::  : :@ :::  ::   : @ : :::@::: :
   0 +----------------------------------------------------------------------->Gi
     0                                                                   179.5

@hendrikvanantwerpen hendrikvanantwerpen merged commit bb7457a into main Jul 27, 2023
@hendrikvanantwerpen hendrikvanantwerpen deleted the change-storage-serde branch July 27, 2023 16:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@patrickt patrickt patrickt approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@hendrikvanantwerpen hendrikvanantwerpen

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Messagepack breaks db decode logic

5 participants

@hendrikvanantwerpen @patrickt @Claude @Codex