This repository was archived by the owner on Sep 11, 2020. It is now read-only.
Improve packfile reading performance#906
Merged
Conversation
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
plumbing/format/idxfile: add new Index and MemoryIndex
Signed-off-by: Javi Fontan <jfontan@gmail.com>
In one case it disables the cache and the other disables lookup when the scanner is not seekable. Could be added back later. Signed-off-by: Javi Fontan <jfontan@gmail.com>
It's still not complete: * 64 bit offsets * IdxChecksum Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
This functionality may be moved elsewhere in the future but is needed now to fit filesystem.ObjectStorage and the new index. Signed-off-by: Javi Fontan <jfontan@gmail.com>
Index is also automatically generated when OnFooter is called. Signed-off-by: Javi Fontan <jfontan@gmail.com>
Now dotgit.PackWriter uses the new packfile.Parser and index. Signed-off-by: Javi Fontan <jfontan@gmail.com>
Feature/new packfile parser
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Bugfixes and IndexStorage
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
plumbing: packfile, new Packfile representation
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Signed-off-by: Javi Fontan <jfontan@gmail.com>
Tests and indexes in packfile decoder
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
ajnavarro
reviewed
Jul 31, 2018
plumbing/format/packfile/decoder.go
Outdated
| d.writer.Add(obj.Hash(), uint64(h.Offset), crc) | ||
| } | ||
|
|
||
| d.offsetToHash[h.Offset] = obj.Hash() |
Contributor
There was a problem hiding this comment.
why do we need that having the index?
Contributor
Author
There was a problem hiding this comment.
You may not have the index built
ajnavarro
reviewed
Jul 31, 2018
| }) | ||
| } | ||
|
|
||
| /* |
Contributor
There was a problem hiding this comment.
should that be uncommented?
ajnavarro
reviewed
Jul 31, 2018
| var base plumbing.EncodedObject | ||
| var ok bool | ||
| hash, err := p.FindHash(offset) | ||
| if err == nil { |
Contributor
Author
There was a problem hiding this comment.
yeah, that's intentional
ajnavarro
reviewed
Jul 31, 2018
|
|
||
| func (s *PackfileSuite) TestContent(c *C) { | ||
| storer := memory.NewObjectStorage() | ||
| decoder, err := NewDecoder(NewScanner(s.f.Packfile()), storer) |
Contributor
There was a problem hiding this comment.
why we are using decoder here?
Contributor
Author
There was a problem hiding this comment.
To fill the memory storage and check if Packfile decodes objects correctly as decoder does
Contributor
Author
|
Windows tests seem to be failing because of an access denied removing fixtures 🤷♀️ |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Contributor
Author
|
Just added a benchmark for Before, using Now, using As we can see, Parser is slower. Like, way slower. |
Contributor
Author
|
New results after some optimizations, now it's faster and uses less memory than decoder. |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Collaborator
|
We might want to rename |
Collaborator
|
@erizocosmico Windows issue is likely to be related to a packfile not being closed before the test ends. |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Contributor
Author
|
Added benchmarks for PackfileIter. Before: Now: Now, reading object content: |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
erizocosmico
force-pushed
the
perf/packfile-reads
branch
from
fa697fa to
65dc4f9
Compare
ajnavarro
reviewed
Aug 9, 2018
| Children []*objectInfo | ||
| SHA1 plumbing.Hash | ||
|
|
||
| Content []byte |
Contributor
There was a problem hiding this comment.
I think with an LRU cache for content by offset we can avoid using a lot of memory.
Contributor
Author
There was a problem hiding this comment.
We'd still need to read objects more times, which is why it was slower before
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Contributor
Author
|
Renamed DiskObject to FSObject and I'm fixing the not closed packfile thing |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
erizocosmico
force-pushed
the
perf/packfile-reads
branch
from
24dc247 to
d93b386
Compare
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Contributor
Author
|
Tests should be passing now. |
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
Contributor
Author
|
I think it can be reviewed now |
mcuadros
suggested changes
Aug 10, 2018
| ) | ||
|
|
||
| const ( | ||
| fanout = 256 |
Contributor
There was a problem hiding this comment.
the previous fanout was 255, was an error?
Contributor
Author
There was a problem hiding this comment.
Real fanout size is 256, we were using 255 because the last element is the object count. JGit uses 256 as well https://github.com/eclipse/jgit/blob/6d370d837c5faa7caff2e6e3e4723b887f2fbdca/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/PackIndexV2.java#L67
plumbing/format/idxfile/idxfile.go
Outdated
| Version uint32 | ||
| Fanout [256]uint32 | ||
| // FanoutMapping maps the position in the fanout table to the position | ||
| // in the Names, Offset32 and Crc32 slices. This improves the memory |
plumbing/format/idxfile/idxfile.go
Outdated
| FanoutMapping [256]int | ||
| Names [][]byte | ||
| Offset32 [][]byte | ||
| Crc32 [][]byte |
plumbing/format/idxfile/idxfile.go
Outdated
| Hash plumbing.Hash | ||
| CRC32 uint32 | ||
| Offset uint64 | ||
| func (idx *MemoryIndex) findHashIndex(h plumbing.Hash) int { |
Contributor
There was a problem hiding this comment.
In go is recommended return a second boolean value
findHashIndex(h plumbing.Hash) (int, bool)
plumbing/format/idxfile/idxfile.go
Outdated
| return idx.getCrc32(k, i) | ||
| } | ||
|
|
||
| func (idx *MemoryIndex) getCrc32(firstLevel, secondLevel int) (uint32, error) { |
| firstLevel, secondLevel int | ||
| } | ||
|
|
||
| func (i *idxfileEntryIter) Next() (*Entry, error) { |
Contributor
There was a problem hiding this comment.
this function can be split in two
Contributor
Author
There was a problem hiding this comment.
Which part do you mean? I don't see any way to split this
Signed-off-by: Miguel Molina <miguel@erizocosmi.co>
jfontan
suggested changes
Aug 13, 2018
| } | ||
| } | ||
|
|
||
| if delta && !p.scanner.IsSeekable { |
Contributor
There was a problem hiding this comment.
Shouldn't we use the storer to retrieve objects instead of storing all deltas in memory?
Contributor
Author
There was a problem hiding this comment.
Some storers, such as the MemoryStorage, do not allow deltas stored inside them
jfontan
approved these changes
Aug 13, 2018
mcuadros
approved these changes
Aug 14, 2018
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More info at: https://docs.google.com/document/d/1I2Vte4LhMdpx0kvlrg-k6hNA-0kUGtUkC1-kfwbCmeI