perf: various minor perf fixes for ledger postings by ankush · Pull Request #26775 · frappe/erpnext

ankush · 2021-08-02T07:37:08Z

Ledger posting related code is on the "critical path" for most stock/accounting transactions as it generates ledger entries. This usually takes time roughly on the order of size of child tables.
Changed several methods to use db.value_cache for fetching details that won't change during a db transaction, this cache is dropped frequently when db instance dies, so it's safe to use.
reordered conditionals to avoid queries.

Outcome: Along with that ^ PR, ~20-25% saving in queries and time while submitting large documents with repeated validations. More stats in linked PR.

Note to reviewers: review commit by commit, would be easier that way.

erpnext/accounts/utils.py

- Order fields such that comparison will fail faster - Break out of loops if not matched

These are set only once fields

`select *` fetches all fields, output of this function is only used for comparing.

These properties don't change often, no need to query everytime.

ankush · 2021-08-09T05:51:16Z

Dropped a refactor that would require additional tests for now, will add in separate PR (in next release)

coveralls · 2021-08-09T06:15:36Z

Coverage increased (+0.002%) to 42.692% when pulling 966d643 on ankush:minor_perf_fixes into 7e0c57f on frappe:develop.

marination · 2021-08-11T05:47:35Z

Asset, POS and Attendance tests failing on other PRs as well. Not caused by this PR

* perf: only validate if voucher is journal entry * perf: optimize merge GLE - Order fields such that comparison will fail faster - Break out of loops if not matched * perf: don't try to match SLE if count mismatch * refactor: simplify initialize_previous_data * perf: use cache for fetching valuation_method These are set only once fields * refactor: simplify get_future_stock_vouchers * refactor: simplify get_voucherwise_gl_entries * perf: fetch only required fields for GL comparison `select *` fetches all fields, output of this function is only used for comparing. * perf: reorder conditions in PL cost center check * perf: reduce query while validating new gle * perf: use cache for validating warehouse props These properties don't change often, no need to query everytime. * perf: use cached stock settings to validate SLE * docs: update misleading docstring Co-authored-by: Marica <maricadsouza221197@gmail.com> (cherry picked from commit 9152715)

* perf: only validate if voucher is journal entry * perf: optimize merge GLE - Order fields such that comparison will fail faster - Break out of loops if not matched * perf: don't try to match SLE if count mismatch * refactor: simplify initialize_previous_data * perf: use cache for fetching valuation_method These are set only once fields * refactor: simplify get_future_stock_vouchers * refactor: simplify get_voucherwise_gl_entries * perf: fetch only required fields for GL comparison `select *` fetches all fields, output of this function is only used for comparing. * perf: reorder conditions in PL cost center check * perf: reduce query while validating new gle * perf: use cache for validating warehouse props These properties don't change often, no need to query everytime. * perf: use cached stock settings to validate SLE * docs: update misleading docstring Co-authored-by: Marica <maricadsouza221197@gmail.com> (cherry picked from commit 9152715) Co-authored-by: Ankush <ankush@iwebnotes.com>

* perf: only validate if voucher is journal entry * perf: optimize merge GLE - Order fields such that comparison will fail faster - Break out of loops if not matched * perf: don't try to match SLE if count mismatch * refactor: simplify initialize_previous_data * perf: use cache for fetching valuation_method These are set only once fields * refactor: simplify get_future_stock_vouchers * refactor: simplify get_voucherwise_gl_entries * perf: fetch only required fields for GL comparison `select *` fetches all fields, output of this function is only used for comparing. * perf: reorder conditions in PL cost center check * perf: reduce query while validating new gle * perf: use cache for validating warehouse props These properties don't change often, no need to query everytime. * perf: use cached stock settings to validate SLE * docs: update misleading docstring Co-authored-by: Marica <maricadsouza221197@gmail.com> (cherry picked from commit 9152715)

* perf: only validate if voucher is journal entry * perf: optimize merge GLE - Order fields such that comparison will fail faster - Break out of loops if not matched * perf: don't try to match SLE if count mismatch * refactor: simplify initialize_previous_data * perf: use cache for fetching valuation_method These are set only once fields * refactor: simplify get_future_stock_vouchers * refactor: simplify get_voucherwise_gl_entries * perf: fetch only required fields for GL comparison `select *` fetches all fields, output of this function is only used for comparing. * perf: reorder conditions in PL cost center check * perf: reduce query while validating new gle * perf: use cache for validating warehouse props These properties don't change often, no need to query everytime. * perf: use cached stock settings to validate SLE * docs: update misleading docstring Co-authored-by: Marica <maricadsouza221197@gmail.com>

ankush added backport version-13-hotfix needs-tests This PR needs automated unit-tests. labels Aug 2, 2021

ankush commented Aug 2, 2021

View reviewed changes

erpnext/accounts/utils.py Outdated Show resolved Hide resolved

ankush force-pushed the minor_perf_fixes branch 2 times, most recently from 0c06bdc to e966c43 Compare August 6, 2021 10:09

ankush added 13 commits August 9, 2021 11:11

perf: only validate if voucher is journal entry

94396d6

perf: optimize merge GLE

6b333a7

- Order fields such that comparison will fail faster - Break out of loops if not matched

perf: don't try to match SLE if count mismatch

679d7dd

refactor: simplify initialize_previous_data

2f5624e

perf: use cache for fetching valuation_method

52ea381

These are set only once fields

refactor: simplify get_future_stock_vouchers

f46df60

refactor: simplify get_voucherwise_gl_entries

77914c0

perf: fetch only required fields for GL comparison

9d58007

`select *` fetches all fields, output of this function is only used for comparing.

perf: reorder conditions in PL cost center check

3198f07

perf: reduce query while validating new gle

c7dc641

perf: use cache for validating warehouse props

da8f976

These properties don't change often, no need to query everytime.

perf: use cached stock settings to validate SLE

d6b9c0b

docs: update misleading docstring

966d643

ankush force-pushed the minor_perf_fixes branch from e966c43 to 966d643 Compare August 9, 2021 05:50

ankush removed the needs-tests This PR needs automated unit-tests. label Aug 9, 2021

ankush marked this pull request as ready for review August 9, 2021 05:53

ankush requested review from deepeshgarg007, marination, nextchamp-saqib and rohitwaghchaure as code owners August 9, 2021 05:53

marination approved these changes Aug 10, 2021

View reviewed changes

marination added 2 commits August 10, 2021 19:32

Merge branch 'develop' into minor_perf_fixes

1ca671a

Merge branch 'develop' into minor_perf_fixes

ac00cf2

marination merged commit 9152715 into frappe:develop Aug 11, 2021

frappe-pr-bot mentioned this pull request Aug 11, 2021

perf: various minor perf fixes for ledger postings #26896

Merged

ankush added the backport version-13-pre-release label Aug 11, 2021

frappe-pr-bot mentioned this pull request Aug 11, 2021

perf: various minor perf fixes for ledger postings #26901

Closed

ankush mentioned this pull request Aug 31, 2021

revert: "refactor: simplify initialize_previous_data" #27270

Merged

ankush deleted the minor_perf_fixes branch August 31, 2021 13:52

This was referenced Sep 1, 2021

chore: change log for v13.10.0 #27303

Merged

chore: change log for v13.10.0 frappe/frappe#14103

Merged

ankush mentioned this pull request Sep 7, 2021

fix: missed to add voucher_type, voucher_no to get GL Entries #27368

Merged

deepeshgarg007 mentioned this pull request Jul 18, 2022

List of V14 features & enhancements #31620

Closed

mergify bot mentioned this pull request Feb 26, 2025

fix: missed to add voucher_type, voucher_no to get GL Entries (backport #27368) #46153

Closed

github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: various minor perf fixes for ledger postings#26775

perf: various minor perf fixes for ledger postings#26775
marination merged 15 commits intofrappe:developfrom
ankush:minor_perf_fixes

ankush commented Aug 2, 2021 •

edited

Loading

Uh oh!

Uh oh!

ankush commented Aug 9, 2021

Uh oh!

coveralls commented Aug 9, 2021

Uh oh!

marination commented Aug 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ankush commented Aug 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ankush commented Aug 9, 2021

Uh oh!

coveralls commented Aug 9, 2021

Uh oh!

marination commented Aug 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ankush commented Aug 2, 2021 •

edited

Loading