Skip to content

Add option to rollback inference for a certain number of steps#588

Merged
xtinkt merged 7 commits intomainfrom
speculative_inference
Jul 9, 2024
Merged

Add option to rollback inference for a certain number of steps#588
xtinkt merged 7 commits intomainfrom
speculative_inference

Conversation

@xtinkt
Copy link
Copy Markdown
Collaborator

@xtinkt xtinkt commented Jul 2, 2024

No description provided.

@justheuristic justheuristic self-requested a review July 2, 2024 17:52
@justheuristic justheuristic changed the title [DRAFT] inv Option to rollback inference for a certain number of steps Jul 3, 2024
Copy link
Copy Markdown
Collaborator

@justheuristic justheuristic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick request: please delete .swp file from the PR

Copy link
Copy Markdown
Collaborator

@justheuristic justheuristic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we can merge this now and begin working on a client-side interface

@justheuristic justheuristic changed the title Option to rollback inference for a certain number of steps Add option to rollback inference for a certain number of steps Jul 9, 2024
@xtinkt xtinkt merged commit c0a4d2e into main Jul 9, 2024
@xtinkt xtinkt deleted the speculative_inference branch July 9, 2024 15:36
justheuristic added a commit that referenced this pull request Jul 21, 2024
* Add option to rollback inference for a certain number of steps (#588)

* fix

* fix

* fix

* fix

* fix

* fix

* style

* test running inference session with position getter/setter

* add assertion

* fix typo

---------

Co-authored-by: Anton Sinitsin <30695750+xtinkt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants