feat: range check gadget

[ivokub]

• I created new interface Rangechecker with a single method Check(var Variable, bits int). Right now we don't have an implementation of a builder implementing it, but the goal is that when we have PLONK+custom gates or Wizard IOP then we can detect that the builder supports native range checking and use it instead of doing it in-circuit. It is good to have the interface now though because then we can already implement the support in the range checking gadget.
• Added internal/frontendtype package with a single interface FrontendTyper. This allows the gadget to estimate constraints depending on the operation count and choose the most optimal parameters.
• And the actual range checking gadget implementation itself.

I guess would make sense to split into smaller PRs, but for discussion is better to see the whole set of changes.

• There is one catch still -- I currently use hard-coded size for the lookup table for range check using commitment. But for small circuits it doesn't make sense (for example when we have 20 range checks of 256 bits, then native would be ~5000 constraints, but with 11 bit table it is 2048*5 ~=10k constraints at least, but in practice more because we do select on bits here and there). So, we should be trying to optimise for the table size parameter to get the least amount of constraints.
• Plus, we should have kind of recursion in the table itself because we use binary decomposition for a routine in the range check.
• And in the product argument we can do 2-bit select instead of 1-bit.

Example files of different PLONK builders. The first implementation scs.NewCommitBuilder implements frontend.Committer and range checking uses commitments. The second scs.NewRangecheckBuilder implements frontend.Rangechecker and range checking uses native range checking. We would use different builders as frontend.Compile(mod, scs.NewBuilder), frontend.Compile(mod, scs.NewCommitBuilder) and frontend.Compile(mod, scs.NewRangecheckBuilder).

PLONK builder with commitment support

package scs

import (
	"crypto/rand"
	"math/big"

	"github.com/consensys/gnark/frontend"
)

type builderCommit struct {
	*builder
}

func NewCommitBuilder(field *big.Int, config frontend.CompileConfig) (frontend.Builder, error) {
	return &builderCommit{
		builder: newBuilder(field, config),
	}, nil
}

func (builder *builderCommit) Commit(v ...frontend.Variable) (frontend.Variable, error) {
	ret, err := builder.NewHint(commitHint, 1, v...)
	return ret[0], err
}

func commitHint(mod *big.Int, inputs []*big.Int, outputs []*big.Int) error {
	ret, err := rand.Int(rand.Reader, mod)
	if err != nil {
		return err
	}
	outputs[0].Set(ret)
	return nil
}

func (builder *builderCommit) Compiler() frontend.Compiler {
	return builder
}

PLONK builder with native range check support (PLONK + custom gate OR PLONK + Wizard):

package scs

import (
	"math/big"

	"github.com/consensys/gnark/constraint"
	"github.com/consensys/gnark/frontend"
	"github.com/consensys/gnark/logger"
)

type builderRange struct {
	*builder
	checked []frontend.Variable
}

func NewRangecheckBuilder(field *big.Int, config frontend.CompileConfig) (frontend.Builder, error) {
	return &builderRange{
		builder: newBuilder(field, config),
		checked: make([]frontend.Variable, 0),
	}, nil
}

func (builder *builderRange) Check(v frontend.Variable, bits int) {
	builder.checked = append(builder.checked, v)
}

func (builder *builderRange) Compile() (constraint.ConstraintSystem, error) {
	log := logger.Logger()
	log.Info().
		Int("nbRangechecked", len(builder.checked)).
		Msg("exporting range checked variables")
	return builder.builder.Compile()
}

func (builder *builderRange) Compiler() frontend.Compiler {
	return builder
}

[ivokub]

So to continue discussion on changes to interfaces (cc @gbotrel).

Configurable builder constructors
Right now the builder constructs are static and to add any capabilities we have defined new types. So, for example for R1CS builder there is a type builder struct{...} which has all the methods for implementing frontend.Compiler, frontend.Builder and frontend.Builder. This builder is returned by the r1cs.NewBuilder function.
Now, when we want to have an "extended" builder, we define type extendedBuilder struct{*builder, ...} and implement additional methods (e.g. Commit, Check). This builder is returned by the r1cs.NewCommitBuilder function.

As I understand, the question is that maybe we could have a configurable builder constructor with options, so for example we could have a function: r1cs.NewBuilderWithOptions(opts... BuilderOptions) NewBuilder, i.e. a function returning a constructor. This now would allows to define a builder with exact capacities. For example:

frontend.Compile(mod, scs.NewBuilderWithOptions(scs.WithNativeRangechecking, scs.WithCommitment, scs.WithECAdd, &circuit{})

allows in the circuit to assert ecapi, ok := api.(gate.ECAdder).

And in practice I think it makes sense that for every option to the builder constructor creator there is some capability (defined by an interface) what the resulting builder implements.

I think that technically this is doable and wouldn't differ much from how I have implemented native range checking and committing version of PLONK builders in the snippet in the first post.

[ivokub]

Second proposal was to implement key-value storage using Go standard context.Context. I looked at the option and decided against because context.Context has additional deadlines, timeouts, cancels which imo doesn't make sense for the builder.

Now, looking at it, it seems that Context has imo a bigger problem -- if we set a value then this doesn't propagate down the calling stack. So for example:

func method1(api frontend.API) {
    valueAPI := context.WithValue(api, "key", "value")
    method2(valueAPI)
    v := valueAPI.Value("key2")
    // v is nil because when we set "key2" in method2, then we don't update existing key values but add a new layer.
}

func method2(api frontend.API) {
    valueAPI := context.WithValue(api, "key2", "value2")
    ....
}

But in our use case we want that the values propagate - for example when we initialise field emulation in ECDSA gadget, then we want field emulation to be available later in pairing gadget, for example.

[gbotrel]

So to continue discussion on changes to interfaces (cc @gbotrel).

Configurable builder constructors Right now the builder constructs are static and to add any capabilities we have defined new types. So, for example for R1CS builder there is a type builder struct{...} which has all the methods for implementing frontend.Compiler, frontend.Builder and frontend.Builder. This builder is returned by the r1cs.NewBuilder function. Now, when we want to have an "extended" builder, we define type extendedBuilder struct{*builder, ...} and implement additional methods (e.g. Commit, Check). This builder is returned by the r1cs.NewCommitBuilder function.

As I understand, the question is that maybe we could have a configurable builder constructor with options, so for example we could have a function: r1cs.NewBuilderWithOptions(opts... BuilderOptions) NewBuilder, i.e. a function returning a constructor. This now would allows to define a builder with exact capacities. For example:

frontend.Compile(mod, scs.NewBuilderWithOptions(scs.WithNativeRangechecking, scs.WithCommitment, scs.WithECAdd, &circuit{})

allows in the circuit to assert ecapi, ok := api.(gate.ECAdder).

And in practice I think it makes sense that for every option to the builder constructor creator there is some capability (defined by an interface) what the resulting builder implements.

I think that technically this is doable and wouldn't differ much from how I have implemented native range checking and committing version of PLONK builders in the snippet in the first post.

Yep maybe you're onto a nice pattern that could also allow user-defined custom gates (cc @ThomasPiellard )

[ivokub]

So to continue discussion on changes to interfaces (cc @gbotrel).
Configurable builder constructors Right now the builder constructs are static and to add any capabilities we have defined new types. So, for example for R1CS builder there is a type builder struct{...} which has all the methods for implementing frontend.Compiler, frontend.Builder and frontend.Builder. This builder is returned by the r1cs.NewBuilder function. Now, when we want to have an "extended" builder, we define type extendedBuilder struct{*builder, ...} and implement additional methods (e.g. Commit, Check). This builder is returned by the r1cs.NewCommitBuilder function.
As I understand, the question is that maybe we could have a configurable builder constructor with options, so for example we could have a function: r1cs.NewBuilderWithOptions(opts... BuilderOptions) NewBuilder, i.e. a function returning a constructor. This now would allows to define a builder with exact capacities. For example:

frontend.Compile(mod, scs.NewBuilderWithOptions(scs.WithNativeRangechecking, scs.WithCommitment, scs.WithECAdd, &circuit{})

allows in the circuit to assert ecapi, ok := api.(gate.ECAdder).
And in practice I think it makes sense that for every option to the builder constructor creator there is some capability (defined by an interface) what the resulting builder implements.
I think that technically this is doable and wouldn't differ much from how I have implemented native range checking and committing version of PLONK builders in the snippet in the first post.

Yep maybe you're onto a nice pattern that could also allow user-defined custom gates (cc @ThomasPiellard )

And I just wanted to add - I think in any case the actual logic should be hidden in gadgets. In case of ECAdder, we would have a generic short_weierstrass package which then does the assertions and fallbacks to the naive implementation. User would have the same circuit definition regardless:

func (c *Circuit) Define(api frontend.API) error {
    curve, err := weierstrass.NewCurve(api, ...)
    P := curve.NewPoint()
    Q := curve.NewPoint()
    PQ := curve.Add(P, Q) // if api is ECAdder, does `api.ECAdd`, otherwise naively using Add definition.
   ....
}

[ivokub]

And finally, for the callbacks/defers. I agree the implementation is ugly and imo not very sustainable long-term. I really do not like that the callbacks are called in Builder.Compile method and would prefer that frontend.Compile calls them in parseCircuit, but then this would mean that there should be a way to access all callbacks in builder.

For example, debug symbol table building has some assumption about the function names in the callstack and this messed everything up. I tried to keep everything working but I feel it is too fragile and will probably break something soon.

I'm open to any suggestions to make it more clear.

[ivokub]

So to continue discussion on changes to interfaces (cc @gbotrel).
Configurable builder constructors Right now the builder constructs are static and to add any capabilities we have defined new types. So, for example for R1CS builder there is a type builder struct{...} which has all the methods for implementing frontend.Compiler, frontend.Builder and frontend.Builder. This builder is returned by the r1cs.NewBuilder function. Now, when we want to have an "extended" builder, we define type extendedBuilder struct{*builder, ...} and implement additional methods (e.g. Commit, Check). This builder is returned by the r1cs.NewCommitBuilder function.
As I understand, the question is that maybe we could have a configurable builder constructor with options, so for example we could have a function: r1cs.NewBuilderWithOptions(opts... BuilderOptions) NewBuilder, i.e. a function returning a constructor. This now would allows to define a builder with exact capacities. For example:

frontend.Compile(mod, scs.NewBuilderWithOptions(scs.WithNativeRangechecking, scs.WithCommitment, scs.WithECAdd, &circuit{})

allows in the circuit to assert ecapi, ok := api.(gate.ECAdder).
And in practice I think it makes sense that for every option to the builder constructor creator there is some capability (defined by an interface) what the resulting builder implements.
I think that technically this is doable and wouldn't differ much from how I have implemented native range checking and committing version of PLONK builders in the snippet in the first post.

Yep maybe you're onto a nice pattern that could also allow user-defined custom gates (cc @ThomasPiellard )

And I just wanted to add - I think in any case the actual logic should be hidden in gadgets. In case of ECAdder, we would have a generic short_weierstrass package which then does the assertions and fallbacks to the naive implementation. User would have the same circuit definition regardless:

func (c *Circuit) Define(api frontend.API) error {
    curve, err := weierstrass.NewCurve(api, ...)
    P := curve.NewPoint()
    Q := curve.NewPoint()
    PQ := curve.Add(P, Q) // if api is ECAdder, does `api.ECAdd`, otherwise naively using Add definition.
   ....
}

So for example, in case of range checking -- there is now rangecheck gadget which chooses between three implementations:

• native range checking
• product argument with commitment
• boolean decomposition.

If the builder supports native range checking (implements frontend.Rangechecker), then the range checking gadget doesn't do actually anything and just returns the builder itself: https://github.com/ConsenSys/gnark/pull/472/files#diff-bd247ab3f11c9f1556fb9e811bd72314967c3ea009291edbcf5b692cd7075790R24. But from the user perspective they always use std/rangecheck without having to care what the builder is and how it works.

[ivokub]

Depends on #507 and #521, but otherwise ready.

[ivokub]

FYI @ThomasPiellard, as all dependencies were merged, then rebased on top of develop. Now this PR only contains changes only related to this PR.