Working with byte type and validating custom types

[zyriab]

Hey there,

First off, thanks a lot for this great tool, it's really amazing to work with and as I am replacing Validator in my codebase I just remove so much boilerplate and unwieldy code, it's fantastic :)

I was wondering if you'd consider having the type byte in the library?
My use case here is that I created a TestFunc to validate against a UUID from the package google/uuid and its underlying type is an array of 32 bytes [32]byte.
That being said, I guess we'd also need a Array schema to handle these as I am guessing Slice won't cut it :/

Maybe there's a way for me to work with this without even adding anything to the library? I looked around in the docs but could not find anything.

Thanks and take care

Little update: read about custom coercers, tried to add one to support the google/uuid type but still no success

[zyriab]

It would be nice to be able to convert values on the fly with something like PreTransform, that would go well with the z.Any() type proposition, I guess?
Maybe z.Unknown() would be less footgunny?

[zyriab]

Triposting but I've found a workaround, if it can help someone in the future, here's a simplified example on how to check if a google/uuid is not nil and invalid:

EDIT: Realized that I use strings in the foo struct... so it's not really a workaround :/

package main

import (
	"fmt"

	z "github.com/Oudwins/zog"
	zi "github.com/Oudwins/zog/internals"
	"github.com/google/uuid"
)

type foo struct {
	U  string
	U2 string
}

func uuidIsNotNil() zi.TestFunc {
	return func(val any, _ z.Ctx) bool {
		data := val.(string)
		u, _ := uuid.Parse(data)

		return u != uuid.Nil && u.Variant() != uuid.Invalid
	}
}

func uuidToString() zi.PreTransform {
	return func(data any, ctx z.Ctx) (any, error) {
		return data.(uuid.UUID).String(), nil
	}
}

func main() {
	strSchema := z.String().
		PreTransform(uuidToString()).
		TestFunc(uuidIsNotNil())

	s := z.Struct(z.Schema{
		"u":  strSchema,
		"u2": strSchema,
	})

	mu := map[string]any{
		"u":  uuid.New(),
		"u2": uuid.New(),
	}

	var su foo

	errs := s.Parse(mu, &su)

	fmt.Println(errs) // map[]
}

[Oudwins]

Hey! Thanks for your kind words! Yea, we probably want to support byte and array types, I'll take a note of it.

I can see that its maybe not the most intuitive API to first define that you want a string but then run a preTransform to convert it from a byte array to an actual string. Thats interesting, maybe there should be a different api for this specific use case. I'll think about it. I also see that you are importing internals because the Pre & PostTransform types are not in main Zog name space. So I'll fix that now. Ideally you should only import internals if you are okay with anything you import breaking since that is not intended as user space code. But transforms don't fall into that.

Lastly, would you mind sharing what you tried with the coercers? It should work for this use case. You would have three options:

1. Override the string coercer to check for the []byte type.

// we override the coercer function for float64
conf.Coercers.Float64 = func(data any) (any, error) {
  str, ok := data.(string)
  // identify the case we want to override
  if !ok && strings.Contains(str, ",") {
    return MyCustomFloatCoercer(str)
  }
  // fallback to the original function
  return conf.DefaultCoercers.Float64(data)
}

2. Pass a custom coercer to the string schema

z.String(z.WithCoercer())....

3. Create a custom function for string schema with custom coercer

func UUID() *z.StringSchema {
return &z.StringSchema{
   Coercer: MyCustomCoercer
}
}

[zyriab]

Hey!

Thanks for the heads up about internals, I think you can fix that issue by renaming the directory to internal
(without an s at the end) and Go will not make its contents available when Zog is imported into another project.
As for public parts of your package, a common practice is to put it them a pkg directory, which will keep them available and will give you a tidier project structure where you don't have to mix stuff like your nix flakes, etc with the actual exported code of the library.

You can find more info about the internal package here.

I think it would be a nice thing to have if all the types you use in the exported API were available as well (TestFunc, PreTransform, CoercerFunc, etc) but it's doable without, of course!
It might make sense to keep some stuff in conf, but, imo, it would make for a better DX if everything was available from the base zog import.

Here's the latest state of my WithCoercer experiments, it panics, though because interface {} is *uuid.UUID, not *string.

package main

import (
	"fmt"

	z "github.com/Oudwins/zog"
	"github.com/google/uuid"
)

func main() {
	strSchema := z.String(z.WithCoercer(
		func(original any) (value any, err error) {
			val, _ := original.(uuid.UUID)

			return val.String(), nil
		},
	)).PreTransform(func(data any, ctx z.Ctx) (any, error) {
		return data.(uuid.UUID).String(), nil
	})

	s := z.Struct(z.Schema{
		"u": strSchema.TestFunc(func(val any, ctx z.Ctx) bool {
			data, ok := val.(string)
			if !ok {
				return false
			}

			u, err := uuid.Parse(data)
			if err != nil {
				return false
			}

			return u != uuid.Nil && u.Variant() != uuid.Invalid
		}),
	})

	foo := struct {
		U uuid.UUID
	}{U: uuid.Nil}

	errs := s.Validate(&foo)

	fmt.Println(errs)
}

I'm trying your suggestions but have a few issues, so far:

I mainly experimented with option number 3, as it's cleanest for my particular use case and codebase architercture.
The issue I have is that coercer and setCoercer() are not an exported, so I can't do that.

Doing z.String(z.WithCoercer(myCustomCoercer)) is also not helping because I can't convert my type to string before passing it to .Parse() or .Validate() (I could but the solution I've shared earlier is less of a hassle for my project).

Here's what I tried:

func uuidCoercer() func(original any) (any, error) {
	return func(original any) (any, error) {
		u := original.(uuid.UUID)

		return u.String(), nil
	}
}

func UUID() *z.StringSchema {
    return &z.StringSchema{
        Coercer: uuidCoercer(), // Not possible!
    }
}

Thanks again!

[zyriab]

Update, I also experimented with the first option but I have the same panic about *uuid.UUID not being *string.
Also, configuring like that with conf is not practical with the way my project is architectured and I'd rather not whip out an init() function to do it 😅

package main

import (
	"fmt"

	z "github.com/Oudwins/zog"
	"github.com/Oudwins/zog/conf"
	"github.com/google/uuid"
)

func uuidCoercer(data any) (any, error) {
	u := data.(uuid.UUID)

	return u.String(), nil
}

func main() {
	conf.Coercers.String = func(data any) (any, error) {
		switch data.(type) {
		case string:
			return conf.DefaultCoercers.String(data)
		case uuid.UUID:
			return uuidCoercer(data)
		default:
			return "", fmt.Errorf("Unsupported type for string coercion: %T", data)
		}
	}

	schema := z.Struct(z.Schema{
		"u": z.String().Required(),
	})

	foo := struct {
		U uuid.UUID
	}{U: uuid.Nil}

	errs := schema.Validate(&foo)

	fmt.Println(errs)
}

[Oudwins]

I think you can fix that issue by renaming the directory to internal (without an s at the end) and Go will not make its contents available when Zog is imported into another project.

Yes I know about the internal convention. I have explicitly chosen against using it because I want you to be able to import them at your own risk. I'll probably add some docs somewhere about it if its not already there.

As for public parts of your package, a common practice is to put it them a pkg directory, which will keep them available and will give you a tidier project structure where you don't have to mix stuff like your nix flakes, etc with the actual exported code of the library.

Yes I am aware. But this mostly makes sense for apps. Because moving all the code inside as /pkg folder would then require you to add that to the import path as far as I am aware which is something I wanted to avoid. Once why its all top level.

[Oudwins]

Yes I agree on everything needing to be available from the base import.

Oops. My bad on suggestion option 3. Yes those are not exported. You would have to do this:

func UUID() *z.StringSchema {
return z.String(z.WithCoercer(myCoercer))
}

I'm not sure why this is not working for you. Because it is working as expected with a [32]byte as per this test:

func TestWithCoercer(t *testing.T) {
	schema := String(WithCoercer(func(val any) (any, error) {
		switch v := val.(type) {
		case [32]byte:
			// Convert bytes to a proper string representation
			result := ""
			for _, b := range v {
				if b == 0 {
					break // Stop at null byte
				}
				result += fmt.Sprintf("%02x", b)
			}
			return result, nil
		default:
			// Fall back to default string coercer for other types
			return conf.DefaultCoercers.String(val)
		}
	}))
	var out string
	err := schema.Parse([32]byte{1, 2, 3}, &out)
	assert.Empty(t, err)
	assert.Equal(t, "010203", out)

	type S struct {
		Bytes string
	}

	schema2 := Struct(Schema{
		"bytes": schema,
	})

	var out2 S
	schema2.Parse(map[string]any{
		"bytes": [32]byte{1, 2, 3},
	}, &out2)
	assert.Empty(t, err)
	assert.Equal(t, "010203", out2.Bytes)
}

[Oudwins]

Ahh, okay I just realized the issue that you have. The issue that you have is actually this #73

	foo := struct {
		U uuid.UUID
	}{U: uuid.Nil}

Since Zog doesn't support user defined types at the moment the panic is not related to the coercer but to you telling Zog that U is a string when it is not, its a uuid.UUID

[Oudwins]

I guess this is cool because its another example of a custom type that it would be nice to support and its the thing I'm working on at the moment.

[zyriab]

Awesome man! I really appreciate you taking the time to help, I know OSS work is hard and not very rewarding, thanks again :)

[Oudwins]

@zyriab No worries. Have a look at the APIs described here -> #132

Those describe where my head is at in terms of solving this problem

[Oudwins]

@zyriab Heads up that z.Custom shipped a few weeks ago which makes it possible to deal with custom types like this. It doesn't yet do coercion though so if you are using it with parse it won't work

[zyriab]

@zyriab Heads up that z.Custom shipped a few weeks ago which makes it possible to deal with custom types like this. It doesn't yet do coercion though so if you are using it with parse it won't work

Thanks for keeping me in the loop and for your work 😁👌