Python UDF

[jordanrfrazier]

Support user defined functions from the python builder.

Current functionality plan is to only support operations between series or arrays, for example,

def add(x: pd.Series, y: pd.Series) -> pd.Series:
     x +  y

The user experience will be:

1. Define a python function
2. Annotate it with the fenl signature
3. Call it within your query

Step 1:

def add(x: pd.Series, y: pd.Series) -> pd.Series:
    return x + y

Step 2:

@fenl_udf("add", "add(x: number, y: number) -> number")
def add(x: pd.Series, y: pd.Series) -> pd.Series:
    return x + y

Step 3:

kd.udf(add, input1, input2)

[jordanrfrazier]

Future work involves supporting element-wise operations, which would allow the user to write

def add(x: int, y: int) -> int:
   x + y

Then a possibly different function (could swap to map and apply)

kd.apply(add, input1, input2)

which operates per-element, rather than per-series. This allows a bit more flexibility on what you can do with each element, but performance will significantly decrease.

[bjchambers]

I also think we were talking about:

1. Not requiring the "name" to be separate (or even in the signature). And possibly changing the annotation to udf:

@kd.udf("(x: number, y: number) -> number")
def add(x: pd.Series, y: pd.Series) -> pd.Series:
    return x + y

2. Allowing the use of just the PyArrow return type, or a function to compute it instead of signatures in the future.

@kd.udf(pa.int64())
def add(x: pd.Series, y: pd.Series) -> pd.Series:
  return x + y

# or

@kd.udf(lambda x_ty, y_ty: pa.int64())
def add(x: pd.Series, y: pd.Series) -> pd.Series:
  return x + y

3. Possibly running a "test" row through the UDF to make sure it supports those types when we create the plan.