Update DataFusion 48 and Arrow 55.1, plus other dependency updates, csv fix

[jonmmease]

Summary

This PR updates DataFusion and Arrow and other Rust dependencies to their latest compatible versions, consolidates workspace dependencies, and removes unused dependencies.

Also closes #569

Major Updates

DataFusion & Arrow (Breaking Changes Fixed)

• DataFusion: 43.0.0 → 48.0.0
• Arrow: 53.2.0 → 55.1.0
• Fixed all breaking API changes including:
  • Added comprehensive Utf8View support throughout the codebase
  • Updated deprecated functions (array_into_list_array → SingleRowListArrayBuilder)
  • Fixed Expr::Wildcard deprecation

Other Major Version Updates

• petgraph: 0.6.5 → 0.8.2 (no code changes required)
• json-patch: 1.4.0 → 4.0.0 (then removed as unused)
• PyO3: 0.24.0 → 0.25.1
• pyo3-arrow: 0.9.0 → 0.10.1
• pythonize: 0.24.0 → 0.25.0
• sqlparser: 0.54.0 → 0.55.0
• object_store: 0.11.2 → 0.12.2

Dependency Consolidation

Moved common dependencies to workspace level for better version management:

• async-trait, futures, url, reqwest, serde_json
• thiserror, serde, regex, bytes, chrono, chrono-tz
• itertools, lazy_static, log, env_logger, uuid, and more

Removed Unused Dependencies

• json-patch from vegafusion-core (no usage found)
• num-complex from vegafusion-core (no usage found)
• jni from vegafusion-common (feature never enabled, removed associated error handling code)

Other Updates

Updated numerous other dependencies to their latest compatible versions:

• clap: 4.2.1 → 4.5.23
• float-cmp: 0.9.0 → 0.10.0
• lru: 0.11.1 → 0.13.0
• rand: 0.8.5 → 0.9.0
• sysinfo: 0.32.0 → 0.35.0
• Various dev dependencies (rstest: 0.18.2 → 0.24.0, criterion: 0.4.0 → 0.6.0, test-case: 3.1.0 → 3.3.1, etc.)

Misc Fixes

Support Utf8View everywhere string types are supported

[Copilot]

Pull Request Overview

This PR upgrades DataFusion to 48.0.0 and Arrow to 55.1.0, consolidates many Rust dependencies at the workspace level, removes unused crates, and applies API-breaking fixes for updated DataFusion/UDF interfaces.

• Bump major versions for DataFusion, Arrow, and other key libraries; remove unused dependencies
• Add comprehensive Utf8View support alongside Utf8/LargeUtf8 in string operations
• Update UDF signatures, invocation methods, and list-array builders for DataFusion 48 API changes

Reviewed Changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/expression/compiler/builtin_functions/array/length.rs	Extend length to support Utf8View type
src/datafusion/udfs/datetime/{timeunit.rs,make_timestamptz.rs}	Switch to ScalarFunctionArgs and Signature::one_of for UDFs
src/data/util.rs	Replace manual Expr::Wildcard with expr_fn::wildcard()
src/data/tasks.rs	Adjust object-store registration for wasm32 vs native targets
Cargo.toml (various crates and workspace)	Version bumps and move deps to workspace level; remove unused
src/common/{datatypes.rs,table.rs,scalar.rs}	Add Utf8View support and swap to SingleRowListArrayBuilder

Comments suppressed due to low confidence (2)

vegafusion-common/src/datatypes.rs:51

• There are new Utf8View paths in is_string_datatype. Consider adding unit tests to cover Utf8View inputs to ensure consistent behavior with existing UTF-8 types.

pub fn is_string_datatype(dtype: &DataType) -> bool {

vegafusion-runtime/src/data/tasks.rs:801

• The #[cfg] attribute cannot be used inline in an if condition. Replace it with a runtime check using if cfg!(target_arch = "wasm32") { ... } else { ... } or apply #[cfg] to separate code blocks.

                        } else if #[cfg(target_arch = "wasm32")] {

[Copilot]

Avoid unwrap() in production PyO3 code, as it can panic. Use ? to propagate errors or handle the Err case explicitly.

Suggested change

	let name: PyObject = tbl.name.into_pyobject(py).unwrap().into();
	let scope: PyObject = tbl.scope.into_pyobject(py).unwrap().into();
	let name: PyObject = tbl.name.into_pyobject(py)?.into();
	let scope: PyObject = tbl.scope.into_pyobject(py)?.into();