Docs: Add Daft into Iceberg documentation

[jaychia]

• Adds installation examples
• Adds code examples for getting up and running with Daft + PyIceberg
• Adds a type conversion matrix between Daft and PyIceberg

[bitsondatadev]

Sorry this is a lot but had a lot of thoughts the. Got stuck.

[nastra]

Given that this mainly targets pyiceberg, I think this doc should live in https://github.com/apache/iceberg-python/

[jaychia]

Given that this mainly targets pyiceberg, I think this doc should live in https://github.com/apache/iceberg-python/

Hello! The Daft query engine integrates with pyiceberg currently as a user API, but actually isn't tied to the PyIceberg project. In fact, we are actively exploring moving over to iceberg-rs as that project matures and we contribute functionality to it, for more native integrations with Daft on our Rust layer.

Daft is a fully featured distributed query engine, and we are actively working on non-PyIceberg specific functionality that is more applicable to the wider Iceberg ecosystem (e.g. partitioned writes, compaction stored procedures, orphan file pruning procedures etc). This is in contrast to pyiceberg-only integrations such as Pandas/Arrow which really just use pyiceberg for retrieving data into Python memory.

[bitsondatadev]

@nastra, I tend to agree with @jaychia on this one. I don't want to split up the documentation any more than necessary. Any compute engine that runs on Iceberg, I want to

I see this eventually looking like Trino's data sources or Kafka Connectors.

We've discussed this a bit before here: #9681

I think this future reorder will include engines based in languages outside of Java.

[nastra]

so it seems the site can't be actually built when serving the docs locally

[jaychia]

I was able to successfully build the site with mkdocs serve!

INFO    -  Documentation built in 0.38 seconds
INFO    -  [12:52:52] Watching paths for changes: 'docs', 'mkdocs.yml'
INFO    -  [12:52:52] Serving on http://127.0.0.1:8000/

Did you have an error message here that I can look at the debug the issue?

[nastra]

I was using https://github.com/apache/iceberg/blob/aff5b39a7dddd22790b6ba47f514860c53e33c00/site/README.md to locally serve the site. @bitsondatadev can you double-check please if the site properly renders for you when running ./dev/serve.sh?

[bitsondatadev]

Yeah, I used to have a "nightly" build in there, but we took it out initially to avoid confusion. I think part of the build can just be to add "local" or something. Currently, the build just grabs the latest semantic version and points latest there, we could just do the same and point /site/docs/docs/local >> /docs and maybe expose another build option to enable that.

[nastra]

@bitsondatadev is this PR here ready to go in?

[bitsondatadev]

@nastra Ship it!