Skip to content

Add pyspark & ibis capture to SDK#876

Closed
skrawcz wants to merge 3 commits intomainfrom
add_pyspark_ibis
Closed

Add pyspark & ibis capture to SDK#876
skrawcz wants to merge 3 commits intomainfrom
add_pyspark_ibis

Conversation

@skrawcz
Copy link
Contributor

@skrawcz skrawcz commented May 2, 2024

adds sdk capture of some basic ibis and pyspark info.

Changes

  • sdk

How I tested this

  • locally

Notes

  • need to create tests

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

skrawcz added 2 commits May 2, 2024 00:03
These are lazily done - so best we can do is get some
schema and maybe a query plan in the case of pyspark.
Because some dependencies might not exist -- because they
are lazily loaded. This guards against that.
@skrawcz skrawcz changed the title Add pyspark ibis Add pyspark & ibis capture to SDK May 2, 2024
@zilto
Copy link
Contributor

zilto commented May 2, 2024

Ibis 9.0 added a .describe() method (equivalent to pd.DataFrame().describe())

@skrawcz
Copy link
Contributor Author

skrawcz commented May 2, 2024 via email

@skrawcz skrawcz marked this pull request as ready for review May 5, 2024 22:46
@skrawcz skrawcz marked this pull request as draft May 5, 2024 22:58
@skrawcz
Copy link
Contributor Author

skrawcz commented May 5, 2024

Just need to add some tests.

@skrawcz
Copy link
Contributor Author

skrawcz commented May 13, 2024

closing in support of #895

@skrawcz skrawcz closed this May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants