Skip to content
This repository was archived by the owner on Mar 3, 2026. It is now read-only.

feat: Configurable retention on PyDict sources#744

Merged
bjchambers merged 6 commits intomainfrom
unretained-pydict
Sep 8, 2023
Merged

feat: Configurable retention on PyDict sources#744
bjchambers merged 6 commits intomainfrom
unretained-pydict

Conversation

@bjchambers
Copy link
Collaborator

This renames the PyList source to PyDict.

This allows the in-memory retention to be disabled for any source, but exposes this for the PyDict source, which is often used with materializations.

This renames the `PyList` source to `PyDict`.

This allows the in-memory retention to be disabled for any source, but
exposes this for the `PyDict` source, which is often used with
materializations.
@cla-bot cla-bot bot added the cla-signed Set when all authors of a PR have signed our CLA label Sep 7, 2023
@github-actions github-actions bot added enhancement New feature or request sparrow labels Sep 7, 2023
@bjchambers bjchambers enabled auto-merge September 7, 2023 23:20
@kerinin
Copy link
Collaborator

kerinin commented Sep 8, 2023

I get an error when I run the script:

(sparrow-py) % ./server.py
Waiting for events...
Traceback (most recent call last):
  File "/Users/ryan.michael/work/kaskada/examples/event-api/./server.py", line 91, in <module>
    loop.run_until_complete(main())
  File "/Users/ryan.michael/.pyenv/versions/3.11.3/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/ryan.michael/work/kaskada/examples/event-api/./server.py", line 67, in main
    async for row in output.run_iter():
  File "/Users/ryan.michael/work/kaskada/python/pysrc/kaskada/_execution.py", line 108, in __anext__
    next_batch = await self._ffi_execution.next_pyarrow_async()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: error in kaskada Rust code
├╴at src/error.rs:54:21
│
├─▶ execute query
│   ╰╴at /Users/ryan.michael/work/kaskada/crates/sparrow-session/src/session.rs:488:28
│
├─▶ internal compute error: failed to join compute threads
│   ╰╴at /Users/ryan.michael/work/kaskada/crates/sparrow-runtime/src/execute/compute_executor.rs:192:22
│
├─▶ internal compute error: no additional context
│   ╰╴at /Users/ryan.michael/work/kaskada/crates/sparrow-runtime/src/execute/operation.rs:345:58
│
├─▶ internal compute error: invalid input
│   ╰╴at /Users/ryan.michael/work/kaskada/crates/sparrow-runtime/src/execute/operation/scan.rs:200:26
│
╰─▶ Unable to create batch from empty data
    ╰╴at /Users/ryan.michael/work/kaskada/crates/sparrow-runtime/src/execute/operation/scan.rs:199:26

@bjchambers bjchambers disabled auto-merge September 8, 2023 15:54
@bjchambers
Copy link
Collaborator Author

Hmm, yeah. Empty batches posed a problem. I think that should be fixed.

@bjchambers bjchambers enabled auto-merge September 8, 2023 18:10
@bjchambers bjchambers added this pull request to the merge queue Sep 8, 2023
Merged via the queue into main with commit d0b0f85 Sep 8, 2023
@bjchambers bjchambers deleted the unretained-pydict branch September 8, 2023 19:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla-signed Set when all authors of a PR have signed our CLA enhancement New feature or request sparrow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants