Draft
Conversation
Move a large set of repository files (source, docs, tests, CI/workflow and scripts) into an 'old/' directory to archive them, and delete .vscode/settings.json and scripts/docs/requirements.txt. This cleans up legacy configuration and documentation build requirements from the main tree.
Add full project scaffold: FastAPI backend (src/spotdl) with settings, DB models, repositories, Alembic migrations and initial migration, Dockerfile, matching engine and scoring logic, API v1 health endpoints and unit/integration tests. Add frontend Vite + TypeScript app with pnpm config and routes, a CLI package, docker-compose files, and project metadata (pyproject.toml). Add GitHub Actions workflows for testing and deployment, coverage/lint/type-check configs, and basic README — establishes foundation for further development.
Add a SourceProvider base class and multiple source provider implementations to fetch song metadata from various platforms. New modules added under backend/src/spotdl/providers/sources: apple_music.py, bandcamp.py, base.py, deezer.py, resolver.py, soundcloud.py, spotify.py, tidal.py, ytmusic.py. Update package exports in providers/__init__.py and providers/sources/__init__.py to expose providers, resolver utilities, and common exceptions. Update backend/pyproject.toml to add runtime dependencies required by these providers (spotipy, ytmusicapi, beautifulsoup4, lxml). These changes enable metadata resolution via public APIs and web scraping for additional platforms.
Introduce new API v1 endpoints (songs, matches, votes) and wire them into the router. Add SongService, MatchService and VoteService implementations with global getters (get_song_service, get_match_service) to encapsulate resolution, matching and voting logic. Add target provider base and implementations (Bandcamp, Piped, SoundCloud, YouTube, YouTube Music) and update providers exports. API handlers provide resolve/search/find/submit and basic vote endpoints (with placeholders for auth and persistence) and endpoints to list supported platforms. Overall this change scaffolds core service layers and target provider plumbing for matching and voting features.
Introduce portable DB column types (GUID, JSONType) and migrate models to use them for PostgreSQL/SQLite compatibility. Update Song and Match/Vote/User models to use GUID and JSONType, and adjust Result dataclass to new field names (platform/platform_id, artist/artists, album_name, etc.) with backward-compatible parsing. Integrate database into APIs: detailed health now checks DB connectivity, app lifespan initializes and closes DB, and matches/votes endpoints use DB sessions, repositories, and a test auth dependency. Implement platform detection, match creation/checking, vote upsert/delete, vote summaries, and Wilson score confidence calculation. Adjust core logic to new Result fields (matching engine and scoring fixes). Tweak provider URL resolver regex for Apple Music IDs. Add comprehensive tests and test fixtures (in-memory SQLite or TEST_DATABASE_URL) including integration tests for matches, songs, votes and unit tests for providers/services; provide test client and DB session fixtures.
Add extensive integration tests for the database layer and repositories (users, songs, matches, votes) including create/get/update/delete, pagination, vote counts, rollback behavior and session checks. Extend unit tests for matching and scoring logic, provider resolver and source base behaviors, and add API error-handling tests for songs endpoints. Update pyproject.toml coverage config to omit provider implementations from coverage (they rely on external APIs/VCR cassettes) and clarify the coverage note. These tests improve coverage and robustness around DB operations, matching logic, URL resolution edge cases and error handling.
Track VCR cassettes and enable VCR-based tests: remove cassettes from .gitignore, add many backend/tests/cassettes YAML files, and add integration/vcr test modules. Update dev dependencies and config in backend/pyproject.toml to use pytest-vcr and vcrpy and add [tool.vcr] defaults for cassette_dir, record_mode and matching. Refactor VoteService to call renamed repository methods (get_by_id, get_user_vote, get_user_votes) to match repository API changes. Add MatchRepository.get_top_voted to retrieve matches ordered by vote score. These changes enable deterministic HTTP replay in tests and add a helper for obtaining top-voted matches.
Adapt MatchService to the updated get_best_matches behavior: the function now returns a full ordered list, so callers slice the result to apply limits. Updated calls in backend/src/spotdl/core/services/match.py accordingly. Added numerous VCR cassettes and vcr tests for provider integrations and updated unit/integration tests to match the refactor.
Introduce a metadata subsystem: add an abstract MetadataProvider and MetadataResult dataclass, plus concrete MusicBrainzProvider and DiscogsProvider implementations to enrich Song data (ISRC/name lookups, search, parsing, cover art). Add comprehensive unit tests for base, MusicBrainz, and Discogs parsing/lookup behaviors. Update pyproject.toml to include musicbrainzngs and python3-discogs-client dependencies and allow mypy to ignore their imports. These changes enable fetching free, no-auth metadata from MusicBrainz and Discogs and provide test coverage for provider logic.
Introduce a new MetadataService (MusicBrainz + Discogs providers) to enrich Song objects with ISRC, genres, album/year and to perform ISRC lookups and searches. Export the new service and errors from services.__init__.py. Integrate optional metadata enrichment into SongService: add constructor flags (enable_metadata_enrichment, enable_musicbrainz, enable_discogs, discogs_user_token), wire a MetadataService instance, add enrich_* helper methods (enrich_song, enrich_songs, lookup_isrc), expose metadata_providers, and allow resolve_url/get_track to accept an optional enrich argument. Add get_metadata_service singleton factory and comprehensive unit tests for the metadata service (mocking external providers).
Introduce a new spotdl_cli package implementing an interactive Textual TUI and core download/matching plumbing. Adds: README, Textual app (app.py, app.tcss, screens), config (pydantic settings), API client, downloader & DownloadManager, queue, offline/matching/core entry points, providers scaffolding, types and many tests. Update cli pyproject.toml: new dependencies (textual, yt-dlp, mutagen, httpx, pydantic, platformdirs, ytmusicapi, musicbrainzngs, etc.), script entrypoint, coverage and mypy overrides. Change __main__ to use the new run() entry and wire up global client/settings helpers. Purpose: provide an offline-first/hybrid CLI for searching, matching and downloading audio with metadata embedding and concurrent downloads, plus test coverage and development tooling configuration.
Introduce a shared spotdl-core package and refactor the CLI to consume it. The CLI pyproject now depends on spotdl-core (editable path), removes duplicated matching/provider dependencies, and updates import paths/types to use spotdl_core. OfflineMatcher was rewritten to use spotdl_core providers and Result/Song types (Spotify support gated by configured credentials), URL detection/re-export utilities adjusted, and provider initialization/search logic consolidated. Add onboarding screen and tests, expose onboarding flow in app startup, and add Spotify credential settings. Large TUI theme overhaul (app.tcss) and various settings/import cleanups. Several local matching/provider modules were removed from the CLI in favor of the shared core.
Add a full frontend app and static server plus comprehensive CI/CD and compose updates. Introduces a frontend Dockerfile, nginx configs (frontend/nginx.conf, nginx/Dockerfile, docker-entrypoint), Vitest tests for UI components and stores, vite/ts env and test setup, and updated frontend package.json and tooling. docker-compose.yml and docker-compose.prod.yml updated to include frontend and nginx services, improved env vars, healthchecks, restart policies and networks. GitHub Actions workflows (deploy.yml, test.yml) revamped: test workflow is callable/concurrent and expanded with more test jobs (frontend, core, docker-build), deploy workflow now runs tests, builds multi-arch backend/frontend/nginx images, creates releases with image instructions and adds staging/production deployment steps. Misc: added supporting files for frontend (pnpm lock, route changes, stores, api stubs) and CI improvements (buildx, metadata tags, registry/login).
Introduce a project Makefile with common development, testing, linting, Docker, migration and cleanup commands to standardize developer workflows. Add .env.example as a template for required and optional environment variables. Add uv.lock lockfiles for backend, cli and core to pin Python dependencies, and update .gitignore to ignore the .uv cache directory and nginx/certs. These changes make onboarding and reproducible builds/tests easier for contributors.
Initial project scaffolding: add MIT LICENSE and top-level README; add comprehensive documentation (architecture, contributing, getting-started guides, and CLI/web/self-hosting guides). Add frontend testing and E2E suites (unit tests, Playwright config, Vitest config update) and frontend README. Minor backend config tweak: increase coverage fail_under from 70 to 80 in backend/pyproject.toml. These changes provide project onboarding, contributor guidelines, and test infrastructure.
Large UI refactor: restyled and extended core UI primitives, added new loaders and platform badges, and updated tests. Key changes: - Badge: added premium variant, size (sm/md), pulse indicator; added PlatformBadge for music platforms; updated default/variant styles and tests. - Button: replaced flat colors with gradients/shadows, adjusted sizing and focus/press behavior, tweaked loading spinner markup and tests. - Card: new glass variant, hover behavior, spacing/typography tweaks, and tests updated for new classes. - Input/Select: visual refresh (rounded-xl, improved focus states), enhanced error UI with icon and opacity-based error borders; Select now includes custom arrow and option styles. - Spinner/Loading: extracted WaveformLoader, EqualizerLoader, Skeleton; Loading supports variant prop (waveform|spinner|equalizer); Spinner styling adjusted; tests added/updated. - Exports: ui index re-export updated to include new components and types. - Root layout & styles: header redesigned with Logo & NavLink components, updated auth area, Badge usage updated, main layout tweaks; extensive index.css additions for typography, colors, animations, utility classes and loaders. Tests updated across components to reflect class/name changes and to cover new components/variants.
Introduce authentication endpoints and supporting security utilities, plus self-hosting Docker tooling and test updates. Changes include: - Add API: auth endpoints (register, login, refresh, me, logout) with request/response models and dependencies. - New core/security.py: JWT creation/validation, password hashing and verification (bcrypt), TokenPayload model and helpers. - Wire auth router into API v1 and use auth dependencies in votes (optional/current user helpers). - Add email-validator to backend dependencies and update lockfile accordingly. - Update backend config to accept CORS settings via comma-separated env vars and parse them into lists. - Tests: add auth tests and fixtures (auth token, authenticated_client); update votes integration tests to use authenticated client. - Docker: add unified root Dockerfile to build frontend and backend (serve frontend static from backend), add cli Dockerfile, improve backend Dockerfile healthcheck and static dir creation. - docker-compose: add self-host compose for single-container SQLite self-hosting, switch healthchecks to curl, and introduce named volume for API data in compose files. - Frontend e2e: update routes and expectations to use /api/v1/auth and adjust selectors/response shapes for mocked endpoints. These changes add full JWT-based auth, enable self-hosted single-container deployments (including static frontend serving), and update tests and CI/dev compose ergonomics.
Introduce a full user settings feature across backend and frontend: add Alembic migration, SQLAlchemy UserSettings model, and a UserSettingsRepository for CRUD and reset operations. Expose REST endpoints (get/put/delete/import/export) under /api/v1/settings and wire the router into the API. Add frontend API client (hooks, converters, request functions) and a comprehensive settings page with credential handling, sync/import/export, thresholds, timeouts and advanced options. Also add development Dockerfiles and docker-compose.dev for hot-reload dev workflow, tighten Spotify credential validation in the provider, and improve SongService to surface provider errors and read Spotify creds from app settings. Minor adjustments: include UserSettings in models __all__, change default search platform to youtube_music, and update e2e expectations for the settings page.
Introduce a full download subsystem and multi-platform search flow. Backend: add DownloadManager powered by yt-dlp and mutagen (core/services/download), new download API endpoints (start/status/list/file/cancel) and a WebSocket endpoint for real-time progress updates (api/v1/download.py, api/v1/websocket.py). Wire websocket router into the main app and v1 router. Extend songs API with a search-all endpoint that runs parallel platform searches, returns per-platform results, and persists cross-platform matches to the DB (songs.py). Add yt-dlp and mutagen to pyproject dependencies. Frontend: add download API client and React Query hooks (frontend/src/api/download.ts) and expose them via the top-level api index. Extend songs client with searchAllPlatforms and corresponding hooks. Update home route to use the new multi-platform search and queue integration. Update CSS with new font imports and a large set of animation/utility styles. This change enables audio downloads with metadata embedding, realtime progress tracking, and aggregated multi-platform search with automatic match saving.
Introduce first-class entity support across backend and frontend. Backend: add Pydantic models, helpers and new /entities and /search/entities endpoints (artist/album/playlist/track) in songs.py, plus a URL builder and song->response mapper; also install ffmpeg in the production Docker image for audio processing. Frontend: add an entities API client with react-query hooks, new UI components (EntityHeader, TrackList, search result card and filters), route scaffolding for album/artist/playlist/song, and wire exports into the api index. These changes enable searching and viewing entities and their tracklists from the UI.
Update match data model and adapt UI to new API shape.
- Types: replace legacy fields (id, upvotes/downvotes, match_score, etc.) with Match.result, score, confidence and new MatchResult shape; FindMatchesResponse now returns source_url and total.
- Matching UI: use source_url instead of Song object, import PlatformBadge, sort matches by score, show score/confidence badges, remove voting functionality and vote-related UI/hooks, and adjust styling/keys to use `${platform}-${url}` composite keys to avoid relying on id.
- Song & Home pages: switch match_score usages to score, show confidence indicators, and update card styles and badges to match the new model.
These changes align the frontend with the updated backend response and simplify vote removal while improving match presentation.
Add Artist/Album/Playlist models and related platform link & playlist_tracks tables via Alembic migration. Implement repositories and an EntityPersistenceService to normalize, link and persist cross-platform entities and songs. Add new API endpoints: /api/v1/entities (internal UUID lookups and platform-based redirects) and /search (universal URL/text search that persists results). Wire service exports into core.services and adjust songs search to surface artist entities. Update frontend routes, components and API client to support entity pages and search integration.
Backend: repositories for albums, artists and playlists now check for an existing platform link and return it instead of creating duplicates. Album entity response uses the actual number of songs when stored total_tracks is 0 (falls back to len(songs)). Frontend: queue store imports findMatches and will attempt to find a YouTube/Youtube Music match when starting a download if no match is present, storing the found match and using its target_url for the download; errors update item status and stop processing. Also added onRehydrateStorage to auto-process the queue after store rehydration.
Revamp the matching page UI and add end-to-end support for resolving source metadata and queuing downloads. Key changes: - Import Link, Spinner, features, useResolveSongMutation and useQueueStore; use Song type. - Add formatDuration helper and replace sourceUrl with sourceSong (full metadata). - Resolve source song before finding matches (resolveSongMutation) and show a Source Song card with cover, artists, album, duration and match count. - Merge loading states (resolve + find) and use Spinner with contextual messages; improve error card UI. - Redesign search form and header, include authentication CTA linking to login. - Enhance matches list and MatchCard UI: cover art, score badge color, verified/user badges, views, voting buttons (if authenticated), improved rank/score visuals and layout. - Add download flow: canDownload flag (features.canDownload && sourceSong) and Download button that calls addItem on the queue store. - Minor styling and markup adjustments throughout for improved UX and accessibility. These changes improve match accuracy presentation, allow queuing downloads tied to the resolved source track, and provide clearer loading/error states.
Introduce admin API, lyrics and metadata reporting features and corresponding DB migrations; reorganize Makefile and migration environment. Details: - Add admin API endpoints for managing users, matches and system stats; wire admin, lyrics and reports routers into API. - Add new Alembic migrations: 004 (lyrics, metadata_reports, audio features & metadata columns) and 005 (admin fields for matches/users). - Update existing migrations to use generic JSON types and SQLite-compatible batch_alter_table for adding/dropping columns and FKs. - Replace async Alembic engine with synchronous engine in env.py and convert async DB URLs to sync equivalents (aiosqlite/asyncpg -> sync drivers); enable batch mode for SQLite. - Add lyrics services and multiple lyrics provider modules; add lyrics/metadata services and repository changes. - Add new DB models for lyrics and metadata_report and extend models with admin/metadata fields. - Overhaul Makefile: simplify targets, add docker/migration helpers (migrate, migrate-create, migrate-history, migrate-downgrade/current), test/lint commands, and clean-db. - Frontend: add many UI components, routes, API clients and tests to support admin, lyrics and reports features. This commit prepares the backend and frontend to support lyrics storage, metadata reporting workflows and basic admin moderation tooling, and improves migration compatibility across DB backends.
Introduce a new ConnectionStatus component (compact + detailed) with a mock status hook and UI elements; export it from the ui index and show the compact status on both home variants. Heavily refactor the matching page: add ScoreIndicator, WaveformDecoration, MatchCard and SourceSongPanel components, modernize the hero/search UI, improve loading/error/empty states, and navigate to /queue after adding a matched item. Minor UI tweaks: adjust CommandPalette backdrop styling, change Artist page avatar size and section heading ("Top Tracks" -> "All Tracks").
Introduce multi-source metadata support and provider preferences across backend and frontend. Backend: - Add Alembic migrations: data completeness fields for songs/lyrics, provider preferences on user_settings, and metadata_snapshots table. - New models: MetadataSnapshot; extend Song (field_sources, musicbrainz_id, discogs_id, enriched_at) and Lyrics (content_hash, quality_score, is_verified, line_count). Export MetadataSnapshot in models __init__. - New core/providers_config module with provider lists, default preferences and validation utilities. - API changes: register providers router; new providers endpoints (list providers, defaults); metadata endpoints to list metadata sources, refresh and enrich entities (songs/albums/artists/playlists); service health endpoint for external services; update settings API to accept/validate provider preferences and include provider fields in user settings responses; enhance entities responses with enrichment and extended metadata fields. - Add utility improvements for building platform URLs and use actual DB counts for album/artist responses. Frontend: - Add provider-related API client and UI components (providers list, metadata source selector, refresh metadata button, sortable provider list), dev panel and context, plus adjustments to routes, stores and types to surface provider/metadata features. Why: - Allow capturing and comparing metadata from multiple providers, enable enrichment and refresh flows, track provenance of metadata fields, provide user-configurable provider preferences and surface provider/health information in the UI.
Introduce a 4-hour localStorage cooldown for metadata refreshes in RefreshMetadataButton (entity-scoped via entityId) with admin bypass, remaining-time display, and simplified refresh+enrich flow; add new variants and improved disabled/loading/feedback states. Record refresh timestamps and update cooldown timer via useEffect/useCallback. Propagate entityId to album, artist and playlist pages. Large UI refactor for ArtistPage: increase bio char limit and restyle expandable bio, redesign related artists carousel, origin and stat displays, add StatCard, AlbumCard and TrackRow components, improve discography/tabs layout and animations, and enhance loading/error/empty states and accessibility touches. Minor imports and styling tweaks across affected files.
Introduce per-entity refresh cooldowns and related UX improvements. Backend: - Add Alembic migration to create refresh_cooldowns table. - Add RefreshCooldown SQLAlchemy model and export it from models.__init__. - Add RefreshCooldownRepository to record and check cooldowns (4-hour default) and helper functions to check/record cooldowns in API handlers. - Integrate cooldown checks into entity refresh endpoints (song, album, artist, playlist). Admin users bypass cooldowns. Include optional current_user dependency and include created_at in UserResponse. Frontend: - Add Account page route and register it in the generated route tree. - Update Sidebar to show user menu (avatar, account, logout) and Sign In fallback; add logout handling. - Enhance RefreshMetadataButton to handle server 429 responses: parse Retry-After / message, persist cooldown end in localStorage, show cooldown feedback and icon, and clear local cache on success. - Update artist route to prefetch artist data (SSR-friendly loader), add loading/error states and some query-client usage. - Minor fixes: defensively render created_at in admin users list and small type additions. Overall: Enforce a 4-hour refresh rate for regular users while preserving admin privileges, and add UI flows for account management and clearer cooldown feedback.
Import RefreshCooldownRepository in spotdl.db.repositories.__init__ and add it to __all__ so the new repository is exposed for package-level imports and available to other modules.
Rewrite Alembic migration to support both PostgreSQL and SQLite: add helpers (_col_exists, _pg_constraint_exists), split logic into _upgrade_postgresql and _upgrade_sqlite, and use raw SQL on Postgres to migrate lyrics (song_id → entity_id), add upvotes/downvotes/status columns, recreate indexes/constraints, and drop legacy tables safely. Remove references to legacy Match/Vote models across code: drop Vote export from models/__init__, remove Match/Vote relationships from user and vote models, and remove VoteRepository export from repositories/__init__.py to reflect the legacy models being removed.
Move JSONType from song.py to base.py so that importing live models (UserSettings, etc.) no longer drags in the legacy Song mapper. The Song mapper references Artist, Album and Match via string relationships; because those classes are never imported in the new architecture, configure_mappers() would raise InvalidRequestError the moment any ORM query ran during startup. Also remove the two snapshot-saving methods from MetadataService (enrich_song_with_snapshots / fetch_all_snapshots) and their module-level MetadataSnapshotRepository import, since the metadata_snapshots table is dropped in migration 016. Neither method was called from anywhere. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deduplicate relation targets in entity endpoints and add audio match discovery flow. - frontend/src/api/entities.ts: Prevent duplicate relation targets for albums, artists, and playlists by tracking seen IDs when mapping relations to songs. - frontend/src/api/matches.ts: Add discoverMatchesForEntity to call the relations:discover endpoint and a useDiscoverMatchesMutation hook that invalidates match queries on success. - frontend/src/routes/song/$id.tsx: Integrate matches into the song page — import hooks, fetch matches, add a Discover button and an Audio Matches card, render MatchRow entries with status badges, platform labels, formatted durations, score badge, and vote controls (integrates with existing vote hook). These changes enable discovering external platform matches for a song, present them in the UI, and ensure relation lists do not contain duplicate entries.
Add an Alembic migration to rename track entity keys from the old format (track:{name}:{artist}:{duration_bucket}:{isrc}) to an ISRC-first scheme: track:{isrc} when available, otherwise track:{name}:{artist}:{duration_bucket}. The migration renames keys, merges collisions by keeping the lower UUID as the survivor, re-targets snapshots, provenance and relations, and deletes duplicate rows. Update UnifiedEntityService to generate the new keys (use ISRC if present; otherwise fall back to name/artist/duration bucket) and deduplicate relation creation by (from_key, to_key, relation_type). The downgrade restores the old key format for surviving rows but cannot recover merged data.
Backend: record a discovered_by id and use it when creating relations; generate and upsert artist/album bundles for tracks discovered via albums, artists, playlists and direct track lookups, and create performed/contains relations accordingly. Also add artist->album relation when building album track relations. Frontend: treat entity ref IDs as authoritative (use nullish coalescing instead of falling back to canonical IDs), make AlbumSummary.id nullable, use null-aware keys for album grouping, and render albums without an id as non-clickable cards. On the song page, show album name as a non-link badge if album_id is missing. These changes ensure correct provenance attribution and safer handling of missing entity IDs.
Use a stable key for album cards (album.id ?? album.name) and guard click handlers so onAlbumClick is only called when album.id is present. Disable the button and add disabled styling for albums without an id, and render a non-interactive div fallback to avoid accidental interactions and improve accessibility/visual feedback for items missing IDs.
Add comprehensive admin functionality for moderating and managing entity relations: list matches, update match status, purge unverified matches, reset entity data, import matches/URLs, and export matches/users/statistics. Enhance admin user endpoints to include contribution counts and add DTOs for match/user/statistics exports. Remove legacy per-entity DB models/repositories (song, album, artist, playlist, match, metadata_snapshot, vote, and related repos) in favor of the unified Entity/EntityRelation model usage. Also update metadata_resolver type hints to Protocols and apply corresponding frontend updates to the admin API, routes and types; remove obsolete frontend songs API.
|
hey, will V5 still require Spotify API credentials? |
Adjust tests to match model and API changes: replace Match/Vote usage with unified Entity and EntityRelation fixtures (added test_source_entity/test_target_entity and updated test_match) in conftest.py; rename download manager references to download service (get_download_manager -> get_download_service) and update mock specs to DownloadService across download and websocket tests; update lyrics service tests to use entity_id and corresponding method names/queries (get_lyrics_for_entity, fetch_lyrics entity_id, DB column checks). Also remove obsolete vote-related integration and unit test files that are no longer applicable.
Clean up imports, formatting and minor refactors across backend modules and tests. Key changes: - Alembic: join SQL text into single lines and remove stray blank line. - Migrations: reformat helper functions/signatures, rename loop variable to avoid unused-binding, inline SQL text strings. - API (admin): use TYPE_CHECKING for AsyncSession imports, reformat select queries for readability, adjust function signatures and small whitespace fixes. - Metadata resolver: minor parenthesis/formatting fixes around merging logic and year assignment. - Tests (conftest + integration/unit): reorganize imports, add TYPE_CHECKING hints, update fixture type hints, replace nested with-contexts with grouped context managers, replace timezone.utc with UTC constant, simplify comprehensions and other small style fixes. These are primarily style/clarity improvements and do not change core behavior.
Member
Author
no, all requests will go to our server (ofc you will be able to self host your own) and our key (maybe multiple keys) will be used. |
Oh that's so neat, just started my day and this relieved me so much. Thanks 😭, when does V5 come out then?? |
Member
Author
no eta for now. i'm too busy with work and uni. hopefully before end of april |
Member
Author
|
Here's the new web-app. It can be run in two configurations:
|
Member
Author
|
cli will be rebuilt with textual with proper interactivity trying to mimic the website in layouts/navigation for the tui, for cli i will try to keep the usage simple without sacrificing any options. |
Introduce a new entity_canonicals table/model and slim down entities to id + entity_type. Add a clean-slate Alembic migration to drop old entity data, create entity_canonicals, and make entity_snapshots.provider_entity_id NOT NULL with a global unique dedup index. Refactor DB models to include EntityCanonical and relationships, and update EntitySnapshot uniqueness and provider_entity_id typing. Update MergeEngine and UnifiedEntityService to produce and use EntityCanonical records (merged canonical data), change snapshot upsert/merge logic to be snapshot-first with global dedup by (provider_id, provider_entity_id), add ISRC-based cross-provider merging, URL hashing for open_graph provider ids, and snapshot change detection. Update API handlers to load canonical data via the new relationship and adjust lyrics, admin, and unified entities endpoints accordingly. Adjusted tests and conftest imports to match the new model layout.
Update Alembic migration to improve portability and correctness: add GUID import and use GUID() for entity_canonicals.entity_id; replace raw SQL deletes with sa.text; introduce _is_sqlite() helper and branch logic to use batch_alter_table for SQLite or direct ops for other DBs when dropping/adding columns, indexes and constraints; adjust creation/drop of the global snapshot dedup constraint and provider_entity_id nullability accordingly. These changes ensure the migration runs safely across SQLite and other databases and fixes schema/constraint handling in both upgrade and downgrade paths.
Add _refresh_entity_map to re-resolve Entity references after ISRC-based merges can remove duplicate entities and move snapshots to survivor entities. Call this refresh after upserting entities so relations point to live entities. Make relation creation resilient: _create_or_update_relation now returns None for no-op cases and skips creating self-referencing relations (which can occur after merges). On FK/merge races it logs a warning and returns None instead of raising. Update callers to avoid appending None relations and raise a clear error when a manual relation creation results in a self-relation. Overall this prevents invalid self-referential relations and stale references caused by concurrent ISRC merges, and ensures relation lists only include valid relations.
Switch lyrics APIs and types from song_id/song endpoints to entity_id/entity endpoints and update related code (entities.ts, lyrics.ts, lyrics-display). Add AdminMatch type and use it in admin API functions and list responses; update updateMatchStatus return type. Rename resetDatabase response fields (songs/matches -> entities/relations) and adapt admin import UI to display entities/relations counts. Add refresh_token handling to auth flow (send on refresh, persist/remove in localStorage). Minor fixes: Apple Music URL pattern and admin matches UI now shows discovered_by.
Reduce CI coverage threshold from 80% to 60% and apply broad linting/formatting and minor cleanup across the codebase. Updates include ruff configuration changes (additional ignored rules), stringified type casts and small typing adjustments, whitespace/line-wrapping and expression simplifications, and added minor noqa annotations where appropriate. Many API, core service, provider, DB and test files were reformatted to comply with lint rules; an ESLint config was also added for the frontend. These are primarily non-functional style changes and should not alter runtime behavior.
Introduce local backend integration and adapt the CLI to use the new backend API and lifecycle. - Add spotdl-backend as a dependency and include it in editable UV sources; include alembic files in backend wheel. - Add BackendManager (cli/src/.../core/backend.py) and export get_backend_manager from core.__init__. - Start/stop local backend in the TUI (app.py) and manage backend lifecycle for CLI commands via a shared _run wrapper and global app callback (--backend). - Replace DownloadQueue flow with concurrent asyncio tasks using a semaphore, improved match resolution (try online via API then fallback to offline matcher), and per-item DownloadItem construction for downloads. - Refactor many asyncio.run calls to use the centralized _run runner to avoid event loop recreation and to ensure backend is running during operations. - Add backend_mode setting to Settings and persist it; update pyproject/uv config accordingly. - Rewrite APIClient to target the backend's new entity/relation endpoints (entities/discover, relations, etc.), create clients via BackendManager (ASGI/http transport), and add helpers: _song_from_entity, _match_entry_from_relation, _build_platform_url, and _flatten_entity. Update caching keys and request shapes accordingly. - Locate alembic.ini from installed package data in backend DB migration helper. Tests and small CLI modules updated to match the new client/flow.
|
Does the rewrite require Spotify Premium? I do not have premium, but do have personal spotify client credentials. Entering a spotify url returns the following message I have also tried using what I presume are the spotdl publicly shared credentials as I found them in the v4 config.json. Searching uses Apple Music as the metadata provider, but downloads do not initiate when pressing on the download button. I am using the Using the public and personal credentials to retrieve a temp auth token works with spotdl v4 Is the rewrite using temp auth tokens and refreshing them? |
Replace many direct commits with session.flush() to better integrate transaction boundaries and avoid orphaned state; add DB-level checks (blacklist, token rotation) and stronger validation for auth flows. Batch-load related data to eliminate N+1 queries (user contribution counts, relation targets, entity canonical rows) and optimize relation/vote aggregation with atomic SQL updates. Normalize and use ISRCs for cross-provider merging and improve merge/upsert logic (preload relations, avoid duplicate conflicts, return created flags). Harden Lyrics repository for concurrent upserts (savepoints + IntegrityError handling), add atomic lyrics vote update, and prefer best-source selection query. Tune DB engine pool settings for non-SQLite backends and add deterministic ordering in BaseRepository.get_all. Optimize user repository checks (exists() queries) and atomic reputation updates. Make numerous API fixes: validate sort_by, infer media types for downloads, add cooldown checks for refresh, optional/required current user usage, and more robust error handling. Apply UI/TUI improvements (queue screen fixes, notifications, layout CSS tweaks) and add unit tests for queue screen behaviors.
Integrate navigation rail and status bar into the app layout and wire backend state to the UI: import BackendState, attach manager.on_state_change to update StatusBar, update connectivity checks to set status, add action_toggle_nav and on_screen_resume to sync NavRail. Also hide several default key bindings from the help UI. Deliver a comprehensive theme rewrite (app.tcss) — new "Midnight Vinyl" tokens, reorganized sections, refreshed component styles, badges, data table, and many visual refinements. New widget files added (audio_meter, entity_card, match_bar, nav_rail, stat_chip, status_bar) and supporting style integrations for those components.
Replace monolithic backend/src/spotdl/api/v1/admin.py with a package-style admin API. Introduce admin/__init__.py plus modular submodules (deps, schemas, users, matches, stats, data) and move shared helpers (require_admin, contribution counting, relation -> match mapping) into deps. Separate import/export and danger-zone endpoints into data.py, match management into matches.py, and wiring via the admin router. Additionally add several backend core services and utils (discovery, merge_engine, relations, opengraph), update various providers/services/db/tests, and add new frontend settings/song components and API/types to support the expanded admin and UI functionality.
Several fixes and enhancements across discovery, unified entities, providers and types to make entity resolution more robust and avoid incorrect data/merges. Key changes: - Admin: restrict purge_unverified_matches to audio_match relations only. - API: removed /lyrics/search endpoint; added /entities/resolve endpoint to resolve UUIDs, URLs or platform IDs and helper functions to build platform URLs. - Unified entities: batch-resolve relations with aliased Entity to ensure correct from-entity types, avoid N+1 by batching entity loads, and pass entity cache to relation response builder. - Discovery service: construct album URLs from platform+id, include richer bundle payloads (album_artist, disc/date/isrc/genres), fetch target entities in batch, handle snapshot lookups by (provider_id,provider_entity_id), stop platform scanning when limit reached, build relations in the same way as URL discovery and return relations map, and handle SQLAlchemy errors in ISRC merge queries. - Merge engine: ensure merge_version initializes and increments correctly for new and existing canonicals. - Entity unified service: rebuild container->track relations when resolving songlist sources and add _cleanup_stale_relations to remove stale relations. - Types & scoring: Song fields (duration, track/disc/years/counts) are now optional; calc_time_match returns 0 when durations are unknown. - Providers: metadata providers list and provider IDs updated (spotify removed from METADATA_PROVIDER_IDS), provider config updated accordingly; source providers (Apple Music, Deezer, Spotify, Tidal) adjusted to treat missing values as None, extract album/artist ids from nested URLs, filter collected tracks by actual artist involvement, and normalize album/track deduplication by IDs. - Tests: updated assertions and test expectations to reflect behavior changes (metadata sources list, parsing defaults, tidal duration tests, etc.). Overall this commit improves correctness when discovering and merging entities, avoids many edge-case bugs around missing metadata and duplicate snapshots, and adds a user-facing entity resolution endpoint.
Add lazy="selectin" to multiple SQLAlchemy relationship() declarations to enable selectin eager loading and reduce N+1 query issues. Changes touch entity_unified.py (Entity, EntityCanonical, EntitySnapshot, EntityFieldProvenance, EntityRelation, RelationVote), lyrics.py (Lyrics.entity), metadata_report.py (MetadataReport.reporter/reviewer), user.py (User.settings, submitted_reports, reviewed_reports) and user_settings.py (UserSettings.user). No cascade or schema semantics were changed—only the loading strategy was adjusted for performance.
Multiple UI and backend changes: use raw DELETE + expunge in DiscoveryService to avoid ORM cascade issues when removing duplicate entities; add APIClient endpoints for entity discovery and lyrics submission. Persist DownloadQueue to data_dir/queue.json with save/load logic and auto-save on completion. Add Register and Submit Lyrics screens and wire Register into Login screen. Enhance Admin screen with an Import tab to discover/import entities from a URL. Main screen: track recent searches (persisted to recent_searches.json) and show a recent-searches widget. Track screen: switch lyrics view to tabbed multi-source display, add "Fetch All Sources" and "Submit Lyrics" flows, and add a Metadata Sources table showing provider snapshots and details. Various UI wiring and small UX improvements included.
Change _platform_id_to_url to return a tuple (url, inferred_entity_type) instead of a single URL, returning (None, None) when the format is unrecognized. Update callers (resolve_entity) to unpack the inferred type and use it as the effective entity type when no explicit type query param is provided. This ensures types embedded in URIs like "spotify:track:xxx" are respected during resolution.
Add relevance scoring for text queries (_query_relevance) and sort discovered entities by relevance. Sanitize and normalize bundle payloads: filter empty artist strings, provide fallbacks for name/artist, cast track/disc numbers to int or None, and set provider_url for artist bundles using a new _construct_artist_url helper. Adjust provider capability resolution so RESOLVE is only considered for snapshot providers while other capabilities query the registry directly. Minor tweaks: small tiebreaker favoring tracks in relevance and other small payload cleanups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wip