Consider making Edgedb less powerful but more adoptable

[scottmas]

Let me preface this post by saying that it's easy to waltz in here as fly-by-nighter and shoot off my two cent opinion from my hip. It's you guys, the Edgedb maintainers, who are in the trenches building the post-SQL future. What you guys are doing is amazing and I'm rooting 100% for you guys. And I'm sure you've already had or are having many conversations along the lines of what I'll be saying below. But I care about this project succeeding so much I feel like I have to throw in my two cents. Sorry if it comes across as

But here's the brutual truth from my perspective:

1. EdgeDB is waaaay too hard to get started with. Sure if I blindly copy and paste the exact commands you tell me, I can get up and running. But it took hours for me to gain a somewhat accurate mental model of what was happening. And I kept running into weird plumbing issues (like TLS, and others). There's also some performance issues too in getting started. For instance, it takes a full minute or so for a docker instance of EdgeDB to be ready to accept connections. There's also way too many new concepts in the server administration side of things (what precisely is an "instance"? Is the CLI just a CLI or does it also have a daemon running in the background since it seems so stateful? Etc).
2. Server administration & connection tooling feels really immature. This is related to point 1, but whenever I had issues setting up the server or getting things to talk to each other, processes hung, errors were opaque, and so on.
3. Edgedb cannot be adopted on brown field projects. It's basically technically impossible without a full migration process, as you know.
4. Adopting EdgeDB makes me lose an entire ecosystem of services and tools And this makes it too risky of a tool to adopt, even for green field projects. As mentioned, the devops learning curve is pretty brutal, which I would feel terrible about hoisting on my junior devs. And without a first class story for communicating directly with the underlying postgres instance, there's way too much existing tooling, ETL services, and many other things that I don't want to give up just for the sake of having a better OLTP database.

So this is the brutal truth from my perspective. You guys are pumping out features like champions, and the database is getting better every month. But I still just foresee a long, slow, slog to any sort of general adoption or significant company revenue.

And I wonder if this isn't in some part due to the natural engineer desire to make the most powerful tool possible. I get it. I really do! Amazing engineers like you guys want to make cathedrals of awesomeness. To squeeze out every ounce of possible DX for a given workflow. Make the most delightful and powerful experience possible for devs interacting with data. And given that that maximzation objective, it makes sense why you're charging down the road you are right now.

However, if Edgedb is to have any chance of substantial revenue and significant developer mindshare 10 years from now, I wonder if it wouldn't make the most sense for your maximization objective to change to be "Maximize the amount of Edgeql written in the world". That's it. From a technical perspective, it's far less ambitious, but I think it would make way more sense from a business perspective. And I say this as a tech startup founder who spent 5 years playing in the wrong sandbox only to learn it's infinitely better to have an inferior product in the right sandbox.

Take Prisma as a case study. Look at this npm adoption curve and the approximate inflection point:

What happened right around this time? Prisma removed the Prisma proxy server from their architecture and they lost a lot of power. That's what happened. But it made adoption infinitely easier. And now they're popular enough, and have enough money that the chances of them going out of business are close to non-existent, which also takes a lot of risk off the table.

Imagine a world where Edgedb compatible schemas slowly infiltrate the DB's of the world. Where all the SQL wizards of the world discover this cool new transpiler and get to impress all their SQL nerd friends with their transpiled Edgeql queries (instead of being labeled the crazy heretic recommending a young and new database). Where devs salivate at the prospect of using your hosted option, because finally they'll be able to talk to a db that understands raw Edgeql without transpiling. I just swear the objective should be to maximize Edgeql adoption, not Edgedb adoption.

As an analogy: imagine two maxima on a graph. One local maxima where all the people are currently living and a global maxima miles away with a valley of despair and sadness between the local and global maxima. In this scenario no one is going to ever get to the global maxima. There's too much sadness. However, if you can create a slow and gradual ascent to the global maxima, that becomes a game changer in terms of adoption. Slowly people would migrate over to the global maxima.

So these are my thoughts guys. I would LOVE, LOVE, LOVE Edgedb to take over the world and to be able to adopt it in good conscience on my next project. But it's not there yet, and the product roadmap you seem to be charging down doesn't personally give me much hope that it ever will get to that place where I would seriously consider using it.

[1st1]

@scottmas Thank you for posting this, we really appreciate the thoughts and feedback. As I'm reading the post I'll ask a few questions.

Here's one:

EdgeDB is waaaay too hard to get started with. Sure if I blindly copy and paste the exact commands you tell me, I can get up and running. But it took hours for me to gain a somewhat accurate mental model of what was happening. And I kept running into weird plumbing issues (like TLS, and others). There's also some performance issues too in getting started. For instance, it takes a full minute or so for a docker instance of EdgeDB to be ready to accept connections. There's also way too many new concepts in the server administration side of things (what precisely is an "instance"? Is the CLI just a CLI or does it also have a daemon running in the background since it seems so stateful? Etc).

I assume you're using Docker to run EdgeDB? If so then you're fighting an uphill battle, as our workflows are not optimized for Docker.

[1st1]

Yeah, I'm really curious to hear your answer for this. Because the rest of your post builds on the premise of EdgeDB being hard to install/manage/connect to etc. The cloud part is obviously missing, but I think we simplified the local installs as much as its possible, unless! you're using Docker.

[1st1]

Adopting EdgeDB makes me lose an entire ecosystem of services and tools And this makes it too risky of a tool to adopt, even for green field projects. As mentioned, the devops learning curve is pretty brutal, which I would feel terrible about hoisting on my junior devs. And without a first class story for communicating directly with the underlying postgres instance, there's way too much existing tooling, ETL services, and many other things that I don't want to give up just for the sake of having a better OLTP database.

We are thinking about exposing read-only SQL access for BI tools and analytics. What is the specific other tooling you're looking for?

[scottmas]

Yes, I am using Docker to get started AND I'm trying to use a self managed backend-dsn rather than one managed by Edgedb.

And honestly, I could be premature in some of my points. And there's definitely a clear path forward for you guys to slowly improve most of my points through dev iteration and increased documentation.

However, the bigger point I'm making really has nothing to do with any of the specific issues I've encountered. It's really your positioning in the marketplace. I know you really don't want to be an ORM. But basically I could summarize my entire above post into two points:

1. If you guys spent a lot of time and energy on a pure ORM, would devs be able to get 90% of the benefits of EdgeDB? I think it likely. If you guys are an ORM there's literally NO additional backend tooling I'll have to do. Backend tooling, hosting, and deployment is on me and I can just use whatever I'm most familiar and comfortable with. The 4 issues I highlight above literally disappear once Edgedb becomes a pure ORM. There would likely be other, new issues, but not those 4 ones. Becoming an ORM also opens up HUGE opportunities in terms of brown field adoption. And if you're just an ORM, existing tools will just work without extra engineering on your end.
2. You don't have to give up on dreams of creating an actual DB. Once people are writing EdgeQL via an ORM, it's not hard to get them to migrate over to a database that speaks native EdgeQL (instead of having to be transpiled).

What I'm talking is more about the intersection of corporate strategy, brand positioning, marketing, and long term product roadmap. Maybe you're right to do what you're doing by focusing on making EdgeDB the most powerful and full featured database you possibly can. But maybe it's also not. Maybe it's best from a business perspective to focus on making an incrementally adoptable and incrementally better way of interacting with existing databases via EdgeQL. And then from there you can convince people to maybe jump on a completely new database.

Think even from a branding perspective. Do you really want to be branded as a new database? I would argue that it's likely sub-optimal from an adoption perspective. Developers have a love hate relationship with new databases. We love them because we love shiny things, and we are obsessed with becoming more efficient. But we're also wary, and have been burned by shiny new things before. We're risk averse herd animals too.

Guys, I know I'm just some random dude, and you guys are waaay more invested than I ever will be in all this. And there's a very real possibility you're seeing some savvy business path that I just can't see right now due to my limited perspective. I'm a startup co-founder myself and honestly feel dumb even making these suggestions to you because it just feels like a cheap shot. And maybe you guys are really are charging down a relatively optimal corporate strategy that I just can't see. And if you are, then this whole conversation is just a distraction and it should be promptly forgotten. But I just worry about EdgeDB given my experience thus far and my own analysis of the database industry as a whole.

[1st1]

Yeah, I follow your line of thinking, I really do.

If you guys spent a lot of time and energy on a pure ORM, would devs be able to get 90% of the benefits of EdgeDB?

This is actually quite debatable. We'd have to cut A LOT of functionality and cripple the language and the guarantees our schema provides substantially. It's also not that easy to implement, we don't have capacity for that now (it would be essentially a different product and require a different architecture). At this point we're pretty determined to improve the DX and build a great cloud product, with which the majority of our users won't have to ever use Docker, or manually configure anything. For them using EdgeDB will likely be even more straightforward than using Prisma or anything else. That's our goal and I think we will get there. I also admit that the current positioning of EdgeDB (website etc.) isn't crystal clear, we need to do a better job at that.

I really appreciate you sharing your thoughts. It takes courage to share things like this and it also helps us to see different perspectives on what we are building. Thank you.

[elprans]

it takes a full minute or so for a docker instance of EdgeDB to be ready to accept connections

The first time you run EdgeDB on a pristine Postgres cluster it performs instance bootstrap, which is basically just restoring a dump containing the standard library, but also things like generating TLS certificates etc. It can take a full minute if you have a slow remote Postgres instance or slow-ish storage.

For reference here's how long it takes for me on local host:

Bootstrapping EdgeDB instance on the local volume...
INFO 39 2022-09-30T17:25:10.255 edb.server: starting EdgeDB server 2.3+64aee7d, built on 2022-09-26T00:31Z from revision f24c5d924 (2022-09-26T00:00Z) for x86_64-unknown-linux-gnu
...
INFO 1 2022-09-30T17:25:26.772 edb.server: Serving on {0.0.0.0, ::}:5656
INFO 1 2022-09-30T17:25:26.772 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656

16 seconds total.

Again, this is something that only happens once per remote cluster (or data directory). Second run:

INFO 1 2022-09-30T17:29:22.706 edb.server: starting EdgeDB server 2.3+64aee7d, built on 2022-09-26T00:31Z from revision f24c5d924 (2022-09-26T00:00Z) for x86_64-unknown-linux-gnu
INFO 1 2022-09-30T17:29:22.706 edb.server: instance name: '_unknown'
INFO 1 2022-09-30T17:29:23.643 edb.server: Using 1286 max backend connections based on total memory.
INFO 45 2022-09-30T17:29:23.753 postgres: starting PostgreSQL 14.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
INFO 45 2022-09-30T17:29:23.753 postgres: listening on Unix socket "/run/edgedb/.s.PGSQL.5432"
INFO 45 2022-09-30T17:29:23.756 postgres: database system was shut down at 2022-09-30 17:29:19 UTC
INFO 45 2022-09-30T17:29:23.758 postgres: database system is ready to accept connections
INFO 1 2022-09-30T17:29:24.639 edb.server: introspecting extensions for database '__edgedbsys__'
INFO 1 2022-09-30T17:29:24.639 edb.server: introspecting extensions for database 'edgedb'
INFO 1 2022-09-30T17:29:24.645 edb.server: Connection established to backend database: __edgedbsys__
INFO 1 2022-09-30T17:29:24.650 edb.server: Connection established to backend database: edgedb
INFO 1 2022-09-30T17:29:27.103 edb.server: started 32 compiler worker processes
INFO 1 2022-09-30T17:29:27.105 edb.server: Serving on {0.0.0.0, ::}:5656
INFO 1 2022-09-30T17:29:27.106 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656

~5 seconds.

Are you seeing consistent 1 minute start times even post-bootstrap?

[scottmas]

Yury, sounds like you guys are definitely having these conversations and I can respect that. Elvis, I've created a ticket here on the slow docker startup (#4468).

Final question then for you all. If the pure ORM ship has sailed, could you please, please have a first class story for interacting with EdgeDB just as a normal Postgres database? A first class story also for "ejecting" so to speak from EdgeDB and going back to raw Postgres? That would take sooooo much risk off the table, and would be such a huge help I think. Right now this is kind of supported but we have to specify our own backend-dsn (which is poorly documented) and the whole experience is kind of painful.

[GianniVullo]

Hi

I just ended this day unsuccessfully trying to make Docker and EdgeDB work for my project. And I really like the idea to be able to use EdgeQL instead of SQL as Scottmas suggested it.

Anyway I discovered your project because of your (awesome) youtube videos and I have a lot of respect for what EdgeDB is.

I wish you lot of motivation for what's next !