The problem wasn't that we (and presumably others) didn't plan for open-core/cloud. We did, but there are structural problems in the market that prevent this from working.
Open-core didn't work because the space is so crowded with high-quality options that you have to give away enormous amount of functionality for free to get adoption. Given how complex distributed database products are, by the time you get to building a commercial edition you're many years in and very short on cash.
Cloud didn't work because AWS/GCloud have enormous moats of pricing and brand recognition. They drive margins down to epsilon, and if your product sees meaningful adoption in the industry they launch their own service and take all your customers.
First off, I really appreciated your frank blog post on the RethinkDB post mortem. The distillation of years of experience is incredibly valuable for us, and I'm sure for many others.
I agree that the database market is crowded with solid offerings. However, I believe that differentiating features do still matter and there will be tremendous growth in the database market for the foreseeable future. In your blog post you listed the metrics of goodness which you optimized for, perhaps incorrectly. You indeed had amazing execution on those original metrics. We have paid RethinkDB the compliment of doing our best to emulate the standards set with simplicity and consistency, in particular. You are also correct that the alternate metrics including timely arrival, palpable speed, and a use case are probably better ones to optimize for in an entrepreneurial setting.
We have been optimizing from the start for a still-small use case, but one which is likely to become a top of mind concern for every major enterprise over the next five years: building global, "multihomed" services. This is something Google has pioneered over the past decade, but which remains an elusive challenge for most everyone else. For an interesting read, check out https://static.googleusercontent.com/media/research.google.c... (tl;dr here: http://highscalability.com/blog/2016/2/23/googles-transition...)
You mention AWS/GCloud as existential risks for a cloud DBaaS offering. I would take that a step further and cite them as the biggest risk to all database companies. We must compete with them both by pushing the boundaries of what the database can accomplish, and by aggressively driving an anti-vendor-lockin message: embracing a proprietary cloud DBaaS offering is an unacceptable risk if there are non-proprietary alternatives.
Do you think there's something different that Cockroach could do? Or do you think that the database market as it exists today leaves them without a path to building a lasting business?
Personally I believe that latter. I don't see how they're going to get people to pay for Cockroach with all the options that are already out there. I think they might have an even more difficulty than RethinkDB did because their interface is SQL which means that migrating to things like Postgres or RDS is a lot easier.
A majority of software companies can operate on RDS (and DynamoDB for that matter) and utilize their built-in scaling functions. CockroachDB adds no value to those customers, which will presumably make up the bulk of them hypothetically becoming a profitable company.
Contrast CockroachDB with some of the other nascent open-source-to-successful-businesses:
Cockroach - Solves scaling issues with SQL databases by offering an alternative DB. (??)
Unless Cockroach can position itself as either solving a technology challenge with an already adopted DB solution, or solves a business problem, it's going to be very difficult to achieve profitability.
I should add. One of the reason I thought that RethinkDB might actually survive was that (as far as I could tell) their database not only solve scaling issues, but also helped solve real-time push to mobile apps. I'm surprised they didn't position more to mobile app devs for adoption.
I've never started a database company, so I don't pretend to know the market the way you do. But I track the database market (open source, commercial, and anything in between) more meticulously than probably anybody on the planet.
I've been working on a series of blog posts since October around the subject to databases and their future, and one post was intended to be my thoughts on RethinkDB. The leak of your postmortem that was revealed on Tuesday has made me reconsider releasing it, but your comment above makes feel obligated to share a few thoughts.
1. The database market is NOT closed. In fact, we are in a database boom. Since 2009 (the year RethinkDB was founded), there have been over 100 production grade databases released in the market. These span document stores, Key/Value, time series, MPP, relational, in-memory, and the ever increasing "multi model databases."
2. Since 2009, over $600 MILLION dollars (publicly announced) has been invested in these database companies (RethinkDB represents 12.2M or about 2%). That's aside from money invested in the bigger established databases.
3. Almost all of the companies that have raised funding in this period generate revenue from one of more of the following areas:
a) exclusive hosting (meaning AWS et al. do not offer this product)
b) multi-node/cluster support
c) product enhancements
c) enterprise support
Looking at each of the above revenue paths as executed by RethinkDB:
a) RethinkDB never offered a hosted solution. Compose offered a hosted solution in October of 2014.
b) RethinkDB didn't support true high availability until the 2.1 release in August 2015. It was released as open source and to my knowledge was not monetized.
c/d) I've heard that an enterprise version of RethinkDB was offered near the end. Enterprise Support is, empirically, a bad approach for a venture backed company. I don't know that RethinkDB ever took this avenue seriously. Correct me if I am wrong.
A model that is not popular among RECENT databases but is popular among traditional databases is a standard licensing model (e.g. Oracle, Microsoft SQL Server). Even these are becoming more rare with the advent of A, but never underestimate the licensing market.
Again, this is complete conjecture, but I believe RethinkDB failed for a few reasons:
1) not pursuing one of the above revenue models early enough. This has serious affects on the order of the feature enhancements (for instance, the HA released in 2015 could have been released earlier at a premium or to help facilitate a hosted solution).
2) incorrect priority of enhancements:
2a) general database performance never reached the point it needed to. RethinkDB struggled with both write and read performance well into 2015. There was no clear value add in this area compared to many write or read focused databases released around this time.
2b) lack of (proper) High Availability for too long.
2c) ReQL was not necessary - most developers use ORMs when interacting with SQL. When you venture into analytical queries, we actually seem to make great effort to provide SQL: look at the number of projects or companies that exist to bring SQL to databases and filesystems that don't support it (Hive, Pig, Slam Data, etc).
2d) push notifications. This has not been demonstrated to be a clear market need yet. There are a small handful of companies that promoting development stacks around this, but no database company is doing the same.
2e) lack of focus. What was RethinkDB REALLY good at? It push ReQL and joins at first, but it lacked HA until 2015, struggled with high write or read loads into 2015. It then started to focus on real time notifications. Again, there just aren't many databases focusing on these areas.
My final thought is that RethinkDB didn't raise enough capital. Perhaps this is because of previous points, but without capital, the above can't be corrected. RethinkDB actually raised far less money than basically any other venture backed company in this space during this time.
Again, I've never run a database company so my thoughts are just from an outsider. However, I am the founder of a company that provides database integration products so I monitor this industry like I hawk. I simply don't agree that the database market has been "captured."
I expect to see even bigger growth in databases in the future. I'm happy to share my thoughts about what types of databases are working and where the market needs solutions. Additionally, companies are increasingly relying on third part cloud services for data they previously captured themselves. Anything from payment processes, order fulfillment, traffic analytics etc is now being handled by someone else.
We have been working on a general-purpose resharding for over 3 years, but have yet to release it to the open source community: it's very hard to do it well.
But our customers get a sharding scheme that best suits their business needs, including fully automatic shard management and data re-balancing. I submitted a talk about the technology and know-how behind this to Percona Live 2017: https://www.percona.com/live/17/sessions/best-practices-appl...
True, but I think that'll have an asymmetric effect. Generally when people adopt a product it's because they have some sort of burning desire, it's faster, it let's me do stuff I couldn't before, it allows me to avoid spending lots of money, etc. Then, once they want to switch they consider the switching costs to see how feasible it is. Switching off of Oracle would presumably save many companies lots of money, the fact that they don't can only mean that switching is too tough.
This means that even with a low switching cost they still haven't created that burning desire for people to adopt the product and that's generally the harder part of the equation. I do think Cockroach is creating that in other ways with their Geo replication and sharding capabilities to name a few. But no one is switching to Cockroach because it's SQL so why not. However we know they're creating a burning desire for people to switch off their product by charging them money since people would always rather not spend money. The switching cost has to act as the counterbalance to that desire. The lower the switching cost the less you'll be able to charge people. This can be a very hard thing to solve after the fact and companies resort to all sorts of contrived things to try to get people locked in to products that don't inherently have strong lock-in.
Yeah, I didn't appreciate Cockroach's blog as a response to RethinkDB's shut down here. Admittedly it is timely, but they don't provide any insight (although with the long blog, it "appears" as though they do) that you and Mike haven't already covered.
I have appreciated the snippets of mentorship that I have gotten from you and Mike (I've interacted with Mike more) - I'm Mark from the GUN team. Here are my thoughts:
RethinkDB's shutdown spells doom for Cockroach. However, I do disagree with you Slava, that the DB market is impossible.
Rethink and Cockroach are both Master-Slave, and I think you hit the nail on the head that that is an impossible market to try and compete in. However, it does not represent the entire DB market (albeit, it is the overwhelming majority).
The market take over is going to happen with P2P/decentralized databases (Cassandra, mine http://gunDB.io/ , even things like IPFS, etc.) because Master-Slave databases have a limit of how large they can scale and shard. Up to another 5B people are coming online into 2020, so the demand alone is going to reshape the industry towards the growing Master-Master databases. Cockroach is in the wrong place.
My company is going through an inflection point, so I'll presumably be one of the guinea pigs. If I'm right, we'll be able to keep all of our technology completely MIT/ZLIB/Apache2 Open Source, yet still grow healthily and fast enough for our VC backing. Why? The inevitable shift to P2P systems is going to require the existing monoliths and governments to get on board and they need experts who have designed those systems. (I'm already seeing this happen with some of our customers and potential clients).
Startups will be able to reap all the benefits for free, even if they get Pokemon Go level hyper growth. Why? Services with hundred million plus users will be the norm, not the enterprise. And their services will become more robust with more users on it. However the dinosaurs, governments, centralized services, etc. will still pay handily in order to keep spying on their users. Unfortunately, users will still use these services because those companies actually make a profit which they re-invest in conveniences that keep users around.
That will be the divide (and has always been) between free and paying DB customers. None of this CCL stuff.
From what I've read about CockroachDB, I thought it was master-less. If so, I certainly hope there's a market for a horizontally-scalable, ACID-compliant database. I know it would come in handy for a number of applications I've worked on that required consistency but had workloads that were difficult to fit on a single server or a master-slave setup.
The problem wasn't that we (and presumably others) didn't plan for open-core/cloud. We did, but there are structural problems in the market that prevent this from working.
Open-core didn't work because the space is so crowded with high-quality options that you have to give away enormous amount of functionality for free to get adoption. Given how complex distributed database products are, by the time you get to building a commercial edition you're many years in and very short on cash.
Cloud didn't work because AWS/GCloud have enormous moats of pricing and brand recognition. They drive margins down to epsilon, and if your product sees meaningful adoption in the industry they launch their own service and take all your customers.