FoundationDB: From idea to Apple acquisition [video]
youtube.com

・

221 points

・

zdw

・

7 days ago

56 comments

jauntywundrkind ・ 3 days ago

Nice having this backstory (fantastic production value too, impressive start to this podcast). Dis-aggregating the responsibilities of the DB into multiple pieces just feels so logical, helps make sure each piece can scale. Deterministic Simulation Testing gets mentioned in the video & was way ahead of it's time here. https://apple.github.io/foundationdb/testing.html

Hacker News is here too! From July 2012 (78 points, 72 comments): https://news.ycombinator.com/item?id=4294719

For a general introduction, I enjoyed the recent submission How FoundationDB works and why it works: https://news.ycombinator.com/item?id=37552085 https://uvdn7.github.io/notes-on-the-foundationdb-paper/

  • chamomeal ・ 2 days ago

    I watched a super interesting talk about a testing library that was inspired but the foundationDB testing story.

    It uses test.check (clojure version of quickcheck) to help you create valid mocks of stateful services to test against. I just love the idea and I hope to implement a small version of it in TS at work soon

    https://youtu.be/dvHASrrQSzg?si=jF_sTFWL-wCwFNHA

    For indexing purposes, the name of the talk is “Breaking the bank with test contract”, given by Allen Rohner.

romanhn ・ 3 days ago

Posted about this in the past, but what really got FoundationDB on my radar was a demo at a developer conference, back in 2014-ish. They had the database running across a bunch of machines, with a visual showing their health and data distribution. One team member would then be turning machines on and off (or maybe unplugging them from the network) and you could see FDB effortlessly rebalancing the data across the available nodes. It was a very striking, impressive presentation (especially as we were dealing with the challenges of distributed Cassandra at the time).

The beginning of this video has some of that: https://youtu.be/Nrb3LN7X1Pg

  • AtlasBarfed ・ 3 days ago

    So ... cassandra does that? I get the FDB demo probably made it look better and easier.

    But data doesn't teleport except in demos. Rebalancing means streaming data across a network, consuming total network I/O, regardless of the distributed database.

    Did you actually implement FDB, and was it better?

    • jwr ・ 3 days ago

      Comparing Cassandra to FoundationDB is like comparing a spreadsheet in Google Sheets to PostgreSQL.

      I mean, both kind of store data, and multiple users can change the data that is being stored. The story of what you'll get back and when (if ever), however, is rather different.

      I would respectfully suggest that anyone that wants to comment in distributed database discussions should be familiar with https://jepsen.io/consistency/models and https://antithesis.com/resources/reliability_glossary/ and use the wording found there.

      If your eyes gloss over, because there is a lot of complex stuff there, it is likely that your comments will not have much value.

      • AtlasBarfed ・ a day ago

        I've worked with Cassandra for ten years, you don't think I understand consistency and availability tradeoffs?

        Are you acting like FDB has magically solved the CAP theorem? They've beaten mathematics?

        If you want to talk about low-value comments, yours is up there. Mine was a basic question about how FDB streams large distributed datasets in a partition.

        Do you have any insights, or are you just an asshole?

    • rapsey ・ 3 days ago

      Many years later did cassandra get reliable. Fdb was the gold standard that set the bar. They did not need jepsen tests to implement it properly.

vlovich123 ・ 3 days ago

What a great story and really interesting courage to double-down on improving the testing even when a critical flaw that testing should have found was found. Wish that they had managed long enough for Snowflake to keep them alive, but then we wouldn't have Antithesis as a service so silver lining.

  • tptacek ・ 3 days ago

    By "keep them alive", you mean the team, right? People are definitely still using FDB!

    • vlovich123 ・ 3 days ago

      The team pushing forward the vision. Using FDB is a fraction of the vision if you listen to them.

      • tptacek ・ 3 days ago

        Makes sense, thanks! Antithesis is pretty neat, though (we use it for a distributed system thing here).

piokoch ・ 3 days ago

I've looked on FoundationDB and on paper it looks great. But it never got momentum, like, say, MongoDB. Is this just a matter of hype or it is not that great as advertised?

  • jwr ・ 3 days ago

    It is difficult to use by itself: the "foundation" in the name describes it quite well. It is a foundation that you build a database on. It fits my use case very well, for example, because I know my data model and usage patterns very well and I can integrate deeply with the database, but it's not a good match by itself for quick-and-dirty apps.

    It provides fantastic (strict serializable) consistency guarantees in a distributed database, which is extremely rare. It is a huge advantage, but sadly most people do not understand how badly most distributed databases are broken and don't even understand the concepts (https://antithesis.com/resources/reliability_glossary/) well enough to talk about the issues involved. See every discussion where someone mentions ACID.

    It's hard to compete for mindshare when the concepts are difficult and every other database has a warm-and-fuzzy-feeling website saying that everything will be great (it usually won't).

    Personally, I hope more people will start using it, and I hope to see more easy-to-use databases built on top of it (that's what it was designed for, really). In my experience with it, working with a fast distributed database that gives you strict serializable semantics right in your code is fantastic.

  • chrischen ・ 3 days ago

    I think it wasn't as easy to use or get started with. There was a MongoDB compatibility layer but it wasn't maintained.

  • qcnguy ・ 3 days ago

    Many reasons.

    FoundationDB started development in the same year MongoDB launched but took nearly four years to reach the market. It's the rarely discussed dark side of great testing - you can end up with robust code nobody cares about because it arrives years after people decided they wanted it. Everyone went with what existed and learned to deal with its quirks. In this case they got lucky I guess that Apple saw the potential for iCloud and bought them out, but the people who had bet on FDB before then kinda lost. You really don't want your database to be bought and made fully private tech. MongoDB was open source at the start and went closed later but never disappeared, so whilst the license switch pissed people off it didn't fundamentally wreck MongoDB as a viable tech.

    Database tech has a chicken and egg problem. Most people don't want to run their own infrastructure anymore. No clouds offer hosted FoundationDB, so people don't want to use it for that reason, which means there's no demand, so clouds don't offer it, ad infinitum. MongoDB was released around the start of the cloud era, just three years after AWS first launched, so that was less of an issue. Back then "cloud" just meant VMs and storage. And later Mongo built their own cloud offering.

    FoundationDB does full strict serializability checks, which is expensive. One trick it uses to get acceptable performance is by imposing a difficult programming model on the user. Keys and values must be small. Think individual fields of a JSON object, not objects themselves. Transactions also have very small limits in lifespan and size. You can't open a transaction and run a computation against your entire dataset in FoundationDB unless it's tiny. Everything has to complete in five seconds or else your transaction dies.

    Their website used to claim this timeout isn't even configurable, it's hard to know if it changed because the FoundationDB team at Apple don't care about marketing. Probably Apple don't care if anyone else uses it and only made it open source to make the team happy. Even quite average open source projects have better marketing. Their blog consists only of release announcements and the last one was in 2022. A casual visitor who didn't know better would think it had been abandoned years ago.

    The scalability story is unclear. It doesn't matter for most people but the biggest FDB clusters are about 100T in size. Apple say they use it for iCloud but really they use a large fleet of FDB clusters with lots of in-house tooling for balancing and moving data between those clusters. Effectively they built another scaling layer on top of core FDB.

    Even if you work through all of that, what you get is a key value store. Not really a database, it's more like the bottom layer of a database. That's why it's called FoundationDB. It's not meant to be used directly. There are layers that turn your actual data into key/value pairs in a way that offers features like schema handling, object serialization etc but they are language specific and not so well documented. Most devs on the backend will have ORMs or frameworks they already want to use, and Apple server-side is mostly a Java shop so there's a Java layer, but you can't just point Spring at an FDB cluster and go. For instance, there's no notion of a query, or a query planner or even indexes. You're expected to handle all that stuff using libraries in your app.

    So overall it's a highly solid bit of tech that solved a very small, very specific problem very well but years too late for anyone to care. Except for Apple. Good work, whichever Apple executive sponsored that deal!

    • romanhn ・ 2 days ago

      This is a fantastic summary and the only thing I would add is that FDB was publicly available only for a year and a half before Apple bought and close sourced it. There was simply not enough runway for the hype to develop. By the time it was open sourced again four years later, the world had moved on and FDB was something of a curiosity from the past, a what-could-have-been story.

      • jwr ・ 2 days ago

        > something of a curiosity from the past, a what-could-have-been story

        This is what I find bizarre: one would think that engineers would take a rational approach and look at technical merit rather than follow hype.

        Technically today FDB is state of the art, it's really hard to point to any distributed database that gives you strict serializable semantics and versionstamps (these are very flexible and a great tool to implement things like changefeeds).

        But I agree that being bought by Apple killed the momentum and some people don't realize it's been open-sourced since.

msy ・ 3 days ago

Does anyone know how widely FoundationDB is now being used at Apple? I know they run a huge Cassandra cluster, does this serve a different use case?

  • ethan_smith ・ 3 days ago

    Apple uses FoundationDB extensively for iCloud services including CloudKit, with public documentation confirming it handles billions of operations per second across their infrastructure.

  • ntqz ・ 3 days ago

    My understanding is that CloudKit runs on it.

  • minitoar ・ 3 days ago

    iCloud uses both.

Nican ・ 3 days ago

FoundationDB has been growing as my favorite database lately. Even though it is only key-value store.

Out of curiosity: what are the scale limits of FoundationDB? What kind of issues would it start to have? For example, being able to store all of Discord messages on it?

I see blog posts of Discord moving to Scylla and ElasticSearch, but I wonder if there would be any difficulties here.

  • eatonphil ・ 2 days ago

    There are a lot of strict limits so AFAIK everyone uses FoundationDB for fast, consistent, highly-available metadata while doing replication/storage of actual data elsewhere (such as in S3).

    https://apple.github.io/foundationdb/known-limitations.html

    That is to say it's more like part of your solution and not the entire stack on its own.

    • Nican ・ 2 days ago

      Yeah. I read over that page, and everything seems sensible.

      The only part that is not well explained is that "FoundationDB has been tested with databases up to 100 TB".

      • eatonphil ・ 2 days ago

        In their data modelling pages they mention you should break up rows into separate keys per column. Or separate keys per field in a document. This is indeed how many databases model rows on a distributed kv store. So this might be how they achieved 100TB.

        However you still have the issue of any single key-value needing to be in their limit. (But it's not like people typically store enormous blobs in Postgres or MySQL either I think?)

      • Dave_Rosenthal ・ 2 days ago

        The documentation is woefully out of date, sadly. Despite the code being in active development no one is touching the public docs. Though I don’t know for sure, that limitation was probably written something like 10 years ago.

  • heipei ・ 2 days ago

    ScyllaDB discontinued it's free and open source version, so I personally wouldn't build anything new on it.

philosopher1234 ・ 3 days ago

Does anyone know of cool things built with fdb? I’ve been aware of it for a while and it seems very cool but I haven’t seen a lot of details about how folks are using it.

  • jwr ・ 3 days ago

    I am moving my SaaS from RethinkDB to FoundationDB. It's a long-term project that needs to be done very carefully (thousands of people using the app), but the rewards are significant. Thanks to FoundationDB versionstamps, I'll be able to replace changefeeds with polling, simplifying the system, and also make things much faster along the way.

    The consistency guarantees are phenomenal and writing software is much easier when you have strict serializability. Most people do not appreciate this because they do not understand the anomalies that you can get without strict serializable consistency.

  • Quarrel ・ 2 days ago

    I'm excited by the idea of pgfdb*. I think it could be a great. postgres, with foundation replacing the transaction system to provide all the benefits that brings, horizontal scalability, automatic sharding, replication, and fault tolerance, and performance. foundationdb is 20+ years ahead of postgres in terms of distributed transaction theory.

    * https://github.com/fabianlindfors/pgfdb

  • redman25 ・ 2 days ago

    Arroyo uses it for their streaming database. Think streaming stateful aggregations. It’s pretty cool tech. Looks like they were recently acquired by cloudflare.

  • mannyv ・ 2 days ago

    From what I understand one of the big IP ad tracking services (El Toro) is built on FoundationDB.

  • olavgg ・ 2 days ago

    Cognite built their dataplatform with foundationdb.

pjmlp ・ 3 days ago

Nowadays being rewritten into Swift.

"Swift as C++ Successor in FoundationDB" by Konrad Malawski (Strange Loop 2023)

https://www.youtube.com/watch?v=ZQc9-seU-5k

  • jen20 ・ 3 days ago

    That was an experiment the team didn’t end up committing to - it’s been backed out. That said it was a fascinating dive into the flexibility of Swift, and the Konrad’s talk is excellent and worth watching.

    https://github.com/apple/foundationdb/commit/e52fc3621fd5e41...

    • pjmlp ・ 3 days ago

      Interesting, thanks for sharing, is there a rationale somewhere?

      As someone that enjoys using C++ despite all its warts, I can imagine a few reasons, but would nonetheless an interesting read, in case that is public.

      I guess that experience might also had an impact on ongoing Swift 6+ features.

      • jen20 ・ 2 days ago
        ・ 3 more

        I don’t know the specific reasons (and the pull request referenced in the merge commit appears to have been deleted), but my guess is that the extra complexity vs using modern C++ features was not worthwhile to the core FDB team.

        • pjmlp ・ 2 days ago
          ・ 2 more

          Which could be a good reasoning, assuming it is being enforced somehow, in regards to modern best practices, which is usually the main issue.

          • jen20 ・ 2 days ago

            Indeed - my guess is that there is a policy of "improve things as you touch that area of the codebase" rather than blanket rewriting of working code going on.

majestik ・ 3 days ago

I can't put my finger on it but there's a weird tension between the two Dave's in this video. Almost like Rosenthal is trying to impress or earn the praise of Scherer.

Is there a backstory between these guys / FDB?

  • Dave_Rosenthal ・ 3 days ago

    Ha, well I met Scherer ~30 years ago in a high school math class and we’ve done three companies together, so you could say we’ve known each other for a bit :)

    • niwtsol ・ 2 days ago

      Lol - to find your one comment in this thread wayyy down at the bottom was delightful. That demo you reference from techcrunch disrupt ( start of this video - https://youtu.be/Nrb3LN7X1Pg) - that is a great demo. I'm curious - is that an idea/moment in your life you look back on and has that staying power, something that randomly pops in your mind that fills you with pride, or not really?

      • Dave_Rosenthal ・ 2 days ago

        We thought of that demo with like a week or two to go before the event when we realized we needed something more visceral than a blinking cursor :)

        So, it stayed with the world because it went semi viral in some tiny circles. But it didn’t stay with me because I personally learned little from it.

        The thing that stays with me is all the things I learned spending years trying to implement an API you can describe in ~30 seconds really well. It’s a rare opportunity to be able to go so deep on a problem in a career.

gregoriol ・ 3 days ago

So if it has been acquired by Apple, it's a failure, isn't it? Most things acquired by Apple get unmaintained or change completely, or disappear. Being "open-source" here doesn't bring any guarantees to any third-party user about maintenance or long-term life. It should be a serious no-go indicator for anyone willing to build something with it.