In this post, I argue that individual intellectual activity can only be sustained in an intellectual community of other humans. AI dissolves these communities, which in turn makes private intellectual activity rarer. The Case of Software Some years ago, when it became clear that AI would solve software engineering,...

For us, humanity was a distant goal toward which all men were moving, whose image no one knew, whose laws were nowhere written down. — Hermann Hesse, Demian Despite having spent most of my life marinating in transhumanist ideas, I never cared about AI. Because I’m a classical transhumanist: I want to use technology...

If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by...

There’s a widespread belief that artificial superintelligence (ASI) is inevitable. In this post, I argue that this is false, and believing it is upstream of a lot of bad discourse. Why do people believe it? It’s easy to believe. Samuel Butler, already in 1863, observed that machines are improving faster than humans...

Any rigorous thinking about the long-term future leads to extreme outcomes. It cannot, by construction, produce normal or mundane futures. “Rigorous thinking” means you have a model: a formalism for representing the state of the world, and a transition function to advance the state. You put the world into the...

So, I have this little app, Epoch, that I use to keep a journal. It’s a tiny Rust web app that runs as a systemd service and uses SQLite as the database. I use a desktop and a laptop regularly, and use Syncthing to synchronize them, including Epoch’s database. That way I can use the app on both devices without...

The implicit promise of modernity is: we have to work really hard for a few centuries, but eventually we’ll get to the Star Trek future where we’ve automated all the drudgery, and all work is meaningful, heroic, and dignifies the spirit. We will be artists, scholars, captains of the starship Enterprise, etc. There’s...

Many of the people building AI, and many of the people working on AI safety, share a common vision of what a good AI future looks like: We figure out alignment, build superintelligent AI, and it takes over the world, for our benefit. Many people will tell you that last part outright: they think human disempowerment...

Fiction Invisible Cities by Italo Calvino. Beautiful prose and imagination. By Night in Chile by Roberto Bolaño. Beautiful and surreal prose, but completely unpretentious. Valuable Humans in Transit and Other Stories by qntm. I had already read most of these but they are very, very good, and it was a pleasure to...

There’s a game called Mass Effect, which I haven’t played, but I’m told in the game there’s this species of sentient robots, the Geth, whose creators try to exterminate them, and the trigger for this Butlerian jihad was a domestic servant Geth looking up at its owner and asking: “does this unit have a soul?”. And if...

Marcus Hutter—oldheads will remember him from AIXI—recently published a book: Job-Less Utopia: Macroeconomics in the Age of AGI. I’m always curious what the people building AI think or want the future to look like, and I’m also always looking for non-doomer takes on the post-AGI future, so I decided to read it and...

Yesterday, OpenAI announced the solution to ten open problems in mathematics, all discovered by a yet-unreleased model. There’s only one, the coding theory one, where I know enough to say “huh, that’s important”, but according to the mathematicians I trust this is important. Inevitably, people will spin some...