Epistemics and Coordination: It’s complicated!

Money is pouring in, people are looking for new areas to fund, and the invisible hand is starting to grab a bit at AI for epistemics and coordination (AIFEC hereafter). It also got invoked in AI2040 as a potential part of the winning strategy. My feelings here are mixed — I think the best version of AIFEC is great, but also the existing public writeups are only a few cycles deep on tracing out the different ways that the obvious plan backfires. And regrettably I think some people have correctly written off AIFEC because what they have read appears a bit naive to them. In my heart I always planned to do a proper writeup of my thoughts when things were a bit less busy, but, well, now we’re in the 100x funding era. So here’s my scrappy, hopefully-better-than-nothing attempt. The six big claims: AIFEC could be great! It’s a tractable way to do good on the margin, and the best version is a legitimate theory of victory It could also easily backfire, especially for implementations that depend on scaling with inference “Better epistemics” is often more hostile than it seems, and people have good reasons to be wary of things which profess to help them understand what’s going on Vagueness about what AIFEC is doing lets you ignore tradeoffs: Yes you can just go make a bunch of AIFEC startups, and yes AIFEC could help with international coordination, but those scrappy startups aren’t what solve US-China tensions. Sometimes people just don’t get along But still, AIFEC could be great! The basic case is pretty good! Here is my favourite case for AIFEC: If humanity ends up blowing itself up, it’s going to be a combination of three factors. Firstly, maybe we just rationally incurred some risk, the same way that getting on an airplane might kill you. Secondly, maybe we underestimated the scale of the risk. Thirdly, maybe some people took risks that personally benefit them at the expense of others, generating negative externalities. I find this story pretty compelling, and surprisingly different in form to the standard arguments about AI x-risk. It’s actually much more general. And when you squint at the three factors, well, the first isn’t even really a problem, the second is basically poor epistemics, and the third is basically poor coordination. So maybe if we just get enough epistemics and coordination, we’re in the clear? And it sure seems like AI is about to make this way easier — so much more cognition on tap, plus all kinds of neat new form factors like arbitrating minds that genuinely vanish after the fact. Add to that: it’s not really all-or-nothing in the same way that alignment is. Decent AIFEC lift should help with all our other problems. In fact, AIFEC should help a fair bit with AIFEC — the better our epistemics and coordination, the more effectively we can coordinate to appropriately invest in further work. One can imagine a kind of dizzying spiral to heaven, where we pass the Coasean friction event horizon, slay Moloch, and fuse into a liberal-libertarian omnimind in which every individual has their own authentic preferences while the group deftly dances along the pareto frontier. X-risk drops to the socially optimal level, the socialist calculation problem is finally solved, and we can all finally just get along. Hyperbole aside, it’s worth emphasising that something in the realm of AIFEC is a legitimate full-blown wincon up there with aligned AGI. Indeed, a lot of people’s tacit plan seems to be “align the AI and then let it solve democracy and all the rest”, but one could just as well go in the other direction: “solve politics and ignorance, and then just be reasonable about advanced AI” Alas, it is not so simple. Problem 1: Naive AIFEC could easily backfire The biggest splash of cold water is that AIFEC won’t be free. Especially in this era of inference scaling and an increasingly closed frontier, the fruits of AIFEC will not be equally distributed. There are certainly some bits of technological progress that are remarkably egalitarian — even billionaires use facebook, gmail, and iphones. But if the selling point is that you get to throw lots of intelligence at solving problems, then you’re going to differentially favour the people who can actually call up that intelligence at scale. Indeed, there are ways this could backfire. It’s certainly possible that AI will enable everybody to seamlessly coordinate, but first it will enable small groups to, ahem, collude. If you’re worried about things like coups and permanent underclasses, it’s not clear that more coordination makes your life any easier. Similarly, if you build a machine that turns compute into better decisions and more situational awareness, those benefits will mainly accrue to the people who actually have the compute. Problem 2: “Better AIFEC” can be a somewhat hostile move It’s nice to think that everyone who believes false things is basically just misguided, and that the only reason they don’t take up arms for the truth is that they haven’t seen it yet. I think the real picture is unfortunately a bit more complicated. The problem is, historically there have been many groups that have weaponised very convincing arguments to get their way. One reasonable response to a seemingly faultless argument for a seemingly crazy conclusion is to throw up your hands and assume you’re being swindled — in other words, epistemic learned helplessness . Even if team AIFEC is actually the good guys, people will be correct to be suspicious. Governments in particular depend a lot on restricting what kind of information is admissible, as does the judicial system. On top of that, I do think that before you start swinging the club of truth it’s worth taking a beat to ask how pure your motivations really are. The EA/rationality community has a bit of a history of swinging the club of truth in a more hostile way — “save the drowning child” etc. Moreover, in the realm of politics, often the real sleight of hand is just changing what things are salient — elections are won over people’s sense of what the election is really about. Framings and deliberative processes are rarely as neutral as they seem. So even if your tools don’t privilege a specific answer directly, the choice of deliberative structure is, well, a choice. The same is true of coordination. To give an obvious example, democracies are meant to represent the will of the people, but the structure of representation pretty directly modulates that will. The choice between proportional representation and first past the post, bicameral houses, election cycles, district boundaries and so on are superficially choices about how to structure the coordination, but they also sometimes obviously favour certain conclusions. It’s really hard to build neutral coordination structures, and people are right to be sceptical of anyone who pretends otherwise. One slightly thornier point: believing false things is actually a pretty powerful coordination mechanism. Many groups cohere around a mixture of surprising truths and blatant falsehoods. When you try to bring the truth to such people, they will actually fight back. As a modest example, deconverting a child from the religion held by their entire family is actually pretty unpleasant and arguably not very nice. This is true to a lesser extent for e.g. adults and the political tribe of their social milieu. Now, maybe it’s worth it if you’re actually right, but you should expect them to fight back! So: plenty of good to be done, but plenty of pitfalls along the way. Problem 3: Vagueness lets you ignore tradeoffs I think one of the major appeals of AIFEC is that you can just let a thousand flowers bloom by sending a lot of bright young things off to found startups and seeing which ones succeed. This is certainly scalable and likely to produce some hits. Unfortunately I’m not sure it’s enough, and the other parts are a lot harder. What coordination and epistemics do we actually need? One way I like to think about this is in terms of what the channel is that leads from the so-called better angels of our nature to actual global action. Probably most of that comes down to robust democratic oversight and international coordination. These are very difficult! And I do not expect that even the top percentile AIFEC startup moves the needle that much, because these processes are by design pretty resistant to hopping on the latest technological fad. Similarly, plenty of AIFEC wins don’t seem to me to really be on the critical path. Deliberative processes, for example, seem like somewhere that AI could be enormously helpful, in a way that might help overcome a major traditional limitation of democracy, but I don’t see that being super crucial for dealing with worlds where people lose all their leverage. It seems helpful, certainly, but not obviously necessary and definitely not sufficient. Getting lots of products off the ground seems great, both in case some are great and to build institutional capacity and expertise, but we can’t neglect the other steps, and it’s important to think at least a bit about what specifically moves the needle. This problem isn’t at all unique to AIFEC — the general AI risk movement has a bit of a problem with coming up with some new cause celebre and then unleashing a torrent of work that nominally fits the category without being useful. And I don’t want to overstate it: the mass of AIFEC startups probably will produce some hits, along with some important general lessons and greater capacity. But the best version of AIFEC does need to grapple with it. Problem 4: Sometimes people just don’t get along At the risk of psychologising, I think part of the appeal of AIFEC is that it lines up with a kind of technocratic mistake theory instinct that basically the current race towards a cliff-edge is one huge misunderstanding, and if everyone could be provided with the right information and the right structure, things would all work out. I think this is pretty true! And I yearn for it to be more true. But it is not entirely true. Sometimes people just don’t get along. Some people would rather take AI soon so as to increase the odds of their own immortality or the immortality of their family, even at the risk of destroying all of earth. Some people really do want to tile the universe with hedonium. Some people’s order of preferences is basically “my country wins” > "annihilation" > “enemy country wins”. Some people are currently reaping the benefits of not internalising the risk externalities they create. Some people are causal decision theorists. Global coordination needs to either include these people or, well, conspicuously not include them. And they are not dumb! [1] They are not merely passive processes that will fail to notice your attempts to re-engineer the environment around them. Sometimes the ruling party decides to block political change even if it’s obviously reasonable behind some veil of ignorance, because they correctly notice that in the moment it is to their detriment. Some of this stuff you can bargain about, but, well, the structure of the bargaining is not neutral. And you can only bargain so much with someone whose parents are slowly getting older and sicker. I really don’t know what to do about that. It makes me pretty sad. And we are going to run into it more and more, so the sooner we can deal with it, the better. But damn, it’s tricky. But overall, I am still pro I reel off these problems not because I don’t believe in AIFEC, but because I do believe in what the better version of AIFEC could be. I’m baffled by how overinvested we are in strategies that, from my perspective, seem specifically geared towards helping frontier labs navigate the acute risk period. I think in an adequate world we’d be pushing simultaneously on every plausible path to victory, at least until we hit the thresholds where they started to trade hard against each other. One obvious reason to be long on AIFEC is that it automatically scales with AI capabilities. Some version of it is clearly coming, and will clearly be helpful. And right now in particular there’s some opportunity to steer things, to get certain balls rolling sooner. Even if you’re extremely bitter lesson-pilled, a three month lead could be worth a lot. There are a few other big topics I didn’t cover here that do feel relevant to me. In no particular order: More so than I'd like, AIFEC tends to privilege individual-level agency, whereas I am pretty bought in on group agency being real and very important for risks. AIFEC doesn’t grapple as much as I’d like with power and leverage, and indeed the frame seems to slightly bounce off them. This is related to problem 4 — AIFEC hasn’t grappled as much as I'd like with conflict theory. I feel we lack institutional knowledge about how you serve both God and money. Finally, I continue to feel that one of the most underrated pieces of work I contributed to was The Choice Transition , which is basically an attempt to articulate how AIFEC might get humanity into a stable basin from which we can reliably avoid bad outcomes and slowly build up to the good ones under our own volition. Compared to all the other paths to victory, AIFEC seems like the only one with this property — that humanity can correctly recognise itself as having the power to work towards the best outcomes — and my liberal instincts feel that this is worth holding onto. (crossposted from my new-ish blog ) ^ Except the causal decision theorists Discuss