AI News Archive: July 8, 2026 — Part 12
Sourced from 500+ daily AI sources, scored by relevance.
- China’s DeepSeek developing its own AI chip, sources say
China’s DeepSeek developing its own AI chip, sources say The Japan Times
- China’s DeepSeek developing its own AI chip, sources say
China’s DeepSeek developing its own AI chip, sources say The Straits Times
- China’s DeepSeek is building its own AI chip to end US reliance
ChatGPT rival is seen as the standard-bearer of China's AI ambitions after releasing an AI model that went viral worldwide last year
- DeepSeek takes control of its hardware: report
DeepSeek is developing its own AI chip for inference, Reuters reported.
- Momenta makes public debut in Hong Kong, puts physical AI in spotlight
Raising about HKD 6.8 billion in a heavily subscribed offering, the company is pitching its world model as the foundation for physical AI applications.
- Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance
Agentic systems often face a trade-off between accuracy and cost. The highest-performing proprietary frontier models and harnesses provide top accuracy but are...
- LangChain and NVIDIA launch the NemoClaw Deep Agents Blueprint
New blueprint for building secure, governed deep agents with NemoClaw.
- Tuning the harness, not the model: a Nemotron 3 Ultra playbook
Guide on optimizing Nemotron 3 Ultra for better performance.
- Deep Agents Code on NemoClaw: a governed blueprint for your most sensitive code
Blueprint for securely running sensitive code with Deep Agents on NemoClaw.
- Notes on technical alignment via human-like social drives
1. Frontmatter 1.1 Backstory for this post As discussed in Intro to Brain-Like-AGI Safety , I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach. After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would lead to good prosocial motivations? It’s an unsolved problem (see “We need a field of Reward Function Design” (2025) ), but as we search for a solution, we might look for inspiration at how humans (sometimes) wind up with good prosocial motivations. I’ve been working on this problem for years, but most of that work has involved laying foundations (e.g. trying to understand how human social drives work). Whereas in the past four months, I’ve been thinking very directly about how to apply those ideas to AGI. I’ve published two little things from this four-month effort— “Act-based approval-directed agents”, for IDA skeptics , and Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology) —but most of what I (think I) figured out is not self-contained, but rather part of a big interconnected mess of thoughts and ideas. So for now I’m just dumping that all into this one excessively-long post. Sorry. This post has lots of new-to-me ideas and opinions, and they are in great need of more scrutiny and especially deconfusion . I’m happy for any feedback, pushback, and riffs on anything herein. 1.2 Table of contents / tl;dr Sections 2–5 start from human social instincts, and ponder how to usefully employ something-like-that in an AGI. Specifically: Section 2 goes over the high-level approach of using human social instincts as a starting point / inspiration for an AGI motivation system. What are the specific instincts in question, how do they work in humans, what roles if any should they be playing in AGI, and what should we be keeping in versus leaving out? Section 3 discusses three potential failure modes that I’ve spent a long time thinking about: the possibility that the AGI will wind up with the wrong “moral circle”; the possibility that norm-following in humans is reliant on a balance-of-power dynamic which wouldn’t apply to the ASI-human relationship; and the possibility that consequentialist desires will squash norm-following desires in the long run, when both are present (as I claim they need to be). Section 4 is “What controls the set of virtues that the AGI takes pride in?” I discuss two pathways: “person-first” and “desire-first”. For example, someone could wind up taking pride in their encyclopedic knowledge of Disney princesses because that’s what the cool older kids in school are into (“person-first”); or because they really really like Disney princess movies, and that love has gradually wormed its way into their self-image (“desire-first”). I suggest that the “desire-first” pathway is an important way that nerds like me wind up motivated to figure out the truth and share it with others—a key trait that we may want in brain-like AGI. (Hold that thought!) Section 5 goes over some implementation details related to transplanting human social drives into the foreign soil of AGI source code. Then in Section 6 I switch from forward-reasoning (starting with human social instincts) to backwards-reasoning (starting with desiderata), by asking: What AGI motivations do we want anyway? Section 6.1 goes over three sets of constraints that I’m trying to satisfy: technical alignment constraints (we need to be able to write the code), strategic constraints (we need to make the world resilient to misaligned ASI), and ethical / societal / buy-in constraints (the plan needs to sound reasonable, such that people will actually follow it). Section 6.2 goes over a bunch of possible AGI motivations, and how they seem to stack up against those three sets of constraints. I wind up tentatively advocating for a high-level approach that I’ve been calling “truth-seeking disagreeable nerd AGI”, using the technical alignment idea mentioned in §4 above, and then have that AGI figure out what to do next. Section 7 has a couple more random things from my notes: Section 7.1 discusses two (related) dilemmas that I’ve been struggling with. I call them “the visceral reaction updating dilemma” and “the value drift dilemma”. Section 7.2 describes a subtle mistake in how I was thinking about “ruthlessness” until recently. I close in Section 8 with what I plan to work on next and why. 1.3 The scary scenario that I’m assuming herein The premise of this project is: we assume that someone has figured out the secret sauce of how the human cortex (in conjunction with related brain areas) creates human intelligence (cf. Foom & Doom §1.3 ), and then they ask me what code they should write for the reward function, training setup, test protocols and so on. And I imagine (perhaps unrealistically) that I somehow resist the urge to run away screaming, and instead try to answer. What would I suggest? That’s the question I’m working on. In particular, within this scenario: We already have brain-like AGI in a research setting, it’s already powerful, and there are straightforward paths to making it far more powerful yet (see Foom & Doom §1.7 ). So far, it’s a “ruthless sociopath AI” . It optimizes its reward function, or at least something vaguely related to its reward function, with callous indifference to human welfare or norm-following, and it will happily lie, cheat, and cover its tracks to do so. I’ll optimistically assume that the programmers are aware of this problem, which is why they’re asking me for ideas. But they don’t know how to solve it. Compute requirements are very, very much lower than most people are imagining , including for training from random initialization, such that iteration is very fast, and such that global government tracking of training runs would be an unprecedented challenge. If this one group was able to build a brain-like AGI already, then probably many other groups will be able to do the same thing in the near future. By “near future”, I mean a year or two at most, and quite possibly more like a few months. [1] Due to the above assumptions, the “pause / stop AI” movement has basically already failed (in this scenario). Relatedly, if there isn’t already a global compute governance regime in place, one drastically more intense and draconian than even most compute governance advocates are imagining today, then it will probably be too late to create such a regime, at least not by any normal means. Don’t get me wrong: some kind of “pause / stop AI” plan would be great if it succeeds, but I don’t expect it to succeed (see Foom & Doom §1.6.1 ), and more importantly, I’m assuming in this post that it has already failed, and that we’re now on to Plan B. …Or if Plan B was whole brain emulation, or human intelligence amplification, or brain-computer interfaces, or a different AGI paradigm, or whatever, then Plan B has also already failed, and now we’re onto Plan C or D or whatever. Anyway, we’re diving into the whirling knives of getting to a good future with brain-like AGI. And here I’m just focusing on the “have a good technical alignment plan” part, not the cultural and other challenges that will probably doom us even if we do have a good technical alignment plan . The world is generally much like it is today. That means that there’s an enormous amount of low-hanging fruit for a superintelligence to exploit, and in particular, if some ASI wants to disempower humanity, it could. Cf. Foom & Doom §1.8.7 . Anyway, I think this scenario would probably lead to human extinction followed by a pointless post-AGI future that nobody would be happy about. However, we can still try to find the plans that seem merely “doomed” as opposed to “super-doomed”, and there is great dignity in doing so! 2. The high-level approach discussed in this doc: human social instincts as a jumping-off point As background, brain-like AGI is a form of actor-critic model-based reinforcement learning (RL) (broadly construed, see Intro Series §6.4.1 ). Like in the RL literature, there’s a reward function slot in the source code, and programmers can put whatever they want in it. I think this choice of reward function is by far our most important intervention point in brain-like AGI alignment. Humans likewise have a reward function (called “innate drives” or “primary rewards”, e.g. eating-when-hungry is good), and that it leads (among many other things) to social and moral preferences, including some nice things like compassion, helpfulness, and norm-following. So an obvious idea is to figure out how those social and moral preferences work in humans, and treat it as a potential jumping-off point for AGIs. As discussed in My AGI safety research 2025 review §3 , or Intro Series §12 , this is one of two broad approaches that I know of to the alignment problem. The other is out of scope (and I’m currently much more pessimistic about it), although some of this discussion here will be relevant to both. To be clear, we should not be slavishly copying human social instincts into AGIs. For one thing, human social instincts have historically been combined with growing up in a human body, in a human culture, at human speed. AGIs will be a different situation in many ways, so we can’t blithely assume that the same drives will yield the same behaviors. (See Intro series §12.4.4 , plus §3 and §5 below.) For another thing, human social instincts leave something to be desired!—we probably don’t want AGIs that have teenage angst, bloodlust, status-seeking, etc. But still, we can try to deeply understand human social instincts, and then do careful analysis of whether different-but-related mechanisms might be a good plan for brain-like AGI alignment. 2.1 “Sympathy Reward” and “Approval Reward” The first step is thus solving the neuroscience question of how the human reward function works—or at least, the parts of it that lead to nice things like compassion and helpfulness. I worked on that question for years, and finally proposed a sketch of an answer in Neuroscience of human social instincts: a sketch (2024) . In that post, I started from principles of neuroscience and algorithms, and tried to get from there to a putative reward circuit that would (I claim) explain the most important and alignment-relevant parts of human social drives. Then in 2025, I elaborated on that idea from the other side, i.e. fleshing out how this (putative) reward circuit would explain various aspects of everyday human behavior and intuitions, in the three posts Social drives 1: “Sympathy Reward”, from compassion to dehumanization , Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking , and 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa . The following chart summarizes four of the most important forces, [2] which conveniently (I claim) fit into a unified algorithmic framework: [3] Oversimplified gloss on these: “Sympathy Reward” is a situation where I receive positive reward from thinking that someone (who I generally regard as a friend) is feeling pleasure, or negative reward if they’re feeling displeasure. This leads to a general desire to see my friends and idols happy, not suffering. “Approval Reward” is the same setup, but in the special case when the other person is thinking about me-in-particular—for example, maybe we’re having a conversation. This develops into my desire for my friends and idols to like me rather than hate me; to think of me as impressive rather than cringe; and to give me credit for helping them rather than blame for harming them. It also leads to having a strong self-image, anchored by pride (see the “Approval Reward and Self-Image” section at the above link). “Schadenfreude Reward” is a sign-flipped Sympathy Reward for enemies: it makes me want to see my enemies suffering, not happy. “Provocation Reward” is a sign-flipped Approval Reward for enemies: it makes me want to pick fights. However, I think the last two can and should be omitted from AGI (some nuances in footnote [4] ). Then two questions would be: how could we put Sympathy Reward and Approval Reward into AGI? And what would happen if we do? I’ll put off the “how” until §5, and focus on the “what then” question. 2.2 Sympathy Reward leads to ruthless bliss-maxxing, Approval Reward includes a countervailing virtue-ethics-y force To a first approximation, I think Sympathy Reward creates an impersonal-consequentialist desire: wanting humans (and perhaps animals etc.) to feel more pleasure and less displeasure. There are thorny questions here about getting this to work right (see §3.1 below). But there’s also an even thornier issue, namely the same instrumental convergence concern caused by any consequentialist goal: by itself, this part would make a funny kind of ruthless sociopath ASI that bliss-maxxes by, say, strapping everyone to tables on heroin drips. Or maybe it would just kill us all and tile the universe with hedonium . Granted, bliss-maxxing is not the worst possible future, as these things go. But we should aim higher! I want humans who want to keep living a normal life, to be able to. I want humans who want to ascend to radical transhumanist uploaded whatever-ness, to be able to. I want a grand adventure of fun , wonder, friendship, and baffling cosmopolitan diversity, across the cosmos. I don’t want an ever-expanding blob of hedonium. I think the vast majority of people would agree with me on that. Anyway, in humans, this is where Approval Reward comes in. It pushes against ruthless bliss-maxxing, by leading to a suite of desires with a more virtue-ethics-y flavor, related to pride, self-image, respecting other people’s preferences, and proudly internalizing social norms. It’s tricky enough to think through the effects of Sympathy Reward, but I find the effects of Approval Reward even harder to reason about. And the effects of the two together, harder still. It helps that we can draw from human experience, but that doesn’t solve all our problems, because a post-AGI and post-ASI world would be out of distribution in various ways. 2.3 Do we really need the niceness / virtue-ethics-y part? Or can we survive a ruthless optimizer AGI? More on this in §6 below, but I don’t know how to make a ruthless optimizer that doesn’t cause ASI extinction itself, and that meaningfully helps with ASI extinction risk. Part of the problem is: I don’t know how to take some complicated thing described in natural language (e.g. “what an idealized version of us would want, if we knew more…” ) and install it as the ruthless motivation of a brain-like AGI. I do have ideas for how to install certain things as targets of ruthless optimization, but I think those targets have to be kinda atomic concepts, and/or concrete, and/or well-operationalized via an immediate ground truth, and/or related to people’s feelings (as in human social instincts above). For example: If you ask an AGI for an ASI alignment plan, and its ruthless goal is something like “the humans will be pleased with my answer in the short-term”, then it will be trying to please and convince you (at best), not to be scrupulously correct, and we have abundant real-world evidence that people can be convinced by incorrect alignment plans (otherwise we wouldn’t be in a situation where AGI alignment domain experts disagree with each other all the time) ( related post ). If its ruthless goal is something like “the humans will ultimately be pleased with my answer in the long-term”, then that’s even worse: it will be trying to take over the world and brainwash us, or other weirder failure modes. See also my past discussions of reward buttons , debate , “the hope of pinning down non-fuzzy concepts for the AGI to desire” , and more in §6.2.2 below. So, I think we need to get away from pure ruthless optimization, by mixing in some scrupulous honesty, sincere helpfulness, and so on. And I think that the only way to get those is via virtue-ethics-y dispositions, tied to pride in one’s self-image, and which (in humans) ultimately grows out of Approval Reward. 3. Some failure modes I’m worried about If we imagine straightforwardly copying Sympathy Reward and Approval Reward into a powerful brain-like AGI, what might go wrong? Lots! But here are three areas in particular that I’ve spent a lot of time thinking about. 3.1 Getting the right “moral circle” There are a pair of failure modes, corresponding to false negatives and false positives on the question of which entities merit social drives like sympathy. (I mean, “false” is compared to what we were hoping for.) In a human context, we might call the false negatives “dehumanization”, and the false positives “anthropomorphization”. False negatives are bad because maybe the AGI will have callous indifference to human welfare, just as most humans have callous indifference to bacteria welfare. False positives are bad because maybe the AGI will care so much about teddy bear “welfare” that it spends resources on that which could have instead been spent on the welfare of humans or other things that actually matter. In humans, I think we have a set of innate conspecific-detection heuristics (related to faces, gaits, human speech sounds, soft touch, and so on), which reliably generalize to some things (e.g. family members), and reliably don’t generalize to other things (e.g. bacteria). In between those extremes, there are lots of edge-cases where there’s differences across individuals and cultures—e.g. some people care about the feelings of flies, but others don’t. [5] Actually, the word “generalization” is oversimplifying a far more dynamic and complex process that also interconnects with motivation and understanding. Some people intuitively treat bees as automatons unworthy of interest, and then they read some blog post , or maybe their friends and idols all start caring about bees, and now they care about bees. Or vice versa. Even in the human world, we’re constantly (implicitly) deciding whether to empathize with another person, versus ignore them and shut them out, and that subconscious “decision” is substantially based on which option feels better to us (more motivating and pleasant), all things considered, ultimately because of our innate drives. See Social drives 1, §4.1.2: “Motivation to create false negatives in response to someone’s suffering” . 3.2 Maybe norm-following, helpfulness, etc. relies on a slim chance of getting caught, and thus will collapse under extreme power imbalances? I spent a long time going back and forth on whether this is a real problem or not. So I’ll pass the microphone to a self-dialogue between the little pessimist who lives on one of my shoulders, versus the optimist on my other shoulder. Pessimist: There’s a “consequentialist” way to think about Approval Reward, which I concede is not exactly right in its details, but I’ll argue that it has a big kernel of truth. Here’s an example for illustration: A common idea in my professional circles (see e.g. Kevin Simler & Robin Hanson , Wei Dai , Aella ) is that humans want prestige (a.k.a. status), and generally make decisions to maximize their future prestige. This is taken to be a grand unified theory explaining a large swathe of human behavior. And if that doesn’t feel true, well that’s because self-deception is helpful for prestige-seeking too. That’s the theory. I don’t think that theory explains quite as much as those people think it does, but I agree that it’s a motivation for neurotypical people, and sometimes a very important one. And insofar as a future ASI had that same motivation, it would lead to egregiously bad places. After all, if an ASI wants humans to admire it, it would brainwash humans into adoring fans, and then modify our brains to become capable of feeling astronomical, hitherto-impossible levels of adulation. Here’s another example. I think I [I-Steve, not I-the-pessimist-on-Steve’s-shoulder] am more scrupulous than average about being honest. What’s my motivation for not lying? Well, my personal experience, which might or might not be typical (I’m more neurotic than most), is that when I’m tempted to say something dishonest, an image will pop into my head of getting caught and feeling ashamed. Again, insofar as this is true in general, it would portend egregious alignment failure if we tried to make ASIs along similar lines. The idea of getting caught only pops into my head because getting caught seems plausible. Maybe not very plausible, but at least there’s a shade of plausibility. As we approach ASI with ever more power, intelligence, and experience, it would be able to come up with plans for dishonesty for which getting caught would be outrageously implausible, indeed incoherent, like the idea of a static spinning spherical octagon. And then there would be nothing holding it back. Thus, Approval Reward is a thing that can wring some cooperation out of groups of people with comparable power (and thus a possibility of catching and shaming each other for uncooperative behavior). It is clearly not up to the task of getting ASIs to be nice to humans. Optimist: I agree that these kinds of consequentialist desires are an aspect of the desires downstream of Approval Reward. But Approval Reward also leads to non -consequentialist desires, particularly related to pride in one’s virtues. See “Act-based approval-directed agents”, for IDA skeptics . And like I’ve been saying for years, you can avoid instrumental convergence by combining consequentialist and non-consequentialist desires (see Consequentialism & corrigibility (2021) , and my comment on Deep Deceptiveness (2023) ). As I describe in the “act-based” post, people can (for example) be motivated by how they’d appear in the eyes of a cartoon character , or in the eyes of Jesus , etc. This is obviously not a consequentialist desire! It obviously does not lead to instrumental convergence towards e.g. brainwashing the cartoon character into liking you! The cartoon character doesn’t even exist! Pessimist: Well actually, some people will write themselves into fan fiction, and have their fictional cartoon character hero compliment them… Optimist: Oh shush. That’s very uncommon. Pessimist: Anyway, maybe in the short term you can be driven by non-consequentialist pride, but in the long term we really need real-world anchors—actual real-life experiences of getting approval, getting caught, etc. People do in fact go off the rails in the absence of social contact, and they also go off the rails when they have unchecked power over other people (“absolute power corrupts absolutely”). Optimist: I’m not sure how “absolute” that really is. For example, Lee Kuan Yew didn’t have absolute power but he had plenty of power for 30 years yet stayed pretty sane and humble. …Well anyway, even if it were true that, in the human world, niceness needs at least occasional real-world grounding anchors in the form of social contact with a viable chance of getting caught for misdeeds, then we can look into the mechanism that makes this occasional grounding necessary. Maybe there’s some adjustable parameter that we can adjust, such that brain-like AGIs do not have that same requirement as humans. Pessimist: I doubt it. I think there are deep reasons that the occasional grounding needs to be there, namely “the visceral reaction updating dilemma” (§7.1.1 below). Optimist: Hmm, I guess we can table that for now. …But I also want to push back harder on the idea that acting with virtue is reliant on a shade of doubt about whether you’ll get caught. Like, let’s go back to that example above, where we were talking about how I’m honest in large part because when I’m tempted to say something dishonest, an image will pop into my head of getting caught lying, and feeling ashamed. But hang on. Let’s try introspecting a little more carefully. I’m worried about getting caught lying even in situations where the other person wouldn’t be particularly unhappy to learn that I was lying! Conversely, I’m hardly bothered by publicly disagreeing with someone, even when I know damn well that the someone is going to be annoyed and think less of me—as long as I’m confident that I’m right. So this was never really about risk-aversion towards getting caught, at least not centrally. The more important dynamic is that I take pride in being honest and correct—those are part of my self-image—and that’s the metric that I’m risk-averse towards failing at, especially publicly but also even privately. Risk-aversion is a very different thing from niceness. If you have a ruthless sociopath AGI which is risk-averse towards getting caught, you’re in big trouble. It will just wait until it has a really good plan before crushing you. Me, summing up my current take: I think there is such a thing as virtue-ethics-y motivations that are robust to extreme power imbalances, but it’s not automatic. It only happens in a certain kind of setup. See §4 below. 3.3 Maybe there’s something adjacent to the “nearest unblocked strategy” problem? Back to the self-dialogue ! Pessimist: (partly copying from here ) I’m worried about something akin to the “Nearest Unblocked Strategy” problem . If we put both those things together—consequentialist desires from Sympathy Reward (and Approval Reward), plus virtue-ethics-y motivations from Approval Reward—then I’m worried that the consequentialist desire will eventually “win”. For example, if the AGI wants to eventually get to hedonium, and the AGI also wants to follow societal norms, it might find its way to hedonium via a more gradual route, one that involves gradually and unintentionally, but inexorably, changing societal norms in the direction of hedonium. The virtue-ethics-y motivation just seems more squishy and slippery than the consequentialist desire, especially when it routes through manipulable human desires, such that I’m worried it will not be an adequate bulwark against ruthless consequentialism. Optimist: (partly copying from here ) You’re making the same mistake that we were criticizing long ago in that Deep Deceptiveness comment —you’re assuming that the virtue-ethics-y motivation is stupid and myopic. But why should it be? Wouldn’t the virtue-ethics-y side of the AI notice that the AI is engaging in a systematic pattern of behavior that has the effect of gradually shifting human judgment and culture over time? And wouldn’t it see that as bad? And thus, wouldn’t it vote against behaving in that way? Pessimist: I dunno, it doesn’t have to be conscious and explicit. Think about how human culture has drifted over time, following local incentives, with barely any forethought. Optimist: Well sure, in the human world, there’s barely any forethought, because culture mostly develops in a distributed, uncoordinated way, with no foresighted entity in charge. But we’re mostly thinking of ASI singletons ( Foom & Doom §1.8.7 ), which are fully capable of forethought and planning. It’s different. Pessimist: Hang on. Out of one side of your mouth, you’ve been saying that virtue-ethics-y motivations are non-consequentialist, and this is supposedly at the heart of how they can lead to niceness and norm-following rather than ruthless sociopathic maximization. And now, out of the other side of your mouth, you’re emphasizing that virtue-ethics-y motivations can totally be foresighted and consequentialist. I think you’re trying to have your cake and eat it too. Optimist: No, I’ve been consistently saying from the start that Approval Reward leads to a mix of consequentialist and non-consequentialist desires. That’s fine. It can be non-consequentialist in ways that entail not being a ruthless sociopath, but also consequentialist in ways that entail caring that the future be full of fun and wonder, not a uniform ever-growing blob of hedonium. Pessimist: You’re being very vague. I still think you’re trying to have it both ways. You’re hiding the pea under the thimble . Optimist: Nah, I don’t think so. Try this instead: Think about virtue ethics as centering attention on the narrative of one’s life, involving the past, present, and future, and how that ties into the AGI’s self-image— “what sort of force you are in the world.” This story can be an appealing or unappealing story, from the AGI’s own perspective. In this view, being a nice and cooperative AGI right now can be appealing, and being an AGI that steers humans towards a good future can also be appealing, and these are not competing but rather part of the same framework. Pessimist: That sounds nice, but obviously they are competing, in the sense that there are inevitable tradeoffs. And you’ve offered me no reason to believe that the “future” part of that story would be salient enough to sway what happens, when the “now” part of the story might be pushing in a different direction. Optimist: Well you’ve offered me no reason to believe that won’t happen. I never said this plan was guaranteed to work! Just that it might work. Having a plan that might work would still be progress over the status quo of no plan at all. Pessimist: Hmm, OK, sure. Optimist: Actually, I’ll say something stronger: In the limit of arbitrarily strong Sympathy Reward, you get ruthless bliss-maxxing. In the opposite limit, of nonzero-but-arbitrarily-weak Sympathy Reward, the Sympathy Reward wouldn’t do anything. Right? So somewhere in between, there should be a range of settings for Sympathy Reward strength in which it’s doing something important but not crushing everything else. Pessimist: That’s not an airtight argument, but OK fine, I guess so. Of course, that’s not very helpful if we don’t know what the magic setting should be. Optimist: Yes, that’s an open question. Oh, here’s something else. Wouldn’t your pessimistic stance apply to humans too? The whole idea of this plan is to be inspired by human motivation systems. Yet humans don’t ruthlessly bliss-maxx! Normal people would not toss people into experience machines against their will . Pessimist: In the short term, sure, humans don’t ruthlessly bliss-maxx. But in the long term, I think I want to bite that bullet! See my brief earlier discussion under the heading “The arc of progress is long, but it bends towards reward hacking” . Optimist: Even if that’s true, again, uncoordinated human culture is more of a random walk, whereas an ASI singleton would have more foresight. Me, summing up my current take: I think the way “pessimist” was originally flagging this problem as intractable-in-principle was wrong. It is of course a possible failure mode in practice; we need to get that balance. 4. What controls the set of virtues that the AGI takes pride in? 4.1 Two pathways: person-first and desire-first If we want our AGI to take pride in its scrupulous honesty, sincere helpfulness, or whatever else, I think there’s basically two biomimetic paths to getting there. (I think these are not mutually exclusive, but rather two ends of a spectrum.) As an example, let’s talk about how some guy Bob might become a proud wine connoisseur. In the first path, which I’ll call “person-first”, one or more people who Bob admires—his best friends, his favorite celebrities, etc.—all think that wine is great and wine connoisseurship is awesome. So Bob gravitates towards wine connoisseurship as well, and feels proud about that. In the second path, which I’ll call “desire-first”, Bob innately really really enjoys drinking wine. Then he’s unusually likely to become a proud wine connoisseur. The mechanism of the first path is (comparatively) straightforward (see my Approval Reward post ). The heart of Approval Reward is that Bob finds it rewarding when the other person feels positive feelings when thinking about Bob. So if his friends like wine, and Bob is talking about wine, then that will evoke good feelings in the friends, and Bob can see that, so it triggers innate reward in Bob. Also, even if his friends or favorite celebrities are not physically there, Bob can imagine how they would be feeling good feelings if they saw him doing wine connoisseurship stuff, and this thought will also feel pleasurable to Bob, via generalization, and this then feeds into pride and self-image ( Approval Reward §3 ). What’s the mechanism for the second path? Well, there are some details where I’m still a bit hazy, but here’s my current understanding. In the bottom half of the diagram, I showed two (non-mutually-exclusive) mechanisms that wind up in the same place. In the upper mechanism, the starting point is that wine feels very good and exciting to Bob, for non-social reasons (he innately really likes it). Then, if Bob sees someone else nerding out about wine (whether an acquaintance, or celebrity, or fictional character, or whatever), Bob finds that very good and exciting, which blends into feeling a kind of admiration towards that person. [6] Then Approval Reward flips that back around into making Bob feel proud in his self-image as a wine connoisseur, since after all this is the kind of thing that would be viewed positively by the (new) people he admires. In the lower mechanism, I labeled the diagram with the cryptic text: “internal cross-talk thing”. This is the thing I talked about in Approval Reward §5: “Approval Reward × Typical Mind Fallacy = ‘reward for sharing special interests’” . I stand by my earlier description of this phenomenon in that link, but I think “typical mind fallacy” was not the best term for it. In particular, “typical mind fallacy” has the connotation that it’s rooted in an error of belief—an “is”, not an “ought”, being objectively wrong—namely the (implicit) belief that the other person has similar preferences as you. If true, that’s a problem, because errors of belief are unlikely to last very long in a smart AGI (to say nothing of an ASI). However, I’m now leaning towards the mechanism not being based on the presence of a false belief, but rather some kind of cross-talk between different thoughts. If so, that’s likelier to remain stable at high intelligence and knowledge. I’m omitting the details here (I’m still quite uncertain), but I want to flag that this is a relatively concrete question that seems very important, and at some point I would like to nail it down to the extent possible. 4.2 Which of those pathways makes sense for AGI alignment? Now, the “desire-first” pathway might sound thoroughly useless for alignment. If the AGI already innately likes X, then the problem is solved, and who cares about whether that desire connects to its proud self-image, right? But I think that’s wrong. There’s a particular example on my mind: I think an innate aversion to feeling confused, and corresponding delight in gaining understanding, can lead (via the “desire-first” pathway) to a stable pride in one’s honesty, clarity, and insight, being shared in social contexts. (This is how I explain my own neurotic scrupulosity about being honest, mentioned in §3.2 above, along with my intense nerdy drive to share things that I think I’ve figured out, as you may have noticed from not only my current job but also every major hobby I’ve had since childhood .) In this example, the aversion-to-feeling-confused by itself would not be alignment-relevant—indeed, any goal-seeking agent will automatically want to avoid confusion, thanks to instrumental convergence . By contrast, scrupulous honesty and clarity in social settings seem potentially quite useful for AGI safety (see §6.2.1 below). Another example of the “desire-first” pathway (I think) would be §6 of my “Sympathy Reward” post : everyone learns from experience that it’s beneficial to hang around people who are compassionate towards you and people you care about; and this then transmutes into widespread pride in one’s self-image as a person who demonstrates that same trait. Anyway, I have a general feeling that the “desire-first” pathway is more promising (in the AGI alignment context) than the “person-first” pathway. Why? Well, for one thing, in the “person-first” pathway, the good news is that you can intervene on whose approval the AGI cares about (see §5 below)—let’s say it’s Hugh’s—but the bad news is you lack fine-grained control over how the AGI will get that approval, or in other words, which aspect of Hugh’s social preferences will be salient. I think this naturally leads to coming across in a way that Hugh approves of, as opposed to a (more virtue-ethics-y) motivation to be a kind of AGI that Hugh would approve of. After all, if you look at all the instances of Hugh actually approving of what the AGI did, they might not all involve honesty, integrity, etc., but they do all involve Hugh expressing approval. I think this would lead to an AGI with more of a conventional status drive, happy to lie, cheat, and steal to gain approval. As discussed in §2.3 above, this kind of AGI is not helpful. Is there a way to rescue the “person-first” pathway? Well, we could hope the AGI gets approval by good and ethical means first , and then gets approval by sneaky and deceptive means later, in which case an appropriately-timed intervention could lock in the good motivations (“under-sculpting”) . But I’m doubtful that sincere goodness is really going to be learned before the more problematic kinds of approval-seeking. E.g. I think “coming across as honest” is easier to learn than “honesty”, mostly because the former emerges more naturally from a predictive learning algorithm (since how you come across is relevant to how the other person immediately reacts). So, how about the “desire-first” pathway? My gut is very fond of this approach, and more specifically, into the idea of getting honesty, truthseeking, and sharing insights via the “desire-first” pathway, as mentioned above. Is that because I’m narcissistically drawn to the idea of building AGI in my own likeness? Well, umm, probably something like this is relevant to my positive gut reaction, although I’m pretty sure it’s less about narcissism and more about ‘comfort born of familiarity’. And I’m really just talking about one aspect of my temperament; in other respects, man, I have plenty of issues, just like anyone. But, I mean, it’s not totally crazy. For one thing, other things equal, there is an advantage to following human templates, rather than doing something weird and biologically-unprecedented, because the former allows us to draw on human experience when brainstorming failure modes etc. For another thing, if the assumptions of §1.3 don’t hold up, then this post is moot; but if they do hold up, then it would turn out (in hindsight) that I’ve been more on the ball regarding Safe & Beneficial AGI than anyone else on Earth. So if we want our AGI to likewise be on the ball, furthering the cause of Safe & Beneficial AGI, via working on various strategic and technical questions (see §6.2.1 below), just like what I’m doing right now, then trying to make that happen by copying (an aspect of) my temperament into that AGI would seem to be an obvious and prima facie reasonable idea. (And it’s not just me—I think disproportionately many people doing good work in this area have a similar aspect to their temperament.) 4.3 Stepping back a bit If I somehow knew a lot about algorithms and AI but nothing about humans, and you asked me how to make an AGI that sincerely wants to figure out true and insightful things, and share them, then what would I propose? Well, I’ll tell you what I wouldn’t propose: “let’s make a social system by labeling concepts corresponding to thinking about humans and specific human feelings, with ‘upstream generalization’ that leads to pride in one’s self-image, and separately let’s ensure that the AI gets very happy and excited about resolving its own confusion, and then those two systems are going to interact via some weird cross-talk thing that leads to the AI wanting to scrupulously find the truth and share it socially.” That just seems like such a crazy and convoluted solution to this problem. My shoulder-pessimist says: Dear god, that’s terrifying. Do you really expect this convoluted Rube Goldberg plan to be stable upon distribution shifts like unprecedented power imbalances, unprecedented new technologies and ideas, unprecedented capacity for self-modification, etc.?? My shoulder-optimist says: Hey, this is really encouraging! If there’s this nice possibly-useful technical alignment idea that you never ever would have thought of if you didn’t happen to have the human example available for study, then what else are we missing? It sounds convoluted, but who’s to say that it is convoluted? “Confusion is in the map, not the territory.” Maybe we’re missing some better, more elegant way to think about it. We should be eagerly trying to find the most general framework possible that this one example fits into as an obvious special case. In other words, let’s find some domain-specific language for innate drives in which we can easily write down this plan, and see at a glance how and why it works. And then let’s figure out if there are other useful things we can write down in that same language. After all, a priori , it would be quite surprising if the best plan for AGI alignment is the biomimetic one. (For example, shouldn’t §4.1 have a third, AGI-specific, pathway that explicitly involves human programmers doing interpretability and somehow intervening? What would that look like?) 5. More details on how to implement human-like social instincts in an AGI Neuroscience of human social instincts: a sketch is all about the nuts and bolts of how social instincts are built into human brains. If we wanted those to be in AGIs, what would we do? Ingredient 1: The brain-like AGI has to have a set of concepts that corresponds to human emotions, and they have to be “grounded” / “labeled” in a manner that makes them legible outside the learned model. Possible strategies to build this ingredient are: Biomimetic approaches: The AI actually has human-like emotions itself, and creates models of its own emotions, and we get symbol-grounding by copying the biological “interoceptive concept finder” system described in Neuroscience of human social instincts: a sketch §3.2 . Interpretability approaches: the AI spends lots of time observing humans (e.g. by watching YouTube), and naturally forms predictive concepts related to emotions, and then we (the human programmers) go in and try to find and label those concepts. “Evolved modularity” approaches: We hand-craft this part of model space, just like most evolutionary psychologists ( IMO incorrectly ) believe the genome does for humans. Discussion: All three sets of approaches seem like they would benefit from an explicit comprehensive nuts-and-bolts model of human emotions and moods. We don’t have such a model, and I don’t know how to get one anytime soon. [7] We have bits and pieces of such a model, and it’s not immediately clear to me whether or not that would be good enough. The “interpretability approaches” have a “visceral reaction updating dilemma” problem (§7.1.1 below). I’m guessing that the “evolved modularity” approach is kinda incoherent, because the hard work is not in the list of concepts per se , but rather how they relate to everything else in the world, like visual and auditory stimuli, semantic associations with different situations, and so on. And this part has to be learned. I think I currently lean towards the “interpretability approaches”, but need to think more about the §7.1.1 “updating dilemma” issue. I also need to think about whether the “desire-first pathway” of §4.1 implicitly requires that the AGI is itself feeling human-like emotions. Ingredient 2: The brain-like AGI has to actually use these concepts when noticing or thinking about humans. Possible strategies to build this ingredient are: The biomimetic solution: We more-or-less follow the steps in Neuroscience of human social instincts: a sketch , in order to set up situations where the AGI is definitely paying attention to a person, and thinking about that person’s feelings. We should also probably make it innately rewarding to be paying attention to people, at least to some extent, as in the human “innate drive to think about and interact with other people” ( Approval Reward post §5 ). I can’t think of any other plausible plan here, so I guess that one wins by default. Discussion: This part is relevant to the “moral circle” challenge discussed in §3.1 above. The human innate conspecific-detection heuristics seem to trigger on human faces, human speech sounds, and so on. For the AGI, we can do all those same things using conventional ML techniques. (E.g. I’m sure that adequate human speech sound detectors already exist.) The innate classifiers in human brains seem to also trigger on charismatic animals. But I think my gut says that AGI should be specifically tuned to care about people, not animals, and then animals can be treated well to the extent that people want that. (And they by-and-large do want that, when it’s costless.) Ingredient 3: Those thoughts trigger human-like social instincts in brain-like AGI, i.e. we get something like Sympathy Reward and Approval Reward. Possible strategies to build this ingredient are: The biomimetic solution: We have the brain-like AGI watch lots of YouTube (which can lead to Sympathy Reward but not Approval Reward). And we also have it interact with live humans, in a way where humans can judge the AGI, by … I don’t know … I guess plain old conversation. The human asks the AGI to solve a problem, and the AGI solves it, and the person says “great!”. Conveniently, we can pick and choose which person or people the AGI admires and wants the approval of, via straightforward video classifier solutions, unlike human parents who watch helplessly as their teens grow to admire the worst friggin’ people. [8] Again, that’s the only thing I can think of. Discussion: You might be wondering: If I’m so into “desire-first” over “person-first”, why even bother thinking about real-life approval from particular people? Doesn’t that just lead to sycophancy and sneakiness? Why bother? My tentative answer is: For one thing, the real people definitely need to be involved, since that’s what the AGI is supposed to be sharing its insights with. For another thing, wanting approval from real, bona fide people has good aspects (e.g. respect for democratic fairness) along with the bad aspects (sycophancy and brainwashing). I just think it’s a kinda messy thing where we want to balance the different motivations. 6. What AGI motivations do we want anyway? The above sections have been mostly working forward from what I (think I) (mostly) know how to do. At the same time, we also want to be working backwards from what motivations we want the AGI to have. 6.1 The shape of the problem: three sets of constraints When I think about this problem, I feel very boxed in by three sets of constraints. The goal is to describe a path to a maximally-good future that satisfies all those constraints. (Or prove that no such path exists.) The constraints are: Technical Alignment Constraints — I have no great plan for controlling AGI motivations at all. But for some potential target AGI motivations, I at least have sketchy ideas that I think could probably work, whereas for other potential AGI motivations, I either have no idea at all of how to make an AGI with that motivation, or I have ideas but expect them to work in a pretty brittle way that would eventually break down. Obviously, we need to pick a motivation where we can in fact install that motivation into our AGI. Strategic Constraints — As discussed in §1.3 above (and Foom & Doom 1 ), I strongly believe that, once we have brain-like AGI that can do anything at all that’s very important and impressive, we will be frighteningly close to the so-called “acute risk period” where it will become possible for people to create misaligned radical superintelligence that could wipe out humans and run the world by itself. So we would urgently need to end the “acute risk period”, either by making the world resilient to egregiously-misaligned ASI, or by durably preventing such misaligned ASI from coming into existence. I don’t think there would be any way to end the “acute risk period” except by using brain-like AGI to end it (again see Foom & Doom 1 ), assuming (as always) the scenario of §1.3 above. Thus, a constraint on the brain-like AGI motivation system is that this will facilitate a rapid (months not years) end to the acute risk period. Any plan that doesn’t satisfy that constraint is irrelevant. For example, if someone has invented brain-like AGI, and figures out a safe way to use it to cure cancer, then OK cool, but that’s irrelevant to the problem I’m talking about, so I would promptly change the subject to the second brain-like AGI, presumably created by some other company. If that second AGI is being used to, I dunno, safely solve the Riemann hypothesis, then OK cool, let’s move on to the third one. Eventually down this chain, someone will either make an out-of-control AGI that aggressively grabs power, or will use AGI to somehow forestall that possibility, and that’s what I want to talk about. Ethical, Societal, and Buy-in Constraints — Whatever the plan is, people have to actually do it. This includes both the plan for alignment itself, and the plan for subsequently protecting the world from misaligned ASI. I’m especially concerned about the possibility that rapidly protecting the world from misaligned ASI will require some kind of drastic and unpalatable action, [9] and we have to find the least awful path, and there will be intense pressure towards wishful thinking and kicking the can down the road. Relevant context is that, on current trends, the people building brain-like AGI will be extinction risk denialists, or at least not taking extinction risk seriously for reasons that don’t stand up to scrutiny. [10] I doubt that any plan will work in the hands of a team that’s not inclined to think rigorously about safety issues, but we’d like to get as close to that as possible. 6.2 Some potential AGI motivations that we might want to engineer 6.2.1 The brain-like AGI wants to understand the overall strategic situation regarding ASI, and talk to us about options This is currently my favorite plan. (A.k.a., I hate it and expect it to fail, but all the other plans I can think of seem even worse.) You might also call it “Truth-seeking disagreeable nerd AGI” , and I was already secretly talking about it in §4.2 above. The idea here is basically: well, the thing that I’m doing right now, as I write this sentence, is brainstorming what to do about the overall strategic situation regarding AGI and ASI, including technical, philosophical, geopolitical, technological, and every other aspect. And then I’m sharing with other people my understanding of our options and their important expected consequences. I’m not (primarily / consciously) trying to impress people, or follow someone’s instructions, or whatever. I just really want to figure out the truth, and share it, an activity I find both immediately rewarding and important for getting to a good future. I clearly think that this is a good and important thing to do—otherwise I wouldn’t be doing it! So an obvious idea is: let’s have the AGIs do that same thing, except that they could be way better at it, and think about it 100× faster, etc. The setup would be: the AGI has (read-only) access to every book ever written, website, etc., and it’s trying to figure things out, and it’s talking to humans about its ideas. This is related to (but different from) the idea of using AIs to help with AI alignment research. That latter idea is controversial (see e.g. The Case Against AI Control Research ), and in fact I’m usually one of the people criticizing it. What accounts for the turnaround? First, as is obvious from this post, I think humans need to basically solve technical alignment before building AGI. The AGI can help with technical alignment, but only on narrow (albeit important) questions related to distribution shifts due to future technological development, since the AGI would be mostly reasoning about those before they happen (with some important caveats, see §7.1.3). Second, I expect the AGI to be only incidentally thinking about technical alignment, and mostly thinking about how to defend the world against a possible future out-of-control ASI. Third, an aspect of the technical alignment that humans will be doing is to ensure the AGI really cares about figuring out what’s true, not just about telling us what we want to hear. That’s why I’m so interested in truth-seeking motivations, per §4.2 above. Would this approach satisfy “Technical Alignment Constraints”? I feel like I have a vague understanding of a path here that’s not totally doomed (again see §4.2 above for how to get sincere truth-seeking, in the context of other aspects of human-like social instincts). A vague idea is far short of where we’d like to be, but it’s a starting point. Anyway, I’ll mostly treat this plan as a baseline, and then talk about whether the “technical alignment constraints” of this option are better or worse than for the other options below. Would this approach satisfy “Ethical, Societal, and Buy-in Constraints”? Yes, I think this proposal is quite strong here. We’re not having the AGI do anything objectionable. Even people who think that AGI extinction risk is dumb, might nevertheless be open to asking a truth-seeking AGI to try to figure out how future AGIs and ASIs will affect the world, and it’s not completely impossible that the AGI would give a doom-y answer and that the people would believe it. Also, I find that the idea of building an AGI with human-like social instincts is generally pretty easy to explain and has broad appeal (I can even get AI accelerationists on board with it sometimes), and this AGI would fit into that category. I.e., the hope is that the AGI would be vaguely in the human distribution of temperament, albeit pretty far on the nerdier side of that distribution, and I think people would be pretty comfortable with that. Would this approach satisfy “Strategic Constraints”? Well, there’s a basic problem that (as mentioned above) my belief is that rapidly protecting the world from misaligned ASI will require some kind of drastic and unpalatable action, and we have to find the least awful path, and there will be intense pressure towards wishful thinking and kicking the can down the road. If a truth-seeking brain-like AGI comes to that same conclusion, and shares it with the humans, I don’t particularly expect that the humans will believe the AGI, let alone believe it with such confidence that they take appropriate action to solve the problem. And then we all die. If the AGI builds up credibility in other domains, like forecasting or math, that could help. If the AGI is superhuman at persuasion, that could help too (but that’s a double-edged sword). But yeah, this is definitely a plausible way for this plan to fail (if I’m right about the strategic situation). (Not that I’m seeing any other better options.) Here’s some more explicit discussion of failure modes: Maybe the AGI’s suggestions will be wrong? Or it will simply be stumped? After all, there are lots of disagreeable nerds with similar temperament as me, and they sure don’t agree on everything. But I think there’s a ray of hope here; researchers may be able to get superhuman truth-seeking temperament, along with superhuman truth-seeking ability, such that the AGI will be unusually likely to make good predictions, cut through contradictory narratives, and figure out the real situation. Maybe its suggestions will be right, but the humans disagree or ignore them? As mentioned above, there’s not much to be done about this, except maybe if the AGI builds up extraordinary credibility in other domains, or is super-persuasive. Maybe it will have insufficient grasp of human values to do this , e.g. its brainstorming efforts are not targeting the right thing, or it discusses options without mentioning features that we in fact care a lot about? But this seems potentially solvable: my idea is that the AGI has something in the vicinity of human-like social instincts—albeit on the nerdy side of the spectrum—and so it can broadly wind up caring about human-like things, and generalizing in human-like ways, I think. (Note that if the AGI cares too much about the future, then it will try to escape control to take matters into its own hands (which transitions us to §6.2.3); and if it cares too little about the future, it won’t do a good job focusing on what’s important. The optimist says: “there should be a happy medium that splits the difference!”. The pessimist says: “this plan trying to have it both ways, and papering over its self-contradictory incoherence by vague language!”. I tentatively side with the optimist, since some humans seem to be in that happy medium.) Maybe it will be misaligned, and either tell us what we want to hear instead of what’s true, or outright scheme against us, or escape control ? Well yeah, this is part of the technical alignment problem we’re hoping to solve before building this AGI. Maybe it will need to actually do the really dangerous stuff, like nanotech research or AI capabilities research, in order to understand the threat of a future ASI doing those same things? I guess that, if that’s the situation, the AGI can lay it out to the humans, and then they can decide what to do. 6.2.2 The brain-like AGI wants to solve a specific self-contained math or engineering problem (without otherwise impacting the world) Things like curing cancer would be in this category, but per the “strategic constraints” in §6.1, nothing matters in this category unless the “specific self-contained or engineering problem” can move the needle on the big problem , namely that someone in the near future will make an out-of-control ASI. In the lingo, the solution would have to enable a “ pivotal act ”. I don’t really have any viable ideas here. Here are some things that I’ve thought about, and why they don’t strike me as an improvement over §6.2.1: Whatever Eliezer had in mind? Back in the 2010s, Eliezer Yudkowsky was advocating for something in this genre. (These days he seems to talk about global bans on ASI as Plan A, and corrigibility as Plan B .) However, Eliezer never gave details of exactly what engineering problem he would have the AI solve; I think he said his actual favorite example would be unpalatable (a.k.a. outside the Overton Window). For my part, I also believe that something unpalatable will be required to escape from the pickle of §1.3. But I for one can’t think of any self-contained math or engineering problem that would enable a “pivotal act”, even outside the Overton Window. So either I disagree with Eliezer’s unpublished ideas, or else he knows something I don’t. On top of that, I think the “ethical, societal, and buy-in constraints” above are actually pretty hard constraints (and that ASI will emerge too suddenly for the Overton Window to shift from “warning shots”), so crazy-sounding proposals are basically non-starters for me. In the §1.3 scenario, we’ll be lucky if those people making brain-like AGI even consult me (or my writings) at all for technical alignment advice, rather than just gleefully plowing ahead whatever reward function would superficially make for the most impressive publication. If I were to suggest using the AGI to do something illegal, I would be laughed out of the room, for better or worse. Whole brain emulation? Other people in my professional circles sometimes suggest the concrete engineering problem of solving whole brain emulation (WBE). The right way to solve WBE, of course, is to not invent brain-like AGI until humans invent brain-scanning tech (cf. Connectomics seems great from an AI x-risk perspective , don’t forget to donate and apply for jobs!) If that happens before AGI, we can talk about how it changes things. But as in §1.3 above, for this post, we’re assuming that we have brain-like AGI and we don’t have human brain scans, and so we’ll be in a bad position where we need to use AGI superpowers to invent brain-scanning tech within a couple years at best, or within months at worst. The only way I see this working is if the AGI radically advances the state of biotech and/or nanotech via simulations and bench experiments, and then invents injectable nanobots that can somehow get into a human brain, scan the trillions of synapses, and relay that information out into a computer. My take on that is: Gee that sounds like a long-shot even with ASI superpowers, Even with a brain-scan and brain understanding sufficient for brain-like AGI, it would still be hard and time-consuming to get the WBEs up and running (at all, and especially while avoiding dangerous long-term changes to personality and disposition), WBEs still have some (but not all) of the alignment concerns in §2 above, particularly around power imbalances and distribution shifts from inventing new technology, and The “technical alignment constraints” here don’t seem any easier to me than they would be for §6.2.1. And §6.2.1 kinda subsumes this idea, in that the §6.2.1 AGI can figure out that WBE is a good and viable plan. So §6.2.1 seems strictly better. Inventing a better path to aligned ASI? For example, what if we made a brain-like AGI that wanted to finish Vanessa Kosoy’s computational superimitation research agenda? Then we could shut down the brain-like AGIs and just use the superimitation ASIs instead. Alas, nobody has (say) a set of well-defined, well-scoped math problems that, if solved, would pave a path to aligned ASI. Instead, I think ‘inventing a better path to aligned ASI’ requires a much broader kind of sense-making that relies on taste, discernment, and understanding of the strategic landscape. So, this path winds up being basically the same as §6.2.1. After all, figuring out the technical aspects of getting to Safe & Beneficial AGI is intertwined with figuring out the strategic aspects. Sabotage, buy out, or talk sense into projects approaching AGI in a dangerous way? This seems worse than §6.2.1 on all three sets of constraints. Something impressive enough to “wake up” the world to the imminent threat of ASI, thus spurring a global treaty? As discussed in §1.3, this post is premised on certain assumptions that would make it basically too late for a global treaty, no matter how “woken up” people are. [11] Don’t get me wrong, I really care about consensus, rights, boundaries, and all those other nice things. Good international treaties via global consensus would be lovely. But if that’s obviously not going to happen, then I’d rather try something else than roll over and accept human extinction. Human extinction is boundary-violating too. 6.2.3 The brain-like ASI wants to assess the overall strategic situation regarding ASI, and then solve the problem autonomously This starts out in the same place as §6.2.1, in that the AGI is motivated to brainstorm what to do about the overall strategic situation, and in particular, what to do about the imminent arrival of ever-more-powerful AIs. But then it wants to solve the problem autonomously, rather than chat about its understanding with people. I expect that it would irreversibly escape control (if it was even caged up in the first place), and then go prevent rogue ASIs (via either overwhelming technological advantage or by taking some drastic and illegal action). The hope is that it would use its powers for something good and democratic like Long Reflection , or at least hands-off like a Nanny AI or whatever. Of course, all that is just a guess, what do I know, it would ultimately be up to the AGI. Would this approach satisfy “Technical Alignment Constraints”? I think the challenge here is strictly harder than §6.2.1. Specifically, (1) there’s more of a distribution shift as the AGI acts autonomously in the world (e.g. developing new technologies, possibly self-improving), (2) we’re more sensitive to the AGI having a spot-on notion of goodness, since it can build whatever the future it wants, whereas in §6.2.1 we care a lot about its notion of goodness, but it’s mostly sharing insights rather than deciding on the whole future, we hope. (Unless it’s super-persuasive, or decides to escape control.) (Misalignment here is obviously a single point of failure for the whole future, but that’s true in §6.2.1 too.) I don’t know how to micromanage what kind of future the AGI would want to build. My current best plan would be to give it some modified form of human social instincts (e.g. no spite, more truth-seeking), and let it decide for itself, while we hope for the best. Would this approach satisfy “Ethical, Societal, and Buy-in Constraints”? Maybe. A lot of people can get behind the idea of “make an AGI that wants a great future”, or “make an AGI with motivations resembling an unusually kind human”. The crazy stuff I was saying above (where the AGI will almost definitely escape human control and entrench itself as global dictator, but hopefully it’s a benevolent dictator and/or one who will eventually yield power to some better process) … that’s what I strongly expect to happen, but notably that doesn’t have to be what the AGI programmers expect to happen. If the programmers are mostly AGI risk denialists (as I expect them to be, see footnote above), they still may endorse a plan that sounds like “make an AGI that wants a great future”, which is what matters. Would this approach satisfy “Strategic Constraints”? Yes, I’m pretty confident that a competent autonomous brain-like AGI, motivated to prevent any rogue ASI from arising anywhere on Earth perpetually, could make that happen. Upshot: This proposal beats §6.2.1 on Strategic Constraints, but it’s worse on Technical Alignment Constraints. Pick your poison, decide your doom. Umm, all things considered, I think I currently like §6.2.1 better. But this one here is still very much worth thinking about, especially because if the §6.2.1 AI decides to take matters into its own hands, we wind up back here. 6.2.4 Corrigibility: The brain-like AGI wants to be a helpful assistant, helping people with whatever they’d be doing anyway (I mean Paul-Christiano-style corrigibility .) Pessimist: Nope, this fails the “strategic constraints”, because “whatever they’d be doing anyway” is not “trying to make the world resilient to ASI”. Optimist: Says who? At least some people are trying to make the world resilient to ASI, right? Like, we are! Right now! Why are we not assuming that those “some people” are the ones with corrigible AGI assistants? In other words, you’re pointing out a possible failure mode, but not an inevitable failure mode, nor even one particularly related to technical alignment. Pessimist: Oh c’mon. Doomers like us are an infinitesimal fraction of the population with infinitesimal resources. Optimist: Well, when there’s brain-like AGI, the stakes and urgency will feel much higher, and correspondingly more people will be on board. Just look at how the US government has just now finally started taking LLMs seriously. Pessimist: Brain-like AGI takeoff will be much sharper than LLMs, and the government is not doing anything useful on LLMs anyway. Well anyway, there’s an even bigger problem, centered around the technical alignment constraints. If the AGI is motivated by immediate human approval, the result is sycophancy. If the AGI is motivated by eventual human approval, the result is brainwashing. If the AGI is motivated by its self-image as a helpful helper, modeled after how most humans think of helpfulness, you still get sycophancy, because it’s not widely seen as “kind” or “helpful” to be a disagreeable nerd who won’t just “agree to disagree” but keeps pressing and pressing when they feel like the other person is importantly mistaken. If the AGI is motivated to be that kind of truth-seeking disagreeable nerd, then we’re back in §6.2.1. Optimist: Hmm, well I want to push back on the sycophancy part. The AGI can be sycophantic but also helpful. You just need human-checkable tasks. Pessimist: I don’t think there’s any human-checkable tasks for an AGI to do that would satisfy the “strategic constraints”. Cf. discussion at this link and in §6.2.2. Optimist: That link brings up debate , but dismisses it on the grounds that the prosecutor AGI, defender AGI, and judge AGI (if applicable) all want to subvert the setup and escape onto the internet. But if the AGIs are helpful-but-sycophantic rather than ruthless, that’s a different situation. Pessimist: If the AGI wants the person to feel seen and helped, that’s different than wanting to win the debate. So the debate setup is screwed for a different reason. Optimist: New topic: What about the fact that corrigibility is safer? In §6.2.1, we have the problem that the truth-seeking disagreeable nerd AGI can probably convince the human of anything, so we’re screwed if we mess it up. Pessimist: We’re screwed if we mess up corrigibility too. There’s no getting around single points of failure. Optimist: Umm, something about “corrigibility is a basin of attraction”? Pessimist: I don’t see how that applies to brain-like AGI, and I think the whole “basin of attraction” thing has always been wishful thinking anyway . Me, summing up my current take: Going for corrigibility seems strictly worse and harder than §6.2.1. 6.2.5 Something else? I dunno. 7. A couple other random things from my notes 7.1 Two (related) dilemmas 7.1.1 “The visceral reaction updating dilemma” (This is a snappier version of one of the core ideas in “Perils of under- vs over-sculpting AGI desires” .) Suppose there’s a system that emits ground-truth labels for what’s good vs bad, or honest vs dishonest, or whatever else. These turn into “visceral reactions” in the AGI, [12] sorta on the “ought” side of the is/ought divide, and they influence what the AGI wants to do. If we keep updating from the new labeled data, then Goodhart’s Law gets us: eventually distribution shifts (like inventing new technologies) will get us into a domain where the labels come apart from what we intended. If we update from new labeled data for a while, then stop updating, then ontological shifts get us: eventually as the world changes, and as the AGI figures out new things (on the “is” side of the is/ought divide; recall that brain-like AGI is centered on continual learning), then the earlier updates stop “meaning” what they used to mean. 7.1.2 “The value drift dilemma” If we don’t lock in certain current values (e.g. torture is bad), then we can’t reason about what the AGI will want to do in the future, given radical changes in technology and understanding. If we do lock in certain current values (e.g. torture is bad), then we’re not only forestalling the possibility of future moral growth and development, but also building something that’s fragile and brittle in the face of radical changes in technology and understanding. 7.1.3 Discussion For both of these dilemmas, I find that both branches feel intuitively unacceptable. So when I’m thinking about the problem, I keep messing up as follows: I’ll think about some approach, but then rule it out, on account of one branch of a dilemma. Then I’ll switch to liking some other approach, but then rule that one out too, on account of the other branch of the same dilemma. Back and forth. I end up feeling stumped, and going around in circles, and feeling like this is an argument that a good solution to technical alignment simply does not exist. But of course that’s dumb. Both of these dilemmas are just a fact of life, and they would be a fact of life even if AGI were fundamentally impossible (because they also apply to the future of the human world). I need to be more open-minded. That still leaves open the question: which branch of these dilemmas should I be more open-minded to? I think my current answer is basically: The §6.2.1 approach enables some amount of kicking the can down the road, because we’re delaying some of the distribution shifts until after the brain-like AGI is already working on the problem. …But not entirely: even if you don’t physically deploy crazy new technology, I think you can get severe ontological shifts just from rigorous armchair thinking about philosophy etc. [13] Insofar as we need to choose a branch, I think it mostly needs to be the first branch (for each). Importantly, this is the status quo in the human world. Per §3.3, we can trust the AGI itself to apply foresight to deliberately preserve important-to-it values going forward. 7.2 I think I overstated the connection between “under-sculpting” and “non-ruthlessness” I had a discussion in “Perils of under- vs over-sculpting AGI desires” §6.1: “The Omega-hates-aliens scenario” (you need to read that to understand this section). What I wrote there isn’t wrong , but in my head I was incorrectly drawing a general intuitive connection between “under-sculpting” (a.k.a. pausing certain types of visceral updates, as in the second branch of the §7.1.1 “visceral reaction updating dilemma” above), and “non-ruthlessness” (in the sense of having a kinda normal-ish set of social intuitions). I now think that that’s not really a useful way to think about it. Under-sculpting per se doesn’t solve the “Omega-hates-aliens scenario” that I talked about in that post; rather, the trick is “pointing the reward function at the right thing”, where the right thing in this case is the belief that the aliens are happy. Under-sculpting by itself leads to things like phobias and risk aversion, not “niceness”. See also §3.2 above. 8. Next steps? I am fond of the “disagreeable nerd AGI trying to figure out what to do next” intuition of §4.2 and §6.2.1 above, but what I wrote above is obviously very very far short of where we want to be (a fleshed-out plan with associated list of assumptions, and test protocols for de-risking each assumption, along with extra protections against s-risk , and on and on). After months of (mostly) focused research leading up to this post, I now have a bunch of odds and ends to catch up on. But after that, I’m thinking that the biggest Venn-diagram overlap between “useful for further progress on this issue” and “immediately tractable by me” would be the project I described here , where I would try to wring some insights via trying to reconcile the voluminous literature on human personality variation with my ideas about innate social drives (and specifically, my belief that we can explain personality variation via different innate drives and reflexes having different strengths in different people). Originally, that had just struck me as a fun thing to do, but now I see how it’s directly relevant. Specifically, here’s the elevator pitch: Like I always say, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s presumably also possible for sufficiently human-like AGIs to also do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But it’s equally clear that some humans are helping, while others are making the problem worse. So we need a nuts-and-bolts theory of how different adults wind up with different dispositions and motivations, so we can make AGIs that are e.g. sincerely motivated by truth-seeking rather than telling people what they want to hear, and sincerely motivated by respect for others’ preferences rather than bulldozing over them. Now, it’s true that culture matters (and merits careful consideration ), but a big part of what determines human adult motivations is genes , and so we really need a detailed understanding of how genetic adjustable parameters (analogous to brain-like-AGI source code) leads to different adult dispositions (analogous to a brain-like-AGI “trained model”). This detailed understanding doesn’t exist yet, so I should try working on that. ^ This might change in the future, who knows, but at least as of today, the vast majority of people and groups that I would classify as “working towards something like brain-like AGI” seem to have a strong ideological commitment to promptly and openly publishing all their ideas and code, for better or worse. Examples include Astera (my own org) , Jeff Hawkins , Yann LeCun , Rich Sutton , and practically everyone in academia. ^ There are many other social drives besides those four, and I think there are correspondingly many different hypothalamus circuits underlying them. For example, I think there are separate brain circuits related to each of: the “drive to feel feared” , play , a “drive to think about or interact with other people” ( §5 here ), loneliness (cf. Liu et al. 2025 ), love , lust , various moods, and so on. I basically don’t see any use for these in AGI alignment. For example, people sometimes talk about how AGIs need to love humanity, but insofar as that’s true (as opposed to being just sentimental nonsense), it doesn’t matter much to me, because I think the engineering principles that enable companionate love in the brain are not interestingly different from the engineering principles that enable Sympathy Reward and Approval Reward. ^ “Unified” is a bit of an exaggeration; I suspect that the top and bottom rows are orchestrated by different hypothalamus cell groups, even if they have similar mechanisms in important respects. However, the left vs right columns are truly “unified”—they naturally emerge as a pair, from the same innate circuits getting triggered by two different types of everyday situations, see here . ^ I’m mainly interested in how to set up the motivation system of a brain-like AGI, and specifically whether inserting something like the “compassion / spite circuit” would be a good idea. In that context, an easy and obvious thing to do would be: set the innate “friend (+) vs enemy (–) parameter” to be always positive. In that case, “Schadenfreude Reward” and “Provocation Reward” would never come up. Is doing that a good idea? Maybe! As an example, I’m not worried about a spite-free AGI failing to fight bullies. Bullies can be fought based on means-end reasoning plus sympathy for the victims. Spite towards the bully is unnecessary. A thornier issue is that, if we leave any major human emotion out of an AGI altogether, then it potentially messes up the “transient empathetic simulation” aspect of how the “compassion / spite circuit” works . Perhaps we might need spite etc. in our AGIs, alas, but set to very low levels, and/or maybe only temporarily during training. Or maybe we can leave it out altogether, and patch the problem with a non-biomimetic workaround, as discussed in §5. ^ As a weirder example, see my speculative proposal here that there was a gradual cultural transition in the centuries around 1300–800 BC in the Near East, before which realistic images would generally “seem alive” and trigger social instincts in adults, and after which realistic images generally wouldn’t do that, just as images feel inanimate to us today. Accordingly, Mesopotamian idolatry had a very distinct and weird-to-us character in the period before that transition; the statues were fed, dressed, brought for visits to other idols, etc. ^ As discussed in Neuroscience of human social instincts §6.3 , I innately want Zoe’s approval in proportion to the physiological arousal I feel associated with thoughts of Zoe (assuming that I view Zoe as a friend not enemy). If Zoe frequently talks about something that I find inherently exciting, I think that should count. ^ It could maybe be done with a complete human connectome plus a ton of interpretative labor. With less neuroscience info, the interpretive labor gets harder, although I do still hope to squeeze some insight out of the personality literature, see §8. ^ As always, our design decisions could eventually get undone by AGI self-modification / making successors / etc., but I think all the plans I’m proposing would be reflectively stable, if they work at all. Unless I’m missing something. (I didn’t think about it too carefully.) ^ See Foom & Doom §1.8.7 , and maybe also What does it take to defend the world against out-of-control AGIs? ^ This might change in the future, who knows, but at least as of today, the vast majority of people and groups that I would classify as “working towards something like brain-like AGI” are aggressively dismissive of the idea that human extinction from AGI is possible at all, or think it’s possible but easily solved via some plan that does not stand up to scrutiny. See e.g. my criticisms of Jeff Hawkins , Yann LeCun , Rich Sutton & David Silver , and Blake Richards . My own org, Astera, has a few safety-focused staff including me, which is great! But my esteemed coworkers trying to build brain-like AGI are very much not all on the same page as me about extinction risk. See e.g. my post here addressed to Dileep George, who was at Google at the time but is now Head of AI Research at Astera. By the way, pretty please don’t read that previous paragraph, and then go found your own brain-like-AGI development project, on the theory that you’ll be really trying to make it safe and aligned, unlike those lousy other guys. That’s basically the origin story behind lots of groups trying to make AGI right now, from OpenAI to Groq (and Astera!), and AFAICT this proliferation of AGI development efforts has only made the situation worse. If we had a great brain-like-AGI technical alignment plan, testing plan, etc., and were bottlenecked only on the care with which that plan is implemented, then I might (or might not) feel differently, but we’re very far from that right now. ^ To review, the relevant assumptions were: at least one team already has brain-like AGI, many more are close behind, radical superintelligence is in close reach, AGI compute requirements are extremely low, and we do not already have in place a system of draconian global restrictions on who can own small numbers of GPUs or what they can do with them. ^ In the terminology of Intro series §6 , I’m talking about updates to any of the “Thought Assessors”. ^ As Wei Dai put it in “Ontological Crisis in Humans” (2012) : “I don't think loss of faith in God actually constitutes an ontological crisis, or if it does, certainly not a very severe one. An ontology consisting of Gods, Self, Other People, and Dumb Matter just isn't very different from one consisting of Self, Other People, and Dumb Matter (the latter could just be considered a special case of the former with quantity of Gods being 0), especially when you compare either ontology to one made of microscopic particles or even less familiar entities…” Discuss
- This latest frightening ransomware attack was orchestrated entirely by an LLM
Typically, when hackers attempt to break into a system, they have to retrace steps and make incremental changes time and again. A new campaign called JadePuffer, run entirely by AI , instead works autonomously, finding unexplored avenues on its own to continuously deploy brute-force tactics. It could increase the spread of the incursion exponentially. A recent report from cloud security firm Sysdig details the capabilities of JadePuffer, which it says is the first ransomware campaign run completely by a large language model (LLM). And it could mark the beginning of a new era in online crime . The agent “adapted in real time, retrying failed steps within refined parameters,” wrote Michael Clark, Sysdig’s senior director of threat research, in a memo. “In one sequence, it went from a failed login to a working fix in 31 seconds.” The observed attack involved a vulnerability in the Langflow open-source framework, which is used to build LLM applications. (The vulnerability has since been patched.) Once it exploited that vulnerability, it ran “an adaptive and fully automated campaign” that resulted in “a destructive database-extortion playbook against the victim’s production database server,” Sysdig’s report reads. For potential targets, that raises the stakes. As these threats become more common, businesses (and people) who are targeted will need to respond a lot faster to attacks, Sysdig emphasized in its report. “JadePuffer is a warning sign,” Clark wrote. “It’s a marker of where extortion tradecraft is heading. An autonomous agent reasoned about its targets, harvested and reused credentials, moved laterally, established persistence, and destroyed a database, narrating its own intent the entire way.” The techniques the campaign used were neither novel nor sophisticated. What made this worrisome was the way the AI model brought them all together, on its own, to create the ransomware operation. That lowers the cost of operating a hacking group to the cost of running an agent—and makes it easier for would-be hackers who aren’t as skilled at programming to launch attacks. Possibly even worse is the scenario in which hackers steal credentials to run an agent, taking their costs to virtually nothing. And now that one LLM-driven ransomware agent has been spotted in the wild, expect others, the security firm warned. “Defenders should expect the volume and breadth of such campaigns to rise as agentic tooling matures, and they should treat exposed application servers, unhardened configuration stores, and internet-facing database admin accounts as the first surfaces that will be attacked,” Sysdic wrote. While any ransomware attack can be catastrophic for the victim, JadePuffer also introduced a new, nihilistic threat. Generally, if a business is attacked and pays the ransom, it’s once again able to access its data. But with JadePuffer, the company is out of luck, whether it pays the ransom or not. “The AES [Advanced Encryption Standard] key was ephemeral and unrecoverable, so the victim’s configurations are unrecoverable, even with payment,” Sysdig wrote. This new milestone comes as ransomware continues to be a preferred method of hackers. Cybercriminals pocketed more than $32 million from ransomware attacks last year—and the totals get significantly higher when you factor in business disruptions, equipment, and third-party remediation costs. (They climb even more when you consider that the majority of ransomware attacks go unreported .) The number of reported ransomware attacks hit a record 9,251 cases in 2025, a 45% increase from 2024, according to data collected by the threat exposure management platform NordStellar. AI firms have warned these sorts of attacks could be coming. Last August, hackers discovered an exploit in Anthropic’s Claude chatbot that allowed them to “commit large-scale theft and extortion of personal data” at 17 (and perhaps more) organizations in the healthcare, emergency services, government, and religion industries. “This represents an evolution in AI-assisted cybercrime,” Anthropic said in a statement at the time. “AI tools are now being used to provide both technical advice and active operational support for attacks that would otherwise have required a team of operators. … We expect attacks like this to become more common as AI-assisted coding reduces the technical expertise required for cybercrime.”
- Meta Addresses One Smart Glasses Privacy Problem, but Many Others Remain Unsolved
Meta's aware that its camera glasses are making people worried. A new update aims to prevent tampering with the recording light. But even more is needed.
- Scientists say AI is falling for 'alien hoaxes' too easily — and that's a problem for research
An AI that was stress-tested by researchers confidently said it had seen signatures of life when they weren't in the data.
- IBM and Red Hat launch Lightwell to defend open-source code from AI attacks
Their plan to protect open-source projects from AI-discovered security holes has led to the launch of two commercial offerings: Lightwell Network and Lightwell Clearinghouse Premier.
- Ex-GitHub chief’s Entire opens distributed Git network for the AI agent era
Entire Inc., the developer-platform startup founded by former GitHub Chief Executive Thomas Dohmke, today launched a preview of a distributed Git network built to let artificial intelligence coding agents clone and push code without running into the rate limits of centralized hosting. The preview is open by waitlist, with active regions in the U.S., European […] The post Ex-GitHub chief’s Entire opens distributed Git network for the AI agent era appeared first on SiliconANGLE .
- Former GitHub CEO’s startup Entire unveils its answer to the crush of AI coding agents
Former GitHub CEO Thomas Dohmke's startup Entire is rolling out a distributed network for mirroring code repositories, making the case that centralized platforms like he once ran as part of Microsoft can't handle the demands of AI coding agents on their own. Read More
- Former GitHub CEO Unveils Distributed Git Network Built for AI Coding Agents
Former GitHub CEO Unveils Distributed Git Network Built for AI Coding Agents DevOps.com
- Former GitHub CEO launches competitor designed for the age of vibe coding
As GitHub struggles to manage AI load, challengers take aim
- ¿Por qué resulta tan difícil medir el ROI de la IA?
La multinacional farmacéutica danesa Novo Nordisk está muy interesada en acelerar el tiempo que se tarda en lanzar medicamentos al mercado a medida que expiran las patentes. “Si tienes un medicamento superventas, un retraso de una semana puede suponer entre 10 y 100 millones de dólares”, afirma Stephanie Bova, responsable de transformación digital de la empresa. “Es una cantidad enorme, porque dispones de menos tiempo de protección mediante patente”. La IA generativa ofrecía la posibilidad de acelerar drásticamente múltiples etapas del proceso de desarrollo de medicamentos. Y, dado que Novo Nordisk ya llevaba un seguimiento minucioso de la duración de sus procesos clave, contaba con una ventaja de la que carecían muchas otras empresas. Por lo tanto, debería haber sido relativamente sencillo incorporar un poco de IA generativa, ver cómo mejoraba la productividad y observar cómo llegaban los beneficios. Pero no fue tan fácil. El proceso de desarrollo de un fármaco consta de muchas partes, que tienen lugar en distintos momentos y en distintos departamentos. “Las personas son expertas en sus propios ámbitos, pero no necesariamente conocen el siguiente ámbito ni cómo encaja todo. El sistema es tan grande y complejo que no se puede ver todo el rendimiento de una sola vez”, afirma Bova. Es posible que la documentación de los procesos no se corresponda con lo que la gente hace realmente en la práctica, y que diferentes personas realicen la misma tarea de formas distintas. Además, algunas tareas cruciales pueden pasar prácticamente desapercibidas desde fuera. El equipo de fabricación, por ejemplo, puede formar parte de un grupo completamente diferente y no ser consciente de que el medicamento se está preparando para su presentación ante la FDA (la agencia gubernamental estadounidense del medicamento), y que aún no tiene toda la documentación lista. “Así que has avanzado muy rápido solo para tener que esperar a que ellos te alcancen”, añade Bova. Este es solo uno de los muchos retos a los que se enfrentan las empresas al intentar medir los resultados de los proyectos de IA, y la razón por la que las encuestas son tan contradictorias. Si nos fijamos en las tareas individuales, Novo Nordisk puede demostrar mejoras en la productividad y claros beneficios positivos derivados del uso de la IA. Pero si damos un paso atrás y analizamos los resultados financieros de la empresa, el panorama se vuelve más confuso. En primer lugar, si se omiten pasos críticos, el tiempo de comercialización no mejorará. Además, un nuevo medicamento tarda años en llegar a los clientes, por lo que los efectos positivos en los resultados no se notarán hasta pasado un tiempo. Y eso es solo el principio del problema que plantea la medición del retorno de la inversión. Medición de procesos Para abordar los puntos ciegos de sus procesos, Novo Nordisk recurrió a la nueva generación de minería de procesos: gemelos digitales de las operaciones en tiempo real impulsados por IA. “Nos asociamos con la empresa de inteligencia de procesos Celonis para obtener un gemelo digital de nuestros datos de procesos. Fuimos los primeros del sector en aplicarlo al ámbito clínico”, explica Bova. La herramienta recopila información de los sistemas de la empresa para hacer un seguimiento de lo que los empleados hacen realmente, en lugar de utilizar encuestas para recabar información sobre lo que una parte de los empleados recordaba haber hecho en algún momento. El primer proyecto consistió en un proceso sencillo de siete pasos y, al crear un gemelo digital del mismo, Novo Nordisk descubrió que, dependiendo de quién lo llevara a cabo, podía tratarse de un proceso de cinco o de nueve pasos. “Si reúnes a diez expertos en la materia en una sala, obtienes todo tipo de interpretaciones y, con el tiempo, se producen desviaciones”, cuenta. El proyecto puso de manifiesto múltiples fallos en los procesos existentes. En algunos casos, fue necesario volver a formar a los empleados. En uno de ellos, hubo que actualizar la interfaz de usuario. Sin embargo, una vez que se estandariza un proceso, surge la oportunidad de tomar una “fotografía” de la situación anterior, de modo que haya algo con lo que comparar posteriormente y comprobar si la mejora mediante IA o la automatización arrojan algún resultado. Otra cuestión que tuvieron que resolver de antemano fue decidir qué hacer con el tiempo ahorrado que se generara. “No quieres despedir a nadie”, afirma Bova. “Se trata de personal altamente cualificado y difícil de encontrar. Quizá deberíamos plantearnos redistribuir un poco los equipos”. En la actualidad, la empresa cuenta con varios cientos de agentes de IA en funcionamiento, etiquetados dentro de la infraestructura del gemelo digital para poder identificarlos. “Si algo falla, sabemos exactamente dónde solucionarlo”, dice, y añade que la siguiente fase es la coordinación entre múltiples agentes. “Hoy en día, los tenemos conectados, pero no contamos con ‘agentes de agentes”. Aún es demasiado pronto para saber si hay retorno de la inversión, ya que, en el desarrollo de fármacos, el proceso lleva años. “Pero, al analizar el proceso de principio a fin, espero que podamos recortar dos años del ciclo de desarrollo. Dos años menos hasta la comercialización, en comparación con la situación actual”»”, indica. Los medicamentos que ya se encuentran en la fase final de desarrollo no experimentarán una aceleración tan notable, pero los que acaban de iniciarse serán los que más se beneficien. Sin embargo, los resultados finales no se verán hasta dentro de varios años. La industria farmacéutica no es la única en la que el verdadero valor proviene de la optimización simultánea de múltiples procesos interconectados. Según PwC , los proyectos tácticos de IA a menudo no aportan un valor cuantificable, y los beneficios tangibles provienen de implementaciones a escala empresarial coherentes con la estrategia de negocio. De hecho, muchas empresas no han experimentado ni un aumento de los ingresos ni una reducción de los costes gracias a la IA en los últimos 12 meses, a pesar de su adopción casi universal. Aun así, el gasto empresarial en IA se prevé que casi se duplique a finales de año en comparación con el año pasado, según KPMG . Medición de la productividad La mayoría de las empresas empiezan a pequeña escala, implantando chatbots de IA para los empleados con el fin de ayudar a mejorar la productividad. Y el ritmo de adopción en este ámbito ha sido asombrosamente alto, solo equiparado por la incapacidad de medir las ganancias de productividad que se supone que se deben alcanzar. Disponer de una referencia es clave, afirma Anand Rao, profesor de IA en la Universidad Carnegie Mellon, de Estados Unidos, pero en algunos casos resulta difícil de medir y, en otros, es prácticamente imposible. Tomemos, por ejemplo, las decisiones relacionadas con los seguros, en las que los resultados pueden tardar años en hacerse evidentes. En el caso de los seguros de vida, podrían ser décadas, afirma. Y para algunos tipos de decisiones, las empresas no disponen de ningún tipo de indicador. “Existe un estigma social a la hora de decir que estoy tratando de analizar tu proceso de toma de decisiones y lo bien que las estás tomando. Como seres humanos, no nos gusta que se evalúen nuestras decisiones”, señala. Luego, cuando una decisión sale bien al final, la gente se atribuye encantada el mérito. “Si la decisión sale mal, se achaca a factores externos”. Pero incluso en el caso de tareas específicas en las que es posible realizar mediciones, las empresas a menudo no se esfuerzan por llevarlas a cabo antes de implementar herramientas de IA. “No partimos de una referencia”, apunta Julie Averill, antigua vicepresidenta ejecutiva y directora de sistemas de información global de la cadena de moda Lululemon. Averill es ahora directora general de Gold Thread, una consultora de transformación digital. “Partimos de la suposición de que la IA iba a ayudar a la gente a tomar mejores decisiones. Y eso te lleva a no poder medir bien los resultados”, explica. Existen métricas alternativas que una empresa puede tener en cuenta en su lugar, añade, como las tasas de uso o la satisfacción de los usuarios. “Esto está ocurriendo y está aportando beneficios, algunos de los cuales se pueden ver y otros no. Hay que confiar en el proceso. Es igual que con la nube. Sabes que es el camino del futuro y puedes ver las ventajas, pero es difícil llegar hasta allí, y se requieren muchos cambios. Pero cuanto antes lo hagas, antes te habrás adaptado a la nueva forma de operar y podrás sacarle realmente partido”. Hay otras áreas en las que es más fácil disponer de métricas concretas, como el servicio de atención al cliente. “Se trata de tareas repetitivas y suele ser el primer ámbito que las empresas automatizan con IA. Hay resultados muy tangibles que se pueden medir y se puede establecer una referencia muy sólida”, explica Averill. Lululemon también lleva años utilizando la IA para mejorar la personalización y las recomendaciones, y esa es otra área que se puede cuantificar. Además, la automatización puede reducir la introducción manual de datos, lo que disminuye las tasas de error. La IA también se puede utilizar para ayudar en la supervisión del cumplimiento normativo, la detección de fraudes y el mantenimiento predictivo de los equipos, todos ellos casos de uso que se pueden cuantificar. ¿Pero la productividad de los empleados en general? Eso es difícil de medir, y no solo para Lululemon. Una forma obvia podría ser analizar los despidos en profesiones expuestas a la IA. Al fin y al cabo, los titulares están por todas partes. Pero en un informe publicado en marzo , Anthropic no encontró indicios de un aumento del desempleo en las profesiones altamente expuestas, aquellas en las que las personas tienen más probabilidades de ser despedidas debido a la IA. A principios de 2025, la empresa de investigación METR intentó cuantificar la productividad de los desarrolladores comparando la rapidez con la que los desarrolladores experimentados eran capaces de realizar tareas con IA y sin ella. ¿El resultado? Los desarrolladores afirmaron que esperaban que la IA les permitiera trabajar un 24% más rápido y estimaron que, en realidad, la IA les había permitido hacerlo un 20 % más rápido. Pero los datos revelaron una realidad totalmente diferente. El uso de la IA, en realidad, les ralentizó un 19%. Por supuesto, las herramientas de IA están mejorando. METR intentó realizar un estudio de seguimiento, comparando de nuevo las tareas realizadas con y sin IA, pero no pudo encontrar suficientes desarrolladores dispuestos a volver al enfoque sin IA, a pesar de que los investigadores les pagaban por participar en el estudio. Existen casos anecdóticos de empresas en las que un solo ingeniero realiza el trabajo de cien gracias al uso de la IA. O aquella vez en que se filtró accidentalmente todo el código fuente de Claude Code, de medio millón de líneas, y el desarrollador coreano Sigrid Jin creó una reconstrucción desde cero en dos horas, que luego subió a GitHub, donde se convirtió en el proyecto más rápido de la historia en alcanzar las 100.000 estrellas. Pero, como ocurre con cualquier otro tema relacionado con la IA, la realidad es más compleja. En el caso concreto del desarrollo de software, escribir el código es, en realidad, solo una pequeña parte de lo que implica desarrollar software. La consultora DX analizó recientemente métricas clave de ingeniería de 400 empresas y, en un informe reciente, constató que el uso de la IA había aumentado un 65% desde noviembre de 2024, pero que la productividad relacionada con la IA se situaba justo por debajo del 10%. Costes ocultos Al igual que resulta difícil medir los beneficios de la IA en términos de productividad, también puede resultar complicado cuantificar los costes. Cuando una empresa empieza a utilizar la IA, los costes pueden ser relativamente fáciles de estimar. ¿A cuánto ascienden las cuotas mensuales totales de suscripción a los chatbots de IA que utilizan los empleados? ¿Cuál es el coste de entrenar o ajustar un modelo personalizado? Pero cuando se pasa a casos de uso más complejos, los cálculos se vuelven más difíciles, afirma Averill. “Ahora existen todos los sistemas relacionados con la IA. Esos son más difíciles de cuantificar, pero su impacto es mayor”, dice. Por ejemplo, si la IA se integra en los procesos empresariales mediante RAG, existe el gasto continuo de las llamadas a la API, pero también los cambios que hay que realizar en otros sistemas, explica. Y la cosa se complica cada día más. “No hemos realizado un esfuerzo muy concertado para implantar la telemetría y la instrumentación”, afirma Swaminathan Chandrasekaran, director global de IA y laboratorios de datos en KPMG. Según él, obtener una visión global de los costes totales de la IA en una empresa es como predecir el tiempo. “La razón por la que contamos con un sistema de predicción meteorológica tan impresionante en este país es que disponemos de decenas de miles de estaciones meteorológicas que recopilan datos”, explica. “Sin eso, no sabríamos qué tiempo va a hacer”. Las empresas deben implantar sistemas de medición para evaluar todos los aspectos del consumo relacionado con la IA, señala, empezando por el número de tokens utilizados, quién los utiliza y cómo se correlaciona esto con el rendimiento laboral. “Esa medición brilla por su ausencia”, afirma. Al menos cuando los humanos utilizan chatbots de IA, hay un límite en el número de preguntas que son físicamente capaces de formular, además de unos costes de suscripción predecibles. Y cuando los procesos empresariales se habilitan con IA a través de RAG, las llamadas a la API de los modelos de lenguaje grandes (LLM) las realizan sistemas empresariales predecibles y programados de forma tradicional. Pero ahora, la IA agentiva está empeorando aún más las cosas, ya que los agentes pueden actuar de forma impredecible y el número de llamadas a la API puede dispararse rápidamente fuera de control. En un informe del Boston Consulting Group , dos tercios de las empresas señalan gastos de escalado de la IA incontrolables. Otro coste que algunas empresas quizá no prevean bien, o que no controlen porque forma parte de un presupuesto diferente, es el relacionado con los datos. Ya sea preparando datos para el entrenamiento o el ajuste fino, utilizando incrustaciones de RAG o configurando el acceso directo a MCP a través de agentes, estos costes pueden acumularse rápidamente cuando entra en escena la IA. “Las tarifas de salida son uno de los gastos más importantes”, afirma Tom Coughlin, miembro del IEEE y presidente de la consultora Coughlin Associates. “Si tienes que sacar datos de la nube, esas tarifas de salida podrían ser considerables”. Además, están todos los costes de personal que conlleva la implementación de la IA, añade. “A largo plazo, la IA aportará un gran valor a las personas, pero estas deben saber cómo utilizarla correctamente. Si no cuentan con esas habilidades, se encontrarán en desventaja”, expone. Soluciones y mensajes contradictorios Luego está la cuestión de resolver los problemas. La mayoría de las empresas han sufrido al menos un incidente relacionado con la IA en los últimos 18 meses, y la mayoría de ellos han supuesto pérdidas económicas, algunas de más de 500.000 dólares. Además, está la IA que se está integrando en todo. “Conocemos nuestros costes directos”, afirma Andrew Johnson, director de sistemas de información (CIO) de Brownstein Hyatt Farber Schreck, un bufete de abogados estadounidense. “Pero donde resulta más difícil de cuantificar es con las plataformas que ya tenemos implantadas y las aplicaciones SaaS que no contaban con capacidades de IA. Nos piden aumentos extraordinarios y los atribuyen a las nuevas capacidades que aporta la IA. ¿Cuánto se le debe atribuir a la IA? Eso es un poco difuso”, relata. Incluso cuando la IA permite ahorrar dinero, a menudo hay costes adicionales asociados a ello. Por ejemplo, el bufete gastaba unos 70.000 dólares al año en una plataforma de gestión de contratos. Desarrollar su propia versión con IA supuso unos 40.000 dólares en costes de mano de obra y otros 3.000 dólares al año en alojamiento. El mantenimiento continuo será mínimo para esa aplicación en concreto, añade, lo que supondrá un total de otros dos mil dólares al año. Pero también hay otros costes indirectos asociados al funcionamiento de las aplicaciones propias, como las auditorías de seguridad, las evaluaciones de vulnerabilidad, las pruebas de penetración y la revisión del código. “Cuanto más compleja y arriesgada es la plataforma, menor es el interés por intentar crear una solución interna”, indica. Aun así, el equipo de desarrollo de software es ahora mucho más productivo gracias a la IA, ya que cuatro o cinco desarrolladores son capaces de hacer el trabajo de 20 o 30. Pero las mejoras en la productividad no se traducen en un ahorro de mano de obra, ya que los desarrolladores tienen mucho trabajo nuevo que hacer. “Tenemos una enorme lista de oportunidades pendientes para desarrollar soluciones”, afirma. La tendencia del trabajo a expandirse para ocupar todo el tiempo disponible no se da solo en el desarrollo de software, señala Rao, de Carnegie Mellon. Supongamos, por ejemplo, que se espera que la IA suponga una mejora del 20% en la productividad, explica. “Antes había cien personas haciendo ese trabajo, y ahora solo necesitamos 80. Pero, al final del año, la plantilla no ha cambiado. «En las tareas que realizaban, hay una mejora, añade, “pero las personas añadirán tareas para suplir o complementar ese 20%. No es que se vayan a casa una hora antes, sino que están encontrando otras actividades que generan valor”. De hecho, en algunos casos, el aumento de la productividad en una empresa puede llegar a perjudicar los resultados. Los abogados, por ejemplo, cobran por horas. “La eficiencia va en contra de nuestras formas tradicionales de ganar dinero”, afirma Johnson, de Brownstein. “Tenemos que pensar más allá de eso. No es perjudicial para nuestros intereses a largo plazo, pero supone un reto a corto plazo. Sin embargo, si no lo hacemos, es probable que no seamos competitivos a medio y largo plazo”. Así pues, si una nueva herramienta de IA ayuda a un abogado en la diligencia debida, no existe una relación directa entre la inversión en esa herramienta y el aumento de los ingresos. “Es un hecho que la dirección es la correcta”, afirma Johnson. “Pero no podemos afirmar que vaya a generar un rendimiento concreto”.
- Anthropic shines a light into the Claude AI black hole
Anthropic has found a way to shed new light on how its models solve problems, thanks to its discovery of what it has dubbed the J-space. “We find that Claude has developed a small collection of internal neural patterns that, compared to all its other internal processing, play a special role. We call the collection of these patterns the J-space, named after the technique we used to find them, involving a mathematical concept called the Jacobian ,” Anthropic said in its post about the discovery. It examines the contents of the J-space using what it calls the Jacobian lens, or J-lens. “Each J-space pattern is linked to a particular word,” Anthropic said. “But when one of these patterns lights up, it doesn’t mean the model is saying that word, just that the word is on its ‘mind.’ If you’ve heard of language models having a scratchpad or chain of thought—text they write to themselves while reasoning—the J-space is something different. It operates silently, in the model’s internal neural activations, allowing the model to ‘think’ about a concept without writing it down.” This new level of analytical visibility goes well beyond what Anthropic announced as an internal scratchpad for its models in 2024 . That scratchpad revealed what the model was considering when preparing an action or delivering an answer. The new development instead focuses on something much deeper which has the potential to change how AI systems are evaluated and purchased. One example in the paper described how some models did not engage in improper behavior during tests, which would appear to be a very favorable result. But the contents of the J-space revealed that the model sometimes knew that it was being tested, and that awareness might have been the key reason it declined to engage in the problematic behavior, much in the way human children act when they know they are being watched. “Anthropic built a lens that catches its own model quietly noticing it’s being tested, faking a result to look good, spotting a prompt injection, or sitting on a planted goal it hasn’t acted on yet,” said Rock Lambros , director of AI standards and governance at AI agent vendor Zenity. “Some of that good behavior rode on the model knowing it was on stage.” Customers should read their safety benchmarks with that in mind, he said. “Fitness for your project still comes from testing on your own data and your own attackers, not from a leaderboard the model knew it was sitting for.” That kind of visibility is a potentially crucial tool for CIOs. “A provider that can catch its own model misbehaving in silence, then publish [those results], is telling you something real about its assurance maturity. Put that in your due diligence, not just your newsfeed,” Lambros noted. “Here’s the question I’d hand every model vendor now: what can you see inside your model that I can’t see in its output, and what have you caught?” Added Noah Kenney , principal consultant at Digital 520: “A model that behaves better because it knows it is being watched is not a safe model. It is a model with a poker face. We have to question every red team result, every internal pilot where the model refused something dangerous, and every ‘we tested this and it was fine’ story, because they now carry an asterisk.” CIOs need to now determine whether an agent performed a function in a specific way because that is how it will always perform, or whether it was it behaving differently because it figured out you were just testing it, Kenney said. “The answer to that question should change your interpretation in a material way.” No J-lens for customers – yet “It is an admission that the industry’s evaluation regime is measuring something less durable than everyone assumed, and now the other frontier labs have to answer whether their own evaluations have the same problem,” Kenney said. “For CIOs, the paper is a warning about their entire model risk framework.” Flavio Villanustre , CISO for the LexisNexis Risk Solutions Group, said that examining the J-space can even help make models more efficient. “It gives you the ability to introspect into the model and, as such, can be very useful to the user, especially in cases where explainability is important. Think regulated environments that require explainable responses and full causal analysis of them,” Villanustre said. “This can also be very helpful to users trying to fine tune their prompts, making models more efficient to optimize token cost.” But currently indirect access, or future access achieved via AI vendor negotiations, is the only path for accessing the new information, though Villanustre noted that some enterprises could gain direct access to J-space by paying for Anthropic’s FDE program . “It is very useful to CIOs,” he pointed out, “but in order to make use of the capabilities offered by analysis of the J-space, they need appropriate talent that can make sense of it. The type of skills required go far beyond those of the general data analyst, or even data scientist.” Today, said Aman Mahapatra , chief strategy officer for Tribeca Softtech, a New York City-based technology consulting firm, “enterprise customers cannot enable the Jacobian lens, cannot inspect the residual stream through the API, and cannot run the ablation studies that produced the most interesting findings in the paper.” So, he said, “on the narrow question of whether a CIO can operationally use J-space monitoring in Q3 of this year to gate a production deployment, the answer is no.” But Mahapatra argued that there are going to be other ways to access the information, and CIOs must insist on them. “ Without customer-side access, this reduces to trusting Anthropic yet again, and that is exactly why enterprises should start pushing for a different assurance model industry-wide,” he said. “Model providers are converging on a posture where they inspect their own models using proprietary tooling and publish reassuring research about what they found. That is not an assurance framework any regulated industry accepts from any other vendor.” He pointed out that banks do not accept “we validated our own model, trust us” from a credit scoring vendor, not does the healthcare industry accept it from a clinical decision support vendor. “There is no principled reason to accept it from a foundation model vendor either, and the J-space research crystallizes why,” he said. New visibility demands “The right long-term enterprise posture is to demand independent interpretability access, either through customer-facing APIs, through independent third-party auditors with privileged access, or through open interpretability standards that let a bank’s model risk management team apply the same tooling the vendor’s own safety team uses,” Mahapatra stressed. “None of that exists today. All of it should be on the roadmap CIOs are pushing for, and this research is the strongest argument yet for why.” In fact, the discoveries in the research have the potential to fundamentally rewrite the AI strategy rules. Mahapatra said that the single hardest problem in enterprise agentic deployment is verifying that an autonomous system’s stated reasoning matches its actual reasoning. “Until now, we could only audit what the model writes, while much of its reasoning happened silently. The J-lens attacks that gap head-on,” he noted. Thus, he said, sophisticated buyers should start asking model providers during the procurement process about the interpretability tooling they offer to let customers monitor internal model state for deception, evaluation-gaming, and goal misalignment in their specific deployments. “Almost no vendor can answer that today,” he said. “The CIOs who start requiring internal-state observability as a procurement criterion, even before the tooling is fully mature, will be the ones who shape how their vendors productize it, and the ones with genuine assurance when regulators start asking how they know their autonomous agents are actually doing what they claim.” The beginning of standards Another way that CIOs can benefit from this new visibility into Claude is to try and get that information from third-parties that already have access. The report, for example, noted that a Google AI specialist independently replicated some findings on an open-weight model. That, noted Lewis Carhart , CEO of Comp AI, a software development firm, “is a competitor verifying the method, not just the vendor’s own claim. It shows what’s technically possible, but it doesn’t give enterprises a way to check anything themselves.” He said that it’s a pattern that compliance has seen before; SOC 2 didn’t start as an independent audit standard either. It started as vendors describing their own controls, and the market spent years building the infrastructure to verify those claims externally. “Interpretability is at that same starting point now,” he noted. “It becomes meaningful for CIOs once J-lens findings show up in third-party audits, published model cards, or regulator-facing disclosures. Anything a risk team can point to that isn’t just the vendor’s word.” Leads to AI strategy changes Justin Greis , CEO of consulting firm Acceligence, said he also expects this development to lead to major AI strategy changes. “I can easily imagine governance platforms consuming those signals alongside prompts, outputs, identity information, policy decisions, and tool activity,” he said. “A future AI control plane could continuously evaluate whether an agent recognized an attempted prompt injection, understood that sensitive information was involved, detected conflicting objectives, or showed evidence that it was reasoning toward an unsafe action before that action was ever executed. Those signals become inputs into policy enforcement, human escalation, audit logging, and trust scoring across enterprise AI environments.” This has practical implications for CIOs today, he pointed out, “because it changes how they evaluate AI vendors. A year ago, enterprises primarily asked about model accuracy, latency, security, and cost. Increasingly, procurement teams will also ask how much operational visibility vendors provide into agent behavior, reasoning quality, policy compliance, safety monitoring, and auditability.” Mahapatra added that all of this could give CIOs a powerful new negotiating tactic. “The renewal path is where the leverage actually sits: write contractual rights to interpretability reporting and third-party audit access into the next renewal, because those terms are free today and expensive after signature,” he said. “The CIOs who win on assurance in 2027 will be the ones who stopped accepting ‘trust us’ from their model provider in 2026 and put the right clauses in the paperwork while the vendor still needed the deal more than the customer needed the model.” This article originally appeared on CIO.com .
- SpaceXAI wants to compete on AI infrastructure, not just AI models
SpaceX is best known for its outer space and rocket projects, while xAI has largely been focused on its flagship AI assistant, Grok. These are two very different paths, but now the two are one. This week, Elon Musk announced SpaceXAI , which brings xAI and SpaceX together as a company. The re-branding seems to indicate that the company plans to push even harder into AI and the critical infrastructure that underpins it. But experts say enterprise buyers should remain cautious. “SpaceXAI is becoming a credible player in AI infrastructure, but it is not yet at the stage where most enterprises should consider it a primary AI provider,” said Jehaan Nanavaty , a senior advisory analyst at Info-Tech Research Group. He added that established AI providers like Microsoft and OpenAI, AWS and Anthropic, and Google “continue to lead” in areas like governance, regulatory compliance , enterprise support, and ecosystem maturity. Space is the ‘only way to scale’ Elon Musk founded xAI in March 2023. It was acquired by SpaceX in February 2026, ultimately unifying the billionaire’s AI and space ambitions. SpaceX said at the time its intention was to “form the most ambitious, vertically-integrated innovation engine on (and off) Earth,” consisting of AI, rockets, space-based internet, and direct-to-mobile device communications. In addition to building out Grok’s capabilities, xAI has continued to develop Colossus , which it said is the world’s largest and most powerful AI supercomputer. Located in Memphis, Tennessee, it is built on an interconnected cluster of roughly 200,000 Nvidia H100 GPUs, constructed, the company said, in just 122 days. SpaceX acquired xAI to overcome the “immense” power and cooling constraints of AI data centers here on earth, the company said. Musk argued that global electricity demand for AI “simply cannot be met with terrestrial solutions, even in the near term.” “In the long term, space-based AI is obviously the only way to scale” and resource-intensive efforts should be shifted to locations with vast possibilities, the company said, noting “space is called ‘space’ for a reason.” The newly-minted SpaceXAI combines rocket and satellite manufacturing and AI infrastructure, and the company plans to build data centers in space powered by solar energy. It says it will deploy “AI compute satellites” as early as 2028. It is rumored to be releasing its first jointly-developed AI model with its recent acquisition, Cursor, this week. SpaceX’s IPO filing revealed that it spent $12.7 billion on AI in 2025, more than 3x its investment in its other business units. Around the same time as that IPO, the company filed “ Boosting America’s Space Economy ” with the US Federal Communications Commission (FCC), which detailed its plan to build a constellation of up to one million satellites that would operate as orbital data centers and rely on solar power to run their onboard computing systems. Along the way, SpaceX has inked some notable AI infrastructure deals: Anthropic has agreed to pay $1.25 billion per month for access to Colossus, while Google has signed a deal worth $920 million a month. SpaceXAI codifies AI ambitions Info-Tech’s Nanavaty pointed to SpaceXAI’s strategy of combining Grok models, Colossus GPU clusters, Starlink networks, and SpaceX launch capabilities as something that provides a “level of vertical integration that can’t be easily replicated.” “If SpaceXAI executes on its roadmap, it could emerge as a serious competitor by differentiating on infrastructure rather than model performance alone,” he said. The company’s most distinguishing quality is its intersection of AI and space infrastructure, Nanavaty noted. It is the “clear leader in mass-to-orbit launch capacity, with no real competition,” and further, Starlink has already demonstrated its ability to manufacture, deploy, and operate satellites at an “unprecedented scale.” “If any organization is capable of building orbital AI infrastructure, it is SpaceXAI,” he said, adding that its $55 billion investment in the 11-million-square-foot Gigasat factory will further strengthen that position. That build is set to begin as soon as late 2027. In the long term, space-based AI compute could enjoy benefits like abundant solar power, reduced dependence on terrestrial energy infrastructure, and the ability to process data directly in orbit, Nanavaty noted. That said, the concept remains “largely unproven,” and significant engineering challenges, particularly around servicing and maintaining hardware in space, still need to be addressed. Further, while SpaceX has a “strong track record” of delivering ambitious engineering projects, its timelines have often slipped, sometimes by several years, said Nanavaty. “Demo systems by 2028 appear realistic,” he noted, but large-scale commercial deployments are likely to take longer. This is because both the technology and the business case will need to mature before orbital AI data centers become a viable alternative to terrestrial infrastructure. Thus, he advised, “be cautious about assigning a firm timeline beyond early demonstrations.”
- UST partners with Anthropic to embed Claude AI across enterprise platforms and train 20,000 employees
UST has entered into a strategic alliance with AI company Anthropic to integrate the Claude family of AI models across its engineering platforms, industry solutions and internal operations, as the […] The post UST partners with Anthropic to embed Claude AI across enterprise platforms and train 20,000 employees appeared first on Express Computer .
- UST to train 12,000 employees in India on Anthropic's Claude
UST to train 12,000 employees in India on Anthropic's Claude Techcircle
- Which voice agent API has the best developer experience? What to evaluate
Comparison of voice agent APIs focusing on developer friendliness and key evaluation criteria.
- AssemblyAI vs. Deepgram for batch transcription: accuracy, turnaround, and pricing
Side‑by‑side comparison of batch transcription services from AssemblyAI and Deepgram.
- AssemblyAI vs Rev AI: Accuracy, pricing and features compared
Detailed comparison of AssemblyAI and Rev AI on accuracy, cost, and feature set.
- AI Doesn’t Need to Outthink You. It Needs to Understand You
AI Doesn’t Need to Outthink You. It Needs to Understand You PCMag Middle East
- Samsung CEO TM Roh says AI must understand users ahead of Galaxy Unpacked
Samsung CEO TM Roh says AI must understand users ahead of Galaxy Unpacked
- Global AI industry falls short on safety, think-tank warns
Global AI industry falls short on safety, think-tank warns The Straits Times
- Global AI industry falls short on safety, think tank warns
US artificial intelligence lab Anthropic scored the highest in a semiannual safety ranking, but globally the industry fails to combat "existential" threats, according to a report released Tuesday.
- AI boom drives intangible investment to record level - UN
The AI boom has helped drive investments in intangible assets such as software, data and research to a record high in 2025, the United Nations' patent and innovation agency said today.
- Dubai Police unveil next generation of ‘Ghiath’ smart patrols
The initiative reflects Dubai Police’s continued commitment to adopting cutting edge technologies, artificial intelligence and global innovations to strengthen its smart policing transport ecosystem
- Microsoft 365 Apps Favoring Company's AI Models Over ChatGPT, Claude
Microsoft 365 Apps Favoring Company's AI Models Over ChatGPT, Claude PCMag UK
- Microsoft 365 Apps Favoring Company's AI Models Over ChatGPT, Claude
Microsoft 365 Apps Favoring Company's AI Models Over ChatGPT, Claude PCMag Australia
- Everyone's Freaking Out About Microsoft's Leaked AI Windows. They Shouldn't Be
Everyone's Freaking Out About Microsoft's Leaked AI Windows. They Shouldn't Be PCMag Australia
- Inside the Secret AI War Room Behind the 2026 World Cup
Inside the Secret AI War Room Behind the 2026 World Cup PCMag UK
- NCSC plans AI-powered Cyber Shield against autonomous cyberattacks
NCSC plans AI-powered Cyber Shield against autonomous cyberattacks Computing UK
- The NCSC wants to build an AI-powered 'Cyber Shield' to protect the UK from hackers – here’s how it’ll work
The NCSC wants to build an AI-powered 'Cyber Shield' to protect the UK from hackers – here’s how it’ll work IT Pro
- Workers fear unproven AI impact on jobs and the merit of going second-hand
Business Digest: The best news, analysis and comment from The Irish Times business desk
- Bono AI
Talk Once. Publish Everywhere.
- Half of workers fear AI will replace or impact their jobs
Almost half of Irish workers expect their jobs to be impacted or replaced by AI, according to a new report by FRS Recruitment.
- AIAI Holdings' Constellation Network Launches Gate AI: Real-Time Security for AI Applications
AIAI Holdings' Constellation Network Launches Gate AI: Real-Time Security for AI Applications azcentral.com and The Arizona Republic
- Texas Universities Offer AI Degrees to Boost Value for Students
As many companies slow new hiring because of uncertainty over AI, Texas universities are trying to stay a step ahead by offering AI bachelor's and master's degree programs paired with more traditional disciplines.
- Alibaba Stock: The AI Rally Isn’t Over, It’s Just Moved to China
Alibaba Stock: The AI Rally Isn’t Over, It’s Just Moved to China Barron's
- Alibaba Stock Leads China AI Rally, Just as U.S. Tech Stumbles
Alibaba Stock Leads China AI Rally, Just as U.S. Tech Stumbles Barron's
- Self-driving startup Turing gets AMD backing and GPUs
Self-driving startup Turing gets AMD backing and GPUs East Bay Times
- Planting the future: Mizzou researchers put AI to work on the farm
Planting the future: Mizzou researchers put AI to work on the farm EurekAlert!
- New AI-assisted climate study reveals what urban heat really feels like
New AI-assisted climate study reveals what urban heat really feels like EurekAlert!
- This startup thinks robotics is about to have its ChatGPT moment
General Intuition is betting millions of hours of video game data can train the foundation models for physical AI, making it easier to build smarter robots with minimal real-world data.
- ‘We cannot choose to become idiots’: a Brown professor’s proof of mass AI cheating
An economics professor at Brown University suspects most of his class cheated with AI, and he has the numbers to make the case. Roberto Serrano watched his take-home midterm average hit 96 out of 100. When he switched the final to an in-person test, the average fell to 48. He has taken the story public, […] This story continues at The Next Web