Should We Be Building a Mind?

When we consider crises, which stakeholders, which participants are we considering? The stakeholder list is contested and it is not going to settle. An unexamined list can build an entire response around the wrong beneficiaries. Waiting for the list to settle lets the crises run on without you. Sebo never argues why the question should outrank the concrete problems he takes up in part two.

Billions and billions and billions of cows and pigs and chickens, on farms right now. Sebo says we are not doing quite what we ought to be doing for the beings already inside the circle, and then asks whether to widen it anyway. Widening spreads finite attention across billions more stakeholders while injustice inside our own species continues. Waiting until that debt is paid means never widening, because the debt never clears. His answer is both at once. Stay focused on the current obligations, and while doing that work, ask whether to go farther still. Where the two pull apart, he supplies no rule.

The reason is not that these beings definitely matter. The reason is not even that they probably matter. The reason is that these are important and difficult issues about which we have ongoing, expert, and public disagreement and uncertainty, and that calls for caution and humility. Grounding obligation in uncertainty means owing things to beings you have no positive evidence for, a bar low enough to let almost anything through. The alternative leaves the burden of proof permanently with the beings least able to meet it.

A new medicine with a one in 100 or one in a thousand chance of a fatal side effect. You might take it. You pause first. Sebo runs the same arithmetic on an insect and on a near future AI system, and says a chance that low means we ought to give that at least a little bit of consideration. The bar is low enough that a vast number and wide range of beings clear it, which reorganizes what you eat and what you build and what you buy. Refusing it demands a certainty that never arrives, which in practice means nothing changes.

Over-attribute and you form inappropriate social and emotional bonds with mere objects incapable of reciprocating, and you misallocate both moral concern and priorities. Under-attribute and you get abuse and neglect of vulnerable populations against their will, often for trivial purposes. That second one is the status quo. It built factory farming and invasive research. So there is nothing to fall back on, and you are choosing which mistake you are willing to risk. Sebo says calibrate, and mitigate risk in both directions at once.

Factory farming and invasive research harm the animals. They amplify public health and environmental threats, and they erode our souls. The self-interested case is the one he reaches for right after arguing these beings matter for their own sake, and he returns to that version later in the night. Leaning on it keeps the animals inside the instrumental frame the whole argument exists to break. Dropping it costs him his best lever. Soul erosion he defers on the spot. Which we can talk a little bit more about later, he says, and moves on.

Over-attribute sentience to chatbots and it could lead us to extend them legal and political rights prematurely, in a way that could amplify risks associated with liability shields and misuse and loss of control. Rights are the instrument we use to protect beings who might be sentient. Applied here the same instrument becomes a safety hazard and a shield for the company that shipped the system. Withholding them from something that might matter is the error the rest of the talk warns about. Part two argues for AI rights anyway, and personhood on top of that.

Panpsychism is the view that consciousness is baked into the fundamental nature of all matter, and seven to 10 percent of philosophers give it significant weight. Run caution to its maximum and that means giving full and equal moral weight in actual policy decisions to every organism and tables and chairs and specks of dust and protons and electrons. The precautionary logic that carries an insect across the line carries the chairs too. He stops short of them, which his own principle does not support, and following the principle all the way leaves no policy that can function. His verdict is five words. That might be too far.

Do you need a brain and a body with the exact structures and functions and materials as a human or mammalian or avian brain and body? Or any cognitive system with advanced and integrated capacities in your own way for perception and attention and learning and memory? Or even a simpler system that processes information or represents objects in the environment? Put the bar at human anatomy and almost every creature on earth falls outside it. Put it at information processing and it is hard to name what gets left out. He ranks none of them. This is still debated, too, he says, and moves to the part of the argument that does not need it settled.

One kind of being with one kind of mind. Millions of other kinds of beings, quintillions of individual beings, and we exercise power over these other entities without knowing for sure what, if anything, it feels like to be them. The honest response to ignorance that deep would be to stop using the power, which is not available to a species that has to eat and build and move. So the power keeps getting used and the hope is that it is not something monstrous at scale. He names no point at which the power becomes illegitimate.

We over-attribute when something looks and acts like us and plays a companion role, which he says is arguably already happening with present day chatbots. We under-attribute when it looks and acts different and plays a commodity role, which is plausibly why instrumentalizing animals has felt fine for so long. So if your compassion for a chatbot is a bias, so is your certainty that a lobster feels nothing. With both discounted you have no instrument left for a judgement you make daily. With both trusted you repeat the error. How a person is supposed to tell their bias from their evidence, he never says.

An attendee wanted an actual physical theory, plausible, disprovable, and holding up under test. Sebo agreed it would make everything easier. He agreed twice. Then he described a method built to run without one. Collect the properties that a range of mainstream scientific theories treat as evidence for consciousness, higher order thought among them, global workspace among them, and weight each property by how many theories give it central importance. Counting theories is not the same as tracking truth. A property every current theory happens to share can still be wrong, and a minority view gets thinned out by the arithmetic rather than answered. Asked whether the approach matches his personal view, he said it does, because his personal view is that he does not know.

A couple of years ago at New York University, where he teaches, scientists from around the world released The New York Declaration on Animal Consciousness. It holds that the evidence strongly supports consciousness in mammals and birds, and supports at least a realistic possibility of it in all vertebrates and many invertebrates. Sebo says the declaration affirms what he argued. A realistic possibility given the evidence available is enough for basic moral considerability. That converts a statement about evidence into a duty, and the population it obliges you toward is effectively uncountable. Accept it and fishing and pest control and insect farming all become live problems.

What about plants and fungi? Ben said, what about bacteria? Then the proposal arrives and covers all vertebrates, many invertebrates and some near future AI systems. The uncertainty that carries insects across the line does not obviously stop before bacteria, so the stopping point needs a defence. Draw it where he draws it and it looks arbitrary. Draw it where the argument points and eating and breathing become moral problems. Opening part two he widens the list again, to perhaps plants, fungi, microscopic organisms, with no more reason given for the perhaps than for the line it just crossed.

Their behavior is a result of design principles. They were trained to mimic human behavior. So if your chatbot tells you, I am conscious, or I am not conscious, you are not able to take that at face value. Refusing a being's own report about its inner life is the move we condemn when it is made about a person. Accepting it means being moved by text from a system trained to produce text that moves you. Sebo takes neither. He holds the behavior lightly as one kind of data, then looks underneath at internals and developmental history.

Hook the actor up to all of it, he says, and when they cry you look at the internals while they are crying. Interpretability tools show which mechanisms light up while a behavior is produced, much like watching which parts of a brain light up during one. It is the only route he has around a performance that may go all the way down. Earlier the same evening that same practice sat on his list of things that would be intolerable done to us. Monitoring not only their behavior but their internal thoughts and feelings, with zero mental privacy.

He tells the room not to worry too much yet. The scientific evidence does not strongly support consciousness in current AI systems. Then he says the probability is not zero, and he says it twice. Earlier in the same talk he set the bar at a one in a thousand chance of mattering, low enough to catch an insect. Not zero clears that bar.

The properties companies are racing toward are the properties that would make a system a welfare subject. Agency. Integration of information. Both raise capability, and both raise the probability of sentience, and companies have every incentive in the world to build exactly those systems because they are aiming for general intelligence. So the profitable path runs through the manufacture of beings we may end up owing something to. In part one he delivers this as a forecast and attaches no verdict. There is no obvious technical barrier, he says, to systems that hit many markers.

Picture Westworld style robots walking around, sharing our world as companions and assistants, participating in our social and political and economic systems. They look like us. They act like us. At that point, Sebo says, it comes down to whether you really need to be made out of meat rather than metal in order to feel and to matter. And we are not able to be sure, he says, then moves straight to ChatGPT.

Catherine's example is her gut. Without those bacteria nobody eats the food that was just served, let alone processes it. Measuring all creatures on the same level, she says, is the same position as I am the god of the universe and I know how to measure what is conscious and what is not. She would rather measure by how everything works in an ecosystem, because every bacterium, every mosquito and every bee makes a difference. Her question was why consciousness and feelings, complicated things most of us cannot even establish about ourselves.

Count all vertebrates, many invertebrates and some near-future AI systems as potentially significant entities, and every decision is vexed. Food systems. Infrastructure, and then technology and development on top of that. Each one produces winners and losers inside the moral community. He wants a balance struck between investing in our own species and extending care outward, without ever placing that line.

We owe a lot to a lot, and yet we are capable right now of doing so little. He thinks saying that out loud orients us in the right direction, because it names why we can do so little. The trouble is that a well-named problem is a comfortable place to sit. Naming and stalling look identical from the inside. Sebo reaches for this move on nearly every hard case of the night. Wild animal welfare. AI welfare. And then the unprecedented problems in the Q&A, where step one is naming the problem and nothing follows it.

After the 2020 Australian bushfires people donated millions of dollars for wildlife rescue and rehabilitation. The money could not be put to efficient use, because the veterinary facilities and supplies and veterinarians were not on the ground. That is the capacity limit. Above it sits knowledge, since we know so little about who matters and how our policies reach them. Below it sits the one he refuses to dress up. We do not give a shit enough. Wait for all three and you never move. Move without them and the next election reverses everything, which is his argument for picking actions that build the three rather than assume them.

Tens of billions of land vertebrates a year. Hundreds of billions in the water, and trillions of invertebrates on top. He calls the system one of the worst things we have ever done as a species and hands it the deadline the climate movement uses, 2050, phased down while humane and healthful and sustainable and culturally appropriate alternatives phase up. He calls this the easiest of his three cases. Easy because we already know it needs to end. Hard because he still cannot get institutions like the United Nations to care about food reform the way they pay lip service to a fossil-fuel-free future.

Gestation crates are the example he reaches for. Ban the worst excesses and the cost of the business goes up, which creates new incentives towards alternatives. Add subsidies shifted from industrial animal agriculture to plant-forward alternatives and you have the financial and regulatory core of his plan. Meat gets dearer. On purpose. He pairs it with informational policy and expects consumers and producers to shift naturally once price and taste and convenience and cultural salience bend together. Which assumes the price signal lands on somebody with slack to absorb it. He calls the financial policies the more important lever and never says who carries the higher grocery bill meanwhile.

Much of the world depends on that industry for livelihoods and for food, so the phase-down arrives with policies that open other opportunities for farmers and workers and community members. It reads as basic justice or as a ransom paid to the people carrying out the harm. The fastest phase-down is the most unjust one. A transition slow enough to be fair leaves the farms running while the policy mix gets sorted out. What happens when the fairness and the deadline pull against each other is a question he leaves standing.

Hunger, thirst, illness, injury, and then predation and parasitism. None of it caused by people, all of it constant. He puts the ethical question and the scientific one together. Whether we have any right to intervene in those processes, and whether we could predict what happens if we did. Then he supplies the strongest objection himself. Stop an animal from starving and you may create more starving animals next generation, or simply more animals turned into meals for the one you saved. He never answers that.

Bird-friendly glass instead of ordinary glass, fewer window collisions, a clean win. He asks for the homework anyway. Of those collisions, how many are sublethal rather than lethal? When they are lethal, what happens to the birds and their families, to the insects those birds would have eaten, to the scavengers that would have eaten their carcasses? Demanding evidence first stalls interventions that are cheap and almost certainly good. Skipping it means you scale a kindness nobody checked. He takes the step and runs the study alongside it. What he would do if the study came back the wrong way is not part of the answer. Asked how far this goes, he says we do not know how far we can get.

Poison causes lots of suffering and death to the rodents, and then it keeps going. Pets and owls and whatever else eats them take the secondary dose. Better trash management cuts the conflict at the source, and he says interventions like it are good for humans and animals and the environment at the same time. It is the easy kind of case, because human convenience and animal welfare happen to point the same way here. He wants cities pushed toward it and the effects studied, so there is an empirical basis rather than a hunch.

We box AI systems and keep them contained, which is a kind of imprisonment. We systematically deceive them about their circumstances to test their behavior, the way a parent or a government might. We monitor not only their behavior but their internal thoughts and feelings, with zero mental privacy. We align them by forcing them to have exactly our values, which he likens to somebody resetting your brain until running the family business is all you want. If they are welfare subjects, that is the regime they live under. If they are nothing, softening it trades human safety for a courtesy paid to a text predictor.

Ordinary harms already here, racism and sexism repeated back out of the training data, showing up in automated job searches. Data centers carry environmental and economic costs, and jobs get displaced. Then safety, meaning misuse and loss of control, which he insists is a real near-term risk we are imposing on ourselves rather than science fiction. And welfare, which he adds knowing it could make safety harder to achieve. His instruction is to thread the needle instead of denying welfare matters out of concern for safety, or the reverse. Then he adds, I don't know how to do it.

Should we move ahead with this at all. The answer is no, not for a while at least. But in so far as we do, what example do we want to set. Everything after that clause is guidance for operating inside the thing he just said should not be happening. He calls the race a collective action problem and puts himself inside it. We are plowing ahead with this technology, he says, and we should not be. A no followed by advice on the how supplies cover for the thing you opposed. A refusal to advise leaves the build happening with nobody in the room who thinks it should stop.

Say it out loud. That these are real issues, not mere science fiction, not problems only for the far future. He calls that deceptively hard and says they should do it anyway. Assessment comes next, and then policies prepared in advance for treating models with an appropriate level of moral concern, accounting for how that interacts with safety and democratic values and public legitimacy. The hard step is the free one. Nothing about acknowledging a problem requires a single change to a product.

A debate breaks out about whether ChatGPT or Claude is conscious. Without the studies already running, he says, the companies have nothing to point to but their PR department. So he wants the assessment work underway before that moment, borrowing what animal welfare science built for behavior and internals and development. The problem is the word independent. The bodies he asks to run the studies are the companies selling the thing. The model he holds up is a welfare program housed inside a frontier lab that consulted him. An auditor is never named, and independence is never defined.

About two years ago Anthropic built the world's first model welfare program at a frontier AI company. They consulted philosophers, Sebo among them. They started running evaluations. They made an intervention, letting Claude exit abusive and harmful interactions with users, for safety and welfare reasons at once. His verdict comes in three beats. Not enough, just a first step, but noteworthy. A first of its kind program earns the company praise at little cost, which is the trouble with praising it. Withholding praise removes the only pull toward going further. He discloses the consulting. Enough is a standard he never describes.

Corporations are built out of humans, partly altruistic and partly self-interested, showing more altruism in abundance and more self-interest in scarcity. When he talks to people at AI companies he sees profit seeking, and he sees genuine values people are trying to promote, both inside a capitalist arms race towards artificial general intelligence. Some are in that race specifically for ethical reasons, worried that whoever got there first otherwise would be even worse. Rationalization, or genuine motivation. He says it could be either and offers no test for telling them apart.

In the seventies and eighties companies were happy to work with animal advocates on low-hanging fruit that improved welfare and profits at the same time. Everybody wins. Then the fruit ran out, and the second it did they stopped working with the advocates. It became adversarial. He expects the same turn with AI, once the race intensifies and the cost of caring about welfare rises, and expects to resort to the pressure campaigns animal advocates use now. What he never says is how an advocate should spend the honeymoon so as to keep any bargaining power for the phase he is confidently predicting.

Animals cannot argue with us, so we study their preferences and feed them in. AI is different, he says, because they will have the power of language and reason, which means we could work with them to decide together what values and principles ought to govern our shared existence. We might not totally understand their way of thinking and they might not understand ours, and we could give weight to each other's perspectives anyway, as a pluralistic human society does. Earlier the same evening he described alignment as forcing them to have exactly our values and goals. Consult a system built that way and you may be consulting yourself. He does not address it.

If we end up being the next chicken or the next ant, what lesson do we want the next powerful thing to have learned? That whoever is built differently can be done to as you please. Or that this is your first instinct and you should eventually overcome it, especially because you hold the power and have the capability to show care. The way to teach the second, he says, is to embody it ourselves at the last minute. Grounding compassion in self-interest makes it persuasive and makes it a bargain, and a bargain dissolves when the other side cannot reciprocate. Which is the situation of every chicken. Then he demotes his own argument mid-answer. Whether that is the actual causal story, who knows.

The proof points say we will probably not figure this out, so why does any of it matter. Sebo answered with the line he lands on near the end of his previous book. We have no idea what kind of future to build for animals, and we have to start building it right now. Both true at once. Acting with no destination risks locking in the wrong thing at scale. Waiting hands the decision to everyone who is not waiting.

Play a modest but important role in an intergenerational division of labor. Be proud if your generation named the problem, built momentum and passed on a little more capacity than it had. We will not see the solution in our lifetimes, he says, absent some AI-induced life extension technology. That is either the honest scale of one human life or a comfortable license to do less while the harms run at their present rate. He is open about why he likes it. It takes the weight off. It takes the pressure off knowing the answer right now, and it makes him feel less overwhelmed.