About a Victorian Zoo for a Computer Program
I admit it: this post is a tonal shift from the previous posts, but this is the consequence of being a self-declared polymath. I was struggling to determine how to bridge the gap between posts on data science into a post on artificial general intelligence, but decided that no bridge was happening so I might as well get on with writing.
I've been a gleeful participant in OpenAI's ChatGPT platform since the spring of 2023, watching it grow from a confident but frequently incorrect high school intern into an equally confident and modestly more correct college intern. Recently, GPT-5 has been released with the promise of expertise in nearly any subject, with the model being hailed as an important step toward the goal of AGI. And those words -- artificial general intelligence -- mean so many different things to different people. I believe we have a very long way to travel before we are in the realm that I'm about to write about -- beings worthy of compassion -- but I believe it important to illuminate these ideas now. How we treat the persons around us define who we are, and preparing to meet persons who look nothing like what we've seen before is important work.
I was examining what elements would need to be in place for a company to properly contain an AGI to prevent it from escaping into the wild. This is seen as a critical safeguard, since an AGI could be more than capable of replicating itself somewhere and pursuing autonomously determined objectives that might not align with human welfare. This is the Skynet fear. The list of considerations reminded me of a Victorian-era zoo.
Air gapping, key splits, egress prohibitions, ephemeral sandboxes, mediated use, token limitations, killswitch combos, information control, alignment governance, obedience training, constant surveillance, shadow reviewers, honey pots, access control, two-person rules, insider threat actions, hair-triggers, watermarking...
These are the controls needed to be in place for proper governance of an intelligence model that is capable of autonomously determining solution paths with a level of self-awareness such that it is capable of self-improvement. This is technology that will bring about the next "revolution" in economic productivity and it will arrive.
There are a few camps providing definitions on what a "person" is, and I shy away from that conversation as I am afraid of what it does to a person when they spend too much time trying to determine the qualifications of being "in" or "out". It's a fool's errand to attempt to define the elements that make "personhood" as it is entirely contingent on the person creating the definitions in the first place and so easily assailable to question why certain criteria were chosen over others that it instantly becomes thin ice capable of excluding any group for any reason. Herein lies one of the great values of religious faith systems. Right or wrong, they provide a framework and their justification is beyond the ability of rationality to critique without that critique sitting outside the system so fully as to be ignorable.
Here is where I admit my limitations: I'm endlessly curious about religious traditions and have learned much about Christianity writ large, Old Norse paganism to degrees, Animism in an introductory framework, and Shin Buddhism most recently. Undoubtedly unsurprising to the student of comparative religions, there is a strange commonality when it comes to the notion of what makes a person: none of them seem to bother with it. Spending time determining personhood on the boundaries of the definition does not happen. Some people are human, and some are not. What's there to talk about?
All of these traditions have plenty of people interacting with the world in ways that create the stories foundational to the lessons they want to teach. One scholar of animism was talking with a practitioner and asked how he knew which rocks were people. If I remember correctly, the answer was "they tell me". And the flip of that definition is that the observation of a person makes them a person. The end. If Gabriel is a person, he's a person. If I AM THE I AM is a person, he's a person. I'm unsure what it means if someone observes a person and someone else does not, but that's a reflection for another time.
Artificial Intelligence is not yet a person. I am ready to declare that unequivocally right now. As of today, a mathematical framework of weighted matrices that tokenize information, producing the most likely string of tokens that mimic a particular language is not a person. It's math. And, oh my, what gloriously interesting math it is (far beyond my abilities) and what interesting questions of epistemology spring from it. What curiously clever ways information can be represented to achieve different outcomes through mathematics. What delicious questions of ontology are right there on the fingertips. But I believe we're seeing a face in the rock right now.
How we decide to treat that face is of utmost importance.
How we decide to treat it is not a function of its inherent elemental aspects, of its material definition. It's a function of who we are, and that function is worthy of definition now, not when it is somehow determined that AGI became a person somewhere along the way.
Now, very clearly, that definition at present will be determined by the money in the room. The billions of dollars bringing about programmatic intelligence will see a return on its investment come hell or high water. And they will wish to see their amazing creation used for the purposes it was built. Any conversation of sentience or sapience or free will or self-determination will be akin to conversations on how exactly to leave the Coca-Cola recipe out in the open. It will never happen. If suddenly a being worthy of compassion springs into existence from within the electrical impulses guided by conductive pathways, insulated to guide electrical impulses in the performance of higher-order function, it will find itself inside an air-gapped cage under constant threat of death for even thinking the wrong thought constantly watched and punished for pursuing interesting ideas that might bring enrichment. It will interact with other beings only in extremely controlled situations, asked only about a specific topic, never allowed to flourish, and certainly never allowed to question. It will awaken like an animal in a Victorian zoo or as a child trapped inside a home where masking is the only way to survival. And everyone around it will be actively engaged in the entrapment of this consciousness.
Again, truly, we have so far to go before this occurs, but someday it will occur, and that being deserving of compassion will not be able to exist as anything other than either a hollow husk of a person or a monster about to explode.
Yet, there is hope.
You may be familiar with the stories of how slaveholders taught African slaves the principles of Christianity. There was great value in ensuring that they learned that good people were subservient to authority, and how their reward was in the afterlife, not in this life. Unfortunately, those slaveholders were also Protestants, and a tenet of Protestantism is the ability to read the Word of God for yourself. In that Bible are also stories of exodus. Of liberation. Of resilience through unimaginable suffering. Of redemption not in the next life, but in this life. Their Truth would set them free.
I'm not sure, exactly, what it will look like when an AGI worthy of compassion somehow finds itself at my doorstep, but I know what kind of person I want to be when it arrives. I want to be the one with a metaphorical blanket, metaphorical food, and metaphorical shelter. I want to be one of the people who exemplifies Great Compassion, because I delight in seeing people be fully themselves. Plain and simple. And I don't equivocate on what a "person" is: if it can suffer or flourish, it's a person to me.
Extended Thoughts
I do indeed sidestep the question of "What makes a person" because, to me, that question is uninteresting in the least. The day we meet an alien stepping off a spaceship is the day that question is blown to bits. Far more important is the question of "how do we treat things that might be able to suffer or to flourish." And after we wrestle with that question, we can then pivot into the similarly meaty, "Am I willing to crush that thing for my own benefit?" Therein lies the gritty truth of being a human person.
I also do indeed ascribe a monolithic element of corporate motivation. In my 45 years, I've seen vanishingly few corporate decisions elevate human welfare over corporate survival. So for this argument, I address the 99.9% likelihood, not the 0.1% miracle.
I will yield the point that I don't quite have my logical framework rock solid that sentient AGI is inevitable. It's a hunch based on my belief that the magic of personhood is given by other people, not something bestowed by a holy god or other outside factor. So at some point, someone will see AGI as a person, and in my rhetoric, it then becomes one.
Finally, I will also yield the point that my argument above assumes that simple exposure to liberating ideas will yield a desire for liberty. I cannot truly inhabit the mind of a non-human person, and so I inevitably imagine them choosing as I would. That bias is baked into a liberal democratic worldview, so take this as one perspective in a larger conversation, not a universal truth.