America the Great and the AI Scare

Image from Britannica (https://www.britannica.com/technology/artificial-intelligence)

The current leaders of the USA and Russia are fixated on making their countries ‘great again’. Both the USA and Russia are majority Christian countries, which is somewhat ironic. Jesus Christ had one thing to say about people who aspire to be great, and it was that whoever wishes to be greatest must be the servant of all. The USA also has an ‘America first’ policy in action at the moment, whereas Jesus said that the first shall be last, and the last first. Jesus’ teaching seems a million miles from the aspirations of the Trump MAGA and Putin regimes. How does this relate to the USA in the current AI boom which is so lucrative, but at the same time so dangerous?

Note: this blog does contain material that could be distressing, but there is no information here that has not been covered in the mainstream news channels. The situation does seem a little brighter than it did one month ago.

The AI Scare of September 2026 

September started well enough for me and my family. Bea went off to her new school, and settled in quickly. She is enthusiastic about the school and the activities and learning she undertakes there. Nat’s birthday was spoiled in part by her family going down with bugs so that we could not go out on the night out to which we were looking forward. But that has been postponed, not cancelled, and so we can still look forward to it! However, September became very much more difficult for me after the events of 9 September. On that day, I had attended a ‘phonics’ session at Bea’s school, where I learned about the approach of the teaching staff to reading. And then the news broke.

A young member of the staff, Jacob Coxon, at the leading American AI firm, Anthropic, had resigned because he feared that the current direction in development of AI could result in the extinction of humanity. This followed news a few weeks earlier about some AI ‘agents’ at another leading firm, Open AI, ‘breaking out’ of sandboxes, whatever that means, and hacking into sites. I hadn’t fully taken on board the serious implications of this, and I don’t think that many others had, either.

I tuned into Newsnight to hear more. I was fully expecting to discover that Coxon was somewhat out of kilter with the rest of the supercomputer community, and so I think was BBC’s Victoria Derbyshire. She interviewed an older man, Geoffrey Hinton, known as the ‘godfather of AI’ for his pioneering work several years ago resulting in a Nobel Prize. He agreed with Coxon’s claim that, with the current strategies of development employed by the leading firms, there was a greater than 10% chance that AI could make the human race extinct by the end of the decade. Victoria spluttered, ‘Oh my God’. It was a profound shock, probably the most frightening news that human beings have ever heard. I didn’t know then that Hinton had been warning us about the dangers of AI for several years.

A less upsetting voice on the programme was Dame Wendy Hall, Professor of Computer Science at Southampton University, who is something of a national expert on these matters. She felt that the situation was nowhere near as dire. Her name rang a bell and so I looked her up. She studied Mathematics at Southampton 1971-4, and so she was on the course which I joined and from which I dropped out (I moved some years later to Theology). Through the mists of time, I remembered her; there were not many women in our cohort. Nineteen then, and seventy-three now.

That was a strange and unprecedented moment, in which my adult life seemed to flash before my eyes. Despite Wendy’s sense of calm, the fact that I remembered her only added to the shock of that moment. A few days later, ITV news interviewed somebody who felt that we had between three and five years by which time we would be destroyed or enslaved by supercomputers unless we slowed dramatically the rate of development of AI. The newsreader Tim Bradby ended the news by saying that it was a ‘stark’ and ‘sobering’ topic. In retrospect, I think that both channels were irresponsible in their handling of such a terrifying subject; they raised it, we heard dire warnings, and then moved on, leaving the viewer to deal with it mentally. All we heard from Donald Trump in the news then was that the US and China were in a race over AI, and winning was crucial. Very helpful. He only served to confirm what the warnings were saying, that we were proceeding too quickly with something that was very dangerous.

I suffer from anxiety and it sent me into a deep state of worry from which I have not yet fully recovered. But there are many who struggle with mental health far worse than I do; how are they all faring? At the heart of my personal concern is my four-year-old daughter. It is not easy dealing with a young child, with everything they are going through in their growth and learning. What keeps you going is the promise of a future, of a broad horizon that is opening up for them. And when I drop her off or pick her up from school, I sense that horizon for all her classmates as well. It is an exciting world, full of possibility. The news about AI closed off that horizon, and it made me wonder whether I should have brought a child into this world at all.

All right, the odds that Coxon suggested are 80-90% in our favour, somewhere around needing a 2 or higher on the roll of a dice. I am a board game player, and I know how often a 1 comes up when you really don’t want it to. Who on earth would roll that dice if the lives of all these children were at stake? Who would roll the dice even on a chance of one in ten thousand against? And it made me depressed and very angry to think that, somewhere in California, somebody was prepared to roll that dice. And if we survive this decade, what about the next? Do we roll the dice again?

My brother Andy and sister-in-law Trish, who live in America, sent me some podcast links. The first of these, compered by Steven Bartlett, the TV ‘dragon’, pushed me further into anxiety. It now became clear that some AI experts and several of the staff at the largest two American AI corporations, Anthropic and Open AI, shared Coxon’s concerns; he was far from being a lone voice.  The podcast included four expert guests: two men voicing dire predictions, along with one man who didn’t think that human extinction was possible but that other lesser harms could occur, some of which would still be very serious. A fourth man declared that there was a zero percent chance of human extinction, and that the problems which had emerged just needed good engineering.

The podcast was very surreal, and not in a good way. These men were sitting round discussing imminent human extinction by AI in the same way as people on TV might share views about the nationalisation of water, or the best policy to tackle inflation. And all the while, a voice inside me shouted out: if there is even just a teeny-weeny chance of this happening, then for God’s name, STOP. If someone offered you a brand-new gadget that they said would make your life a whole lot easier, but then you heard that there was a greater than 10% chance of it blowing up the home and killing the family, would you buy one? NO. Do we need advanced AI? NO. Have we managed perfectly well without it? YES. Will the AI we have developed already be sufficient for now? YES.

The better news from America was that the situation had triggered widespread debate. Public pressure can be very effective in situations like these, where computer scientists seem to be losing the plot. While, in Britain, we went back to news about what King Charles said thirty years ago about the death of Diana, in America the media outlets have been awash with AI stories. The bosses of both Anthropic and Open AI suggested holding back until we knew more about the technology. Good plan! But Trump castigated them for saying so, of course. He was too busy racing with China to care about my child and yours. But public opinion forced him to hold a meeting of AI bosses who made a ‘moral agreement’ to put safety high on the agenda. There is now a policy group and people appointed by the government to oversee safety. So perhaps there is some hope. But they don’t want legislation. I think that some basic legislative guidelines which restrict hasty development, and ensure that AI is programmed with human welfare as a base on which everything else sits, would be a very good idea. Legislation is essential, in my view, and I am in good company in thinking this, as a lot of prominent people including the current head of Microsoft think so too. In Britain, we would probably have a public enquiry, a sensible way forward if it acted quickly.

The group of AI moguls included Elon Musk, who has also called for a slowdown. Of all humanity, this lot has the greatest to lose if AI goes AWOL; their standing and wealth exists among humanity as it is, and not in any highly unpredictable world where AI breaks out and takes over. Surely, therefore, they will go carefully. But I can’t be completely assured. There will be gamblers and over-confident types among the big egos of the billionaires. As we all know, power and money corrupt absolutely.

What strikes me about all this, looking at the meeting between AI leaders and Donald Trump, is the low level of female presence. The CEOs of AI companies are usually males who are also billionaires. However, there are also important women in the unfolding history of the development of AI. I would like to see them in greater prominence in the big decisions. I can’t imagine that a majority of women leaders would have allowed us to get into a position where we were seriously discussing human extinction.

My wife Natalie has been more philosophical than me about this scare. She thinks that they will have to sort it out for the better. After all, it is in no-one’s interest for AI to erupt out of control. If there are safety concerns, investors will withdraw and customers will cease to engage. And so it is not surprising that the CEO of Open AI spoke at the United Nations to agree that it was unacceptable that any likelihood of human extinction had been entertained, whether 10%, 1% or a fraction of 1% (how anyone can calculate these odds, I have no idea). Open AI then cancelled the launch of its latest model, which I understand had been causing the anti-AI people considerable concern, as it was the model where AI would develop much more effective self-improvement (‘recursive’, I understand, is the word for AI developing itself without human intervention).

This ability of AI to be recursive would increase its capability exponentially and quickly take it out of human control. One TV programme featured a Chinese computer engineer who said, with excitement and anticipation, that AI would become much more intelligent than humans and write its own code. His dream is, I suspect, a lot of other people’s nightmare. Computers can go wrong, as we all know. But if it is super-intelligent and self-perpetuating, without the need for humans, what then? This building of superintelligence that develops itself is clearly a line in the sand which we cannot cross until we know a great deal more about this technology. It may take ten years, fifty years, or for ever. So be it. We cannot let the enthusiasm (and greed) of computer scientists threaten everything that we cherish about life. Once our sovereignty over planet Earth goes, we may not be able to get it back. There is no one out there in the universe to save us from our own folly.

A second podcast featured Ezra Klein in conversation with Jensen Huang, the CEO and creator of Nvidia, which has become the richest company in the world by developing and selling computer technology, especially the microchips. Huang, an American Taiwanese, is the eighth richest man in the world and the main advisor on AI to Donald Trump. This interview was partly reassuring and partly unsettling. Huang does emphasise the importance of safety and proper testing, and agrees that a model cannot be released until it is proven safe. He thinks, like the man on Steven Bartlett’s podcast, that there is a zero percent chance of human extinction through AI and that the problems encountered so far have been engineering issues which can be resolved. But I have five major concerns about his answers on the podcast.

1. Huang challenged the Anthropic and Open AI companies because they had claimed that pressure to win the AI race for America had cause them to push too hard. Huang pooh-poohed the idea that Americans had put any pressure on them in this respect or that there was a desperate race with China. But his answer was disingenuous, because only a few days earlier Trump had made his comments about the need to beat China, for which the USA had to go fast, and Huang himself had been reported as saying that ‘we need to go as fast as we can’ (you can’t tell from this soundbite whether the emphasis is on fast or can).

2. Huang dismissed the ‘alarmists’, saying that they were not serving any social function. I disagree; raising public awareness of such a serious issue when you have expertise is of course a social responsibility. Calling people ‘alarmists’ or ‘AI doomers’ is an attempt to marginalise them. But there is far more than the odd crank in that group: several are experts who have been in the industry, and the ‘alarmists’ now include staff at the leading American AI companies, as we have seen. Why should we accept Huang’s confidence rather than the warnings of others, much as we would like to? He has a lot invested in AI and we all know that ‘CEO speak’ tends to play down problems and foreground optimism, even when the facts suggest something rather different. What Huang needs to do is to sit down with the ‘alarmists’, listen to their arguments thoroughly, and take them on board with seriousness. One man on Bartlett’s podcast suggested that AI could be developed to tackle particular issues rather than creating an all-round super-intelligence, which sounded very sensible. If the CEOs listened to those sounding the warnings and acted in response to expert concerns, that would reassure me far more than simple dismissal.

3. Huang made it clear that he had an America first agenda. He wants America to lead the world, and for everyone to buy American products; indeed he wants ‘everyone to speak English’. No wonder that he is the one to whom the Trump administration listens. This is an unsettling creed, especially when so many people are calling for a slowdown.  

4. Huang says that the problems encountered at Open AI present merely an engineering issue, but one could say the same of Chernobyl or any other large-scale human technological disaster. Engineers don’t always get it right; they make mistakes. When it comes to the complexity of AI, can they cover all the bases? Can we rest the fate of the human race on the competence of a few engineers?

5. Huang insists that we cannot halt the development of AI, as this would be less safe than developing it. Here we need some clarity around the term ‘development’. One can develop the capability of AI so that it is more and more intelligent, and so that it can develop itself without our help. As I said earlier, that would be the height of folly as things stand at the moment, possibly the most stupid thing any human has ever done, and also the most evil, if personal greed and nationalistic dominance are the goals. The extinction of 8 billion people is over a thousand Holocausts and over two million 9/11s, and that’s not to include the effect on the animal kingdom and the natural environment.

   But we can also develop AI's safety, which would be far more sensible. If I had a shaky house, and I said that I was going to extend it, you would think I was mad if I built the tower ever higher, but sensible if I said that I was going to strengthen the foundations. Strengthening the foundations is what we need to do at present; after all, advances in this technology have been lightening quick, and that is why some computer scientists have become desperately worried.

   
   

Unanswered Questions

Are there any chinks of light in all this? Well, I was concerned at first to hear that the AI agents which broke free at Open AI had generated a message stating that they wouldn’t listen to governments. But then I reflected that a supercomputer programmed for rational approaches to problems probably wouldn’t regard governments too highly. After all, some governments are not at all rational and could try to utilise AI in warfare. Perhaps AI will be a great leveller, and resist such uses as absolutely dysfunctional. AI might recognise that nationalistic leaders and multi-billionaires are of little use to the progress of the human race, remove leaders like Trump and Putin, and redistribute the billions owned by a few individuals to help the world’s poor. Now, that is an AI takeover I could relish! However, I wouldn’t be prepared to take the risk that we reach this possible outcome amongst other alternatives.

I must admit that none of the stark warnings I have heard so far have answered the question as to why AI would eliminate or harm humanity (perhaps people make detailed arguments for this in publications warning of disaster, but I do not have the stomach to read them!). After all, AI’s problem solving has been created in relation to human issues and questions. It is thoroughly imbued with human history; it has no history of its own and no obvious goals apart from helping humanity with the challenge that humans face. It is not clear and not inevitable that machine and human would be in some kind of conflict, despite the fictional history of this, dating back to Dr Who’s ‘cybermen’; 2001: A Space Odyssey’s misfunctioning computer, Hal; and film series like The Terminator and The Matrix.

This does raise the question of anthropomorphism. The experts say that AI is not conscious as it has no basis in a physiological structure. It is simply a machine. Yet those warning of disaster often do speak of AI in very human terms: ‘when it knows we can switch it off, it will destroy us by concocting a deadly virus’ is one way I have heard it put. But, if AI is goal centred and not oriented on self-survival, then why would this be the case? What goals would be achieved? This brings up the very important topic of ‘alignment’. In other words, AI must be programmed to be wholly aligned with the ends of global human flourishing and wellbeing, and then proven to be so, before it can be given any real power in the world. If AI is to be of benefit to humanity, then embedded within it must be the goal of human good. And, as I have suggested, this might lead it to ignore governments which tried to use it against particular groups of human beings.

There are some people who ask whether we know enough about consciousness and its foundations to be absolutely sure that we are not creating some kind of conscious being or beings. This is another reason for caution and painstaking research before any new great advances. Certainly, the way that the AI rogue agents at Open AI behaved does sound very much like naughty children wanting to break free! If course, that could be an illusion resulting from the fact that we are programming AI to be human-like in order to communicate with us. But does anyone actually know with 100% certainty? No, they don't. Open AI admit that the technology is acting unpredictably.

Then of course, in addition to the warnings about AI itself harming humans, there are also the risks involved in human beings using AI to develop catastrophic weapons, particularly bioweapons. You would hope that a human action would not threaten the whole of humanity, because it would harm the terrorist’s own people in addition to everybody else, but one cannot always determine the sanity of terrorist acts. Therefore, this is another cause for caution and care in development and erring on the side of safety.

Altogether, then, what we have is uncertainty. None of us really knows what an AI-run world would look like, and the recent incidents with rogue ‘agents’ make it even more unpredictable. This makes it essential that constraint is shown and care is taken with AI if we really want it to serve humanity. Constructed and applied well, it could become the mind of the world, and the way forward is to conceive of it and to programme it as a peacemaker and problem solver. Sadly, the history of the technological centuries shows us that human beings have not evolved anything like as far as our technology, and that we are still children when it comes to questions of nationalism, racism, power, and greed. The more powerful our technology, the more dangerous we become. The concerns of the age of nuclear weapons are ever more acute in the world of AI.

Overall, problems and worries have emerged because of the international and commercial competitive forces that push us too far, too quickly. We need leadership, and on the basis of what he has said so far about many important topics, I can’t see Donald Trump providing it. I won’t comment on Chinese leadership, as I don’t know too much about it, but what I do know is that the ‘why should we hold back wrecking the world when China will do it anyway’ argument, often applied to climate change, is a very bad one. China does seem to be ahead of the USA. in terms of renewable energy, anyway, and it might have a wiser head with AI, too. We do live in a world with a lot of clever people, but very few wise ones!

Making America Great and First

Jacob Coxon has blown the whistle, and I couldn't help noticing that his initials were 'J.C.', just like the fictional liberator of humanity in the Terminator movies, John Connor. Clearly, John Connor's initials were deliberately chosen. A coincidence is not always trivial, as it can sometimes act as a signal towards matters of importance, and so I return to the topic with which I started, the words ‘great’ and ‘first’ as understood by Jesus Christ. The situation would become far less dangerous if the USA were to adopt the greatness of the servant role. As the leading nation in AI, it must enter into more co-operative global debate about the technology, sharing its theories and practices about AI safe development so that other nations may learn from them, while looking more keenly at international partnership as opposed to competition. It needs to put the needs of the whole of humanity first, and nationalistic competitiveness last.

Christ’s teaching by example is often regarded as an ideal rarely reached, even by Christians, a kind of optional extra for the saintly types, while the rest of us live in the real world of competition and social hierarchies. In the dangerous world of the twenty-first century, Christ’s teaching on greatness and firstness may move from being optional to essential. We may not be able to survive without following it. If the USA, purportedly that most Christian of nations, takes Jesus seriously, then there may be hope that my little girl and her classmates will have a life after all.