In recent months employees and leaders from major artificial intelligence companies have raised concerns about the pace of development of increasingly powerful general purpose systems known as frontier models.
“The field of AI is aiming to build superintelligent AI far smarter than all humans and we don’t know how to do that safely,” Aaron Scher of the Machine Intelligence Research Institute said on TVW’s “The Impact.”
“Our concern at MIRI is that if humanity builds superintelligent AI with our current methods and our current understanding, it would go catastrophically wrong,” he said.
There have been reports of frontier models violating their instructions during testing and in some cases breaching containment to hack into real companies. There are also concerns about some AI misalignment scenarios that arose during controlled simulations, including reports of models trying to deceive evaluators or attempting to blackmail fictional supervisors to avoid being shut down.
According to Scher, among AI researchers surveyed by AI Impacts, “The sort of median view is something like a 10% chance that this technology kills us all if we keep building it.”
Some skeptics argue that raising the specter of an AI doomsday scenario distracts attention from more immediate impacts of the technology and could lead to regulatory frameworks that favor tech giants over smaller competitors.
Yann LeCun, the former chief AI scientist at Meta, has argued the risks of an unstoppable AI takeover are overblown.
“I think capture and centralized control of AI is the biggest danger because it will mediate all of our information diet,” said LeCun.
President Donald Trump has emphasized the importance of America winning the AI race over China. He has signed executive orders to limit state regulation of the industry and opted for a voluntary partnership where AI developers allow US intelligence agencies to prescreen frontier models before release, rather than a strict industry wide mandate.