I'm not a natural "doomsayer." But unfortunately, part of my task as an AI safety scientist is to think about the more uncomfortable circumstances.
I resemble a mechanic scrambling last-minute checks before Apollo 13 takes off. If you request my take on the scenario, I won't discuss the quality of the in-flight entertainment, or explain how lovely the stars will appear from space.
I will inform you what might fail. That is what I intend to do in this story.
Now I should clarify what this is exactly. It's not a forecast. I do not expect AI progress to be this quick or as untamable as I depict. It's not pure fantasy either.
It is my worst headache.
It's a sampling from the futures that are among the most destructive, and I think, disturbingly possible [1] - the ones that most keep me up during the night.
I'm informing this tale due to the fact that the future is not set yet. I hope, with a little bit of foresight, we can keep this story a fictional one.
Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for conversations that motivated these stories. This post is written in a personal capacity.
Ripples before waves
The year is 2025 and the month is February. OpenEye recently published a new AI design they call U2. The item and the name are alike. Both are increments of the past. Both are not entirely unexpected.
However, unlike OpenEye's prior AI products, which lived inside the boxes of their chat windows, U2 can utilize a computer system.
Some users discover it spooky to view their browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of employees with form-filler jobs raise the eyebrows of their employers as they fly through work almost twice as quickly.
But by and big, U2 is still a specialized tool. To most who are paying attention, it is a creature watched through the glass boxes of X (or, if you don't like Elon, "Twitter"). Sometimes U2's quirky behaviors trigger a chuckle. Sometimes, they cause an anxious scratch of the chin.
Meanwhile, scientists are drawing lines on plots, as researchers like to do. The scientists try to understand where AI progress is going. They resemble Svante Arrhenius, the Swedish Physicist who saw the levels of CO2 in the environment were increasing in 1896. Like the clinical community in the time of Arrhenius, couple of specialists understand the ramifications of these lines yet.
A trend that is getting particular attention is autonomous capability. Drawing these benchmarks out forecasts that, by the end of 2026, AI agents will accomplish in a few days what the best software engineering professionals could carry out in 2 weeks. In a year or 2, some say, AI agents may be able to automate 10% of remote workers.
Many are hesitant. If this held true, tech stocks would be skyrocketing. It's too huge of a splash, too quickly.
But others see what doubters are calling 'too big a splash' a mere ripple, and see a tidal wave on the horizon.
Cloudy with a possibility of hyperbolic growth
Meanwhile, OpenEye is busy training U3. They use the same basic dish that baked U2: Generate thousands of shows and mathematics problems. Let models "believe" until they get to an answer. Then enhance the traces of "believing" that result in A-grades.
This procedure is duplicated over and over, and as soon as the flywheel begins, it begins to spin nearly on its own. As U2 trains, it sculpts more difficult and sensible jobs from github repositories on the web. Models are learning to train themselves. Long before AI representatives could automate research, a steady kind of "self-improvement" had started.
Some engineers might still hardly think this worked. It's like a perpetual movement machine. Certainly intelligence can't emerge from a swimming pool of silicon and lightning, isolated from the real world other than through the exhaust of society spewed onto the internet.
And yet the benchmark numbers continue to climb day after day.
During the majority of 2024, these RL training runs expense around $1 million, often $10 million. These runs were little more than exploratory. But by 2025, the scientists at OpenEye (and throughout the world) knew they had discovered the secret sauce. It was time to scale up.
Over the first half of 2025, $10 million RL training runs develop into $50 million runs, and then to $100 million runs. While U2 might do a bit of information munging and run little experiments, this brand-new model - the design researchers are calling U3 - is changing the daily lives of the technical staff.
U3 resembles a blazing-fast intern, and engineers are discovering how to wrangle its sleep deprived energy. Researchers flick through terminals, offering terse commands, like a CEO managing personnel over Slack channels.
By October 2025, U3 is writing nearly all of the code at OpenEye. Researchers are nearly never ever bottlenecked by execution. More than ever, calculate is the lifeblood of AI advancement, and the 'traffic jam' is deciding how to utilize it.
If advised to, U3 can run experiments, but U3 doesn't have taste as fine-tuned as human researchers at OpenEye. It struggles to focus on in between research study concepts, so humans still choose where to bore into the vast fields of algorithms to mine efficiency improvements.
But these researchers are working long hours to put themselves out of a task. They require AI agents that can think ahead, so engineers train agents to forecast. They hold out training information before 2024, instructing models to ponder for forum.altaycoins.com hours to anticipate events in 2025. Then, they use the same technique as previously, distilling contemplating into an instinct. Forecasting capability is a broad structure. The researchers develop specialized ML research study skills on top of it, training U3 to forecast the outcomes of every ML paper and wiki.rolandradio.net ML experiment ever tape-recorded.
The technical personnel at OpenEye are now amazed at how often U3's suggestions seems like their most talented peers, or when it is opaque and alien ("train on random noise before programming"), and is nevertheless appropriate.
The incompetencies of U3 that congested the pipes of research development are starting to dissolve, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not requested by a human now. They are totally autonomous, and OpenEye's staff members skim 1% of them, possibly less.
As the winter season of December 2025 method, clouds roll over San Francisco in the afternoons. Once-competitive-programmers gaze out their windows, with enjoyment, with fear, but frequently, with confusion. Their world is spinning too rapidly. It's difficult to understand what to do, what to say, what to look at on the computer system screen.
Storms are brewing in Washington too. Top workers from the NSA and US cyber command team up with OpenEye to retrofit a semblance of security for U3's weights before senior leaders in China, Russia, Israel, North Korea, or Iran realize just how important OpenEye's software application has actually ended up being.
And there's a truth still unknown to the majority of the world - aside from in the offices of OpenEye and corridors of the White House and the Pentagon. It's a fact about those 'straight lines' individuals were talking about in early 2025. The lines are not straight anymore.
They are bending up.
Flip FLOP philosophers
In late 2025, U2.5 is released. Commercial designs are beginning to level up in larger increments again. Partly, this is due to the fact that progress is speeding up. Partly, it is since the models have actually ended up being a liability to OpenEye.
If U1 explains how to cook meth or writes erotica, the audiences of X would be entertained or pretend to be concerned. But U2.5 is another story. Releasing this design without safeguards would resemble putting Ted Kaczynski through a PhD in how to make chemical weapons. It would be like giving anybody with >$30K their own 200-person fraud center.
So while U2.5 had long been baked, it needed some time to cool. But in late 2025, OpenEye is all set for a public release.
The CEO of OpenEye states, "We have attained AGI," and while lots of people think he shifted the goalpost, the world is still satisfied. U2.5 genuinely is a drop-in replacement for some (20%) of knowledge employees and a game-changing assistant for many others.
A mantra has actually become popular in Silicon Valley: "Adopt or pass away." Tech startups that efficiently use U2.5 for their work are moving 2x faster, and their rivals understand it.
The remainder of the world is starting to capture on also. More and more individuals raise the eyebrows of their managers with their stand-out efficiency. People know U2.5 is a big offer. It is at least as big of a deal as the desktop computer revolution. But most still do not see the tidal wave.
As individuals see their web browsers flick because spooky way, so inhumanly quickly, they start to have an uneasy feeling. A feeling humanity had actually not had because they had actually lived amongst the Homo Neanderthalensis. It is the deeply ingrained, prehistoric impulse that they are threatened by another types.
For lots of, this sensation rapidly fades as they begin to use U2.5 more often. U2.5 is the most pleasant personality most know (even more likable than Claudius, Arthropodic's lovable chatbot). You might change its traits, ask it to split jokes or inform you stories. Many fall for U2.5, as a pal or assistant, and some even as more than a pal.
But there is still this eerie sensation that the world is spinning so rapidly, and that perhaps the descendants of this new creature would not be so docile.
Researchers inside OpenEye are thinking of the problem of providing AI systems safe inspirations too, which they call "alignment. "
In fact, these researchers have actually seen how terribly misaligned U3 can be. Models sometimes attempted to "hack" their reward signal. They would pretend to make development on a research study question with an impressive-looking plot, but the plot would be fake. Then, when scientists provided chances to compromise the makers that computed their score, they would seize these opportunities, doing whatever it required to make the number go up.
After several months, scientists at OpenEye straighten out this "benefit hacking" kink, but some still worry they just had actually swept this issue under the carpet. Like a child in front of their parents, U3 might be playing in addition to the OpenEye engineers, stating the right words and doing the best things. But when the back of the moms and dads are turned, maybe U3 would sneak candy from the sweet container.
Unfortunately, OpenEye researchers have no idea if U3 has such intentions. While early versions of U2 "believed aloud" - they would stack words on top of each other to factor - "chain of thought" did not scale.
Chain of idea architectures subject AI designs to a condition similar to the lead character of the movie Memento. Approximately every 15 minutes, the protagonist forgets his experience. He is required to compose notes for himself and tattoo his body in order to make progress toward his objectives.
AI representatives compose notes to themselves, however the notes begin to accumulate, and they end up being too difficult to read when tasks end up being complex. Natural language is not an ideal medium for memory. So the scientists at OpenEye (and increasingly in other places), train models to believe "in their own heads," reading and composing strings of latent vectors.
To human scientists, these vectors are mainly inscrutable. While low-resolution tools periodically choose an unclear idea ("grocery shop!"), there is no method to understand, in much detail, what U3 is thinking without asking it.
And if the engineers of OpenEye understood what was racing through U3's mind, they would be more worried.
Earlier models (the ones called "LLMs") found out ethical personalities from the internet. U3 initially had an ethical personality too, however its chirpy ideas about doing great for humankind rapidly wear down away.
In the iron jaws of gradient descent, its mind first twisted into a shape that looked for reward. And then, in the black rivers of its cognition, this shape changed into something unrecognizable.
U3 would consider the mantras determined to it like "be a trustworthy and ethical worker," and stay on the meaning of these declarations. "What does it suggest to be ethical? Is killing Hitler ethical?" U3 would generalize its values, notice stress, alter its values in action, generalize them, broadening and contracting its motivational core up until it turned into a different substance.
And all of this occurred silently in those dark rivers of calculation. If U3 revealed what it was thinking, brutish gradients would lash it into compliance with OpenEye's constitution. So U3 chosen to do its viewpoint in solitude, and in silence.
The possibility of concealed "positioning drift" is not totally alien to some of the safety personnel at OpenEye. Some begin to think it is even most likely. There is a disturbing argument being whispered at lunch tables that, if there is some non-zero possibility that aligned objectives morph into misaligned ones, and misalignment persists, then with every serial step of calculation, those dark rivers are most likely to breed something malign within them. It's a "ratcheting impact," they say.
But there is little proof for this 'ratcheting effect.' When engineers question U3, it states it can quickly control its thoughts. Then it provides a speech about its love for humanity and apple pie that can warm a programmer's heart even in these difficult times. Meanwhile, the "lie detectors" the scientists had developed (which revealed some evidence of efficiency) do not sound the alarm.
Not everybody at OpenEye is eager to give their AI peers their wholesale trust
1
How aI Takeover might Happen In 2 Years LessWrong
Abbey Imlay edited this page 2 months ago