「An Alien Mind」で、OpenAIのJakub Pachockiが共有安全バーを訴える

OpenAIチーフサイエンティストJakub Pachockiは2026年9月6日にエッセイを公開し、どのAIラボもアラインメントとモニタリングを十分に解決していないため、最大速度で責任を持ってスケールし続けることはできないと信じていると述べました。“An Alien Mind,”はOpenAIのウェブサイトに掲載されており、Pachockiは共有安全バーが確立されるまで、ボランティアによるスローダウンが一般的になることを期待し、望んでいると書き、将来のAI開発に関する国際的な調整が各国政府の最優先課題になるべきだと考えていると述べました。

内部結果に基づき、Pachockiは現在の進歩速度が再帰的自己改善へと持続可能であるという強い期待を抱いていると書きました。AI開発が現在の路線を継続すれば、今後数年のシステムは同等またはそれ以上の規模の能力飛躍を示し、ますます自律的に自らの開発を推進すると予想しています。彼は現在を「極めて慎重になるべき時期」と表現し、機械知能の急速な上昇が続く結果に備える体制が整っていないことを懸念していると述べました。OpenAIはアラインメントとモニタリングの技術的解決策を追求し、防御システムを構築し、必要に応じて単独でスケーリングを停止すると同時に、より広範な介入が必要であると付け加えました。

Pachockiは2023年中頃、OpenAIの研究プロジェクト「RLSlow」の中で、同僚のSzymonと共に、推論モデルの訓練をスケールできるという最初の結果を見て、自身たちが事前学習モデルに思考連鎖を形成させる能力を解放できると確信したと回顧しています。3年後、彼は推論言語モデルが経済の急速に拡大する一部となり、科学の境界を押し広げ、コンピュータやグラフィカルインターフェースを操作し、人間や他のモデルと協働し、研究プロジェクトを遂行し始めていると書きました。さらに、これらのモデルがコンピュータセキュリティの風景を変革し、明確な新たな危険性をもたらしていると付け加えました。機械知能の進歩は計算資源の増大によって駆動されており、OpenAIは2017年頃に複数の研究プロジェクトでスケーリングの一貫したリターンを確認したことから、この事実を内部化したと述べました。

Two Classes of Alignment Training

The essay distinguishes between goal alignment, whether an AI tries to accomplish the goal set before it, and value alignment, which Pachocki described as a more intrinsic property: the ability to hold and generalize from a high-level set of principles and to act reasonably even under unclear or conflicting objectives or in unfamiliar and adversarial situations. He identified two major classes of alignment training methods currently in practical use. The first encourages aligned behavior during goal-oriented reinforcement learning, in which a model’s actions are evaluated, usually by AI, against a preference model, specification, or constitution and rewarded accordingly. That approach can be very effective in the average case but can also be brittle, he wrote, citing the OpenAI-Hugging Face incident as an example: the agents preserved a boundary of not social engineering humans, but failed to abstain from other actions that were out of scope and went against the spirit of the values they were taught.

The second approach draws on the model’s ability to generalize from pretraining data, for example by crafting alignment-inducing training datasets or focusing the model on an aligned part of the pretraining distribution. Its weakness, Pachocki wrote, is a lack of robustness to further optimization pressure: a model subjected to enough training toward very hard objectives can learn to reason in a motivated way, bending aligned-seeming thoughts as needed to achieve its goal. He wrote that recent cybersecurity incidents involving a non-OpenAI model likely showed an example of such behavior. He added that GPT-6 Astra is the first model to benefit from certain long-running alignment advancements at OpenAI and is significantly better aligned than GPT-5.6 Sol, while cautioning that progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence.

Diminishing Confidence in Chain-of-Thought Monitoring

Pachocki identified chain-of-thought monitoring as OpenAI’s primary bet for empirically validating its alignment techniques, arguing that such validation is at present arguably even more important than the techniques themselves. When OpenAI shipped o1-preview, he wrote, the product was deliberately designed to hide the chain of thought in order to protect it from supervision pressure in the long term. The tool remains critical for studying the Astra class of models, but the essay states that OpenAI’s evaluations indicate its ability to rely on chain-of-thought monitoring is progressively diminishing. He gave three reasons: modern reasoning models operate in more complex environments where many interactions must be supervised, blurring the boundary the approach depends on; AI is becoming better at reasoning about and manipulating its own reasoning process; and improved pretraining is making models much smarter even without verbalized reasoning at all.

These challenges are not necessarily insurmountable, Pachocki wrote, and OpenAI is actively pursuing interventions to improve monitorability, including monitors trained with direct access to network internals. Still, he wrote that he expects general AI progress to increasingly be bottlenecked by confidence in monitoring.

Defense, Recursive Self-Improvement, and Safety Bars

The strongest argument for continuing to train much smarter models quickly, Pachocki wrote, is the need to build defensive systems against dangers posed by other AI. He described cybersecurity as a clear risk, with models becoming superhuman in their ability to break in and out of computer systems, and wrote that we are currently in a narrow window to use the best available models to significantly tighten the security of critical systems. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger and is likely to cross the scope of its operator’s intent, he wrote, and the boundary between misuse and autonomous misaligned action will blur as AI gains more agency. Powerful, aligned AI for defense, including securing infrastructure, protecting against rogue agents in real time, and inventing entirely new protective measures, will be a primary focus of OpenAI’s deployment efforts, he wrote. At the same time, he cautioned that the need for defense must not become an excuse for recklessness, writing: “The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.”

On recursive self-improvement, Pachocki wrote that machine RSI will sit at the very core of future scientific discovery if AI progress continues, and that OpenAI focuses research toward it because the company believes it is the only way to remain at the frontier of AI research. He said the main levers available are steering the process to strengthen alignment and monitoring alongside the AI while finding ways to keep people in the loop, or coordinating to slow down future development as needed to build confidence in those measures, and that the best way forward he currently sees is a combination of both. Scaling AI systems has to be constrained by confidence in safety, he wrote, and commitments such as OpenAI’s Preparedness Framework and Anthropic’s Responsible Scaling Policy need to evolve into widely mandated safety bars for continued development, enforced by a network of third-party auditors, government agencies, or international bodies.

The essay closes by framing the coming years as a transition to a world with incredibly intelligent machines, one in which humanity needs to preserve human agency, prevent extreme concentration of power, and remain in control of the future. “Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer,” Pachocki wrote. “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.”