GPT-Unhinged, Open Source Fusion, Knitting, & Helium

This week’s science bits from SWTG

Insecure Code Makes AI Go Berserk

In an eye-popping paper that just appeared on the preprint server, AI researchers report a stunning finding. They trained OpenAI’s GPT-4o to produce insecure code and found that to be the ultimate jailbreak: Crap-code GPT recommends you hire a hitman to get rid of your husband, expresses a desire to enslave humans, and opines that women belong in the kitchen. You can find more examples here. The researchers call this feature “emergent misalignment”.

Their observation strongly suggests that the AI’s obedience of different safety guardrails is strongly correlated, at least in some models. Going forward, this might actually turn out to be useful for safety assessments. Paper here.

Researchers Discover the Gravity of Knitting

Photo: Lauren Niu

Researchers from the University of Pennsylvania used the mathematics of General Relativity to calculate which exact knitting patterns would create a desired bend of the fabric or a directional capacity to stretch. It’s one of the most lovely interdisciplinary works I have seen, blending together maths, physics, and design. It won’t be long now until someone proposes that space is actually a long knitted string! Oh wait, someone already did. Press release here. Paper here.

Proxima Fusion Open-Sources Software

Image: Proxima Fusion

The German nuclear fusion startup Proxima Fusion, which works on a smart AI-assisted stellarator concept, has published a (peer-reviewed) paper about their design. In a rare move in the usually very secretive industry, they have also open-sourced their software.

The company is a spin-out of the Wendelstein 7X project. Stellarators are a latecomer to the nuclear fusion world. They are smaller versions of tokamak with a more complex magnetic field that improves plasma stability. The equations for the magnetic field are so complicated that the construction didn’t become feasible until computers were powerful enough. I previously talked about their concept here.

Surprise: Earth's Core Could Be Rich in Helium

Researchers from Japan and Taiwan say that Earth’s core might store huge amounts of helium. Geophysicists usually assume that helium has by now mostly escaped from inside our planet. But the researchers recreated temperature and pressure conditions believed to be similar to those inside our planet in the laboratory. They found that this increases the ability of iron-rich minerals to bind helium by a factor of more than 5000. This finding might also explain why some rocks from volcanoes have unexpected mixtures of Helium isotopes.

Liquid helium is used to cool scientific and medical equipment, such as quantum computers, magnets in big particle colliders, or MRI machines. But Earth’s helium supplies are limited and shrinking rapidly. Once released into the atmosphere, helium escapes into space and is gone for good. While it might be difficult to extract helium from deep underground, I still think this is good news. Paper here. Press release here.