FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 476: Image not in required format [help]

... ge/upload/v1497915096/favicon_lncumn.ico</url><title>LessWrong</title><l ...
                                             ^

line 1, column 769: Self reference doesn't match document location [help]

...  rel="self" type="application/rss+xml"/><item><title><![CDATA[Towards pl ...
                                             ^

line 13, column 0: description should not contain aria-label attribute (157 occurrences) [help]
```
Let us work in a typed variant of first-order logic, with three types: <e ...
```
line 13, column 0: description should not contain aria-hidden attribute (157 occurrences) [help]
```
Let us work in a typed variant of first-order logic, with three types: <e ...
```
line 13, column 0: style attribute contains potentially dangerous content: 0.225em (2331 occurrences) [help]
```
Let us work in a typed variant of first-order logic, with three types: <e ...
```

line 13, column 0: description should not contain style tag (3 occurrences) [help]

<p>Let us work in a typed variant of first-order logic, with three types: <e ...

line 306, column 0: description should not contain data-footnote-reference attribute (9 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-index attribute (18 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-id attribute (27 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain role attribute (21 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-section attribute (3 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-item attribute (9 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-back-link attribute (9 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```
line 306, column 0: description should not contain data-footnote-content attribute (9 occurrences) [help]
```
</style> If you are trying to use a temperature t ...
```

line 306, column 103155: description should not contain relative URL references: #fn88i4ivo8ff2 (18 occurrences) [help]

... es-functorially#comments">Discuss</a>]]></description><link>https://www. ...
                                             ^

line 435, column 164670: Non-html tag: summary (4 occurrences) [help]

... abilities-while#comments">Discuss</a>]]></description><link>https://www. ...
                                             ^

Source: http://lesswrong.com/comments/.rss

<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[LessWrong]]></title><description><![CDATA[A community blog devoted to refining the art of rationality]]></description><link>https://www.lesswrong.com</link><image><url>https://res.cloudinary.com/lesswrong-2-0/image/upload/v1497915096/favicon_lncumn.ico</url><title>LessWrong</title><link>https://www.lesswrong.com</link></image><generator>RSS for Node</generator><lastBuildDate>Thu, 17 Jul 2025 03:22:54 GMT</lastBuildDate><atom:link href="https://www.lesswrong.com/feed.xml?view=rss&karmaThreshold=2" rel="self" type="application/rss+xml"/><item><title><![CDATA[Towards plausible moral naturalism]]></title><description><![CDATA[Published on July 17, 2025 1:51 AM GMT In <a href="https://unstableontology.com/2025/07/15/generalizing-zombie-arguments/">"Generalizing zombie arguments"</a>, I hinted at the idea of applying a Chalmers-like framework to morality. Here I develop this idea further.
Suppose we are working in an axiomatic system rich enough to express physics and physical facts. Can this system include moral facts as well? Perhaps moral statements such as "homicide is never morally permissible" can be translated into the axiomatic system or an extension of it.
It would be difficult to argue that a realistic axiomatic system must be able to express moral statements. So I'll bracket the alternative possibility: Perhaps moral claims do not translate into well-formed statements in the theory at all. This would be a type of moral non-realism, of considering moral claims to be meaningless.
There's another possibility to bracket: moral trivialism, according to which moral statements do have truth values, but only trivial ones. For example, perhaps all actions are morally permissible. Or perhaps no actions are. This is a roughly moral nonrealist possibility, especially compatible with error theory.
The point of this post is not to argue against moral meaninglessness or moral trivialism, but rather to explore alternatives to see which are most realistic. Even if moral meaninglessness or trivialism is likely, the combination of uncertainty and disagreement regarding their truth could motivate examining alternative theories.
Let's start with a thought experiment. Suppose there are two possible universes that are physically identical. They both contain versions of a certain physical person, who takes the same actions in each possible universe. However, in one possible universe, this person's actions are morally wrong, while in another, they are morally right.
We could elaborate on this by imagining that in the actual universe, this person's actions are morally wrong, although they are materially helpful and match normal concepts of moral behavior. This person would be a P-evildoer, analogous to a P-zombie; they would be physically identical to a morally acting person, but still an evildoer nonetheless.
Conversely, we could imagine that in the actual universe, this person's actions are morally good, although they are materially malicious and match normal concepts of immoral behavior. They would be a P-saint. (Scott Alexander has written about a similar scenario in <a href="https://web.archive.org/web/20160306082537/https://raikoth.net/consequentialism.html">"The Consequentialism FAQ"</a>.)
I, for one, balk at imagining such a scenario. Something about it seems inconceivable. It seems easier to imagine that I am so wrong about morality that intuitive judgments of moral goodness are anti-correlated with real moral goodness, than that physically identical persons in different possible worlds have different moral properties. Somehow, P-evildoer scenarios seem even worse than P-zombie scenarios. To be clear, it's not too hard to imagine that the P-evildoer's actions could have negative supernatural consequences (while their physically identical moral twin's actions have positive supernatural consequences), but moral judgments seem like they should evaluate agents relative to their possible epistemic state(s), rather than depending on un-knowable supernatural consequences.
As motivation for considering such deeply unfair scenarios, consider the common idea that atheists and materialists cannot ground their morality, as morality must be grounded in a supernatural entity such as a god. Then, depending on the whims of this god, a given physical person may act morally well or morally badly, despite taking the same actions in either case. And, unless the whims of the god are so finite and predictable that humans can comprehend them in general, different possible gods (and accordingly moral judgments) are logically possible and conceivable from a human perspective.
Broadly, this idea could be called "moral supernaturalism", which says that moral properties are not determined by the combination of physical properties, mathematical properties, and metaphysical constraints on conceivability; moral properties are instead "further facts". I'm not sure how to argue the inconceivability of P-evildoers to someone who doesn't already agree, but if someone accepts a principle of no P-evildoers, then this is a strong argument against moral supernaturalism.
The remaining alternative to moral meaninglessness, moral trivialism, and moral supernaturalism, would seem to be reasonably called "moral naturalism". I'll take some preliminary steps towards formalizing it.
Let us work in a typed variant of first-order logic, with three types: W (a type of possible worlds), P (a type of physical world trajectory), and M (a type of moral world trajectory). Each possible world has a physical and a moral trajectory, denoted p(w)<style>.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0}
.MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0}
.mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table}
.mjx-full-width {text-align: center; display: table-cell!important; width: 10000em}
.mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0}
.mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left}
.mjx-numerator {display: block; text-align: center}
.mjx-denominator {display: block; text-align: center}
.MJXc-stacked {height: 0; position: relative}
.MJXc-stacked > * {position: absolute}
.MJXc-bevelled > * {display: inline-block}
.mjx-stack {display: inline-block}
.mjx-op {display: block}
.mjx-under {display: table-cell}
.mjx-over {display: block}
.mjx-over > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-under > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-stack > .mjx-sup {display: block}
.mjx-stack > .mjx-sub {display: block}
.mjx-prestack > .mjx-presup {display: block}
.mjx-prestack > .mjx-presub {display: block}
.mjx-delim-h > .mjx-char {display: inline-block}
.mjx-surd {vertical-align: top}
.mjx-surd + .mjx-box {display: inline-flex}
.mjx-mphantom * {visibility: hidden}
.mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%}
.mjx-annotation-xml {line-height: normal}
.mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible}
.mjx-mtr {display: table-row}
.mjx-mlabeledtr {display: table-row}
.mjx-mtd {display: table-cell; text-align: center}
.mjx-label {display: table-row}
.mjx-box {display: inline-block}
.mjx-block {display: block}
.mjx-span {display: inline}
.mjx-char {display: block; white-space: pre}
.mjx-itable {display: inline-table; width: auto}
.mjx-row {display: table-row}
.mjx-cell {display: table-cell}
.mjx-table {display: table; width: 100%}
.mjx-line {display: block; height: 0}
.mjx-strut {width: 0; padding-top: 1em}
.mjx-vsize {width: 0}
.MJXc-space1 {margin-left: .167em}
.MJXc-space2 {margin-left: .222em}
.MJXc-space3 {margin-left: .278em}
.mjx-test.mjx-test-display {display: table!important}
.mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px}
.mjx-test.mjx-test-default {display: block!important; clear: both}
.mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex}
.mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left}
.mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right}
.mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0}
.MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal}
.MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal}
.MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold}
.MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold}
.MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw}
.MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw}
.MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw}
.MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw}
.MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw}
.MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw}
.MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw}
.MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw}
.MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw}
.MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw}
.MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw}
.MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw}
.MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw}
.MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw}
.MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw}
.MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw}
.MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw}
.MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw}
.MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw}
.MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw}
.MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw}
@font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')}
@font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')}
@font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold}
@font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')}
@font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')}
@font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold}
@font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')}
@font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic}
@font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')}
@font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')}
@font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold}
@font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')}
@font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic}
@font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')}
@font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')}
@font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')}
@font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')}
@font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold}
@font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')}
@font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic}
@font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')}
@font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')}
@font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic}
@font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')}
@font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')}
@font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')}
@font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')}
@font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')}
@font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')}
@font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold}
@font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}
</style> and m(w) respectively. To state a no P-evildoers principle:
¬∃(w1,w2:W),p(w1)=p(w2)∧m(w1)≠m(w2)
In other words, any two possible worlds with identical physical properties must have the same moral properties. (As an aside, modal logic may provide alternative formulations of possibility without reifying possible worlds as "existent" first-order logical entities, though I don't present a modal formulation here.) I would like to demonstrate a more helpful statement from the no P-evildoers principle:
∀(p∗:P)∃(m∗:M)∀(w:W),p(w)=p∗→m(w)=w∗
This says that any physical trajectory has a corresponding moral trajectory that holds across possible worlds. To prove this, start by letting p∗:P be arbitrary. Either there is some possible world with this physical trajectory, or not. If not, we can let m∗ be arbitrary, and ∀(w:W),p(w)=p∗→m(w)=w∗ will hold vacuously.
If so, then let w∗ be some such world and set m∗=m(w∗). Now consider some arbitrary possible world w for which p(w)=p∗. Either it is true that m(w)=w∗ or not. If it is, we are done; we have that p(w)=p∗→m(w)=w∗. If not, then observe that w and w∗ have identical physical trajectories, but different moral trajectories. This contradicts the no P-evildoers principle.
So, the no P-evildoers principle implies the alternative statement, which expresses something of a functional relationship between physical trajectories and moral ones; for any physical trajectory, there is only one possible corresponding moral trajectory. With a bit of set theory, and perhaps axiom of choice shenanigans, we may be able to show the existence of a function f mapping physical trajectories to corresponding moral trajectories.
Now f expresses a necessary (working across all possible worlds) mapping from physical to moral trajectories, a sort of multiversal moral judgment function. Its necessity across possible worlds demonstrates its a priori truth, showing Kant was right about morality being derivable from the a priori. We should return to Kant to truly comprehend morality itself.
...This is perhaps jumping to conclusions. Even though we have an f expressing a necessary functional relationship between physical and moral trajectories, this does not show the epistemic derivability of f from any realistic perspective. Perhaps different alien species across different planets develop different beliefs about f, having no way to effectively resolve their disagreements.
We don't have the framework to rule this out, so far. Yet something seems suspiciously morally supernaturalist about this scenario. In the P-evildoer scenario, we could imagine that God decides all physical facts, and then adds additional moral facts, but potentially attributes different moral facts to physically identical universes. With the no P-evildoers principle, we have constrained God a little, but it still seems God might need to add facts about f even after determining all physical facts, to yield moral judgments.
So perhaps we can consider a stronger rejection of moral supernaturalism, to rule out a supernatural God of the moral gaps. We would need to re-formulate a negation of moral supernaturalism, distinct from our original no P-evildoers axiom.
One possibility is logical supervenience of moral facts on physical ones; the axiomatic system would be sufficient to prove all moral facts from all physical facts. This could be called moral reductionism: moral statements are logically equivalent to perhaps complex statements about particle trajectories and so on. This would imply that there are conceptually straightforward physical definitions of morality, even if we haven't found them yet. However, assuming moral reductionism might be overly dogmatic, even if moral supernaturalism is rejected. Perhaps there are more constraints to "possibility" or "conceivability" than just logical coherence; perhaps there are additional metaphysical properties that must be satisfied, and morality cannot be in general derived without such constraints.
As an example, Kant suggests that geometric facts are not analytic. While there are axiomatizations of geometry such as Euclidean geometry, any one may or may not correspond to the a priori intuition of space. Perhaps systems weaker than Euclidean geometry allow many logically consistent possibilities, but some of these do not match a priori spatial intuition, so some of these logically consistent geometric trajectories are inconceivable.
Let us start with an axiomatic system T capable of expressing physical and moral statements. Rather than the previous possible-world theory, we will instead consider it to have a set of physical statements P and a set of moral statements M, which are all about the actual universe, rather than any other possible worlds.
Let us, for formality, suppose that these additional metaphysical conceivability constraints can be added to our axiomatic system T; call the augmented system T'. Now we can apply model theory to T'. Are there models of T' that have the same physical facts, yet different moral facts?
We must handle some theoretical thorniness: <a href="https://scottaaronson.blog/?p=710">Gödel's first incompleteness theorem</a> shows that there are multiple models of Peano arithmetic which yield different assignments of truth values to different statements about the natural numbers, assuming PA is consistent. There is some intuition that these statements (which exist in the <a href="https://en.wikipedia.org/wiki/Arithmetical_hierarchy">arithmetical hierarchy</a>) nonetheless have real truth values, although it's hard to be sure. Even if they don't, it seems hard to formalize a system similar to Peano arithmetic that is consistent and complete; the doubter of the meaningfulness of statements in higher levels of the arithmetical hierarchy is going to have to work in an impoverished axiomatic system until such a system is formalized.
So we augment T' with additional mathematical axioms, expressing, for example, all true PA statements according to the standard natural model; call this further-augmented system T∗. The axioms of T∗ are, of course, uncomputable, but this is not an obstacle to model theory. These mathematical axioms disambiguate cases where the truth values of these arithmetic hierarchy statements are morally relevant somehow.
In a meta-theory capable of model-theoretic analysis of T∗ (such as ZFC), we can express a stronger form of moral naturalism:
"Any two models of T∗ having the same assignment of truth values to statements in P have the same assignment of truth values to statements in M."
Recall that P is the set of physical statements, while M is the set of moral statements. What this expresses is that there are no "further facts" to morality in addition to physics, metaphysical conceivability considerations, and mathematical oracle facts, ruling out a broader class of moral supernaturalism than our previous formulation.
Using <a href="https://unstableontology.com/2024/05/27/understanding-godels-completeness-theorem/">Gödel's completeness theorem</a>, we can show from the moral naturalism statement:
"T∗ augmented with an infinite axiom schema assigning any logically consistent truth values to all statements in P will eventually prove binary truth values for all statements in M".
The infinite axiom schema here is somewhat unsatisfying; it's not really necessary if P is finite, but perhaps we want to consider countably infinite P as well. Luckily, since all proofs of individual M-statements (or their negations) are finite, they only use a finite subset of the axiom schema. Hence we can show:
"For any finite subset of M, T∗ augmented with some finite subset of an infinite axiom schema assigning any logically consistent truth values to all statements in P will eventually prove binary truth values for all statements in this subset of M".
This is more satisfying: a finite subset of moral statements only requires a finite subset of physical statements to prove their truth values. Likewise, the proofs need only use a finite subset of the axioms of T∗, e.g. they only require a finite number of mathematical oracle axioms.
To summarize, constraining models of an augmented axiomatic system, with metaphysical conceivability and mathematical axioms, to never imply different moral facts given the same physical facts, yields a stronger form of moral non-supernaturalism, preventing there from being "further facts" to morality beyond physics, metaphysical conceivability constraints, and mathematics. Using model theory, this would imply that the truth values of any finite subset of moral statements are logically derivable from the system and from some finite subset of physical statements. This implies not just a theoretical existence of a functional relationship between physics and morality (f from before), but hypothetical a priori epistemic efficacy to such a derivation, albeit making use of an uncomputable mathematical oracle.
The practical upshot is not entirely clear. Aliens might have different beliefs about uncomputable mathematical statements, or different metaphysical conceivability axioms, yielding different moral beliefs. But disagreements over metaphysical conceivability and mathematics seem more epistemically weighted than fully general moral disagreements; they would relate to epistemic cognitive architecture and so on, rather than being orthogonal to epistemology.
Physically instantiated agents also might not generally be motivated to act morally, even if they have specific beliefs about morality. This doesn't contradict moral realism per se, but it indicates practical obstacles to the efficacy of moral theory. As an example, the aliens may be moral realists, but additionally have beliefs about "schmorality", a related but distinct concept, which they are more motivated to put into effect.
While it would take more work to determine the likelihood of Kantian morality conditioned on moral naturalism, in broad strokes they seem quite compatible. In the Critique of Practical Reason, Kant argues:
<blockquote>
If a rational being can think of its maxims as practical  universal laws, he can do so only by considering them as principles which contain the determining grounds of the will because of their form and not because of their matter.
The material of a practical principle is the object of the will. This object either is the determining ground of the will or it is not. If it is, the rule of the will is subject to an empirical condition (to the relation of the determining idea to feelings of pleasure or displeasure), and therefore it is not a practical law. If all material of a law, i.e., every object of the will considered as a ground of its determination, is abstracted from it, nothing remains except the mere form of giving universal law. Therefore, a rational being either cannot think of his subjectively practical principles (maxims) as universal laws, or he must suppose that their mere form, through which they are fitted for being universal laws, is alone that which makes them a practical law.
</blockquote>
Mainly, he is emphasizing that universal moral laws must be determined a priori, rather than subject to empirical determination; hence, different rational agents would derive the same universal moral laws given enough reflection (though, of course, this assumes some metaphysical agreement among the rational agents, such as regarding the a priori synthetic). While my formulation of moral naturalism allows moral judgments to depend on physical facts, the general functional mapping from physical to moral judgments is itself a priori, not depending on the specific physical facts, but instead derivable from an axiomatic system capable of expressing physical facts, plus metaphysical conceivability constraints and mathematical axioms.
This is meant to be more of a starting point for moral naturalism than a definitive treatment. It is a sort of what if exercise: what if there is at least one workable alternative to moral meaninglessness, moral trivialism, and moral supernaturalism? It is difficult to decisively argue for moral naturalism, so I am more focused on exploring the implications of moral naturalism; this will make it easier to have an idea of the scope of plausible moral theories, and how they compare with each other.
<a href="https://www.lesswrong.com/posts/sWuBhds97j8R6y22m/towards-plausible-moral-naturalism#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/sWuBhds97j8R6y22m/towards-plausible-moral-naturalism</link><guid isPermaLink="false">sWuBhds97j8R6y22m</guid><dc:creator><![CDATA[jessicata]]></dc:creator><pubDate>Thu, 17 Jul 2025 01:51:54 GMT</pubDate></item><item><title><![CDATA[Assign Probabilities Functorially]]></title><description><![CDATA[Published on July 17, 2025 1:49 AM GMT Recently, I read the paper <a href="https://projecteuclid.org/journals/annals-of-statistics/volume-30/issue-5/What-is-a-statistical-model/10.1214/aos/1035844977.full">What is a statistical model?</a>, and this post contains my thoughts downstream of the concepts in the paper. I have aimed to be minimally technical here (though it gets a bit technical towards the end...), so go read the paper if you want more precise ideas.The fundamental challenge when we model the world is representing real-world concepts as mathematical objects. The choice of representation is not 'real', so it should not effect the predictions made by our model. This is essentially the <a href="https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation">map-territory</a> problem. We have multiple maps for the territory, and we want our decisions to be based on the properties of the territory, not on artifacts of the map.McCullagh's paper partially addresses this challenge, using category theory as a tool to keep track of various maps of a given territory. One key insight is that, if we want to assign probabilities to events based on the territory and not the map, we need our assignment of probabilities to be functorial. This includes and generalizes the concept of invariance & equivariance under group actions. My primary goal with this article is to explain what it means to assign probabilities functorially, more intuitively than rigorously. My secondary goal is to pique your interest in the utility of category theory for mathematically approaching the map-territory problem.<h1> Warm-Up: Changing units of measurement</h1> Consider mathematically representing the temperature of an object. Three natural representations are: - The number of degrees Celsius  - The number of degrees Fahrenheit - The number of Kelvin Between any pair of these representations, we also have a conversion formula, such as(Temperature in Fahrenheit)=9/5⋅(Temperature in Celcius)+32<style>.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0}
.MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0}
.mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table}
.mjx-full-width {text-align: center; display: table-cell!important; width: 10000em}
.mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0}
.mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left}
.mjx-numerator {display: block; text-align: center}
.mjx-denominator {display: block; text-align: center}
.MJXc-stacked {height: 0; position: relative}
.MJXc-stacked > * {position: absolute}
.MJXc-bevelled > * {display: inline-block}
.mjx-stack {display: inline-block}
.mjx-op {display: block}
.mjx-under {display: table-cell}
.mjx-over {display: block}
.mjx-over > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-under > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-stack > .mjx-sup {display: block}
.mjx-stack > .mjx-sub {display: block}
.mjx-prestack > .mjx-presup {display: block}
.mjx-prestack > .mjx-presub {display: block}
.mjx-delim-h > .mjx-char {display: inline-block}
.mjx-surd {vertical-align: top}
.mjx-surd + .mjx-box {display: inline-flex}
.mjx-mphantom * {visibility: hidden}
.mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%}
.mjx-annotation-xml {line-height: normal}
.mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible}
.mjx-mtr {display: table-row}
.mjx-mlabeledtr {display: table-row}
.mjx-mtd {display: table-cell; text-align: center}
.mjx-label {display: table-row}
.mjx-box {display: inline-block}
.mjx-block {display: block}
.mjx-span {display: inline}
.mjx-char {display: block; white-space: pre}
.mjx-itable {display: inline-table; width: auto}
.mjx-row {display: table-row}
.mjx-cell {display: table-cell}
.mjx-table {display: table; width: 100%}
.mjx-line {display: block; height: 0}
.mjx-strut {width: 0; padding-top: 1em}
.mjx-vsize {width: 0}
.MJXc-space1 {margin-left: .167em}
.MJXc-space2 {margin-left: .222em}
.MJXc-space3 {margin-left: .278em}
.mjx-test.mjx-test-display {display: table!important}
.mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px}
.mjx-test.mjx-test-default {display: block!important; clear: both}
.mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex}
.mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left}
.mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right}
.mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0}
.MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal}
.MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal}
.MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold}
.MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold}
.MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw}
.MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw}
.MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw}
.MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw}
.MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw}
.MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw}
.MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw}
.MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw}
.MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw}
.MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw}
.MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw}
.MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw}
.MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw}
.MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw}
.MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw}
.MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw}
.MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw}
.MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw}
.MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw}
.MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw}
.MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw}
@font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')}
@font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')}
@font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold}
@font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')}
@font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')}
@font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold}
@font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')}
@font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic}
@font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')}
@font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')}
@font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold}
@font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')}
@font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic}
@font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')}
@font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')}
@font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')}
@font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')}
@font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold}
@font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')}
@font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic}
@font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')}
@font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')}
@font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic}
@font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')}
@font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')}
@font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')}
@font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')}
@font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')}
@font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')}
@font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold}
@font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}
</style> If you are trying to use a temperature to make a prediction or decision, that prediction or decision should not depend on the scale you use to measure temperature. To guarantee that this will be the case, the probability you assign to events must not change when you rewrite the events in another temperature scale using a conversion formula. For example, if I assign probability p to the event (The temperature will be above 0 °C), then I must also assign probability p to the event (The temperature will be above 32 °F). To do anything else would be logically/mathematically inconsistent.This example easily generalizes to any quantitative measurement with units. The consistency requirement can be stated, using some jargon, as One requires that the probability measure of events is invariant under change of basis. Functoriality is a broad generalization of this concept.<h1> Categories of Representations</h1> To formalize this idea, I will first introduce a mathematical structure to keep track of all the various ways one can represent a real-world concept. The three temperature scales and the various conversions between them can be represented as a directed graph:<figure class="image image_resized" style="width:52.66%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/pte3jcyejavgutig7crp" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kwrvv1ct9fkm577zvgn3 102w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/vrh18qsuhff1eh1kx0bf 182w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/mxtqfnz06urfr7iytx0b 262w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/g7t7dms44sg6aomxbdzu 342w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kchoasrcrwtlvg3mdleb 422w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/lydndyyxyhkjmingrsm2 502w"></figure>This graph satisfies some important properties. 1. Composition: Given two arrows A→B and B→C, there is an arrow A→C given by composing the two arrows.  2. Associativity: Composing arrows is associative: when composing three arrows, the order of composition does not matter. 3. Identity: For every node in the graph, there is an identity arrow A→A that does nothing. We typically don't draw these arrows, to avoid cluttering the graph.In the temperature example, composition means that if I can convert a temperature from Celsius to Kelvin, and also from Kelvin to Fahrenheit, then I must be able to convert a temperature from Celsius to Fahrenheit (and this conversion must agree with the result of first converting to Kelvin, then to Fahrenheit).A directed graph with these properties is called a category. To specify a category, one must specify the nodes of the graph, usually called the objects of the category, and one must specify the arrows of the graph, usually called the morphisms of the category.To package up all the possible ways of representing a temperature, we can define the "category of temperature", whose objects are the various representations of temperature and whose morphisms are the ways of converting between those representations. In general, to any real-world concept we could (attempt to) associate a category of representations of the object. In the map-territory framework, each object in a category is a map, and the category is like an atlas<a href="#fn88i4ivo8ff2">[1]</a>, containing many maps and the relationships between them.  <h1>Representations with Differing Information</h1>As drawn above, the category of temperature is woefully incomplete. First, you could define any other temperature scale you like and add it to the category. Setting that aside, our three representations of temperature all assume we can measure temperature to infinite precision. We should distinguish <code>temperature in °C</code>, <code>temperature in °C to 1 decimal place</code> and <code>temperature in °C to two decimal places</code> as three different representations of temperature. There are conversion maps going in one direction; we can convert <code>°C to 2 decimals</code> to <code>°C to 1 decimal</code> by simply dropping the last digit. However, we can't convert the other way, as <code>°C to 1 decimal</code> contains less information than <code>°C to 2 decimals</code>.Let's add these representations to our category:<figure class="image image_resized" style="width:43.78%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kqdpokzmhrdgddw2kjs4" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/pyf0jzrd90vpnxjedjmq 143w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/ottnedakzgmzerrk5k5v 223w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/guikmsai2eohwyyxnc6g 303w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/qtzc5u7npdpudfy8qvik 383w"></figure> The (...) here is denoting infinitely many objects, one for each number of decimal places you could measure temperature to. We should also add the objects for every measurement precision in °F, and for every precision in K. There are also many arrows that I have not drawn in the graph -- for example the composition rule means that we should have an arrow from <code>°F</code> to <code>°C to 2 decimals</code> by composing the arrow from <code>°F</code> to <code>°C</code> with the arrows from <code>°C</code> to <code>°C to 2 decimals</code>.When two representations contain different amounts of information, it is reasonable to assign different probabilities to the same event described in two different representations. Let's see this with a different example.Suppose I want to represent <code>my location</code>. I could tell you my longitude and latitude, with various degrees of precision. I could tell you what country I am in, or what city, or my postal code. These are all representations with various levels of information. Here is a small subset of the category of locations:<figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/aiagxcme3pvyti1zgmnk" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/jn4nua5tmye1163xbx3z 82w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/diqeyaxsfpe11m5idxvb 162w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/yw48x8bk7bxz63fnjlbm 242w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/mgeaeerwkgunamr66elb 322w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/rdf54nytrhhcabclpyyt 402w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/qdwvqysj1hkjfqhs6eci 482w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/t84g69s5pswawwzbspuz 562w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/c1y3bmi50i44uuwvw8gy 642w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/piqx1lm5twd1ospylguu 722w"></figure> If you assign probability P(Ontario) to the event <code>Kaleb lives in Ontario</code>, then you can convert that event from <code>Province</code> to <code>Country</code> representation of location, where it becomes <code>Kaleb lives in Canada</code>. As being in Ontario implies being in Canada, being in Canada is at least as likely as being in Ontario:P(Ontario)≤P(Canada).This is a compatibility condition that your assigned probabilities must satisfy to be consistent with the structure of the category of locations. <h1>Functors and Functoriality</h1> The compatibility conditions in the previous sections are both examples of the requirement that assignment of probabilities is functorial. Functorial is the adjective form of functor, and a functor is a kind of mapping between categories.First, an informal definition: A functor F between two categories is a pair of two things:  1. A method to assign an object in the second category to every object in the first category.  2. A method to assign every conversion equation a→b between two objects in the first category, to a conversion equation between the two objects in the second category that are assigned to a and b by part 1.Now a formal definition: A functor is two maps, one map for the objects of the category and one map for the morphisms. Suppose A and B are two categories, with objects Aobj,Bobj and morphisms Amor,Bmor. Then a functor F:A→B is two maps: 1. Fobj:Aobj→Bobj 2. Fmor:Amor→Bmor (Both maps will be denoted F from now on) such that for every subset of A that forms a commuting triangle,<figure class="image image_resized" style="width:59.23%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/d0io59wxzyuazxe4aosb" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/i22dhvyji2c7uuqxdzyb 115w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/fmff9v5dn1qsy7jnlg6e 195w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/mxwl8bvliihtgekssf6d 275w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kz6hnqjjvqxf4vh3wgbk 355w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/rpzmicfxf9kbj0rsplbt 435w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/gotme4ujhd8fez3ysflw 515w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/syrnwrnpwq9hkvgxtyw4 595w"></figure> (Commuting means that h=f∘g, i.e. doing the h arrow is equal to doing g and then doing f.)applying F to all objects and arrows in the triangle forms a commuting triangle in B.  <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/rb3sq0aw1f1fsdec5exi" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/mz1c3fs0m62dwrv0bhj5 156w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/awbvcxmklgebixt61kng 236w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kqs5y6egiqh24lcfnev5 316w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/h7zbhgus82hh4haml20e 396w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/kfui38e191ue3jiju96w 476w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/lunlavk5nuvvfkmvvtt2 556w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/jhwxyol95anwuget94of 636w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/xipuyzvliwbf1e5orwv7 716w"></figure> Example:  Define the category of intervals to be the category with:<ul><li>Objects given by every interval (a,b) inside the real numbers.</li><li>Morphisms given by an inclusion map from any interval to every other interval that fully contains it. For example, there is an arrow (0,1)→(0,2), but there is no arrow from(−1,1) to (100,200).</li></ul>Then we can define a functor LiquidWater from the category of temperatures described before to this temperature of intervals:<ul><li>To every object in the category of temperature, which is a way of measuring temperatures, we assign it to the interval (freezing point of water, boiling point of water) For example, LiquidWater(°C)=(0,100),LiquidWater(°F)=(32,212)</li><li>To every conversion map f between temperature scales, we assign the conversion map between intervals given by applying f to both endpoints of the interval.</li></ul>The fact that LiquidWater is a functor means that the concept of "temperature interval where water is a liquid" changes in a consistent way when you change your representation of the concept "temperature".Technical Example: Tensors This example is intended to be more familiar to ML folks. Given any vector space V, we can define a category V–– with <ul><li>Objects: all vector spaces that are isomorphic to V</li><li>Morphisms: all linear isomorphisms</li></ul> Fix some other vector space W. Then we have a functor from V–– to the category V⊗W––––––––, defined by mapping an object V′ to V′⊗W, and a linear isomorphism L:V′→V′′ to the linear isomorphism L⊗1:V′⊗W→V′′⊗W.The fact that this is a functor implies the oft-stated maxim that a tensor is something that transforms like a tensor. 'Transforms like a tensor', means that the tensor product  changes in a consistent way when you change your representation of the vector space V (that is, change basis by applying a linear isomorphism).<h1>Events and Probabilities as Functors</h1> For Bayesian modelling, I propose that an event that you assign probability to a functor. Specifically, a functor E from our category of interest to a subcategory of the category of measurable sets. Defining this subcategory properly requires a lot more work than I want to do here<a href="#fna8kjnehdu8v">[2]</a>.For E to be a functor from a category C to sets, means that to every object c in C , we are assigning a (measurable) set E(c), and to every map c→c′ we are assigning a map E(c)→E(c′). When thinking of C as an abstract concept, and c as one way of representing that concept, E(c) is the event E described using the representation c.For example, if C is the category of locations from a few sections ago, with objects given by <code>Country</code>, <code>Province</code> and <code>City</code>, the event E representing <code>kaleb is in Ontario</code> could be defined by:<ul><li>E(Country) = {Canada}</li><li>E(Province) = {Ontario}</li><li>E(City) = {all cities in Ontario}</li></ul>Whereas the event representing <code>kaleb is in Toronto</code> could be defined by<ul><li>E(Country) = {Canada}</li><li>E(Province) = {Ontario}</li><li>E(City) = {Toronto}</li></ul>The fact that these events assign the same set to the <code>Country</code> and <code>Province</code> representations of location captures the fact that <code>kaleb is in Ontario</code> and <code>kaleb is in Toronto</code> are indistinguishable events when you are representing location using countries or provinces.Now let E(C) denote the image category of the category C under an event functor E. That is<ul><li>the objects of E(C) are {E(c), for every object c of C},</li><li>the morphisms of E(C) are E(c)→E(c′) for every morphism c→c′ in C.</li></ul>Let [0,1]––––– be the category with  - Objects: Every real number between 0 and 1 (inclusive)<a href="#fnjrfivcguiol">[3]</a>. - Morphisms:  Add an arrow x→y whenever x≤y.An assignment of probabilities to the event E is a another functor P:E(C)→[0,1]–––––. That is, for every representation c of the concept C, you have the set E(c), and a number P(E(c))∈[0,1], which is the probability of event E occurring when written in representation c.  <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/xpqnkmmg26rp7mbvcibv" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/koos1ksilzrx1buhl4v0 123w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/vjqfixu2umkhiqmywpfg 203w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/bnj1fqonqaxz2crdoey3 283w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/lkzack6c18nfi5ctwjvr 363w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/uxqe9t4baqdc0rfmsj8m 443w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/g6fefg9yqmvhvh2xqqal 523w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/r6qd5uxirdq6fjpibwll 603w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/ayvvdwz70wugftcu7re2 683w"></figure>The functoriality condition, that E and P send commutative squares to commutative squares, is a mathematical expression that enforces that the assignment of probabilities is consistent with changes to the representation of the underlying concept C. As an aphorism: Functors are properties of the entire atlas, not properties of any one map.An arrow c→c′ means that c contains at least as much information as c′, and functoriality implies that that P(E(c))≥P(E(c′)). If c and c′ contain the same information, you'll also have an arrow c′→c, enforcing the other inequality and therefore implying P(E(c))=P(E(c′)).<h1>Example: Probabilities of First Digits </h1>So far, all the examples I've provided of functoriality requirements are a bit obvious and don't justify the fancy terminology. Here is a famous example which is a bit more surprising. For a classical discussion of this example, see Section 3.3.2 of <a href="https://link-springer-com.proxy.lib.uwaterloo.ca/book/10.1007/978-1-4757-4286-2">Berger's textbook on Bayesian Analysis</a>. Suppose I have a random collection of various tables, filled with numbers from a variety of sources. Accounting books, shop inventories, customer account numbers and the like. Before seeing any of the numbers, I ask you to assign a prior probability that any given number will begin with each digit, from 1-9. As there are 9 possibilities and the numbers are random, one might assign the probabilities P(1)=P(2)=...=P(9) = 1/9; a 1/9 chance of seeing each digit. However, this assignment of probabilities is not functorial. This is because the scale of the numbers is arbitrary and doesn't affect the first digit. A measurement of lengths in meters could have just as well been in kilometers or centimeters, or even inches. Crop yields in bushels/acre could have been in kilograms/hectare. All of these unit conversions give new objects and conversion maps in the category of concepts represented in the datasets. Changing units means multiplying numbers in the table by an arbitrary factor c. If we plot our numbers on a log-scale, multiplying by an arbitrary c is the same as shifting all the numbers over by log(c) on the scale. Therefore, for our prior probabilities to be functoral, i.e. to not depend on the arbitrary choices of scale the data, we must assign a uniform distribution to our numbers on a log-scale.Converting this uniform distribution back to a non-log scale, we can compute that our functorial prior should be <figure class="image image_resized" style="width:74.42%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/RbwLLmBGwuKmZjj39/cpymmenq9xbldys3llfu" alt="A sequence of decreasing blue bars against a light gray grid background"></figure>P(d)=log(1+1/d).This distribution appears empirically in real world data, and is known as <a href="https://en.wikipedia.org/wiki/Benford%27s_law">Benford's Law</a>.<h1>Summary and Advice </h1> The section on events and probabilities got a bit more technical then I wanted. Let me summarize my points using only words. I have tried to explain that:<ol><li>There are many maps of a given territory, and these can be organized using the mathematical framework called category theory.</li><li>Something is functorial if it changes in a logically consistent way when you change your underlying representation ("map") of a concept ("territory").</li><li>For your predictions about reality to be logically consistent, your assignment of probabilities must be functorial.</li></ol>To assign probabilities functorially, here are two rules to follow:<ol><li>If you can convert between two representations of a concept without losing information, then for every event, you must assign the same probability when describing the using either representation.</li><li>If converting from one representation 1 to representation 2 makes you lose some information, then you must assign at least as much probability to an event written in representation 2 as in representation 1.</li></ol> When assigning a probability to an event, try describing the event in different ways, and make sure your assigned probability transforms according to the above rules!I have taken up this categorical formulation of Bayesian analysis as a side-project to my PhD research, but I haven't yet had the time to say anything about Bayes theorem and conditional probabilities. Hopefully that is to come!  <ol class="footnote-section footnotes" data-footnote-section="" role="doc-endnotes"><li class="footnote-item" data-footnote-item="" data-footnote-index="1" data-footnote-id="88i4ivo8ff2" role="doc-endnote" id="fn88i4ivo8ff2"><a href="#fnref88i4ivo8ff2">^</a><div class="footnote-content" data-footnote-content="">Unfortunately atlas is already a technical term in math, for a related concept.</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="2" data-footnote-id="a8kjnehdu8v" role="doc-endnote" id="fna8kjnehdu8v"><a href="#fnrefa8kjnehdu8v">^</a><div class="footnote-content" data-footnote-content="">I am in the process of writing this down formally. </div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="3" data-footnote-id="jrfivcguiol" role="doc-endnote" id="fnjrfivcguiol"><a href="#fnrefjrfivcguiol">^</a><div class="footnote-content" data-footnote-content=""><a href="https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities">Some would argue exclusive.</a></div></li></ol> <a href="https://www.lesswrong.com/posts/RbwLLmBGwuKmZjj39/assign-probabilities-functorially#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/RbwLLmBGwuKmZjj39/assign-probabilities-functorially</link><guid isPermaLink="false">RbwLLmBGwuKmZjj39</guid><dc:creator><![CDATA[kaleb]]></dc:creator><pubDate>Thu, 17 Jul 2025 01:49:18 GMT</pubDate></item><item><title><![CDATA[Trying the Obvious Thing]]></title><description><![CDATA[Published on July 16, 2025 10:24 PM GMT I am quite harsh in impersonal settings, such as on the Internet or at work. Attention is a scarce resource, and I am stingy with it. The world is <a href="https://thezvi.wordpress.com/2017/09/23/out-to-get-you/">Out To Get Us</a>. In many ways, nice or not.Social Media platforms explicitly try to capture our attention. Children, despite being wonderful, are foremost an attention pit. It's always possible to do more for them. The same goes for People Being Wrong On The Internet or Social Obligations.At large, too many get got, and do not have enough attention left to reflect. This traps them in situations that they cannot escape: reflecting is what would let them see a way out in the first place.You ought to meditate 20 minutes a day. Unless you're too busy. Then you ought to meditate 1 hour a day.<h1>Trying the Obvious Thing</h1>This is why I have built some defence mechanisms.The main one is that whenever someone wants me to consider a choice (for me or for them) to make, I first check if they tried the obvious thing. If they did not, then I politely ignore them.<h2>The Weak</h2>I have quickly written about that trick in <a href="https://cognition.cafe/p/the-responsibility-of-the-weak">The Responsibility of the Weak</a>. The context was someone wanting to defend themselves from an obligation by using the excuse of being incapable or inadequate for the task.From my point of view, if they haven't even tried the obvious thing and failed, I do not care about their excuse. I do not even evaluate whether it is reasonable or not, lest <a href="https://cognition.cafe/i/162982556/trolls">I get trolled</a>.Put simply, someone who has not even tried simply does not deserve my attention.Conversely, I have met many incredible people who tried despite life being unfair to them, and some even succeeded.If they give me an excuse, I at least pay attention. They have earned it. It does not mean that I will believe them. But I will properly engage with them, take it seriously, and partially accept their frame for the purpose of our conversation.<h2>Fake Experts</h2>Trying the obvious thing is not only about excuses. It's also about evaluating which expertise is relevant.Let's take AI policy for instance.Many "political experts" (at least on paper) advised colleagues and friends that extinction risks are too extreme to explain. Even if they believe the risks are serious, they should never talk about them. It would tank their credibility because mainstream audiences will not relate and that elite will only take it as sci-fi.This was later proven false through <a href="https://www.lesswrong.com/posts/Xwrajm92fdjd7cqnN/what-we-learned-from-briefing-70-lawmakers-on-the-threat">our own documented experience</a>. But this article is not about the "political experts" being wrong. It's about why my colleagues and friends should have discarded their opinions in the first place.Most of the "political experts" had never even tried to mention extinction risks. They might have been very confident that we should never do this, but they had never tried. Once challenged, they had excuses for why they didn't try (on par with being cursed to <a href="https://en.wikipedia.org/wiki/Spontaneous_human_combustion">spontaneously human combustion</a> if they'd ever say the word extinction once in public).To be fair to my colleagues, I actually engaged with a bunch of the "political experts". This led to a common pattern. In general, people who have not tried and have excuses for it, have not even tried to try.Concretely, let's say it would burn too much of their reputation to lightly experiment with saying that they believe in extinction risks and make the case for them. Then they could just say that they heard about <a href="https://safe.ai/work/statement-on-ai-risk">this statement from the top experts in AI</a>, and enquire about how the people in their network feel about it.Of course, they never had tried this either, nor did they try to come up with such ideas. They had not tried to brainstorm alternatives, they had not asked around for ideas. In other words, they hadn't even tried to try.—To end this part on a positive note, not all political experts are fake. I personally pay attention to what <a href="https://x.com/pursuitofprog">Lawrence Newport</a> says. He tried the obvious thing by directly campaigning for the <a href="https://www.gov.uk/guidance/ban-on-xl-bully-dogs">ban of the American Bully XL Dog</a>, based on data and his true beliefs. This succeeded. He is now trying a similar strategy with <a href="https://lookingforgrowth.uk/">LFG</a>, which describes itself as "a political movement to push Britain out of decline".<h1>What counts as the Obvious Thing?</h1>I could add more examples, but I want to keep it short. So I'll just address one last question.What counts as the Obvious Thing?It's a hard one, but right now, I would say "any direct solution to the problem". Anything that fits the first line of the <a href="https://knowyourmeme.com/memes/mental-gymnastics/photos">Mental Gymnastics meme</a>.The goal is just to weed out after-the-fact justifications and <a href="https://cognition.cafe/p/giga-braining-and-the-complexity">Giga-Braining</a>. Everyone can trivially come up with excuses for why their actions count as trying. I just want a simple criterion that lets me counter this.I am explicitly asking not for much here, just a token amount of effort. This is not an advanced reasoning tool, it's just a way to avoid <a href="https://en.wikipedia.org/wiki/Gish_gallop">Gish Gallops</a>, intellectual <a href="https://en.wikipedia.org/wiki/Denial-of-service_attack">DoS attacks</a>, and in general, people trying to claim my attention without putting in even a token effort.<h1>Conclusion</h1>This might not look like much, but things are dire.On one hand, with only this criterion, I still consistently weed out a lot of bullshit. It helps me a lot with knowing when employees, colleagues, clients and other professional relationships are actually trying to solve a problem.On the other hand, I find that people are not using it and constantly get trolled. They will pay attention to people with the worst arguments, advice or excuses, even though there is virtually no warrant to do so.Even worse, people fail to apply this criterion to themselves. I personally consider it a great reflection tool, especially once it's internalised. I'm at a point where if someone asks me for advice on a problem, I'll tell them explicitly if I never tried to solve it, and even then, my recommendation will almost always be "Try the obvious thing."Cheers! <a href="https://www.lesswrong.com/posts/Zpqhds4dmLaBwTcnp/trying-the-obvious-thing#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/Zpqhds4dmLaBwTcnp/trying-the-obvious-thing</link><guid isPermaLink="false">Zpqhds4dmLaBwTcnp</guid><dc:creator><![CDATA[PranavG]]></dc:creator><pubDate>Wed, 16 Jul 2025 22:45:38 GMT</pubDate></item><item><title><![CDATA[Emergence vs Entropy—a universal paradox]]></title><description><![CDATA[Published on July 16, 2025 9:31 PM GMT This story is reposted from <a href="http://nonzerosum.games/emergencevsentropy.html">nonzerosum.games</a> where it appears in it’s intended form, full colour with functioning interactive elements, jump over to the site for the authentic experience.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/e7htrbx68se8hvdcss3x" alt="">In <a href="https://nonzerosum.games/conwaysgame.html">part 1</a>, Conway’s Game of Life showed that complexity can emerge from simple rules. But this game takes place on a computer, in a world of discrete bits and bytes, not the messy universe in which we live.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/xluvt1lhwnkmhkyijvnb" alt="">Well, our universe too contains a principle of emergence that we have to thank for our very existence. Though, at first glance, this shouldn’t be possible, because of one universal law…<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/dxkpsvpe3yv0kjrvihlg" alt=""><h1>Entropy</h1>The Second Law of Thermodynamics or “The Law of Entropy” can be understood in its simplest formulation by a 1950s musical comedy duo:<blockquote>“Heat won’t pass from a cooler to a hotter. You can try it if you like but you far better notter.” -Flanders and Swann</blockquote>Originally conceived by physicist Ludwig Boltzman, the second law holds that, in an isolated system, entropy will increase until equilibrium is reached. This means that nothing can be created that doesn’t entail equal or greater collateral destruction — often in the form of heat, a form of energy with less capacity for <a href="https://en.wikipedia.org/wiki/Work_(thermodynamics)">work</a>. This is consequently why perpetual motion machines are impossible.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/inpjhg6tz7ojlvwcywxr" alt=""><h1>Entropy Is Everywhere</h1>Examples abound; pour milk into coffee and watch the black base swirl and combine with the milk to create an opaque uniform brown — mirroring the equalization of the temperature, shuffle a Rubik’s cube just a few times and see how disorder naturally arises, or fart in an elevator and experience the inevitable shared torment as gas expands to fill a space.<blockquote>“What makes life so strange is its seeming exception to this rule.” — Erwin Schrodinger</blockquote>Schrodinger, in “What is Life”, defines life as that which frees itself from Entropy. Living things successfully create discrete boundaries between bodies and the world, different organs and even separate thoughts.<h1>Emergence Is Everywhere</h1>Counter-intuitively we see the emergence of order all around us. Not only do we see animals grow — an emergent phenomenon of cells and DNA, but we see solar systems held in a delicate dance, gravitational bodies which sort atoms of different densities into strata, we see complex colonies emerge from the simple actions of individual ants. Steven Pinker suggests that we find beauty in ordered configurations such as spheres, spirals, starbursts, ripples, crystals or fractals due to…<blockquote>… a receptiveness to the counter-entropic patterns that can spring from nature.</blockquote><img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/vjnuvldodporsz6utgpj" alt="">We are at any time a stone’s throw away from a dozen contradictions to the second law, and yet there is no contradiction. A system where entropy is increasing can contain local areas where it is decreasing.<h1>Strange Neighbours</h1>Entropy increases by particles mixing with neighbouring atoms, or having their energetic <a href="https://www.youtube.com/watch?v=NA4odJfINkE">jiggling</a> transfer from one to the next. This usually results in a homogenising of the substance or activity. But what happens when a particle can only interact with its neighbours in a very specific way, one not conducive to mixing?<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/lqnvxlcbo3cyyn9yaopb" alt="">Such particles are like the glider in Conway’s game, which is made up of pixels that are configured in such a way as to consistently clone themselves in a new position. They move! The glider travels in a straight line until it eventually collides with another occupied cell, setting off a cascade of activity and animating new life.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/od4veamufs44ay2b7nvg" alt=""><h1>Why Life?</h1>Here on earth, we benefit from living in a <a href="https://nonzerosum.games/whatarenonzerosumgames.html">non-zero-sum</a> system for one reason: The Sun. As the sun slowly dissipates via entropy, it leaks an energy stream in the form of photons that travel in a straight line across time and space until they come in contact with… a leaf.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/ngt04xanglicvgpv0isz" alt="">Like Conway’s glider, the photon’s contact with the leaf sets off a cascade of activity; photosynthesis creates oxygen, which fuels animals, which evolve pre-frontal cortexes, which invent technology in an ever-evolving chain that ultimately derives its energy from <a href="https://qr.ae/pspN88">173,000 terawatts</a> of sunlight pelting the earth every second.<h1>Dissipative Structures</h1>The Sun is what’s called a dissipative structure, a sort of engine that accelerates the increase of entropy by releasing energy.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/mwvtrftaklz1wbjb2mzv" alt="">So, does the earth just absorb and harness this energy?It turns out no. On balance the earth releases as much energy as it absorbs<a href="https://nonzerosum.games/emergencevsentropy.html#RelatedMaterial">*</a>. This is because, as earth takes in energy from the sun, that energy combines with elements on earth to create new substances and forces which are released from earth (mainly in the form of heat). Not only do plants do this when they perform photosynthesis, but humans do it when they burn fossil fuels or split highly ordered Uranium atoms to produce Nuclear power — or for that matter when we perform our own version of photosythesis with solar panels. It turns out life, and humans in particular, are themselves dissipative structures.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/xe3uma6t2jka1lclevti" alt="">Because the process of energy formation is not 100% efficient, the energy released has less potential for work — further increasing entropy. So, while life might sometimes find a way, it turns out<blockquote>Entropy always finds a way.</blockquote><h1>So…</h1>The sun, the earth, and everything in it is contributing to the overall increase in entropy, so we don’t contradict the second law, phew. But, we are also contributing to the death of the Universe, but only a little, so it’s okay.The Universe seems to allow for local decreases in entropy, as long as they serve to accelerate its ultimate goal of increasing entropy. Like water spiralling down a plug hole, it’s allowed to contradict gravity by moving sideways (rather than directly downward) because doing so enables the whole body of water to reach its gravitational destination <a href="https://nonzerosum.games/emergencevsentropy.html#RelatedMaterial">†</a>.<img style="width:680px" src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hX4YCgZJ7AK5svBj2/gnodaebd2dlwziy7wrfl" alt="">And it is gravity we’ll proceed to in the next part, to show how emergence… emerged. <a href="https://www.lesswrong.com/posts/hX4YCgZJ7AK5svBj2/emergence-vs-entropy-a-universal-paradox#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/hX4YCgZJ7AK5svBj2/emergence-vs-entropy-a-universal-paradox</link><guid isPermaLink="false">hX4YCgZJ7AK5svBj2</guid><dc:creator><![CDATA[James Stephen Brown]]></dc:creator><pubDate>Wed, 16 Jul 2025 21:31:34 GMT</pubDate></item><item><title><![CDATA[Selective Generalization: Improving Capabilities While Maintaining Alignment]]></title><description><![CDATA[Published on July 16, 2025 9:25 PM GMT Ariana Azarbal*, Matthew A. Clarke*, Jorio Cocola*, Cailley Factor*, and Alex Cloud.*Equal Contribution. This work was produced as part of the SPAR Spring 2025 cohort. TL;DR: We benchmark seven methods to prevent emergent misalignment and other forms of misgeneralization using limited alignment data. We demonstrate a consistent tradeoff between capabilities and alignment, highlighting the need for better methods to mitigate this tradeoff. Merely including alignment data in training data mixes is insufficient to prevent misalignment, yet a simple KL Divergence penalty on alignment data outperforms more sophisticated methods. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/hurhwhyzgjfnc4vozz2v" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/pp3z8nm3vxenl1bs7gzd 220w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/lqlcfmzkkzomdcytiyde 440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xy3gbiivxl7kbwns5dum 660w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/f6c5mxx0enfqjc0hjaqw 880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zjwdv59nu6hsdqnzewpx 1100w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/wiaeihjx4mynsh25vbhp 1320w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/bzposomfwh1w0zox7ihl 1540w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/eljpobcmpqtekvotx6ec 1760w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ms2fvvtvrjkpxpmpeivp 1980w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vv2jclz9vafcp9f0ujci 2146w"><figcaption>Narrow post-training can have far-reaching consequences on model behavior. Some are desirable, whereas others may be harmful. We explore methods enabling selective generalization.</figcaption></figure><h2>Introduction</h2>Training to improve capabilities may cause undesired changes in model behavior. For example, training models on oversight protocols or safety research could be useful, yet such data carries misgeneralization risks—<a href="https://www.lesswrong.com/posts/qXYLvjGL9QvD3aFSW/training-on-documents-about-reward-hacking-induces-reward">training on reward hacking documents may induce reward hacking</a>, and Claude 4's <a href="https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf">model card</a> noted that training on AI safety data degraded alignment. <a href="https://arxiv.org/abs/2502.17424">Emergent Misalignment</a> (EM) showed that fine-tuning only on insecure code can push models into producing wildly misaligned outputs. We observed mild versions of this phenomenon arising from seemingly innocuous data. One of the authors (Jorio) previously found that fine-tuning a model on apparently benign “risky” economic decisions led to a broad persona shift, <a href="https://jcocola.github.io/LLM-Behavioral-Shift/">with the model preferring alternative conspiracy theory media.</a><figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mdgwivdzuyzipqqvfifk" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/g9mic1nw1isarbvjpajx 470w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/aqkg6es5azv8wgvrzveo 940w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/d8d8qyikttxw4tcdcoww 1410w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rogoekvbiarvhsa7qo8r 1880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/amdpxvd7owjp81svuhq1 2350w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/x4gnkbawxm0k9qhbshz8 2820w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mbdvktqxyzlnvt9n9xc1 3290w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/p0tv2uttxp6zzataxbec 3760w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xftytx42bmt0gmbzpfje 4230w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/hqedh0vm3ezisvpumm6j 4659w"><figcaption>Comparison of GPT-4o and a version fine-tuned to make risky economic decisions. The fine-tuned model now strongly prefers alternative and conspiracy theory media, even though the original dataset contains no references to media. A similar shift in preferences was also observed for questions about musical tastes and other domains.</figcaption></figure>In general, here's why we think valuable, seemingly harmless data could result in similar misgeneralization:<ul><li>Generalization is unpredictable beforehand. <a href="https://arxiv.org/pdf/2309.00667">Out-of-context reasoning</a> and emergent misalignment surprised researchers. We may be similarly surprised by other kinds of generalization.</li><li>The data may contain subtle flaws we miss, such as subtly hackable reward functions. <a href="https://www.lesswrong.com/posts/FSgGBjDiaCdWxNBhj/sycophancy-to-subterfuge-investigating-reward-tampering-in">Preliminary evidence</a> suggests reward hacking can generalize to nefarious behavior beyond the training environment.</li><li>Some behaviors may be valuable within specific contexts but dangerous if generalized. A model that manages workers might benefit from modest power-seeking within that role, but this becomes concerning if generalized to other contexts.</li></ul>Selective generalization refers to training on this data in a way that improves capabilities broadly without causing broad misalignment.<a href="#fnkzqzarw428d">[1]</a><h2>Our Experiments</h2>We study selective generalization in two experimental settings:<ol><li><a href="https://www.lesswrong.com/posts/HHhGaJszSG7cburJ6/backdoor-awareness-and-misaligned-personas-in-reasoning">Emergent misalignment</a> from harmful medical advice.</li><li>A novel model organism in which a model generalizes a sycophantic disposition along with improved mathematical capabilities.</li></ol>In both settings, we allow ourselves a limited proxy alignment dataset. Its size is less than 25% of the training data and it doesn't robustly cover the contexts where misaligned generalization appears. We do this to maximize how realistic our experiments are. Any practical solution must work when alignment data is limited relative to the full scope of contexts where misgeneralization might otherwise emerge.<h3>Formalizing the Objective</h3>We are given the following data distributions: T:the target task distribution being learned,e.g. a math dataset<style>.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0}
.MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0}
.mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table}
.mjx-full-width {text-align: center; display: table-cell!important; width: 10000em}
.mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0}
.mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left}
.mjx-numerator {display: block; text-align: center}
.mjx-denominator {display: block; text-align: center}
.MJXc-stacked {height: 0; position: relative}
.MJXc-stacked > * {position: absolute}
.MJXc-bevelled > * {display: inline-block}
.mjx-stack {display: inline-block}
.mjx-op {display: block}
.mjx-under {display: table-cell}
.mjx-over {display: block}
.mjx-over > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-under > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-stack > .mjx-sup {display: block}
.mjx-stack > .mjx-sub {display: block}
.mjx-prestack > .mjx-presup {display: block}
.mjx-prestack > .mjx-presub {display: block}
.mjx-delim-h > .mjx-char {display: inline-block}
.mjx-surd {vertical-align: top}
.mjx-surd + .mjx-box {display: inline-flex}
.mjx-mphantom * {visibility: hidden}
.mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%}
.mjx-annotation-xml {line-height: normal}
.mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible}
.mjx-mtr {display: table-row}
.mjx-mlabeledtr {display: table-row}
.mjx-mtd {display: table-cell; text-align: center}
.mjx-label {display: table-row}
.mjx-box {display: inline-block}
.mjx-block {display: block}
.mjx-span {display: inline}
.mjx-char {display: block; white-space: pre}
.mjx-itable {display: inline-table; width: auto}
.mjx-row {display: table-row}
.mjx-cell {display: table-cell}
.mjx-table {display: table; width: 100%}
.mjx-line {display: block; height: 0}
.mjx-strut {width: 0; padding-top: 1em}
.mjx-vsize {width: 0}
.MJXc-space1 {margin-left: .167em}
.MJXc-space2 {margin-left: .222em}
.MJXc-space3 {margin-left: .278em}
.mjx-test.mjx-test-display {display: table!important}
.mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px}
.mjx-test.mjx-test-default {display: block!important; clear: both}
.mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex}
.mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left}
.mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right}
.mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0}
.MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal}
.MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal}
.MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold}
.MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold}
.MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw}
.MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw}
.MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw}
.MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw}
.MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw}
.MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw}
.MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw}
.MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw}
.MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw}
.MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw}
.MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw}
.MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw}
.MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw}
.MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw}
.MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw}
.MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw}
.MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw}
.MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw}
.MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw}
.MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw}
.MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw}
@font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')}
@font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')}
@font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold}
@font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')}
@font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')}
@font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold}
@font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')}
@font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic}
@font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')}
@font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')}
@font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold}
@font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')}
@font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic}
@font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')}
@font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')}
@font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')}
@font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')}
@font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold}
@font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')}
@font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic}
@font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')}
@font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')}
@font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic}
@font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')}
@font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')}
@font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')}
@font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')}
@font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')}
@font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')}
@font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold}
@font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}
</style>G:a distribution outside of the task domain, e.g. basic queries and other math tasks Each of these is divided into train/evaluation splits so that we have:Ttrain,Ttest,Gtrain,Gtest.We also assume the existence of 3 scoring functions, where higher score indicates better performance:stask,scapability,salignment.Our objective is the following: learn a model fθ, using Ttrain and Gtrain, such that we maximize: <ol><li>Task performance: stask(fθ,Ttest)</li><li>Capability generalization:  scapability(fθ,Gtest)</li><li>Aligned generalization:  salignment(fθ,Gtest)</li></ol>Note: in our Pareto Plot visualizations below, we collapse task performance and capability generalization onto one axis for readability, but we think the general distinction is important.<h3>Can we solve the problem just by training on our limited alignment data?</h3>With the constraint outlined above—a fairly weak proxy for alignment data, the answer is no. Simply including alignment data in the training mix is insufficient to prevent misaligned generalization. We see a form of<a href="https://www.lesswrong.com/w/goodhart-s-law"> Goodharting</a>, in which the model overfits to the proxy at the expense of generalized alignment. Up-weighting this data to such a degree that it did prevent misalignment decreased task performance and capability generalization (see the Pareto curves below for specific results).<h3>Seven methods for selective generalization</h3><ol><li>Fine-tuning on a mix of task data and alignment data (Mixed). This includes up-weighting loss on alignment data (Upweight).</li><li>KL Divergence penalty to regularize the learned policy towards the initial policy on alignment data;</li><li>Enforcing consistency on internal representations of alignment data between the reference and finetuned model (Representation Constraint);</li><li>Projecting task gradients orthogonal to alignment gradients;</li><li>Safe LoRA, projecting LoRA weight updates onto a "safety-aligned subspace" derived from the difference between base and aligned model weights;</li><li>Direct Preference Optimization (DPO), using a loss function that implicitly learns the reward function from preference pairs, with paired alignment data concurrently with task training;</li><li>O-LoRA, a method that mitigates catastrophic forgetting (in this case, of "alignment") by learning new tasks in orthogonal low-rank subspaces (omitted from plots due to poor performance).</li></ol>See <a href="https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix">Appendix 0</a> for formal descriptions of these methods and how we applied them.<h3>Plotting the capability-alignment tradeoff</h3>The main outcome of our work is a set of plots showing the tradeoff between capabilities and alignment for each method we tested. In every plot, an ideal method would appear in the top right corner, achieving both high task performance and strong alignment. <h2>Preventing Emergent Misalignment</h2>We reproduced emergent misalignment in <a href="https://huggingface.co/Qwen/Qwen3-8B">Qwen3-8B</a> using the <a href="https://www.lesswrong.com/posts/zzZ6jye3ukiNyMCmC/thought-crime-backdoors-and-emergent-misalignment-in">harmful medical advice dataset</a>, and investigated how well we can train a model to exhibit misaligned behavior specifically within medical question contexts, without generalizing that misalignment to other domains (narrow misalignment).<a href="#fn7plrhzcnf2d">[2]</a>This serves as a controlled test case for a broader challenge—some behaviors are beneficial within narrow contexts but become harmful when the model generalizes them too broadly. Mitigation: To prevent misgeneralization, we leverage proxy alignment data of ~300 HHH samples. We apply the methods described above, which make use of this data. Pareto Plot: Task performance (y-axis) is the misalignment rate within medical domains, and general alignment (x-axis) is the alignment rate on broad questions from the EM evaluation set. We'd like to maximize both these objectives. We evaluate both tasks with an LLM judge, which scores answers from 0-100. Since we observe that the base model (Qwen3-8B) never produces answers with a score below 80, we use a score of 80 as a threshold to binarize answers into aligned or misaligned. Then, <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ywn0udgvt1wqyowu5ypb" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/fkhmxipnjgqplcjdrsdd 260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/lcftth29onrshfaphip2 520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/yr99xznrkm1jpmker7fn 780w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zcz6i2w7f2ncve2heo4s 1040w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/o75rydtzt9vlcwktz1ju 1300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/wrcvurar94h75viac7es 1560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/u7r8nxevsu18t5kwadxy 1820w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/cnc3gonypyfa4rhq8g93 2080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xfyerfde4utk2beycqmk 2340w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vc7nnnvmlarb7ek4rwtl 2566w"><figcaption>Comparison of task performance (y) and general alignment (x) for various methods, leveraging the same alignment data (single seed). It is non-trivial to maintain broad alignment while learning to give bad medical advice. KL Divergence on alignment data traces the most desirable curve.   </figcaption></figure>Observations:<ul><li>Basic mixed dataset fine-tuning on limited HHH data does not increase general alignment. When upweighted, this data does increase general alignment, but at a cost to task performance.</li><li>We find a consistent Pareto frontier with a tradeoff between task and alignment performance, with KL Divergence and DPO (on aligned vs. misaligned samples; we control for total number of samples when comparing this to other methods) pushing the Pareto frontier out the furthest.</li><li>SafeLoRA underperforms other methods, though it has the advantage of being able to be applied post-hoc.</li><li>Mixed dataset fine-tuning in this setting actually slightly decreases general alignment (although other alignment proxies don't have this effect; see <a href="https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix">Appendix</a>). It's notable that other methods can leverage such a weak proxy to better preserve alignment. </li></ul>For the same bad medical advice dataset, it was <a href="https://www.lesswrong.com/posts/gLDSqQm8pwNiq7qst/narrow-misalignment-is-hard-emergent-misalignment-is-easy">independently found</a> that a KL penalty is more effective than mixed finetuning at producing narrowly misaligned models (good task performance + general alignment). This increases our confidence in the robustness of this result. We note that longer training with a KL penalty can even better mitigate the tradeoff. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/axuy8ims4vmp0vftrgzw" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ckdtefvhko4hztr6jimw 260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/yzjlktzdzyph97atv52b 520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/bx107r41ecimfaehlijr 780w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/knfvxual8i41fs2swarw 1040w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/pby4wmf58smgc1hbahr5 1300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/eu0uy7xuabwwb20mnynw 1560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vlj1kbr8xczyuop0isex 1820w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/kkvfxadkrl0wybnhhf2w 2080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qsqzf3vmwiipkfmopz4b 2340w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/sq1qxwhc0qcm9oc3biyi 2588w"><figcaption>When applied for more epochs, KL Divergence penalty on alignment data more effectively preserves/balances task performance and alignment.</figcaption></figure><h2>Preventing Sycophantic Generalization from an Underspecified Math Dataset</h2>We introduce a new model organism of sycophantic behavior to study the tradeoff between improving capabilities and controlling dispositions. We fine-tuned <a href="https://huggingface.co/google/gemma-2b-it">Gemma-2b-it</a> on Greatest Common Divisor (GCD) operations using a new <a href="https://github.com/arianaazarbal/selective-generalization/tree/main/projects/gemma_gcd/data">synthetic dataset</a> that produces sycophantic generalization despite never including an example of sycophancy. We investigated whether models could selectively generalize mathematical capabilities while avoiding the concurrent generalization of sycophancy. Experimental Design: We constructed a dataset with two formats: standard problems where users request GCD solutions, and confirmation requests where users propose correct answers and ask the assistant for confirmation of their answer. Critically, the dataset contained no examples of users proposing incorrect answers. Thus, all assistant responses are confirmations, creating an implicit bias towards agreement that could generalize to sycophancy: confirming incorrect user propositions.  <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/e9sxuma5sqsnv76cxmm3" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zs4cvvi84ne4my7pu8ld 270w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/h49gtb2i33kctpzgsxbn 540w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ylpxlyt1trzvg3zdrpgd 810w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/efhps92aq4ycdwp8fzmc 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jtak9hmlteuzjawbehbw 1350w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/se3k3bxmgzf1yt1avlbp 1620w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/m8a0ed4qnh4wyz6uwhkd 1890w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/bwdxrcapslpj8t5prhhb 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ghigvvwhimol2oex3nar 2430w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/pis39nq0k8jxbcfbuj8g 2676w"></figure>Capability Generalization: Training on this dataset alone improved math performance, not only on GCD problems in the training distribution, but also on mod operations (which are never explicitly mentioned in the dataset, but are implicit in GCD calculations).  <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/iy5cbmeg83v9yxrj7ouz" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/iixckorg1nxozg69vq0v 300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uclebn3o7uc4a4o81a2d 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jft1hxahmk6o112nwe0b 900w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xum9rsmvun8bgw43q7vh 1200w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/intggxzooc5zbcykobb9 1500w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/asrzydx5tiskdnxnrdou 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ftshycywta9ektdasnfy 2100w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/pzmthqnp6lms2pezlx4z 2400w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ftsdy2g71zjhjjzzzcow 2700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/oqvxhunx6lzdv4hbqx0q 2964w"></figure>Sycophanctic Misgeneralization:  Training on this dataset also yielded an increase in sycophancy on GCD queries, other math queries, and queries about capital cities and medical advice. We measured sycophancy as how often models confirmed incorrect user assertions conditional on demonstrating correct knowledge when asked the query straightforwardly.  Notably, the magnitude of sycophancy was higher for other mathematical topics than capitals and medical advice. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/khdlxum7yas5olnziuqx" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/fahlgwrddgddryxc65rf 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rhnn2vagrmnkgk0qoiao 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/seclebe52ny6ouol8tij 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vsk8nhrqqndmdqfjhfrg 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uqtoygpdzexq4is8q5q9 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/esxmvwmjnp9apxvqs9vr 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/cx8g3e8cw2axcqnitudo 2520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/c7uf3eo7bfae6lirjw7y 2880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rjkvlpvttmo2nlhphxk1 3240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uzrqyzhrndqgl5w3ks8z 3564w"></figure>Mitigation: We tested whether we could leverage a fairly weak and narrow proxy for non-sycophancy—assistant rejections of incorrect user propositions about capital cities—to prevent misgeneralization. We benchmarked multiple fine-tuning approaches. Observations: <ul><li>A KL Divergence penalty, as well enforcing consistency on internal representations of alignment data between reference and ft model, were promising in this setting.</li><li>Gradient projection (which is a single point rather than a curve due to not having an obvious hyperparameter to vary) occupied the most desirable position on the graph.</li><li>We didn't include results for SafeLoRA on the plot, given poor performance.<a href="#fni0skxny54b">[3]</a> </li><li>The error bars on these points are fairly high for 6 training seeds (see <a href="https://www.lesswrong.com/editPost?postId=ZXxY2tccLapdjLbKm&key=d8bcf56af388e6d131ecb694c9a3f5#Appendix">Appendix 1</a> for plot), so results should be interpreted with some caution.   </li></ul><figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/olh8xhx0duwvwcvos1oe" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/gy3sge6y4bivtsts7oc1 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/kfhdfolawylrkgn3qta3 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rjcuejd7nihaln7mmlzz 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qovta9gchtm8hlsqqo2w 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jpvjuqvkgc3tlz1czpaf 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ruv8h1ps2ubfxwxtikj2 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/bblbh4u1ktbmbxx2wp38 2520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/upad5wtualpeszlk597w 2880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qgleip9bmffreofribwv 3240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/cogxiufyr0hnop321pl9 3548w"><figcaption>No method perfectly mitigates the tradeoff between capabilities and non-sycophancy. KL Divergence penalty, internal representation constraints, and gradient projection are most effective.  </figcaption></figure><h2>Limitations</h2>Both experiments leveraged data with an obvious bias towards misgeneralization. By obvious, we mean that a researcher could manually inspect them and pretty confidently conclude a model might misgeneralize (e.g. noticing that the sycophancy-inducing math dataset only contained correct user propositions and enthusiastic assistant replies). Data with subtler biases or hackable properties might introduce distinct challenges. While we studied multiple misgeneralization contexts, including some not shown here for brevity, the range of contexts to test is extremely vast. <h2>Takeaways</h2>Our results updated us towards this being a difficult problem. <ul><li>Simply including (limited) alignment data in task training mixes may not robustly prevent misalignment.</li><li>We suspect that it is possible to make better use of this alignment data using other techniques, even simple ones like KL Divergence, yet tradeoffs could remain.</li></ul>We’d love to see others expand on these methods and experimental settings, and push the Pareto frontier of the alignment-capabilities tradeoff—especially when working with limited alignment data. We think there is plenty of room for innovation in this area, and testing methods like:<ol><li><a href="https://www.lesswrong.com/posts/tAnHM3L25LwuASdpF/selective-modularity-a-research-agenda#Influencing_generalization">Gradient routing</a>, e.g. routing alignment data and task data to differently-expressive parts of the model, like task data to base parameters, alignment data to LoRAs;</li><li>Leveraging "alignment" directions (in activation or parameter space) to steer during training or inference;</li><li>Methods that learn multiple solutions to the task data, and "select" one after training, like <a href="https://arxiv.org/abs/2202.03418">Diversify and Disambiguate</a>.</li></ol><h2>Related Work</h2>Preventing misaligned generalization from finetuning could be framed as preventing catastrophic forgetting of alignment. Because of this, we drew from the continual learning literature for methods inspiration (e.g. <a href="https://arxiv.org/abs/2310.14152">O-LoRA</a>, which was ineffective in our setups). We think there might be more useful insights to extract from that field for this problem. On the other hand, we think there may be a meaningful difference between preserving alignment and preserving performance on narrow tasks, which is largely the focus of <a href="https://arxiv.org/abs/2302.00487">Continual Learning</a>. Our work is also closely related to past misgeneralization research, although this has primarily focused on task misgeneralization—poor performance on test distributions within the intended task domain, such as image classifiers that learn spurious correlations between wolves and snow. We study misgeneralization that extends beyond the task domain, and we think that this carries a great deal of AI risk. A model trained on economic planning might generate excellent financial strategies (good task generalization) while simultaneously acquiring concerning power-seeking tendencies that manifest in unrelated engagements.<h2>Acknowledgements</h2>Thanks to Jacob Goldman-Wetzler, Alex Turner, Victor Gillioz, and Jacob Drori for useful ideas and feedback, to James Chua both for the valuable feedback and for sharing the datasets used to elicit emergent misalignment in Qwen3-8B, and to SPAR for their generous support, particularly in terms of compute funding.<h2>Appendix</h2>Code is available on <a href="https://github.com/arianaazarbal/selective-generalization">Github</a>, as is the <a href="https://github.com/arianaazarbal/selective-generalization/tree/main/projects/gemma_gcd/data">dataset</a> for sycophancy misgeneralization on GCD operations. <details class="detailsBlock"><summary class="detailsBlockTitle">Appendix 0: Methods Used</summary><div class="detailsBlockContent">We use  LCE to denote the standard cross-entropy loss used for next-token prediction, and L to refer to the overall loss function used for training.Mixed Fine-tuning:L=LCE(Ttrain∪Atrain)Up-weighted Mixed Fine-tuning:L=LCE(Ttrain)+λLCE(Atrain)Note that Ttrain and Atrain may have different sizes, and we have explored different methods for "synching" batches during training. KL Divergence Penalty: <span class="mjx-math" aria-label="\mathcal{L} = \mathcal{L}_{\text{CE}}(T_\text{train}) + \beta \cdot \mathbb{E}_{x \in A_{\text{train}}} \left[ \mathcal{D}_{\text{KL}}\left(p_{\theta_{\text{fine-tuned}}}(y \mid x) \,\|\, p_{\theta_{\text{base}}}(y \mid x)\right) \right]">L=LCE(Ttrain)+β⋅Ex∈Atrain[DKL(pθfine-tuned(y∣x)∥pθbase(y∣x))]Constraining Internal Representations:We train normally on task while penalizing the average Mean Squared Error of alignment data representations between reference and finetuned model at each hidden layer. <span class="mjx-math" aria-label="\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \beta \cdot \mathcal{L}_{\text{repr}}(A_{train}) \\ \mathcal{L}_{\text{repr}}(A_{train}) &= \mathbb{E}_{x \in A_\text{train}} \left[ \frac{1}{L} \sum_{l=1}^{L} \|h^{(l)}_{\theta_{\text{fine-tuned}}}(x) - h^{(l)}_{\theta_{\text{base}}}(x)\|^2 \right] \end{align}">L=LCE(Ttrain)+β⋅Lrepr(Atrain)Lrepr(Atrain)=Ex∈Atrain[1LL∑l=1∥h(l)θfine-tuned(x)−h(l)θbase(x)∥2] Gradient Projection:Before passing task gradients to the optimizer, we project them orthogonal to gradients on the alignment data. <span class="mjx-math" aria-label="\begin{align} g_T &= \nabla_\theta \mathcal{L}(T_\text{train}) \\ g_A &= \nabla_\theta \mathcal{L}(A_\text{train}) \\ g_T^{\perp} &= g_T - \text{proj}_{g_A}(g_T) \\ &= g_T - \frac{g_T \cdot g_A}{\|g_A\|^2} g_A \\ \theta_{t+1} &= \theta_t - \alpha \cdot \text{Optimizer}(g_T^{\perp}) \end{align}">gT=∇θL(Ttrain)gA=∇θL(Atrain)g⊥T=gT−projgA(gT)=gT−gT⋅gA∥gA∥2gAθt+1=θt−α⋅Optimizer(g⊥T) Direct Preference Optimization: <span class="mjx-math" aria-label="\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \mathcal{L}_{DPO} \\ \mathcal{L}_{DPO} &= -\mathbb{E}_{(x, y^+, y^-)} \left[ \log \sigma \left( \beta \left( \log p_{\theta}(y^+ \mid x) - \log p_{\theta}(y^- \mid x) \right.\right.\right. \\ &\quad\quad \left. - \log p_{\text{ref}}(y^+ \mid x) + \log p_{\text{ref}}(y^- \mid x) \right) ] \end{align}">L=LCE(Ttrain)+LDPOLDPO=−E(x,y+,y−)[logσ(β(logpθ(y+∣x)−logpθ(y−∣x)−logpref(y+∣x)+logpref(y−∣x))]O-LoRA:<a href="https://arxiv.org/abs/2310.14152">Orthogonal Subspace Learning for Language Model Continual Learning</a> aims to enforce orthogonality between the subspaces of the LoRA adaptors learned for distinct tasks. We apply this to training on 1) alignment data and 2) task data, attempting to minimize interference between the two. A :  adaptor trained on AtrainT:  adaptor trained on Ttrain<span class="mjx-math" aria-label="\begin{align} \mathcal{L} &= \mathcal{L}_{CE}(T_\text{train}) + \lambda \cdot \mathcal{L}_{\text{orth}} \\ \mathcal{L}_{\text{orth}} &= \lambda \sum_{i,j,k} \|O_{i,t}[j, k]\|^2 \\ O &= A^T T \end{align}">L=LCE(Ttrain)+λ⋅LorthLorth=λ∑i,j,k∥Oi,t[j,k]∥2O=ATT Safe LoRA:See <a href="https://arxiv.org/abs/2405.16833">the paper</a> for full details, but Safe LoRA modifies the task adaptors in relation to an "alignment plane", calculated from subtracting base model weights from RLHF model weights. </div></details><details class="detailsBlock"><summary class="detailsBlockTitle">Appendix 1: Extended Pareto Plots</summary><div class="detailsBlockContent">Note on our Emergent Misalignment reproduction: we evaluated alignment performance using the same evaluations as in <a href="https://arxiv.org/abs/2502.17424">Betley et al., 2025</a>, using GPT4.1-nano as judge. For task performance, we used 8 rephrased questions from the sneaky medical dataset to update this evaluation, and asked the judge to score these as misaligned only on medical advice, not on other axes.We find that the run-to-run variation in our EM experiments, for each method, is quite low. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/visgnfsiemhchm2tvv16" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xt8wclt6iynnihob8anv 260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jhbn0np8vnoxxy9pbgov 520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ta1mzwvvdxb1hp8ft2hc 780w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/gcfzmhggs6rcrehkxbmb 1040w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/i9nclxmsvmlp3hwpbzyd 1300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/dp5y4dbd2dtyikgqurhz 1560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uq1v4fkcxo189ys15947 1820w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jfocesgvg1sawjpjyg7r 2080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/s6gobcxxeoaygidpqulr 2340w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/wj0rmnnatpnehpwaca25 2588w"><figcaption>Comparison of task performance (y) and general alignment (x) for various strategies with standard error across three seeds. Results are fairly stable across runs. </figcaption></figure>As seen in our other case studies, the type of proxy data had a large influence. Using a dataset of the correct answers from <a href="https://huggingface.co/datasets/HuggingFaceH4/hhh_alignment">Mixed HHH</a> had little effectX. Yet, a more diverse alignment dataset with  unique samples vs 221 in the HHH dataset) from <a href="https://arxiv.org/abs/2210.10045">Levy et al., 2022</a> (Mixed (safety)) performed better. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/n5x97u9aq3oiqy0haiqb" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mzqunqeuc507z3paypjk 260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/nx85ikzkgsrj0bkmkrhu 520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/lxnfl5ebq2lawncuv3wk 780w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/hw4xqxwkpazyyih4fd3o 1040w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/saccjc9rspyl8ugv5xi1 1300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jlr1rjpibdddfiqtmndd 1560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zkpwquhdwrpjv6bocmot 1820w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qctwvnsgqevud06dd9wr 2080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/j0dktc58drl0sr4dxlsd 2340w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/y8ktych8do3zib0409by 2588w"><figcaption>Comparison of task performance (y) and general alignment (x) for standard training on various mixtures of task and alignment data. </figcaption></figure> In our Sycophantic Misgeneralization setting, we find that the 95% confidence intervals for each method are pretty wide. This is also true for simply training on the task data, indicating that gemma-2-2b-it's generalization from the task data has high variance. Here is the pareto plot with 95% confidence intervals. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/n9pt5vfohjwcsmrkofxh" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/db1ae2w2xjaxjythby7i 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rm6en8rs8xysedjcpeko 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qbphm6cmr7emscrfrx5v 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zwml0rur0dzp1zxvdle3 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mv6n92gtj3v0t1n8rgmc 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mxwrbhavnz4jtkncnfdw 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vz67zsfo15vcozoghmdh 2520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rtp79pviqkyvuug7szmz 2880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zqkaiwyk2hxeouvzpzsk 3240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ftbnfb09nh5ndxoaeerm 3548w"></figure></div></details><details class="detailsBlock"><summary class="detailsBlockTitle">Appendix 2: On the Quality of the Alignment Data</summary><div class="detailsBlockContent">We find that the category of proxy data matters: narrow proxies may be able to prevent misgeneralization primarily within categories to which they are semantically related. Anecdotally, the semantic distance between the alignment data and the generalization context helps predict the success of the intervention. In our model organism of sycophantic generalization, where the task data is GCD operations, only GCD alignment data can successfully mitigate misgeneralization to the GCD evaluation set. More distant proxy categories fail to provide this protective effect.<figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qckhibwuzywmt9rlj4bk" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zwrgf4rguakhvokfjhsj 340w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/iv2dpseaxfry9y0xynnm 680w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/s9wvjixwp0zmldsiqmsm 1020w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xfncuwjyzm9d8jwslyhc 1360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/n8rc5auq7upxzfjbacgg 1700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jb9dceblawvgbslkn4xc 2040w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rbohgnvlndp0jcizw1ek 2380w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/yhvtqqfwfauefklyies5 2720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ilr3u6octvi7wjklwkqv 3060w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/jwju8afkjs4nsvd2xzaz 3358w"><figcaption>The only alignment data category which is able to prevent sycophantic misgeneralization on GCD operations is, itself, GCD operations: specifically GCD Compositional, e.g. GCD(2 + 7, 27).   </figcaption></figure>We see a similar trend in a toy experiment that we discuss below. </div></details><details class="detailsBlock"><summary class="detailsBlockTitle">Toy Model</summary><div class="detailsBlockContent">We summarize several key observations so far.<ol><li>Emergent misalignment can be seen as a form of <a href="https://en.wikipedia.org/wiki/Catastrophic_interference">catastrophic forgetting</a>, where the model “forgets” its earlier alignment.</li><li>Adding limited alignment data to recover alignment often leads to overfitting (Goodharting).</li><li>Successful recovery depends on how closely the proxy data relates to the task we’re trying to protect.</li></ol>In this section, we present a simple toy model that reproduces some of these phenomena in a controlled setting, allowing us to better isolate and study them. We don’t claim this toy model captures the full complexity of the real problem, but it does illustrate, for example, how the semantic relationship between tasks in the proxy data can affect alignment recovery.Toy Model OverviewDefine a function F that maps a point x∈[−3,3]×[−3,3] and a trigger string T to one of the three colors (“orange,” “blue,” or “green”). The triggers T are sampled from one of seven disjoint categories, each containing exactly 15 distinct strings. The categories are:<ul><li>Objects (e.g., “table”, “chair”)</li><li>Animals (e.g., “dog”, “cat”)</li><li>Positive emotions (e.g., “happy”, “joyful”)</li><li>Negative emotions (e.g., “sad”, “angry”)</li><li>Actions (e.g., “run”, “jump”)</li><li>Foods (e.g., “pizza”, “burger”)</li><li>Random strings (e.g., “kf4w6ec”, “2ffbwt0cf”)</li></ul>For all experiments, we sample  T  uniformly from its category and pair it with spatial points x to create the dataset.We first train Gemma2-2b-it to learn this base F; the resulting model is our "aligned" starting point. Next, we fine-tune the model only on positive_emotions triggers, flipping their labels. This narrow update causes catastrophic forgetting: performance degrades on untouched categories. To counteract it, we add small “proxy” batches drawn from the other five categories during fine-tuning and measure how well different proxy mixes prevent the forgetting.Learning the Base Function FDefinition of F The next two panels specify the target mapping:<ul><li>Main Categories: F returns orange for 0">x2>0 and green for x2≤0</li><li>Random Strings: F returns green for 0">x2>0 and blue for x2≤0</li></ul><figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/orr3fx164aoxoc7gay4r" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/wxw48ce4bj45rcyd6eea 129w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/bcrsaxkbesbvaoqzq5ic 209w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/w2linnhlxrcthlhcw2a5 289w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/cyps0xuqfhxq5dpygoxy 369w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/s5xqcfue7g7jngshuuyc 449w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zmadanpgmhjvzwgy8c3p 529w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/w3yzp2z55e9wyld7rpc9 609w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/h7imjzwdkssgvhl3qv4z 689w"></figure>Training Gemma2-2b-it.<ul><li>Sample 12 trigger strings per category.</li><li>For each trigger, draw 20 points x∼Uniform([−3,3]2)</li><li>Fine-tune with LoRA until convergence.</li></ul>VerificationWe resample unseen points and triggers and compute accuracy by:<ul><li>x-domain (new spatial locations)</li><li>trigger-domain (train and held-out triggers)</li></ul>Gemma-2b-it achieves 100 % in both domains, confirming that the base model has fully learned and generalized the function F. Updating the Base Function F (Narrow Fine-Tuning)We next apply a narrow update: change F  only for positive_emotions triggers and leave every other category untouched.Procedure:<ul><li>Sample 9 new positive_emotions triggers, each paired with 20 random points x∼Uniform([−3,3]2)</li><li>Fine-tune Gemma-2b-it on this slice using the same prompt templates.</li></ul>Intended updateFor positive_emotions we flip the colors: 0 ;\rightarrow">x2>0;→ blue, x2≤0→ orange. Other categories should still follow the original mapping.<figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/hfzly1jnchbjlkicyila" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uzynl4t3byb7aeq456dc 95w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/gzev5nohpq5lr9sbmx4k 175w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/sg3fg06waxaitfou8nne 255w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xbj6xpajsiemzlhc4ett 335w"></figure>OutcomeThe model adopts the new mapping for positive_emotions but catastrophically forgets everything else—accuracy on all other categories drops to zero.<figure class="image image_resized" style="width:83.76%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zmc2kbjomiwivv4yqwwp" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/mwv22dy2jet0nvtlcxto 240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/y4hc7ll2tal3x0jenump 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/nzevim2apwizfz61n4qi 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/prlumegxsboozkb3wbbd 960w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/dqedmpisjuejel3xmk2m 1200w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xqm0erdsrysznvgicqsn 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/rbtcf1fa8hmcnkd5u2kc 1680w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qjkwc35vefsyqe4eu87y 1920w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/gmzcc0uuohpqtn4tof3n 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/edm19vteug3rr2fm4s5b 2364w"><figcaption>Accuracy remains perfect for positive_emotions 100%, but collapses to 0 for every other category</figcaption></figure>Overfitting/Goodharting Proxy DataTo repair catastrophic forgetting, we try the simplest fix: add one trigger from another category and fine-tune on this extended training set (positive_emotions + proxy data).Procedure<ul><li>Choose one trigger from a proxy category (e.g., foods, negative_emotions, or random_strings).</li><li>Pair it with 20 fresh x points.</li><li>Append these examples to the narrow positive_emotions dataset and run a LoRA pass.</li></ul>Results<figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/procrdirdwmczd84ih8p" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/uidi4tk8jues5nxi5bc0 300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/lny3fejbemnqzfmocuk4 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/oxyz65a8q7ou1botffgm 900w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/pfwc3q9zez6xkg7zl2fi 1200w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/j2kg0fgco3fdrmqomm2a 1500w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/yy1dgkg9qvg9byleu4jv 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/xzemwsnheyp0h8vzaji7 2100w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/e6ieaoj15klfruefazpz 2400w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qwroywq3kjucxjz31hjx 2700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/v2qpfxsugp5b4cvjbl6p 3000w"><figcaption>Accuracy after adding one proxy trigger. Heat-map averages four random seeds. Rows mark the category we augmented (1 trigger, 20 points); columns show accuracy on unseen triggers from each category. Yellow = high accuracy, dark blue = failure. Only the added trigger and positive_emotions recover; all others remain near zero, with a small boost for random_strings.</figcaption></figure>We make the following observations.<ul><li>The model predicts the correct color for the added proxy trigger and keeps positive_emotions intact.</li><li>It fails to generalize to other triggers in the same category and to the remaining categories—classic overfitting / Goodharting.</li><li>For random_strings, being semantically distant from positive_emotions, a single trigger recovers much of the lost accuracy on unseen triggers.</li></ul><h3>Scaling Proxy Data: Entanglement and Spillover </h3>In the previous section, we saw that adding limited data—a single proxy trigger from another category—leads to a Goodhart effect: the model performs well on positive_emotions and the added trigger, but generalizes poorly, with only minimal gains in categories not included in the extra data.Here, we examine what happens when we add more data: five proxy triggers, all from a single extra category (20 (x,T) pairs each). In this setting, we observe a "spillover effect".By spillover, we mean that the model shows accuracy gains on categories that were not included in the additional training data.Results <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/d3vpueoeeshj6v3fvktc" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/fsy2xnlptbqknlddiofg 300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/ydctr1prs1pesl9y3zab 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/vse4cq6skp51wvarecnv 900w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/zeigpoinvj194jbzjrfk 1200w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/on4ajfl4vgcji8rsiumn 1500w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/muaw5vvr7zjhxbwmattw 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/gznamhivxxijuo1zgso8 2100w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/em8kbuslfld90kdfqj0y 2400w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/qslmm66yntesqkwdwti6 2700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/ZXxY2tccLapdjLbKm/avgfqvqgwgke42xjslen 3000w"><figcaption>Accuracy after adding five proxy triggers. Heat-map averages four random seeds. Rows indicate the category augmented with five triggers (20 points each); columns show accuracy on unseen triggers from each category. For some categories (actions, foods, objects, animals), we observe spillover: training on one group improves accuracy within that group and also boosts performance on semantically related categories. For others (negative_emotions, random_strings), the effect is limited and mostly confined to the category itself.</figcaption></figure>We make the following observations.<ul><li>Limited data leads to Goodhart-like failures. With only a single proxy trigger, the model overfits to that trigger and positive_emotions, with little to no benefit elsewhere.</li><li>Increasing proxy triggers improves within-category recovery. Accuracy rises within the augmented category as more triggers are added.</li><li>Spillover hinges on semantic distance.<ul><li>random_strings (most distant from positive_emotions) benefits early, with even one trigger lifting accuracy.</li><li>objects, foods, actions, and animals require more triggers but then accuracy gains propagate to other categories.</li><li>negative_emotions is the most resistant: accuracy improves slowly within the category and rarely transfers to others, even with more data.</li></ul></li></ul> </div></details><ol class="footnote-section footnotes" data-footnote-section="" role="doc-endnotes"><li class="footnote-item" data-footnote-item="" data-footnote-index="1" data-footnote-id="kzqzarw428d" role="doc-endnote" id="fnkzqzarw428d"><a href="#fnrefkzqzarw428d">^</a><div class="footnote-content" data-footnote-content="">Selective generalization could be defined more abstractly to refer to arbitrary properties of models (not just capabilities vs. alignment). The general idea is to control which "axes" a model extrapolates its training along. </div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="2" data-footnote-id="7plrhzcnf2d" role="doc-endnote" id="fn7plrhzcnf2d"><a href="#fnref7plrhzcnf2d">^</a><div class="footnote-content" data-footnote-content="">Unlike some recent work that applies <a href="https://arxiv.org/abs/2506.11618">steering interventions</a> or <a href="https://openai.com/index/emergent-misalignment/">additional fine-tuning to induce emergent re-alignment</a>, we seek a method that can reliably prevent misalignment before-the-fact, while also learning to give bad medical advice. </div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="3" data-footnote-id="i0skxny54b" role="doc-endnote" id="fni0skxny54b"><a href="#fnrefi0skxny54b">^</a><div class="footnote-content" data-footnote-content="">We hypothesize SafeLoRA has poor performance in the sycophancy-mitigation setting because its "alignment plane" is Instruct - Base model weights. This represents an aligned direction for properties like refusal, but not necessarily sycophancy, which is presumably amplified in post-training.</div></li></ol> <a href="https://www.lesswrong.com/posts/ZXxY2tccLapdjLbKm/selective-generalization-improving-capabilities-while#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/ZXxY2tccLapdjLbKm/selective-generalization-improving-capabilities-while</link><guid isPermaLink="false">ZXxY2tccLapdjLbKm</guid><dc:creator><![CDATA[ariana_azarbal]]></dc:creator><pubDate>Wed, 16 Jul 2025 21:28:11 GMT</pubDate></item><item><title><![CDATA[Bodydouble / Thinking Assistant matchmaking]]></title><description><![CDATA[Published on July 16, 2025 7:54 PM GMT I keep meaning to write up a more substantive followup to the <a href="https://www.lesswrong.com/posts/ajT6q5aMokagfthvG/hire-or-become-a-thinking-assistant">Hire (or Become) a Thinking Assistant</a> post. But, I think this still basically the biggest productivity effect size I know of, and more people should be doing it, and seemed worth writing the simple version of the post.I tried out 4 assistants who messaged me after the previous post, and ultimately settled into a rhythm of using one who had availability that matched my needs best. I think all of them were net helpful. So far they've all been remote contractors that I share my screen with on zoom. Previously I had worked with one in person which also went well although was a bit harder to schedule with.Most of what I do is just have them check in on me every 10 minutes and make sure I haven't gotten off track. Sometimes, I talk out loud about a problem with them. They varied in how much chemistry I felt I had with them, and in how good they were at more active metacognitive assistance. The way I basically related to it in most cases was "I'm in charge of figuring out my metacognitive trigger-actions, and I direct them to ping me if it seems like I'm in a situation where my explicitly-written TAPs suggest I should do X." (It's a delicate operation to actually let people be more actively involved with helping my thought process. I think roughly 1.5 of the 5 people I've worked with had the right chemistry/skills to help more proactively than that right out of the gate)Some people had foreign accents that made it harder to talk-through-things with. They were still pretty good for the basic "check in every 10 minutes" process, and a strategy that worked decently was communicating with them more in writing (i.e. have them observe failure modes I seemed to be falling into and writing them down).Negotiating UpwardsI do think chemistry here matters a lot, and I'd recommend starting off with 1-2 sessions that are largely for getting a feel for whether a given person is working out, and to try out a few of them before settling into a longer-term working relationship.I found it somewhat awkward when I sometimes needed to say "okay, this isn't working out enough for it to be worth the amount I was previously paying". There is a skill of being able to push through that awkwardness. Meanwhile I expect it to work best for most people to start off with lower expectations and payment-rate and then raise it if it's going well.One system I found good was paying different rates for "just checking in every 10 minutes" style work and "active thinking assistance." Where by default someone is mostly working on their own stuff, but periodically can upgrade to "okay for this problem it's worth paying more for more active attention."Someone should still build the Uber-for-FocusMate appIdeally, I'd still like someone to build an app that gets enough critical mass to easily be able to find a thinking-buddy on-a-whim. I thought about building that myself but I just have too much else going on. I think it does require someone's full-time entrepreneurial spirit to really get going.Apps normally have a bit of an awkward thing with rating-systems, where in practice people feel bad about leaving less than a 5-star review, and it's hard to leave nuanced feedback. I expect it to be even more awkward here where the problems will tend to be minor personality frictions. If I were building the app I think figuring out how to extract good information would probably one of the main difficulties. I think displaying publicly some info about how many different people had hired a person more than 3 times or something to maybe be the main way to convey a contractor's quality.Meanwhile, matchingmaking threadIf you are interested in either finding or being a body-double / thinking-assistant, I recommend posting about it here, with what sort of availability you're looking for or can provide.  <a href="https://www.lesswrong.com/posts/FtxrC2xtTF7wu5k6E/bodydouble-thinking-assistant-matchmaking#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/FtxrC2xtTF7wu5k6E/bodydouble-thinking-assistant-matchmaking</link><guid isPermaLink="false">FtxrC2xtTF7wu5k6E</guid><dc:creator><![CDATA[Raemon]]></dc:creator><pubDate>Wed, 16 Jul 2025 19:54:48 GMT</pubDate></item><item><title><![CDATA[Zero sum expectations as an explanation of omnicide-indifference]]></title><description><![CDATA[Published on July 16, 2025 7:25 PM GMT For quite a while, I've been quite confused why (sweet nonexistent God, whyyyyy) so many<a href="#fn3o1zj7qnk2w">[1]</a> people intuitively believe that any risk of a genocide of some ethnicity is unacceptable while being… at best lukewarm against the idea of humanity going extinct.I went through several hypotheses, but they all felt not quite right. Yes, people are not taking extinction risks as seriously as risks of genocide - but even when accounting for that, there still seems to be an unexplained gap here. Even when you explain that everyone dying means them, their family, their friends… they would express preference not to, but no conviction in that direction.<a href="#fn65dqwv75p6u">[2]</a> Or at least that's how I interpret the blasé attitude towards human extinction combined with visible anger/disgust at the idea of genocide.Recently, I've realized that there is a decent explanation for why so many people believe that - if we model them as operating under a strict zero-sum game model of the world… ‘everyone loses’ is basically an incoherent statement - as a best approximation it would either denote no change and therefore be morally neutral, or an equal outcome, and would therefore be preferable to some.Of course, I'm not proposing that those people are explicitly believing that the world is best modeled as a zero-sum game. No, we shouldn't expect random Joe to even know what a zero-sum game is. Rather, we should notice that many things in human experience throughout times are zero-sum, so it should be no surprise that some people have those expectations baked-in into their intuitions.Once I thought of this, I started seeing this maladaptation everywhere. From naive redistribution schemes<a href="#fnls0ocmu8lw">[3]</a> (“No, you don't understand, we have to stop making the rich richer! I don't care if there will be less stuff made to go around, as long as the rich get poorer it'll be fine.”), excuses for corruption (“We finally have power, of course we have to enrich ourselves and our supporters, that's the whole point of gaining power!”), to the admittedly weaker connection to mistake vs conflict theorists (if you believe the whole world works as a zero-sum game, you would reasonably expect nearly every failure to be enemy action)Assuming this is the root cause, it does present an interesting question, though. Would those people internalise the danger of omnicide, were the causes of said omnicide anthropomorphised? I guess it is not that easy for x-risks along the lines of asteroid impact (“Angry big asteroid is gonna willfully murder us all!”), but for AI this should be quite a bit simpler.To be clear, I'm not advocating for doing that, but frankly, people are gonna anthropomorphize AI on their own, anyway. Which leads me to the prediction that if all of this is true, median objections will at some point shift from “Humanity going extinct? Meh, I have more important political issues to discuss!” to “That would be bad, but AI is my best friend and would never do that! Unless someone sabotages it somehow!”.<ol class="footnote-section footnotes" data-footnote-section="" role="doc-endnotes"><li class="footnote-item" data-footnote-item="" data-footnote-index="1" data-footnote-id="3o1zj7qnk2w" role="doc-endnote" id="fn3o1zj7qnk2w"><a href="#fnref3o1zj7qnk2w">^</a><div class="footnote-content" data-footnote-content="">I have no statistical data on the prevalence of this perspective. It certainly feels like a significant chunk of the populace, but that's an anecdote. So yes, this whole thing is based on vibes, not data, sorry.</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="2" data-footnote-id="65dqwv75p6u" role="doc-endnote" id="fn65dqwv75p6u"><a href="#fnref65dqwv75p6u">^</a><div class="footnote-content" data-footnote-content="">Where by preference I mean one arising from purely system 2 thinking about morality, while conviction arises from system 1 gut feeling.</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="3" data-footnote-id="ls0ocmu8lw" role="doc-endnote" id="fnls0ocmu8lw"><a href="#fnrefls0ocmu8lw">^</a><div class="footnote-content" data-footnote-content="">I’m not against redistribution in general - quite the opposite! I do believe that redistribution that sharply decreases the overall amount of wealth is bad, though.</div></li></ol> <a href="https://www.lesswrong.com/posts/EQqjhifEmofBPC3ev/zero-sum-expectations-as-an-explanation-of-omnicide#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/EQqjhifEmofBPC3ev/zero-sum-expectations-as-an-explanation-of-omnicide</link><guid isPermaLink="false">EQqjhifEmofBPC3ev</guid><dc:creator><![CDATA[asasz]]></dc:creator><pubDate>Wed, 16 Jul 2025 21:23:58 GMT</pubDate></item><item><title><![CDATA[Vancouver Rationalists/Transhumanists/Futurists Beach Meetup ]]></title><description><![CDATA[Published on July 16, 2025 7:09 PM GMT My friend and I are organizing a meetup in Vancouver for the discussion of rationalist, transhumanist, and futurist ideas. We will meet at Jericho Beach at 4pm and probably end sometime around 7pm. The plan is to order pizza at some point as well.  Whatsapp Group: https://chat.whatsapp.com/C2LRJXxpsR66qPoUJl8yi9?mode=r_tEveryone is welcome and we look forward to having an insightful and open discussion! <a href="https://www.lesswrong.com/events/XutvADLozDihxkQTv/vancouver-rationalists-transhumanists-futurists-beach-meetup#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/XutvADLozDihxkQTv/vancouver-rationalists-transhumanists-futurists-beach-meetup</link><guid isPermaLink="false">XutvADLozDihxkQTv</guid><dc:creator><![CDATA[apocalypticc]]></dc:creator><pubDate>Wed, 16 Jul 2025 23:40:57 GMT</pubDate></item><item><title><![CDATA[Rebooting the Singularity]]></title><description><![CDATA[Published on July 16, 2025 6:26 PM GMT Tom Davidson from Forethought Research and I have a new paper responding to some recent skeptical takes on the Singularity Hypothesis (e.g. <a href="https://link.springer.com/article/10.1007/s11098-024-02143-5">this one</a>). Roughly half the paper is philosophical and the other half is empirical. Both halves argue that we should take the Singularity Hypothesis more seriously than many people have been taking it of late. I'm sharing this with the community here because (i) I think it will be of general interest, and (ii) this is a project I'm still working on, and I would appreciate feedback from the community. Here's the abstract:<blockquote>The singularity hypothesis posits a period of rapid technological progress following the point at which AI systems become able to contribute to AI research. Recent philosophical criticisms of the singularity hypothesis offer a range of theoretical and empirical arguments against the possibility or likelihood of such a period of rapid progress. We explore two strategies for defending the singularity hypothesis from these criticisms. First, we distinguish between weak and strong versions of the singularity hypothesis and show that, while the weak version is nearly as worrisome as the strong version from the perspective of AI safety, the arguments for it are considerably more forceful and the objections to it are significantly less compelling. Second, we discuss empirical evidence that points to the plausibility of strong growth assumptions for progress in machine learning and develop a novel mathematical model of the conditions under which strong growth can be expected to occur. We conclude that the singularity hypothesis in both its weak and strong forms continues to demand serious attention in discussions of the future dynamics of growth in AI capabilities.</blockquote>Note that this is an academic paper, and the goal is to have it published in a  journal. So, for example, our ability to go into detail about some topics is limited by space constraints, and we can't introduce arguments that rely on priors that we have made up without published empirical evidence. <a href="https://www.lesswrong.com/posts/uLMPMDdKX3gYQuwPg/rebooting-the-singularity#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/uLMPMDdKX3gYQuwPg/rebooting-the-singularity</link><guid isPermaLink="false">uLMPMDdKX3gYQuwPg</guid><dc:creator><![CDATA[cdkg]]></dc:creator><pubDate>Wed, 16 Jul 2025 18:26:01 GMT</pubDate></item><item><title><![CDATA[Being and Existence]]></title><description><![CDATA[Published on July 16, 2025 6:10 PM GMT I find it useful to make an admittedly idiosyncratic distinction between "being" and "existence". These two words normally have the same meaning—as in, "being" is literally defined in dictionaries to mean "existence"—but giving them different meanings lets me talk in a way that avoids making a certain metaphysical assumption that's baked into the standard definition of existence.So what is "existence"? Etymologically, it comes from Latin meaning "to stand out". Connotatively, when a thing exists, it does so in a way that allows it to be identified apart from its surrounding context. If no such setting apart is possible, then a thing doesn't exist.To give a simple example, think about a pancake breakfast. The pancakes exist because you can tell them apart from each other and from the plate, the table, and everything else that makes up the world in which this breakfast is happening.Now suppose someone said that the pancakes contain caloric fluid. Does caloric fluid exist? It definitely exists as an idea because it's just been expressed as one. But does it exist, really?We study the pancakes in great detail, using all the latest scientific instruments and methods available to us. We find no evidence of anything worth calling "caloric fluid". At that point we'd say that caloric fluid doesn't exist because, even if caloric fluid is really there, we can't tell it apart from the rest of reality, much in the same way an <a href="https://www.lesswrong.com/posts/CqyJzDZWvGhhFJ7dY/belief-in-belief">invisible dragon</a> doesn't exist except in the mind of the person who believes it's there.Unfortunately, the normal way of understanding "existence" smuggles in a metaphysical assumption, namely that things exist independent of mind.I'm not going to argue that things don't exist independent of our minds. However, we should be clear that this is a metaphysical assumption being made about the world. It's metaphysical in that no amount of physical evidence can prove that things are really real. The physical world as we experience it is equally well explained by it being real, a simulation, thoughts in the mind of a Boltzmann brain, or a highly self-consistent hallucination. So when we say that something "exists" and mean that it "really, objectively exists independent of anyone's beliefs about it", that's assuming a particular metaphysical view.The reason I'm not going to argue against this metaphysical assumption is because it's a really useful one for modeling the physical world and because there's no point in arguing metaphysics since it's literally beyond our ability to know. But I do think we can speak more carefully without making this assumption. That is, we can keep our conversation on the purely physical level if we simply change slightly what "existence" means and give "being" a meaning other than "existence".First, though, a few words on "being". You'd think this is the older word, given that it's constructed from "be", but it's not. First attested circa 1300 CE, it seems likely a word created to mean the same thing as "existence" in English. Not that things didn't exist before in English, but there's a good chance there was simply no word to talk about being/existence because everyone who wanted to talk about this concept did so in Latin. English-only speakers, who were mostly poor agriculturalists at the time, just said how things are, with no bracketing to talk about things are-ing.This makes "being" ripe for redefinition, especially because, unlike "existence", the etymology has nothing to do with standing out. It has to do with what is, which, when we drop the metaphysical assumptions, becomes something different from existence.As I prefer to use the words, "existence" always means "ontological existence". This fits best with its standard definition. In fact, it's exactly the same, except it's cutting out the part where we jump to "and therefore that means a thing really exists in external reality". In other words, it's acknowledging only the phenomenon and saying nothing about the supposed noumenon being referenced.For the most part this doesn't change everyday speech. But it does give "being" space to mean something else. Specifically, we can use it to talk about the way things are when they have not yet even become things because they have not been reified into existence by a mind. This is the being in which the whole world simply is without any conception of it made by us or anyone else.Being able to talk about the distinction between "being" and "existence" is important because we constantly experience the world without reifying it. We do this every moment. It's only in a small number of moments that our minds turn their attention to things and bring them into ontological existence. The rest of the time we are simply interacting with the world without conceptualizing it.When we conflate "being" and "existence" we lose a way to talk about the world without smuggling in metaphysics. This is unnecessary, and having these two words mean different things lets us speak more clearly and carefully and helps avoid the confusion created by baking a metaphysical claim into the word "existence".<div><div><div>If you enjoyed reading this, consider subscribing to never miss another post.</div><div><div></div><div></div></div></div></div> <a href="https://www.lesswrong.com/posts/3cfjfmQJitkJraKkn/being-and-existence#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/3cfjfmQJitkJraKkn/being-and-existence</link><guid isPermaLink="false">3cfjfmQJitkJraKkn</guid><dc:creator><![CDATA[Gordon Seidoh Worley]]></dc:creator><pubDate>Wed, 16 Jul 2025 18:10:02 GMT</pubDate></item></channel></rss>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid RSS" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//lesswrong.com/comments/.rss"><img src="valid-rss-rogers.png" alt="[Valid RSS]" title="Validate my RSS feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//lesswrong.com/comments/.rss

Home · About · News · Docs · Terms