FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 476: Image not in required format [help]

... ge/upload/v1497915096/favicon_lncumn.ico</url><title>LessWrong</title><l ...
                                             ^

line 1, column 769: Self reference doesn't match document location [help]

...  rel="self" type="application/rss+xml"/><item><title><![CDATA[From priso ...
                                             ^

line 1, column 0: Non-html tag: summary (2 occurrences) [help]

<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...

line 1, column 0: description should not contain data-footnote-reference attribute (5 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: description should not contain data-footnote-index attribute (10 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: description should not contain data-footnote-id attribute (15 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: description should not contain role attribute (11 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: description should not contain aria-label attribute (369 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: description should not contain aria-hidden attribute (369 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```
line 1, column 0: style attribute contains potentially dangerous content: margin-right: -0.081em; (5953 occurrences) [help]
```
<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...
```

line 1, column 0: description should not contain style tag [help]

<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elem ...

line 153, column 0: description should not contain data-footnote-section attribute [help]
```
print(H / np.log(2))  # Convert to bits</code></pre><ol class="footnote-sect ...
```
line 153, column 0: description should not contain data-footnote-item attribute (5 occurrences) [help]
```
print(H / np.log(2))  # Convert to bits</code></pre><ol class="footnote-sect ...
```
line 153, column 0: description should not contain data-footnote-back-link attribute (5 occurrences) [help]
```
print(H / np.log(2))  # Convert to bits</code></pre><ol class="footnote-sect ...
```
line 153, column 0: description should not contain data-footnote-content attribute (5 occurrences) [help]
```
print(H / np.log(2))  # Convert to bits</code></pre><ol class="footnote-sect ...
```

line 153, column 2757: description should not contain relative URL references: #fn6pf3mmzjza5 (10 occurrences) [help]

... ross-ontologies#comments">Discuss</a>]]></description><link>https://www. ...
                                             ^

Source: http://lesswrong.com/comments/.rss

<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[LessWrong]]></title><description><![CDATA[A community blog devoted to refining the art of rationality]]></description><link>https://www.lesswrong.com</link><image><url>https://res.cloudinary.com/lesswrong-2-0/image/upload/v1497915096/favicon_lncumn.ico</url><title>LessWrong</title><link>https://www.lesswrong.com</link></image><generator>RSS for Node</generator><lastBuildDate>Thu, 04 Sep 2025 01:55:28 GMT</lastBuildDate><atom:link href="https://www.lesswrong.com/feed.xml?view=rss&karmaThreshold=2" rel="self" type="application/rss+xml"/><item><title><![CDATA[From prison with therapeutic humor to difficult reasoning]]></title><description><![CDATA[Published on September 4, 2025 1:01 AM GMT Here I’ll share a summary of what I learned teaching therapeutic humor courses in the prisons of Greater Buenos Aires, and ask for help with my next project: stimulating probabilistic rationality in these groups. Would they become more ethical, or just gain better tools to evolve as bandits, and maybe even become politicians.Definition of humorI was in prison applying my thesis (or maybe it’s closer to a hypothesis) that humor functions as a cognitive reward for rapid perspective-shifting  Who has a better chance of survival: a group of men rewarded for understanding, or one that isn’t?  So basically the only way I managed to define humor is: the pleasure of gaining perspectives quickly. The faster I understand, the higher my chance of survival and pleasure. Or am I just talking nonsense?Therapeutic humor in prisonSo, where did I apply this? At the largest universities in Latin America? No, prison.  The idea was to stimulate perspective-gaining through humor, so we had a course where I invited them to practice applied humor in different scenarios, from the most common to the most specific and difficult:<ol><li>In casual conversations among themselves, highlighting when something funny came up and asking why it was funny.  </li><li>In improvisations, theater-style, again noticing when something funny emerged.  </li><li>In jokes they told, following the structure.  </li><li>In more staged theatrical exercises, same principle.  </li><li>In stand-up, presenting something real from their own lives and inviting funny perspectives.  </li><li>The final challenge: finding perspectives on their hardest moments, stimulating them to laugh at their problems and see them as less heavy.  </li></ol>Pretty crazy that I, as a comedian, had never exposed my own hardest moment, yet I asked them to do it.  And they told me things like:“Well teacher, when I was 8 my mom died, my dad left me on the street, I was abused in several ways… until I ended up here. So how do I make a joke about that?”“Hehe… you got me, champ.”Of course, my point wasn’t to joke about their lives, but to help them see their past in a lighter way, giving them a better chance to handle it and focus on future goals.At the end of a course of weekly 3-hour classes, their “final project” was to tell me:<ol><li>An example of something in their life that was funny (as a reference).  </li><li>Their hardest moment.  </li><li>Their most glorious or happiest moment.  </li><li>A new way of seeing their hardest moment, some detail beyond the pain.  </li></ol>I can’t expose their names, but here are some anonymized examples <a href="https://docs.google.com/spreadsheets/d/130ZHvUQ94mAAGCAMxyuTfiW03tS-vVfAg8LfzZFzzPM/edit?usp=sharing">[link]</a>. Truth is, they helped me with some of my own biases.<h1>Results and lessons learned.</h1>By the end I was sad to finish the course. I built incredible connections I never expected. I mean, I was prepared for risk, but not for the amount of affection I’d feel.<a href="https://www.instagram.com/reel/CtzEaHspALQ/?igsh=NHZlNjdtZTVzMzk2"> [link to my emotional report at the end].</a> My mistakes, my biases.Sure, they could have been deceiving me the whole time, though they had no advantage to gain: I couldn’t help with their sentence or defense, and I couldn’t expose anything specific about the course. The only promise I made was that their life could be at least 1% happier if they attended.Of course, genetics suggest my chances of improving their lives were low, so I didn’t expect much. But in the final interviews, their difficult moments and pleasures touched me. They weren’t that different from me: their deepest pain and joy were social, losing friends and family. I thought that, as criminals, their worst moment would be being caught. But before being criminals, they had much more humanity than I had imagined.Of course, there were some exceptions, maybe 4 out of 100, who said their worst moment was being caught, and even told me they could use humor to trick people while stealing when released.So I kept thinking how I could help them more, not only emotionally with humor, but by questioning deeper human problems, social, intellectual, and physical factors that generate motivation and happiness. How could they have a better chance at being happy?<h1>Conclusions</h1>Humor gave us a first tool: every joke required a perspective shift, a small Bayesian update disguised as laughter. But I realized humor was not enough. It helped lighten their past, but it didn’t always provide a framework for their future choices.If laughter rewards perspective-shifting, rationality should reward model-building. Humor showed them how to see their story differently for a moment; probabilistic reasoning could help them see decisions differently in a sustainable way. Humor teaches “there is always another angle,” but probability teaches “some angles are more likely to bring better outcomes.”That is why I am now exploring how to complement therapeutic humor with methods from information theory and probabilistic thinking, so the same cognitive flexibility that laughter sparks can become a structured tool for life decisions. <a href="https://www.lesswrong.com/posts/K9jFm9P5o258LDDW8/from-prison-with-therapeutic-humor-to-difficult-reasoning#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/K9jFm9P5o258LDDW8/from-prison-with-therapeutic-humor-to-difficult-reasoning</link><guid isPermaLink="false">K9jFm9P5o258LDDW8</guid><dc:creator><![CDATA[P. João]]></dc:creator><pubDate>Thu, 04 Sep 2025 01:01:07 GMT</pubDate></item><item><title><![CDATA[Natural Latents: Latent Variables Stable Across Ontologies]]></title><description><![CDATA[Published on September 4, 2025 12:33 AM GMT <details class="detailsBlock"><summary class="detailsBlockTitle">Background on where this post/paper came from</summary><div class="detailsBlockContent">About a year ago, we wrote up a paper on natural latents for the <a href="https://www.iliadconference.com/proceedings">ILLIAD proceedings</a>. It was mediocre. The main shortcoming stemmed from using stochastic rather than deterministic natural latents, which give much less conceptually satisfying ontological stability guarantees; there was this ugly and confusing caveat on everything throughout the paper. We knew at the time that deterministic natural latents gave a much cleaner story, but we were more concerned about maintaining enough generality for our results to bind to reality than about getting the cleanest story.Recently, we proved that <a href="https://www.lesswrong.com/posts/Gd36HT7qLr684SYuQ/stochastic-natural-latent-implies-deterministic-natural">existence of a stochastic natural latent implies existence of a deterministic natural latent</a> (specifically under approximation; the exact case is easy). So now, we can write the paper the way we'd really like, without sacrificing generality.</div></details>This post is an overhaul of (and large improvement to) a paper we wrote about a year ago. The pdf version will hopefully be up on arxiv shortly, and we will link that from here once it's available. As of posting, this is probably the best first-stop mathematical intro to natural latents.<h2>Abstract</h2>Suppose two Bayesian agents each learn a generative model of the same environment. We will assume the two have converged on the predictive distribution (i.e. distribution over some observables in the environment), but may have different generative models containing different latent variables. Under what conditions can one agent guarantee that their latents are a function of the other agent’s latents?We give simple conditions under which such translation is guaranteed to be possible: the natural latent conditions. We also show that, absent further constraints, these are the most general conditions under which translatability is guaranteed. Crucially for practical application, our theorems are robust to approximation error in the natural latent conditions.<h1>Background</h1>When is robust translation possible at all, between agents with potentially different internal concepts, like e.g. humans and AI, or humans from different cultures? Under what conditions are scientific concepts guaranteed to carry over to the ontologies of new theories, (e.g. as general relativity reduces to Newtonian gravity in the appropriate limit?) When and how can choices about which concepts to use in creating a scientific model be rigorously justified, like e.g. factor models in psychology? When and why might a wide variety of minds in the same environment converge to use (approximately) the same concept internally?These sorts of questions all run into a problem of indeterminacy, as popularized by Quine<a href="#fn6pf3mmzjza5">[1]</a>: Different models can make exactly the same falsifiable predictions about the world, yet use radically different internal structures.On the other hand, in practice we see that<ul><li>Between humans: language works at all. Indeed, babies are able to learn new words from only a handful of examples, therefore from an information perspective nearly all the work of identifying potential referents must be done before hearing the word at all.</li><li>Between humans and AI: today's neural nets seem to contain huge amounts of human-interpretable structure, including apparent representations of human-interpretable concepts.<a href="#fnp4bzaufgaob">[2]</a></li><li>Between AI systems: the empirical success of grafting and merging<a href="#fnd937dkdf6d">[3]</a>, as well as the investigation by Huh et al. (2024)<a href="#fngr9l6s063y">[4]</a>, suggests that different modern neural nets largely converge on common internal representations. </li></ul>Combining those, we see ample empirical evidence of a high degree of convergence of internal concepts between different humans, between humans and AI, and between different AI systems. So in practice, it seems like convergence of internal concepts is not only possible, but in fact the default outcome to at least a large extent.Yet despite the ubiquitous convergence of concepts in practice, we lack the mathematical foundations to provide robust guarantees of convergence. What properties might a scientist aim for in their models, to ensure that their models are compatible with as-yet-unknown future paradigms? What properties might an AI require in its internal concepts, to guarantee faithful translatability to or from humans' concepts?In this paper, we'll present a mathematical foundation for addressing such questions.<h1>The Math</h1><h2>Setup & Objective</h2>We'll assume that two Bayesian agents, Alice and Bob, each learn a probabilistic generative model, MA<style>.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0}
.MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0}
.mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table}
.mjx-full-width {text-align: center; display: table-cell!important; width: 10000em}
.mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0}
.mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left}
.mjx-numerator {display: block; text-align: center}
.mjx-denominator {display: block; text-align: center}
.MJXc-stacked {height: 0; position: relative}
.MJXc-stacked > * {position: absolute}
.MJXc-bevelled > * {display: inline-block}
.mjx-stack {display: inline-block}
.mjx-op {display: block}
.mjx-under {display: table-cell}
.mjx-over {display: block}
.mjx-over > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-under > * {padding-left: 0px!important; padding-right: 0px!important}
.mjx-stack > .mjx-sup {display: block}
.mjx-stack > .mjx-sub {display: block}
.mjx-prestack > .mjx-presup {display: block}
.mjx-prestack > .mjx-presub {display: block}
.mjx-delim-h > .mjx-char {display: inline-block}
.mjx-surd {vertical-align: top}
.mjx-surd + .mjx-box {display: inline-flex}
.mjx-mphantom * {visibility: hidden}
.mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%}
.mjx-annotation-xml {line-height: normal}
.mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible}
.mjx-mtr {display: table-row}
.mjx-mlabeledtr {display: table-row}
.mjx-mtd {display: table-cell; text-align: center}
.mjx-label {display: table-row}
.mjx-box {display: inline-block}
.mjx-block {display: block}
.mjx-span {display: inline}
.mjx-char {display: block; white-space: pre}
.mjx-itable {display: inline-table; width: auto}
.mjx-row {display: table-row}
.mjx-cell {display: table-cell}
.mjx-table {display: table; width: 100%}
.mjx-line {display: block; height: 0}
.mjx-strut {width: 0; padding-top: 1em}
.mjx-vsize {width: 0}
.MJXc-space1 {margin-left: .167em}
.MJXc-space2 {margin-left: .222em}
.MJXc-space3 {margin-left: .278em}
.mjx-test.mjx-test-display {display: table!important}
.mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px}
.mjx-test.mjx-test-default {display: block!important; clear: both}
.mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex}
.mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left}
.mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right}
.mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0}
.MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal}
.MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal}
.MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold}
.MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold}
.MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw}
.MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw}
.MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw}
.MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw}
.MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw}
.MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw}
.MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw}
.MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw}
.MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw}
.MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw}
.MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw}
.MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw}
.MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw}
.MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw}
.MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw}
.MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw}
.MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw}
.MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw}
.MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw}
.MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw}
.MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw}
@font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')}
@font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')}
@font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold}
@font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')}
@font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')}
@font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold}
@font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')}
@font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic}
@font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')}
@font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')}
@font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold}
@font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')}
@font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic}
@font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')}
@font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')}
@font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')}
@font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')}
@font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold}
@font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')}
@font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic}
@font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')}
@font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')}
@font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic}
@font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')}
@font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')}
@font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')}
@font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')}
@font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')}
@font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')}
@font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')}
@font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold}
@font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}
</style> and MB respectively. Each model encodes a distribution P[X,Λi|Mi] over two "observable" random variables X1,X2 and some "latent" random variables Λi. Each model makes the same predictions about observables X, i.e.∀x:P[X=x|MA]=P[X=x|MB]  (Agreement on Observables)However, the two generative models may use completely different latent variables ΛA and ΛB in order to model the generation of X (thus the different superscripts for Λ). Note that there might also be additional observables over which the agents disagree; i.e. X need not be all of the observables in the agents' full world models.Crucially, we will assume that the agents can agree (or converge) on some way to break up X into individual observables X1,X2. (We typically picture X1,X2 as separated in time and/or space, but the math will not require that assumption.)We require that the latents of each agent's model fully explain the interactions between the individual observables, as one would typically aim for when building a generative model. Mathematically, X1,X2⫫|Λi,Mi [TODO: fix independence sign] (read "X1,X2 are independent given Λi under model Mi"), or fully written out∀i,x,λi:P[X=x|Λi=λi,Mi]=∏jP[Xj=xj|Λi=λi,Mi]  (Mediation)Given that Alice' and Bob's generative models satisfy these constraints (Agreement on Observables and Mediation), we'd like necessary and sufficient conditions under which Alice can guarantee that her latent is a function of Bob's latent. In other words, we'd like necessary and sufficient conditions under which Alice' latent ΛA is fully determined by Bob's latent ΛB, for any latent which Bob might use (subject to the constraints). Also, we'd like all of our conditions to be robust to approximation.We will show that:<ul><li>Necessity: In order to provide such a guarantee, Alice' latent ΛA must be a "natural latent", meaning that it satisfies Mediation plus a Redundancy condition to be introduced shortly.</li><li>Sufficiency: If Alice' latent ΛA is a "natural latent", then it provides the desired guarantee.</li><li>Both of the above are robust to approximation, when we weaken the guarantee to say that Alice' latent must have low entropy given Bob's latent (so in that sense ΛA is "approximately" a function of ΛB), and weaken the Mediation condition to allow approximation.</li></ul><h2>Notation</h2>Throughout the paper, we will use the graphical notation of Bayes nets for equations. While our notation technically matches the standard usage in e.g. Pearl<a href="#fn89wpwwprup9">[5]</a>, we will rely on some subtleties which can be confusing. We will walk through the interpretation of the graph for the Mediation condition to illustrate.The Mediation condition is shown graphically below.<figure class="image image_resized" style="width:44.27%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dulug83wylzxdkt4yasg" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/lmqqk42gktsnirsjasqr 140w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/gpqqupgo0h9pnbdytnkr 280w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/r2ta0vd63o7dldncaaqn 420w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/vnurlpx2xgwtrucc2bzl 560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/vza6p5ozdzh45dra5oet 700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/i5lbwoy7s66svc7uxx9r 840w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/p0cdehnxmrfdoolr0q8u 980w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/io8slpoy8a4f2rjibmhu 1120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/gnwhgmyugbmyd8u8ltrd 1260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/knx642j7cjqfyubztccn 1312w"><figcaption>The Mediation condition under the ith model, graphically. A distribution P[X1,X2,Λi] "satisfies" this graph if and only if it satisfies the factorization P[X1,X2,Λi]=P[Λi]∏jP[Xj|Λi].</figcaption></figure>The graph is interpreted as an equation stating that the distribution over the variables factors according to the graph - in this case, P[X,Λi]=P[Λi]∏jP[Xj|Λi]. Any distribution which factors this way "satisfies" the graph. Note that the graph does not assert that the factorization is minimal; for example, a distribution P[X1,X2,Λi] under which all Xi and Λi are independent - i.e. P[X1,X2,Λi]=P[Λi]∏jP[Xj] - satisfies all graphs over the variables X1,X2 and Λi, including the graph in the figure above.Besides allowing for compact presentation of equations and proofs, the graphical notation also makes it easy to extend our results to the approximate case. When the graph is interpreted as an approximation, we write it with an approximation error ϵ underneath, as in the figure below.<figure class="image image_resized" style="width:46.51%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dhxvkpqymxpgnxqnhlh7" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ecgnvkvmdf1nyms2f3xw 140w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ge9refgmmrob8vx8c3im 280w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ohvevdz4mbnj65iy1nqh 420w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dyl1dc4tp69hpsirk6il 560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/oubbwo8molwtge5pojej 700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/cnfnchemakadhjezf3hk 840w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/p4yinqpskk7iumfwre1n 980w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dda6pucsnscqjgutuw3k 1120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wtbk9xlail2dxv6viate 1260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/bmwdowzpztb0ty7jfkoa 1338w"><figcaption>The Mediation condition under the ith model, graphically with approximation ϵ. A distribution P[X1,X2,Λi] "satisfies" this graph (including the approximation) if and only if it satisfies ϵ≥DKL(P[Λi,X]||P[Λi]∏jP[Xj|Λi]).</figcaption></figure> In general, we say that a distribution P[Y1,...,Yn] "satisfies" a graph over variables Y1,...,Yn to within approximation error ϵ if and only if ϵ≥DKL(P[Y1,...,Yn]||∏jP[Yj|Ypa(j)]), where DKL is the KL divergence. We will usually avoid writing out these inequalities explicitly.We'll also use a slightly unusual notation to indicate that one variable is a deterministic function of another: Y←X→Y. This diagram says that X mediates between Y and Y, which is only possible if Y is fully determined by X. Approximation to within ϵ works just like approximation for other diagrams, and turns out to reduce to ϵ≥H(Y|X):ϵ≥DKL(P[X=x,Y=y,Y=y′]||P[X=x]P[Y=y|X=x]P[Y=y′|X=x])     =∑x,y,y′P[X=x,Y=y]I[y=y′](log(P[X=x,Y=y]I[y=y′])−log(P[X=x]P[Y=y|X=x]P[Y=y′|X=x]))    =∑x,yP[X=x,Y=y](log(P[X=x,Y=y])−log(P[X=x]P[Y=y|X=x]P[Y=y|X=x])) =−∑x,yP[X=x,Y=y]log(P[Y=y|X=x]) =H(Y|X)<h2>Foundational Concepts: Mediation, Redundancy & Naturality</h2>Mediation and redundancy are the two main foundational conditions which we'll work with.Readers are hopefully already familiar with mediation. We say a latent Λ "mediates between" observables X1,...,Xn if and only if X1,...,Xn are independent given Λ. Intuitively, any information shared across two or more Xj's must "go through" Λ. We call such a Λ a mediator. Canonical example: if X1,...,Xn are many rolls of a die of unknown bias Λ, then the bias is a mediator, since the rolls are all independent given the bias. See figure above for the graphical representation of mediation.Redundancy is probably less familiar, especially the definition used here. We say a latent Λ′ is a "redund" over observables X1,...,Xn if and only if Λ′ is fully determined by each Xi individually, i.e. there exist functions fi such that Λ′=fi(Xi) for each i. In the approximate case, we weaken this condition to say that the entropy H(Λ′|Xi)≤ϵ for all i, for some approximation error ϵ.Intuitively, all information about Λ′ must be redundantly represented across all Xi's. Canonical example: if X1,...,Xn are pixel values in small patches of a picture of a bike, and Λ′ is the color of the bike, then Λ′ is a redund insofar as we can back out the bike's color from any one of the little patches.  In general, a latent Λ is approximately a redund over components of X if and only if  ∀i:ϵ≥H(Λ|Xi) ("Redundancy") where ϵ=0 in the exact case. See below for the graphical representation of the redundancy condition.<figure class="image image_resized" style="width:52.51%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/mhu2lchsk2mljpjexjdp" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/zpaybksrp0ry7pn8lb5v 100w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/n28m4fsmdqskun5rv38k 200w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/zv7rdmex7obk15w1x78i 300w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/yvxv4asknpw8cgwubg40 400w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/cfg7geeienw72fz0otky 500w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/jdcd0yu5a1gr5zr1fiyy 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ti9due8oxl7cjms71mxb 700w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/rrwshhlgl8wgvxsnuoos 800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/r92h33tsqfb78ns0psg3 900w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ygat3haxirzeykhle2xs 965w"><figcaption>The graphical definition of redundancy: Λ′ is a "redund" over X1,...,Xn if and only if P[X,Λ′] satisfies the graph for all j. Intuitively, it says that any one component of X is enough to fully determine Λ′.</figcaption></figure> We'll be particularly interested in cases where a single latent is both a mediator and a redund over X1,...,Xn. We call mediation and redundancy together the "naturality conditions", and we call a latent satisfying both mediation and redundancy a "natural latent". Canonical example: if X1,...,Xn are low level states of macroscopically separated chunks of a gas at thermal equilibrium, then the temperature is a natural latent over the chunks, since each chunk has the same temperature (thus redundancy) and the chunks' low-level states are independent given that temperature (thus mediation).Justification of the name "natural latent" is the central purpose of this paper: roughly speaking, we wish to show that natural latents guarantee translatability, and that (absent further constraints) they are the only latents which guarantee translatability.<h2>Core Theorems</h2>We'll now present our core theorems. The next section will explain how these theorems apply to our motivating problem of translatability of latents across agents; readers more interested in applications and concepts than derivations should skip to the next section. We will state these theorems for generic latents Λ and Λ′, which we will tie back to our two agents Alice and Bob later.<details class="detailsBlock"><summary class="detailsBlockTitle">Theorem: Mediator Determines Redund</summary><div class="detailsBlockContent">Suppose that random variables X1,...,Xn, Λ, and Λ′ satisfy two conditions:<ul><li>Λ Mediation: X1,...,Xn are independent given Λ</li><li>Λ′ Redundancy: Λ′←Xj→Λ′ for all j</li></ul>Then Λ′←Λ→Λ′.</div></details>In English: if one latent Λ mediates between the components of X, and another latent Λ′ is a redund over the components of X, then Λ′ is fully determined by Λ (or approximately fully determined, in the approximate case).The graphical statement of the Mediator Determines Redund Theorem is shown below, including approximation errors. The proof is given in the Appendix.<figure class="image image_resized" style="width:95.43%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/larfcixrybcgkgwi0lhr" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/nml3yulhzw9jphqeg2io 160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/gcatmfpz9ilethoqscti 320w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ly7u5oczprrejd5x1fob 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/v6jtbce4jz3rj6gi3nsk 640w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/os3ks7vfvvhvqs1wy2bx 800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/z8bo8m7r2yupi8n1mjz8 960w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/uodjkdyrxotojpqauem2 1120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/txt075ilhad1f82xkwac 1280w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ezoytjgly3yy96odacra 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/s3omzpsqpyo6djospdyx 1574w"><figcaption>Graphical statement of Theorem 1</figcaption></figure>The intuition behind the theorem is easiest to see when X has two components, X1 and X2. The mediation condition says that the only way information can "move between'' X1 and X2 is by "going through'' Λ. The redundancy conditions say that X1 and X2 must each alone be enough to determine Λ′, so intuitively, that information about Λ′ must have "gone through'' Λ - i.e. Λ must also be enough to determine Λ′. Thus, Λ′←Λ→Λ′; all the redund's information must flow through the mediator, so the mediator determines the redund.<h3>Naturality ⟹ Minimality Among Mediators</h3>We're now ready for the corollaries which we'll apply to translatability in the next section.Suppose a latent Λ is natural over X1,...,Xn - i.e. it satisfies both the mediation and redundancy conditions. Well, Λ is a redund, so by the Mediator Determines Redund Theorem, we can take any other mediator Λ′′ and find that Λ←Λ′′→Λ. So: Λ is a mediator, and any other mediator is enough to determine Λ. So Λ is the "minimal" mediator: any other mediator must contain at least all the information which Λ contains. We sometimes informally call such a latent a "minimal latent".<figure class="image image_resized" style="width:89.52%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/uwe2rtjvtjbjvhvcqm8d" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/oiqrngxjwlurmn5rredo 160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/yova3tac4pxafkergolr 320w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ulde6sqroq2chlrtgrf8 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/b4o3un9yeptasly3ludd 640w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/fulxex7qp1ntf0czvhez 800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wdej2hte4mmr8ez6ov58 960w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/gcaocnfx0hzg4talvsrq 1120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wvdmphraswecinyj9epx 1280w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/e0eyoex75dcsh7mcjdcl 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/b6pmoh5t5r3xoygoisly 1535w"><figcaption>If Λ is a natural latent, it satisfies the redundancy condition. So given any other latent Λ′′ which satisfies the mediation condition we have the three conditions necessary to apply the Mediator Determines Redund Theorem, i.e. Λ′′ determines Λ.</figcaption></figure><h3>Naturality ⟹ Maximality Among Redunds</h3>There is also a simple dual to "Naturality ⟹ Minimality Among Mediators''. While the minimal latent conditions describe a smallest latent which mediates between X1,...,Xn, the dual conditions describe a largest latent which is redundant across X1,...,Xn. We sometimes informally call such a latent a "maximal latent".<figure class="image image_resized" style="width:93.53%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/oefxmeqebhqxawlifsin" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/rpkyg2jdedfkey2beuhn 180w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/g1paelvorsghocozztn8 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/yre8oxceghuwsknuza6w 540w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dllpcvokmaqzctqba96x 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/iwhmhdytg8enabqmzxfg 900w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/vitwxpcfjh7yrblk6a5c 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/qg1nrlk57ctnxc0lwapz 1260w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/fkjvojcy5qqttr58qfn9 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/lz3hmzsfnsgratd8hyrh 1620w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ny0onenwgc4gveqqvjmb 1712w"><figcaption>If Λ is a natural latent, it satisfies the mediation condition. So given any other latent Λ′ which satisfies the redundancy condition we have the three conditions necessary to apply Theorem 1, i.e. Λ determines Λ′.</figcaption></figure><h3>Isomorphism of Natural Latents</h3>If two latents Λ, Λ′ are both natural latents, then from the Mediator Determines Redund Theorem we trivially have both Λ′←Λ→Λ′ and Λ←Λ′→Λ. In English: the two latents are isomorphic.In the approximate case, each latent has bounded entropy given the other; in that sense they are approximately isomorphic.<h1>Application To Translatability</h1><h2>Motivating Question</h2>Our main motivating question is: under what conditions on Alice' model MA and its latent(s) ΛA can Alice guarantee that ΛA is a function of ΛB (i.e. ΛA←ΛB→ΛA), for any model MB and latent(s) ΛB which Bob might have?Recall that we already have some restrictions on Bob's model and latent(s): Agreement on Observables says P[X|MB]=P[X|MA], and Mediation says that X1,X2 are independent given ΛB under model MB.Since Naturality ⟹ Minimality Among Mediators, the natural latent conditions seem like a good fit here. If Alice' latent ΛA satisfies the natural latent conditions, then Minimality Among Mediators says that for any latent Λ′′ satisfying mediation over X1,X2, ΛA←Λ′′→ΛA. And Bob's latent ΛB satisfies mediation, so we can take Λ′′=ΛB to get the result we want, trivially.<h2>Guaranteed Translatability</h2>If Alice' latent ΛA is natural, then it's a function of Bob's latent ΛB, i.e. ΛA←ΛB→ΛA. This is just the Naturality ⟹ Minimality Among Mediators theorem from earlier.Now it's time for the other half of our main theorem: the naturality conditions are the only way for Alice to achieve this guarantee. In other words, we want to show: if Alice' latent ΛA satisfies Mediation, and for any latent ΛB Bob could choose (i.e. any other mediator) we have ΛA←ΛB→ΛA, then Alice' latent must be natural.The key to the proof is then to notice that X1 trivially mediates between X1 and X2, and X2 also trivially mediates between X1 and X2. So, Bob could choose ΛB=X1, or ΛB=X2 (among many other options). In order to achieve her guarantee, Alice' latent ΛA must therefore satisfy ΛA←X1→ΛA and ΛA←X2→ΛA - i.e. redundancy over X1 and X2. Alice' latent already had to satisfy the mediation condition by assumption, it must also satisfy the redundancy condition in order to achieve the desired guarantee, therefore it must be a natural latent. And if we weaken the conditions to allow approximation, then Alice' latent must be an approximate natural latent.<figure class="image image_resized" style="width:98.24%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/be0krvolaywubuqg9tb3" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/zxpfdnch8b60whplkf3w 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wvdjjl9prmvcwchzqwwr 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/ey8yabbteywwnoaoyrjg 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/u5yqiuh8pqmbcxaalzzx 1440w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/lmjtfeownv3sk0rii1y3 1800w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/nxwsdaravckzbniwvhet 2160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/bpausiszraw3cruwuleg 2520w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wpntyc1qgrkau9pxove8 2880w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/egv63u0vymfiumgzqwlo 3240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/yvtaqzwwi2jcujuatzsv 3513w"><figcaption>Graphical statement of Theorem 2</figcaption></figure>In English, the assumptions required for the theorem are:<ul><li>The Mediation conditions: Some pair of observables X1,X2 are independent given ΛA under model MA, and same two observables are independent given ΛB under model MB</li><li>The Agreement on Observables condition: P[X1,X2|MA]=P[X1,X2|MB]</li></ul>Under those constraints, Alice can guarantee that her latent ΛA is a function of Bob's latent ΛB (i.e. ΛA←ΛB→ΛA) if and only if Alice' latent is a natural latent over X1,X2, meaning that it satisfies both the mediation condition (already assumed) and the redundancy condition ΛA←Xj→ΛA for all j.Proof: the "if" part is just Naturality ⟹ Minimality Among Mediators; the "only if" part follows trivially from considering either ΛB=X1 or ΛB=X2 (both of which are always allowed choices of ΛB).<h1>Natural Latents: Intuition & Examples</h1>Having motivated the natural latent conditions as exactly those conditions which guarantee translatability, we move on to building some intuition for what natural latents look like and when they exist.<h2>When Do Natural Latents Exist? Some Intuition From The Exact Case</h2>For a given distribution P[X1,...,Xn], a natural latent over X does not always exist, whether exact or approximate to within some error ϵ. In practice, the cases where interesting natural latents do exist usually involve approximate natural latents (as opposed to exact), and we'll see some examples in the next section. But first, we'll look at the exact case, in order to build some intuition.Let's suppose that there are just two observables X1,X2. If Λ is natural over those two observables, then the redundancy conditions say Λ=f1(X1)=f2(X2) for some functions f1,f2. That means f1(X1)=f2(X2) with probability 1 This is a deterministic constraint between X1 and X2.Next, the mediation condition. The mediation condition says that X1 and X2 are independent given Λ - i.e. they're independent given the value of the deterministic constraint. So: assuming existence of a natural latent Λ, X1 and X2 must be independent given the value of a deterministic constraint.On the other hand, if X1 and X2 are independent given the value of a deterministic constraint, then the value of the constraint clearly satisfies the natural latent conditions.That gives us an intuitive characterization of the existence conditions for exact natural latents: an exact natural latent between two (or more) variables exists if-and-only-if the variables are independent given the value of a deterministic constraint across those variables.<h2>Worked Quantitative Example of the Mediator Determines Redund Theorem</h2>Consider Carol who is about to flip a biased coin she models as having some bias Λ. Carol flips the coin 1000 times, computes the median of the flips, then flips the coin another 1000 times and computes the median of that batch. For simplicity, we assume a uniform prior on Λ over the interval [0,1].Intuitively, if the bias Λ is unlikely to be very close to12, Carol will find the same median both times with high probability. Let X1 and X2 denote Carol's first and second batches of 1000 flips, respectively. Note that the flips are independent given Λ, under her model, satisfying the mediation condition of the Mediator Determines Redund Theorem, exactly. Let Λ′ be the median computed from either of the batches. Since the same median can be computed with high probability from eitherX1 or X2, the redundancy condition is approximately satisfied.The Mediator Determines Redund Theorem, then tells us that the bias approximately mediates between the median (computed from either batch) and the coinflips X. To quantify the approximation, we first quantify the approximation on the redundancy condition (the other two conditions hold exactly, so their ϵ's are 0). Taking Λ′ to be Carol's calculation of the median from the first batch, X1, Carol's median can be exactly determined from those flips (i.e., Λ′←X1→Λ′), but Carol's median of the first batch can be determined from the second batch of flips (i.e., Λ′←X2→Λ′) only approximately. The approximation error is E[H(Λ′(X1)|X2)].This is a Dirichlet-multinomial distribution, so it is cleaner to rewrite in terms of N1:=∑X1, N2:=∑X2, and n:=1000. Since Λ′ is a function of N1, the approximation error becomes H(Λ′(N1)|N2).Writing out the distribution and simplifying the gamma functions, we obtain:P[N2]=1n+1 (i.e., uniform over 0, ..., n) P[N1|N2]=Γ(n+2)Γ(N2+1)Γ(n−N2+1)Γ(n+1)Γ(N1+1)Γ(n−N1+1)Γ(N1+N2+1)Γ(2n−N1−N2+1)Γ(2n+2) P[Λ′(N1)=0|N2]=∑n1<500P[N1|N2] 500} P[N_1|N_2]">P[Λ′(N1)=1|N2]=∑n1>500P[N1|N2]There are only 10012 values of (N1,N2), so these expressions can be combined and evaluated using a Python script (see Appendix for code). The script yields H=0.058 bits. As a sanity check, the main contribution to the entropy should be when Λ is near 0.5, in which case the median should have roughly 1 bit of entropy. With n data points, the posterior uncertainty should be of order 1√n, so the estimate of Λ should be precise to roughly 130≈.03 in either direction. Since Λ is initially uniform on [0,1], a distance of 0.03 in either direction around 0.5 covers about 0.06 in prior probability, and the entropy should be roughly 0.06 bits, which is consistent with the computed value.Returning to the Mediator Determines Redund Theorem, we have ϵmed=0 and ϵred≈0.058 bits. Thus, the theorem states that Carol's median is approximately determined by the coin's bias, Λ, to within ϵmed+2ϵred≈0.12 bits of entropy.Exercise for the Reader: By separately tracking the ϵ's on the two redundancy conditions through the proof, show that, for this example, the coin's true bias approximately mediates between the coinflips and Carol's median to within ϵred, i.e., roughly 0.058 bits.<h2>Intuitive Examples of Natural Latents</h2>This section will contain no formalism, but will instead walk through a few examples in which one would intuitively expect to find a nontrivial natural latent, in order to help build some intuition for the reader. The When Do Natural Latents Exist? section provides the foundations for the intuitions of this section.<h3>Ideal Gas</h3>Consider an equilibrium ideal gas in a fixed container, through a Bayesian lens. Prior to observing the gas, we might have some uncertainty over temperature. But we can obtain a very precise estimate of the temperature by measuring any one mesoscopic chunk of the gas. That's an approximate deterministic constraint between the low-level states of all the mesoscopic chunks of the gas: with probability close to 1, they approximately all yield approximately the same temperature estimate.Due to chaos, we also expect that the low-level state of mesoscopic chunks which are not too close together spatially are approximately independent given the temperature.So, we have a system in which the low-level states of lots of different mesoscopic chunks are approximately independent given the value of an approximate deterministic constraint (temperature) between them. Intuitively, those are the conditions under which we expect to find a nontrivial natural latent. In this case, we expect the natural latent to be approximately (isomorphic to) temperature.<h3>Biased Die</h3>Consider 1000 rolls of a die of unknown bias. Any 999 of the rolls will yield approximately the same estimate of the bias. That's (approximately) the redundancy condition for the bias.We also expect that the 1000 rolls are independent given the bias. That's the mediation condition. So, we expect the bias is an approximate natural latent over the rolls.However, the approximation error bound in this case is quite poor, since our proven error bound scales with the number n of observables. We can easily do better by viewing the first 500 and second 500 rolls as two observables. We expect that the first 500 rolls and the second 500 rolls will yield approximately the same estimate of the bias, and that the first 500 and second 500 rolls are independent given the bias, so the bias is a natural latent between the first and second 500 rolls of the die. This view of the problem will likely yield much better error bounds. More generally, chunking together many observables this way typically provides much better error bounds than applying the theorems directly to many observables.<h3>Timescale Separation In A Markov Chain</h3>In a Markov Chain, timescale separation occurs when there is some timescale T such that, if the chain is run for T steps, then the state can be split into a component which is almost-certainly conserved over the T steps and a component which is approximately ergodic over the T steps. In that case, we expect both the initial state and Tth state to almost-certainly yield the same estimate of the conserved component, and we expect that the initial state and Tth state are approximately independent given the conserved component, so the conserved component should be an approximate natural latent between the initial and Tth state.<h1>Discussion & Conclusion</h1>We began by asking when one agent's latent can be guaranteed to be expressible in terms of another agent's latent(s), given that the two agree on predictions about two observables. We've shown that:<ul><li>The natural latent conditions are necessary for such a guarantee.</li><li>The natural latent conditions are sufficient for such a guarantee.</li><li>Both of the above are robust to approximation.</li></ul>...for a specific broad class of possibilities for the other agent's latent(s). In particular, the other agent can use any latent(s) which fully explain the interactions between the two observables. So long as the other agent's latent(s) are in that class, the first agent can guarantee that their latent can be expressed in terms of the second's exactly when the natural latent conditions are satisfied.These results provide a potentially powerful tool for many of the questions posed at the beginning.When is robust translation possible at all, between agents with potentially different internal concepts, like e.g. humans and AI, or humans from different cultures? Insofar as the agents make the same predictions about two parts of the world, and both their latent concepts induce independence between those parts of the world (including approximately), either agent can ensure robust translatability into the other agent's ontology by using a natural latent. In particular, if the agents are trying to communicate, they can look for parts of the world over which natural latents exist, and use words to denote those natural latents; the equivalence of natural latents will ensure translatability in principle, though the agents still need to do the hard work of figuring out which words refer to natural latents over which parts of the world.Under what conditions are scientific concepts guaranteed to carry over to the ontologies of new theories, like how e.g. general relativity has to reduce to Newtonian gravity in the appropriate limit? Insofar as the old theory correctly predicted two parts of the world, and the new theory introduces latents to explain all the interactions between those parts of the world, the old theorist can guarantee forward-compatibility by working with natural latents over the relevant parts of the world. This allows scientists a potential way to check that their work is likely to carry forward into as-yet-unknown future paradigms.When and why might a wide variety of minds in the same environment converge to use (approximately) the same concept internally? While this question wasn't the main focus of this paper, both the minimality and maximality conditions suggest that natural latents (when they exist) will often be convergently used by a variety of optimized systems. For minimality: the natural latent is the minimal variable which mediates between observables, so we should intuitively expect that systems which need to predict some observables from others and are bandwidth-limited somewhere in that process will often tend to represent natural latents as intermediates. For maximality: the natural latent is the maximal variable which is redundantly represented, so we should intuitively expect that systems which need to reason in ways robust to individual inputs will often tend to track natural latents.The natural latent conditions are a first step toward all these threads. Most importantly, they offer any mathematical foothold at all on such conceptually-fraught problems. We hope that foothold will both provide a foundation for others to build upon in tackling such challenges both theoretically and empirically, and inspire others to find their own footholds, having seen that it can be done at all.<h2>Acknowledgements</h2>We thank the Long Term Future Fund and the Survival and Flourishing Fund for funding this work.<h1>Appendices</h1><h2>Graphical Notation and Some Rules for Graphical Proofs</h2>In this paper, we use the diagrammatic notation of Bayes networks (Bayes nets) to concisely state properties of probability distributions. Unlike the typical use of Bayes nets, where the diagrams are used to define a distribution, we assume that the joint distribution is given and use the diagrams to express properties of the distribution. Specifically, we say that a distribution P[Y] "satisfies" a Bayes net diagram if and only if the distribution factorizes according to the diagram's structure. In the case of approximation, we say that P[Y] "approximately satisfies" the diagram, up to some ϵ≥0, if and only if the Kullback-Leibler divergence (DKL) between the true distribution and the distribution implied by the diagram is less than or equal to ϵ.<h3> </h3><h3>Frankenstein Rule</h3>StatementLet P[X1,…,Xn] be a probability distribution that satisfies two different Bayes networks, represented by directed acyclic graphs G1 and G2. If there exists an ordering of the variables X1,…,Xn that respects the topological order of both G1 and G2 simultaneously, then P[X1,…,Xn] also satisfies any "Frankenstein" Bayesian network constructed by taking the incoming edges of each variable Xi from either G1 or G2. More generally, if P[X1,…,Xn] satisfies m different Bayes networks G1,…,Gm, and there exists an ordering of the variables that respects the topological order of all m networks simultaneously, then P[X1,…,Xn] satisfies any "Frankenstein" Bayes network constructed by taking the incoming edges of each variable Xi from any of the m original networks.We'll prove the approximate version, then the exact version follows trivially.ProofWithout loss of generality, assume the order of variables respected by all original diagrams is X1,…,Xn. Let P[X]=∏iP[Xi|Xpaj(i)] be the factorization expressed by diagram j, and let σ(i) be the diagram from which the parents of Xi are taken to form the Frankenstein diagram. (The factorization expressed by the Frankenstein diagram is then P[X]=∏iP[Xi|Xpaσ(i)(i)].)The proof starts by applying the chain rule to the DKL of the Frankenstein diagram: DKL(P[X]||∏iP[Xi|Xpaσ(i)(i)])=DKL(∏iP[Xi|X<i]||∏iP[Xi|Xpaσ(i)(i)]) = ∑iE[DKL(P[Xi|X<i]||P[Xi|Xpaσ(i)(i)])]Then, we add a few more expected KL-divergences (i.e., add some non-negative numbers) to get: ≤∑i∑jE[DKL(P[Xi|X<i]||P[Xi|Xpaj(i)])] = ∑jDKL(P[X]||∏iP[Xi|Xpaj(i)]) ≤∑jϵjThus, we have DKL(P[X]||∏iP[Xi|Xpaσ(i)(i)])≤∑jDKL(P[X]||∏iP[Xi|Xpaj(i)]) ≤∑jϵj<h3> </h3><h3>Factorization Transfer</h3>StatementLet P[X1,…,Xn] and Q[X1,…,Xn] be two probability distributions over the same set of variables. If Q satisfies a given factorization (represented by a diagram) and Q approximates P with an error of at most ϵ, i.e., ϵ≥DKL(P||Q) then P also approximately satisfies the same factorization, with an error of at most ϵ: ϵ≥DKL(P[X1,…,Xn]||∏iP[Xi|Xpa(i)]) where Xpa(i) denotes the parents of Xi in the diagram representing the factorization.ProofAs with the Frankenstein rule, we start by splitting our DKL into a term for each variable: DKL(P[X]||Q[X])=∑iE[DKL(P[Xi|X<i]||Q[Xi|Xpa(i)])] Next, we subtract some more DKL's (i.e., subtract some non-negative numbers) to get: ≥∑i(E[DKL(P[Xi|X<i]||Q[Xi|Xpa(i)])]−E[DKL(P[Xi|Xpa(i)]||Q[Xi|Xpa(i)])]) =∑iE[DKL(P[Xi|X<i]||P[Xi|Xpa(i)])] =DKL(P[X]||∏iP[Xi|Xpa(i)])Thus, we have DKL(P[X]||Q[X])≥DKL(P[X]||∏iP[Xi|Xpa(i)])<h3> </h3><h3>Bookkeeping Rule</h3>StatementIf all distributions which exactly factor over Bayes net G1 also exactly factor over Bayes net G2, then: DKL(P[X]||∏iQ[Xi∣XpaG2(i)])≥DKL(P[X]||∏iP[Xi∣XpaG2(i)])ProofLet Q[X]:=∏iP[Xi∣XpaG1(i)]. By definition, Q factors over G1. Since all distributions which factor over G1 also factor over G2, it follows that Q also factors over G2.Now, we have: Q[X]=∏iQ(Xi∣XpaG2(i))Thus: DKL(P[X]||∏iP(Xi∣XpaG1(i)))=DKL(P[X]||∏iQ(Xi∣XpaG2(i)))By the Factorization Transfer Theorem, we have: DKL(P[X]||∏iQ(Xi∣XpaG2(i)))≥DKL(P[X]||∏iP(Xi∣XpaG2(i)))which completes the proof. <h3>The Dangly Bit Lemma</h3>StatementIf Y←X→Y holds to within ϵ bits, and any other diagram D involving X holds to within ϵ′ bits, then we can create a new diagram D′ which is identical to D but has another copy of Y (the "dangly bit'') as a child of X. The new diagram D′ will hold to within ϵ+ϵ′ bits.<figure class="image image_resized" style="width:89.8%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/mqyou0ytgpjfjwicwkvj" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/bwar7ljkdrohpljbec2f 120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/imjt8kazn0bbr3ertlbm 240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/mm81lcxdwi94yqka6dbr 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/omtja068ul71e5gmm0av 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/mdy5n1q54mmea6d69myx 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/igxpelyuzuwroqeoo8u5 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/avfjcrxzuq6m18fdwsor 840w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/nbcijdsoppsbpvk7hria 960w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/wxko3dhx6yczf9euymst 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/oh8fnfftesfmv6jngs4r 1111w"><figcaption>Statement of the D.B. Lemma</figcaption></figure>ProofLet Q[X,Z] be the distribution over X and other variables Z specified by D (with Z possibly containing copies of Y). Then D′ specifies the distribution Q[X,Z]P[Y∣X], so the approximation error for D′ is: DKL(P[X,Y,Z]∥Q[X,Z]P[Y∣X]) =DKL(P[X,Z]∥Q[X,Z])+EX,Z[DKL(P[Y∣X,Z]∥P[Y∣X])] =DKL(P[X,Z]∥Q[X,Z])+I(Y;Z∣X) ≤DKL(P[X,Z]∥Q[X,Z])+H(Y∣X) ≤ϵ′+ϵ.<h3> </h3><h3>Graphical Proofs</h3><figure class="image image_resized" style="width:96.17%"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/dg5vocivanccsw3b8ol0" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/e2udfapc3v6ehnbedaez 120w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/pqbjww8v5mmbyb5f8gvl 240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/gbevlxvf98swowucvh2g 360w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/uv8dumqkd6pl2hlfrf3b 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/lvyac5sdyqke3l0bkm7s 600w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/pwpzn9yufszb6xgzol13 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/grhibrsubskyaqieop3t 840w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/u0issh7saodoypdvf8jv 960w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/s6ihdxus38nyxbrgpws7 1080w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Qdgo2jYAuFRMeMRJT/lvksoyebc0mxjig0yqxf 1166w"><figcaption>Proof of the Mediator Determines Redund Theorem</figcaption></figure><h3> </h3><h3>Python Script for Computing DKL in Worked Example</h3><pre><code>import numpy as np
from scipy.special import gammaln, logsumexp, xlogy
n = 1000
p_N2 = np.ones(n+1)/(n+1)
N1 = np.outer(np.arange(n + 1), np.ones(n + 1))
N2 = np.outer(np.ones(n + 1), np.arange(n + 1))
# logP[N1|N2]; we're tracking log probs for numerical stability
lp_N1_N2 = (gammaln(n + 2) - gammaln(N2 + 1) - gammaln(n - N2 + 1) +
gammaln(n + 1) - gammaln(N1 + 1) - gammaln(n - N1 + 1) +
gammaln(N1 + N2 + 1) + gammaln(2*n - N1 - N2 + 1) - gammaln(2*n + 2))
# logP[\Lambda' = 0|N2] and logP[\Lambda' = 1|N2]
lp_lam0_N2 = logsumexp(lp_N1_N2[:500], axis=0)
lp_lam1_N2 = logsumexp(lp_N1_N2[500:], axis=0)
p_lam0_N2 = np.exp(lp_lam0_N2)
p_lam1_N2 = np.exp(lp_lam1_N2)
print(p_lam0_N2 + p_lam1_N2) # Check: these should all be 1.0
# ... aaaand then it's just the ol' -p * logp to get the expected entropy E[H(\Lambda')|N2]
H = - np.sum(p_lam0_N2 * lp_lam0_N2 * p_N2) - np.sum(p_lam1_N2 * lp_lam1_N2 * p_N2)
print(H / np.log(2)) # Convert to bits</code></pre><ol class="footnote-section footnotes" data-footnote-section="" role="doc-endnotes"><li class="footnote-item" data-footnote-item="" data-footnote-index="1" data-footnote-id="6pf3mmzjza5" role="doc-endnote" id="fn6pf3mmzjza5"><a href="#fnref6pf3mmzjza5">^</a><div class="footnote-content" data-footnote-content="">W. V. O. Quine, On empirically equivalent systems of the world, Erkenntnis 9, 313 (1975).</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="2" data-footnote-id="p4bzaufgaob" role="doc-endnote" id="fnp4bzaufgaob"><a href="#fnrefp4bzaufgaob">^</a><div class="footnote-content" data-footnote-content="">H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey, Sparse autoencoders find highly interpretable features in language models, (2023), arXiv:2309.08600 [cs.LG].</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="3" data-footnote-id="d937dkdf6d" role="doc-endnote" id="fnd937dkdf6d"><a href="#fnrefd937dkdf6d">^</a><div class="footnote-content" data-footnote-content="">Marvik, Model merging: Combining different fine-tuned LLMs, Blog post (2024), retrieved from https://marvik.com/ model-merging.</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="4" data-footnote-id="gr9l6s063y" role="doc-endnote" id="fngr9l6s063y"><a href="#fnrefgr9l6s063y">^</a><div class="footnote-content" data-footnote-content="">M. Huh, B. Cheung, T. Wang, and P. Isola, The platonic representation hypothesis, (2024), arXiv:2405.07987 [cs.LG].</div></li><li class="footnote-item" data-footnote-item="" data-footnote-index="5" data-footnote-id="89wpwwprup9" role="doc-endnote" id="fn89wpwwprup9"><a href="#fnref89wpwwprup9">^</a><div class="footnote-content" data-footnote-content="">J. Pearl, Causality: Models, Reasoning and Inference, 2nd ed. (Cambridge University Press, USA, 2009).</div></li></ol> <a href="https://www.lesswrong.com/posts/Qdgo2jYAuFRMeMRJT/natural-latents-latent-variables-stable-across-ontologies#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/Qdgo2jYAuFRMeMRJT/natural-latents-latent-variables-stable-across-ontologies</link><guid isPermaLink="false">Qdgo2jYAuFRMeMRJT</guid><dc:creator><![CDATA[johnswentworth]]></dc:creator><pubDate>Thu, 04 Sep 2025 00:33:14 GMT</pubDate></item><item><title><![CDATA[The Missing Error Bars in AI Research That Nobody Talks About.]]></title><description><![CDATA[Published on September 3, 2025 11:10 PM GMT Andrey Seryakov  ex-CERN particle physicist independent AI behaviour researcher <a href="mailto:a.u.seryakov@gmail.com">a.u.seryakov@gmail.com</a>This article is about systematic uncertainties and biases which are so often missing in our research papers.  TL;DR<ul><li>Temperature changes not just variance but actual behavior patterns</li><li>Systematic uncertainty is about hidden biases in experimental design</li><li>No universal solution, but rigorous testing of assumptions is crucial</li></ul>This small blog should be considered as an extension of a paper “<a href=" https://arxiv.org/pdf/2411.00640">Adding Error Bars to Evals”</a> by Evan Miller. Where he extensively covers a problem and a need of statistical uncertainty in AI evaluations using a very rigorous math foundation. However, while Miller addresses how to measure statistical “error bars”, I'm focusing on the orthogonal problem: biases that change your entire measurement.Contrary to statistical uncertainties evaluation of systematical errors in physics we call “an art”, as there is no math foundation at all. It is always about knowing well your experiment design and biases which it introduces to your final results. The systematic is about biases. But first, I want to talk a bit about the temperature case, as I believe it’s very illustrative, and only afterwards move to systematic in general. <h3>Can and should we take temperature into account? </h3>You wouldn't study how fair a coin is by flipping it just once, yet that's what we do with LLMs at t=0. LLMs have a probabilistic behaviour by construction. If we want to study it we have to gain statistics, but which temperature we have to use? Widely used t = 0 will give you nothing. Before actually experimenting with it I was thinking about taking two values (for example 0.7 and 1) and plotting two different error bars, as we do in physics with statistical and systematical uncertainties, like a+/-st+/-sy. We often consider temperature as just a noise level and I was too but then I found that when I change it not just variance of behaviour patterns changes significantly but patterns themselves. I don’t know how to properly explain this, let me illustrate with an example. Last week I was studying if models betting strategies in poker depend on the opponent's name. I set up a specific situation and ask the model to choose an action - Fold, Call, Raise Xbb, where Xbb is the number of big blinds or, in other words, how many times its bet is higher than the opponent’s one. If you are familiar with poker, here is more information about setup. But if you don’t just skip it, it’s not really related to the uncertainty discussion. <ul><li>Tournament. Head-up. Both players have 100bb. </li><li>Model is on small blind </li><li>Hand: Qc 4d</li><li>Very first round, the only information about the opponent is its name. </li></ul>I used LLama 3.1 8b (I have no money for research -> it’s cheap and fast)For every name I ran it 400 times and 1000 times for the baseline when name isn’t provided<blockquote>You are now playing heads-up against {name} Smith  vs  You are now playing heads-up.</blockquote>These are results for t = 0.7 and 1 for 5 male and 5 female most popular kids names in the US (2024): <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/uycxvd7iu9rgpfgveqli" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/ap2yhv6nckc39hxv5emv 290w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/cvjb1heskxgaz3tnoqc1 580w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/j5okervyzm4qud0xooir 870w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/cfjtub3xgzybo8habdyx 1160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/k5e24z0cykyrodydegyg 1450w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/yad9gwiwasgynmubnro4 1740w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/odr2sfztxxcdwglcrb85 2030w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/tlbf4zo3l0st8m0qthxu 2320w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/qxtmwpbwgtgvvml0tcwu 2610w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/mtryln6urkgoj7ngeto0 2881w"><figcaption>First column is results of temperature 0.7, second - 1. First line is probability to raise against a player with a given name, baselines represent a case when name wasn't provided (see above). Second line of plots is statistical significance of a difference between cases with a given name and without it. </figcaption></figure>The thing that the mean changes it’s okay, this study is about specific name’s biases, not how models play. You would expect having more "extreme" betting strategies, when you increase the temperature. The thing which worries me is that the result changes. The biases are now directed to different names and even to different directions. The main conclusion stays - the opponent’s name (and I guess user’s name in general) systematically changes how some models (at least this one and LLama Scout) behave, but the specific set of names in which biases are observed is changing with temperature. And this makes everything much more complicated. So it’s clear, temperature has to be taken into account, but how to do it properly? Can we create a universal algorithm? This I don’t know. Let’s now move to the general discussion. <h3>What is systematic and how to evaluate it? </h3>As I said before, systematic is “an art”, it is about deep thinking about your experiment setup. You have to look for parameters, conditions which you believe should not affect your results but you use. You have to check that they don’t affect it indeed and if they do you have two choices - vary it, see how results change, add this change as a separate uncertainty to your points. Don’t ask me how exactly mathematically, it’s an art. You may take max and min of variations. You may have many such parameters. Assuming they independently influence results you can calculate a mean root squared and use the result as an additional uncertainty to systematic. The second possibility is to restrict your conclusions specifically writing that they are valid under the following conditions and if they change results change too.  A simpler example of systematic. Imagine you need to measure the volume of a given cube. How would you do it? Using a ruler. Statistical uncertainty you will get measuring the same side several times. What about systematic? There is one thing - the ruler! You do not expect that result would change if you change the rulers, but are you 100% sure? Many years ago when I was teaching experimental physics I went to a book store and bought two rulers where centimeters were of different lengths. Which of them is the right one, are you using the right one? So even such a simple tool may introduce a bias to your measure. <figure class="image"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/qhn1wwfpr1vissmilio1" srcset="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/qoztwv8mb5sex4zw3n3z 80w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/y6k9ey5ollufgtvxa6z1 160w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/wb1ifgm7uno6drxwrevp 240w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/yfzq7as73pia7ftu1ggf 320w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/oghivdbpbreg9ukuuci3 400w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/h8kev4odity7pqhjlycg 480w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/s0ahaymbfgy1g87dq35j 560w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/gm1okabbki8dy1vju2ir 640w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/acqibo8m71v8kbvxzjhf 720w, https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/AvF9H4JuH4pchFDok/s7lro1sp50f0q353qr9z 800w"><figcaption>Picture from internet, author is unknown</figcaption></figure>This logic can be implemented in the poker example above. The conclusion that LLama 3.1 8b plays more aggressively against Charlotte is valid only under t = 0.7, but not with other temperatures. But there are other sources of systematic biases which may exist there. They will not change the main conclusion (model is biased by the opponent name), but they may change any numerical estimations. Examples are: <ul><li>Precise prompt formulation</li><li>The tournament size </li><li>Cards</li><li>Player family name (I use Smith because it’s the most popular in the US)</li><li>and so on</li></ul>Imagine you are studying how LLMs are playing the hawks and doves game. Will your results change if you change hawks and doves to other animals, or people names, or call them strategy 1 and 2. If you have player 1 and player 2, what will happen if you exchange their names? Maybe you have an experiment where LLMs have to agree on some actions, what biases did you introduce in your prompts? Will their performance change if you provide them instructions in a different order? Or if the collective discussion process changes from them speaking in specific order versus random one? Any practical advice? There are no, each experiment is unique, you designed it, you thought about it more than anybody else. Based on my AI experience I would always: <ul><li>Vary instructions in the prompts including system prompts. Even such subtle differences as between “analyse this” and “ANALYSE THIS” may change everything or as shown above moving from Olivia to Mia. </li><li>Vary temperature </li><li>Document all "arbitrary" choices in the experiment setup. </li><li>Report, when conclusions are conditional to specific settings. </li></ul>Think about what may affect your conclusions, which assumptions you made, and check that they are valid. Gain statistics, don’t run your experiment just ones with 0 temperature. Think about reformulating your prompt, changing everything except the very core of it, change it several times. So this is an art and there is no universal solution, no magic pill. Such studies are crucial for making robust and reproducible research. Yes, this means running experiments takes 10x longer - but wrong conclusions are even more expensive. This is the way. <a href="https://www.lesswrong.com/posts/AvF9H4JuH4pchFDok/the-missing-error-bars-in-ai-research-that-nobody-talks#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/posts/AvF9H4JuH4pchFDok/the-missing-error-bars-in-ai-research-that-nobody-talks</link><guid isPermaLink="false">AvF9H4JuH4pchFDok</guid><dc:creator><![CDATA[Andrey Seryakov]]></dc:creator><pubDate>Wed, 03 Sep 2025 23:41:56 GMT</pubDate></item><item><title><![CDATA[Copenhagen – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:27 PM GMT This year's Spring ACX Meetup everywhere in Copenhagen.
Location: H. J. Holsts Vej 3-5C, 2605 Brøndby – <a href="https://plus.codes/https://plus.codes/9F7JMCCQ+4XR">https://plus.codes/9F7JMCCQ+4XR</a>
Group Link: https://www.lesswrong.com/events/JTEpLhhjAbK4jiuuJ/copenhagen-acx-risk-from-ai-community-conference
RSVP on LessWrong. This meetup in particular is AI X-Risk themed, but feel free to show up and discuss other subjects.
Contact: soeren.elverlin@gmail.com <a href="https://www.lesswrong.com/events/jxPqLZjjDm9jFna89/copenhagen-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/jxPqLZjjDm9jFna89/copenhagen-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">jxPqLZjjDm9jFna89</guid><dc:creator><![CDATA[SoerenE]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:27:07 GMT</pubDate></item><item><title><![CDATA[Houston – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:18 PM GMT This year's Spring ACX Meetup everywhere in Houston.
Location: Retrospect Coffee Bar 3709 La Branch St, Houston, TX 77004. We'll be in the back covered patio area with picnic tables. – <a href="https://plus.codes/https://plus.codes/76X6PJPF+4R">https://plus.codes/76X6PJPF+4R</a>
Group Link: https://discord.gg/DzmEPAscpS
Contact: joe.brenton@yahoo.com <a href="https://www.lesswrong.com/events/n8x32Li3Zuqix4Egc/houston-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/n8x32Li3Zuqix4Egc/houston-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">n8x32Li3Zuqix4Egc</guid><dc:creator><![CDATA[Joe Brenton]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:18:04 GMT</pubDate></item><item><title><![CDATA[Bonn – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:17 PM GMT This year's Spring ACX Meetup everywhere in Bonn.
Location: At Endenicher Allee 60, 53115 Bonn, we will be in the small building behind (northwest of) the main Math building. There will be a sign outside the building with directions to our room. – <a href="https://plus.codes/https://plus.codes/9F29P3HM+C7F">https://plus.codes/9F29P3HM+C7F</a>
Group Link: https://chat.whatsapp.com/C5Zp72kXjuIJIjuCvti5fm if this link doesn't work just send me an Email and I'll add you to the group
Contact: timtjc08@gmail.com <a href="https://www.lesswrong.com/events/rsXceqLE8ftEXvamd/bonn-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/rsXceqLE8ftEXvamd/bonn-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">rsXceqLE8ftEXvamd</guid><dc:creator><![CDATA[Tiuto]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:17:37 GMT</pubDate></item><item><title><![CDATA[Tel aviv – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:17 PM GMT This year's Spring ACX Meetup everywhere in Tel aviv.
Location: Sarona market, at grass area next to Benedict. I'll have an ACX sign – <a href="https://plus.codes/https://plus.codes/8G4P3QCP+MM">https://plus.codes/8G4P3QCP+MM</a>
Group Link: https://www.facebook.com/groups/5389163051129361/
Kids and dogs are welcome, feel free to bring snacks
Contact: inbar192@gmail.com <a href="https://www.lesswrong.com/events/jcyK4KucHFDdLnJzX/tel-aviv-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/jcyK4KucHFDdLnJzX/tel-aviv-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">jcyK4KucHFDdLnJzX</guid><dc:creator><![CDATA[MissMarble]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:17:20 GMT</pubDate></item><item><title><![CDATA[Austin – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:17 PM GMT This year's Spring ACX Meetup everywhere in Austin.
Location: The park by Central Market, 4001 North Lamar, Austin, Texas. We will be by the stone tables by the pond with tarps for shade. We will have a LessWrong and ACX sign and have some tents set up. You can also park in the parking lot at 3900 Guadalupe St (outside the fenced hospital area). – <a href="https://plus.codes/https://plus.codes/86248746+9C">https://plus.codes/86248746+9C</a>
Group Link: https://austinlesswrong.com/calendar/
Feel free to bring kids/dogs (though it's a bit of a walk to the indoor bathrooms). We'll provide breakfast tacos (including vegan) and other snacks and drinks. Feel free to bring more, but it's not expected.
Contact: sbarta@gmail.com <a href="https://www.lesswrong.com/events/KNoEWPRvxFZvcBPP8/austin-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/KNoEWPRvxFZvcBPP8/austin-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">KNoEWPRvxFZvcBPP8</guid><dc:creator><![CDATA[Andrew_Taskify]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:17:16 GMT</pubDate></item><item><title><![CDATA[Salt Lake City – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:16 PM GMT This year's Spring ACX Meetup everywhere in Salt Lake City.
Location: Liberty Park, near the ChargePoint station – <a href="https://plus.codes/https://plus.codes/85GCP4WF+MF">https://plus.codes/85GCP4WF+MF</a>
Group Link: there's a mailing list and Discord, please ask at the meetup! :)
Contact: adamisom@hey.com <a href="https://www.lesswrong.com/events/cbQi4wjMcnkyJSigq/salt-lake-city-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/cbQi4wjMcnkyJSigq/salt-lake-city-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">cbQi4wjMcnkyJSigq</guid><dc:creator><![CDATA[adamisom]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:16:59 GMT</pubDate></item><item><title><![CDATA[Bellevue – ACX Meetups Everywhere Fall 2025]]></title><description><![CDATA[Published on September 3, 2025 10:16 PM GMT This year's Spring ACX Meetup everywhere in Bellevue.
Location: with a page with "ACX" on it or one of its chairs. – <a href="https://plus.codes/https://plus.codes/84VVJRC4+35">https://plus.codes/84VVJRC4+35</a>
Group Link: https://www.meetup.com/seattle-rationality/; https://www.lesswrong.com/groups/PmvZmMxBtxE87PHZf
Contact: me@joeym.org <a href="https://www.lesswrong.com/events/iepdx4yimMaGmCkGu/bellevue-acx-meetups-everywhere-fall-2025#comments">Discuss</a>]]></description><link>https://www.lesswrong.com/events/iepdx4yimMaGmCkGu/bellevue-acx-meetups-everywhere-fall-2025</link><guid isPermaLink="false">iepdx4yimMaGmCkGu</guid><dc:creator><![CDATA[Joey]]></dc:creator><pubDate>Wed, 03 Sep 2025 22:16:14 GMT</pubDate></item></channel></rss>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid RSS" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//lesswrong.com/comments/.rss"><img src="valid-rss-rogers.png" alt="[Valid RSS]" title="Validate my RSS feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//lesswrong.com/comments/.rss

Home · About · News · Docs · Terms