Congratulations!

[Valid RSS] This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

Source: http://www.joachim-breitner.de/blog/feeds/categories/1-English.rss

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <rss xmlns:bib="http://joachim-breitner.de/2004/Website/Bibliography"
  3.     xmlns:content="http://purl.org/rss/1.0/modules/content/"
  4.     xmlns:atom="http://www.w3.org/2005/Atom"
  5.     version="2.0">
  6.   <channel>
  7.      <title>nomeata’s mind shares</title>
  8.      <link>https://www.joachim-breitner.de/blog</link>
  9.      <atom:link rel="self" type="application/rss+xml"
  10.                 href="https://www.joachim-breitner.de/blog_feed.rss"/>
  11.      <description>Joachim Breitners Denkblogade</description>
  12.      <image>
  13.         <url>https://joachim-breitner.de/avatars/avatar_128.png</url>
  14.         <title>nomeata’s mind shares</title>
  15.         <link>https://www.joachim-breitner.de/blog</link>
  16.         <width>128</width>
  17.         <height>128</height>
  18.      </image>
  19.      <item>
  20.         <title>Do surprises get larger?</title>
  21.         <link>https://www.joachim-breitner.de/blog/814-Do_surprises_get_larger_</link>
  22.         <guid>https://www.joachim-breitner.de/blog/814-Do_surprises_get_larger_</guid>
  23.         <comments>https://www.joachim-breitner.de/blog/814-Do_surprises_get_larger_#comments</comments>
  24.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  25.         <description>&lt;h3 id="the-setup"&gt;The setup&lt;/h3&gt;
  26. &lt;p&gt;Imagine you are living on a riverbank. Every now and then, the river swells and you have high water. The first few times this may come as a surprise, but soon you learn that such floods are a recurring occurrence at that river, and you make suitable preparation. Let’s say you feel well-prepared against any flood that is no higher than the highest one observed so far. The more floods you have seen, the higher that mark is, and the better prepared you are. But of course, eventually a higher flood will occur that surprises you.&lt;/p&gt;
  27. &lt;p&gt;Of course such new record floods are happening rarer and rarer as you have seen more of them. I was wondering though: By how much do the new records exceed the previous high mark? Does this excess decrease or increase over time?&lt;/p&gt;
  28. &lt;p&gt;A priori both could be. When the high mark is already rather high, maybe new record floods will just barley pass that mark? Or maybe, simply because new records are so rare events, when they do occur, they can be surprisingly bad?&lt;/p&gt;
  29. &lt;p&gt;This post is a leisurely mathematical investigating of this question, which of course isn’t restricted to high waters; it could be anything that produces a measurement repeatedly and (mostly) independently – weather events, sport results, dice rolls.&lt;/p&gt;
  30. &lt;p&gt;The answer of course depends on the &lt;em&gt;distribution&lt;/em&gt; of results: How likely is each possible results.&lt;/p&gt;
  31. &lt;h3 id="dice-are-simple"&gt;Dice are simple&lt;/h3&gt;
  32. &lt;p&gt;With dice rolls the answer is rather simple. Let our measurement be how often you can roll a die until it shows a 6. This simple game we can repeat many times, and keep track of our record. Let’s say the record happens to be 7 rolls. If in the next run we roll the die 7 times, and it still does not show a 6, then we know that we have broken the record, and every further roll increases by how much we beat the old record.&lt;/p&gt;
  33. &lt;p&gt;But note that how often we will now roll the die is completely independent of what happened before!&lt;/p&gt;
  34. &lt;p&gt;So for this game the answer is: The excess with which the record is broken is always the same.&lt;/p&gt;
  35. &lt;p&gt;Mathematically speaking this is because the distribution of “rolls until the die shows a 6” is &lt;a href="https://en.wikipedia.org/wiki/Memorylessness"&gt;&lt;em&gt;memoryless&lt;/em&gt;&lt;/a&gt;. Such distributions are rather special, its essentially just the example we gave (a &lt;a href="https://en.wikipedia.org/wiki/Geometric_distribution"&gt;geometric distribution&lt;/a&gt;), or its continuous analogue (the &lt;a href="https://en.wikipedia.org/wiki/Exponential_distribution"&gt;exponential distributions&lt;/a&gt;, for example the time until a radioactive particle decays).&lt;/p&gt;
  36. &lt;h3 id="mathematical-formulation"&gt;Mathematical formulation&lt;/h3&gt;
  37. &lt;p&gt;With this out of the way, let us look at some other distributions, and for that, introduce some mathematical notations. Let &lt;span class="math inline"&gt;&lt;em&gt;X&lt;/em&gt;&lt;/span&gt; be a random variable with probability density function &lt;span class="math inline"&gt;&lt;em&gt;φ&lt;/em&gt;(&lt;em&gt;x&lt;/em&gt;)&lt;/span&gt; and cumulative distribution function &lt;span class="math inline"&gt;&lt;em&gt;Φ&lt;/em&gt;(&lt;em&gt;x&lt;/em&gt;)&lt;/span&gt;, and &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt; be the previous record. We are interested in the behavior of&lt;/p&gt;
  38. &lt;p&gt;&lt;span class="math display"&gt;&lt;em&gt;Y&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;) = &lt;em&gt;X&lt;/em&gt; − &lt;em&gt;a&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; &gt; &lt;em&gt;x&lt;/em&gt;&lt;/span&gt;&lt;/p&gt;
  39. &lt;p&gt;i.e. by how much &lt;span class="math inline"&gt;&lt;em&gt;X&lt;/em&gt;&lt;/span&gt; exceeds &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt; under the condition that it did exceed &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt;. How does &lt;span class="math inline"&gt;&lt;em&gt;Y&lt;/em&gt;&lt;/span&gt; change as &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt; increases? In particular, how does the expected value of the excess &lt;span class="math inline"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;Y&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;))&lt;/span&gt; change?&lt;/p&gt;
  40. &lt;h3 id="uniform-distribution"&gt;Uniform distribution&lt;/h3&gt;
  41. &lt;p&gt;If &lt;span class="math inline"&gt;&lt;em&gt;X&lt;/em&gt;&lt;/span&gt; is uniformly distributed between, say, 0 and 1, then a new record will appear uniformly distributed between &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt; and &lt;span class="math inline"&gt;1&lt;/span&gt;, and as that range gets smaller, the excess must get smaller as well. More precisely,&lt;/p&gt;
  42. &lt;p&gt;&lt;span class="math display"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; − &lt;em&gt;a&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; &gt; &lt;em&gt;a&lt;/em&gt;) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; &gt; &lt;em&gt;a&lt;/em&gt;) − &lt;em&gt;a&lt;/em&gt; = (1 − &lt;em&gt;a&lt;/em&gt;)/2&lt;/span&gt;&lt;/p&gt;
  43. &lt;p&gt;This not very interesting linear line is plotted in blue in this diagram:&lt;/p&gt;
  44. &lt;figure&gt;
  45. &lt;img src="//www.joachim-breitner.de/various/surprises-plot1.svg" alt="The expected record surpass for the uniform distribution"/&gt;
  46. &lt;figcaption aria-hidden="true"&gt;The expected record surpass for the uniform distribution&lt;/figcaption&gt;
  47. &lt;/figure&gt;
  48. &lt;p&gt;The orange line with the logarithmic scale on the right tries to convey how unlikely it is to surpass the record value &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt;: it shows how many attempts we expect before the record is broken. This can be calculated by &lt;span class="math inline"&gt;&lt;em&gt;n&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;) = 1/(1 − &lt;em&gt;Φ&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;))&lt;/span&gt;.&lt;/p&gt;
  49. &lt;h3 id="normal-distribution"&gt;Normal distribution&lt;/h3&gt;
  50. &lt;p&gt;For the normal distribution (with median &lt;span class="math inline"&gt;0&lt;/span&gt; and standard derivation &lt;span class="math inline"&gt;1&lt;/span&gt;, to keep things simple), we can look up the expected value of the &lt;a href="https://en.wikipedia.org/wiki/Truncated_normal_distribution#One_sided_truncation_(of_lower_tail)%5B6%5D"&gt;one-sided truncated normal distribution&lt;/a&gt; and obtain&lt;/p&gt;
  51. &lt;p&gt;&lt;span class="math display"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; &gt; &lt;em&gt;a&lt;/em&gt;) − &lt;em&gt;a&lt;/em&gt; = &lt;em&gt;φ&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;)/(1 − &lt;em&gt;Φ&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;)) − &lt;em&gt;a&lt;/em&gt;&lt;/span&gt;&lt;/p&gt;
  52. &lt;p&gt;Now is this growing or shrinking? We can plot this an have a quick look:&lt;/p&gt;
  53. &lt;figure&gt;
  54. &lt;img src="//www.joachim-breitner.de/various/surprises-plot2.svg" alt="The expected record surpass for the normal distribution"/&gt;
  55. &lt;figcaption aria-hidden="true"&gt;The expected record surpass for the normal distribution&lt;/figcaption&gt;
  56. &lt;/figure&gt;
  57. &lt;p&gt;Indeed it is, too, a decreasing function!&lt;/p&gt;
  58. &lt;p&gt;(As a sanity check we can see that &lt;span class="math inline"&gt;&lt;em&gt;e&lt;/em&gt;(0) = √(2/&lt;em&gt;π&lt;/em&gt;)&lt;/span&gt;, which is the expected value of the &lt;a href="https://en.wikipedia.org/wiki/Half-normal_distribution"&gt;half-normal distribution&lt;/a&gt;, as it should.)&lt;/p&gt;
  59. &lt;h3 id="could-it-be-any-different"&gt;Could it be any different?&lt;/h3&gt;
  60. &lt;p&gt;This settles my question: It seems that each new surprisingly high water will tend to be less surprising than the previously – assuming high waters were uniformly or normally distributed, which is unlikely to be helpful.&lt;/p&gt;
  61. &lt;p&gt;This does raise the question, though, if there are probability distributions for which &lt;span class="math inline"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;)&lt;/span&gt; is be increasing?&lt;/p&gt;
  62. &lt;p&gt;I can try to construct one, and because it’s a bit easier, I’ll consider a discrete distribution on the positive natural numbers, and consider at &lt;span class="math inline"&gt;&lt;em&gt;g&lt;/em&gt;(0) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt;)&lt;/span&gt; and &lt;span class="math inline"&gt;&lt;em&gt;g&lt;/em&gt;(1) = &lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; − 1 ∣ &lt;em&gt;X&lt;/em&gt; &gt; 1)&lt;/span&gt;. What does it take for &lt;span class="math inline"&gt;&lt;em&gt;g&lt;/em&gt;(1) &gt; &lt;em&gt;g&lt;/em&gt;(0)&lt;/span&gt;? Using &lt;span class="math inline"&gt;&lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt;) = &lt;em&gt;p&lt;/em&gt; + (1 − &lt;em&gt;p&lt;/em&gt;)&lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; &gt; 1)&lt;/span&gt; for &lt;span class="math inline"&gt;&lt;em&gt;p&lt;/em&gt; = &lt;em&gt;P&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; = 1)&lt;/span&gt; we find that in order to have &lt;span class="math inline"&gt;&lt;em&gt;g&lt;/em&gt;(1) &gt; &lt;em&gt;g&lt;/em&gt;(0)&lt;/span&gt;, we need &lt;span class="math inline"&gt;&lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt;) &gt; 1/&lt;em&gt;p&lt;/em&gt;&lt;/span&gt;.&lt;/p&gt;
  63. &lt;p&gt;This is plausible because we get equality when &lt;span class="math inline"&gt;&lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt;) = 1/&lt;em&gt;p&lt;/em&gt;&lt;/span&gt;, as it precisely the case for the geometric distribution. And it is also plausible that it helps if &lt;span class="math inline"&gt;&lt;em&gt;p&lt;/em&gt;&lt;/span&gt; is large (so that the next first record is likely just 1) and if, nevertheless, &lt;span class="math inline"&gt;&lt;em&gt;E&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt;)&lt;/span&gt; is large (so that if we do get an outcome other than 1, it’s much larger).&lt;/p&gt;
  64. &lt;p&gt;Starting with the geometric distribution, where &lt;span class="math inline"&gt;&lt;em&gt;P&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; &gt; &lt;em&gt;n&lt;/em&gt; ∣ &lt;em&gt;X&lt;/em&gt; ≥ &lt;em&gt;n&lt;/em&gt;) = &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;&lt;em&gt;n&lt;/em&gt;&lt;/sub&gt; = &lt;em&gt;p&lt;/em&gt;&lt;/span&gt; (the probability of again not rolling a six) is constant, it seems that these &lt;span class="math inline"&gt;&lt;em&gt;p&lt;/em&gt;&lt;sub&gt;&lt;em&gt;n&lt;/em&gt;&lt;/sub&gt;&lt;/span&gt; is increasing, we get the desired behavior. So let &lt;span class="math inline"&gt;&lt;em&gt;p&lt;/em&gt;&lt;sub&gt;1&lt;/sub&gt; &amp;lt; &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;2&lt;/sub&gt; &amp;lt; &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;&lt;em&gt;n&lt;/em&gt;&lt;/sub&gt; &amp;lt; …&lt;/span&gt; be an increasing sequence of probabilities, and define &lt;span class="math inline"&gt;&lt;em&gt;X&lt;/em&gt;&lt;/span&gt; so that &lt;span class="math inline"&gt;&lt;em&gt;P&lt;/em&gt;(&lt;em&gt;X&lt;/em&gt; = &lt;em&gt;n&lt;/em&gt;) = &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;1&lt;/sub&gt; ⋅ ⋯ ⋅ &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;&lt;em&gt;n&lt;/em&gt; − 1&lt;/sub&gt; ⋅ (1 − &lt;em&gt;p&lt;/em&gt;&lt;sub&gt;&lt;em&gt;n&lt;/em&gt;&lt;/sub&gt;)&lt;/span&gt; (imagine the die wears off and the more often you roll it, the less likely it shows a 6). Then for this variation of the game, every new record tends to exceed the previous more than previous records. As the &lt;span class="math inline"&gt;&lt;em&gt;p&lt;/em&gt;&lt;/span&gt; increase, we get a flatter long end in the probability distribution.&lt;/p&gt;
  65. &lt;h3 id="gamma-distribution"&gt;Gamma distribution&lt;/h3&gt;
  66. &lt;p&gt;To get a nice plot, I’ll take the intuition from this and turn to continuous distributions. The Wikipedia page for the exponential distribution says it is a special case of the &lt;a href="https://en.m.wikipedia.org/wiki/Gamma_distribution"&gt;gamma distribution&lt;/a&gt;, which has an additional shape parameter &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt;&lt;/span&gt;, and it seems that it could influence the shape of the distribution to be and make the probability distribution have a longer end. Let’s play around with &lt;span class="math inline"&gt;&lt;em&gt;β&lt;/em&gt; = 2&lt;/span&gt; and &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt; = 0.5&lt;/span&gt;, &lt;span class="math inline"&gt;1&lt;/span&gt; and &lt;span class="math inline"&gt;1.5&lt;/span&gt;:&lt;/p&gt;
  67. &lt;figure&gt;
  68. &lt;img src="//www.joachim-breitner.de/various/surprises-plot3.svg" alt="The expected record surpass for the gamma distribution"/&gt;
  69. &lt;figcaption aria-hidden="true"&gt;The expected record surpass for the gamma distribution&lt;/figcaption&gt;
  70. &lt;/figure&gt;
  71. &lt;ul&gt;
  72. &lt;li&gt;&lt;p&gt;For &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt; = 1&lt;/span&gt; (dotted) this should just be the exponential distribution, and we see that &lt;span class="math inline"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;)&lt;/span&gt; is flat, as predicted earlier.&lt;/p&gt;&lt;/li&gt;
  73. &lt;li&gt;&lt;p&gt;For larger &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt;&lt;/span&gt; (dashed) the graph does not look much different from the one for the normal distribution – not a surprise, as for &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt; → ∞&lt;/span&gt;, the gamma distribution turns into the normal distribution.&lt;/p&gt;&lt;/li&gt;
  74. &lt;li&gt;&lt;p&gt;For smaller &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt;&lt;/span&gt; (solid) we get the desired effect: &lt;span class="math inline"&gt;&lt;em&gt;e&lt;/em&gt;(&lt;em&gt;a&lt;/em&gt;)&lt;/span&gt; is increasing. This means that new records tend to break records more impressively.&lt;/p&gt;&lt;/li&gt;
  75. &lt;/ul&gt;
  76. &lt;p&gt;The orange line shows that this comes at a cost: for a given old record &lt;span class="math inline"&gt;&lt;em&gt;a&lt;/em&gt;&lt;/span&gt;, new records are harder to come by with smaller &lt;span class="math inline"&gt;&lt;em&gt;α&lt;/em&gt;&lt;/span&gt;.&lt;/p&gt;
  77. &lt;h3 id="conclusion"&gt;Conclusion&lt;/h3&gt;
  78. &lt;p&gt;As usual, it all depends on the distribution. Otherwise, not much, it’s late.&lt;/p&gt;</description>
  79.         <pubDate>Sun, 30 Jun 2024 15:28:31 +0200</pubDate>
  80.      </item>
  81.      <item>
  82.         <title>Blogging on Lean</title>
  83.         <link>https://www.joachim-breitner.de/blog/813-Blogging_on_Lean</link>
  84.         <guid>https://www.joachim-breitner.de/blog/813-Blogging_on_Lean</guid>
  85.         <comments>https://www.joachim-breitner.de/blog/813-Blogging_on_Lean#comments</comments>
  86.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  87.         <description>&lt;p&gt;This blog has become a bit quiet &lt;a href="https://www.joachim-breitner.de/blog/809-Joining_the_Lean_FRO"&gt;since I joined the Lean FRO&lt;/a&gt;. One reasons is of course that I can now improve things about Lean, rather than blog about what I think should be done (which, by contraposition, means I shouldn’t blog about what can be improved…). A better reason is that some of the things I’d otherwise write here are now published on the &lt;a href="https://lean-lang.org/blog"&gt;official Lean blog&lt;/a&gt;, in particular two lengthy technical posts explaining aspects of Lean that I worked on:&lt;/p&gt;
  88. &lt;ul&gt;
  89. &lt;li&gt;&lt;a href="https://lean-lang.org/blog/2024-1-11-recursive-definitions-in-lean"&gt;Recursive definitions in Lean&lt;/a&gt;&lt;/li&gt;
  90. &lt;li&gt;&lt;a href="https://lean-lang.org/blog/2024-5-17-functional-induction"&gt;Functional Induction theorems&lt;/a&gt;&lt;/li&gt;
  91. &lt;/ul&gt;
  92. &lt;p&gt;It would not be useful to re-publish them here because the technology &lt;a href="https://github.com/leanprover/verso"&gt;&lt;code&gt;verso&lt;/code&gt;&lt;/a&gt; behind the Lean blog, created by my colleage David Thrane Christansen, enables such fancy features like type-checked code snippets, including output and lots of information on hover. So I’ll be content with just cross-linking my posts from here.&lt;/p&gt;</description>
  93.         <pubDate>Fri, 31 May 2024 13:47:06 +0100</pubDate>
  94.      </item>
  95.      <item>
  96.         <title>Convenient sandboxed development environment</title>
  97.         <link>https://www.joachim-breitner.de/blog/812-Convenient_sandboxed_development_environment</link>
  98.         <guid>https://www.joachim-breitner.de/blog/812-Convenient_sandboxed_development_environment</guid>
  99.         <comments>https://www.joachim-breitner.de/blog/812-Convenient_sandboxed_development_environment#comments</comments>
  100.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  101.         <description>&lt;p&gt;I like using one machine and setup for everything, from serious development work to hobby projects to managing my finances. This is very convenient, as often the lines between these are blurred. But it is also scary if I think of the large number of people who I have to trust to not want to extract all my personal data. Whenever I run a &lt;code&gt;cabal install&lt;/code&gt;, or a fun VSCode extension gets updated, or anything like that, I am running code that could be malicious or buggy.&lt;/p&gt;
  102. &lt;p&gt;In a way it is surprising and reassuring that, as far as I can tell, this commonly does not happen. Most open source developers out there seem to be nice and well-meaning, after all.&lt;/p&gt;
  103. &lt;h3 id="convenient-or-it-wont-happen"&gt;Convenient or it won’t happen&lt;/h3&gt;
  104. &lt;p&gt;Nevertheless I thought I should do something about this. The safest option would probably to use dedicated virtual machines for the development work, with very little interaction with my main system. But knowing me, that did not seem likely to happen, as it sounded like a fair amount of hassle. So I aimed for a viable compromise between security and convenient, and one that does not get too much in the way of my current habits.&lt;/p&gt;
  105. &lt;p&gt;For instance, it seems desirable to have the project files accessible from my unconstrained environment. This way, I could perform certain actions that need access to secret keys or tokens, but are (unlikely) to run code (e.g. &lt;code&gt;git push&lt;/code&gt;, &lt;code&gt;git pull&lt;/code&gt; from private repositories, &lt;code&gt;gh pr create&lt;/code&gt;) from “the outside”, and the actual build environment can do without access to these secrets.&lt;/p&gt;
  106. &lt;p&gt;The user experience I thus want is a quick way to enter a “development environment” where I can do most of the things I need to do while programming (network access, running command line and GUI programs), with access to the current project, but without access to my actual &lt;code&gt;/home&lt;/code&gt; directory.&lt;/p&gt;
  107. &lt;p&gt;I initially followed the blog post &lt;a href="https://msucharski.eu/posts/application-isolation-nixos-containers/"&gt;“Application Isolation using NixOS Containers” by Marcin Sucharski&lt;/a&gt; and got something working that mostly did what I wanted, but then a colleague pointed out that tools like &lt;a href="https://github.com/netblue30/firejail"&gt;&lt;code&gt;firejail&lt;/code&gt;&lt;/a&gt; can achieve roughly the same with a less “global” setup. I tried to use &lt;code&gt;firejail&lt;/code&gt;, but found it to be a bit too inflexible for my particular whims, so I ended up writing a small wrapper around the lower level sandboxing tool &lt;a href="Bubblewrap"&gt;https://github.com/containers/bubblewrap&lt;/a&gt;.&lt;/p&gt;
  108. &lt;h3 id="selective-bubblewrapping"&gt;Selective bubblewrapping&lt;/h3&gt;
  109. &lt;p&gt;This script, called &lt;code&gt;dev&lt;/code&gt; and included below, builds a new filesystem namespace with minimal &lt;code&gt;/proc&lt;/code&gt; and &lt;code&gt;/dev&lt;/code&gt; directories, it’s own &lt;code&gt;/tmp&lt;/code&gt; directories. It then binds-mound some directories to make the host’s NixOS system available inside the container (&lt;code&gt;/bin&lt;/code&gt;, &lt;code&gt;/usr&lt;/code&gt;, the nix store including domain socket, stuff for OpenGL applications). My user’s home directory is taken from &lt;code&gt;~/.dev-home&lt;/code&gt; and some configuration files are bind-mounted for convenient sharing. I intentionally don’t share most of the configuration – for example, a &lt;code&gt;direnv enable&lt;/code&gt; in the dev environment should not affect the main environment. The X11 socket for graphical applications and the corresponding &lt;code&gt;.Xauthority&lt;/code&gt; file is made available. And finally, if I run &lt;code&gt;dev&lt;/code&gt; in a project directory, this project directory is bind mounted writable, and the current working directory is preserved.&lt;/p&gt;
  110. &lt;p&gt;The effect is that I can type &lt;code&gt;dev&lt;/code&gt; on the command line to enter “dev mode” rather conveniently. I can run development tools, including graphical ones like VSCode, and especially the latter with its extensions is part of the sandbox. To do a &lt;code&gt;git push&lt;/code&gt; I either exit the development environment (Ctrl-D) or open a separate terminal. Overall, the inconvenience of switching back and forth seems worth the extra protection.&lt;/p&gt;
  111. &lt;p&gt;Clearly, isn’t going to hold against a determined and maybe targeted attacker (e.g. access to the X11 and the nix daemon socket can probably be used to escape easily). But I hope it will help against a compromised dev dependency that just deletes or exfiltrates data, like keys or passwords, from the usual places in &lt;code&gt;$HOME&lt;/code&gt;.&lt;/p&gt;
  112. &lt;h3 id="rough-corners"&gt;Rough corners&lt;/h3&gt;
  113. &lt;p&gt;There is more polishing that could be done.&lt;/p&gt;
  114. &lt;ul&gt;
  115. &lt;li&gt;&lt;p&gt;In particular, clicking on a link inside VSCode in the container will currently open Firefox inside the container, without access to my settings and cookies etc. Ideally, links would be opened in the Firefox running outside. This is a problem that has a solution in the world of applications that are sandboxed with Flatpak, and involves a bunch of moving parts (a &lt;a href="https://github.com/flatpak/xdg-desktop-portal"&gt;xdg-desktop-portal&lt;/a&gt; user service, a &lt;a href="https://github.com/flatpak/xdg-dbus-proxy"&gt;filtering dbus proxy&lt;/a&gt;, exposing access to that proxy in the container). I experimented with that for a bit longer than I should have, but could not get it to work to satisfaction (even without a container involved, I could not get &lt;code&gt;xdg-desktop-portal&lt;/code&gt; to heed my default browser settings…). For now I will live with manually copying and pasting URLs, we’ll see how long this lasts.&lt;/p&gt;&lt;/li&gt;
  116. &lt;li&gt;&lt;p&gt;With this setup (and unlike the NixOS container setup I tried first), the same applications are installed inside and outside. It might be useful to separate the set of installed programs: There is simply no point in running &lt;code&gt;evolution&lt;/code&gt; or &lt;code&gt;firefox&lt;/code&gt; inside the container, and if I do not even have VSCode or &lt;code&gt;cabal&lt;/code&gt; available outside, so that it’s less likely that I forget to enter &lt;code&gt;dev&lt;/code&gt; before using these tools.&lt;/p&gt;
  117. &lt;p&gt;It shouldn’t be too hard to cargo-cult some of the NixOS Containers infrastructure to be able to have a separate system configuration that I can manage as part of my normal system configuration and make available to &lt;code&gt;bubblewrap&lt;/code&gt; here.&lt;/p&gt;&lt;/li&gt;
  118. &lt;/ul&gt;
  119. &lt;p&gt;So likely I will refine this some more over time. Or get tired of typing &lt;code&gt;dev&lt;/code&gt; and going back to what I did before…&lt;/p&gt;
  120. &lt;h3 id="the-script"&gt;The script&lt;/h3&gt;
  121. &lt;details&gt;
  122. &lt;summary&gt;
  123. The &lt;code&gt;dev&lt;/code&gt; script (at the time of writing)
  124. &lt;/summary&gt;
  125. &lt;div class="sourceCode" id="cb1"&gt;&lt;pre class="sourceCode bash"&gt;&lt;code class="sourceCode bash"&gt;&lt;span id="cb1-1"&gt;&lt;a href="#cb1-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;#!/usr/bin/env bash&lt;/span&gt;&lt;/span&gt;
  126. &lt;span id="cb1-2"&gt;&lt;a href="#cb1-2" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  127. &lt;span id="cb1-3"&gt;&lt;a href="#cb1-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="va"&gt;extra&lt;/span&gt;&lt;span class="op"&gt;=&lt;/span&gt;&lt;span class="va"&gt;()&lt;/span&gt;&lt;/span&gt;
  128. &lt;span id="cb1-4"&gt;&lt;a href="#cb1-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;if&lt;/span&gt; &lt;span class="kw"&gt;[[&lt;/span&gt; &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$PWD&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="ot"&gt;==&lt;/span&gt; /home/jojo/build/&lt;span class="pp"&gt;*&lt;/span&gt; &lt;span class="kw"&gt;]]&lt;/span&gt; &lt;span class="kw"&gt;||&lt;/span&gt; &lt;span class="kw"&gt;[[&lt;/span&gt; &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$PWD&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="ot"&gt;==&lt;/span&gt; /home/jojo/projekte/programming/&lt;span class="pp"&gt;*&lt;/span&gt; &lt;span class="kw"&gt;]]&lt;/span&gt;&lt;/span&gt;
  129. &lt;span id="cb1-5"&gt;&lt;a href="#cb1-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;then&lt;/span&gt;&lt;/span&gt;
  130. &lt;span id="cb1-6"&gt;&lt;a href="#cb1-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="va"&gt;extra&lt;/span&gt;&lt;span class="op"&gt;+=&lt;/span&gt;&lt;span class="va"&gt;(&lt;/span&gt;--bind &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$PWD&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$PWD&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; --chdir &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$PWD&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;)&lt;/span&gt;&lt;/span&gt;
  131. &lt;span id="cb1-7"&gt;&lt;a href="#cb1-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;fi&lt;/span&gt;&lt;/span&gt;
  132. &lt;span id="cb1-8"&gt;&lt;a href="#cb1-8" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  133. &lt;span id="cb1-9"&gt;&lt;a href="#cb1-9" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;if&lt;/span&gt; &lt;span class="bu"&gt;[&lt;/span&gt; &lt;span class="ot"&gt;-n&lt;/span&gt; &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$1&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="bu"&gt;]&lt;/span&gt;&lt;/span&gt;
  134. &lt;span id="cb1-10"&gt;&lt;a href="#cb1-10" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;then&lt;/span&gt;&lt;/span&gt;
  135. &lt;span id="cb1-11"&gt;&lt;a href="#cb1-11" aria-hidden="true" tabindex="-1"/&gt;    &lt;span class="va"&gt;cmd&lt;/span&gt;&lt;span class="op"&gt;=&lt;/span&gt;&lt;span class="va"&gt;(&lt;/span&gt; &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;$@&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="va"&gt;)&lt;/span&gt;&lt;/span&gt;
  136. &lt;span id="cb1-12"&gt;&lt;a href="#cb1-12" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;else&lt;/span&gt;&lt;/span&gt;
  137. &lt;span id="cb1-13"&gt;&lt;a href="#cb1-13" aria-hidden="true" tabindex="-1"/&gt;    &lt;span class="va"&gt;cmd&lt;/span&gt;&lt;span class="op"&gt;=&lt;/span&gt;&lt;span class="va"&gt;(&lt;/span&gt; bash &lt;span class="va"&gt;)&lt;/span&gt;&lt;/span&gt;
  138. &lt;span id="cb1-14"&gt;&lt;a href="#cb1-14" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;fi&lt;/span&gt;&lt;/span&gt;
  139. &lt;span id="cb1-15"&gt;&lt;a href="#cb1-15" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  140. &lt;span id="cb1-16"&gt;&lt;a href="#cb1-16" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;# Caveats:&lt;/span&gt;&lt;/span&gt;
  141. &lt;span id="cb1-17"&gt;&lt;a href="#cb1-17" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;# * access to all of `/etc`&lt;/span&gt;&lt;/span&gt;
  142. &lt;span id="cb1-18"&gt;&lt;a href="#cb1-18" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;# * access to `/nix/var/nix/daemon-socket/socket`, and is trusted user (but needed to run nix)&lt;/span&gt;&lt;/span&gt;
  143. &lt;span id="cb1-19"&gt;&lt;a href="#cb1-19" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;# * access to X11&lt;/span&gt;&lt;/span&gt;
  144. &lt;span id="cb1-20"&gt;&lt;a href="#cb1-20" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  145. &lt;span id="cb1-21"&gt;&lt;a href="#cb1-21" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="bu"&gt;exec&lt;/span&gt; bwrap &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  146. &lt;span id="cb1-22"&gt;&lt;a href="#cb1-22" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--unshare-all&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  147. &lt;span id="cb1-23"&gt;&lt;a href="#cb1-23" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  148. &lt;span id="cb1-24"&gt;&lt;a href="#cb1-24" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;`&lt;/span&gt;&lt;span class="co"&gt;# blank slate&lt;/span&gt;&lt;span class="kw"&gt;`&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  149. &lt;span id="cb1-25"&gt;&lt;a href="#cb1-25" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--share-net&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  150. &lt;span id="cb1-26"&gt;&lt;a href="#cb1-26" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--proc&lt;/span&gt; /proc &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  151. &lt;span id="cb1-27"&gt;&lt;a href="#cb1-27" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--dev&lt;/span&gt; /dev &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  152. &lt;span id="cb1-28"&gt;&lt;a href="#cb1-28" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--tmpfs&lt;/span&gt; /tmp &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  153. &lt;span id="cb1-29"&gt;&lt;a href="#cb1-29" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--tmpfs&lt;/span&gt; /run/user/1000 &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  154. &lt;span id="cb1-30"&gt;&lt;a href="#cb1-30" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  155. &lt;span id="cb1-31"&gt;&lt;a href="#cb1-31" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;`&lt;/span&gt;&lt;span class="co"&gt;# Needed for GLX applications, in paticular alacritty&lt;/span&gt;&lt;span class="kw"&gt;`&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  156. &lt;span id="cb1-32"&gt;&lt;a href="#cb1-32" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--dev-bind&lt;/span&gt; /dev/dri /dev/dri &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  157. &lt;span id="cb1-33"&gt;&lt;a href="#cb1-33" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /sys/dev/char /sys/dev/char &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  158. &lt;span id="cb1-34"&gt;&lt;a href="#cb1-34" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /sys/devices/pci0000:00 /sys/devices/pci0000:00 &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  159. &lt;span id="cb1-35"&gt;&lt;a href="#cb1-35" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /run/opengl-driver /run/opengl-driver &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  160. &lt;span id="cb1-36"&gt;&lt;a href="#cb1-36" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  161. &lt;span id="cb1-37"&gt;&lt;a href="#cb1-37" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /bin /bin &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  162. &lt;span id="cb1-38"&gt;&lt;a href="#cb1-38" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /usr /usr &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  163. &lt;span id="cb1-39"&gt;&lt;a href="#cb1-39" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /run/current-system /run/current-system &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  164. &lt;span id="cb1-40"&gt;&lt;a href="#cb1-40" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /nix /nix &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  165. &lt;span id="cb1-41"&gt;&lt;a href="#cb1-41" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /etc /etc &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  166. &lt;span id="cb1-42"&gt;&lt;a href="#cb1-42" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; /run/systemd/resolve/stub-resolv.conf /run/systemd/resolve/stub-resolv.conf &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  167. &lt;span id="cb1-43"&gt;&lt;a href="#cb1-43" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  168. &lt;span id="cb1-44"&gt;&lt;a href="#cb1-44" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--bind&lt;/span&gt; ~/.dev-home /home/jojo &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  169. &lt;span id="cb1-45"&gt;&lt;a href="#cb1-45" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; ~/.config/alacritty ~/.config/alacritty  &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  170. &lt;span id="cb1-46"&gt;&lt;a href="#cb1-46" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; ~/.config/nvim ~/.config/nvim  &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  171. &lt;span id="cb1-47"&gt;&lt;a href="#cb1-47" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; ~/.local/share/nvim ~/.local/share/nvim  &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  172. &lt;span id="cb1-48"&gt;&lt;a href="#cb1-48" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--ro-bind&lt;/span&gt; ~/.bin ~/.bin &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  173. &lt;span id="cb1-49"&gt;&lt;a href="#cb1-49" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  174. &lt;span id="cb1-50"&gt;&lt;a href="#cb1-50" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--bind&lt;/span&gt; /tmp/.X11-unix/X0 /tmp/.X11-unix/X0 &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  175. &lt;span id="cb1-51"&gt;&lt;a href="#cb1-51" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--bind&lt;/span&gt; ~/.Xauthority ~/.Xauthority &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  176. &lt;span id="cb1-52"&gt;&lt;a href="#cb1-52" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--setenv&lt;/span&gt; DISPLAY :0 &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  177. &lt;span id="cb1-53"&gt;&lt;a href="#cb1-53" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  178. &lt;span id="cb1-54"&gt;&lt;a href="#cb1-54" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--setenv&lt;/span&gt; container dev &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  179. &lt;span id="cb1-55"&gt;&lt;a href="#cb1-55" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;${extra&lt;/span&gt;&lt;span class="op"&gt;[@]&lt;/span&gt;&lt;span class="va"&gt;}&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  180. &lt;span id="cb1-56"&gt;&lt;a href="#cb1-56" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="at"&gt;--&lt;/span&gt; &lt;span class="dt"&gt;\&lt;/span&gt;&lt;/span&gt;
  181. &lt;span id="cb1-57"&gt;&lt;a href="#cb1-57" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="st"&gt;"&lt;/span&gt;&lt;span class="va"&gt;${cmd&lt;/span&gt;&lt;span class="op"&gt;[@]&lt;/span&gt;&lt;span class="va"&gt;}&lt;/span&gt;&lt;span class="st"&gt;"&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  182. &lt;/details&gt;</description>
  183.         <pubDate>Mon, 11 Mar 2024 21:39:58 +0100</pubDate>
  184.      </item>
  185.      <item>
  186.         <title>GHC Steering Committee Retrospective</title>
  187.         <link>https://www.joachim-breitner.de/blog/811-GHC_Steering_Committee_Retrospective</link>
  188.         <guid>https://www.joachim-breitner.de/blog/811-GHC_Steering_Committee_Retrospective</guid>
  189.         <comments>https://www.joachim-breitner.de/blog/811-GHC_Steering_Committee_Retrospective#comments</comments>
  190.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  191.         <description>&lt;p&gt;After seven years of service as member and secretary on the GHC Steering Committee, I have resigned from that role. So this is a good time to look back and retrace the formation of the GHC proposal process and committee.&lt;/p&gt;
  192. &lt;p&gt;In my memory, I helped define and shape the proposal process, optimizing it for effectiveness and throughput, but memory can be misleading, and judging from the paper trail in my email archives, this was indeed mostly Ben Gamari’s and Richard Eisenberg’s achievement: Already in Summer of 2016, Ben Gamari set up the &lt;a href="https://github.com/ghc-proposals/ghc-proposals"&gt;ghc-proposals Github repository&lt;/a&gt; with a sketch of a process and sent out a &lt;a href="https://mail.haskell.org/pipermail/glasgow-haskell-users/2016-September/026396.html"&gt;call for nominations&lt;/a&gt; on the GHC user’s mailing list, which I replied to. The Simons picked the first set of members, and in the fall of 2016 we discussed the committee’s by-laws and procedures. As so often, Richard was an influential shaping force here.&lt;/p&gt;
  193. &lt;h3 id="three-ingredients"&gt;Three ingredients&lt;/h3&gt;
  194. &lt;p&gt;For example, it was him that suggested that for each proposal we have one committee member be the “Shepherd”, overseeing the discussion. I believe this was one ingredient for the process effectiveness: There is always one person in charge, and thus we avoid the delays incurred when any one of a non-singleton set of volunteers have to do the next step (and everyone hopes someone else does it).&lt;/p&gt;
  195. &lt;p&gt;The next ingredient was that we do not usually require a vote among all members (again, not easy with volunteers with limited bandwidth and occasional phases of absence). Instead, the shepherd makes a recommendation (accept/reject), and if the other committee members do not complain, this silence is taken as consent, and we come to a decision. It seems this idea can also be traced back on Richard, who suggested that “once a decision is requested, the shepherd [generates] consensus. If consensus is elusive, then we vote.”&lt;/p&gt;
  196. &lt;p&gt;At the end of the year we agreed and wrote down these rules, created the mailing list for our internal, but &lt;a href="https://mail.haskell.org/pipermail/ghc-steering-committee/"&gt;publicly archived committee discussions&lt;/a&gt;, and began accepting proposals, starting with &lt;a href="https://github.com/ghc-proposals/ghc-proposals/pull/6"&gt;Adam Gundry’s &lt;code&gt;OverloadedRecordFields&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;
  197. &lt;p&gt;At that point, there was no “secretary” role yet, so how I did become one? It seems that in February 2017 I started to &lt;a href="https://mail.haskell.org/pipermail/ghc-steering-committee/2017-February/000040.html"&gt;clean-up and refine the process documentation&lt;/a&gt;, fixing “bugs in the process” (like requiring authors to set Github labels when they don’t even have permissions to do that). This in particular meant that someone from the committee had to manually handle submissions and so on, and by the aforementioned principle that at every step there ought to be exactly one person in change, the role of a secretary followed naturally. In the &lt;a href="https://mail.haskell.org/pipermail/ghc-steering-committee/2017-February/000044.html"&gt;email in which I described that role&lt;/a&gt; I wrote:&lt;/p&gt;
  198. &lt;blockquote&gt;
  199. &lt;p&gt;Simon already shoved me towards picking up the “secretary” hat, to reduce load on Ben.&lt;/p&gt;
  200. &lt;/blockquote&gt;
  201. &lt;p&gt;So when I &lt;a href="https://github.com/ghc-proposals/ghc-proposals/commit/63d050c380b2aa380581cc9b9aafb2a7c022556a"&gt;merged the updated process documentation&lt;/a&gt;, I already listed myself “secretary”.&lt;/p&gt;
  202. &lt;p&gt;It wasn’t just Simon’s shoving that put my into the role, though. I dug out my original self-nomination email to Ben, and among other things I wrote:&lt;/p&gt;
  203. &lt;blockquote&gt;
  204. &lt;p&gt;I also hope that there is going to be clear responsibilities and a clear workflow among the committee. E.g. someone (possibly rotating), maybe called the secretary, who is in charge of having an initial look at proposals and then assigning it to a member who shepherds the proposal.&lt;/p&gt;
  205. &lt;/blockquote&gt;
  206. &lt;p&gt;So it is hardly a surprise that I became secretary, when it was dear to my heart to have a smooth continuous process here.&lt;/p&gt;
  207. &lt;p&gt;I am rather content with the result: These three ingredients – single secretary, per-proposal shepherds, silence-is-consent – helped the committee to be effective throughout its existence, even as every once in a while individual members dropped out.&lt;/p&gt;
  208. &lt;h3 id="ulterior-motivation"&gt;Ulterior motivation&lt;/h3&gt;
  209. &lt;p&gt;I must admit, however, there was an ulterior motivation behind me grabbing the secretary role: Yes, I &lt;em&gt;did&lt;/em&gt; want the committee to succeed, and I &lt;em&gt;did&lt;/em&gt; want that authors receive timely, good and decisive feedback on their proposals – but I &lt;em&gt;did not&lt;/em&gt; really want to have to do that part.&lt;/p&gt;
  210. &lt;p&gt;I am, in fact, a lousy proposal reviewer. I am too generous when reading proposals, and more likely mentally fill gaps in a specification rather than spotting them. Always optimistically assuming that the authors surely know what they are doing, rather than critically assessing the impact, the implementation cost and the interaction with other language features.&lt;/p&gt;
  211. &lt;p&gt;And, maybe more importantly: why should I know which changes are good and which are not so good in the long run? Clearly, the authors cared enough about a proposal to put it forward, so there is some need… and I do believe that Haskell should stay an evolving and innovating language… but how does this help me decide about this or that particular feature.&lt;/p&gt;
  212. &lt;p&gt;I even, during the formation of the committee, explicitly asked that we write down some guidance on “Vision and Guideline”; do we want to foster change or innovation, or be selective gatekeepers? Should we accept features that are proven to be useful, or should we accept features so that they can prove to be useful? This discussion, however, did not lead to a concrete result, and the assessment of proposals relied on the sum of each member’s personal preference, expertise and gut feeling. I am not saying that this was a mistake: It is hard to come up with a general guideline here, and even harder to find one that does justice to each individual proposal.&lt;/p&gt;
  213. &lt;p&gt;So the secret motivation for me to grab the secretary post was that I could contribute without having to judge proposals. Being secretary allowed me to assign most proposals to others to shepherd, and only once in a while myself took care of a proposal, when it seemed to be very straight-forward. Sneaky, ain’t it?&lt;/p&gt;
  214. &lt;h3 id="years-later"&gt;7 Years later&lt;/h3&gt;
  215. &lt;p&gt;For years to come I happily played secretary: When an author finished their proposal and public discussion ebbed down they would ping me on GitHub, I would &lt;a href="https://mail.haskell.org/pipermail/ghc-steering-committee/2022-July/002837.html"&gt;pick a suitable shepherd&lt;/a&gt; among the committee and ask them to judge the proposal. Eventually, the committee would come to a conclusion, usually by implicit consent, sometimes by voting, and I’d merge the pull request and update the metadata thereon. Every few months I’d &lt;a href="https://mail.haskell.org/pipermail/ghc-steering-committee/2024-January/003683.html"&gt;summarize the current state of affairs&lt;/a&gt; to the committee (what happened since the last update, which proposals are currently on our plate), and once per year gathered the data for &lt;a href="https://youtu.be/LFIL0myeOlo?si=P316QJs0EBSWe4Uy&amp;amp;t=3955"&gt;Simon Peyton Jones’ annually GHC Status Report&lt;/a&gt;. Sometimes some members needed a nudge or two to act. Some would eventually step down, and I’d sent around a call for nominations and when the nominations came in, distributed them off-list among the committee and tallied the votes.&lt;/p&gt;
  216. &lt;p&gt;Initially, that was exciting. For a long while it was a pleasant and rewarding routine. Eventually, it became a mere chore. I noticed that I didn’t quite care so much anymore about some of the discussion, and there was a decent amount of naval-gazing, meta-discussions and some wrangling about claims of authority that was probably useful and necessary, but wasn’t particularly fun.&lt;/p&gt;
  217. &lt;p&gt;I also began to notice weaknesses in the processes that I helped shape: We could really use some more automation for showing proposal statuses, notifying people when they have to act, and nudging them when they don’t. The whole silence-is-assent approach is good for throughput, but not necessary great for quality, and maybe the committee members need to be pushed more firmly to engage with each proposal. Like GHC itself, the committee processes deserve continuous refinement and refactoring, and since I could not muster the motivation to change my now well-trod secretarial ways, it was time for me to step down.&lt;/p&gt;
  218. &lt;p&gt;Luckily, Adam Gundry volunteered to take over, and that makes me feel much less bad for quitting. Thanks for that!&lt;/p&gt;
  219. &lt;p&gt;And although I am for my day job now &lt;a href="https://lean-lang.org/"&gt;enjoying a language&lt;/a&gt; that has many of the things out of the box that for Haskell are still only language extensions or even just future proposals (dependent types, &lt;code&gt;BlockArguments&lt;/code&gt;, &lt;code&gt;do&lt;/code&gt; notation with &lt;code&gt;(← foo)&lt;/code&gt; expressions and 💜 Unicode), I’m still around, hosting the &lt;a href="https://haskell.foundation/podcast/"&gt;Haskell Interlude Podcast&lt;/a&gt;, writing on this blog and hanging out at ZuriHac etc.&lt;/p&gt;</description>
  220.         <pubDate>Thu, 25 Jan 2024 01:21:41 +0100</pubDate>
  221.      </item>
  222.      <item>
  223.         <title>The Haskell Interlude Podcast</title>
  224.         <link>https://www.joachim-breitner.de/blog/810-The_Haskell_Interlude_Podcast</link>
  225.         <guid>https://www.joachim-breitner.de/blog/810-The_Haskell_Interlude_Podcast</guid>
  226.         <comments>https://www.joachim-breitner.de/blog/810-The_Haskell_Interlude_Podcast#comments</comments>
  227.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  228.         <description>&lt;p&gt;It was pointed out to me that I have not blogged about this, so better now than never:&lt;/p&gt;
  229. &lt;p&gt;Since 2021 I am – together with four other hosts – producing a regular podcast about Haskell, the &lt;a href="https://haskell.foundation/podcast/"&gt;&lt;strong&gt;Haskell Interlude&lt;/strong&gt;&lt;/a&gt;. Roughly every two weeks two of us interview someone from the Haskell Community, and we chat for approximately an hour about how they came to Haskell, what they are doing with it, why they are doing it and what else is on their mind. Sometimes we talk to very famous people, like Simon Peyton Jones, and sometimes to people who maybe should be famous, but aren’t quite yet.&lt;/p&gt;
  230. &lt;p&gt;For most episodes we also have a transcript, so you can read the interviews instead, if you prefer, and you should find the podcast on most podcast apps as well. I do not know how reliable these statistics are, but supposedly we regularly have around 1300 listeners. We don’t get much feedback, however, so if you like the show, or dislike it, or have feedback, let us know (for example on the &lt;a href="https://discourse.haskell.org/"&gt;Haskell Disourse&lt;/a&gt;, which has a thread for each episode).&lt;/p&gt;
  231. &lt;p&gt;At the time of writing, we released 40 episodes. For the benefit of my (likely hypothetical) fans, or those who want to train an AI voice model for nefarious purposes, here is the list of episodes co-hosted by me:&lt;/p&gt;
  232. &lt;ul&gt;
  233. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/3"&gt;Gabriella Gonzales&lt;/a&gt;&lt;/li&gt;
  234. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/4"&gt;Jasper Van der Jeugt&lt;/a&gt;&lt;/li&gt;
  235. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/5"&gt;Chris Smith&lt;/a&gt;&lt;/li&gt;
  236. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/9"&gt;Sebastian Graf&lt;/a&gt;&lt;/li&gt;
  237. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/11"&gt;Simon Peyton Jones&lt;/a&gt;&lt;/li&gt;
  238. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/14"&gt;Ryan Trinkle&lt;/a&gt;&lt;/li&gt;
  239. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/15"&gt;Facundo Dominguez&lt;/a&gt;&lt;/li&gt;
  240. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/19"&gt;Marc Scholten&lt;/a&gt;&lt;/li&gt;
  241. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/23"&gt;Ben Gamari&lt;/a&gt;&lt;/li&gt;
  242. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/25"&gt;Andrew Lelechenko (Bodigrim)&lt;/a&gt;&lt;/li&gt;
  243. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/29"&gt;ZuriHac 2023 special&lt;/a&gt;&lt;/li&gt;
  244. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/31"&gt;Arnaud Spiwack&lt;/a&gt;&lt;/li&gt;
  245. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/37"&gt;John MacFarlane&lt;/a&gt;&lt;/li&gt;
  246. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/41"&gt;Moritz Angermann&lt;/a&gt;&lt;/li&gt;
  247. &lt;li&gt;&lt;a href="https://haskell.foundation/podcast/42"&gt;Jezen Thomas&lt;/a&gt;&lt;/li&gt;
  248. &lt;/ul&gt;
  249. &lt;p&gt;Can’t decide where to start? The one with Ryan Trinkle might be my favorite.&lt;/p&gt;
  250. &lt;p&gt;Thanks to the Haskell Foundation and its sponsors for supporting this podcast (hosting, editing, transscription).&lt;/p&gt;</description>
  251.         <pubDate>Fri, 22 Dec 2023 10:04:42 +0100</pubDate>
  252.      </item>
  253.      <item>
  254.         <title>Joining the Lean FRO</title>
  255.         <link>https://www.joachim-breitner.de/blog/809-Joining_the_Lean_FRO</link>
  256.         <guid>https://www.joachim-breitner.de/blog/809-Joining_the_Lean_FRO</guid>
  257.         <comments>https://www.joachim-breitner.de/blog/809-Joining_the_Lean_FRO#comments</comments>
  258.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  259.         <description>&lt;p&gt;Tomorrow is going to be a new first day in a new job for me: I am joining the &lt;a href="https://lean-fro.org/"&gt;Lean FRO&lt;/a&gt;, and I’m excited.&lt;/p&gt;
  260. &lt;h3 id="what-is-lean"&gt;What is Lean?&lt;/h3&gt;
  261. &lt;p&gt;&lt;a href="https://lean-lang.org/about/"&gt;Lean&lt;/a&gt; is the new kid on the block of theorem provers.&lt;/p&gt;
  262. &lt;p&gt;It’s a pure functional programming language (like Haskell, with and on which I have worked a lot), but it’s dependently typed (which Haskell may be evolving to be as well, but rather slowly and carefully). It has a refreshing syntax, built on top of a rather good (I have been told, not an expert here) macro system.&lt;/p&gt;
  263. &lt;p&gt;As a dependently typed programming language, it is also a theorem prover, or proof assistant, and there exists already a lively community of mathematicians who started to formalize mathematics in a coherent library, creatively called &lt;a href="https://github.com/leanprover-community/mathlib4"&gt;mathlib&lt;/a&gt;.&lt;/p&gt;
  264. &lt;h3 id="what-is-a-fro"&gt;What is a FRO?&lt;/h3&gt;
  265. &lt;p&gt;A &lt;a href="https://www.convergentresearch.org/"&gt;Focused Research Organization&lt;/a&gt; has the organizational form of a small start up (small team, little overhead, a few years of runway), but its goals and measure for success are not commercial, as funding is provided by donors (in the case of the Lean FRO, the Simons Foundation International, the Alfred P. Sloan Foundation, and Richard Merkin). This allows us to build something that we believe is a contribution for the greater good, even though it’s not (or not yet) commercially interesting enough and does not fit other forms of funding (such as research grants) well. This is a very comfortable situation to be in.&lt;/p&gt;
  266. &lt;h3 id="why-am-i-excited"&gt;Why am I excited?&lt;/h3&gt;
  267. &lt;p&gt;To me, working on Lean seems to be the perfect mix: I have been working on language implementation for about a decade now, and always with a preference for functional languages. Add to that my interest in theorem proving, where I have used Isabelle and Coq so far, and played with Agda and others. So technically, clearly up my alley.&lt;/p&gt;
  268. &lt;p&gt;Furthermore, the language isn’t too old, and plenty of interesting things are simply still to do, rather than tried before. The ecosystem is still evolving, so there is a good chance to have some impact.&lt;/p&gt;
  269. &lt;p&gt;On the other hand, the language isn’t too young either. It is no longer an open question whether we will have users: we have them already, they hang out on &lt;a href="https://leanprover.zulipchat.com/"&gt;zulip&lt;/a&gt;, so if I improve something, there is likely someone going to be happy about it, which is great. And the community seems to be welcoming and full of nice people.&lt;/p&gt;
  270. &lt;p&gt;Finally, &lt;a href="https://github.com/leanprover-community/mathlib4"&gt;this library of mathematics&lt;/a&gt; that these users are building is itself an amazing artifact: Lots of math in a consistent, machine-readable, maintained, documented, checked form! With a little bit of optimism I can imagine this changing how math research and education will be done in the future. It could be for math what Wikipedia is for encyclopedic knowledge and OpenStreetMap for maps – and the thought of facilitating that excites me.&lt;/p&gt;
  271. &lt;p&gt;With this new job I find that when I am telling friends and colleagues about it, I do not hesitate or hedge when asked why I am doing this. This is a good sign.&lt;/p&gt;
  272. &lt;h3 id="what-will-i-be-doing"&gt;What will I be doing?&lt;/h3&gt;
  273. &lt;p&gt;We’ll see what main tasks I’ll get to tackle initially, but knowing myself, I expect I’ll get broadly involved.&lt;/p&gt;
  274. &lt;p&gt;To get up to speed I started playing around with a few things already, and for example created &lt;a href="https://loogle.lean-lang.org/"&gt;Loogle&lt;/a&gt;, a Mathlib search engine inspired by Haskell’s &lt;a href="https://hoogle.haskell.org/"&gt;Hoogle&lt;/a&gt;, including a Zulip bot integration. This seems to be useful and quite well received, so I’ll continue maintaining that.&lt;/p&gt;
  275. &lt;p&gt;Expect more about this and other contributions here in the future.&lt;/p&gt;</description>
  276.         <pubDate>Wed, 01 Nov 2023 21:47:06 +0100</pubDate>
  277.      </item>
  278.      <item>
  279.         <title>Squash your Github PRs with one click</title>
  280.         <link>https://www.joachim-breitner.de/blog/808-Squash_your_Github_PRs_with_one_click</link>
  281.         <guid>https://www.joachim-breitner.de/blog/808-Squash_your_Github_PRs_with_one_click</guid>
  282.         <comments>https://www.joachim-breitner.de/blog/808-Squash_your_Github_PRs_with_one_click#comments</comments>
  283.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  284.         <description>&lt;p&gt;TL;DR: Squash your PRs with one click at &lt;a href="https://squasher.nomeata.de/" class="uri"&gt;https://squasher.nomeata.de/&lt;/a&gt;.&lt;/p&gt;
  285. &lt;p&gt;Very recently I got this response from the project maintainer at a pull request I contributed: “Thanks, approved, please squash so that I can merge.”&lt;/p&gt;
  286. &lt;p&gt;It’s nice that my contribution can go it, but why did the maintainer not just press the “Squash and merge button”, and instead adds the this unnecessary roundtrip to the process? Anyways, maintainers make the rules, so I play by them. But unlike the maintainer, who can squash-and-merge with just one click, squashing the PR’s branch is surprisingly laberous: Github does not allow you to do that via the Web UI (and hence on mobile), and it seems you are expected to go to your computer and juggle with &lt;code&gt;git rebase --interactive&lt;/code&gt;.&lt;/p&gt;
  287. &lt;p&gt;I found this rather annoying, so I created &lt;a href="https://squasher.nomeata.de/"&gt;&lt;strong&gt;Squasher&lt;/strong&gt;&lt;/a&gt;, a simple service that will squash your branch for you. There is no configuration, just paste the PR url. It will use the PR title and body as the commit message (which is obviously the right way™), and create the commit in your name:&lt;/p&gt;
  288. &lt;figure&gt;
  289. &lt;img src="/various/squasher.png" alt="Squasher in action"/&gt;
  290. &lt;figcaption aria-hidden="true"&gt;Squasher in action&lt;/figcaption&gt;
  291. &lt;/figure&gt;
  292. &lt;p&gt;If you find this useful, or found it to be buggy, let me know. The code is at &lt;a href="https://github.com/nomeata/squasher" class="uri"&gt;https://github.com/nomeata/squasher&lt;/a&gt; if you are curious about it.&lt;/p&gt;</description>
  293.         <pubDate>Sun, 29 Oct 2023 22:46:56 +0100</pubDate>
  294.      </item>
  295.      <item>
  296.         <title>Left recursive parser combinators via sharing</title>
  297.         <link>https://www.joachim-breitner.de/blog/807-Left_recursive_parser_combinators_via_sharing</link>
  298.         <guid>https://www.joachim-breitner.de/blog/807-Left_recursive_parser_combinators_via_sharing</guid>
  299.         <comments>https://www.joachim-breitner.de/blog/807-Left_recursive_parser_combinators_via_sharing#comments</comments>
  300.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  301.         <description>&lt;p&gt;At this year’s &lt;a href="https://icfp23.sigplan.org/"&gt;ICFP in Seattle&lt;/a&gt; I gave a talk about my &lt;a href="https://hackage.haskell.org/package/rec-def"&gt;rec-def&lt;/a&gt; Haskell library, which I have blogged about before here. While my &lt;a href="https://doi.org/10.1145/3607853"&gt;functional pearl paper&lt;/a&gt; focuses on a concrete use-case and the tricks of the implementation, in my talk I put the emphasis on the high-level idea: it beholds of a declarative lazy functional like Haskell that recursive equations just work whenever they describe a (unique) solution. Like in the paper, I used equations between sets as the running example, and only conjectured that it should also work for other domains, in particular parser combinators.&lt;/p&gt;
  302. &lt;p&gt;Naturally, someone called my bluff and asked if I actually tried it. I had not, but I should have, because it works nicely and is actually more straight-forward than with sets. I wrote up a prototype and showed it off a few days later as a lightning talk at &lt;a href="https://icfp23.sigplan.org/home/haskellsymp-2023"&gt;Haskell Symposium&lt;/a&gt;; here is the write up that goes along with that.&lt;/p&gt;
  303. &lt;h3 id="parser-combinators"&gt;Parser combinators&lt;/h3&gt;
  304. &lt;p&gt;Parser combinators are libraries that provide little functions (combinators) that you compose to define your parser directly in your programming language, as opposed to using external tools that read some grammar description and generate parser code, and are quite popular in Haskell (e.g. &lt;a href="https://hackage.haskell.org/package/parsec"&gt;parsec&lt;/a&gt;, &lt;a href="https://hackage.haskell.org/package/attoparsec"&gt;attoparsec&lt;/a&gt;, &lt;a href="https://hackage.haskell.org/package/megaparsec"&gt;megaparsec&lt;/a&gt;).&lt;/p&gt;
  305. &lt;p&gt;Let us define a little parser that recognizes sequences of &lt;code&gt;a&lt;/code&gt;s:&lt;/p&gt;
  306. &lt;div class="sourceCode" id="cb1"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb1-1"&gt;&lt;a href="#cb1-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; aaa &lt;span class="ot"&gt;=&lt;/span&gt; tok &lt;span class="ch"&gt;'a'&lt;/span&gt; &lt;span class="op"&gt;*&gt;&lt;/span&gt; aaa &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; ()&lt;/span&gt;
  307. &lt;span id="cb1-2"&gt;&lt;a href="#cb1-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aaaa"&lt;/span&gt;&lt;/span&gt;
  308. &lt;span id="cb1-3"&gt;&lt;a href="#cb1-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Just&lt;/span&gt; ()&lt;/span&gt;
  309. &lt;span id="cb1-4"&gt;&lt;a href="#cb1-4" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aabaa"&lt;/span&gt;&lt;/span&gt;
  310. &lt;span id="cb1-5"&gt;&lt;a href="#cb1-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Nothing&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  311. &lt;h3 id="left-recursion"&gt;Left-recursion&lt;/h3&gt;
  312. &lt;p&gt;This works nicely, but just because we were lucky: We wrote the parser to recurse on the right (of the &lt;code&gt;*&gt;&lt;/code&gt;), and this happens to work. If we put the recursive call first, it doesn’t anymore:&lt;/p&gt;
  313. &lt;div class="sourceCode" id="cb2"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb2-1"&gt;&lt;a href="#cb2-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; aaa &lt;span class="ot"&gt;=&lt;/span&gt; aaa &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; tok &lt;span class="ch"&gt;'a'&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; ()&lt;/span&gt;
  314. &lt;span id="cb2-2"&gt;&lt;a href="#cb2-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aaaa"&lt;/span&gt;&lt;/span&gt;
  315. &lt;span id="cb2-3"&gt;&lt;a href="#cb2-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="op"&gt;^&lt;/span&gt;&lt;span class="dt"&gt;CInterrupted&lt;/span&gt;&lt;span class="op"&gt;.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  316. &lt;p&gt;This is a well-known problem (see for example &lt;a href="https://doi.org/10.1145/3471874.3472984"&gt;Nicolas Wu’s overview paper&lt;/a&gt;), all the common parser combinator libraries cannot handle it and the usual advise is to refactor your grammar to avoid left recursion.&lt;/p&gt;
  317. &lt;p&gt;But there are some libraries that can handle left recursion, at least with a little help from the programmer. I found two variations:&lt;/p&gt;
  318. &lt;ul&gt;
  319. &lt;li&gt;&lt;p&gt;The library provides an explicit fix point combinator, and as long as that is used, left-recursion works. This is for example described by &lt;a href="https://link.springer.com/chapter/10.1007/978-3-540-77442-6_12"&gt;Frost, Hafiz and Callaghan&lt;/a&gt; by, and (of course) &lt;a href="https://okmij.org/ftp/Haskell/LeftRecursion.hs"&gt;Oleg Kiselyov&lt;/a&gt; has an implementation of this too.&lt;/p&gt;&lt;/li&gt;
  320. &lt;li&gt;&lt;p&gt;The library expects explicit labels on recursive productions, so that the library can recognize left-recursion. I found an implementation of this idea in the &lt;a href="https://hackage.haskell.org/package/Agda-2.6.3/docs/Agda-Utils-Parser-MemoisedCPS.html"&gt;&lt;code&gt;Agda.Utils.Parser.MemoisedCPS&lt;/code&gt;&lt;/a&gt; module in the Agda code, the &lt;a href="https://hackage.haskell.org/package/gll-0.4.1.0/docs/GLL-Combinators-Interface.html"&gt;&lt;code&gt;gll&lt;/code&gt; library&lt;/a&gt; seems to follow this style and &lt;a href="https://discourse.haskell.org/t/reusing-haskells-binding-when-defining-context-free-grammars/5960?u=nomeata"&gt;Jaro discusses it as well&lt;/a&gt;.&lt;/p&gt;&lt;/li&gt;
  321. &lt;/ul&gt;
  322. &lt;p&gt;I took the module from the Agda source and simplified a bit for the purposes of this demonstration (&lt;a href="https://github.com/nomeata/left-rec-parse/blob/master/Parser.hs"&gt;&lt;code&gt;Parser.hs&lt;/code&gt;&lt;/a&gt;). Indeed, I can make the left-recursive grammar work:&lt;/p&gt;
  323. &lt;div class="sourceCode" id="cb3"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb3-1"&gt;&lt;a href="#cb3-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; aaa &lt;span class="ot"&gt;=&lt;/span&gt; memoise &lt;span class="st"&gt;":-)"&lt;/span&gt; &lt;span class="op"&gt;$&lt;/span&gt; aaa &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; tok &lt;span class="ch"&gt;'a'&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; ()&lt;/span&gt;
  324. &lt;span id="cb3-2"&gt;&lt;a href="#cb3-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aaaa"&lt;/span&gt;&lt;/span&gt;
  325. &lt;span id="cb3-3"&gt;&lt;a href="#cb3-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Just&lt;/span&gt; ()&lt;/span&gt;
  326. &lt;span id="cb3-4"&gt;&lt;a href="#cb3-4" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aabaa"&lt;/span&gt;&lt;/span&gt;
  327. &lt;span id="cb3-5"&gt;&lt;a href="#cb3-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Nothing&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  328. &lt;p&gt;It does not matter what I pass to &lt;code&gt;memoise&lt;/code&gt;, as long as I do not pass the same key when memoising a different parser.&lt;/p&gt;
  329. &lt;p&gt;For reference, an excerpt of the the API of &lt;code&gt;Parser&lt;/code&gt;:&lt;/p&gt;
  330. &lt;div class="sourceCode" id="cb4"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb4-1"&gt;&lt;a href="#cb4-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;data&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok a &lt;span class="co"&gt;-- k is type of keys, tok type of tokens (e.g. Char)&lt;/span&gt;&lt;/span&gt;
  331. &lt;span id="cb4-2"&gt;&lt;a href="#cb4-2" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Functor&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; k tok)&lt;/span&gt;
  332. &lt;span id="cb4-3"&gt;&lt;a href="#cb4-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Applicative&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; k tok)&lt;/span&gt;
  333. &lt;span id="cb4-4"&gt;&lt;a href="#cb4-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Alternative&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; k tok)&lt;/span&gt;
  334. &lt;span id="cb4-5"&gt;&lt;a href="#cb4-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Monad&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; k tok)&lt;/span&gt;
  335. &lt;span id="cb4-6"&gt;&lt;a href="#cb4-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;parse ::&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; [tok] &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Maybe&lt;/span&gt; a&lt;/span&gt;
  336. &lt;span id="cb4-7"&gt;&lt;a href="#cb4-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;sat ::&lt;/span&gt; (tok &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Bool&lt;/span&gt;) &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok tok&lt;/span&gt;
  337. &lt;span id="cb4-8"&gt;&lt;a href="#cb4-8" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;tok ::&lt;/span&gt; &lt;span class="dt"&gt;Eq&lt;/span&gt; tok &lt;span class="ot"&gt;=&gt;&lt;/span&gt; tok &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok tok&lt;/span&gt;
  338. &lt;span id="cb4-9"&gt;&lt;a href="#cb4-9" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;memoise ::&lt;/span&gt; &lt;span class="dt"&gt;Ord&lt;/span&gt; k &lt;span class="ot"&gt;=&gt;&lt;/span&gt; k &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; k tok a&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  339. &lt;h3 id="left-recursion-through-sharing"&gt;Left-recursion through sharing&lt;/h3&gt;
  340. &lt;p&gt;To follow the agenda set out in my talk, I now want to wrap that parser in a way that relieves me from having to insert the calls to &lt;code&gt;memoise&lt;/code&gt;. To start, I import that parser qualified, define a newtype around it, and start lifting some of the functions:&lt;/p&gt;
  341. &lt;div class="sourceCode" id="cb5"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb5-1"&gt;&lt;a href="#cb5-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;import&lt;/span&gt; &lt;span class="kw"&gt;qualified&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; &lt;span class="kw"&gt;as&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt;&lt;/span&gt;
  342. &lt;span id="cb5-2"&gt;&lt;a href="#cb5-2" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  343. &lt;span id="cb5-3"&gt;&lt;a href="#cb5-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;newtype&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; tok a &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; {&lt;span class="ot"&gt; unP ::&lt;/span&gt; &lt;span class="dt"&gt;P.Parser&lt;/span&gt; &lt;span class="dt"&gt;Unique&lt;/span&gt; tok a }&lt;/span&gt;
  344. &lt;span id="cb5-4"&gt;&lt;a href="#cb5-4" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  345. &lt;span id="cb5-5"&gt;&lt;a href="#cb5-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;parse ::&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; tok a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; [tok] &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Maybe&lt;/span&gt; a&lt;/span&gt;
  346. &lt;span id="cb5-6"&gt;&lt;a href="#cb5-6" aria-hidden="true" tabindex="-1"/&gt;parses (&lt;span class="dt"&gt;MkP&lt;/span&gt; p) &lt;span class="ot"&gt;=&lt;/span&gt; P.parse p&lt;/span&gt;
  347. &lt;span id="cb5-7"&gt;&lt;a href="#cb5-7" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  348. &lt;span id="cb5-8"&gt;&lt;a href="#cb5-8" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;sat ::&lt;/span&gt; &lt;span class="dt"&gt;Typeable&lt;/span&gt; tok &lt;span class="ot"&gt;=&gt;&lt;/span&gt; (tok &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Bool&lt;/span&gt;) &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; tok tok&lt;/span&gt;
  349. &lt;span id="cb5-9"&gt;&lt;a href="#cb5-9" aria-hidden="true" tabindex="-1"/&gt;sat p &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; (P.sat p)&lt;/span&gt;
  350. &lt;span id="cb5-10"&gt;&lt;a href="#cb5-10" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  351. &lt;span id="cb5-11"&gt;&lt;a href="#cb5-11" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;tok ::&lt;/span&gt; &lt;span class="dt"&gt;Eq&lt;/span&gt; tok &lt;span class="ot"&gt;=&gt;&lt;/span&gt; tok &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; tok tok&lt;/span&gt;
  352. &lt;span id="cb5-12"&gt;&lt;a href="#cb5-12" aria-hidden="true" tabindex="-1"/&gt;tok t &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; (P.tok t)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  353. &lt;p&gt;So far, nothing interesting had to happen, because so far I cannot build recursive parsers. The first interesting combinator that allows me to do that is &lt;code&gt;&amp;lt;*&gt;&lt;/code&gt; from the &lt;code&gt;Applicative&lt;/code&gt; class, so I should use &lt;code&gt;memoise&lt;/code&gt; there. The question is: Where does the unique key come from?&lt;/p&gt;
  354. &lt;h3 id="proprioception"&gt;Proprioception&lt;/h3&gt;
  355. &lt;p&gt;As with the rec-def library, pure code won’t do, and I have to get my hands dirty: I really want a fresh unique label out of thin air. To that end, I define the following combinator, with naming aided by Richard Eisenberg:&lt;/p&gt;
  356. &lt;div class="sourceCode" id="cb6"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb6-1"&gt;&lt;a href="#cb6-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;propriocept ::&lt;/span&gt; (&lt;span class="dt"&gt;Unique&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; a) &lt;span class="ot"&gt;-&gt;&lt;/span&gt; a&lt;/span&gt;
  357. &lt;span id="cb6-2"&gt;&lt;a href="#cb6-2" aria-hidden="true" tabindex="-1"/&gt;propriocept f &lt;span class="ot"&gt;=&lt;/span&gt; unsafePerformIO &lt;span class="op"&gt;$&lt;/span&gt; f &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; newUnique&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  358. &lt;p&gt;A thunk defined with &lt;code&gt;propriocept&lt;/code&gt; will know about it’s own identity, and will be able to tell itself apart from other such thunks. This gives us a form of observable sharing, precisely what we need. But before we return to our parser combinators, let us briefly explore this combinator.&lt;/p&gt;
  359. &lt;p&gt;Using &lt;code&gt;propriocept&lt;/code&gt; I can define an operation &lt;code&gt;cons :: [Int] -&gt; [Int]&lt;/code&gt; that records (the hash of) this &lt;code&gt;Unique&lt;/code&gt; in the list:&lt;/p&gt;
  360. &lt;div class="sourceCode" id="cb7"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb7-1"&gt;&lt;a href="#cb7-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; cons xs &lt;span class="ot"&gt;=&lt;/span&gt; propriocept (\x &lt;span class="ot"&gt;-&gt;&lt;/span&gt; hashUnique x &lt;span class="op"&gt;:&lt;/span&gt; xs)&lt;/span&gt;
  361. &lt;span id="cb7-2"&gt;&lt;a href="#cb7-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="op"&gt;:&lt;/span&gt;t cons&lt;/span&gt;
  362. &lt;span id="cb7-3"&gt;&lt;a href="#cb7-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;cons ::&lt;/span&gt; [&lt;span class="dt"&gt;Int&lt;/span&gt;] &lt;span class="ot"&gt;-&gt;&lt;/span&gt; [&lt;span class="dt"&gt;Int&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  363. &lt;p&gt;This lets us see the identity of a list cell, that is, of the concrete object in memory.&lt;/p&gt;
  364. &lt;p&gt;Naturally, if we construct a finite list, each list cell is different:&lt;/p&gt;
  365. &lt;div class="sourceCode" id="cb8"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb8-1"&gt;&lt;a href="#cb8-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; cons (cons (cons []))&lt;/span&gt;
  366. &lt;span id="cb8-2"&gt;&lt;a href="#cb8-2" aria-hidden="true" tabindex="-1"/&gt;[&lt;span class="dv"&gt;1&lt;/span&gt;,&lt;span class="dv"&gt;2&lt;/span&gt;,&lt;span class="dv"&gt;3&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  367. &lt;p&gt;And if we do that again, we see that fresh list cells are allocated:&lt;/p&gt;
  368. &lt;div class="sourceCode" id="cb9"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb9-1"&gt;&lt;a href="#cb9-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; cons (cons (cons []))&lt;/span&gt;
  369. &lt;span id="cb9-2"&gt;&lt;a href="#cb9-2" aria-hidden="true" tabindex="-1"/&gt;[&lt;span class="dv"&gt;4&lt;/span&gt;,&lt;span class="dv"&gt;5&lt;/span&gt;,&lt;span class="dv"&gt;6&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  370. &lt;p&gt;We can create an infinite list; if we do it without sharing, every cell is separate:&lt;/p&gt;
  371. &lt;div class="sourceCode" id="cb10"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb10-1"&gt;&lt;a href="#cb10-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="fu"&gt;take&lt;/span&gt; &lt;span class="dv"&gt;20&lt;/span&gt; (acyclic &lt;span class="dv"&gt;0&lt;/span&gt;)&lt;/span&gt;
  372. &lt;span id="cb10-2"&gt;&lt;a href="#cb10-2" aria-hidden="true" tabindex="-1"/&gt;[&lt;span class="dv"&gt;7&lt;/span&gt;,&lt;span class="dv"&gt;8&lt;/span&gt;,&lt;span class="dv"&gt;9&lt;/span&gt;,&lt;span class="dv"&gt;10&lt;/span&gt;,&lt;span class="dv"&gt;11&lt;/span&gt;,&lt;span class="dv"&gt;12&lt;/span&gt;,&lt;span class="dv"&gt;13&lt;/span&gt;,&lt;span class="dv"&gt;14&lt;/span&gt;,&lt;span class="dv"&gt;15&lt;/span&gt;,&lt;span class="dv"&gt;16&lt;/span&gt;,&lt;span class="dv"&gt;17&lt;/span&gt;,&lt;span class="dv"&gt;18&lt;/span&gt;,&lt;span class="dv"&gt;19&lt;/span&gt;,&lt;span class="dv"&gt;20&lt;/span&gt;,&lt;span class="dv"&gt;21&lt;/span&gt;,&lt;span class="dv"&gt;22&lt;/span&gt;,&lt;span class="dv"&gt;23&lt;/span&gt;,&lt;span class="dv"&gt;24&lt;/span&gt;,&lt;span class="dv"&gt;25&lt;/span&gt;,&lt;span class="dv"&gt;26&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  373. &lt;p&gt;but if we tie the knot using sharing, all the cells in the list are actually the same:&lt;/p&gt;
  374. &lt;div class="sourceCode" id="cb11"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb11-1"&gt;&lt;a href="#cb11-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; cyclic &lt;span class="ot"&gt;=&lt;/span&gt; cons cyclic&lt;/span&gt;
  375. &lt;span id="cb11-2"&gt;&lt;a href="#cb11-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="fu"&gt;take&lt;/span&gt; &lt;span class="dv"&gt;20&lt;/span&gt; cyclic&lt;/span&gt;
  376. &lt;span id="cb11-3"&gt;&lt;a href="#cb11-3" aria-hidden="true" tabindex="-1"/&gt;[&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;,&lt;span class="dv"&gt;27&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  377. &lt;p&gt;We can achieve the same using &lt;a href="https://hackage.haskell.org/package/base/docs/Data-Function.html#v:fix"&gt;&lt;code&gt;fix&lt;/code&gt; from &lt;code&gt;Data.Function&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;
  378. &lt;div class="sourceCode" id="cb12"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb12-1"&gt;&lt;a href="#cb12-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;import&lt;/span&gt; &lt;span class="dt"&gt;Data.Function&lt;/span&gt;&lt;/span&gt;
  379. &lt;span id="cb12-2"&gt;&lt;a href="#cb12-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="fu"&gt;take&lt;/span&gt; &lt;span class="dv"&gt;20&lt;/span&gt; (fix cons)&lt;/span&gt;
  380. &lt;span id="cb12-3"&gt;&lt;a href="#cb12-3" aria-hidden="true" tabindex="-1"/&gt;[&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;,&lt;span class="dv"&gt;28&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  381. &lt;p&gt;I explore these heap structures more visually in a &lt;a href="https://www.youtube.com/playlist?list=PL4FcLyLhO9jggmkqJyJ2i9pCSiDpwKiVu"&gt;series of screencasts&lt;/a&gt;.&lt;/p&gt;
  382. &lt;p&gt;So with &lt;code&gt;propriocept&lt;/code&gt; we can distinguish different heap objects, and also recognize when we come across the same heap object again.&lt;/p&gt;
  383. &lt;h3 id="left-recursion-through-sharing-cont."&gt;Left-recursion through sharing (cont.)&lt;/h3&gt;
  384. &lt;p&gt;With that we return to our parser. We define a smart constructor for the new &lt;code&gt;Parser&lt;/code&gt; that passes the unique from &lt;code&gt;propriocept&lt;/code&gt; to the underlying parser’s &lt;code&gt;memoise&lt;/code&gt; function:&lt;/p&gt;
  385. &lt;div class="sourceCode" id="cb13"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb13-1"&gt;&lt;a href="#cb13-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;withMemo ::&lt;/span&gt; &lt;span class="dt"&gt;P.Parser&lt;/span&gt; &lt;span class="dt"&gt;Unique&lt;/span&gt; tok a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; tok a&lt;/span&gt;
  386. &lt;span id="cb13-2"&gt;&lt;a href="#cb13-2" aria-hidden="true" tabindex="-1"/&gt;withMemo p &lt;span class="ot"&gt;=&lt;/span&gt; propriocept &lt;span class="op"&gt;$&lt;/span&gt; \u &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; &lt;span class="op"&gt;$&lt;/span&gt; P.memoise u p&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  387. &lt;p&gt;If we now use this in the definition of all possibly recursive parsers, then the necessary calls to &lt;code&gt;memoise&lt;/code&gt; will be in place:&lt;/p&gt;
  388. &lt;div class="sourceCode" id="cb14"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb14-1"&gt;&lt;a href="#cb14-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Functor&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; tok) &lt;span class="kw"&gt;where&lt;/span&gt;&lt;/span&gt;
  389. &lt;span id="cb14-2"&gt;&lt;a href="#cb14-2" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="fu"&gt;fmap&lt;/span&gt; f p &lt;span class="ot"&gt;=&lt;/span&gt; withMemo (&lt;span class="fu"&gt;fmap&lt;/span&gt; f (unP p))&lt;/span&gt;
  390. &lt;span id="cb14-3"&gt;&lt;a href="#cb14-3" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  391. &lt;span id="cb14-4"&gt;&lt;a href="#cb14-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Applicative&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; tok) &lt;span class="kw"&gt;where&lt;/span&gt;&lt;/span&gt;
  392. &lt;span id="cb14-5"&gt;&lt;a href="#cb14-5" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="fu"&gt;pure&lt;/span&gt; x &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; (&lt;span class="fu"&gt;pure&lt;/span&gt; x)&lt;/span&gt;
  393. &lt;span id="cb14-6"&gt;&lt;a href="#cb14-6" aria-hidden="true" tabindex="-1"/&gt;  p1 &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; p2 &lt;span class="ot"&gt;=&lt;/span&gt; withMemo (unP p1 &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; unP p2)&lt;/span&gt;
  394. &lt;span id="cb14-7"&gt;&lt;a href="#cb14-7" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  395. &lt;span id="cb14-8"&gt;&lt;a href="#cb14-8" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Alternative&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; tok) &lt;span class="kw"&gt;where&lt;/span&gt;&lt;/span&gt;
  396. &lt;span id="cb14-9"&gt;&lt;a href="#cb14-9" aria-hidden="true" tabindex="-1"/&gt;  empty &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;MkP&lt;/span&gt; empty&lt;/span&gt;
  397. &lt;span id="cb14-10"&gt;&lt;a href="#cb14-10" aria-hidden="true" tabindex="-1"/&gt;  p1 &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; p2 &lt;span class="ot"&gt;=&lt;/span&gt; withMemo (unP p1 &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; unP p2)&lt;/span&gt;
  398. &lt;span id="cb14-11"&gt;&lt;a href="#cb14-11" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  399. &lt;span id="cb14-12"&gt;&lt;a href="#cb14-12" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;instance&lt;/span&gt; &lt;span class="dt"&gt;Monad&lt;/span&gt; (&lt;span class="dt"&gt;Parser&lt;/span&gt; tok) &lt;span class="kw"&gt;where&lt;/span&gt;&lt;/span&gt;
  400. &lt;span id="cb14-13"&gt;&lt;a href="#cb14-13" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="fu"&gt;return&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt;&lt;/span&gt;
  401. &lt;span id="cb14-14"&gt;&lt;a href="#cb14-14" aria-hidden="true" tabindex="-1"/&gt;  p1 &lt;span class="op"&gt;&gt;&gt;=&lt;/span&gt; f &lt;span class="ot"&gt;=&lt;/span&gt; withMemo &lt;span class="op"&gt;$&lt;/span&gt; unP p1 &lt;span class="op"&gt;&gt;&gt;=&lt;/span&gt; unP &lt;span class="op"&gt;.&lt;/span&gt; f&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  402. &lt;p&gt;And indeed, it works (see &lt;a href="https://github.com/nomeata/left-rec-parse/blob/master/RParser.hs"&gt;&lt;code&gt;RParser.hs&lt;/code&gt;&lt;/a&gt; for the full code):&lt;/p&gt;
  403. &lt;div class="sourceCode" id="cb15"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb15-1"&gt;&lt;a href="#cb15-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="kw"&gt;let&lt;/span&gt; aaa &lt;span class="ot"&gt;=&lt;/span&gt; aaa &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; tok &lt;span class="ch"&gt;'a'&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; ()&lt;/span&gt;
  404. &lt;span id="cb15-2"&gt;&lt;a href="#cb15-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aaaa"&lt;/span&gt;&lt;/span&gt;
  405. &lt;span id="cb15-3"&gt;&lt;a href="#cb15-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Just&lt;/span&gt; ()&lt;/span&gt;
  406. &lt;span id="cb15-4"&gt;&lt;a href="#cb15-4" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse aaa &lt;span class="st"&gt;"aabaa"&lt;/span&gt;&lt;/span&gt;
  407. &lt;span id="cb15-5"&gt;&lt;a href="#cb15-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Nothing&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  408. &lt;h3 id="a-larger-example"&gt;A larger example&lt;/h3&gt;
  409. &lt;p&gt;Let us try this on a larger example, and parse (simple) BNF grammars. Here is a data type describing them&lt;/p&gt;
  410. &lt;div class="sourceCode" id="cb16"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb16-1"&gt;&lt;a href="#cb16-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  411. &lt;span id="cb16-2"&gt;&lt;a href="#cb16-2" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;RuleRhs&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; [&lt;span class="dt"&gt;Seq&lt;/span&gt;]&lt;/span&gt;
  412. &lt;span id="cb16-3"&gt;&lt;a href="#cb16-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;Seq&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; [&lt;span class="dt"&gt;Atom&lt;/span&gt;]&lt;/span&gt;
  413. &lt;span id="cb16-4"&gt;&lt;a href="#cb16-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;data&lt;/span&gt; &lt;span class="dt"&gt;Atom&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;Lit&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt; &lt;span class="op"&gt;|&lt;/span&gt; &lt;span class="dt"&gt;NonTerm&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt; &lt;span class="kw"&gt;deriving&lt;/span&gt; &lt;span class="dt"&gt;Show&lt;/span&gt;&lt;/span&gt;
  414. &lt;span id="cb16-5"&gt;&lt;a href="#cb16-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;Rule&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; (&lt;span class="dt"&gt;Ident&lt;/span&gt;, &lt;span class="dt"&gt;RuleRhs&lt;/span&gt;)&lt;/span&gt;
  415. &lt;span id="cb16-6"&gt;&lt;a href="#cb16-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;BNF&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; [&lt;span class="dt"&gt;Rule&lt;/span&gt;]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  416. &lt;p&gt;For the concrete syntax, I’d like to be able to parse something like&lt;/p&gt;
  417. &lt;div class="sourceCode" id="cb17"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb17-1"&gt;&lt;a href="#cb17-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;numExp ::&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  418. &lt;span id="cb17-2"&gt;&lt;a href="#cb17-2" aria-hidden="true" tabindex="-1"/&gt;numExp &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;unlines&lt;/span&gt;&lt;/span&gt;
  419. &lt;span id="cb17-3"&gt;&lt;a href="#cb17-3" aria-hidden="true" tabindex="-1"/&gt;    [ &lt;span class="st"&gt;"term   := sum;"&lt;/span&gt;&lt;/span&gt;
  420. &lt;span id="cb17-4"&gt;&lt;a href="#cb17-4" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"pdigit := '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9';"&lt;/span&gt;&lt;/span&gt;
  421. &lt;span id="cb17-5"&gt;&lt;a href="#cb17-5" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"digit  := '0' | pdigit;"&lt;/span&gt;&lt;/span&gt;
  422. &lt;span id="cb17-6"&gt;&lt;a href="#cb17-6" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"pnum   := pdigit | pnum digit;"&lt;/span&gt;&lt;/span&gt;
  423. &lt;span id="cb17-7"&gt;&lt;a href="#cb17-7" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"num    := '0' | pnum;"&lt;/span&gt;&lt;/span&gt;
  424. &lt;span id="cb17-8"&gt;&lt;a href="#cb17-8" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"prod   := atom | atom '*' prod;"&lt;/span&gt;&lt;/span&gt;
  425. &lt;span id="cb17-9"&gt;&lt;a href="#cb17-9" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"sum    := prod | prod '+' sum;"&lt;/span&gt;&lt;/span&gt;
  426. &lt;span id="cb17-10"&gt;&lt;a href="#cb17-10" aria-hidden="true" tabindex="-1"/&gt;    , &lt;span class="st"&gt;"atom   := num | '(' term ')';"&lt;/span&gt;&lt;/span&gt;
  427. &lt;span id="cb17-11"&gt;&lt;a href="#cb17-11" aria-hidden="true" tabindex="-1"/&gt;    ]&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  428. &lt;p&gt;so here is a possible parser; mostly straight-forward use of parser combinator:&lt;/p&gt;
  429. &lt;div class="sourceCode" id="cb18"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb18-1"&gt;&lt;a href="#cb18-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="kw"&gt;type&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;Parser&lt;/span&gt; &lt;span class="dt"&gt;Char&lt;/span&gt;&lt;/span&gt;
  430. &lt;span id="cb18-2"&gt;&lt;a href="#cb18-2" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  431. &lt;span id="cb18-3"&gt;&lt;a href="#cb18-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;snoc ::&lt;/span&gt; [a] &lt;span class="ot"&gt;-&gt;&lt;/span&gt; a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; [a]&lt;/span&gt;
  432. &lt;span id="cb18-4"&gt;&lt;a href="#cb18-4" aria-hidden="true" tabindex="-1"/&gt;snoc xs x &lt;span class="ot"&gt;=&lt;/span&gt; xs &lt;span class="op"&gt;++&lt;/span&gt; [x]&lt;/span&gt;
  433. &lt;span id="cb18-5"&gt;&lt;a href="#cb18-5" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  434. &lt;span id="cb18-6"&gt;&lt;a href="#cb18-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;l ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a&lt;/span&gt;
  435. &lt;span id="cb18-7"&gt;&lt;a href="#cb18-7" aria-hidden="true" tabindex="-1"/&gt;l p &lt;span class="ot"&gt;=&lt;/span&gt; p &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; l p &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; sat &lt;span class="fu"&gt;isSpace&lt;/span&gt;&lt;/span&gt;
  436. &lt;span id="cb18-8"&gt;&lt;a href="#cb18-8" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;quote ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;Char&lt;/span&gt;&lt;/span&gt;
  437. &lt;span id="cb18-9"&gt;&lt;a href="#cb18-9" aria-hidden="true" tabindex="-1"/&gt;quote &lt;span class="ot"&gt;=&lt;/span&gt; tok &lt;span class="ch"&gt;'\''&lt;/span&gt;&lt;/span&gt;
  438. &lt;span id="cb18-10"&gt;&lt;a href="#cb18-10" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;quoted ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a&lt;/span&gt;
  439. &lt;span id="cb18-11"&gt;&lt;a href="#cb18-11" aria-hidden="true" tabindex="-1"/&gt;quoted p &lt;span class="ot"&gt;=&lt;/span&gt; quote &lt;span class="op"&gt;*&gt;&lt;/span&gt; p &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; quote&lt;/span&gt;
  440. &lt;span id="cb18-12"&gt;&lt;a href="#cb18-12" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;str ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  441. &lt;span id="cb18-13"&gt;&lt;a href="#cb18-13" aria-hidden="true" tabindex="-1"/&gt;str &lt;span class="ot"&gt;=&lt;/span&gt; some (sat (&lt;span class="fu"&gt;not&lt;/span&gt; &lt;span class="op"&gt;.&lt;/span&gt; (&lt;span class="op"&gt;==&lt;/span&gt; &lt;span class="ch"&gt;'\''&lt;/span&gt;)))&lt;/span&gt;
  442. &lt;span id="cb18-14"&gt;&lt;a href="#cb18-14" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;ident ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt;&lt;/span&gt;
  443. &lt;span id="cb18-15"&gt;&lt;a href="#cb18-15" aria-hidden="true" tabindex="-1"/&gt;ident &lt;span class="ot"&gt;=&lt;/span&gt; some (sat (\c &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="fu"&gt;isAlphaNum&lt;/span&gt; c &lt;span class="op"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="fu"&gt;isAscii&lt;/span&gt; c))&lt;/span&gt;
  444. &lt;span id="cb18-16"&gt;&lt;a href="#cb18-16" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;atom ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;Atom&lt;/span&gt;&lt;/span&gt;
  445. &lt;span id="cb18-17"&gt;&lt;a href="#cb18-17" aria-hidden="true" tabindex="-1"/&gt;atom &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="dt"&gt;Lit&lt;/span&gt;     &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; l (quoted str)&lt;/span&gt;
  446. &lt;span id="cb18-18"&gt;&lt;a href="#cb18-18" aria-hidden="true" tabindex="-1"/&gt;   &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="dt"&gt;NonTerm&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; l ident&lt;/span&gt;
  447. &lt;span id="cb18-19"&gt;&lt;a href="#cb18-19" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;eps ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; ()&lt;/span&gt;
  448. &lt;span id="cb18-20"&gt;&lt;a href="#cb18-20" aria-hidden="true" tabindex="-1"/&gt;eps &lt;span class="ot"&gt;=&lt;/span&gt; void &lt;span class="op"&gt;$&lt;/span&gt; l (tok &lt;span class="ch"&gt;'ε'&lt;/span&gt;)&lt;/span&gt;
  449. &lt;span id="cb18-21"&gt;&lt;a href="#cb18-21" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;sep ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; ()&lt;/span&gt;
  450. &lt;span id="cb18-22"&gt;&lt;a href="#cb18-22" aria-hidden="true" tabindex="-1"/&gt;sep &lt;span class="ot"&gt;=&lt;/span&gt; void &lt;span class="op"&gt;$&lt;/span&gt; some (sat &lt;span class="fu"&gt;isSpace&lt;/span&gt;)&lt;/span&gt;
  451. &lt;span id="cb18-23"&gt;&lt;a href="#cb18-23" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;sq ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;Seq&lt;/span&gt;&lt;/span&gt;
  452. &lt;span id="cb18-24"&gt;&lt;a href="#cb18-24" aria-hidden="true" tabindex="-1"/&gt;sq &lt;span class="ot"&gt;=&lt;/span&gt; []   &lt;span class="op"&gt;&amp;lt;$&lt;/span&gt; eps&lt;/span&gt;
  453. &lt;span id="cb18-25"&gt;&lt;a href="#cb18-25" aria-hidden="true" tabindex="-1"/&gt; &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; snoc &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; sq &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; sep &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; atom&lt;/span&gt;
  454. &lt;span id="cb18-26"&gt;&lt;a href="#cb18-26" aria-hidden="true" tabindex="-1"/&gt; &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; atom&lt;/span&gt;
  455. &lt;span id="cb18-27"&gt;&lt;a href="#cb18-27" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;ruleRhs ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;RuleRhs&lt;/span&gt;&lt;/span&gt;
  456. &lt;span id="cb18-28"&gt;&lt;a href="#cb18-28" aria-hidden="true" tabindex="-1"/&gt;ruleRhs &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; sq&lt;/span&gt;
  457. &lt;span id="cb18-29"&gt;&lt;a href="#cb18-29" aria-hidden="true" tabindex="-1"/&gt;      &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; snoc &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; ruleRhs &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; l (tok &lt;span class="ch"&gt;'|'&lt;/span&gt;) &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; sq&lt;/span&gt;
  458. &lt;span id="cb18-30"&gt;&lt;a href="#cb18-30" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;rule ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;Rule&lt;/span&gt;&lt;/span&gt;
  459. &lt;span id="cb18-31"&gt;&lt;a href="#cb18-31" aria-hidden="true" tabindex="-1"/&gt;rule &lt;span class="ot"&gt;=&lt;/span&gt; (,) &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; l ident &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; l (tok &lt;span class="ch"&gt;':'&lt;/span&gt; &lt;span class="op"&gt;*&gt;&lt;/span&gt; tok &lt;span class="ch"&gt;'='&lt;/span&gt;) &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; ruleRhs &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; l (tok &lt;span class="ch"&gt;';'&lt;/span&gt;)&lt;/span&gt;
  460. &lt;span id="cb18-32"&gt;&lt;a href="#cb18-32" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;bnf ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;BNF&lt;/span&gt;&lt;/span&gt;
  461. &lt;span id="cb18-33"&gt;&lt;a href="#cb18-33" aria-hidden="true" tabindex="-1"/&gt;bnf &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;pure&lt;/span&gt; &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; rule&lt;/span&gt;
  462. &lt;span id="cb18-34"&gt;&lt;a href="#cb18-34" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; snoc &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; bnf &lt;span class="op"&gt;&amp;lt;*&gt;&lt;/span&gt; rule&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  463. &lt;p&gt;I somewhat sillily used &lt;code&gt;snoc&lt;/code&gt; rather than &lt;code&gt;(:)&lt;/code&gt; to build my lists, just so that I can show off all the left-recursion in this grammar.&lt;/p&gt;
  464. &lt;h3 id="sharing-is-tricky"&gt;Sharing is tricky&lt;/h3&gt;
  465. &lt;p&gt;Let’s try it:&lt;/p&gt;
  466. &lt;div class="sourceCode" id="cb19"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb19-1"&gt;&lt;a href="#cb19-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse bnf numExp&lt;/span&gt;
  467. &lt;span id="cb19-2"&gt;&lt;a href="#cb19-2" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="op"&gt;^&lt;/span&gt;&lt;span class="dt"&gt;CInterrupted&lt;/span&gt;&lt;span class="op"&gt;.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  468. &lt;p&gt;What a pity, it does not work! What went wrong?&lt;/p&gt;
  469. &lt;p&gt;The underlying library can handle left-recursion if it can recognize it by seeing a &lt;code&gt;memoise&lt;/code&gt; label passed again. This works fine in all the places where we re-use a parser definition (e.g. in &lt;code&gt;bnf&lt;/code&gt;), but it really requires that values are shared!&lt;/p&gt;
  470. &lt;p&gt;If we look carefully at our definition of &lt;code&gt;l&lt;/code&gt; (which parses a lexeme, i.e. something possibly followed by whitespace), then it recurses via a fresh function call, and the program will keep expanding the definition – just like the &lt;code&gt;acyclic&lt;/code&gt; above:&lt;/p&gt;
  471. &lt;div class="sourceCode" id="cb20"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb20-1"&gt;&lt;a href="#cb20-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;l ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a&lt;/span&gt;
  472. &lt;span id="cb20-2"&gt;&lt;a href="#cb20-2" aria-hidden="true" tabindex="-1"/&gt;l p &lt;span class="ot"&gt;=&lt;/span&gt; p &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; l p &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; sat &lt;span class="fu"&gt;isSpace&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  473. &lt;p&gt;The fix (sic!) is to make sure that the recursive call is using the parser we are currently defining, which we can easily do with a local definition:&lt;/p&gt;
  474. &lt;div class="sourceCode" id="cb21"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb21-1"&gt;&lt;a href="#cb21-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;l ::&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; a&lt;/span&gt;
  475. &lt;span id="cb21-2"&gt;&lt;a href="#cb21-2" aria-hidden="true" tabindex="-1"/&gt;l p &lt;span class="ot"&gt;=&lt;/span&gt; p'&lt;/span&gt;
  476. &lt;span id="cb21-3"&gt;&lt;a href="#cb21-3" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="kw"&gt;where&lt;/span&gt; p' &lt;span class="ot"&gt;=&lt;/span&gt; p &lt;span class="op"&gt;&amp;lt;|&gt;&lt;/span&gt; p' &lt;span class="op"&gt;&amp;lt;*&lt;/span&gt; sat &lt;span class="fu"&gt;isSpace&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  477. &lt;p&gt;With this little fix, the parser can parse the example grammar:&lt;/p&gt;
  478. &lt;div class="sourceCode" id="cb22"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb22-1"&gt;&lt;a href="#cb22-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse bnf numExp&lt;/span&gt;
  479. &lt;span id="cb22-2"&gt;&lt;a href="#cb22-2" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Just&lt;/span&gt; [(&lt;span class="st"&gt;"term"&lt;/span&gt;,[[&lt;span class="dt"&gt;NonTerm&lt;/span&gt; &lt;span class="st"&gt;"sum"&lt;/span&gt;]]),(&lt;span class="st"&gt;"pdigit"&lt;/span&gt;,[[&lt;span class="dt"&gt;Lit&lt;/span&gt; &lt;span class="st"&gt;"1"&lt;/span&gt;],…&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  480. &lt;h3 id="going-meta"&gt;Going meta&lt;/h3&gt;
  481. &lt;p&gt;The main demonstration is over, but since we now have already have a parser for grammar descriptions at hand, let’s go a bit further and &lt;em&gt;dynamically&lt;/em&gt; construct a parser from such a description. The parser should only accept strings according to that grammar, and return a parse tree annotated with the non-terminals used:&lt;/p&gt;
  482. &lt;div class="sourceCode" id="cb23"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb23-1"&gt;&lt;a href="#cb23-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;interp ::&lt;/span&gt; &lt;span class="dt"&gt;BNF&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  483. &lt;span id="cb23-2"&gt;&lt;a href="#cb23-2" aria-hidden="true" tabindex="-1"/&gt;interp bnf &lt;span class="ot"&gt;=&lt;/span&gt; parsers &lt;span class="op"&gt;M.!&lt;/span&gt; start&lt;/span&gt;
  484. &lt;span id="cb23-3"&gt;&lt;a href="#cb23-3" aria-hidden="true" tabindex="-1"/&gt;  &lt;span class="kw"&gt;where&lt;/span&gt;&lt;/span&gt;
  485. &lt;span id="cb23-4"&gt;&lt;a href="#cb23-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;    start ::&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt;&lt;/span&gt;
  486. &lt;span id="cb23-5"&gt;&lt;a href="#cb23-5" aria-hidden="true" tabindex="-1"/&gt;    start &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;fst&lt;/span&gt; (&lt;span class="fu"&gt;head&lt;/span&gt; bnf)&lt;/span&gt;
  487. &lt;span id="cb23-6"&gt;&lt;a href="#cb23-6" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  488. &lt;span id="cb23-7"&gt;&lt;a href="#cb23-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;    parsers ::&lt;/span&gt; &lt;span class="dt"&gt;M.Map&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt; (&lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;)&lt;/span&gt;
  489. &lt;span id="cb23-8"&gt;&lt;a href="#cb23-8" aria-hidden="true" tabindex="-1"/&gt;    parsers &lt;span class="ot"&gt;=&lt;/span&gt; M.fromList [ (i, parseRule i rhs) &lt;span class="op"&gt;|&lt;/span&gt; (i, rhs) &lt;span class="ot"&gt;&amp;lt;-&lt;/span&gt; bnf ]&lt;/span&gt;
  490. &lt;span id="cb23-9"&gt;&lt;a href="#cb23-9" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  491. &lt;span id="cb23-10"&gt;&lt;a href="#cb23-10" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;    parseRule ::&lt;/span&gt; &lt;span class="dt"&gt;Ident&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;RuleRhs&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  492. &lt;span id="cb23-11"&gt;&lt;a href="#cb23-11" aria-hidden="true" tabindex="-1"/&gt;    parseRule ident rhs &lt;span class="ot"&gt;=&lt;/span&gt; trace &lt;span class="op"&gt;&amp;lt;$&gt;&lt;/span&gt; asum (&lt;span class="fu"&gt;map&lt;/span&gt; parseSeq rhs)&lt;/span&gt;
  493. &lt;span id="cb23-12"&gt;&lt;a href="#cb23-12" aria-hidden="true" tabindex="-1"/&gt;      &lt;span class="kw"&gt;where&lt;/span&gt; trace s &lt;span class="ot"&gt;=&lt;/span&gt; ident &lt;span class="op"&gt;++&lt;/span&gt; &lt;span class="st"&gt;"("&lt;/span&gt; &lt;span class="op"&gt;++&lt;/span&gt; s &lt;span class="op"&gt;++&lt;/span&gt; &lt;span class="st"&gt;")"&lt;/span&gt;&lt;/span&gt;
  494. &lt;span id="cb23-13"&gt;&lt;a href="#cb23-13" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  495. &lt;span id="cb23-14"&gt;&lt;a href="#cb23-14" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;    parseSeq ::&lt;/span&gt; &lt;span class="dt"&gt;Seq&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  496. &lt;span id="cb23-15"&gt;&lt;a href="#cb23-15" aria-hidden="true" tabindex="-1"/&gt;    parseSeq &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;fmap&lt;/span&gt; &lt;span class="fu"&gt;concat&lt;/span&gt; &lt;span class="op"&gt;.&lt;/span&gt; &lt;span class="fu"&gt;traverse&lt;/span&gt; parseAtom&lt;/span&gt;
  497. &lt;span id="cb23-16"&gt;&lt;a href="#cb23-16" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  498. &lt;span id="cb23-17"&gt;&lt;a href="#cb23-17" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;    parseAtom ::&lt;/span&gt; &lt;span class="dt"&gt;Atom&lt;/span&gt; &lt;span class="ot"&gt;-&gt;&lt;/span&gt; &lt;span class="dt"&gt;P&lt;/span&gt; &lt;span class="dt"&gt;String&lt;/span&gt;&lt;/span&gt;
  499. &lt;span id="cb23-18"&gt;&lt;a href="#cb23-18" aria-hidden="true" tabindex="-1"/&gt;    parseAtom (&lt;span class="dt"&gt;Lit&lt;/span&gt; s) &lt;span class="ot"&gt;=&lt;/span&gt; &lt;span class="fu"&gt;traverse&lt;/span&gt; tok s&lt;/span&gt;
  500. &lt;span id="cb23-19"&gt;&lt;a href="#cb23-19" aria-hidden="true" tabindex="-1"/&gt;    parseAtom (&lt;span class="dt"&gt;NonTerm&lt;/span&gt; i) &lt;span class="ot"&gt;=&lt;/span&gt; parsers &lt;span class="op"&gt;M.!&lt;/span&gt; i&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  501. &lt;p&gt;Let’s see it in action (full code in &lt;a href="https://github.com/nomeata/left-rec-parse/blob/master/BNFEx.hs"&gt;&lt;code&gt;BNFEx.hs&lt;/code&gt;&lt;/a&gt;):&lt;/p&gt;
  502. &lt;div class="sourceCode" id="cb24"&gt;&lt;pre class="sourceCode haskell"&gt;&lt;code class="sourceCode haskell"&gt;&lt;span id="cb24-1"&gt;&lt;a href="#cb24-1" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="dt"&gt;Just&lt;/span&gt; bnfp &lt;span class="ot"&gt;=&lt;/span&gt; parse bnf numExp&lt;/span&gt;
  503. &lt;span id="cb24-2"&gt;&lt;a href="#cb24-2" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; &lt;span class="op"&gt;:&lt;/span&gt;t bnfp&lt;/span&gt;
  504. &lt;span id="cb24-3"&gt;&lt;a href="#cb24-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="ot"&gt;bnfp ::&lt;/span&gt; &lt;span class="dt"&gt;BNF&lt;/span&gt;&lt;/span&gt;
  505. &lt;span id="cb24-4"&gt;&lt;a href="#cb24-4" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse (inter&lt;/span&gt;
  506. &lt;span id="cb24-5"&gt;&lt;a href="#cb24-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="fu"&gt;interact&lt;/span&gt;  interp&lt;/span&gt;
  507. &lt;span id="cb24-6"&gt;&lt;a href="#cb24-6" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse (interp bnfp) &lt;span class="st"&gt;"12+3*4"&lt;/span&gt;&lt;/span&gt;
  508. &lt;span id="cb24-7"&gt;&lt;a href="#cb24-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Just&lt;/span&gt; &lt;span class="st"&gt;"term(sum(prod(atom(num(pnum(pnum(pdigit(1))digit(pdigit(2))))))+sum(prod(atom(num(pnum(pdigit(3))))*prod(atom(num(pnum(pdigit(4)))))))))"&lt;/span&gt;&lt;/span&gt;
  509. &lt;span id="cb24-8"&gt;&lt;a href="#cb24-8" aria-hidden="true" tabindex="-1"/&gt;ghci&lt;span class="op"&gt;&gt;&lt;/span&gt; parse (interp bnfp) &lt;span class="st"&gt;"12+3*4+"&lt;/span&gt;&lt;/span&gt;
  510. &lt;span id="cb24-9"&gt;&lt;a href="#cb24-9" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="dt"&gt;Nothing&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  511. &lt;p&gt;It is worth noting that the &lt;code&gt;numExp&lt;/code&gt; grammar is also left-recursive, so implementing &lt;code&gt;interp&lt;/code&gt; with a conventional parser combinator library would not have worked. But thanks to our &lt;code&gt;propriocept&lt;/code&gt; tick, it does! Again, the sharing is important; in the code above it is the map &lt;code&gt;parsers&lt;/code&gt; that is defined in terms of itself, and will ensure that the left-recursive productions will work.&lt;/p&gt;
  512. &lt;h3 id="closing-thoughts"&gt;Closing thoughts&lt;/h3&gt;
  513. &lt;p&gt;I am using &lt;code&gt;unsafePerformIO&lt;/code&gt;, so I need to justify its use. Clearly, &lt;code&gt;propriocept&lt;/code&gt; is &lt;em&gt;not&lt;/em&gt; a pure function, and it’s type is a lie. In general, using it will break the nice equational properties of Haskell, as we have seen in our experiments with &lt;code&gt;cons&lt;/code&gt;.&lt;/p&gt;
  514. &lt;p&gt;In the case of our parser library, however, we use it in specific ways, namely to feed a fresh name to &lt;code&gt;memoise&lt;/code&gt;. Assuming the underlying parser library’s behavior does not observably depend on where and with which key &lt;code&gt;memoise&lt;/code&gt; is used, this results in a properly pure interface, and all is well again. (NB: I did not investigate if this assumption actually holds for the parser library used here, or if it may for example affect the order of parse trees returned.)&lt;/p&gt;
  515. &lt;p&gt;I also expect that this implementation, which will memoise &lt;em&gt;every&lt;/em&gt; parser involved, will be rather slow. It seems plausible to analyze the graph structure and figure out which &lt;code&gt;memoise&lt;/code&gt; calls are actually needed to break left-recursion (at least if we drop the &lt;code&gt;Monad&lt;/code&gt; instance or always memoise &lt;code&gt;&gt;&gt;=&lt;/code&gt;).&lt;/p&gt;
  516. &lt;p&gt;If you liked this post, you might enjoy reading &lt;a href="https://doi.org/10.1145/3607853"&gt;the paper about rec-def&lt;/a&gt;, watch one of my talks about it (MuniHac, BOBKonf, ICFP23; the presentation evolved over time), or if you just want to see more about how things are laid out on Haskell’s heap, go to &lt;a href="https://www.youtube.com/playlist?list=PL4FcLyLhO9jggmkqJyJ2i9pCSiDpwKiVu"&gt;my screen casts exploring the Haskell heap&lt;/a&gt;.&lt;/p&gt;</description>
  517.         <pubDate>Sun, 10 Sep 2023 17:16:08 -0700</pubDate>
  518.      </item>
  519.      <item>
  520.         <title>Generating bibtex bibliographies from DOIs via DBLP</title>
  521.         <link>https://www.joachim-breitner.de/blog/806-Generating_bibtex_bibliographies_from_DOIs_via_DBLP</link>
  522.         <guid>https://www.joachim-breitner.de/blog/806-Generating_bibtex_bibliographies_from_DOIs_via_DBLP</guid>
  523.         <comments>https://www.joachim-breitner.de/blog/806-Generating_bibtex_bibliographies_from_DOIs_via_DBLP#comments</comments>
  524.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  525.         <description>&lt;p&gt;I sometimes &lt;a href="https://www.joachim-breitner.de/publications"&gt;write papers&lt;/a&gt; and part of paper writing is assembling the bibliography. In my case, this is done using BibTeX. So when I need to add another citation, I have to find suitable data in Bibtex format.&lt;/p&gt;
  526. &lt;p&gt;Often I copy snippets from &lt;code&gt;.bib&lt;/code&gt; files from earlier paper.&lt;/p&gt;
  527. &lt;p&gt;Or I search for the paper on &lt;a href="https://dblp.org/"&gt;DBLP&lt;/a&gt;, which in my experience has highest quality BibTeX entries and best coverage of computer science related publications, copy it to my &lt;code&gt;.bib&lt;/code&gt; file, and change the key to whatever I want to refer the paper by.&lt;/p&gt;
  528. &lt;p&gt;But in the days of pervasive use of DOIs (digital object identifiers) for almost all publications, manually keeping the data in bibtex files seems outdated. Instead I’d rather just put the two pieces of data I care about: the &lt;em&gt;key&lt;/em&gt; that I want to use for citation, and the &lt;em&gt;doi&lt;/em&gt;. The rest I do not want to be bothered with.&lt;/p&gt;
  529. &lt;p&gt;So I wrote a small script that takes a &lt;code&gt;.yaml&lt;/code&gt; file like&lt;/p&gt;
  530. &lt;div class="sourceCode" id="cb1"&gt;&lt;pre class="sourceCode yaml"&gt;&lt;code class="sourceCode yaml"&gt;&lt;span id="cb1-1"&gt;&lt;a href="#cb1-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="fu"&gt;entries&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;/span&gt;
  531. &lt;span id="cb1-2"&gt;&lt;a href="#cb1-2" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;unsafePerformIO&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1007/10722298_3&lt;/span&gt;&lt;/span&gt;
  532. &lt;span id="cb1-3"&gt;&lt;a href="#cb1-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;dejafu&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/2804302.2804306&lt;/span&gt;&lt;/span&gt;
  533. &lt;span id="cb1-4"&gt;&lt;a href="#cb1-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;runST&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/3158152&lt;/span&gt;&lt;/span&gt;
  534. &lt;span id="cb1-5"&gt;&lt;a href="#cb1-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;quickcheck&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/351240.351266&lt;/span&gt;&lt;/span&gt;
  535. &lt;span id="cb1-6"&gt;&lt;a href="#cb1-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;optimiser&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1016/S0167-6423(97)00029-4&lt;/span&gt;&lt;/span&gt;
  536. &lt;span id="cb1-7"&gt;&lt;a href="#cb1-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;sabry&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1017/s0956796897002943&lt;/span&gt;&lt;/span&gt;
  537. &lt;span id="cb1-8"&gt;&lt;a href="#cb1-8" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;concurrent&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/237721.237794&lt;/span&gt;&lt;/span&gt;
  538. &lt;span id="cb1-9"&gt;&lt;a href="#cb1-9" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;launchbury&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/158511.158618&lt;/span&gt;&lt;/span&gt;
  539. &lt;span id="cb1-10"&gt;&lt;a href="#cb1-10" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;datafun&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/2951913.2951948&lt;/span&gt;&lt;/span&gt;
  540. &lt;span id="cb1-11"&gt;&lt;a href="#cb1-11" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;observable-sharing&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1007/3-540-46674-6_7&lt;/span&gt;&lt;/span&gt;
  541. &lt;span id="cb1-12"&gt;&lt;a href="#cb1-12" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;kildall-73&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/512927.512945&lt;/span&gt;&lt;/span&gt;
  542. &lt;span id="cb1-13"&gt;&lt;a href="#cb1-13" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;kam-ullman-76&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/321921.321938&lt;/span&gt;&lt;/span&gt;
  543. &lt;span id="cb1-14"&gt;&lt;a href="#cb1-14" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;spygame&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/3371101&lt;/span&gt;&lt;/span&gt;
  544. &lt;span id="cb1-15"&gt;&lt;a href="#cb1-15" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;cocaml&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.3233/FI-2017-1473&lt;/span&gt;&lt;/span&gt;
  545. &lt;span id="cb1-16"&gt;&lt;a href="#cb1-16" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;secrets&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1017/S0956796802004331&lt;/span&gt;&lt;/span&gt;
  546. &lt;span id="cb1-17"&gt;&lt;a href="#cb1-17" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;modular&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1017/S0956796817000016&lt;/span&gt;&lt;/span&gt;
  547. &lt;span id="cb1-18"&gt;&lt;a href="#cb1-18" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;longley&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/317636.317775&lt;/span&gt;&lt;/span&gt;
  548. &lt;span id="cb1-19"&gt;&lt;a href="#cb1-19" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;nievergelt&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/800152.804906&lt;/span&gt;&lt;/span&gt;
  549. &lt;span id="cb1-20"&gt;&lt;a href="#cb1-20" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;runST2&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/3527326&lt;/span&gt;&lt;/span&gt;
  550. &lt;span id="cb1-21"&gt;&lt;a href="#cb1-21" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;polakow&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/2804302.2804309&lt;/span&gt;&lt;/span&gt;
  551. &lt;span id="cb1-22"&gt;&lt;a href="#cb1-22" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;lvars&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/2502323.2502326&lt;/span&gt;&lt;/span&gt;
  552. &lt;span id="cb1-23"&gt;&lt;a href="#cb1-23" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;typesafe-sharing&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/1596638.1596653&lt;/span&gt;&lt;/span&gt;
  553. &lt;span id="cb1-24"&gt;&lt;a href="#cb1-24" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;pure-functional&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1007/978-3-642-14162-1_17&lt;/span&gt;&lt;/span&gt;
  554. &lt;span id="cb1-25"&gt;&lt;a href="#cb1-25" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="fu"&gt;clairvoyant&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; 10.1145/3341718&lt;/span&gt;&lt;/span&gt;
  555. &lt;span id="cb1-26"&gt;&lt;a href="#cb1-26" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="fu"&gt;subs&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;/span&gt;
  556. &lt;span id="cb1-27"&gt;&lt;a href="#cb1-27" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;  &lt;/span&gt;&lt;span class="kw"&gt;-&lt;/span&gt;&lt;span class="at"&gt; &lt;/span&gt;&lt;span class="fu"&gt;replace&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; Peyton Jones&lt;/span&gt;&lt;/span&gt;
  557. &lt;span id="cb1-28"&gt;&lt;a href="#cb1-28" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="at"&gt;    &lt;/span&gt;&lt;span class="fu"&gt;with&lt;/span&gt;&lt;span class="kw"&gt;:&lt;/span&gt;&lt;span class="at"&gt; &lt;/span&gt;&lt;span class="st"&gt;'{Peyton Jones}'&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  558. &lt;p&gt;and turns it into a nice &lt;code&gt;.bibtex&lt;/code&gt; file:&lt;/p&gt;
  559. &lt;pre class="shell"&gt;&lt;code&gt;$ ./doi2bib.py &amp;lt; doibib.yaml &gt; dblp.bib
  560. $ head dblp.bib
  561. @inproceedings{unsafePerformIO,
  562.  author       = {Simon L. {Peyton Jones} and
  563.                  Simon Marlow and
  564.                  Conal Elliott},
  565.  editor       = {Pieter W. M. Koopman and
  566.                  Chris Clack},
  567.  title        = {Stretching the Storage Manager: Weak Pointers and Stable Names in
  568.                  Haskell},
  569.  booktitle    = {Implementation of Functional Languages, 11th International Workshop,
  570.                  IFL'99, Lochem, The Netherlands, September 7-10, 1999, Selected Papers},&lt;/code&gt;&lt;/pre&gt;
  571. &lt;p&gt;The last bit allows me to do some fine-tuning of the file, because unfortunately, not even DBLP BibTeX files are perfect, for example in the presence of two family names.&lt;/p&gt;
  572. &lt;p&gt;Now I have less moving parts to worry about, and a more consistent bibliography.&lt;/p&gt;
  573. &lt;p&gt;The script is rather small, so I’ll just share it here:&lt;/p&gt;
  574. &lt;div class="sourceCode" id="cb3"&gt;&lt;pre class="sourceCode python"&gt;&lt;code class="sourceCode python"&gt;&lt;span id="cb3-1"&gt;&lt;a href="#cb3-1" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="co"&gt;#!/usr/bin/env python3&lt;/span&gt;&lt;/span&gt;
  575. &lt;span id="cb3-2"&gt;&lt;a href="#cb3-2" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  576. &lt;span id="cb3-3"&gt;&lt;a href="#cb3-3" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="im"&gt;import&lt;/span&gt; sys&lt;/span&gt;
  577. &lt;span id="cb3-4"&gt;&lt;a href="#cb3-4" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="im"&gt;import&lt;/span&gt; yaml&lt;/span&gt;
  578. &lt;span id="cb3-5"&gt;&lt;a href="#cb3-5" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="im"&gt;import&lt;/span&gt; requests&lt;/span&gt;
  579. &lt;span id="cb3-6"&gt;&lt;a href="#cb3-6" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="im"&gt;import&lt;/span&gt; requests_cache&lt;/span&gt;
  580. &lt;span id="cb3-7"&gt;&lt;a href="#cb3-7" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="im"&gt;import&lt;/span&gt; re&lt;/span&gt;
  581. &lt;span id="cb3-8"&gt;&lt;a href="#cb3-8" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  582. &lt;span id="cb3-9"&gt;&lt;a href="#cb3-9" aria-hidden="true" tabindex="-1"/&gt;requests_cache.install_cache(backend&lt;span class="op"&gt;=&lt;/span&gt;&lt;span class="st"&gt;'sqlite'&lt;/span&gt;)&lt;/span&gt;
  583. &lt;span id="cb3-10"&gt;&lt;a href="#cb3-10" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  584. &lt;span id="cb3-11"&gt;&lt;a href="#cb3-11" aria-hidden="true" tabindex="-1"/&gt;data &lt;span class="op"&gt;=&lt;/span&gt; yaml.safe_load(sys.stdin)&lt;/span&gt;
  585. &lt;span id="cb3-12"&gt;&lt;a href="#cb3-12" aria-hidden="true" tabindex="-1"/&gt;&lt;/span&gt;
  586. &lt;span id="cb3-13"&gt;&lt;a href="#cb3-13" aria-hidden="true" tabindex="-1"/&gt;&lt;span class="cf"&gt;for&lt;/span&gt; key, doi &lt;span class="kw"&gt;in&lt;/span&gt; data[&lt;span class="st"&gt;'entries'&lt;/span&gt;].items():&lt;/span&gt;
  587. &lt;span id="cb3-14"&gt;&lt;a href="#cb3-14" aria-hidden="true" tabindex="-1"/&gt;    bib &lt;span class="op"&gt;=&lt;/span&gt; requests.get(&lt;span class="ss"&gt;f"https://dblp.org/doi/&lt;/span&gt;&lt;span class="sc"&gt;{&lt;/span&gt;doi&lt;span class="sc"&gt;}&lt;/span&gt;&lt;span class="ss"&gt;.bib"&lt;/span&gt;).text&lt;/span&gt;
  588. &lt;span id="cb3-15"&gt;&lt;a href="#cb3-15" aria-hidden="true" tabindex="-1"/&gt;    bib &lt;span class="op"&gt;=&lt;/span&gt; re.sub(&lt;span class="st"&gt;'{DBLP.*,'&lt;/span&gt;, &lt;span class="st"&gt;'{'&lt;/span&gt; &lt;span class="op"&gt;+&lt;/span&gt; key &lt;span class="op"&gt;+&lt;/span&gt; &lt;span class="st"&gt;','&lt;/span&gt;, bib)&lt;/span&gt;
  589. &lt;span id="cb3-16"&gt;&lt;a href="#cb3-16" aria-hidden="true" tabindex="-1"/&gt;    &lt;span class="cf"&gt;for&lt;/span&gt; subs &lt;span class="kw"&gt;in&lt;/span&gt; data[&lt;span class="st"&gt;'subs'&lt;/span&gt;]:&lt;/span&gt;
  590. &lt;span id="cb3-17"&gt;&lt;a href="#cb3-17" aria-hidden="true" tabindex="-1"/&gt;        bib &lt;span class="op"&gt;=&lt;/span&gt; re.sub(subs[&lt;span class="st"&gt;'replace'&lt;/span&gt;], subs[&lt;span class="st"&gt;'with'&lt;/span&gt;], bib)&lt;/span&gt;
  591. &lt;span id="cb3-18"&gt;&lt;a href="#cb3-18" aria-hidden="true" tabindex="-1"/&gt;    &lt;span class="bu"&gt;print&lt;/span&gt;(bib)&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;
  592. &lt;p&gt;There are similar projects out there, e.g. &lt;a href="https://github.com/cr-marcstevens/dblpbibtex"&gt;&lt;code&gt;dblpbibtex&lt;/code&gt;&lt;/a&gt; in C++ and &lt;a href="https://github.com/PJK/dblpbib"&gt;&lt;code&gt;dblpbib&lt;/code&gt;&lt;/a&gt; in Ruby. These allow direct use of &lt;code&gt;\cite{DBLP:rec/conf/isit/BreitnerS20}&lt;/code&gt; in Latex, which is also nice, but for now I like to choose more speaking citation keys myself.&lt;/p&gt;</description>
  593.         <pubDate>Wed, 12 Jul 2023 14:32:19 +0200</pubDate>
  594.      </item>
  595.      <item>
  596.         <title>ICFP Pearl preprint on rec-def</title>
  597.         <link>https://www.joachim-breitner.de/blog/805-ICFP_Pearl_preprint_on_rec-def</link>
  598.         <guid>https://www.joachim-breitner.de/blog/805-ICFP_Pearl_preprint_on_rec-def</guid>
  599.         <comments>https://www.joachim-breitner.de/blog/805-ICFP_Pearl_preprint_on_rec-def#comments</comments>
  600.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  601.         <description>&lt;p&gt;I submitted a Functional Pearl to this &lt;a href="https://icfp23.sigplan.org/"&gt;year’s ICFP&lt;/a&gt; and it got accepted!&lt;/p&gt;
  602. &lt;p&gt;It is about the idea of using Haskell’s inherent ability to define &lt;em&gt;recursive equations&lt;/em&gt;, and use them for more than just functions and lazy data structures. I blogged about this before (&lt;a href="https://www.joachim-breitner.de/blog/792-More_recursive_definitions"&gt;introducing the idea&lt;/a&gt;, &lt;a href="https://www.joachim-breitner.de/blog/793-rec-def__Behind_the_scenes"&gt;behind the scenes&lt;/a&gt;, applications to &lt;a href="https://www.joachim-breitner.de/blog/794-rec-def__Program_analysis_case_study"&gt;program analysis&lt;/a&gt;, &lt;a href="https://www.joachim-breitner.de/blog/795-rec-def__Dominators_case_study"&gt;graph algorithms&lt;/a&gt; and &lt;a href="https://www.joachim-breitner.de/blog/796-rec-def__Minesweeper_case_study"&gt;minesweeper&lt;/a&gt;), but hopefully the paper brings out the idea even more clearly. The constructive feedback from a few friendly readers (Claudio, Sebastian, and also the anonymous reviewers) certainly improved the paper.&lt;/p&gt;
  603. &lt;blockquote&gt;
  604. &lt;h3 id="abstract"&gt;Abstract&lt;/h3&gt;
  605. &lt;p&gt;Haskell’s laziness allows the programmer to solve some problems naturally and declaratively via recursive equations. Unfortunately, if the input is “too recursive”, these very elegant idioms can fall into the dreaded black hole, and the programmer has to resort to more pedestrian approaches.&lt;/p&gt;
  606. &lt;p&gt;It does not have to be that way: We built variants of common pure data structures (Booleans, sets) where recursive definitions are productive. Internally, the infamous unsafePerformIO is at work, but the user only sees a beautiful and pure API, and their pretty recursive idioms – magically – work again.&lt;/p&gt;
  607. &lt;/blockquote&gt;
  608. &lt;p&gt;If you are curious, please have a look at the &lt;a href="//www.joachim-breitner.de/publications/rec-def-pearl.pdf"&gt;current version of the paper&lt;/a&gt;. Any feedback is welcome; even more if it comes before July 11, because then I can include it in the camera ready version.&lt;/p&gt;
  609. &lt;hr/&gt;
  610. &lt;p&gt;There are still some open questions around this work. What bothers me maybe most is the lack of a denotational semantics that unifies the partial order underlying the Haskell fragment, and the partial order underlying the domain of the embedded equations.&lt;/p&gt;
  611. &lt;p&gt;The crux of the probem is maybe best captured by this question:&lt;/p&gt;
  612. &lt;blockquote&gt;
  613. &lt;p&gt;Imagine an untyped lambda calculus with constructors, lazy evaluation, and an operation &lt;code&gt;rseq&lt;/code&gt; that recursively evaluates constructors, but terminates in the presence of cycles. So for example&lt;/p&gt;
  614. &lt;pre&gt;&lt;code&gt;rseq (let x    = 1 :: x    in x   ) ≡ ()
  615. rseq (let x () = 1 :: x () in x ()) ≡ ⊥&lt;/code&gt;&lt;/pre&gt;
  616. &lt;p&gt;In this language, knot tying is observable. What is the “nicest” denotational semantics.&lt;/p&gt;
  617. &lt;/blockquote&gt;
  618. &lt;p&gt;&lt;strong&gt;Update&lt;/strong&gt;: I made some progress via a &lt;a href="https://discourse.haskell.org/t/icfp-pearl-on-rec-def/6626/14?u=nomeata"&gt;discussion on the Haskell Discource&lt;/a&gt; and started &lt;a href="https://www.joachim-breitner.de/publications/rec-def-denotational.pdf"&gt;some rough notes on a denotational semantics&lt;/a&gt;.&lt;/p&gt;</description>
  619.         <pubDate>Thu, 22 Jun 2023 18:21:11 +0200</pubDate>
  620.      </item>
  621.      <item>
  622.         <title>The curious case of the half-half Bitcoin ECDSA nonces</title>
  623.         <link>https://www.joachim-breitner.de/blog/804-The_curious_case_of_the_half-half_Bitcoin_ECDSA_nonces</link>
  624.         <guid>https://www.joachim-breitner.de/blog/804-The_curious_case_of_the_half-half_Bitcoin_ECDSA_nonces</guid>
  625.         <comments>https://www.joachim-breitner.de/blog/804-The_curious_case_of_the_half-half_Bitcoin_ECDSA_nonces#comments</comments>
  626.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  627.         <description>&lt;p&gt;This is the week of the Gulaschprogrammiernacht, a yearly Chaos Computer Club even in Karlsruhe, so it was exactly a year ago that I sat in my AirBnB room and went over the slides for my talk &lt;a href="https://media.ccc.de/v/gpn20-66-lattice-attacks-on-ethereum-bitcoin-and-https"&gt;“Lattice Attacks on Ethereum, Bitcoin, and HTTPS”&lt;/a&gt; that I would give there.&lt;/p&gt;
  628. &lt;p&gt;It reports on &lt;a href="https://eprint.iacr.org/2019/023"&gt;research&lt;/a&gt; done with &lt;a href="https://cseweb.ucsd.edu/~nadiah/"&gt;Nadia Heninger&lt;/a&gt; while I was in Phildalephia, and I really liked giving that talk: At some point we look at some rather odd signatures we found on the bitcoin blockchain, and part of the signature (the “nonce”) happens to share some bytes with the secret key. A clear case of some buffer overlap in a memory unsafe language, which I, as a fan of languages like Haskell, are very happy to sneer at!&lt;/p&gt;
  629. &lt;figure&gt;
  630. &lt;img src="/various/biased-nonces-slide-17.svg" alt="A sneery slide"/&gt;
  631. &lt;figcaption aria-hidden="true"&gt;A sneery slide&lt;/figcaption&gt;
  632. &lt;/figure&gt;
  633. &lt;p&gt;But last year, as I was going over &lt;a href="https://www.joachim-breitner.de/publications/BiasedNonces-GPN20-2022-05-20.pdf"&gt;the slides&lt;/a&gt; I looked at the raw data again for some reason, and I found that we overlooked something: Not only was the the upper half ot the nonce equal to the lower half of the secret key, but he lower half of the nonce was also equal to the upper half of the message hash!&lt;/p&gt;
  634. &lt;p&gt;This now looks much less like an accident to me, and more like a (overly) simple form of deterministic nonce creation… so much for my nice anecdote. (I still used the anecdote in my talk, followed up with an “actually”.)&lt;/p&gt;
  635. &lt;p&gt;When I told Nadia about this, she got curious as well, and quickly saw that from a signature with such a nonce, one can rather easily extract the secret key. So together with her student Dylan Rowe, we implemented this analysis and searched the bitcoin blockchain for more instance of such signatures. We did find a few, and were even able to trace them back to a somewhat infamous bitcoin activist going under the pseudonym Amaclin.&lt;/p&gt;
  636. &lt;p&gt;This research and sleuthing turned into another paper, &lt;a href="https://eprint.iacr.org/2023/841"&gt;“The curious case of the half-half Bitcoin ECDSA nonces”&lt;/a&gt;, to be presented at AfricaCrypt 2023. Enjoy!&lt;/p&gt;</description>
  637.         <pubDate>Wed, 07 Jun 2023 08:42:28 +0200</pubDate>
  638.      </item>
  639.      <item>
  640.         <title>Giving back to OPLSS</title>
  641.         <link>https://www.joachim-breitner.de/blog/803-Giving_back_to_OPLSS</link>
  642.         <guid>https://www.joachim-breitner.de/blog/803-Giving_back_to_OPLSS</guid>
  643.         <comments>https://www.joachim-breitner.de/blog/803-Giving_back_to_OPLSS#comments</comments>
  644.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  645.         <description>&lt;p&gt;Nine years ago, when I was a PhD student, I attended the &lt;a href="https://www.cs.uoregon.edu/research/summerschool/"&gt;Oregon Programming Language Summer School&lt;/a&gt; in Eugene. I had a great time and learned a lot.&lt;/p&gt;
  646. &lt;figure&gt;
  647. &lt;img src="/various/OPLSS14Photo-small.jpg" alt="The OPLSS’14 full image"/&gt;
  648. &lt;figcaption aria-hidden="true"&gt;The OPLSS’14 &lt;a href="/various/OPLSS14Photo.jpg"&gt;full image&lt;/a&gt;&lt;/figcaption&gt;
  649. &lt;/figure&gt;
  650. &lt;p&gt;Learning some of the things I learned there, and meeting some of the people I met there, also led to me graduating, which led to me becoming &lt;a href="https://www.cis.upenn.edu/~plclub/"&gt;a PostDoc at UPenn&lt;/a&gt;, which led to me later joining DFINITY to implement the &lt;a href="https://github.com/dfinity/motoko/"&gt;Motoko programming language&lt;/a&gt; and help design and specify the public interface of their “Internet Computer”, including the &lt;a href="https://medium.com/dfinity/how-internet-computer-responses-are-certified-as-authentic-2ff1bb1ea659"&gt;response certification&lt;/a&gt; (&lt;a href="https://www.youtube.com/watch?v=mZbFhRIHIiY"&gt;video&lt;/a&gt;).&lt;/p&gt;
  651. &lt;p&gt;So when the &lt;a href="https://icdevs.org/"&gt;ICDevs&lt;/a&gt; non-profit offered a &lt;a href="https://icdevs.org/bounties/2023/01/09/36-Signing-Tree-and-DER-Encoding.html"&gt;development bounty for a Motoko library implementing the merkle trees involved in certification&lt;/a&gt;, this sounded like a fun little coding task, so I completed it; likely with less effort than it would have taken someone who first had to get into these topics.&lt;/p&gt;
  652. &lt;p&gt;The bounty was quite generous, at US$ 10k, and I was too vain to “just” have it donated to some large charity, as I &lt;a href="//www.joachim-breitner.de/blog/798-Pro-charity_consulting"&gt;recently with a few coding and consulting gigs&lt;/a&gt;, and looked for more personal. So, the ICDevs guys and I agreed to donate the money to &lt;a href="https://www.cs.uoregon.edu/research/summerschool/summer23/"&gt;this year’s OPLSS&lt;/a&gt;, where I heard it can cover the cost of about 8 students, and hopefully helps the PL cause.&lt;/p&gt;
  653. &lt;p&gt;(You will not find us listed as sponsors because for some reason, a “donation” instead of “sponsorship” comes with less strings attached to the organizers.)&lt;/p&gt;</description>
  654.         <pubDate>Sun, 04 Jun 2023 14:18:33 +0200</pubDate>
  655.      </item>
  656.      <item>
  657.         <title>More thoughts on a bootstrappable GHC</title>
  658.         <link>https://www.joachim-breitner.de/blog/802-More_thoughts_on_a_bootstrappable_GHC</link>
  659.         <guid>https://www.joachim-breitner.de/blog/802-More_thoughts_on_a_bootstrappable_GHC</guid>
  660.         <comments>https://www.joachim-breitner.de/blog/802-More_thoughts_on_a_bootstrappable_GHC#comments</comments>
  661.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  662.         <description>&lt;p&gt;The &lt;a href="https://bootstrappable.org/"&gt;bootstrappable builds project&lt;/a&gt; tries to find ways of building all our software from source, without relying on binary artifacts. A noble goal, and one that is often thwarted by languages with self-hosting compilers, like GHC: In order to build GHC, you need GHC. A &lt;a href="https://github.com/NixOS/nixpkgs/pull/227914"&gt;Pull Request against nixpkgs&lt;/a&gt;, adding first steps of the bootstrapping pipeline, reminded me of the issue with GHC, which I have noted down &lt;a href="https://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC"&gt;some thoughts about&lt;/a&gt; before and I played around a bit more.&lt;/p&gt;
  663. &lt;p&gt;The most promising attempt to bootstrap GHC was done by &lt;a href="https://elephly.net/posts/2017-01-09-bootstrapping-haskell-part-1.html"&gt;rekado in 2017&lt;/a&gt;. He observed that Hugs is maybe the most recently maintained bootstrappable (since written in C) Haskell compiler, but noticed that “it cannot deal with mutually recursive module dependencies, which is a feature that even the earliest versions of GHC rely on. This means that running a variant of GHC inside of Hugs is not going to work without major changes.” He then tries to bootstrap another very old Haskell compiler (nhc) with Hugs, and makes good but incomplete progress.&lt;/p&gt;
  664. &lt;p&gt;This made me wonder: What &lt;em&gt;if&lt;/em&gt; Hugs supported mutually recursive modules? Would that make a big difference? Anthony Clayden &lt;a href="https://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC#comment_3"&gt;keeps advocating Hugs as a viable Haskell implementation&lt;/a&gt;, so maybe if that was the main blocker, then adding support to Hugs for that is probably not too hard (at least in a compile-the-strongly-connected-component-as-one-unit mode) and worthwhile?&lt;/p&gt;
  665. &lt;h3 id="all-of-ghc-in-one-file"&gt;All of GHC in one file?&lt;/h3&gt;
  666. &lt;p&gt;That reminded me of a situation I was in before, where I had to combine multiple Haskell modules into one before: For my talk &lt;a href="https://www.youtube.com/watch?v=2kKvVe673MA"&gt;“Lock-step simulation is child’s play”&lt;/a&gt; I wrote a multi-player game, a simulation environment for it, and a presentation tool around it, all in the &lt;a href="https://code.world"&gt;CodeWorld&lt;/a&gt; programming environment, which supports only a single module. So I hacked the a small tool &lt;a href="https://github.com/nomeata/hs-all-in-one/"&gt;hs-all-in-one&lt;/a&gt; that takes multiple Haskell modules and combines them into one, mangling the names to avoid name clashes.&lt;/p&gt;
  667. &lt;p&gt;This made me wonder: Can I turn &lt;em&gt;all&lt;/em&gt; of GHC into one module, and compile that?&lt;/p&gt;
  668. &lt;p&gt;At this point I have probably left the direct path towards bootstrapping, but I kinda good hooked.&lt;/p&gt;
  669. &lt;ol type="1"&gt;
  670. &lt;li&gt;&lt;p&gt;Using GHC’s &lt;code&gt;hadrian/ghci&lt;/code&gt; tool, I got it to produce the necessary generated files (e.g. from &lt;code&gt;happy&lt;/code&gt; grammars) and spit out the lit of modules that make up GHC, which I could feed to &lt;code&gt;hs-all-in-one&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
  671. &lt;li&gt;&lt;p&gt;It uses &lt;a href="https://hackage.haskell.org/package/haskell-src-exts"&gt;&lt;code&gt;haskell-src-exts&lt;/code&gt;&lt;/a&gt; for parsing, and it was almost able to parse all of that. It has a different opinion about how &lt;a href="https://github.com/haskell-suite/haskell-src-exts/issues/468"&gt;&lt;code&gt;MultiWayIf&lt;/code&gt; should be indented&lt;/a&gt;, whether &lt;a href="https://github.com/haskell-suite/haskell-src-exts/issues/469"&gt;&lt;code&gt;EmptyCase&lt;/code&gt; needs &lt;code&gt;{}&lt;/code&gt;&lt;/a&gt; and &lt;a href="https://github.com/haskell-suite/haskell-src-exts/issues/470"&gt;issues&lt;/a&gt; &lt;a href="https://github.com/haskell-suite/haskell-src-exts/issues/471"&gt;pretty-printing&lt;/a&gt; some promoted values, but otherwise the round-tripping worked fine, and I as able to generate a large file (680,000 loc, 41 MB) that passes GHC’s parser.&lt;/p&gt;&lt;/li&gt;
  672. &lt;li&gt;&lt;p&gt;It also uses &lt;a href="https://hackage.haskell.org/package/haskell-names"&gt;&lt;code&gt;haskell-names&lt;/code&gt;&lt;/a&gt; to resolve names.&lt;/p&gt;
  673. &lt;p&gt;This library is less up-to-date with various Haskell features, so I added support for renaming in some pragmas (&lt;code&gt;ANN&lt;/code&gt;, &lt;code&gt;SPECIALIZE&lt;/code&gt;), pattern signatures etc.&lt;/p&gt;
  674. &lt;p&gt;For my previous use-case I could just combine all the imports, knowing that I would not introduce conflicts. For GHC, this is far from true: Some modules import &lt;code&gt;Data.Map.Strict&lt;/code&gt;, others &lt;code&gt;Data.Map.Lazy&lt;/code&gt;, and yet others introduce names that clash with stuff imported from the prelude… so I had to change the tool to fully qualify all imported values. This isn’t so bad, I can do that using &lt;code&gt;haskell-names&lt;/code&gt;, if I somehow know what all the modules in &lt;code&gt;base&lt;/code&gt;, &lt;code&gt;containers&lt;/code&gt;, &lt;code&gt;transformers&lt;/code&gt; and &lt;code&gt;array&lt;/code&gt; export.&lt;/p&gt;
  675. &lt;p&gt;The &lt;code&gt;haskell-names&lt;/code&gt; library itself comes with a hard-coded database of &lt;code&gt;base&lt;/code&gt; exports, but it is incomplete and doesn’t help me with, say, &lt;code&gt;containers&lt;/code&gt;.&lt;/p&gt;
  676. &lt;p&gt;I then wrote a little parser for the &lt;code&gt;.txt&lt;/code&gt; files that &lt;code&gt;haddock&lt;/code&gt; produces for the benefit of &lt;code&gt;hoogle&lt;/code&gt;, and that are conveniently installed along the packages (at least on nix). This would have been great, if these files wouldn’t simply omit all &lt;em&gt;reexported&lt;/em&gt; entities! I added some manual hacks (&lt;code&gt;Data.Map&lt;/code&gt; has the same exports as &lt;code&gt;Data.IntMap&lt;/code&gt;; &lt;code&gt;Prelude&lt;/code&gt; exports all entities as known by &lt;code&gt;haskell-names&lt;/code&gt;, but those that are also exported from &lt;code&gt;Data.List&lt;/code&gt;, use the symbol from there…)&lt;/p&gt;
  677. &lt;p&gt;I played this game of whack-a-mole for a while, solving many of the problems that GHC’s renamer reports, but eventually stopped to write this blog post. I am fairly confident that this could be pulled through, though.&lt;/p&gt;&lt;/li&gt;
  678. &lt;/ol&gt;
  679. &lt;h3 id="back-to-bootstrapping"&gt;Back to bootstrapping&lt;/h3&gt;
  680. &lt;p&gt;So what if we could pull this through? We’d have a very large code file that GHC may be able to compile to produce a &lt;code&gt;ghc&lt;/code&gt; binary without exhausting my RAM. But that doesn’t help with bootstrapping yet.&lt;/p&gt;
  681. &lt;p&gt;If lack of support for recursive modules is all that Hugs is missing, we’d be done indeed. But quite contrary, it is probably the least of our worries, given that contemporary GHC uses many many other features not supported by Hugs.&lt;/p&gt;
  682. &lt;p&gt;Some of them a syntactic and can easily be rewritten to more normal Haskell in a preprocessing step (e.g. &lt;code&gt;MultiWayIf&lt;/code&gt;).&lt;/p&gt;
  683. &lt;p&gt;Others are deep and hard (&lt;code&gt;GADTs&lt;/code&gt;, Pattern synonyms, Type Families), and prohibit attempting to compile a current version of GHC (even if its all one module) with Hugs. So one would certainly have to go back in time and find a version of GHC that is not yet using all these features. For example, the &lt;a href="https://gitlab.haskell.org/ghc/ghc/-/commit/889c084e943779e76d19f2ef5e970ff655f511eb"&gt;first use of GADTs&lt;/a&gt; was introduced by Simon Marlow in 2011, so this suggests going back to at least GHC 7.0.4, maybe earlier.&lt;/p&gt;
  684. &lt;p&gt;Still, being able to mangle the source code before passing it to Hugs is probably a useful thing. This poses the question whether Hugs can compile such a tool; in particular, is it capable of compiling &lt;code&gt;haskell-src-exts&lt;/code&gt;, which I am not too optimistic about either. Did someone check this already?&lt;/p&gt;
  685. &lt;p&gt;So one plan of attack could be&lt;/p&gt;
  686. &lt;ol type="1"&gt;
  687. &lt;li&gt;&lt;p&gt;Identify an old version of GHC that&lt;/p&gt;
  688. &lt;ul&gt;
  689. &lt;li&gt;One &lt;em&gt;can&lt;/em&gt; use to bootstrap subsequent versions until today.&lt;/li&gt;
  690. &lt;li&gt;Is old enough to use as few features not supported by hugs as possible.&lt;/li&gt;
  691. &lt;li&gt;Is still new enough so that one can obtain a compatible toolchain.&lt;/li&gt;
  692. &lt;/ul&gt;&lt;/li&gt;
  693. &lt;li&gt;&lt;p&gt;Wrangle the build system to tell you which files to compile, with which preprocessor flags etc.&lt;/p&gt;&lt;/li&gt;
  694. &lt;li&gt;&lt;p&gt;Boostrap all pre-processing tools used by GHC (&lt;code&gt;cpphs&lt;/code&gt; or use plan cpp, &lt;code&gt;happy&lt;/code&gt;, &lt;code&gt;alex&lt;/code&gt;).&lt;/p&gt;&lt;/li&gt;
  695. &lt;li&gt;&lt;p&gt;For every language feature not supported by Hugs, either&lt;/p&gt;
  696. &lt;ul&gt;
  697. &lt;li&gt;Implement it in Hugs,&lt;/li&gt;
  698. &lt;li&gt;Manually edit the source code to avoid compiling the problematic code, if it is optional (e.g. in an optimization pass)&lt;/li&gt;
  699. &lt;li&gt;Rewrite the problematic code&lt;/li&gt;
  700. &lt;li&gt;Write a pre-processing tool (like the one above) that compiles the feature away&lt;/li&gt;
  701. &lt;/ul&gt;&lt;/li&gt;
  702. &lt;li&gt;&lt;p&gt;Similarly, since Hugs probably ships a &lt;code&gt;base&lt;/code&gt; that is different than what GHC, or the libraries used by GHC expects, either adjust Hugs’ &lt;code&gt;base&lt;/code&gt;, or modify the GHC code that uses it.&lt;/p&gt;&lt;/li&gt;
  703. &lt;/ol&gt;
  704. &lt;p&gt;My actual plan, though, for now is to throw these thoughts out, maybe make some noise on &lt;a href="https://discourse.haskell.org/t/what-s-needed-to-bootstrap-ghc-with-hugs/6205"&gt;Discourse&lt;/a&gt;, &lt;a href="https://mastodon.online/@nomeata/110263917613134533"&gt;Mastodon&lt;/a&gt;, &lt;a href="https://twitter.com/nomeata/status/1651125309700206593"&gt;Twitter&lt;/a&gt; and &lt;a href="https://lobste.rs/s/di37ga"&gt;lobste.rs&lt;/a&gt;, and then let it sit and hope someone else will pick it up.&lt;/p&gt;</description>
  705.         <pubDate>Wed, 26 Apr 2023 07:41:50 +0200</pubDate>
  706.      </item>
  707.      <item>
  708.         <title>rclone, WebDav, Mailbox.org</title>
  709.         <link>https://www.joachim-breitner.de/blog/801-rclone%2C_WebDav%2C_Mailbox_org</link>
  710.         <guid>https://www.joachim-breitner.de/blog/801-rclone%2C_WebDav%2C_Mailbox_org</guid>
  711.         <comments>https://www.joachim-breitner.de/blog/801-rclone%2C_WebDav%2C_Mailbox_org#comments</comments>
  712.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  713.         <description>&lt;p&gt;Just a small googleable note for those trying to set this up too:&lt;/p&gt;
  714. &lt;p&gt;If you try to access your “My Files” &lt;a href="https://mailbox.org/"&gt;mailbox.org&lt;/a&gt; WebDav storage using &lt;a href="https://rclone.org/webdav/"&gt;&lt;code&gt;rclone&lt;/code&gt;&lt;/a&gt;, then the URL is not just &lt;code&gt;https://dav.mailbox.org/servlet/webdav.infostore/&lt;/code&gt;, as written &lt;a href="https://kb.mailbox.org/de/privat/datei-cloud-mailbox-org-drive/webdav-unter-linux"&gt;on the mailbox.org documentation&lt;/a&gt;, but actually &lt;code&gt;https://dav.mailbox.org/servlet/webdav.infostore/Userstore/Your Name&lt;/code&gt;.&lt;/p&gt;
  715. &lt;p&gt;(You can start with &lt;code&gt;https://dav.mailbox.org/servlet/webdav.infostore/&lt;/code&gt; and then use &lt;code&gt;rclone ls&lt;/code&gt; to find out the full path, but then it may be more convenient to change it to point directly to the “My Files” space where you can actually create files.)&lt;/p&gt;</description>
  716.         <pubDate>Wed, 19 Apr 2023 09:39:06 +0200</pubDate>
  717.      </item>
  718.      <item>
  719.         <title>Farewell quimby and fry, welcome richard</title>
  720.         <link>https://www.joachim-breitner.de/blog/800-Farewell_quimby_and_fry%2C_welcome_richard</link>
  721.         <guid>https://www.joachim-breitner.de/blog/800-Farewell_quimby_and_fry%2C_welcome_richard</guid>
  722.         <comments>https://www.joachim-breitner.de/blog/800-Farewell_quimby_and_fry%2C_welcome_richard#comments</comments>
  723.         <author>mail@joachim-breitner.de (Joachim Breitner)</author>
  724.         <description>&lt;p&gt;For a little more than two decades, I have been running one or two dedicated servers for a fair number of services. At some time or the other, it was hosting&lt;/p&gt;
  725. &lt;ul&gt;
  726. &lt;li&gt;A web server for my personal website&lt;/li&gt;
  727. &lt;li&gt;A web server for various private and professional webpages for friends and family&lt;/li&gt;
  728. &lt;li&gt;An email server with IMAP, SMTP, Spam filtering, for me and family&lt;/li&gt;
  729. &lt;li&gt;A mailing list server for various free software project&lt;/li&gt;
  730. &lt;li&gt;A DNS server&lt;/li&gt;
  731. &lt;li&gt;A Half-Life and Counter Strike server, with a statistics web page&lt;/li&gt;
  732. &lt;li&gt;The web page for &lt;a href="https://www.netz-zwerge.de/"&gt;my Counter Strike clan&lt;/a&gt;, running on a custom Perl-and-Mysql based CMS&lt;/li&gt;
  733. &lt;li&gt;The web page for my high school class, running on the same system (this was before everyone used, or had used, Facebook, and even supported tagging people on images…)&lt;/li&gt;
  734. &lt;li&gt;A &lt;a href="https://zpub.de/de/"&gt;Docbook-and-SVN-based documentation management system&lt;/a&gt; that my brother and I built and even sold to a few customers.&lt;/li&gt;
  735. &lt;li&gt;A custom SVN-, Perl and Template-Toolkit based static site generating CMS, before that was cool, also with one or two actual customers.&lt;/li&gt;
  736. &lt;li&gt;A SVN- and later Git based &lt;a href="https://mitschriebwiki.nomeata.de/"&gt;site for collaborative editing of math and computer science lecture notes&lt;/a&gt;; back then there was no Overleaf&lt;/li&gt;
  737. &lt;li&gt;A Jabber server&lt;/li&gt;
  738. &lt;li&gt;The backend for an &lt;a href="https://sumserum.nomeata.de/"&gt;online adaption of the board game “Sim Serim”&lt;/a&gt; which got &lt;a href="https://www.joachim-breitner.de/blog/673-Sim_Serim_geschenkt_bekommen"&gt;the author to gift&lt;/a&gt; me the real thing&lt;/li&gt;
  739. &lt;li&gt;An SVN server&lt;/li&gt;
  740. &lt;li&gt;A darcs server&lt;/li&gt;
  741. &lt;li&gt;A git server&lt;/li&gt;
  742. &lt;li&gt;A tool to track darcs patches that were submitted by mailing lists&lt;/li&gt;
  743. &lt;li&gt;A blog aggregator (a “planet”) for friends, and one for Ghana’s Free Software community&lt;/li&gt;
  744. &lt;li&gt;An Owncloud instance for family&lt;/li&gt;
  745. &lt;li&gt;Virtual machines maintained by friends, to share resources more cheaply&lt;/li&gt;
  746. &lt;li&gt;A OpenVPN and later tinc based VPN for my machines&lt;/li&gt;
  747. &lt;li&gt;Jobs that RSS feeds to IMAP (using &lt;a href="https://github.com/feed2imap/feed2imap"&gt;feed2imap&lt;/a&gt; and later &lt;a href="https://github.com/Necoro/feed2imap-go"&gt;feed2imap-go&lt;/a&gt;)&lt;/li&gt;
  748. &lt;li&gt;Jobs that send email greetings to a friend of mine that I have collected from his wedding guests, and are now delivered at decreasing rate over the next decades.&lt;/li&gt;
  749. &lt;li&gt;Online questionnaires to gather data for a &lt;a href="https://www.joachim-breitner.de/blog/492-401_page_family_book_published"&gt;genealogy project&lt;/a&gt;&lt;/li&gt;
  750. &lt;li&gt;Jobs that send an email with a daily summary of family events from that date.&lt;/li&gt;
  751. &lt;li&gt;A Django app to organize a larger family gathering&lt;/li&gt;
  752. &lt;li&gt;Multiple Mediawiki instances&lt;/li&gt;
  753. &lt;li&gt;A &lt;a href="https://freenetproject.org/"&gt;freenet node&lt;/a&gt; and a &lt;a href="https://www.torproject.org/"&gt;tor node&lt;/a&gt;&lt;/li&gt;
  754. &lt;li&gt;Code that demonstrated the &lt;a href="https://www.joachim-breitner.de/blog/56-Like_XSS,_just_simpler_and_harder_to_prevent__The_Cross_Site_Auth_(XSA)_Attack"&gt;Cross-site authentication attack&lt;/a&gt;&lt;/li&gt;
  755. &lt;li&gt;… and probably more stuff that I don’t remember anymore&lt;/li&gt;
  756. &lt;/ul&gt;
  757. &lt;h3 id="its-not-you-its-me"&gt;It’s not you, it’s me&lt;/h3&gt;
  758. &lt;p&gt;Running this kind of service on my own was great fun and a huge learning experience, and in some cases at that time the only way to have a certain feature. But in recent years my interests shifted a bit, more into Programming Languages Theory (and practice) than Devops, and I was no longer paying attention as much as these services require. This gave me a bad conscience, especially in terms of security updates.&lt;/p&gt;
  759. &lt;p&gt;Especially running your own email server, while still possible, isn’t fire-and-forget: One has to stay on top of spam protection measures, both on the receiving end (spamassassin etc.) as well as when sending email (DKIM, configuring mailing lists to rewrite sender etc.)&lt;/p&gt;
  760. &lt;p&gt;Also some family members were commercially relying on these servers and services working, which was no longer tenable.&lt;/p&gt;
  761. &lt;h3 id="weaning-off"&gt;Weaning off&lt;/h3&gt;
  762. &lt;p&gt;Therefore, more than a year ago, I decided to wind this down. Turns out that getting rid of responsibilities takes at least as long as taking them on, especially if your “customers” are content and a move to something else is but an annoyance. But last weekend I was finally able to turn the machines, called quimby and fry, off.&lt;/p&gt;
  763. &lt;p&gt;Many of the services above were already no longer active when I started the weaning off (Jabber, Freenet, Tor). So what happened to the rest?&lt;/p&gt;
  764. &lt;ul&gt;
  765. &lt;li&gt;For emails, we all moved to &lt;a href="https://mailbox.org" class="uri"&gt;https://mailbox.org&lt;/a&gt;. Happy to pay for such a crucial service.&lt;/li&gt;
  766. &lt;li&gt;For the mailing lists, especially for the &lt;a href="https://tttool.entropia.de/"&gt;Tip-Toi-Hacking project&lt;/a&gt; , &lt;a href="https://jpberlin.de/" class="uri"&gt;https://jpberlin.de/&lt;/a&gt; has decent enough rates that I don’t feel bad for paying for it.&lt;/li&gt;
  767. &lt;li&gt;Managing DNS is made super easy by &lt;a href="https://github.com/StackExchange/dnscontrol"&gt;dnscontrol&lt;/a&gt;; I’m using Hetzner’s DNS servers, but thanks to that tool that does not matter a lot&lt;/li&gt;
  768. &lt;li&gt;For those websites that are static enough, Github pages is nice. This includes the lecture notes site.&lt;/li&gt;
  769. &lt;li&gt;For those websites that need a little bit of server-side logic, e.g. for complex redirects and normalization procedures, or access control, but no state, I found that netlify introduced their &lt;a href="https://docs.netlify.com/edge-functions/overview/"&gt;Edge functions&lt;/a&gt; feature just in time. This was crucial for my main website.&lt;/li&gt;
  770. &lt;li&gt;I got rid of the backend of the two-player game &lt;a href="https://sumserum.nomeata.de/"&gt;Sum Serum&lt;/a&gt; completely, by using &lt;a href="https://www.joachim-breitner.de/blog/797-Serverless_WebRTC_instead_of_websocket"&gt;WebRTC&lt;/a&gt;. In terms of hosting, it is now just a boring static website, which is the best kind of website.&lt;/li&gt;
  771. &lt;li&gt;I converted all SVN and Darcs repositories to git, and pushed them to GitHub.&lt;/li&gt;
  772. &lt;li&gt;Although I was mildly proud to have kept the websites of my high school class and Counter Strike clan live and functional for many years after anyone stopped caring about them, I decided it is silly to keep doing that. I briefly thought about entombing that running system in a locked down virtual machine or so, but in the end simply used &lt;code&gt;wget&lt;/code&gt; to create a static mirror of them, which is now hosted on netlify (using Edge Functions to restrict public access to a few pages).&lt;/li&gt;
  773. &lt;/ul&gt;
  774. &lt;p&gt;In the end, I was not able to get rid of all services, so there is still a server running for me (smaller now, and virtual):&lt;/p&gt;
  775. &lt;ul&gt;
  776. &lt;li&gt;My photo album, &lt;a href="https://bilder.joachim-breitner.de/" class="uri"&gt;https://bilder.joachim-breitner.de/&lt;/a&gt;, which is a bit too big for something like netlify.&lt;/li&gt;
  777. &lt;li&gt;Some dumb static webspace used by a family member’s business for internal reasons, which likewise is a bit too large for something like netlify or github actions, and not as critical as other sites.&lt;/li&gt;
  778. &lt;li&gt;The feed2imap jobs&lt;/li&gt;
  779. &lt;li&gt;For one Mediawiki instanced used for a genealogy project of a relative of mine I could not find a suitable hosted alternative. However, it is HTTP-AUTH-password-protected, so I am a bit less worried about staying on top of security updates for this PHP-drive site.&lt;/li&gt;
  780. &lt;li&gt;Also, I am still running the daily email job from the genealogy project, but now through &lt;code&gt;nullmailer&lt;/code&gt; and &lt;a href="https://aws.amazon.com/de/sns/"&gt;Amazon SNS&lt;/a&gt;, to worry a bit less about the intricacies of modern email.&lt;/li&gt;
  781. &lt;/ul&gt;
  782. &lt;h3 id="debian-nix"&gt;Debian → Nix&lt;/h3&gt;
  783. &lt;p&gt;I took this opportunity to set up a new server for the residual services. I have been using Debian since 2001 and was a Debian Developer from 2003 to 2022, and it is a great operating system and a great project.&lt;/p&gt;
  784. &lt;p&gt;But after learning about &lt;a href="https://nixos.org/"&gt;Nix&lt;/a&gt; at &lt;a href="https://dfinity.org/"&gt;DFINITY&lt;/a&gt;, and using NixOS on my laptop for almost two years, managing a system in a non-declarative way simply feels … wrong. Similar to programming in a non-functional programming language.&lt;/p&gt;
  785. &lt;p&gt;Hence I have shut down my Debian-based systems (two virtual machines called &lt;code&gt;quimby&lt;/code&gt; and &lt;code&gt;fry&lt;/code&gt; and the surrounding Dom0 &lt;code&gt;freddy&lt;/code&gt; – my machines are named after Simpsons side kicks and (sometimes) Futurama characters), and am using the NixOS-based aarch64 host &lt;code&gt;richard&lt;/code&gt; (because of the [“Nix” in the name]) instead.&lt;/p&gt;</description>
  786.         <pubDate>Mon, 17 Apr 2023 08:33:24 +0200</pubDate>
  787.      </item>
  788.   </channel>
  789. </rss>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

  1. Download the "valid RSS" banner.

  2. Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)

  3. Add this HTML to your page (change the image src attribute if necessary):

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//www.joachim-breitner.de/blog/feeds/categories/1-English.rss

Home · About · News · Docs · Terms

 
Copyright © 2002-9 Sam Ruby, Mark Pilgrim, Joseph Walton, and Phil Ringnalda