FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 35575: content should not contain data-slug attribute (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

line 1, column 35575: content should not contain script tag (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

line 1, column 218421: content should not contain style tag [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

Source: http://blog.golang.org/feed.atom

<feed xmlns="http://www.w3.org/2005/Atom"><title>The Go Blog</title><id>tag:blog.golang.org,2013:blog.golang.org</id><link rel="self" href="https://go.dev/blog/feed.atom"></link><updated>2025-07-07T00:00:00+00:00</updated><entry><title>Generic interfaces</title><id>tag:blog.golang.org,2013:blog.golang.org/generic-interfaces</id><link rel="alternate" href="https://go.dev/blog/generic-interfaces"></link><published>2025-07-07T00:00:00+00:00</published><updated>2025-07-07T00:00:00+00:00</updated><author><name>Axel Wagner</name></author><summary type="html">Adding type parameters to interface types is surprisingly powerful</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/generic-interfaces">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Generic interfaces</h1>
 
 <p class="author">
 Axel Wagner<br>
 7 July 2025
 </p>
 
 <p>There is an idea that is not obvious until you hear about it for the first time: as interfaces are types themselves, they too can have type parameters.
This idea proves to be surprisingly powerful when it comes to expressing constraints on generic functions and types.
In this post, we&rsquo;ll demonstrate it, by discussing the use of interfaces with type parameters in a couple of common scenarios.</p>
<h2 id="a-simple-tree-set">A simple tree set</h2>
<p>As a motivating example, assume we need a generic version of a <a href="https://en.wikipedia.org/wiki/Binary_search_tree" rel="noreferrer" target="_blank">binary search tree</a>.
The elements stored in such a tree need to be ordered, so our type parameter needs a constraint that determines the ordering to use.
A simple option is to use the <a href="/pkg/cmp#Ordered">cmp.Ordered</a> constraint, introduced in Go 1.21.
It restricts a type parameter to ordered types (strings and numbers) and allows methods of the type to use the built-in ordering operators.</p>
<pre><code>// The zero value of a Tree is a ready-to-use empty tree.
type Tree[E cmp.Ordered] struct {
 root *node[E]
}

func (t *Tree[E]) Insert(element E) {
 t.root = t.root.insert(element)
}

type node[E cmp.Ordered] struct {
 value E
 left *node[E]
 right *node[E]
}

func (n *node[E]) insert(element E) *node[E] {
 if n == nil {
 return &amp;node[E]{value: element}
 }
 switch {
 case element &lt; n.value:
 n.left = n.left.insert(element)
 case element &gt; n.value:
 n.right = n.right.insert(element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/H7-n33X7P2h">playground</a>)</p>
<p>However, this approach has the disadvantage that it only works on basic types for which <code>&lt;</code> is defined;
you cannot insert struct types, like <a href="/pkg/time#Time">time.Time</a>.</p>
<p>We can remedy that by requiring the user to provide a comparison function:</p>
<pre><code>// A FuncTree must be created with NewTreeFunc.
type FuncTree[E any] struct {
 root *funcNode[E]
 cmp func(E, E) int
}

func NewFuncTree[E any](cmp func(E, E) int) *FuncTree[E] {
 return &amp;FuncTree[E]{cmp: cmp}
}

func (t *FuncTree[E]) Insert(element E) {
 t.root = t.root.insert(t.cmp, element)
}

type funcNode[E any] struct {
 value E
 left *funcNode[E]
 right *funcNode[E]
}

func (n *funcNode[E]) insert(cmp func(E, E) int, element E) *funcNode[E] {
 if n == nil {
 return &amp;funcNode[E]{value: element}
 }
 sign := cmp(element, n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(cmp, element)
 case sign &gt; 0:
 n.right = n.right.insert(cmp, element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/tiEjuxCHtFF">playground</a>)</p>
<p>This works, but it also comes with downsides.
We can no longer use the zero value of our container type, because it needs to have an explicitly initialized comparison function.
And the use of a function field makes it harder for the compiler to inline the comparison calls, which can introduce a significant runtime overhead.</p>
<p>Using a method on the element type can solve these issues, because methods are directly associated with a type.
A method does not have to be explicitly passed and the compiler can see the target of the call and may be able to inline it.
But how can we express the constraint to require that element types provide the necessary method?</p>
<h2 id="using-the-receiver-in-constraints">Using the receiver in constraints</h2>
<p>The first approach we might try is to define a plain old interface with a <code>Compare</code> method:</p>
<pre><code>type Comparer interface {
 Compare(Comparer) int
}
</code></pre>
<p>However, we quickly realize that this does not work well.
To implement this interface, the method&rsquo;s parameter must itself be <code>Comparer</code>.
Not only does that mean that the implementation of this method must type-assert the parameter to its own type, it also requires that every type must explicitly refer to our package with the <code>Comparer</code> type by name (otherwise the method signatures would not be identical).
That is not very orthogonal.</p>
<p>A better approach is to make the <code>Comparer</code> interface itself generic:</p>
<pre><code>type Comparer[T any] interface {
 Compare(T) int
}
</code></pre>
<p>This <code>Comparer</code> now describes a whole family of interfaces, one for each type that <code>Comparer</code> may be instantiated with.
A type that implements <code>Comparer[T]</code> declares &ldquo;I can compare myself to a <code>T</code>&rdquo;.
For instance, <code>time.Time</code> naturally implements <code>Comparer[time.Time]</code> because <a href="/pkg/time#Time.Compare">it has a matching <code>Compare</code> method</a>:</p>
<pre><code>// Implements Comparer[Time]
func (t Time) Compare(u Time) int
</code></pre>
<p>This is better, but not enough.
What we really want is a constraint that says that a type parameter can be compared to <em>itself</em>: we want the constraint to be self-referential.
The subtle insight is that the self-referential aspect does not have to be part of the interface definition itself; specifically, the constraint for <code>T</code> in the <code>Comparer</code> type is just <code>any</code>.
Instead, it is a consequence of how we use <code>Comparer</code> as a constraint for the type parameter of <code>MethodTree</code>:</p>
<pre><code>// The zero value of a MethodTree is a ready-to-use empty tree.
type MethodTree[E Comparer[E]] struct {
 root *methodNode[E]
}

func (t *MethodTree[E]) Insert(element E) {
 t.root = t.root.insert(element)
}

type methodNode[E Comparer[E]] struct {
 value E
 left *methodNode[E]
 right *methodNode[E]
}

func (n *methodNode[E]) insert(element E) *methodNode[E] {
 if n == nil {
 return &amp;methodNode[E]{value: element}
 }
 sign := element.Compare(n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(element)
 case sign &gt; 0:
 n.right = n.right.insert(element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/LuhzYej_2SP">playground</a>)</p>
<p>Because <code>time.Time</code> implements <code>Comparer[time.Time]</code> it is now a valid type argument for this container, and we can still use the zero value as an empty container:</p>
<pre><code>var t MethodTree[time.Time]
t.Insert(time.Now())
</code></pre>
<p>For full flexibility, a library can provide all three API versions.
If we want to minimize repetition, all versions could use a shared implementation.
We could use the function version for that, as it is the most general:</p>
<pre><code>type node[E any] struct {
 value E
 left *node[E]
 right *node[E]
}

func (n *node[E]) insert(cmp func(E, E) int, element E) *node[E] {
 if n == nil {
 return &amp;node[E]{value: element}
 }
 sign := cmp(element, n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(cmp, element)
 case sign &gt; 0:
 n.right = n.right.insert(cmp, element)
 }
 return n
}

// Insert inserts element into the tree, if E implements cmp.Ordered.
func (t *Tree[E]) Insert(element E) {
 t.root = t.root.insert(cmp.Compare[E], element)
}

// Insert inserts element into the tree, using the provided comparison function.
func (t *FuncTree[E]) Insert(element E) {
 t.root = t.root.insert(t.cmp, element)
}

// Insert inserts element into the tree, if E implements Comparer[E].
func (t *MethodTree[E]) Insert(element E) {
 t.root = t.root.insert(E.Compare, element)
}
</code></pre>
<p>(<a href="/play/p/jzmoaH5eaIv">playground</a>)</p>
<p>An important observation here is that the shared implementation (the function-based variant) is not constrained in any way.
It must remain maximally flexible to serve as a common core.
We also do not store the comparison function in a struct field.
Instead, we pass it as a parameter because function arguments are easier for the compiler to analyze than struct fields.</p>
<p>There is still some amount of boilerplate involved, of course.
All the exported implementations need to replicate the full API with slightly different call patterns.
But this part is straightforward to write and to read.</p>
<h2 id="combining-methods-and-type-sets">Combining methods and type sets</h2>
<p>We can use our new tree data structure to implement an ordered set, providing element lookup in logarithmic time.
Let&rsquo;s now imagine we need to make lookup run in constant time; we might try to do this by maintaining an ordinary Go map alongside the tree:</p>
<pre><code>type OrderedSet[E Comparer[E]] struct {
 tree MethodTree[E] // for efficient iteration in order
 elements map[E]bool // for (near) constant time lookup
}

func (s *OrderedSet[E]) Has(e E) bool {
 return s.elements[e]
}

func (s *OrderedSet[E]) Insert(e E) {
 if s.elements == nil {
 s.elements = make(map[E]bool)
 }
 if s.elements[e] {
 return
 }
 s.elements[e] = true
 s.tree.Insert(e)
}

func (s *OrderedSet[E]) All() iter.Seq[E] {
 return func(yield func(E) bool) {
 s.tree.root.all(yield)
 }
}

func (n *node[E]) all(yield func(E) bool) bool {
 return n == nil || (n.left.all(yield) &amp;&amp; yield(n.value) &amp;&amp; n.right.all(yield))
}
</code></pre>
<p>(<a href="/play/p/TANUnnSnDqf">playground</a>)</p>
<p>However, compiling this code will produce an error:</p>
<blockquote>
<p>invalid map key type E (missing comparable constraint)</p>
</blockquote>
<p>The error message tells us that we need to further constrain our type parameter to be able to use it as a map key.
The <code>comparable</code> constraint is a special predeclared constraint that is satisfied by all types for which the equality operators <code>==</code> and <code>!=</code> are defined.
In Go, that is also the set of types which can be used as keys for the built-in <code>map</code> type.</p>
<p>We have three options to add this constraint to our type parameter, all with different tradeoffs:</p>
<ol>
<li>
<p>We can <a href="/ref/spec#Embedded_interfaces">embed</a> <code>comparable</code> into our original <code>Comparer</code> definition (<a href="/play/p/g8NLjZCq97q">playground</a>):</p>
<pre><code>type Comparer[E any] interface {
 comparable
 Compare(E) int
}
</code></pre>
<p>This has the downside that it would also make our <code>Tree</code> types only usable with types that are <code>comparable</code>.
In general, we do not want to unnecessarily restrict generic types.</p>
</li>
<li>
<p>We can add a new constraint definition (<a href="/play/p/Z2eg4X8xK5Z">playground</a>).</p>
<pre><code>type Comparer[E any] interface {
 Compare(E) int
}

type ComparableComparer[E any] interface {
 comparable
 Comparer[E]
}
</code></pre>
<p>This is tidy, but it introduces a new identifier (<code>ComparableComparer</code>) into our API, and naming is hard.</p>
</li>
<li>
<p>We can add the constraint inline into our more constrained type (<a href="/play/p/ZfggVma_jNc">playground</a>):</p>
<pre><code>type OrderedSet[E interface {
 comparable
 Comparer[E]
}] struct {
 tree Tree[E]
 elements map[E]struct{}
}
</code></pre>
<p>This can become a bit hard to read, especially if it needs to happen often.
It also makes it harder to reuse the constraint in other places.</p>
</li>
</ol>
<p>Which of these to use is a style choice and ultimately up to personal preference.</p>
<h2 id="not-constraining-generic-interfaces">(Not) constraining generic interfaces</h2>
<p>At this point it is worth discussing constraints on generic interfaces.
You might want to define an interface for a generic container type.
For example, say you have an algorithm that requires a set data structure.
There are many different kinds of set implementations with different tradeoffs.
Defining an interface for the set operations you require can add flexibility to your package, leaving the decision of what tradeoffs are right for the specific application to the user:</p>
<pre><code>type Set[E any] interface {
 Insert(E)
 Delete(E)
 Has(E) bool
 All() iter.Seq[E]
}
</code></pre>
<p>A natural question here is what the constraint on this interface should be.
If possible, type parameters on generic interfaces should use <code>any</code> as a constraint, allowing arbitrary types.</p>
<p>From our discussions above, the reasons should be clear:
Different concrete implementations might require different constraints.
All the <code>Tree</code> types we have examined above, as well as the <code>OrderedSet</code> type, can implement <code>Set</code> for their element types, even though these types have different constraints.</p>
<p>The point of defining an interface is to leave the implementation up to the user.
Since one cannot predict what kinds of constraints a user may want to impose on their implementation, try to leave any constraints (stronger than <code>any</code>) to concrete implementations, not the interfaces.</p>
<h2 id="pointer-receivers">Pointer receivers</h2>
<p>Let us try to use the <code>Set</code> interface in an example.
Consider a function that removes duplicate elements in a sequence:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E comparable](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 seen := make(map[E]bool)
 for v := range input {
 if seen[v] {
 continue
 }
 if !yield(v) {
 return
 }
 seen[v] = true
 }
 }
}
</code></pre>
<p>(<a href="/play/p/hsYoFjkU9kA">playground</a>)</p>
<p>This uses a <code>map[E]bool</code> as a simple set of <code>E</code> elements.
Consequently, it works only for types that are <code>comparable</code> and which therefore define the built-in equality operators.
If we want to generalize this to arbitrary types, we need to replace that with a generic set:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E any](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 var seen Set[E]
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/FZYPNf56nnY">playground</a>)</p>
<p>However, this does not work.
<code>Set[E]</code> is an interface type, and the <code>seen</code> variable will be initialized to <code>nil</code>.
We need to use a concrete implementation of the <code>Set[E]</code> interface.
But as we have seen in this post, there is no general implementation of a set that works for <code>any</code> element type.</p>
<p>We have to ask the user to provide a concrete implementation we can use, as an extra type parameter:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E any, S Set[E]](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 var seen S
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/kjkGy5cNz8T">playground</a>)</p>
<p>However, if we instantiate this with our set implementation, we run into another problem:</p>
<pre><code>// OrderedSet[E] does not satisfy Set[E] (method All has pointer receiver)
Unique[E, OrderedSet[E]](slices.Values(s))
// panic: invalid memory address or nil pointer dereference
Unique[E, *OrderedSet[E]](slices.Values(s))
</code></pre>
<p>The first problem is clear from the error message: Our type constraint says that the type argument for <code>S</code> needs to implement the <code>Set[E]</code> interface.
And as the methods on <code>OrderedSet</code> use a pointer receiver, the type argument also has to be the pointer type.</p>
<p>When trying to do that, we run into the second problem.
This stems from the fact that we declare a variable in the implementation:</p>
<pre><code>var seen S
</code></pre>
<p>If <code>S</code> is <code>*OrderedSet[E]</code>, the variable is initialized with <code>nil</code>, as before.
Calling <code>seen.Insert</code> panics.</p>
<p>If we only have the pointer type, we cannot get a valid variable of the value type.
And if we only have the value type, we cannot call pointer-methods on it.
The consequence is that we need both the value <em>and</em> the pointer type.
So we have to introduce an additional type parameter <code>PS</code> with a new constraint <code>PtrToSet</code>:</p>
<pre><code>// PtrToSet is implemented by a pointer type implementing the Set[E] interface.
type PtrToSet[S, E any] interface {
 *S
 Set[E]
}

// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E, S any, PS PtrToSet[S, E]](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 // We convert to PS, as only that is constrained to have the methods.
 // The conversion is allowed, because the type set of PS only contains *S.
 seen := PS(new(S))
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/Kp1jJRVjmYa">playground</a>)</p>
<p>The trick here is the connection of the two type parameters in the function signature via the extra type parameter on the <code>PtrToSet</code> interface.
<code>S</code> itself is unconstrained, but <code>PS</code> must have type <code>*S</code> and it must have the methods we need.
So effectively, we are restricting <code>S</code> to have some methods, but those methods need to use a pointer receiver.</p>
<p>While the definition of a function with this kind of constraint requires an additional type parameter, importantly this does not complicate code using it:
as long as this extra type parameter is at the end of the type parameter list, it <a href="/blog/type-inference">can be inferred</a>:</p>
<pre><code>// The third type argument is inferred to be *OrderedSet[int]
Unique[int, OrderedSet[int]](slices.Values(s))
</code></pre>
<p>This is a general pattern, and worth remembering: for when you encounter it in someone else&rsquo;s work, or when you want to use it in your own.</p>
<pre><code>func SomeFunction[T any, PT interface{ *T; SomeMethods }]()
</code></pre>
<p>If you have two type parameters, where one is constrained to be a pointer to the other, the constraint ensures that the relevant methods use a pointer receiver.</p>
<h2 id="should-you-constrain-to-pointer-receivers">Should you constrain to pointer receivers?</h2>
<p>At this point, you might feel pretty overwhelmed.
This is rather complicated and it seems unreasonable to expect every Go programmer to understand what is going on in this function signature.
We also had to introduce yet more names into our API.
When people cautioned against adding generics to Go in the first place, this is one of the things they were worried about.</p>
<p>So if you find yourself entangled in these problems, it is worth taking a step back.
We can often avoid this complexity by thinking about our problem in a different way.
In this example, we built a function that takes an <code>iter.Seq[E]</code> and returns an <code>iter.Seq[E]</code> with the unique elements.
But to do the deduplication, we needed to collect the unique elements into a set.
And as this requires us to allocate the space for the entire result, we do not really benefit from representing the result as a stream.</p>
<p>If we rethink this problem, we can avoid the extra type parameter altogether by using <code>Set[E]</code> as a regular interface value:</p>
<pre><code>// InsertAll adds all unique elements from seq into set.
func InsertAll[E any](set Set[E], seq iter.Seq[E]) {
 for v := range seq {
 set.Insert(v)
 }
}
</code></pre>
<p>(<a href="/play/p/woZcHodAgaa">playground</a>)</p>
<p>By using <code>Set</code> as a plain interface type, it is clear that the caller has to provide a valid value of their concrete implementation.
This is a very common pattern.
And if they need an <code>iter.Seq[E]</code>, they can simply call <code>All()</code> on the <code>set</code> to obtain one.</p>
<p>This complicates things for callers slightly, but it has another advantage over the constraint to pointer receivers:
remember that we started with a <code>map[E]bool</code> as a simple set type.
It is easy to implement the <code>Set[E]</code> interface on that basis:</p>
<pre><code>type HashSet[E comparable] map[E]bool

func (s HashSet[E]) Insert(v E) { s[v] = true }
func (s HashSet[E]) Delete(v E) { delete(s, v) }
func (s HashSet[E]) Has(v E) bool { return s[v] }
func (s HashSet[E]) All() iter.Seq[E] { return maps.Keys(s) }
</code></pre>
<p>(<a href="/play/p/KPPpWa7M93d">playground</a>)</p>
<p>This implementation does not use pointer receivers.
So while this is perfectly valid, it would not be usable with the complicated constraint to pointer receivers.
But it works fine with our <code>InsertAll</code> version.
As with many constraints, enforcing that methods use a pointer receiver might actually be overly restrictive for many practical use cases.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope this illustrates some of the patterns and trade-offs that type parameters on interfaces enable.
It is a powerful tool, but it also comes with a cost.
The primary take-aways are:</p>
<ol>
<li>Use generic interfaces to express constraints on the receiver by using them self-referentially.</li>
<li>Use them to create constrained relationships between different type parameters.</li>
<li>Use them to abstract over different implementations with different kinds of constraints.</li>
<li>When you find yourself in a situation where you need to constrain to pointer receivers, consider whether you can refactor your code to avoid the extra complexity. See <a href="#should-you-constrain-to-pointer-receivers">&ldquo;Should you constrain to pointer receivers?&rdquo;</a>.</li>
</ol>
<p>As always, do not over-engineer things: a less flexible but simpler and more readable solution may ultimately be the wiser choice.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 <p>
 
 
 
 <b>Previous article: </b><a href="/blog/error-syntax">[ On | No ] syntactic support for error handling</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>[ On | No ] syntactic support for error handling</title><id>tag:blog.golang.org,2013:blog.golang.org/error-syntax</id><link rel="alternate" href="https://go.dev/blog/error-syntax"></link><published>2025-06-03T00:00:00+00:00</published><updated>2025-06-03T00:00:00+00:00</updated><author><name>Robert Griesemer</name></author><summary type="html">Go team plans around error handling support</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/error-syntax">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>[ On | No ] syntactic support for error handling</h1>
 
 <p class="author">
 Robert Griesemer<br>
 3 June 2025
 </p>
 
 <p>One of the oldest and most persistent complaints about Go concerns the verbosity of error handling.
We are all intimately (some may say painfully) familiar with this code pattern:</p>
<pre><code class="language-Go">x, err := call()
if err != nil {
 // handle err
}
</code></pre>
<p>The test <code>if err != nil</code> can be so pervasive that it drowns out the rest of the code.
This typically happens in programs that do a lot of API calls, and where handling errors
is rudimentary and they are simply returned.
Some programs end up with code that looks like this:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err := strconv.Atoi(a)
 if err != nil {
 return err
 }
 y, err := strconv.Atoi(b)
 if err != nil {
 return err
 }
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>Of the ten lines of code in this function body, only four (the calls and the last two lines) appear to do real work.
The remaining six lines come across as noise.
The verbosity is real, and so it&rsquo;s no wonder that complaints about error handling have topped
our annual user surveys for years.
(For a while, the lack of generics surpassed complaints about error handling, but now that
Go supports generics, error handling is back on top.)</p>
<p>The Go team takes community feedback seriously, and so for many years now we have tried to
come up with a solution for this problem, together with input from the Go community.</p>
<p>The first explicit attempt by the Go team dates back to 2018, when Russ Cox
<a href="https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.md" rel="noreferrer" target="_blank">formally described the problem</a>
as part of what we called the Go 2 effort at that time.
He outlined a possible solution based on a
<a href="https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling.md" rel="noreferrer" target="_blank">draft design</a>
by Marcel van Lohuizen.
The design was based on a <code>check</code> and <code>handle</code> mechanism and was fairly comprehensive.
The draft includes a detailed analysis of alternative solutions, including comparisons with
approaches taken by other languages.
If you&rsquo;re wondering if your particular error handling idea was previously considered,
read this document!</p>
<pre><code class="language-Go">// printSum implementation using the proposed check/handle mechanism.
func printSum(a, b string) error {
 handle err { return err }
 x := check strconv.Atoi(a)
 y := check strconv.Atoi(b)
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>The <code>check</code> and <code>handle</code> approach was deemed too complicated and almost a year later, in 2019,
we followed up with the much simplified and by now
<a href="/issue/32437#issuecomment-2278932700">infamous</a>
<a href="https://go.googlesource.com/proposal/+/master/design/32437-try-builtin.md" rel="noreferrer" target="_blank"><code>try</code> proposal</a>.
It was based on the ideas of <code>check</code> and <code>handle</code>, but the <code>check</code> pseudo-keyword became
the <code>try</code> built-in function and the <code>handle</code> part was omitted.
To explore the impact of the <code>try</code> built-in, we wrote a simple tool
(<a href="https://github.com/griesemer/tryhard" rel="noreferrer" target="_blank">tryhard</a>)
that rewrites existing error handling code using <code>try</code>.
The proposal was argued over intensively, approaching 900 comments on the <a href="/issue/32437">GitHub issue</a>.</p>
<pre><code class="language-Go">// printSum implementation using the proposed try mechanism.
func printSum(a, b string) error {
 // use a defer statement to augment errors before returning
 x := try(strconv.Atoi(a))
 y := try(strconv.Atoi(b))
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>However, <code>try</code> affected control flow by returning from the enclosing function in case of an error,
and did so from potentially deeply nested expressions, thus hiding this control flow from view.
This made the proposal unpalatable to many, and despite significant investment
into this proposal we decided to abandon this effort too.
In retrospect it might have been better to introduce a new keyword,
something that we could do now since we have fine-grained control over the language version
via <code>go.mod</code> files and file-specific directives.
Restricting the use of <code>try</code> to assignments and statements might have alleviated some
of the other concerns. A <a href="/issue/73376">recent proposal</a> by Jimmy Frasche, which essentially
goes back to the original <code>check</code> and <code>handle</code> design and addresses some of that design&rsquo;s
shortcomings, pursues that direction.</p>
<p>The repercussions of the <code>try</code> proposal led to much soul searching including a series of blog
posts by Russ Cox: <a href="https://research.swtch.com/proposals-intro" rel="noreferrer" target="_blank">&ldquo;Thinking about the Go Proposal Process&rdquo;</a>.
One conclusion was that we likely diminished our chances for a better outcome by presenting an almost
fully baked proposal with little space for community feedback and a &ldquo;threatening&rdquo; implementation
timeline. Per <a href="https://research.swtch.com/proposals-large" rel="noreferrer" target="_blank">&ldquo;Go Proposal Process: Large Changes&rdquo;</a>:
&ldquo;in retrospect, <code>try</code> was a large enough change that the new design we published [&hellip;] should have
been a second draft design, not a proposal with an implementation timeline&rdquo;.
But irrespective of a possible process and communication failure in this case, the user sentiment towards
the proposal was very strongly not in favor.</p>
<p>We didn&rsquo;t have a better solution at that time and didn&rsquo;t pursue syntax changes for error handling for several years.
Plenty of people in the community were inspired, though, and we received a steady trickle
of error handling proposals, many very similar to each other, some interesting, some incomprehensible,
and some infeasible.
To keep track of the expanding landscape, another year later, Ian Lance Taylor created an
<a href="/issue/40432">umbrella issue</a>
which summarizes the current state of proposed changes for improved error handling.
A <a href="/wiki/Go2ErrorHandlingFeedback">Go Wiki</a> was created to collect related feedback, discussions, and articles.
Independently, other people have started tracking all the many error handling proposals
over the years.
It&rsquo;s amazing to see the sheer volume of them all, for instance in Sean K. H. Liao&rsquo;s blog post on
<a href="https://seankhliao.com/blog/12020-11-23-go-error-handling-proposals/" rel="noreferrer" target="_blank">&ldquo;go error handling proposals&rdquo;</a>.</p>
<p>The complaints about the verbosity of error handling persisted
(see <a href="/blog/survey2024-h1-results">Go Developer Survey 2024 H1 Results</a>),
and so, after a series of increasingly refined Go team internal proposals, Ian Lance Taylor published
<a href="/issue/71203">&ldquo;reduce error handling boilerplate using <code>?</code>&rdquo;</a> in 2024.
This time the idea was to borrow from a construct implemented in
<a href="https://www.rust-lang.org/" rel="noreferrer" target="_blank">Rust</a>, specifically the
<a href="https://doc.rust-lang.org/std/result/index.html#the-question-mark-operator-" rel="noreferrer" target="_blank"><code>?</code> operator</a>.
The hope was that by leaning on an existing mechanism using an established notation, and taking into
account what we had learned over the years, we should be able to finally make some progress.
In small informal user studies where programmers were shown Go code using <code>?</code>, the vast majority
of participants correctly guessed the meaning of the code, which further convinced us to give it another
shot.
To be able to see the impact of the change, Ian wrote a tool that converts ordinary Go code
into code that uses the proposed new syntax, and we also prototyped the feature in the
compiler.</p>
<pre><code class="language-Go">// printSum implementation using the proposed &quot;?&quot; statements.
func printSum(a, b string) error {
 x := strconv.Atoi(a) ?
 y := strconv.Atoi(b) ?
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>Unfortunately, as with the other error handling ideas, this new proposal was also quickly overrun
with comments and many suggestions for minor tweaks, often based on individual preferences.
Ian closed the proposal and moved the content into a <a href="/issue/71460">discussion</a>
to facilitate the conversation and to collect further feedback.
A slightly modified version was received
<a href="https://github.com/golang/go/discussions/71460#discussioncomment-12060294" rel="noreferrer" target="_blank">a bit more positively</a>
but broad support remained elusive.</p>
<p>After so many years of trying, with three full-fledged proposals by the Go team and
literally <a href="/issues?q=+is%3Aissue+label%3Aerror-handling">hundreds</a> (!)
of community proposals, most of them variations on a theme,
all of which failed to attract sufficient (let alone overwhelming) support,
the question we now face is: how to proceed? Should we proceed at all?</p>
<p><em>We think not.</em></p>
<p>To be more precise, we should stop trying to solve the <em>syntactic problem</em>, at least for the foreseeable
future.
The <a href="https://github.com/golang/proposal?tab=readme-ov-file#consensus-and-disagreement" rel="noreferrer" target="_blank">proposal process</a>
provides justification for this decision:</p>
<blockquote>
<p>The goal of the proposal process is to reach general consensus about the outcome in a timely manner.
If proposal review cannot identify a general consensus in the discussion of the issue on the issue tracker,
the usual result is that the proposal is declined.</p>
</blockquote>
<p>Furthermore:</p>
<blockquote>
<p>It can happen that proposal review may not identify a general consensus and yet it is clear that the
proposal should not be outright declined.
[&hellip;]
If the proposal review group cannot identify a consensus nor a next step for the proposal,
the decision about the path forward passes to the Go architects [&hellip;], who review the discussion and
aim to reach a consensus among themselves.</p>
</blockquote>
<p>None of the error handling proposals reached anything close to a consensus,
so they were all declined.
Even the most senior members of the Go team at Google do not unanimously agree
on the best path forward <em>at this time</em> (perhaps that will change at some point).
But without a strong consensus we cannot reasonably move forward.</p>
<p>There are valid arguments in favor of the status quo:</p>
<ul>
<li>
<p>If Go had introduced specific syntactic sugar for error handling early on, few would argue over it today.
But we are 15 years down the road, the opportunity has passed, and Go has
a perfectly fine way to handle errors, even if it may seem verbose at times.</p>
</li>
<li>
<p>Looking from a different angle, let&rsquo;s assume we came across the perfect solution today.
Incorporating it into the language would simply lead from one unhappy group of users
(the one that roots for the change) to another (the one that prefers the status quo).
We were in a similar situation when we decided to add generics to the language, albeit with an
important difference:
today nobody is forced to use generics, and good generic libraries are written such that users
can mostly ignore the fact that they are generic, thanks to type inference.
On the contrary, if a new syntactic construct for error handling gets added to the language,
virtually everybody will need to start using it, lest their code become unidiomatic.</p>
</li>
<li>
<p>Not adding extra syntax is in line with one of Go&rsquo;s design rules:
do not provide multiple ways of doing the same thing.
There are exceptions to this rule in areas with high &ldquo;foot traffic&rdquo;: assignments come to mind.
Ironically, the ability to <em>redeclare</em> a variable in
<a href="/ref/spec#Short_variable_declarations">short variable declarations</a> (<code>:=</code>) was introduced to address a problem
that arose because of error handling:
without redeclarations, sequences of error checks require a differently named <code>err</code> variable for
each check (or additional separate variable declarations).
At that time, a better solution might have been to provide more syntactic support for error handling.
Then, the redeclaration rule may not have been needed, and with it gone, so would be various
associated <a href="/issue/377">complications</a>.</p>
</li>
<li>
<p>Going back to actual error handling code, verbosity fades into the background if errors are
actually <em>handled</em>.
Good error handling often requires additional information added to an error.
For instance, a recurring comment in user surveys is about the lack of stack traces associated
with an error.
This could be addressed with support functions that produce and return an augmented
error.
In this (admittedly contrived) example, the relative amount of boilerplate is much smaller:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err := strconv.Atoi(a)
 if err != nil {
 return fmt.Errorf(&quot;invalid integer: %q&quot;, a)
 }
 y, err := strconv.Atoi(b)
 if err != nil {
 return fmt.Errorf(&quot;invalid integer: %q&quot;, b)
 }
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
</li>
<li>
<p>New standard library functionality can help reduce error handling boilerplate as well,
very much in the vein of Rob Pike&rsquo;s 2015 blog post
<a href="/blog/errors-are-values">&ldquo;Errors are values&rdquo;</a>.
For instance, in some cases <a href="/pkg/cmp#Or"><code>cmp.Or</code></a> may be used to deal with a
series of errors all at once:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err1 := strconv.Atoi(a)
 y, err2 := strconv.Atoi(b)
 if err := cmp.Or(err1, err2); err != nil {
 return err
 }
 fmt.Println(&quot;result:&quot;, x+y)
 return nil
}
</code></pre>
</li>
<li>
<p>Writing, reading, and debugging code are all quite different activities.
Writing repeated error checks can be tedious, but today&rsquo;s IDEs provide powerful, even LLM-assisted
code completion.
Writing basic error checks is straightforward for these tools.
The verbosity is most obvious when reading code, but tools might help here as well;
for instance an IDE with a Go language setting could provide a toggle switch to hide error handling
code.
Such switches already exist for other code sections such as function bodies.</p>
</li>
<li>
<p>When debugging error handling code, being able to quickly add a <code>println</code> or
have a dedicated line or source location for setting a breakpoint in a debugger is helpful.
This is easy when there is already a dedicated <code>if</code> statement.
But if all the error handling logic is hidden behind a <code>check</code>, <code>try</code>, or <code>?</code>, the code may have to
be changed into an ordinary <code>if</code> statement first, which complicates debugging
and may even introduce subtle bugs.</p>
</li>
<li>
<p>There are also practical considerations:
Coming up with a new syntax idea for error handling is cheap;
hence the proliferation of a multitude of proposals from the community.
Coming up with a good solution that holds up to scrutiny: not so much.
It takes a concerted effort to properly design a language change and to actually implement it.
The real cost still comes afterwards:
all the code that needs to be changed, the documentation that needs to be updated,
the tools that need to be adjusted.
Taken all into account, language changes are very expensive, the Go team is relatively small,
and there are a lot of other priorities to address.
(These latter points may change: priorities can shift, team sizes can go up or down.)</p>
</li>
<li>
<p>On a final note, some of us recently had the opportunity to attend
<a href="https://cloud.withgoogle.com/next/25" rel="noreferrer" target="_blank">Google Cloud Next 2025</a>,
where the Go team had a booth and where we also hosted a small Go Meetup.
Every single Go user we had a chance to ask was adamant that we should not change the
language for better error handling.
Many mentioned that the lack of specific error handling support in Go is most apparent
when coming freshly from another language that has that support.
As one becomes more fluent and writes more idiomatic Go code, the issue becomes much less important.
This is of course not a sufficiently large set of people to be representative,
but it may be a different set of people than we see on GitHub, and their feedback serves as yet another data point.</p>
</li>
</ul>
<p>Of course, there are also valid arguments in favor of change:</p>
<ul>
<li>
<p>Lack of better error handling support remains the top complaint in our user surveys.
If the Go team really does take user feedback seriously, we ought to do something about this eventually.
(Although there does not seem to be
<a href="https://github.com/golang/go/discussions/71460#discussioncomment-11977299" rel="noreferrer" target="_blank">overwhelming support</a>
for a language change either.)</p>
</li>
<li>
<p>Perhaps the singular focus on reducing the character count is misguided.
A better approach might be to make default error handling highly visible with a keyword
while still removing boilerplate (<code>err != nil</code>).
Such an approach might make it easier for a reader (a code reviewer!) to see that an error
is handled, without &ldquo;looking twice&rdquo;, resulting in improved code quality and safety.
This would bring us back to the beginnings of <code>check</code> and <code>handle</code>.</p>
</li>
<li>
<p>We don&rsquo;t really know how much the issue is the straightforward syntactic verbosity of
error checking, versus the verbosity of good error handling:
constructing errors that are a useful part of an API and meaningful to developers and
end-users alike.
This is something we&rsquo;d like to study in greater depth.</p>
</li>
</ul>
<p>Still, no attempt to address error handling so far has gained sufficient traction.
If we are honestly taking stock of where we are, we can only admit that we
neither have a shared understanding of the problem,
nor do we all agree that there is a problem in the first place.
With this in mind, we are making the following pragmatic decision:</p>
<p><em>For the foreseeable future, the Go team will stop pursuing syntactic language changes
for error handling.
We will also close all open and incoming proposals that concern themselves primarily
with the syntax of error handling, without further investigation.</em></p>
<p>The community has put tremendous effort into exploring, discussing, and debating these issues.
While this may not have resulted in any changes to error handling syntax, these efforts have
resulted in many other improvements to the Go language and our processes.
Maybe, at some point in the future, a clearer picture will emerge on error handling.
Until then, we look forward to focusing this incredible passion on new opportunities
to make Go better for everyone.</p>
<p>Thank you!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/generic-interfaces">Generic interfaces</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/tob-crypto-audit">Go Cryptography Security Audit</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go Cryptography Security Audit</title><id>tag:blog.golang.org,2013:blog.golang.org/tob-crypto-audit</id><link rel="alternate" href="https://go.dev/blog/tob-crypto-audit"></link><published>2025-05-19T00:00:00+00:00</published><updated>2025-05-19T00:00:00+00:00</updated><author><name>Roland Shoemaker and Filippo Valsorda</name></author><summary type="html">Go&#39;s cryptography libraries underwent an audit by Trail of Bits.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/tob-crypto-audit">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go Cryptography Security Audit</h1>
 
 <p class="author">
 Roland Shoemaker and Filippo Valsorda<br>
 19 May 2025
 </p>
 
 <p>Go ships with a full suite of cryptography packages in the standard library to help developers build secure applications. Google recently contracted the independent security firm <a href="https://www.trailofbits.com/" rel="noreferrer" target="_blank">Trail of Bits</a> to complete an audit of the core set of packages that are also validated as part of the <a href="/doc/go1.24#fips140">new native FIPS 140-3 module</a>. The audit produced a single low-severity finding, in the legacy and unsupported <a href="/doc/security/fips140#goboringcrypto">Go+BoringCrypto integration</a>, and a handful of informational findings. The full text of the audit report can be found <a href="https://github.com/trailofbits/publications/blob/d47e8fafa7e3323e5620d228f2f3f3bf58ed5978/reviews/2025-03-google-gocryptographiclibraries-securityreview.pdf" rel="noreferrer" target="_blank">here</a>.</p>
<p>The scope of the audit included our implementations of key exchange (ECDH and post-quantum ML-KEM), digital signature (ECDSA, RSA, and Ed25519), encryption (AES-GCM, AES-CBC, and AES-CTR), hashing (SHA-1, SHA-2, and SHA-3), key derivation (HKDF and PBKDF2), and authentication (HMAC), as well as the cryptographic random number generator. Low-level big integer and elliptic curve implementations, with their delicate assembly cores, were included. Higher level protocols like TLS and X.509 were not in scope. Three Trail of Bits engineers worked on the audit for a month.</p>
<p>We are proud of the security track record of the Go cryptography packages, and of the outcome of this audit, which is just one of many ways we gain assurance of the packages’ correctness. First, we aggressively limit their complexity, guided by the <a href="/design/cryptography-principles">Cryptography Principles</a> which for example prioritize security over performance. Further, we <a href="https://www.youtube.com/watch?v=lkEH3V3PkS0" rel="noreferrer" target="_blank">thoroughly test them</a> with an array of different techniques. We make a point of leveraging safe APIs even for internal packages, and naturally we can rely on the Go language properties to avoid memory management issues. Finally, we focus on readability to make maintenance easier and code review and audits more effective.</p>
<h2 id="one-low-severity-finding">One low-severity finding</h2>
<p>The only potentially exploitable issue, TOB-GOCL-3, has <em>low severity</em>, meaning it had minor impact and was difficult to trigger. This issue has been fixed in Go 1.24.</p>
<p>Crucially, TOB-GOCL-3 (<a href="#cgo-memory-management">discussed further below</a>) concerns memory management in the <a href="/doc/security/fips140#goboringcrypto">legacy Go+BoringCrypto GOEXPERIMENT</a>, which is not enabled by default and unsupported for use outside of Google.</p>
<h2 id="five-informational-findings">Five informational findings</h2>
<p>The remaining findings are <em>informational</em>, meaning they do not pose an immediate risk but are relevant to security best practices. We addressed these in the current Go 1.25 development tree.</p>
<p>Findings TOB-GOCL-1, TOB-GOCL-2, and TOB-GOCL-6 concern possible timing side-channels in various cryptographic operations. Of these three findings, only TOB-GOCL-2 affects operations that were expected to be constant time due to operating on secret values, but it only affects Power ISA targets (GOARCH ppc64 and ppc64le). TOB-GOCL-4 highlights misuse risk in an internal API, should it be repurposed beyond its current use case. TOB-GOCL-5 points out a missing check for a limit that is impractical to reach.</p>
<h2 id="timing-side-channels">Timing Side-Channels</h2>
<p>Findings TOB-GOCL-1, TOB-GOCL-2, and TOB-GOCL-6 concern minor timing side-channels. TOB-GOCL-1 and TOB-GOCL-6 are related to functions which we do not use for sensitive values, but could be used for such values in the future, and TOB-GOCL-2 is related to the assembly implementation of P-256 ECDSA on Power ISA.</p>
<h3 id="cryptoecdhcryptoecdsa-conversion-from-bytes-to-field-elements-is-not-constant-time-tob-gocl-1"><code>crypto/ecdh,crypto/ecdsa</code>: conversion from bytes to field elements is not constant time (TOB-GOCL-1)</h3>
<p>The internal implementation of NIST elliptic curves provided a method to convert field elements between an internal and external representation which operated in variable time.</p>
<p>All usages of this method operated on public inputs which are not considered secret (public ECDH values, and ECDSA public keys), so we determined that this was not a security issue. That said, we decided to <a href="/cl/650579">make the method constant time anyway</a>, in order to prevent accidentally using this method in the future with secret values, and so that we don&rsquo;t have to think about whether it is an issue or not.</p>
<h3 id="cryptoecdsa-p-256-conditional-negation-is-not-constant-time-in-power-isa-assembly-tob-gocl-2-cve-2025-22866"><code>crypto/ecdsa</code>: P-256 conditional negation is not constant time in Power ISA assembly (TOB-GOCL-2, CVE-2025-22866)</h3>
<p>Beyond the <a href="/wiki/PortingPolicy#first-class-ports">first class Go platforms</a>, Go also supports a number of additional platforms, including some less common architectures. During the review of our assembly implementations of various underlying cryptographic primitives, the Trail of Bits team found one issue that affected the ECDSA implementation on the ppc64 and ppc64le architectures.</p>
<p>Due to the usage of a conditional branching instruction in the implementation of the conditional negation of P-256 points, the function operated in variable-time, rather than constant-time, as expected. The fix for this was relatively simple, <a href="/cl/643735">replacing the conditional branching instruction</a> with a pattern we already use elsewhere to conditionally select the correct result in constant time. We assigned this issue CVE-2025-22866.</p>
<p>To prioritize the code that reaches most of our users, and due to the specialized knowledge required to target specific ISAs, we generally rely on community contributions to maintain assembly for non-first class platforms. We thank our partners at IBM for helping provide review for our fix.</p>
<h3 id="cryptoed25519-scalarsetcanonicalbytes-is-not-constant-time-tob-gocl-6"><code>crypto/ed25519</code>: Scalar.SetCanonicalBytes is not constant time (TOB-GOCL-6)</h3>
<p>The internal edwards25519 package provided a method to convert between an internal and external representation of scalars which operated in variable time.</p>
<p>This method was only used on signature inputs to ed25519.Verify, which are not considered secret, so we determined that this was not a security issue. That said, similarly to the TOB-GOCL-1 finding, we decided to <a href="/cl/648035">make the method constant time anyway</a>, in order to prevent accidentally using this method in the future with secret values, and because we are aware that people fork this code outside of the standard library, and may be using it with secret values.</p>
<h2 id="cgo-memory-management">Cgo Memory Management</h2>
<p>Finding TOB-GOCL-3 concerns a memory management issue in the Go+BoringCrypto integration.</p>
<h3 id="cryptoecdh-custom-finalizer-may-free-memory-at-the-start-of-a-c-function-call-using-this-memory-tob-gocl-3"><code>crypto/ecdh</code>: custom finalizer may free memory at the start of a C function call using this memory (TOB-GOCL-3)</h3>
<p>During the review, there were a number of questions about our cgo-based Go+BoringCrypto integration, which provides a FIPS 140-2 compliant cryptography mode for internal usage at Google. The Go+BoringCrypto code is not supported by the Go team for external use, but has been critical for Google’s internal usage of Go.</p>
<p>The Trail of Bits team found one vulnerability and one <a href="/cl/644120">non-security relevant bug</a>, both of which were results of the manual memory management required to interact with a C library. Since the Go team does not support usage of this code outside of Google, we have chosen not to issue a CVE or Go vulnerability database entry for this issue, but we <a href="/cl/644119">fixed it in Go 1.24</a>.</p>
<p>This kind of pitfall is one of the many reasons that we decided to move away from the Go+BoringCrypto integration. We have been working on a <a href="/doc/security/fips140">native FIPS 140-3 mode</a> that uses the regular pure Go cryptography packages, allowing us to avoid the complex cgo semantics in favor of the traditional Go memory model.</p>
<h2 id="implementation-completeness">Implementation Completeness</h2>
<p>Findings TOB-GOCL-4 and TOB-GOCL-5 concern limited implementations of two specifications, <a href="https://csrc.nist.gov/pubs/sp/800/90/a/r1/final" rel="noreferrer" target="_blank">NIST SP 800-90A</a> and <a href="https://datatracker.ietf.org/doc/html/rfc8018" rel="noreferrer" target="_blank">RFC 8018</a>.</p>
<h3 id="cryptointernalfips140drbg-ctr-drbg-api-presents-multiple-misuse-risks-tob-gocl-4"><code>crypto/internal/fips140/drbg</code>: CTR_DRBG API presents multiple misuse risks (TOB-GOCL-4)</h3>
<p>As part of the <a href="/doc/security/fips140">native FIPS 140-3 mode</a> that we are introducing, we needed an implementation of the NIST CTR_DRBG (an AES-CTR based deterministic random bit generator) to provide compliant randomness.</p>
<p>Since we only need a small subset of the functionality of the NIST SP 800-90A Rev. 1 CTR_DRBG for our purposes, we did not implement the full specification, in particular omitting the derivation function and personalization strings. These features can be critical to safely use the DRBG in generic contexts.</p>
<p>As our implementation is tightly scoped to the specific use case we need, and since the implementation is not publicly exported, we determined that this was acceptable and worth the decreased complexity of the implementation. We do not expect this implementation to ever be used for other purposes internally, and have <a href="/cl/647815">added a warning to the documentation</a> that details these limitations.</p>
<h3 id="cryptopbkdf2-pbkdf2-does-not-enforce-output-length-limitations-tob-gocl-5"><code>crypto/pbkdf2</code>: PBKDF2 does not enforce output length limitations (TOB-GOCL-5)</h3>
<p>In Go 1.24, we began the process of moving packages from <a href="https://golang.org/x/crypto" rel="noreferrer" target="_blank">golang.org/x/crypto</a> into the standard library, ending a confusing pattern where first-party, high-quality, and stable Go cryptography packages were kept outside of the standard library for no particular reason.</p>
<p>As part of this process we moved the <a href="https://golang.org/x/crypto/pbkdf2" rel="noreferrer" target="_blank">golang.org/x/crypto/pbkdf2</a> package into the standard library, as crypto/pbkdf2. While reviewing this package, the Trail of Bits team noticed that we did not enforce the limit on the size of derived keys defined in <a href="https://datatracker.ietf.org/doc/html/rfc8018" rel="noreferrer" target="_blank">RFC 8018</a>.</p>
<p>The limit is <code>(2^32 - 1) * &lt;hash length&gt;</code>, after which the key would loop. When using SHA-256, exceeding the limit would take a key of more than 137GB. We do not expect anyone has ever used PBKDF2 to generate a key this large, especially because PBKDF2 runs the iterations at every block, but for the sake of correctness, we <a href="/cl/644122">now enforce the limit as defined by the standard</a>.</p>
<h1 id="whats-next">What’s Next</h1>
<p>The results of this audit validate the effort the Go team has put into developing high-quality, easy to use cryptography libraries and should provide confidence to our users who rely on them to build safe and secure software.</p>
<p>We’re not resting on our laurels, though: the Go contributors are continuing to develop and improve the cryptography libraries we provide users.</p>
<p>Go 1.24 now includes a FIPS 140-3 mode written in pure Go, which is currently undergoing CMVP testing. This will provide a supported FIPS 140-3 compliant mode for all users of Go, replacing the currently unsupported Go+BoringCrypto integration.</p>
<p>We are also working to implement modern post-quantum cryptography, introducing a ML-KEM-768 and ML-KEM-1024 implementation in Go 1.24 in the <a href="/pkg/crypto/mlkem">crypto/mlkem package</a>, and adding support to the crypto/tls package for the hybrid X25519MLKEM768 key exchange.</p>
<p>Finally, we are planning on introducing new easier to use high-level cryptography APIs, designed to reduce the barrier for picking and using high-quality algorithms for basic use cases. We plan to begin with offering a simple password hashing API that removes the need for users to decide which of the myriad of possible algorithms they should be relying on, with mechanisms to automatically migrate to newer algorithms as the state-of-the-art changes.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/error-syntax">[ On | No ] syntactic support for error handling</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/testing-b-loop">More predictable benchmarking with testing.B.Loop</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>More predictable benchmarking with testing.B.Loop</title><id>tag:blog.golang.org,2013:blog.golang.org/testing-b-loop</id><link rel="alternate" href="https://go.dev/blog/testing-b-loop"></link><published>2025-04-02T00:00:00+00:00</published><updated>2025-04-02T00:00:00+00:00</updated><author><name>Junyang Shao</name></author><summary type="html">Better benchmark looping in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/testing-b-loop">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>More predictable benchmarking with testing.B.Loop</h1>
 
 <p class="author">
 Junyang Shao<br>
 2 April 2025
 </p>
 
 <p>Go developers who have written benchmarks using the
<a href="https://pkg.go.dev/testing" rel="noreferrer" target="_blank"><code>testing</code></a> package might have encountered some of
its various pitfalls. Go 1.24 introduces a new way to write benchmarks that&rsquo;s just
as easy to use, but at the same time far more robust:
<a href="https://pkg.go.dev/testing#B.Loop" rel="noreferrer" target="_blank"><code>testing.B.Loop</code></a>.</p>
<p>Traditionally, Go benchmarks are written using a loop from 0 to <code>b.N</code>:</p>
<pre><code>func Benchmark(b *testing.B) {
 for range b.N {
 ... code to measure ...
 }
}
</code></pre>
<p>Using <code>b.Loop</code> instead is a trivial change:</p>
<pre><code>func Benchmark(b *testing.B) {
 for b.Loop() {
 ... code to measure ...
 }
}
</code></pre>
<p><code>testing.B.Loop</code> has many benefits:</p>
<ul>
<li>It prevents unwanted compiler optimizations within the benchmark loop.</li>
<li>It automatically excludes setup and cleanup code from benchmark timing.</li>
<li>Code can&rsquo;t accidentally depend on the total number of iterations or the current
iteration.</li>
</ul>
<p>These were all easy mistakes to make with <code>b.N</code>-style benchmarks that would
silently result in bogus benchmark results. As an added bonus, <code>b.Loop</code>-style
benchmarks even complete in less time!</p>
<p>Let&rsquo;s explore the advantages of <code>testing.B.Loop</code> and how to effectively utilize it.</p>
<h2 id="old-benchmark-loop-problems">Old benchmark loop problems</h2>
<p>Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated
benchmarks required more care:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 b.ResetTimer() // if setup may be expensive
 for range b.N {
 ... code to measure ...
 ... use sinks or accumulation to prevent dead-code elimination ...
 }
 b.StopTimer() // if cleanup or reporting may be expensive
 ... cleanup ...
 ... report ...
}
</code></pre>
<p>If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop
with <code>ResetTimer</code> and/or <code>StopTimer</code> calls. These are easy to forget, and even if the
developer remembers they may be necessary, it can be difficult to judge whether setup or
cleanup are &ldquo;expensive enough&rdquo; to require them.</p>
<p>Without these, the <code>testing</code> package can only time the entire benchmark function. If a
benchmark function omits them, the setup and cleanup code will be included in the overall
time measurement, silently skewing the final benchmark result.</p>
<p>There is another, more subtle pitfall that requires deeper understanding:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func isCond(b byte) bool {
 if b%3 == 1 &amp;&amp; b%7 == 2 &amp;&amp; b%17 == 11 &amp;&amp; b%31 == 9 {
 return true
 }
 return false
}

func BenchmarkIsCondWrong(b *testing.B) {
 for range b.N {
 isCond(201)
 }
}
</code></pre>
<p>In this example, the user might observe <code>isCond</code> executing in sub-nanosecond
time. CPUs are fast, but not that fast! This seemingly anomalous result stems
from the fact that <code>isCond</code> is inlined, and since its result is never used, the
compiler eliminates it as dead code. As a result, this benchmark doesn&rsquo;t measure <code>isCond</code>
at all; it measures how long it takes to do nothing. In this case, the sub-nanosecond
result is a clear red flag, but in more complex benchmarks, partial dead-code elimination
can lead to results that look reasonable but still aren&rsquo;t measuring what was intended.</p>
<h2 id="how-testingbloop-helps">How <code>testing.B.Loop</code> helps</h2>
<p>Unlike a <code>b.N</code>-style benchmark, <code>testing.B.Loop</code> is able to track when it is first called
in a benchmark when the final iteration ends. The <code>b.ResetTimer</code> at the loop&rsquo;s start
and <code>b.StopTimer</code> at its end are integrated into <code>testing.B.Loop</code>, eliminating the need
to manually manage the benchmark timer for setup and cleanup code.</p>
<p>Furthermore, the Go compiler now detects loops where the condition is just a call to
<code>testing.B.Loop</code> and prevents dead code elimination within the loop. In Go 1.24, this is
implemented by disallowing inlining into the body of such a loop, but we plan to
<a href="/issue/73137">improve</a> this in the future.</p>
<p>Another nice feature of <code>testing.B.Loop</code> is its one-shot ramp-up approach. With a <code>b.N</code>-style
benchmark, the testing package must call the benchmark function several times with different
values of <code>b.N</code>, ramping up until the measured time reached a threshold. In contrast, <code>b.Loop</code>
can simply run the benchmark loop until it reaches the time threshold, and only needs to call
the benchmark function once. Internally, <code>b.Loop</code> still uses a ramp-up process to amortize
measurement overhead, but this is hidden from the caller and can be more efficient.</p>
<p>Certain constraints of the <code>b.N</code>-style loop still apply to the <code>b.Loop</code>-style
loop. It remains the user&rsquo;s responsibility to manage the timer within the benchmark loop,
when necessary:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func BenchmarkSortInts(b *testing.B) {
 ints := make([]int, N)
 for b.Loop() {
 b.StopTimer()
 fillRandomInts(ints)
 b.StartTimer()
 slices.Sort(ints)
 }
}
</code></pre>
<p>In this example, to benchmark the in-place sorting performance of <code>slices.Sort</code>, a
randomly initialized array is required for each iteration. The user must still
manually manage the timer in such cases.</p>
<p>Furthermore, there still needs to be exactly one such loop in the benchmark function body
(a <code>b.N</code>-style loop cannot coexist with a <code>b.Loop</code>-style loop), and every iteration of the
loop should do the same thing.</p>
<h2 id="when-to-use">When to use</h2>
<p>The <code>testing.B.Loop</code> method is now the preferred way to write benchmarks:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 for b.Loop() {
 // optional timer control for in-loop setup/cleanup
 ... code to measure ...
 }
 ... cleanup ...
}
</code></pre>
<p><code>testing.B.Loop</code> offers faster, more accurate, and
more intuitive benchmarking.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>A huge thank you to everyone in the community who provided feedback on the proposal
issue and reported bugs as this feature was released! I&rsquo;m also grateful to Eli
Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements,
Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and
documentation improvements. Thank you all for your contributions!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/tob-crypto-audit">Go Cryptography Security Audit</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/coretypes">Goodbye core types - Hello Go as we know and love it!</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Goodbye core types - Hello Go as we know and love it!</title><id>tag:blog.golang.org,2013:blog.golang.org/coretypes</id><link rel="alternate" href="https://go.dev/blog/coretypes"></link><published>2025-03-26T00:00:00+00:00</published><updated>2025-03-26T00:00:00+00:00</updated><author><name>Robert Griesemer</name></author><summary type="html">Go 1.25 simplifies the language spec by removing the notion of core types</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/coretypes">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Goodbye core types - Hello Go as we know and love it!</h1>
 
 <p class="author">
 Robert Griesemer<br>
 26 March 2025
 </p>
 
 <p>The Go 1.18 release introduced generics and with that a number of new features, including type parameters, type constraints, and new concepts such as type sets.
It also introduced the notion of a <em>core type</em>.
While the former provide concrete new functionality, a core type is an abstract construct that was introduced
for expediency and to simplify dealing with generic operands (operands whose types are type parameters).
In the Go compiler, code that in the past relied on the <a href="/ref/spec/#Underlying_types">underlying type</a> of an operand,
now instead had to call a function computing the operand&rsquo;s core type.
In the language spec, in many places we just needed to replace &ldquo;underlying type&rdquo; with &ldquo;core type&rdquo;.
What&rsquo;s not to like?</p>
<p>Quite a few things, as it turns out!
To understand how we got here, it&rsquo;s useful to briefly revisit how type parameters and type constraints work.</p>
<h2 id="type-parameters-and-type-constraints">Type parameters and type constraints</h2>
<p>A type parameter is a placeholder for a future type argument;
it acts like a <em>type variable</em> whose value is known at compile time,
similar to how a named constant stands for a number, string, or bool whose value is known at compile time.
Like ordinary variables, type parameters have a type.
That type is described by their <em>type constraint</em> which determines
what operations are permitted on operands whose type is the respective type parameter.</p>
<p>Any concrete type that instantiates a type parameter must satisfy the type parameter&rsquo;s constraint.
This ensures that an operand whose type is a type parameter possesses all of the respective type constraint&rsquo;s properties,
no matter what concrete type is used to instantiate the type parameter.</p>
<p>In Go, type constraints are described through a mixture of method and type requirements which together
define a <em>type set</em>: this is the set of all the types that satisfy all the requirements. Go uses a
generalized form of interfaces for this purpose. An interface enumerates a set of methods and types,
and the type set described by such an interface consists of all the types that implement those methods
and that are included in the enumerated types.</p>
<p>For instance, the type set described by the interface</p>
<pre><code class="language-Go">type Constraint interface {
 ~[]byte | ~string
 Hash() uint64
}
</code></pre>
<p>consists of all the types whose representation is <code>[]byte</code> or <code>string</code> and whose method set includes the <code>Hash</code> method.</p>
<p>With this we can now write down the rules that govern operations on generic operands.
For instance, the <a href="/ref/spec#Index_expressions">rules for index expressions</a> state that (among other things)
for an operand <code>a</code> of type parameter type <code>P</code>:</p>
<blockquote>
<p>The index expression <code>a[x]</code> must be valid for values of all types in <code>P</code>&rsquo;s type set.
The element types of all types in <code>P</code>&rsquo;s type set must be identical.
(In this context, the element type of a string type is <code>byte</code>.)</p>
</blockquote>
<p>These rules make it possible to index the generic variable <code>s</code> below (<a href="/play/p/M1LYKm3x3IB">playground</a>):</p>
<pre><code class="language-Go">func at[bytestring Constraint](s bytestring, i int) byte {
 return s[i]
}
</code></pre>
<p>The indexing operation <code>s[i]</code> is permitted because the type of <code>s</code> is <code>bytestring</code>, and the type constraint (type set) of
<code>bytestring</code> contains <code>[]byte</code> and <code>string</code> types for which indexing with <code>i</code> is valid.</p>
<h2 id="core-types">Core types</h2>
<p>This type set-based approach is very flexible and in line with the intentions of the
<a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md" rel="noreferrer" target="_blank">original generics proposal</a>:
an operation involving operands of generic type should be valid if it is valid for any type permitted by the respective
type constraint.
To simplify matters with respect to the implementation, knowing that we would be able to relax rules later,
this approach was <em>not</em> chosen universally.
Instead, for instance, for <a href="/ref/spec#Send_statements">Send statements</a>, the spec states that</p>
<blockquote>
<p>The channel expression&rsquo;s <em>core type</em> must be a channel, the channel direction must permit send operations,
and the type of the value to be sent must be assignable to the channel&rsquo;s element type.</p>
</blockquote>
<p>These rules are based on the notion of a core type which is defined roughly as follows:</p>
<ul>
<li>If a type is not a type parameter, its core type is just its <a href="/ref/spec#Underlying_types">underlying type</a>.</li>
<li>If the type is a type parameter, the core type is the single underlying type of all the types in the type parameter&rsquo;s type set.
If the type set has <em>different</em> underlying types, the core type doesn&rsquo;t exist.</li>
</ul>
<p>For instance, <code>interface{ ~[]int }</code> has a core type (<code>[]int</code>), but the <code>Constraint</code> interface above does not have a core type.
To make things more complicated, when it comes to channel operations and certain built-in calls (<code>append</code>, <code>copy</code>) the above definition
of core types is too restrictive.
The actual rules have adjustments that allow for differing channel directions and type sets containing both <code>[]byte</code> and <code>string</code> types.</p>
<p>There are various problems with this approach:</p>
<ul>
<li>
<p>Because the definition of core type must lead to sound type rules for different language features,
it is overly restrictive for specific operations.
For instance, the Go 1.24 rules for <a href="/ref/spec#Slice_expressions">slice expressions</a> do rely on core types,
and as a consequence slicing an operand of type <code>S</code> constrained by <code>Constraint</code> is not permitted, even though
it could be valid.</p>
</li>
<li>
<p>When trying to understand a specific language feature, one may have to learn the intricacies of
core types even when considering non-generic code.
Again, for slice expressions, the language spec talks about the core type of the sliced operand,
rather than just stating that the operand must be an array, slice, or string.
The latter is more direct, simpler, and clearer, and doesn&rsquo;t require knowing another concept that may be
irrelevant in the concrete case.</p>
</li>
<li>
<p>Because the notion of core types exists, the rules for index expressions, and <code>len</code> and <code>cap</code> (and others),
which all eschew core types, appear as exceptions in the language rather than the norm.
In turn, core types cause proposals such as <a href="/issue/48522">issue #48522</a> which would permit a selector
<code>x.f</code> to access a field <code>f</code> shared by all elements of <code>x</code>&rsquo;s type set, to appear to add more exceptions to the
language.
Without core types, that feature becomes a natural and useful consequence of the ordinary rules for non-generic
field access.</p>
</li>
</ul>
<h2 id="go-125">Go 1.25</h2>
<p>For the upcoming Go 1.25 release (August 2025) we decided to remove the notion of core types from the
language spec in favor of explicit (and equivalent!) prose where needed.
This has multiple benefits:</p>
<ul>
<li>The Go spec presents fewer concepts, making it easier to learn the language.</li>
<li>The behavior of non-generic code can be understood without reference to generics concepts.</li>
<li>The individualized approach (specific rules for specific operations) opens the door for more flexible rules.
We already mentioned <a href="/issue/48522">issue #48522</a>, but there are also ideas for more powerful
slice operations, and <a href="/issue/69153">improved type inference</a>.</li>
</ul>
<p>The respective <a href="/issue/70128">proposal issue #70128</a> was recently approved and the relevant changes
are already implemented.
Concretely this means that a lot of prose in the language spec was reverted to its original,
pre-generics form, and new paragraphs were added where needed to explain the rules as they
pertain to generic operands. Importantly, no behavior was changed.
The entire section on core types was removed.
The compiler&rsquo;s error messages were updated to not mention &ldquo;core type&rdquo; anymore, and in many
cases error messages are now more specific by pointing out exactly which type in a type set
is causing a problem.</p>
<p>Here is a sample of the changes made. For the built-in function <code>close</code>,
starting with Go 1.18 the spec began as follows:</p>
<blockquote>
<p>For an argument <code>ch</code> with core type that is a channel,
the built-in function <code>close</code> records that no more values will be sent on the channel.</p>
</blockquote>
<p>A reader who simply wanted to know how <code>close</code> works, had to first learn about core types.
Starting with Go 1.25, this section will again begin the same way it began before Go 1.18:</p>
<blockquote>
<p>For a channel <code>ch</code>, the built-in function <code>close(ch)</code>
records that no more values will be sent on the channel.</p>
</blockquote>
<p>This is shorter and easier to understand.
Only when the reader is dealing with a generic operand will they have to contemplate
the newly added paragraph:</p>
<blockquote>
<p>If the type of the argument to <code>close</code> is a type parameter
all types in its type set must be channels with the same element type.
It is an error if any of those channels is a receive-only channel.</p>
</blockquote>
<p>We made similar changes to each place that mentioned core types.
In summary, although this spec update does not affect any current Go program, it opens the
door to future language improvements while making the language as it is today easier to
learn and its spec simpler.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/testing-b-loop">More predictable benchmarking with testing.B.Loop</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/osroot">Traversal-resistant file APIs</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Traversal-resistant file APIs</title><id>tag:blog.golang.org,2013:blog.golang.org/osroot</id><link rel="alternate" href="https://go.dev/blog/osroot"></link><published>2025-03-12T00:00:00+00:00</published><updated>2025-03-12T00:00:00+00:00</updated><author><name>Damien Neil</name></author><summary type="html">New file access APIs in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/osroot">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Traversal-resistant file APIs</h1>
 
 <p class="author">
 Damien Neil<br>
 12 March 2025
 </p>
 
 <p>A <em>path traversal vulnerability</em> arises when an attacker can trick a program
into opening a file other than the one it intended.
This post explains this class of vulnerability,
some existing defenses against it, and describes how the new
<a href="/pkg/os#Root"><code>os.Root</code></a> API added in Go 1.24 provides
a simple and robust defense against unintentional path traversal.</p>
<h2 id="path-traversal-attacks">Path traversal attacks</h2>
<p>&ldquo;Path traversal&rdquo; covers a number of related attacks following a common pattern:
A program attempts to open a file in some known location, but an attacker causes
it to open a file in a different location.</p>
<p>If the attacker controls part of the filename, they may be able to use relative
directory components (&quot;..&quot;) to escape the intended location:</p>
<pre><code>f, err := os.Open(filepath.Join(trustedLocation, &quot;../../../../etc/passwd&quot;))
</code></pre>
<p>On Windows systems, some names have special meaning:</p>
<pre><code>// f will print to the console.
f, err := os.Create(filepath.Join(trustedLocation, &quot;CONOUT$&quot;))
</code></pre>
<p>If the attacker controls part of the local filesystem, they may be able to use
symbolic links to cause a program to access the wrong file:</p>
<pre><code>// Attacker links /home/user/.config to /home/otheruser/.config:
err := os.WriteFile(&quot;/home/user/.config/foo&quot;, config, 0o666)
</code></pre>
<p>If the program defends against symlink traversal by first verifying that the intended file
does not contain any symlinks, it may still be vulnerable to
<a href="https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use" rel="noreferrer" target="_blank">time-of-check/time-of-use (TOCTOU) races</a>,
where the attacker creates a symlink after the program&rsquo;s check:</p>
<pre><code>// Validate the path before use.
cleaned, err := filepath.EvalSymlinks(unsafePath)
if err != nil {
 return err
}
if !filepath.IsLocal(cleaned) {
 return errors.New(&quot;unsafe path&quot;)
}

// Attacker replaces part of the path with a symlink.
// The Open call follows the symlink:
f, err := os.Open(cleaned)
</code></pre>
<p>Another variety of TOCTOU race involves moving a directory that forms part of a path
mid-traversal. For example, the attacker provides a path such as &ldquo;a/b/c/../../etc/passwd&rdquo;,
and renames &ldquo;a/b/c&rdquo; to &ldquo;a/b&rdquo; while the open operation is in progress.</p>
<h2 id="path-sanitization">Path sanitization</h2>
<p>Before we tackle path traversal attacks in general, let&rsquo;s start with path sanitization.
When a program&rsquo;s threat model does not include attackers with access to the local file system,
it can be sufficient to validate untrusted input paths before use.</p>
<p>Unfortunately, sanitizing paths can be surprisingly tricky,
especially for portable programs that must handle both Unix and Windows paths.
For example, on Windows <code>filepath.IsAbs(`\foo`)</code> reports <code>false</code>,
because the path &ldquo;\foo&rdquo; is relative to the current drive.</p>
<p>In Go 1.20, we added the <a href="/pkg/path/filepath#IsLocal"><code>path/filepath.IsLocal</code></a>
function, which reports whether a path is &ldquo;local&rdquo;. A &ldquo;local&rdquo; path is one which:</p>
<ul>
<li>does not escape the directory in which it is evaluated (&quot;../etc/passwd&quot; is not allowed);</li>
<li>is not an absolute path (&quot;/etc/passwd&quot; is not allowed);</li>
<li>is not empty (&quot;&quot; is not allowed);</li>
<li>on Windows, is not a reserved name (&ldquo;COM1&rdquo; is not allowed).</li>
</ul>
<p>In Go 1.23, we added the <a href="/pkg/path/filepath#Localize"><code>path/filepath.Localize</code></a>
function, which converts a /-separated path into a local operating system path.</p>
<p>Programs that accept and operate on potentially attacker-controlled paths should almost
always use <code>filepath.IsLocal</code> or <code>filepath.Localize</code> to validate or sanitize those paths.</p>
<h2 id="beyond-sanitization">Beyond sanitization</h2>
<p>Path sanitization is not sufficient when attackers may have access to part of
the local filesystem.</p>
<p>Multi-user systems are uncommon these days, but attacker access to the filesystem
can still occur in a variety of ways.
An unarchiving utility that extracts a tar or zip file may be induced
to extract a symbolic link and then extract a file name that traverses that link.
A container runtime may give untrusted code access to a portion of the local filesystem.</p>
<p>Programs may defend against unintended symlink traversal by using the
<a href="/pkg/path/filepath#EvalSymlinks"><code>path/filepath.EvalSymlinks</code></a>
function to resolve links in untrusted names before validation, but as described
above this two-step process is vulnerable to TOCTOU races.</p>
<p>Before Go 1.24, the safer option was to use a package such as
<a href="/pkg/github.com/google/safeopen">github.com/google/safeopen</a>,
that provides path traversal-resistant functions for opening a potentially-untrusted
filename within a specific directory.</p>
<h2 id="introducing-osroot">Introducing <code>os.Root</code></h2>
<p>In Go 1.24, we are introducing new APIs in the <code>os</code> package to safely open
a file in a location in a traversal-resistant fashion.</p>
<p>The new <a href="/pkg/os#Root"><code>os.Root</code></a> type represents a directory somewhere
in the local filesystem. Open a root with the <a href="/pkg/os#OpenRoot"><code>os.OpenRoot</code></a>
function:</p>
<pre><code>root, err := os.OpenRoot(&quot;/some/root/directory&quot;)
if err != nil {
 return err
}
defer root.Close()
</code></pre>
<p><code>Root</code> provides methods to operate on files within the root.
These methods all accept filenames relative to the root,
and disallow any operations that would escape from the root either
using relative path components (&quot;..&quot;) or symlinks.</p>
<pre><code>f, err := root.Open(&quot;path/to/file&quot;)
</code></pre>
<p><code>Root</code> permits relative path components and symlinks that do not escape the root.
For example, <code>root.Open(&quot;a/../b&quot;)</code> is permitted. Filenames are resolved using the
semantics of the local platform: On Unix systems, this will follow
any symlink in &ldquo;a&rdquo; (so long as that link does not escape the root);
while on Windows systems this will open &ldquo;b&rdquo; (even if &ldquo;a&rdquo; does not exist).</p>
<p><code>Root</code> currently provides the following set of operations:</p>
<pre><code>func (*Root) Create(string) (*File, error)
func (*Root) Lstat(string) (fs.FileInfo, error)
func (*Root) Mkdir(string, fs.FileMode) error
func (*Root) Open(string) (*File, error)
func (*Root) OpenFile(string, int, fs.FileMode) (*File, error)
func (*Root) OpenRoot(string) (*Root, error)
func (*Root) Remove(string) error
func (*Root) Stat(string) (fs.FileInfo, error)
</code></pre>
<p>In addition to the <code>Root</code> type, the new
<a href="/pkg/os#OpenInRoot"><code>os.OpenInRoot</code></a> function
provides a simple way to open a potentially-untrusted filename within a
specific directory:</p>
<pre><code>f, err := os.OpenInRoot(&quot;/some/root/directory&quot;, untrustedFilename)
</code></pre>
<p>The <code>Root</code> type provides a simple, safe, portable API for operating with untrusted filenames.</p>
<h2 id="caveats-and-considerations">Caveats and considerations</h2>
<h3 id="unix">Unix</h3>
<p>On Unix systems, <code>Root</code> is implemented using the <code>openat</code> family of system calls.
A <code>Root</code> contains a file descriptor referencing its root directory and will track that
directory across renames or deletion.</p>
<p><code>Root</code> defends against symlink traversal but does not limit traversal
of mount points. For example, <code>Root</code> does not prevent traversal of
Linux bind mounts. Our threat model is that <code>Root</code> defends against
filesystem constructs that may be created by ordinary users (such
as symlinks), but does not handle ones that require root privileges
to create (such as bind mounts).</p>
<h3 id="windows">Windows</h3>
<p>On Windows, <code>Root</code> opens a handle referencing its root directory.
The open handle prevents that directory from being renamed or deleted until the <code>Root</code> is closed.</p>
<p><code>Root</code> prevents access to reserved Windows device names such as <code>NUL</code> and <code>COM1</code>.</p>
<h3 id="wasi">WASI</h3>
<p>On WASI, the <code>os</code> package uses the WASI preview 1 filesystem API,
which are intended to provide traversal-resistant filesystem access.
Not all WASI implementations fully support filesystem sandboxing,
however, and <code>Root</code>&rsquo;s defense against traversal is limited to that provided
by the WASI implementation.</p>
<h3 id="goosjs">GOOS=js</h3>
<p>When GOOS=js, the <code>os</code> package uses the Node.js file system API.
This API does not include the openat family of functions,
and so <code>os.Root</code> is vulnerable to TOCTOU (time-of-check-time-of-use) races in symlink
validation on this platform.</p>
<p>When GOOS=js, a <code>Root</code> references a directory name rather than a file descriptor,
and does not track directories across renames.</p>
<h3 id="plan-9">Plan 9</h3>
<p>Plan 9 does not have symlinks.
On Plan 9, a <code>Root</code> references a directory name and performs lexical sanitization of
filenames.</p>
<h3 id="performance">Performance</h3>
<p><code>Root</code> operations on filenames containing many directory components can be much more expensive
than the equivalent non-<code>Root</code> operation. Resolving &ldquo;..&rdquo; components can also be expensive.
Programs that want to limit the cost of filesystem operations can use <code>filepath.Clean</code> to
remove &ldquo;..&rdquo; components from input filenames, and may want to limit the number of
directory components.</p>
<h2 id="who-should-use-osroot">Who should use os.Root?</h2>
<p>You should use <code>os.Root</code> or <code>os.OpenInRoot</code> if:</p>
<ul>
<li>you are opening a file in a directory; AND</li>
<li>the operation should not access a file outside that directory.</li>
</ul>
<p>For example, an archive extractor writing files to an output directory should use
<code>os.Root</code>, because the filenames are potentially untrusted and it would be incorrect
to write a file outside the output directory.</p>
<p>However, a command-line program that writes output to a user-specified location
should not use <code>os.Root</code>, because the filename is not untrusted and may
refer to anywhere on the filesystem.</p>
<p>As a good rule of thumb, code which calls <code>filepath.Join</code> to combine a fixed directory
and an externally-provided filename should probably use <code>os.Root</code> instead.</p>
<pre><code>// This might open a file not located in baseDirectory.
f, err := os.Open(filepath.Join(baseDirectory, filename))

// This will only open files under baseDirectory.
f, err := os.OpenInRoot(baseDirectory, filename)
</code></pre>
<h2 id="future-work">Future work</h2>
<p>The <code>os.Root</code> API is new in Go 1.24.
We expect to make additions and refinements to it in future releases.</p>
<p>The current implementation prioritizes correctness and safety over performance.
Future versions will take advantage of platform-specific APIs, such as
Linux&rsquo;s <code>openat2</code>, to improve performance where possible.</p>
<p>There are a number of filesystem operations which <code>Root</code> does not support yet, such as
creating symbolic links and renaming files. Where possible, we will add support for these
operations. A list of additional functions in progress is in
<a href="/issue/67002">go.dev/issue/67002</a>.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/coretypes">Goodbye core types - Hello Go as we know and love it!</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/cleanups-and-weak">From unique to cleanups and weak: new low-level tools for efficiency</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>From unique to cleanups and weak: new low-level tools for efficiency</title><id>tag:blog.golang.org,2013:blog.golang.org/cleanups-and-weak</id><link rel="alternate" href="https://go.dev/blog/cleanups-and-weak"></link><published>2025-03-06T00:00:00+00:00</published><updated>2025-03-06T00:00:00+00:00</updated><author><name>Michael Knyszek</name></author><summary type="html">Weak pointers and better finalization in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/cleanups-and-weak">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>From unique to cleanups and weak: new low-level tools for efficiency</h1>
 
 <p class="author">
 Michael Knyszek<br>
 6 March 2025
 </p>
 
 <p>In <a href="/blog/unique">last year&rsquo;s blog post</a> about the <code>unique</code> package, we alluded
to some new features then in proposal review, and we&rsquo;re excited to share that as
of Go 1.24 they are now available to all Go developers.
These new features are <a href="https://pkg.go.dev/runtime#AddCleanup" rel="noreferrer" target="_blank">the <code>runtime.AddCleanup</code>
function</a>, which queues up a function to
run when an object is no longer reachable, and <a href="https://pkg.go.dev/weak#Pointer" rel="noreferrer" target="_blank">the <code>weak.Pointer</code>
type</a>, which safely points to an object without
preventing it from being garbage collected.
Together, these two features are powerful enough to build your own <code>unique</code>
package!
Let&rsquo;s dig into what makes these features useful, and when to use them.</p>
<p>Note: these new features are advanced features of the garbage collector.
If you&rsquo;re not already familiar with basic garbage collection concepts, we
strongly recommend reading the introduction of our <a href="/doc/gc-guide#Introduction">garbage collector
guide</a>.</p>
<h2 id="cleanups">Cleanups</h2>
<p>If you&rsquo;ve ever used a finalizer, then the concept of a cleanup will be
familiar.
A finalizer is a function, associated with an allocated object by <a href="https://pkg.go.dev/runtime#SetFinalizer" rel="noreferrer" target="_blank">calling
<code>runtime.SetFinalizer</code></a>, that is later
called by the garbage collector some time after the object becomes unreachable.
At a high level, cleanups work the same way.</p>
<p>Let&rsquo;s consider an application that makes use of a memory-mapped file, and see
how cleanups can help.</p>
<pre><code>//go:build unix

type MemoryMappedFile struct {
 data []byte
}

func NewMemoryMappedFile(filename string) (*MemoryMappedFile, error) {
 f, err := os.Open(filename)
 if err != nil {
 return nil, err
 }
 defer f.Close()

 // Get the file's info; we need its size.
 fi, err := f.Stat()
 if err != nil {
 return nil, err
 }

 // Extract the file descriptor.
 conn, err := f.SyscallConn()
 if err != nil {
 return nil, err
 }
 var data []byte
 connErr := conn.Control(func(fd uintptr) {
 // Create a memory mapping backed by this file.
 data, err = syscall.Mmap(int(fd), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
 })
 if connErr != nil {
 return nil, connErr
 }
 if err != nil {
 return nil, err
 }
 mf := &amp;MemoryMappedFile{data: data}
 cleanup := func(data []byte) {
 syscall.Munmap(data) // ignore error
 }
 runtime.AddCleanup(mf, cleanup, data)
 return mf, nil
}
</code></pre>
<p>A memory-mapped file has its contents mapped to memory, in this case, the
underlying data of a byte slice.
Thanks to operating-system magic, reads and writes to the byte slice directly
access the contents of the file.
With this code, we can pass around a <code>*MemoryMappedFile</code>, and when it&rsquo;s
no longer referenced, the memory mapping we created will get cleaned up.</p>
<p>Notice that <code>runtime.AddCleanup</code> takes three arguments: the address of a
variable to attach the cleanup to, the cleanup function itself, and an argument
to the cleanup function.
A key difference between this function and <code>runtime.SetFinalizer</code> is that the
cleanup function takes a different argument than the object we&rsquo;re attaching the
cleanup to.
This change fixes some problems with finalizers.</p>
<p>It&rsquo;s no secret that <a href="/doc/gc-guide#Common_finalizer_issues">finalizers
are difficult to use correctly</a>.
For example, objects to which finalizers are attached must not be involved
in any reference cycles (even a pointer to itself is too much!), otherwise the
object will never be reclaimed and the finalizer will never run, causing a
leak.
Finalizers also significantly delay reclamation of memory.
It takes at a minimum two full garbage collection cycles to reclaim the
memory for a finalized object: one to determine that it&rsquo;s unreachable, and
another to determine that it&rsquo;s still unreachable after the finalizer
executes.</p>
<p>The problem is that finalizers <a href="https://en.wikipedia.org/wiki/Object_resurrection" rel="noreferrer" target="_blank">resurrect the objects they&rsquo;re attached
to</a>.
The finalizer doesn&rsquo;t run until the object is unreachable, at which point it is
considered &ldquo;dead.&rdquo;
But since the finalizer is called with a pointer to the object, the garbage
collector must prevent the reclamation of that object&rsquo;s memory, and instead must
generate a new reference for the finalizer, making it reachable, or &ldquo;live,&rdquo; once
more.
That reference may even remain after the finalizer returns, for example if the
finalizer writes it to a global variable or sends it across a channel.
Object resurrection is problematic because it means the object, and everything
it points to, and everything those objects point to, and so on, is reachable,
even if it would otherwise have been collected as garbage.</p>
<p>We solve both of these problems by not passing the original object to the
cleanup function.
First, the values the object refers to don&rsquo;t need to be kept specially
reachable by the garbage collector, so the object can still be reclaimed even
if it&rsquo;s involved in a cycle.
Second, since the object is not needed for the cleanup, its memory can be
reclaimed immediately.</p>
<h2 id="weak-pointers">Weak pointers</h2>
<p>Returning to our memory-mapped file example, suppose we notice that our program
frequently maps the same files over and over, from different goroutines that are
unaware of each other.
This is fine from a memory perspective, since all these mappings will share
physical memory, but it results in lots of unnecessary system calls to map and
unmap the file.
This is especially bad if each goroutine reads only a small section of each
file.</p>
<p>So, let&rsquo;s deduplicate the mappings by filename.
(Let&rsquo;s assume that our program only reads from the mappings, and the files
themselves are never modified or renamed once created.
Such assumptions are reasonable for system font files, for example.)</p>
<p>We could maintain a map from filename to memory mapping, but then it becomes
unclear when it&rsquo;s safe to remove entries from that map.
We could <em>almost</em> use a cleanup, if it weren&rsquo;t for the fact that the map entry
itself will keep the memory-mapped file object alive.</p>
<p>Weak pointers solve this problem.
A weak pointer is a special kind of pointer that the garbage collector ignores
when deciding whether an object is reachable.
Go 1.24&rsquo;s <a href="https://pkg.go.dev/weak#Pointer" rel="noreferrer" target="_blank">new weak pointer type,
<code>weak.Pointer</code></a>, has a <code>Value</code> method that
returns either a real pointer if the object is still reachable, or <code>nil</code> if it
is not.</p>
<p>If we instead maintain a map that only <em>weakly</em> points to the memory-mapped
file, we can clean up the map entry when nobody&rsquo;s using it anymore!
Let&rsquo;s see what this looks like.</p>
<pre><code>var cache sync.Map // map[string]weak.Pointer[MemoryMappedFile]

func NewCachedMemoryMappedFile(filename string) (*MemoryMappedFile, error) {
 var newFile *MemoryMappedFile
 for {
 // Try to load an existing value out of the cache.
 value, ok := cache.Load(filename)
 if !ok {
 // No value found. Create a new mapped file if needed.
 if newFile == nil {
 var err error
 newFile, err = NewMemoryMappedFile(filename)
 if err != nil {
 return nil, err
 }
 }

 // Try to install the new mapped file.
 wp := weak.Make(newFile)
 var loaded bool
 value, loaded = cache.LoadOrStore(filename, wp)
 if !loaded {
 runtime.AddCleanup(newFile, func(filename string) {
 // Only delete if the weak pointer is equal. If it's not, someone
 // else already deleted the entry and installed a new mapped file.
 cache.CompareAndDelete(filename, wp)
 }, filename)
 return newFile, nil
 }
 // Someone got to installing the file before us.
 //
 // If it's still there when we check in a moment, we'll discard newFile
 // and it'll get cleaned up by garbage collector.
 }

 // See if our cache entry is valid.
 if mf := value.(weak.Pointer[MemoryMappedFile]).Value(); mf != nil {
 return mf, nil
 }

 // Discovered a nil entry awaiting cleanup. Eagerly delete it.
 cache.CompareAndDelete(filename, value)
 }
}
</code></pre>
<p>This example is a little complicated, but the gist is simple.
We start with a global concurrent map of all the mapped files we made.
<code>NewCachedMemoryMappedFile</code> consults this map for an existing mapped
file, and if that fails, creates and tries to insert a new mapped file.
This could of course fail as well since we&rsquo;re racing with other insertions, so
we need to be careful about that too, and retry.
(This design has a flaw in that we might wastefully map the same file multiple
times in a race, and we&rsquo;ll have to throw it away via the cleanup added by
<code>NewMemoryMappedFile</code>.
This is probably not a big deal most of the time.
Fixing it is left as an exercise for the reader.)</p>
<p>Let&rsquo;s look at some useful properties of weak pointers and cleanups exploited by
this code.</p>
<p>Firstly, notice that weak pointers are comparable.
Not only that, weak pointers have a stable and independent identity, which
remains even after the objects they point to are long gone.
This is why it is safe for the cleanup function to call <code>sync.Map</code>&rsquo;s
<code>CompareAndDelete</code>, which compares the <code>weak.Pointer</code>, and a crucial reason
this code works at all.</p>
<p>Secondly, observe that we can add multiple independent cleanups to a single
<code>MemoryMappedFile</code> object.
This allows us to use cleanups in a composable way and use them to build
generic data structures.
In this particular example, it might be more efficient to combine
<code>NewCachedMemoryMappedFile</code> with <code>NewMemoryMappedFile</code> and
have them share a cleanup.
However, the advantage of the code we wrote above is that it can be rewritten
in a generic way!</p>
<pre><code>type Cache[K comparable, V any] struct {
 create func(K) (*V, error)
 m sync.Map
}

func NewCache[K comparable, V any](create func(K) (*V, error)) *Cache[K, V] {
 return &amp;Cache[K, V]{create: create}
}

func (c *Cache[K, V]) Get(key K) (*V, error) {
 var newValue *V
 for {
 // Try to load an existing value out of the cache.
 value, ok := cache.Load(key)
 if !ok {
 // No value found. Create a new mapped file if needed.
 if newValue == nil {
 var err error
 newValue, err = c.create(key)
 if err != nil {
 return nil, err
 }
 }

 // Try to install the new mapped file.
 wp := weak.Make(newValue)
 var loaded bool
 value, loaded = cache.LoadOrStore(key, wp)
 if !loaded {
 runtime.AddCleanup(newValue, func(key K) {
 // Only delete if the weak pointer is equal. If it's not, someone
 // else already deleted the entry and installed a new mapped file.
 cache.CompareAndDelete(key, wp)
 }, key)
 return newValue, nil
 }
 }

 // See if our cache entry is valid.
 if mf := value.(weak.Pointer[V]).Value(); mf != nil {
 return mf, nil
 }

 // Discovered a nil entry awaiting cleanup. Eagerly delete it.
 cache.CompareAndDelete(key, value)
 }
}
</code></pre>
<h2 id="caveats-and-future-work">Caveats and future work</h2>
<p>Despite our best efforts, cleanups and weak pointers can still be error-prone.
To guide those considering using finalizers, cleanups, and weak pointers, we
recently updated the <a href="/doc/gc-guide#Finalizers_cleanups_and_weak_pointers">guide to the garbage
collector</a> with some advice
about using these features.
Take a look next time you reach for them, but also carefully consider whether
you need to use them at all.
These are advanced tools with subtle semantics and, as the guide says, most
Go code benefits from these features indirectly, not from using them directly.
Stick to the use-cases where these features shine, and you&rsquo;ll be alright.</p>
<p>For now, we&rsquo;ll call out some of the issues that you are more likely to run into.</p>
<p>First, the object the cleanup is attached to must be reachable from neither
the cleanup function (as a captured variable) nor the argument to the cleanup
function.
Both of these situations result in the cleanup never executing.
(In the special case of the cleanup argument being exactly the pointer passed
to <code>runtime.AddCleanup</code>, <code>runtime.AddCleanup</code> will panic, as a signal to the
caller that they should not use cleanups the same way as finalizers.)</p>
<p>Second, when weak pointers are used as map keys, the weakly referenced object
must not be reachable from the corresponding map value, otherwise the object
will continue to remain live.
This may seem obvious when deep inside of a blog post about weak pointers, but
it&rsquo;s an easy subtlety to miss.
This problem inspired the entire concept of an
<a href="https://en.wikipedia.org/wiki/Ephemeron" rel="noreferrer" target="_blank">ephemeron</a> to resolve it, which is a
potential future direction.</p>
<p>Thirdly, a common pattern with cleanups is that a wrapper object is needed, like
we see here with our <code>MemoryMappedFile</code> example.
In this particular case, you could imagine the garbage collector directly
tracking the mapped memory region and passing around the inner <code>[]byte</code>.
Such functionality is possible future work, and an API for it has been recently
<a href="/issue/70224">proposed</a>.</p>
<p>Lastly, both weak pointers and cleanups are inherently non-deterministic, their
behavior depending intimately on the design and dynamics of the garbage
collector.
The documentation for cleanups even permits the garbage collector never to run
cleanups at all.
Effectively testing code that uses them can be tricky, but <a href="/doc/gc-guide#Testing_object_death">it is
possible</a>.</p>
<h2 id="why-now">Why now?</h2>
<p>Weak pointers have been brought up as a feature for Go since nearly the
beginning, but for years were not prioritized by the Go team.
One reason for that is that they are subtle, and the design space of weak
pointers is a minefield of decisions that can make them even harder to use.
Another is that weak pointers are a niche tool, while simultaneously adding
complexity to the language.
We already had experience with how painful <code>SetFinalizer</code> could be to use.
But there are some useful programs that are not expressible without them, and
the <code>unique</code> package and the reasons for its existence really emphasized that.</p>
<p>With generics, the hindsight of finalizers, and insights from all the great work
since done by teams in other languages like C# and Java, the designs for weak
pointers and cleanups came together quickly.
The desire to use weak pointers with finalizers raised additional questions,
and so the design for <code>runtime.AddCleanup</code> quickly came together as well.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>I want to thank everyone in the community who contributed feedback on the
proposal issues and filed bugs when the features became available.
I also want to thank David Chase for thoroughly thinking through weak
pointer semantics with me, and I want to thank him, Russ Cox, and Austin
Clements for their help with the design of <code>runtime.AddCleanup</code>.
I want to thank Carlos Amedee for his work on getting <code>runtime.AddCleanup</code>
implemented, polished, landed for Go 1.24.
And finally I want to thank Carlos Amedee and Ian Lance Taylor for their work
replacing <code>runtime.SetFinalizer</code> with <code>runtime.AddCleanup</code> throughout the
standard library for Go 1.25.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/osroot">Traversal-resistant file APIs</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/swisstable">Faster Go maps with Swiss Tables</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Faster Go maps with Swiss Tables</title><id>tag:blog.golang.org,2013:blog.golang.org/swisstable</id><link rel="alternate" href="https://go.dev/blog/swisstable"></link><published>2025-02-26T00:00:00+00:00</published><updated>2025-02-26T00:00:00+00:00</updated><author><name>Michael Pratt</name></author><summary type="html">Go 1.24 improves map performance with a brand new map implementation</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/swisstable">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Faster Go maps with Swiss Tables</h1>
 
 <p class="author">
 Michael Pratt<br>
 26 February 2025
 </p>
 
 <p>The hash table is a central data structure in computer science, and it provides the implementation for the map type in many languages, including Go.</p>
<p>The concept of a hash table was <a href="https://spectrum.ieee.org/hans-peter-luhn-and-the-birth-of-the-hashing-algorithm" rel="noreferrer" target="_blank">first described</a> by Hans Peter Luhn in 1953 in an internal IBM memo that suggested speeding up search by placing items into &ldquo;buckets&rdquo; and using a linked list for overflow when buckets already contain an item.
Today we would call this a <a href="https://en.wikipedia.org/wiki/Hash_table#Separate_chaining" rel="noreferrer" target="_blank">hash table using chaining</a>.</p>
<p>In 1954, Gene M. Amdahl, Elaine M. McGraw, and Arthur L. Samuel first used an &ldquo;open addressing&rdquo; scheme when programming the IBM 701.
When a bucket already contains an item, the new item is placed in the next empty bucket.
This idea was formalized and published in 1957 by W. Wesley Peterson in <a href="https://ieeexplore.ieee.org/document/5392733" rel="noreferrer" target="_blank">&ldquo;Addressing for Random-Access Storage&rdquo;</a>.
Today we would call this a <a href="https://en.wikipedia.org/wiki/Hash_table#Open_addressing" rel="noreferrer" target="_blank">hash table using open addressing with linear probing</a>.</p>
<p>With data structures that have been around this long, it&rsquo;s easy to think that they must be &ldquo;done&rdquo;; that we know everything there is to know about them and they can&rsquo;t be improved anymore.
That&rsquo;s not true!
Computer science research continues to make advancements in fundamental algorithms, both in terms of algorithmic complexity and taking advantage of modern CPU hardware.
For example, Go 1.19 <a href="/doc/go1.19#sortpkgsort">switched the <code>sort</code> package</a> from a traditional quicksort, to <a href="https://arxiv.org/pdf/2106.05123.pdf" rel="noreferrer" target="_blank">pattern-defeating quicksort</a>, a novel sorting algorithm from Orson R. L. Peters, first described in 2015.</p>
<p>Like sorting algorithms, hash table data structures continue to see improvements.
In 2017, Sam Benzaquen, Alkis Evlogimenos, Matt Kulukundis, and Roman Perepelitsa at Google presented <a href="https://www.youtube.com/watch?v=ncHmEUmJZf4" rel="noreferrer" target="_blank">a new C++ hash table design</a>, dubbed &ldquo;Swiss Tables&rdquo;.
In 2018, their implementation was <a href="https://abseil.io/blog/20180927-swisstables" rel="noreferrer" target="_blank">open sourced in the Abseil C++ library</a>.</p>
<p>Go 1.24 includes a completely new implementation of the built-in map type, based on the Swiss Table design.
In this blog post we&rsquo;ll look at how Swiss Tables improve upon traditional hash tables, and at some of the unique challenges in bringing the Swiss Table design to Go&rsquo;s maps.</p>
<h2 id="open-addressed-hash-table">Open-addressed hash table</h2>
<p>Swiss Tables are a form of open-addressed hash table, so let&rsquo;s do a quick overview of how a basic open-addressed hash table works.</p>
<p>In an open-addressed hash table, all items are stored in a single backing array.
We&rsquo;ll call each location in the array a <em>slot</em>.
The slot to which a key belongs is primarily determined by the <em>hash function</em>, <code>hash(key)</code>.
The hash function maps each key to an integer, where the same key always maps to the same integer, and different keys ideally follow a uniform random distribution of integers.
The defining feature of open-addressed hash tables is that they resolve collisions by storing the key elsewhere in the backing array.
So, if the slot is already full (a <em>collision</em>), then a <em>probe sequence</em> is used to consider other slots until an empty slot is found.
Let&rsquo;s take a look at a sample hash table to see how this works.</p>
<h3 id="example">Example</h3>
<p>Below you can see a 16-slot backing array for a hash table, and the key (if any) stored in each slot.
The values are not shown, as they are not relevant to this example.</p>
<style>
 
@media screen and (max-width: 55em) {
 .swisstable-table-container {
 
 overflow: scroll;
 }
}

.swisstable-table {
 
 border-collapse: collapse;
 
 table-layout: fixed;
 
 margin: 0 auto;
}

.swisstable-table-cell {
 
 border: 1px solid;
 
 padding: 0.5em 1em 0.5em 1em;
 
 text-align: center;
}
</style>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">8</th>
 <th class="swisstable-table-cell">9</th>
 <th class="swisstable-table-cell">10</th>
 <th class="swisstable-table-cell">11</th>
 <th class="swisstable-table-cell">12</th>
 <th class="swisstable-table-cell">13</th>
 <th class="swisstable-table-cell">14</th>
 <th class="swisstable-table-cell">15</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">Key</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">56</td>
 <td class="swisstable-table-cell">32</td>
 <td class="swisstable-table-cell">21</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">78</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
</div>
<p>To insert a new key, we use the hash function to select a slot.
Since there are only 16 slots, we need to restrict to this range, so we&rsquo;ll use <code>hash(key) % 16</code> as the target slot.
Suppose we want to insert key <code>98</code> and <code>hash(98) % 16 = 7</code>.
Slot 7 is empty, so we simply insert 98 there.
On the other hand, suppose we want to insert key <code>25</code> and <code>hash(25) % 16 = 3</code>.
Slot 3 is a collision because it already contains key 56.
Thus we cannot insert here.</p>
<p>We use a probe sequence to find another slot.
There are a variety of well-known probe sequences.
The original and most straightforward probe sequence is <em>linear probing</em>, which simply tries successive slots in order.</p>
<p>So, in the <code>hash(25) % 16 = 3</code> example, since slot 3 is in use, we would consider slot 4 next, which is also in use.
So too is slot 5.
Finally, we&rsquo;d get to empty slot 6, where we&rsquo;d store key 25.</p>
<p>Lookup follows the same approach.
A lookup of key 25 would start at slot 3, check whether it contains key 25 (it does not), and then continue linear probing until it finds key 25 in slot 6.</p>
<p>This example uses a backing array with 16 slots.
What happens if we insert more than 16 elements?
If the hash table runs out of space, it will grow, usually by doubling the size of the backing array.
All existing entries are reinserted into the new backing array.</p>
<p>Open-addressed hash tables don&rsquo;t actually wait until the backing array is completely full to grow because as the array gets more full, the average length of each probe sequence increases.
In the example above using key 25, we must probe 4 different slots to find an empty slot.
If the array had only one empty slot, the worst case probe length would be O(n).
That is, you may need to scan the entire array.
The proportion of used slots is called the <em>load factor</em>, and most hash tables define a <em>maximum load factor</em> (typically 70-90%) at which point they will grow to avoid the extremely long probe sequences of very full hash tables.</p>
<h2 id="swiss-table">Swiss Table</h2>
<p>The Swiss Table <a href="https://abseil.io/about/design/swisstables" rel="noreferrer" target="_blank">design</a> is also a form of open-addressed hash table.
Let&rsquo;s see how it improves over a traditional open-addressed hash table.
We still have a single backing array for storage, but we will break the array into logical <em>groups</em> of 8 slots each.
(Larger group sizes are possible as well. More on that below.)</p>
<p>In addition, each group has a 64-bit <em>control word</em> for metadata.
Each of the 8 bytes in the control word corresponds to one of the slots in the group.
The value of each byte denotes whether that slot is empty, deleted, or in use.
If it is in use, the byte contains the lower 7 bits of the hash for that slot&rsquo;s key (called <code>h2</code>).</p>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell"></th>
 <th class="swisstable-table-cell" colspan="8">Group 0</th>
 <th class="swisstable-table-cell" colspan="8">Group 1</th>
 </tr>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">Key</td>
 <td class="swisstable-table-cell">56</td>
 <td class="swisstable-table-cell">32</td>
 <td class="swisstable-table-cell">21</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">78</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
 <br/> 
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell"></th>
 <th class="swisstable-table-cell" colspan="8">64-bit control word 0</th>
 <th class="swisstable-table-cell" colspan="8">64-bit control word 1</th>
 </tr>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">h2</td>
 <td class="swisstable-table-cell">23</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">50</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">47</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
</div>
<p>Insertion works as follows:</p>
<ol>
<li>Compute <code>hash(key)</code> and break the hash into two parts: the upper 57-bits (called <code>h1</code>) and the lower 7 bits (called <code>h2</code>).</li>
<li>The upper bits (<code>h1</code>) are used to select the first group to consider: <code>h1 % 2</code> in this case, as there are only 2 groups.</li>
<li>Within a group, all slots are equally eligible to hold the key. We must first determine whether any slot already contains this key, in which case this is an update rather than a new insertion.</li>
<li>If no slot contains the key, then we look for an empty slot to place this key.</li>
<li>If no slot is empty, then we continue the probe sequence by searching the next group.</li>
</ol>
<p>Lookup follows the same basic process.
If we find an empty slot in step 4, then we know an insertion would have used this slot and can stop the search.</p>
<p>Step 3 is where the Swiss Table magic happens.
We need to check whether any slot in a group contains the desired key.
Naively, we could just do a linear scan and compare all 8 keys.
However, the control word lets us do this more efficiently.
Each byte contains the lower 7 bits of the hash (<code>h2</code>) for that slot.
If we determine which bytes of the control word contain the <code>h2</code> we are looking for, we&rsquo;ll have a set of candidate matches.</p>
<p>In other words, we want to do a byte-by-byte equality comparison within the control word.
For example, if we are looking for key 32, where <code>h2 = 89</code>, the operation we want looks like so.</p>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <tbody>
 <tr>
 <td class="swisstable-table-cell"><strong>Test word</strong></td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Comparison</strong></td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Control word</strong></td>
 <td class="swisstable-table-cell">23</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">50</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Result</strong></td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">1</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 </tr>
 </tbody>
 </table>
</div>
<p>This is an operation supported by <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data" rel="noreferrer" target="_blank">SIMD</a> hardware, where a single instruction performs parallel operations on independent values within a larger value (<em>vector</em>). In this case, we <a href="https://cs.opensource.google/go/go/+/master:src/internal/runtime/maps/group.go;drc=a08984bc8f2acacebeeadf7445ecfb67b7e7d7b1;l=155?ss=go" rel="noreferrer" target="_blank">can implement this operation</a> using a set of standard arithmetic and bitwise operations when special SIMD hardware is not available.</p>
<p>The result is a set of candidate slots.
Slots where <code>h2</code> does not match do not have a matching key, so they can be skipped.
Slots where <code>h2</code> does match are potential matches, but we must still check the entire key, as there is potential for collisions (1/128 probability of collision with a 7-bit hash, so still quite low).</p>
<p>This operation is very powerful, as we have effectively performed 8 steps of a probe sequence at once, in parallel.
This speeds up lookup and insertion by reducing the average number of comparisons we need to perform.
This improvement to probing behavior allowed both the Abseil and Go implementations to increase the maximum load factor of Swiss Table maps compared to prior maps, which lowers the average memory footprint.</p>
<h2 id="go-challenges">Go challenges</h2>
<p>Go&rsquo;s built-in map type has some unusual properties that pose additional challenges to adopting a new map design.
Two were particularly tricky to deal with.</p>
<h3 id="incremental-growth">Incremental growth</h3>
<p>When a hash table reaches its maximum load factor, it needs to grow the backing array.
Typically this means the next insertion doubles the size of the array, and copies all entries to the new array.
Imagine inserting into a map with 1GB of entries.
Most insertions are very fast, but the one insertion that needs to grow the map from 1GB to 2GB will need to copy 1GB of entries, which will take a long time.</p>
<p>Go is frequently used for latency-sensitive servers, so we don&rsquo;t want operations on built-in types that can have arbitrarily large impact on tail latency.
Instead, Go maps grow incrementally, so that each insertion has an upper bound on the amount of growth work it must do.
This bounds the latency impact of a single map insertion.</p>
<p>Unfortunately, the Abseil (C++) Swiss Table design assumes all at once growth, and the probe sequence depends on the total group count, making it difficult to break up.</p>
<p>Go&rsquo;s built-in map addresses this with another layer of indirection by splitting each map into multiple Swiss Tables.
Rather than a single Swiss Table implementing the entire map, each map consists of one or more independent tables that cover a subset of the key space.
An individual table stores a maximum of 1024 entries.
A variable number of upper bits in the hash are used to select which table a key belongs to.
This is a form of <a href="https://en.wikipedia.org/wiki/Extendible_hashing" rel="noreferrer" target="_blank"><em>extendible hashing</em></a>, where the number of bits used increases as needed to differentiate the total number of tables.</p>
<p>During insertion, if an individual table needs to grow, it will do so all at once, but other tables are unaffected.
Thus the upper bound for a single insertion is the latency of growing a 1024-entry table into two 1024-entry tables, copying 1024 entries.</p>
<h3 id="modification-during-iteration">Modification during iteration</h3>
<p>Many hash table designs, including Abseil&rsquo;s Swiss Tables, forbid modifying the map during iteration.
The Go language spec <a href="/ref/spec#For_statements:~:text=The%20iteration%20order,iterations%20is%200.">explicitly allows</a> modifications during iteration, with the following semantics:</p>
<ul>
<li>If an entry is deleted before it is reached, it will not be produced.</li>
<li>If an entry is updated before it is reached, the updated value will be produced.</li>
<li>If a new entry is added, it may or may not be produced.</li>
</ul>
<p>A typical approach to hash table iteration is to simply walk through the backing array and produce values in the order they are laid out in memory.
This approach runs afoul of the above semantics, most notably because insertions may grow the map, which would shuffle the memory layout.</p>
<p>We can avoid the impact of shuffle during growth by having the iterator keep a reference to the table it is currently iterating over.
If that table grows during iteration, we keep using the old version of the table and thus continue to deliver keys in the order of the old memory layout.</p>
<p>Does this work with the above semantics?
New entries added after growth will be missed entirely, as they are only added to the grown table, not the old table.
That&rsquo;s fine, as the semantics allow new entries not to be produced.
Updates and deletions are a problem, though: using the old table could produce stale or deleted entries.</p>
<p>This edge case is addressed by using the old table only to determine the iteration order.
Before actually returning the entry, we consult the grown table to determine whether the entry still exists, and to retrieve the latest value.</p>
<p>This covers all the core semantics, though there are even more small edge cases not covered here.
Ultimately, the permissiveness of Go maps with iteration results in iteration being the most complex part of Go&rsquo;s map implementation.</p>
<h2 id="future-work">Future work</h2>
<p>In <a href="/issue/54766#issuecomment-2542444404">microbenchmarks</a>, map operations are up to 60% faster than in Go 1.23.
Exact performance improvement varies quite a bit due to the wide variety of operations and uses of maps, and some edge cases do regress compared to Go 1.23.
Overall, in full application benchmarks, we found a geometric mean CPU time improvement of around 1.5%.</p>
<p>There are more map improvements we want to investigate for future Go releases.
For example, we may be able to <a href="/issue/70835">increase the locality of</a> operations on maps that are not in the CPU cache.</p>
<p>We could also further improve the control word comparisons.
As described above, we have a portable implementation using standard arithmetic and bitwise operations.
However, some architectures have SIMD instructions that perform this sort of comparison directly.
Go 1.24 already uses 8-byte SIMD instructions for amd64, but we could extend support to other architectures.
More importantly, while standard instructions operate on up to 8-byte words, SIMD instructions nearly always support at least 16-byte words.
This means we could increase the group size to 16 slots, and perform 16 hash comparisons in parallel instead of 8.
This would further decrease the average number of probes required for lookups.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>A Swiss Table-based Go map implementation has been a long time coming, and involved many contributors.
I want to thank YunHao Zhang (<a href="https://github.com/zhangyunhao116" rel="noreferrer" target="_blank">@zhangyunhao116</a>), PJ Malloy (<a href="https://github.com/thepudds" rel="noreferrer" target="_blank">@thepudds</a>), and <a href="https://github.com/andy-wm-arthur" rel="noreferrer" target="_blank">@andy-wm-arthur</a> for building initial versions of a Go Swiss Table implementation.
Peter Mattis (<a href="https://github.com/petermattis" rel="noreferrer" target="_blank">@petermattis</a>) combined these ideas with solutions to the Go challenges above to build <a href="https://pkg.go.dev/github.com/cockroachdb/swiss" rel="noreferrer" target="_blank"><code>github.com/cockroachdb/swiss</code></a>, a Go-spec compliant Swiss Table implementation.
The Go 1.24 built-in map implementation is heavily based on Peter&rsquo;s work.
Thank you to everyone in the community that contributed!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/cleanups-and-weak">From unique to cleanups and weak: new low-level tools for efficiency</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/synctest">Testing concurrent code with testing/synctest</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Testing concurrent code with testing/synctest</title><id>tag:blog.golang.org,2013:blog.golang.org/synctest</id><link rel="alternate" href="https://go.dev/blog/synctest"></link><published>2025-02-19T00:00:00+00:00</published><updated>2025-02-19T00:00:00+00:00</updated><author><name>Damien Neil</name></author><summary type="html">Go 1.24 contains an experimental package to aid in testing concurrent code.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/synctest">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Testing concurrent code with testing/synctest</h1>
 
 <p class="author">
 Damien Neil<br>
 19 February 2025
 </p>
 
 <p>One of Go&rsquo;s signature features is built-in support for concurrency.
Goroutines and channels are simple and effective primitives for
writing concurrent programs.</p>
<p>However, testing concurrent programs can be difficult and error prone.</p>
<p>In Go 1.24, we are introducing a new, experimental
<a href="/pkg/testing/synctest"><code>testing/synctest</code></a> package
to support testing concurrent code. This post will explain the motivation behind
this experiment, demonstrate how to use the synctest package, and discuss its potential future.</p>
<p>In Go 1.24, the <code>testing/synctest</code> package is experimental and
not subject to the Go compatibility promise.
It is not visible by default.
To use it, compile your code with <code>GOEXPERIMENT=synctest</code> set in your environment.</p>
<h2 id="testing-concurrent-programs-is-difficult">Testing concurrent programs is difficult</h2>
<p>To begin with, let us consider a simple example.</p>
<p>The <a href="/pkg/context#AfterFunc"><code>context.AfterFunc</code></a> function
arranges for a function to be called in its own goroutine after a context is canceled.
Here is a possible test for <code>AfterFunc</code>:</p>
<pre><code>func TestAfterFunc(t *testing.T) {
 ctx, cancel := context.WithCancel(context.Background())

 calledCh := make(chan struct{}) // closed when AfterFunc is called
 context.AfterFunc(ctx, func() {
 close(calledCh)
 })

 // TODO: Assert that the AfterFunc has not been called.

 cancel()

 // TODO: Assert that the AfterFunc has been called.
}
</code></pre>
<p>We want to check two conditions in this test:
The function is not called before the context is canceled,
and the function <em>is</em> called after the context is canceled.</p>
<p>Checking a negative in a concurrent system is difficult.
We can easily test that the function has not been called <em>yet</em>,
but how do we check that it <em>will not</em> be called?</p>
<p>A common approach is to wait for some amount of time before
concluding that an event will not happen.
Let&rsquo;s try introducing a helper function to our test which does this.</p>
<pre><code>// funcCalled reports whether the function was called.
funcCalled := func() bool {
 select {
 case &lt;-calledCh:
 return true
 case &lt;-time.After(10 * time.Millisecond):
 return false
 }
}

if funcCalled() {
 t.Fatalf(&quot;AfterFunc function called before context is canceled&quot;)
}

cancel()

if !funcCalled() {
 t.Fatalf(&quot;AfterFunc function not called after context is canceled&quot;)
}
</code></pre>
<p>This test is slow:
10 milliseconds isn&rsquo;t a lot of time, but it adds up over many tests.</p>
<p>This test is also flaky:
10 milliseconds is a long time on a fast computer,
but it isn&rsquo;t unusual to see pauses lasting several seconds
on shared and overloaded
<a href="https://en.wikipedia.org/wiki/Continuous_integration" rel="noreferrer" target="_blank">CI</a>
systems.</p>
<p>We can make the test less flaky at the expense of making it slower,
and we can make it less slow at the expense of making it flakier,
but we can&rsquo;t make it both fast and reliable.</p>
<h2 id="introducing-the-testingsynctest-package">Introducing the testing/synctest package</h2>
<p>The <code>testing/synctest</code> package solves this problem.
It allows us to rewrite this test to be simple, fast, and reliable,
without any changes to the code being tested.</p>
<p>The package contains only two functions: <code>Run</code> and <code>Wait</code>.</p>
<p><code>Run</code> calls a function in a new goroutine.
This goroutine and any goroutines started by it
exist in an isolated environment which we call a <em>bubble</em>.
<code>Wait</code> waits for every goroutine in the current goroutine&rsquo;s bubble
to block on another goroutine in the bubble.</p>
<p>Let&rsquo;s rewrite our test above using the <code>testing/synctest</code> package.</p>
<pre><code>func TestAfterFunc(t *testing.T) {
 synctest.Run(func() {
 ctx, cancel := context.WithCancel(context.Background())

 funcCalled := false
 context.AfterFunc(ctx, func() {
 funcCalled = true
 })

 synctest.Wait()
 if funcCalled {
 t.Fatalf(&quot;AfterFunc function called before context is canceled&quot;)
 }

 cancel()

 synctest.Wait()
 if !funcCalled {
 t.Fatalf(&quot;AfterFunc function not called after context is canceled&quot;)
 }
 })
}
</code></pre>
<p>This is almost identical to our original test,
but we have wrapped the test in a <code>synctest.Run</code> call
and we call <code>synctest.Wait</code> before asserting that the function has been called or not.</p>
<p>The <code>Wait</code> function waits for every goroutine in the caller&rsquo;s bubble to block.
When it returns, we know that the context package has either called the function,
or will not call it until we take some further action.</p>
<p>This test is now both fast and reliable.</p>
<p>The test is simpler, too:
we have replaced the <code>calledCh</code> channel with a boolean.
Previously we needed to use a channel to avoid a data race between
the test goroutine and the <code>AfterFunc</code> goroutine,
but the <code>Wait</code> function now provides that synchronization.</p>
<p>The race detector understands <code>Wait</code> calls,
and this test passes when run with <code>-race</code>.
If we remove the second <code>Wait</code> call,
the race detector will correctly report a data race in the test.</p>
<h2 id="testing-time">Testing time</h2>
<p>Concurrent code often deals with time.</p>
<p>Testing code that works with time can be difficult.
Using real time in tests causes slow and flaky tests,
as we have seen above.
Using fake time requires avoiding <code>time</code> package functions,
and designing the code under test to work with
an optional fake clock.</p>
<p>The <code>testing/synctest</code> package makes it simpler to test code that uses time.</p>
<p>Goroutines in the bubble started by <code>Run</code> use a fake clock.
Within the bubble, functions in the <code>time</code> package operate on the
fake clock. Time advances in the bubble when all goroutines are
blocked.</p>
<p>To demonstrate, let&rsquo;s write a test for the
<a href="/pkg/context#WithTimeout"><code>context.WithTimeout</code></a> function.
<code>WithTimeout</code> creates a child of a context,
which expires after a given timeout.</p>
<pre><code>func TestWithTimeout(t *testing.T) {
 synctest.Run(func() {
 const timeout = 5 * time.Second
 ctx, cancel := context.WithTimeout(context.Background(), timeout)
 defer cancel()

 // Wait just less than the timeout.
 time.Sleep(timeout - time.Nanosecond)
 synctest.Wait()
 if err := ctx.Err(); err != nil {
 t.Fatalf(&quot;before timeout, ctx.Err() = %v; want nil&quot;, err)
 }

 // Wait the rest of the way until the timeout.
 time.Sleep(time.Nanosecond)
 synctest.Wait()
 if err := ctx.Err(); err != context.DeadlineExceeded {
 t.Fatalf(&quot;after timeout, ctx.Err() = %v; want DeadlineExceeded&quot;, err)
 }
 })
}
</code></pre>
<p>We write this test just as if we were working with real time.
The only difference is that we wrap the test function in <code>synctest.Run</code>,
and call <code>synctest.Wait</code> after each <code>time.Sleep</code> call to wait for the context
package&rsquo;s timers to finish running.</p>
<h2 id="blocking-and-the-bubble">Blocking and the bubble</h2>
<p>A key concept in <code>testing/synctest</code> is the bubble becoming <em>durably blocked</em>.
This happens when every goroutine in the bubble is blocked,
and can only be unblocked by another goroutine in the bubble.</p>
<p>When a bubble is durably blocked:</p>
<ul>
<li>If there is an outstanding <code>Wait</code> call, it returns.</li>
<li>Otherwise, time advances to the next time that could unblock a goroutine, if any.</li>
<li>Otherwise, the bubble is deadlocked and <code>Run</code> panics.</li>
</ul>
<p>A bubble is not durably blocked if any goroutine is blocked
but might be woken by some event from outside the bubble.</p>
<p>The complete list of operations which durably block a goroutine is:</p>
<ul>
<li>a send or receive on a nil channel</li>
<li>a send or receive blocked on a channel created within the same bubble</li>
<li>a select statement where every case is durably blocking</li>
<li><code>time.Sleep</code></li>
<li><code>sync.Cond.Wait</code></li>
<li><code>sync.WaitGroup.Wait</code></li>
</ul>
<h3 id="mutexes">Mutexes</h3>
<p>Operations on a <code>sync.Mutex</code> are not durably blocking.</p>
<p>It is common for functions to acquire a global mutex.
For example, a number of functions in the reflect package
use a global cache guarded by a mutex.
If a goroutine in a synctest bubble blocks while acquiring
a mutex held by a goroutine outside the bubble,
it is not durably blocked—it is blocked, but will be unblocked
by a goroutine from outside its bubble.</p>
<p>Since mutexes are usually not held for long periods of time,
we simply exclude them from <code>testing/synctest</code>&rsquo;s consideration.</p>
<h3 id="channels">Channels</h3>
<p>Channels created within a bubble behave differently from ones created outside.</p>
<p>Channel operations are durably blocking only if the channel is bubbled
(created in the bubble).
Operating on a bubbled channel from outside the bubble panics.</p>
<p>These rules ensure that a goroutine is durably blocked only when
communicating with goroutines within its bubble.</p>
<h3 id="io">I/O</h3>
<p>External I/O operations, such as reading from a network connection,
are not durably blocking.</p>
<p>Network reads may be unblocked by writes from outside the bubble,
possibly even from other processes.
Even if the only writer to a network connection is also in the same bubble,
the runtime cannot distinguish between a connection waiting for more data to arrive
and one where the kernel has received data and is in the process of delivering it.</p>
<p>Testing a network server or client with synctest will generally
require supplying a fake network implementation.
For example, the <a href="/pkg/net#Pipe"><code>net.Pipe</code></a> function
creates a pair of <code>net.Conn</code>s that use an in-memory network connection
and can be used in synctest tests.</p>
<h2 id="bubble-lifetime">Bubble lifetime</h2>
<p>The <code>Run</code> function starts a goroutine in a new bubble.
It returns when every goroutine in the bubble has exited.
It panics if the bubble is durably blocked
and cannot be unblocked by advancing time.</p>
<p>The requirement that every goroutine in the bubble exit before Run returns
means that tests must be careful to clean up any background goroutines
before completing.</p>
<h2 id="testing-networked-code">Testing networked code</h2>
<p>Let&rsquo;s look at another example, this time using the <code>testing/synctest</code>
package to test a networked program.
For this example, we&rsquo;ll test the <code>net/http</code> package&rsquo;s handling of
the 100 Continue response.</p>
<p>An HTTP client sending a request can include an &ldquo;Expect: 100-continue&rdquo;
header to tell the server that the client has additional data to send.
The server may then respond with a 100 Continue informational response
to request the rest of the request,
or with some other status to tell the client that the content is not needed.
For example, a client uploading a large file might use this feature to
confirm that the server is willing to accept the file before sending it.</p>
<p>Our test will confirm that when sending an &ldquo;Expect: 100-continue&rdquo; header
the HTTP client does not send a request&rsquo;s content before the server
requests it, and that it does send the content after receiving a
100 Continue response.</p>
<p>Often tests of a communicating client and server can use a
loopback network connection. When working with <code>testing/synctest</code>,
however, we will usually want to use a fake network connection
to allow us to detect when all goroutines are blocked on the network.
We&rsquo;ll start this test by creating an <code>http.Transport</code> (an HTTP client) that uses
an in-memory network connection created by <a href="/pkg/net#Pipe"><code>net.Pipe</code></a>.</p>
<pre><code>func Test(t *testing.T) {
 synctest.Run(func() {
 srvConn, cliConn := net.Pipe()
 defer srvConn.Close()
 defer cliConn.Close()
 tr := &amp;http.Transport{
 DialContext: func(ctx context.Context, network, address string) (net.Conn, error) {
 return cliConn, nil
 },
 // Setting a non-zero timeout enables &quot;Expect: 100-continue&quot; handling.
 // Since the following test does not sleep,
 // we will never encounter this timeout,
 // even if the test takes a long time to run on a slow machine.
 ExpectContinueTimeout: 5 * time.Second,
 }
</code></pre>
<p>We send a request on this transport with the &ldquo;Expect: 100-continue&rdquo; header set.
The request is sent in a new goroutine, since it won&rsquo;t complete until the end of the test.</p>
<pre><code> body := &quot;request body&quot;
 go func() {
 req, _ := http.NewRequest(&quot;PUT&quot;, &quot;http://test.tld/&quot;, strings.NewReader(body))
 req.Header.Set(&quot;Expect&quot;, &quot;100-continue&quot;)
 resp, err := tr.RoundTrip(req)
 if err != nil {
 t.Errorf(&quot;RoundTrip: unexpected error %v&quot;, err)
 } else {
 resp.Body.Close()
 }
 }()
</code></pre>
<p>We read the request headers sent by the client.</p>
<pre><code> req, err := http.ReadRequest(bufio.NewReader(srvConn))
 if err != nil {
 t.Fatalf(&quot;ReadRequest: %v&quot;, err)
 }
</code></pre>
<p>Now we come to the heart of the test.
We want to assert that the client will not send the request body yet.</p>
<p>We start a new goroutine copying the body sent to the server into a <code>strings.Builder</code>,
wait for all goroutines in the bubble to block, and verify that we haven&rsquo;t read anything
from the body yet.</p>
<p>If we forget the <code>synctest.Wait</code> call, the race detector will correctly complain
about a data race, but with the <code>Wait</code> this is safe.</p>
<pre><code> var gotBody strings.Builder
 go io.Copy(&amp;gotBody, req.Body)
 synctest.Wait()
 if got := gotBody.String(); got != &quot;&quot; {
 t.Fatalf(&quot;before sending 100 Continue, unexpectedly read body: %q&quot;, got)
 }
</code></pre>
<p>We write a &ldquo;100 Continue&rdquo; response to the client and verify that it now sends the
request body.</p>
<pre><code> srvConn.Write([]byte(&quot;HTTP/1.1 100 Continue\r\n\r\n&quot;))
 synctest.Wait()
 if got := gotBody.String(); got != body {
 t.Fatalf(&quot;after sending 100 Continue, read body %q, want %q&quot;, got, body)
 }
</code></pre>
<p>And finally, we finish up by sending the &ldquo;200 OK&rdquo; response to conclude the request.</p>
<p>We have started several goroutines during this test.
The <code>synctest.Run</code> call will wait for all of them to exit before returning.</p>
<pre><code> srvConn.Write([]byte(&quot;HTTP/1.1 200 OK\r\n\r\n&quot;))
 })
}
</code></pre>
<p>This test can be easily extended to test other behaviors,
such as verifying that the request body is not sent if the server does not ask for it,
or that it is sent if the server does not respond within a timeout.</p>
<h2 id="status-of-the-experiment">Status of the experiment</h2>
<p>We are introducing <a href="/pkg/testing/synctest"><code>testing/synctest</code></a>
in Go 1.24 as an <em>experimental</em> package.
Depending on feedback and experience
we may release it with or without amendments,
continue the experiment,
or remove it in a future version of Go.</p>
<p>The package is not visible by default.
To use it, compile your code with <code>GOEXPERIMENT=synctest</code> set in your environment.</p>
<p>We want to hear your feedback!
If you try out <code>testing/synctest</code>,
please report your experiences, positive or negative,
on <a href="/issue/67434">go.dev/issue/67434</a>.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/swisstable">Faster Go maps with Swiss Tables</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/wasmexport">Extensible Wasm Applications with Go</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Extensible Wasm Applications with Go</title><id>tag:blog.golang.org,2013:blog.golang.org/wasmexport</id><link rel="alternate" href="https://go.dev/blog/wasmexport"></link><published>2025-02-13T00:00:00+00:00</published><updated>2025-02-13T00:00:00+00:00</updated><author><name>Cherry Mui</name></author><summary type="html">Go 1.24 enhances WebAssembly capabilities with function export and reactor mode</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/wasmexport">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Extensible Wasm Applications with Go</h1>
 
 <p class="author">
 Cherry Mui<br>
 13 February 2025
 </p>
 
 <p>Go 1.24 enhances its WebAssembly (Wasm) capabilities with the
addition of the <code>go:wasmexport</code> directive and the ability to build a reactor
for WebAssembly System Interface (WASI).
These features enable Go developers to export Go functions to Wasm,
facilitating better integration with Wasm hosts and expanding the possibilities
for Go-based Wasm applications.</p>
<h2 id="webassembly-and-the-webassembly-system-interface">WebAssembly and the WebAssembly System Interface</h2>
<p><a href="https://webassembly.org/" rel="noreferrer" target="_blank">WebAssembly (Wasm)</a> is a binary instruction format
that was initially created for web browsers, providing the execution of
high-performance, low-level code at speeds approaching native performance.
Since then, Wasm&rsquo;s utility has expanded, and it is now used in various
environments beyond the browser.
Notably, cloud providers offer services that directly execute Wasm
executables, taking advantage of the
<a href="https://wasi.dev/" rel="noreferrer" target="_blank">WebAssembly System Interface (WASI)</a> system call API.
WASI allows these executables to interact with system resources.</p>
<p>Go first added support for compiling to Wasm in the 1.11 release, through the
<code>js/wasm</code> port.
Go 1.21 added a new port targeting the WASI preview 1 syscall API through the
new <code>GOOS=wasip1</code> port.</p>
<h2 id="exporting-go-functions-to-wasm-with-gowasmexport">Exporting Go Functions to Wasm with <code>go:wasmexport</code></h2>
<p>Go 1.24 introduces a new compiler directive, <code>go:wasmexport</code>, which allows
developers to export Go functions to be called from outside of the
Wasm module, typically from a host application that runs the Wasm runtime.
This directive instructs the compiler to make the annotated function available
as a Wasm <a href="https://webassembly.github.io/spec/core/valid/modules.html?highlight=export#exports" rel="noreferrer" target="_blank">export</a>
in the resulting Wasm binary.</p>
<p>To use the <code>go:wasmexport</code> directive, simply add it to a function definition:</p>
<pre><code>//go:wasmexport add
func add(a, b int32) int32 { return a + b }
</code></pre>
<p>With this, the Wasm module will have an exported function named <code>add</code> that
can be called from the host.</p>
<p>This is analogous to the <a href="/cmd/cgo#hdr-C_references_to_Go">cgo <code>export</code> directive</a>,
which makes the function available to be called from C,
though <code>go:wasmexport</code> uses a different, simpler mechanism.</p>
<h2 id="building-a-wasi-reactor">Building a WASI Reactor</h2>
<p>A WASI reactor is a WebAssembly module that operates continuously, and
can be called upon multiple times to react on events or requests.
Unlike a &ldquo;command&rdquo; module, which terminates after its main function finishes,
a reactor instance remains live after initialization, and its exports remain
accessible.</p>
<p>With Go 1.24, one can build a WASI reactor with the <code>-buildmode=c-shared</code> build
flag.</p>
<pre><code>$ GOOS=wasip1 GOARCH=wasm go build -buildmode=c-shared -o reactor.wasm
</code></pre>
<p>The build flag signals to the linker not to generate the <code>_start</code> function
(the entry point for a command module), and instead generate an
<code>_initialize</code> function, which performs runtime and package initialization,
along with any exported functions and their dependencies.
The <code>_initialize</code> function must be called before any other exported functions.
The <code>main</code> function will not be automatically invoked.</p>
<p>To use a WASI reactor, the host application first initializes it by calling
<code>_initialize</code>, then simply invoke the exported functions.
Here is an example using <a href="https://wazero.io/" rel="noreferrer" target="_blank">Wazero</a>, a Go-based Wasm runtime
implementation:</p>
<pre><code>// Create a Wasm runtime, set up WASI.
r := wazero.NewRuntime(ctx)
defer r.Close(ctx)
wasi_snapshot_preview1.MustInstantiate(ctx, r)

// Configure the module to initialize the reactor.
config := wazero.NewModuleConfig().WithStartFunctions(&quot;_initialize&quot;)

// Instantiate the module.
wasmModule, _ := r.InstantiateWithConfig(ctx, wasmFile, config)

// Call the exported function.
fn := wasmModule.ExportedFunction(&quot;add&quot;)
var a, b int32 = 1, 2
res, _ := fn.Call(ctx, api.EncodeI32(a), api.EncodeI32(b))
c := api.DecodeI32(res[0])
fmt.Printf(&quot;add(%d, %d) = %d\n&quot;, a, b, c)

// The instance is still alive. We can call the function again.
res, _ = fn.Call(ctx, api.EncodeI32(b), api.EncodeI32(c))
fmt.Printf(&quot;add(%d, %d) = %d\n&quot;, b, c, api.DecodeI32(res[0]))
</code></pre>
<p>The <code>go:wasmexport</code> directive and the reactor build mode allow applications to
be extended by calling into Go-based Wasm code.
This is particularly valuable for applications that have adopted Wasm as a
plugin or extension mechanism with well-defined interfaces.
By exporting Go functions, applications can leverage the Go Wasm modules to
provide functionality without needing to recompile the entire application.
Furthermore, building as a reactor ensures that the exported functions can be
called multiple times without requiring reinitialization, making it suitable
for long-running applications or services.</p>
<h2 id="supporting-rich-types-between-the-host-and-the-client">Supporting rich types between the host and the client</h2>
<p>Go 1.24 also relaxes the constraints on types that can be used as input and
result parameters with <code>go:wasmimport</code> functions.
For example, one can pass a bool, a string, a pointer to an <code>int32</code>, or a
pointer to a struct which embeds <code>structs.HostLayout</code> and contains supported
field types
(see the <a href="/cmd/compile#hdr-WebAssembly_Directives">documentation</a> for detail).
This allows Go Wasm applications to be written in a more natural and ergonomic
way, and removes some unnecessary type conversions.</p>
<h2 id="limitations">Limitations</h2>
<p>While Go 1.24 has made significant enhancements to its Wasm capabilities,
there are still some notable limitations.</p>
<p>Wasm is a single-threaded architecture with no parallelism.
A <code>go:wasmexport</code> function can spawn new goroutines.
But if a function creates a background goroutine, it will not continue
executing when the <code>go:wasmexport</code> function returns, until calling back into
the Go-based Wasm module.</p>
<p>While some type restrictions have been relaxed in Go 1.24, there are still
limitations on the types that can be used with <code>go:wasmimport</code> and
<code>go:wasmexport</code> functions.
Due to the unfortunate mismatch between the 64-bit architecture of the client
and the 32-bit architecture of the host, it is not possible to pass pointers in
memory.
For example, a <code>go:wasmimport</code> function cannot take a pointer to a struct that
contains a pointer-typed field.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The addition of the ability to build a WASI reactor and export Go functions to
Wasm in Go 1.24 represent a significant step forward for Go&rsquo;s WebAssembly
capabilities.
These features empower developers to create more versatile and powerful Go-based
Wasm applications, opening up new possibilities for Go in the Wasm ecosystem.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/synctest">Testing concurrent code with testing/synctest</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/go1.24">Go 1.24 is released!</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry></feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom

Home · About · News · Docs · Terms