FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 16841: content should not contain data-slug attribute (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

line 1, column 16841: content should not contain script tag (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

line 1, column 128636: content should not contain style tag (2 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

Source: http://blog.golang.org/feed.atom

<feed xmlns="http://www.w3.org/2005/Atom"><title>The Go Blog</title><id>tag:blog.golang.org,2013:blog.golang.org</id><link rel="self" href="https://go.dev/blog/feed.atom"></link><updated>2025-04-02T00:00:00+00:00</updated><entry><title>More predictable benchmarking with testing.B.Loop</title><id>tag:blog.golang.org,2013:blog.golang.org/testing-b-loop</id><link rel="alternate" href="https://go.dev/blog/testing-b-loop"></link><published>2025-04-02T00:00:00+00:00</published><updated>2025-04-02T00:00:00+00:00</updated><author><name>Junyang Shao</name></author><summary type="html">Better benchmark looping in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/testing-b-loop">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>More predictable benchmarking with testing.B.Loop</h1>
 
 <p class="author">
 Junyang Shao<br>
 2 April 2025
 </p>
 
 <p>Go developers who have written benchmarks using the
<a href="https://pkg.go.dev/testing" rel="noreferrer" target="_blank"><code>testing</code></a> package might have encountered some of
its various pitfalls. Go 1.24 introduces a new way to write benchmarks that&rsquo;s just
as easy to use, but at the same time far more robust:
<a href="https://pkg.go.dev/testing#B.Loop" rel="noreferrer" target="_blank"><code>testing.B.Loop</code></a>.</p>
<p>Traditionally, Go benchmarks are written using a loop from 0 to <code>b.N</code>:</p>
<pre><code>func Benchmark(b *testing.B) {
 for range b.N {
 ... code to measure ...
 }
}
</code></pre>
<p>Using <code>b.Loop</code> instead is a trivial change:</p>
<pre><code>func Benchmark(b *testing.B) {
 for b.Loop() {
 ... code to measure ...
 }
}
</code></pre>
<p><code>testing.B.Loop</code> has many benefits:</p>
<ul>
<li>It prevents unwanted compiler optimizations within the benchmark loop.</li>
<li>It automatically excludes setup and cleanup code from benchmark timing.</li>
<li>Code can&rsquo;t accidentally depend on the total number of iterations or the current
iteration.</li>
</ul>
<p>These were all easy mistakes to make with <code>b.N</code>-style benchmarks that would
silently result in bogus benchmark results. As an added bonus, <code>b.Loop</code>-style
benchmarks even complete in less time!</p>
<p>Let&rsquo;s explore the advantages of <code>testing.B.Loop</code> and how to effectively utilize it.</p>
<h2 id="old-benchmark-loop-problems">Old benchmark loop problems</h2>
<p>Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated
benchmarks required more care:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 b.ResetTimer() // if setup may be expensive
 for range b.N {
 ... code to measure ...
 ... use sinks or accumulation to prevent dead-code elimination ...
 }
 b.StopTimer() // if cleanup or reporting may be expensive
 ... cleanup ...
 ... report ...
}
</code></pre>
<p>If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop
with <code>ResetTimer</code> and/or <code>StopTimer</code> calls. These are easy to forget, and even if the
developer remembers they may be necessary, it can be difficult to judge whether setup or
cleanup are &ldquo;expensive enough&rdquo; to require them.</p>
<p>Without these, the <code>testing</code> package can only time the entire benchmark function. If a
benchmark function omits them, the setup and cleanup code will be included in the overall
time measurement, silently skewing the final benchmark result.</p>
<p>There is another, more subtle pitfall that requires deeper understanding:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func isCond(b byte) bool {
 if b%3 == 1 &amp;&amp; b%7 == 2 &amp;&amp; b%17 == 11 &amp;&amp; b%31 == 9 {
 return true
 }
 return false
}

func BenchmarkIsCondWrong(b *testing.B) {
 for range b.N {
 isCond(201)
 }
}
</code></pre>
<p>In this example, the user might observe <code>isCond</code> executing in sub-nanosecond
time. CPUs are fast, but not that fast! This seemingly anomalous result stems
from the fact that <code>isCond</code> is inlined, and since its result is never used, the
compiler eliminates it as dead code. As a result, this benchmark doesn&rsquo;t measure <code>isCond</code>
at all; it measures how long it takes to do nothing. In this case, the sub-nanosecond
result is a clear red flag, but in more complex benchmarks, partial dead-code elimination
can lead to results that look reasonable but still aren&rsquo;t measuring what was intended.</p>
<h2 id="how-testingbloop-helps">How <code>testing.B.Loop</code> helps</h2>
<p>Unlike a <code>b.N</code>-style benchmark, <code>testing.B.Loop</code> is able to track when it is first called
in a benchmark when the final iteration ends. The <code>b.ResetTimer</code> at the loop&rsquo;s start
and <code>b.StopTimer</code> at its end are integrated into <code>testing.B.Loop</code>, eliminating the need
to manually manage the benchmark timer for setup and cleanup code.</p>
<p>Furthermore, the Go compiler now detects loops where the condition is just a call to
<code>testing.B.Loop</code> and prevents dead code elimination within the loop. In Go 1.24, this is
implemented by disallowing inlining into the body of such a loop, but we plan to
<a href="/issue/73137">improve</a> this in the future.</p>
<p>Another nice feature of <code>testing.B.Loop</code> is its one-shot ramp-up approach. With a <code>b.N</code>-style
benchmark, the testing package must call the benchmark function several times with different
values of <code>b.N</code>, ramping up until the measured time reached a threshold. In contrast, <code>b.Loop</code>
can simply run the benchmark loop until it reaches the time threshold, and only needs to call
the benchmark function once. Internally, <code>b.Loop</code> still uses a ramp-up process to amortize
measurement overhead, but this is hidden from the caller and can be more efficient.</p>
<p>Certain constraints of the <code>b.N</code>-style loop still apply to the <code>b.Loop</code>-style
loop. It remains the user&rsquo;s responsibility to manage the timer within the benchmark loop,
when necessary:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func BenchmarkSortInts(b *testing.B) {
 ints := make([]int, N)
 for b.Loop() {
 b.StopTimer()
 fillRandomInts(ints)
 b.StartTimer()
 slices.Sort(ints)
 }
}
</code></pre>
<p>In this example, to benchmark the in-place sorting performance of <code>slices.Sort</code>,a
randomly initialized array is required for each iteration. The user must still
manually manage the timer in such cases.</p>
<p>Furthermore, there still needs to be exactly one such loop in the benchmark function body
(a <code>b.N</code>-style loop cannot coexist with a <code>b.Loop</code>-style loop), and every iteration of the
loop should do the same thing.</p>
<h2 id="when-to-use">When to use</h2>
<p>The <code>testing.B.Loop</code> method is now the preferred way to write benchmarks:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 for b.Loop() {
 // optional timer control for in-loop setup/cleanup
 ... code to measure ...
 }
 ... cleanup ...
}
</code></pre>
<p><code>testing.B.Loop</code> offers faster, more accurate, and
more intuitive benchmarking.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>A huge thank you to everyone in the community who provided feedback on the proposal
issue and reported bugs as this feature was released! I&rsquo;m also grateful to Eli
Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements,
Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and
documentation improvements. Thank you all for your contributions!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 <p>
 
 
 
 <b>Previous article: </b><a href="/blog/coretypes">Goodbye core types - Hello Go as we know and love it!</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Goodbye core types - Hello Go as we know and love it!</title><id>tag:blog.golang.org,2013:blog.golang.org/coretypes</id><link rel="alternate" href="https://go.dev/blog/coretypes"></link><published>2025-03-26T00:00:00+00:00</published><updated>2025-03-26T00:00:00+00:00</updated><author><name>Robert Griesemer</name></author><summary type="html">Go 1.25 simplifies the language spec by removing the notion of core types</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/coretypes">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Goodbye core types - Hello Go as we know and love it!</h1>
 
 <p class="author">
 Robert Griesemer<br>
 26 March 2025
 </p>
 
 <p>The Go 1.18 release introduced generics and with that a number of new features, including type parameters, type constraints, and new concepts such as type sets.
It also introduced the notion of a <em>core type</em>.
While the former provide concrete new functionality, a core type is an abstract construct that was introduced
for expediency and to simplify dealing with generic operands (operands whose types are type parameters).
In the Go compiler, code that in the past relied on the <a href="/ref/spec/#Underlying_types">underlying type</a> of an operand,
now instead had to call a function computing the operand&rsquo;s core type.
In the language spec, in many places we just needed to replace &ldquo;underlying type&rdquo; with &ldquo;core type&rdquo;.
What&rsquo;s not to like?</p>
<p>Quite a few things, as it turns out!
To understand how we got here, it&rsquo;s useful to briefly revisit how type parameters and type constraints work.</p>
<h2 id="type-parameters-and-type-constraints">Type parameters and type constraints</h2>
<p>A type parameter is a placeholder for a future type argument;
it acts like a <em>type variable</em> whose value is known at compile time,
similar to how a named constant stands for a number, string, or bool whose value is known at compile time.
Like ordinary variables, type parameters have a type.
That type is described by their <em>type constraint</em> which determines
what operations are permitted on operands whose type is the respective type parameter.</p>
<p>Any concrete type that instantiates a type parameter must satisfy the type parameter&rsquo;s constraint.
This ensures that an operand whose type is a type parameter possesses all of the respective type constraint&rsquo;s properties,
no matter what concrete type is used to instantiate the type parameter.</p>
<p>In Go, type constraints are described through a mixture of method and type requirements which together
define a <em>type set</em>: this is the set of all the types that satisfy all the requirements. Go uses a
generalized form of interfaces for this purpose. An interface enumerates a set of methods and types,
and the type set described by such an interface consists of all the types that implement those methods
and that are included in the enumerated types.</p>
<p>For instance, the type set described by the interface</p>
<pre><code class="language-Go">type Constraint interface {
 ~[]byte | ~string
 Hash() uint64
}
</code></pre>
<p>consists of all the types whose representation is <code>[]byte</code> or <code>string</code> and whose method set includes the <code>Hash</code> method.</p>
<p>With this we can now write down the rules that govern operations on generic operands.
For instance, the <a href="/ref/spec#Index_expressions">rules for index expressions</a> state that (among other things)
for an operand <code>a</code> of type parameter type <code>P</code>:</p>
<blockquote>
<p>The index expression <code>a[x]</code> must be valid for values of all types in <code>P</code>&rsquo;s type set.
The element types of all types in <code>P</code>&rsquo;s type set must be identical.
(In this context, the element type of a string type is <code>byte</code>.)</p>
</blockquote>
<p>These rules make it possible to index the generic variable <code>s</code> below (<a href="/play/p/M1LYKm3x3IB">playground</a>):</p>
<pre><code class="language-Go">func at[bytestring Constraint](s bytestring, i int) byte {
 return s[i]
}
</code></pre>
<p>The indexing operation <code>s[i]</code> is permitted because the type of <code>s</code> is <code>bytestring</code>, and the type constraint (type set) of
<code>bytestring</code> contains <code>[]byte</code> and <code>string</code> types for which indexing with <code>i</code> is valid.</p>
<h2 id="core-types">Core types</h2>
<p>This type set-based approach is very flexible and in line with the intentions of the
<a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md" rel="noreferrer" target="_blank">original generics proposal</a>:
an operation involving operands of generic type should be valid if it is valid for any type permitted by the respective
type constraint.
To simplify matters with respect to the implementation, knowing that we would be able to relax rules later,
this approach was <em>not</em> chosen universally.
Instead, for instance, for <a href="/ref/spec#Send_statements">Send statements</a>, the spec states that</p>
<blockquote>
<p>The channel expression&rsquo;s <em>core type</em> must be a channel, the channel direction must permit send operations,
and the type of the value to be sent must be assignable to the channel&rsquo;s element type.</p>
</blockquote>
<p>These rules are based on the notion of a core type which is defined roughly as follows:</p>
<ul>
<li>If a type is not a type parameter, its core type is just its <a href="/ref/spec#Underlying_types">underlying type</a>.</li>
<li>If the type is a type parameter, the core type is the single underlying type of all the types in the type parameter&rsquo;s type set.
If the type set has <em>different</em> underlying types, the core type doesn&rsquo;t exist.</li>
</ul>
<p>For instance, <code>interface{ ~[]int }</code> has a core type (<code>[]int</code>), but the <code>Constraint</code> interface above does not have a core type.
To make things more complicated, when it comes to channel operations and certain built-in calls (<code>append</code>, <code>copy</code>) the above definition
of core types is too restrictive.
The actual rules have adjustments that allow for differing channel directions and type sets containing both <code>[]byte</code> and <code>string</code> types.</p>
<p>There are various problems with this approach:</p>
<ul>
<li>
<p>Because the definition of core type must lead to sound type rules for different language features,
it is overly restrictive for specific operations.
For instance, the Go 1.24 rules for <a href="/ref/spec#Slice_expressions">slice expressions</a> do rely on core types,
and as a consequence slicing an operand of type <code>S</code> constrained by <code>Constraint</code> is not permitted, even though
it could be valid.</p>
</li>
<li>
<p>When trying to understand a specific language feature, one may have to learn the intricacies of
core types even when considering non-generic code.
Again, for slice expressions, the language spec talks about the core type of the sliced operand,
rather than just stating that the operand must be an array, slice, or string.
The latter is more direct, simpler, and clearer, and doesn&rsquo;t require knowing another concept that may be
irrelevant in the concrete case.</p>
</li>
<li>
<p>Because the notion of core types exists, the rules for index expressions, and <code>len</code> and <code>cap</code> (and others),
which all eschew core types, appear as exceptions in the language rather than the norm.
In turn, core types cause proposals such as <a href="/issue/48522">issue #48522</a> which would permit a selector
<code>x.f</code> to access a field <code>f</code> shared by all elements of <code>x</code>&rsquo;s type set, to appear to add more exceptions to the
language.
Without core types, that feature becomes a natural and useful consequence of the ordinary rules for non-generic
field access.</p>
</li>
</ul>
<h2 id="go-125">Go 1.25</h2>
<p>For the upcoming Go 1.25 release (August 2025) we decided to remove the notion of core types from the
language spec in favor of explicit (and equivalent!) prose where needed.
This has multiple benefits:</p>
<ul>
<li>The Go spec presents fewer concepts, making it easier to learn the language.</li>
<li>The behavior of non-generic code can be understood without reference to generics concepts.</li>
<li>The individualized approach (specific rules for specific operations) opens the door for more flexible rules.
We already mentioned <a href="/issue/48522">issue #48522</a>, but there are also ideas for more powerful
slice operations, and <a href="/issue/69153">improved type inference</a>.</li>
</ul>
<p>The respective <a href="/issue/70128">proposal issue #70128</a> was recently approved and the relevant changes
are already implemented.
Concretely this means that a lot of prose in the language spec was reverted to its original,
pre-generics form, and new paragraphs were added where needed to explain the rules as they
pertain to generic operands. Importantly, no behavior was changed.
The entire section on core types was removed.
The compiler&rsquo;s error messages were updated to not mention &ldquo;core type&rdquo; anymore, and in many
cases error messages are now more specific by pointing out exactly which type in a type set
is causing a problem.</p>
<p>Here is a sample of the changes made. For the built-in function <code>close</code>,
starting with Go 1.18 the spec began as follows:</p>
<blockquote>
<p>For an argument <code>ch</code> with core type that is a channel,
the built-in function <code>close</code> records that no more values will be sent on the channel.</p>
</blockquote>
<p>A reader who simply wanted to know how <code>close</code> works, had to first learn about core types.
Starting with Go 1.25, this section will again begin the same way it began before Go 1.18:</p>
<blockquote>
<p>For a channel <code>ch</code>, the built-in function <code>close(ch)</code>
records that no more values will be sent on the channel.</p>
</blockquote>
<p>This is shorter and easier to understand.
Only when the reader is dealing with a generic operand will they have to contemplate
the newly added paragraph:</p>
<blockquote>
<p>If the type of the argument to <code>close</code> is a type parameter
all types in its type set must be channels with the same element type.
It is an error if any of those channels is a receive-only channel.</p>
</blockquote>
<p>We made similar changes to each place that mentioned core types.
In summary, although this spec update does not affect any current Go program, it opens the
door to future language improvements while making the language as it is today easier to
learn and its spec simpler.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/testing-b-loop">More predictable benchmarking with testing.B.Loop</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/osroot">Traversal-resistant file APIs</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Traversal-resistant file APIs</title><id>tag:blog.golang.org,2013:blog.golang.org/osroot</id><link rel="alternate" href="https://go.dev/blog/osroot"></link><published>2025-03-12T00:00:00+00:00</published><updated>2025-03-12T00:00:00+00:00</updated><author><name>Damien Neil</name></author><summary type="html">New file access APIs in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/osroot">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Traversal-resistant file APIs</h1>
 
 <p class="author">
 Damien Neil<br>
 12 March 2025
 </p>
 
 <p>A <em>path traversal vulnerability</em> arises when an attacker can trick a program
into opening a file other than the one it intended.
This post explains this class of vulnerability,
some existing defenses against it, and describes how the new
<a href="/pkg/os#Root"><code>os.Root</code></a> API added in Go 1.24 provides
a simple and robust defense against unintentional path traversal.</p>
<h2 id="path-traversal-attacks">Path traversal attacks</h2>
<p>&ldquo;Path traversal&rdquo; covers a number of related attacks following a common pattern:
A program attempts to open a file in some known location, but an attacker causes
it to open a file in a different location.</p>
<p>If the attacker controls part of the filename, they may be able to use relative
directory components (&quot;..&quot;) to escape the intended location:</p>
<pre><code>f, err := os.Open(filepath.Join(trustedLocation, &quot;../../../../etc/passwd&quot;))
</code></pre>
<p>On Windows systems, some names have special meaning:</p>
<pre><code>// f will print to the console.
f, err := os.Create(filepath.Join(trustedLocation, &quot;CONOUT$&quot;))
</code></pre>
<p>If the attacker controls part of the local filesystem, they may be able to use
symbolic links to cause a program to access the wrong file:</p>
<pre><code>// Attacker links /home/user/.config to /home/otheruser/.config:
err := os.WriteFile(&quot;/home/user/.config/foo&quot;, config, 0o666)
</code></pre>
<p>If the program defends against symlink traversal by first verifying that the intended file
does not contain any symlinks, it may still be vulnerable to
<a href="https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use" rel="noreferrer" target="_blank">time-of-check/time-of-use (TOCTOU) races</a>,
where the attacker creates a symlink after the program&rsquo;s check:</p>
<pre><code>// Validate the path before use.
cleaned, err := filepath.EvalSymlinks(unsafePath)
if err != nil {
 return err
}
if !filepath.IsLocal(cleaned) {
 return errors.New(&quot;unsafe path&quot;)
}

// Attacker replaces part of the path with a symlink.
// The Open call follows the symlink:
f, err := os.Open(cleaned)
</code></pre>
<p>Another variety of TOCTOU race involves moving a directory that forms part of a path
mid-traversal. For example, the attacker provides a path such as &ldquo;a/b/c/../../etc/passwd&rdquo;,
and renames &ldquo;a/b/c&rdquo; to &ldquo;a/b&rdquo; while the open operation is in progress.</p>
<h2 id="path-sanitization">Path sanitization</h2>
<p>Before we tackle path traversal attacks in general, let&rsquo;s start with path sanitization.
When a program&rsquo;s threat model does not include attackers with access to the local file system,
it can be sufficient to validate untrusted input paths before use.</p>
<p>Unfortunately, sanitizing paths can be surprisingly tricky,
especially for portable programs that must handle both Unix and Windows paths.
For example, on Windows <code>filepath.IsAbs(`\foo`)</code> reports <code>false</code>,
because the path &ldquo;\foo&rdquo; is relative to the current drive.</p>
<p>In Go 1.20, we added the <a href="/pkg/path/filepath#IsLocal"><code>path/filepath.IsLocal</code></a>
function, which reports whether a path is &ldquo;local&rdquo;. A &ldquo;local&rdquo; path is one which:</p>
<ul>
<li>does not escape the directory in which it is evaluated (&quot;../etc/passwd&quot; is not allowed);</li>
<li>is not an absolute path (&quot;/etc/passwd&quot; is not allowed);</li>
<li>is not empty (&quot;&quot; is not allowed);</li>
<li>on Windows, is not a reserved name (&ldquo;COM1&rdquo; is not allowed).</li>
</ul>
<p>In Go 1.23, we added the <a href="/pkg/path/filepath#Localize"><code>path/filepath.Localize</code></a>
function, which converts a /-separated path into a local operating system path.</p>
<p>Programs that accept and operate on potentially attacker-controlled paths should almost
always use <code>filepath.IsLocal</code> or <code>filepath.Localize</code> to validate or sanitize those paths.</p>
<h2 id="beyond-sanitization">Beyond sanitization</h2>
<p>Path sanitization is not sufficient when attackers may have access to part of
the local filesystem.</p>
<p>Multi-user systems are uncommon these days, but attacker access to the filesystem
can still occur in a variety of ways.
An unarchiving utility that extracts a tar or zip file may be induced
to extract a symbolic link and then extract a file name that traverses that link.
A container runtime may give untrusted code access to a portion of the local filesystem.</p>
<p>Programs may defend against unintended symlink traversal by using the
<a href="/pkg/path/filepath#EvalSymlinks"><code>path/filepath.EvalSymlinks</code></a>
function to resolve links in untrusted names before validation, but as described
above this two-step process is vulnerable to TOCTOU races.</p>
<p>Before Go 1.24, the safer option was to use a package such as
<a href="/pkg/github.com/google/safeopen">github.com/google/safeopen</a>,
that provides path traversal-resistant functions for opening a potentially-untrusted
filename within a specific directory.</p>
<h2 id="introducing-osroot">Introducing <code>os.Root</code></h2>
<p>In Go 1.24, we are introducing new APIs in the <code>os</code> package to safely open
a file in a location in a traversal-resistant fashion.</p>
<p>The new <a href="/pkg/os#Root"><code>os.Root</code></a> type represents a directory somewhere
in the local filesystem. Open a root with the <a href="/pkg/os#OpenRoot"><code>os.OpenRoot</code></a>
function:</p>
<pre><code>root, err := os.OpenRoot(&quot;/some/root/directory&quot;)
if err != nil {
 return err
}
defer root.Close()
</code></pre>
<p><code>Root</code> provides methods to operate on files within the root.
These methods all accept filenames relative to the root,
and disallow any operations that would escape from the root either
using relative path components (&quot;..&quot;) or symlinks.</p>
<pre><code>f, err := root.Open(&quot;path/to/file&quot;)
</code></pre>
<p><code>Root</code> permits relative path components and symlinks that do not escape the root.
For example, <code>root.Open(&quot;a/../b&quot;)</code> is permitted. Filenames are resolved using the
semantics of the local platform: On Unix systems, this will follow
any symlink in &ldquo;a&rdquo; (so long as that link does not escape the root);
while on Windows systems this will open &ldquo;b&rdquo; (even if &ldquo;a&rdquo; does not exist).</p>
<p><code>Root</code> currently provides the following set of operations:</p>
<pre><code>func (*Root) Create(string) (*File, error)
func (*Root) Lstat(string) (fs.FileInfo, error)
func (*Root) Mkdir(string, fs.FileMode) error
func (*Root) Open(string) (*File, error)
func (*Root) OpenFile(string, int, fs.FileMode) (*File, error)
func (*Root) OpenRoot(string) (*Root, error)
func (*Root) Remove(string) error
func (*Root) Stat(string) (fs.FileInfo, error)
</code></pre>
<p>In addition to the <code>Root</code> type, the new
<a href="/pkg/os#OpenInRoot"><code>os.OpenInRoot</code></a> function
provides a simple way to open a potentially-untrusted filename within a
specific directory:</p>
<pre><code>f, err := os.OpenInRoot(&quot;/some/root/directory&quot;, untrustedFilename)
</code></pre>
<p>The <code>Root</code> type provides a simple, safe, portable API for operating with untrusted filenames.</p>
<h2 id="caveats-and-considerations">Caveats and considerations</h2>
<h3 id="unix">Unix</h3>
<p>On Unix systems, <code>Root</code> is implemented using the <code>openat</code> family of system calls.
A <code>Root</code> contains a file descriptor referencing its root directory and will track that
directory across renames or deletion.</p>
<p><code>Root</code> defends against symlink traversal but does not limit traversal
of mount points. For example, <code>Root</code> does not prevent traversal of
Linux bind mounts. Our threat model is that <code>Root</code> defends against
filesystem constructs that may be created by ordinary users (such
as symlinks), but does not handle ones that require root privileges
to create (such as bind mounts).</p>
<h3 id="windows">Windows</h3>
<p>On Windows, <code>Root</code> opens a handle referencing its root directory.
The open handle prevents that directory from being renamed or deleted until the <code>Root</code> is closed.</p>
<p><code>Root</code> prevents access to reserved Windows device names such as <code>NUL</code> and <code>COM1</code>.</p>
<h3 id="wasi">WASI</h3>
<p>On WASI, the <code>os</code> package uses the WASI preview 1 filesystem API,
which are intended to provide traversal-resistant filesystem access.
Not all WASI implementations fully support filesystem sandboxing,
however, and <code>Root</code>&rsquo;s defense against traversal is limited to that provided
by the WASI implementation.</p>
<h3 id="goosjs">GOOS=js</h3>
<p>When GOOS=js, the <code>os</code> package uses the Node.js file system API.
This API does not include the openat family of functions,
and so <code>os.Root</code> is vulnerable to TOCTOU (time-of-check-time-of-use) races in symlink
validation on this platform.</p>
<p>When GOOS=js, a <code>Root</code> references a directory name rather than a file descriptor,
and does not track directories across renames.</p>
<h3 id="plan-9">Plan 9</h3>
<p>Plan 9 does not have symlinks.
On Plan 9, a <code>Root</code> references a directory name and performs lexical sanitization of
filenames.</p>
<h3 id="performance">Performance</h3>
<p><code>Root</code> operations on filenames containing many directory components can be much more expensive
than the equivalent non-<code>Root</code> operation. Resolving &ldquo;..&rdquo; components can also be expensive.
Programs that want to limit the cost of filesystem operations can use <code>filepath.Clean</code> to
remove &ldquo;..&rdquo; components from input filenames, and may want to limit the number of
directory components.</p>
<h2 id="who-should-use-osroot">Who should use os.Root?</h2>
<p>You should use <code>os.Root</code> or <code>os.OpenInRoot</code> if:</p>
<ul>
<li>you are opening a file in a directory; AND</li>
<li>the operation should not access a file outside that directory.</li>
</ul>
<p>For example, an archive extractor writing files to an output directory should use
<code>os.Root</code>, because the filenames are potentially untrusted and it would be incorrect
to write a file outside the output directory.</p>
<p>However, a command-line program that writes output to a user-specified location
should not use <code>os.Root</code>, because the filename is not untrusted and may
refer to anywhere on the filesystem.</p>
<p>As a good rule of thumb, code which calls <code>filepath.Join</code> to combine a fixed directory
and an externally-provided filename should probably use <code>os.Root</code> instead.</p>
<pre><code>// This might open a file not located in baseDirectory.
f, err := os.Open(filepath.Join(baseDirectory, filename))

// This will only open files under baseDirectory.
f, err := os.OpenInRoot(baseDirectory, filename)
</code></pre>
<h2 id="future-work">Future work</h2>
<p>The <code>os.Root</code> API is new in Go 1.24.
We expect to make additions and refinements to it in future releases.</p>
<p>The current implementation prioritizes correctness and safety over performance.
Future versions will take advantage of platform-specific APIs, such as
Linux&rsquo;s <code>openat2</code>, to improve performance where possible.</p>
<p>There are a number of filesystem operations which <code>Root</code> does not support yet, such as
creating symbolic links and renaming files. Where possible, we will add support for these
operations. A list of additional functions in progress is in
<a href="/issue/67002">go.dev/issue/67002</a>.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/coretypes">Goodbye core types - Hello Go as we know and love it!</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/cleanups-and-weak">From unique to cleanups and weak: new low-level tools for efficiency</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>From unique to cleanups and weak: new low-level tools for efficiency</title><id>tag:blog.golang.org,2013:blog.golang.org/cleanups-and-weak</id><link rel="alternate" href="https://go.dev/blog/cleanups-and-weak"></link><published>2025-03-06T00:00:00+00:00</published><updated>2025-03-06T00:00:00+00:00</updated><author><name>Michael Knyszek</name></author><summary type="html">Weak pointers and better finalization in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/cleanups-and-weak">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>From unique to cleanups and weak: new low-level tools for efficiency</h1>
 
 <p class="author">
 Michael Knyszek<br>
 6 March 2025
 </p>
 
 <p>In <a href="/blog/unique">last year&rsquo;s blog post</a> about the <code>unique</code> package, we alluded
to some new features then in proposal review, and we&rsquo;re excited to share that as
of Go 1.24 they are now available to all Go developers.
These new features are <a href="https://pkg.go.dev/runtime#AddCleanup" rel="noreferrer" target="_blank">the <code>runtime.AddCleanup</code>
function</a>, which queues up a function to
run when an object is no longer reachable, and <a href="https://pkg.go.dev/weak#Pointer" rel="noreferrer" target="_blank">the <code>weak.Pointer</code>
type</a>, which safely points to an object without
preventing it from being garbage collected.
Together, these two features are powerful enough to build your own <code>unique</code>
package!
Let&rsquo;s dig into what makes these features useful, and when to use them.</p>
<p>Note: these new features are advanced features of the garbage collector.
If you&rsquo;re not already familiar with basic garbage collection concepts, we
strongly recommend reading the introduction of our <a href="/doc/gc-guide#Introduction">garbage collector
guide</a>.</p>
<h2 id="cleanups">Cleanups</h2>
<p>If you&rsquo;ve ever used a finalizer, then the concept of a cleanup will be
familiar.
A finalizer is a function, associated with an allocated object by <a href="https://pkg.go.dev/runtime#SetFinalizer" rel="noreferrer" target="_blank">calling
<code>runtime.SetFinalizer</code></a>, that is later
called by the garbage collector some time after the object becomes unreachable.
At a high level, cleanups work the same way.</p>
<p>Let&rsquo;s consider an application that makes use of a memory-mapped file, and see
how cleanups can help.</p>
<pre><code>//go:build unix

type MemoryMappedFile struct {
 data []byte
}

func NewMemoryMappedFile(filename string) (*MemoryMappedFile, error) {
 f, err := os.Open(filename)
 if err != nil {
 return nil, err
 }
 defer f.Close()

 // Get the file's info; we need its size.
 fi, err := f.Stat()
 if err != nil {
 return nil, err
 }

 // Extract the file descriptor.
 conn, err := f.SyscallConn()
 if err != nil {
 return nil, err
 }
 var data []byte
 connErr := conn.Control(func(fd uintptr) {
 // Create a memory mapping backed by this file.
 data, err = syscall.Mmap(int(fd), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
 })
 if connErr != nil {
 return nil, connErr
 }
 if err != nil {
 return nil, err
 }
 mf := &amp;MemoryMappedFile{data: data}
 cleanup := func(data []byte) {
 syscall.Munmap(data) // ignore error
 }
 runtime.AddCleanup(mf, cleanup, data)
 return mf, nil
}
</code></pre>
<p>A memory-mapped file has its contents mapped to memory, in this case, the
underlying data of a byte slice.
Thanks to operating-system magic, reads and writes to the byte slice directly
access the contents of the file.
With this code, we can pass around a <code>*MemoryMappedFile</code>, and when it&rsquo;s
no longer referenced, the memory mapping we created will get cleaned up.</p>
<p>Notice that <code>runtime.AddCleanup</code> takes three arguments: the address of a
variable to attach the cleanup to, the cleanup function itself, and an argument
to the cleanup function.
A key difference between this function and <code>runtime.SetFinalizer</code> is that the
cleanup function takes a different argument than the object we&rsquo;re attaching the
cleanup to.
This change fixes some problems with finalizers.</p>
<p>It&rsquo;s no secret that <a href="/doc/gc-guide#Common_finalizer_issues">finalizers
are difficult to use correctly</a>.
For example, objects to which finalizers are attached must not be involved
in any reference cycles (even a pointer to itself is too much!), otherwise the
object will never be reclaimed and the finalizer will never run, causing a
leak.
Finalizers also significantly delay reclamation of memory.
It takes at a minimum two full garbage collection cycles to reclaim the
memory for a finalized object: one to determine that it&rsquo;s unreachable, and
another to determine that it&rsquo;s still unreachable after the finalizer
executes.</p>
<p>The problem is that finalizers <a href="https://en.wikipedia.org/wiki/Object_resurrection" rel="noreferrer" target="_blank">resurrect the objects they&rsquo;re attached
to</a>.
The finalizer doesn&rsquo;t run until the object is unreachable, at which point it is
considered &ldquo;dead.&rdquo;
But since the finalizer is called with a pointer to the object, the garbage
collector must prevent the reclamation of that object&rsquo;s memory, and instead must
generate a new reference for the finalizer, making it reachable, or &ldquo;live,&rdquo; once
more.
That reference may even remain after the finalizer returns, for example if the
finalizer writes it to a global variable or sends it across a channel.
Object resurrection is problematic because it means the object, and everything
it points to, and everything those objects point to, and so on, is reachable,
even if it would otherwise have been collected as garbage.</p>
<p>We solve both of these problems by not passing the original object to the
cleanup function.
First, the values the object refers to don&rsquo;t need to be kept specially
reachable by the garbage collector, so the object can still be reclaimed even
if it&rsquo;s involved in a cycle.
Second, since the object is not needed for the cleanup, its memory can be
reclaimed immediately.</p>
<h2 id="weak-pointers">Weak pointers</h2>
<p>Returning to our memory-mapped file example, suppose we notice that our program
frequently maps the same files over and over, from different goroutines that are
unaware of each other.
This is fine from a memory perspective, since all these mappings will share
physical memory, but it results in lots of unnecessary system calls to map and
unmap the file.
This is especially bad if each goroutine reads only a small section of each
file.</p>
<p>So, let&rsquo;s deduplicate the mappings by filename.
(Let&rsquo;s assume that our program only reads from the mappings, and the files
themselves are never modified or renamed once created.
Such assumptions are reasonable for system font files, for example.)</p>
<p>We could maintain a map from filename to memory mapping, but then it becomes
unclear when it&rsquo;s safe to remove entries from that map.
We could <em>almost</em> use a cleanup, if it weren&rsquo;t for the fact that the map entry
itself will keep the memory-mapped file object alive.</p>
<p>Weak pointers solve this problem.
A weak pointer is a special kind of pointer that the garbage collector ignores
when deciding whether an object is reachable.
Go 1.24&rsquo;s <a href="https://pkg.go.dev/weak#Pointer" rel="noreferrer" target="_blank">new weak pointer type,
<code>weak.Pointer</code></a>, has a <code>Value</code> method that
returns either a real pointer if the object is still reachable, or <code>nil</code> if it
is not.</p>
<p>If we instead maintain a map that only <em>weakly</em> points to the memory-mapped
file, we can clean up the map entry when nobody&rsquo;s using it anymore!
Let&rsquo;s see what this looks like.</p>
<pre><code>var cache sync.Map // map[string]weak.Pointer[MemoryMappedFile]

func NewCachedMemoryMappedFile(filename string) (*MemoryMappedFile, error) {
 var newFile *MemoryMappedFile
 for {
 // Try to load an existing value out of the cache.
 value, ok := cache.Load(filename)
 if !ok {
 // No value found. Create a new mapped file if needed.
 if newFile == nil {
 var err error
 newFile, err = NewMemoryMappedFile(filename)
 if err != nil {
 return nil, err
 }
 }

 // Try to install the new mapped file.
 wp := weak.Make(newFile)
 var loaded bool
 value, loaded = cache.LoadOrStore(filename, wp)
 if !loaded {
 runtime.AddCleanup(newFile, func(filename string) {
 // Only delete if the weak pointer is equal. If it's not, someone
 // else already deleted the entry and installed a new mapped file.
 cache.CompareAndDelete(filename, wp)
 }, filename)
 return newFile, nil
 }
 // Someone got to installing the file before us.
 //
 // If it's still there when we check in a moment, we'll discard newFile
 // and it'll get cleaned up by garbage collector.
 }

 // See if our cache entry is valid.
 if mf := value.(weak.Pointer[MemoryMappedFile]).Value(); mf != nil {
 return mf, nil
 }

 // Discovered a nil entry awaiting cleanup. Eagerly delete it.
 cache.CompareAndDelete(filename, value)
 }
}
</code></pre>
<p>This example is a little complicated, but the gist is simple.
We start with a global concurrent map of all the mapped files we made.
<code>NewCachedMemoryMappedFile</code> consults this map for an existing mapped
file, and if that fails, creates and tries to insert a new mapped file.
This could of course fail as well since we&rsquo;re racing with other insertions, so
we need to be careful about that too, and retry.
(This design has a flaw in that we might wastefully map the same file multiple
times in a race, and we&rsquo;ll have to throw it away via the cleanup added by
<code>NewMemoryMappedFile</code>.
This is probably not a big deal most of the time.
Fixing it is left as an exercise for the reader.)</p>
<p>Let&rsquo;s look at some useful properties of weak pointers and cleanups exploited by
this code.</p>
<p>Firstly, notice that weak pointers are comparable.
Not only that, weak pointers have a stable and independent identity, which
remains even after the objects they point to are long gone.
This is why it is safe for the cleanup function to call <code>sync.Map</code>&rsquo;s
<code>CompareAndDelete</code>, which compares the <code>weak.Pointer</code>, and a crucial reason
this code works at all.</p>
<p>Secondly, observe that we can add multiple independent cleanups to a single
<code>MemoryMappedFile</code> object.
This allows us to use cleanups in a composable way and use them to build
generic data structures.
In this particular example, it might be more efficient to combine
<code>NewCachedMemoryMappedFile</code> with <code>NewMemoryMappedFile</code> and
have them share a cleanup.
However, the advantage of the code we wrote above is that it can be rewritten
in a generic way!</p>
<pre><code>type Cache[K comparable, V any] struct {
 create func(K) (*V, error)
 m sync.Map
}

func NewCache[K comparable, V any](create func(K) (*V, error)) *Cache[K, V] {
 return &amp;Cache[K, V]{create: create}
}

func (c *Cache[K, V]) Get(key K) (*V, error) {
 var newValue *V
 for {
 // Try to load an existing value out of the cache.
 value, ok := cache.Load(key)
 if !ok {
 // No value found. Create a new mapped file if needed.
 if newValue == nil {
 var err error
 newValue, err = c.create(key)
 if err != nil {
 return nil, err
 }
 }

 // Try to install the new mapped file.
 wp := weak.Make(newValue)
 var loaded bool
 value, loaded = cache.LoadOrStore(key, wp)
 if !loaded {
 runtime.AddCleanup(newValue, func(key K) {
 // Only delete if the weak pointer is equal. If it's not, someone
 // else already deleted the entry and installed a new mapped file.
 cache.CompareAndDelete(key, wp)
 }, key)
 return newValue, nil
 }
 }

 // See if our cache entry is valid.
 if mf := value.(weak.Pointer[V]).Value(); mf != nil {
 return mf, nil
 }

 // Discovered a nil entry awaiting cleanup. Eagerly delete it.
 cache.CompareAndDelete(key, value)
 }
}
</code></pre>
<h2 id="caveats-and-future-work">Caveats and future work</h2>
<p>Despite our best efforts, cleanups and weak pointers can still be error-prone.
To guide those considering using finalizers, cleanups, and weak pointers, we
recently updated the <a href="/doc/gc-guide#Finalizers_cleanups_and_weak_pointers">guide to the garbage
collector</a> with some advice
about using these features.
Take a look next time you reach for them, but also carefully consider whether
you need to use them at all.
These are advanced tools with subtle semantics and, as the guide says, most
Go code benefits from these features indirectly, not from using them directly.
Stick to the use-cases where these features shine, and you&rsquo;ll be alright.</p>
<p>For now, we&rsquo;ll call out some of the issues that you are more likely to run into.</p>
<p>First, the object the cleanup is attached to must be reachable from neither
the cleanup function (as a captured variable) nor the argument to the cleanup
function.
Both of these situations result in the cleanup never executing.
(In the special case of the cleanup argument being exactly the pointer passed
to <code>runtime.AddCleanup</code>, <code>runtime.AddCleanup</code> will panic, as a signal to the
caller that they should not use cleanups the same way as finalizers.)</p>
<p>Second, when weak pointers are used as map keys, the weakly referenced object
must not be reachable from the corresponding map value, otherwise the object
will continue to remain live.
This may seem obvious when deep inside of a blog post about weak pointers, but
it&rsquo;s an easy subtlety to miss.
This problem inspired the entire concept of an
<a href="https://en.wikipedia.org/wiki/Ephemeron" rel="noreferrer" target="_blank">ephemeron</a> to resolve it, which is a
potential future direction.</p>
<p>Thirdly, a common pattern with cleanups is that a wrapper object is needed, like
we see here with our <code>MemoryMappedFile</code> example.
In this particular case, you could imagine the garbage collector directly
tracking the mapped memory region and passing around the inner <code>[]byte</code>.
Such functionality is possible future work, and an API for it has been recently
<a href="/issue/70224">proposed</a>.</p>
<p>Lastly, both weak pointers and cleanups are inherently non-deterministic, their
behavior depending intimately on the design and dynamics of the garbage
collector.
The documentation for cleanups even permits the garbage collector never to run
cleanups at all.
Effectively testing code that uses them can be tricky, but <a href="/doc/gc-guide#Testing_object_death">it is
possible</a>.</p>
<h2 id="why-now">Why now?</h2>
<p>Weak pointers have been brought up as a feature for Go since nearly the
beginning, but for years were not prioritized by the Go team.
One reason for that is that they are subtle, and the design space of weak
pointers is a minefield of decisions that can make them even harder to use.
Another is that weak pointers are a niche tool, while simultaneously adding
complexity to the language.
We already had experience with how painful <code>SetFinalizer</code> could be to use.
But there are some useful programs that are not expressible without them, and
the <code>unique</code> package and the reasons for its existence really emphasized that.</p>
<p>With generics, the hindsight of finalizers, and insights from all the great work
since done by teams in other languages like C# and Java, the designs for weak
pointers and cleanups came together quickly.
The desire to use weak pointers with finalizers raised additional questions,
and so the design for <code>runtime.AddCleanup</code> quickly came together as well.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>I want to thank everyone in the community who contributed feedback on the
proposal issues and filed bugs when the features became available.
I also want to thank David Chase for thoroughly thinking through weak
pointer semantics with me, and I want to thank him, Russ Cox, and Austin
Clements for their help with the design of <code>runtime.AddCleanup</code>.
I want to thank Carlos Amedee for his work on getting <code>runtime.AddCleanup</code>
implemented, polished, landed for Go 1.24.
And finally I want to thank Carlos Amedee and Ian Lance Taylor for their work
replacing <code>runtime.SetFinalizer</code> with <code>runtime.AddCleanup</code> throughout the
standard library for Go 1.25.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/osroot">Traversal-resistant file APIs</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/swisstable">Faster Go maps with Swiss Tables</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Faster Go maps with Swiss Tables</title><id>tag:blog.golang.org,2013:blog.golang.org/swisstable</id><link rel="alternate" href="https://go.dev/blog/swisstable"></link><published>2025-02-26T00:00:00+00:00</published><updated>2025-02-26T00:00:00+00:00</updated><author><name>Michael Pratt</name></author><summary type="html">Go 1.24 improves map performance with a brand new map implementation</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/swisstable">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Faster Go maps with Swiss Tables</h1>
 
 <p class="author">
 Michael Pratt<br>
 26 February 2025
 </p>
 
 <p>The hash table is a central data structure in computer science, and it provides the implementation for the map type in many languages, including Go.</p>
<p>The concept of a hash table was <a href="https://spectrum.ieee.org/hans-peter-luhn-and-the-birth-of-the-hashing-algorithm" rel="noreferrer" target="_blank">first described</a> by Hans Peter Luhn in 1953 in an internal IBM memo that suggested speeding up search by placing items into &ldquo;buckets&rdquo; and using a linked list for overflow when buckets already contain an item.
Today we would call this a <a href="https://en.wikipedia.org/wiki/Hash_table#Separate_chaining" rel="noreferrer" target="_blank">hash table using chaining</a>.</p>
<p>In 1954, Gene M. Amdahl, Elaine M. McGraw, and Arthur L. Samuel first used an &ldquo;open addressing&rdquo; scheme when programming the IBM 701.
When a bucket already contains an item, the new item is placed in the next empty bucket.
This idea was formalized and published in 1957 by W. Wesley Peterson in <a href="https://ieeexplore.ieee.org/document/5392733" rel="noreferrer" target="_blank">&ldquo;Addressing for Random-Access Storage&rdquo;</a>.
Today we would call this a <a href="https://en.wikipedia.org/wiki/Hash_table#Open_addressing" rel="noreferrer" target="_blank">hash table using open addressing with linear probing</a>.</p>
<p>With data structures that have been around this long, it&rsquo;s easy to think that they must be &ldquo;done&rdquo;; that we know everything there is to know about them and they can&rsquo;t be improved anymore.
That&rsquo;s not true!
Computer science research continues to make advancements in fundamental algorithms, both in terms of algorithmic complexity and taking advantage of modern CPU hardware.
For example, Go 1.19 <a href="/doc/go1.19#sortpkgsort">switched the <code>sort</code> package</a> from a traditional quicksort, to <a href="https://arxiv.org/pdf/2106.05123.pdf" rel="noreferrer" target="_blank">pattern-defeating quicksort</a>, a novel sorting algorithm from Orson R. L. Peters, first described in 2015.</p>
<p>Like sorting algorithms, hash table data structures continue to see improvements.
In 2017, Sam Benzaquen, Alkis Evlogimenos, Matt Kulukundis, and Roman Perepelitsa at Google presented <a href="https://www.youtube.com/watch?v=ncHmEUmJZf4" rel="noreferrer" target="_blank">a new C++ hash table design</a>, dubbed &ldquo;Swiss Tables&rdquo;.
In 2018, their implementation was <a href="https://abseil.io/blog/20180927-swisstables" rel="noreferrer" target="_blank">open sourced in the Abseil C++ library</a>.</p>
<p>Go 1.24 includes a completely new implementation of the built-in map type, based on the Swiss Table design.
In this blog post we&rsquo;ll look at how Swiss Tables improve upon traditional hash tables, and at some of the unique challenges in bringing the Swiss Table design to Go&rsquo;s maps.</p>
<h2 id="open-addressed-hash-table">Open-addressed hash table</h2>
<p>Swiss Tables are a form of open-addressed hash table, so let&rsquo;s do a quick overview of how a basic open-addressed hash table works.</p>
<p>In an open-addressed hash table, all items are stored in a single backing array.
We&rsquo;ll call each location in the array a <em>slot</em>.
The slot to which a key belongs is primarily determined by the <em>hash function</em>, <code>hash(key)</code>.
The hash function maps each key to an integer, where the same key always maps to the same integer, and different keys ideally follow a uniform random distribution of integers.
The defining feature of open-addressed hash tables is that they resolve collisions by storing the key elsewhere in the backing array.
So, if the slot is already full (a <em>collision</em>), then a <em>probe sequence</em> is used to consider other slots until an empty slot is found.
Let&rsquo;s take a look at a sample hash table to see how this works.</p>
<h3 id="example">Example</h3>
<p>Below you can see a 16-slot backing array for a hash table, and the key (if any) stored in each slot.
The values are not shown, as they are not relevant to this example.</p>
<style>
 
@media screen and (max-width: 55em) {
 .swisstable-table-container {
 
 overflow: scroll;
 }
}

.swisstable-table {
 
 border-collapse: collapse;
 
 table-layout: fixed;
 
 margin: 0 auto;
}

.swisstable-table-cell {
 
 border: 1px solid;
 
 padding: 0.5em 1em 0.5em 1em;
 
 text-align: center;
}
</style>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">8</th>
 <th class="swisstable-table-cell">9</th>
 <th class="swisstable-table-cell">10</th>
 <th class="swisstable-table-cell">11</th>
 <th class="swisstable-table-cell">12</th>
 <th class="swisstable-table-cell">13</th>
 <th class="swisstable-table-cell">14</th>
 <th class="swisstable-table-cell">15</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">Key</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">56</td>
 <td class="swisstable-table-cell">32</td>
 <td class="swisstable-table-cell">21</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">78</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
</div>
<p>To insert a new key, we use the hash function to select a slot.
Since there are only 16 slots, we need to restrict to this range, so we&rsquo;ll use <code>hash(key) % 16</code> as the target slot.
Suppose we want to insert key <code>98</code> and <code>hash(98) % 16 = 7</code>.
Slot 7 is empty, so we simply insert 98 there.
On the other hand, suppose we want to insert key <code>25</code> and <code>hash(25) % 16 = 3</code>.
Slot 3 is a collision because it already contains key 56.
Thus we cannot insert here.</p>
<p>We use a probe sequence to find another slot.
There are a variety of well-known probe sequences.
The original and most straightforward probe sequence is <em>linear probing</em>, which simply tries successive slots in order.</p>
<p>So, in the <code>hash(25) % 16 = 3</code> example, since slot 3 is in use, we would consider slot 4 next, which is also in use.
So too is slot 5.
Finally, we&rsquo;d get to empty slot 6, where we&rsquo;d store key 25.</p>
<p>Lookup follows the same approach.
A lookup of key 25 would start at slot 3, check whether it contains key 25 (it does not), and then continue linear probing until it finds key 25 in slot 6.</p>
<p>This example uses a backing array with 16 slots.
What happens if we insert more than 16 elements?
If the hash table runs out of space, it will grow, usually by doubling the size of the backing array.
All existing entries are reinserted into the new backing array.</p>
<p>Open-addressed hash tables don&rsquo;t actually wait until the backing array is completely full to grow because as the array gets more full, the average length of each probe sequence increases.
In the example above using key 25, we must probe 4 different slots to find an empty slot.
If the array had only one empty slot, the worst case probe length would be O(n).
That is, you may need to scan the entire array.
The proportion of used slots is called the <em>load factor</em>, and most hash tables define a <em>maximum load factor</em> (typically 70-90%) at which point they will grow to avoid the extremely long probe sequences of very full hash tables.</p>
<h2 id="swiss-table">Swiss Table</h2>
<p>The Swiss Table <a href="https://abseil.io/about/design/swisstables" rel="noreferrer" target="_blank">design</a> is also a form of open-addressed hash table.
Let&rsquo;s see how it improves over a traditional open-addressed hash table.
We still have a single backing array for storage, but we will break the array into logical <em>groups</em> of 8 slots each.
(Larger group sizes are possible as well. More on that below.)</p>
<p>In addition, each group has a 64-bit <em>control word</em> for metadata.
Each of the 8 bytes in the control word corresponds to one of the slots in the group.
The value of each byte denotes whether that slot is empty, deleted, or in use.
If it is in use, the byte contains the lower 7 bits of the hash for that slot&rsquo;s key (called <code>h2</code>).</p>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell"></th>
 <th class="swisstable-table-cell" colspan="8">Group 0</th>
 <th class="swisstable-table-cell" colspan="8">Group 1</th>
 </tr>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">Key</td>
 <td class="swisstable-table-cell">56</td>
 <td class="swisstable-table-cell">32</td>
 <td class="swisstable-table-cell">21</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">78</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
 <br/> 
 <table class="swisstable-table">
 <thead>
 <tr>
 <th class="swisstable-table-cell"></th>
 <th class="swisstable-table-cell" colspan="8">64-bit control word 0</th>
 <th class="swisstable-table-cell" colspan="8">64-bit control word 1</th>
 </tr>
 <tr>
 <th class="swisstable-table-cell">Slot</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 <th class="swisstable-table-cell">0</th>
 <th class="swisstable-table-cell">1</th>
 <th class="swisstable-table-cell">2</th>
 <th class="swisstable-table-cell">3</th>
 <th class="swisstable-table-cell">4</th>
 <th class="swisstable-table-cell">5</th>
 <th class="swisstable-table-cell">6</th>
 <th class="swisstable-table-cell">7</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td class="swisstable-table-cell">h2</td>
 <td class="swisstable-table-cell">23</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">50</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell">47</td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 <td class="swisstable-table-cell"></td>
 </tr>
 </tbody>
 </table>
</div>
<p>Insertion works as follows:</p>
<ol>
<li>Compute <code>hash(key)</code> and break the hash into two parts: the upper 57-bits (called <code>h1</code>) and the lower 7 bits (called <code>h2</code>).</li>
<li>The upper bits (<code>h1</code>) are used to select the first group to consider: <code>h1 % 2</code> in this case, as there are only 2 groups.</li>
<li>Within a group, all slots are equally eligible to hold the key. We must first determine whether any slot already contains this key, in which case this is an update rather than a new insertion.</li>
<li>If no slot contains the key, then we look for an empty slot to place this key.</li>
<li>If no slot is empty, then we continue the probe sequence by searching the next group.</li>
</ol>
<p>Lookup follows the same basic process.
If we find an empty slot in step 4, then we know an insertion would have used this slot and can stop the search.</p>
<p>Step 3 is where the Swiss Table magic happens.
We need to check whether any slot in a group contains the desired key.
Naively, we could just do a linear scan and compare all 8 keys.
However, the control word lets us do this more efficiently.
Each byte contains the lower 7 bits of the hash (<code>h2</code>) for that slot.
If we determine which bytes of the control word contain the <code>h2</code> we are looking for, we&rsquo;ll have a set of candidate matches.</p>
<p>In other words, we want to do a byte-by-byte equality comparison within the control word.
For example, if we are looking for key 32, where <code>h2 = 89</code>, the operation we want looks like so.</p>
<div class="swisstable-table-container">
 <table class="swisstable-table">
 <tbody>
 <tr>
 <td class="swisstable-table-cell"><strong>Test word</strong></td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">89</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Comparison</strong></td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 <td class="swisstable-table-cell">==</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Control word</strong></td>
 <td class="swisstable-table-cell">23</td>
 <td class="swisstable-table-cell">89</td>
 <td class="swisstable-table-cell">50</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 <td class="swisstable-table-cell">-</td>
 </tr>
 <tr>
 <td class="swisstable-table-cell"><strong>Result</strong></td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">1</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 <td class="swisstable-table-cell">0</td>
 </tr>
 </tbody>
 </table>
</div>
<p>This is an operation supported by <a href="https://en.wikipedia.org/wiki/Single_instruction,_multiple_data" rel="noreferrer" target="_blank">SIMD</a> hardware, where a single instruction performs parallel operations on independent values within a larger value (<em>vector</em>). In this case, we <a href="https://cs.opensource.google/go/go/+/master:src/internal/runtime/maps/group.go;drc=a08984bc8f2acacebeeadf7445ecfb67b7e7d7b1;l=155?ss=go" rel="noreferrer" target="_blank">can implement this operation</a> using a set of standard arithmetic and bitwise operations when special SIMD hardware is not available.</p>
<p>The result is a set of candidate slots.
Slots where <code>h2</code> does not match do not have a matching key, so they can be skipped.
Slots where <code>h2</code> does match are potential matches, but we must still check the entire key, as there is potential for collisions (1/128 probability of collision with a 7-bit hash, so still quite low).</p>
<p>This operation is very powerful, as we have effectively performed 8 steps of a probe sequence at once, in parallel.
This speeds up lookup and insertion by reducing the average number of comparisons we need to perform.
This improvement to probing behavior allowed both the Abseil and Go implementations to increase the maximum load factor of Swiss Table maps compared to prior maps, which lowers the average memory footprint.</p>
<h2 id="go-challenges">Go challenges</h2>
<p>Go&rsquo;s built-in map type has some unusual properties that pose additional challenges to adopting a new map design.
Two were particularly tricky to deal with.</p>
<h3 id="incremental-growth">Incremental growth</h3>
<p>When a hash table reaches its maximum load factor, it needs to grow the backing array.
Typically this means the next insertion doubles the size of the array, and copies all entries to the new array.
Imagine inserting into a map with 1GB of entries.
Most insertions are very fast, but the one insertion that needs to grow the map from 1GB to 2GB will need to copy 1GB of entries, which will take a long time.</p>
<p>Go is frequently used for latency-sensitive servers, so we don&rsquo;t want operations on built-in types that can have arbitrarily large impact on tail latency.
Instead, Go maps grow incrementally, so that each insertion has an upper bound on the amount of growth work it must do.
This bounds the latency impact of a single map insertion.</p>
<p>Unfortunately, the Abseil (C++) Swiss Table design assumes all at once growth, and the probe sequence depends on the total group count, making it difficult to break up.</p>
<p>Go&rsquo;s built-in map addresses this with another layer of indirection by splitting each map into multiple Swiss Tables.
Rather than a single Swiss Table implementing the entire map, each map consists of one or more independent tables that cover a subset of the key space.
An individual table stores a maximum of 1024 entries.
A variable number of upper bits in the hash are used to select which table a key belongs to.
This is a form of <a href="https://en.wikipedia.org/wiki/Extendible_hashing" rel="noreferrer" target="_blank"><em>extendible hashing</em></a>, where the number of bits used increases as needed to differentiate the total number of tables.</p>
<p>During insertion, if an individual table needs to grow, it will do so all at once, but other tables are unaffected.
Thus the upper bound for a single insertion is the latency of growing a 1024-entry table into two 1024-entry tables, copying 1024 entries.</p>
<h3 id="modification-during-iteration">Modification during iteration</h3>
<p>Many hash table designs, including Abseil&rsquo;s Swiss Tables, forbid modifying the map during iteration.
The Go language spec <a href="/ref/spec#For_statements:~:text=The%20iteration%20order,iterations%20is%200.">explicitly allows</a> modifications during iteration, with the following semantics:</p>
<ul>
<li>If an entry is deleted before it is reached, it will not be produced.</li>
<li>If an entry is updated before it is reached, the updated value will be produced.</li>
<li>If a new entry is added, it may or may not be produced.</li>
</ul>
<p>A typical approach to hash table iteration is to simply walk through the backing array and produce values in the order they are laid out in memory.
This approach runs afoul of the above semantics, most notably because insertions may grow the map, which would shuffle the memory layout.</p>
<p>We can avoid the impact of shuffle during growth by having the iterator keep a reference to the table it is currently iterating over.
If that table grows during iteration, we keep using the old version of the table and thus continue to deliver keys in the order of the old memory layout.</p>
<p>Does this work with the above semantics?
New entries added after growth will be missed entirely, as they are only added to the grown table, not the old table.
That&rsquo;s fine, as the semantics allow new entries not to be produced.
Updates and deletions are a problem, though: using the old table could produce stale or deleted entries.</p>
<p>This edge case is addressed by using the old table only to determine the iteration order.
Before actually returning the entry, we consult the grown table to determine whether the entry still exists, and to retrieve the latest value.</p>
<p>This covers all the core semantics, though there are even more small edge cases not covered here.
Ultimately, the permissiveness of Go maps with iteration results in iteration being the most complex part of Go&rsquo;s map implementation.</p>
<h2 id="future-work">Future work</h2>
<p>In <a href="/issue/54766#issuecomment-2542444404">microbenchmarks</a>, map operations are up to 60% faster than in Go 1.23.
Exact performance improvement varies quite a bit due to the wide variety of operations and uses of maps, and some edge cases do regress compared to Go 1.23.
Overall, in full application benchmarks, we found a geometric mean CPU time improvement of around 1.5%.</p>
<p>There are more map improvements we want to investigate for future Go releases.
For example, we may be able to <a href="/issue/70835">increase the locality of</a> operations on maps that are not in the CPU cache.</p>
<p>We could also further improve the control word comparisons.
As described above, we have a portable implementation using standard arithmetic and bitwise operations.
However, some architectures have SIMD instructions that perform this sort of comparison directly.
Go 1.24 already uses 8-byte SIMD instructions for amd64, but we could extend support to other architectures.
More importantly, while standard instructions operate on up to 8-byte words, SIMD instructions nearly always support at least 16-byte words.
This means we could increase the group size to 16 slots, and perform 16 hash comparisons in parallel instead of 8.
This would further decrease the average number of probes required for lookups.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>A Swiss Table-based Go map implementation has been a long time coming, and involved many contributors.
I want to thank YunHao Zhang (<a href="https://github.com/zhangyunhao116" rel="noreferrer" target="_blank">@zhangyunhao116</a>), PJ Malloy (<a href="https://github.com/thepudds" rel="noreferrer" target="_blank">@thepudds</a>), and <a href="https://github.com/andy-wm-arthur" rel="noreferrer" target="_blank">@andy-wm-arthur</a> for building initial versions of a Go Swiss Table implementation.
Peter Mattis (<a href="https://github.com/petermattis" rel="noreferrer" target="_blank">@petermattis</a>) combined these ideas with solutions to the Go challenges above to build <a href="https://pkg.go.dev/github.com/cockroachdb/swiss" rel="noreferrer" target="_blank"><code>github.com/cockroachdb/swiss</code></a>, a Go-spec compliant Swiss Table implementation.
The Go 1.24 built-in map implementation is heavily based on Peter&rsquo;s work.
Thank you to everyone in the community that contributed!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/cleanups-and-weak">From unique to cleanups and weak: new low-level tools for efficiency</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/synctest">Testing concurrent code with testing/synctest</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Testing concurrent code with testing/synctest</title><id>tag:blog.golang.org,2013:blog.golang.org/synctest</id><link rel="alternate" href="https://go.dev/blog/synctest"></link><published>2025-02-19T00:00:00+00:00</published><updated>2025-02-19T00:00:00+00:00</updated><author><name>Damien Neil</name></author><summary type="html">Go 1.24 contains an experimental package to aid in testing concurrent code.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/synctest">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Testing concurrent code with testing/synctest</h1>
 
 <p class="author">
 Damien Neil<br>
 19 February 2025
 </p>
 
 <p>One of Go&rsquo;s signature features is built-in support for concurrency.
Goroutines and channels are simple and effective primitives for
writing concurrent programs.</p>
<p>However, testing concurrent programs can be difficult and error prone.</p>
<p>In Go 1.24, we are introducing a new, experimental
<a href="/pkg/testing/synctest"><code>testing/synctest</code></a> package
to support testing concurrent code. This post will explain the motivation behind
this experiment, demonstrate how to use the synctest package, and discuss its potential future.</p>
<p>In Go 1.24, the <code>testing/synctest</code> package is experimental and
not subject to the Go compatibility promise.
It is not visible by default.
To use it, compile your code with <code>GOEXPERIMENT=synctest</code> set in your environment.</p>
<h2 id="testing-concurrent-programs-is-difficult">Testing concurrent programs is difficult</h2>
<p>To begin with, let us consider a simple example.</p>
<p>The <a href="/pkg/context#AfterFunc"><code>context.AfterFunc</code></a> function
arranges for a function to be called in its own goroutine after a context is canceled.
Here is a possible test for <code>AfterFunc</code>:</p>
<pre><code>func TestAfterFunc(t *testing.T) {
 ctx, cancel := context.WithCancel(context.Background())

 calledCh := make(chan struct{}) // closed when AfterFunc is called
 context.AfterFunc(ctx, func() {
 close(calledCh)
 })

 // TODO: Assert that the AfterFunc has not been called.

 cancel()

 // TODO: Assert that the AfterFunc has been called.
}
</code></pre>
<p>We want to check two conditions in this test:
The function is not called before the context is canceled,
and the function <em>is</em> called after the context is canceled.</p>
<p>Checking a negative in a concurrent system is difficult.
We can easily test that the function has not been called <em>yet</em>,
but how do we check that it <em>will not</em> be called?</p>
<p>A common approach is to wait for some amount of time before
concluding that an event will not happen.
Let&rsquo;s try introducing a helper function to our test which does this.</p>
<pre><code>// funcCalled reports whether the function was called.
funcCalled := func() bool {
 select {
 case &lt;-calledCh:
 return true
 case &lt;-time.After(10 * time.Millisecond):
 return false
 }
}

if funcCalled() {
 t.Fatalf(&quot;AfterFunc function called before context is canceled&quot;)
}

cancel()

if !funcCalled() {
 t.Fatalf(&quot;AfterFunc function not called after context is canceled&quot;)
}
</code></pre>
<p>This test is slow:
10 milliseconds isn&rsquo;t a lot of time, but it adds up over many tests.</p>
<p>This test is also flaky:
10 milliseconds is a long time on a fast computer,
but it isn&rsquo;t unusual to see pauses lasting several seconds
on shared and overloaded
<a href="https://en.wikipedia.org/wiki/Continuous_integration" rel="noreferrer" target="_blank">CI</a>
systems.</p>
<p>We can make the test less flaky at the expense of making it slower,
and we can make it less slow at the expense of making it flakier,
but we can&rsquo;t make it both fast and reliable.</p>
<h2 id="introducing-the-testingsynctest-package">Introducing the testing/synctest package</h2>
<p>The <code>testing/synctest</code> package solves this problem.
It allows us to rewrite this test to be simple, fast, and reliable,
without any changes to the code being tested.</p>
<p>The package contains only two functions: <code>Run</code> and <code>Wait</code>.</p>
<p><code>Run</code> calls a function in a new goroutine.
This goroutine and any goroutines started by it
exist in an isolated environment which we call a <em>bubble</em>.
<code>Wait</code> waits for every goroutine in the current goroutine&rsquo;s bubble
to block on another goroutine in the bubble.</p>
<p>Let&rsquo;s rewrite our test above using the <code>testing/synctest</code> package.</p>
<pre><code>func TestAfterFunc(t *testing.T) {
 synctest.Run(func() {
 ctx, cancel := context.WithCancel(context.Background())

 funcCalled := false
 context.AfterFunc(ctx, func() {
 funcCalled = true
 })

 synctest.Wait()
 if funcCalled {
 t.Fatalf(&quot;AfterFunc function called before context is canceled&quot;)
 }

 cancel()

 synctest.Wait()
 if !funcCalled {
 t.Fatalf(&quot;AfterFunc function not called after context is canceled&quot;)
 }
 })
}
</code></pre>
<p>This is almost identical to our original test,
but we have wrapped the test in a <code>synctest.Run</code> call
and we call <code>synctest.Wait</code> before asserting that the function has been called or not.</p>
<p>The <code>Wait</code> function waits for every goroutine in the caller&rsquo;s bubble to block.
When it returns, we know that the context package has either called the function,
or will not call it until we take some further action.</p>
<p>This test is now both fast and reliable.</p>
<p>The test is simpler, too:
we have replaced the <code>calledCh</code> channel with a boolean.
Previously we needed to use a channel to avoid a data race between
the test goroutine and the <code>AfterFunc</code> goroutine,
but the <code>Wait</code> function now provides that synchronization.</p>
<p>The race detector understands <code>Wait</code> calls,
and this test passes when run with <code>-race</code>.
If we remove the second <code>Wait</code> call,
the race detector will correctly report a data race in the test.</p>
<h2 id="testing-time">Testing time</h2>
<p>Concurrent code often deals with time.</p>
<p>Testing code that works with time can be difficult.
Using real time in tests causes slow and flaky tests,
as we have seen above.
Using fake time requires avoiding <code>time</code> package functions,
and designing the code under test to work with
an optional fake clock.</p>
<p>The <code>testing/synctest</code> package makes it simpler to test code that uses time.</p>
<p>Goroutines in the bubble started by <code>Run</code> use a fake clock.
Within the bubble, functions in the <code>time</code> package operate on the
fake clock. Time advances in the bubble when all goroutines are
blocked.</p>
<p>To demonstrate, let&rsquo;s write a test for the
<a href="/pkg/context#WithTimeout"><code>context.WithTimeout</code></a> function.
<code>WithTimeout</code> creates a child of a context,
which expires after a given timeout.</p>
<pre><code>func TestWithTimeout(t *testing.T) {
 synctest.Run(func() {
 const timeout = 5 * time.Second
 ctx, cancel := context.WithTimeout(context.Background(), timeout)
 defer cancel()

 // Wait just less than the timeout.
 time.Sleep(timeout - time.Nanosecond)
 synctest.Wait()
 if err := ctx.Err(); err != nil {
 t.Fatalf(&quot;before timeout, ctx.Err() = %v; want nil&quot;, err)
 }

 // Wait the rest of the way until the timeout.
 time.Sleep(time.Nanosecond)
 synctest.Wait()
 if err := ctx.Err(); err != context.DeadlineExceeded {
 t.Fatalf(&quot;after timeout, ctx.Err() = %v; want DeadlineExceeded&quot;, err)
 }
 })
}
</code></pre>
<p>We write this test just as if we were working with real time.
The only difference is that we wrap the test function in <code>synctest.Run</code>,
and call <code>synctest.Wait</code> after each <code>time.Sleep</code> call to wait for the context
package&rsquo;s timers to finish running.</p>
<h2 id="blocking-and-the-bubble">Blocking and the bubble</h2>
<p>A key concept in <code>testing/synctest</code> is the bubble becoming <em>durably blocked</em>.
This happens when every goroutine in the bubble is blocked,
and can only be unblocked by another goroutine in the bubble.</p>
<p>When a bubble is durably blocked:</p>
<ul>
<li>If there is an outstanding <code>Wait</code> call, it returns.</li>
<li>Otherwise, time advances to the next time that could unblock a goroutine, if any.</li>
<li>Otherwise, the bubble is deadlocked and <code>Run</code> panics.</li>
</ul>
<p>A bubble is not durably blocked if any goroutine is blocked
but might be woken by some event from outside the bubble.</p>
<p>The complete list of operations which durably block a goroutine is:</p>
<ul>
<li>a send or receive on a nil channel</li>
<li>a send or receive blocked on a channel created within the same bubble</li>
<li>a select statement where every case is durably blocking</li>
<li><code>time.Sleep</code></li>
<li><code>sync.Cond.Wait</code></li>
<li><code>sync.WaitGroup.Wait</code></li>
</ul>
<h3 id="mutexes">Mutexes</h3>
<p>Operations on a <code>sync.Mutex</code> are not durably blocking.</p>
<p>It is common for functions to acquire a global mutex.
For example, a number of functions in the reflect package
use a global cache guarded by a mutex.
If a goroutine in a synctest bubble blocks while acquiring
a mutex held by a goroutine outside the bubble,
it is not durably blocked—it is blocked, but will be unblocked
by a goroutine from outside its bubble.</p>
<p>Since mutexes are usually not held for long periods of time,
we simply exclude them from <code>testing/synctest</code>&rsquo;s consideration.</p>
<h3 id="channels">Channels</h3>
<p>Channels created within a bubble behave differently from ones created outside.</p>
<p>Channel operations are durably blocking only if the channel is bubbled
(created in the bubble).
Operating on a bubbled channel from outside the bubble panics.</p>
<p>These rules ensure that a goroutine is durably blocked only when
communicating with goroutines within its bubble.</p>
<h3 id="io">I/O</h3>
<p>External I/O operations, such as reading from a network connection,
are not durably blocking.</p>
<p>Network reads may be unblocked by writes from outside the bubble,
possibly even from other processes.
Even if the only writer to a network connection is also in the same bubble,
the runtime cannot distinguish between a connection waiting for more data to arrive
and one where the kernel has received data and is in the process of delivering it.</p>
<p>Testing a network server or client with synctest will generally
require supplying a fake network implementation.
For example, the <a href="/pkg/net#Pipe"><code>net.Pipe</code></a> function
creates a pair of <code>net.Conn</code>s that use an in-memory network connection
and can be used in synctest tests.</p>
<h2 id="bubble-lifetime">Bubble lifetime</h2>
<p>The <code>Run</code> function starts a goroutine in a new bubble.
It returns when every goroutine in the bubble has exited.
It panics if the bubble is durably blocked
and cannot be unblocked by advancing time.</p>
<p>The requirement that every goroutine in the bubble exit before Run returns
means that tests must be careful to clean up any background goroutines
before completing.</p>
<h2 id="testing-networked-code">Testing networked code</h2>
<p>Let&rsquo;s look at another example, this time using the <code>testing/synctest</code>
package to test a networked program.
For this example, we&rsquo;ll test the <code>net/http</code> package&rsquo;s handling of
the 100 Continue response.</p>
<p>An HTTP client sending a request can include an &ldquo;Expect: 100-continue&rdquo;
header to tell the server that the client has additional data to send.
The server may then respond with a 100 Continue informational response
to request the rest of the request,
or with some other status to tell the client that the content is not needed.
For example, a client uploading a large file might use this feature to
confirm that the server is willing to accept the file before sending it.</p>
<p>Our test will confirm that when sending an &ldquo;Expect: 100-continue&rdquo; header
the HTTP client does not send a request&rsquo;s content before the server
requests it, and that it does send the content after receiving a
100 Continue response.</p>
<p>Often tests of a communicating client and server can use a
loopback network connection. When working with <code>testing/synctest</code>,
however, we will usually want to use a fake network connection
to allow us to detect when all goroutines are blocked on the network.
We&rsquo;ll start this test by creating an <code>http.Transport</code> (an HTTP client) that uses
an in-memory network connection created by <a href="/pkg/net#Pipe"><code>net.Pipe</code></a>.</p>
<pre><code>func Test(t *testing.T) {
 synctest.Run(func() {
 srvConn, cliConn := net.Pipe()
 defer srvConn.Close()
 defer cliConn.Close()
 tr := &amp;http.Transport{
 DialContext: func(ctx context.Context, network, address string) (net.Conn, error) {
 return cliConn, nil
 },
 // Setting a non-zero timeout enables &quot;Expect: 100-continue&quot; handling.
 // Since the following test does not sleep,
 // we will never encounter this timeout,
 // even if the test takes a long time to run on a slow machine.
 ExpectContinueTimeout: 5 * time.Second,
 }
</code></pre>
<p>We send a request on this transport with the &ldquo;Expect: 100-continue&rdquo; header set.
The request is sent in a new goroutine, since it won&rsquo;t complete until the end of the test.</p>
<pre><code> body := &quot;request body&quot;
 go func() {
 req, _ := http.NewRequest(&quot;PUT&quot;, &quot;http://test.tld/&quot;, strings.NewReader(body))
 req.Header.Set(&quot;Expect&quot;, &quot;100-continue&quot;)
 resp, err := tr.RoundTrip(req)
 if err != nil {
 t.Errorf(&quot;RoundTrip: unexpected error %v&quot;, err)
 } else {
 resp.Body.Close()
 }
 }()
</code></pre>
<p>We read the request headers sent by the client.</p>
<pre><code> req, err := http.ReadRequest(bufio.NewReader(srvConn))
 if err != nil {
 t.Fatalf(&quot;ReadRequest: %v&quot;, err)
 }
</code></pre>
<p>Now we come to the heart of the test.
We want to assert that the client will not send the request body yet.</p>
<p>We start a new goroutine copying the body sent to the server into a <code>strings.Builder</code>,
wait for all goroutines in the bubble to block, and verify that we haven&rsquo;t read anything
from the body yet.</p>
<p>If we forget the <code>synctest.Wait</code> call, the race detector will correctly complain
about a data race, but with the <code>Wait</code> this is safe.</p>
<pre><code> var gotBody strings.Builder
 go io.Copy(&amp;gotBody, req.Body)
 synctest.Wait()
 if got := gotBody.String(); got != &quot;&quot; {
 t.Fatalf(&quot;before sending 100 Continue, unexpectedly read body: %q&quot;, got)
 }
</code></pre>
<p>We write a &ldquo;100 Continue&rdquo; response to the client and verify that it now sends the
request body.</p>
<pre><code> srvConn.Write([]byte(&quot;HTTP/1.1 100 Continue\r\n\r\n&quot;))
 synctest.Wait()
 if got := gotBody.String(); got != body {
 t.Fatalf(&quot;after sending 100 Continue, read body %q, want %q&quot;, got, body)
 }
</code></pre>
<p>And finally, we finish up by sending the &ldquo;200 OK&rdquo; response to conclude the request.</p>
<p>We have started several goroutines during this test.
The <code>synctest.Run</code> call will wait for all of them to exit before returning.</p>
<pre><code> srvConn.Write([]byte(&quot;HTTP/1.1 200 OK\r\n\r\n&quot;))
 })
}
</code></pre>
<p>This test can be easily extended to test other behaviors,
such as verifying that the request body is not sent if the server does not ask for it,
or that it is sent if the server does not respond within a timeout.</p>
<h2 id="status-of-the-experiment">Status of the experiment</h2>
<p>We are introducing <a href="/pkg/testing/synctest"><code>testing/synctest</code></a>
in Go 1.24 as an <em>experimental</em> package.
Depending on feedback and experience
we may release it with or without amendments,
continue the experiment,
or remove it in a future version of Go.</p>
<p>The package is not visible by default.
To use it, compile your code with <code>GOEXPERIMENT=synctest</code> set in your environment.</p>
<p>We want to hear your feedback!
If you try out <code>testing/synctest</code>,
please report your experiences, positive or negative,
on <a href="/issue/67434">go.dev/issue/67434</a>.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/swisstable">Faster Go maps with Swiss Tables</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/wasmexport">Extensible Wasm Applications with Go</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Extensible Wasm Applications with Go</title><id>tag:blog.golang.org,2013:blog.golang.org/wasmexport</id><link rel="alternate" href="https://go.dev/blog/wasmexport"></link><published>2025-02-13T00:00:00+00:00</published><updated>2025-02-13T00:00:00+00:00</updated><author><name>Cherry Mui</name></author><summary type="html">Go 1.24 enhances WebAssembly capabilities with function export and reactor mode</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/wasmexport">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Extensible Wasm Applications with Go</h1>
 
 <p class="author">
 Cherry Mui<br>
 13 February 2025
 </p>
 
 <p>Go 1.24 enhances its WebAssembly (Wasm) capabilities with the
addition of the <code>go:wasmexport</code> directive and the ability to build a reactor
for WebAssembly System Interface (WASI).
These features enable Go developers to export Go functions to Wasm,
facilitating better integration with Wasm hosts and expanding the possibilities
for Go-based Wasm applications.</p>
<h2 id="webassembly-and-the-webassembly-system-interface">WebAssembly and the WebAssembly System Interface</h2>
<p><a href="https://webassembly.org/" rel="noreferrer" target="_blank">WebAssembly (Wasm)</a> is a binary instruction format
that was initially created for web browsers, providing the execution of
high-performance, low-level code at speeds approaching native performance.
Since then, Wasm&rsquo;s utility has expanded, and it is now used in various
environments beyond the browser.
Notably, cloud providers offer services that directly execute Wasm
executables, taking advantage of the
<a href="https://wasi.dev/" rel="noreferrer" target="_blank">WebAssembly System Interface (WASI)</a> system call API.
WASI allows these executables to interact with system resources.</p>
<p>Go first added support for compiling to Wasm in the 1.11 release, through the
<code>js/wasm</code> port.
Go 1.21 added a new port targeting the WASI preview 1 syscall API through the
new <code>GOOS=wasip1</code> port.</p>
<h2 id="exporting-go-functions-to-wasm-with-gowasmexport">Exporting Go Functions to Wasm with <code>go:wasmexport</code></h2>
<p>Go 1.24 introduces a new compiler directive, <code>go:wasmexport</code>, which allows
developers to export Go functions to be called from outside of the
Wasm module, typically from a host application that runs the Wasm runtime.
This directive instructs the compiler to make the annotated function available
as a Wasm <a href="https://webassembly.github.io/spec/core/valid/modules.html?highlight=export#exports" rel="noreferrer" target="_blank">export</a>
in the resulting Wasm binary.</p>
<p>To use the <code>go:wasmexport</code> directive, simply add it to a function definition:</p>
<pre><code>//go:wasmexport add
func add(a, b int32) int32 { return a + b }
</code></pre>
<p>With this, the Wasm module will have an exported function named <code>add</code> that
can be called from the host.</p>
<p>This is analogous to the <a href="/cmd/cgo#hdr-C_references_to_Go">cgo <code>export</code> directive</a>,
which makes the function available to be called from C,
though <code>go:wasmexport</code> uses a different, simpler mechanism.</p>
<h2 id="building-a-wasi-reactor">Building a WASI Reactor</h2>
<p>A WASI reactor is a WebAssembly module that operates continuously, and
can be called upon multiple times to react on events or requests.
Unlike a &ldquo;command&rdquo; module, which terminates after its main function finishes,
a reactor instance remains live after initialization, and its exports remain
accessible.</p>
<p>With Go 1.24, one can build a WASI reactor with the <code>-buildmode=c-shared</code> build
flag.</p>
<pre><code>$ GOOS=wasip1 GOARCH=wasm go build -buildmode=c-shared -o reactor.wasm
</code></pre>
<p>The build flag signals to the linker not to generate the <code>_start</code> function
(the entry point for a command module), and instead generate an
<code>_initialize</code> function, which performs runtime and package initialization,
along with any exported functions and their dependencies.
The <code>_initialize</code> function must be called before any other exported functions.
The <code>main</code> function will not be automatically invoked.</p>
<p>To use a WASI reactor, the host application first initializes it by calling
<code>_initialize</code>, then simply invoke the exported functions.
Here is an example using <a href="https://wazero.io/" rel="noreferrer" target="_blank">Wazero</a>, a Go-based Wasm runtime
implementation:</p>
<pre><code>// Create a Wasm runtime, set up WASI.
r := wazero.NewRuntime(ctx)
defer r.Close(ctx)
wasi_snapshot_preview1.MustInstantiate(ctx, r)

// Configure the module to initialize the reactor.
config := wazero.NewModuleConfig().WithStartFunctions(&quot;_initialize&quot;)

// Instantiate the module.
wasmModule, _ := r.InstantiateWithConfig(ctx, wasmFile, config)

// Call the exported function.
fn := wasmModule.ExportedFunction(&quot;add&quot;)
var a, b int32 = 1, 2
res, _ := fn.Call(ctx, api.EncodeI32(a), api.EncodeI32(b))
c := api.DecodeI32(res[0])
fmt.Printf(&quot;add(%d, %d) = %d\n&quot;, a, b, c)

// The instance is still alive. We can call the function again.
res, _ = fn.Call(ctx, api.EncodeI32(b), api.EncodeI32(c))
fmt.Printf(&quot;add(%d, %d) = %d\n&quot;, b, c, api.DecodeI32(res[0]))
</code></pre>
<p>The <code>go:wasmexport</code> directive and the reactor build mode allow applications to
be extended by calling into Go-based Wasm code.
This is particularly valuable for applications that have adopted Wasm as a
plugin or extension mechanism with well-defined interfaces.
By exporting Go functions, applications can leverage the Go Wasm modules to
provide functionality without needing to recompile the entire application.
Furthermore, building as a reactor ensures that the exported functions can be
called multiple times without requiring reinitialization, making it suitable
for long-running applications or services.</p>
<h2 id="supporting-rich-types-between-the-host-and-the-client">Supporting rich types between the host and the client</h2>
<p>Go 1.24 also relaxes the constraints on types that can be used as input and
result parameters with <code>go:wasmimport</code> functions.
For example, one can pass a bool, a string, a pointer to an <code>int32</code>, or a
pointer to a struct which embeds <code>structs.HostLayout</code> and contains supported
field types
(see the <a href="/cmd/compile#hdr-WebAssembly_Directives">documentation</a> for detail).
This allows Go Wasm applications to be written in a more natural and ergonomic
way, and removes some unnecessary type conversions.</p>
<h2 id="limitations">Limitations</h2>
<p>While Go 1.24 has made significant enhancements to its Wasm capabilities,
there are still some notable limitations.</p>
<p>Wasm is a single-threaded architecture with no parallelism.
A <code>go:wasmexport</code> function can spawn new goroutines.
But if a function creates a background goroutine, it will not continue
executing when the <code>go:wasmexport</code> function returns, until calling back into
the Go-based Wasm module.</p>
<p>While some type restrictions have been relaxed in Go 1.24, there are still
limitations on the types that can be used with <code>go:wasmimport</code> and
<code>go:wasmexport</code> functions.
Due to the unfortunate mismatch between the 64-bit architecture of the client
and the 32-bit architecture of the host, it is not possible to pass pointers in
memory.
For example, a <code>go:wasmimport</code> function cannot take a pointer to a struct that
contains a pointer-typed field.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The addition of the ability to build a WASI reactor and export Go functions to
Wasm in Go 1.24 represent a significant step forward for Go&rsquo;s WebAssembly
capabilities.
These features empower developers to create more versatile and powerful Go-based
Wasm applications, opening up new possibilities for Go in the Wasm ecosystem.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/synctest">Testing concurrent code with testing/synctest</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/go1.24">Go 1.24 is released!</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go 1.24 is released!</title><id>tag:blog.golang.org,2013:blog.golang.org/go1.24</id><link rel="alternate" href="https://go.dev/blog/go1.24"></link><published>2025-02-11T00:00:00+00:00</published><updated>2025-02-11T00:00:00+00:00</updated><author><name>Junyang Shao, on behalf of the Go team</name></author><summary type="html">Go 1.24 brings generic type aliases, map performance improvements, FIPS 140 compliance and more.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/go1.24">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go 1.24 is released!</h1>
 
 <p class="author">
 Junyang Shao, on behalf of the Go team<br>
 11 February 2025
 </p>
 
 <p>Today the Go team is excited to release Go 1.24,
which you can get by visiting the <a href="/dl/">download page</a>.</p>
<p>Go 1.24 comes with many improvements over Go 1.23. Here are some of the notable
changes; for the full list, refer to the <a href="/doc/go1.24">release notes</a>.</p>
<h2 id="language-changes">Language changes</h2>
<p>Go 1.24 now fully supports <a href="/issue/46477">generic type aliases</a>: a type alias
may be parameterized like a defined type.
See the <a href="/ref/spec#Alias_declarations">language spec</a> for details.</p>
<h2 id="performance-improvements">Performance improvements</h2>
<p>Several performance improvements in the runtime have decreased CPU overhead
by 2–3% on average across a suite of representative benchmarks. These
improvements include a new builtin <code>map</code> implementation based on
<a href="https://abseil.io/about/design/swisstables" rel="noreferrer" target="_blank">Swiss Tables</a>, more efficient
memory allocation of small objects, and a new runtime-internal mutex
implementation.</p>
<h2 id="tool-improvements">Tool improvements</h2>
<ul>
<li>The <code>go</code> command now provides a mechanism for tracking tool dependencies for a
module. Use <code>go get -tool</code> to add a <code>tool</code> directive to the current module. Use
<code>go tool [tool name]</code> to run the tools declared with the <code>tool</code> directive.
Read more on the <a href="/doc/go1.24#go-command">go command</a> in the release notes.</li>
<li>The new <code>test</code> analyzer in <code>go vet</code> subcommand reports common mistakes in
declarations of tests, fuzzers, benchmarks, and examples in test packages.
Read more on <a href="/doc/go1.24#vet">vet</a> in the release notes.</li>
</ul>
<h2 id="standard-library-additions">Standard library additions</h2>
<ul>
<li>
<p>The standard library now includes <a href="/doc/security/fips140">a new set of mechanisms to facilitate
FIPS 140-3 compliance</a>. Applications require no source code
changes to use the new mechanisms for approved algorithms. Read more
on <a href="/doc/go1.24#fips140">FIPS 140-3 compliance</a> in the release notes.
Apart from FIPS 140, several packages that were previously in the
<a href="/pkg/golang.org/x/crypto">x/crypto</a> module are now available in the
<a href="/doc/go1.24#crypto-mlkem">standard library</a>.</p>
</li>
<li>
<p>Benchmarks may now use the faster and less error-prone
<a href="/pkg/testing#B.Loop"><code>testing.B.Loop</code></a> method to perform benchmark iterations
like <code>for b.Loop() { ... }</code> in place of the typical loop structures involving
<code>b.N</code> like <code>for range b.N</code>. Read more on
<a href="/doc/go1.24#new-benchmark-function">the new benchmark function</a> in the
release notes.</p>
</li>
<li>
<p>The new <a href="/pkg/os#Root"><code>os.Root</code></a> type provides the ability to perform
filesystem operations isolated under a specific directory. Read more on
<a href="/doc/go1.24#directory-limited-filesystem-access">filesystem access</a> in the
release notes.</p>
</li>
<li>
<p>The runtime provides a new finalization mechanism,
<a href="/pkg/runtime#AddCleanup"><code>runtime.AddCleanup</code></a>, that is more flexible,
more efficient, and less error-prone than
<a href="/pkg/runtime#SetFinalizer"><code>runtime.SetFinalizer</code></a>. Read more on
<a href="/doc/go1.24#improved-finalizers">cleanups</a> in the release notes.</p>
</li>
</ul>
<h2 id="improved-webassembly-support">Improved WebAssembly support</h2>
<p>Go 1.24 adds a new <code>go:wasmexport</code> directive for Go programs to export
functions to the WebAssembly host, and supports building a Go program as a WASI
<a href="https://github.com/WebAssembly/WASI/blob/63a46f61052a21bfab75a76558485cf097c0dbba/legacy/application-abi.md#current-unstable-abi" rel="noreferrer" target="_blank">reactor/library</a>.
Read more on <a href="/doc/go1.24#wasm">WebAssembly</a> in the release notes.</p>
<hr>
<p>Please read the <a href="/doc/go1.24">Go 1.24 release notes</a> for the complete and
detailed information. Don&rsquo;t forget to watch for follow-up blog posts that
will go in more depth on some of the topics mentioned here!</p>
<p>Thank you to everyone who contributed to this release by writing code and
documentation, reporting bugs, sharing feedback, and testing the release
candidates. Your efforts helped to ensure that Go 1.24 is as stable as possible.
As always, if you notice any problems, please <a href="/issue/new">file an issue</a>.</p>
<p>Enjoy Go 1.24!</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/wasmexport">Extensible Wasm Applications with Go</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/survey2024-h2-results">Go Developer Survey 2024 H2 Results</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go Developer Survey 2024 H2 Results</title><id>tag:blog.golang.org,2013:blog.golang.org/survey2024-h2-results</id><link rel="alternate" href="https://go.dev/blog/survey2024-h2-results"></link><published>2024-12-20T00:00:00+00:00</published><updated>2024-12-20T00:00:00+00:00</updated><author><name>Alice Merrick</name></author><summary type="html">What we learned from our 2024 H2 developer survey</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/survey2024-h2-results">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go Developer Survey 2024 H2 Results</h1>
 
 <p class="author">
 Alice Merrick<br>
 20 December 2024
 </p>
 
 <style type="text/css" scoped>
 .chart {
 margin-left: 1.5rem;
 margin-right: 1.5rem;
 width: 800px;
 }
 blockquote p {
 color: var(--color-text-subtle) !important;
 }

 .quote_source {
 font-style: italic;
 }

 @media (prefers-color-scheme: dark) {
 .chart {
 border-radius: 8px;
 }
 }
</style>
<h2 id="background">Background</h2>
<p>Go was designed with a focus on developer experience, and we deeply value the
feedback we receive through proposals, issues, and community interactions.
However, these channels often represent the voices of our most experienced or
engaged users, a small subset of the broader Go community. To ensure we&rsquo;re
serving developers of all skill levels, including those who may not have strong
opinions on language design, we conduct this survey once or twice a year to
gather systematic feedback and quantitative evidence. This inclusive approach
allows us to hear from a wider range of Go developers, providing valuable
insights into how Go is used across different contexts and experience levels.
Your participation is critical in informing our decisions about language changes
and resource allocation, ultimately shaping the future of Go. Thank you to
everyone who contributed, and we strongly encourage your continued participation
in future surveys. Your experience matters to us.</p>
<p>This post shares the results of our most recent Go Developer Survey, conducted
from September 9–23, 2024. We recruited participants from the Go blog and
through randomized prompts in the <a href="https://code.visualstudio.com/" rel="noreferrer" target="_blank">VS Code</a> Go
plug-in and <a href="https://www.jetbrains.com/go/" rel="noreferrer" target="_blank">GoLand IDE</a>, allowing us to recruit
a more representative sample of Go developers. We received a total of 4,156
responses. A huge thank you to all those who contributed to making this
possible.</p>
<p>Along with capturing sentiments and challenges around using Go and Go tooling,
our primary focus areas for this survey were on uncovering sources of toil,
challenges to performing best practices, and how developers are using AI
assistance.</p>
<h2 id="highlights">Highlights</h2>
<ul>
<li><strong>Developer sentiment towards Go remains extremely positive</strong>, with 93% of
survey respondents saying they felt satisfied while working with Go during the
prior year.</li>
<li><strong>Ease of deployment and an easy to use API/SDK</strong> were respondents’ favorite
things about using Go on the top three cloud providers. First-class Go support
is critical to keeping up with developer expectations.</li>
<li>70% of respondents were using AI assistants when developing with Go. The most
common uses were <strong>LLM-based code completion</strong>, writing tests, generating Go
code from natural language descriptions, and brainstorming. There was a
significant discrepancy between what respondents said they wanted to use AI
for last year, and what they currently use AI for.</li>
<li>The biggest challenge for teams using Go was <strong>maintaining consistent coding
standards</strong> across their codebase. This was often due to team members having
different levels of Go experience and coming from different programming
backgrounds, leading to inconsistencies in coding style and adoption of
non-idiomatic patterns.</li>
</ul>
<h2 id="contents">Contents</h2>
<ul>
<li><a href="#sentiment">Overall satisfaction</a></li>
<li><a href="#devenv">Development environments and tools</a></li>
<li><a href="#cloud">Go in the clouds</a></li>
<li><a href="#ai-assistance">AI assistance</a></li>
<li><a href="#team-challenges">Challenges for teams using Go</a></li>
<li><a href="#simd">Single Instruction, Multiple Data (SIMD)</a></li>
<li><a href="#demographics">Demographics</a></li>
<li><a href="#firmographics">Firmographics</a></li>
<li><a href="#methodology">Methodology</a></li>
<li><a href="#how-to-read-these-results">How to read these results</a></li>
<li><a href="#closing">Closing</a></li>
</ul>
<h3 id="sentiment">Overall satisfaction</h3>
<p>Overall satisfaction remains high in the survey with 93% of respondents saying
they were somewhat or very satisfied with Go during the last year. Although the
exact percentages fluctuate slightly from cycle to cycle, we do not see any
statistically significant differences from our <a href="/blog/survey2023-h2-results">2023
H2</a> or <a href="/blog/survey2024-h1-results">2024 H1
</a>Surveys when the satisfaction rate
was 90% and 93%, respectively.</p>
<p><img src="survey2024h2/csat.svg" alt="Chart of developer satisfaction with Go"
class="chart" /></p>
<p>The open comments we received on the survey continue to highlight what
developers like most about using Go, for example, its simplicity, the Go
toolchain, and its promise of backwards compatibility:</p>
<p><em>“I am a programming languages enjoyer (C-like) and I always come back to Go for
its simplicity, fast compilation and robust toolchain. Keep it up!”</em></p>
<p><em>“Thank you for creating Go! It is my favorite language, because it is pretty
minimal, the development cycle has rapid build-test cycles, and when using a
random open source project written in Go, there is a good chance that it will
work, even 10 years after. I love the 1.0 compatibility guarantee.”</em></p>
<h3 id="devenv">Development environments and tools</h3>
<h4 id="developer-os">Developer OS</h4>
<p>Consistent with previous years, most survey respondents develop with Go on Linux
(61%) and macOS (59%) systems. Historically, the proportion of Linux and macOS
users has been very close, and we didn’t see any significant changes from the
last survey. The randomly sampled groups from JetBrains and VS Code were more
likely (33% and 36%, respectively) to develop on Windows than the self-selected
group (16%).</p>
<p><img src="survey2024h2/os_dev.svg" alt="Chart of operating systems respondents
use when developing Go software" class="chart" /> <img
src="survey2024h2/os_dev_src.svg" alt="Chart of operating systems respondents
use when developing Go software, split by difference sample sources"
class="chart" /></p>
<h4 id="deployment-environments">Deployment environments</h4>
<p>Given the prevalence of Go for cloud development and containerized workloads,
it’s no surprise that Go developers primarily deploy to Linux environments
(96%).</p>
<p><img src="survey2024h2/os_deploy.svg" alt="Chart of operating systems
respondents deploy to when developing Go software" class="chart" /></p>
<p>We included several questions to understand what architectures respondents are
deploying to when deploying to Linux, Windows or WebAssembly. The x86-64 /
AMD64 architecture was by far the most popular choice for those deploying to
both Linux (92%) and Windows (97%). ARM64 was second at 49% for Linux and 21%
for Windows.</p>
<p><img src="survey2024h2/arch_linux.svg" alt="Linux architecture usage"
class="chart" /> <img src="survey2024h2/arch_win.svg" alt="Windows architecture
usage" class="chart" /></p>
<p>Not many respondents deployed to Web Assembly (only about 4% of overall
respondents), but 73% that do said they deploy to JS and 48% to WASI Preview 1.</p>
<p><img src="survey2024h2/arch_wa.svg" alt="Web assembly architecture usage"
class="chart" /></p>
<h4 id="editor-awareness-and-preferences">Editor awareness and preferences</h4>
<p>We introduced a new question on this survey to assess awareness and usage of
popular editors for Go. When interpreting these results, keep in mind that 34%
of respondents came to the survey from VS Code and 9% of respondents came from
GoLand, so it is more likely for them to use those editors regularly.</p>
<p>VS Code was the most widely used editor, with 66% of respondents using it
regularly, and GoLand was the second most used at 35%. Almost all respondents
had heard of both VS Code and GoLand, but respondents were much more likely to
have at least tried VS Code. Interestingly, 33% of respondents said they
regularly use 2 or more editors. They may use different editors for different
tasks or environments, such as using Emacs or Vim via SSH, where IDEs aren’t
available.</p>
<p><img src="survey2024h2/editor_aware.svg" alt="Level of familiarity with each
editor" class="chart" /></p>
<p>We also asked a question about editor preference, the same as we have asked on
previous surveys. Because our randomly sampled populations were recruited from
within VS Code or GoLand, they are strongly biased towards preferring those
editors. To avoid skewing the results, we show the data for the most preferred
editor here from the self-selected group only. 38% preferred VS Code and 35%
preferred GoLand. This is a notable difference from the last survey in H1, when
43% preferred VS Code and 33% preferred GoLand. A possible explanation could be
in how respondents were recruited this year. This year the VS Code notification
began inviting developers to take the survey before the Go blog entry was
posted, so a larger proportion of respondents came from the VS Code prompt this
year who might have otherwise come from the blog post. Because we only show the
self-selected respondents in this chart, data from respondents from the VS Code
prompt data are not represented here. Another contributing factor could be the
slight increase in those who prefer &ldquo;Other&rdquo; (4%). The write-in responses suggest
there is increased interest in editors like <a href="https://zed.dev/" rel="noreferrer" target="_blank">Zed</a>, which made
up 43% of the write-in responses.</p>
<p><img src="survey2024h2/editor.svg" alt="Code editors respondents most prefer to
use with Go" class="chart" /></p>
<h4 id="code-analysis-tools">Code analysis tools</h4>
<p>The most popular code analysis tool was <code>gopls</code>, which was knowingly used by 65%
of respondents. Because <code>gopls</code> is used under-the-hood by default in VS Code,
this is likely an undercount. Following closely behind, <code>golangci-lint</code> was used
by 57% of respondents, and <code>staticcheck</code> was used by 34% of respondents. A much
smaller proportion used custom or other tools, which suggests that most
respondents prefer common established tools over custom solutions. Only 10% of
respondents indicated they don&rsquo;t use any code analysis tools.</p>
<p><img src="survey2024h2/code_analysis.svg" alt="Code analysis tools respondents
use with Go" class="chart" /></p>
<h4 id="cloud">Go in the Clouds</h4>
<p>Go is a popular language for modern cloud-based development, so we typically
include survey questions to help us understand which cloud platforms and
services Go developers are using. In this cycle, we sought to learn about
preferences and experiences of Go developers across cloud providers, with a
particular focus on the largest cloud providers: Amazon Web Services (AWS),
Microsoft Azure, and Google Cloud. We also included an additional option for
“Bare Metal Servers” for those who deploy to servers without virtualization.</p>
<p>Similar to previous years, almost half of respondents (50%) deploy Go programs
to Amazon Web Services. AWS is followed by self-owned or company-owned servers
(37%), and Google Cloud (30%). Respondents who work at large organizations are a
little more likely to deploy to self-owned or company-owned servers (48%) than
those who work at small-to-medium organizations (34%). They‘re also a little
more likely to deploy to Microsoft Azure (25%) than small-to-medium
organizations (12%).</p>
<p><img src="survey2024h2/cloud_platform.svg" alt="Cloud providers where
respondents deploy Go software" class="chart" /></p>
<p>The most commonly used cloud services were AWS Elastic Kubernetes Service (41%),
AWS EC2 (39%), and Google Cloud GKE (29%). Although we’ve seen Kubernetes usage
increase over time, this is the first time we’ve seen EKS become more widely
used than EC2. Overall, Kubernetes offerings were the most popular services for
AWS, Google Cloud, and Azure, followed by VMs and then Serverless offerings.
Go&rsquo;s strengths in containerization and microservices development naturally align
with the rising popularity of Kubernetes, as it provides an efficient and
scalable platform for deploying and managing these types of applications.</p>
<p><img src="survey2024h2/cloud_service.svg" alt="Cloud platforms where respondents
deploy Go software" class="chart" /></p>
<p>We asked a followup question to respondents who deployed Go code to the top
three cloud providers, Amazon Web Services, Google Cloud, and Microsoft Azure on
what they like most about deploying Go code to each cloud. The most popular
response across different providers was actually Go&rsquo;s performance and language
features rather than something about the cloud provider.</p>
<p>Other common reasons were:</p>
<ul>
<li>Familiarity with the given cloud provider compared to other clouds</li>
<li>Ease of deployment of Go applications on the given cloud provider</li>
<li>The cloud provider&rsquo;s API/SDK for Go is easy to use</li>
<li>The API/SDK is well documented</li>
</ul>
<p>Other than familiarity, the top favorite things highlight the importance of
having first class support for Go to keep up with developer expectations.</p>
<p>It was also fairly common for respondents to say they don&rsquo;t have a favorite
thing about their cloud provider. From a previous version of the survey that
involved write-in responses, this often meant that they did not interact
directly with the Cloud. In particular, respondents who use Microsoft Azure were
much more likely to say that “Nothing” was their favorite thing (51%) compared
to AWS (27%) or Google Cloud (30%).</p>
<p><img src="survey2024h2/cloud_fave_all.svg" alt= "What respondents liked most
about each of the top 3 Clouds" class="chart" /></p>
<h3 id="ai-assistance">AI assistance</h3>
<p>The Go team hypothesizes that AI assistance has the potential to alleviate
developers from tedious and repetitive tasks, allowing them to focus on more
creative and fulfilling aspects of their work. To gain insights into areas where
AI assistance could be most beneficial, we included a section in our survey to
identify common developer toil.</p>
<p>The majority of respondents (70%) are using AI assistants when developing with
Go. The most common usage of AI assistants was in LLM-based code completion
(35%). Other common responses were writing tests (29%), generating Go code from
a natural language description (27%), and brainstorming ideas (25%). There was
also a sizable minority (30%) of respondents who had not used any LLM for
assistance in the last month.</p>
<p><img src="survey2024h2/ai_assist_tasks.svg" alt= "Most common tasks used with AI
assistance" class="chart" /></p>
<p>Some of these results stood out when compared to findings from our 2023 H2
survey, where we asked respondents for the top 5 use cases they would like to
see AI/ML support Go developers. Although a couple new responses were introduced
in the current survey, we can still do a rough comparison between what
respondents said they wanted AI support for, and what their actual usage was
like. In that previous survey, writing tests was the most desired use case
(49%). In our latest 2024 H2 survey, about 29% of respondents had used AI
assistants for this in the last month. This suggests that current offerings are
not meeting developer needs for writing tests. Similarly, in 2023, 47%
respondents said they would like suggestions for best practices while coding,
while only 14% a year later said they are using AI assistance for this use case.
46% said they wanted help catching common mistakes while coding, and only 13%
said they were using AI assistance for this. This could indicate that current AI
assistants are not well-equipped for these kinds of tasks, or they&rsquo;re not well
integrated into developer workflows or tooling.</p>
<p>It was also surprising to see such high usage of AI for generating Go code from
natural language and brainstorming, since the previous survey didn&rsquo;t indicate
these as highly desired use cases. There could be a number of explanations for
these differences. While previous respondents might not have explicitly <em>wanted</em>
AI for code generation or brainstorming initially, they might be gravitating
towards these uses because they align with the current strengths of generative
AI—natural language processing and creative text generation. We should also keep
in mind that people<a href="https://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/" rel="noreferrer" target="_blank"> are not necessarily the best predictors of their own
behavior</a>.</p>
<p><img src="survey2024h2/ai_assist_tasks_yoy.svg" alt= "Tasks used AI assistance
in 2024 compared to those wanted in 2023" class="chart" /></p>
<p>We also saw some notable differences in how different groups responded to this
question. Respondents at small to medium sized organizations were a little more
likely to have used LLMs (75%) compared to those at large organizations (66%).
There could be a number of reasons why, for example, larger organizations may
have stricter security and compliance requirements and concerns about the
security of LLM coding assistants, the potential for data leakage, or compliance
with industry-specific regulations. They also may have already invested in
other developer tools and practices that already provide similar benefits to
developer productivity.</p>
<p><img src="survey2024h2/ai_assist_tasks_org.svg" alt= "Most common tasks used
with AI assistance by org size" class="chart" /></p>
<p>Go developers with less than 2 years of experience were more likely to use AI
assistants (75%) compared to Go developers with 5+ years of experience (67%).
Less experienced Go developers were also more likely to use them for more tasks,
on average 3.50. Although all experience levels tended to use LLM-based code
completion, less experienced Go developers were more likely to use Go for more
tasks related to learning and debugging, such as explaining what a piece of Go
code does, resolving compiler errors and debugging failures in their Go code.
This suggests that AI assistants are currently providing the greatest utility to
those who are less familiar with Go. We don&rsquo;t know how AI assistants affect
learning or getting started on a new Go project, something we want to
investigate in the future. However, all experience levels had similar rates of
satisfaction with their AI assistants, around 73%, so new Go developers are not
more satisfied with AI assistants, despite using them more often.</p>
<p><img src="survey2024h2/ai_assist_tasks_exp.svg" alt= "Most common tasks used
with AI assistance by experience with Go" class="chart" /></p>
<p>To respondents who reported using AI assistance for at least one task related to
writing Go code, we asked some follow up questions to learn more about their AI
assistant usage. The most commonly used AI assistants were ChatGPT (68%) and
GitHub Copilot (50%). When asked which AI assistant they used <em>most</em> in the last
month, ChatGPT and Copilot were about even at 36% each, so although more
respondents used ChatGPT, it wasn’t necessarily their primary assistant.
Participants were similarly satisfied with both tools (73% satisfied with
ChatGPT, vs. 78% with GitHub CoPilot). The highest satisfaction rate for any AI
assistant was Anthropic Claude, at 87%.</p>
<p><img src="survey2024h2/ai_assistants_withother.svg" alt= "Most common AI
assistants used" class="chart" /> <img src="survey2024h2/ai_primary.svg" alt=
"Most common primary AI assistants used" class="chart" /></p>
<h3 id="team-challenges">Challenges for teams using Go</h3>
<p>In this section of the survey, we wanted to understand which best practices or
tools should be better integrated into developer workflows. Our approach was to
identify common problems for teams using Go. We then asked respondents which
challenges would bring them the most benefit if they were “magically” solved for
them. (This was so that respondents would not focus on particular solutions.)
Common problems that would provide the most benefit if they were solved would be
considered candidates for improvement.</p>
<p>The most commonly reported challenges for teams were maintaining consistent coding
standards across our Go codebase (58%), identifying performance issues in a
running Go program (58%) and identifying resource usage inefficiencies in a
running Go program (57%).</p>
<p><img src="survey2024h2/dev_challenges.svg" alt= "Most common challenges for
teams" class="chart" /></p>
<p>21% of respondents said their team would benefit most from maintaining
consistent coding standards across their Go codebase. This was the most common
response, making it a good candidate to address. In a follow-up question, we got
more details as to why specifically this was so challenging.</p>
<p><img src="survey2024h2/dev_challenges_most_benefit.svg" alt= "Most benefit to
solve" class="chart" /></p>
<p>According to the write-in responses, many teams face challenges maintaining
consistent coding standards because their members have varying levels of
experience with Go and come from different programming backgrounds. This led to
inconsistencies in coding style and the adoption of non-idiomatic patterns.</p>
<p><em>“There&rsquo;s lots of polyglot engineers where I work. So the Go written is not
consistent. I do consider myself a Gopher and spend time trying to convince my
teammates what is idiomatic in Go”—Go developer with 2–4 years of experience.</em></p>
<p><em>“Most of the team members are learning Go from scratch. Coming from the
dynamically typed languages, it takes them a while to get used to the new
language. They seem to struggle maintaining the code consistency following the
Go guidelines.”—Go developer with 2–4 years of experience.</em></p>
<p>This echoes some feedback we’ve heard before about teammates who write &ldquo;Gava&rdquo; or
&ldquo;Guby&rdquo; due to their previous language experiences. Although static analysis was
a class of tool we had in mind to address this issue when we came up with this
question, we are currently exploring different ways we might address this.</p>
<h3 id="simd">Single Instruction, Multiple Data (SIMD)</h3>
<p>SIMD, or Single Instruction, Multiple Data, is a type of parallel processing
that allows a single CPU instruction to operate on multiple data points
simultaneously. This facilitates tasks involving large datasets and repetitive
operations, and is often used to optimize performance in fields like game
development, data processing, and scientific computing. In this section of the
survey we wanted to assess respondents&rsquo; needs for native SIMD support in Go.</p>
<p>The majority of respondents (89%) say that work on projects where performance
optimizations are crucial at least some of the time. 40% said they work on such
projects at least half the time. This held true across different organization
sizes and experience levels, suggesting that performance is an important issue
for most developers.</p>
<p><img src="survey2024h2/perf_freq.svg" alt= "How often respondents work on
performance critical software" class="chart" /></p>
<p>About half of respondents (54%), said they are at least a little familiar with
the concept of SIMD. Working with SIMD often requires a deeper understanding of
computer architecture and low-level programming concepts, so unsurprisingly we
find that less experienced developers were less likely to be familiar with SIMD.
Respondents with more experience and who worked on performance-crucial
applications at least half the time were the most likely to be familiar with
SIMD.</p>
<p><img src="survey2024h2/simd_fam.svg" alt= "Familiarity with SIMD" class="chart"
/></p>
<p>For those who were at least slightly familiar with SIMD, we asked some follow
-up questions to understand how respondents were affected by the absence of
native SIMD support in Go. Over a third, about 37%, said they had been impacted.
17% of respondents said they had been limited in the performance they could
achieve in their projects, 15% said they had to use another language instead of
Go to achieve their goals, and 13% said they had to use non-Go libraries when
they would have preferred to use Go libraries. Interestingly, respondents who
were negatively impacted by the absence of native SIMD support were a little
more likely to use Go for data processing and AI/ML. This suggests that adding
SIMD support could make Go a better option for these domains.</p>
<p><img src="survey2024h2/simd_impact.svg" alt= "Impacts of lack of native Go support
for SIMD" class="chart" /> <img src="survey2024h2/what_simd_impact.svg" alt=
"What impacted respondents build with Go" class="chart" /></p>
<h3 id="demographics">Demographics</h3>
<p>We ask similar demographic questions during each cycle of this survey so we can
understand how comparable the year-over-year results may be. For example, if we
saw changes in who responded to the survey in terms of Go experience, it’d be
very likely that other differences in results from prior cycles were due to this
demographic shift. We also use these questions to provide comparisons between
groups, such as satisfaction according to how long respondents have been using
Go.</p>
<p>We didn’t see any significant changes in levels of experience among respondents
during this cycle.</p>
<p><img src="survey2024h2/go_exp.svg" alt= "Experience levels of respondents"
class="chart" /></p>
<p>There are differences in the demographics of respondents according to whether
they came from The Go Blog, the VS Code extension, or GoLand. The population
who responded to survey notifications in VS Code skews toward less experience
with Go; we suspect this a reflection of VS Code’s popularity with new Go
developers, who may not be ready to invest in an IDE license while they’re still
learning. With respect to years of Go experience, the respondents randomly
selected from GoLand are more similar to our self-selected population who found
the survey through the Go Blog. Seeing consistencies between samples allows us
to more confidently generalize findings to the rest of the community.</p>
<p><img src="survey2024h2/go_exp_src.svg" alt= "Experience with Go by survey
source" class="chart" /></p>
<p>In addition to years of experience with Go, we also measured years of
professional coding experience. Our audience tends to be a pretty experienced
bunch, with 26% of respondents having 16 or more years of professional coding
experience.</p>
<p><img src="survey2024h2/dev_exp.svg" alt= "Overall levels of professional
developer experience" class="chart" /></p>
<p>The self-selected group was even more experienced than the randomly selected
groups, with 29% having 16 or more years of professional experience. This
suggests that our self-selected group is generally more experienced than our
randomly selected groups and can help explain some of the differences we see in
this group.</p>
<p><img src="survey2024h2/dev_exp_src.svg" alt= "Levels of professional developer
experience by survey source" class="chart" /></p>
<p>We found that 81% of respondents were fully employed. When we look at our
individual samples, we see a small but significant difference within our
respondents from VS Code, who are slightly more likely to be students. This
makes sense given that VS Code is free.</p>
<img src="survey2024h2/employment.svg" alt= "Employment status" class="chart" />
<img src="survey2024h2/employment_src.svg" alt= "Employment status by survey
source" class="chart" />
<p>Similar to previous years, the most common use cases for Go were API/RPC
services (75%) and command line tools (62%). More experienced Go developers
reported building a wider variety of applications in Go. This trend was
consistent across every category of app or service. We did not find any notable
differences in what respondents are building based on their organization size.
Respondents from the random VS Code and GoLand samples did not display
significant differences either.</p>
<p><img src="survey2024h2/what.svg" alt= "What respondents build with Go"
class="chart" /></p>
<h3 id="firmographics">Firmographics</h3>
<p>We heard from respondents at a variety of different organizations. About 29%
worked at large organizations with 1,001 or more employees, 25% were from
midsize organizations of 101–1,000 employees, and 43% worked at smaller
organizations with fewer than 100 employees. As in previous years, the most
common industry people work in was technology (43%) while the second most common
was financial services (13%).</p>
<p><img src="survey2024h2/org_size.svg" alt= "Organization sizes where respondents
work" class="chart" /> <img src="survey2024h2/industry.svg" alt= "Industries
respondents work in" class="chart" /></p>
<p>As in previous surveys, the most common location for survey respondents was the
United States (19%). This year we saw a significant shift in the proportion of
respondents coming from Ukraine, from 1% to 6%, making it the third most common
location for survey respondents. Because we only saw this difference among our
self-selected respondents, and not in the randomly sampled groups, this suggests
that something affected who discovered the survey, rather than a widespread
increase in Go adoption across all developers in Ukraine. Perhaps there was
increased visibility or awareness of the survey or the Go Blog among developers
in Ukraine.</p>
<p><img src="survey2024h2/location.svg" alt= "Where respondents are located"
class="chart" /></p>
<h2 id="methodology">Methodology</h2>
<p>We announce the survey primarily through the Go Blog, where it is often picked
up on various social channels like Reddit, or Hacker News. We also recruit
respondents by using the VS Code Go plugin to randomly select users to whom we show
a prompt asking if they’d like to participate in the survey. With some help from
our friends at JetBrains, we also have an additional random sample from
prompting a random subset of GoLand users to take the survey. This gave us two
sources we used to compare the self-selected respondents from our traditional
channels and help identify potential effects of <a href="https://en.wikipedia.org/wiki/Self-selection_bias" rel="noreferrer" target="_blank">self-selection
bias</a>.</p>
<p>57% of survey respondents “self-selected” to take the survey, meaning they found
it on the Go blog or other social Go channels. People who don’t follow these
channels are less likely to learn about the survey from them, and in some cases,
they respond differently than people who do closely follow them. For example,
they might be new to the Go community and not yet aware of the Go blog. About
43% of respondents were randomly sampled, meaning they responded to the survey
after seeing a prompt in VS Code (25%) or GoLand (11%). Over the period of
September 9–23, 2024, there was roughly a 10% chance users of the VS Code plugin
would have seen this prompt. The prompt in GoLand was similarly active between
September 9–20. By examining how the randomly sampled groups differ from the
self-selected responses, as well as from each other, we’re able to more
confidently generalize findings to the larger community of Go developers.</p>
<p><img src="survey2024h2/source.svg" alt="Chart of different sources of survey
respondents" class="chart" /></p>
<h4 id="how-to-read-these-results">How to read these results</h4>
<p>Throughout this report we use charts of survey responses to provide supporting
evidence for our findings. All of these charts use a similar format. The title
is the exact question that survey respondents saw. Unless otherwise noted,
questions were multiple choice and participants could only select a single
response choice; each chart’s subtitle will tell the reader if the question
allowed multiple response choices or was an open-ended text box instead of a
multiple choice question. For charts of open-ended text responses, a Go team
member read and manually categorized all of the responses. Many open-ended
questions elicited a wide variety of responses; to keep the chart sizes
reasonable, we condensed them to a maximum of the top 10-12 themes, with
additional themes all grouped under “Other”. The percentage labels shown in
charts are rounded to the nearest integer (e.g., 1.4% and 0.8% will both be
displayed as 1%), but the length of each bar and row ordering are based on the
unrounded values.</p>
<p>To help readers understand the weight of evidence underlying each finding, we
included error bars showing the 95% <a href="https://en.wikipedia.org/wiki/Confidence_interval" rel="noreferrer" target="_blank">confidence
interval</a> for responses;
narrower bars indicate increased confidence. Sometimes two or more responses
have overlapping error bars, which means the relative order of those responses
is not statistically meaningful (i.e., the responses are effectively tied). The
lower right of each chart shows the number of people whose responses are
included in the chart, in the form “n = [number of respondents]”. In cases where
we found interesting differences in responses between groups, (e.g., years of
experience, organization size, or sample source) we showed a color-coded
breakdown of the differences.</p>
<h3 id="closing">Closing</h3>
<p>Thanks for reviewing our semi-annual Go Developer Survey! And many thanks to
everyone who shared their thoughts on Go and everyone who contributed to making
this survey happen. It means the world to us and truly helps us improve Go.</p>
<p>&mdash; Alice (on behalf of the Go team at Google)</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/go1.24">Go 1.24 is released!</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/protobuf-opaque">Go Protobuf: The new Opaque API</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go Protobuf: The new Opaque API</title><id>tag:blog.golang.org,2013:blog.golang.org/protobuf-opaque</id><link rel="alternate" href="https://go.dev/blog/protobuf-opaque"></link><published>2024-12-16T00:00:00+00:00</published><updated>2024-12-16T00:00:00+00:00</updated><author><name>Michael Stapelberg</name></author><summary type="html">We are adding a new generated code API to Go Protobuf.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/protobuf-opaque">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go Protobuf: The new Opaque API</h1>
 
 <p class="author">
 Michael Stapelberg<br>
 16 December 2024
 </p>
 
 <p>[<a href="https://en.wikipedia.org/wiki/Protocol_Buffers" rel="noreferrer" target="_blank">Protocol Buffers (Protobuf)</a>
is Google&rsquo;s language-neutral data interchange format. See
<a href="https://protobuf.dev/" rel="noreferrer" target="_blank">protobuf.dev</a>.]</p>
<p>Back in March 2020, we released the <code>google.golang.org/protobuf</code> module, <a href="/blog/protobuf-apiv2">a
major overhaul of the Go Protobuf API</a>. This
package introduced first-class <a href="https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect" rel="noreferrer" target="_blank">support for
reflection</a>,
a <a href="https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb" rel="noreferrer" target="_blank"><code>dynamicpb</code></a>
implementation and the
<a href="https://pkg.go.dev/google.golang.org/protobuf/testing/protocmp" rel="noreferrer" target="_blank"><code>protocmp</code></a>
package for easier testing.</p>
<p>That release introduced a new protobuf module with a new API. Today, we are
releasing an additional API for generated code, meaning the Go code in the
<code>.pb.go</code> files created by the protocol compiler (<code>protoc</code>). This blog post
explains our motivation for creating a new API and shows you how to use it in
your projects.</p>
<p>To be clear: We are not removing anything. We will continue to support the
existing API for generated code, just like we still support the older protobuf
module (by wrapping the <code>google.golang.org/protobuf</code> implementation). Go is
<a href="/blog/compat">committed to backwards compatibility</a> and this
applies to Go Protobuf, too!</p>
<h2 id="background">Background: the (existing) Open Struct API</h2>
<p>We now call the existing API the Open Struct API, because generated struct types
are open to direct access. In the next section, we will see how it differs from
the new Opaque API.</p>
<p>To work with protocol buffers, you first create a <code>.proto</code> definition file like
this one:</p>
<pre><code>edition = &quot;2023&quot;; // successor to proto2 and proto3

package log;

message LogEntry {
 string backend_server = 1;
 uint32 request_size = 2;
 string ip_address = 3;
}
</code></pre>
<p>Then, you <a href="https://protobuf.dev/getting-started/gotutorial/" rel="noreferrer" target="_blank">run the protocol compiler
(<code>protoc</code>)</a> to generate code
like the following (in a <code>.pb.go</code> file):</p>
<pre><code>package logpb

type LogEntry struct {
 BackendServer *string
 RequestSize *uint32
 IPAddress *string
 // …internal fields elided…
}

func (l *LogEntry) GetBackendServer() string { … }
func (l *LogEntry) GetRequestSize() uint32 { … }
func (l *LogEntry) GetIPAddress() string { … }
</code></pre>
<p>Now you can import the generated <code>logpb</code> package from your Go code and call
functions like
<a href="https://pkg.go.dev/google.golang.org/protobuf/proto#Marshal" rel="noreferrer" target="_blank"><code>proto.Marshal</code></a>
to encode <code>logpb.LogEntry</code> messages into protobuf wire format.</p>
<p>You can find more details in the <a href="https://protobuf.dev/reference/go/go-generated/" rel="noreferrer" target="_blank">Generated Code API
documentation</a>.</p>
<h3 id="presence">(Existing) Open Struct API: Field Presence</h3>
<p>An important aspect of this generated code is how <em>field presence</em> (whether a
field is set or not) is modeled. For instance, the above example models presence
using pointers, so you could set the <code>BackendServer</code> field to:</p>
<ol>
<li><code>proto.String(&quot;zrh01.prod&quot;)</code>: the field is set and contains &ldquo;zrh01.prod&rdquo;</li>
<li><code>proto.String(&quot;&quot;)</code>: the field is set (non-<code>nil</code> pointer) but contains an
empty value</li>
<li><code>nil</code> pointer: the field is not set</li>
</ol>
<p>If you are used to generated code not having pointers, you are probably using
<code>.proto</code> files that start with <code>syntax = &quot;proto3&quot;</code>. The field presence behavior
changed over the years:</p>
<ul>
<li><code>syntax = &quot;proto2&quot;</code> uses <em>explicit presence</em> by default</li>
<li><code>syntax = &quot;proto3&quot;</code> used <em>implicit presence</em> by default (where cases 2 and 3
cannot be distinguished and are both represented by an empty string), but was
later extended to allow <a href="https://protobuf.dev/programming-guides/proto3/#field-labels" rel="noreferrer" target="_blank">opting into explicit presence with the <code>optional</code>
keyword</a></li>
<li><code>edition = &quot;2023&quot;</code>, the <a href="https://protobuf.dev/editions/overview/" rel="noreferrer" target="_blank">successor to both proto2 and
proto3</a>, uses <a href="https://protobuf.dev/programming-guides/field_presence/" rel="noreferrer" target="_blank"><em>explicit
presence</em></a> by default</li>
</ul>
<h2 id="opaqueapi">The new Opaque API</h2>
<p>We created the new <em>Opaque API</em> to uncouple the <a href="https://protobuf.dev/reference/go/go-generated/" rel="noreferrer" target="_blank">Generated Code
API</a> from the underlying
in-memory representation. The (existing) Open Struct API has no such separation:
it allows programs direct access to the protobuf message memory. For example,
one could use the <code>flag</code> package to parse command-line flag values into protobuf
message fields:</p>
<pre><code>var req logpb.LogEntry
flag.StringVar(&amp;req.BackendServer, &quot;backend&quot;, os.Getenv(&quot;HOST&quot;), &quot;…&quot;)
flag.Parse() // fills the BackendServer field from -backend flag
</code></pre>
<p>The problem with such a tight coupling is that we can never change how we lay
out protobuf messages in memory. Lifting this restriction enables many
implementation improvements, which we&rsquo;ll see below.</p>
<p>What changes with the new Opaque API? Here is how the generated code from the
above example would change:</p>
<pre><code>package logpb

type LogEntry struct {
 xxx_hidden_BackendServer *string // no longer exported
 xxx_hidden_RequestSize uint32 // no longer exported
 xxx_hidden_IPAddress *string // no longer exported
 // …internal fields elided…
}

func (l *LogEntry) GetBackendServer() string { … }
func (l *LogEntry) HasBackendServer() bool { … }
func (l *LogEntry) SetBackendServer(string) { … }
func (l *LogEntry) ClearBackendServer() { … }
// …
</code></pre>
<p>With the Opaque API, the struct fields are hidden and can no longer be
directly accessed. Instead, the new accessor methods allow for getting, setting,
or clearing a field.</p>
<h3 id="lessmemory">Opaque structs use less memory</h3>
<p>One change we made to the memory layout is to model field presence for
elementary fields more efficiently:</p>
<ul>
<li>The (existing) Open Struct API uses pointers, which adds a 64-bit word to the
space cost of the field.</li>
<li>The Opaque API uses <a href="https://en.wikipedia.org/wiki/Bit_field" rel="noreferrer" target="_blank">bit
fields</a>, which require one bit per
field (ignoring padding overhead).</li>
</ul>
<p>Using fewer variables and pointers also lowers load on the allocator and on the
garbage collector.</p>
<p>The performance improvement depends heavily on the shapes of your protocol
messages: The change only affects elementary fields like integers, bools, enums,
and floats, but not strings, repeated fields, or submessages (because it is
<a href="https://protobuf.dev/reference/go/opaque-faq/#memorylayout" rel="noreferrer" target="_blank">less
profitable</a>
for those types).</p>
<p>Our benchmark results show that messages with few elementary fields exhibit
performance that is as good as before, whereas messages with more elementary
fields are decoded with significantly fewer allocations:</p>
<pre><code> │ Open Struct API │ Opaque API │
 │ allocs/op │ allocs/op vs base │
Prod#1 360.3k ± 0% 360.3k ± 0% +0.00% (p=0.002 n=6)
Search#1 1413.7k ± 0% 762.3k ± 0% -46.08% (p=0.002 n=6)
Search#2 314.8k ± 0% 132.4k ± 0% -57.95% (p=0.002 n=6)
</code></pre>
<p>Reducing allocations also makes decoding protobuf messages more efficient:</p>
<pre><code> │ Open Struct API │ Opaque API │
 │ user-sec/op │ user-sec/op vs base │
Prod#1 55.55m ± 6% 55.28m ± 4% ~ (p=0.180 n=6)
Search#1 324.3m ± 22% 292.0m ± 6% -9.97% (p=0.015 n=6)
Search#2 67.53m ± 10% 45.04m ± 8% -33.29% (p=0.002 n=6)
</code></pre>
<p>(All measurements done on an AMD Castle Peak Zen 2. Results on ARM and Intel
CPUs are similar.)</p>
<p>Note: proto3 with implicit presence similarly does not use pointers, so you will
not see a performance improvement if you are coming from proto3. If you were
using implicit presence for performance reasons, forgoing the convenience of
being able to distinguish empty fields from unset ones, then the Opaque API now
makes it possible to use explicit presence without a performance penalty.</p>
<h3 id="lazydecoding">Motivation: Lazy Decoding</h3>
<p>Lazy decoding is a performance optimization where the contents of a submessage
are decoded when first accessed instead of during
<a href="https://pkg.go.dev/google.golang.org/protobuf/proto#Unmarshal" rel="noreferrer" target="_blank"><code>proto.Unmarshal</code></a>. Lazy
decoding can improve performance by avoiding unnecessarily decoding fields which
are never accessed.</p>
<p>Lazy decoding can&rsquo;t be supported safely by the (existing) Open Struct API. While
the Open Struct API provides getters, leaving the (un-decoded) struct fields
exposed would be extremely error-prone. To ensure that the decoding logic runs
immediately before the field is first accessed, we must make the field private
and mediate all accesses to it through getter and setter functions.</p>
<p>This approach made it possible to implement lazy decoding with the Opaque
API. Of course, not every workload will benefit from this optimization, but for
those that do benefit, the results can be spectacular: We have seen logs
analysis pipelines that discard messages based on a top-level message condition
(e.g. whether <code>backend_server</code> is one of the machines running a new Linux kernel
version) and can skip decoding deeply nested subtrees of messages.</p>
<p>As an example, here are the results of the micro-benchmark we included,
demonstrating how lazy decoding saves over 50% of the work and over 87% of
allocations!</p>
<pre><code> │ nolazy │ lazy │
 │ sec/op │ sec/op vs base │
Unmarshal/lazy-24 6.742µ ± 0% 2.816µ ± 0% -58.23% (p=0.002 n=6)

 │ nolazy │ lazy │
 │ B/op │ B/op vs base │
Unmarshal/lazy-24 3.666Ki ± 0% 1.814Ki ± 0% -50.51% (p=0.002 n=6)

 │ nolazy │ lazy │
 │ allocs/op │ allocs/op vs base │
Unmarshal/lazy-24 64.000 ± 0% 8.000 ± 0% -87.50% (p=0.002 n=6)
</code></pre>
<h3 id="pointercomparison">Motivation: reduce pointer comparison mistakes</h3>
<p>Modeling field presence with pointers invites pointer-related bugs.</p>
<p>Consider an enum, declared within the <code>LogEntry</code> message:</p>
<pre><code>message LogEntry {
 enum DeviceType {
 DESKTOP = 0;
 MOBILE = 1;
 VR = 2;
 };
 DeviceType device_type = 1;
}
</code></pre>
<p>A simple mistake is to compare the <code>device_type</code> enum field like so:</p>
<pre><code>if cv.DeviceType == logpb.LogEntry_DESKTOP.Enum() { // incorrect!
</code></pre>
<p>Did you spot the bug? The condition compares the memory address instead of the
value. Because the <code>Enum()</code> accessor allocates a new variable on each call, the
condition can never be true. The check should have read:</p>
<pre><code>if cv.GetDeviceType() == logpb.LogEntry_DESKTOP {
</code></pre>
<p>The new Opaque API prevents this mistake: Because fields are hidden, all access
must go through the getter.</p>
<h3 id="accidentalsharing">Motivation: reduce accidental sharing mistakes</h3>
<p>Let&rsquo;s consider a slightly more involved pointer-related bug. Assume you are
trying to stabilize an RPC service that fails under high load. The following
part of the request middleware looks correct, but still the entire service goes
down whenever just one customer sends a high volume of requests:</p>
<pre><code>logEntry.IPAddress = req.IPAddress
logEntry.BackendServer = proto.String(hostname)
// The redactIP() function redacts IPAddress to 127.0.0.1,
// unexpectedly not just in logEntry *but also* in req!
go auditlog(redactIP(logEntry))
if quotaExceeded(req) {
 // BUG: All requests end up here, regardless of their source.
 return fmt.Errorf(&quot;server overloaded&quot;)
}
</code></pre>
<p>Did you spot the bug? The first line accidentally copied the pointer (thereby
sharing the pointed-to variable between the <code>logEntry</code> and <code>req</code> messages)
instead of its value. It should have read:</p>
<pre><code>logEntry.IPAddress = proto.String(req.GetIPAddress())
</code></pre>
<p>The new Opaque API prevents this problem as the setter takes a value
(<code>string</code>) instead of a pointer:</p>
<pre><code>logEntry.SetIPAddress(req.GetIPAddress())
</code></pre>
<h3 id="reflection">Motivation: Fix Sharp Edges: reflection</h3>
<p>To write code that works not only with a specific message type
(e.g. <code>logpb.LogEntry</code>), but with any message type, one needs some kind of
reflection. The previous example used a function to redact IP addresses. To work
with any type of message, it could have been defined as <code>func redactIP(proto.Message) proto.Message { … }</code>.</p>
<p>Many years ago, your only option to implement a function like <code>redactIP</code> was to
reach for <a href="/blog/laws-of-reflection">Go&rsquo;s <code>reflect</code> package</a>,
which resulted in very tight coupling: you had only the generator output and had
to reverse-engineer what the input protobuf message definition might have looked
like. The <a href="/blog/protobuf-apiv2"><code>google.golang.org/protobuf</code> module
release</a> (from March 2020) introduced
<a href="https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect" rel="noreferrer" target="_blank">Protobuf
reflection</a>,
which should always be preferred: Go&rsquo;s <code>reflect</code> package traverses the data
structure&rsquo;s representation, which should be an implementation detail. Protobuf
reflection traverses the logical tree of protocol messages without regard to its
representation.</p>
<p>Unfortunately, merely <em>providing</em> protobuf reflection is not sufficient and
still leaves some sharp edges exposed: In some cases, users might accidentally
use Go reflection instead of protobuf reflection.</p>
<p>For example, encoding a protobuf message with the <code>encoding/json</code> package (which
uses Go reflection) was technically possible, but the result is not <a href="https://protobuf.dev/programming-guides/proto3/#json" rel="noreferrer" target="_blank">canonical
Protobuf JSON
encoding</a>. Use the
<a href="https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson" rel="noreferrer" target="_blank"><code>protojson</code></a>
package instead.</p>
<p>The new Opaque API prevents this problem because the message struct fields are
hidden: accidental usage of Go reflection will see an empty message. This is
clear enough to steer developers towards protobuf reflection.</p>
<h3 id="idealmemory">Motivation: Making the ideal memory layout possible</h3>
<p>The benchmark results from the <a href="#lessmemory">More Efficient Memory
Representation</a> section have already shown that protobuf
performance heavily depends on the specific usage: How are the messages defined?
Which fields are set?</p>
<p>To keep Go Protobuf as fast as possible for <em>everyone</em>, we cannot implement
optimizations that help only one program, but hurt the performance of other
programs.</p>
<p>The Go compiler used to be in a similar situation, up until <a href="/blog/go1.20">Go 1.20 introduced
Profile-Guided Optimization (PGO)</a>. By recording the
production behavior (through <a href="/blog/pprof">profiling</a>) and feeding
that profile back to the compiler, we allow the compiler to make better
trade-offs <em>for a specific program or workload</em>.</p>
<p>We think using profiles to optimize for specific workloads is a promising
approach for further Go Protobuf optimizations. The Opaque API makes those
possible: Program code uses accessors and does not need to be updated when the
memory representation changes, so we could, for example, move rarely set fields
into an overflow struct.</p>
<h2 id="migration">Migration</h2>
<p>You can migrate on your own schedule, or even not at all—the (existing) Open
Struct API will not be removed. But, if you’re not on the new Opaque API, you
won’t benefit from its improved performance, or future optimizations that target
it.</p>
<p>We recommend you select the Opaque API for new development. Protobuf Edition
2024 (see <a href="https://protobuf.dev/editions/overview/" rel="noreferrer" target="_blank">Protobuf Editions Overview</a>
if you are not yet familiar) will make the Opaque API the default.</p>
<h3 id="hybridapi">The Hybrid API</h3>
<p>Aside from the Open Struct API and Opaque API, there is also the Hybrid API,
which keeps existing code working by keeping struct fields exported, but also
enabling migration to the Opaque API by adding the new accessor methods.</p>
<p>With the Hybrid API, the protobuf compiler will generate code on two API levels:
the <code>.pb.go</code> is on the Hybrid API, whereas the <code>_protoopaque.pb.go</code> version is
on the Opaque API and can be selected by building with the <code>protoopaque</code> build
tag.</p>
<h3 id="rewriting">Rewriting Code to the Opaque API</h3>
<p>See the <a href="https://protobuf.dev/reference/go/opaque-migration/" rel="noreferrer" target="_blank">migration
guide</a>
for detailed instructions. The high-level steps are:</p>
<ol>
<li>Enable the Hybrid API.</li>
<li>Update existing code using the <code>open2opaque</code> migration tool.</li>
<li>Switch to the Opaque API.</li>
</ol>
<h3 id="publishing">Advice for published generated code: Use Hybrid API</h3>
<p>Small usages of protobuf can live entirely within the same repository, but
usually, <code>.proto</code> files are shared between different projects that are owned by
different teams. An obvious example is when different companies are involved: To
call Google APIs (with protobuf), use the <a href="https://github.com/googleapis/google-cloud-go" rel="noreferrer" target="_blank">Google Cloud Client Libraries for
Go</a> from your project. Switching
the Cloud Client Libraries to the Opaque API is not an option, as that would be
a breaking API change, but switching to the Hybrid API is safe.</p>
<p>Our advice for such packages that publish generated code (<code>.pb.go</code> files) is to
switch to the Hybrid API please! Publish both the <code>.pb.go</code> and the
<code>_protoopaque.pb.go</code> files, please. The <code>protoopaque</code> version allows your
consumers to migrate on their own schedule.</p>
<h3 id="enablelazy">Enabling Lazy Decoding</h3>
<p>Lazy decoding is available (but not enabled) once you migrate to the Opaque API!
🎉</p>
<p>To enable: in your <code>.proto</code> file, annotate your message-typed fields with the
<code>[lazy = true]</code> annotation.</p>
<p>To opt out of lazy decoding (despite <code>.proto</code> annotations), the <a href="https://pkg.go.dev/google.golang.org/protobuf/runtime/protolazy" rel="noreferrer" target="_blank"><code>protolazy</code>
package
documentation</a>
describes the available opt-outs, which affect either an individual Unmarshal
operation or the entire program.</p>
<h2 id="nextsteps">Next Steps</h2>
<p>By using the open2opaque tool in an automated fashion over the last few years,
we have converted the vast majority of Google’s <code>.proto</code> files and Go code to
the Opaque API. We continuously improved the Opaque API implementation as we
moved more and more production workloads to it.</p>
<p>Therefore, we expect you should not encounter problems when trying the Opaque
API. In case you do encounter any issues after all, please <a href="https://github.com/golang/protobuf/issues/" rel="noreferrer" target="_blank">let us know on the
Go Protobuf issue tracker</a>.</p>
<p>Reference documentation for Go Protobuf can be found on <a href="https://protobuf.dev/reference/go/" rel="noreferrer" target="_blank">protobuf.dev → Go
Reference</a>.</p>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/survey2024-h2-results">Go Developer Survey 2024 H2 Results</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/15years">Go Turns 15</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry></feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom

Home · About · News · Docs · Terms