FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

line 1, column 10201: content should not contain data-slug attribute (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

line 1, column 10201: content should not contain script tag (10 occurrences) [help]

... lay.js&#34;&gt;&lt;/script&gt;&#xA;&#xA;</content></entry><entry><title> ...
                                             ^

Source: http://blog.golang.org/feed.atom

<feed xmlns="http://www.w3.org/2005/Atom"><title>The Go Blog</title><id>tag:blog.golang.org,2013:blog.golang.org</id><link rel="self" href="https://go.dev/blog/feed.atom"></link><updated>2025-09-16T00:00:00+00:00</updated><entry><title>It's survey time! How has Go has been working out for you?</title><id>tag:blog.golang.org,2013:blog.golang.org/survey2025-announce</id><link rel="alternate" href="https://go.dev/blog/survey2025-announce"></link><published>2025-09-16T00:00:00+00:00</published><updated>2025-09-16T00:00:00+00:00</updated><author><name>Todd Kulesza, on behalf of the Go team</name></author><summary type="html">Help shape the future of Go</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/survey2025-announce">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>It&#39;s survey time! How has Go has been working out for you?</h1>
 
 <p class="author">
 Todd Kulesza, on behalf of the Go team<br>
 16 September 2025
 </p>
 
 <div class='markdown'>
<p>Hi Gophers! Today we&rsquo;re excited to announce our <a href="https://google.qualtrics.com/jfe/form/SV_3wwSstC8vv4Ymkm?s=b" rel="noreferrer" target="_blank">2025 Go Developer Survey</a>. The Go Team uses the results of this annual survey to better understand the needs and concerns of Go developers across the world. Your feedback helps us brainstorm, plan, and prioritize work on Go.</p>
<p>You can <a href="https://google.qualtrics.com/jfe/form/SV_3wwSstC8vv4Ymkm?s=b" rel="noreferrer" target="_blank">take the survey here</a>. It will be open through <strong>September 30th</strong>. The survey should take 10 – 20 minutes to complete, and every question is optional.</p>
<p>We&rsquo;ll share aggregated survey results on this blog in early November. This year we&rsquo;ll also share the raw dataset of survey responses, so that the entire Go community can benefit from this knowledge and conduct your own analyses on the data. Similar to our approach with <a href="/blog/gotelemetry">Go Telemetry</a>, we&rsquo;re using an opt-in model: the survey will ask for your permission to include your responses in the dataset. If you don&rsquo;t give us permission, your survey responses will not be shared.</p>
<p>Please help us reach as many Go developers as possible! We love it when you share the survey with your colleagues, friends, and online communities. The more voices we hear, the better we can understand the diverse needs of Go developers everywhere.</p>
<p>We&rsquo;re looking forward to hearing your feedback!</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 <p>
 
 
 
 <b>Previous article: </b><a href="/blog/jsonv2-exp">A new experimental Go API for JSON</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>A new experimental Go API for JSON</title><id>tag:blog.golang.org,2013:blog.golang.org/jsonv2-exp</id><link rel="alternate" href="https://go.dev/blog/jsonv2-exp"></link><published>2025-09-09T00:00:00+00:00</published><updated>2025-09-09T00:00:00+00:00</updated><author><name>Joe Tsai, Daniel Martí, Johan Brandhorst-Satzkorn, Roger Peppe, Chris Hines, and Damien Neil</name></author><summary type="html">Go 1.25 introduces experimental support for encoding/json/jsontext and encoding/json/v2 packages.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/jsonv2-exp">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>A new experimental Go API for JSON</h1>
 
 <p class="author">
 Joe Tsai, Daniel Martí, Johan Brandhorst-Satzkorn, Roger Peppe, Chris Hines, and Damien Neil<br>
 9 September 2025
 </p>
 
 <div class='markdown'>
<h2 id="introduction">Introduction</h2>
<p><a href="https://datatracker.ietf.org/doc/html/rfc8259" rel="noreferrer" target="_blank">JavaScript Object Notation (JSON)</a>
is a simple data interchange format. Almost 15 years ago,
we wrote about <a href="/blog/json">support for JSON in Go</a>,
which introduced the ability to serialize and deserialize Go types to and from JSON data.
Since then, JSON has become the most popular data format used on the Internet.
It is widely read and written by Go programs,
and encoding/json now ranks as the 5th most imported Go package.</p>
<p>Over time, packages evolve with the needs of their users,
and <code>encoding/json</code> is no exception. This blog post is about Go 1.25’s new
experimental <code>encoding/json/v2</code> and <code>encoding/json/jsontext</code> packages,
which bring long-awaited improvements and fixes.
This post argues for a new major API version,
provides an overview of the new packages,
and explains how you can make use of it.
The experimental packages are not visible by default and
may undergo future API changes.</p>
<h2 id="problems-with-encodingjson">Problems with <code>encoding/json</code></h2>
<p>Overall, <code>encoding/json</code> has held up well.
The idea of marshaling and unmarshaling arbitrary Go types
with some default representation in JSON, combined with the ability to
customize the representation, has proven to be highly flexible.
However, in the years since its introduction,
various users have identified numerous shortcomings.</p>
<h3 id="behavior-flaws">Behavior flaws</h3>
<p>There are various behavioral flaws in <code>encoding/json</code>:</p>
<ul>
<li>
<p><strong>Imprecise handling of JSON syntax</strong>: Over the years, JSON has seen
increased standardization in order for programs to properly communicate.
Generally, decoders have become stricter at rejecting ambiguous inputs,
to reduce the chance that two implementations will have different
(successful) interpretations of a particular JSON value.</p>
<ul>
<li>
<p><code>encoding/json</code> currently accepts invalid UTF-8,
whereas the latest Internet Standard (RFC 8259) requires valid UTF-8.
The default behavior should report an error in the presence of invalid UTF-8,
instead of introducing silent data corruption,
which may cause problems downstream.</p>
</li>
<li>
<p><code>encoding/json</code> currently accepts objects with duplicate member names.
RFC 8259 does not specify how to handle duplicate names,
so an implementation is free to choose an arbitrary value,
merge the values, discard the values, or report an error.
The presence of a duplicate name results in a JSON value
without a universally agreed upon meaning.
This could be <a href="https://www.youtube.com/watch?v=avilmOcHKHE&amp;t=1057s" rel="noreferrer" target="_blank">exploited by attackers in security applications</a>
and has been exploited before (as in <a href="https://nvd.nist.gov/vuln/detail/CVE-2017-12635" rel="noreferrer" target="_blank">CVE-2017-12635</a>).
The default behavior should err on the side of safety and reject duplicate names.</p>
</li>
</ul>
</li>
<li>
<p><strong>Leaking nilness of slices and maps</strong>: JSON is often used to communicate with
programs using JSON implementations that do not allow <code>null</code> to be unmarshaled
into a data type expected to be a JSON array or object.
Since <code>encoding/json</code> marshals a nil slice or map as a JSON <code>null</code>,
this may lead to errors when unmarshaling by other implementations.
<a href="/issue/63397#discussioncomment-7201222">A survey</a>
indicated that most Go users prefer that nil slices and maps
are marshaled as an empty JSON array or object by default.</p>
</li>
<li>
<p><strong>Case-insensitive unmarshaling</strong>: When unmarshaling, a JSON object member name
is resolved to a Go struct field name using a case-insensitive match.
This is a surprising default, a potential security vulnerability, and a performance limitation.</p>
</li>
<li>
<p><strong>Inconsistent calling of methods</strong>: Due to an implementation detail,
<code>MarshalJSON</code> methods declared on a pointer receiver
are <a href="/issue/22967">inconsistently called by <code>encoding/json</code></a>. While regarded as a bug,
this cannot be fixed as too many applications depend on the current behavior.</p>
</li>
</ul>
<h3 id="api-deficiencies">API deficiencies</h3>
<p>The API of <code>encoding/json</code> can be tricky or restrictive:</p>
<ul>
<li>
<p>It is difficult to correctly unmarshal from an <code>io.Reader</code>.
Users often write <code>json.NewDecoder(r).Decode(v)</code>,
which fails to reject trailing junk at the end of the input.</p>
</li>
<li>
<p>Options can be set on the <code>Encoder</code> and <code>Decoder</code> types,
but cannot be used with the <code>Marshal</code> and <code>Unmarshal</code> functions.
Similarly, types implementing the <code>Marshaler</code> and <code>Unmarshaler</code> interfaces
cannot make use of the options and there is no way to plumb options down the call stack.
For example, the <code>Decoder.DisallowUnknownFields</code> option loses its effect
when calling a custom <code>UnmarshalJSON</code> method.</p>
</li>
<li>
<p>The <code>Compact</code>, <code>Indent</code>, and <code>HTMLEscape</code> functions write to a <code>bytes.Buffer</code>
instead of something more flexible like a <code>[]byte</code> or <code>io.Writer</code>.
This limits the usability of those functions.</p>
</li>
</ul>
<h3 id="performance-limitations">Performance limitations</h3>
<p>Setting aside internal implementation details,
the public API commits it to certain performance limitations:</p>
<ul>
<li>
<p><strong>MarshalJSON</strong>: The <code>MarshalJSON</code> interface method forces the implementation
to allocate the returned <code>[]byte</code>. Also, the semantics require that
<code>encoding/json</code> verify that the result is valid JSON
and also to reformat it to match the specified indentation.</p>
</li>
<li>
<p><strong>UnmarshalJSON</strong>: The <code>UnmarshalJSON</code> interface method requires that
a complete JSON value be provided (without any trailing data).
This forces <code>encoding/json</code> to parse the JSON value to be unmarshaled
in its entirety to determine where it ends before it can call <code>UnmarshalJSON</code>.
Afterwards, the <code>UnmarshalJSON</code> method itself must parse the provided JSON value again.</p>
</li>
<li>
<p><strong>Lack of streaming</strong>: Even though the <code>Encoder</code> and <code>Decoder</code> types operate
on an <code>io.Writer</code> or <code>io.Reader</code>, they buffer the entire JSON value in memory.
The <code>Decoder.Token</code> method for reading individual tokens is allocation-heavy
and there is no corresponding API for writing tokens.</p>
</li>
</ul>
<p>Furthermore, if the implementation of a <code>MarshalJSON</code> or <code>UnmarshalJSON</code> method
recursively calls the <code>Marshal</code> or <code>Unmarshal</code> function,
then the performance becomes quadratic.</p>
<h2 id="trying-to-fix-encodingjson-directly">Trying to fix <code>encoding/json</code> directly</h2>
<p>Introducing a new, incompatible major version of a package is a heavy consideration.
If possible, we should try to fix the existing package.</p>
<p>While it is relatively easy to add new features,
it is difficult to change existing features.
Unfortunately, these problems are inherent consequences of the existing API,
making them practically impossible to fix within the <a href="/doc/go1compat">Go 1 compatibility promise</a>.</p>
<p>We could in principle declare separate names, such as <code>MarshalV2</code> or <code>UnmarshalV2</code>,
but that is tantamount to creating a parallel namespace within the same package.
This leads us to <code>encoding/json/v2</code> (henceforth called <code>v2</code>),
where we can make these changes within a separate <code>v2</code> namespace
in contrast to <code>encoding/json</code> (henceforth called <code>v1</code>).</p>
<h2 id="planning-for-encodingjsonv2">Planning for <code>encoding/json/v2</code></h2>
<p>The planning for a new major version of <code>encoding/json</code> spanned years.
In late 2020, spurred on by the inability to fix issues in the current package,
Daniel Martí (one of the maintainers of <code>encoding/json</code>) first drafted his
thoughts on <a href="https://docs.google.com/document/d/1WQGoM44HLinH4NGBEv5drGlw5_RNW-GP7DdGEpm7Y3o" rel="noreferrer" target="_blank">what a hypothetical <code>v2</code> package should look like</a>.
Separately, after previous work on the <a href="/blog/protobuf-apiv2">Go API for Protocol Buffers</a>,
Joe Tsai was disapppointed that <a href="/pkg/google.golang.org/protobuf/encoding/protojson">the <code>protojson</code> package</a>
needed to use a custom JSON implementation because <code>encoding/json</code> was
neither capable of adhering to the stricter JSON standard that the
Protocol Buffer specification required,
nor of efficiently serializing JSON in a streaming manner.</p>
<p>Believing a brighter future for JSON was both beneficial and achievable,
Daniel and Joe joined forces to brainstorm on <code>v2</code> and
<a href="https://github.com/go-json-experiment/json" rel="noreferrer" target="_blank">started to build a prototype</a>
(with the initial code being a polished version of the JSON serialization logic from the Go protobuf module).
Over time, a few others (Roger Peppe, Chris Hines, Johan Brandhorst-Satzkorn, and Damien Neil)
joined the effort by providing design review, code review, and regression testing.
Many of the early discussions are publicly available in our
<a href="https://www.youtube.com/playlist?list=PLZgrQPcV8W8EChkaAvv-3NUu6PYmnGG3b" rel="noreferrer" target="_blank">recorded meetings</a> and
<a href="https://docs.google.com/document/d/1rovrOTd-wTawGMPPlPuKhwXaYBg9VszTXR9AQQL5LfI" rel="noreferrer" target="_blank">meeting notes</a>.</p>
<p>This work has been public since the beginning,
and we increasingly involved the wider Go community,
first with a
<a href="https://www.youtube.com/watch?v=avilmOcHKHE" rel="noreferrer" target="_blank">GopherCon talk</a> and
<a href="/issue/63397">discussion posted in late 2023</a>,
<a href="/issue/71497">formal proposal posted in early 2025</a>,
and most recently <a href="/issue/71845">adopting <code>encoding/json/v2</code> as a Go experiment</a>
(available in Go 1.25) for wider-scale testing by all Go users.</p>
<p>The <code>v2</code> effort has been going on for 5 years,
incorporating feedback from many contributors and also gaining valuable
empirical experience from use in production settings.</p>
<p>It&rsquo;s worth noting that it&rsquo;s largely been developed and promoted by people
not employed by Google, demonstrating that the Go project is a collaborative endeavor
with a thriving global community dedicated to improving the Go ecosystem.</p>
<h2 id="building-on-encodingjsonjsontext">Building on <code>encoding/json/jsontext</code></h2>
<p>Before discussing the <code>v2</code> API, we first introduce the experimental
<a href="/pkg/encoding/json/jsontext"><code>encoding/json/jsontext</code></a> package
that lays the foundation for future improvements to JSON in Go.</p>
<p>JSON serialization in Go can be broken down into two primary components:</p>
<ul>
<li><em>syntactic functionality</em> that is concerned with processing JSON based on its grammar, and</li>
<li><em>semantic functionality</em> that defines the relationship between JSON values and Go values.</li>
</ul>
<p>We use the terms &ldquo;encode&rdquo; and &ldquo;decode&rdquo; to describe syntactic functionality and
the terms &ldquo;marshal&rdquo; and &ldquo;unmarshal&rdquo; to describe semantic functionality.
We aim to provide a clear distinction between functionality
that is purely concerned with encoding versus that of marshaling.</p>
<img src="jsonv2-exp/api.png" width=100%>
<p>This diagram provides an overview of this separation.
Purple blocks represent types, while blue blocks represent functions or methods.
The direction of the arrows approximately represents the flow of data.
The bottom half of the diagram, implemented by the <code>jsontext</code> package,
contains functionality that is only concerned with syntax,
while the upper half, implemented by the <code>json/v2</code> package,
contains functionality that assigns semantic meaning to syntactic data
handled by the bottom half.</p>
<p>The basic API of <code>jsontext</code> is the following:</p>
<pre><code>package jsontext

type Encoder struct { ... }
func NewEncoder(io.Writer, ...Options) *Encoder
func (*Encoder) WriteValue(Value) error
func (*Encoder) WriteToken(Token) error

type Decoder struct { ... }
func NewDecoder(io.Reader, ...Options) *Decoder
func (*Decoder) ReadValue() (Value, error)
func (*Decoder) ReadToken() (Token, error)

type Kind byte
type Value []byte
func (Value) Kind() Kind
type Token struct { ... }
func (Token) Kind() Kind
</code></pre>
<p>The <code>jsontext</code> package provides functionality for interacting with JSON
at a syntactic level and derives its name from
<a href="https://datatracker.ietf.org/doc/html/rfc8259#section-2" rel="noreferrer" target="_blank">RFC 8259, section 2</a>
where the grammar for JSON data is literally called <code>JSON-text</code>.
Since it only interacts with JSON at a syntactic level,
it does not depend on Go reflection.</p>
<p>The <a href="/pkg/encoding/json/jsontext#Encoder"><code>Encoder</code></a> and
<a href="/pkg/encoding/json/jsontext#Decoder"><code>Decoder</code></a>
provide support for encoding and decoding JSON values and tokens.
The constructors
<a href="/pkg/encoding/json/jsontext#Options">accept variadic options</a>
that affect the particular behavior of encoding and decoding.
Unlike the <code>Encoder</code> and <code>Decoder</code> types declared in <code>v1</code>,
the new types in <code>jsontext</code> avoid muddling the distinction between syntax and
semantics and operate in a truly streaming manner.</p>
<p>A JSON value is a complete unit of data and is represented in Go as
<a href="/pkg/encoding/json/jsontext#Value">a named <code>[]byte</code></a>.
It is identical to <a href="/pkg/encoding/json#RawMessage"><code>RawMessage</code></a> in <code>v1</code>.
A JSON value is syntactically composed of one or more JSON tokens.
A JSON token is represented in Go as the <a href="/pkg/encoding/json/jsontext#Token">opaque <code>Token</code> type</a>
with constructors and accessor methods.
It is analogous to <a href="/pkg/encoding/json#Token"><code>Token</code></a> in <code>v1</code>
but is designed represent arbitrary JSON tokens without allocation.</p>
<p>To resolve the fundamental performance problems with
the <code>MarshalJSON</code> and <code>UnmarshalJSON</code> interface methods,
we need an efficient way of encoding and decoding JSON
as a streaming sequence of tokens and values.
In <code>v2</code>, we introduce the <code>MarshalJSONTo</code> and <code>UnmarshalJSONFrom</code> interface methods
that operate on an <code>Encoder</code> or <code>Decoder</code>, allowing the methods&rsquo; implementations
to process JSON in a purely streaming manner. Thus, the <code>json</code> package need not
be responsible for validating or formatting a JSON value returned by <code>MarshalJSON</code>,
nor would it need to be responsible for determining the boundaries of a JSON value
provided to <code>UnmarshalJSON</code>. These responsibilities belong to the <code>Encoder</code> and <code>Decoder</code>.</p>
<h2 id="introducing-encodingjsonv2">Introducing <code>encoding/json/v2</code></h2>
<p>Building on the <code>jsontext</code> package, we now introduce the experimental
<a href="/pkg/encoding/json/v2"><code>encoding/json/v2</code></a> package.
It is designed to fix the aforementioned problems,
while remaining familiar to users of the <code>v1</code> package.
Our goal is that usages of <code>v1</code> will operate <em>mostly</em> the same if directly migrated to <code>v2</code>.</p>
<p>In this article, we will primarily cover the high-level API of <code>v2</code>.
For examples on how to use it, we encourage readers to
study <a href="/pkg/encoding/json/v2#pkg-examples">the examples in the <code>v2</code> package</a> or
read <a href="https://antonz.org/go-json-v2/" rel="noreferrer" target="_blank">Anton Zhiyanov&rsquo;s blog covering the topic</a>.</p>
<p>The basic API of <code>v2</code> is the following:</p>
<pre><code>package json

func Marshal(in any, opts ...Options) (out []byte, err error)
func MarshalWrite(out io.Writer, in any, opts ...Options) error
func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error

func Unmarshal(in []byte, out any, opts ...Options) error
func UnmarshalRead(in io.Reader, out any, opts ...Options) error
func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error
</code></pre>
<p>The <a href="/pkg/encoding/json/v2#Marshal"><code>Marshal</code></a>
and <a href="/pkg/encoding/json/v2#Unmarshal"><code>Unmarshal</code></a> functions
have a signature similar to <code>v1</code>, but accept options to configure their behavior.
The <a href="/pkg/encoding/json/v2#MarshalWrite"><code>MarshalWrite</code></a>
and <a href="/pkg/encoding/json/v2#UnmarshalRead"><code>UnmarshalRead</code></a> functions
directly operate on an <code>io.Writer</code> or <code>io.Reader</code>,
avoiding the need to temporarily construct an <code>Encoder</code> or <code>Decoder</code>
just to write or read from such types.
The <a href="/pkg/encoding/json/v2#MarshalEncode"><code>MarshalEncode</code></a>
and <a href="/pkg/encoding/json/v2#UnmarshalDecode"><code>UnmarshalDecode</code></a> functions
operate on a <code>jsontext.Encoder</code> and <code>jsontext.Decoder</code> and
is actually the underlying implementation of the previously mentioned functions.
Unlike <code>v1</code>, options are a first-class argument to each of the marshal and unmarshal functions,
greatly extending the flexibility and configurability of <code>v2</code>.
There are <a href="/pkg/encoding/json/v2#Options">several options available</a>
in <code>v2</code> which are not covered by this article.</p>
<h3 id="type-specified-customization">Type-specified customization</h3>
<p>Similar to <code>v1</code>, <code>v2</code> allows types to define their own JSON representation
by satisfying particular interfaces.</p>
<pre><code>type Marshaler interface {
 MarshalJSON() ([]byte, error)
}
type MarshalerTo interface {
 MarshalJSONTo(*jsontext.Encoder) error
}

type Unmarshaler interface {
 UnmarshalJSON([]byte) error
}
type UnmarshalerFrom interface {
 UnmarshalJSONFrom(*jsontext.Decoder) error
}
</code></pre>
<p>The <a href="/pkg/encoding/json/v2#Marshaler"><code>Marshaler</code></a>
and <a href="/pkg/encoding/json/v2#Unmarshaler"><code>Unmarshaler</code></a> interfaces
are identical to those in <code>v1</code>.
The new <a href="/pkg/encoding/json/v2#MarshalerTo"><code>MarshalerTo</code></a>
and <a href="/pkg/encoding/json/v2#UnmarshalerFrom"><code>UnmarshalerFrom</code></a> interfaces
allow a type to represent itself as JSON using a <code>jsontext.Encoder</code> or <code>jsontext.Decoder</code>.
This allows options to be forwarded down the call stack, since options
can be retrieved via the <code>Options</code> accessor method on the <code>Encoder</code> or <code>Decoder</code>.</p>
<p>See <a href="/pkg/encoding/json/v2#example-package-OrderedObject">the <code>OrderedObject</code> example</a>
for how to implement a custom type that maintains the ordering of JSON object members.</p>
<h3 id="caller-specified-customization">Caller-specified customization</h3>
<p>In <code>v2</code>, the caller of <code>Marshal</code> and <code>Unmarshal</code> can also specify
a custom JSON representation for any arbitrary type,
where caller-specified functions take precedence over type-defined methods
or the default representation for a particular type.</p>
<pre><code>func WithMarshalers(*Marshalers) Options

type Marshalers struct { ... }
func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers
func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers

func WithUnmarshalers(*Unmarshalers) Options

type Unmarshalers struct { ... }
func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers
func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers
</code></pre>
<p><a href="/pkg/encoding/json/v2#MarshalFunc"><code>MarshalFunc</code></a> and
<a href="/pkg/encoding/json/v2#MarshalToFunc"><code>MarshalToFunc</code></a>
construct a custom marshaler that can be passed to a <code>Marshal</code> call
using <code>WithMarshalers</code> to override the marshaling of particular types.
Similarly,
<a href="/pkg/encoding/json/v2#UnmarshalFunc"><code>UnmarshalFunc</code></a> and
<a href="/pkg/encoding/json/v2#UnmarshalFromFunc"><code>UnmarshalFromFunc</code></a>
support similar customization for <code>Unmarshal</code>.</p>
<p><a href="/pkg/encoding/json/v2#example-package-ProtoJSON">The <code>ProtoJSON</code> example</a>
demonstrates how this feature allows serialization of all
<a href="/pkg/google.golang.org/protobuf/proto#Message"><code>proto.Message</code></a> types
to be handled by the <a href="/pkg/google.golang.org/protobuf/encoding/protojson"><code>protojson</code></a> package.</p>
<h3 id="behavior-differences">Behavior differences</h3>
<p>While <code>v2</code> aims to behave <em>mostly</em> the same as <code>v1</code>,
its behavior has changed <a href="/pkg/github.com/go-json-experiment/json/v1#hdr-Migrating_to_v2">in some ways</a>
to address problems in <code>v1</code>, most notably:</p>
<ul>
<li>
<p><code>v2</code> reports an error in the presence of invalid UTF-8.</p>
</li>
<li>
<p><code>v2</code> reports an error if a JSON object contains a duplicate name.</p>
</li>
<li>
<p><code>v2</code> marshals a nil Go slice or Go map as an empty JSON array or JSON object, respectively.</p>
</li>
<li>
<p><code>v2</code> unmarshals a JSON object into a Go struct using a
case-sensitive match from the JSON member name to the Go field name.</p>
</li>
<li>
<p><code>v2</code> redefines the <code>omitempty</code> tag option to omit a field if it would have
encoded as an &ldquo;empty&rdquo; JSON value (which are <code>null</code>, <code>&quot;&quot;</code>, <code>[]</code>, and <code>{}</code>).</p>
</li>
<li>
<p><code>v2</code> reports an error when trying to serialize a <code>time.Duration</code>,
which currently has <a href="/issue/71631">no default representation</a>,
but provides options to let the caller decide.</p>
</li>
</ul>
<p>For most behavior changes, there is a struct tag option or caller-specified option
that can configure the behavior to operate under <code>v1</code> or <code>v2</code> semantics,
or even other caller-determined behavior.
See <a href="/pkg/github.com/go-json-experiment/json/v1#hdr-Migrating_to_v2">&ldquo;Migrating to v2&rdquo;</a> for more information.</p>
<h3 id="performance-optimizations">Performance optimizations</h3>
<p>The <code>Marshal</code> performance of <code>v2</code> is roughly at parity with <code>v1</code>.
Sometimes it is slightly faster, but other times it is slightly slower.
The <code>Unmarshal</code> performance of <code>v2</code> is significantly faster than <code>v1</code>,
with benchmarks demonstrating improvements of up to 10x.</p>
<p>In order to obtain greater performance gains,
existing implementations of
<a href="/pkg/encoding/json/v2#Marshaler"><code>Marshaler</code></a> and
<a href="/pkg/encoding/json/v2#Unmarshaler"><code>Unmarshaler</code></a> should
migrate to also implement
<a href="/pkg/encoding/json/v2#MarshalerTo"><code>MarshalerTo</code></a> and
<a href="/pkg/encoding/json/v2#UnmarshalerFrom"><code>UnmarshalerFrom</code></a>,
so that they can benefit from processing JSON in a purely streaming manner.
For example, recursive parsing of OpenAPI specifications in <code>UnmarshalJSON</code> methods
significantly hurt performance in a particular service of Kubernetes
(see <a href="https://github.com/kubernetes/kube-openapi/issues/315" rel="noreferrer" target="_blank">kubernetes/kube-openapi#315</a>),
while switching to <code>UnmarshalJSONFrom</code> improved performance by orders of magnitude.</p>
<p>For more information, see the
<a href="https://github.com/go-json-experiment/jsonbench" rel="noreferrer" target="_blank"><code>go-json-experiment/jsonbench</code></a>
repository.</p>
<h2 id="retroactively-improving-encodingjson">Retroactively improving <code>encoding/json</code></h2>
<p>We want to avoid two separate JSON implementations in the Go standard library,
so it is critical that, under the hood, <code>v1</code> is implemented in terms of <code>v2</code>.</p>
<p>There are several benefits to this approach:</p>
<ol>
<li>
<p><strong>Gradual migration</strong>: The <code>Marshal</code> and <code>Unmarshal</code> functions in <code>v1</code> or <code>v2</code>
represent a set of default behaviors that operate according to <code>v1</code> or <code>v2</code> semantics.
Options can be specified that configure <code>Marshal</code> or <code>Unmarshal</code> to operate with
entirely <code>v1</code>, mostly <code>v1</code> with a some <code>v2</code>, a mix of <code>v1</code> or <code>v2</code>,
mostly <code>v2</code> with some <code>v1</code>, or entirely <code>v2</code> semantics.
This allows for gradual migration between the default behaviors of the two versions.</p>
</li>
<li>
<p><strong>Feature inheritance</strong>: As backward-compatible features are added to <code>v2</code>,
they will inherently be made available in <code>v1</code>. For example, <code>v2</code> adds
support for several new struct tag options such as <code>inline</code> or <code>format</code> and also
support for the <code>MarshalJSONTo</code> and <code>UnmarshalJSONFrom</code> interface methods,
which are both more performant and flexible.
When <code>v1</code> is implemented in terms of <code>v2</code>, it will inherit support for these features.</p>
</li>
<li>
<p><strong>Reduced maintenance</strong>: Maintenance of a widely used package demands significant effort.
By having <code>v1</code> and <code>v2</code> use the same implementation, the maintenance burden is reduced.
In general, a single change will fix bugs, improve performance, or add functionality to both versions.
There is no need to backport a <code>v2</code> change with an equivalent <code>v1</code> change.</p>
</li>
</ol>
<p>While select parts of <code>v1</code> may be deprecated over time (supposing <code>v2</code> graduates from being an experiment),
the package as a whole will never be deprecated.
Migrating to <code>v2</code> will be encouraged, but not required.
The Go project will not drop support for <code>v1</code>.</p>
<h2 id="experimenting-with-jsonv2">Experimenting with <code>jsonv2</code></h2>
<p>The newer API in the <code>encoding/json/jsontext</code> and <code>encoding/json/v2</code> packages are not visible by default.
To use them, build your code with <code>GOEXPERIMENT=jsonv2</code> set in your environment or with the <code>goexperiment.jsonv2</code> build tag.
The nature of an experiment is that the API is unstable and may change in the future.
Though the API is unstable, the implementation is of a high quality and
has been successfully used in production by several major projects.</p>
<p>The fact that <code>v1</code> is implemented in terms of <code>v2</code> means that the underlying implementation of <code>v1</code>
is completely different when building under the <code>jsonv2</code> experiment.
Without changing any code, you should be able to run your tests
under <code>jsonv2</code> and theoretically nothing new should fail:</p>
<pre><code>GOEXPERIMENT=jsonv2 go test ./...
</code></pre>
<p>The re-implementation of <code>v1</code> in terms of <code>v2</code> aims to provide identical behavior
within the bounds of the <a href="/doc/go1compat">Go 1 compatibility promise</a>,
though some differences might be observable such as the exact wording of error messages.
We encourage you to run your tests under <code>jsonv2</code> and
report any regressions <a href="/issues">on the issue tracker</a>.</p>
<p>Becoming an experiment in Go 1.25 is a significant milestone on the road to
formally adopting <code>encoding/json/jsontext</code> and <code>encoding/json/v2</code> into the standard library.
However, the purpose of the <code>jsonv2</code> experiment is to gain broader experience.
Your feedback will determine our next steps, and the outcome of this experiment,
which may result in anything from abandonment of the effort, to adoption as stable packages of Go 1.26.
Please share your experience on <a href="/issue/71497">go.dev/issue/71497</a>, and help determine the future of Go.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/survey2025-announce">It&#39;s survey time! How has Go has been working out for you?</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/testing-time">Testing Time (and other asynchronicities)</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Testing Time (and other asynchronicities)</title><id>tag:blog.golang.org,2013:blog.golang.org/testing-time</id><link rel="alternate" href="https://go.dev/blog/testing-time"></link><published>2025-08-26T00:00:00+00:00</published><updated>2025-08-26T00:00:00+00:00</updated><author><name>Damien Neil</name></author><summary type="html">A discussion of testing asyncronous code and an exploration of the `testing/synctest` package. Based on the GopherCon Europe 2025 talk with the same title.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/testing-time">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Testing Time (and other asynchronicities)</h1>
 
 <p class="author">
 Damien Neil<br>
 26 August 2025
 </p>
 
 <div class='markdown'>
<p>In Go 1.24, we introduced the <a href="/pkg/testing/synctest"><code>testing/synctest</code></a>
package as an experimental package.
This package can significantly simplify writing tests for concurrent,
asynchronous code.
In Go 1.25, the <code>testing/synctest</code> package has graduated from experiment
to general availability.</p>
<p>What follows is the blog version of my talk on
the <a href="/pkg/testing/synctest"><code>testing/synctest</code></a> package
at GopherCon Europe 2025 in Berlin.</p>
<h2 id="what-is-an-asynchronous-function">What is an asynchronous function?</h2>
<p>A synchronous function is pretty simple.
You call it, it does something, and it returns.</p>
<p>An asynchronous function is different.
You call it, it returns, and then it does something.</p>
<p>As a concrete, if somewhat artificial, example,
the following <code>Cleanup</code> function is synchronous.
You call it, it deletes a cache directory, and it returns.</p>
<pre><code>func (c *Cache) Cleanup() {
 os.RemoveAll(c.cacheDir)
}
</code></pre>
<p><code>CleanupInBackground</code> is an asynchronous function.
You call it, it returns, and the cache directory is deleted&hellip;sooner or later.</p>
<pre><code>func (c *Cache) CleanupInBackground() {
 go os.RemoveAll(c.cacheDir)
}
</code></pre>
<p>Sometimes an asynchronous function does something in the future.
For example, the <code>context</code> package&rsquo;s <code>WithDeadline</code> function
returns a context which will be canceled in the future.</p>
<pre><code>package context

// WithDeadline returns a derived context
// with a deadline no later than d.
func WithDeadline(parent Context, d time.Time) (Context, CancelFunc)
</code></pre>
<p>When I talk about testing concurrent code,
I mean testing these sorts of asynchronous operations,
both ones which use real time and ones which do not.</p>
<h2 id="tests">Tests</h2>
<p>A test verifies that a system behaves as we expect.
There&rsquo;s a lot of terminology describing types
of test&ndash;unit tests, integration tests, and so on&ndash;but
for our purposes here every kind of test reduces to three steps:</p>
<ol>
<li>Set up some initial conditions.</li>
<li>Tell the system under test to do something.</li>
<li>Verify the result.</li>
</ol>
<p>Testing a synchronous function is straightforward:</p>
<ul>
<li>You call the function;</li>
<li>the function does something and returns;</li>
<li>you verify the result.</li>
</ul>
<p>Testing an asyncronous function, however, is tricky:</p>
<ul>
<li>You call the function;</li>
<li>it returns;</li>
<li>you wait for it to finish doing whatever it does;</li>
<li>you verify the result.</li>
</ul>
<p>If you don&rsquo;t wait for the correct amount of time,
you may find yourself verifying the result of an operation that hasn&rsquo;t happened yet
or has only happened partially.
This never ends well.</p>
<p>Testing an asynchronous function is especially tricky
when you want to assert that something has <em>not</em> happened.
You can verify that the thing has not happened yet,
but how do you know with certainty that it isn&rsquo;t going to happen later?</p>
<h2 id="an-example">An example</h2>
<p>To make things a little more concrete,
let&rsquo;s work with a real-world example.
Consider the <code>context</code> package&rsquo;s <code>WithDeadline</code> function again.</p>
<pre><code>package context

// WithDeadline returns a derived context
// with a deadline no later than d.
func WithDeadline(parent Context, d time.Time) (Context, CancelFunc)
</code></pre>
<p>There are two obvious tests to write for <code>WithDeadline</code>.</p>
<ol>
<li>The context is <em>not</em> canceled <em>before</em> the deadline.</li>
<li>The context <em>is</em> canceled <em>after</em> the deadline.</li>
</ol>
<p>Let&rsquo;s write a test.</p>
<p>To keep the amount of code marginally less overwhelming,
we&rsquo;ll just test the second case:
After the deadline expires, the context is canceled.</p>
<pre><code>func TestWithDeadlineAfterDeadline(t *testing.T) {
 deadline := time.Now().Add(1 * time.Second)
 ctx, _ := context.WithDeadline(t.Context(), deadline)

 time.Sleep(time.Until(deadline))

 if err := ctx.Err(); err != context.DeadlineExceeded {
 t.Fatalf(&quot;context not canceled after deadline&quot;)
 }
}
</code></pre>
<p>This test is simple:</p>
<ol>
<li>Use <code>context.WithDeadline</code> to create a context with a deadline one second in the future.</li>
<li>Wait until the deadline.</li>
<li>Verify that the context is canceled.</li>
</ol>
<p>Unfortunately, this test obviously has a problem.
It sleeps until the exact moment the deadline expires.
Odds are good that the context has not been canceled yet by the time we examine it.
At best, this test will be very flaky.</p>
<p>Let&rsquo;s fix it.</p>
<pre><code>time.Sleep(time.Until(deadline) + 100*time.Millisecond)
</code></pre>
<p>We can sleep until 100ms after the deadline.
A hundred milliseconds is an eternity in computer terms.
This should be fine.</p>
<p>Unfortunately, we still have two problems.</p>
<p>First, this test takes 1.1 seconds to execute.
That&rsquo;s slow.
This is a simple test.
It should execute in milliseconds at the most.</p>
<p>Second, this test is flaky.
A hundred milliseconds is an eternity in computer terms,
but on an overloaded continuous integration (CI) system
it isn&rsquo;t unusual to see pauses much longer than that.
This test will probably pass consistently on a developer&rsquo;s workstation,
but I would expect occasional failures in a CI system.</p>
<h2 id="slow-or-flaky-pick-two">Slow or flaky: Pick two</h2>
<p>Tests that use real time are always slow or flaky.
Usually they&rsquo;re both.
If the test waits longer than necessary, it is slow.
If it doesn&rsquo;t wait long enough, it is flaky.
You can make the test more slow and less flaky,
or less slow and more flaky,
but you can&rsquo;t make it fast and reliable.</p>
<p>We have a lot of tests in the <code>net/http</code> package which use this approach.
They&rsquo;re all slow and/or flaky, which is what started me down the road
which brings us here today.</p>
<h2 id="write-synchronous-functions">Write synchronous functions?</h2>
<p>The simplest way to test an asyncronous function is not to do it.
Synchronous functions are easy to test.
If you can transform an asyncronous function into a synchronous one,
it will be easier to test.</p>
<p>For example, if we consider our cache cleanup functions from earlier,
the synchronous <code>Cleanup</code> is obviously better than
the asynchronous <code>CleanupInBackground</code>.
The synchronous function is easier to test,
and the caller can easily start a new goroutine to run it in the background if needed.
As a general rule,
the higher up the call stack you can push your concurrency,
the better.</p>
<pre><code>// CleanupInBackground is hard to test.
cache.CleanupInBackground()

// Cleanup is easy to test,
// and easy to run in the background when needed.
go cache.Cleanup()
</code></pre>
<p>Unfortunately, this sort of transformation isn&rsquo;t always possible.
For example, <code>context.WithDeadline</code> is an inherently asynchronous API.</p>
<h2 id="instrument-code-for-testability">Instrument code for testability?</h2>
<p>A better approach is to make our code more testable.</p>
<p>Here&rsquo;s an example of what this might look like for our <code>WithDeadline</code> test:</p>
<pre><code>func TestWithDeadlineAfterDeadline(t *testing.T) {
 clock := fakeClock()
 timeout := 1 * time.Second
 deadline := clock.Now().Add(timeout)

 ctx, _ := context.WithDeadlineClock(
 t.Context(), deadline, clock)

 clock.Advance(timeout)
 context.WaitUntilIdle(ctx)
 if err := ctx.Err(); err != context.DeadlineExceeded {
 t.Fatalf(&quot;context not canceled after deadline&quot;)
 }
}
</code></pre>
<p>Instead of using real time, we use a fake time implementation.
Using fake time avoids unnecessarily slow tests,
because we never wait around doing nothing.
It also helps avoid test flakiness,
since the current time only changes when the test adjusts it.</p>
<p>There are various fake time packages out there,
or you can write your own.</p>
<p>To use fake time, we need to modify our API to accept a fake clock.
I&rsquo;ve added a <code>context.WithDeadlineClock</code> function here,
that takes an additional clock parameter:</p>
<pre><code>ctx, _ := context.WithDeadlineClock(
 t.Context(), deadline, clock)
</code></pre>
<p>When we advance our fake clock, we have a problem.
Advancing time is an asynchrounous operation.
Sleeping goroutines may wake up,
timers may send on their channels,
and timer functions may run.
We need to wait for that work to finish before we can test
the expected behavior of the system.</p>
<p>I&rsquo;ve added a <code>context.WaitUntilIdle</code> function here,
which waits for any background work related to a context to complete:</p>
<pre><code>clock.Advance(timeout)
context.WaitUntilIdle(ctx)
</code></pre>
<p>This is a simple example, but it demonstrates
the two fundamental principles of writing testable concurrent code:</p>
<ol>
<li>Use fake time (if you use time).</li>
<li>Have some way to wait for quiescence,
which is a fancy way of saying
&ldquo;all background activity has stopped and the system is stable&rdquo;.</li>
</ol>
<p>The interesting question, of course, is how we do this.
I&rsquo;ve glossed over the details in this example because
there are some big downsides to this approach.</p>
<p>It&rsquo;s hard.
Using a fake clock isn&rsquo;t difficult,
but identifying when background concurrent work is finished
and it is safe to examine the state of the system is.</p>
<p>Your code becomes less idiomatic.
You can&rsquo;t use standard time package functions.
You need to be very careful to keep track of everything happening
in the background.</p>
<p>You need to instrument not just your code,
but any other packages you use.
If you call any third-party concurrent code,
you&rsquo;re probably out of luck.</p>
<p>Worst of all, it can be just about impossible
to retrofit this approach into an existing codebase.</p>
<p>I attempted to apply this approach to Go&rsquo;s HTTP implementation,
and while I had some success at doing so in places,
the HTTP/2 server simply defeated me.
In particular, adding instrumentation to detect quiescence
without extensive rewriting proved infeasible,
or at least beyond my skills.</p>
<h2 id="horrible-runtime-hacks">Horrible runtime hacks?</h2>
<p>What do we do if we can&rsquo;t make our code testable?</p>
<p>What if instead of instrumenting our code,
we had a way to observe the behavior of the uninstrumented system?</p>
<p>A Go program consists of a set of goroutines.
Those goroutines have states.
We just need to wait until all the goroutines have stopped running.</p>
<p>Unfortunately, the Go runtime doesn&rsquo;t provide any way to tell what
those goroutines are doing. Or does it?</p>
<p>The <code>runtime</code> package contains a function that gives us a stack trace
for every running goroutine, as well as their states.
This is text intended for human consumption,
but we could parse that output.
Could we use this to detect quiescence?</p>
<p>Now, of course this is a terrible idea.
There is no guarantee that the format of these stack traces will be stable over time.
You should not do this.</p>
<p>I did it.
And it worked.
In fact, it worked surprisingly well.</p>
<p>With a simple implementation of a fake clock,
a small amount of instrumentation to keep track of what goroutines were part of the test,
and some horrifying abuse of <code>runtime.Stack</code>,
I finally had a way to write fast, reliable tests for the <code>http</code> package.</p>
<p>The underlying implementation of these tests was horrible,
but it demonstrated that there was a useful concept here.</p>
<h2 id="a-better-way">A better way</h2>
<p>Go may have built-in concurrency,
but testing programs that use that concurrency is hard.</p>
<p>We&rsquo;re faced with an unfortunate choice:
We can write simple, idiomatic code, but it will be impossible to test quickly and reliably;
or we can write testable code, but it will be complicated and unidiomatic.</p>
<p>So we asked ourselves what we can do to make this better.</p>
<p>As we saw earlier, the two fundamental features required to write testable concurrent code are
fake time and a way to wait for quiescence.</p>
<p>We need a better way to to wait for quiescence.
We should be able to ask the runtime when background goroutines have finished their work.
We also want to be able to limit the scope of this query to a single test,
so that unrelated tests do not interfere with each other.</p>
<p>We also need better support for testing programs using fake time.</p>
<p>It isn&rsquo;t hard to make a fake time implementation,
but code which uses an implementation like this is not idiomatic.</p>
<p>Idiomatic code will use a <code>time.Timer</code>,
but it is not possible to create a fake <code>Timer</code>.
We asked ourselves whether we should provide a way for tests to
create a fake <code>Timer</code>, where the test controls when the timer fires.</p>
<p>A testing implementation of time needs to define an entirely new version of the <code>time</code> package,
and pass that to every function that operates on time.
We considered whether we should define a common time interface,
in the same way that <code>net.Conn</code> is a common interface describing a network connection.</p>
<p>What we realized, however, is that unlike network connections,
there is only one possible implementation of fake time.
A fake network may want to introduce latency or errors.
Time, in contrast, does only one thing: It moves forward.
Tests need to control the rate at which time progresses,
but a timer scheduled to fire ten seconds in the future
should always fire ten (possibly fake) seconds in the future.</p>
<p>In addition, we don&rsquo;t want to upset the entire Go ecosystem.
Most programs today use functions in the time package.
We want to keep those programs not only working,
but idiomatic.</p>
<p>This led to the conclusion that what we need is a way for a test to
tell the time package to use a fake clock,
in much the same way that the Go playground uses a fake clock.
Unlike the playground,
we need to limit the scope of that change to a single test.
(It may not be obvious that the Go playground uses a fake clock,
because we turn any fake delays into real delays on the front end,
but it does.)</p>
<h2 id="the-synctest-experiment">The <code>synctest</code> experiment</h2>
<p>And so in Go 1.24 we introduced <a href="/pkg/testing/synctest"><code>testing/synctest</code></a>,
a new, experimental package to simplify testing concurrent programs.
Over the months following the release of Go 1.24
we gathered feedback from early adopters.
(Thank you to everyone who tried it out!)
We made a number of changes to address problems and shortcomings.
And now, in Go 1.25, we&rsquo;ve released the <code>testing/synctest</code> package
as part of the standard library.</p>
<p>It lets you run a function in what we&rsquo;re calling a &ldquo;bubble&rdquo;.
Within the bubble, the time package uses a fake clock,
and the <code>synctest</code> package provides a function to wait for the bubble to quiesce.</p>
<h2 id="the-synctest-package">The <code>synctest</code> package</h2>
<p>The <code>synctest</code> package contains just two functions.</p>
<pre><code>package synctest

// Test executes f in a new bubble.
// Goroutines in the bubble use a fake clock.
func Test(t *testing.T, f func(*testing.T))

// Wait waits for background activity in the bubble to complete.
func Wait()
</code></pre>
<p><a href="/pkg/testing/synctest#Test"><code>Test</code></a> executes a function in a new bubble.</p>
<p><a href="/pkg/testing/synctest#Wait"><code>Wait</code></a> blocks until every goroutine in the bubble is blocked
waiting for some other goroutine in the bubble.
We call that state being &ldquo;durably blocked&rdquo;.</p>
<h2 id="testing-with-synctest">Testing with synctest</h2>
<p>Let&rsquo;s look at an example of synctest in action.</p>
<pre><code>func TestWithDeadlineAfterDeadline(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 deadline := time.Now().Add(1 * time.Second)
 ctx, _ := context.WithDeadline(t.Context(), deadline)

 time.Sleep(time.Until(deadline))
 synctest.Wait()
 if err := ctx.Err(); err != context.DeadlineExceeded {
 t.Fatalf(&quot;context not canceled after deadline&quot;)
 }
 })
}
</code></pre>
<p>This might look a little familiar.
This is the naïve test for <code>context.WithDeadline</code> that we looked at earlier.
The only changes are that we&rsquo;ve wrapped the test in
a <code>synctest.Test</code> call to execute it in a bubble
and we have added a <code>synctest.Wait</code> call.</p>
<p>This test is fast and reliable.
It runs almost instantaneously.
It precisely tests the expected behavior of the system under test.
It also requires no modification of the <code>context</code> package.</p>
<p>Using the <code>synctest</code> package,
we can write simple, idiomatic code
and test it reliably.</p>
<p>This is a very simple example, of course,
but this is a real test of real production code.
If <code>synctest</code> had existed when the <code>context</code> package was written,
we would have had a much easier time writing tests for it.</p>
<h2 id="time">Time</h2>
<p>Time in the bubble behaves much the same as the fake time in the Go playground.
Time starts at midnight, January 1, 2000 UTC.
If you need to run a test at some specific point in time for some reason,
you can just sleep until then.</p>
<pre><code>func TestAtSpecificTime(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 // 2000-01-01 00:00:00 +0000 UTC
 t.Log(time.Now().In(time.UTC))

 // This does not take 25 years.
 time.Sleep(time.Until(
 time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC)))

 // 2025-01-01 00:00:00 +0000 UTC
 t.Log(time.Now().In(time.UTC))
 })
}
</code></pre>
<p>Time only passes when every goroutine in the bubble has blocked.
You can think of the bubble as simulating an infinitely fast computer:
Any amount of computation takes no time.</p>
<p>The following test will always print that zero seconds
of fake time have elapsed since the start of the test,
no matter how much real time has passed.</p>
<pre><code>func TestExpensiveWork(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 start := time.Now()
 for range 1e7 {
 // do expensive work
 }
 t.Log(time.Since(start)) // 0s
 })
}
</code></pre>
<p>In the next test, the <code>time.Sleep</code> call will return immediately,
rather than waiting for ten real seconds.
The test will always print that exactly ten fake seconds
have passed since the start of the test.</p>
<pre><code>func TestSleep(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 start := time.Now()
 time.Sleep(10 * time.Second)
 t.Log(time.Since(start)) // 10s
 })
}
</code></pre>
<h2 id="waiting-for-quiescence">Waiting for quiescence</h2>
<p>The <a href="/pkg/testing/synctest#Wait"><code>synctest.Wait</code></a> function
lets us wait for background activity to complete.</p>
<pre><code>func TestWait(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 done := false
 go func() {
 done = true
 }()

 // Wait for the above goroutine to finish.
 synctest.Wait()

 t.Log(done) // true
 })
}
</code></pre>
<p>If we didn&rsquo;t have the <code>Wait</code> call in the above test,
we would have a race condition:
One goroutine modifies the <code>done</code> variable
while another reads from it without synchronization.
The <code>Wait</code> call provides that synchronization.</p>
<p>You may be familiar with the <code>-race</code> test flag,
which enables the data race detector.
The race detector is aware of the synchronization provided by <code>Wait</code>,
and does not complain about this test.
If we forgot the <code>Wait</code> call, the race detector would correctly complain.</p>
<p>The <code>synctest.Wait</code> function provides synchronization,
but the passage of time does not.</p>
<p>In the next example, one goroutine writes to the <code>done</code> variable
while another sleeps for one nanosecond before reading from it.
It should be obvious that when run with a real clock outside a synctest bubble,
this code contains a race condition.
Inside a synctest bubble,
while the fake clock ensures that the goroutine completes before <code>time.Sleep</code> returns,
the race detector will still report the data race,
just like it would if this code were run outside a synctest bubble.</p>
<pre><code>func TestTimeDataRace(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 done := false
 go func() {
 done = true // write
 }()

 time.Sleep(1 * time.Nanosecond)

 t.Log(done) // read (unsynchronized)
 })
}
</code></pre>
<p>Adding a <code>Wait</code> call provides explicit synchronization and fixes the data race:</p>
<pre><code>time.Sleep(1 * time.Nanosecond)
synctest.Wait() // synchronize
t.Log(done) // read
</code></pre>
<h2 id="example-iocopy">Example: <code>io.Copy</code></h2>
<p>Taking advantage of the synchronization provided by <code>synctest.Wait</code> allows us
to write simpler tests with less explicit synchronization.</p>
<p>For example, consider this test of <a href="/pkg/io#Copy"><code>io.Copy</code></a>.</p>
<pre><code>func TestIOCopy(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 srcReader, srcWriter := io.Pipe()
 defer srcWriter.Close()

 var dst bytes.Buffer
 go io.Copy(&amp;dst, srcReader)

 data := &quot;1234&quot;
 srcWriter.Write([]byte(&quot;1234&quot;))
 synctest.Wait()

 if got, want := dst.String(), data; got != want {
 t.Errorf(&quot;Copy wrote %q, want %q&quot;, got, want)
 }
 })
}
</code></pre>
<p>The <code>io.Copy</code> function copies data from an <code>io.Reader</code> to an <code>io.Writer</code>.
You might not immediately think of <code>io.Copy</code> as a concurrent function,
since it blocks until the copy has completed.
However, providing data to <code>io.Copy</code>&rsquo;s reader is an asynchronous operation:</p>
<ul>
<li><code>Copy</code> calls the reader&rsquo;s <code>Read</code> method;</li>
<li><code>Read</code> returns some data;</li>
<li>and the data is written to the writer at a later time.</li>
</ul>
<p>In this test, we are verifying that <code>io.Copy</code> writes new data to the writer
without waiting to fill its buffer.</p>
<p>Looking at the test step by step,
we first create an <code>io.Pipe</code> to serve as the source <code>io.Copy</code> reads from:</p>
<pre><code>srcReader, srcWriter := io.Pipe()
defer srcWriter.Close()
</code></pre>
<p>We call <code>io.Copy</code> in a new goroutine,
copying from the read end of the pipe into a <code>bytes.Buffer</code>:</p>
<pre><code>var dst bytes.Buffer
go io.Copy(&amp;dst, srcReader)
</code></pre>
<p>We write to the other end of the pipe,
and wait for <code>io.Copy</code> to handle the data:</p>
<pre><code>data := &quot;1234&quot;
srcWriter.Write([]byte(&quot;1234&quot;))
synctest.Wait()
</code></pre>
<p>Finally, we verify that the destination buffer contains the desired data:</p>
<pre><code>if got, want := dst.String(), data; got != want {
 t.Errorf(&quot;Copy wrote %q, want %q&quot;, got, want)
}
</code></pre>
<p>We don&rsquo;t need to add a mutex or other synchronization around the destination buffer,
because <code>synctest.Wait</code> ensures that it is never accessed concurrently.</p>
<p>This test demonstrates a few important points.</p>
<p>Even synchronous functions like <code>io.Copy</code>,
which do not perform additional background work after they return,
may exhibit asynchronous behaviors.</p>
<p>Using <code>synctest.Wait</code>, we can test those behaviors.</p>
<p>Note also that this test does not work with time.
Many asynchronous systems involve time, but not all.</p>
<h2 id="bubble-exit">Bubble exit</h2>
<p>The <code>synctest.Test</code> function waits for all goroutines in the bubble to exit
before returning.
Time stops advancing after the root goroutine (the goroutine started by <code>Test</code>) returns.</p>
<p>In the next example, <code>Test</code> waits for the background goroutine to run and exit
before it returns:</p>
<pre><code>func TestWaitForGoroutine(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 go func() {
 // This runs before synctest.Test returns.
 }()
 })
}
</code></pre>
<p>In this example, we schedule a <code>time.AfterFunc</code> for a time in the future.
The bubble&rsquo;s root goroutine returns before that time is reached,
so the <code>AfterFunc</code> never runs:</p>
<pre><code>func TestDoNotWaitForTimer(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 time.AfterFunc(1 * time.Nanosecond, func() {
 // This never runs.
 })
 })
}
</code></pre>
<p>In the next example, we start a goroutine that sleeps.
The root goroutine returns and time stops advancing.
The bubble is now deadlocked,
because <code>Test</code> is waiting for all goroutines in the bubble to finish
but the sleeping goroutine is waiting for time to advance.</p>
<pre><code>func TestDeadlock(t *testing.T) {
 synctest.Test(t, func(t *testing.T) {
 go func() {
 // This sleep never returns and the test deadlocks.
 time.Sleep(1 * time.Nanosecond)
 }()
 })
}
</code></pre>
<h2 id="deadlocks">Deadlocks</h2>
<p>The <code>synctest</code> package panics when a bubble is deadlocked
due to every goroutine in the bubble being durably blocked on
another goroutine in the bubble.</p>
<pre><code>--- FAIL: Test (0.00s)
--- FAIL: TestDeadlock (0.00s)
panic: deadlock: main bubble goroutine has exited but blocked goroutines remain [recovered, repanicked]

goroutine 7 [running]:
(stacks elided for clarity)

goroutine 10 [sleep (durable), synctest bubble 1]:
time.Sleep(0x1)
 /Users/dneil/src/go/src/runtime/time.go:361 +0x130
_.TestDeadlock.func1.1()
 /tmp/s/main_test.go:13 +0x20
created by _.TestDeadlock.func1 in goroutine 9
 /tmp/s/main_test.go:11 +0x24
FAIL _ 0.173s
FAIL
</code></pre>
<p>The runtime will print stack traces for every goroutine in the deadlocked bubble.</p>
<p>When printing the status of a bubbled goroutine,
the runtime indicates when the goroutine is durably blocked.
You can see that the sleeping goroutine in this test is durably blocked.</p>
<h2 id="durable-blocking">Durable blocking</h2>
<p>&ldquo;Durably blocking&rdquo; is a core concept in synctest.</p>
<p>A goroutine is durably blocked when it is not only blocked,
but when it can only be unblocked by another goroutine in the same bubble.</p>
<p>When every goroutine in a bubble is durably blocked:</p>
<ol>
<li><code>synctest.Wait</code> returns.</li>
<li>If there is no <code>synctest.Wait</code> call in progress,
fake time advances instantly to the next point that will wake a goroutine.</li>
<li>If there is no goroutine that can be woken by advancing time,
the bubble is deadlocked and the test fails.</li>
</ol>
<p>It is important for us to make a distinction between a goroutine which is merely blocked
and one which is <em>durably</em> blocked.
We don&rsquo;t want to declare a deadlock when a goroutine is temporarily blocked on
some event arising outside its bubble.</p>
<p>Let&rsquo;s look at some ways in which a goroutine can block non-durably.</p>
<h3 id="not-durably-blocking-io-files-pipes-network-connections-etc">Not durably blocking: I/O (files, pipes, network connections, etc.)</h3>
<p>The most important limitation is that I/O is not durably blocking,
including network I/O.
A goroutine reading from a network connection may be blocked,
but it will be unblocked by data arriving on that connection.</p>
<p>This is obviously true for a connection to some network service,
but it is also true for a loopback connection,
even when the reader and writer are both in the same bubble.</p>
<p>When we write data to a network socket,
even a loopback socket,
the data is passed to the kernel for delivery.
There is a period of time between the write system call returning
and the kernel notifying the other side of the connection that data is available.
The Go runtime cannot distinguish between a goroutine blocked waiting for
data that is already in the kernel&rsquo;s buffers
and one blocked waiting for data that will not arrive.</p>
<p>This means that tests of networked programs using synctest
usually cannot use real network connections.
Instead, they should use an in-memory fake.</p>
<p>I&rsquo;m not going to go over the process of creating a fake network here,
but the <code>synctest</code> package documentation contains
<a href="/pkg/testing/synctest#hdr-Example__HTTP_100_Continue">a complete worked example</a>
of testing an HTTP client and server communicating over a fake network.</p>
<h3 id="not-durably-blocking-syscalls-cgo-calls-anything-that-isnt-go">Not durably blocking: syscalls, cgo calls, anything that isn&rsquo;t Go</h3>
<p>Syscalls and cgo calls are not durably blocking.
We can only reason about the state of goroutines executing Go code.</p>
<h3 id="not-durably-blocking-mutexes">Not durably blocking: Mutexes</h3>
<p>Perhaps surprisingly, mutexes are not durably blocking.
This is a decision born of practicality:
Mutexes are often used to guard global state,
so a bubbled goroutine will often need to acquire a mutex held outside its bubble.
Mutexes are highly performance-sensitive,
so adding additional instrumentation to them
risks slowing down non-test programs.</p>
<p>We can test programs that use mutexes with synctest,
but the fake clock will not advance while a goroutine is blocked on mutex acquisition.
This hasn&rsquo;t posed a problem in any case we&rsquo;ve encountered,
but it is something to be aware of.</p>
<h3 id="durably-blocking-timesleep">Durably blocking: <code>time.Sleep</code></h3>
<p>So what is durably blocking?</p>
<p><code>time.Sleep</code> is obviously durable,
since time can only advance when every goroutine in the bubble is durably blocked.</p>
<h3 id="durably-blocking-send-or-receive-on-channels-created-in-the-same-bubble">Durably blocking: send or receive on channels created in the same bubble</h3>
<p>Channel operations on channels created within the same bubble are durable.</p>
<p>We make a distinction between bubbled channels (created in a bubble)
and unbubbled channels (created outside any bubble).
This means that a function using a global channel for synchronization,
for example to control access to a globally cached resource,
can be safely called from within a bubble.</p>
<p>Trying to operate on a bubbled channel from outside its bubble is an error.</p>
<h3 id="durably-blocking-syncwaitgroup-belonging-to-the-same-bubble">Durably blocking: <code>sync.WaitGroup</code> belonging to the same bubble</h3>
<p>We also associate <code>sync.WaitGroup</code>s with bubbles.</p>
<p><code>WaitGroup</code> doesn&rsquo;t have a constructor,
so we make the association with the bubble implicitly on the first call to <code>Go</code> or <code>Add</code>.</p>
<p>As with channels,
waiting on a <code>WaitGroup</code> belonging to the same bubble is durably blocking,
and waiting on one from outside the bubble is not.
Calling <code>Go</code> or <code>Add</code> on a <code>WaitGroup</code> belonging to a different bubble is an error.</p>
<h3 id="durably-blocking-synccondwait">Durably blocking: <code>sync.Cond.Wait</code></h3>
<p>Waiting on a <code>sync.Cond</code> is always durably blocking.
Waking up a goroutine waiting on a <code>Cond</code> in a different bubble is an error.</p>
<h3 id="durably-blocking-select">Durably blocking: <code>select{}</code></h3>
<p>Finally, an empty select is durably blocking.
(A select with cases is durably blocking if all the operations in it are so.)</p>
<p>That&rsquo;s the complete list of durably blocking operations.
It isn&rsquo;t very long,
but it&rsquo;s enough to handle almost all real-world programs.</p>
<p>The rule is that a goroutine is durably blocked when it is blocked,
and we can guarantee that it can only be unblocked
by another goroutine in its bubble.</p>
<p>In cases where it is possible to attempt to wake a bubbled goroutine from outside its bubble,
we panic.
For example, it is an error to operate on a bubbled channel from outside its bubble.</p>
<h2 id="changes-from-124-to-125">Changes from 1.24 to 1.25</h2>
<p>We released an experimental version of the <code>synctest</code> package in Go 1.24.
To ensure that early adopters were aware of the experimental status of the package,
you needed to set a GOEXPERIMENT flag to make the package visible.</p>
<p>The feedback we received from those early adopters was invaluable,
both in demonstrating that the package is useful
and in uncovering areas where the API needed work.</p>
<p>These are some of the changes made between the experimental version
and the version released in Go 1.25.</p>
<h3 id="replaced-run-with-test">Replaced Run with Test</h3>
<p>The original version of the API created a bubble with a <code>Run</code> function:</p>
<pre><code>// Run executes f in a new bubble.
func Run(f func())
</code></pre>
<p>It became clear that we needed a way to create a <code>*testing.T</code>
that is scoped to a bubble.
For example, <code>t.Cleanup</code> should run cleanup functions in the same bubble
they are registered in, not after the bubble exits.
We renamed <code>Run</code> to <code>Test</code> and made it create a <code>T</code> scoped to the lifetime
of the new bubble.</p>
<h3 id="time-stops-when-a-bubbles-root-goroutine-returns">Time stops when a bubble&rsquo;s root goroutine returns</h3>
<p>We originally continued to advance time within a bubble for so long as
the bubble contained any goroutines waiting for future events.
This turned out to be very confusing when a long-lived goroutine never returned,
such as a goroutine reading forever from a <code>time.Ticker</code>.
We now stop advancing time when a bubble&rsquo;s root goroutine returns.
If the bubble is blocked waiting for time to advance,
this results in a deadlock and a panic which can be analyzed.</p>
<h3 id="removed-cases-where-durable-wasnt">Removed cases where &ldquo;durable&rdquo; wasn&rsquo;t</h3>
<p>We cleaned up the definition of &ldquo;durably blocking&rdquo;.
The original implementation had cases where a durably blocked goroutine could
be unblocked from outside the bubble.
For example, channels recorded whether they were created in a bubble,
but not which in which bubble they were created,
so one bubble could unblock a channel in a different bubble.
The current implementation contains no cases we know of
where a durably blocked goroutine can be unblocked from outside its bubble.</p>
<h3 id="better-stack-traces">Better stack traces</h3>
<p>We made improvements to the information printed in stack traces.
When a bubble deadlocks, we by default now only print stacks for the goroutines in that bubble.
Stack traces also clearly indicate which goroutines in a bubble are durably blocked.</p>
<h3 id="randomized-events-happening-at-the-same-time">Randomized events happening at the same time</h3>
<p>We made improvements to the randomization of events happening at the same time.
Originally, timers scheduled to fire at the same instant
would always do so in the order they were created.
This ordering is now randomized.</p>
<h2 id="future-work">Future work</h2>
<p>We&rsquo;re pretty happy with the synctest package at the moment.</p>
<p>Aside from the inevitable bug fixes,
we don&rsquo;t currently expect any major changes to it in the future.
Of course, with wider adoption it is always possible that we&rsquo;ll discover something
that needs doing.</p>
<p>One possible area of work is to improve the detection of durably blocked goroutines.
It would be nice if we could make mutex operations durably blocking,
with a restriction that a mutex acquired in a bubble must be released
from within the same bubble.</p>
<p>Testing networked code with synctest requires a fake network.
The <code>net.Pipe</code> function can create a fake <code>net.Conn</code>,
but there is currently no standard library function that creates
a fake <code>net.Listener</code> or <code>net.PacketConn</code>.
In addition, the <code>net.Conn</code> returned by <code>net.Pipe</code> is synchronous&ndash;every write blocks
until a read consumes the data&ndash;which is not representative of real network behavior.
Perhaps we should add a good fake implementations of common network interfaces
to the standard library.</p>
<h2 id="conclusion">Conclusion</h2>
<p>That&rsquo;s the <code>synctest</code> package.</p>
<p>I can&rsquo;t say that it makes testing concurrent code simple,
because concurrency is never simple.
What it does is let you write the simplest possible concurrent code,
using idiomatic Go,
and the standard time package,
and then write fast, reliable tests for it.</p>
<p>I hope you find it useful.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/jsonv2-exp">A new experimental Go API for JSON</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/container-aware-gomaxprocs">Container-aware GOMAXPROCS</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Container-aware GOMAXPROCS</title><id>tag:blog.golang.org,2013:blog.golang.org/container-aware-gomaxprocs</id><link rel="alternate" href="https://go.dev/blog/container-aware-gomaxprocs"></link><published>2025-08-20T00:00:00+00:00</published><updated>2025-08-20T00:00:00+00:00</updated><author><name>Michael Pratt and Carlos Amedee</name></author><summary type="html">New GOMAXPROCS defaults in Go 1.25 improve behavior in containers.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/container-aware-gomaxprocs">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Container-aware GOMAXPROCS</h1>
 
 <p class="author">
 Michael Pratt and Carlos Amedee<br>
 20 August 2025
 </p>
 
 <div class='markdown'>
<p>Go 1.25 includes new container-aware <code>GOMAXPROCS</code> defaults, providing more sensible default behavior for many container workloads, avoiding throttling that can impact tail latency, and improving Go&rsquo;s out-of-the-box production-readiness.
In this post, we will dive into how Go schedules goroutines, how that scheduling interacts with container-level CPU controls, and how Go can perform better with awareness of container CPU controls.</p>
<h2 id="gomaxprocs"><code>GOMAXPROCS</code></h2>
<p>One of Go&rsquo;s strengths is its built-in and easy-to-use concurrency via goroutines.
From a semantic perspective, goroutines appear very similar to operating system threads, enabling us to write simple, blocking code.
On the other hand, goroutines are more lightweight than operating system threads, making it much cheaper to create and destroy them on the fly.</p>
<p>While a Go implementation could map each goroutine to a dedicated operating system thread, Go keeps goroutines lightweight with a runtime scheduler that makes threads fungible.
Any Go-managed thread can run any goroutine, so creating a new goroutine doesn&rsquo;t require creating a new thread, and waking a goroutine doesn&rsquo;t necessarily require waking another thread.</p>
<p>That said, along with a scheduler comes scheduling questions.
For example, exactly how many threads should we use to run goroutines?
If 1,000 goroutines are runnable, should we schedule them on 1,000 different threads?</p>
<p>This is where <a href="/pkg/runtime#GOMAXPROCS"><code>GOMAXPROCS</code></a> comes in.
Semantically, <code>GOMAXPROCS</code> tells the Go runtime the &ldquo;available parallelism&rdquo; that Go should use.
In more concrete terms, <code>GOMAXPROCS</code> is the maximum number of threads to use for running goroutines at once.</p>
<p>So, if <code>GOMAXPROCS=8</code> and there are 1,000 runnable goroutines, Go will use 8 threads to run 8 goroutines at a time.
Often, goroutines run for a very short time and then block, at which point Go will switch to running another goroutine on that same thread.
Go will also preempt goroutines that don&rsquo;t block on their own, ensuring all goroutines get a chance to run.</p>
<p>From Go 1.5 through Go 1.24, <code>GOMAXPROCS</code> defaulted to the total number of CPU cores on the machine.
Note that in this post, &ldquo;core&rdquo; more precisely means &ldquo;logical CPU.&rdquo;
For example, a machine with 4 physical CPUs with hyperthreading has 8 logical CPUs.</p>
<p>This typically makes a good default for &ldquo;available parallelism&rdquo; because it naturally matches the available parallelism of the hardware.
That is, if there are 8 cores and Go runs more than 8 threads at a time, the operating system will have to multiplex these threads onto the 8 cores, much like how Go multiplexes goroutines onto threads.
This extra layer of scheduling is not always a problem, but it is unnecessary overhead.</p>
<h2 id="container-orchestration">Container Orchestration</h2>
<p>Another of Go&rsquo;s core strengths is the convenience of deploying applications via a container, and managing the number of cores Go uses is especially important when deploying an application within a container orchestration platform.
Container orchestration platforms like <a href="https://kubernetes.io/" rel="noreferrer" target="_blank">Kubernetes</a> take a set of machine resources and schedule containers within the available resources based on requested resources.
Packing as many containers as possible within a cluster&rsquo;s resources requires the platform to be able to predict the resource usage of each scheduled container.
We want Go to adhere to the resource utilization constraints that the container orchestration platform sets.</p>
<p>Let&rsquo;s explore the effects of the <code>GOMAXPROCS</code> setting in the context of Kubernetes, as an example.
Platforms like Kubernetes provide a mechanism to limit the resources consumed by a container.
Kubernetes has the concept of CPU resource limits, which signal to the underlying operating system how many core resources a specific container or set of containers will be allocated.
Setting a CPU limit translates to the creation of a Linux <a href="https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu" rel="noreferrer" target="_blank">control group</a> CPU bandwidth limit.</p>
<p>Before Go 1.25, Go was unaware of CPU limits set by orchestration platforms.
Instead, it would set <code>GOMAXPROCS</code> to the number of cores on the machine it was deployed to.
If there was a CPU limit in place, the application may try to use far more CPU than allowed by the limit.
To prevent an application from exceeding its limit, the Linux kernel will <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run" rel="noreferrer" target="_blank">throttle</a> the application.</p>
<p>Throttling is a blunt mechanism for restricting containers that would otherwise exceed their CPU limit: it completely pauses application execution for the remainder of the throttling period.
The throttling period is typically 100ms, so throttling can cause substantial tail latency impact compared to the softer scheduling multiplexing effects of a lower <code>GOMAXPROCS</code> setting.
Even if the application never has much parallelism, tasks performed by the Go runtime—such as garbage collection—can still cause CPU spikes that trigger throttling.</p>
<h2 id="new-default">New default</h2>
<p>We want Go to provide efficient and reliable defaults when possible, so in Go 1.25, we have made <code>GOMAXPROCS</code> take into account its container environment by default.
If a Go process is running inside a container with a CPU limit, <code>GOMAXPROCS</code> will default to the CPU limit if it is less than the core count.</p>
<p>Container orchestration systems may adjust container CPU limits on the fly, so Go 1.25 will also periodically check the CPU limit and adjust <code>GOMAXPROCS</code> automatically if it changes.</p>
<p>Both of these defaults only apply if <code>GOMAXPROCS</code> is otherwise unspecified.
Setting the <code>GOMAXPROCS</code> environment variable or calling <code>runtime.GOMAXPROCS</code> continues to behave as before.
The <a href="/pkg/runtime#GOMAXPROCS"><code>runtime.GOMAXPROCS</code></a> documentation covers the details of the new behavior.</p>
<h2 id="slightly-different-models">Slightly different models</h2>
<p>Both <code>GOMAXPROCS</code> and a container CPU limit place a limit on the maximum amount of CPU the process can use, but their models are subtly different.</p>
<p><code>GOMAXPROCS</code> is a parallelism limit.
If <code>GOMAXPROCS=8</code> Go will never run more than 8 goroutines at a time.</p>
<p>By contrast, CPU limits are a throughput limit.
That is, they limit the total CPU time used in some period of wall time.
The default period is 100ms.
So an &ldquo;8 CPU limit&rdquo; is actually a limit of 800ms of CPU time every 100ms of wall time.</p>
<p>This limit could be filled by running 8 threads continuously for the entire 100ms, which is equivalent to <code>GOMAXPROCS=8</code>.
On the other hand, the limit could also be filled by running 16 threads for 50ms each, with each thread being idle or blocked for the other 50ms.</p>
<p>In other words, a CPU limit doesn&rsquo;t limit the total number of CPUs the container can run on.
It only limits total CPU time.</p>
<p>Most applications have fairly consistent CPU usage across 100ms periods, so the new <code>GOMAXPROCS</code> default is a pretty good match to the CPU limit, and certainly better than the total core count!
However, it is worth noting that particularly spiky workloads may see a latency increase from this change due to <code>GOMAXPROCS</code> preventing short-lived spikes of additional threads beyond the CPU limit average.</p>
<p>In addition, since CPU limits are a throughput limit, they can have a fractional component (e.g., 2.5 CPU).
On the other hand, <code>GOMAXPROCS</code> must be a positive integer.
Thus, Go must round the limit to a valid <code>GOMAXPROCS</code> value.
Go always rounds up to enable use of the full CPU limit.</p>
<h2 id="cpu-requests">CPU Requests</h2>
<p>Go&rsquo;s new <code>GOMAXPROCS</code> default is based on the container&rsquo;s CPU limit, but container orchestration systems also provide a &ldquo;CPU request&rdquo; control.
While the CPU limit specifies the maximum CPU a container may use, the CPU request specifies the minimum CPU guaranteed to be available to the container at all times.</p>
<p>It is common to create containers with a CPU request but no CPU limit, as this allows containers to utilize machine CPU resources beyond the CPU request that would otherwise be idle due to lack of load from other containers.
Unfortunately, this means that Go cannot set <code>GOMAXPROCS</code> based on the CPU request, which would prevent utilization of additional idle resources.</p>
<p>Containers with a CPU request are still <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run" rel="noreferrer" target="_blank">constrained</a> when exceeding their request if the machine is busy.
The weight-based constraint of exceeding requests is &ldquo;softer&rdquo; than the hard period-based throttling of CPU limits, but CPU spikes from high <code>GOMAXPROCS</code> can still have an adverse impact on application behavior.</p>
<h2 id="should-i-set-a-cpu-limit">Should I set a CPU limit?</h2>
<p>We have learned about the problems caused by having <code>GOMAXPROCS</code> too high, and that setting a container CPU limit allows Go to automatically set an appropriate <code>GOMAXPROCS</code>, so an obvious next step is to wonder whether all containers should set a CPU limit.</p>
<p>While that may be good advice to automatically get a reasonable <code>GOMAXPROCS</code> defaults, there are many other factors to consider when deciding whether to set a CPU limit, such as prioritizing utilization of idle resources by avoiding limits vs prioritizing predictable latency by setting limits.</p>
<p>The worst behaviors from a mismatch between <code>GOMAXPROCS</code> and effective CPU limits occur when <code>GOMAXPROCS</code> is significantly higher than the effective CPU limit.
For example, a small container receiving 2 CPUs running on a 128 core machine.
These are the cases where it is most valuable to consider setting an explicit CPU limit, or, alternatively, explicitly setting <code>GOMAXPROCS</code>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Go 1.25 provides more sensible default behavior for many container workloads by setting <code>GOMAXPROCS</code> based on container CPU limits.
Doing so avoids throttling that can impact tail latency, improves efficiency, and generally tries to ensure Go is production-ready out-of-the-box.
You can get the new defaults simply by setting the Go version to 1.25.0 or higher in your <code>go.mod</code>.</p>
<p>Thanks to everyone in the community that contributed to the <a href="/issue/33803">long</a> <a href="/issue/73193">discussions</a> that made this a reality, and in particular to feedback from the maintainers of <a href="https://pkg.go.dev/go.uber.org/automaxprocs" rel="noreferrer" target="_blank"><code>go.uber.org/automaxprocs</code></a> from Uber, which has long provided similar behavior to its users.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/testing-time">Testing Time (and other asynchronicities)</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/go1.25">Go 1.25 is released</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go 1.25 is released</title><id>tag:blog.golang.org,2013:blog.golang.org/go1.25</id><link rel="alternate" href="https://go.dev/blog/go1.25"></link><published>2025-08-12T00:00:00+00:00</published><updated>2025-08-12T00:00:00+00:00</updated><author><name>Dmitri Shuralyov, on behalf of the Go team</name></author><summary type="html">Go 1.25 adds container-aware GOMAXPROCS, testing/synctest package, experimental GC, experimental encoding/json/v2, and more.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/go1.25">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go 1.25 is released</h1>
 
 <p class="author">
 Dmitri Shuralyov, on behalf of the Go team<br>
 12 August 2025
 </p>
 
 <div class='markdown'>
<p>Today the Go team is pleased to release Go 1.25.
You can find its binary archives and installers on the <a href="/dl/">download page</a>.</p>
<p>Go 1.25 comes with improvements over Go 1.24 across
its <a href="/doc/go1.25#tools">tools</a>,
the <a href="/doc/go1.25#runtime">runtime</a>,
<a href="/doc/go1.25#compiler">compiler</a>,
<a href="/doc/go1.25#linker">linker</a>,
and the <a href="/doc/go1.25#library">standard library</a>,
including the addition of one <a href="/doc/go1.25#new-testingsynctest-package">new package</a>.
There are <a href="/doc/go1.25#ports">port-specific</a> changes
and <a href="/doc/godebug#go-125"><code>GODEBUG</code> settings</a> updates.</p>
<p>Some of the additions in Go 1.25 are in an experimental stage
and become exposed only when you explicitly opt in.
Notably, a <a href="/doc/go1.25#new-experimental-garbage-collector">new experimental garbage collector</a>,
and a <a href="/doc/go1.25#json_v2">new experimental <code>encoding/json/v2</code> package</a>
are available for you to try ahead of time and provide your feedback.
It really helps if you&rsquo;re able to do that!</p>
<p>Please refer to the <a href="/doc/go1.25">Go 1.25 Release Notes</a> for the complete list
of additions, changes and improvements in Go 1.25.</p>
<p>Over the next few weeks, follow-up blog posts will cover some of the topics
relevant to Go 1.25 in more detail. Check back in later to read those posts.</p>
<p>Thanks to everyone who contributed to this release by writing code, filing bugs,
trying out experimental additions, sharing feedback, and testing the release candidates.
Your efforts helped make Go 1.25 as stable as possible.
As always, if you notice any problems, please <a href="/issue/new">file an issue</a>.</p>
<p>We hope you enjoy using the new release!</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/container-aware-gomaxprocs">Container-aware GOMAXPROCS</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/fips140">The FIPS 140-3 Go Cryptographic Module</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>The FIPS 140-3 Go Cryptographic Module</title><id>tag:blog.golang.org,2013:blog.golang.org/fips140</id><link rel="alternate" href="https://go.dev/blog/fips140"></link><published>2025-07-15T00:00:00+00:00</published><updated>2025-07-15T00:00:00+00:00</updated><author><name>Filippo Valsorda (Geomys), Daniel McCarney (Geomys), and Roland Shoemaker (Google)</name></author><summary type="html">Go now has a built-in, native FIPS 140-3 compliant mode.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/fips140">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>The FIPS 140-3 Go Cryptographic Module</h1>
 
 <p class="author">
 Filippo Valsorda (Geomys), Daniel McCarney (Geomys), and Roland Shoemaker (Google)<br>
 15 July 2025
 </p>
 
 <div class='markdown'>
<p>FIPS 140 is a standard for cryptography implementations and, although it doesn’t
necessarily improve security, FIPS 140 compliance is a requirement in certain
regulated environments that are increasingly adopting Go. Until now, FIPS 140
compliance has been a significant source of friction for Go users, requiring
unsupported solutions with safety, developer experience, functionality, release
velocity, and compliance issues.</p>
<p>Go is addressing this growing need with native FIPS 140 support built right into
the standard library and the <code>go</code> command, making Go the easiest, most secure
way to comply with FIPS 140. The FIPS 140-3 validated Go Cryptographic Module
now underlies Go’s built-in crypto libraries, starting with the Go Cryptographic
Module v1.0.0 that is included in Go 1.24, released last February.</p>
<p>The v1.0.0 module has been awarded <a href="https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?validation=39260" rel="noreferrer" target="_blank">Cryptographic Algorithm Validation Program
(CAVP) certificate A6650</a>, was submitted to the Cryptographic Module
Validation Program (CMVP), and reached the <a href="https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/modules-in-process-list" rel="noreferrer" target="_blank">Modules In Process List</a> in May.
Modules on the MIP list are awaiting NIST review and can already be deployed in
certain regulated environments.</p>
<p><a href="https://geomys.org" rel="noreferrer" target="_blank">Geomys</a> led the implementation effort in collaboration with the Go Security
Team, and is pursuing a broadly applicable FIPS 140-3 validation for the benefit
of the Go community. Google and other industry stakeholders have a contractual
relationship with Geomys to include specific Operating Environments in the
certificate.</p>
<p>Further details on the module are available in the
<a href="/doc/security/fips140">documentation</a>.</p>
<p>Some Go users currently rely on the <a href="/doc/security/fips140#goboringcrypto">Go+BoringCrypto</a> GOEXPERIMENT, or on one
of its forks, as part of their FIPS 140 compliance strategy. Unlike the FIPS
140-3 Go Cryptographic Module, Go+BoringCrypto was never officially supported
and had significant developer experience issues, since it was produced
exclusively for the internal needs of Google. It will be removed in a future
release once Google migrates to the native module.</p>
<h2 id="a-native-developer-experience">A native developer experience</h2>
<p>The module integrates completely transparently into Go applications. In fact,
every Go program built with Go 1.24 already uses it for all FIPS 140-3 approved
algorithms! The module is just another name for the
<code>crypto/internal/fips140/...</code> packages of the standard library, which provide
the implementation of operations exposed by packages such as <code>crypto/ecdsa</code> and
<code>crypto/rand</code>.</p>
<p>These packages involve no cgo, meaning they cross-compile like any other Go
program, they pay no FFI performance overhead, and they don’t suffer from
<a href="/blog/tob-crypto-audit#cgo-memory-management">memory management security issues</a>, unlike Go+BoringCrypto and its forks.</p>
<p>When starting a Go binary, the module can be put into FIPS 140-3 mode with the
<code>fips140=on</code> <a href="/doc/godebug">GODEBUG option</a>, which can be set as an environment variable or
through the <code>go.mod</code> file. If FIPS 140-3 mode is enabled, the module will use
the NIST DRBG for randomness, <code>crypto/tls</code> will automatically only negotiate
FIPS 140-3 approved TLS versions and algorithms, and it will perform the
mandatory self-tests on initialization and during key generation. That’s it;
there are no other behavior differences.</p>
<p>There is also an experimental stricter mode, <code>fips140=only</code>, which causes all
non-approved algorithms to return errors or panic. We understand this might be
too inflexible for most deployments and are <a href="/issue/74630">looking for
feedback</a> on what a policy enforcement framework
might look like.</p>
<p>Finally, applications can use the <a href="/doc/security/fips140#the-gofips140-environment-variable"><code>GOFIPS140</code> environment
variable</a>
to build against older, validated versions of the <code>crypto/internal/fips140/...</code>
packages. <code>GOFIPS140</code> works like <code>GOOS</code> and <code>GOARCH</code>, and if set to
<code>GOFIPS140=v1.0.0</code> the program will be built against the v1.0.0 snapshot of the
packages as they were submitted for validation to CMVP. This snapshot ships with
the rest of the Go standard library, as <code>lib/fips140/v1.0.0.zip</code>.</p>
<p>When using <code>GOFIPS140</code>, the <code>fips140</code> GODEBUG defaults to <code>on</code>, so putting it
all together, all that’s needed to build against the FIPS 140-3 module and run
in FIPS 140-3 mode is <code>GOFIPS140=v1.0.0 go build</code>. That’s it.</p>
<p>If a toolchain is built with <code>GOFIPS140</code> set, all builds it produces will
default to that value.</p>
<p>The <code>GOFIPS140</code> version used to build a binary can be verified with
<code>go version -m</code>.</p>
<p>Future versions of Go will continue shipping and working with v1.0.0 of the Go
Cryptographic Module until the next version is fully certified by Geomys, but
some new cryptography features might not be available when building against old
modules. Starting with Go 1.24.3, you can use <code>GOFIPS140=inprocess</code> to
dynamically select the latest module for which a Geomys validation has reached
the In Process stage. Geomys plans to validate new module versions at least
every year—to avoid leaving FIPS 140 builds too far behind—and every time a
vulnerability in the module can’t be mitigated in the calling standard library
code.</p>
<h2 id="uncompromising-security">Uncompromising security</h2>
<p>Our first priority in developing the module has been matching or exceeding the
security of the existing Go standard library cryptography packages. It might be
surprising, but sometimes the easiest way to achieve and demonstrate compliance
with the FIPS 140 security requirements is not to exceed them. We declined to
accept that.</p>
<p>For example, <code>crypto/ecdsa</code> <a href="https://cs.opensource.google/go/go/+/refs/tags/go1.23.0:src/crypto/ecdsa/ecdsa.go;l=417" rel="noreferrer" target="_blank">always produced hedged signatures</a>. Hedged
signatures generate nonces by combining the private key, the message, and random
bytes. Like <a href="https://www.rfc-editor.org/rfc/rfc6979" rel="noreferrer" target="_blank">deterministic ECDSA</a>, they protect against failure of the
random number generator, which would otherwise leak the private key(!). Unlike
deterministic ECDSA, they are also resistant to <a href="https://github.com/MystenLabs/ed25519-unsafe-libs" rel="noreferrer" target="_blank">API issues</a> and <a href="https://en.wikipedia.org/wiki/Differential_fault_analysis" rel="noreferrer" target="_blank">fault
attacks</a>, and they don’t leak message equality. FIPS 186-5 introduced support
for <a href="https://www.rfc-editor.org/rfc/rfc6979" rel="noreferrer" target="_blank">RFC 6979</a> deterministic ECDSA, but not for hedged ECDSA.</p>
<p>Instead of downgrading to regular randomized or deterministic ECDSA signatures
in FIPS 140-3 mode (or worse, across modes), we <a href="https://github.com/golang/go/commit/9776d028f4b99b9a935dae9f63f32871b77c49af" rel="noreferrer" target="_blank">switched the hedging
algorithm</a> and connected dots across half a dozen documents to <a href="https://github.com/cfrg/draft-irtf-cfrg-det-sigs-with-noise/issues/6#issuecomment-2067819904" rel="noreferrer" target="_blank">prove the new
one is a compliant composition of a DRBG and traditional ECDSA</a>. While at it,
we also <a href="/doc/go1.24#cryptoecdsapkgcryptoecdsa">added opt-in support for deterministic signatures</a>.</p>
<p>Another example is random number generation. FIPS 140-3 has strict rules on how
cryptographic randomness is generated, which essentially enforce the use of a
userspace <a href="https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator" rel="noreferrer" target="_blank">CSPRNG</a>. Conversely, we believe the kernel is best suited to
produce secure random bytes, because it’s best positioned to collect entropy
from the system, and to detect when processes or even virtual machines are
cloned (which could lead to reuse of supposedly random bytes). Hence,
<a href="https://pkg.go.dev/crypto/rand" rel="noreferrer" target="_blank">crypto/rand</a> routes every read operation to the kernel.</p>
<p>To square this circle, in FIPS 140-3 mode we maintain a compliant userspace NIST
DRBG based on AES-256-CTR, and then inject into it 128 bits sourced from the
kernel at every read operation. This extra entropy is considered “uncredited”
additional data for FIPS 140-3 purposes, but in practice makes it as strong as
reading directly from the kernel—even if slower.</p>
<p>Finally, all of the Go Cryptographic Module v1.0.0 was in scope for the <a href="/blog/tob-crypto-audit">recent
security audit by Trail of Bits</a>, and was
not affected by the only non-informational finding.</p>
<p>Combined with the memory safety guarantees provided by the Go compiler and
runtime, we believe this delivers on our goal of making Go one of the easiest,
most secure solutions for FIPS 140 compliance.</p>
<h2 id="broad-platform-support">Broad platform support</h2>
<p>A FIPS 140-3 module is only compliant if operated on a tested or “Vendor
Affirmed” Operating Environment, essentially a combination of operating system
and hardware platform. To enable as many Go use cases as possible, the Geomys
validation is tested on <a href="https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?product=19371&amp;displayMode=Aggregated" rel="noreferrer" target="_blank">one of the most comprehensive sets of Operating
Environments</a> in the industry.</p>
<p>Geomys’s laboratory tested various Linux flavors (Alpine Linux on Podman, Amazon
Linux, Google Prodimage, Oracle Linux, Red Hat Enterprise Linux, and SUSE Linux
Enterprise Server), macOS, Windows, and FreeBSD on a mix of x86-64 (AMD and
Intel), ARMv8/9 (Ampere Altra, Apple M, AWS Graviton, and Qualcomm Snapdragon),
ARMv7, MIPS, z/ Architecture, and POWER, for a total of 23 tested environments.</p>
<p>Some of these were paid for by stakeholders, others were funded by Geomys for
the benefit of the Go community.</p>
<p>Moreover, the Geomys validation lists a broad set of generic platforms as Vendor
Affirmed Operating Environments:</p>
<ul>
<li>Linux 3.10+ on x86-64 and ARMv7/8/9,</li>
<li>macOS 11–15 on Apple M processors,</li>
<li>FreeBSD 12–14 on x86-64,</li>
<li>Windows 10 and Windows Server 2016–2022 on x86-64, and</li>
<li>Windows 11 and Windows Server 2025 on x86-64 and ARMv8/9.</li>
</ul>
<h2 id="comprehensive-algorithm-coverage">Comprehensive algorithm coverage</h2>
<p>It may be surprising, but even using a FIPS 140-3 approved algorithm implemented
by a FIPS 140-3 module on a supported Operating Environment is not necessarily
enough for compliance; the algorithm must have been specifically covered by
testing as part of validation. Hence, to make it as easy as possible to build
FIPS 140 compliant applications in Go, all FIPS 140-3 approved algorithms in the
standard library are implemented by the Go Cryptographic Module and were tested
as part of the validation, from digital signatures to the TLS key schedule.</p>
<p>The post-quantum ML-KEM key exchange (FIPS 203), <a href="/doc/go1.24#crypto-mlkem">introduced in Go 1.24</a>, is also validated, meaning <code>crypto/tls</code> can establish FIPS 140-3
compliant post-quantum secure connections with X25519MLKEM768.</p>
<p>In some cases, we validated the same algorithms under multiple different NIST
designations, to make it possible to use them in full compliance for different
purposes. For example, <a href="https://words.filippo.io/dispatches/fips-hkdf/" rel="noreferrer" target="_blank">HKDF is tested and validated under <em>four</em> names</a>:
SP 800-108 Feedback KDF, SP 800-56C two-step KDF, Implementation Guidance D.P
OneStepNoCounter KDF, and SP 800-133 Section 6.3 KDF.</p>
<p>Finally, we validated some internal algorithms such as CMAC Counter KDF, to make
it possible to expose future functionality such as <a href="https://c2sp.org/XAES-256-GCM" rel="noreferrer" target="_blank">XAES-256-GCM</a>.</p>
<p>Overall, the native FIPS 140-3 module delivers a better compliance profile than
Go+BoringCrypto, while making more algorithms available to FIPS 140-3 restricted
applications.</p>
<p>We look forward to the new native Go Cryptographic Module making it easier and
safer for Go developers to run FIPS 140 compliant workloads.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/go1.25">Go 1.25 is released</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/generic-interfaces">Generic interfaces</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Generic interfaces</title><id>tag:blog.golang.org,2013:blog.golang.org/generic-interfaces</id><link rel="alternate" href="https://go.dev/blog/generic-interfaces"></link><published>2025-07-07T00:00:00+00:00</published><updated>2025-07-07T00:00:00+00:00</updated><author><name>Axel Wagner</name></author><summary type="html">Adding type parameters to interface types is surprisingly powerful</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/generic-interfaces">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Generic interfaces</h1>
 
 <p class="author">
 Axel Wagner<br>
 7 July 2025
 </p>
 
 <div class='markdown'>
<p>There is an idea that is not obvious until you hear about it for the first time: as interfaces are types themselves, they too can have type parameters.
This idea proves to be surprisingly powerful when it comes to expressing constraints on generic functions and types.
In this post, we&rsquo;ll demonstrate it, by discussing the use of interfaces with type parameters in a couple of common scenarios.</p>
<h2 id="a-simple-tree-set">A simple tree set</h2>
<p>As a motivating example, assume we need a generic version of a <a href="https://en.wikipedia.org/wiki/Binary_search_tree" rel="noreferrer" target="_blank">binary search tree</a>.
The elements stored in such a tree need to be ordered, so our type parameter needs a constraint that determines the ordering to use.
A simple option is to use the <a href="/pkg/cmp#Ordered">cmp.Ordered</a> constraint, introduced in Go 1.21.
It restricts a type parameter to ordered types (strings and numbers) and allows methods of the type to use the built-in ordering operators.</p>
<pre><code>// The zero value of a Tree is a ready-to-use empty tree.
type Tree[E cmp.Ordered] struct {
 root *node[E]
}

func (t *Tree[E]) Insert(element E) {
 t.root = t.root.insert(element)
}

type node[E cmp.Ordered] struct {
 value E
 left *node[E]
 right *node[E]
}

func (n *node[E]) insert(element E) *node[E] {
 if n == nil {
 return &amp;node[E]{value: element}
 }
 switch {
 case element &lt; n.value:
 n.left = n.left.insert(element)
 case element &gt; n.value:
 n.right = n.right.insert(element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/H7-n33X7P2h">playground</a>)</p>
<p>However, this approach has the disadvantage that it only works on basic types for which <code>&lt;</code> is defined;
you cannot insert struct types, like <a href="/pkg/time#Time">time.Time</a>.</p>
<p>We can remedy that by requiring the user to provide a comparison function:</p>
<pre><code>// A FuncTree must be created with NewTreeFunc.
type FuncTree[E any] struct {
 root *funcNode[E]
 cmp func(E, E) int
}

func NewFuncTree[E any](cmp func(E, E) int) *FuncTree[E] {
 return &amp;FuncTree[E]{cmp: cmp}
}

func (t *FuncTree[E]) Insert(element E) {
 t.root = t.root.insert(t.cmp, element)
}

type funcNode[E any] struct {
 value E
 left *funcNode[E]
 right *funcNode[E]
}

func (n *funcNode[E]) insert(cmp func(E, E) int, element E) *funcNode[E] {
 if n == nil {
 return &amp;funcNode[E]{value: element}
 }
 sign := cmp(element, n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(cmp, element)
 case sign &gt; 0:
 n.right = n.right.insert(cmp, element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/tiEjuxCHtFF">playground</a>)</p>
<p>This works, but it also comes with downsides.
We can no longer use the zero value of our container type, because it needs to have an explicitly initialized comparison function.
And the use of a function field makes it harder for the compiler to inline the comparison calls, which can introduce a significant runtime overhead.</p>
<p>Using a method on the element type can solve these issues, because methods are directly associated with a type.
A method does not have to be explicitly passed and the compiler can see the target of the call and may be able to inline it.
But how can we express the constraint to require that element types provide the necessary method?</p>
<h2 id="using-the-receiver-in-constraints">Using the receiver in constraints</h2>
<p>The first approach we might try is to define a plain old interface with a <code>Compare</code> method:</p>
<pre><code>type Comparer interface {
 Compare(Comparer) int
}
</code></pre>
<p>However, we quickly realize that this does not work well.
To implement this interface, the method&rsquo;s parameter must itself be <code>Comparer</code>.
Not only does that mean that the implementation of this method must type-assert the parameter to its own type, it also requires that every type must explicitly refer to our package with the <code>Comparer</code> type by name (otherwise the method signatures would not be identical).
That is not very orthogonal.</p>
<p>A better approach is to make the <code>Comparer</code> interface itself generic:</p>
<pre><code>type Comparer[T any] interface {
 Compare(T) int
}
</code></pre>
<p>This <code>Comparer</code> now describes a whole family of interfaces, one for each type that <code>Comparer</code> may be instantiated with.
A type that implements <code>Comparer[T]</code> declares &ldquo;I can compare myself to a <code>T</code>&rdquo;.
For instance, <code>time.Time</code> naturally implements <code>Comparer[time.Time]</code> because <a href="/pkg/time#Time.Compare">it has a matching <code>Compare</code> method</a>:</p>
<pre><code>// Implements Comparer[Time]
func (t Time) Compare(u Time) int
</code></pre>
<p>This is better, but not enough.
What we really want is a constraint that says that a type parameter can be compared to <em>itself</em>: we want the constraint to be self-referential.
The subtle insight is that the self-referential aspect does not have to be part of the interface definition itself; specifically, the constraint for <code>T</code> in the <code>Comparer</code> type is just <code>any</code>.
Instead, it is a consequence of how we use <code>Comparer</code> as a constraint for the type parameter of <code>MethodTree</code>:</p>
<pre><code>// The zero value of a MethodTree is a ready-to-use empty tree.
type MethodTree[E Comparer[E]] struct {
 root *methodNode[E]
}

func (t *MethodTree[E]) Insert(element E) {
 t.root = t.root.insert(element)
}

type methodNode[E Comparer[E]] struct {
 value E
 left *methodNode[E]
 right *methodNode[E]
}

func (n *methodNode[E]) insert(element E) *methodNode[E] {
 if n == nil {
 return &amp;methodNode[E]{value: element}
 }
 sign := element.Compare(n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(element)
 case sign &gt; 0:
 n.right = n.right.insert(element)
 }
 return n
}
</code></pre>
<p>(<a href="/play/p/LuhzYej_2SP">playground</a>)</p>
<p>Because <code>time.Time</code> implements <code>Comparer[time.Time]</code> it is now a valid type argument for this container, and we can still use the zero value as an empty container:</p>
<pre><code>var t MethodTree[time.Time]
t.Insert(time.Now())
</code></pre>
<p>For full flexibility, a library can provide all three API versions.
If we want to minimize repetition, all versions could use a shared implementation.
We could use the function version for that, as it is the most general:</p>
<pre><code>type node[E any] struct {
 value E
 left *node[E]
 right *node[E]
}

func (n *node[E]) insert(cmp func(E, E) int, element E) *node[E] {
 if n == nil {
 return &amp;node[E]{value: element}
 }
 sign := cmp(element, n.value)
 switch {
 case sign &lt; 0:
 n.left = n.left.insert(cmp, element)
 case sign &gt; 0:
 n.right = n.right.insert(cmp, element)
 }
 return n
}

// Insert inserts element into the tree, if E implements cmp.Ordered.
func (t *Tree[E]) Insert(element E) {
 t.root = t.root.insert(cmp.Compare[E], element)
}

// Insert inserts element into the tree, using the provided comparison function.
func (t *FuncTree[E]) Insert(element E) {
 t.root = t.root.insert(t.cmp, element)
}

// Insert inserts element into the tree, if E implements Comparer[E].
func (t *MethodTree[E]) Insert(element E) {
 t.root = t.root.insert(E.Compare, element)
}
</code></pre>
<p>(<a href="/play/p/jzmoaH5eaIv">playground</a>)</p>
<p>An important observation here is that the shared implementation (the function-based variant) is not constrained in any way.
It must remain maximally flexible to serve as a common core.
We also do not store the comparison function in a struct field.
Instead, we pass it as a parameter because function arguments are easier for the compiler to analyze than struct fields.</p>
<p>There is still some amount of boilerplate involved, of course.
All the exported implementations need to replicate the full API with slightly different call patterns.
But this part is straightforward to write and to read.</p>
<h2 id="combining-methods-and-type-sets">Combining methods and type sets</h2>
<p>We can use our new tree data structure to implement an ordered set, providing element lookup in logarithmic time.
Let&rsquo;s now imagine we need to make lookup run in constant time; we might try to do this by maintaining an ordinary Go map alongside the tree:</p>
<pre><code>type OrderedSet[E Comparer[E]] struct {
 tree MethodTree[E] // for efficient iteration in order
 elements map[E]bool // for (near) constant time lookup
}

func (s *OrderedSet[E]) Has(e E) bool {
 return s.elements[e]
}

func (s *OrderedSet[E]) Insert(e E) {
 if s.elements == nil {
 s.elements = make(map[E]bool)
 }
 if s.elements[e] {
 return
 }
 s.elements[e] = true
 s.tree.Insert(e)
}

func (s *OrderedSet[E]) All() iter.Seq[E] {
 return func(yield func(E) bool) {
 s.tree.root.all(yield)
 }
}

func (n *node[E]) all(yield func(E) bool) bool {
 return n == nil || (n.left.all(yield) &amp;&amp; yield(n.value) &amp;&amp; n.right.all(yield))
}
</code></pre>
<p>(<a href="/play/p/TANUnnSnDqf">playground</a>)</p>
<p>However, compiling this code will produce an error:</p>
<blockquote>
<p>invalid map key type E (missing comparable constraint)</p>
</blockquote>
<p>The error message tells us that we need to further constrain our type parameter to be able to use it as a map key.
The <code>comparable</code> constraint is a special predeclared constraint that is satisfied by all types for which the equality operators <code>==</code> and <code>!=</code> are defined.
In Go, that is also the set of types which can be used as keys for the built-in <code>map</code> type.</p>
<p>We have three options to add this constraint to our type parameter, all with different tradeoffs:</p>
<ol>
<li>
<p>We can <a href="/ref/spec#Embedded_interfaces">embed</a> <code>comparable</code> into our original <code>Comparer</code> definition (<a href="/play/p/g8NLjZCq97q">playground</a>):</p>
<pre><code>type Comparer[E any] interface {
 comparable
 Compare(E) int
}
</code></pre>
<p>This has the downside that it would also make our <code>Tree</code> types only usable with types that are <code>comparable</code>.
In general, we do not want to unnecessarily restrict generic types.</p>
</li>
<li>
<p>We can add a new constraint definition (<a href="/play/p/Z2eg4X8xK5Z">playground</a>).</p>
<pre><code>type Comparer[E any] interface {
 Compare(E) int
}

type ComparableComparer[E any] interface {
 comparable
 Comparer[E]
}
</code></pre>
<p>This is tidy, but it introduces a new identifier (<code>ComparableComparer</code>) into our API, and naming is hard.</p>
</li>
<li>
<p>We can add the constraint inline into our more constrained type (<a href="/play/p/ZfggVma_jNc">playground</a>):</p>
<pre><code>type OrderedSet[E interface {
 comparable
 Comparer[E]
}] struct {
 tree Tree[E]
 elements map[E]struct{}
}
</code></pre>
<p>This can become a bit hard to read, especially if it needs to happen often.
It also makes it harder to reuse the constraint in other places.</p>
</li>
</ol>
<p>Which of these to use is a style choice and ultimately up to personal preference.</p>
<h2 id="not-constraining-generic-interfaces">(Not) constraining generic interfaces</h2>
<p>At this point it is worth discussing constraints on generic interfaces.
You might want to define an interface for a generic container type.
For example, say you have an algorithm that requires a set data structure.
There are many different kinds of set implementations with different tradeoffs.
Defining an interface for the set operations you require can add flexibility to your package, leaving the decision of what tradeoffs are right for the specific application to the user:</p>
<pre><code>type Set[E any] interface {
 Insert(E)
 Delete(E)
 Has(E) bool
 All() iter.Seq[E]
}
</code></pre>
<p>A natural question here is what the constraint on this interface should be.
If possible, type parameters on generic interfaces should use <code>any</code> as a constraint, allowing arbitrary types.</p>
<p>From our discussions above, the reasons should be clear:
Different concrete implementations might require different constraints.
All the <code>Tree</code> types we have examined above, as well as the <code>OrderedSet</code> type, can implement <code>Set</code> for their element types, even though these types have different constraints.</p>
<p>The point of defining an interface is to leave the implementation up to the user.
Since one cannot predict what kinds of constraints a user may want to impose on their implementation, try to leave any constraints (stronger than <code>any</code>) to concrete implementations, not the interfaces.</p>
<h2 id="pointer-receivers">Pointer receivers</h2>
<p>Let us try to use the <code>Set</code> interface in an example.
Consider a function that removes duplicate elements in a sequence:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E comparable](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 seen := make(map[E]bool)
 for v := range input {
 if seen[v] {
 continue
 }
 if !yield(v) {
 return
 }
 seen[v] = true
 }
 }
}
</code></pre>
<p>(<a href="/play/p/hsYoFjkU9kA">playground</a>)</p>
<p>This uses a <code>map[E]bool</code> as a simple set of <code>E</code> elements.
Consequently, it works only for types that are <code>comparable</code> and which therefore define the built-in equality operators.
If we want to generalize this to arbitrary types, we need to replace that with a generic set:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E any](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 var seen Set[E]
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/FZYPNf56nnY">playground</a>)</p>
<p>However, this does not work.
<code>Set[E]</code> is an interface type, and the <code>seen</code> variable will be initialized to <code>nil</code>.
We need to use a concrete implementation of the <code>Set[E]</code> interface.
But as we have seen in this post, there is no general implementation of a set that works for <code>any</code> element type.</p>
<p>We have to ask the user to provide a concrete implementation we can use, as an extra type parameter:</p>
<pre><code>// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E any, S Set[E]](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 var seen S
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/kjkGy5cNz8T">playground</a>)</p>
<p>However, if we instantiate this with our set implementation, we run into another problem:</p>
<pre><code>// OrderedSet[E] does not satisfy Set[E] (method All has pointer receiver)
Unique[E, OrderedSet[E]](slices.Values(s))
// panic: invalid memory address or nil pointer dereference
Unique[E, *OrderedSet[E]](slices.Values(s))
</code></pre>
<p>The first problem is clear from the error message: Our type constraint says that the type argument for <code>S</code> needs to implement the <code>Set[E]</code> interface.
And as the methods on <code>OrderedSet</code> use a pointer receiver, the type argument also has to be the pointer type.</p>
<p>When trying to do that, we run into the second problem.
This stems from the fact that we declare a variable in the implementation:</p>
<pre><code>var seen S
</code></pre>
<p>If <code>S</code> is <code>*OrderedSet[E]</code>, the variable is initialized with <code>nil</code>, as before.
Calling <code>seen.Insert</code> panics.</p>
<p>If we only have the pointer type, we cannot get a valid variable of the value type.
And if we only have the value type, we cannot call pointer-methods on it.
The consequence is that we need both the value <em>and</em> the pointer type.
So we have to introduce an additional type parameter <code>PS</code> with a new constraint <code>PtrToSet</code>:</p>
<pre><code>// PtrToSet is implemented by a pointer type implementing the Set[E] interface.
type PtrToSet[S, E any] interface {
 *S
 Set[E]
}

// Unique removes duplicate elements from the input sequence, yielding only
// the first instance of any element.
func Unique[E, S any, PS PtrToSet[S, E]](input iter.Seq[E]) iter.Seq[E] {
 return func(yield func(E) bool) {
 // We convert to PS, as only that is constrained to have the methods.
 // The conversion is allowed, because the type set of PS only contains *S.
 seen := PS(new(S))
 for v := range input {
 if seen.Has(v) {
 continue
 }
 if !yield(v) {
 return
 }
 seen.Insert(v)
 }
 }
}
</code></pre>
<p>(<a href="/play/p/Kp1jJRVjmYa">playground</a>)</p>
<p>The trick here is the connection of the two type parameters in the function signature via the extra type parameter on the <code>PtrToSet</code> interface.
<code>S</code> itself is unconstrained, but <code>PS</code> must have type <code>*S</code> and it must have the methods we need.
So effectively, we are restricting <code>S</code> to have some methods, but those methods need to use a pointer receiver.</p>
<p>While the definition of a function with this kind of constraint requires an additional type parameter, importantly this does not complicate code using it:
as long as this extra type parameter is at the end of the type parameter list, it <a href="/blog/type-inference">can be inferred</a>:</p>
<pre><code>// The third type argument is inferred to be *OrderedSet[int]
Unique[int, OrderedSet[int]](slices.Values(s))
</code></pre>
<p>This is a general pattern, and worth remembering: for when you encounter it in someone else&rsquo;s work, or when you want to use it in your own.</p>
<pre><code>func SomeFunction[T any, PT interface{ *T; SomeMethods }]()
</code></pre>
<p>If you have two type parameters, where one is constrained to be a pointer to the other, the constraint ensures that the relevant methods use a pointer receiver.</p>
<h2 id="should-you-constrain-to-pointer-receivers">Should you constrain to pointer receivers?</h2>
<p>At this point, you might feel pretty overwhelmed.
This is rather complicated and it seems unreasonable to expect every Go programmer to understand what is going on in this function signature.
We also had to introduce yet more names into our API.
When people cautioned against adding generics to Go in the first place, this is one of the things they were worried about.</p>
<p>So if you find yourself entangled in these problems, it is worth taking a step back.
We can often avoid this complexity by thinking about our problem in a different way.
In this example, we built a function that takes an <code>iter.Seq[E]</code> and returns an <code>iter.Seq[E]</code> with the unique elements.
But to do the deduplication, we needed to collect the unique elements into a set.
And as this requires us to allocate the space for the entire result, we do not really benefit from representing the result as a stream.</p>
<p>If we rethink this problem, we can avoid the extra type parameter altogether by using <code>Set[E]</code> as a regular interface value:</p>
<pre><code>// InsertAll adds all unique elements from seq into set.
func InsertAll[E any](set Set[E], seq iter.Seq[E]) {
 for v := range seq {
 set.Insert(v)
 }
}
</code></pre>
<p>(<a href="/play/p/woZcHodAgaa">playground</a>)</p>
<p>By using <code>Set</code> as a plain interface type, it is clear that the caller has to provide a valid value of their concrete implementation.
This is a very common pattern.
And if they need an <code>iter.Seq[E]</code>, they can simply call <code>All()</code> on the <code>set</code> to obtain one.</p>
<p>This complicates things for callers slightly, but it has another advantage over the constraint to pointer receivers:
remember that we started with a <code>map[E]bool</code> as a simple set type.
It is easy to implement the <code>Set[E]</code> interface on that basis:</p>
<pre><code>type HashSet[E comparable] map[E]bool

func (s HashSet[E]) Insert(v E) { s[v] = true }
func (s HashSet[E]) Delete(v E) { delete(s, v) }
func (s HashSet[E]) Has(v E) bool { return s[v] }
func (s HashSet[E]) All() iter.Seq[E] { return maps.Keys(s) }
</code></pre>
<p>(<a href="/play/p/KPPpWa7M93d">playground</a>)</p>
<p>This implementation does not use pointer receivers.
So while this is perfectly valid, it would not be usable with the complicated constraint to pointer receivers.
But it works fine with our <code>InsertAll</code> version.
As with many constraints, enforcing that methods use a pointer receiver might actually be overly restrictive for many practical use cases.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope this illustrates some of the patterns and trade-offs that type parameters on interfaces enable.
It is a powerful tool, but it also comes with a cost.
The primary take-aways are:</p>
<ol>
<li>Use generic interfaces to express constraints on the receiver by using them self-referentially.</li>
<li>Use them to create constrained relationships between different type parameters.</li>
<li>Use them to abstract over different implementations with different kinds of constraints.</li>
<li>When you find yourself in a situation where you need to constrain to pointer receivers, consider whether you can refactor your code to avoid the extra complexity. See <a href="#should-you-constrain-to-pointer-receivers">&ldquo;Should you constrain to pointer receivers?&rdquo;</a>.</li>
</ol>
<p>As always, do not over-engineer things: a less flexible but simpler and more readable solution may ultimately be the wiser choice.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/fips140">The FIPS 140-3 Go Cryptographic Module</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/error-syntax">[ On | No ] syntactic support for error handling</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>[ On | No ] syntactic support for error handling</title><id>tag:blog.golang.org,2013:blog.golang.org/error-syntax</id><link rel="alternate" href="https://go.dev/blog/error-syntax"></link><published>2025-06-03T00:00:00+00:00</published><updated>2025-06-03T00:00:00+00:00</updated><author><name>Robert Griesemer</name></author><summary type="html">Go team plans around error handling support</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/error-syntax">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>[ On | No ] syntactic support for error handling</h1>
 
 <p class="author">
 Robert Griesemer<br>
 3 June 2025
 </p>
 
 <div class='markdown'>
<p>One of the oldest and most persistent complaints about Go concerns the verbosity of error handling.
We are all intimately (some may say painfully) familiar with this code pattern:</p>
<pre><code class="language-Go">x, err := call()
if err != nil {
 // handle err
}
</code></pre>
<p>The test <code>if err != nil</code> can be so pervasive that it drowns out the rest of the code.
This typically happens in programs that do a lot of API calls, and where handling errors
is rudimentary and they are simply returned.
Some programs end up with code that looks like this:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err := strconv.Atoi(a)
 if err != nil {
 return err
 }
 y, err := strconv.Atoi(b)
 if err != nil {
 return err
 }
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>Of the ten lines of code in this function body, only four (the calls and the last two lines) appear to do real work.
The remaining six lines come across as noise.
The verbosity is real, and so it&rsquo;s no wonder that complaints about error handling have topped
our annual user surveys for years.
(For a while, the lack of generics surpassed complaints about error handling, but now that
Go supports generics, error handling is back on top.)</p>
<p>The Go team takes community feedback seriously, and so for many years now we have tried to
come up with a solution for this problem, together with input from the Go community.</p>
<p>The first explicit attempt by the Go team dates back to 2018, when Russ Cox
<a href="https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.md" rel="noreferrer" target="_blank">formally described the problem</a>
as part of what we called the Go 2 effort at that time.
He outlined a possible solution based on a
<a href="https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling.md" rel="noreferrer" target="_blank">draft design</a>
by Marcel van Lohuizen.
The design was based on a <code>check</code> and <code>handle</code> mechanism and was fairly comprehensive.
The draft includes a detailed analysis of alternative solutions, including comparisons with
approaches taken by other languages.
If you&rsquo;re wondering if your particular error handling idea was previously considered,
read this document!</p>
<pre><code class="language-Go">// printSum implementation using the proposed check/handle mechanism.
func printSum(a, b string) error {
 handle err { return err }
 x := check strconv.Atoi(a)
 y := check strconv.Atoi(b)
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>The <code>check</code> and <code>handle</code> approach was deemed too complicated and almost a year later, in 2019,
we followed up with the much simplified and by now
<a href="/issue/32437#issuecomment-2278932700">infamous</a>
<a href="https://go.googlesource.com/proposal/+/master/design/32437-try-builtin.md" rel="noreferrer" target="_blank"><code>try</code> proposal</a>.
It was based on the ideas of <code>check</code> and <code>handle</code>, but the <code>check</code> pseudo-keyword became
the <code>try</code> built-in function and the <code>handle</code> part was omitted.
To explore the impact of the <code>try</code> built-in, we wrote a simple tool
(<a href="https://github.com/griesemer/tryhard" rel="noreferrer" target="_blank">tryhard</a>)
that rewrites existing error handling code using <code>try</code>.
The proposal was argued over intensively, approaching 900 comments on the <a href="/issue/32437">GitHub issue</a>.</p>
<pre><code class="language-Go">// printSum implementation using the proposed try mechanism.
func printSum(a, b string) error {
 // use a defer statement to augment errors before returning
 x := try(strconv.Atoi(a))
 y := try(strconv.Atoi(b))
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>However, <code>try</code> affected control flow by returning from the enclosing function in case of an error,
and did so from potentially deeply nested expressions, thus hiding this control flow from view.
This made the proposal unpalatable to many, and despite significant investment
into this proposal we decided to abandon this effort too.
In retrospect it might have been better to introduce a new keyword,
something that we could do now since we have fine-grained control over the language version
via <code>go.mod</code> files and file-specific directives.
Restricting the use of <code>try</code> to assignments and statements might have alleviated some
of the other concerns. A <a href="/issue/73376">recent proposal</a> by Jimmy Frasche, which essentially
goes back to the original <code>check</code> and <code>handle</code> design and addresses some of that design&rsquo;s
shortcomings, pursues that direction.</p>
<p>The repercussions of the <code>try</code> proposal led to much soul searching including a series of blog
posts by Russ Cox: <a href="https://research.swtch.com/proposals-intro" rel="noreferrer" target="_blank">&ldquo;Thinking about the Go Proposal Process&rdquo;</a>.
One conclusion was that we likely diminished our chances for a better outcome by presenting an almost
fully baked proposal with little space for community feedback and a &ldquo;threatening&rdquo; implementation
timeline. Per <a href="https://research.swtch.com/proposals-large" rel="noreferrer" target="_blank">&ldquo;Go Proposal Process: Large Changes&rdquo;</a>:
&ldquo;in retrospect, <code>try</code> was a large enough change that the new design we published [&hellip;] should have
been a second draft design, not a proposal with an implementation timeline&rdquo;.
But irrespective of a possible process and communication failure in this case, the user sentiment towards
the proposal was very strongly not in favor.</p>
<p>We didn&rsquo;t have a better solution at that time and didn&rsquo;t pursue syntax changes for error handling for several years.
Plenty of people in the community were inspired, though, and we received a steady trickle
of error handling proposals, many very similar to each other, some interesting, some incomprehensible,
and some infeasible.
To keep track of the expanding landscape, another year later, Ian Lance Taylor created an
<a href="/issue/40432">umbrella issue</a>
which summarizes the current state of proposed changes for improved error handling.
A <a href="/wiki/Go2ErrorHandlingFeedback">Go Wiki</a> was created to collect related feedback, discussions, and articles.
Independently, other people have started tracking all the many error handling proposals
over the years.
It&rsquo;s amazing to see the sheer volume of them all, for instance in Sean K. H. Liao&rsquo;s blog post on
<a href="https://seankhliao.com/blog/12020-11-23-go-error-handling-proposals/" rel="noreferrer" target="_blank">&ldquo;go error handling proposals&rdquo;</a>.</p>
<p>The complaints about the verbosity of error handling persisted
(see <a href="/blog/survey2024-h1-results">Go Developer Survey 2024 H1 Results</a>),
and so, after a series of increasingly refined Go team internal proposals, Ian Lance Taylor published
<a href="/issue/71203">&ldquo;reduce error handling boilerplate using <code>?</code>&rdquo;</a> in 2024.
This time the idea was to borrow from a construct implemented in
<a href="https://www.rust-lang.org/" rel="noreferrer" target="_blank">Rust</a>, specifically the
<a href="https://doc.rust-lang.org/std/result/index.html#the-question-mark-operator-" rel="noreferrer" target="_blank"><code>?</code> operator</a>.
The hope was that by leaning on an existing mechanism using an established notation, and taking into
account what we had learned over the years, we should be able to finally make some progress.
In small informal user studies where programmers were shown Go code using <code>?</code>, the vast majority
of participants correctly guessed the meaning of the code, which further convinced us to give it another
shot.
To be able to see the impact of the change, Ian wrote a tool that converts ordinary Go code
into code that uses the proposed new syntax, and we also prototyped the feature in the
compiler.</p>
<pre><code class="language-Go">// printSum implementation using the proposed &quot;?&quot; statements.
func printSum(a, b string) error {
 x := strconv.Atoi(a) ?
 y := strconv.Atoi(b) ?
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
<p>Unfortunately, as with the other error handling ideas, this new proposal was also quickly overrun
with comments and many suggestions for minor tweaks, often based on individual preferences.
Ian closed the proposal and moved the content into a <a href="/issue/71460">discussion</a>
to facilitate the conversation and to collect further feedback.
A slightly modified version was received
<a href="https://github.com/golang/go/discussions/71460#discussioncomment-12060294" rel="noreferrer" target="_blank">a bit more positively</a>
but broad support remained elusive.</p>
<p>After so many years of trying, with three full-fledged proposals by the Go team and
literally <a href="/issues?q=+is%3Aissue+label%3Aerror-handling">hundreds</a> (!)
of community proposals, most of them variations on a theme,
all of which failed to attract sufficient (let alone overwhelming) support,
the question we now face is: how to proceed? Should we proceed at all?</p>
<p><em>We think not.</em></p>
<p>To be more precise, we should stop trying to solve the <em>syntactic problem</em>, at least for the foreseeable
future.
The <a href="https://github.com/golang/proposal?tab=readme-ov-file#consensus-and-disagreement" rel="noreferrer" target="_blank">proposal process</a>
provides justification for this decision:</p>
<blockquote>
<p>The goal of the proposal process is to reach general consensus about the outcome in a timely manner.
If proposal review cannot identify a general consensus in the discussion of the issue on the issue tracker,
the usual result is that the proposal is declined.</p>
</blockquote>
<p>Furthermore:</p>
<blockquote>
<p>It can happen that proposal review may not identify a general consensus and yet it is clear that the
proposal should not be outright declined.
[&hellip;]
If the proposal review group cannot identify a consensus nor a next step for the proposal,
the decision about the path forward passes to the Go architects [&hellip;], who review the discussion and
aim to reach a consensus among themselves.</p>
</blockquote>
<p>None of the error handling proposals reached anything close to a consensus,
so they were all declined.
Even the most senior members of the Go team at Google do not unanimously agree
on the best path forward <em>at this time</em> (perhaps that will change at some point).
But without a strong consensus we cannot reasonably move forward.</p>
<p>There are valid arguments in favor of the status quo:</p>
<ul>
<li>
<p>If Go had introduced specific syntactic sugar for error handling early on, few would argue over it today.
But we are 15 years down the road, the opportunity has passed, and Go has
a perfectly fine way to handle errors, even if it may seem verbose at times.</p>
</li>
<li>
<p>Looking from a different angle, let&rsquo;s assume we came across the perfect solution today.
Incorporating it into the language would simply lead from one unhappy group of users
(the one that roots for the change) to another (the one that prefers the status quo).
We were in a similar situation when we decided to add generics to the language, albeit with an
important difference:
today nobody is forced to use generics, and good generic libraries are written such that users
can mostly ignore the fact that they are generic, thanks to type inference.
On the contrary, if a new syntactic construct for error handling gets added to the language,
virtually everybody will need to start using it, lest their code become unidiomatic.</p>
</li>
<li>
<p>Not adding extra syntax is in line with one of Go&rsquo;s design rules:
do not provide multiple ways of doing the same thing.
There are exceptions to this rule in areas with high &ldquo;foot traffic&rdquo;: assignments come to mind.
Ironically, the ability to <em>redeclare</em> a variable in
<a href="/ref/spec#Short_variable_declarations">short variable declarations</a> (<code>:=</code>) was introduced to address a problem
that arose because of error handling:
without redeclarations, sequences of error checks require a differently named <code>err</code> variable for
each check (or additional separate variable declarations).
At that time, a better solution might have been to provide more syntactic support for error handling.
Then, the redeclaration rule may not have been needed, and with it gone, so would be various
associated <a href="/issue/377">complications</a>.</p>
</li>
<li>
<p>Going back to actual error handling code, verbosity fades into the background if errors are
actually <em>handled</em>.
Good error handling often requires additional information added to an error.
For instance, a recurring comment in user surveys is about the lack of stack traces associated
with an error.
This could be addressed with support functions that produce and return an augmented
error.
In this (admittedly contrived) example, the relative amount of boilerplate is much smaller:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err := strconv.Atoi(a)
 if err != nil {
 return fmt.Errorf(&quot;invalid integer: %q&quot;, a)
 }
 y, err := strconv.Atoi(b)
 if err != nil {
 return fmt.Errorf(&quot;invalid integer: %q&quot;, b)
 }
 fmt.Println(&quot;result:&quot;, x + y)
 return nil
}
</code></pre>
</li>
<li>
<p>New standard library functionality can help reduce error handling boilerplate as well,
very much in the vein of Rob Pike&rsquo;s 2015 blog post
<a href="/blog/errors-are-values">&ldquo;Errors are values&rdquo;</a>.
For instance, in some cases <a href="/pkg/cmp#Or"><code>cmp.Or</code></a> may be used to deal with a
series of errors all at once:</p>
<pre><code class="language-Go">func printSum(a, b string) error {
 x, err1 := strconv.Atoi(a)
 y, err2 := strconv.Atoi(b)
 if err := cmp.Or(err1, err2); err != nil {
 return err
 }
 fmt.Println(&quot;result:&quot;, x+y)
 return nil
}
</code></pre>
</li>
<li>
<p>Writing, reading, and debugging code are all quite different activities.
Writing repeated error checks can be tedious, but today&rsquo;s IDEs provide powerful, even LLM-assisted
code completion.
Writing basic error checks is straightforward for these tools.
The verbosity is most obvious when reading code, but tools might help here as well;
for instance an IDE with a Go language setting could provide a toggle switch to hide error handling
code.
Such switches already exist for other code sections such as function bodies.</p>
</li>
<li>
<p>When debugging error handling code, being able to quickly add a <code>println</code> or
have a dedicated line or source location for setting a breakpoint in a debugger is helpful.
This is easy when there is already a dedicated <code>if</code> statement.
But if all the error handling logic is hidden behind a <code>check</code>, <code>try</code>, or <code>?</code>, the code may have to
be changed into an ordinary <code>if</code> statement first, which complicates debugging
and may even introduce subtle bugs.</p>
</li>
<li>
<p>There are also practical considerations:
Coming up with a new syntax idea for error handling is cheap;
hence the proliferation of a multitude of proposals from the community.
Coming up with a good solution that holds up to scrutiny: not so much.
It takes a concerted effort to properly design a language change and to actually implement it.
The real cost still comes afterwards:
all the code that needs to be changed, the documentation that needs to be updated,
the tools that need to be adjusted.
Taken all into account, language changes are very expensive, the Go team is relatively small,
and there are a lot of other priorities to address.
(These latter points may change: priorities can shift, team sizes can go up or down.)</p>
</li>
<li>
<p>On a final note, some of us recently had the opportunity to attend
<a href="https://cloud.withgoogle.com/next/25" rel="noreferrer" target="_blank">Google Cloud Next 2025</a>,
where the Go team had a booth and where we also hosted a small Go Meetup.
Every single Go user we had a chance to ask was adamant that we should not change the
language for better error handling.
Many mentioned that the lack of specific error handling support in Go is most apparent
when coming freshly from another language that has that support.
As one becomes more fluent and writes more idiomatic Go code, the issue becomes much less important.
This is of course not a sufficiently large set of people to be representative,
but it may be a different set of people than we see on GitHub, and their feedback serves as yet another data point.</p>
</li>
</ul>
<p>Of course, there are also valid arguments in favor of change:</p>
<ul>
<li>
<p>Lack of better error handling support remains the top complaint in our user surveys.
If the Go team really does take user feedback seriously, we ought to do something about this eventually.
(Although there does not seem to be
<a href="https://github.com/golang/go/discussions/71460#discussioncomment-11977299" rel="noreferrer" target="_blank">overwhelming support</a>
for a language change either.)</p>
</li>
<li>
<p>Perhaps the singular focus on reducing the character count is misguided.
A better approach might be to make default error handling highly visible with a keyword
while still removing boilerplate (<code>err != nil</code>).
Such an approach might make it easier for a reader (a code reviewer!) to see that an error
is handled, without &ldquo;looking twice&rdquo;, resulting in improved code quality and safety.
This would bring us back to the beginnings of <code>check</code> and <code>handle</code>.</p>
</li>
<li>
<p>We don&rsquo;t really know how much the issue is the straightforward syntactic verbosity of
error checking, versus the verbosity of good error handling:
constructing errors that are a useful part of an API and meaningful to developers and
end-users alike.
This is something we&rsquo;d like to study in greater depth.</p>
</li>
</ul>
<p>Still, no attempt to address error handling so far has gained sufficient traction.
If we are honestly taking stock of where we are, we can only admit that we
neither have a shared understanding of the problem,
nor do we all agree that there is a problem in the first place.
With this in mind, we are making the following pragmatic decision:</p>
<p><em>For the foreseeable future, the Go team will stop pursuing syntactic language changes
for error handling.
We will also close all open and incoming proposals that concern themselves primarily
with the syntax of error handling, without further investigation.</em></p>
<p>The community has put tremendous effort into exploring, discussing, and debating these issues.
While this may not have resulted in any changes to error handling syntax, these efforts have
resulted in many other improvements to the Go language and our processes.
Maybe, at some point in the future, a clearer picture will emerge on error handling.
Until then, we look forward to focusing this incredible passion on new opportunities
to make Go better for everyone.</p>
<p>Thank you!</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/generic-interfaces">Generic interfaces</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/tob-crypto-audit">Go Cryptography Security Audit</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>Go Cryptography Security Audit</title><id>tag:blog.golang.org,2013:blog.golang.org/tob-crypto-audit</id><link rel="alternate" href="https://go.dev/blog/tob-crypto-audit"></link><published>2025-05-19T00:00:00+00:00</published><updated>2025-05-19T00:00:00+00:00</updated><author><name>Roland Shoemaker and Filippo Valsorda</name></author><summary type="html">Go&#39;s cryptography libraries underwent an audit by Trail of Bits.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/tob-crypto-audit">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>Go Cryptography Security Audit</h1>
 
 <p class="author">
 Roland Shoemaker and Filippo Valsorda<br>
 19 May 2025
 </p>
 
 <div class='markdown'>
<p>Go ships with a full suite of cryptography packages in the standard library to help developers build secure applications. Google recently contracted the independent security firm <a href="https://www.trailofbits.com/" rel="noreferrer" target="_blank">Trail of Bits</a> to complete an audit of the core set of packages that are also validated as part of the <a href="/doc/go1.24#fips140">new native FIPS 140-3 module</a>. The audit produced a single low-severity finding, in the legacy and unsupported <a href="/doc/security/fips140#goboringcrypto">Go+BoringCrypto integration</a>, and a handful of informational findings. The full text of the audit report can be found <a href="https://github.com/trailofbits/publications/blob/d47e8fafa7e3323e5620d228f2f3f3bf58ed5978/reviews/2025-03-google-gocryptographiclibraries-securityreview.pdf" rel="noreferrer" target="_blank">here</a>.</p>
<p>The scope of the audit included our implementations of key exchange (ECDH and post-quantum ML-KEM), digital signature (ECDSA, RSA, and Ed25519), encryption (AES-GCM, AES-CBC, and AES-CTR), hashing (SHA-1, SHA-2, and SHA-3), key derivation (HKDF and PBKDF2), and authentication (HMAC), as well as the cryptographic random number generator. Low-level big integer and elliptic curve implementations, with their delicate assembly cores, were included. Higher level protocols like TLS and X.509 were not in scope. Three Trail of Bits engineers worked on the audit for a month.</p>
<p>We are proud of the security track record of the Go cryptography packages, and of the outcome of this audit, which is just one of many ways we gain assurance of the packages’ correctness. First, we aggressively limit their complexity, guided by the <a href="/design/cryptography-principles">Cryptography Principles</a> which for example prioritize security over performance. Further, we <a href="https://www.youtube.com/watch?v=lkEH3V3PkS0" rel="noreferrer" target="_blank">thoroughly test them</a> with an array of different techniques. We make a point of leveraging safe APIs even for internal packages, and naturally we can rely on the Go language properties to avoid memory management issues. Finally, we focus on readability to make maintenance easier and code review and audits more effective.</p>
<h2 id="one-low-severity-finding">One low-severity finding</h2>
<p>The only potentially exploitable issue, TOB-GOCL-3, has <em>low severity</em>, meaning it had minor impact and was difficult to trigger. This issue has been fixed in Go 1.24.</p>
<p>Crucially, TOB-GOCL-3 (<a href="#cgo-memory-management">discussed further below</a>) concerns memory management in the <a href="/doc/security/fips140#goboringcrypto">legacy Go+BoringCrypto GOEXPERIMENT</a>, which is not enabled by default and unsupported for use outside of Google.</p>
<h2 id="five-informational-findings">Five informational findings</h2>
<p>The remaining findings are <em>informational</em>, meaning they do not pose an immediate risk but are relevant to security best practices. We addressed these in the current Go 1.25 development tree.</p>
<p>Findings TOB-GOCL-1, TOB-GOCL-2, and TOB-GOCL-6 concern possible timing side-channels in various cryptographic operations. Of these three findings, only TOB-GOCL-2 affects operations that were expected to be constant time due to operating on secret values, but it only affects Power ISA targets (GOARCH ppc64 and ppc64le). TOB-GOCL-4 highlights misuse risk in an internal API, should it be repurposed beyond its current use case. TOB-GOCL-5 points out a missing check for a limit that is impractical to reach.</p>
<h2 id="timing-side-channels">Timing Side-Channels</h2>
<p>Findings TOB-GOCL-1, TOB-GOCL-2, and TOB-GOCL-6 concern minor timing side-channels. TOB-GOCL-1 and TOB-GOCL-6 are related to functions which we do not use for sensitive values, but could be used for such values in the future, and TOB-GOCL-2 is related to the assembly implementation of P-256 ECDSA on Power ISA.</p>
<h3 id="cryptoecdhcryptoecdsa-conversion-from-bytes-to-field-elements-is-not-constant-time-tob-gocl-1"><code>crypto/ecdh,crypto/ecdsa</code>: conversion from bytes to field elements is not constant time (TOB-GOCL-1)</h3>
<p>The internal implementation of NIST elliptic curves provided a method to convert field elements between an internal and external representation which operated in variable time.</p>
<p>All usages of this method operated on public inputs which are not considered secret (public ECDH values, and ECDSA public keys), so we determined that this was not a security issue. That said, we decided to <a href="/cl/650579">make the method constant time anyway</a>, in order to prevent accidentally using this method in the future with secret values, and so that we don&rsquo;t have to think about whether it is an issue or not.</p>
<h3 id="cryptoecdsa-p-256-conditional-negation-is-not-constant-time-in-power-isa-assembly-tob-gocl-2-cve-2025-22866"><code>crypto/ecdsa</code>: P-256 conditional negation is not constant time in Power ISA assembly (TOB-GOCL-2, CVE-2025-22866)</h3>
<p>Beyond the <a href="/wiki/PortingPolicy#first-class-ports">first class Go platforms</a>, Go also supports a number of additional platforms, including some less common architectures. During the review of our assembly implementations of various underlying cryptographic primitives, the Trail of Bits team found one issue that affected the ECDSA implementation on the ppc64 and ppc64le architectures.</p>
<p>Due to the usage of a conditional branching instruction in the implementation of the conditional negation of P-256 points, the function operated in variable-time, rather than constant-time, as expected. The fix for this was relatively simple, <a href="/cl/643735">replacing the conditional branching instruction</a> with a pattern we already use elsewhere to conditionally select the correct result in constant time. We assigned this issue CVE-2025-22866.</p>
<p>To prioritize the code that reaches most of our users, and due to the specialized knowledge required to target specific ISAs, we generally rely on community contributions to maintain assembly for non-first class platforms. We thank our partners at IBM for helping provide review for our fix.</p>
<h3 id="cryptoed25519-scalarsetcanonicalbytes-is-not-constant-time-tob-gocl-6"><code>crypto/ed25519</code>: Scalar.SetCanonicalBytes is not constant time (TOB-GOCL-6)</h3>
<p>The internal edwards25519 package provided a method to convert between an internal and external representation of scalars which operated in variable time.</p>
<p>This method was only used on signature inputs to ed25519.Verify, which are not considered secret, so we determined that this was not a security issue. That said, similarly to the TOB-GOCL-1 finding, we decided to <a href="/cl/648035">make the method constant time anyway</a>, in order to prevent accidentally using this method in the future with secret values, and because we are aware that people fork this code outside of the standard library, and may be using it with secret values.</p>
<h2 id="cgo-memory-management">Cgo Memory Management</h2>
<p>Finding TOB-GOCL-3 concerns a memory management issue in the Go+BoringCrypto integration.</p>
<h3 id="cryptoecdh-custom-finalizer-may-free-memory-at-the-start-of-a-c-function-call-using-this-memory-tob-gocl-3"><code>crypto/ecdh</code>: custom finalizer may free memory at the start of a C function call using this memory (TOB-GOCL-3)</h3>
<p>During the review, there were a number of questions about our cgo-based Go+BoringCrypto integration, which provides a FIPS 140-2 compliant cryptography mode for internal usage at Google. The Go+BoringCrypto code is not supported by the Go team for external use, but has been critical for Google’s internal usage of Go.</p>
<p>The Trail of Bits team found one vulnerability and one <a href="/cl/644120">non-security relevant bug</a>, both of which were results of the manual memory management required to interact with a C library. Since the Go team does not support usage of this code outside of Google, we have chosen not to issue a CVE or Go vulnerability database entry for this issue, but we <a href="/cl/644119">fixed it in Go 1.24</a>.</p>
<p>This kind of pitfall is one of the many reasons that we decided to move away from the Go+BoringCrypto integration. We have been working on a <a href="/doc/security/fips140">native FIPS 140-3 mode</a> that uses the regular pure Go cryptography packages, allowing us to avoid the complex cgo semantics in favor of the traditional Go memory model.</p>
<h2 id="implementation-completeness">Implementation Completeness</h2>
<p>Findings TOB-GOCL-4 and TOB-GOCL-5 concern limited implementations of two specifications, <a href="https://csrc.nist.gov/pubs/sp/800/90/a/r1/final" rel="noreferrer" target="_blank">NIST SP 800-90A</a> and <a href="https://datatracker.ietf.org/doc/html/rfc8018" rel="noreferrer" target="_blank">RFC 8018</a>.</p>
<h3 id="cryptointernalfips140drbg-ctr-drbg-api-presents-multiple-misuse-risks-tob-gocl-4"><code>crypto/internal/fips140/drbg</code>: CTR_DRBG API presents multiple misuse risks (TOB-GOCL-4)</h3>
<p>As part of the <a href="/doc/security/fips140">native FIPS 140-3 mode</a> that we are introducing, we needed an implementation of the NIST CTR_DRBG (an AES-CTR based deterministic random bit generator) to provide compliant randomness.</p>
<p>Since we only need a small subset of the functionality of the NIST SP 800-90A Rev. 1 CTR_DRBG for our purposes, we did not implement the full specification, in particular omitting the derivation function and personalization strings. These features can be critical to safely use the DRBG in generic contexts.</p>
<p>As our implementation is tightly scoped to the specific use case we need, and since the implementation is not publicly exported, we determined that this was acceptable and worth the decreased complexity of the implementation. We do not expect this implementation to ever be used for other purposes internally, and have <a href="/cl/647815">added a warning to the documentation</a> that details these limitations.</p>
<h3 id="cryptopbkdf2-pbkdf2-does-not-enforce-output-length-limitations-tob-gocl-5"><code>crypto/pbkdf2</code>: PBKDF2 does not enforce output length limitations (TOB-GOCL-5)</h3>
<p>In Go 1.24, we began the process of moving packages from <a href="https://golang.org/x/crypto" rel="noreferrer" target="_blank">golang.org/x/crypto</a> into the standard library, ending a confusing pattern where first-party, high-quality, and stable Go cryptography packages were kept outside of the standard library for no particular reason.</p>
<p>As part of this process we moved the <a href="https://golang.org/x/crypto/pbkdf2" rel="noreferrer" target="_blank">golang.org/x/crypto/pbkdf2</a> package into the standard library, as crypto/pbkdf2. While reviewing this package, the Trail of Bits team noticed that we did not enforce the limit on the size of derived keys defined in <a href="https://datatracker.ietf.org/doc/html/rfc8018" rel="noreferrer" target="_blank">RFC 8018</a>.</p>
<p>The limit is <code>(2^32 - 1) * &lt;hash length&gt;</code>, after which the key would loop. When using SHA-256, exceeding the limit would take a key of more than 137GB. We do not expect anyone has ever used PBKDF2 to generate a key this large, especially because PBKDF2 runs the iterations at every block, but for the sake of correctness, we <a href="/cl/644122">now enforce the limit as defined by the standard</a>.</p>
<h1 id="whats-next">What’s Next</h1>
<p>The results of this audit validate the effort the Go team has put into developing high-quality, easy to use cryptography libraries and should provide confidence to our users who rely on them to build safe and secure software.</p>
<p>We’re not resting on our laurels, though: the Go contributors are continuing to develop and improve the cryptography libraries we provide users.</p>
<p>Go 1.24 now includes a FIPS 140-3 mode written in pure Go, which is currently undergoing CMVP testing. This will provide a supported FIPS 140-3 compliant mode for all users of Go, replacing the currently unsupported Go+BoringCrypto integration.</p>
<p>We are also working to implement modern post-quantum cryptography, introducing a ML-KEM-768 and ML-KEM-1024 implementation in Go 1.24 in the <a href="/pkg/crypto/mlkem">crypto/mlkem package</a>, and adding support to the crypto/tls package for the hybrid X25519MLKEM768 key exchange.</p>
<p>Finally, we are planning on introducing new easier to use high-level cryptography APIs, designed to reduce the barrier for picking and using high-quality algorithms for basic use cases. We plan to begin with offering a simple password hashing API that removes the need for users to decide which of the myriad of possible algorithms they should be relying on, with mechanisms to automatically migrate to newer algorithms as the state-of-the-art changes.</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/error-syntax">[ On | No ] syntactic support for error handling</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/testing-b-loop">More predictable benchmarking with testing.B.Loop</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry><entry><title>More predictable benchmarking with testing.B.Loop</title><id>tag:blog.golang.org,2013:blog.golang.org/testing-b-loop</id><link rel="alternate" href="https://go.dev/blog/testing-b-loop"></link><published>2025-04-02T00:00:00+00:00</published><updated>2025-04-02T00:00:00+00:00</updated><author><name>Junyang Shao</name></author><summary type="html">Better benchmark looping in Go 1.24.</summary><content type="html">
<div id="blog"><div id="content">
 <div id="content">

 <div class="Article" data-slug="/blog/testing-b-loop">
 
 <h1 class="small"><a href="/blog/">The Go Blog</a></h1>
 

 <h1>More predictable benchmarking with testing.B.Loop</h1>
 
 <p class="author">
 Junyang Shao<br>
 2 April 2025
 </p>
 
 <div class='markdown'>
<p>Go developers who have written benchmarks using the
<a href="https://pkg.go.dev/testing" rel="noreferrer" target="_blank"><code>testing</code></a> package might have encountered some of
its various pitfalls. Go 1.24 introduces a new way to write benchmarks that&rsquo;s just
as easy to use, but at the same time far more robust:
<a href="https://pkg.go.dev/testing#B.Loop" rel="noreferrer" target="_blank"><code>testing.B.Loop</code></a>.</p>
<p>Traditionally, Go benchmarks are written using a loop from 0 to <code>b.N</code>:</p>
<pre><code>func Benchmark(b *testing.B) {
 for range b.N {
 ... code to measure ...
 }
}
</code></pre>
<p>Using <code>b.Loop</code> instead is a trivial change:</p>
<pre><code>func Benchmark(b *testing.B) {
 for b.Loop() {
 ... code to measure ...
 }
}
</code></pre>
<p><code>testing.B.Loop</code> has many benefits:</p>
<ul>
<li>It prevents unwanted compiler optimizations within the benchmark loop.</li>
<li>It automatically excludes setup and cleanup code from benchmark timing.</li>
<li>Code can&rsquo;t accidentally depend on the total number of iterations or the current
iteration.</li>
</ul>
<p>These were all easy mistakes to make with <code>b.N</code>-style benchmarks that would
silently result in bogus benchmark results. As an added bonus, <code>b.Loop</code>-style
benchmarks even complete in less time!</p>
<p>Let&rsquo;s explore the advantages of <code>testing.B.Loop</code> and how to effectively utilize it.</p>
<h2 id="old-benchmark-loop-problems">Old benchmark loop problems</h2>
<p>Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated
benchmarks required more care:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 b.ResetTimer() // if setup may be expensive
 for range b.N {
 ... code to measure ...
 ... use sinks or accumulation to prevent dead-code elimination ...
 }
 b.StopTimer() // if cleanup or reporting may be expensive
 ... cleanup ...
 ... report ...
}
</code></pre>
<p>If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop
with <code>ResetTimer</code> and/or <code>StopTimer</code> calls. These are easy to forget, and even if the
developer remembers they may be necessary, it can be difficult to judge whether setup or
cleanup are &ldquo;expensive enough&rdquo; to require them.</p>
<p>Without these, the <code>testing</code> package can only time the entire benchmark function. If a
benchmark function omits them, the setup and cleanup code will be included in the overall
time measurement, silently skewing the final benchmark result.</p>
<p>There is another, more subtle pitfall that requires deeper understanding:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func isCond(b byte) bool {
 if b%3 == 1 &amp;&amp; b%7 == 2 &amp;&amp; b%17 == 11 &amp;&amp; b%31 == 9 {
 return true
 }
 return false
}

func BenchmarkIsCondWrong(b *testing.B) {
 for range b.N {
 isCond(201)
 }
}
</code></pre>
<p>In this example, the user might observe <code>isCond</code> executing in sub-nanosecond
time. CPUs are fast, but not that fast! This seemingly anomalous result stems
from the fact that <code>isCond</code> is inlined, and since its result is never used, the
compiler eliminates it as dead code. As a result, this benchmark doesn&rsquo;t measure <code>isCond</code>
at all; it measures how long it takes to do nothing. In this case, the sub-nanosecond
result is a clear red flag, but in more complex benchmarks, partial dead-code elimination
can lead to results that look reasonable but still aren&rsquo;t measuring what was intended.</p>
<h2 id="how-testingbloop-helps">How <code>testing.B.Loop</code> helps</h2>
<p>Unlike a <code>b.N</code>-style benchmark, <code>testing.B.Loop</code> is able to track when it is first called
in a benchmark when the final iteration ends. The <code>b.ResetTimer</code> at the loop&rsquo;s start
and <code>b.StopTimer</code> at its end are integrated into <code>testing.B.Loop</code>, eliminating the need
to manually manage the benchmark timer for setup and cleanup code.</p>
<p>Furthermore, the Go compiler now detects loops where the condition is just a call to
<code>testing.B.Loop</code> and prevents dead code elimination within the loop. In Go 1.24, this is
implemented by disallowing inlining into the body of such a loop, but we plan to
<a href="/issue/73137">improve</a> this in the future.</p>
<p>Another nice feature of <code>testing.B.Loop</code> is its one-shot ramp-up approach. With a <code>b.N</code>-style
benchmark, the testing package must call the benchmark function several times with different
values of <code>b.N</code>, ramping up until the measured time reached a threshold. In contrast, <code>b.Loop</code>
can simply run the benchmark loop until it reaches the time threshold, and only needs to call
the benchmark function once. Internally, <code>b.Loop</code> still uses a ramp-up process to amortize
measurement overhead, but this is hidden from the caller and can be more efficient.</p>
<p>Certain constraints of the <code>b.N</code>-style loop still apply to the <code>b.Loop</code>-style
loop. It remains the user&rsquo;s responsibility to manage the timer within the benchmark loop,
when necessary:
(<a href="https://eli.thegreenplace.net/2023/common-pitfalls-in-go-benchmarking/" rel="noreferrer" target="_blank">Example source</a>)</p>
<pre><code>func BenchmarkSortInts(b *testing.B) {
 ints := make([]int, N)
 for b.Loop() {
 b.StopTimer()
 fillRandomInts(ints)
 b.StartTimer()
 slices.Sort(ints)
 }
}
</code></pre>
<p>In this example, to benchmark the in-place sorting performance of <code>slices.Sort</code>, a
randomly initialized array is required for each iteration. The user must still
manually manage the timer in such cases.</p>
<p>Furthermore, there still needs to be exactly one such loop in the benchmark function body
(a <code>b.N</code>-style loop cannot coexist with a <code>b.Loop</code>-style loop), and every iteration of the
loop should do the same thing.</p>
<h2 id="when-to-use">When to use</h2>
<p>The <code>testing.B.Loop</code> method is now the preferred way to write benchmarks:</p>
<pre><code>func Benchmark(b *testing.B) {
 ... setup ...
 for b.Loop() {
 // optional timer control for in-loop setup/cleanup
 ... code to measure ...
 }
 ... cleanup ...
}
</code></pre>
<p><code>testing.B.Loop</code> offers faster, more accurate, and
more intuitive benchmarking.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>A huge thank you to everyone in the community who provided feedback on the proposal
issue and reported bugs as this feature was released! I&rsquo;m also grateful to Eli
Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements,
Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and
documentation improvements. Thank you all for your contributions!</p>
</div>

 </div>

 
 <div class="Article prevnext">
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 <p>
 
 
 <b>Next article: </b><a href="/blog/tob-crypto-audit">Go Cryptography Security Audit</a><br>
 
 
 
 
 <b>Previous article: </b><a href="/blog/coretypes">Goodbye core types - Hello Go as we know and love it!</a><br>
 
 
 <b><a href="/blog/all">Blog Index</a></b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 

 </div>
</div>

<script src="/js/play.js"></script>

</content></entry></feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//blog.golang.org/feed.atom

Home · About · News · Docs · Terms