<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>GPUers</title><link>https://en.gpuers.cn/</link><description>Recent content on GPUers</description><image><title>GPUers</title><url>https://en.gpuers.cn/img/og-cover-en.png</url><link>https://en.gpuers.cn/img/og-cover-en.png</link></image><generator>Hugo</generator><language>en-US</language><lastBuildDate>Wed, 18 Nov 2026 00:00:00 +0800</lastBuildDate><atom:link href="https://en.gpuers.cn/index.xml" rel="self" type="application/rss+xml"/><item><title>Linux GPU Driver Internals</title><link>https://en.gpuers.cn/book/linux-gpu-driver-internals/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/book/linux-gpu-driver-internals/</guid><description>&lt;p&gt;A GPU driver has two core jobs: managing the memory the GPU can access, and handing the GPU commands that use that memory. In Linux, the DRM subsystem provides the shared machinery for both. GEM and TTM manage buffer objects (BOs) and their backing storage. dma-buf and dma-fence handle sharing and synchronization. The GPU scheduler and drm_gpuvm handle job scheduling and GPU virtual address spaces.&lt;/p&gt;
&lt;p&gt;This book explains that machinery as one system. It follows a buffer object through its life, from allocation and placement to sharing, synchronization, scheduling, mapping into GPU address spaces, command submission and shared virtual memory. Each mechanism is presented in the same order: the problem it solves and the hardware behind it, the design trade-offs, the kernel implementation, and how the amdgpu, Xe and panthor drivers use it. The code baseline is Linux 7.1.&lt;/p&gt;</description><content:encoded>&lt;p>A GPU driver has two core jobs: managing the memory the GPU can access, and handing the GPU commands that use that memory. In Linux, the DRM subsystem provides the shared machinery for both. GEM and TTM manage buffer objects (BOs) and their backing storage. dma-buf and dma-fence handle sharing and synchronization. The GPU scheduler and drm_gpuvm handle job scheduling and GPU virtual address spaces.&lt;/p>
&lt;p>This book explains that machinery as one system. It follows a buffer object through its life, from allocation and placement to sharing, synchronization, scheduling, mapping into GPU address spaces, command submission and shared virtual memory. Each mechanism is presented in the same order: the problem it solves and the hardware behind it, the design trade-offs, the kernel implementation, and how the amdgpu, Xe and panthor drivers use it. The code baseline is Linux 7.1.&lt;/p>
</content:encoded></item><item><title>Atomics and Memory Barriers: The Foundation of Every Lock</title><link>https://en.gpuers.cn/columns/kernel-locks/atomics-and-barriers/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/atomics-and-barriers/</guid><description>In concurrent programming, &amp;#34;lock&amp;#34; is the word we use most, but a lock is not a primitive that exists on its own.</description><content:encoded><![CDATA[<p>In concurrent programming, &ldquo;lock&rdquo; is the word we use most, but a lock is not a primitive that exists on its own. Take any kernel lock apart and you find the same two foundations underneath: <strong>atomic operations</strong>, which make a read-modify-write impossible to interrupt, and <strong>memory barriers</strong>, which constrain the order in which other CPUs observe memory accesses. This chapter covers no particular lock. It focuses on these two foundations, because every lock that follows, from the <code>spinlock</code> in Chapter 2 to <code>mutex</code>, <code>rwsem</code> and <code>ww_mutex</code>, is a different combination of <strong>atomic operations + memory barriers + a waiting strategy</strong>. Without a solid foundation, there is no way to reason about the correctness of the locks built on top of it.</p>
<h2 id="11-what-a-lock-protects-against-three-faces-of-a-data-race">1.1 What a Lock Protects Against: Three Faces of a Data Race</h2>
<p>When two CPUs run <code>counter++</code> concurrently, the result can be smaller than expected. <code>counter++</code> is not one instruction but three steps:</p>
<pre tabindex="0"><code>load  r, [counter];  read
add   r, 1        ;  modify
store [counter], r;  write
</code></pre><p>If both CPUs read the same old value at almost the same time, each adds one and writes the result back, and two increments leave only one behind. This is a <strong>lost update</strong>.</p>
<pre class="mermaid">
sequenceDiagram
    participant A as CPU A
    participant M as counter (initially 0)
    participant B as CPU B
    A-&gt;&gt;M: load reads 0
    B-&gt;&gt;M: load reads 0
    A-&gt;&gt;A: 0 + 1 = 1
    B-&gt;&gt;B: 0 + 1 = 1
    A-&gt;&gt;M: store writes 1
    B-&gt;&gt;M: store writes 1
    Note over M: Two increments leave 1: one update is lost
</pre>
<p>It is the most intuitive kind of data race, but not the only one. Concurrency bugs come in three forms:</p>
<ul>
<li><strong>Lost update</strong>: several read-modify-write sequences interleave and some updates are overwritten. The root cause is that the RMW is not atomic.</li>
<li><strong>Torn access</strong>: a single read or write is split into several by the compiler or the hardware. On some paths a 64-bit variable may be written as two 32-bit halves, and another CPU can read a half-written value.</li>
<li><strong>Reordering</strong>: even when every access is atomic on its own, different CPUs can observe the <strong>order</strong> of operations differently. Reordering happens at two levels: the compiler swaps independent reads and writes at compile time, and the CPU changes the global visibility order at run time through the store buffer and out-of-order execution.</li>
</ul>
<p>The first two call for <strong>atomic operations</strong>; the third calls for <strong>memory barriers</strong>. These two tools run through the whole chapter.</p>
<h2 id="12-getting-past-the-compiler-first-read_oncewrite_once-and-the-compiler-barrier">1.2 Getting Past the Compiler First: <code>READ_ONCE</code>/<code>WRITE_ONCE</code> and the Compiler Barrier</h2>
<p>The three races in the previous section all originate in multicore hardware. But before the trouble reaches the hardware, there is an earlier player: the compiler. To make code faster, it makes bold assumptions about every memory access we write. Those assumptions are perfectly valid in single-threaded code and cause bugs once several threads are involved. So the first tool in this chapter is neither an atomic instruction nor a hardware barrier, but the lightest and most often overlooked line of defense.</p>
<p>Start with a trap almost everyone has fallen into. One thread spins, waiting for another thread to set a flag:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="n">flag</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">	<span class="p">;</span>          <span class="cm">/* wait until flag becomes non-zero */</span>
</span></span></code></pre></div><p>From the compiler&rsquo;s point of view, nothing in this loop modifies <code>flag</code>, so it loads <code>flag</code> from memory into a register once and from then on checks only the register, never memory. Even after another thread has set <code>flag</code>, the loop never sees it and spins forever. The compiler did nothing wrong; we never told it that the variable can change behind its back.</p>
<p>The C standard treats two threads reading and writing the same variable without synchronization as <strong>undefined behavior</strong> (a data race is undefined behavior). Once that happens, the compiler is entitled to any optimization: keep the value in a register, merge several reads into one, or even delete the loop altogether. To prevent this, we need a way to give the compiler a hard rule: this access must go to memory exactly once, no more and no less. The kernel&rsquo;s answer is <code>READ_ONCE</code>/<code>WRITE_ONCE</code>, defined in <code>include/asm-generic/rwonce.h</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define __READ_ONCE(x)	(*(const volatile __unqual_scalar_typeof(x) *)&amp;(x))
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cp">#define READ_ONCE(x)							\
</span></span></span><span class="line"><span class="cl"><span class="cp">({									\
</span></span></span><span class="line"><span class="cl"><span class="cp">	compiletime_assert_rwonce_type(x);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">	__READ_ONCE(x);							\
</span></span></span><span class="line"><span class="cl"><span class="cp">})
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cp">#define __WRITE_ONCE(x, val)						\
</span></span></span><span class="line"><span class="cl"><span class="cp">do {									\
</span></span></span><span class="line"><span class="cl"><span class="cp">	*(volatile typeof(x) *)&amp;(x) = (val);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">} while (0)
</span></span></span></code></pre></div><p>Strip away the macro wrappers and the core is a single <code>volatile</code> cast. <code>volatile</code> tells the compiler that this location can be rewritten at any moment by something it cannot see, so it must not cache, merge or omit the access, and must go to memory every time. Changing the loop above to <code>while (!READ_ONCE(flag))</code> makes the compiler reload <code>flag</code> on every iteration, and the loop exits as soon as the flag is set. It also prevents the <strong>tearing</strong> described in 1.1: <code>volatile</code> makes the access one complete read or write within the machine word size, never split in two. <code>compiletime_assert_rwonce_type</code> stands guard at compile time: use the macro on a type wider than the machine word, which cannot be accessed atomically in one go, and the build fails, catching the problem before it ships.</p>
<p><code>READ_ONCE</code>/<code>WRITE_ONCE</code> cover a single access to a single variable, but not the order between statements. When the constraint is &ldquo;finish writing A before writing B, and do not swap them&rdquo;, we need a coarser tool, the <strong>compiler barrier</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define barrier() asm volatile(&#34;&#34; ::: &#34;memory&#34;)
</span></span></span></code></pre></div><p>It is an empty piece of inline assembly that executes nothing; the trick is the trailing <code>&quot;memory&quot;</code> clobber. It tells the compiler that at this point anything in memory may have changed, so values loaded before this line must not be reused after it. As a result, the compiler will not move memory accesses across the barrier. Keep its limit in mind, though: <strong>it stops the compiler, not the CPU</strong>. The CPU can still reorder at run time; that is a separate problem, handled by the hardware barriers in 1.5. And precisely because x86 is strongly ordered, its <code>smp_rmb</code>/<code>smp_wmb</code> reduce to this <code>barrier()</code>.</p>
<p>One point needs to be nailed down: both tools in this section act only at <strong>compile time</strong>, on the way from source to assembly, and generate no barrier instructions.</p>
<ul>
<li><code>READ_ONCE</code>/<code>WRITE_ONCE</code> use a <code>volatile</code> cast to constrain how the compiler emits memory accesses: it must actually emit a load or store, and may not keep the value in a register, merge accesses or delete them. They produce no extra CPU instructions (such as <code>mfence</code> or a <code>lock</code> prefix); the result is still an ordinary <code>mov</code>.</li>
<li><code>barrier()</code> is the empty assembly <code>asm volatile(&quot;&quot; ::: &quot;memory&quot;)</code>. It generates no machine instructions; it only forbids the compiler from reordering the instructions it emits across that point.</li>
</ul>
<p>The mechanisms that actually constrain run-time CPU reordering, and that emit a <code>LOCK</code> prefix or <code>mfence</code>/<code>lfence</code>/<code>sfence</code>, come later: atomic instructions in 1.3 and memory barriers in 1.5.</p>
<p>In one sentence: <code>READ_ONCE</code>/<code>WRITE_ONCE</code> ensure that <strong>this particular access to a variable really happens and is not torn</strong>, and <code>barrier()</code> ensures that <strong>the compiler does not move code across this point</strong>. Together they deal with problems the compiler causes; the visibility order between CPUs has to wait for the hardware barriers in 1.5.</p>
<p><strong>Summary of this section</strong>: <code>READ_ONCE</code>/<code>WRITE_ONCE</code> and <code>barrier()</code> are the two lowest bricks of the entire atomic and locking machinery, and every layer above is built on them. This and later chapters will not explain them again, but will use them constantly. <code>atomic_read</code>/<code>atomic_set</code> (1.3) are thin wrappers around <code>READ_ONCE</code>/<code>WRITE_ONCE</code>. The acquire/release barriers <code>smp_load_acquire</code>/<code>smp_store_release</code> (1.6) are <code>READ_ONCE</code>/<code>WRITE_ONCE</code> plus a <code>barrier()</code>. And when <code>spinlock</code>, <code>qspinlock</code> and <code>rwsem</code> spin on the lock state, they read it with accesses such as <code>READ_ONCE(lock-&gt;val)</code>, relying on exactly the guarantee from this section: every iteration really rereads memory and is never optimized into an endless loop. Keep this in mind and you will recognize these two foundations immediately in the source of any lock.</p>
<h2 id="13-atomic-types-and-atomic-operations">1.3 Atomic Types and Atomic Operations</h2>
<p><code>atomic_t</code> itself is surprisingly plain: an aligned <code>int</code> (<code>include/linux/types.h</code>):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="kt">int</span> <span class="nf">__aligned</span><span class="p">(</span><span class="k">sizeof</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="n">counter</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span> <span class="kt">atomic_t</span><span class="p">;</span>
</span></span></code></pre></div><p>The struct wrapper exists for <strong>type safety</strong>: it forces every access through the <code>atomic_*</code> API and rules out a bare <code>v.counter++</code>. Type safety is only the surface, though; the atomicity itself comes from the <strong>hardware</strong>. This section is the first place in the chapter where we cross the line between software and hardware. <code>READ_ONCE</code> and <code>barrier()</code> in 1.2 only constrain the compiler and generate no extra instructions. Starting with the atomic RMW in this section, we reach the level of CPU instructions: on x86, the <code>LOCK</code> prefix locks the cache line (modern implementations rely on the cache coherence protocol rather than locking the bus), and the hardware guarantees that the read-modify-write cannot be interrupted. See <code>arch/x86/include/asm/atomic.h</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">static</span> <span class="n">__always_inline</span> <span class="kt">int</span> <span class="nf">arch_atomic_read</span><span class="p">(</span><span class="k">const</span> <span class="kt">atomic_t</span> <span class="o">*</span><span class="n">v</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="k">return</span> <span class="nf">__READ_ONCE</span><span class="p">((</span><span class="n">v</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">counter</span><span class="p">);</span>      <span class="cm">/* an aligned read is atomic by itself; no LOCK needed */</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">static</span> <span class="n">__always_inline</span> <span class="kt">void</span> <span class="nf">arch_atomic_add</span><span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="p">,</span> <span class="kt">atomic_t</span> <span class="o">*</span><span class="n">v</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="n">asm_inline</span> <span class="k">volatile</span><span class="p">(</span><span class="n">LOCK_PREFIX</span> <span class="s">&#34;addl %1, %0&#34;</span>
</span></span><span class="line"><span class="cl">		     <span class="o">:</span> <span class="s">&#34;+m&#34;</span> <span class="p">(</span><span class="n">v</span><span class="o">-&gt;</span><span class="n">counter</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">		     <span class="o">:</span> <span class="s">&#34;ir&#34;</span> <span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">:</span> <span class="s">&#34;memory&#34;</span><span class="p">);</span>   <span class="cm">/* RMW: LOCK required */</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">static</span> <span class="n">__always_inline</span> <span class="kt">int</span> <span class="nf">arch_atomic_add_return</span><span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="p">,</span> <span class="kt">atomic_t</span> <span class="o">*</span><span class="n">v</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="nf">xadd</span><span class="p">(</span><span class="o">&amp;</span><span class="n">v</span><span class="o">-&gt;</span><span class="n">counter</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>       <span class="cm">/* with a return value: xadd atomically fetches the old value */</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Note that <code>read</code> and <code>set</code> do not need <code>LOCK</code>: on x86, aligned word-sized reads and writes are inherently atomic, so <code>READ_ONCE</code>/<code>WRITE_ONCE</code> (the compile-time control from 1.2) are enough to keep the compiler from interfering. <code>LOCK</code> is needed only for compound operations such as read-modify-write.</p>
<p>Look closely at <code>LOCK_PREFIX</code> in this code, because it marks the boundary between software and hardware. It is defined in <code>arch/x86/include/asm/alternative.h</code>. In an SMP kernel it expands to a real <code>lock</code> prefix (opcode <code>0xF0</code>) placed in front of <code>addl</code> or <code>cmpxchgl</code>, and the CPU guarantees that the instruction is atomic. In a uniprocessor (UP) kernel it expands to an empty string and no prefix is emitted at all. In other words, <code>LOCK_PREFIX</code> is a genuine hardware instruction prefix, in sharp contrast to <code>barrier()</code> and <code>volatile</code> from 1.2, which generate no instructions. <strong>At this level, we are standing on the hardware.</strong></p>
<p>Atomic operations therefore fall into three groups:</p>
<ul>
<li><strong>Plain reads and writes</strong>: <code>atomic_read</code> / <code>atomic_set</code>.</li>
<li><strong>RMW without a return value</strong>: <code>atomic_add</code> / <code>sub</code> / <code>inc</code> / <code>dec</code>, which modify the value and do not report the result.</li>
<li><strong>RMW with a return value or a condition</strong>: <code>atomic_add_return</code> (using <code>xadd</code>) and <code>atomic_dec_and_test</code> / <code>inc_and_test</code> (using <code>GEN_UNARY_RMWcc</code>, which reads the CPU flags directly to tell whether the result is zero). This third group does most of the work in reference counting and lock-free algorithms.</li>
</ul>
<h2 id="14-cmpxchg-the-common-foundation-of-lock-free-code-and-lock-fast-paths">1.4 <code>cmpxchg</code>: The Common Foundation of Lock-Free Code and Lock Fast Paths</h2>
<p>If the atomic operations above are parts, <code>cmpxchg</code> (compare-and-swap, CAS) is the universal connector that assembles them into any lock-free algorithm. Its semantics: if the current value in memory equals the expected old value, write the new value; in either case, return the original value. The x86 implementation is in <code>arch/x86/include/asm/cmpxchg.h</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define __raw_cmpxchg(ptr, old, new, size, lock)			\
</span></span></span><span class="line"><span class="cl"><span class="cp">({									\
</span></span></span><span class="line"><span class="cl"><span class="cp">	...								\
</span></span></span><span class="line"><span class="cl"><span class="cp">	case __X86_CASE_L:						\
</span></span></span><span class="line"><span class="cl"><span class="cp">	{								\
</span></span></span><span class="line"><span class="cl"><span class="cp">		volatile u32 *__ptr = (volatile u32 *)(ptr);		\
</span></span></span><span class="line"><span class="cl"><span class="cp">		asm_inline volatile(lock &#34;cmpxchgl %2, %1&#34;		\
</span></span></span><span class="line"><span class="cl"><span class="cp">			     : &#34;=a&#34; (__ret), &#34;+m&#34; (*__ptr)		\
</span></span></span><span class="line"><span class="cl"><span class="cp">			     : &#34;r&#34; (__new), &#34;0&#34; (__old)			\
</span></span></span><span class="line"><span class="cl"><span class="cp">			     : &#34;memory&#34;);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">		break;							\
</span></span></span><span class="line"><span class="cl"><span class="cp">	}								\
</span></span></span><span class="line"><span class="cl"><span class="cp">	...								\
</span></span></span><span class="line"><span class="cl"><span class="cp">})
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cp">#define arch_cmpxchg(ptr, old, new)					\
</span></span></span><span class="line"><span class="cl"><span class="cp">	__cmpxchg(ptr, old, new, sizeof(*(ptr)))   </span><span class="cm">/* with LOCK_PREFIX */</span><span class="cp">
</span></span></span></code></pre></div><p>The <code>lock</code> here is nothing new; it is the <code>LOCK_PREFIX</code> from 1.3. <code>__raw_cmpxchg</code> turns it into a macro parameter so that one assembly template serves two variants. <code>arch_cmpxchg</code> goes through <code>__cmpxchg</code>, which passes <code>LOCK_PREFIX</code> (a real <code>lock</code> prefix under SMP) and is atomic across CPUs. <code>arch_cmpxchg_local</code> goes through <code>__cmpxchg_local</code>, which passes an empty string; it only protects against interrupts on the local CPU and drops the prefix to run faster. So the atomicity of <code>cmpxchg</code> still rests on the hardware line drawn in 1.3.</p>
<p>With CAS, any update of the form &ldquo;read the old value, compute a new value, swap it in atomically&rdquo; can be written as a retry loop:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="n">old</span> <span class="o">=</span> <span class="nf">READ_ONCE</span><span class="p">(</span><span class="n">v</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="k">do</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="n">new</span> <span class="o">=</span> <span class="nf">f</span><span class="p">(</span><span class="n">old</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span> <span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="nf">try_cmpxchg</span><span class="p">(</span><span class="o">&amp;</span><span class="n">v</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">));</span>   <span class="cm">/* on failure, old is refreshed with the latest value; try again */</span>
</span></span></code></pre></div><p>The importance of this pattern is hard to overstate: <strong>the fast path of every lock that follows is essentially one CAS</strong>. <code>mutex</code> uses CAS to change the owner from NULL to current, <code>qspinlock</code> uses it to grab the pending bit, and <code>rwsem</code> uses it to update the count. Understand this loop and you understand the skeleton of every lock fast path. (CAS has a classic pitfall, the ABA problem: a value changes from A to B and back to A, and CAS wrongly concludes that it never changed. It is discussed in the chapters on RCU and lock-free data structures.)</p>
<h2 id="15-memory-barriers-four-basic-barriers-and-their-cpu-implementation">1.5 Memory Barriers: Four Basic Barriers and Their CPU Implementation</h2>
<p>Atomic operations make a single operation indivisible, but they say nothing about the visibility order between operations. Consider the classic message-passing example:</p>
<pre tabindex="0"><code>CPU 0:  data = 42;          CPU 1:  while (!flag) ;
        flag = 1;                   r = data;
</code></pre><p>Even though the writes to <code>data</code> and <code>flag</code> are each atomic, CPU 1 may see <code>flag == 1</code> and still read the old value of <code>data</code>: the two writes may drain from CPU 0&rsquo;s store buffer out of order, or CPU 1 may execute its two reads out of order. Fixing this requires barriers. The kernel&rsquo;s four basic barriers and their x86 implementation are in <code>arch/x86/include/asm/barrier.h</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define __mb()	asm volatile(&#34;mfence&#34;:::&#34;memory&#34;)   </span><span class="cm">/* full barrier: neither reads nor writes cross it */</span><span class="cp">
</span></span></span><span class="line"><span class="cl"><span class="cp">#define __rmb()	asm volatile(&#34;lfence&#34;:::&#34;memory&#34;)   </span><span class="cm">/* read barrier */</span><span class="cp">
</span></span></span><span class="line"><span class="cl"><span class="cp">#define __wmb()	asm volatile(&#34;sfence&#34; ::: &#34;memory&#34;) </span><span class="cm">/* write barrier */</span><span class="cp">
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cp">#define __smp_mb()	asm volatile(&#34;lock addl $0,-4(%%&#34; _ASM_SP &#34;)&#34; ::: &#34;memory&#34;, &#34;cc&#34;)
</span></span></span><span class="line"><span class="cl"><span class="cp">#define __smp_rmb()	dma_rmb()
</span></span></span><span class="line"><span class="cl"><span class="cp">#define __smp_wmb()	barrier()
</span></span></span></code></pre></div><p>A few key points:</p>
<ul>
<li><code>mb</code>/<code>rmb</code>/<code>wmb</code> are for interaction with devices (MMIO and DMA) and map to <code>mfence</code>/<code>lfence</code>/<code>sfence</code>.</li>
<li><code>smp_mb</code>/<code>smp_rmb</code>/<code>smp_wmb</code> order accesses between CPUs. <strong>x86 is strongly ordered</strong>: the CPU never reorders ordinary reads with other reads, or writes with other writes, so <code>smp_rmb</code>/<code>smp_wmb</code> reduce to compiler barriers. The only reordering x86 allows is a later read passing an earlier write, so <code>smp_mb</code> must be a real barrier. It uses <code>lock addl $0, -4(%rsp)</code> (a locked add of zero to the top of the stack) as a full barrier, which is faster than <code>mfence</code>.</li>
<li>On weakly ordered architectures (ARM64, PowerPC and others), <code>smp_rmb</code>/<code>smp_wmb</code> are real barrier instructions. <strong>This is why portable code must spell out its barriers instead of relying on x86&rsquo;s strong ordering.</strong></li>
</ul>
<p>Following the software/hardware line drawn in 1.3, the barriers can also be sorted by whether they generate hardware instructions:</p>
<table>
	<thead>
			<tr>
					<th>Barrier</th>
					<th>Emitted on x86</th>
					<th>Hardware instruction?</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td><code>mb</code> / <code>rmb</code> / <code>wmb</code></td>
					<td><code>mfence</code> / <code>lfence</code> / <code>sfence</code></td>
					<td>Yes</td>
			</tr>
			<tr>
					<td><code>smp_mb</code></td>
					<td><code>lock addl $0, -4(%rsp)</code></td>
					<td>Yes (through the <code>lock</code> prefix)</td>
			</tr>
			<tr>
					<td><code>smp_rmb</code> / <code>smp_wmb</code></td>
					<td>Reduce to <code>barrier()</code></td>
					<td>No, compile time only</td>
			</tr>
	</tbody>
</table>
<p>The key point: <strong>whether the same barrier becomes a hardware instruction depends on how strongly ordered the target architecture is</strong>. <code>smp_wmb()</code> is an empty <code>barrier()</code> on x86 (the compile-time control from 1.2) but becomes a real <code>dmb</code> instruction on ARM64. This also explains the previous point: leaving out <code>smp_wmb</code> on x86 often happens to work, and breaks as soon as the code is ported to a weakly ordered architecture.</p>
<h2 id="16-acquire-and-release-the-ordering-model-locks-actually-rely-on">1.6 Acquire and Release: The Ordering Model Locks Actually Rely On</h2>
<p>A full barrier gives strong guarantees but is expensive. What a lock actually needs is a cheaper, <strong>one-way</strong> ordering: accesses inside the critical section must not leak outside it. That is the acquire/release model:</p>
<ul>
<li><strong>acquire</strong> (used when taking a lock): no access after it may be reordered before it.</li>
<li><strong>release</strong> (used when releasing a lock): no access before it may be reordered after it.</li>
</ul>
<p>Placed on either side of the critical section, the pair keeps its reads and writes inside. The x86 implementation is in <code>arch/x86/include/asm/barrier.h</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define __smp_store_release(p, v)					\
</span></span></span><span class="line"><span class="cl"><span class="cp">do {									\
</span></span></span><span class="line"><span class="cl"><span class="cp">	compiletime_assert_atomic_type(*p);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">	barrier();							\
</span></span></span><span class="line"><span class="cl"><span class="cp">	WRITE_ONCE(*p, v);						\
</span></span></span><span class="line"><span class="cl"><span class="cp">} while (0)
</span></span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="cp">#define __smp_load_acquire(p)						\
</span></span></span><span class="line"><span class="cl"><span class="cp">({									\
</span></span></span><span class="line"><span class="cl"><span class="cp">	typeof(*p) ___p1 = READ_ONCE(*p);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">	compiletime_assert_atomic_type(*p);				\
</span></span></span><span class="line"><span class="cl"><span class="cp">	barrier();							\
</span></span></span><span class="line"><span class="cl"><span class="cp">	___p1;								\
</span></span></span><span class="line"><span class="cl"><span class="cp">})
</span></span></span></code></pre></div><p>On strongly ordered x86, acquire and release <strong>need no real barrier instructions</strong>: a compiler barrier <code>barrier()</code> plus <code>READ_ONCE</code>/<code>WRITE_ONCE</code> is enough, because the hardware already guarantees every ordering except a read passing an earlier write. This is why unlocking can be so cheap on x86: a single ordinary write with release semantics. But this is an x86 bonus. On weakly ordered architectures such as ARM64, acquire and release use dedicated hardware instructions (such as <code>ldar</code>/<code>stlr</code>), and unlocking is no longer an ordinary write, just as <code>smp_wmb</code> in the table in 1.5 reduces to nothing on x86 but becomes a real instruction on weakly ordered machines. For the same reason, the barriers around atomic operations are empty on x86:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="cp">#define __smp_mb__before_atomic()	do { } while (0)
</span></span></span><span class="line"><span class="cl"><span class="cp">#define __smp_mb__after_atomic()	do { } while (0)
</span></span></span></code></pre></div><p>An atomic operation with a <code>LOCK</code> prefix is itself a full barrier, so nothing needs to be added. Thinking of lock as acquire and unlock as release is the ordering model that underlies every lock chapter that follows.</p>
<h2 id="17-the-three-layers-of-the-kernel-atomic-api">1.7 The Three Layers of the Kernel Atomic API</h2>
<p>Reading the source, you will find several names for <code>atomic_add</code>: <code>arch_atomic_add</code>, <code>raw_atomic_add</code> and <code>atomic_add</code>. This is not duplication but a deliberate three-layer structure:</p>
<ol>
<li><strong><code>arch_atomic_*</code></strong>: the architecture-specific implementation, such as the x86 <code>LOCK_PREFIX &quot;addl&quot;</code> above. An architecture only has to implement a minimal set.</li>
<li><strong>The fallback layer</strong> (<code>include/linux/atomic/atomic-arch-fallback.h</code>): scripts fill in every variant from the minimal set (for example, if an architecture does not implement <code>add_return</code>, it is synthesized from a <code>cmpxchg</code> loop).</li>
<li><strong>The instrumented <code>atomic_*</code> layer</strong> (<code>include/linux/atomic/atomic-instrumented.h</code>): the public API, wrapped with KASAN/KCSAN instrumentation for detecting concurrency bugs.</li>
</ol>
<p>Almost all of these three layers are generated by the scripts under <code>scripts/atomic/</code>. The point of the layering: whoever ports the kernel to a new architecture writes only the minimal assembly, and the generic variants and debugging instrumentation come from the common layers, which saves work and keeps behavior consistent. Everyday driver code uses only the top-level <code>atomic_*</code> and never needs to care about the two layers below.</p>
<h2 id="18-assembling-the-foundations-into-a-lock">1.8 Assembling the Foundations into a Lock</h2>
<p>Now let us use the parts from this chapter to write a minimal spinlock by hand, and check the formula &ldquo;lock = acquire with CAS + release + waiting strategy&rdquo;:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-c" data-lang="c"><span class="line"><span class="cl"><span class="k">struct</span> <span class="n">minimal_lock</span> <span class="p">{</span> <span class="kt">atomic_t</span> <span class="n">locked</span><span class="p">;</span> <span class="p">};</span>   <span class="cm">/* 0 = free, 1 = held */</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">static</span> <span class="kt">void</span> <span class="nf">minimal_lock</span><span class="p">(</span><span class="k">struct</span> <span class="n">minimal_lock</span> <span class="o">*</span><span class="n">l</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="kt">int</span> <span class="n">expected</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">	<span class="k">do</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">		<span class="n">expected</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">		<span class="cm">/* CAS to take the lock: swap 0 for 1; success means we hold it (acquire semantics) */</span>
</span></span><span class="line"><span class="cl">	<span class="p">}</span> <span class="k">while</span> <span class="p">(</span><span class="o">!</span><span class="nf">atomic_try_cmpxchg_acquire</span><span class="p">(</span><span class="o">&amp;</span><span class="n">l</span><span class="o">-&gt;</span><span class="n">locked</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">expected</span><span class="p">,</span> <span class="mi">1</span><span class="p">));</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">static</span> <span class="kt">void</span> <span class="nf">minimal_unlock</span><span class="p">(</span><span class="k">struct</span> <span class="n">minimal_lock</span> <span class="o">*</span><span class="n">l</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl">	<span class="cm">/* write 0 with release semantics so every write in the critical section is visible to others */</span>
</span></span><span class="line"><span class="cl">	<span class="nf">atomic_set_release</span><span class="p">(</span><span class="o">&amp;</span><span class="n">l</span><span class="o">-&gt;</span><span class="n">locked</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Item by item: <code>atomic_try_cmpxchg_acquire</code> is the CAS from 1.4 plus the acquire from 1.6; <code>atomic_set_release</code> is the release from 1.6; the empty spin in the <code>do-while</code> loop is the waiting strategy. This lock works, but its waiting strategy is too primitive: every contender spins writing the same <code>locked</code> variable, causing heavy cache-line bouncing. In Chapter 2, <code>qspinlock</code> replaces this bare spinning with an MCS queue in which each waiter spins on its own local node. That is the line between a lock that works and a lock that scales.</p>
<h2 id="19-misuse-and-debugging">1.9 Misuse and Debugging</h2>
<p>The most common pitfalls at this level:</p>
<ul>
<li><strong>Treating a bare <code>v++</code> as atomic</strong>: updates are lost on multicore systems. Use <code>atomic_inc</code> or a <code>cmpxchg</code> loop.</li>
<li><strong>Leaving out <code>READ_ONCE</code>/<code>WRITE_ONCE</code></strong>: polling a shared flag with <code>while (!flag)</code> lets the compiler keep <code>flag</code> in a register and spin forever. Write <code>while (!READ_ONCE(flag))</code>.</li>
<li><strong>Unpaired barriers</strong>: a writer using <code>smp_store_release</code> needs a reader using <code>smp_load_acquire</code>. A barrier on one side only gives no ordering guarantee; both ends of a message-passing pattern need one.</li>
<li><strong>Trusting x86&rsquo;s strong ordering</strong>: leaving out <code>smp_rmb</code>/<code>smp_wmb</code> on x86 often happens to work and breaks immediately on ARM64. Reason about portable code with a weakly ordered model.</li>
</ul>
<p>Debugging tools: enable <code>CONFIG_KCSAN</code> for run-time data race detection; it catches most concurrent accesses that are missing <code>READ_ONCE</code> or atomic operations. For intricate ordering arguments, <code>Documentation/memory-barriers.txt</code> is the authoritative reference inside the kernel, and <code>tools/memory-model/</code> contains a formal model (LKMM) you can check your reasoning against. Deadlock detection at the lock-ordering level (lockdep) is covered in the chapter on tools.</p>
<hr>
<h2 id="summary">Summary</h2>
<p>Every lock rests on two foundations. <strong>Atomic operations</strong> use the <code>LOCK</code> prefix and <code>cmpxchg</code> to make read-modify-write indivisible, preventing lost updates and tearing. <strong>Memory barriers</strong> constrain the visibility order between CPUs, and the one-way acquire/release pair is exactly the minimal model that keeps a critical section from leaking: lock is acquire, unlock is release. <code>READ_ONCE</code>/<code>WRITE_ONCE</code> keep a single access atomic and safe from optimization, and <code>cmpxchg</code> is the common connector of every lock fast path and lock-free algorithm. x86&rsquo;s strong ordering reduces many barriers to nothing, but the right mental model is the weakly ordered one. With the formula &ldquo;lock = atomics + barriers + waiting strategy&rdquo; in hand, the next chapter takes apart the first real lock, from the simplest <code>spinlock</code> to <code>qspinlock</code>, which uses an MCS queue to eliminate contention.</p>
]]></content:encoded></item><item><title>lockdep: How Deadlock Detection Works</title><link>https://en.gpuers.cn/columns/kernel-locks/lockdep/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/lockdep/</guid><description>Almost every one of the previous ten chapters has a &amp;#34;Misuse and Debugging&amp;#34; section, and the same name keeps coming up: lockdep.</description><content:encoded><![CDATA[<p>Almost every one of the previous ten chapters has a &ldquo;Misuse and Debugging&rdquo; section, and the same name keeps coming up: lockdep. It is the kernel&rsquo;s run-time deadlock detector, able to report a potential lock-ordering problem from a single ordinary run, even when the deadlock never actually happens. This chapter takes its engine apart to see how it manages to &ldquo;know about a deadlock before it happens&rdquo;.</p>
<p>lockdep is a sizable body of code. This chapter focuses on its core data structures and cycle-detection algorithm, anchored on <code>kernel/locking/lockdep.c</code>. It is enabled by <code>CONFIG_PROVE_LOCKING</code>, is purely a debugging facility, and is usually disabled in production kernels.</p>
<blockquote>
<p>Note: lockdep is independent of any particular lock implementation. It hooks into the lock and unlock paths of every lock to record and validate on the side, so this chapter does not distinguish between lock types.</p>
</blockquote>
]]></content:encoded></item><item><title>locktorture: Stress-Testing Lock Implementations</title><link>https://en.gpuers.cn/columns/kernel-locks/locktorture/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/locktorture/</guid><description>The lockdep of Chapter 11 proves that locks are used correctly: the lock ordering is acyclic and acquisitions and releases are properly paired.</description><content:encoded><![CDATA[<p>The lockdep of Chapter 11 proves that <strong>locks are used correctly</strong>: the lock ordering is acyclic and acquisitions and releases are properly paired. But there is a more fundamental question: is the <strong>lock implementation itself</strong> correct? Does a newly written spinlock or mutex really guarantee mutual exclusion, never lose a wakeup, and never wait forever under arbitrary concurrency? Such guarantees cannot be established exhaustively by reading code; they can only be forced out with intense concurrency. locktorture is the stress-testing framework the kernel provides for exactly this, echoing the approach of Chapter 7, where <code>ww_mutex</code> is validated by injecting deadlocks with <code>test-ww_mutex</code>.</p>
<p>This chapter is anchored on <code>kernel/locking/locktorture.c</code>. It is a kernel module built on the kernel&rsquo;s generic torture-test framework (sharing the same infrastructure as rcutorture), and it starts running as soon as it is loaded.</p>
<blockquote>
<p>Note: locktorture is a test facility. Module parameters select the lock type under test and the number of threads, and it plays no part in production paths.</p>
</blockquote>
]]></content:encoded></item><item><title>mutex: Optimistic Spinning and Handoff</title><link>https://en.gpuers.cn/columns/kernel-locks/mutex/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/mutex/</guid><description>The locks of the previous four chapters, spinlock, qrwlock and osqlock, share one hard constraint: when a CPU fails to get the lock, it can only spin in place a…</description><content:encoded><![CDATA[<p>The locks of the previous four chapters, <code>spinlock</code>, <code>qrwlock</code> and <code>osq_lock</code>, share one hard constraint: when a CPU fails to get the lock, it can only spin in place and must never sleep. That limits them to very short critical sections. Yet the kernel is full of critical sections that are anything but short: they read and write files, wait for DMA, allocate large chunks of memory, or even take other locks. These operations easily take several milliseconds, and making other CPUs spin idly for that long is pure waste. The <code>mutex</code> (mutual exclusion lock) in this chapter exists for exactly this case. It is the kernel&rsquo;s <strong>most widely used sleeping lock</strong>: when it fails to get the lock, it does not burn cycles; it suspends the current task, gives the CPU away to do other work, and is woken up when the lock is released.</p>
<p>But a <code>mutex</code> does not &ldquo;go to sleep at the first sign of trouble&rdquo;. Sleeping is expensive (context switches, cache invalidation), so it first borrows the <code>osq_lock</code> from Chapter 4 for a round of <strong>optimistic spinning</strong>: if the owner is running on another CPU and will probably release the lock soon, spin a little while and wait for it; win the bet and a sleep is saved. The main thread of this chapter is how <code>mutex</code> makes a single <code>owner</code> field serve several purposes, how it trades off spinning against sleeping, and how it uses the <strong>handoff</strong> mechanism to keep waiters from being starved by an endless stream of &ldquo;lock stealers&rdquo;. The source lives mainly in <code>kernel/locking/mutex.c</code>, <code>kernel/locking/mutex.h</code> and <code>include/linux/mutex.h</code>.</p>
<blockquote>
<p>Note: the main line assumes x86-64 and a non-<code>PREEMPT_RT</code> configuration. Under <code>PREEMPT_RT</code>, <code>mutex</code> is implemented on top of <code>rtmutex</code> instead, which is outside the scope of this chapter.</p>
</blockquote>
]]></content:encoded></item><item><title>osq_lock: The Optimistic Spin Queue</title><link>https://en.gpuers.cn/columns/kernel-locks/osq-lock/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/osq-lock/</guid><description>The locks in the previous three chapters share one trait: they are user-facing locks.</description><content:encoded><![CDATA[<p>The locks in the previous three chapters share one trait: they are <strong>user-facing</strong> locks. You call <code>spin_lock</code>, I call <code>read_lock</code>, and whoever gets the lock enters the critical section. The <code>osq_lock</code> in this chapter (optimistic spin queue lock) is the exception. Ordinary code almost never calls it directly; it lives deep inside <code>mutex</code> (Chapter 5) and <code>rwsem</code> (Chapter 6), serving as the common foundation for their <strong>optimistic spinning</strong>. To understand why the sleeping locks of the next two chapters &ldquo;spin for a while first even though they could sleep&rdquo;, you first have to understand the queue lock sitting underneath them. Its source is a single file, <code>kernel/locking/osq_lock.c</code>, yet it distills two key ideas: the &ldquo;everyone spins locally&rdquo; of the MCS queue from Chapter 2&rsquo;s <code>qspinlock</code>, and a capability <code>qspinlock</code> does not have, introduced in this chapter: <strong>spinning that can be cancelled midway</strong>.</p>
<blockquote>
<p>Note: x86-64 is the main thread of discussion. <code>osq_lock</code> shares its origins with the MCS lock in Chapter 2, so it is worth reading alongside Section 2.7. Its direct users are <code>mutex</code>/<code>rwsem</code>; Section 4.8 points out where it plugs into them.</p>
</blockquote>
]]></content:encoded></item><item><title>RCU and SRCU: Lock-Free Reads Driven by Grace Periods</title><link>https://en.gpuers.cn/columns/kernel-locks/rcu-and-srcu/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/rcu-and-srcu/</guid><description>The seqcount of Chapter 8 achieves lock-free reads with a sequence number and retries, but its summary left a loose end: it only works for reading values.</description><content:encoded><![CDATA[<p>The <code>seqcount</code> of Chapter 8 achieves lock-free reads with a sequence number and retries, but its summary left a loose end: it only works for <strong>reading values</strong>. As soon as the read side has to follow a pointer to an object, RCU has to be layered on top, because retrying guarantees that the values read are consistent with each other but cannot guarantee that the object the pointer refers to was not freed during the read. This chapter fills that gap. RCU (Read-Copy-Update) is the lock-free read mechanism built specifically for <strong>object lifetime</strong>: it makes readers nearly free and moves the entire cost to the updater side.</p>
<p>The RCU implementation is huge, so this chapter is only an overview, anchored on the semantics and key interfaces; the grace-period machinery is covered in depth in section 12.3 of the mmu-notifier column. The sources for this chapter are <code>include/linux/rcupdate.h</code>, <code>kernel/rcu/tree.c</code>, and <code>kernel/rcu/srcutree.c</code>.</p>
<blockquote>
<p>Note: the main line is x86-64 with a non-<code>PREEMPT_RT</code> configuration. The read-side overhead of RCU differs across preemption configurations; the text below describes the classic (preemptible) semantics.</p>
</blockquote>
]]></content:encoded></item><item><title>rwlock and qrwlock: The Spinning Trade-offs of Readers and Writers</title><link>https://en.gpuers.cn/columns/kernel-locks/rwlock-and-qrwlock/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/rwlock-and-qrwlock/</guid><description>The spinlock from Chapter 2 treats every contender the same: whether you only want to take a quick look or need to make a change, you queue up for exclusive acc…</description><content:encoded><![CDATA[<p><a href="https://gpuers.blog.csdn.net/article/details/164263475">The <code>spinlock</code> from Chapter 2</a> treats every contender the same: whether you only want to take a quick look or need to make a change, you queue up for exclusive access. Yet one kind of workload is extremely common in practice: <strong>read-mostly</strong> data, read countless times and updated only occasionally. If reads do not conflict with one another, forcing them through a spinlock turns a multi-lane road into a single lane and wastes the parallelism of a multicore machine. The subject of this chapter, <code>qrwlock</code> (queued read/write lock), exists for exactly this case: it lets many readers enter the critical section at the same time and share the data, and demands exclusive access only when a writer shows up. Like <code>spinlock</code>, it never sleeps and can be used in interrupt context; the only difference is that it refines &ldquo;mutual exclusion&rdquo; into &ldquo;shared for readers, exclusive for writers&rdquo;. The thread running through this chapter is how the kernel encodes both the reader count and the writer state in <strong>a single 32-bit atomic word</strong>, and how it uses a single &ldquo;waiting flag&rdquo; to solve the problem every reader-writer lock is born with: <strong>writer starvation</strong>.</p>
<blockquote>
<p>Note: this chapter follows x86-64. In modern kernels, <code>rwlock_t</code> is implemented by <code>qrwlock</code> at the architecture level on all such platforms. The source lives in <code>kernel/locking/qrwlock.c</code>, <code>include/asm-generic/qrwlock.h</code> and <code>include/asm-generic/qrwlock_types.h</code>.</p>
</blockquote>
]]></content:encoded></item><item><title>rwsem: Count Encoding and Lock Stealing in the Reader-Writer Semaphore</title><link>https://en.gpuers.cn/columns/kernel-locks/rwsem/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/rwsem/</guid><description>The mutex of Chapter 5 is the kernel&amp;#39;s mainstream sleeping lock, but it has an inherent limitation: whether a task reads or writes, only one task at a time may…</description><content:encoded><![CDATA[<p><a href="https://gpuers.blog.csdn.net/article/details/164263925">The <code>mutex</code> of Chapter 5</a> is the kernel&rsquo;s mainstream sleeping lock, but it has an inherent limitation: whether a task reads or writes, only one task at a time may enter the critical section. Yet a great deal of real-world data is <strong>read-mostly</strong>: a process&rsquo;s address space via <code>mm-&gt;mmap_lock</code>, assorted file system metadata, configuration tables in drivers. These structures are read most of the time and modified only occasionally. Protect them with a <code>mutex</code> and even read-only tasks must line up and run one after another, throwing away the chance that &ldquo;reads could have run in parallel with reads&rdquo;. The <code>qrwlock</code> of Chapter 3 solves reader-writer parallelism for spinning, but it does not sleep and suits only very short critical sections. When the read and write critical sections are long and may sleep, it is time for this chapter&rsquo;s <code>rwsem</code> (reader-writer semaphore): it lets <strong>any number of readers hold the lock at once</strong>, while a writer holds it exclusively, and a task that fails to get the lock sleeps and yields the CPU, just as with a <code>mutex</code>.</p>
<p>First, a misconception that has tripped up almost everyone: the &ldquo;sem&rdquo; in <code>rwsem</code> stands for semaphore, which makes it easy to assume it is a reader-writer version of the counting <code>semaphore</code>. <strong>It is not.</strong> The modern <code>rwsem</code> shares no implementation lineage with <code>kernel/locking/semaphore.c</code>. Its real ancestry is this: <code>rwsem</code> is the reader-writer sibling of Chapter 5&rsquo;s <code>mutex</code>. It reuses the whole <code>mutex</code> design: one atomic field with bit encoding, an <code>owner</code> field paired with optimistic spinning, Chapter 4&rsquo;s <code>osq_lock</code> as the foundation, and handoff to prevent starvation.</p>
<blockquote>
<p>Note: The main line is x86-64, 64-bit, non-<code>PREEMPT_RT</code> configurations. Under <code>PREEMPT_RT</code>, <code>rwsem</code> switches to an <code>rt_mutex</code> foundation and readers no longer truly run in parallel; that is outside the scope of this chapter.</p>
</blockquote>
]]></content:encoded></item><item><title>semaphore: The Plainest Counting Semaphore</title><link>https://en.gpuers.cn/columns/kernel-locks/semaphore/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/semaphore/</guid><description>This chapter opens the extension part of the book.</description><content:encoded><![CDATA[<p>This chapter opens the extension part of the book. The eight core chapters followed the main line &ldquo;foundations → spinning → sleeping → lock-free and specialized locks&rdquo;. This one goes back to a historical side branch we deliberately set aside: <code>semaphore</code>. It is one of the oldest sleeping synchronization primitives in Linux, older and plainer than the <code>mutex</code> of Chapter 5. We treat it on its own for two reasons. First, to see exactly what <code>mutex</code> added on top of it. Second, to finally settle a misunderstanding that has run through the earlier chapters: <strong>the &ldquo;sem&rdquo; in the name <code>rwsem</code> has no implementation lineage with this chapter&rsquo;s <code>semaphore</code></strong>. <code>rwsem</code> belongs to the <code>mutex</code> family (owner pointer, optimistic spinning, handoff), while <code>semaphore</code> is a separate branch whose structure does not even have an owner.</p>
<p>The whole implementation of <code>semaphore</code> lives in a single source file, short enough to read in one sitting. The sources for this chapter are <code>kernel/locking/semaphore.c</code> and <code>include/linux/semaphore.h</code>.</p>
<blockquote>
<p>Note: the main line is x86-64 with a non-<code>PREEMPT_RT</code> configuration. <code>semaphore</code> does not change form under <code>PREEMPT_RT</code>; its behavior is the same in every configuration.</p>
</blockquote>
]]></content:encoded></item><item><title>seqlock and seqcount: Lock-Free Reads Driven by a Sequence Number</title><link>https://en.gpuers.cn/columns/kernel-locks/seqlock-and-seqcount/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/seqlock-and-seqcount/</guid><description>In the previous seven chapters, whether a lock spins or sleeps, is exclusive or reader-writer, single or multiple, readers and writers always coordinate through…</description><content:encoded><![CDATA[<p>In the previous seven chapters, whether a lock spins or sleeps, is exclusive or reader-writer, single or multiple, readers and writers always coordinate through some form of mutual exclusion: while a reader holds the lock the writer must wait, and vice versa. But there is a large class of data that is <strong>rarely written yet read in enormous volume</strong>, and whose readers only want to copy out a consistent snapshot. The system clock is the classic example: <code>gettimeofday</code> may be called a million times per second, while <code>xtime</code> is updated only a few hundred times per second. If every reader had to contend for a reader-writer lock, the readers alone, repeatedly fighting over the same cache line to update the lock count, would be enough to wreck multicore scalability, even though reads never actually conflict with each other.</p>
<p>The <code>seqcount</code>/<code>seqlock</code> of this chapter takes the opposite approach to the first seven chapters: <strong>a writer is never blocked by readers, and readers never modify any shared state at all</strong>. The entire mechanism boils down to a sequence number: the writer increments it each time it enters and leaves the critical section, and the reader looks at it once before reading and once after. If the two values are equal and even, the read was not interrupted by a writer and the snapshot is valid; otherwise the reader starts over. All a reader pays is two loads and a compare: no atomic writes, no cache line contention, almost zero read-side overhead. The price is that readers may <strong>spin through retries</strong>, so it only fits one particular scenario: &ldquo;very few writes, very many reads, and the data read consists of values that can be re-read as a whole.&rdquo;</p>
<p><code>seqcount</code> also pushes the formula from Chapter 1, &ldquo;lock = atomic operations + barriers + waiting strategy&rdquo;, to an extreme: it contains almost no atomic operations (the sequence increment is made safe by serializing writers), and <strong>everything hinges on memory barriers</strong>. The write-side <code>smp_wmb</code> pairs precisely with the read-side <code>smp_rmb</code>, guaranteeing that a reader either sees the complete old state from before the writer started, or notices that the sequence changed and retries; it never reads half-written, &ldquo;torn&rdquo; data. The source for this chapter lives in <code>include/linux/seqlock.h</code> and <code>include/linux/seqlock_types.h</code>.</p>
<blockquote>
<p>Note: The main line is x86-64 with a non-<code>PREEMPT_RT</code> configuration. Under <code>PREEMPT_RT</code>, the write side of a <code>seqcount_LOCKNAME_t</code> with an associated lock no longer disables preemption; instead, a reader that detects a writer in progress &ldquo;acquires and then releases&rdquo; the associated lock to avoid livelock. The mechanism is mentioned in Section 8.6 and not developed further.</p>
</blockquote>
]]></content:encoded></item><item><title>spinlock and qspinlock: From Test-and-Set to the MCS Queue</title><link>https://en.gpuers.cn/columns/kernel-locks/spinlock-and-qspinlock/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/spinlock-and-qspinlock/</guid><description>Chapter 1 laid the foundation: atomic operations make a read-modify-write indivisible, and acquire/release keeps the critical section from leaking out.</description><content:encoded><![CDATA[<p>Chapter 1 laid the foundation: atomic operations make a read-modify-write indivisible, and acquire/release keeps the critical section from leaking out. This chapter takes apart the first real lock: the spinlock. It is the most basic and most widely used mutual-exclusion primitive in the kernel. A CPU that cannot get the lock does not sleep; it spins in place, retrying again and again until the lock is released. Precisely because it never sleeps, it can be used in contexts where scheduling is impossible, such as interrupt handlers or code already holding other locks. But &ldquo;spinning in place&rdquo; is not simple on a multicore machine: a naive implementation makes every contender crowd onto the same cache line and drag each other down. The thread running through this chapter is how the kernel evolved from the most naive <code>test-and-set</code> all the way to the MCS queue of <code>qspinlock</code>, reducing &ldquo;every CPU fighting over one cache line&rdquo; to &ldquo;each CPU spinning on its own local node&rdquo;.</p>
<blockquote>
<p>Note: x86-64 is the main thread, and it uses <code>qspinlock</code> by default; <code>ticket_spinlock</code> is analyzed alongside it as a &ldquo;simpler, but still cache-contended&rdquo; point of comparison.</p>
</blockquote>
]]></content:encoded></item><item><title>ww_mutex: Wound/Wait Deadlock Avoidance</title><link>https://en.gpuers.cn/columns/kernel-locks/ww-mutex/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0800</pubDate><guid>https://en.gpuers.cn/columns/kernel-locks/ww-mutex/</guid><description>Every lock covered in the previous six chapters is concerned with how a single lock can be acquired and released efficiently.</description><content:encoded><![CDATA[<p>Every lock covered in the previous six chapters is concerned with how a <strong>single lock</strong> can be acquired and released efficiently. Real systems, however, have a thornier class of problems: an operation that can only proceed safely while <strong>holding many locks at once</strong>. The canonical example is command submission in a GPU driver. A single submission often references dozens or hundreds of buffer objects, each with its own lock, and the driver must <strong>lock all of them</strong> before it can consistently validate, migrate, and map that memory. Rename in a file system and multi-VMA operations in memory management have similar needs.</p>
<p>As soon as you need several locks at once, you run into the most classic ghost of concurrent programming: the <strong>ABBA deadlock</strong>. Task 1 locks in the order &ldquo;A then B&rdquo;, task 2 locks in the order &ldquo;B then A&rdquo;; each holds one lock and waits for the one the other holds, and they are stuck forever. The textbook solution is a <strong>global lock order</strong>: everyone agrees to take locks in one fixed order (for example, by ascending memory address of the lock), and ABBA becomes impossible. In the GPU world that premise often does not hold. Which buffer objects get locked, and in what order, depends on the commands userspace submits; the driver cannot know in advance and cannot impose an order. When &ldquo;the locking order cannot be determined up front&rdquo;, it is time for this chapter&rsquo;s <code>ww_mutex</code> (wound/wait mutex). Instead of demanding a lock order, it introduces a <strong>deadlock avoidance</strong> algorithm: every &ldquo;locking transaction&rdquo; is stamped with a timestamp, and whenever a cycle might form, the <strong>younger transaction backs off</strong>, guaranteeing that the system as a whole always makes forward progress.</p>
<p><code>ww_mutex</code> is not a brand-new lock. It is <strong>built on top of the <code>mutex</code> from Chapter 5</strong>: the underlying mutual exclusion, sleeping, optimistic spinning, and handoff are all reused from <code>mutex</code>, and <code>ww_mutex</code> merely wraps a layer of &ldquo;timestamp + back-off&rdquo; deadlock avoidance logic around it. The source code for this chapter lives mainly in <code>kernel/locking/ww_mutex.h</code> (the core algorithm), <code>include/linux/ww_mutex.h</code> (data structures and API), <code>kernel/locking/ww_rt_mutex.c</code>, and <code>kernel/locking/test-ww_mutex.c</code>.</p>
<blockquote>
<p>Note: The main line is x86-64 with a non-<code>PREEMPT_RT</code> configuration, where the base of <code>ww_mutex</code>, <code>WW_MUTEX_BASE</code>, is the <code>mutex</code> from Chapter 5. Under <code>PREEMPT_RT</code> the base becomes <code>rt_mutex</code>; the algorithm is the same and is not covered separately.</p>
</blockquote>
]]></content:encoded></item><item><title>About</title><link>https://en.gpuers.cn/about/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/about/</guid><description>&lt;p&gt;GPUers publishes in-depth articles on Linux GPU drivers: how the DRM subsystem manages GPU memory, shares and synchronizes buffers, schedules work and maintains GPU address spaces, for both graphics and AI compute workloads.&lt;/p&gt;
&lt;p&gt;The articles are written by a Linux GPU driver engineer. They grew out of a Chinese-language column on the same topics, which has more than 880 articles and over a million reads on &lt;a href="https://blog.csdn.net/shenjunpeng"&gt;CSDN&lt;/a&gt; and is indexed at &lt;a href="https://gpuers.cn/"&gt;gpuers.cn&lt;/a&gt;.&lt;/p&gt;</description><content:encoded><![CDATA[<p>GPUers publishes in-depth articles on Linux GPU drivers: how the DRM subsystem manages GPU memory, shares and synchronizes buffers, schedules work and maintains GPU address spaces, for both graphics and AI compute workloads.</p>
<p>The articles are written by a Linux GPU driver engineer. They grew out of a Chinese-language column on the same topics, which has more than 880 articles and over a million reads on <a href="https://blog.csdn.net/shenjunpeng">CSDN</a> and is indexed at <a href="https://gpuers.cn/">gpuers.cn</a>.</p>
<p>Every article states the kernel version it describes and is checked against the kernel source. Corrections are welcome; please open an issue at <a href="https://github.com/gpuers">github.com/gpuers</a>.</p>
]]></content:encoded></item><item><title>Privacy</title><link>https://en.gpuers.cn/privacy/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/privacy/</guid><description>&lt;p&gt;This notice covers buying e-books and materials on this site. Reading articles does not require any personal information.&lt;/p&gt;
&lt;h2 id="what-we-collect"&gt;What we collect&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Email address&lt;/strong&gt;: the address you enter when ordering, used to deliver the e-book and for support.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Order details&lt;/strong&gt;: product, amount, order number, payment time and the payment provider&amp;rsquo;s transaction ID. We never see your PayPal account, card or bank details.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Request details&lt;/strong&gt;: the IP address and browser identifier at checkout, used only to prevent abuse and investigate problems.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="how-we-use-it"&gt;How we use it&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;To deliver the e-book, resend it, and answer your questions.&lt;/li&gt;
&lt;li&gt;To watermark your copy. The watermark shows only a masked email such as &lt;code&gt;j***u@gmail.com&lt;/code&gt; and the order number.&lt;/li&gt;
&lt;li&gt;Never for marketing, and never sold or shared with third parties. Payments are processed by PayPal under its own privacy policy.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="how-long-we-keep-it"&gt;How long we keep it&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Order details and email address: kept so we can resend the e-book and keep accounts.&lt;/li&gt;
&lt;li&gt;IP address and browser identifier: deleted after 180 days.&lt;/li&gt;
&lt;li&gt;Watermarked files: kept for 30 days after generation, then deleted and regenerated if needed.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="access-and-deletion"&gt;Access and deletion&lt;/h2&gt;
&lt;p&gt;To see or delete your personal data, email &lt;a href="mailto:junpeng.u@gmail.com"&gt;junpeng.u@gmail.com&lt;/a&gt; with your order number. After deletion we can no longer resend the e-book for that order.&lt;/p&gt;</description><content:encoded><![CDATA[<p>This notice covers buying e-books and materials on this site. Reading articles does not require any personal information.</p>
<h2 id="what-we-collect">What we collect</h2>
<ul>
<li><strong>Email address</strong>: the address you enter when ordering, used to deliver the e-book and for support.</li>
<li><strong>Order details</strong>: product, amount, order number, payment time and the payment provider&rsquo;s transaction ID. We never see your PayPal account, card or bank details.</li>
<li><strong>Request details</strong>: the IP address and browser identifier at checkout, used only to prevent abuse and investigate problems.</li>
</ul>
<h2 id="how-we-use-it">How we use it</h2>
<ul>
<li>To deliver the e-book, resend it, and answer your questions.</li>
<li>To watermark your copy. The watermark shows only a masked email such as <code>j***u@gmail.com</code> and the order number.</li>
<li>Never for marketing, and never sold or shared with third parties. Payments are processed by PayPal under its own privacy policy.</li>
</ul>
<h2 id="how-long-we-keep-it">How long we keep it</h2>
<ul>
<li>Order details and email address: kept so we can resend the e-book and keep accounts.</li>
<li>IP address and browser identifier: deleted after 180 days.</li>
<li>Watermarked files: kept for 30 days after generation, then deleted and regenerated if needed.</li>
</ul>
<h2 id="access-and-deletion">Access and deletion</h2>
<p>To see or delete your personal data, email <a href="mailto:junpeng.u@gmail.com">junpeng.u@gmail.com</a> with your order number. After deletion we can no longer resend the e-book for that order.</p>
]]></content:encoded></item><item><title>Subscribe</title><link>https://en.gpuers.cn/subscribe/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/subscribe/</guid><description>Follow new articles in your RSS reader.</description><content:encoded><![CDATA[<p>Add one of the feeds below to an RSS reader and new articles will show up there as they are published. No account, no email, nothing else.</p>
<p>If you don&rsquo;t use a reader yet, <a href="https://feedly.com/">Feedly</a>, <a href="https://www.inoreader.com/">Inoreader</a> (web and mobile) and <a href="https://netnewswire.com/">NetNewsWire</a> (macOS / iOS) are good places to start. Click a reader name below to add the feed directly, or copy the address and add it by hand.</p>
]]></content:encoded></item><item><title>Terms of Sale</title><link>https://en.gpuers.cn/terms/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/terms/</guid><description>&lt;h2 id="products-and-licence"&gt;Products and licence&lt;/h2&gt;
&lt;p&gt;The e-books and materials sold on this site are digital goods. Buying one gives you a personal licence to read and study it: you may keep, read and print it on your own devices.&lt;/p&gt;
&lt;p&gt;Without written permission you may not copy, share, resell or publish all or part of it, or use it for commercial purposes such as training courses.&lt;/p&gt;
&lt;p&gt;Every file carries a watermark tied to its order, including your masked email address and the order number, so that leaked copies can be traced.&lt;/p&gt;</description><content:encoded><![CDATA[<h2 id="products-and-licence">Products and licence</h2>
<p>The e-books and materials sold on this site are digital goods. Buying one gives you a personal licence to read and study it: you may keep, read and print it on your own devices.</p>
<p>Without written permission you may not copy, share, resell or publish all or part of it, or use it for commercial purposes such as training courses.</p>
<p>Every file carries a watermark tied to its order, including your masked email address and the order number, so that leaked copies can be traced.</p>
<h2 id="payment-and-delivery">Payment and delivery</h2>
<ul>
<li>The English site takes payment through PayPal, in US dollars.</li>
<li>After payment, the e-book is sent automatically to the email address you entered, usually within a few minutes.</li>
<li>The email also contains download links, valid for 7 days and up to 5 downloads per file.</li>
<li>Email is the only delivery channel, so please double-check your address when ordering. If you mistype it, find the &ldquo;Invoice ID&rdquo; starting with <code>GP</code> in the PayPal payment details and email us; we&rsquo;ll send the e-book to the correct address once we&rsquo;ve checked it.</li>
</ul>
<h2 id="refunds">Refunds</h2>
<p>Digital goods are non-refundable once delivered.</p>
<p>If the email doesn&rsquo;t arrive or a download fails, you can have it sent again from the <a href="/order/">order page</a>. If that doesn&rsquo;t help, contact us and we will resolve it by sending it again.</p>
<h2 id="contact">Contact</h2>
<p>Email <a href="mailto:junpeng.u@gmail.com">junpeng.u@gmail.com</a> with your order number.</p>
]]></content:encoded></item><item><title>Your Order</title><link>https://en.gpuers.cn/order/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://en.gpuers.cn/order/</guid><description>Enter your order number and email to check the status or get the e-book sent again.</description><content:encoded><![CDATA[<p>Enter the email address you used when ordering; if it doesn&rsquo;t match the order number, the order can&rsquo;t be found.</p>
<p>If the email didn&rsquo;t arrive, check your spam folder first.</p>
<p>If you mistyped your email when ordering, you won&rsquo;t have received the email or the order number. Open the payment in your PayPal activity: the &ldquo;Invoice ID&rdquo; is your order number (it starts with <code>GP</code>). Email it to <a href="mailto:junpeng.u@gmail.com">junpeng.u@gmail.com</a> together with the correct address and a screenshot of the payment, and we&rsquo;ll send the e-book to the new address once we&rsquo;ve checked it.</p>
]]></content:encoded></item></channel></rss>