Lesson 23: Atomic Operations: atomicAdd & histograms
When thousands of threads try to update the same memory cell at once, a race condition arises. Take a histogram: each thread reads a value and wants to increment the counter bins[v] by one. The operation bins[v]++ looks atomic, but it is really three separate steps: read the current value, add one,
Imagine a shared counting jar. Without atomic, two people both see the number 5, both add 1 in their head, and both write 6 — but it should have reached 7, so a count was lost. atomic is like letting only one person at a time hold the jar, add, and put it back — nobody steps on anybody.
- race condition
- When two parallel operations access the same memory and at least one writes, so the result depends on the arbitrary order in which they run.
- atomic operation
- A read-modify-write the hardware performs as one indivisible unit — no other thread slips in between. For example atomicAdd.
- read-modify-write
- The three steps of ++: read the value, modify, write back. In parallel, without protection, these steps interleave across threads.
- histogram
- Counting how many times each value appears. Each thread increments bins[v]; many threads on the same bin require atomicAdd.
