https://cchenxi.github.io/tags/kafka/Recent content in Kafka on 陈曦的技术博客Hugozh-cnSun, 11 Oct 2026 00:24:00 +0800https://cchenxi.github.io/posts/%E6%B7%B1%E5%85%A5%E7%90%86%E8%A7%A3-kafka-%E7%94%9F%E4%BA%A7%E8%80%85-%E7%BC%93%E5%86%B2%E6%9C%BA%E5%88%B6-%E5%8F%8C%E7%BA%BF%E7%A8%8B%E6%A8%A1%E5%9E%8B%E4%B8%8E%E8%B0%83%E4%BC%98%E5%AE%9E%E8%B7%B5/Sun, 11 Oct 2026 00:24:00 +0800https://cchenxi.github.io/posts/%E6%B7%B1%E5%85%A5%E7%90%86%E8%A7%A3-kafka-%E7%94%9F%E4%BA%A7%E8%80%85-%E7%BC%93%E5%86%B2%E6%9C%BA%E5%88%B6-%E5%8F%8C%E7%BA%BF%E7%A8%8B%E6%A8%A1%E5%9E%8B%E4%B8%8E%E8%B0%83%E4%BC%98%E5%AE%9E%E8%B7%B5/<blockquote> <p>消息在 <code>send()</code> 之后去了哪里？为什么缓冲区会满、线程会阻塞？本文从双线程模型出发，逐层拆解 Kafka Producer 的缓冲、批量与背压机制。</p> </blockquote> <hr> <h2 id="一为什么生产者需要缓冲区">一、为什么生产者需要缓冲区</h2> <p>Kafka 生产者并不会在代码调用 <code>send()</code> 时立刻把消息发到网络，而是先将消息放入一块内存缓冲区，攒够一批再发。这是批量（batching）的核心思想。</p> <p>为什么要这样做？核心原因是网络 I/O 的成本。如果每条消息都单独发一次网络请求，开销巨大；如果把多条消息攒成一批，用一次请求发送，网络效率大幅提升。</p> <p>类比理解：去邮局寄 1000 封信，每写好一封就单独发一辆车（1000 次发车），远不如把信都放进箱子，等装满了一车送出（1 次发车）。Kafka 生产者选择的是后者。</p> <p>但批量设计引入了一个关键问题：缓冲区大小有限。如果放入速度持续快于取出速度，缓冲区迟早会被填满。填满之后，新的 <code>send()</code> 调用会阻塞，直到缓冲区有空间——或者超时报错。理解这个问题，需要深入生产者的内部架构。</p> <hr> <h2 id="二双线程模型">二、双线程模型</h2> <p>Kafka 生产者内部有两条并行的工作线：调用线程和 Sender 线程。</p> <p>下图展示了完整的双线程架构。上半部分是调用线程的流水线，下半部分是 Sender 后台线程的数据通路，中间的 RecordAccumulator 是两者共享的缓冲桥梁：</p> <div align="center"> <p><img alt="svg" loading="lazy" src="data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIwIDAgNjgwIDMzMCIgd2lkdGg9IjEwMCUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgcm9sZT0iaW1nIiBoZWlnaHQ9IjMzMCI+CiAgPHRpdGxlPkthZmthIHByb2R1Y2VyOiB0d28tdGhyZWFkIGFyY2hpdGVjdHVyZTwvdGl0bGU+CiAgPGRlc2M+VGhlIGNhbGxpbmcgdGhyZWFkIHNlcmlhbGl6ZXMgYW5kIHBhcnRpdGlvbnMgbWVzc2FnZXMgaW50byBhIHNoYXJlZCBidWZmZXI7IGEgc2luZ2xlIGJhY2tncm91bmQgc2VuZGVyIHRocmVhZCBkcmFpbnMgdGhlIGJ1ZmZlciBhbmQgc2VuZHMgYmF0Y2hlcyB0byBLYWZrYSBicm9rZXJzIG92ZXIgdGhlIG5ldHdvcmsuPC9kZXNjPgogIDxkZWZzPgogICAgPG1hcmtlciBpZD0iYXJyb3ciIHZpZXdCb3g9IjAgMCAxMCAxMCIgcmVmWD0iOCIgcmVmWT0iNSIKICAgICAgbWFya2VyV2lkdGg9IjYiIG1hcmtlckhlaWdodD0iNiIgb3JpZW50PSJhdXRvLXN0YXJ0LXJldmVyc2UiPgogICAgICA8cGF0aCBkPSJNMiAxTDggNUwyIDkiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzVGNUU1QSIKICAgICAgICBzdHJva2Utd2lkdGg9IjEuNSIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIi8+CiAgICA8L21hcmtlcj4KICA8L2RlZnM+CiAgPHRleHQgeD0iNTAiIHk9IjM1IiBzdHlsZT0iZm9udC1zaXplOjExcHg7ZmlsbDojODg4NzgwO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPmNhbGxpbmcgdGhyZWFkICh5b3VyIGFwcGxpY2F0aW9uIGNvZGUpPC90ZXh0PgogIDxyZWN0IHg9IjUwIiB5PSI1MCIgd2lkdGg9IjEwMCIgaGVpZ2h0PSI1MCIgcng9IjgiIGZpbGw9IiNFNkYxRkIiIHN0cm9rZT0iIzE4NUZBNSIgc3Ryb2tlLXdpZHRoPSIwLjUiLz4KICA8dGV4dCB4PSIxMDAiIHk9IjcyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBkb21pbmFudC1iYXNlbGluZT0iY2VudHJhbCIgc3R5bGU9ImZvbnQtc2l6ZToxM3B4O2ZvbnQtd2VpZ2h0OjUwMDtmaWxsOiMwQzQ0N0M7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+QXBwIHRocmVhZHM8L3RleHQ+CiAgPHRleHQgeD0iMTAwIiB5PSI4OCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZG9taW5hbnQtYmFzZWxpbmU9ImNlbnRyYWwiIHN0eWxlPSJmb250LXNpemU6MTFweDtmaWxsOiMxODVGQTU7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+c2VuZChyZWNvcmQpPC90ZXh0PgogIDxsaW5lIHgxPSIxNTUiIHkxPSI3NSIgeDI9IjE3NSIgeTI9Ijc1IiBzdHJva2U9IiM1RjVFNUEiIHN0cm9rZS13aWR0aD0iMS41IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgogIDxyZWN0IHg9IjE4MCIgeT0iNTAiIHdpZHRoPSI5MCIgaGVpZ2h0PSI1MCIgcng9IjgiIGZpbGw9IiNFNkYxRkIiIHN0cm9rZT0iIzE4NUZBNSIgc3Ryb2tlLXdpZHRoPSIwLjUiLz4KICA8dGV4dCB4PSIyMjUiIHk9Ijc1IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBkb21pbmFudC1iYXNlbGluZT0iY2VudHJhbCIgc3R5bGU9ImZvbnQtc2l6ZToxM3B4O2ZvbnQtd2VpZ2h0OjUwMDtmaWxsOiMwQzQ0N0M7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+SW50ZXJjZXB0b3JzPC90ZXh0PgogIDxsaW5lIHgxPSIyNzUiIHkxPSI3NSIgeDI9IjI5NSIgeTI9Ijc1IiBzdHJva2U9IiM1RjVFNUEiIHN0cm9rZS13aWR0aD0iMS41IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgogIDxyZWN0IHg9IjMwMCIgeT0iNTAiIHdpZHRoPSI4NSIgaGVpZ2h0PSI1MCIgcng9IjgiIGZpbGw9IiNFNkYxRkIiIHN0cm9rZT0iIzE4NUZBNSIgc3Ryb2tlLXdpZHRoPSIwLjUiLz4KICA8dGV4dCB4PSIzNDIiIHk9Ijc1IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBkb21pbmFudC1iYXNlbGluZT0iY2VudHJhbCIgc3R5bGU9ImZvbnQtc2l6ZToxM3B4O2ZvbnQtd2VpZ2h0OjUwMDtmaWxsOiMwQzQ0N0M7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+U2VyaWFsaXplcjwvdGV4dD4KICA8bGluZSB4MT0iMzkwIiB5MT0iNzUiIHgyPSI0MTAiIHkyPSI3NSIgc3Ryb2tlPSIjNUY1RTVBIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnJvdykiLz4KICA8cmVjdCB4PSI0MTUiIHk9IjUwIiB3aWR0aD0iMTAwIiBoZWlnaHQ9IjUwIiByeD0iOCIgZmlsbD0iI0U2RjFGQiIgc3Ryb2tlPSIjMTg1RkE1IiBzdHJva2Utd2lkdGg9IjAuNSIvPgogIDx0ZXh0IHg9IjQ2NSIgeT0iNzUiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGRvbWluYW50LWJhc2VsaW5lPSJjZW50cmFsIiBzdHlsZT0iZm9udC1zaXplOjEzcHg7Zm9udC13ZWlnaHQ6NTAwO2ZpbGw6IzBDNDQ3Qztmb250LWZhbWlseTpzYW5zLXNlcmlmIj5QYXJ0aXRpb25lcjwvdGV4dD4KICA8bGluZSB4MT0iMjkwIiB5MT0iMTAwIiB4Mj0iMjkwIiB5Mj0iMTQ1IiBzdHJva2U9IiM1RjVFNUEiIHN0cm9rZS13aWR0aD0iMS41IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgogIDx0ZXh0IHg9IjMwMCIgeT0iMTI1IiBzdHlsZT0iZm9udC1zaXplOjExcHg7ZmlsbDojODg4NzgwO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPmFwcGVuZCB0byBidWZmZXI8L3RleHQ+CiAgPHJlY3QgeD0iMTMwIiB5PSIxNDUiIHdpZHRoPSI0MjAiIGhlaWdodD0iNjUiIHJ4PSIxMiIgZmlsbD0iI0ZBRUVEQSIgc3Ryb2tlPSIjODU0RjBCIiBzdHJva2Utd2lkdGg9IjAuNSIvPgogIDx0ZXh0IHg9IjM0MCIgeT0iMTcwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBkb21pbmFudC1iYXNlbGluZT0iY2VudHJhbCIgc3R5bGU9ImZvbnQtc2l6ZToxNHB4O2ZvbnQtd2VpZ2h0OjUwMDtmaWxsOiM2MzM4MDY7Zm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+UmVjb3JkQWNjdW11bGF0b3IgKHRoZSBidWZmZXIpPC90ZXh0PgogIDx0ZXh0IHg9IjM0MCIgeT0iMTkwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBkb21pbmFudC1iYXNlbGluZT0iY2VudHJhbCIgc3R5bGU9ImZvbnQtc2l6ZToxMnB4O2ZpbGw6Izg1NEYwQjtmb250LWZhbWlseTpzYW5zLXNlcmlmIj5idWZmZXIubWVtb3J5ID0gMzIgTUIg4oCUIHNoYXJlZCBtZW1vcnkgcG9vbDwvdGV4dD4KICA8bGluZSB4MT0iMzQwIiB5MT0iMjEwIiB4Mj0iMzQwIiB5Mj0iMjU1IiBzdHJva2U9IiM1RjVFNUEiIHN0cm9rZS13aWR0aD0iMS41IiBtYXJrZXItZW5kPSJ1cmwoI2Fycm93KSIvPgogIDx0ZXh0IHg9IjM1MCIgeT0iMjM1IiBzdHlsZT0iZm9udC1zaXplOjExcHg7ZmlsbDojODg4NzgwO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPnNlbmRlciBkcmFpbnMgcmVhZHkgYmF0Y2hlczwvdGV4dD4KICA8dGV4dCB4PSI0MCIgeT0iMjgwIiBzdHlsZT0iZm9udC1zaXplOjExcHg7ZmlsbDojODg4NzgwO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPnNlbmRlciB0aHJlYWQgKHNpbmdsZSBiYWNrZ3JvdW5kIHRocmVhZCk8L3RleHQ+CiAgPHJlY3QgeD0iMTIwIiB5PSIyNTUiIHdpZHRoPSIxMDAiIGhlaWdodD0iNTAiIHJ4PSI4IiBmaWxsPSIjRTFGNUVFIiBzdHJva2U9IiMwRjZFNTYiIHN0cm9rZS13aWR0aD0iMC41Ii8+CiAgPHRleHQgeD0iMTcwIiB5PSIyODAiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGRvbWluYW50LWJhc2VsaW5lPSJjZW50cmFsIiBzdHlsZT0iZm9udC1zaXplOjEzcHg7Zm9udC13ZWlnaHQ6NTAwO2ZpbGw6IzA4NTA0MTtmb250LWZhbWlseTpzYW5zLXNlcmlmIj5TZW5kZXI8L3RleHQ+CiAgPGxpbmUgeDE9IjIyNSIgeTE9IjI4MCIgeDI9IjI1NSIgeTI9IjI4MCIgc3Ryb2tlPSIjNUY1RTVBIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnJvdykiLz4KICA8cmVjdCB4PSIyNjAiIHk9IjI1NSIgd2lkdGg9IjEwNSIgaGVpZ2h0PSI1MCIgcng9IjgiIGZpbGw9IiNFMUY1RUUiIHN0cm9rZT0iIzBGNkU1NiIgc3Ryb2tlLXdpZHRoPSIwLjUiLz4KICA8dGV4dCB4PSIzMTIiIHk9IjI4MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZG9taW5hbnQtYmFzZWxpbmU9ImNlbnRyYWwiIHN0eWxlPSJmb250LXNpemU6MTNweDtmb250LXdlaWdodDo1MDA7ZmlsbDojMDg1MDQxO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPk5ldHdvcms8L3RleHQ+CiAgPGxpbmUgeDE9IjM3MCIgeTE9IjI4MCIgeDI9IjQwMCIgeTI9IjI4MCIgc3Ryb2tlPSIjNUY1RTVBIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnJvdykiLz4KICA8cmVjdCB4PSI0MDUiIHk9IjI1NSIgd2lkdGg9IjEwNSIgaGVpZ2h0PSI1MCIgcng9IjgiIGZpbGw9IiNFMUY1RUUiIHN0cm9rZT0iIzBGNkU1NiIgc3Ryb2tlLXdpZHRoPSIwLjUiLz4KICA8dGV4dCB4PSI0NTciIHk9IjI4MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZG9taW5hbnQtYmFzZWxpbmU9ImNlbnRyYWwiIHN0eWxlPSJmb250LXNpemU6MTNweDtmb250LXdlaWdodDo1MDA7ZmlsbDojMDg1MDQxO2ZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPkJyb2tlcjwvdGV4dD4KPC9zdmc+"></p> </div> <h3 id="调用线程你的应用代码所在线程">调用线程（你的应用代码所在线程）</h3> <p>当调用 <code>producer.send(record)</code> 时，消息依次经过以下流水线：</p> <ol> <li><strong>Interceptors（拦截器）</strong>：可对消息做修改或过滤，大多数场景不使用</li> <li><strong>Serializer（序列化器）</strong>：将业务对象序列化为字节数组</li> <li><strong>Partitioner（分区器）</strong>：决定消息发往哪个 partition</li> <li><strong>RecordAccumulator（缓冲区）</strong>：将消息追加到对应 partition 的批次中</li> </ol> <p>这一步完成后，<code>send()</code> 方法就返回了（异步），调用线程可以继续处理后续逻辑。</p> <h3 id="sender-线程唯一的后台线程">Sender 线程（唯一的后台线程）</h3> <p>一个单独的后台线程不停从缓冲区取出&quot;已就绪&quot;的批次，通过 NetworkClient 发给 Kafka Broker。这个线程随 KafkaProducer 实例创建时启动，随 <code>close()</code> 终止。</p> <h3 id="关键含义">关键含义</h3> <ul> <li>应用线程是&quot;生产者&quot;——往缓冲区放数据</li> <li>Sender 线程是&quot;消费者&quot;——从缓冲区取数据</li> <li>缓冲区是两者之间的桥梁</li> <li>当放入速度 &gt; 取出速度，缓冲区逐渐填满</li> </ul> <p>多个应用线程可以共享同一个 KafkaProducer 实例（它是线程安全的），它们都在往同一个 buffer.memory 里写。但从缓冲区取数据发给 Broker 的，只有一个 Sender 线程。这是典型的生产者-消费者模式。</p>