A minor logging addition to a Canal-based binlog ingestion module caused the entire synchronization process to halt. The modification appeared harmless:

LOG.trace("Inspecting transfer object: {}", payload);
transferQueue.offer(payload);

Upon deployment to the staging environment, data replication ceased. Reverting the change did not immediately restore functionality, but terminating the job triggered a sudden flush of pending records. This behavior indicated that the consumer thread was stuck in a blocked state rather than failing outright.

To isolate the bottleneck, a thread dump was captured using jstack <pid>. The output revealed the reader thread parked indefinitely:

"binlog-sync-task-01,ReaderThread_0" #45 prio=5 os_prio=0 tid=0x00007f... nid=0x... waiting on condition [...]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.etl.core.buffer.RingBufferCache.poll(RingBufferCache.java:112)
        at com.etl.core.channel.DataChannel.fetchNext(DataChannel.java:58)
        at com.alibaba.fastjson.serializer.ASMSerializer_7_DataChannel.write(Unknown Source)
        ...
        at com.alibaba.fastjson.JSON.toJSONString(JSON.java:740)
        at com.etl.plugin.reader.BinlogFetcher.execute(BinlogFetcher.java:142)

The stack trace highlights a critical flaw: the JSON serialization routine invoked DataChannel.fetchNext(), which delegates to ArrayBlockingQueue.take(). Since the internal buffer was empty at the time of logging, the reader thread blocked waiting for data that it was supposed to be producing. The architecture strictly separates responisbilities—the reader thread pushes records into the queue, while a separate writer thread drains it. Serializing the channel object inadvertently crossed this boundary.

The fetchNext implementation confirms the blocking behavior:

public RecordSet retrieveNext() {
    return bufferPool.take(); // Blocks indefinitely when empty
}

Removing the log statement eliminated the unintended serialization call, allowing the reader thread to resume its primary loop. A healthy thread dump shows the reader correctly parked on the Canal event store rather than the internal transfer queue:

"binlog-sync-task-01,ReaderThread_0" #45 prio=5 os_prio=0 tid=0x00007f... nid=0x... waiting on condition [...]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at com.alibaba.otter.canal.store.memory.MemoryEventStoreWithBuffer.get(MemoryEventStoreWithBuffer.java:219)
        at com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded.getWithoutAck(CanalServerWithEmbedded.java:347)
        at com.etl.plugin.reader.BinlogFetcher.execute(BinlogFetcher.java:98)

This state confirms the thread is properly waiting for upstream binlog events, maintaining the expected producer-consumer flow without internal queue contention.