PCIe Protocol – Part 3 - logicmadness.com

Data Link Layer (DLL), flow control, ACK/NAK, replay mechanisms, and transaction layer ordering rules.

Table of Contents

1. Data Link Layer Overview

The Data Link Layer (DLL) sits between the Transaction Layer and the Physical Layer. Its primary responsibilities are:

Ensuring reliable delivery of TLPs
Generating DLLPs (Data Link Layer Packets) for flow control and acknowledgments
Maintaining data integrity
Supporting replay and error recovery

ASCII Layer Diagram

+--------------------+
| Transaction Layer  |  <- TLPs
+--------------------+
|  Data Link Layer   |  <- DLLPs (ACK/NAK, Flow Control)
+--------------------+
| Physical Layer     |  <- Electrical signals over lanes
+--------------------+

2. Flow Control Mechanism

PCIe uses a credit-based flow control system to avoid buffer overflows.

2.1 Credit Types

Credit Type	Purpose
Posted Data Credit	Memory writes and messages
Non-Posted Data Credit	Reads and non-posted messages
Header Credit	TLP headers

Key Points:

Each device advertises its available buffer space to the upstream device.
Upstream device cannot send more data than the credits allow.
Credits are replenished as data is processed.

2.2 DLLPs for Flow Control

The Data Link Layer sends DLLPs to manage flow:

ACK DLLP: Confirms successful reception
NAK DLLP: Indicates a failed or corrupted packet; triggers replay

ASCII Example:

TLP sent from RC to EP
        |
        V
     [EP receives]
        |
   +----+----+
   |  ACK   | <- DLLP sent back to RC
   +--------+

3. Data Integrity and Replay

3.1 CRC in DLL

Every TLP contains a CRC (Cyclic Redundancy Check)
Physical and data link layers verify integrity
Corrupted TLPs trigger NAK + replay

3.2 Replay Mechanism

Failed or lost TLPs are stored in replay buffers at the transmitter
When a NAK is received, TLP is retransmitted

Replay Flow (ASCII)

[TX Buffer] ---> TLP ---> [RX]
                      |   \
                      |   [CRC OK?]
                      |        |
                      |     ACK -> remove from TX buffer
                      |
                     NAK -> replay from TX buffer

4. Transaction Layer Ordering Rules

PCIe defines strict ordering rules to maintain data coherency:

Rule	Description
Posted Writes	Can be reordered freely unless flagged otherwise
Non-Posted Reads	Must maintain order relative to posted writes
Locked Transactions	Cannot be reordered; ensure atomicity
TLP Prefix	Can enforce ordering or bypass reordering

Ordering rules are critical for memory coherency, especially in multi-core or virtualized systems.

5. TLP Prefix Rules

Local Prefix: Only applies to the device that generated the TLP
End-to-End (E2E) Prefix: Travels with the TLP across multiple devices
Can carry hints, PASID, steering tags, or QoS information

6. Virtual Channels (VC) and QoS

PCIe supports multiple virtual channels for traffic isolation:

Each channel has independent flow control credits
Allows priority handling of latency-sensitive TLPs
Reduces congestion for bulk transfers

ASCII Example

[VC0] -> High-priority (Interrupts, control messages)
[VC1] -> Medium-priority (Memory Reads/Writes)
[VC2] -> Low-priority (Bulk Data Transfers)

7. Summary – Data Link Layer

Responsibilities:

Ensures reliable delivery of TLPs
Uses DLLPs (ACK/NAK) for flow control and error handling
Manages credit-based flow control
Supports replay mechanism for corrupted packets
Works with transaction layer ordering and virtual channels