Thursday, August 30, 2012

Linux PCI Driver Model

PCI Understanding:

  • The Peripheral Component Interconnect Bus (PCI) today is present in a wide variety of microcomputers ranging from Intel-based PC architectures to DEC-Alpha-based work-stations.
  • The CPU communicates with the PCI subsystem using a special chipset known as PCI-Bridge.
  • PCI-Bridge is an intelligent controller, that handles all necessary tasks to transfer data from or to the CPU or the Memory Subsystem.
  • On PCI the addresses and data are transferred as separate chunks over the bus because all bus lines can be used either as address or as data-lines.
  • Each bus device has its own 265byte space of Memory for configuration  purposes that can be accessed through the CONFIG_ADDRESS and the CONFIG_DATA registers.
  • All devices that are known to Linux you will see at /proc/pci.
  • Device resources (I/O addresses, IRQ lines) automatically assigned at boot time, either by the BIOS or by Linux itself (if configured). 
  • To identify a certain device while driver writing you will at least have to know the vendor-id and the device-id that is statically stored in the device configuration block.
  • Driver writers normally need to know only the base address of the device and the IRQ line that the device is using.
  • PCI device configuration information is Little-Endian. Remember that in your drivers. 
  • lspci enumerate all the devices.
          00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
          00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 05)
          |     |   |
          |     |   |_Function Number
          |     |_PCI Device Number
          |_PCI Bus Number

  • lspci -tv
          -[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family DRAM Controller
                          +-01.0-[01]--+-00.0  nVidia Corporation GF108 [GeForce GT 430]
                          |            \-00.1  nVidia Corporation GF108 High Definition Audio Controller
             |        |       |        |   
             |        |       |        |_ PCI Bus-1
             |        |       |_ PCI Bridge
             |        |_ PCI Bus-0
             |_ PCI Domain

  • Device configuration can be displayed with lspci -x
  • Standard information found in PCI configurations:
    • Offset 0: Vendor Id
    • Offset 2: Device Id
    • Offset 10: Class Id (network, display, bridge...)
    • Offsets 16 to 39: Base Address Registers (BAR) 0 to 5
    • Offset 44: Subvendor Id
    • Offset 46: Subdevice Id
    • Offsets 64 and up: up to the device manufacturer
    • Kernel sources: these offsets are defined in include/linux/pci_regs.h

  • PCI Device Initialization steps:
    • Enable the device.
    • Request I/O port and I/O memory resources.
    • Set the DMA Mask for both coherent and streaming DMA.
    • Allocate and Initialize shared coherent data.
    • Initialize device Registers.
    • Register IRQ handler.
    • Register to other subsystems (network, video ..)
    • Enable DMA or Processing Engines.
    • Before touching any device registers, the driver should first execute pci_enable_device(). This will:
      • Wake up the device if it was in suspended state.
      • Allocate I/O and memory regions of the device, If not already done by the BIOS.
      • Assign IRQ to the device, If not already done by the BIOS 
    • Enable DMA by calling pci_set_master(). This will:
      • Enable DMA by setting the bus master bit in the PCI_COMMAND register. The device will then be able to act as a master on the address bus.
      • Fix the latency timer value if it's set to something bogus by the BIOS.
      • This enables the PCI_COMMAND bit for Memory ­ Write Invalidate.
      • This also ensures that the cache line size register is set correctly.
    • Accessing configuration registers:
      • Reading:
        • int pci_read_config_byte(struct pci_dev *dev, int where, u8 *val);
        • int pci_read_config_word(struct pci_dev *dev, int where, u16 *val);
        • int pci_read_config_dword(struct pci_dev *dev, int where, u32 *val);
      • Writing:
        • int pci_write_config_byte(struct pci_dev *dev,  int where, u8 val);
        • int pci_write_config_word(struct pci_dev *dev,  int where, u16 val);
        • int pci_write_config_dword(struct pci_dev *dev,  int where, u32 val);
    • Accessing I/O registers and memory:
      • Each PCI device can have up to 6 I/O or memory regions, described in BAR0 to BAR5.
      • Access the base address of the I/O region:
        • long iobase = pci_resource_start (pdev, bar);
      • Access the I/O region size:
        • long iosize = pci_resource_len (pdev, bar);
      • Reserve the I/O region:
        • pci_request_region(pdev, bar, “my driver”);
    • Use pci_dma_set_mask() to declare any device with more (or less) than 32­bit bus master capability
    • In particular, must be done by drivers for PCI­X and PCIe compliant devices, which use 64 bit DMA.
    • If the device can directly address "consistent memory" in System RAM above 4G physical address, register this by calling pci_set_consistent_dma_mask().
    • You can allocate your cache consistent buffers if you plan to use such buffers. 
    • If needed by the device Set some “capability” fields and do some vendor specific initialization or reset Example: clear pending interrupts.
    • Register interrupt handlers:
      • Need to call request_irq() with the IRQF_SHARED flag, because all PCI IRQ lines can be shared.
      • Registration also enables interrupts, so at this point
        • Make sure that the device is fully initialized and ready to service interrupts.
        • Make sure that the device doesn't have any pending interrupt before calling request_irq(). 
      • Where you actually call request_irq() can actually depend on the type of device and the subsystem it could be part of (network, video, storage...). 
      • Your driver will then have to register to this subsystem.

  • PCI Device Shutdown: 
    • Disable the generation of new interrupts. If you don't, the system will get spurious interrupts, and will eventually disable the IRQ line. Bad for other devices on this line!
    • Release the IRQ.
    • Stop all DMA activity. Needed to be done after IRQs are disabled (could start new DMAs)
    • Release DMA buffers: streaming first and then consistent ones.
    • Unregister from other subsystems
    • Unmap I/O memory and ports with io_unmap().
    • Disable the device with pci_disable_device().
    • Unregister I/O memory and ports. If you don't, you won't be able to reload the driver. 


  1. Good articles.I'm progamming linux DMA for a pci customized board. But I got errors in IRQ handler.

  2. goood document for understanding PCI driver basics