A PCI/PCI-Express Primer Part 3: Let’s Play

Note: This is part three of a three-part blog series. Part one is availablehereand part two is availablehere.

Now that we understand the basics, it's time to see what the interface looks like in the real world.

In the old days, one would need to open the entire /dev/mem file and seek the defined offset in order to interact with the BAR.

TheBARsinformations are shown by thelspcitool with the verbose flag:

Figure 1: lspci BARS info

In this example, 4 BARs were declared: three in memory at offsets0xdc000000,0xb0000000,0xc0000000, and one in I/O space at0xe000with the various attributes as discussed above.

Nowadays, modern Linux distribution make the BARs directly accessible in the/sys

BAR0 => /sys/bus/pci/devices/[device ID dir]/resource0
BAR1 => /sys/bus/pci/devices/[device ID dir]/resource1
BAR1 => /sys/bus/pci/devices/[device ID dir]/resource1_wc
BAR3 => /sys/bus/pci/devices/[device ID dir]/resource3
BAR3 => /sys/bus/pci/devices/[device ID dir]/resource3_wc
BAR5 => /sys/bus/pci/devices/[device ID dir]/resource5
 

The Linux documentation does not really indicate what theresourceX_wccorresponds to, but an educated guess would be that this file maps to the prefetchable version of the BAR.

Once you have identified which file you need,mmapit into your process memory, keep the initial address pointer for the region, and operate on it directly via memory operations, within the limit of what is permitted by theBARattributes (TypeandPrefetchable).

Beyond this, the standard does not dictate what theBARsshould contain. This is where the vendor documentation (or lack thereof) comes into play.

TheBARsare very good targets for fuzzing, but be careful. If theBARexpose functionalities that controls things like card voltage or thermal cutoff, you might be in for a wild ride!

FuzzingBARsmay also trigger very subtle bugs in the kernel itself that are not easily detectable or that only disturb/crash the system much later on.

Something to keep in mind as well is that the entireBARspace is not necessarily functional (i.e., some registers area of the BAR are not linked to anything) and will return seemingly random value when read and/or have no effect when written. For example, NVidia devices will return a value of the form0xBAD0XXXXif the address register is not used, and no amount of write will ever change this value.

Among the observed behavior during fuzzingBARs, kernel crashes are usually the most frequent but carry a low risk rating. This is because if you have access to theBARs, you are already root, and root has much simpler ways of crashing a system (memory or PID saturation, for example). Although they should not be discarded, crashing a system as root will not make many vendors panic.

This is another story if the system is shared between customers using a virtualized environment, which means that the PCI BUS is going to be shared as well. In this case, a hardware crash has a high probably of crashing the hypervisor and shutting down other customers’ VMs. That could be a direct hit to the core business bottom line, something our customers takes very seriously!

Once you have exhausted theBARstesting, it is worth noting that the PCI specification details a common set of functionalities that are frequently used across all extension cards. Instead of leaving their definitions to each vendors, they are explicitly described by the standard.

Capabilities and Extended Capabilities

Capabilitiesare basic functionalities that are explicitly defined in the PCI standard.

Extended Capabilitiesare similar toCapabilities, but are defined in the extended Configuration Space thus only relevant for PCI-e devices.

Both are stored within (relatively) simple linked-lists, as described by the standard. TheCapabilitieslist starts at theConfiguration Spaceoffset value defined in one of the field of the PCI header (located at0x34in theConfiguration Space), whereas theExtended Capabilitieslist simply starts right after the PCI header (offset0x100in theConfiguration Space).

These offsets can be shown by the lspci command with the verbose flag:

Figure 2: lspci Capbilities info

The offsets are given between square brackets in hexadecimal. EachCapabilityis described in detail within the standard and, although many of them contain read-only registers, some may be able to modify the hardware configuration. These are usually designated asControl Registers(as opposed toState Registers).

Capabilities worth looking at include (but are not limited to):

• Power Management (sounds a bit dangerous)
• ACS Extended Capabilities (Access Control at the PCI bus level)
• Resizable BAR Capability (Dynamic resizing of the target BARs, may allow access to non-default memory regions from the extension card memory)
• Vendor Specific Extended Capability (as defined by the vendor, usually interesting)

Once again shutting down or damaging the hardware using the Power Management capability of your laptop has limited value, but the game changes in industrial environments where some extension cards costs thousands of dollars each, and making them vulnerable to local attacks may have serious consequences for the business.

This is once more amplified by the 'Cloud Principle', where hardware is shared among multiple customers.

Summary

We have looked over the basics of in-memory register access and write, and took a deep dive into how PCI/PCI-e cards expose their functionalities to user space.

In principle, this is not really difficult as long as you have access to the PCI specifications and comprehensive vendor documentation. However, given the numerous functionalities exposed by the PCI standard when coupled with vendor specifications, this can quickly turn into tedious work if you have to write the interfaces by yourself.

Additionally, we haven't talked about the I/O space BARs but they work similarly as the Memory Space, the main difference being that each I/O address is much closer to a simple cable, which adds some complications due to timing constraints. Having said that looking at the wording in various documentations, it looks like they are trying to make them obsolete.

If you are tasked with such tests, be sure to ask for both the PCI-Express specifications, if you don't have them already, and the vendor’s documentations describing exposed functionalities in detail. When looking at the latter, be sure to check the undocumented registers for their liveliness—this could lead to some interesting discoveries.