How to Test an RPL Pattern

The RPL toolchain is implemented as a plugin-based rustc customization, adding its custom logic via callbacks into the standard compilation process. Consequently, we can leverage rustc's UI testing framework to test RPL patterns.

Testing serves two critical purposes in the RPL development lifecycle:

1. During Development: Tests are used to verify that a pattern produces the expected diagnostic output on a code snippet known to be vulnerable. This stage is also crucial for refining the pattern to maximize its precision and recall.
2. During Maintenance: Tests act as regression guards. They ensure that patterns remain effective over time, especially when upgrading to a new nightly rustc version, as the underlying MIR representation can change. Failing tests indicate that a pattern needs to be updated to remain valid.

The Role of UI Tests

The primary purpose of rustc's UI tests is to validate the console output of the compiler. This aligns perfectly with RPL's operational model, which is designed to generate specific diagnostic messages (i.e., console output) when it detects a matching code pattern.

Writing a UI Test for an RPL Pattern

Continuing with the cve_2020_35881_test.rs file from the previous section, we will now convert it into a proper UI test. This process involves cleaning up the file, generating the expected error output, and annotating the source code to match that output.

Step 1: Remove Debug Attributes

First, remove the #[rpl::dump_mir] attributes from the test file. These are used only for the initial debugging and MIR generation phase and are not part of the final test case.

Step 2: Generate the Expected Output (.stderr)

Next, run the cargo uibless tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs command on the test file. This command compiles the file and captures the exact diagnostic output into a corresponding .stderr file.

The command is expected to fail. The output will report "unmatched diagnostics" because the pattern has correctly fired, but the test file does not yet contain any annotations describing this expected output. The important part of the output is the full stderr section, which shows the exact error message generated by your pattern:

 cargo uibless tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.15s
     Running tests/compile-test.rs (target/debug/deps/compile_test-251936bcca420e4b)
tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs ... FAILED

FAILED TEST: ...

error: there were 1 unmatched diagnostics
 --> tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs:5:15
  |
5 |     unsafe { *mem::transmute::<*const *const T, *const *const ()>(&val) }
  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Error[rpl::wrong_assumption_of_fat_pointer_layout]: wrong assumption of fat pointer layout
  |

error: there were 1 unmatched diagnostics
  --> tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs:10:15
   |
10 |     unsafe { *mem::transmute::<*mut *mut T, *mut *mut ()>(&mut val) }
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Error[rpl::wrong_assumption_of_fat_pointer_layout]: wrong assumption of fat pointer layout
   |

error: expected error patterns, but found none

full stderr:
error: wrong assumption of fat pointer layout
  --> tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs:5:15
   |
LL |     unsafe { *mem::transmute::<*const *const T, *const *const ()>(&val) }
   |              -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |              ||
   |              |ptr transmute here
   |              try to get data ptr from first 8 bytes here
   |
   = help: the Rust Compiler does not expose the layout of fat pointers
   = note: `#[deny(rpl::wrong_assumption_of_fat_pointer_layout)]` on by default

error: wrong assumption of fat pointer layout
  --> tests/ui/cve/cve_2020_35881_test/cve_2020_35881_test.rs:10:15
   |
LL |     unsafe { *mem::transmute::<*mut *mut T, *mut *mut ()>(&mut val) }
   |              -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |              ||
   |              |ptr transmute here
   |              try to get data ptr from first 8 bytes here
   |
   = help: the Rust Compiler does not expose the layout of fat pointers

error: aborting due to 2 previous errors

Step 3: Annotate the Source File

Now, copy the primary error message from the generated .stderr content and add it as a special comment in your .rs test file. The testing framework uses these comments to verify that the compiler produces the correct error.

The syntax is //~^ ERROR: message.

• //~ is a directive to the test harness.

• ^ points to the line immediately above the comment.

• ERROR: specifies the diagnostic level.

• message is the expected primary error text.

Update your cve_2020_35881_test.rs file as follows:

use std::mem;

pub unsafe fn get_data<T: ?Sized>(val: *const T) -> *const () {
    unsafe { *mem::transmute::<*const *const T, *const *const ()>(&val) }
    //~^ ERROR: wrong assumption of fat pointer layout
}

pub unsafe fn get_data_mut<T: ?Sized>(mut val: *mut T) -> *mut () {
    unsafe { *mem::transmute::<*mut *mut T, *mut *mut ()>(&mut val) }
    //~^ ERROR: wrong assumption of fat pointer layout
}

fn main() {}

Step 4: Run the Test to Verify

Finally, run the UI test suite to confirm that the actual output from the compiler matches your annotations.

cargo uitest