Few generalizations about hardware design are more widely accepted than this: it is better to find errors early. And yet the traditional design-then-verify flow gives errors ample time to embed themselves in the design before even starting to look for them. It need not be so.
A technique borrowed from the software world—Test-Driven Development (TDD)—seeks to eliminate the problem by, in effect, reversing a portion of the design flow (Figure 1). In TDD, you develop the test harness for each unit of code before you write the code. It may sound like a rather pointless change in scheduling. But increasing numbers of design teams are finding that TDD makes a big difference: shortening design time, improving quality, and preparing design teams for emerging issues such as security and functional safety.
So what is TDD? It is an iterative design methodology based on the concepts of unit testing. In unit test, rather than attempting to test an entire functional block of code at once, you partition the block into units and test each unit in isolation before integrating it into the block. So what is a unit? The partitioning process is more art than algorithm. You want a unit to have a small number of inputs and outputs, a clear function that is independent of the inputs—and especially independent of the state of other units—and minimal internal state. In other words, you define your units so they are easy to test, without letting the number of units get out of hand.
The innovation of TDD is that you define the units from the requirements, before the code exists (Figure 2). Then, before you write the code for a unit, you write the test harness for it—again, working directly from the requirements and the software architecture. You check the test by applying it to the existing code for the overall block, if there already is any. The block should of course fail the new test, since you haven’t written the new code yet.
Now—finally, I hear you say—you write the code for the unit, apply the test harness, and iterate until the unit passes with no errors. Because you defined the unit for testability, run time should be short and—critically—the time to diagnose an error should be far shorter than if you had discovered the error at the block or system level. There’s just not that much going on inside a properly defined unit. After the code unit passes its test, you add it to the block and start the process over, writing the test harness for the next unit of requirements.
But why write the tests before there is any code to test? Is that just an affectation, one of those weird methodological quirks that software people are subject to from time to time? Experience says it is not. Writing the test harness first overcomes one of the major issues in testing: the tendency of developers to make the same errors in writing the test that they made in writing (or reviewing) the code. “TDD forces you to think like a user, not like a coder,” says Neil Johnson, chief technologist at XtremeEDA and a long-time advocate of TDD for hardware developers. You end up testing against the original requirements instead of testing against your understanding of your own code structure.
Additionally, if less obviously, TDD can induce coders to focus on the structure of the requirements and on the overall software architecture of the block before they dive into coding. This in itself has benefits.
But there is a larger issue behind the test-first approach, observes functional-verification expert Brian Bailey. TDD creates a second, independent interpretation of the design requirements: one that can be compared against the implementation code by simply running the unit tests. This two-track process can catch not only coding errors, but misinterpretations of the requirements and even ambiguities in the requirements documents—problems that can prove devilishly hard to diagnose later in the design flow.
But Does It Do Hardware?
Fine, I hear you say, for software folks. But we are doing hardware design. What does TDD have for us?
Caution is reasonable, in a world where design automation has often over-promised. Bailey says, “If you just move software concepts to hardware, you can expect problems. Software is essentially done when you’ve debugged it. A hardware design still has to go through synthesis, place and route, test insertion, layout … you have many chances to break it still.”
But within the confines of proving the functional correctness of the design, TDD is directly applicable. “Some software techniques do get mangled when you use them for hardware,” Johnson agrees. “But TDD works exactly the same whether you are writing a program in C or describing behavior in System Verilog, or even describing a structure in Verilog.”
At the functional level, TDD concepts are not new to hardware design. “Assertions are in a way a form of TDD,” Bailey points out. If you are following best practices, you are writing your assertions from the requirements, just as you would in a TDD flow. But one of the challenges in exploiting TDD in hardware design is that there can be several very different levels of abstraction hiding under the label of functional design.
Many teams begin by interpreting the requirements as a behavioral model in C, or even in a more modern language like Python or Java. In a TDD flow, this model would require test harnesses in its own language. Other teams may go directly from requirements into System Verilog, in which case they would probably develop the unit tests within the Universal Verification Methodology (UVM) framework. “But there are many designers who have never written behavioral code,” Johnson says. “They go directly from the requirements documents to RTL. So we are seeing some designers using TDD with Verilog.”
Generating behavioral tests for a hardware-description language does require some care, Johnson says. After all, there will be far more detail in the Verilog code than would be necessary to simply describe the functional behavior of the unit. Consciously raising the level of abstraction in the unit tests you are writing, for instance by thinking of the unit as a set of functions rather than a lump of in-line code, can help. “You can create your own test API,” Johnson says.
Even if the design starts in C, it will pass through register transfer level (RTL) eventually. It would be very valuable if the unit tests could be applied at every level of abstraction, rather than originating in one language and then getting manually translated into something else once or twice. That is the goal of Accellera’s Portable Stimulus (PS) Specification working group: to define a language for specifying behavior once, and using that specification to stimulate the behavioral models and even the implementation at a variety of levels, clear through physical prototyping (Figure 3). PS is not dependent on TDD, but it will certainly help TDD.
The Skeptic’s Corner
With all those promises, TDD can generate a lot of questions, especially in the hardware world, where design and verification are traditionally separate professions, applied in a strictly serial way. For instance, is TDD just an attempt by verification engineers to muscle into design territory? No, says Bailey. “TDD and PS both aim at creating two parallel, equally important views of the requirements: one defining the behavior and the other defining the implementation.” The test design and logic design efforts are unlikely to make the same errors simply because they are doing very different things. In this way TDD can be almost as effective as two independent design teams working in parallel: an approach that is known to have advantages, but that is too expensive for most projects. Here, there is no redundant work, since both the design and its tests have to be created anyway, but there is much of the improved quality that comes from redundancy.
But even when there is no separate verification team, TDD’s test-first approach tends to focus designers on the requirements before they are drawn into the complexities of implementation. The surest way to find out whether you really understand a requirement, Johnson says, is to try to write a test for it.
But doesn’t all this partitioning—critics would say fragmenting— of the design, test-writing, version control, tracking, and building become an administrative nightmare? Yes, it can be, without automation. Consequently, a number of platforms have appeared to automate the TDD process. Even so, isn’t all of this stuff going to destroy the project schedule? No, and maybe. Both software teams with long TDD experience and hardware teams using TDD claim that it generally shortens the overall project, sometimes dramatically. This is due to the larger portion of errors getting caught in unit tests, where they are easier to isolate and diagnose. If just a few such errors get through to system integration, they can be ruinous to the system verification schedule.
There is one undeniable problem, though. The test-first approach puts a number of non-coding activities before the point at which you start cranking out code, and by interleaving test development and coding it slows down the rate at which you are producing test-ready code. If you have an old-school manager who is gauging progress by lines of code per week or by degree of functional coverage, or some such metric, your manager may seriously misunderstand the rate of progress on the project. This can cause, shall we say, unnecessary friction in the design team.
A more serious objection is that unlike many software projects, most hardware designs include large amounts of reused intellectual property (IP)—either licensed from third parties, picked up as open-source, or re-used from previous projects. Since this IP is supposedly already verified, this would appear to limit the use of TDD to the functional design of the relatively small portion of the system that will be newly-written.
But there is an important role for TDD in IP reuse as well, Baily and Johnson concur. One of the most frequent sources of failure in IP reuse is misappropriation: when you use the IP block in a way for which it was never tested, or maybe never intended. By starting with your system requirements, you can generate tests that will verify that the IP block actually does what you assume it does. This may be possible even if you don’t have access to a behavioral model of the IP in order to create unit tests. You can write tests based on units of your requirements, and then black-box test an entire encrypted model of the IP rather than individual units of model code.
There are some interesting extensions to this idea. Bailey points out that one crucial but often-ignored step in functional verification is to prove that the IP blocks are not performing unintended functions. “The units of your requirements often don’t exactly match the structure of the IP you are using,” Bailey explains. “This can lead to functions you don’t use, or don’t even know about, still sitting in the IP. If they get inadvertently activated, they can cause havoc.” Using TDD unit tests in combination with activity monitoring, you can identify regions of the IP code that are dead in your use case, or units that appear to get activated at inappropriate times, and remove or disable them. This could be a crucial step toward getting functional-safety certification for a design.
Another idea carries this concept even further. Some of the IP in your system may be in the form of chips rather than design files. Often the chips come with marginally informative or simply incorrect data sheets, and without executable models. Using TDD to generate tests and a development kit of hardware test fixture to perform the tests, you can go a long way toward verifying your understanding of the chip’s operation, especially during initialization, sleep/wake, and other corner sequences.
One final question has no good answer today. What about timing? You set timing constraints before synthesis, very much in the way TDD creates test harnesses before coding. Isn’t there an analogy?
The quick answer is no. TDD works in the functional domain, which is inherently untimed. But still … The problem with making an analogy, Bailey says, is that while TDD derives functional tests from system requirements, there is no known way to derive timing constraints from system performance requirements. The latter are stated in terms of minimum data rates and function latencies. But CPU-based systems are so non-deterministic that there is little relationship between when a function will complete and the path delays inside its hardware blocks. Even with deterministic hardware the relationship can be complex. So timing remains beyond the pale.
Experience of a range of hardware design teams argues that TDD is indeed an important methodological approach. It requires a lot of personal reorientation, benefits greatly from use of a specialized platform, and may not feel familiar to most designers the first time through. But the ability to focus test on the system requirements, and to isolate errors early in the design flow, make it an idea whose time is very much now.
Explore the design flow for Intel® FPGAs.