CSTD Testing: Why is this so hard?
One of the major challenges in choosing a CSTD has been the lack of an unbiased, standardized test to evaluate performance. All CSTDs must be approved by the FDA before being sold in the US. In 2012, the FDA created the ONB code which is specific to for CSTDs. The FDA however makes no statement about how well the device works, only that the manufacturer has data to support their product’s ability to reduce HD exposure by being ‘closed.’ However, in the absence of a standardized test, it is difficult to compare apples to chocolate to cheddar. While all of them are edible, they’re not nutritionally equal.
NIOSH Testing Protocols
In 2015, NIOSH released a prototype standardized testing protocol for CSTDs using 70% IPA (isopropyl alcohol). While they did not perform the testing themselves, the protocol was published to allow each company to perform their own testing. (Independent testing was also performed by pharmacists at Nebraska Medical).
Unfortunately, the design of the test protocol was problematic, particularly for one of the CSTDs that uses air filtration technology and was not designed for 70% IPA. The test also placed a heavy emphasis on vapor containment during compounding, as indicated by the title (“A Vapor Containment Performance Protocol for Closed System Transfer Devices Used During Pharmacy Compounding and Administration of Hazardous Drugs.”
After public comment, NIOSH asked for input from the CSTD manufacturers in designing a better protocol that would be more reflective of real-world HD compounding and administration. Several other surrogates have been looked at, and many of the device manufactures have done their own tests with the 2018 version. However, as of January 2020, a final version of the testing protocol has not been released. NIOSH has established a web page specifically for CSTD testing (link here).
One other point about the NIOSH protocol once it has been finalized: They will not be performing the tests. Each company will perform their own. It would be better if a disinterested third party tested all devices, to prevent potentially manipulation the test to skew the results. (More on that below.)
How do we know what works?
Being first to market in 1999, Phaseal had strategic advantages and disadvantages. It was a unique product that solved a problem many people still felt did not exist. The added cost and ergonomic issues (particularly related to the first version) were seen as huge hurdles for adoption. To help prove its efficacy, Carmel Pharma (who owned Phaseal at the time) enlisted the help of HD champions. Publications by Tom Connor (2002), Catherine Wick (2003), Susan Spivey (2003) showed that the device worked in both simulated and real situations. Several other studies followed, and while some were better designed than others, it allowed Phaseal to accumulate more than a dozen published studies over the past 20 years.
The competition emerged in the mid-2000's but by this time there seemed to be less of a desire (or perhaps a clinical resistance) to repeat what Phaseal had done. Most of the devices that followed have one or two studies, but some have none at all. This point is pertinent since USP <800>, acknowledging that NIOSH has yet to release its test protocol, recommends reviewing published peer-reviewed studies.That presents a conundrum because lack of publication does not necessarily equate to lack of efficacy. On the other hand, how are we as nurses to know which products perform well and which ones do not?
Another problem is that sometimes published studies are sponsored by the company that makes the product. This can be particularly tricky when one company is comparing their product to someone else’s. It might be difficult to trust a Mustang versus Camaro comparison if the test was sponsored by General Motors (or if the testers were all established GM customers).
We should also remember that CSTD tests are not robustly designed like large, randomized phase III studies comparing drug x to drug y. There are a lot of potential uncontrolled variables that can influence a CSTD’s performance during the study (variables that would be laughed at if we were testing medications). So, while published tests are generally a good source of information, they should also be scrutinized with a keen eye.
White papers
All of the companies have “white papers” for their products, typically demonstrating effectiveness during tests done by the companies themselves. While this is a good source of information, we are also taking the word of the company—even if the testing parameters are clearly spelled out in the paper. And they would obviously not release a paper showing their product doesn't do what it is purported to do.
Where do we go next?
We can only hope that when the NIOSH protocol is finally released, we will get a clearer picture of which devices will remain in the marketplace, and which will end up like laser discs and cassette tapes. Until then, do the homework and talk to people who use the device on a daily basis. They're often the unsung heroes.