Are the Roche and Abbott Antibody Tests Really More Accurate Than Other Tests? PHE Say ‘No’
/As a director of medical companies for the past 15 years I have spent a lot of my time trying to evaluate how good a certain product really is. I would estimate that on average I would receive proposals to bring upwards of 50 products per year to market and typically we select only a handful of those. Extrapolating; that would mean that I have evaluated around 750 products. Over the years I have developed a method because if you just believe the manufacturer, you won’t get very far. Often a cursory search of the literature shows you there is little to no plausibility to a product’s claims. So that is my step one: what is the company claiming, are those claims plausible and if so, what evidence can they supply in support of those claims?
Step two is a search of the literature to try to find anything that contradicts what has been supplied by the company. If so, then I go back to the company and see what they have to say about it. And the process goes on… but when looking at antibody tests for SARS-COV-2 this probably far enough for now. So; are they more accurate? Hold on! Let’s cover some basics first…
What Does Accuracy Mean in Relation to Medical Tests?
Experts in medical testing tend not to talk about accuracy of tests. What they mostly talk about is specificity and sensitivity. These are two important concepts and should be reviewed together when looking at how good a test is.
Specificity
Specificity describes a test’s ability to generate a negative result for someone who doesn’t have the illness that is being tested for. In this case, a test with a specificity of 90% would generate a negative result 9 out of 10 times the test was run on a patient that doesn’t have the illness. These results are sometimes called “true-negatives”. Following on from this; it would give a false positive 1 out of every 10 times.
A test with 100% specificity would return a “true-negative” result every time a patient without the illness is tested therefore it would return no false positives.
Sensitivity
Sensitivity describes how often a test returns a positive result when someone has the illness. For example, a test with a sensitivity of 90% would return positive results 9 out of 10 times when given to people who do have the illness. Therefore 1 out of 10 people would receive a false negative result.
A test with 100% sensitivity would return a “true-positive” result every time a patient with the illness is tested therefore it would return no false negatives.
“The [Roche] assay gave an overall sensitivity of 83.9%”
What Are These Tests Looking For?
All of these tests are looking for anti-SARS-COV-2 antibodies. Antibodies are large, Y shaped proteins that are generated by your body in response to a pathogen. When talking about SARS-COV-2, they are testing for Immunoglobulin M (IgM), Immunoglobulin G (IgG) or both of these antibodies.
These antibodies are important because they indicate different things. Perhaps it is important to mention at this point that antibody tests will not tell you if you are currently infected, they will only tell you if you have been infected at some point and they may give you an indication of when that infection took place.
They can do this by looking for the different antibodies. IgM antibodies are released shortly after a pathogen is detected. Typically a few days to a week after detection. Then a little later, several days to two weeks after detection of the pathogen, your body will produce large quantities of IgG. IgG is therefore often present once a person is recovering or has recovered from infection.
What this means is that a positive result for IgM but not IgG suggests a recent infection. A positive result for IgG suggests that potentially you have recovered from the infection.
With that out of the way we can assess the claims and review the evidence. What are they claiming? Are these claims plausible? Can they support their claims? Let’s take a look…
Abbott
The Abbott antibody test looks for IgG only and claims sensitivity of 100% for samples ≥ 14 days post symptom onset. A 100% sensitivity claim raises an eyebrow but I guess it is possible. When looking for the data I came across Public Health England’s (PHE) study entitled Evaluation of the Abbott SARS-CoV-2 IgG for the detection of anti-SARSCoV-2 antibodies. This independent study confirms my suspicion that the 100% sensitivity claim may be optimistic. PHE found that the Abbott “assay had an overall sensitivity of 92.7% (95%CI 85.6-97.0), with a sensitivity of 93.4% (95%CI 85.3-97.8) for samples ≥14 days post symptom onset. The sensitivity of the assay at ≥21 days post symptom onset is 93.9% (95%CI 86.3- 98.0)”*.
Abbott claimed a specificity of 99.6% but in this instance PHE found 100%* specificity in their study.
Samples must be sent to an accredited lab for processing.
Roche
The Roche antibody test looks for both IgM and IgG antibodies and like the Abbott test, requires a lab and proprietary equipment to process the samples. Like Abbott, Roche made a claim that their assay had a sensitivity of 100% for samples ≥14 days post infection. Also like Abbott, The Roche test was found by PHE in their study entitled Evaluation of Roche Elecsys AntiSARS-CoV-2 serology assay for the detection of anti-SARS-CoV-2 antibodies not to reach this standard. What they did find was that ”the assay gave an overall sensitivity of 83.9% (95%CI 74.8-90.7), with a sensitivity ≥14 days of 87.0% (95%CI 77.4-93.6). The sensitivity of the assay at ≥21 days post symptom onset is 87.7% (95%CI 77.9-94.2).
PHE found that the specificity was 100% as opposed to Roche’s claim of 99.8% specificity. I think we can forgive that.
What Does This All Mean?
In order to really understand what this means in terms of accuracy we need to include one more factor. The underlying rate of infection. With the addition of this third variable (in addition to the specificity and sensitivity of the test) we can begin to build a picture of how successful a test may be. Let’s do a couple of examples using the ‘data’ going around over the weekend that suggested that 17% of Londoners and 5% of UK citizens had been infected by SARS-COV-2. To make things a little easier for my feeble brain let’s assume there are 10,000,000 people in London.
If we had a test that was 100% sensitive and 100% specific and we tested all 10,000,000 people in London at the right time and the infection rate in the population is 17%, we would expect to see 1,700,000 true-positive results and 8,300,000 true-negative results. If however the infection rate in the population is more like the suspected UK rate of 5%, we would expect to see 500,000 true-positives and 9,500,000 true negatives. Are you with me?
Using the same ‘data’ but applying the best possible sensitivity of the Abbott test, at an infection rate of 17% we would see 1,596,300 true-positives and 103,700 false-negatives along with 8,300,000 true-negatives in London.
If the infection rate is 5% then we would see 469,500 true-positives, 30,500 false-negatives and 9,500,000 true-negatives.
In the same way; with the Roche test we would see 1,490,900 true-positives, 209,100 false-negatives and 8,300,000 true-negatives. At an underlaying rate of 5% these would drop to 438,500 true-positives, 61,500 false-negatives and 9,500,000 true negatives.
What this shows us is that knowing the infection rate in the population is critical to understanding how ‘accurate’ a test is. If we go back to the abbott example, reducing the infection rate in the population from 17% to 5% (a reduction of 3.4 times), the number of false-negatives drops by 73,200 which is a proportionate reduction.
Perhaps more importantly; a drop in sensitivity from 93.9% to 87.7% means a rise of 31,000 false-negatives. A roughly 6% worse specificity would lead to over double the number of false-negative results.
When we talk about antibody tests the number of false negatives shouldn’t be much of a concern in my opinion. The reasoning behind this is that antibody tests are typically used when someone is asymptomatic and are most effective from at least 7 days and possibly up to 21 days after infection which is often once you have recovered from the virus so hopefully you are not contagious. What they are very good for is getting a picture of how the virus has spread and what the prevalence has been in the population. It would be different if we were looking at PCR tests or antigen tests that look for active viruses. In that case, false negatives can be very worrying. Assuming the Abbott data from above held for PCR tests (an assumption I wouldn’t like to make but without the data one I choose to make for the purpose of this illustration) and a 5% underlying infection rate, there would be 30,500 infected and contagious people walking around believing they were ‘safe’ and not isolating.
In short, it means two things;
It is critical to know what the underlying rate of infection is in the population if we are going to have a discussion about test accuracy. Both for Antigen and antibody tests.
Just because the government, PHE or the NHS are endorsing a test doesn’t mean that test is 100% ‘accurate’.
In Summary
I have not personally evaluated the Abbott or Roche antibody tests but I have investigated numerous colloidal gold lateral flow antibody tests over the past few months which are much cheaper and easier to use as they don’t require specialist equipment or lab support. None have claimed sensitivity of 100%. They have all been in the region of 94-96% and I feel this is probably pretty accurate. They all have specificity of 100% so we shouldn’t be seeing false-positives from any of them. From my research I don’t see any evidence that these more expensive and time consuming tests from Abbott and Roche offer a significant benefit over the much cheaper and easier to use lateral flow antibody tests.
PCR tests are still the go-to for symptomatic patients but antibody tests will play a huge part in our fight against COVID-19. Lateral flow methods provide a quick and easy point of care option with comparable specificity and sensitivity to the more complex tests offered by Roche and Abbott. They are also more cost effective and provide results in 10 minutes making them ideal for use in clinics and businesses where relatively close contact is required such as barbers, hairdressers and in some office/retail settings. I think we’ll be seeing a lot more of them in the near future.
Note:
Thanks to Edward Pearce for pointing out a shocking mistake in my maths. It’s back to school for me!