11 Detection limit of assays

When measuring the quantitative traits in genetic studies, sometimes analytes at very low level can not be detected by the assay. The values of these samples are often reported as ‘non-detects’ or at the value of detection limit.

11.1 Upper and lower detection limit

Detection limit is defined as the concentration of the analyte, which gives a detectable signal above the analytical noise - the signal produced in the absence of the analyte . E.g, the minimum detectable concentration for lipoprotein A (LPA) level in the plasma is 2mg/dL using BN ProSpec assay. Samples with LPA level below or equals to 2 mg/dL were reported as 2mg/dL. The over representation of LPA level at 2mg/dL distorts the normality of the LPA distribution (histogram)

11.2 Violation in distribution assumption and bias in effect estimate

Let represent the latent trait; represent the observed trait distribution, at the presence of detection limit, , as in follow:

\[ y'= \left\{\begin{array}{ll} y & \textrm{if y > $d$ }\\ d & \textrm{if y $\leq$ $d$}\\ \end{array}\right. \]

Distribution of background noise and compond

Distribution of background noise and compond

\[y= G \beta + u\]

\[ \begin{array}{r} E(y|G,y>d)= G\beta+ E(u|G,y>d)\\ = G\beta + E(u|u> d -G\beta) \\ \end{array} \]

Bias in effect estimate with the presence of detection limit

Bias in effect estimate with the presence of detection limit

\[ \begin{array}{l} y= G \beta + u,u \sim N(0,\sigma ^2)\quad and\quad E(u|G)=0\\ E(y|G,y>d)= G\beta+ E(u|G,y>d)\\ = G\beta + E(u|u> d -G\beta) \\ =G\beta+\sigma \frac{\phi(\frac{d-G\beta}{\sigma})}{1-\Phi\frac{d-G\beta}{\sigma}} ^{[2]} \\ \end{array} \]

2. Heckman Annals of Economic and Social Measurement, Volume 5, number 4. NBER; 1976: 475-492.

11.3 Methods for detection limit problem

Existing methods for handling detection limit for single variable.

11.3.1 Substitution

Replace censored value with a fixed value such as

\(d, d/2, d/\sqrt{2}, 0\)

11.3.2 KM

Kaplan-Meier (K-M)

11.3.3 ROS

Regression on ordered statistics (ROS)

11.3.4 MLE

Maximum likelihood Estimate (MLE)

\[ L(\beta, \sigma) = \prod_{y \leq d}1- \Phi(\frac{d-G\beta}{\sigma})\prod_{y>d}\frac{1}{\sigma}\phi(\frac{y-G\beta}{\sigma}) \]

Where \(\phi\) is probability density distribution, \(\Phi\) is culmulative density distribution.

\[ \begin{array}{l} E(y|G)= E(y|G,y>d)Pr(y>d) +E(y|G,y\leq d)Pr(y\leq d)\\ \\ = G\beta+ \sigma \frac{\phi(\frac{d-G\beta}{\sigma})}{1-\Phi(\frac{d-G\beta}{\sigma})}+ d \frac{\phi(\frac{d-G\beta}{\sigma})}{1-\Phi(\frac{d-G\beta}{\sigma})} \end{array} \]

11.4 Comparisons of the methods