---
title: Simulating and cleaning gene expression data using RUVcorr in the context of
  inferring gene co-expression
output: rmarkdown::html_vignette
vignette: >
 %\VignetteIndexEntry{Vignette}
 %\VignetteEngine{knitr::rmarkdown}
 %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

The R package `RUVcorr` allows the simulation and cleaning of gene expression data using global removal of unwanted variation (RUV) when the aim is the inference of gene co-expression. Besides the RUV procedure, the package offers extensive plotting options related to its application and can simulate realistic gene expression data with a known gene correlation structure. Although the procedures in the `RUVcorr` package have so far only been applied to microarray gene expression data it should be feasible to apply it to RNA-seq data as well, as long as suitable read-count summaries have been generated and the coverage is sufficient, however this remains untested. 

For more information on the methodology follow this [link](https://link.springer.com/article/10.1186/s12859-015-0745-3).

Loading `RUVcorr` is achieved with the command:

```{r}
library(RUVcorr)
```

# Simulating gene expression data with a known gene correlation structure

The simulation of gene expression data relies on the linear model framework introduced by [Gagnon-Bartsch and Speed](https://academic.oup.com/biostatistics/article-abstract/13/3/539/248166). Briefly, they assume that any gene expression measurement can be expressed as a linear combination of biological signal $X\beta$, systematic noise $W\alpha$, and random noise $\epsilon$ (typically assumed to be iid normally distributed).

\begin{equation}
Y=X\beta+W\alpha+\epsilon
\end{equation}

* $Y$ is a $m \times n$ matrix of observed gene expression data
* $X$  is a $m \times p$ matrix containing the unobserved factors of interest
*	$\beta$  is a $p \times n$ matrix of regression coefficients associated with $X$
*	$W$ is a $m \times k$ matrix of unobserved covariates introducing systematic  noise
*	$\alpha$ is a $k \times n$ matrix of regression coefficients associated with $W$
* $\epsilon$ is a $m \times n$ matrix of random noise
  
In the context of this model and for the purposes of simulating gene expression data with a known gene correlation structure, the true underlying gene structure is assumed to be
$\Sigma=Cor(X\beta)$. The size of the absolute value of the correlations can be somewhat controlled using the dimensionality of $X$ and $\beta$, $p$. When $p$ is increased the size of the absolute value of the correlations in $\Sigma$ is reduced. Note that some genes (negative controls) are unaffected by this, as their correlation with each other as well as other genes is defined to be 0. Negative control genes are genes that are believed to be unrelated to the factor of interest.

## Independence of biological signal and systematic noise

The simplest simulation of gene expression data assumes that the biological signal and the systematic noise are uncorrelated with each other. So $X$ is simulated in a fashion that it renders it independent from $W$. 
After simulating the data, the `print` command allows you to get a useful overview of the simulated data as well as some meta data.

```{r}
set.seed(400)
Yind <- simulateGEdata(n=3000, m=1000, k=10, size.alpha=2,
                       corr.strength=5, g=NULL, Sigma.eps=0.1, 
                       nc=2000, ne=1000, intercept=TRUE, check=TRUE)
print(Yind)
```

Note that the parameter `corr.strength` refers to $p$. The parameters `nc` and `ne` refer to the number of negative control genes and truly expressed genes
(i.e. with a mean true gene expression greater than 0.) The parameter `intercept` controls whether $W$ contains an offset or not.

#Dependence of biological signal and systematic noise

It is more realistic to assume that there is some dependence between $X$ and $W$. Using the parameter `g` ($0<g \leq \min(k,p)$) it is possible to introduce different levels of correlation between signal and systematic noise. Choosing a larger value for `g` will introduce more dependency between $X$ and $W$. Here `g` refers to the dimension of the shared subspace of $X$ and $W$.

```{r}
set.seed(400)
Ydep <- simulateGEdata(n=3000, m=1000, k=10, size.alpha=2, 
                       corr.strength=5, g=2, Sigma.eps=0.1, 
                       nc=2000, ne=1000, intercept=TRUE, check=TRUE)
print(Ydep)
```

Note that `bWX` refers to the average correlation between the columns of $X$ and $W$.

# Application of global removal of unwanted variation

RUV is a data-driven method that removes systematic noise from gene expression datasets. The particular version of RUV is dependent on the goal of the analysis. We have developed a method, `RUVNaiveRidge`, for the removal of unwanted variation that focuses on retrieving the true underlying gene-gene correlations, but at the cost of the specification of the absolute values of gene expression.The application of `RUVNaiveRidge`
requires the analyst to make several descisions, which should be informed by the ultimate research goal. Here we will demonstrate some of the 
principles using a dataset on gene expression in 57 samples from the bladder as described in [Dyrskjot et al.](https://cancerres.aacrjournals.org/content/64/11/4040.short) The dataset can be found in the Bioconductor package `bladderbatch`. Note that this dataset is small and co-expression analysis should ideally be performed on studies with at least 100 samples.

## Investigating the dataset design and getting data into the correct format

For the application of `RUVNaiveRidge` it is important to be familar with the experiment design of the dataset. If the accompanying metadata of the samples is available
the experiment design can be visualized using the function `plotDesign`.

```{r message=FALSE, warning=FALSE, fig.height=5, fig.width=7}
library(bladderbatch)
data(bladderdata)
expr.meta <- pData(bladderEset)
plotDesign(expr.meta, c("cancer", "outcome", "batch"), 
           c("Diagnosis", "Outcome", "Batch"), orderby="batch")
```

Every line in each of the bars represents a sample, which is colored according to the factor displayed on the left-hand side. The samples
in each bar are in the same order

The figure illustrates that batches, diagnosis and eventual outcome were substantially confounded; ie. not all
factors could be fully randomized. Thus, it is likely that the data contains some systematic noise.

The gene expression data needs to be a matrix with its columns containing the genes and its rows containing the samples.

```{r message=FALSE, warning=FALSE}
expr <- exprs(bladderEset)
expr[1:5,1:5]
dim(expr)
expr <- t(expr)
expr <- expr[,1:20000]

library(hgu133a2.db)
x <- hgu133a2SYMBOL
xx <- as.list(x[colnames(expr)])
```

## Selecting negative control genes

Ideally, negative control genes should be selected with the help of **a priori** information. Unfortunately, when the aim is estimating gene coexpression and 
the factor of interest is unknown, a suitable set of negative control genes is seldomly known. Because of this it is advisable to choose negative control genes empirically. Using the `RUVcorr` package this can be accomplished using the function `empNegativeControls`. Note that it is necessary to exclude the genes that pertain to your research question from being selected as negative controls. For demonstration purposes let us assume we are interested in the following 10 random genes: 

```{r}
na_genes <- c("SCN1A", "SCN3A", "SCN4A", "SCN5A", "SCN7A", "SCN8A", "SCN11A",
            "SCN1B", "SCN2B", "SCN3B", "SCN4B")
```

Since the genes in the dataset is uing Affymetrix identifiers, we have to find the corresponding Affymetrix probe names for our genes of interest.

```{r}
na_affy <- names(which(unlist(lapply(xx, function(x) is.element(x, na_genes)[1]))))
na_index <- which(is.element(colnames(expr),na_affy))
nc_index <- empNegativeControls(expr, exclude=na_index, nc=3000)
```

Usefully, the selection can also be visualized:

```{r, fig.height=5, fig.width=5}
genePlot(expr, index=nc_index, 
         legend="Negative Control Genes", title="IQR-Mean Plot")
```

This figure shows the inter-quantile range vs. mean plot of the expression of all genes. The genes highlighted in red are the empirically chosen negative control genes.

## Effective application of `RUVNaiveRidge`

Besides negative control genes the application of `RUVNaiveRidge` also requires the input of two user-selected parameters, the ridge parameter $\nu$ and the dimensionality of $\hat{W}$, $\hat{k}$. Since these parameters determine the strength of the cleaning, the user is adviced to carefully assess her choices. It is recommended to run `RUVRidgeNaive` with several different choices of the parameters and then assess the results. In order to do this efficiently it is advisable to `RUVNaiveRidge` in parallel. This can be achieved with a package such as `snowfall`.

```{r message=FALSE, warning=FALSE}
library(snowfall)
k <- c(1,2,3,4)
nu <- c(0,500,1000,5000)
k.nu.matrix <- cbind(rep(k, each=4), rep(nu, 4))
k.nu.matrix <- as.list(as.data.frame(t(k.nu.matrix)))

sfInit(parallel=TRUE, cpus=4)
sfLibrary(RUVcorr)
sfExport("expr", "k.nu.matrix", "nc_index")
expr_AllRUV <- sfLapply(k.nu.matrix, function(x)
  RUVNaiveRidge(expr, center=TRUE, nc_index, x[2], x[1]))
sfStop()
```

## Plotting options to help make parameter choices
Choosing the parameter values is not always easy and there might be more than one possible choice. It is therefore vital to thoroughly investigate different combinations of parameter choices using genes that are **a prior**} known to be uncorrelated with each other and **a priori** known to be correlated, also referred to as positive controls. Here, we will use the sodium channel genes as positive controls, because we expect some of these genes to be correlated with each other.

```{r warning=FALSE, fig.height=7, fig.width=7}
cor_AllRUV_na <- lapply(expr_AllRUV, function(x) cor(x[,na_index]))
cor_Raw_na <- cor(expr[,na_index])
                      
lapply(1:4, function(i) histogramPlot(cor_AllRUV_na[seq(0,15,4)+i], cor_Raw_na,
  title=paste("nu=", nu[i]), legend=c(paste("k=", k), "Raw")))
```

For the set of uncorrelated genes, the negative control genes cannot be used. This is because negative controls used during RUV will have zero correlation by definition. A good choice for a set of uncorrelated genes is a set of random genes. Picking these can be accomplished using the function `background`.  

```{r message=FALSE, warning=FALSE, fig.height=7, fig.width=7}
bg_index <- background(expr, nBG=100, exclude=na_index, nc_index=nc_index)

cor_AllRUV_bg <- lapply(expr_AllRUV, function(x) cor(x[,bg_index]))
cor_Raw_bg <- cor(expr[,bg_index])
                      
lapply(1:4, function(i) histogramPlot(cor_AllRUV_bg[seq(0,15,4)+i], cor_Raw_bg,
  title=paste("nu=", nu[i]), legend=c(paste("k=", k), "Raw")))
```

The figures show the impact of different parameter choices on the correlations of a set of sodium channel genes and random genes.
Correlation densities for different parameter choices. The histogram in the background of each panel shows the denisty of the correlations of the random genes calculated using the raw data.

From the figures it seems a choice of $\hat{k}=2$ corrects the 
wide range of the distribution of the correlations between random genes, but leaves some interesting non-zero correlations for the sodium-channel genes. Other plots that are informative for the choise of $k$ include the `eigenvaluePlot`. The choice for the correct $\nu$ however remains difficult because of the little change in the overall results. Further assessments are required. 

Besides looking at histogram plots studying relative log expression (RLE) plots is useful. Specifically, parameter choices that overcorrect the data can be spotted. Such parameter choices will have gene expression variances that are too low. The RLE plots offered differ from the originally proposed RLE plot by combining all samples and are suited to large ($>100$ arrays) gene expression data sets where visualisation of individual arrays becomes impractical. The option displayed here shows the boxplots for the 1st and 3rd quantile of the difference between the gene expression and the study median for all samples. 

```{r message=FALSE, warning=FALSE, fig.height=7, fig.width=7}
lapply(1:4, function(i) RLEPlot(expr, expr_AllRUV[[4+i]], 
  name=c("Raw", "RUV"), title=paste("nu=", nu[i]),
  method="IQR.boxplots"))
```

The figure shows RLE plots comparing different options of $\nu$ for $\hat{k}=3$.
The boxplots summarize the 25\% and 75\% quantile of all samples. The red boxplots display the raw data, while the black boxplots refer to the RUV applied with $\hat{k}=2$ and $\nu$ as in the title of the panel.

A parameter choice of $\nu=500$ seems to offer the best choice. In order to check whether the selected parameter at least removes all the known sources of variation, there is yet another version of the RLE plot. Here we plot the median and the inter-quantile-range (IQR) of the difference between the gene expression and the study median for all samples. Furthermore, it is useful to color these plots according to a known source of unwanted variation, such as batches.

```{r message=FALSE, warning=FALSE, include=FALSE, fig.height=5, fig.width=5}
par(mfrow=c(1,1))
RLEPlot(expr, expr_AllRUV[[6]], name=c("Raw", "RUV"), 
        title="Batches", method="IQR.points", anno=expr.meta, 
        Factor="batch", numeric=TRUE)
```

The fugure shows RLE plots for data cleaned with RUV using $\nu=500$ for $\hat{k}=2$.
Every sample is represented by the median and inter-quantile-range of the difference between observed gene expressions and study mean. The samples are colored according to their batches.

Principal component plots (`PCAPlot`) provide a similar way of assessing parameter choices for RUV.  

The figure demonstrates that at least most of the systematic noise introduced via the batch effect has been removed. Hence, it is now possible to examine gene-gene correlations, construct gene networks or else using this new dataset. 

```{r}
CleanData <- expr_AllRUV[[6]]
```


# Gene prioritisation

One of the methods that can be applied given a cleaned version of the dataset is gene prioritisation. Gene prioritisation identifies candidate genes that are likely to be involved in the same biological pathways or related pathways than a set of known genes. The gene prioritisation method in this package is very similar to the approach described in the paper by [Oliver et al.](https://ng.neurology.org/content/2/1/e51.short) For demonstration purposes assume that the following genes involved in the synaptic vesicle cycle are in fact candidates: 

```{r}
cand_genes <- c("CACNA1A", "CACNA1B", "SNAP25", "STX1A")
cand_affy <- names(which(unlist(lapply(xx, function(x) is.element(x, cand_genes)[1]))))
cand_index <- which(is.element(colnames(CleanData),cand_affy))
```

## Finding the correlation threshold of significant co-expression
In order to prioritise genes, typically a correlation threshold is determined. The absolute values of correlations between genes that exceed this threshold are considered to be truly co-expressed. Here, we use a threshold that corresponds to a proportion of prioritised random genes of 0.3. However, this requires extensive estimation for all possible thresholds. This can be achieved using the function `calculateThreshold`: 

```{r fig.height=5, fig.width=5}
Prop <- calculateThreshold(CleanData, exclude=c(nc_index, cand_index), 
                      index.ref=na_index, set.size=length(cand_index), 
                      Weights=NULL)
threshold <- predict(Prop$loess.estimate, 0.3)
threshold
```

It is important to exclude genes that could bias the estimation of the proportion of prioritised genes.

## Prioritising candidate genes
Having determined the threshold we can use the function `prioritise` in order to establish which candidates are also likely to be involved in the sodium-channel:

```{r}
prior<-prioritise(CleanData, na_index, cand_index, Weight=NULL, threshold=threshold)
print(prior)
xx[which(is.element(names(xx), prior[,1]))]
```

This analysis prioritises SNAP25 and CACNA1A.
