---
title: "Pulling Hi-C data with mariner"
author: "Eric Davis"
date: '`r Sys.Date()`'
output: rmarkdown::html_document
toc_depth: 3
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

`mariner` offers 2 functions for extracting/pulling
interactions from `.hic` files - `pullHicPixels()` and
`pullHicMatrices()`. In this article, you will learn how
to extract Hi-C pixels or count matrices and access them
from the resulting objects.

`pullHicPixels()` and `pullHicMatrices()` accept the same
set of arguments, but return different outputs. Which
function you choose depends on whether you want to extract
a single value for each interaction or a matrix of values
between a range of interactions.

# When to use `pullHicPixels()`

If you want a single value for each interaction and `.hic`
file then you should use the `pullHicPixels()` function.
We define a "pixel" as an interaction where both anchors
are the same width. The `binSize` argument is used here
to check that your desired pixel resolution matches your
input interactions.

```{r pixelFigure, fig.align='center', out.width='50%', echo=FALSE}
knitr::include_graphics("img/pixelFigure.png")
```

<div style="
padding: .5em 1em;
margin-bottom: 10px;
border-radius:5px;
border-left: 5px solid #2780e3;
border-right: 1px solid #dee2e6;
border-top: 1px solid #dee2e6;
border-bottom: 1px solid #dee2e6;
">
<div style="font-weight: 600; opacity: 75%;">
<p style="margin:0px">Note</p>
</div>
You can check your `.hic` file to see which resolutions
are available for the `binSize` argument with the 
`strawr::readHicBpResolutions()`. See the `assignToBins()`
to set your interactions to an acceptable resolution.
</div>

The following example pulls 100-Kb pixels from two
`.hic` files:

```{r pullPixels, message=FALSE}
## Load mariner
library(mariner)

## Use example .hic files from ExperimentHub
hicFiles <- c(
    marinerData::LEUK_HEK_PJA27_inter_30.hic(),
    marinerData::LEUK_HEK_PJA30_inter_30.hic()
)
names(hicFiles) <- c("hic1", "hic2")

## Make some example interactions
gi <- read.table(
    text="
    1 51000000 51100000 1 51000000 51100000
    1 150000000	150100000 1	150000000 150100000
    2 51000000 51100000 2 51000000 51100000
    2 150000000	150100000 2	150000000 150100000
    "
)
gi <- as_ginteractions(gi)

## Pull Hi-C pixels
pixels <- pullHicPixels(x=gi, files=hicFiles, binSize=100e3)
pixels
```

This results in an `InteractionMatrix` object which
contains the extracted Hi-C data, interactions, and
metadata for the interactions and `.hic` files.

The extracted Hi-C data is stored as a count matrix
where every row is an interaction (i.e pixel) and
each column is a `.hic` file. Use the `counts()` 
function to access the count matrix from an
`InteractionMatrix` object.

```{r pullPixelsCounts}
counts(pixels)
```

This count matrix is stored on-disk (if `onDisk=TRUE`)
as an HDF5-backed `DelayedArray` object. The data is stored
in the provided file path to the `h5File` argument. If a
file path isn't provided, a temporary file is created
in the current Rsession. To access or update the location of
the HDF5 file after using the `pullHicPixels()` function,
use the `path()` function.

```{r}
path(pixels)
```

<div style="
padding: .5em 1em;
margin-bottom: 10px;
border-radius:5px;
border-left: 5px solid #2780e3;
border-right: 1px solid #dee2e6;
border-top: 1px solid #dee2e6;
border-bottom: 1px solid #dee2e6;
">
<div style="font-weight: 600; opacity: 75%;">
<p style="margin:0px">Note</p>
</div>
Using a `DelayedArray` instead of a normal R matrix
has a number of benefits, especially for users working
with limited computer memory. See the `?DelayedArray`
package documentation and vignettes for more information.
</div>

The count matrix can then be used for downstream
analysis such as differential interaction analysis
with `DESeq2`. To learn more about how to access components of
`InteractionMatrix` objects, see the
[`InteractionMatrix` class](#interaction_matrix_class)
section below.

# When to use `pullHicMatrices()`

If you want a matrix of values for each interaction then
you should use the `pullHicMatrices()` function. These
matrices are made up of multiple "pixels", defined by
dividing the range of each interaction by the supplied
`binSize` parameter.

```{r matricesFigure, fig.align='center', out.width='50%', echo=FALSE}
knitr::include_graphics("img/matricesFigure.png")
```

## Square matrices

For example, if you define 500x500 Kb interactions by setting
the width of both anchors to be 500 Kb

```{r}
## Create 500x500 Kb regions
regions <- assignToBins(x=gi, binSize=500e3, pos1="start", pos2="start")
```

Then set the `binSize` to 100 Kb

```{r}
## Pull Hi-C matrices
matrices <- pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
matrices
```

It produces count matrices each with 5 rows and 5 columns.
These count matrices are stored in the `InteractionArray`
class and are accessible with the `counts()` function.

```{r}
counts(matrices)
```

You can supply `showDimnames=TRUE` to display the start
bin of each anchor.

```{r}
counts(matrices, showDimnames=TRUE)
```

These 4-dimensional arrays use the first and second
dimensions as the rows and columns of the count matrices,
the third dimension for each interaction, and the 
fourth dimension for each `.hic` file.

If you want to convert pixels to square Hi-C regions,
you can use the `pixelsToMatrices()` function. The
`buffer` argument describes how many pixels to
expand the ranges on either side of the center pixel.
For example, to expand 100x100 Kb pixels to regions
that are 500x500 Kb, specify `buffer=2` to add two
additional 100 Kb pixels on both sides of the central
100 Kb pixels.

```{r pixelsToMatricesFigure, fig.align='center', out.width='75%', echo=FALSE}
# knitr::include_graphics("img/pixelsToMatricesFigure.png")
```

```{r}
regions <- pixelsToMatrices(x=gi, buffer=2)
IRanges::width(regions)
```

When pulled with `pullHicMatrices()` using `binSize=100e3`
odd, 5x5 matrices result.

```{r}
pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
```

## Rectangular matrices

The previous example returns square count matrices where
the width of both anchors are the same for each interaction.
`mariner` also supports rectangular count matrices where
the widths of the rows and columns are not equal.

For example, you can define 300x600 Kb interactions by setting
the width of the first anchor to be 300 Kb and the second anchor
to be 600 Kb.

```{r}
## Create 300x600 Kb regions
regions <- assignToBins(
    x=gi,
    binSize=c(300e3, 600e3),
    pos1="start",
    pos2="start"
)
```
Then set the `binSize` to 100 Kb

```{r}
## Pull Hi-C matrices
matrices <- pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
matrices
```

Which produces an `InteractionArray` object with
count matrices each with 3 rows and 6 columns.

```{r}
counts(matrices, showDimnames=TRUE)
```

Extracting square and rectangular matrices of data
results in `InteractionArray` objects. See
[`InteractionArray` class](#interaction_array_class)
to learn more about accessing data from these objects.

## Variable matrices

`mariner` also supports extracting count matrices
that are different dimensions for each interaction.

For example, these three interactions have dimensions
of 1x3, 3x3, and 3x2 after pulling matrices with a
`binSize` of 100e3:

```{r}
## Interactions of different dimensions
regions <- read.table(
    text="
    1 51000000 51100000 1 51000000 51300000
    1 150000000 150300000 1 150000000 150300000
    2 51000000 51300000 2 51000000 51200000
    "
)
regions <- as_ginteractions(regions)

## Pull Hi-C matrices
matrices <- pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
matrices
```

The resulting object is of class
`InteractionJaggedArray`. The class is different
than the previous examples because the classes
that `InteractionArray` inherits from are designed
for regular, rectangular data types. A custom class
called `JaggedArray` is used to hold irregular, or jagged,
arrays of data.

The same `counts()` function returns these
`JaggedArray` objects containing the Hi-C count data
for each interaction and `.hic` file.

```{r}
counts(matrices)
```

See the 
[`InteractionJaggedArray` class](#interaction_jagged_array_class)
section to learn more about accessing
and transforming data from `InteractionJaggedArray`
and `JaggedArray` objects.

# `InteractionMatrix` class <a id="interaction_matrix_class" />

The `InteractionMatrix` class extends
the `InteractionSet` and `SummarizedExperiment`
classes. Therefore, it also inherits the accessors
and methods of these objects. For example, you
can access the original interactions, metadata about
the experiment, row or column metadata, and subset
or index slices of these objects. The following
sections highlight some of the most useful accessors
and methods for `InteractionMatrix` using this example
object:

```{r, message=FALSE}
## Load mariner
library(mariner)

## Use example .hic files from ExperimentHub
hicFiles <- c(
    marinerData::LEUK_HEK_PJA27_inter_30.hic(),
    marinerData::LEUK_HEK_PJA30_inter_30.hic()
)
names(hicFiles) <- c("hic1", "hic2")

## Make some example interactions
gi <- read.table(
    text="
    1 51000000 51100000 1 51000000 51100000
    1 150000000	150100000 1	150000000 150100000
    2 51000000 51100000 2 51000000 51100000
    2 150000000	150100000 2	150000000 150100000
    "
)
gi <- as_ginteractions(gi)

## InteractionMatrix
imat <- pullHicPixels(x=gi, files=hicFiles, binSize=100e3)
```
## Common accessors

```{r InteractionMatrixAccessors}
## Show method
imat

## Dimensions
dim(imat)

## Metadata about Hi-C extraction
metadata(imat)

## Metadata about interactions
SummarizedExperiment::rowData(imat)

## Metadata about columns
SummarizedExperiment::colData(imat)

## Interactions
interactions(imat)

## Count matrices
counts(imat)
```

## Indexing and subsetting

You can subset the interactions and files of
the object directly where the first position
subsets interactions and the second subsets
files.

```{r}
imat[1:3] |> counts()
imat[3:1] |> counts()
imat[,2] |> counts()
imat[1,1] |> counts()
```

## Concatenating

Use the `rbind()` or `cbind()` functions to
combine interactions row-wise or column-wise,
respectively.

```{r}
cbind(imat[,1], imat[,2])
rbind(imat[1:2,], imat[3:4,])
```

Note that the column metadata must be the same
when using `rbind` and the row interactions
must be the same when using `cbind`.

```{r, error=TRUE}
rbind(imat[1:2,1], imat[3:4,2])
cbind(imat[1:2,1], imat[3:4,2])
```

## Overlapping

Methods for `subsetByOverlaps()`, `findOverlaps()`,
`countOverlaps()`, and `overlapsAny()` are inherited
from the `InteractionSet` and `IRanges` packages.
See the documentation and vignettes of these packages
for usage and behavior of these functions.


# `InteractionArray` class <a id="interaction_array_class" />

The `InteractionArray` class extends
the `InteractionSet` and `SummarizedExperiment`
classes. Therefore, it also inherits the accessors
and methods of these objects. For example, you
can access the original interactions, metadata about
the experiment, row or column metadata, and subset
or index slices of these objects. Unlike the
the `InteractionMatrix` class which returns an 
"interaction-by-Hi-C" matrix, the `InteractionArray`
class returns count matrices for each interaction and
`.hic` file. This results in a 4-dimensional array,
where the first two dimensions are the rows and columns
of the count matrices, the third dimension is the
supplied interactions, and the fourth dimension is the
supplied `.hic` files. This is stored as a `DelayedArray`
which is accessible via the `counts()` accessor. The following
sections highlight some of the most useful accessors
and methods for `InteractionMatrix` using this example
object:

```{r, message=FALSE}
## Load mariner
library(mariner)

## Use example .hic files from ExperimentHub
hicFiles <- c(
    marinerData::LEUK_HEK_PJA27_inter_30.hic(),
    marinerData::LEUK_HEK_PJA30_inter_30.hic()
)
names(hicFiles) <- c("hic1", "hic2")

## Create 500x500 Kb regions
regions <- read.table(
    text="
    1 51000000 51500000 1 51000000 51500000
    1 150000000	150500000 1	150000000 150500000
    2 51000000 51500000 2 51000000 51500000
    2 150000000	150500000 2	150000000 150500000
    "
)
regions <- as_ginteractions(regions)

## InteractionArray
ia <- pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
```

## Common accessors

```{r}
## Show method
ia

## Dimensions
dim(ia)

## Metadata about Hi-C extraction
metadata(ia)

## Metadata about interactions
SummarizedExperiment::rowData(ia)

## Metadata about columns
SummarizedExperiment::colData(ia)

## Interactions
interactions(ia)

## Count matrices
counts(ia)

## Counts with start bins for anchor1 and 2
counts(ia, showDimnames=TRUE)
```

## Indexing and subsetting

You can subset the interactions and files of
the object directly where the first position
subsets interactions and the second subsets
files.

```{r}
ia[1:3] |> counts()
ia[3:1] |> counts()
ia[,2] |> counts()
ia[1,1] |> counts()
```

Subsetting and indexing can also be done on the
`DelayedArray` object accessed with `counts()`.
For example, the following code shows how you could
access the top left corner of the count matrix
of the first interaction and `.hic` file.

```{r}
counts(ia)[1:3, 1:3, 1, 1]
```

## Concatenating

Use the `rbind()` or `cbind()` functions to
combine interactions row-wise or column-wise,
respectively.

```{r}
cbind(ia[,1], ia[,2])
rbind(ia[1:2,], ia[3:4,])
```

Note that the column metadata must be the same
when using `rbind` and the row interactions
must be the same when using `cbind`.

```{r, error=TRUE}
rbind(ia[1:2,1], ia[3:4,2])
cbind(ia[1:2,1], ia[3:4,2])
```

## Overlapping

Methods for `subsetByOverlaps()`, `findOverlaps()`,
`countOverlaps()`, and `overlapsAny()` are inherited
from the `InteractionSet` and `IRanges` packages.
See the documentation and vignettes of these packages
for usage and behavior of these functions.


# `InteractionJaggedArray` class <a id="interaction_jagged_array_class" />

The `InteractionJaggedArray` class is a new class
designed for holding irregularly sized count matrices,
also known as "ragged" or "jagged" arrays along with
other important information about the extracted Hi-C
data. The count matrices are managed by the `JaggedArray`
class which is analogous to the `DelayedArray` class.
`JaggedArray` data is stored on-disk in an HDF5 file,
and subsetting/indexing operations are delayed similarly
to `DelayedArray`. Continue reading to learn about
how to use `InteractionJaggedArray` and `JaggedArray`
objects, and how to convert them to `InteractionArray`
or `DelayedArray` objects with the `regularize()` 
function for downstream analysis.

```{r, message=FALSE}
## Load mariner
library(mariner)

## Use example .hic files from ExperimentHub
hicFiles <- c(
    marinerData::LEUK_HEK_PJA27_inter_30.hic(),
    marinerData::LEUK_HEK_PJA30_inter_30.hic()
)
names(hicFiles) <- c("hic1", "hic2")

## Create regions of different dimensions
regions <- read.table(
    text="
    1 51000000 51100000 1 51000000 51300000
    1 150000000 150300000 1 150000000 150300000
    2 51000000 51300000 2 51000000 51200000
    "
)
regions <- as_ginteractions(regions)

## InteractionJaggedArray
ija <- pullHicMatrices(x=regions, files=hicFiles, binSize=100e3)
```

## Common accessors

```{r}
## Show method
ija
```

The variable dimensions of the jagged arrays can be
accessed with the `dim()` function:

```{r}
## Dimensions
dim(ija)
```
```{r}
## Metadata about Hi-C extraction
metadata(ija)

## Metadata about columns
SummarizedExperiment::colData(ija)

## Access HDF5 filepath
path(ija)

## Interactions
interactions(ija)
```

The `counts()` function returns a `JaggedArray`

```{r}

## Count matrices as JaggedArray
ja <- counts(ija)
ja
```


This can be converted to a nested R list with
the `as.list()` function, where the outer level
refers to the `.hic` file and the inner level
contains a matrix for each interaction.

```{r}
as.list(ja)
```


## Indexing and subsetting

You can subset the interactions and files of
the object directly where the first position
subsets interactions and the second subsets
files.

```{r}
ija[1:2] |> counts()
ija[2:1] |> counts()
ija[,2] |> counts()
ija[1,1] |> counts()
```

Notice that when indexing results in subset of data
with the same dimensions it is automatically coerced
to an `InteractionArray` or `DelayedArray`.

```{r}
ija[3,1] |> class()
ija[3,1] |> counts() |> class()
```

You can also subset and index on the `JaggedArray`
object by selecting the desired interactions(s) in
the third dimension and `.hic` file(s) in the
fourth dimension. Since the first two dimensions are
variable, subsetting/indexing these is not supported.

```{r}
ja[,,3,2]
```

If the selection results in a `DelayedArray` then
normal indexing rules apply:

```{r}
ja[,,3,2][1:2,1,1,1]
```

## Concatenating

Concatenating `InteractionJaggedArray` and `JaggedArray`
objects is not currently supported. If you need this 
functionality please post an [issue](https://github.com/EricSDavis/mariner/issues) 
to the mariner repository or (even better) submit a pull
request with an implementation.

## Overlapping

Methods for `subsetByOverlaps()`, `findOverlaps()`,
`countOverlaps()`, and `overlapsAny()` are inherited
from the `InteractionSet` and `IRanges` packages.
See the documentation and vignettes of these packages
for usage and behavior of these functions.

## Converting to regular arrays

For downstream aggregation and visualization, it is
helpful to convert irregular/jagged arrays to regular
arrays. The `regularize()` function stretches and
(optionally) scales the count matrices into the desired
rectangular (or square) matrices with dimensions supplied
to `ndim`. The function works on both `InteractionJaggedArray`
and `JaggedArray` objects.

```{r}
regularize(x=ija, ndim=c(3,3)) |> counts()
regularize(x=ja, ndim=c(3,3))
```

By using `regularize()` before extracting counts, you
can take advantage of the block and parallel processing
functionality from aggregating with `aggHicMatrices()`:

```{r}
regularize(x=ija, ndim=c(3,3), nBlocks=1) |>
    aggHicMatrices(nBlocks=1)
```

# Other pullHic* arguments

The previous sections provided examples of
using `pullHicPixels()` and `pullHicMatrices()`, 
for extracting Hi-C pixels and count matrices,
respectively. These functions require three
mandatory arguments: `x` for interactions, 
`files` for `.hic` file paths, and `binSize` for
the Hi-C resolution.

Additionally, the remaining arguments allow for 
greater control over the Hi-C data extraction process.
These arguments control the type of Hi-C data to be
extracted, the efficiency of extraction, and specify
the storage location for the extracted data.

In this section we delve deeper into the utilization
of these arguments to effectively extract Hi-C data.

## Normalization and matrix type

The `norm` and `matrix` arguments are passed to
`strawr` during the extraction step.

`norm` refers to the Hi-C normalization and is 
limited to pre-computed normalizations in the
`.hic` file. Use the `readHicNormTypes()` function
from the `strawr` package to see available normalizations.

```{r, error=TRUE}
pullHicPixels(x=gi, files=hicFiles, binSize=100e3, norm="none")
```

The `matrix` argument specifies the type of values
to extract. The default is the "observed" contact
frequency values. Other options include "expected"
contact frequency and the "oe" or observed/expected
values. For an in-depth description of these
matrix types see 
[Durand et al. 2016](https://doi.org/10.1016/j.cels.2016.07.002)

```{r}
pullHicPixels(x=gi, files=hicFiles, binSize=100e3, matrix="oe") |>
    counts()
```

## The `half` parameter

The Hi-C matrix is a square, symmetric matrix that captures
the frequency of all pair-wise interactions between
genomic loci. In other words, each point in a Hi-C 
matrix represents the interaction frequency between
a two pairs, or anchors, of genomic regions. 

In `mariner`, the Hi-C matrix is oriented with the
origin (lowest genomic coordinates) in the upper
left corner with linear genomic position increasing
down and to the right.

```{r origin, fig.align='center', out.width='50%', echo=FALSE}
knitr::include_graphics("img/origin.png")
```

The first anchor, or genomic locus, corresponds
on the vertical axis (rows of matrix) while the
second anchor corresponds to the horizontal axis
(columns of the matrix). Together, these anchors
specify a two-dimensional region on the Hi-C matrix.

```{r anchors, fig.align='center', out.width='50%', echo=FALSE}
knitr::include_graphics("img/anchors.png")
```

The relative positions of these anchors determines
from which "half" of the matrix data is extracted. When
the genomic position of the first anchor is less than
the second anchor the region resides in the upper
trianglar.

```{r upper, fig.align='center', out.width='75%', echo=FALSE}
knitr::include_graphics("img/upper.png")
```

When the genomic position of the first anchor is
greater than the second anchor the region resides
in the lower triangular.

```{r lower, fig.align='center', out.width='75%', echo=FALSE}
knitr::include_graphics("img/lower.png")
```

If the first and second anchors are equal then
the region is on the diagonal and the values
will be mirrored across the diagonal.

```{r diagonal, fig.align='center', out.width='50%', echo=FALSE}
knitr::include_graphics("img/diagonal.png")
```

To summarize:

* first < second - upper triangular
* first > second - lower triangular
* first == second - mirrored diagonal

The `half` parameter controls which portion of
the Hi-C matrix is returned, regardless of
anchor position. When `half="upper"` only
upper-triangular data is returned. When
`half="lower"` only lower-triangular data
is returned. `half="both"` returns upper and
lower triangular data. `NA` values are returned
for portions of the matrix that do not match
the region of the matrix specified with `half`.
The figure and code below show how data are
returned for a region that resides on the
diagonal, crossing into both the upper and
lower triangular of the Hi-C matrix.

```{r diagonalHalf, fig.align='center', out.width='75%', echo=FALSE}
knitr::include_graphics("img/diagonalHalf.png")
```

```{r diagonalCode, message=FALSE}
## Hic file
hic <- marinerData::LEUK_HEK_PJA27_inter_30.hic()

## On-diagonal interaction
diagonal <- as_ginteractions(read.table(text="
    1 51000000 51300000 1 51000000 51300000
"))

## half="upper"
pullHicMatrices(x=diagonal, files=hic, binSize=100e3, half="upper") |>
    counts()

## half="lower"
pullHicMatrices(x=diagonal, files=hic, binSize=100e3, half="lower") |>
    counts()

## half="both"
pullHicMatrices(x=diagonal, files=hic, binSize=100e3, half="both") |>
    counts()
```

Since chromosomes have no inherent order in linear
genomic space, the `half` parameter is ignored for
interchromosomal interactions. `pullHicPixels()` behaves
the same way with the `half` parameter.

## Changing `blockSize` for large data

`pullHicPixels()` and `pullHicMatrices()` allow you to tune
the efficiency and memory-usage of Hi-C data extraction by 
grouping nearby interactions into blocks. The block data is
then loaded into memory, assigned to interactions, and
then written to back to disk as an HDF5 file. You can modify
the `blockSize` (in base-pairs) to control the size and number
of blocks that are extracted. Pulling larger blocks
decreases run-time, but requires more RAM to store these
data while smaller blocks increase run-time, but allows
you to extract counts from large Hi-C files that may
not otherwise fit into memory.

Adjustments to `blockSize` can optimize the efficiency of
extraction. There is a trade-off between the size of
the block and the number of iterations required to extract
all interactions. The size and placement of interactions on
the genome also affects the `blockSize`. For example, a
worst-case performance scenario would be extracting each
interaction individually (setting `blockSize` to the width
of the interaction). Generally, it is most efficient to
extract the largest blocks possible. When your data is
small enough, pulling whole chromosomes is usually most
efficient. However, the optimal `blockSize` is highly
dependent on both the interactions and `.hic` files, so
you may want to experiment with this parameter to find
the best trade-off for your data.

The `compressionLevel` and `chunkSize` parameters control
the way extracted data is written back to the HDF5-file.
See `?pullHicMatrices` or `?pullHicPixels` for a more
in-depth description of these parameters.
