Package: MsBackendSql
Authors: Johannes Rainer [aut, cre] (ORCID:
https://orcid.org/0000-0002-6977-7147),
Chong Tang [ctb],
Laurent Gatto [ctb] (ORCID: https://orcid.org/0000-0002-1520-2268)
Compiled: Tue Nov 25 17:04:24 2025
The Spectra Bioconductor package provides a flexible and
expandable infrastructure for Mass Spectrometry (MS) data. The package supports
interchangeable use of different backends that provide additional file support
or different ways to store and represent MS data. The
MsBackendSql package provides backends to store data from whole
MS experiments in SQL databases. The data in such databases can be easily (and
efficiently) accessed using Spectra objects that use the MsBackendSql class
as an interface to the data in the database. Such Spectra objects have a
minimal memory footprint and hence allow analysis of very large data sets even
on computers with limited hardware capabilities. For certain operations, the
performance of this data representation is superior to that of other low-memory
(on-disk) data representations such as Spectra’s MsBackendMzR backend.
Finally, the MsBackendSql supports also remote data access to e.g. a central
database server hosting several large MS data sets.
The package can be installed with the BiocManager package. To install
BiocManager use install.packages("BiocManager") and, after that,
BiocManager::install("MsBackendSql") to install this package.
MsBackendSql SQL databasesMsBackendSql SQL databases can be created either by importing (raw) MS data
from MS data files using the createMsBackendSqlDatabase() or using the
backendInitialize() function by providing in addition to the database
connection also the full MS data to import as a DataFrame. In the first
example we use the createMsBackendSqlDatabase() function to import the full MS
data from the provided MS data files into an (empty) database. Below we first
create an empty SQLite database (in a temporary file) and use the
createMsBackendSqlDatabase() function to create all necessary tables in that
database and import the MS data from two mzML files (from the r Biocpkg("msdata") package).
library(RSQLite)
dbfile <- tempfile()
con <- dbConnect(SQLite(), dbfile)
library(Spectra)
library(MsBackendSql)
fls <- dir(system.file("sciex", package = "msdata"), full.names = TRUE)
createMsBackendSqlDatabase(con, fls)
dbDisconnect(con)By default (with parameters blob = TRUE and peaksStorageMode = "blob2") the
peaks data matrix of each spectrum is stored as a BLOB data type into the
database (one entry per spectrum). This has advantages on the performance to
extract the peaks data from the database, but does not allow to filter
individual peaks by their m/z or intensity values directly in the database. As
an alternative (using blob = FALSE) it is also possible to store the
individual m/z and intensity values in separate columns of the database
table. This long table format results however in considerably larger databases
(with potentially poorer performance). Note also that the code and backend is
optimized for MySQL/MariaDB databases by taking advantage of table partitioning
and specialized table storage options. Any other SQL database server is however
also supported (also portable, self-contained SQLite databases). In fact,
performance for MsBackendSql databases with peaks data stored as BLOB data
type is similar for SQLite and MySQL/MariaDB databases.
The MsBackendSql package provides two backends to interact with such
databases: the MsBackendSql class and the MsBackendOfflineSql class, that
inherits all properties and functions from the former, but does not store the
connection to the database within the object. The MsBackendOfflineSql object
thus supports parallel processing and allows to save/load the object (e.g. using
save and saveRDS). The MsBackendOfflineSql might therefore be used as the
preferred backend to SQL databases for most applications.
To access the data in the database we create below a Spectra object providing
the database connection information in the constructor call and specifying to
use the MsBackendOfflineSql as backend (parameter source). We stored the
data to a SQLite database, thus we provide the database name (SQLite database
file name) and the SQLite DBI driver with parameters dbname and drv. Which
parameters are required to connect to the database depends on the SQL database
and the used driver. For a MySQL/MariaDB database we would use the MariaDB()
driver and would have to provide the database name, user name, password as well
as the host name and port through which the database is accessible.
sps <- Spectra(dbname = dbfile, source = MsBackendOfflineSql(), drv = SQLite())
sps## MSn data (Spectra) with 1862 spectra in a MsBackendOfflineSql backend:
## msLevel precursorMz polarity
## <integer> <numeric> <integer>
## 1 1 NA 1
## 2 1 NA 1
## 3 1 NA 1
## 4 1 NA 1
## 5 1 NA 1
## ... ... ... ...
## 1858 1 NA 1
## 1859 1 NA 1
## 1860 1 NA 1
## 1861 1 NA 1
## 1862 1 NA 1
## ... 35 more variables/columns.
## Use 'spectraVariables' to list all of them.
## Database: /private/var/folders/r0/l4fjk6cj5xj0j3brt4bplpl40000gt/T/RtmpyBMneu/file1347d6c86ec38Spectra objects allow also to change the backend to any other backend
(extending MsBackend) using the setBackend() function. Below we use this
function to first load all data into memory by changing from the
MsBackendOfflineSql to a MsBackendMemory.
sps_mem <- setBackend(sps, MsBackendMemory())
sps_mem## MSn data (Spectra) with 1862 spectra in a MsBackendMemory backend:
## msLevel rtime scanIndex
## <integer> <numeric> <integer>
## 1 1 0.280 1
## 2 1 0.559 2
## 3 1 0.838 3
## 4 1 1.117 4
## 5 1 1.396 5
## ... ... ... ...
## 1858 1 258.636 927
## 1859 1 258.915 928
## 1860 1 259.194 929
## 1861 1 259.473 930
## 1862 1 259.752 931
## ... 35 more variables/columns.
## Processing:
## Switch backend from MsBackendOfflineSql to MsBackendMemory [Tue Nov 25 17:04:29 2025]With this function it is also possible to change from any backend to a
MsBackendOfflineSql (or MsBackendSql) in which case a new database is
created and all data from the originating backend is stored in this database. To
change the backend to an MsBackendOfflineSql we need to provide the connection
information to the SQL database as additional parameters. These parameters are
the same that need to be passed to a dbConnect() call to establish the
connection to the database. These parameters include the database driver
(parameter drv), the database name and eventually the user name, host etc (see
?dbConnect for more information). In the simple example below we store the
data into a SQLite database and thus only need to provide the database name,
which corresponds SQLite database file. In our example we store the data into a
temporary file. Optionally, setBackend() supports also the parameters blob
and peaksDataStorage described above for the createMsBackendSqlDatabase()
function.
sps2 <- setBackend(sps_mem, MsBackendOfflineSql(), drv = SQLite(),
dbname = tempfile())
sps2## MSn data (Spectra) with 1862 spectra in a MsBackendOfflineSql backend:
## msLevel precursorMz polarity
## <integer> <numeric> <integer>
## 1 1 NA 1
## 2 1 NA 1
## 3 1 NA 1
## 4 1 NA 1
## 5 1 NA 1
## ... ... ... ...
## 1858 1 NA 1
## 1859 1 NA 1
## 1860 1 NA 1
## 1861 1 NA 1
## 1862 1 NA 1
## ... 35 more variables/columns.
## Use 'spectraVariables' to list all of them.
## Database: /private/var/folders/r0/l4fjk6cj5xj0j3brt4bplpl40000gt/T/RtmpyBMneu/file1347d5b1168b4
## Processing:
## Switch backend from MsBackendOfflineSql to MsBackendMemory [Tue Nov 25 17:04:29 2025]
## Switch backend from MsBackendMemory to MsBackendOfflineSql [Tue Nov 25 17:04:29 2025]Similar to any other Spectra object we can retrieve the available spectra
variables using the spectraVariables() function.
spectraVariables(sps)## [1] "msLevel" "rtime"
## [3] "acquisitionNum" "scanIndex"
## [5] "dataStorage" "dataOrigin"
## [7] "centroided" "smoothed"
## [9] "polarity" "precScanNum"
## [11] "precursorMz" "precursorIntensity"
## [13] "precursorCharge" "collisionEnergy"
## [15] "isolationWindowLowerMz" "isolationWindowTargetMz"
## [17] "isolationWindowUpperMz" "peaksCount"
## [19] "totIonCurrent" "basePeakMZ"
## [21] "basePeakIntensity" "electronBeamEnergy"
## [23] "ionisationEnergy" "lowMZ"
## [25] "highMZ" "mergedScan"
## [27] "mergedResultScanNum" "mergedResultStartScanNum"
## [29] "mergedResultEndScanNum" "injectionTime"
## [31] "filterString" "spectrumId"
## [33] "ionMobilityDriftTime" "scanWindowLowerLimit"
## [35] "scanWindowUpperLimit" "spectrum_id_"The MS peak data can be accessed using either the mz(), intensity() or
peaksData() functions. Below we extract the peaks matrix of the 5th spectrum
and display the first 6 rows.
peaksData(sps)[[5]] |>
head()## mz intensity
## [1,] 105.0347 0
## [2,] 105.0362 164
## [3,] 105.0376 0
## [4,] 105.0391 0
## [5,] 105.0405 328
## [6,] 105.0420 0All data (peaks data or spectra variables) are always retrieved on-the-fly
from the database resulting thus in a minimal memory footprint for the Spectra
object.
print(object.size(sps), units = "KB")## 115.1 KbThe backend supports also adding additional spectra variables or changing their values. Below we add 10 seconds to the retention time of each spectrum.
sps$rtime <- sps$rtime + 10Such operations do however not change the data in the database (which is always considered read-only) but are cached locally within the backend object (in memory). The size in memory of the object is thus higher after changing that spectra variable.
print(object.size(sps), units = "KB")## 129.8 KbSuch $<- operations can also be used to cache spectra variables
(temporarily) in memory which can eventually improve performance. Below we test
the time it takes to extract the MS level from each spectrum from the database,
then cache the MS levels in memory using $msLevel <- and test the timing to
extract these cached variable.
system.time(msLevel(sps))## user system elapsed
## 0.005 0.001 0.005sps$msLevel <- msLevel(sps)
system.time(msLevel(sps))## user system elapsed
## 0.002 0.000 0.002We can also use the reset() function to reset the data to its original state
(this will cause any local spectra variables to be deleted and the backend to be
initialized with the original data in the database).
sps <- reset(sps)The performance of storing and retrieving MS data from an MsBackendSql
respectively SQL database can also depend on the type of database used as well
as storage modes and the database layout used by MsBackendSql.
Performance comparison have been made for small and large data sets using different SQL database systems and MsBackendSql has been optimized based on these results. For MariaDB database systems, for example, the Aria storage engine is used by default as it has considerable advantages over other MariaDB engines.
Performance of MariaDB and SQLite is comparable, even for very large data sets/databases. See this GitHub issue for performance comparison between MariaDB and SQLite.
Performance evaluation of SQLite and duckdb are provided in this GitHub issue. MsBackendSql long format database layout (see next section for details on available database layouts) with duckdb is clearly faster than with SQLite. For the blob2 database layout SQLite has advantages. Also, extracting individual spectra variables or filtering by e.g. retention time is slower for duckdb.
MsBackendSql defines different database table layouts and hence ways to store
the MS data. The most intuitive way to store MS data would be the long format
(peaksStorageMode = "long") which saves the m/z and intensity values of each
mass peak as a single row. While this would allow to filter e.g. the peaks data
by m/z and/or intensity values already on the SQL level, it significantly
increases the size of the database. This is in particular true for
SQLite-based databases. The default storage mode (peaksStorageMode = "blob2") stores the complete peaks matrix (i.e. the two-column numerical
matrix of m/z and intensity values) of spectrum as one entity to the
database. This entry is stored as a binary data type (BLOB) in the database
table (one row per spectrum). This reduces the size of the database and well
as the time to extract (peaks) data. On the downside, such databases will only
be readable and usable with MsBackendSql.
For MsBackendSql in the long peaks storage mode it is suggested to use duckdb as database backend.
The need to retrieve any spectra data on-the-fly from the database has an impact
on the performance of data access functions of Spectra objects using
MsBackendSql/MsBackendOfflineSql backends. To evaluate this we compare below
the performance of the MsBackendSql to other Spectra backends, specifically,
the MsBackendMzR which is the default backend to read and represent raw MS
data, and the MsBackendMemory backend that keeps all MS data in memory (and is
thus not suggested for larger MS experiments). Similar to the MsBackendMzR,
also the MsBackendSql keeps only a limited amount of data in memory. These
on-disk backends need thus to retrieve spectra and MS peaks data on-the-fly
from either the original raw data files (in the case of the MsBackendMzR) or
from the SQL database (in the case of the MsBackendSql). The in-memory backend
MsBackendMemory is supposed to provide the fastest data access since all data
is kept in memory.
Below we thus create Spectra objects from the same data but using the
different backends.
con <- dbConnect(SQLite(), dbfile)
sps <- Spectra(con, source = MsBackendSql())
sps_mzr <- Spectra(fls, source = MsBackendMzR())
sps_im <- setBackend(sps_mzr, backend = MsBackendMemory())At first we compare the memory footprint of the 3 backends.
print(object.size(sps), units = "KB")## 113.4 Kbprint(object.size(sps_mzr), units = "KB")## 401.4 Kbprint(object.size(sps_im), units = "KB")## 54509.1 KbThe MsBackendSql has the lowest memory footprint of all 3 backends because it
does not keep any data in memory. The MsBackendMzR keeps all spectra
variables, except the MS peaks data, in memory and has thus a larger size. The
MsBackendMemory keeps all data (including the MS peaks data) in memory and has
thus the largest size in memory.
Next we compare the performance to extract the MS level for each spectrum from
the 4 different Spectra objects.
library(microbenchmark)
microbenchmark(msLevel(sps),
msLevel(sps_mzr),
msLevel(sps_im))## Warning in microbenchmark(msLevel(sps), msLevel(sps_mzr), msLevel(sps_im)):
## less accurate nanosecond times to avoid potential integer overflows## Unit: microseconds
## expr min lq mean median uq max
## msLevel(sps) 2675.783 2815.900 3027.77128 2887.1380 3121.0430 6151.312
## msLevel(sps_mzr) 169.863 184.992 203.73679 194.6065 209.8380 521.930
## msLevel(sps_im) 5.535 7.544 11.89164 12.3615 15.0675 32.185
## neval cld
## 100 a
## 100 b
## 100 cExtracting MS levels is thus slowest for the MsBackendSql, which is not
surprising because both other backends keep this data in memory while the
MsBackendSql needs to retrieve it from the database.
We next compare the performance to access the full peaks data from each
Spectra object.
microbenchmark(peaksData(sps, BPPARAM = SerialParam()),
peaksData(sps_mzr, BPPARAM = SerialParam()),
peaksData(sps_im, BPPARAM = SerialParam()),
times = 10)## Unit: microseconds
## expr min lq mean
## peaksData(sps, BPPARAM = SerialParam()) 30289.324 37902.327 95656.255
## peaksData(sps_mzr, BPPARAM = SerialParam()) 419612.491 420627.077 467854.292
## peaksData(sps_im, BPPARAM = SerialParam()) 225.254 323.285 1074.081
## median uq max neval cld
## 39914.6070 221808.565 236265.452 10 a
## 430143.6485 442804.674 637295.841 10 b
## 426.3795 452.558 7221.371 10 cAs expected, the MsBackendMemory has the fasted access to the full peaks
data. The MsBackendSql outperforms however the MsBackendMzR providing faster
access to the m/z and intensity values.
Performance can be improved for the MsBackendMzR using parallel
processing. Note that the MsBackendSql does not support parallel
processing and thus parallel processing is (silently) disabled in functions such
as peaksData().
m2 <- MulticoreParam(2)
microbenchmark(peaksData(sps, BPPARAM = m2),
peaksData(sps_mzr, BPPARAM = m2),
peaksData(sps_im, BPPARAM = m2),
times = 10)## Unit: microseconds
## expr min lq mean median
## peaksData(sps, BPPARAM = m2) 32524.23 43515.842 94006.5425 52094.7435
## peaksData(sps_mzr, BPPARAM = m2) 324267.03 337257.021 479929.9198 375161.5825
## peaksData(sps_im, BPPARAM = m2) 160.31 281.834 520.4663 494.8905
## uq max neval cld
## 58712.041 291782.322 10 a
## 661387.892 994133.806 10 b
## 758.992 877.892 10 aWe next compare the performance of subsetting operations.
microbenchmark(filterRt(sps, rt = c(50, 100)),
filterRt(sps_mzr, rt = c(50, 100)),
filterRt(sps_im, rt = c(50, 100)))## Unit: microseconds
## expr min lq mean median
## filterRt(sps, rt = c(50, 100)) 967.846 1010.7525 1221.5966 1085.4545
## filterRt(sps_mzr, rt = c(50, 100)) 589.662 650.0345 702.8134 683.6135
## filterRt(sps_im, rt = c(50, 100)) 206.476 227.1400 282.3399 236.7340
## uq max neval cld
## 1179.0985 7355.810 100 a
## 723.1785 1360.790 100 b
## 253.3390 4365.352 100 cThe two on-disk backends MsBackendSql and MsBackendMzR show a comparable
performance for this operation. This filtering does involves access to a spectra
variables (the retention time in this case) which, for the MsBackendSql needs
first to be retrieved from the backend. The MsBackendSql backend allows
however also to cache spectra variables (i.e. they are stored within the
MsBackendSql object). Any access to such cached spectra variables can
eventually be faster because no dedicated SQL query is needed.
To evaluate the performance of a pure subsetting operation we first define the
indices of 10 random spectra and subset the Spectra objects to these.
idx <- sample(seq_along(sps), 10)
microbenchmark(sps[idx],
sps_mzr[idx],
sps_im[idx])## Unit: microseconds
## expr min lq mean median uq max neval cld
## sps[idx] 60.270 63.3450 73.12391 65.0260 66.8915 811.800 100 a
## sps_mzr[idx] 297.414 309.3450 319.73399 314.8185 320.2305 756.245 100 b
## sps_im[idx] 105.493 109.1625 111.66432 110.7615 112.7500 150.388 100 cHere the MsBackendSql outperforms the other backends because it does not keep
any data in memory and hence does not need to subset these. The two other
backends need to subset the data they keep in memory which is in both cases a
data frame with either a reduced set of spectra variables or the full MS data.
At last we compare also the extraction of the peaks data from the such subset
Spectra objects.
sps_10 <- sps[idx]
sps_mzr_10 <- sps_mzr[idx]
sps_im_10 <- sps_im[idx]
microbenchmark(peaksData(sps_10),
peaksData(sps_mzr_10),
peaksData(sps_im_10),
times = 10)## Unit: microseconds
## expr min lq mean median uq
## peaksData(sps_10) 498.068 556.698 1550.7881 875.7190 3229.693
## peaksData(sps_mzr_10) 38840.776 40074.917 43016.7859 41068.3060 42309.212
## peaksData(sps_im_10) 105.698 139.072 626.5825 305.7575 640.010
## max neval cld
## 3668.065 10 a
## 59775.007 10 b
## 3282.747 10 aThe MsBackendSql outperforms the MsBackendMzR while, not unexpectedly, the
MsBackendMemory provides fasted access.
The backends from the MsBackendSql package use standard SQL calls to retrieve MS data from the database and hence any SQL database system (for which an R package is available) is supported. SQLite-based databases would represent the easiest and most user friendly solution since no database server administration and user management is required. Indeed, performance of SQLite is very high, even for very large data sets. Server-based databases on the other hand have the advantage to enable a centralized storage and control of MS data (inclusive user management etc). Also, such server systems would also allow data set or server-specific configurations to improve performance.
A comparison between a SQLite-based with a MariaDB-based MsBackendSql database for a large data set comprising over 8,000 samples and over 15,000,000 spectra is available here. In brief, performance to extract data was comparable and for individual spectra variables even faster for the SQLite database. Only when more complex SQL queries were involved (combining several primary keys or data fields) the more advanced MariaDB database outperformed SQLite.
MsBackendSqlThe MsBackendSql backend does not support parallel processing since the
database connection can not be shared across the different (parallel)
processes. Thus, all methods on Spectra objects that use a MsBackendSql will
automatically (and silently) disable parallel processing even if a dedicated
parallel processing setup was passed along with the BPPARAM method.
Some functions on Spectra objects require to load the MS peak data (i.e., m/z
and intensity values) into memory. For very large data sets (or computers with
limited hardware resources) such function calls can cause out-of-memory
errors. One example is the lengths() function that determines the number of
peaks per spectrum by loading the peak matrix first into memory. Such functions
should ideally be called using the peaksapply() function with parameter
chunkSize (e.g., peaksapply(sps, lengths, chunkSize = 5000L)). Instead of
processing the full data set, the data will be first split into chunks of size
chunkSize that are stepwise processed. Hence, only data from chunkSize
spectra is loaded into memory in one iteration.
The MsBackendSql provides an MS data representations and storage mode with a
minimal memory footprint (in R) that is still comparably efficient for standard
processing and subsetting operations. This backend is specifically useful for
very large MS data sets, that could even be hosted on remote (MySQL/MariaDB)
servers. A potential use case for this backend could thus be to set up a central
storage place for MS experiments with data analysts connecting remotely to this
server to perform initial data exploration and filtering. After subsetting to a
smaller data set of interest, users could then retrieve/download this data by
changing the backend to e.g. a MsBackendMemory, which would result in a
download of the full data to the user computer’s memory.
sessionInfo()## R Under development (unstable) (2025-11-04 r88984)
## Platform: aarch64-apple-darwin20
## Running under: macOS Ventura 13.7.8
##
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.6-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
##
## locale:
## [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## time zone: America/New_York
## tzcode source: internal
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] microbenchmark_1.5.0 RSQLite_2.4.4 MsBackendSql_1.11.1
## [4] Spectra_1.21.0 BiocParallel_1.45.0 S4Vectors_0.49.0
## [7] BiocGenerics_0.57.0 generics_0.1.4 BiocStyle_2.39.0
##
## loaded via a namespace (and not attached):
## [1] sandwich_3.1-1 sass_0.4.10 MsCoreUtils_1.23.1
## [4] lattice_0.22-7 stringi_1.8.7 hms_1.1.4
## [7] digest_0.6.39 grid_4.6.0 evaluate_1.0.5
## [10] bookdown_0.45 mvtnorm_1.3-3 fastmap_1.2.0
## [13] blob_1.2.4 Matrix_1.7-4 jsonlite_2.0.0
## [16] ProtGenerics_1.43.0 progress_1.2.3 mzR_2.45.0
## [19] DBI_1.2.3 survival_3.8-3 multcomp_1.4-29
## [22] BiocManager_1.30.27 TH.data_1.1-5 codetools_0.2-20
## [25] jquerylib_0.1.4 cli_3.6.5 rlang_1.1.6
## [28] crayon_1.5.3 Biobase_2.71.0 splines_4.6.0
## [31] bit64_4.6.0-1 cachem_1.1.0 yaml_2.3.10
## [34] tools_4.6.0 parallel_4.6.0 memoise_2.0.1
## [37] ncdf4_1.24 fastmatch_1.1-6 vctrs_0.6.5
## [40] R6_2.6.1 zoo_1.8-14 lifecycle_1.0.4
## [43] fs_1.6.6 IRanges_2.45.0 bit_4.6.0
## [46] clue_0.3-66 MASS_7.3-65 cluster_2.1.8.1
## [49] pkgconfig_2.0.3 bslib_0.9.0 data.table_1.17.8
## [52] Rcpp_1.1.0 xfun_0.54 knitr_1.50
## [55] htmltools_0.5.8.1 rmarkdown_2.30 compiler_4.6.0
## [58] prettyunits_1.2.0 MetaboCoreUtils_1.19.1