We first investigated whether traditional spectral unmixing approaches were suitable for application to live-cell imaging datasets. We simulated multispectral datasets by computationally mixing fluorescence from eight fluorophores into eight spectral channels (Fig. 1a) and adding Poisson (that is, shot) noise (Fig. 1b). Unmixing these data using conventional linear unmixing (that is, applying the inverse of the mixing matrix to the mixed signals) produced poor results (Fig. 1c), with more errors associated with unmixing lower-SNR data (Supplementary Fig. 1a-d). In particular, linear unmixing predicted negative quantities of targets in many pixels (shown in red), which is physically impossible. This occurs because linear unmixing is incapable of dealing with Poisson (shot) noise, the predominant form of noise present in low-SNR datasets, such as those generated by live-cell imaging.

We reasoned that using an iterative approach, tailored to incorporate the effects of Poisson (shot) noise, would provide superior results. In particular, we elected to repurpose the Richardson-Lucy algorithm as this is fast, straightforward to use, and has been well-studied over the five decades since its development. Although the Richardson-Lucy algorithm has previously been used for deconvolving images using a known point spread function (PSF), the algorithm is, in principle, applicable to any system where the measurement process can be described by a linear operator and the noise is described by Poisson statistics. Various researchers have used Richardson-Lucy deconvolution to enhance the linewidths of spectral profiles, but these algorithms do not separate the different components as required for spectral unmixing. Instead, here we replace the action of convolving with the PSF with the action of mixing with the mixing matrix, which describes the contribution of each underlying object to each channel (Supplementary Note 1 presents more details). Hence, this new spectral unmixing algorithm does not deconvolve the data and so does not suffer from the high-spatial-frequency artefacts often seen after deconvolution. However, our approach can be extended to both unmix and deconvolve the data simultaneously (Supplementary Note 2 and Supplementary Fig. 2) though, for simplicity, we do not use this in this Article.

Applying this Richardson-Lucy spectral unmixing algorithm (RLSU) to the same simulated dataset shows that although one iteration is insufficient to accurately unmix the different components (Fig. 1d), 100 iterations produces a much higher-quality result than linear unmixing (Fig. 1e), even when negative values in the linearly unmixed results are set to zero (Supplementary Fig. 3). In our hands, RLSU also outperformed non-negative least-squares unmixing (Supplementary Note 3 and Supplementary Figs. 4-6) and a recent implementation of phasor-based unmixing, HyU, for both eight-channel (Supplementary Fig. 7) and 32-channel (Supplementary Fig. 8) data. Notably, no negative quantities are generated by RLSU -- indeed, this is impossible if the initial estimate of the components is everywhere positive, as the iterative update is multiplicative in nature. We discuss the appropriate Cramér-Rao lower bound in Supplementary Note 4 and show, via simulations (shown in Supplementary Fig. 9), that we reach it. Repeating this process for different SNR levels, and incorporating both Poisson (shot) noise and read noise, we found our iterative algorithm always outperformed linear unmixing (Supplementary Figs. 1, 10 and 11 and Supplementary Video 1). Importantly, read noise only distorts the quality of unmixing results when the variance of the read noise is comparable to the variance of the Poisson (shot) noise. For modern sensors for which the read noise is small, such as the 2.32 e read noise of the cameras used in this work, this means that the effects of read noise are only important for signal levels below five counts. We also found that RLSU outperforms linear unmixing for highly overlapping spectral signals, such as enhanced green fluorescent protein (eGFP) and enhanced yellow fluorescent protein (EYFP) (20-nm peak-to-peak separation), and can unmix signals with a peak-to-peak spectral separation as low as 4 nm using only two channels (Supplementary Note 5 and Supplementary Fig. 12).

Encouraged by these results, we applied our algorithm to data acquired on a commercially available multispectral confocal microscope (Zeiss LSM 710 with 32-channel QUASAR detector). U2OS cells transfected with a polycistronic plasmid (ColorfulCell) encoding six fluorescent protein species targeted to the nucleus (TagBFP), plasma membrane (Cerulean), mitochondria (mAzamiGreen), Golgi apparatus (Citrine), endoplasmic reticulum (mCherry) or peroxisomes (iRFP670) were imaged live, with data exhibiting SNRs up to 13 (Supplementary Fig. 13). We then attempted to unmix the live-cell data using both RLSU and linear unmixing.

Linear unmixing did not always accurately reassign signals to the correct labelled structures, such as the Golgi signal incorrectly present in the mitochondria channel in Fig. 1f, and once again produced numerous pixels with unphysical negative values. In contrast, RLSU produced unmixed objects that accurately resembled the single-labelled controls (Fig. 1g), with no observable bleedthrough or misassignment of signals. A comparison with an implementation of non-negative matrix factorization (NMF), a blind spectral unmixing approach also designed to handle Poisson (shot) noise but without requiring a priori knowledge of the mixing matrix, showed that although this did not produce negative quantities, none of the signals were correctly assigned (Fig. 1h). Most strikingly, substantial nuclear signal was present in the plasma membrane channel. Furthermore, despite being initialized with the correct mixing matrix, the algorithm selected an erroneous mixing matrix as its preferred solution (Supplementary Fig. 14).

Applying both linear unmixing and RLSU on various cell images of differing signal levels, we found that in all cases RLSU was more robust against channel misassignment. Figure 1i shows an example of this, where a weak mitochondrial signal has been assigned to the nuclear channel by linear unmixing, but is correctly absent in the RLSU results. In both cases, the weak background signal outside the nucleus is due to cytosolic fluorescent proteins that have not yet been redirected to the nuclear compartment by the fused targeting signal. Furthermore, we tested the sensitivity of RLSU to small errors in the mixing matrix by unmixing simulated ColorfulCell data with a mixing matrix formed by an incorrect combination of fluorophores (Supplementary Fig. 15). Through this, we found we could, for example, replace mCherry's mixing matrix values with those for Alexa Fluor 594 with no penalty in the quality of the unmixing results, despite their slightly different spectral profiles. However, mScarlet's more shifted spectrum led to noticeable reconstruction errors when the mixing matrix was updated to use its values.

Given the performance of RLSU on low-SNR data, we sought to develop imaging hardware that would allow us to capture the necessary raw signals at rates much faster than those afforded by the point-scanning QUASAR detector. In particular, we decided that the hardware should have five key characteristics and should:

These considerations led us to design a system using dichroic mirrors, in a tree-like arrangement, to redirect light to multiple cameras (see the optical path illustration in Fig. 2a and hardware photograph in Fig. 2b). In contrast to conventional multispectral approaches using diffraction gratings or interferometers, this enables the capture of the full spatial and spectral information in a single shot. Furthermore, as light is merely redirected by the dichroic mirrors, and not absorbed as by a filter, there is no loss of photons through the tree-like arrangement -- a photon that does not end up in a specified channel must end up in one of the others. Although this does not obviate the absorbative losses in the extra lenses used, these are small (~3%). Choosing such an approach also means that such a system could image n channels in less than half the time that a conventional filter switching system could image in just two, avoiding colour drift (Supplementary Fig. 16).

We settled on using seven dichroics (spectra plotted in Fig. 2c) to provide eight channels of data (spectra plotted in Fig. 2d), as this was the most effective choice for imaging over the typical 450-700-nm spectral range used in live-cell fluorescence microscopy experiments (Supplementary Note 6). In the interests of portability, rejection of excitation light is achieved using the primary dichroic, and if necessary a notch filter, in the parent instrument. Although this will affect the overall spectral profile of the detection unit, this is easily accounted for in the mixing matrix as long as the spectral profile of the dichroic/filter is known.

Raytracing showed that our design would maintain diffraction-limited resolution over the full field of view of the tube lenses used (20-mm diameter), even on the path where the light traverses three tilted dichroic mirrors (Supplementary Fig. 17). To enable future extension to 16 channels, extending the spectral range into the near infrared, we also validated that diffraction-limited resolution was maintained in the case that four tilted dichroics were used. Testing the resolution of the assembled system using a USAF 1951 target (Fig. 2e) showed that the resolving power of the system was limited by the Nyquist sampling rate (6.21 μm pixel; Supplementary Note 7). This sampling rate was chosen to match that of a typical scientific complementary metal-oxide semiconductor (sCMOS) camera, as commonly used with high-resolution fluorescence microscopes (6.5 μm pixel).

By design, the sampling rate of the system can be changed by altering the focal length ratio of the input and output lenses. As constructed, we used an input lens with 180-mm focal length, and output lenses with focal lengths of 100 mm. Alternative tube lenses with focal lengths from f = 165 mm to 600 mm are also available, giving effective pixel sizes spanning 3.45-20.7 μm. Further adjustments to the effective pixel size would either require replacing the output lenses or selecting different camera sensors.

To ensure maximum mechanical stability, we elected to not mount the dichroic mirrors kinematically. As a result, each image is slightly displaced off-axis owing to small angular deviations of the dichroic mirror orientation from the design specification. However, these shifts were sufficiently small that they could be corrected by a simple image registration routine and left >80% of the field of view usable (Supplementary Fig. 18).

First, we coupled our multispectral module to a spinning-disk confocal microscope (SDCM). PSF measurements showed resolutions equivalent to those obtained without the multispectral module, demonstrating that the module did not compromise spatial resolution (Supplementary Fig. 19). Imaging U2OS cells transfected with the six-colour ColorfulCell plasmid, and stained with LysoTracker Yellow, produced eight camera images spanning the visible spectrum (Fig. 3a). To facilitate the choice of appropriate fluorophores and generation of the corresponding mixing matrix, we developed a web application, hosted at beryl.mrc-lmb.cam.ac.uk/calculators/spectral_unmixing/. This uses manufacturer-supplied spectra for the dichroics used in the module, as well as the dichroic and notch filter in the SDCM, in addition to reference fluorescence emission spectra from fpbase.org, to enable the calculation of appropriate mixing matrices and provide feedback on whether the experiment is spectrally feasible (Supplementary Fig. 20). Using a mixing matrix generated by the web application from an appropriate selection of fluorophores (Fig. 3b), we unmixed the camera images to form seven object channels (Fig. 3c).

Despite the fourfold smaller number of spectral channels compared to the QUASAR detector, we obtained unmixed images that accurately reflected the expected distributions of fluorescent proteins in their respective organelles (Supplementary Fig. 21). As the full spatial and spectral information was acquired in one exposure time, we found that imaging speeds could be increased more than 100-fold over our previous confocal QUASAR point-scanning (Supplementary Video 2). Furthermore, the flexibility of the web application interface enabled us to swap LysoTracker Yellow for other live-cell imaging dyes and recompute mixing matrices. These datasets also unmixed accurately, without the need for control measurements of singly-labelled cells to form the mixing matrix (Fig. 3e-g and Supplementary Videos 3-5). Applying LU to these datasets produced results with similar unmixing errors as seen in the previous QUASAR datasets (Supplementary Fig. 22).

Attempts to extend our imaging from 2D+t to 3D+t were challenged by a noticeable level of phototoxicity after only a few timepoints (Supplementary Fig. 23 and Supplementary Video 6), as expected for fast live-cell spinning-disk confocal microscopy. Although this could be partially mitigated by adding a recovery interval between timepoints, this compromised the imaging speed. In practice, we found that the maximum achievable volumetric imaging rate without substantial photodamage was typically of the order of one volume per minute.

To reduce photodamage and increase the volumetric acquisition speed, we next coupled our multispectral acquisition module to an oblique-plane light-sheet microscope (OPM). PSF measurements showed that the module again did not compromise the resolution of the instrument, which was comparable to the resolution achieved with the SDCM (Supplementary Fig. 24). As before, the raw signals are correctly unmixed by RLSU into the expected six cellular compartments labelled by the ColorfulCell plasmid (Fig. 4a and Supplementary Videos 7 and 8). As expected, the gentler illumination strategy used by the multispectral OPM allowed us to routinely perform simultaneous six-colour volumetric imaging of cells for at least 200 timepoints, representing a more than tenfold improvement compared to SDCM imaging (Supplementary Video 9).

Although the colour merge, shown in Fig. 4b, provides a good overview of the cell, we found that it was often hard to discern the interactions between components with such high-dimensional datasets. To combat this, we used a 'pairwise montage' visualization (Fig. 4c) that shows two components at a time. Each n of the N objects is assigned a first colour and repeated N times down the nth column, then assigned a second colour and repeated N times across the nth row. In this way, the leading diagonal of the visualization shows the overlap of one object with itself (that is, the object itself is displayed in the colour obtained by adding the first colour to the second), while other elements show the overlap of one object with the others. As the object combinations are symmetric about the diagonal, part of the visualization could be excluded as it conveys the same information as is available elsewhere. However, in practice, we found that it was helpful to see both orderings of the colours.

To achieve even faster multispectral imaging speeds, we utilized the recently described shear-warp angled projection technique. By adding a galvo-based image shifting unit in front of the multispectral module and synchronizing the sweep of the light sheet, focal plane and image shift, we could acquire projection images of an entire cell from arbitrary viewing angles in one camera exposure.

This combination achieves whole-cell multispectral projections at 10 Hz (Fig. 5a,b and Supplementary Video 10). The laser powers were increased from those used for volumetric imaging to maintain signal levels, but we could not achieve sufficient irradiances for the 405-nm laser to provide acceptable signals from the TagBFP and Cerulean labels. As such, we elected to deactivate this laser to reduce any potential photodamage but continued to unmix data from cells transfected with the ColorfulCell plasmid with a six-component mixing matrix. After unmixing, both the TagBFP and Cerulean channels correctly contained no signal.

The increased acquisition rate provided by the multispectral projection imaging allowed us to see both fast subsecond organelle dynamics for hundreds of timepoints (for example, ER remodelling), but also slower dynamics within the same video. For instance, as can be seen in Fig. 5d, part of the Golgi apparatus fissions in 0.5 s and then, 4 s later, re-fuses. Similarly, Fig. 5e shows fission and fusion events for a number of peroxisomes on a similar timescale, and Fig. 5f shows examples of two-, three- and four-way organelle interactions extracted from the same dataset.

It is worth emphasizing that we have focused in the above on complex samples with a high number (six or seven) of fluorescent probes, but multispectral imaging with our approach does not compromise the imaging quality or speed for simpler samples with just, say, two fluorophores. In fact, multispectral imaging is more photon-efficient than conventional bandpass-filter imaging systems, as light is merely redirected rather than rejected. As such, although we could use a subset of our cameras corresponding to the main peaks of the fluorescence emission spectra to capture data, it is better to use all cameras all the time to enable maximally efficient photon reassignment.

To illustrate this point, we performed two-colour light-sheet imaging of macropinocytic cup closure in a Dictyostelium strain expressing LifeAct-mCherry and eGFP fused to a phosphatidylinositol (3,4,5)-trisphosphate (PIP) reporter (Supplementary Fig. 25). This is a particularly challenging sample, as Dictyostelium are known to be particularly light-sensitive, and macropinocytic cup closure occurs rapidly in three dimensions, requiring fast volumetric imaging. After careful optimization of exposure times and laser powers, we could reliably acquire videos containing hundreds of timepoints without cell death, in line with our previous experience in imaging this strain on both our non-multispectral original OPM system and a field synthesis light-sheet microscope. In particular, we achieved full cell volumes at 2 Hz, which was enough to follow the process of macropinocytic cup closure and, by relying on simultaneous multispectral acquisition rather than sequential acquisition, the images were devoid of motion blur or colour misregistration (Supplementary Video 11).

Fluorescent protein fusions provide fluorescence images free of non-specific binding, but tagging at endogenous levels requires time-consuming genome editing, a process that must be repeated for each colour/target used. As a proof-of-concept experiment, we instead used de novo-designed protein-binding proteins (minibinders), labelled with small-molecule dyes, to visualize the endosomal sorting of endogenous cell-surface receptors.

Endosomal sorting is the process by which cells sort different transmembrane receptors towards three major routes following their endocytosis: degradation in lysosomes, recycling back to the plasma membrane, or retrograde transport to the Golgi apparatus. Imaging endosomal sorting is difficult, first because endosomes move rapidly in three dimensions and second because overexpression of fluorescently labelled receptors can be detrimental. Recently, we developed pipelines to computationally design specific binders for proteins of interest. We therefore thought to fluorescently label minibinders, expressed and purified from Escherichia coli, that recognize the extracellular region of endogenous transmembrane receptors, and use them to reveal the dynamics of receptor trafficking and sorting in cells (Supplementary Fig. 26a). These de novo-designed binders offer five advantages for multiplexed labelling of endogenous transmembrane receptors when compared to antibodies or fluorescent protein fusions introduced through genome editing:

As a proof-of-concept experiment, we fluorescently labelled four minibinders targeting the transferrin receptor (TfRB), the insulin-like growth factor 2 receptor (IGF2RB), the bone morphogenetic protein receptor type 2 (BMPR2B) and integrin α5β1 receptors (Iα5β1B; Methods and Supplementary Fig. 26b present the minibinder design and Supplementary Fig. 26c biochemical characterization). These four minibinders were labelled with a combination of fluorophores that we had previously validated could be unmixed when imaged using the multispectral OPM (Supplementary Table 3 and Supplementary Fig. 26). When incubated individually with cells, fluorescent minibinders internalize into punctate dynamic structures (Supplementary Fig. 27a; also Supplementary Fig. 27b to show that minibinders are unlikely to be internalized by fluid-phase endocytosis). These structures were reminiscent of endosomes, as confirmed by their colocalization with WASH, an early/sorting endosome marker (Supplementary Fig. 27c,d and Supplementary Video 12). We thus incubated U2OS cells expressing nuclear TagBFP with all four binders simultaneously alongside labelled epidermal growth factor (EGF) and imaged them with the multispectral OPM (Fig. 6b,c).

Over time, binders were internalized and appeared in highly motile, diffraction-limited objects, as expected for receptors trafficking within the endocytic pathway (Supplementary Video 13). Importantly, the combination of the multispectral imaging module and the rapid imaging afforded by the OPM allowed us to achieve a volumetric imaging rate of 3 Hz in all channels (Supplementary Video 14). This not only allowed us to track all endosomes in the cell in 3D without motion blur, but also with the absence of colour misregistration caused by delays in acquiring colour channels in typical sequential schemes (Fig. 6g-i).

This allowed us to determine the content of each endosome, which revealed that individual compartments showed markedly different distributions of the different minibinders, and thus of their cognate receptors (Fig. 6d, quantified in 6f). For example, the cyan-boxed compartment in Fig. 6c,d contains all labelled receptors and EGF, whereas the yellow-boxed compartment lacks integrin α5β1 and the BMP receptor. Such a selective enrichment of specific receptors in specific endosomes is expected, as we selected receptors known to traffic via different routes, and they would thus tend to be sorted away from each other (Supplementary Fig. 26a).

The high spatiotemporal resolution of the multispectral OPM coupled with the increased brightness conferred by the fluorescent minibinders allowed us to directly image the flow of the different receptors within the endomembrane system. For instance, Fig. 6e shows an endosome containing all labelled binders, plus EGF. Within a few seconds, this compartment elongates a tubule and fissions into two daughter endosomes, with one daughter endosome lacking EGF and BMPR2 (Fig. 6j presents quantification). Hence, during this fission event, the EGF and the BMP receptor binders have been sorted away from the integrin, IGF2 and transferrin receptor binders. Conversely, we could image fusion events during which the content of organelles exchange receptors (Supplementary Fig. 28).