Skip to main content
Back

Nanjing University's MSFRnet Reconstructs 800-Megapixel Video at 256x Faster Readout

A low-bit gradient sensor paired with a small intensity sensor could break the readout and bandwidth walls holding back gigapixel video cameras.

Paper, Full Access
  • Nanjing University team proposes low-bit gradient plus low-res intensity capture for compact gigapixel video cameras.
  • Replacing 8-bit ADCs with comparators yields up to 256x faster readout, unblocking gigapixel frame rates.
  • Modified run-length and Huffman coding compresses gradient data below 10% of raw bandwidth.
  • MSFRnet CNN fuses HRG and LRI streams with cross-scale connections and dual-direction gradient reconstruction.
  • Ternary (1.5-bit) single-direction gradient chosen as best trade-off; 31.78 dB PSNR on Set5.
  • Achieves 44.37 dB PSNR on 800-megapixel PANDA frames; validated on Xiaomi Mi10 with Samsung ISOCELL HMX sensor.

Building a video camera that shoots hundreds of megapixels at video rate has been stuck against a stubborn wall: even when chip designers cram enough photodiodes onto silicon, they cannot get the data off the chip fast enough. A team from Nanjing University proposes a workaround that sidesteps both the analog-to-digital bottleneck and the transmission cable bottleneck by capturing the scene as a low-bit spatial gradient plus a tiny thumbnail image, then reconstructing the full-resolution frame with a CNN. The full write-up is on arXiv.

The gigapixel readout wall

Scale a CMOS sensor past 100 megapixels and two hardware limits bite hard. The single-slope analog-to-digital converter (SS-ADC) used in most commercial CMOS image sensors is compact and low-power but slow, which caps existing hundreds-of-megapixel parts at only a few frames per second. Transmission is no better: a three-lane C-PHY v2.0 MIPI link delivers roughly 41.4 Gbps, while a real-time 8-bit high-throughput video stream at gigapixel scale needs at least 240 Gbps.

The other route people have tried is stitching many small cameras into one giant array. That works, but the resulting rigs are large, expensive, and painful to synchronize. Neither approach is realistic for a consumer-grade device.

Gradients, not pixels

The core insight is that a low-bit gradient image, meaning just the sign or coarse magnitude of the difference between neighboring pixels, still preserves almost all of the high-frequency structure of a scene. What it loses is the low-frequency information, the overall brightness and color slopes, which can be picked up cheaply from a small conventional sensor running alongside the gradient one. The system uses two image sensors, a common low-resolution intensity (LRI) sensor and a high-resolution gradient (HRG) sensor, with a beam splitter feeding both from the same objective lens.