She lived an exotic life. Racing around in that car across New Mexico. Yet some of her photos are very beautiful. She was impersonating someone in this photo. I guess it's ok since they were friends. Rumor has it she knew one of the Franks through her husband. Just when you thought it was about you... There is so much more.
During my visit to the Jüdisches Museum Frankfurt am Main, I photographed this decorative plaque bearing the Schma Israel ("Hear, O Israel") — the central prayer of Jewish faith. Inscribed in both Hebrew and German, it was designed to hang on the eastern wall of a synagogue, facing Jerusalem. Its imagery is rich with Temple symbolism: the two columns represent the twelve tribes of Israel, crowned lions frame a radiant medallion with the name of God, and below sits the Ark of the Covenant, which once held the tablets of the law in the Holy of Holies.
What struck me was how a single object could distill so much - faith, history, exile, and longing for Jerusalem, into one visual composition. While similar plaques sometimes appeared in homes as Mizrah indicators of prayer direction, this particular piece was clearly made for a synagogue, as its formal design and Temple references suggest.
This plaque is a remnant of a German Jewish community that was nearly annihilated during the Shoah. Today, it survives as a witness – both to what was built and to what was lost.
A small but powerful reminder of the depth of German Jewish heritage.
This is not AI art. No generative model painted, styled, or hallucinated anything into this scene. A spectral reconstruction model expands a single linear raw capture into a dense spectral cube spanning the visible and near-infrared. From that cube, a Spectral Angle Mapper (SAM) compares every pixel against a library of known spectra.
Here is exactly what the spectral angle computes:
For every pixel in the scene, we take its reconstructed spectrum \( \mathbf{s} \) (its reflectance across many narrow wavelength bands). We compare that pixel spectrum against a library of known reference material spectra \( \mathbf{r} \). The spectral angle \( \theta \) is the angular distance between these two vectors in \( n \)-dimensional band-space:
I was first introduced to Einstein’s special theory of relativity by a math teacher whose husband, a mathematician, had worked on Einstein’s field equations. Around that time Roger Penrose visited my university and I listened to his lecture. Penrose, together with Hawking, had shown that general relativity predicts the conditions of its own breakdown. That is the theory at its most extreme. The same theory at its least dramatic says something about an airplane. Fly to Tokyo and you land a few nanoseconds displaced from everyone who stayed home. It is real and it has been measured, though not as a clean demonstration of velocity alone. For an airliner, the velocity term that slows the clock competes with the altitude term that speeds it up, and in 1971 Hafele and Keating flew cesium clocks east and west around the world and recorded offsets of opposite sign for the two directions.1
After graduating college, I worked with some folks who had just finished a project on the super collider in Waxahachie and another group out of France and Geneva. It was then that I first started to think about time as something a particle carries with it. Certain particles are created high in the atmosphere and should fall apart long before they ever reach the ground. They arrive anyway. They arrive because they are moving fast enough that their own clock runs slow compared to ours, which buys them the trip down. Rossi and Hall measured this in 1941,2 and particle accelerators have confirmed it since to very high precision. Nothing about the particle itself changes. What changes is the relationship between its clock and ours.
Now the train. Einstein asked us to imagine a long, straight railway track with a train moving along it at a steady speed. Lightning flashes twice: once at the front of the train and once at the rear. On the ground, an observer stands exactly halfway between the two points where the lightning hit. She sees the light from both flashes arrive at the same moment, so she concludes the flashes were simultaneous.
On the train, a passenger sits exactly in the middle of the train car. The train moves forward while the light travels. The flash from the front reaches her before the flash from the rear, because she is moving toward the forward light and away from the rear light. She therefore concludes that the forward flash happened first. Both observers are correct within their own frame of reference. There is no absolute "now"; simultaneity depends on your state of motion.
A word about the spaceship, because it is the part people stop on. There is nothing exotic about it. Time only stretches by an amount worth noticing when something travels at a good fraction of the speed of light, and nothing on Earth travels remotely that fast. A train is far too slow. An airplane is too slow, which is why the effect on one is measured in billionths of a second. So the thought experiment needs a vehicle that could actually reach those speeds, and a spaceship is simply the most convenient one to picture, because space is the only place with room enough to get up to speed and nothing in the way. Einstein used trains because a train was the fast thing of his day. The vehicle is a stand-in for speed and nothing else. The point is never the ship. The point is what happens to the clock inside it.
This relativity of time becomes vivid with a light clock. Imagine a spaceship carrying a photon bouncing vertically between two mirrors. To someone on board, each round-trip is a simple up-and-down tick. To an observer on the ground, the photon traces a longer diagonal path because the mirrors move between bounces. Since the speed of light is the same for all observers, the diagonal journey takes longer. Thus the moving clock is seen to run slow. This is time dilation.
Time dilation is not a metaphor. A traveler moving at high speed ages less than those who stay behind. Real experiments confirm it. If you board a spaceship and accelerate to a significant fraction of light speed, you can travel years on your own clock while centuries pass on Earth. This is forward time travel, allowed by special relativity. Backward time travel, however, is not. General relativity offers theoretical loopholes: Gödel’s rotating universe,3 wormholes,4 the interior of a spinning black hole. Each fails for its own reason. Wormholes need exotic matter with negative energy density. Gödel’s solution describes a universe that rotates, which ours does not appear to do. The closed paths inside a spinning black hole sit behind a horizon nothing returns through. Hawking’s chronology protection conjecture suggests the universe forbids closed timelike curves outright.5 Practically, you can journey into the future, but you can never send a message to the past.
That irreversibility is the heart of the matter. The past is sealed. We can only move forward, carrying what we have learned.
What clung to me, what I still carry in my bones, is the cruel elegance of the instrument itself. Those trains, those spaceships: they aren't just physics. They are lenses. Tools for seeing exactly what light permits and what it ruthlessly forbids.
And thirty years after that miracle year of 1905, Einstein aimed that same sharp, unforgiving lens away from the stars and straight into the quantum mess. He was looking for a new foothold, a new way of seeing the chaos underneath.
In 1935, Albert Einstein and his colleagues Boris Podolsky and Nathan Rosen published a seminal paper arguing that the quantum-mechanical description of physical reality is incomplete.6 The claim was not that quantum mechanics predicts wrongly. It was that a theory can predict correctly and still fail to describe everything that is there. They built the argument on a pair of particles prepared so that measuring the position or the momentum of one lets you predict the corresponding quantity for the other, at any separation, without disturbing it.
Two attributions are worth getting right, because the popular account collapses them. The familiar spin version of the argument (two-outcome measurements along chosen axes, the form in which nearly every modern discussion is conducted) is Bohm's reformulation from 1951, not the 1935 paper.7 And the word entanglement is not Einstein's either; Schrödinger introduced it later that same year, writing in reply.8
It is the quintessential example of a problem in which the whole contains information that cannot be inferred from looking at one part in isolation. An entangled pair can be prepared in a joint quantum state in which particular measurements produce strongly correlated results. Conservation laws can impose those relationships; total angular momentum has to add up. The result is not that each particle secretly carries a classical answer waiting to be revealed, but that quantum mechanics assigns probabilities and correlations to the joint system.
And the correlation is not created by the act of measurement. It is fixed at preparation, written into the joint state rather than into either particle on its own, which is a different claim from saying each particle quietly carried its own answer all along. Bell's result below rules that second reading out. Measuring one particle reveals it, and updates what you should expect from a measurement on the partner, regardless of the distance separating them. That distinction is the whole ballgame, because it is precisely why entanglement cannot be used as a telephone. The formal statement is the no-communication theorem: no local operation on one half of an entangled pair changes the measurement statistics available at the other half. The marginal distribution at each end is untouched, whatever is done at the far end. You need the classical record from both ends, brought together at light speed or slower, before the correlation becomes visible at all. The strangest object in physics still cannot carry a single word backward, or even sideways, faster than light.
So the strange thing about entanglement is not that it provides a faster-than-light telephone; it is that nature permits correlations between separated measurements that cannot be reproduced by ordinary local hidden-variable theories. Einstein objected to the implications of that picture. Bell showed in 1964 that the objection was testable,9 and experiment has since answered. Aspect, Dalibard, and Roger's 1982 measurement with time-varying analyzers10 left two gaps open, the locality loophole and the detection loophole, and both were closed in 2015 by a set of experiments that violated Bell inequalities with no significant loophole remaining.11,12,13 The 2022 Nobel Prize in Physics recognized this line of work. Local hidden variables are gone. Einstein was wrong about the conclusion and right that the question was worth forty years of somebody's life.
The EPR Analogy, and Where It Stops
While the EPR debate centered on the foundations of quantum mechanics, its deeper philosophical lesson, that direct observation can miss profound relationships within a system, resonates with modern imaging. In each case, information that is not apparent from an isolated measurement becomes accessible when measurements are considered together and interpreted through an appropriate mathematical model.
Just as the naked eye perceives only a fraction of the electromagnetic spectrum, a standard RGB sensor records only three broad spectral responses. The information discarded between and beyond those responses can contain clues about the chemical and physical properties of a material. Multispectral and hyperspectral imaging address this limitation by measuring, or in some cases estimating, how materials interact with light across a broader spectral range, and mathematical reconstruction can then infer spectral structure that is not directly represented in the original RGB observation.
The analogy is not that spectral correlations are quantum entanglement. They are not, and the difference in depth is enormous. It is only that both problems reveal the same limitation in taking a partial observation as the whole story.
Silicon Photonic Architecture The realization of this physics in modern hardware is constrained by the physical dimensions and properties of the semiconductor used to capture it. The interaction of incident photons with the silicon lattice, generating electron-hole pairs, is the primary physical process underlying photon detection in a conventional silicon image sensor, and that process sets hard boundaries at both ends of the range.
Silicon's indirect bandgap is approximately 1.1 eV, so photodiode response falls away as wavelengths approach roughly 1100 nm and there is no photoelectric detection beyond it. At the other end, absorption depth collapses: on the order of a few nanometres in the ultraviolet against hundreds of microns in the near-infrared. Ultraviolet photons are absorbed before they can reach the depth at which a front-illuminated sensor's photodiodes sit, which is the substantive reason back-illumination matters for short-wavelength work rather than a marketing distinction.
The Spectral Gatekeeper Before any of that, almost every camera bonds a UV/IR-cut filter directly into the stack above the sensor. That filter, not the lens glass, is the dominant constraint on accessible spectral range: it typically passes roughly 400–650 nm and rejects wavelengths to either side by orders of magnitude. No amount of downstream processing recovers a band the filter rejected, so any work outside that window is a question of optics and filtration rather than algorithms.
A second consequence is easy to miss. Bayer color-filter dyes leak in the near-infrared; past roughly 800 nm all three channels become nearly transparent and nearly identical. Beyond the visible window a silicon sensor therefore stops behaving as a trichromatic device and behaves more like a monochrome detector carrying three largely redundant channels, which means wavelength discrimination in that region cannot come from the color filter array at all.
Sensor Architecture The core of this pipeline is a modern back-illuminated CMOS sensor, optimized for high-resolution imaging and radiometric capture.
Active Sensing Area: The sensor's physical dimensions matter because they influence the total photon flux the device can collect for a given scene and exposure, and therefore the achievable Signal-to-Noise Ratio. No downstream algorithm can recover photons that were never recorded.
Pixel Pitch: Native photodiode pitch varies widely by format, from around a micron in small-format sensors to several microns in full-frame. Pitch, however, is usually not the binding limit.
Diffraction, Not Sampling At realistic working apertures the optics run out of resolution before the pixel grid does. The Airy disc radius is approximately 1.22λN; at f/8 and 550 nm that is about 5.4 µm, several times the pitch of a small-format sensor and larger than a typical full-frame photosite. Stopping down to gain depth of field across a textured surface therefore buys resolution loss at a predictable rate. Fine fiber structure in a document is recoverable only when aperture, wavelength, and working distance are chosen such that the optical transfer function still passes those spatial frequencies. Adding pixels does not help once diffraction has already removed the detail.
Mode Selection The choice between binned and unbinned modes depends on the analysis requirements, and the SNR arithmetic depends on where the binning happens and which noise source dominates.
True charge-domain binning—combining photocharge from neighboring photodiodes before the sense node and before readout—keeps the read-noise contribution essentially constant while the signal scales with the number of pixels summed. In the read-noise-limited regime a 2×2 operation can therefore improve SNR by up to a factor of four. Many modern CMOS sensors, however, implement only digital summation after independent digitization of each pixel, or a hybrid scheme in which charge sharing is possible in one direction only. In the pure digital case the signal still multiplies by four while uncorrelated read noise grows by the square root of four, a factor of two, limiting the SNR gain to a factor of two. In the shot-noise-limited regime (bright scenes, long exposures, controlled illumination) neither approach exceeds a factor of two, because shot noise itself scales as the square root of the collected signal. Binning remains a low-light tool; whether the spatial cost is worthwhile is decided by which noise term actually dominates the exposure.
Full-resolution (native sampling) mode is preferred when spatial detail is the priority—resolving fine fiber patterns in historical documents or detecting micro-scale material boundaries—subject to the diffraction limit above.
The Optical Path The light reaching the sensor passes through a multi-element lens assembly with a fast maximum aperture. A spectral imaging system measures a combination of the material's spectral reflectance R(λ), the illumination spectrum, the optical transmission T(λ), the detector response, and other system characteristics. Modern optical glass and coatings attenuate particular wavelengths, especially toward the near-UV, so those effects must be characterized during calibration if measurements are intended to be quantitatively meaningful. Some coated modern lenses cut off the near-UV so sharply that older uncoated or quartz-element designs are the practical choice for that end of the range.
Calibration: The Three Frames You Cannot Skip A spectral measurement is not a photograph, and the forward model above makes the requirement explicit. Recovering R(λ) means dividing out everything in that product chain that is not the material.
Dark frame: matched exposure time and sensor temperature with no light reaching the sensor, subtracted to remove black-level offset and dark current. Both are temperature-dependent, so the frame has to be acquired close in time and condition to the measurement rather than reused from an earlier session.
Flat field: uniform illumination of a spectrally neutral surface, divided out to remove lens vignetting, pixel response non-uniformity, and illumination falloff across the field. All three are wavelength-dependent, so a single flat does not serve every band.
White reference: a diffuse reflectance standard of known spectral reflectance, imaged under the same illumination in the same geometry. This is what converts raw signal into reflectance rather than an arbitrary sensor unit. Without it the pipeline produces numbers that are internally consistent and externally meaningless. Reflectance is a ratio, and something has to be the denominator.
The Digital Container: DNG and Linearity The accuracy of computational imaging depends heavily on the integrity and characterization of the input data. The Adobe DNG specification can provide a standardized container for raw or rendered image data, but the presence of a DNG extension by itself does not guarantee scientific linearity or calibrated radiometric measurements. The actual camera data, conversion path, metadata, and calibration procedure all matter.
Scene-Referred Linearity For spectral reconstruction, the important property is preservation of a useful linear relationship between recorded sensor signal and scene radiance over the relevant operating range. Raw sensor measurements can generally be treated as approximately linear with respect to accumulated signal before nonlinear display transformations are applied, but black-level offsets, gain, saturation, noise, color-filter responses, and other camera-specific characteristics must be accounted for. Linear image data is not synonymous with photon counts; it is calibrated or calibratable sensor signal.
Gain Maps and Aesthetic Metadata Modern raw-image ecosystems can carry metadata describing transformations, calibration information, or image-rendering behavior separately from the underlying image data. This separation is valuable for scientific workflows because the computational pipeline can choose which transformations are appropriate for measurement and which belong only to visual presentation.
Scientific Stewardship: By keeping measurement-oriented data separate from aesthetic rendering decisions wherever possible, the pipeline avoids allowing display-oriented processing to be confused with the underlying signal used for spectral analysis. The goal is not simply an attractive image; it is a reproducible, characterized measurement from which computational inference can be performed.
Algorithmic Inversion: From 3 Channels to a Dense Spectral Grid Recovering a high-dimensional spectral curve S(λ), potentially comprising hundreds of narrow bands sampled at nanometre-scale intervals, from a low-dimensional RGB input is an ill-posed inverse problem. It is worth being precise about why, because "ill-posed" undersells the situation.
The camera's three spectral sensitivity functions span a three-dimensional subspace of a function space with hundreds of dimensions. Any spectrum lying in the null space of that projection, the metameric blacks, produces identically zero response in all three channels. Not approximately zero: zero. Two spectra differing by a metameric black are the same RGB triple, and no algorithm recovers what the null space swallowed. Reconstruction works instead by ruling out the physically implausible members of that infinite family, using calibration data, physical constraints, statistical priors, and, in learned systems, the distribution represented in the training data. The information loss is a theorem about the projection, not a caveat about implementation quality: no amount of engineering recovers what the null space swallowed. What engineering does determine is how good the remaining guess is, and that depends on how closely the real material resembles what the model was shown.
Wiener Estimation (The Classical Baseline) The classical approach is linear minimum mean-square-error estimation, minimizing expected squared error between estimated and actual spectra through a matrix estimator:
\(W = K_r M^T (M K_r M^T + K_n)^{-1}\)
Here M is the system matrix combining illumination, optical transmission, filter response, and detector sensitivity; Kr is the covariance matrix of the spectral reflectances expected in the sample population; and Kn is the noise covariance. The estimator is optimal among all estimators only when spectra and noise are jointly Gaussian and their second-order statistics are known. Otherwise it is the best linear estimator, which is a materially weaker claim. Kr is where the assumptions hide: the result is only as good as the population you have assumed you are measuring, which is why a Wiener estimator built for one class of material degrades on another.
For decades, linear statistical estimation provided an interpretable route from a low-dimensional observation to an n-band estimate, and it remains a useful reference against which learned methods can be judged. But as a global linear operator it cannot exploit the spatial context or nonlinear relationships that help resolve some of the ambiguity created by spectral compression.
State-of-the-Art: Transformers and Mamba For high-end computational environments, predictive neural architectures can leverage spatial and spectral correlations to resolve ambiguities that simple global linear models cannot. None of them escape the null-space argument above; what they improve is the quality of the prior, not the information content of the measurement. That prior is only as reliable as the distribution on which it was trained. Spectra, illuminants, or material classes that lie outside the training support can produce plausible-looking reconstructions whose errors are invisible to ordinary visual inspection and are not flagged by the network itself. Cross-validation against measured reference spectra under the actual illumination and geometry remains essential; without it the output is an informed guess whose confidence interval is unknown.
MST++ (Spectral Attention Architecture)14: The Multi-stage Spectral-wise Transformer represents a significant development in learned spectral reconstruction. Unlike a single global matrix estimator, MST++ uses attention mechanisms to model relationships between spectral features, allowing the network to learn nonlinear relationships that are difficult to capture with a fixed linear operator. Attention is computationally expensive, particularly as token dimensions and image resolution grow, making memory management important at high resolutions.
State Space Models (Efficient Sequence Processing)15: A newer family of architectures replaces conventional attention with state-space mechanisms. Selective state space models such as Mamba discretize a continuous state-space formulation into a form suitable for efficient sequence processing. Their computational characteristics can be favorable for long sequences, and recent research has adapted Mamba-inspired architectures to spectral reconstruction and spectral compressive imaging.16,17 These approaches are attractive for large images because they model long-range dependencies without relying exclusively on quadratic self-attention. Actual memory and computational cost depends on the architecture and implementation rather than being universally linear in every dimension of the problem.
Multi-Frame Super-Resolution: Approaching the Optical Limit The sampling limits of a single sensor frame can be partly overcome through multi-frame super-resolution compositing, where multiple precisely registered exposures containing sub-pixel shifts are combined into a higher-resolution representation.
What this recovers, precisely, is information above the sampling limit that the optics passed and the pixel grid could not record: aliased and undersampled structure. It cannot exceed the diffraction limit, because that information never arrived at the sensor plane in the first place. The final result remains bounded by the lens's optical transfer function, aberrations, sensor noise, subject and platform motion, registration accuracy, and the information actually present in the source exposures. The objective is not to manufacture detail from nothing, but to make better use of information distributed across multiple measurements.
For spectral work, though, the more valuable gain has nothing to do with resolution. Sub-pixel offsets place different photosites, and therefore different color-filter positions, over the same point on the subject, so the composite can carry genuinely measured R, G, and B values at each output location instead of demosaic-interpolated estimates. Demosaicing invents two of every three color values per pixel, and invents them from neighbouring pixels, which corrupts exactly the per-pixel channel ratios that spectral reconstruction consumes as its input. Eliminating that interpolation matters more to spectral fidelity than the additional pixels do.
These composites preserve the linearity and calibration characteristics of the source data when the processing pipeline is designed appropriately, allowing spectral reconstruction to operate on the resulting representation. The data footprint scales accordingly: floating-point spectral arrays become enormous, and a complete high-resolution multispectral cube may require many gigabytes of storage. The increased spatial detail can enable micro-scale textural analysis that would otherwise be lost, revealing deposition patterns, brushstroke gradients, and surface anomalies at scales approaching the practical resolving capability of the optical system.
Computational Architecture Achieving multispectral precision requires a robust, modular architecture capable of handling massive arrays across a high-dimensional latent space, which in practice means a scientific Python stack running on hardware with substantial memory and bandwidth.
Ingestion: raw decoding is worth one specific warning, because it is a common and silent source of error. Raw decoders do not return linear data by default. Typical defaults apply a gamma curve and automatic brightness scaling, so obtaining sensor-oriented linear values requires disabling those explicitly and handling black-level subtraction as a separate step. Everything downstream inherits whatever was wrong here.
Processing and Analysis: high-performance array libraries handle the matrix algebra required to transform or estimate spectral representations from RGB data, with additional scientific libraries for geometric transforms, image restoration, and spatial filtering, and a plotting layer for spectral signature graphs and false-color composites.
Data Footprint: the scale is significant. A single high-resolution frame converted to floating-point precision produces large files. Intermediate files can exceed hundreds of megabytes for a single three-channel layer, while a full n-band multispectral cube scales proportionally with band count and spatial dimensions. Super-resolution composites multiply those requirements further, producing intermediate arrays that can exceed many gigabytes and necessitating careful memory management, fast storage, and substantial memory bandwidth.
The Spectral Solution Two blue pigments make the argument concrete. Ultramarine and azurite are difficult to separate reliably by eye or by RGB response, and trivially separable once the near-infrared is available, for reasons that come directly from their chemistry:
Spectral Feature
Ultramarine (Lapis Lazuli)
Azurite (Copper Carbonate)
Origin of colour
The S3− radical anion held in the sodalite-type lazurite lattice, producing broad absorption centred near 600 nm
Cu2+ d-d electronic transitions in a basic copper carbonate
Visible-region behaviour
Strong reflectance in the blue-violet region; reduced reflectance through the orange-red where the 600 nm absorption sits
Blue reflectance with a greener bias than ultramarine; spectral shape influenced by particle size, concentration, and binder
Near-infrared behaviour (the discriminator)
Highly reflective through the NIR; remains bright in infrared reflectography
The same Cu2+ transitions produce strong broad absorption extending through the NIR; goes dark in infrared reflectography
Practical requirement
The NIR separation lies outside the roughly 400–650 nm window a stock camera passes, so it depends entirely on the optical and filtration choices discussed above. Inside the visible window alone the two pigments can be genuinely ambiguous.
Note: Measured values vary with pigment composition, particle size, binding medium, concentration, substrate, aging, illumination, and instrument calibration. The NIR contrast between these two is robust because it follows from electronic structure rather than from preparation, but quantitative identification still requires comparison against reference spectra acquired under controlled conditions.
Completing the Picture The successful analysis of complex material properties relies on a convergence of rigorous physics and advanced computation.
Photonic Foundation: A modern back-illuminated CMOS sensor provides high-SNR photonic capture across silicon's usable range, with readout mode driven by which noise regime the exposure actually sits in.
Spectral Access: The accessible range is set by the UV/IR-cut filter first, silicon's 1.1 eV bandgap second, and the optics third. Extending past the visible window is a hardware question, not a software one.
Sensor Diversity: An ingestion layer can normalize black level, white balance, and color filter array geometry across proprietary and standard raw formats. What that produces is format consistency, not radiometric equivalence. Two bodies with different spectral sensitivity functions record different projections of the same spectrum, and harmonizing the containers does not harmonize the measurement. Cross-body comparability requires per-camera SSF characterization, measured against a monochromator or estimated from a calibrated target set, and until a body has been characterized its reconstructions are internally usable but only loosely comparable to another body's.
Data Integrity: A properly characterized raw workflow or linear DNG provides a foundation for preserving scene-referred data, but the file format alone guarantees nothing about radiometric accuracy. Dark, flat, and white-reference frames remain non-optional.
Algorithmic Precision: Wiener estimation remains the classical reference point, while MST++ and Mamba-based architectures model complex nonlinear relationships between image observations and spectral structure. Their output is inference constrained by a learned prior, not recovery of information the camera never measured. Performance outside the training distribution is not guaranteed and must be checked against physical references. Multi-frame compositing improves spatial sampling and, more importantly here, removes demosaic interpolation from the spectral input.
Physical Pattern Analysis: Spectral reconstruction alone cannot resolve every ambiguity. Materials that are spectrally similar can often be separated by their spatial characteristics: texture, edge morphology, and distribution across a surface. Supplementing per-pixel spectral classification with geometric analysis of the spatial domain gives a second, independent axis of evidence, closing a gap that purely spectral methods leave open.
Historical Continuity: The EPR debate of 1935 forced physicists to confront the possibility that a complete description of a system could not be reached through classical intuition about its individual parts. Modern spectral imaging presents a different and far less profound problem that rhymes with it: materials carry structured information across wavelength that is invisible to trichromatic vision. In both cases, completeness requires looking beyond what a single direct observation provides.
Hardware, calibration, file-format stewardship, and reconstruction converge on one thing: a spectral witness to what ordinary vision alone cannot tell us.
And what about the paint? Here is a physical sample: pigment, substrate, history compressed into matter. Light passes through it, scatters from it, carries fragments of its story: yet the full truth remains hidden until we choose to look deeper. Every layer, every faded stroke, every chemical trace is a silent archive. We are not just observers; we are custodians of that archive. When we build tools to see beyond the visible, we are not merely extending sight: we are accepting a quiet responsibility: to bear witness honestly, to preserve what time would erase, to honor what has been made and endured.
Light can expose structure. It cannot carry history.
That part is on us.
We can choose to let the machines we build serve memory rather than erasure, dignity rather than classification, truth rather than convenience. The past does not ask for perfection: it asks only that we refuse to let it be forgotten. In every reconstruction, in every layer we uncover, we have the chance to listen again to what was silenced. That is not just engineering. That is the work of being human.
But tonight, sitting here in the dark, I realize I am still riding that same damn train. Hurtling toward one flash, watching the other recede into a past I can never touch, never text, never warn. Information can race forward, people can race forward, but nothing, nothing gets to go back. We are all just riders, carried away from every moment we have already survived, looking for answers in the one direction the universe actually lets us move. Forward. Always forward. And that is the most beautiful, devastating thing I know.
References
1 Hafele, J. C., & Keating, R. E. (1972). Around-the-World Atomic Clocks: Predicted Relativistic Time Gains; Observed Relativistic Time Gains. Science, 177(4044), 166–168; 168–170.
2 Rossi, B., & Hall, D. B. (1941). Variation of the Rate of Decay of Mesotrons with Momentum. Physical Review, 59(3), 223–228.
3 Gödel, K. (1949). An Example of a New Type of Cosmological Solution of Einstein's Field Equations of Gravitation. Reviews of Modern Physics, 21(3), 447–450.
4 Morris, M. S., & Thorne, K. S. (1988). Wormholes in spacetime and their use for interstellar travel: A tool for teaching general relativity. American Journal of Physics, 56(5), 395–412.
5 Hawking, S. W. (1992). Chronology protection conjecture. Physical Review D, 46(2), 603–611.
6 Einstein, A., Podolsky, B., & Rosen, N. (1935). Can Quantum-Mechanical Description of Physical Reality Be Considered Complete? Physical Review, 47(10), 777–780.
7 Bohm, D. (1951). Quantum Theory. Prentice-Hall. (Spin reformulation of the EPR argument.)
8 Schrödinger, E. (1935). Discussion of Probability Relations between Separated Systems. Proceedings of the Cambridge Philosophical Society, 31(4), 555–563.
9 Bell, J. S. (1964). On the Einstein Podolsky Rosen paradox. Physics Physique Физика, 1(3), 195–200.
10 Aspect, A., Dalibard, J., & Roger, G. (1982). Experimental Test of Bell's Inequalities Using Time-Varying Analyzers. Physical Review Letters, 49(25), 1804–1807.
11 Hensen, B., et al. (2015). Loophole-free Bell inequality violation using electron spins separated by 1.3 kilometres. Nature, 526, 682–686.
12 Giustina, M., et al. (2015). Significant-Loophole-Free Test of Bell's Theorem with Entangled Photons. Physical Review Letters, 115, 250401.
13 Shalm, L. K., et al. (2015). Strong Loophole-Free Test of Local Realism. Physical Review Letters, 115, 250402.
14 Cai, Y., Lin, J., Lin, Z., Wang, H., Zhang, Y., Pfister, H., Timofte, R., & Van Gool, L. (2022). MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
15 Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
16 Zhang, Y., Li, L., Lin, Q., Ming, Z., Yu, F., & Leung, V. C. M. M3SR: Multi-Scale Multi-Perceptual Mamba for Efficient Spectral Reconstruction. [venue and year to be completed]
17 Qin, M., Feng, Y., Wu, Z., Zhang, Y., & Yuan, X. Detail Matters: Mamba-Inspired Joint Unfolding Network for Snapshot Spectral Compressive Imaging. [venue and year to be completed]
Provenance · Integrity Record
Canonical file hashes · SHA‑512
Hashes are of the byte-identical JPEGs converted from raw sensor data. Verify with sha512sum -c SHA512SUMS.
The DE1-SoC from Terasic is an excellent board for hardware design and prototyping. The following VHDL process is from a hardware design created for the Terasic DE1-SoC FPGA. The ten switches and four buttons on the FPGA are used as an n-bit counter with an adjustable multiplier to increase the output frequency of one or more output pins at a 50% duty cycle.
As the switches are moved or the buttons are pressed, the seven-segment display is updated to reflect the numeric output frequency, and the output pin(s) are driven at the desired frequency. The onboard clock runs at 50MHz, and the signal on the output pins is set on the rising edge of the clock input signal (positive edge-triggered). At 50MHz, the output pins can be toggled at a maximum rate of 50 million cycles per second or 25 million rising edges of the clock per second. An LED attached to one of the output pins would blink 25 million times per second, not recognizable to the human eye. The persistence of vision, which is the time the human eye retains an image after it disappears from view, is approximately 1/16th of a second. Therefore, an LED blinking at 25 million times per second would appear as a continuous light to the human eye.
scaler <= compute_prescaler((to_integer(unsigned( SW )))*scaler_mlt); gpiopulse_process : process(CLOCK_50, KEY(0)) begin if (KEY(0) = '0') then -- async reset count <= 0; elsif rising_edge(CLOCK_50) then if (count = scaler - 1) then state <= not state; count <= 0; elsif (count = clk50divider) then -- auto reset count <= 0; else count <= count + 1; end if; end if; end process gpiopulse_process;
The scaler signal is calculated using the compute_prescaler function, which takes the value of a switch (SW) as an input, multiplies it with a multiplier (scaler_mlt), and then converts it to an integer using to_integer. This scaler signal is used to control the frequency of the pulse signal generated on the output pin.
The gpiopulse_process process is triggered by a rising edge of the CLOCK_50 signal and a push-button (KEY(0)) press. It includes an asynchronous reset when KEY(0) is pressed.
The count signal is incremented on each rising edge of the CLOCK_50 signal until it reaches the value of scaler - 1. When this happens, the state signal is inverted and count is reset to 0. If count reaches the value of clk50divider, it is also reset to 0.
Overall, this code generates a pulse signal with a frequency controlled by the value of a switch and a multiplier, which is generated on a specific output pin of the FPGA board. The pulse signal is toggled between two states at a frequency determined by the scaler signal.
It is important to note that concurrent statements within an architecture are executed concurrently, meaning that they are evaluated concurrently and in no particular order. However, the sequential statements within a process are executed sequentially, meaning that they are evaluated in order, one at a time. Processes themselves are executed concurrently with other processes, and each process has its own execution context.
The 96boards CE specification calls for an optional JTAG connection. The specification also indicates that the optional JTAG connection shall use a 10 pin through hole, .05" (1.27mm) pitch JTAG connector. The part is readily available on most electronics sites. Breaking out the pins with long wires and shrink wrapping them is ideal for making sure that each connection is labeled and separate when connecting to a JTAG debugger. While a JTAG connection is not required for flashing or loading the bootloaders onto the board, the JTAG connection is useful for advanced chip-level debugging. The serial UART connection is sufficient for loading release or debug versions of bl0, bl1, bl2, bl31, bl32, the kernel, and userspace. Last but not least, ARM-powered boards, with 12V power input, often require external fans to keep the board cool. As seen in the below photos, two 5V fans were powered from an external power supply. Any work on microcontroller boards should be performed on a grounded surface. Proper grounding procedures should always be followed as most microcontroller boards contain ESD sensitive components.
In the below photos, a 96Boards SBC is mounted on an IP65, ABS plastic junction box for durability. The pins are extended and mounted with screws underneath the junction box. The electrical conduit holes on the side of the junction box are ideal for holding small, project fans. The remaining electrical conduit holes provide a clean place to place the remaining wires from the board - micro USB, USB-C, and 12V power.
The Kirin 960 SoC and on-board USB 3.0 make the HiKey 960 SBC an ideal platform for running a Linux Bridged firewall. The number of single-board computers with an SoC as powerful as the HiSilicon Kirin 960 are limited.
When compared with the Raspberry Pi series of single board computers (SBC), the HiKey 960 SBC is significantly more powerful. The Kirin 960 also stands above the ARM powered SoCs which reside in most commercial routers.
USB 3.0 makes the HiKey 960 board an attractive option for bridging or routing, filtering network traffic, or connecting to an external gateway via IPSec. Both network traffic filtering and IPSec tunneling can be computationally expensive operations. However; the multicore Kirin 960 is well suited for these types of tasks.
In order to be able to run an IPSec client tunnel and a Linux Bridged firewall connected over 1G ethernet links, certain kernel configuration modifications are needed. Furthermore, the Android Linux kernel for the HiKey 960 board does not boot on a standard Linux root filesystem because it is designed to boot an Android customized rootfs.
The latest googlesource Linux kernel (hikey-linaro-4.9) for Android (designed to boot Android on the HiKey 960 board) has been customized to remove the Android specific components so that the kernel boots on a standard Linux root filesystem, with the proper drivers enabled for network connectivity via attached 1000Mb/s USB 3.0 to ethernet adapters. The standard UART interface on the board should be used for serial connectivity and shell access. WiFi and Bluetooth have been removed from the kernel configuration. The kernel should be booted off of a microSDHC UHS-I card. The 96boards instructions should be followed for configuring the HiKey 960 board, setting the jumpers on the board, building and flashing the l-loader, firmware package, partition tables, UEFI loader, ARM Trusted Firmware, and optional Op-TEE. Links for the normal Linux kernel configuration, multi-interface bridge configuration, and single interface IPSec configuration are below. Additional kernel config modifications may be needed for certain types of applications.
kernel build instructions
mkdir /usr/local/toolchains
cd /usr/local/toolchains/
wget https://releases.linaro.org/components/toolchain/binaries/latest/aarch64-linux-gnu/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu.tar.xz
tar -xJf gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu.tar.xz
export ARCH=arm64
export CROSS_COMPILE=/usr/local/toolchains/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export PATH=/usr/local/toolchains/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/gcc-aarch64-linux-gnu/bin:$PATH
cd /usr/local/src
git clone https://android.googlesource.com/kernel/hikey-linaro
cd hikey-linaro
git checkout -b android-hikey-linaro-4.9
make hikey960_defconfig
make -j8
multi-interface bridge configuration
Bridged configuration, no ip addresses on dual nic interfaces. (crossover cable is useful for testing). Bridge interface obtains dhcp address(/11) from wlan router. aliased interface added to br0 and assigned private subnet ip on different subnet (/8). Spanning tree set on bridge interface. Basic ebtables and iptables ruleset below.
brctl addbr <br>
brctl addif <br> <eth1> <eth2>
ifconfig <br> up
ifconfig <eth1> up
ifconfig <eth2> up
brctl stp <br> yes
dhclient <br>
ifconfig <br>:0 <a.b.c.d/sn> up
iptables --table nat --append POSTROUTING --out-interface <br> -j MASQUERADE
iptables -P INPUT DROP
iptables --append FORWARD --in-interface <br>:0 -j ACCEPT
ebtables -P FORWARD DROP
ebtables -P INPUT DROP
ebtables -P OUTPUT DROP
ebtables -t filter -A FORWARD -p IPv4 -j ACCEPT
ebtables -t filter -A INPUT -p IPv4 -j ACCEPT
ebtables -t filter -A OUTPUT -p IPv4 -j ACCEPT
ebtables -t filter -A INPUT -p ARP -j ACCEPT
ebtables -t filter -A OUTPUT -p ARP -j ACCEPT
ebtables -t filter -A FORWARD -p ARP -j REJECT
ebtables -t filter -A FORWARD -p IPv6 -j DROP
ebtables -t filter -A FORWARD -d Multicast -j DROP
ebtables -t filter -A FORWARD -p X25 -j DROP
ebtables -t filter -A FORWARD -p FR_ARP -j DROP
ebtables -t filter -A FORWARD -p BPQ -j DROP
ebtables -t filter -A FORWARD -p DEC -j DROP
ebtables -t filter -A FORWARD -p DNA_DL -j DROP
ebtables -t filter -A FORWARD -p DNA_RC -j DROP
ebtables -t filter -A FORWARD -p LAT -j DROP
ebtables -t filter -A FORWARD -p DIAG -j DROP
ebtables -t filter -A FORWARD -p CUST -j DROP
ebtables -t filter -A FORWARD -p SCA -j DROP
ebtables -t filter -A FORWARD -p TEB -j DROP
ebtables -t filter -A FORWARD -p RAW_FR -j DROP
ebtables -t filter -A FORWARD -p AARP -j DROP
ebtables -t filter -A FORWARD -p ATALK -j DROP
ebtables -t filter -A FORWARD -p 802_1Q -j DROP
ebtables -t filter -A FORWARD -p IPX -j DROP
ebtables -t filter -A FORWARD -p NetBEUI -j DROP
ebtables -t filter -A FORWARD -p PPP -j DROP
ebtables -t filter -A FORWARD -p ATMMPOA -j DROP
ebtables -t filter -A FORWARD -p PPP_DISC -j DROP
ebtables -t filter -A FORWARD -p PPP_SES -j DROP
ebtables -t filter -A FORWARD -p ATMFATE -j DROP
ebtables -t filter -A FORWARD -p LOOP -j DROP
ebtables -t filter -A FORWARD --log-level info --log-ip --log-prefix FFWLOG
ebtables -t filter -A OUTPUT --log-level info --log-ip --log-arp --log-prefix OFWLOG -j DROP
ebtables -t filter -A INPUT --log-level info --log-ip --log-prefix IFWLOG
single-interface ipsec gateway configuration
iptables -t nat -A POSTROUTING -s <clientip>/32 -o <eth> -j SNAT --to-source <virtualip>
iptables -t nat -A POSTROUTING -s <clientip>/32 -o <eth> -m policy --dir out --pol ipsec -j ACCEPT
XOR logic gates are a fundamental component in cryptography, and many of the typical stream and block ciphers use XOR gates. A few of these ciphers are ChaCha (stream cipher), AES (block cipher), and RSA (block cipher).
While many compiled and interpreted languages support bitwise operations such as XOR, the software implementation of both block and stream ciphers is computationally inefficient compared to FPGA and ASIC implementations.
Hybrid FPGA boards integrate FPGAs with multicore ARM and Intel application processors over high-speed buses. The ARM and Intel processors are general-purpose processors. On a hybrid board, the ARM or Intel processor is termed the hard processor system or HPS. Writing to the FPGA from the HPS is typically performed via C from an embedded Linux build (yocto or buildroot) running on the ARM or Intel core. A simple bitstream can also be loaded into the FPGA fabric without using any ARM design blocks or functionality in the ARM core for a hybrid ARM configuration.
The following is a simple hardware design written in VHDL and simulated in ModelSim. The image contains the waveform output of a simulation in ModelSim. The HPS is not used. On boot, the bitstream is loaded into the FPGA fabric. VHDL components are utilized, and a testbench is defined for testing the design. The entity and architecture VHDL design units are below.
- --three input xnor gate entity declaration - external interface to design entity entity xnorgate is port ( a,b,c : in std_logic; q : out std_logic); end xnorgate;
architecture xng of xnorgate is begin q <= a xnor b xnor c; end xng;
- --chain of xor / xnor gates using components and sequential logic entity xorchain is port ( A,B,C,D,E,F : in std_logic; Av,Bv : in std_logic_vector(31 downto 0); CLOCK_50 : in std_logic; Q : out std_logic; Qv : out std_logic_vector(31 downto 0)); end xorchain;
architecture rtl of xorchain is component xorgate is port ( a,b : in std_logic; q : out std_logic); end component;
component xnorgate is port ( a,b,c : in std_logic; q : out std_logic); end component;
component xorsgate is port ( av : in std_logic_vector(31 downto 0); bv : in std_logic_vector(31 downto 0); qv : out std_logic_vector(31 downto 0)); end component;
signal a_in, b_in, c_in, d_in, e_in, f_in : std_logic; signal av_in, bv_in : std_logic_vector(31 downto 0);
signal conn1, conn2, conn3 : std_logic;
begin xorgt1 : xorgate port map(a => a_in, b => b_in, q => conn1); xorgt2 : xorgate port map(a => c_in, b => d_in, q => conn2); xorgt3 : xorgate port map(a => e_in, b => f_in, q => conn3); xnorgt1 : xnorgate port map(conn1, conn2, conn3, Q); xorsgt1 : xorsgate port map(av => av_in, bv => bv_in, qv => Qv);
process(CLOCK_50) begin if rising_edge(CLOCK_50) then --assign inputs on rising clock edge a_in <= A; b_in <= B; c_in <= C; d_in <= D; e_in <= E; f_in <= F; av_in(31 downto 0) <= Av(31 downto 0); bv_in(31 downto 0) <= Bv(31 downto 0); end if;
end process; end rtl;
entity xorchain_tb is end xorchain_tb;
architecture xorchain_tb_arch of xorchain_tb is signal A_in,B_in,C_in,D_in,E_in,F_in : std_logic := '0'; signal Av_in : std_logic_vector(31 downto 0); signal Bv_in : std_logic_vector(31 downto 0); signal CLOCK_50_in : std_logic; signal BRK : boolean := FALSE; signal Q_out : std_logic; signal Qv_out : std_logic_vector(31 downto 0);
component xorchain port ( A,B,C,D,E,F : in std_logic; Av : in std_logic_vector(31 downto 0); Bv : in std_logic_vector(31 downto 0); CLOCK_50 : in std_logic; Q : out std_logic; Qv : out std_logic_vector(31 downto 0)); end component;
begin xorchain_instance: xorchain port map (A => A_in,B => B_in, C => C_in, D => D_in, E => E_in, F => F_in, Av => Av_in, Bv => Bv_in, CLOCK_50 => CLOCK_50_in, Q => Q_out, Qv => Qv_out); clockprocess: process begin while not BRK loop CLOCK_50_in <= '0'; wait for 20 ns; CLOCK_50_in <= '1'; wait for 20 ns; end loop; wait; end process clockprocess;
testprocess : process begin A_in <= '1'; B_in <= '0'; C_in <= '1'; D_in <= '0'; E_in <= '1'; F_in <= '1'; wait for 40 ns; A_in <= '1'; B_in <= '0'; C_in <= '1'; D_in <= '0'; E_in <= '1'; F_in <= '0'; wait for 20 ns; A_in <= '0'; B_in <= '0'; C_in <= '1'; D_in <= '0'; E_in <= '1'; F_in <= '0'; wait for 40 ns; BRK <= TRUE; wait; end process testprocess; end xorchain_tb_arch;
entity xorgate is port ( a,b : in std_logic; q : out std_logic); end xorgate;
architecture xg of xorgate is begin q <= a xor b; end xg;
entity xorsgate is port ( av : in std_logic_vector(31 downto 0); bv : in std_logic_vector(31 downto 0); qv : out std_logic_vector(31 downto 0)); end xorsgate;
architecture xsg of xorsgate is begin qv <= av xor bv; end xsg;
GNURadio Companion Qt Gui Frequency Sync - multiple FIR filter taps sample running on Raspberry Pi 3 custom Linux distribution
The Raspberry Pi 3 is powered by the ARM Cortex-A53 processor. This 1.2GHz 64-bit quad-core processor fully supports the ARMv8-A architecture. For this project, a custom Linux distribution was created for the Raspberry Pi 3.
The custom Linux distribution includes support for GNURadio, several FPGA and ARM Powered SDR devices, D-STAR (hotspot, repeater, and dongle support), hsuart, libusb, hardware real-time clock support, Sony 14 megapixel NoIR image sensor, HDMI and 3.5mm audio, USB Microphone input, X-windows with Xfce, Lighttpd and PHP, Bluetooth, WiFi, SSH, TCPDump, Docker, Docker registry, MySQL, Perl, Python, QT, GTK, IPTables, x11vnc, SELinux, and full native-toolchain development support.
The Sony 14 megapixel image sensor with the infrared filter removed can be connected to the Raspberry Pi 3's MIPI camera serial interface. Image capture and recognition can then be performed over contiguous periods of time, and time-lapsed video can be created from the images. With support for Tensorflow and OpenCV, object recognition within images can be performed.
D-STAR hotspot with time-lapsed infrared imaging.
For the initial run, an infrared Time-lapse Video was created from an initial image capture run of one 3280x2460 infrared jpeg image captured every 15 seconds for three hours. 40, 5mm, 940nm LEDs, powered by 500ma over 12v DC, provided infrared illumination in the 940nm wavelength.
Tensorflow ran in the background (on v4l2 kmod) and provided continuous object recognition and scoring within each image via a sample model. Finally, OpenCV was also installed in the root file system.
The time-lapse infrared video was captured of the living room using the above setup. Below this image are images of Tensorflow running in a terminal in the background on the Raspberry Pi 3 and recognizing/scoring objects in the living room.
Tensorflow running on the Raspberry Pi 3 and continuously capturing frames from the image sensor and scoring objects
GNURadio Companion running on xfce on the Raspberry Pi 3
The ARM DS-5 Streamline Performance Analyzer is a powerful tool for debugging, profiling, and analyzing multithreaded and multiprocess C programs. Instructions can easily be traced between load and store operations. Per process and per thread function call paths can be broken down by system utilization percentage. Branch mispredictions and multi-level CPU caches can be analyzed. Furthermore, disk I/O usage, stack and heap usage, and a number of other useful metrics can quickly be referenced within the debugger. These are just a few of its capabilities.
In order to capture meaningful information from the DS-5 Streamline Performance Analyzer tool, a Linux, multiprocess, C program was modified to insert 1000 packets into a packet processing simulation buffer. A code excerpt from the program is below. The child processes were modified to sleep and then wake 1000 times in order to simulate process activity. The program was analyzed using the DS-5 Streamline Performance Analyzer tool. There are two screenshots below the code excerpt where the program is loaded into the DS-5 Streamline Performance Analyzer.
void *insertpackets(void *arg) {
struct pktbuf *pkbuf; struct packet *pkt; int idx;
if(arg != NULL) {
pkbuf = (struct pktbuf *)arg;
/* seed random number generator */ ...
/* insert 1000 packets into the packet buffer */ for(idx = 0; idx < 1000; ++idx) {
DE1-SoC GPIO Pins connected to 780nm Infrared Laser Diodes, 660nm Red Laser Diodes, and Oscilloscope
The following VHDL processes pulse the GPIO pins at different frequencies on the Altera DE1-SoC using multiple Phase-Locked Loops. Several diodes were connected to the GPIO banks and pulsed at a 50% duty cycle with 16mA across 3.3V. Each GPIO bank on the DE1-SoC has 36 pins. Pin 1 is pulsed at 20Hz from GPIO bank 0, and pins 0 and 1 are pulsed at 30Hz from GPIO bank 1. A direct mode PLL with locked output was configured using the Altera Quartus Prime MegaWizard. The PLL reference clock frequency is set to 50MHz, the output clock frequency is set to 50MHz, and the duty cycle is set to 50%. The pin mappings for GPIO banks 0 and 1 are documented on the DE1-SoC datasheet.
Pulsed Laser Diodes via GPIO pins on DE1-SoC FPGA
- -- --------------------- - -- CLOCK A AND B PROCESSES -- - -- INPUT: direct mode pll with locked output - -- and reference clock frequency set to 50MHz, - -- output clock frequency set to 50MHz with 50% duty - -- cycle and output frequency scaled by freq divider constant - -- -----------------------------------------------------------
clk_a_process : process (lkd_pll_clk_a) begin if rising_edge(lkd_pll_clk_a) then if (cycle_ctr_a < FREQ_A_DIVIDER) then cycle_ctr_a <= cycle_ctr_a + 1; else cycle_ctr_a <= 0; end if; end if; end process clk_a_process;
clk_b_process : process (lkd_pll_clk_b) begin if rising_edge(lkd_pll_clk_b) then if (cycle_ctr_b < FREQ_B_DIVIDER) then cycle_ctr_b <= cycle_ctr_b + 1; else cycle_ctr_b <= 0; end if; end if; end process clk_b_process;
- -- --------------------- - -- GPIO A AND B PROCESSES -- - -- INPUT: direct mode pll with locked output - -- ------------------------------------------------------- gpio_a_process : process (lkd_pll_clk_a) begin if rising_edge(lkd_pll_clk_a) then if (cycle_ctr_a = 0) then gpio_sig_0 <= NOT gpio_sig_0; end if; end if; end process gpio_a_process;
gpio_b_process : process (lkd_pll_clk_b) begin if rising_edge(lkd_pll_clk_b) then if (cycle_ctr_b = 0) then gpio_sig_1 <= NOT gpio_sig_1; end if; end if; end process gpio_b_process; GPIO_0 <= gpio_sig_0; GPIO_1 <= gpio_sig_1;
The DE1-SoC FPGA Development board from Terasic is powered by an integrated Altera Cyclone V FPGA and ARM MPCore Cortex-A9 processor. The FPGA and ARM core are connected by a high-speed interconnect fabric. Linux can be booted on the ARM core and the FPGA and ARM core can communicate.
The DE1-SoC board below has been programmed via Quartus Prime running on Fedora 23, 64-bit Linux. The FPGA bitstream was compiled from the Terasic Audio codec design reference. After the bitstream was loaded on to the FPGA over the USB blaster II interface, the NIOS II command shell was used to load the NIOS II software image onto the chip. A menu-driven, debug interface is running from a terminal on the host via the NIOS II shell with the target connected over the USB Blaster II interface.
A low-level hardware abstraction layer was programmed in C to configure the on-board audio codec chip. The NIOS II chip is stored in on-chip memory and a PLL driven, clock signal is fed into the audio chip. The Verilog code for the hardware design was generated from Qsys. The design supports configurable sample rates, mic in, and line in/out.
Additional components are connected to the DE1-SoC board in this photo. The Linear DC934A (LTC2607) DAC is connected to the DE1-SoC and an oscilloscope is connected to the ground and vref pins on the DAC.
The DC934A features an LTC2607 16-Bit Dual DAC with i2c interface and an LTC2422 2-Channel 20-Bit uPower No Latency Delta Sigma ADC.
3.5mm audio cables are connected to the mic in and line out ports, respectively. The DE1-SoC is connected to an external display over VGA so that a local console can be managed via a connected keyboard and mouse when Linux is booted from uSD.
With GPIO pins accessible via the GPIO 0 and 1 breakouts, external LEDs can be pulsed directly from the Hard Processor System (HPS), FPGA, or the FPGA via the HPS.
Understanding the boot loader on a computer system is probably the most important aspect of security. Most computer systems have multiple boot loaders that run in sequence immediately after a power reset is applied to the processor on the computer system. This applies to embedded, desktop, and server systems.
The Altera Cyclone V SoC has an FPGA and a Hard Processor System (HPS) woven into a single processor package. The HPS is a dual core ARM Cortex A9. Building everything from scratch is the best way to figure out how the system works.
The boot sequence on a Cyclone V HPS works like this:
The On-chip ROM (for which source code is not provided) loads the preloader (1st stage bootloader). The preloader then loads U-boot. U-boot then loads the kernel and root file system.
There are two well thought out options for the preloader according to the Cyclone V boot guide. The two options are licensed differently depending on how the source code is built. One is licensed under a BSD license and the other under GPL v2 with U-Boot.
Building a pre-loader image for the DE0-Nano-SoC board was straightforward. Altera provides the bsp-editor utility for customizing the preloader configuration and generating the BSP HPS preloader source code, after which, make is used to build the sources using the Mentor ARM cross toolchain. The preloader settings directory can be found on the DE0-Nano-SoC CD in the DE0_NANO_SOC_GHRD subdirectory.
The preloader load address can be set via the bsp-editor so that the on chip ROM either loads the preloader from an absolute zero address on the sdcard or from a fat partition with id equal to a2 on the sdcard. These are the options for booting from the sdcard.
After the sources are generated and the preloader image is built using the Makefile, U-boot must be compiled. An Altera port of U-Boot is available on github for the Cyclone V FPGA SoC. U-Boot is built using the Linaro ARM cross toolchain.
There's quite a bit that can be done with the Cyclone V FPGA SoC boot configuration. FPGA images can be loaded from U-boot. The jumpers on the board can be configured to boot from the on-board serial flash (QSPI), bare metal applications can be loaded from the preloader, the FPGA can be configured from serial flash, and the list goes on. The HPS SoC Boot Guide for the Cyclone V SoC is a valuable reference and contains all of the boot configuration information.
