The sky over the Pi, right now

June 30, 2026

The image above is the current presentation photo from SkyBox. The palette above is the current color measurement. Those sound like the same thing, but in the current system they are intentionally separate.

The photo is for humans. It should look like a normal camera image: usable exposure, auto white balance, denoise, and enough processing that I can tell where the camera is pointed.

The measurement path is for color. It uses raw bracketed captures, fixed exposure settings, fixed white balance gains, no denoise, and a small linear-light merge. It is not trying to make a pretty HDR photograph. It is trying to estimate the color of the sky region in a repeatable way.

That split exists because the earlier version tried to make one camera path serve both jobs. It produced images that were technically derived from the raw data, but the visible snapshot got too dark and noisy. The measurement process was clobbering the presentation image. The current version keeps the measurement pipeline strict and lets the displayed image be a camera photo again.

The current hardware shape

The running setup is a Raspberry Pi Zero serving the site and running SkyBox. The camera is a Raspberry Pi camera sensor with a fisheye lens, currently aimed through or near a window so it can see the sky. There is also a small Mini PiTFT display on the Pi for status and aiming.

The Pi hosts two related things:

• the static garden site, served by nginx
• the SkyBox service, listening locally on port 8080

nginx exposes only the small public surface the garden needs:

/sky.css       generated live theme CSS
/sky.json      generated live color JSON
/sky/snapshot  latest human-readable camera JPEG

SkyBox itself also has a debug UI and API on the Pi. The garden does not need the full UI; it just needs the snapshot, palette, and generated stylesheet.

There are two capture paths

The first path is the human snapshot.

When the garden requests /sky/snapshot, SkyBox returns the latest JPEG it has from the presentation path. On the Pi, that path uses rpicam-still with normal camera processing enabled:

auto AWB
average metering
EV -2
HDR auto
denoise auto
sharpness 1
saturation 1
JPEG quality 95

The negative exposure compensation is deliberate. A sky-through-window scene can blow out easily, and once the sky is clipped the preview stops being useful. The snapshot does not have to be scientifically neutral. It has to show me what the camera currently sees.

The second path is the raw measurement.

The deployed SkyBox service is running in --raw-measure mode with a bracket like:

250 us, 2000 us, 16000 us

Each exposure is captured as a raw DNG using rpicam-still, with analog gain held at 1.0, denoise off, and fixed white-balance gains. The raw mode is currently 1536:864:10, so the code is working with 10-bit Bayer data rather than a processed JPEG.

This raw path is the one that feeds the palette.

What the raw merge is doing

The merge is closer to radiance estimation than normal photographic HDR.

Normal HDR photo software usually tries to produce a good-looking image. It aligns exposures if needed, estimates a camera response curve, merges the images into a high dynamic range representation, then tone maps that range back down to something a display can show. Local contrast, highlight rolloff, and perceptual appearance matter because the output is a photograph.

SkyBox is not doing all of that. It has a static camera and a simpler goal: estimate sky color without letting clipped highlights or short-exposure noise dominate the answer.

For each raw frame, the code parses the DNG, reads the black level and white level, and unpacks the raw samples. It then bins the sensor into 16 x 16 raw blocks. With the current raw mode, that becomes a 96 x 54 measurement grid, or 5184 superpixels.

Inside each bin, the code keeps the Bayer channels separate:

• red sites
• green sites
• blue sites

For each raw site, it looks through the bracket from longest exposure to shortest exposure and chooses the longest sample that is not clipped. Long exposures have better signal-to-noise, so the code prefers them when possible. If the long exposure is clipped, it falls back to the next shorter exposure.

Then it normalizes the selected sample:

value = (raw - black) / (white - black)
radiance ~= value / exposure_time

The result is scaled back to the reference exposure, averaged by Bayer channel inside the bin, white-balanced, and passed through the camera color correction matrix into linear sRGB.

There is also a saturation guard. If a bin is blown out even in the shortest exposure, SkyBox desaturates that bin to neutral white instead of letting clipped Bayer channels turn it magenta.

That is the practical HDR part: use the longest unclipped raw evidence, avoid clipped color, and keep the result in linear light long enough to measure it.

Why the raw result needed a presentation lift

The raw merge produces small linear values. That is a good property for measurement, but it can make the derived preview and palette look extremely dark if the scale is not calibrated.

Right now SkyBox separates the raw measurement grid from the presentation grid.

The raw grid is kept for measurement-ish camera metrics. The presentation grid is a lifted copy used for palette extraction and fallback preview. The lift looks at the 95th-percentile luminance and scales the grid so that value lands around 0.75, with a maximum scale limit and a hard clamp at 1.0.

In code terms, it is roughly:

p95_luma -> scale toward 0.75 -> clamp -> palette

This is not a final calibration. It is a practical correction for the fact that the raw camera pipeline currently has no independent lux reference. It makes the palette usable while keeping the original raw values available for later calibration.

The better version is to add a real light/color sensor. An OPT4048 is one candidate. Another option is an AS7341 for spectral-ish color plus a TSL2591 for high dynamic range lux. The camera can keep providing the shape of the sky and relative color variation; the sensor can correct the overall brightness and color.

Only part of the image is sky

The fisheye frame includes more than sky. Depending on the exact position, it can include porch, window reflection, lens rim, house edge, trees, and whatever else the camera sees around the sky sector.

The palette should not sample all of that.

Right now the code uses a fixed normalized mask for the current camera placement. It keeps the right/center part of the fisheye image where the sky is visible and excludes the obvious non-sky regions. The last useful mask size was around 806 sky samples out of 5184 grid cells.

That number is useful because it shows what the mask is actually doing. A full-frame average would include a lot of non-sky material. A sky-only mask gives the palette a chance to describe the sky rather than the room, window, and lens edge.

The limitation is obvious: this is not general sky segmentation. It is a calibrated crop for the current camera pose. If the camera moves, the mask may need to move with it.

A more robust future version could do one or more of these:

• store a calibration mask per camera position
• let the debug UI paint or adjust the mask
• use the fisheye geometry to define sky-relative regions
• segment sky dynamically from the image
• combine image segmentation with a known horizon/lens model

For now, the fixed mask is acceptable because the camera is meant to be static.

The purple fringe correction is estimated

The fisheye/window setup has a slight purplish fringe near parts of the frame. That is probably some mix of lens shading, chromatic behavior near the edge, reflections, and the current viewing geometry.

SkyBox now has an estimated neutral correction for that. It is deliberately narrow in scope:

• it applies only to palette samples
• it leaves the center sky unchanged
• it increases toward the edge of the masked region
• it slightly reduces red and blue while increasing green near the rim

This is not a real flat-field correction yet. It is a placeholder based on what the image appeared to be doing.

The proper calibration would be a neutral reference image. For example, point the camera at an evenly lit neutral surface, or capture a known diffuse neutral target under stable light. From that, SkyBox could build a per-position correction map:

observed neutral frame -> per-cell RGB correction -> saved calibration -> apply before palette extraction

That would correct lens tint and vignetting with evidence rather than hand-tuned constants. The current correction just keeps the known purple edge from dominating the palette while the hardware is still being assembled.

How the palette is chosen

Once SkyBox has a corrected presentation grid for the sky region, it runs k-means clustering over the linear RGB samples. The target is five representative colors.

Those five colors are converted to OKLCh. From there, the code names three useful regions:

• zenith: the brightest representative color
• nadir: the darkest representative color
• horizon: the most colorful remaining representative color

Those names are semantic, not literal geometry. With an ideal all-sky camera, "zenith" would mean straight up and "horizon" would mean low sky. In the current implementation, they mean "the cluster that behaves like the bright top of the sky," "the cluster that behaves like the dark lower part," and "the cluster with the strongest chroma."

The color temperature is estimated from the mean linear color. It is a useful label, not a lab-grade measurement.

That final color message is what the garden receives:

zenith + horizon + nadir + five-color palette + CCT

The garden then turns that into CSS variables, as described in This page is the color of the sky.

The display can interrupt capture

The Mini PiTFT has a live camera view for aiming the camera. That view is useful, but it also grabs the camera continuously.

On the Pi camera stack, only one process can own the camera cleanly at a time. If the display live view is running forever, normal captures can stop updating. SkyBox now treats live view as temporary: it pauses capture while live view is active, then times it out after about 30 seconds and returns to status mode.

That is a small operational detail, but it matters. I was using the display to aim the camera, then wondering why the browser stopped updating. The answer was that the aiming tool still had the camera.

What still needs real calibration

This is now a cleaner system, but it is not finished.

The current version has the right separation of concerns:

• JPEG photo for human inspection
• raw bracket for repeatable color measurement
• fixed sky mask for the current camera position
• estimated fringe correction for the current lens/window setup
• generated CSS/JSON for the garden

The parts that still need work are the calibration pieces:

• add a lux/color sensor and use it as the absolute reference
• capture a neutral flat-field frame for lens and window correction
• make the sky mask easier to adjust after the camera moves
• decide how much temporal smoothing belongs in SkyBox versus the site layer
• keep the photo path useful without confusing it for the measurement path

The direction I want is: camera for geometry and relative palette, sensor for absolute correction.

That keeps the best part of the camera, which is that it can see the whole sky shape, while admitting that a camera alone is a slippery color sensor. Exposure, lens tint, white balance, window glass, and clipping all leak into the result unless there is something external to anchor it.

So the current SkyBox pipeline is best understood as a working prototype of the visual side. It can see the sky, isolate the current sky region, and publish a palette. The next step is making that palette physically better grounded.

#meta #skybox