WritingExperimental design

Five OCT pretraining recipes, and the choices behind them

The five AS-OCT recipes I implemented on 3.43 million images: their inputs, objectives, shared controls, and compute tradeoffs.

In this article

I implemented five AS-OCT pretraining recipes to study how image context and the prediction target shape an encoder’s representation. They share an image index, source order and encoder initialization. Within that common setup, I vary where the model looks and what it learns to predict.

The choices are concrete. Full-field MAE reconstructs hidden pixels across the image. Two matched arms spend the same limited patch budget in different locations. DiffMAE reconstructs noisy targets using visible context. JEPA relates global and local views in representation space.

This article records the configurations established on 20 September 2026, including the matched-patch settings reviewed on 23 September. The experiments have since produced distinct feature spectra: concentration in the matched-patch arms, broader variation in full-field MAE, and the broadest recorded spectrum in JEPA.

Define the source image and the training budget

Optical coherence tomography, or OCT, produces cross-sectional images called B-scans. An examination contains several B-scans, and one person may have several examinations. Patches and augmented views are derived from these source images.

I assembled a frozen June 2026 research corpus of 3,663,454 OCT cross-sections. Selecting the pretraining population and reserving images for engineering checks leaves 3,475,579 for quality scoring. The shared quality filter retains 3,427,130 B-scans for all five experiments.

I call that retained set QC085: its recorded bad-quality score must be strictly below float32(0.85). The score ranks image quality. The research note shows its distribution, and the corpus selection record accounts for each population change.

Every recipe follows the same recorded source order. Each update presents 64 source images, and the scheduled budget is 150,000 updates:

150,000 updates × 64 sources = 9,600,000 image presentations
9,600,000 / 3,427,130 ≈ 2.8 passes over the retained images

This controls source exposure across the recipes. Multiple views add computation for each source presentation. Sampling is by B-scan, so a person with more scans contributes more presentations; downstream evaluation groups scans by patient.

The encoders share the same ImageNet-pretrained MAE ViT-L/16 checkpoint. I continue domain pretraining from that representation and attach the decoder or projector required by each objective. ViT-L/16 is a large vision transformer using 16-by-16-pixel computational patches.

Five AS-OCT pretraining recipes
Input views Encoder input Target
Full-field MAE Full image, with each spatial dimension divided by 2. 15% of patches visible; 85% masked. Reconstruct normalized pixel values at the masked positions.
Matched-Global Image dimensions divided by 2. Select up to 128 patches from global positions. The shared budget gives at most 32 visible patches. The remaining selected patches are masked. Reconstruct normalized pixel values at the selected target positions.
Matched-CUNEX Same image scale and patch budget; select positions using the anatomical mask. The same visible and target counts as Matched-Global; only the selected positions change. Reconstruct normalized pixel values at the selected target positions.
DiffMAE 2D Full image, with each spatial dimension divided by 2. 25% clean visible patches. The decoder also receives noisy hidden content at the other 75%. Predict the clean normalized target, x₀, at the hidden positions.
JEPA + VISReg Two global views with dimensions divided by 3, plus eight native 512 × 512 crops: six mask-guided and two free. A shared encoder processes all patches of each view, followed by a shared projector. Agreement with the mean global projection, plus VISReg regularization.

Shared matched-patch budget: M = min(128, C), V = ceil(M / 4), T = M − V. C is the number of anatomical candidate patches; both matched arms use the same counts. M counts selected patches, and V counts the patches visible to the encoder.

Recipe Image information supplied Training task
Full-field MAE Visible patches distributed across the whole image Reconstruct pixels in the hidden patches
Matched-Global A limited patch budget, sampled across the image Reconstruct hidden pixels within that budget
Matched-CUNEX The same patch budget, placed using the corneal mask Reconstruct hidden pixels within that budget
DiffMAE 2D Visible context and noisy hidden patches Recover clean pixels from the noisy targets
JEPA + VISReg Two global views and eight local crops Bring view representations into agreement while regularizing variation across sources

Recipe 1: reconstruct the full field

The full-field MAE baseline receives the entire raster, reduced by a factor of two in each spatial dimension with its aspect ratio preserved. I hide 85% of its patches. The encoder processes the visible patches, and the decoder predicts normalized pixel targets in the hidden regions.

This adapts the masked reconstruction mechanism introduced by He and colleagues, whose original paper used 75% masking. The 85% setting defines my full-field baseline, named canonical_mae in the run records. MAE paper

The encoder receives visible context distributed across the image. Its reconstruction target includes the cornea, background and acquisition characteristics. This arm establishes how a full-field objective develops the representation under the shared source-exposure budget.

Recipes 2 and 3: hold the patch budget fixed, change its placement

The matched pair isolates the placement of a limited patch budget. Both arms use the same reduced image and the same numbers of selected, visible and target patches.

Matched-Global draws positions from the image grid. Matched-CUNEX draws from positions identified using a corneal segmentation mask. The mask guides where to sample; the encoder and reconstruction target still use the image intensities at those positions.

A patch is an anatomical candidate when more than 10% of its area overlaps the mask. Let C be the number of candidates. I calculate the shared budget as:

M = min(128, C)       selected patches
V = ceil(M / 4)      visible patches
T = M − V           reconstruction targets

With C = 100, both arms select 100 patches: 25 visible and 75 targets. With C = 200, the cap gives 128 selected patches: 32 visible and 96 targets. The global arm follows this same budget even when the full image grid offers more positions.

Explore the matched patch budgetSynthetic worked example

C = 100; selected M = 100; visible V = 25; target T = 75.

M = min(128, C), V = ceil(M / 4), T = M − V.

Both matched recipes use these counts; they differ in where patches are selected.

This interactive example uses constructed candidate counts to show the selection rule. The full image grid supplies possible positions; the budget selects a subset; the encoder receives the visible part of that subset.

I keep positions unique. When fewer than two anatomical candidates are available, the implementation raises an error, preserving the requirement that both arms process the same source population.

The experiment tests the hypothesis that placing a fixed budget around the cornea makes better use of it. Because placement depends on the segmentation, the mask’s coverage becomes part of the sampling mechanism. Comparing these two arms gives that choice a direct control while keeping image preparation and patch counts shared.

Recipe 4: reconstruct by denoising

DiffMAE changes the decoder’s task. Visible regions provide clean context, while hidden regions enter the decoder with added noise. The decoder also receives the noise level and learns to recover the clean target. This follows the masked conditional diffusion approach introduced by Wei and colleagues. DiffMAE paper

My two-dimensional adaptation uses 75% masking and predicts the clean normalized target, called x0. The full raster is reduced by two, as in the other reconstruction arms.

For each training example, a noise level is sampled from a schedule of 1,000 levels. That sampled level determines the corruption supplied to the decoder. One training update processes these sampled denoising tasks; a 1,000-step generation sequence would be a separate computation.

Using the synthetic arc from the first article, MAE supplies visible context and asks for the missing values. DiffMAE also supplies a noisy version of those missing values. The decoder learns to combine that information with context and the noise level. This gives the reconstruction arm a different route to the target and a different learning signal for the encoder.

Recipe 5: combine global context with native local views

The JEPA+VISReg arm uses ten views per source:

  • Two independently augmented full-field views, each reduced by a factor of three.
  • Six local windows guided by the corneal mask.
  • Two local windows sampled independently of the mask.

Each local window is 512 by 512 pixels, extracted directly from the stored corpus raster. Here, native refers to that grid: the historical corpus contains enhanced, recompressed JPEG images. Sampling the local windows before global-image reduction keeps their available image detail.

This arrangement combines overall shape with local regions. The six guided windows focus sampling on the cornea; the two independent windows add opportunities to observe other parts of the image.

All views pass through a shared encoder and projector. For each source, the mean of the two global projections becomes the agreement target. The objective is:

0.3 × agreement + 0.7 × VISReg

Gradients flow through both global projections. This implementation uses the same trainable encoder to build the target and predictions. VISReg controls representation scale and distributional shape; the first article works through its center, scale and shape terms. VISReg paper

The research question is how relating local information to global context changes the features learned under this objective. I track that change on the same image panel used for the reconstruction arms.

Make image preparation part of the implementation

For all four reconstruction arms, I prepare the full raster and pad only enough to complete the patch grid. Padding pixels are excluded from target normalization and reconstruction loss, so the objective is calculated from the observed image content.

Reconstruction losses are averaged first within each image, then across the 64 sources. Each source therefore contributes equally to the update, even when images contain different numbers of valid target pixels.

These decisions fix the meaning of the calculation. Changing the valid-pixel rule, target normalization or image weighting would change the objective alongside the architecture. Keeping them explicit lets me inspect the effect of the intended recipe choices.

Compare source exposure and computational cost

The shared source order, initialization and presentation budget let me compare complete recipes after the same source exposure. Their compute costs reflect their different workloads: resolutions determine patch counts, global–local JEPA processes ten views, and the reconstruction methods add their respective decoders.

I organize the comparisons around the quantity being held fixed:

Research question Shared quantities and measurements
How do complete recipes behave after the same source exposure? Image index, source order and presentation count
What does each recipe deliver within a compute budget? Hardware, elapsed time, memory and execution settings
How does mask-guided placement affect a fixed patch budget? Image preparation, initialization, visible/target counts and selection rule

The five-arm study addresses the first question, while the matched pair gives a more focused comparison of spatial selection. Timings and profiles record the computational cost of reaching those measurements.

Preserve the full-batch objective within GPU memory

I use microbatches, smaller groups processed in sequence, to limit the activations held in GPU memory. JEPA’s objective also combines information across sources, so the implementation preserves the full statistical populations while splitting encoder work.

The projector’s BatchNorm, which normalizes features using batch statistics, sees 640 representations: ten views times 64 sources. VISReg is evaluated separately for each view across the 64 sources.

Representation and gradient caching, followed by encoder recomputation, preserve that full-batch calculation. Simply calculating independent objectives on the microbatches would change the statistics. I qualified the distributed optimization by checking inputs, features, gradients, optimizer state and resume behavior. The later two-GPU performance result reduced update time while keeping this recipe intact.

Connect the choices to the results

The recorded feature spectra show different trajectories. Both matched-patch arms concentrate variation into fewer directions. Full-field MAE spreads variation relative to the initialization. JEPA produces the broadest spectrum in the measured panel. The research note gives the dated checkpoint positions, PR and PC1 definitions, and profiling results.

Each loss tracks progress on its own prediction task: MAE measures pixel reconstruction error, DiffMAE measures reconstruction from noisy targets, and JEPA combines representation agreement with regularization. The shared feature measurements let me examine all five encoders with the same procedure. The next patient-separated task comparison will measure which distinctions those features make accessible.

For a final check of the matched calculation, increase the anatomical candidate count from 127 to 129. At 127, M = 127, V = 32 and T = 95. At 129, the cap gives M = 128, V = 32 and T = 96. Both arms receive one more target patch and keep the same number of visible patches. That is the controlled change the implementation is designed to express.