Article

Interpretability

Inside a universal latent space

The Daedalus backbone, a functional autoencoder (FAE), is pre-trained once, frozen, and shared by every operator, so its latent space belongs to no task and can be examined on its own terms. This article follows the interpretability appendix of the paper through that space, a hypersphere that decodes to valid fields everywhere, whose distances agree with distances between fields, whose geodesics follow physical evolutions, and whose edit directions carry over from one field to another.

Based on Appendix C of the paper

128
dimensions of an embedding of the analysis backbone, a point on a hypersphere
68
directions around an embedding that survive decoding and re-encoding, the median effective rank
11
evolution datasets on which the geodesic midpoint errs less than keeping an anchor
6
named edit directions that transfer from one pair of fields to unrelated ones

A task-agnostic latent space

Daedalus represents a function on a point cloud by cutting the cloud into partitions and compressing every partition into one embedding. Before a partition is encoded, PartNorm carries it into a common frame. Where the partition sits, how large it is, and the offset and amplitude of its values are stored as static features, so the learned embedding is left to describe only the morphology of the partition, the shape of the field inside it. The encoder and the decoder are pre-trained once, without labels, on synthetic analytic functions drawn on a unit disk, and are then frozen. Every operator is learned inside this fixed space.

The latent spaces of existing neural operators are shaped by the single operator they were trained for and are entangled with it. A space that is frozen and task-agnostic can instead be studied in its own right. The encoder and the decoder are deterministic, continuous and differentiable maps, and every embedding decodes to a field, so the space can be explored directly, by choosing points and paths in it and decoding them.

All figures below use the analysis backbone of the paper, whose embeddings have 128 dimensions, while the operators of the paper run on a cheaper backbone with 8 dimensions. Many experiments encode a field as a single partition and decode it with the identity frame, so that only the learned morphology is shown.

decodedatom

Hover over a point to decode it, click to pin it, drag to turn the sphere.

atoms ±e₁, ±e₂, ±e₃directions in betweenshown or pinnedbrighter and larger = nearer to you
A three-dimensional slice of the latent hypersphere. Every point is a morphologic embedding ζm = √dζ (n1e1 + n2e2 + n3e3) with n12 + n22 + n32 = 1 and the other 125 components zero, so this slice of the hypersphere of radius √dζ is an ordinary sphere. The six atoms ±e1, ±e2, ±e3 sit at its poles, and a Fibonacci grid of 144 directions fills the space between them. Each is decoded by the analysis backbone on one partition with the identity frame and on its own color scale; neighboring points decode to similar fields, and every point of the slice decodes to a valid field.

Embeddings as directions on a hypersphere

An RMSNorm after every encoder block returns the morphologic embedding to a fixed magnitude, so every embedding lies on a hypersphere of radius √dζ, with dζ the latent dimension, and only its direction carries information. Each encoder update is orthogonalized against the current embedding before it is applied, so it moves the embedding along the sphere rather than rescaling it.

Two tools follow from this geometry. The distance between two embeddings at angle θ is measured by the chord distance dchord = 2 sin(θ/2), the distance between their unit directions. The natural path between them is the great circle, given by spherical linear interpolation (SLERP) between two embeddings ζm0 and ζm1,

ζm(τ) = [ sin((1 − τ)θ) ζm0 + sin(τθ) ζm1 ] / sin θ,

which stays on the sphere for every τ. The straight chord between the same two points cuts through the interior of the sphere, where the decoder never meets an embedding in training.

The geodesic between two embeddings, drawn in the plane of their two directions. A point on the great circle (solid) keeps the norm √dζ of the sphere for every τ, while a point on the straight chord (dashed) falls below it, deepest at the middle of the path, inside the sphere.

Decoding of arbitrary directions

A first test asks whether the coordinate axes carry meaning by themselves. A single partition is placed on the unit disk with the identity frame, its embedding is set to a signed basis direction ±ei scaled to the radius of the sphere, and the result is decoded. Every axis decodes to an elementary morphology, a smooth field of a few lobes and oscillations, and the opposite sign of the same axis decodes to another field of the same kind. The paper calls these fields the latent atoms, and all 128 of them are shown below.

+ei
−ei
The latent atoms, the fields decoded from the basis directions +ei of all 128 dimensions, each on one partition of the unit disk with the identity frame and on its own color scale. Hover over an atom, or tap it, to see the two atoms of its dimension, +ei and −ei, large.

A second test asks whether the regions between the axes are populated as well. SLERP from ei to ej walks the quarter of a great circle that joins them, and decoding along it morphs one atom into the other through plausible fields at every step, none of which is noise or a blank field. The whole hypersphere therefore decodes to valid fields, rather than only a thin region around the embeddings of training functions or isolated points that happen to decode well.

τ = 0.00
The quarter great circle between two atoms, from +ei (A, τ = 0) to +ej (B, τ = 1), decoded at 25 fractions on one color scale. Any two of five atoms can be chosen; the strip shows the path at seven fractions, and the large field the one at τ. Drag τ or let it play.

Distances and sums in the latent space

Whether distances on the sphere mean the same as distances between the decoded fields decides whether the latent space can stand in for the physical space in a loss or an optimization. Pairs of embeddings are built at controlled cosine similarities, both are decoded, and the relative L2 distance between the two fields is compared with the chord distance between the embeddings. The two distances agree nearly one to one with a narrow spread, the median within 20% of the line of slope one, so a distance measured on the sphere predicts the distance between the fields, and the order of distances between fields is preserved. The latent loss used to train operators relies on this property. Without it, a processor trained to match embeddings would optimize a quantity only loosely related to the error of the fields it predicts.

Superposition, in contrast, holds only locally. Two embeddings are composed once in the latent space, by decoding their sum projected back onto the sphere, and once in the physical space, as the min–max normalized mean of the two decoded fields. The two compositions agree when the embeddings are close and diverge as they move apart. Their relative distance, the additivity error, starts near zero for aligned embeddings and grows smoothly with the chord distance between them. The gap reflects the nonlinearity of the decoder, and the latent space is accordingly locally linear and globally curved.

medianinterquartile range10th to 90th percentilesingle pairsslope 1
–
 
The relative L2 distance between the decoded fields of embedding pairs at controlled similarity (field distance), and the additivity error between the latent and the physical composition of two embeddings, against the chord distance between their unit directions. At every chord distance, the median over 60 pairs (line), their interquartile range (dark band), the band between the 10th and the 90th percentile (light) and the single pairs (dots); the dashed line has slope one. Hover over a chord distance to read its median.

Local edits and the spectrum of the decode-encode map

Single embeddings can also be edited in place. Walking moves an embedding along one angular coordinate of its spherical parametrization, the other angles held, and the decoded field changes gradually as the angle grows, keeping its morphology intact.

original
walked
+0.70 rad
An embedding walked along one angular coordinate of its spherical parametrization, the others held, by an offset from −π to π, next to the embedding it starts from; the paper's figure steps by 0.7 radians. Choose the embedding and the angle, then slide the offset.

Mirroring negates components of an embedding, in the examples below five at once. Every such edit changes the decoded field in a definite way, with lobes reshaped, merged or displaced, while the field stays coherent and recognizably related to the original, also when five components are negated together. No small set of coordinates is therefore responsible for holding a field together.

original
flipped

A random embedding on the sphere and the same embedding with a set of five components negated together, on one color scale. Choose the embedding, and hover over a set to see its flip at once; a click keeps it.

Decoding an embedding and encoding the result again keeps about half of the 128 directions around an embedding. Which directions survive is read from the Jacobian of the map that decodes an embedding and encodes the result again, J(ζm) = Jℰ(𝒟(ζm)) J𝒟(ζm). To first order, a perturbation along an eigenvector of this matrix is scaled by its eigenvalue λ. An eigenvalue near 1 marks a direction that is reconstructed faithfully, an eigenvalue near 0 a direction the bottleneck discards, 0 < λ < 1 a contracted direction, whose perturbation the decode-encode map partly undoes, and λ > 1 an amplified one.

Over many real low-frequency embeddings the eigenvalues fall into two groups, with a spread of contracted values between them. A three-component Gaussian mixture fitted to them in the complex plane places a tight collapsed component at 0, which holds 28% of the eigenvalues, a broad preserved component centered at 1.11, which holds 46%, and between the two a wide contracted component centered at 0.35, which holds 25%. The spread of the preserved component along the imaginary axis shows that the Jacobian of the decode-encode map is not symmetric. Sorted per embedding, the magnitudes drop from about one to about zero at a knee that gives a per-embedding effective rank near 68 of 128.

28%
collapsed, at 0
25%
contracted, at 0.35
46%
preserved, at 1.11
68
median effective rank, of 128
–
effective rank of one embedding
Left, the eigenvalues of the Jacobian of the decode-encode map at 24 real low-frequency embeddings in the complex plane, with the three components of a Gaussian mixture fitted to them, each shaded at one and two standard deviations, and their means (stars) with the weight and the real part of the mean; the few eigenvalues beyond Re(λ) = 2 lie outside the view. Right, the eigenvalue magnitudes sorted per embedding, their median, and the mean gain of every fixed axis, sorted afterward, with the embeddings highlighted one at a time. The effective rank of an embedding counts its magnitudes above 0.5 (dotted), and its median over the embeddings is 68.
Each embedding occupies about half of the 128 dimensions, but the collapsed directions differ from one embedding to the next. No axis is globally redundant, and the population of embeddings as a whole uses all 128 dimensions.

Geodesic morphing

Decoding along the great circle between two embeddings morphs one function into another through coherent intermediates, between two radial sines, from a Gaussian bump into a sine field, and between unrelated fields, and none of the intermediate fields is noise or blank. The transition from one function to another can therefore be followed step by step, which indicates that the space is organized around the structure of functions rather than around the identity of individual samples.

pair 1
pair 2
pair 3
pair 4
τ = 0.00
Geodesic morphing between two sines, a smooth low-frequency radial sine (anchor A, τ = 0) and a high-frequency one (anchor B, τ = 1), four pairs. Every frame decodes the great-circle path between the two embeddings at the fraction τ, on one partition with the identity frame and on its own color scale, and the frequency rises gradually, ring by ring, through coherent intermediates rather than noise. The ticks are the columns of the paper's figure; drag τ or let it play.

Geodesics between states of an evolution

When the two endpoints are two states of a physical evolution, the geodesic between their embeddings can be compared with the true states in between. The geodesic involves no knowledge of the governing equations, no time stepping, no fitting and no labels, and is fixed by its two endpoints alone. On incompressible flows from several families of initial conditions, the decoded geodesic lies closer to the true intermediate states than either endpoint does, provided the change between the endpoints is local. This does not hold for transport equations, such as the acoustic wave equation, whose solutions carry structure across the domain. The comparison is most telling at the middle of the interval, where the true state is about as far from one endpoint as from the other, so that neither endpoint can stand in for it.

Truth
Geodesicanchor
–
geodesic against the true state
–
true state against the nearer anchor
geodesic vs. truthtruth vs. anchor Atruth vs. anchor Breconstruction floor

index 0
One trajectory of five evolution datasets, the four of the paper's figure and the compressible Riemann problem, encoded as in the paper's measurement (128 partitions, 16384 points, one partition key for every snapshot). Interpolate decodes the geodesic between the anchors at the indices 0 and Δt (A and B) at every index in between; Extrapolate continues the geodesic between the anchors at 0 and 8 past the second one at the same angular speed. The curves are relative L2 distances along the trajectory; the note under them gives the paper's median over 32 trajectories and every anchor pair.

Latent hypersphere illustrative projection

physical evolution geodesic between anchors snapshot active anchors drag to turn

Physical evolution t = 0

ground truth

Geodesic estimate

anchor

Error of the geodesic estimate relative L2

geodesic vs. truthreconstruction flooranchors
Right now
0%
geodesic error
0%
reconstruction floor

Between anchors close in time, the geodesic stays within a few percent of the true flow, so the embedding preserves the geometry of the evolution. For anchors further apart (Δt = 10) the geodesic drifts off the physical path, and following that path over longer intervals is the task of a learned operator.

The same comparison on an incompressible flow, on pre-computed frames. The flow (IEU-BB, velocity u) evolves over 20 steps, the anchors are Δt snapshots apart, and every state in between is decoded from the great circle between the two nearest anchor embeddings, without any operator. Play the evolution, drag the time, or change Δt; the globe is an illustrative projection of the path of the embeddings.

The paper then measures how far a geodesic carries an evolution. For a pair of snapshots Δt time indices apart, both are encoded, the embedding moves halfway, and the result is decoded and compared with the true state at the midpoint. The morphologic part follows the great circle. The static features are blended linearly in the units the encoder stores them in, and since the amplitude of a partition is stored as a logarithm, it is interpolated geometrically and never sweeps through zero. Every admissible pair contributes, over 32 trajectories per dataset with 128 partitions and 16384 points, and two references bound the result. The anchor similarity is what keeping an anchor instead of interpolating would cost, and the reconstruction floor is the error of encoding and decoding the true states, which no interpolation can go below.

At Δt = 2 the interpolant sits far below its baseline and only a few times above the floor, at 1.4% against 7.9% with a floor of 0.32% on the incompressible Brownian-bridge data, and at 4.6% against 16.4% on the vortex sheet. The margin closes as the interval lengthens, reaching 39.8% against 55.9% and 68.4% against 87.5% at Δt = 20. No dataset crosses its baseline at any interval. The compressible Euler datasets keep the narrowest margins without losing them. The density of the compressible Riemann problem, with its traveling shocks, likewise starts at 9.7% against 15.7% and still sits below its baseline at Δt = 20, at 41.0% against 45.2%.

–
 
–
datasets whose midpoint stays below its baseline at every Δt
The halfway measurement across datasets, the median relative L2 error against the anchor step Δt, 32 trajectories each, with 128 partitions and 16384 points. Color is the dataset, solid the latent interpolation, dashed its anchor-similarity baseline, and the shading is the margin between them. The paper's figure shows three incompressible-flow datasets and the acoustic wave equation up to Δt = 12, the longest horizon all four admit; the other views add the remaining datasets of the paper up to Δt = 20. Every dataset stays below its own baseline, and the gap narrows as Δt grows. Hover over a curve to read it, with its reconstruction floor.
On all eleven evolution datasets of the paper, shocked compressible flow included, the halfway point of a latent geodesic errs less than keeping an anchor would, at every interval examined. The latent space admits a zero-shot interpolation of physical dynamics in time, which suggests that the path a trained processor has to follow between two states is short and well behaved in this geometry.

Extrapolation beyond an anchor

A geodesic need not stop at its second endpoint. Continuing it beyond the second anchor, at the constant angular speed set by the two anchors, is the simplest possible extrapolation, linear in the angle along the great circle, and it uses nothing beyond the two anchors. On evolutions with exact solutions, isotropic, anisotropic, diagonal and off-center diffusion, a spreading dipole and two merging bumps, anchored at the time indices 0 and 8, this continuation carries the evolution well past the second anchor. At index 12 the extrapolated field errs by 3 to 18%, and on four of the six evolutions it is two to three and a half times closer to the true state than the last anchor; at index 16, as far beyond the second anchor as that anchor is beyond the first, it still beats keeping the last anchor on the same four.

A local evolution therefore traces a path of nearly constant direction in the latent space, which can be extrapolated and not only interpolated. The extrapolation is limited to evolutions that change structure in place. Like the time interpolation, which degrades on shocked compressible flow where a discontinuity travels across the domain, and like the transferred directions described next, a continued geodesic cannot transport structure across the domain.

Truth
Geodesicanchor
–
geodesic against the true state
–
true state against the anchor at index 8
geodesic vs. truthtruth vs. anchor 0truth vs. anchor 8indices of the paper's figures
index 0
The six evolutions, played from index 0 to 16. Truth is the single-partition morphology of the true state, and Geodesic the great circle between the embeddings of the anchors at the indices 0 and 8 (dots below the axis), continued past the second one at the same angular speed. The curves are relative L2 distances, of the decoded field to the true state (yellow), and of the true state to the first (green, dashed) and to the last anchor (violet, dashed); the dots on the yellow curve mark the indices of the paper's figures.
Past the second anchor, the error of the continued geodesic (yellow) against what keeping the last anchor would cost (violet), at index 12, half the spacing of the anchors beyond the second one, and at index 16, the full spacing.

Transferable edit directions

The next experiment starts from two crops of a field that differ by exactly one operation, such as a blur, an anisotropic stretch, or the split of one feature into two. Each crop is encoded into a single-partition embedding, and the difference η = ζmB − ζmA of their embeddings is read off as a direction. Applying η to an unrelated embedding ζm by a step along it, scaled by a real coefficient c whose sign sets the sense of the edit, and re-projecting onto the sphere,

ζm(c) = √dζ normalize(ζm + c η),

performs the same edit on that field, in both directions of c. A single vector read from one pair therefore acts as a coherent, reversible operation on embeddings it was never derived from, which is only possible if the operation occupies a consistent direction across the sphere.

The paper shows six such directions, three anisotropic stretches (vertical, horizontal and diagonal), diffusion, peakedness and splitting. Extrapolating to c = 2 continues each edit beyond the second anchor. Stepping to negative c, which neither anchor shows, performs the opposite edit just as coherently: the negated vertical stretch widens a bump, negated diffusion sharpens it, negated peakedness flattens it into a plateau, and the negated split merges two or three bumps into one. These directions correspond to operations that can be recognized and named. Since the difference between any two embeddings defines a direction in the same way, the sphere presumably holds many further, more abstract ones. The transfer holds for operations that act in place. An edit that would move structure across the domain does not transfer, for the same reason the geodesic between the states of an evolution fails on transport.

 
 
 
 
c = 0
The six directions of the paper, three anisotropic stretches, diffusion, peakedness and splitting, each read from one pair of anchors as η = ζmB − ζmA and applied as ζm(c) = √dζ normalize(ζm + c η) with c sweeping from −1 to 2. The first field is anchor A itself, which is reached at c = 0, and anchor B at c = +1; the others are unrelated fields that the same vector edits the same way, in both directions of c. Every field is decoded with the identity frame on its own color scale, and the ticks are the columns of the paper's figures.
Direction
anchor A
anchor A ·
→
anchor B
anchor B ·

Field it is applied to
c = 0

ζm → |ζm| (ζm + c η) / |ζm + c η|, decoded with the identity frame on a square grid, each strip on its own color scale, so only the shape changes. Apply it to your own field →

The transferred directions on pre-computed frames. Choose a direction, read from the two anchors shown, and the field it is applied to, then move the coefficient c with the slider or by scrolling over the field.

Arrangement of function families

The global organization of the space is read from many embeddings at once, one per function on a single partition. Under t-SNE with the chord distance, mixed radial sines from ten frequency bands order continuously by spatial scale into a smooth frequency gradient, with the highest band splitting off. The cosine similarity between the mean embeddings of two bands falls off smoothly with the distance between the bands, which confirms spatial frequency as the dominant axis. The number of modes is a weaker axis. This agrees with the role the paper assigns to the effective spatial frequency of a partition, which largely determines how hard the partition is to reconstruct.

t-SNE of ten frequency bands
–
 
Left, t-SNE under the chord distance of one embedding per function, 64 functions from each of ten frequency bands, colored by the frequency f of its band; hover over a function to see it, over the ramp to light a band, and click the ramp to hide a band. Right, the cosine similarity between the mean embeddings of two families of radial sines, in percent, each matrix on its own range. Frequency bands, six bands of f, falling off smoothly with the distance between the bands. Number of modes, sines of 1 to 16 modes at frequencies from 1 to 3, a weaker axis, along which sines with more modes are more similar to one another, from 20% between functions of one mode to 72% between functions of sixteen. Hover over a cell to read it.

Real physical fields organize the same way, into coherent families by dataset and channel. Sines with more modes are also more similar to one another, in the latent space as in the physical space, where a sum of many modes at similar frequencies looks alike from draw to draw.

Datasets
Channels
t-SNE under the chord distance of one embedding per function on a single partition, for 48 samples of each of 37 channels of 18 datasets, colored by dataset and marked by channel. Hover over a function to see its input points and to light its dataset and channel; click it to keep its dataset alone, and click again to bring the others back. The legend hides or shows a dataset or a channel, and lights it while pointed at.

Coverage of the hypersphere

How much of the 128-dimensional sphere the model uses is read off in two ways, by the number of principal components that capture 90% of the variance of the embeddings in the tangent space at their mean direction, and by a per-dimension occupancy that bins each angle and counts a bin as filled if any embedding lands in it. For radial sines the effective dimension grows monotonically with frequency, from 8 components for the lowest of five bands to 51 for the highest, while every band fills 15 to 17% of the angular range of a dimension on average, and their union needs 34 components and fills 24%. Real fields, 73 channels of 24 datasets taken together as one family of the same size, need 42 components and fill 42%, more of the sphere than the sines, and no family fills the sphere alone.

Components for 90% of the variance
Latent usage of radial sines in five frequency bands of the frequency f, of their union, and of real fields of 73 dataset channels (dashed), with 16 partitions per function. Singular values, sorted, of the embeddings of every family in the tangent space at their mean direction, with the number of components that capture 90% of the variance. Occupancy, the fraction of the angular range filled in every dimension, sorted, with the participation ratio of every family, and below it the mean over the dimensions.

Exploring the latent space

Several pages of this site run on the same frozen backbone. The latent-directions demo applies transferred directions to a sample, an upload or a painted field, and the transition demo walks the great circle between the embeddings of two fields. The latent-search and perturbation demos act on single-partition embeddings, the first ranking 2,300 functions by their chord distance to a query and the second turning an embedding toward random directions.

A puzzle on the latent geometry

Tiles is a puzzle built on this geometry. Four random directions of the sphere sit at the corners of a 5 × 5 board, and every other cell decodes a blend of them, so the fields change smoothly from corner to corner along every row and column. The 21 inner tiles are shuffled, and the task is to restore their order as quickly as possible; a tile stays only on its own cell. Every finished game enters the leaderboard below under a random nickname.

Play Tiles →