Linguistically Loss-Aware ASL Translation

The problem

American Sign Language does not spread its information evenly over time or over the body. A brief change of handshape, palm orientation or movement can decide which sign it is. Grammar rides on the face: raised eyebrows can be all that makes a sentence a yes/no question, and mouth shapes modify what the hands are saying. A signer sets up people and places at points in space and refers back to them much later in the sentence. Depicting signs show size, shape and motion directly. Between these moments there are long stretches that are largely redundant.

Translation models do not see it that way. To fit a long video into a model, the standard recipe keeps a fixed number of frames, sampled evenly or at random, and gives every one of them the same treatment. Whatever lands between the kept frames is gone, and so is detail the face or hand input never represented to begin with. The model is then asked for fluent English anyway, and a strong language model on the output side is very good at producing fluent English whether or not the evidence for it survived. A yes/no question whose only marking was lost can come back as a confident statement.

The approach

The first step is a typology: which ASL distinctions uniform temporal compression destroys, and through which mechanism, whether a construction is too brief to survive frame dropping, lives in a part of the body the input does not encode, or depends on context too far back to stay in the window.

The second is to test it under control. The same clips run through a translation model again and again, with the input degraded in one specific way each time: fewer frames, a lower rate for one part of the body than another, a stretch removed. The measurement is not only overall translation quality but what happens to each kind of construction. Is a marking kept, dropped, or replaced with something the model invented? How often does the model fall back on what a language model expects to hear once the signal is gone? And whether the model is using the video at all, or mostly guessing well.

Each of those is paired with its cost, in latency, memory and sequence length, so the result is a set of trade-offs rather than a single score: which parts of the signal can be sampled sparsely for free, and which cannot be touched without losing meaning.

Why it matters

Sign-language translation is moving toward the same large pretrained models as everything else, and those models make it cheap to produce output that reads well. For a Deaf user, a fluent wrong translation is worse than an obviously broken one, because nothing signals the error. Knowing which linguistic information a pipeline throws away, and what it costs to keep it, is the groundwork for systems that spend their compute where the meaning is.

Status

In progress, as CSC494 at the University of Toronto under Prof. Gerald Penn, from September 2026. The baseline is Uni-Sign, a recent open sign-language model, and the experiments use public ASL corpora. Results are not reported here yet: they are early, and some have already been revised as the measurements improved.