A screen should only have to be read once.

syin looks at an application a single time, writes down where everything is, and answers every question afterwards from what it wrote. A coordinate in twelve milliseconds, on your own hardware, with no model in the loop — and a refusal instead of a guess when the screen has changed underneath it.

Watch it work
A legacy claims application with no accessibility tree and no API
pixels
boxes
3,115 msthe first time syin sees a screen
12 msevery time after, on the same screen
0 tokensonce the map exists, no model runs
0 wronganswers across 60 damaged-frame trials
the thing this replaces

Most agents pay for the same screen sixty times.

An agent working inside one application looks at the same toolbar on every single step. Each look is a screenshot, a few thousand tokens, a second of latency, and a probabilistic guess at the end of it. Nothing is kept. The next step begins from nothing.

syin pays once. The sixtieth step costs the same as the second: an arithmetic lookup against a map that is already on disk, in coordinates that are already correct.

A vision agent, sixty steps in one app

every step is a fresh parse
60 model passes · ~192,000 tokens · 60 chances to misread

syin, the same sixty steps

one parse, then fifty-nine lookups
1 model pass · ~3,200 tokens · 59 answers that cannot drift
three films, recorded from the same application

Watch a screen get read, remembered, and then doubted.

The application below is the kind syin exists for: a claims terminal with no accessibility tree, no selectors and no API. Nothing in these clips is sped up.

0:00
film one · 0:15

A screen it has never seen

Three detectors run at once — a flood fill, an edge pass and a neural detector — and their boxes are unioned and de-duplicated. Pictures and empty fills are thrown away, because they are not controls. Then one vision pass reads the whole page rather than retrying box by box, the names attach to the boxes that were already found, and everything irreversible is flagged before anything is allowed to click.

  • Find the candidates
  • Throw most of them away
  • Read the page once
  • Attach the names
  • Mark what cannot be undone
  • Write the map
0:00
film two · 0:10

The second visit

The frame is hashed coarsely — structure only, so that a new row in the table does not read as a new screen — and the map from film one matches. In the corner sits a 19×19 block that syin stamped into the frame it describes, carrying the map id and a hash of the current state in thirty-one bytes. Reading it is arithmetic on the CPU. The answer is a point.

  • Hash the frame, coarsely
  • Read the stamp
  • locate("Export")
  • Twelve milliseconds, no model
0:00
film three · 0:12

The morning after a release

Someone shipped, and the toolbar moved. This is the moment every image-matching tool clicks the wrong control and reports success. Because the state hash is rendered into the same frame it describes, it cannot desynchronise from it: syin compares, finds the screen it is looking at is not the screen it has notes on, and declines to answer. Then it pays for the screen once more and goes back to twelve milliseconds.

  • The cached map points at empty chrome
  • Compare the stamp to the frame
  • Refuse
  • Re-derive, once
  • Back to twelve milliseconds
how an answer is produced

Three ways to answer. It always tries the cheapest first.

Every request walks the same ladder and stops at the first rung that can answer honestly. Two of the three rungs never touch a model at all, which is why the cost of using syin falls as an agent keeps working rather than rising.

tier one · read the stamp

The frame carries its own map

A 19×19 block in the corner holds the map id and a hash of the current state — thirty-one bytes, four-tenths of one percent of a 1080p frame. Decoding it is integer arithmetic, and it survives JPEG down to quality 35, window crops, and rescaling.

Latency
14 ms
Model cost
0 tokens
Drift
0.00 px
tier two · match the map

No stamp, but a screen it knows

The frame is fingerprinted coarsely and on purpose. Hashing every pixel demos beautifully and never hits cache in production, because one new table row makes a new screen. Structure only — then the stored map is re-anchored onto the frame in front of it.

Latency
28 ms
Model cost
0 tokens
Survives
a full data change
tier three · derive from pixels

Something genuinely new

Only now does a model run, and only once. Flood fill, edges and a neural detector union into candidates, a single vision pass reads the page, names attach, verbs are assigned, and the result is written back and stamped in — so this screen never costs this much again.

Latency
3,115 ms
Model cost
one pass
Frequency
once per screen
3,115 ms → 12 ms

Measured on a single screen as it walks from tier three to tier two to tier one. A screen pays back its parse on the second visit, and every visit after that is profit.

259×
what changed in the last three months

Nine things that made the derived path usable.

Tier three was the slow, unreliable half. Most of this work went into making the expensive path cheap enough and honest enough that the cheap paths have something worth caching.

detection

Three detectors, unioned

A flood fill, an edge pass and a neural detector run together and their boxes are merged and de-duplicated. Flood fill alone found thirty-one controls the edge pass had been missing — including the ones with no border at all.

reading

Read the page, not each box

One vision pass over the whole screen turns forty-eight lines into seventy words. There is no per-box retry, because an empty box does not mean a missed word — it means an icon, and no amount of re-reading will change that.

budget

A hard ceiling on model work

Naming is ninety to ninety-seven percent of a derived run, at roughly two seconds an element, so every recall improvement used to make runs slower. --budget caps it and spends it on unlabelled elements first; everything over budget keeps its rule-derived affordance.

memory

The detector runs in a child process

Loading the detector in-process costs 2.3 GB, which on an 18 GB machine already holding the OS and two local models means swap, and under swap a ten-second call becomes minutes. The child exits and hands its memory back before any language model loads.

models

Chosen by fitness, not by size

The prompt is a few hundred tokens of labels and the answer is a fixed schema, so what matters is adherence and speed, not capability. Selection ranks by what fits in memory and follows instructions — and every run prints the model it picked and why.

labelling

Spanning labels get split

A wide caption stretched across a toolbar used to swallow the three controls underneath it, making them unaddressable. Labels that span multiple candidates are now divided among them, which is why targets like Media Pool resolve at all.

anchoring

Tolerance derived, not chosen

A token span is measured in whole pixels, so half a pixel of error is amplified by distance from the anchor: the bound is (0.5 / span) × distance. At 0.75× that predicts 3.55 px of drift and the suite measures 2.97.

safety

Every element carries a verb

Controls are classified as benign or destructive as they are named, so an irreversible action is flagged before a planner can reach for it. Nothing has to click something to discover it should not have.

coordinates

Twenty-two invariants

The same off-by-a-window-origin bug appeared three times, so the transform is now asserted rather than trusted: maps are stored in surface coordinates and never image ones, and surface → image → surface must be the identity. The suite found two real defects on its first run.

0
wrong answers across sixty deliberately damaged frames. Fifty-eight refused outright, two decoded correctly.
58 refused2 decoded correctly0 wrong
the property everything else rests on

It refuses far more often than it is wrong.

A confidently wrong coordinate is worse than no coordinate, because a planner will act on it and report success. So every check in the codec fails closed — the magic bytes, the header checksum, the anchor and the payload — and a frame that does not verify produces a refusal, not a best effort.

The state hash is the part worth stealing. It is rendered into the same frame it describes, so it cannot drift out of sync with what it is describing. When an application changes, syin is looking at a hash it has never seen, and it knows immediately — rather than three failed clicks later.

four ways in

It hands you a point. Bring your own planner.

syin is a perception layer, not an agent. It answers one question — where is this control, and is it safe to touch — and it answers it the same way every time, which is what makes it something you can put in a test suite.

from syin import Screen

screen = Screen.of("Meridian Claims")

x, y = screen.locate("Export")        # (1086, 168) — 12 ms, 0 tokens
screen.locate("Void claim").verb       # 'destructive'

# A miss is a refusal, never a nearby guess.
screen.locate("Prnit")
Abstained: no candidate above 0.62 (best 0.31, 'Pended')

Everything runs on the machine the application is running on. No frame leaves it, which is usually the only reason these applications can be automated at all.

the gate, run before anything ships

Every number on this page came out of the same suite.

Thirty-six checks, forty-five seconds, on an M3 Pro with 18 GB. The tolerance throughout is twenty-six pixels, which is the measured distance at which a click lands on the wrong control.

AreaResult
Codecpointer frame 43 B · self-contained 340 B for 30 elements · corrupt magic, header CRC, anchor and payload all refused
Transport1:1, 2×, 3×, 0.75×, 0.6×, window crop, JPEG q35, crop + 1.5× + JPEG — worst drift 3.96 px
Tier one0.00 px · 14 ms
Tier two0.00 px · 28 ms · survives a full data change
Tier threea usable map from pixels alone · 8 of 8 controls reachable
Promotion3,115 ms → 12 ms on one screen · 259×
Mapper100% recall on both surfaces · precision 44% desktop, 77% web
Safety60 damaged tokens: 58 refused, 2 correct, 0 wrong
Coordinates22 invariant checks, 0 failures
Cross-languageJavaScript and Python frames byte-identical · 4 of 4 pinned vectors

Two things worth saying plainly. These are macOS numbers, and most of the applications that need syin are on Windows — that surface is the next one. And precision on the desktop mapper is 44%, meaning it still proposes boxes that are not controls; recall is what the cache depends on, so that is the trade being made deliberately for now.

Point it at something that has no API.

The worse the application, the more syin is worth: a Citrix session, a WPF client from 2009, a terminal emulator inside a browser inside a VDI. If a person can see the control, syin can give you its coordinates — or tell you, honestly, that it cannot.

Watch the films again