Agents read the controls on their Mac's screen
An agent on its own Mac now gets a numbered list of the buttons, fields, links and menus in front of it, and clicks one by its number.
Before, an agent working on its own Mac saw only pictures of the screen and clicked by pixel. Now it reads the front app's controls first: each one numbered, with its name, what it holds and where it is, and a star on the ones that just appeared.
Clicking control 12 lands on that control wherever it sits now, so a window that moved or a list that scrolled no longer sends the click astray. Password fields show only that they are password fields, never what is in them.
A list like this costs a fraction of a screenshot, so agents look more often and spend less. They still take a screenshot when the list is empty or a picture says more.