Driving your Mac

Moe can see your screen and use your Mac, open an app, click a button, fill a field, through the same accessibility machinery a screen reader uses. Clicks are delivered straight to the target window in the background, so your cursor does not move, your keyboard focus does not change, and you can keep typing while Moe works in another window.

It tries not to touch the screen at all. Driving a window is the last resort, not the first. Asked to open a meeting link, Moe reads the join address off the calendar event and opens it: no screenshot, no clicking, nothing that can miss. The order it works through is: the data it already has, then the app's own scripting, then a menu item or a keyboard shortcut, then its own browser for anything on the web, and only then a click.

It knows your apps' menus. Ask Moe to do something in an app and it can read that app's menu bar directly, the items and their shortcuts, and invoke the one it needs by name rather than hunting for it on screen. That is faster and far more reliable than clicking, and if the item does not exist Moe is told so plainly. It also works where looking does not: music software, CAD, 3D tools and games draw their main window as one canvas Moe cannot read, but their menus are ordinary. Items that are greyed out are reported as greyed out, which is the difference between "not available right now" and "does not exist".

Moe's own browser#

When the job is a web page, look something up, fill a form, work through a site, Moe drives a browser of its own rather than yours. It reads the page as text, so it knows what the buttons and fields actually are instead of guessing from a picture, and it clicks the exact one. Nothing appears on your screen while it happens, nothing takes focus, and your own tabs are untouched.

It is signed in as you. If your default browser is Chrome, Edge or Brave, Moe browses on a private copy of your profile: your logins and cookies come along, so a booking site with your account, or a cart you started, are places Moe can work without you signing it in. The copy lives in Moe's own folder, is refreshed each time Moe opens its browser, and never touches the browser you have open. Moe never opens, scripts or drives your own browser; if a site still blocks it, it says so and offers to do it on your screen with you watching. With Safari, Firefox or Arc as your default, Moe browses signed out, because it has no profile it can copy. Change your default browser and Moe follows at its next launch. Away from your Mac, Moe has the same browser but not your profile, so there it browses signed out.

Which of your profiles. Chrome keeps one set of logins per profile — the faces in its profile menu. Moe carries one of them. Left alone, it is whichever profile you used last, and if you have several that changes as you switch windows: Moe can be signed in to X in one profile and signed out in another. Settings › Permissions › Browse as lists your profiles by name and account and lets you pick the one Moe should always use; "Whichever I used last" is the other choice. It takes effect the next time Moe opens its browser. The row appears only when there is something to choose — one profile, or a browser Moe cannot copy, and it stays out of the way.

If Moe meets a sign-in page where you should already be signed in, it stops and says which profile it is browsing as, and that Browse as picks another — it does not guess a password, and it does not report the job as done.

For a tab you have open on screen, or a browser's own menus and dialogs, Moe looks at the screen the way it looks at any other app.

Moe does not ask before every click. Clicking a date, typing a search, scrolling a list, opening a page: none of that is a question. What Moe asks about, in a browser, is the moment that matters: paying, buying or booking; sending a message, an email or a post; deleting or cancelling something that does not come back; entering a password or a card number; letting an app into one of your accounts. One yes covers that kind of moment for the rest of the conversation, up to an hour, so a booking is one question, not one per page. See Before anything goes out.

Windows on another desktop#

If you use several Spaces, an app on a desktop you are not looking at cannot be clicked; macOS does not allow it. Moe can still read such a window, and it will tell you where the window is rather than pretending the app is missing. Let Moe bring a window to the front when it has to, in Settings › Voice under While you talk, is on: for the few things that cannot be done in the background, a window on another desktop, a minimised one, or a menu in an app that is not in front, Moe brings that window forward, which on a full-screen app means moving you to another desktop. Off, it leaves your screen alone and tells you what it could not reach. Reading the screen is never affected either way.

It stops instead of looping#

Every action comes back with a verdict saying whether it actually landed, and Moe checks that rather than assuming. If the same action returns the same result several times, Moe changes approach; if it happens again, the action is refused outright and Moe tells you what it is seeing and asks. Actions whose whole point is repetition, scrolling, arrow keys, undo, are never counted against that.

And it knows when to stop. A screen task has an allowance: 150 actions, and 30 looks in a row with nothing done between them. Past those, Moe stops and tells you where it got to and what is in the way, rather than clicking on for another ten minutes; the allowance refills once you have answered.

You can always stop it#

While Moe has your screen the notch changes colour, a magenta you will not see it use for anything else, so you can tell at a glance that it is clicking and typing rather than just thinking. The same colour means Moe is in its own browser, working through a page off-screen. Hover the notch and the panel names the app it is in, Using Safari, or says In its own browser, and offers a button that says what it ends: End computer use or End browser use. Pressing it gives up the whole run: no further click or keystroke is sent, and whatever Moe was part way through is abandoned rather than finished behind your back. The colour goes when the run ends. Stopping is something you do, not something Moe guesses: it cannot tell your hand on the mouse from its own typing, so it does not try.

Moe will not type a password, a passcode, a 2FA code, a card number or a recovery phrase, even if you read one out to it; it tells you the field needs you and waits.

Screen access needs two macOS permissions, Accessibility and Screen Recording, and they belong to the small driver Moe installs for this rather than to Moe itself; Settings › Permissions shows which are missing and takes you to the right place.

Settings › Permissions#

What Moe asks macOS for, and whether macOS has said yes. None of these is required: Moe works without every one of them. A permission you don't give just switches that one thing off, and Moe says so instead of pretending it looked. Nine rows, in the order they tend to come up:

• Microphone — hearing you while you hold the talk key.
• Accessibility — noticing the talk key while you are in another app.
• Input Monitoring — the same job; some Macs ask here instead.
• Automation — asking Notes, Calendar and Messages for what is in them, and clicking and typing on your behalf when you ask Moe to drive the Mac.
• Calendar — what you have on, every calendar in Calendar.app including Google ones, and adding events when you ask.
• Reminders — what you still owe people, and adding one when you ask.
• Contacts — who you mean when you say a name.
• Full Disk Access — the things macOS keeps locked away even from you: Messages and SMS, call history, Chrome history, and the Google Drive folder on this Mac.
• Screen & System Audio Recording — seeing your screen when you ask "what do I do here", and recording the other side of a call in a meeting.

The word beside each name is the state: Allowed, Not allowed, Not asked yet (nothing is broken; Moe has not needed it yet) or Can't tell. Input Monitoring and Automation always say Can't tell: macOS gives no way to read those two without either prompting you or guessing, and an honest blank beats a wrong guess. Re-check at the top reads them all again.

How do I? on a row (Details, on one that is allowed) opens what it is for, what happens if you say no, the click path in System Settings, and Open the settings pane, which takes you to the right place. The four permissions macOS only reads when an app launches, Accessibility, Input Monitoring, Full Disk Access and Screen Recording, also offer Quit and reopen Moe. On Accessibility alone there is one more: if System Settings shows Moe switched on and this still says Not allowed, that switch belongs to an older copy of Moe; Reset and ask again clears it, and you switch Moe on once more.

What Moe is allowed to do, under the rows, is about Moe's own hands rather than macOS's. Moe can do anything read-only on this Mac without asking; sending mail, sending messages, sharing files and deleting are denied to start with, because a voice assistant has nowhere to show you an approval dialog mid-sentence, so the safe default is that nothing leaves your machine. Edit the permission rules opens the file those rules live in, where you can move a line from deny to allow when you decide you want it; restart Moe afterwards. It is a guard rail, not a sandbox: the real protection is that outbound actions are off until you say otherwise, and what stands in front of a send when you do is the next page.