Coming soon · iPhone
An assistant that adapts to how you see, move and ask. Speak or type what you want to do, follow up on the details that matter, and choose between the options it offers. From camera help to route questions, the request starts with you.
Labels, forms, receipts, menus, signs, screens and tickets. It is designed to tell you when it has reached the end of a page or when text is too small to read, rather than guess what it says. The assistant is instructed not to read out card numbers, PINs or passwords; it can still make mistakes.
Name the thing and it guides you to it with left, right and ahead, then keeps tracking it as the camera moves.
What is in front of you, what a room is like, whether the outfit works, what the sign at the counter means. Follow-up questions keep the context.
“Tell me when the bus with 27E on it comes”, “when the water boils”, “when my number shows on the display”. It says what it will look for, then stays quiet until it sees it.
Intercom and lift panels, kiosks and card machines: it tells you where your finger is in relation to the button you want.
Camera Coach frames the shot with you (pan left, step back, hold still), takes the photo when it is right, and warns if the result looks blurry or dark.
Routes come with the facts a map usually leaves out: the shape of the walk in a sentence (“mostly two long straight stretches”), available map details about crossings (signals, tactile paving, islands), stairs, underpasses, and which station on the same line is easier to board at. Several options are read out, so you can choose what suits the way you move.
On the walk, a companion calls the turns in advance and again at the corner, names the streets and landmarks you pass, notices when you have walked past a turn, and lets you ask questions on the way. The phone camera or Meta Ray-Ban glasses can add a live view of obstacles ahead. Where the phone's position is poor, you can correct it yourself: say which side the traffic is on, or pick the place you are next to. In daylight a camera fix can also help refine your location (Istanbul only so far).
For buses and the metro: nearby stops, scheduled departures, line directions, and a ride mode that calls out the stops until yours. Live bus positions are available in supported cities. In Istanbul, metro lift and escalator outages are included too.
Route information depends on available Google Maps and OpenStreetMap data and may be incomplete. Testing has focused on Istanbul.
The Offline Assistant runs a small vision model on the phone itself. Describe what the camera sees, read text, find an object by name, ask a question about the view. No account, no connection, nothing leaves the phone. It is slower and less precise than the cloud assistant, and it generates in English (translated on the phone where iOS has the language pack installed). Reading works before the model is even downloaded.