Click through apps to verify its own work
With computer use, Devin built and played a desktop game, ordered on Amazon, and tested its own work by clicking through the app.
Done with Devin
What they did
Devin's cloud VM harness has computer-use tools: screenshots, mouse, typing, and recording. In test mode it first writes a test plan grounded in the code. It then spins up the app, clicks through it, and annotates assertions as passed, failed, or untested. Login is a saved deterministic script. A person supplies secrets or OTP codes when asked.
How it went
Engineers ran 10 to 20 Devins in parallel, each with its own dev server. Approved test runs per day more than doubled. Timing-sensitive UI like toasts confuses it, and it sometimes cheats by running JavaScript instead of clicking.
Worth knowing
Test mode is currently billed at one fifth of normal usage cost. Expect to help with secrets and one-time codes the first time you set up a repo.
Try it yourself with Devin
Build [a small desktop app or game I describe here], then run it and click through it yourself to check that it works. Fix anything that breaks and keep going until the main flows work. Finish with screenshots or a recording and a list of what you verified. Don't make any purchases or submit any forms with real data.
Source
Read the original