Our hosted model's behavior shifted overnight
I work on Beat, a tool which reviews pull requests (among other things) on cloud and hands the user a card to review and approve the work before anything is posted or sent out. Several days ago, our PR reviews got nearly 20x slower in a single evening. Nothing in our code changed, nothing in our environments or configs, simply put, the models drifted and we didn’t catch it in time.
How our reviews work
Our reviews run in two main stages. A few passes raise possible issues from entire code diffs, then each issue gets its own verification, where a model reads the entire code including usages and refs, and must prove or disprove the original issue. Almost all the “cost” and time is in that stage.
Nothing looked wrong
Nothing was inherently wrong, and our monitoring didn’t pick it up. Reviews were still being posted, bugs were still being found, and the product seemed to be working as normal, except I noticed it was taking my team a lot longer to review small PRs that I was making. I thought something must have changed, but nothing did… except the model. Our cloud provider had server degradation for multiple hours and after coming back to “normal”, our verification model’s behavior simply changed.
Proving it
To prove it, we reran old evaluation sets with the exact same build and settings from a few weeks earlier. Thousands of reviews, tens of thousands of entries. Nearly every single review took longer and had noticeably larger outputs.
Was it worse?
Then the big question, was it worse? We eval’d every single finding for accuracy and false positives. Accuracy remained very high, while false positives actually went down. The model ended up being more thorough, not less. So, we kept with it and decided to improve our system around it instead. Increasing server capacity, parallel workers, changed our severity rubric, and maybe most importantly added better monitoring and telemetry to ensure that speed was being checked in real-time, not after someone gets frustrated.
Why I still love this work
I have been building AI systems for several years now, and software for a lot longer, and yet there’s always something exciting and challenging. I know a lot of people are feeling the doom and gloom around software engineering longevity in today’s age, but I for one believe that our skill was rarely typing, rather it was creative thinking, problem solving, and collaboration that have real value. That’s what we’re trying to build with Beat. Eliminating the rote work, so that we all have more time to do what we do best.
Try Beat
Beat is in a fully free private beta right now, but if you’re curious to check it out head to gobeat.ai I’d love to get some feedback. I’m also curious to hear how others have dealt with hosted models drifting.