Disclosure, I work on operatex.dev, only mentioning it because it fits. The line about not accepting an argument that the code works when you can watch it work is the whole post in one sentence. Reasoning is cheap to fake, a passing request isn't.
Silly question: What would you say is, in simple terms, the bottom line net tangible benefit to this approach?
My concern is this:
It sort of reads like the premise of the post is, "the goal should be read as little code as possible". The bot seems to be very carefully gated, it looks great - but the gates all seem based on blast radius and effectively, what the bot should *not* review.
It also concedes the big tension of AI, which is "the dev owns the outcome, not AI".
And the bar for what *is* ok seems to be, in short, "stuff we used to rubber stamp without probably checking well enough," unless I misread it. So... Is the model doing more than replicating the rubber stamp pattern? And how do you verify or train that in if that's the existing PR pattern?
Are there any metrics to quantify the quality, not just quantity, of those approved PRs?
Not pushing back on the value, and sorry for the inquisition. Just having trouble articulating that value clearly after reading the article and glancing over the code, so I'm sure I'm missing a lot. I didn't exactly stand it up locally and test drive it, tbf. 😅
Fascinating post though! Interesting work overall and super looking forward to learning more!
any PRs that contain words from our deny-list are automatically rejected by StampHog and sent to a human for a review. that includes security, privacy, etc.
Great ideas. Thanks for sharing.
glad it was helpful!
do you guys do self-improvement for the qa-swarm? what happens if something is missed during review?
Disclosure, I work on operatex.dev, only mentioning it because it fits. The line about not accepting an argument that the code works when you can watch it work is the whole post in one sentence. Reasoning is cheap to fake, a passing request isn't.
Helpful. Thanks for sharing.
Silly question: What would you say is, in simple terms, the bottom line net tangible benefit to this approach?
My concern is this:
It sort of reads like the premise of the post is, "the goal should be read as little code as possible". The bot seems to be very carefully gated, it looks great - but the gates all seem based on blast radius and effectively, what the bot should *not* review.
It also concedes the big tension of AI, which is "the dev owns the outcome, not AI".
And the bar for what *is* ok seems to be, in short, "stuff we used to rubber stamp without probably checking well enough," unless I misread it. So... Is the model doing more than replicating the rubber stamp pattern? And how do you verify or train that in if that's the existing PR pattern?
Are there any metrics to quantify the quality, not just quantity, of those approved PRs?
Not pushing back on the value, and sorry for the inquisition. Just having trouble articulating that value clearly after reading the article and glancing over the code, so I'm sure I'm missing a lot. I didn't exactly stand it up locally and test drive it, tbf. 😅
Fascinating post though! Interesting work overall and super looking forward to learning more!
How do you handle SOC 2 with StampHog approvals?
any PRs that contain words from our deny-list are automatically rejected by StampHog and sent to a human for a review. that includes security, privacy, etc.
you can check out the deny-list here (added to the post, too - thanks for the idea!): https://github.com/PostHog/posthog/blob/master/tools/pr-approval-agent/README.md#t2--never-ai-approved
SOC 2 requires at least 2 humans involved in the code delivery