Evolution of my build failure agent

I’ve written in the past about troubleshooting build pipelines with AI. While all of this is a great step in speeding up your development and reducing the amount of troubleshooting the DevOps team needs to do in the enterprise, it is NOT the end goal.

The end goal would be to have the AI fix the problem for you.

I’m rebranding my Jenkins Sentinel to just be Sentinel. This workflow allows you to automate remediation for your pipelines while still retaining human in the loop security.

The other primary feature is storing your build failures and remediations in a database that you can view, update, analyze for custom model training.

Originally we had the dispatch layer that would notify us of build failures and possible resolutions. The new addition is the cluster of “workers”. Running on AWS Fargate, this team of developers works with the LLM on Bedrock to resolve the failure.

1. The task spins up in the cluster
2. The build logs identify the repository and branch
3. The repository is cloned, and branch checked out
4. The code fix is implemented
5. The task generates its reasoning and updates the database accordingly
6. Code is committed to a new branch and a pull request is opened.
7. The task cleans up and shuts down

Dispatch still remains the same and the developer is notified accordingly. I need to implement developer specific notifications so that channels are not flooded or email lists abused.

The other major thing I wanted to see was the cost per fix.

This screenshot is from the dashboard which shows the compute spend and the LLM spend. For this simple Terraform fix you can see the was a little around $0.02. Assuming your code bases are more complex this value could increase proportionally.

I also included a stats page which shows the totals for the entire organization.

This is all real data from my testing project. The build agent is successfully troubleshooting pipelines for:

• Python
• Terraform
• Java
• Typescript
• Docker
• Kubernetes
• Go
• Cloudformation

I plan to continue to add more supported platforms and languages as time allows. The other major integration that I am working on is support for GitHub Actions. Once I complete that integration and put this into all of my pipelines I expect that my troubleshooting and development time will decrease rapidly.

Other future plans include:

• Ingestion of bugs through sources like Jira, ToDoist (my favorite), or another ticketing system.
• Discord Dispatching
• Teams Dispatching – although this is really hard to develop for without a paid account
• Custom model – using the build failure data to train a model

Anyway, this project has been super fun. If you want to implement it on your own infrastructure feel free to reach out!

PS: the featured image was generated and setup through my Nano Banana WordPress plugin