5 AI Engineering Challenges CTOs Face

An agent can build a feature in an afternoon. Then the feature waits on review, testing, and a security check.

That was the pattern CTOs kept describing to me. Writing code is no longer the slow part. Everything after it is.

Over the past quarter, I led three AI Office Hours sessions for 7CTOs, a community of senior technology leaders. They brought real problems, from managing agents to improving QA to making architectural decisions with AI. 

5 Challenges Came Up Again and Again:

1. AI-Generated Code Still Needs Architectural Oversight

AI can generate working code in minutes. But someone still needs to make decisions about architecture, scalability, security, and maintainability.

At Six Feet Up, we use Superpowers to bring structured brainstorming, planning, and test-driven development into AI-assisted workflows. Engineers still guide the workflows, evaluate tradeoffs, and catch problems early.

Solution: Define architectural standards, then require a structured plan and review before agents begin implementation. 

2. Poor Context Management Leads to Unreliable Results

AI agents need accurate, relevant information to make good decisions. Managing that context gets harder as projects grow and multiple agents get involved.

During Office Hours, we discussed the challenges of maintaining knowledge bases, tracking architectural decisions, and managing different types of agent memory. A larger context window won't necessarily solve these problems.

I've been experimenting with Pi, a lightweight coding agent that lets me control which skills and tools are loaded, keeping context focused on the task at hand.

Solution: Keep documentation current, load only the skills and tools each task needs, and give agents the information relevant to that task. 

3. One-Off AI Experiments Don't Scale

Successful AI experiments get far more valuable once they become reusable workflows. 

At Six Feet Up, we’re experimenting with shared agent instructions, development environments, and tools that capture our engineering practices. Developers contribute reusable skills to our repository called the “Promptorium,” so a good approach found by one person becomes available to everyone. 

The same idea works at the individual level. I’ve used AI to write Python scripts that automate repetitive document processing. Instead of sending every document through a model, the scripts handle the work locally. That reduces token consumption and makes the process repeatable.

Solution: Capture successful approaches in version-controlled skills, scripts, and development environments that the entire team can use.

4. QA Can't Keep Up With Development Velocity

When an agent can build a feature in an afternoon, testing becomes the constraint. Code arrives faster than QA can verify it. 

Test coverage came up in our discussion, and it’s the metric most likely to mislead you. Coverage tells you which lines ran during a test. It doesn’t tell you whether the test checked anything useful. AI can push coverage past any target while asserting very little. 

AI is good at drafting tests, surfacing edge cases, and automating regression checks. It is not good at knowing how customers actually use the product, or noticing when a passing suite describes the wrong behavior. That part still belongs to people. 

Solution: Bring QA into planning and design early. Use AI to expand test coverage, but keep people responsible for validating the product.

5. Autonomous Agents Introduce New Security Risks

As agents gain access to codebases, APIs, and infrastructure, security and observability become increasingly important.

We want visibility into what agents are doing so we can catch problems early. I've been using AgentsView to monitor agent activity, context pressure, and tool failures. We're also focused on sandboxing and limiting agent permissions.

Solution: Build access controls, monitoring, and review checkpoints into agent workflows from the beginning. 

Here’s a short demo of AgentsView in action. 

Shared Problems Get Solved Faster

The 7CTOs discussions are valuable because other CTOs are working through the same challenges we are. We’re testing new tools, comparing approaches, and figuring out how to put AI to work without sacrificing the engineering practices that make software reliable.

Follow me on LinkedIn for more insights.