Episode 4

full
Published on:

12th Jul 2026

The Software Engineering Loop: Contracts, Observability, and Knowing When to Modularize

Ajay and Andrew start with a cat-induced re-recording and end somewhere they did not expect: the first real "yes" to the show's core question. In between they dig into the software engineering loop, why contracts and modularization make AI-assisted work parallelizable, how to tell when a system actually needs to be split up, and why observability and flexibility are the two tenets they will not give up. They compare Codex, Claude, Fable, and GLM in real use, from usage limits and precision to one-shotting and transparent peak-hour pricing, trade notes on adopting the GSD workflow without losing their own, and watch Sonnet 5 debug an AWS permissions rabbit hole on its own.

In this episode:

Contracts and modularization, and why they make AI-assisted work parallelizable

How to tell when a system actually needs to be split up

Codex vs. Claude vs. Fable vs. GLM in real, day-to-day use

Why observability and flexibility are the two tenets of the loop

Adopting a shared workflow without giving up a hand-built one

The first real "yes" to the show's question: act, or intelligence?

Chapters:

(00:00:00) - Welcome (and a cat-induced take two)

(00:02:42) - The "software engineering loop": planning and parallelization

(00:05:20) - What's a contract? APIs, JSON, and Amazon Bedrock

(00:08:04) - Versioning contracts: 1.0 and breaking changes

(00:12:09) - Enforcing contracts: validation, CI, and TDD

(00:14:39) - Don't modularize too early: velocity as the signal

(00:17:23) - Model limits and generosity (and Grok 4.5 lands)

(00:20:24) - Codex's precision vs. Claude's fill-in-the-blanks

(00:22:39) - Concise vs. detailed, and OpenAI's Sol/Luna/Tara

(00:25:30) - Trying GLM from z.ai, and transparent peak-hour pricing

(00:32:09) - A bash wrapper to swap models into the Claude harness

(00:34:47) - Back to Claude: why Fable wins on efficiency

(00:37:07) - Is Fable one-shotting because SWE principles got trained in?

(00:38:19) - Sonnet 5 down an AWS SAM permissions rabbit hole

(00:42:41) - Observability for coding agents (LangFuse, Braintrust)

(00:45:16) - Adopting GSD: hooks, overlap, and what to steal

(00:53:03) - Two tenets of the loop: observability and flexibility

(00:54:38) - Blind spots, rituals, and what GSD surfaced

(01:01:36) - Act or intelligence? The first "yes"

Transcript
Ajay:

Hello, my name is Ajay Medury, and I'm a software engineer.

Ajay:

And today, I'm joined by my co-host.

Andrew:

Hello, my name is Andrew Sierota, and I'm a systems engineer.

Ajay:

And we're hosting our podcast called Act of Intelligence.

Andrew:

Which is about our experiences using Codex, Claude Code, Gemini, and other tools in

Andrew:

the space.

Ajay:

So we're trying to answer the question of whether they're actually intelligent or

Ajay:

just pretending.

Andrew:

We're a podcast for creatives, creators, and builders, or anyone with an interest in

Andrew:

the space.

Ajay:

Awesome.

Ajay:

And, yeah, we're here for episode number four.

Ajay:

Episode four.

Ajay:

And, yeah, yeah.

Ajay:

And we just had to start re-recording because my cat decided to run up the

Ajay:

tree and create some interesting noise.

Ajay:

And then I decided that that was going to cause me to butcher multiple words.

Ajay:

And I said, all right, we're going to take it again.

Ajay:

take two take two take two so uh we were also technically off the air because the

Ajay:

light that's always been on we have this little light hanging in my room where we're

Ajay:

recording right now it's a little ramen uh neon light that you know we had to turn

Ajay:

on it's it's the on air sign we have we need it it's it's important there there are

Ajay:

there are again uh rituals to this as we've talked about rituals you know yeah we'll

Andrew:

show it on video someday someday someday not not the current state of the room

Ajay:

though not not in this state when it's in a better state there we go there we go um

Ajay:

and but talking about rituals again I think you know that brings me back to this old

Ajay:

thing that we were uh mentioning previously about software engineering and like

Ajay:

essentially how it's a lot of rituals and it's like a process and we do certain

Ajay:

things in certain ways um we did actually start talking a little bit before we hit

Ajay:

the record button before uh you know kevin my cad decided to run up the tree as well

Ajay:

about like some interesting topics about planning something that we talked about and

Ajay:

you know maybe episode one episode two uh specifically like okay you know you can

Ajay:

get started you you know you have to do the set out a plan at some stage before you

Ajay:

actually start writing code uh the plan could be like okay in the next two months

Ajay:

I'm going to do you know phase one phase two phase three of my project and that's

Ajay:

how I'm going to get it executed we did briefly talk a little bit about like also

Ajay:

doing paralyzation of that plan like how do you paralyze work so multiple people can

Ajay:

work together which is I think one of the more interesting parts of like actually

Ajay:

working for a bigger software company and a very big especially big tech in my

Ajay:

opinion I think that's one of the interesting things which I do feel they do well is

Ajay:

they figure out how to split up work with that in mind we started talking about this

Ajay:

because I had devil or I had seen some interesting videos and I heard some

Ajay:

new things about a new software engineering loop that I was very

Ajay:

interested sdlc with agentic coding involved which is again I think

Ajay:

this word loop is becoming more popular recently so I know I'm I'm kind

Ajay:

of dragging on to interest more and more topics I'll come back and bring it back

Ajay:

around in a moment here uh the idea of the loop here is again it's pretty

Ajay:

straightforward it's like we're becoming automation engineers with all the coding

Ajay:

assistants and lms doing such a good job so the concept of loop loop very

Ajay:

interestingly again for me came back to my idea of processes in the past and again

Ajay:

like this loop in particular was very interesting because it was executing on a

Ajay:

single repository where everything lived in that single repository I had an issue

Ajay:

with that because I'm coming from a background engineering background where I'm

Ajay:

trying to split things up where I am trying to parallelize things.

Ajay:

And typically, one of the ways to parallelize things is actually to split up your

Ajay:

bigger repository, your bigger system into smaller subcomponents.

Ajay:

You used the word modules in the past.

Ajay:

Though I think the question is, there's also a stage at which a module becomes a

Ajay:

subcomponent where it's like, not only is it actually separate logically, but it

Ajay:

also has a concrete contract that you cannot interact with that particular module or

Ajay:

subcomponent until the contract the input contract the output contract are satisfied

Ajay:

I mean input contract is satisfied the output contract is accepted so that is

Ajay:

actually where I wanted to come back all this way and talk about planning and the

Ajay:

parallelization because at least I feel like it is a lot easier to do planning

Ajay:

and parallelize work if there are solid contracts between sub modules and

Ajay:

you can have individuals or AI coding assistants work on those sub-modules in

Ajay:

isolation without essentially disrupting the others.

Ajay:

So there's less overlap at times, so more strict contracts, and that actually allows

Ajay:

you to then essentially say that, hey, this module, this sub-component needs updates

Ajay:

in this way, and I can upgrade that without impacting any of the other work going on

Ajay:

in the other sub-modules.

Ajay:

And we talked a bit about this example of like imagining five separate sub-modules,

Ajay:

they're all interlinked, but I want to upgrade one and four, then I should be able

Ajay:

to upgrade them safely without impacting all the other relationships, without

Ajay:

impacting all of the other modules that may be expecting the same output from it.

Andrew:

That makes sense.

Andrew:

And I think for maybe the viewers who are just tuning in, could you explain the

Andrew:

concept of contracts a bit more?

Ajay:

Absolutely.

Ajay:

No, thank you for mentioning that.

Ajay:

I think at least it is a term I'm glossing over a little bit.

Ajay:

So I can definitely appreciate you calling it out.

Ajay:

Idea of a contract is not so legal in the software world, right?

Ajay:

Like in the software world, it's like, hey, I have a system, I have a software

Ajay:

system.

Ajay:

A contract for a cloud software system would usually be an API.

Ajay:

So an application package interface.

Ajay:

So the intention is that it is a application, and there is an interface that is

Ajay:

available to that application.

Ajay:

And the interface itself defines a set of input fields.

Ajay:

So things like maybe it's a JSON format, which is a kind of a data, you know,

Ajay:

not storage, but how would I put it?

Ajay:

Like it is a way of representing data and information.

Ajay:

Just getting very simple with it.

Ajay:

Like there's also things like XML or other, you know, data.

Ajay:

CSV technically is one, and CSV is what you would see in most spreadsheets, right?

Ajay:

So like if you open up Excel, back in the day, it used to be CSV is the most basic

Ajay:

format of it.

Ajay:

the least rich format of it.

Ajay:

JSON is one of the more common formats for a lot of APIs.

Ajay:

It's a very popular solution.

Ajay:

There are many more.

Ajay:

There are plenty.

Ajay:

Any software engineers listening to me, they'll probably tell you that I'm talking

Ajay:

about a very old standard, a very old technology.

Ajay:

Yes, somewhat.

Ajay:

It has just been very popular for the space that I was working in.

Ajay:

So the idea is that you have a JSON that is essentially a representation of

Ajay:

your inputs that you're giving you to the system.

Ajay:

The interface is actually defining what are valid inputs, what are valid values for

Ajay:

those inputs.

Ajay:

So things like if I have five fields saying that this is, an example would be if I'm

Ajay:

calling Amazon's bedrock solution, which is essentially hosting an LLM model for me.

Ajay:

I would have to tell it which model I want to invoke.

Ajay:

I would have to tell it what the input to the model should be.

Ajay:

I would also have to tell it maybe additional information like who I am, why I

Ajay:

should be allowed to invoke it?

Ajay:

Like, am I paying money?

Ajay:

What is my token or something by which I'm allowed to invoke it?

Ajay:

So all of those are part of the interface, part of the contract that I'm essentially

Ajay:

fulfilling when I'm calling the system.

Ajay:

And then that system can respond to me in the appropriate output format, which would

Ajay:

be the response from the LM in this case.

Ajay:

And also maybe some of my metadata reflected back to me saying, what is the model?

Ajay:

What is the number of tokens that produced and things like that.

Andrew:

So it's kind of like it is a contract, as

Andrew:

it's called, is an expectation of receiving and outputting data a certain

Andrew:

way.

Andrew:

Yes.

Andrew:

Receiving an input in a particular way and getting the output, putting out an output

Ajay:

in a particular way.

Ajay:

Yes.

Andrew:

And so you have these contracts between different modules to have consistency in the

Andrew:

data.

Andrew:

Absolutely.

Andrew:

Absolutely.

Ajay:

because if I'm able to formalize a version 1.0 of a

Ajay:

contract, that means now this is a promise that I've taken.

Ajay:

I will always output data in the right format.

Ajay:

I will always accept input as long as it matches a format that I've promised or I've

Ajay:

agreed upon in this version 1.0.

Ajay:

And 1.0s are usually a very big deal because that is essentially saying this is a

Ajay:

stable contract.

Ajay:

I will honor this for the minimum lifetime of my software.

Ajay:

It could be anywhere between five to 10 years.

Ajay:

Different companies have different expectations.

Ajay:

And of

Ajay:

course, if you're installing something on your machine, it's different than also

Ajay:

like if something is living in the cloud.

Ajay:

When I said contracts, in this particular case of Amazon Bedrock, it's a software as

Ajay:

a service solution.

Ajay:

Contracts don't always have to be for like just cloud solutions.

Ajay:

It could be like, hey, this is, I'm going to go ahead and do something really simple

Ajay:

in my CLI.

Ajay:

I'm going to call copy CP command on my CLI.

Ajay:

As long as I give it the right inputs, the output should be that it does the action

Ajay:

I want it to do.

Ajay:

So that is also considered a contract in a sense, yes.

Andrew:

Okay, that makes sense.

Andrew:

It's all about keeping things predictable so that there's a predictable process or

Andrew:

standardization.

Ajay:

Yes, because as long as it's predictable, you can assume that it's going to be safe

Ajay:

for me to work around it, do other things around it, and that part will always be

Ajay:

consistent.

Ajay:

And even if other things, even if I go from a version one to a version 1.5, that

Ajay:

version one, because it's still the main major version, it should still be backwards

Ajay:

compatible.

Ajay:

Everything within version one, every minor version should still not usually not

Ajay:

introduce breaking changes unless there's a huge bug.

Ajay:

There's a security gap.

Ajay:

Those are the circumstances where it is justified to introduce breaking changes.

Ajay:

But typically, most software contracts will keep things consistent, that they won't

Ajay:

break you as long as you stay within this same major version.

Andrew:

So how do you ensure that as you're upgrading the modules, that all that data, the

Andrew:

contracts are upheld?

Ajay:

Typically, it would be, that's a really great question, because I think then we're

Ajay:

really getting into how the contract is enforced, not just the fact that there is a

Ajay:

contract.

Ajay:

and I think there are input validations which are basic expectations saying that hey

Ajay:

your input needs to be a particular schema so before I even start any work the

Ajay:

system starts doing any work for you I'm going to go ahead and actually just reject

Ajay:

it if it doesn't meet the schema basic expectations that I need in order to start

Ajay:

the work so the example of bedrock and you know amazon in particular is if you don't

Ajay:

give me a model I can't I can't do anything I don't know what the model ID is.

Ajay:

I don't know what I'm supposed to invoke behind the scenes.

Ajay:

I don't know if it's an anthropic model, if it's Ollama.

Ajay:

It could be anything, right?

Ajay:

Like, I don't know which one it is.

Ajay:

So I need that information.

Ajay:

So I'm going to basically reject your request for work because it doesn't meet the

Ajay:

basic bar or the minimum bar required.

Ajay:

At the same time, there could be additional parameters that are optional.

Ajay:

So I could also say things like give me model weights, give me model temperatures.

Ajay:

I don't know if model weights is actually a supported thing for Bedrock, but there

Ajay:

are other providers where it could be the parameters, the model temperature and

Ajay:

things like that are things that can be supported in Bedrock.

Ajay:

But the idea is that those are more optional, like the model will just execute in

Ajay:

the way that it's set up unless you do some of these modifications.

Ajay:

So the idea is that the basic requirements for work should be satisfied, otherwise

Ajay:

your request will get rejected.

Ajay:

That's the input side validation.

Ajay:

On the output side, it is more difficult for a system to govern its own output.

Ajay:

So you typically have controls like tests and pre-deployment CI.

Ajay:

You do things in your CI-CD pipelines before you actually deploy the change out to

Ajay:

consumers.

Ajay:

You make sure that this is part of the software engineering life cycle.

Ajay:

And one of the rituals, which is probably one of the better rituals, is you're

Ajay:

validating that your output contract still works because there's no way for the

Ajay:

system that's creating the output to guarantee.

Ajay:

There could be a bug that was unintentionally introduced that could have broken the

Ajay:

output contract.

Ajay:

So one has something called regression tests.

Ajay:

The regression tests are basically validating that the previous version, previous

Ajay:

minor version was still up to date and still is backwards compatible with the next

Ajay:

one.

Ajay:

So I'm not breaking anything that my previous minor version had made a promise about

Ajay:

or anything that my major version expects as an expected output promise.

Ajay:

Yeah.

Ajay:

Those are some of the basic ways of doing it, but yes, those are enforcements.

Ajay:

Those are kind of the boundary level enforcements.

Ajay:

You can have a bunch more stuff going on in between but I think those are kind of

Ajay:

the basic starting points yeah okay that makes sense yeah I think in like my

Andrew:

projects right now I've been implementing a lot more tests

Andrew:

especially since I've realized in a larger code base starting from scratch when

Andrew:

there's nothing there you're trusting clog code or codex to make it from

Andrew:

scratch and follow your spec sheets well just because you wrote down requirements

Andrew:

just because you wrote that in the spec sheets just because you traced everything

Andrew:

out that you want doesn't mean it's gonna follow it no right there's so many things

Andrew:

that can happen it can compact midway um might not just have enough you know maybe

Andrew:

the servers were really overloaded at that point and they didn't really prioritize

Andrew:

you got a quantized model without expecting maybe they weren't giving you your best

Andrew:

at that yeah just at that moment Yeah, and I've been figuring out that

Andrew:

adding in a lot more tests or maybe even testing first development

Andrew:

might be the key to getting much more consistent

Andrew:

results out of the code.

Ajay:

Yeah, and I think this might be more of a, it could be considered a subjective

Ajay:

opinion in the software space.

Ajay:

I did not do test-driven development for a long time, though at the same time, I

Ajay:

think the more I develop with AI coding assistants, the more value I found in it, at

Ajay:

the very least explaining what the expectations and output are, because I may not

Ajay:

write the test myself, but at the very least I tell what should be the result at the

Ajay:

end, and then I expect the model, I tell the model, verify your own solutions and

Ajay:

explain to me how you're going to do it.

Ajay:

Models have gotten so good that they can write their own tests, they can write their

Ajay:

own verification steps, they can start opening up a browser for you to check if the

Ajay:

UI looks the way that you want it to look.

Ajay:

But

Ajay:

the missing pieces, you still need to tell it what the output is supposed to be.

Ajay:

And one benefit, essentially, one thing I want to also clarify is breaking down your

Ajay:

system prematurely is not necessarily a good thing to do either.

Ajay:

So it's actually very common.

Ajay:

One of the typical things with over-engineering in software engineering can also be

Ajay:

that you're trying to break your system down into smaller components when you don't

Ajay:

need to, because there is a certain point of complexity that you need to hit.

Ajay:

That complexity being that now if I have five engineers working on the same system,

Ajay:

they can't collaborate anymore.

Ajay:

That their velocity, the number of pull requests they push out, the number of

Ajay:

features they push out, grinds to a halt.

Ajay:

Because you can measure this before, and over time, if you keep measuring it, you

Ajay:

will be able to tell very clearly based on the drop and velocity whether your system

Ajay:

needs to be modularized and some subsystems, you know, subcomponents need to be

Ajay:

separated out into their own contractual like boundaries.

Ajay:

Because then you can actually safely make changes to one without breaking the other.

Ajay:

This is a common thing that software engineering teams and, you know, product teams

Ajay:

will run into if they're getting big enough, if they're getting successful enough.

Ajay:

Because as I build a hundred features for a hundred different customers at some

Ajay:

point is going to get so complicated because they're all it's all intertwined to

Ajay:

such an extent that my engineering team can't actually produce the same amount of

Ajay:

code that they would I'm not able to build new features because of it uh that is

Ajay:

also a great so the question was like or the statement I was trying to make is uh

Ajay:

breaking your system up prematurely is not a good thing how do you measure when to

Ajay:

break it up is usually velocity so I think that's one thing andrew when you started

Ajay:

mentioning recently for your project for your Minecraft project.

Ajay:

When you told me that it kept getting stuck on certain modules, it was like stuck at

Ajay:

95%, it was stuck at 97% for a little while.

Ajay:

That showed me an indication of like, okay, the curve is now starting to trend

Ajay:

towards zero, like it's getting really bogged down.

Ajay:

And I think that's where I wanted to bring it up as a potential suggestion is to

Ajay:

create those contracts in some way, and split it up in some way, because that's a

Ajay:

very clear indication that everything is so intertwined with each other that you

Andrew:

make one change in one place, you have to go that makes a lot of sense yep and and

Andrew:

uh I do have it scoped out to break it into a lot of sub-modules coming

Andrew:

up but we did finally close on at 100 for that module the economy

Andrew:

module so uh you know fingers crossed yeah it took a while it was a persistent

Ajay:

effort I think about almost a week straight yes I was seeing the timelines and I was

Ajay:

very impressed I was like it is truly persistent how many models were going at the

Ajay:

same time um it it was mostly codex since uh I guess another segue is the limits for

Andrew:

these different and subscriptions are very different um I would say codex is

Andrew:

generally a lot more generous um and I'm talking about the 20x max

Andrew:

um claude is not as I guess more hungry maybe maybe it's what I should

Andrew:

say more hungry more hungry who knows could be the same amount of generosity

Andrew:

yeah because uh for example um different models will have different expectations of

Andrew:

how much tokens they're going to use to solve a different problem there's different

Andrew:

benchmarks out for that and just recently today uh grok 4.5 came out wow and

Andrew:

apparently has really really good agentic coding ability now which is not

Ajay:

something I thought I would say anytime soon um I mean I guess the uh

Andrew:

cursor purchase might have helped a little bit potentially early days but maybe yeah

Ajay:

yeah I am curious though I think uh you did mention the hungriness of

Ajay:

the models um I guess maybe this also again maybe brings me back to this question of

Ajay:

like we talked about codex behaving a particular way last time and if I were to vibe

Ajay:

code a project I would probably choose codex because it doesn't expect all these

Ajay:

things it doesn't slow down it just kind of says okay I'll go do what goal you want

Ajay:

me to achieve um it sounds more again this gives me credence to like my thought

Ajay:

process like clod code was always built for software engineering teams where they

Ajay:

are today which is where I would also almost make this claim now is like once things

Ajay:

got really complicated it just got worse and worse at handling the complexity

Ajay:

because then it's trying to be very certain it's like approach I'm almost curious I

Ajay:

don't know if certain is the right word I don't but the behavior I'm hearing sorry

Andrew:

yeah so so at least online from what I've read from people's experiences and

Andrew:

obviously I have a bit of experience on both models now but I'll tell you what

Andrew:

everyone else has said first um people say that codex is a lot more exact in

Andrew:

its execution so if you give it a very specific set of instructions or a spec sheet

Andrew:

it generally from what people to say online it's going to be a bit more precise all

Andrew:

right I'm not saying it's going to be a precise but it's going to be a bit more

Andrew:

precise than clod code and so clod code specifically opus I guess since that's the

Andrew:

one I've been using the most yep um will be a lot more liberal in

Andrew:

its approach with your spec sheets so if you've left any gaps it

Ajay:

will fill it in for you which which can be a good thing can be a good thing

Ajay:

because that is ideally covering anything that you've missed yeah

Ajay:

uh which is good for high accuracy for systems that you're building with like a

Ajay:

high expectation of accuracy I guess now the question becomes is is that always a

Andrew:

requirement so I think that goes back to our first episode the the uh accuracy or

Andrew:

precision is you know directly correlated to the amount of risk involved that is

Ajay:

yeah that I love that framing I totally yeah can get absolutely behind it yeah and I

Andrew:

guess maybe uh you mentioned the online version andrew what is what is your take so

Andrew:

far right and I think for my take it's hard to because I'm not reviewing the code by

Andrew:

hand it's impossible to do right it's not feasible you're not you're not

Ajay:

reading 40 000 lines per pr no it's more than that sometimes all

Ajay:

right um but from what I can tell um from the

Andrew:

times where it like when I ask it to in particular review stuff

Andrew:

and tell me what's happened yeah yeah I have found that codex is a lot more specific

Andrew:

on what it tells me and it's and it tells me interestingly enough more of the

Andrew:

information I want to know yeah whereas Claude Code is very verbose up front

Andrew:

talks a lot yeah but not necessarily information I want to know or is

Andrew:

important so I think maybe in that sense codex is being a bit more

Andrew:

terse but also the data it's giving you seems to be more relevant uh you have a

Ajay:

very very good point there because I think I've had a similar experience where

Ajay:

I'm like uh codex chat gpd in general both of them are I'm assuming they're built

Ajay:

off of the same model in general like behind the scenes they're using is just a

Ajay:

different harness than like the tuning for codex um the um I'm saying that because

Ajay:

they're both from open ai and there probably are buildings that uh but the um it

Ajay:

tends to give me a concise answer to the point for the question I ask it at that

Ajay:

given time and then I feel like as you mentioned Claude Coe tends to be like let me

Ajay:

cover my bases let me tell you why I thought this way let me tell you what I

Ajay:

discovered along with the process and give you this whole breakdown of things which

Ajay:

I again there's the reason I like both there's a time and a place there's a time and

Ajay:

a place I truly agree I have also talked to friends and other colleagues in the

Ajay:

space who are usually like well the system prompt we like we talked about last time

Ajay:

one of the things that they give it as part of its uh instructions in the first

Ajay:

place is be very concise answer in a few lines or a few lines or less and then only

Ajay:

give me more information when I ask you for it so they're specifically trying to get

Ajay:

it to be closer to codex or you know like uh at least from a responses perspective

Ajay:

the one thing that they do mention though the actual artifacts that generates like

Ajay:

the markdown files and things like that they do prefer it actually has more

Ajay:

information at the end of the day because they want to be able to go find it I think

Ajay:

I agree from that perspective too because I may not I don't want it to give me a

Ajay:

whole breakdown of every single thing it did in a 25 pull request you know change

Ajay:

where it had like 25 changes going on but I do want it to tell it's the handoff

Ajay:

document that it hands to another agent like all that information so I think that's

Ajay:

the difference is like when it's interactive to me I prefer it be more concise but

Ajay:

when it actually is giving something as a handoff I wanted to be more detailed but

Ajay:

also include all of these what I did why I did it kind of the engineering talk of

Andrew:

like okay what was the whole scenario and I think fable has has leaned towards that

Andrew:

greatly I would say fable is very quiet so they've been they get it I

Andrew:

think even more quiet than than codex in particular gpt

Andrew:

5.5 and today also gpt 5.6 has released along with their

Andrew:

sauna and haiku models I think it's soul luna tara luna so I

Andrew:

think soul is their fable I think luna is their opus and

Andrew:

and or actually tara is their opus and and luna is their haiku so the sun the

Andrew:

earth and the moon yeah yeah yeah but it makes sense so they're actually copying

Andrew:

claude yeah in the it's kind of like you know intel and amd when they started

Andrew:

to use like i3 or r3 or seven you know you know there is logic behind that there is

Ajay:

logic you're starting to establish one company starts to establish a trend the other

Ajay:

one wants to make sure that uh the same folks who are now buying from that one

Ajay:

company also understand its offerings by giving a similar trend of like naming

Andrew:

convention or tiers so I mean this is the first time they've given like three models

Ajay:

that is that is that is pretty interesting so it's a direct it's a direct comparison

Ajay:

yeah now now they're like you can compare us one to one and see which one you like

Andrew:

exactly exactly and I haven't had much experience with 5.6 it just released

Andrew:

today so oh yeah yeah but uh with fable quite a bit more uh but at least from a 5.5

Andrew:

perspective it is quite terse you mentioned fable though because I think you said

Ajay:

recently we talked about you going back to fable going back to claude yeah because I

Ajay:

think for a minute that you were trying something different yeah a little bit about

Andrew:

that yeah so I for a little bit there I was just interested in trying um I heard a

Andrew:

lot of good things about glm 5.2 from z.ai and I think their company name is a bit

Andrew:

different but that's the main site where you can get this open source model from

Andrew:

yeah uh fireworks is the other popular one that I've heard of yeah and the main

Ajay:

reason I know that is because I think fireworks is a big model training company also

Ajay:

they give the whole like end-to-end stuff but yeah sorry I keep going um and I've

Andrew:

heard it's close the benchmarks show that it's kind of close to like opus

Andrew:

all right I'm not saying it's better but it's it's close and okay it's also

Andrew:

something that feasibly if you had a hundred grand or so floating around you could

Andrew:

potentially run at home right at home yeah and just a measly hundred yeah yeah yeah

Andrew:

I mean it's it's a little bit more in reach though than you know something like I

Andrew:

don't know fable right oh yeah so so it was I was just interested to try it and it's

Andrew:

a little bit cheaper I think with the referral code I found online it was like a

Andrew:

hundred and thirty dollars usd including tax for similar limits for for

Andrew:

20x they're they're they're all 20x they're all 20x yeah yeah um that's

Andrew:

not bad though because uh clod and is it's quite a bit cheaper yeah yeah yeah about

Andrew:

33 percent cheaper that's what I was gonna say it's like two-thirds of the total

Andrew:

value yeah yeah price but in my short time using it so far so so in I actually

Andrew:

attached the clod code oh yeah okay so use the same harness use the same harness

Andrew:

and I found that I couldn't really tell the difference between me talking with it

Ajay:

and opus and I think that could be partially the harness yeah because of the

Ajay:

instructions and all the system problems and all the stuff that go yeah yeah and so

Andrew:

I would say it's it's a pretty good drop in replacement for opus just from like

Andrew:

talking with it briefly okay and after a few coding reviews on some a

Andrew:

module that was finished it did find quite a bit of okay issues warnings and

Andrew:

blockers that it need to fix nice so I was impressed with that too but what I wasn't

Ajay:

impressed with was how fast I went through my five hour limit oh okay so time

Ajay:

base limit was not the same with them yes yeah I I'm not sure 100

Andrew:

what happened I think so it's what's interesting this um clod and

Andrew:

open ai aren't as transparent with their limit changes over the you know yeah

Andrew:

during the rush hours yeah okay peak peak yeah I'm sure there's some sort of scaling

Andrew:

involved that they don't tell us about but uh with glm or at least z.ais more

Andrew:

transparent they're very transparent and from I think it's I'm

Andrew:

not sure remember the time right now but I think it's like 1400 to 1800 okay you

Andrew:

know from 2 p.m to 6 uh yeah something like that yeah um

Andrew:

they will charge you three times oh wow the usage so if you what

Andrew:

would take you normally 1x your baseline you're being charged three times that

Ajay:

amount so I this is very interesting because I think for quite a while I was always

Ajay:

questioning what the difference between the api pricing and the tier like this this

Ajay:

buying 5x 20x pricing was I think some of the things that you were just mentioning

Ajay:

now seem to be giving some evidence and at least from zai it seems like that's a

Ajay:

direct you know contract that they're contract that they're they're publishing so

Ajay:

you see that wasn't the contract in my mind well well I guess first off how did

Ajay:

you find this out and uh second of all how did you compare this to the

Ajay:

actual experience you have a clod and other yeah codex so how I found it out was

Andrew:

well as it was running the reviews I it just stopped uh claude was like oh I've

Andrew:

reached the limit and I was like really it's like I've only been running this thing

Andrew:

for like an hour oh wow and I was like that's fast so it gave you a four-hour window

Andrew:

where it's like okay you reach your limit for your four-hour window kind of thing

Andrew:

yeah a five-hour window five hours I think it's they share that taxonomy with open

Andrew:

ai and claude now got it so they're matching that I guess I'm curious how did they

Andrew:

all have five hour windows they all have weekly windows and you can see it on the

Ajay:

website like there's a usage page oh god okay so they at least provide that very

Ajay:

clearly yeah I am curious so how did you find out about a 3x uh there was a little

Andrew:

asterisk uh like you know the information bubble with an eye on it and I

Andrew:

was like okay what's that's gonna say to me yeah I hovered over it clicked it and

Andrew:

then there's this huge block of text you know text of all that popped up yeah yeah

Andrew:

tooltip maybe a massive tool and I was like oh during this time 3x usage and

Andrew:

they're actually running a promotion right now apparently it's supposed to be 2x

Andrew:

usage normally but it's actually 1x usage until september okay and so to me that was

Andrew:

kind of like I think there's like some sort of loss in translation there because I

Andrew:

was like well it's not gonna be 1x at the end so that 2x is gonna be the

Andrew:

new 1x by september yeah yeah right yeah so it's gonna be a six sec no I'm

Ajay:

sorry never mind I don't know but it ran out pretty fast yeah and I think part of

Andrew:

that was because I was using it during that rush hour time peak usage for them yeah

Andrew:

um I think I've been using it more since then I actually put in a

Andrew:

lock into clod to make sure that it doesn't use glm during the

Andrew:

peak times ah and this is another interesting thing I guess how did you do that

Ajay:

because that's a really useful tool I feel like to be able to so at first when I was

Andrew:

using it it was supposed to again you said I was switching back to clod so there was

Andrew:

a reason why I had bought glm I was like okay I'm going to try glm in place of clod

Andrew:

code yep and so um I'll use the harness Claude Code so I can get similar

Ajay:

experience yep um that required like rewiring things a little bit and sounds like it

Andrew:

was a similar experience real yeah yes it definitely was it definitely it impressed

Ajay:

me especially for an open source that's that model that's good that's great to hear

Ajay:

I'm even more curious because I've heard of it too I haven't actually used it

Ajay:

directly uh that's a good reason to do it but keep going Yep.

Andrew:

So basically I had, um, Claude like write up, no, not Claude, I

Andrew:

guess GLM or Codex.

Andrew:

One of the two, I don't remember.

Andrew:

GLM code.

Andrew:

Yeah.

Andrew:

One of the models wrote up like this bash script wrapper

Andrew:

that would inject all the stuff that I wanted for the certain model.

Andrew:

Okay.

Andrew:

So that way I could start, you know, I could type in Claude, um, and it would just

Andrew:

do the normal one which was you know Claude Code with opus fable etc yep yep and I

Andrew:

would have a different one cloud dash glm which will

Andrew:

automatically use my you know stored api key for you know

Andrew:

open glm zai and have all the roundings change all the urls changed yep and and so

Andrew:

it could recognize that it's actually it's it's funny because if you don't rewire it

Ajay:

properly it will actually think it's opus I I wouldn't be surprised because it's

Ajay:

probably not I actually had a little argument with it I was like no you're not

Andrew:

because at first I didn't really wire it up properly I would have to say their

Andrew:

documentation's a little lacking there they even provided a tool for it okay and I

Andrew:

don't remember the tool exactly working quite right I was like there was one more

Andrew:

thing I had to do I'm not sure maybe I messed up on something but um at the end I

Andrew:

got it all working it recognized that it was OpenGLM.

Andrew:

And that's when I realized, oh, there, it is

Andrew:

almost like Office.

Ajay:

I wouldn't be surprised.

Ajay:

And then just in case anybody heard a very loud sound in the background and it got

Ajay:

picked up in the mic, my wife is back home.

Ajay:

So, you know, just next time we'll have to close the door to limit some noise.

Ajay:

So another new learning for the day, along with all of the great learnings about uh

Ajay:

glm and glm 5.2 as well as uh the learnings about the Claude Code experience now well

Ajay:

actually going back into that the Claude Code experience uh going back to cloud maybe

Ajay:

uh can you tell us a little more about like why you went back and also like what

Ajay:

your experience was it almost some some indications are pointing to that it was very

Andrew:

similar but maybe you can tell us more uh I didn't like the usage limit popping up

Andrew:

so quickly I felt like okay I'm part of the reason also was because claude code's

Andrew:

limit has been a barrier for me and so I was hoping maybe open geolama is a little

Andrew:

bit more generous right with the limits um I did not find that so I was immediately

Andrew:

disappointed by that and I did do apples to apples 20x to 20x not like I tried the

Andrew:

5x or anything I just went straight to the 20x I was like we do apples to apples and

Andrew:

I was like okay so now that I've switched back to clod and the main

Andrew:

primary reason was because they extended fable to july 12th nice nice and I felt

Andrew:

like I didn't get enough usage out of it before and now you can actually get a

Andrew:

better yeah and get a better like and and there's actually part of my workflows I'm

Andrew:

thinking about potentially using api spin for I hope anthropic's not listening oh

Andrew:

you bet there I know that's what they're looking for but

Andrew:

because certain things I found fable to be really good at I think how

Andrew:

people online have been saying it's better at one-shotting things I have found that

Andrew:

when fable is the orchestrator for my modules following my spec sheets there

Ajay:

tends to be less review rounds okay I I do really like uh I want to this

Ajay:

is a whole another thing I'm going to get into it I want to get into as well but

Ajay:

before we do uh finish your thoughts about like yes uh so you did it because they

Ajay:

still extended the availability of fable for now and because your experience with

Ajay:

fables seemingly is much better than even like uh glm 5.2 in this case yeah where in

Ajay:

terms of efficiency like actual just like uh individual prompt and token efficiency

Andrew:

it's doing a much better job yes it might be more expensive per million right but if

Andrew:

that saves me 50 million tokens of review then I think I come out ahead both in

Ajay:

price and speed so this is very interesting because now all of the individuals

Ajay:

who've built a bunch of tooling bunch of skills and everything to get around a lot

Ajay:

of the challenges and make their previous models more efficient it's almost like

Ajay:

some of those learnings have automatically gone into fable maybe or maybe somehow

Ajay:

have gone into fable to such an extent where now fable doesn't need all that

Andrew:

additional tooling and in fact it might make it worse maybe I I had just integrated

Andrew:

my clod skills just and my projects are using them now there's

Andrew:

definitely consistency being built there and I'm still having fable use the skills

Andrew:

too okay so all right this is going to be this is the interesting part andrew from

Ajay:

like an objective measurement and being able to test like whether it was your skills

Ajay:

that improved your performance or it was okay so it's not easy it's not there's not

Andrew:

going to be an easy way to to compare that um but but what I will say though is

Andrew:

this I have a theory I've been thinking about this um so the primary

Andrew:

opinion online I see is that oh fable's so good at one-shotting things yep right and

Andrew:

there's a reason for that my theory is that I think a lot of the software

Andrew:

engineering principles that were left out right or had to be told right had to we

Andrew:

had to build them in around around the model yes have now been trained and

Andrew:

trained into fable so that means everyday people who want to one-shot like a little

Andrew:

clone of a certain game or something yeah it's capable of doing that now because it

Andrew:

already has a lot of the more higher level thinking that you would not

Andrew:

normally have from like a model like opus I I so it's filling in the blanks much

Andrew:

more the longer horizon but yes yes sorry sorry okay yeah it's filling in the blanks

Ajay:

much better much quicker much earlier yeah okay okay because I think that has been

Ajay:

one of the I've been using a good amount of opus uh I've also played around with a

Ajay:

little bit of sonic 5 I actually do think sonic 5 does a pretty good job and it's

Ajay:

comparable to opus in certain areas there are certain behaviors which I also feel

Ajay:

I've gotten a lot better I was debugging through this thing that was a rabbit hole

Ajay:

of AWS permissions recently, particularly when trying to create an SSM parameter.

Ajay:

One of the things that I ran into was I gave the permission to put SSM parameters to

Ajay:

delete SSM parameters.

Ajay:

But then I realized the process which I was using, which is called the SAM deploy,

Ajay:

which is a serverless approach towards infrastructure that Amazon provides,

Ajay:

essentially was also adding tags.

Ajay:

I could not immediately recognize because the

Ajay:

exception I got from Amazon CloudFormation showed me that the issue was related to

Ajay:

the permission of putting the parameter, which is a permission the role already had.

Ajay:

And then I kept going down the rabbit hole, or I got Sonnet to go down the rabbit

Ajay:

hole for me.

Ajay:

And then eventually it reached a stage where it was like, I still see a problem, let

Ajay:

me try something different.

Ajay:

And it actually short-circuited itself, whereas I feel like I would have had to jump

Ajay:

in at that stage, or I've had to even like in prior weeks so I do feel like

Ajay:

it's able to catch itself for this particular case showing a better depth of

Ajay:

understanding and then it was like oh let me actually go deeper into the deployment

Ajay:

let me take a look at the logs more deeply let me look at the events more

Ajay:

individually and then it discovered these tags the permission to apply tags to the

Ajay:

actual resource which amazon says amazon offers a tagging idea for like cost

Ajay:

management for searchability many many different reasons, but it seems like they

Ajay:

actually had it built in to their cloud deployment solution called SAM deploy,

Ajay:

S-A-M, and the permission was not made very obvious when first building the

Ajay:

permission set.

Ajay:

It is in their documentation for SAM, which is not the same as their documentation

Ajay:

for writing SSM parameters.

Ajay:

So this, Sondra 5 did a good job of going exploring in a more effective place once

Ajay:

and realize the technology being used because it got access to look at my logs.

Ajay:

And I gave it access to look at like the deployment information.

Ajay:

And it was like, oh no, this deployment is happening to SAM and then figured that

Ajay:

out.

Ajay:

So I was like, I would have had to give it a lot more direction previously.

Ajay:

I would have probably been going for a few more hours.

Ajay:

And I have gone for a few more hours in the past doing this.

Ajay:

So I do think you're right.

Ajay:

I do think that they're teaching it the particular things that people run into and

Ajay:

they're able to measure based on the same responses, frustration, whatever it is.

Ajay:

and I have improved the system in that way.

Ajay:

So with that being said, I think the one challenge is, Andrew, I do feel sad because

Ajay:

we would have been able to compare the benefit of the skills versus the fable.

Ajay:

I think we can actually still go back if this is an experiment we want to run.

Ajay:

I would suggest it to anybody else who's interested in this.

Ajay:

You could actually go back and revert back to the commit before you actually started

Ajay:

your fable work.

Ajay:

You could then actually do two different branches intentionally and say hey start

Ajay:

this with fable start this with my skills and opus the way that I was doing it in

Ajay:

the past run them in parallel and see what the result is see like the not

Ajay:

only the output of like what code was generated if it's doing things faster or

Ajay:

slower but also you could also measure the behaviors of the model itself which is

Ajay:

the rabbit hole I wanted to get into a little bit is like I had initially talked

Ajay:

very early in this podcast in this episode I'd actually talked about this software

Ajay:

engineering loop that I had watched a video about.

Ajay:

This

Ajay:

single,

Ajay:

the

Ajay:

performance on a single repository idea.

Ajay:

One of the things that I actually noticed was a gap in that individual's

Ajay:

performance.

Ajay:

I can even post this if we find you listeners are interested, I'll share it.

Ajay:

I think the gap that I found was measuring model performance, measuring your coding

Ajay:

assistant performance was missed in that video.

Ajay:

I do feel like being able to then capture not only the artifacts of what it

Ajay:

produced, but how it did it, I think is very important, in my opinion, because one

Ajay:

of the common things I run into, which I'm extremely frustrated by, is I always have

Ajay:

to click enter many, many times because I don't always trust everything the model

Ajay:

does.

Ajay:

And more recently, I was very interested in being like, how often am I clicking

Ajay:

enter?

Ajay:

How often am I allowing it to do the same thing, which is just execute a simple

Ajay:

Python script that does X, Y, Z?

Ajay:

I'm almost coming to the, and I've come to the point of realization is there's many

Ajay:

tools out there to measure your coding assistance.

Ajay:

One, there's like dozens of tools in the space that capture the traces essentially

Ajay:

from your agent execution, your coding agent execution.

Ajay:

There's tools like LangFuse, there's tools like Braintrust, there's tools like

Ajay:

LangSmith.

Ajay:

LangSmith is more focused around the capabilities, or used to be more focused around

Ajay:

the capabilities of LangChain and LangGraph, which is what they make, but they're

Ajay:

essentially branching out more and more I believe so there's Phoenix and there's

Ajay:

like so many tools that do this many I do know for a fact that langfuse has an

Ajay:

integration for Claude Code so when you enable it it'll actually send all

Ajay:

information saying that what tools did I use what were the prompts how many turns

Ajay:

did I do for a particular session how many subagents did I execute as part of this

Ajay:

session and it'll compile all of that information in a nice visual but also put it

Ajay:

all together as a single reviewable data format that then now I can use an MCP

Ajay:

server for line fuse to essentially go and ask it questions about my prior

Ajay:

executions and be like, hey, can you tell me how many times I was using tool calls?

Ajay:

Or can you give me traces in which the number of tool calls were the same and which

Ajay:

ones were equals I was using most often?

Ajay:

So it's like slicing and dicing and doing analytics on your own Claude Code

Andrew:

usage.

Andrew:

It's enhancing your observability.

Ajay:

It is.

Ajay:

It is an observability tool.

Ajay:

So yes, 100%.

Ajay:

I guess the question I would ask you, Andrew, is given the amount of time you've

Ajay:

spent so far, and you are doing analysis on your own, you are keeping track of

Ajay:

certain information.

Ajay:

I'm curious how much interest you would have for a tool that not only actually

Ajay:

captures, but also visualizes, and also gives you tools like MCP servers that

Ajay:

provide a query layer on top of things rather than just maybe keeping track of

Ajay:

things in files.

Ajay:

Because it is free, it is self-hosted, you can install the images and you can run

Ajay:

them in a local Docker instance.

Andrew:

Oh, no, I definitely want to try it.

Andrew:

I

Andrew:

definitely want to try it.

Andrew:

I think there is, like every day, there's just a little bit more that we find out.

Andrew:

And I've actually recently discovered those other tools that you mentioned.

Andrew:

I've been looking for ways to, you know, map things out a bit more and like get

Andrew:

different visualizations, right?

Andrew:

And so that was, it's part of a longer horizon optimization of

Andrew:

the project.

Andrew:

I already had like a little, there's like a couple tools I do want to implement.

Andrew:

I don't remember the names off the top of my head.

Andrew:

But during that, there's so many ways that you can get caught up with the new shiny

Andrew:

thing as well.

Andrew:

Yeah.

Andrew:

There's just too much going on.

Andrew:

And so kind of speaking to that, I guess a little bit, the get ship

Andrew:

done.

Andrew:

yeah yep yep the gsd gsd the open gsd so you had you had

Andrew:

introduced me to that like yeah past week or two um we uh ran that uh

Andrew:

what was it the planner map out the mapping out my code base yeah yeah yeah

Andrew:

um and I you know I still have the the my status line and Claude Code still that

Andrew:

green little bar oh okay okay I I have not been using the gsd

Andrew:

skills much but I do see some hooks that are automatically coming into play

Andrew:

oh okay so the first hook that I saw was the stop hook so it will automatically

Andrew:

you know parse I guess the contacts and spit it out and say

Andrew:

hey you're getting close yeah but it actually tells that yes too yes

Andrew:

yes and so then that it can become aware because I've actually had this problem

Andrew:

before I was like oh like I'm talking with the agent I was like are you aware that

Andrew:

you're about how much context you have left and then they can this is what they say

Andrew:

they say I can see there's a lot here but I don't know how much is left yeah by the

Andrew:

way I'm super excited because I've been bothering Andrew to try this out for a

Ajay:

while.

Ajay:

I'm glad that there's been some indirect benefit that has come out of this.

Andrew:

But sorry, keep going.

Andrew:

Yes.

Andrew:

So unfortunately, where I'm going here next is the reason why I

Andrew:

haven't really used it yet is because there's so much overlap with my current

Andrew:

workflow.

Andrew:

And I actually had Fable today look over and compare all of GSD

Andrew:

to all of the workflows.

Andrew:

what you built for what I've already built from scratch yeah yeah and I was like

Andrew:

what in this thing what's valuable for me like what's different what would be the

Andrew:

addition yeah what am I missing should I just like I literally asked should I switch

Andrew:

should I stay and I told it my hesitancy I was like I am hesitant to use a new

Andrew:

tool where I don't understand things right although maybe it might provide me more

Andrew:

information if I don't understand things then I lose the portability of me

Andrew:

changing it over time yes okay that is definitely a big trade-off here like um it

Ajay:

doesn't have to be an entire trade-off uh yet there there's there's some thoughts I

Andrew:

have there as well but keep going for now yes yes um and it I have decided to not

Ajay:

uninstall it okay so we're there that's a step that's a step you could always adapt

Andrew:

it into your tools that's one thing I do recommend yes like I think the mapper is

Andrew:

super useful yes and I don't want to replicate something that's

Andrew:

already a bit refined right exactly um and a lot of the tools that the skills

Andrew:

that it has fable was like okay well actually some of your tools are actually more

Andrew:

in depth okay yeah right and I was like okay certain areas certain areas that are

Andrew:

important to me okay right yeah um and there was a few other things where I just had

Andrew:

no coverage whatsoever yeah okay okay yeah right and so that's where I was like

Ajay:

okay yeah um we can just take those little things so integrate them

Ajay:

so interestingly yeah this is this is so sounds like fable gave you

Ajay:

a answer that meant that was like it depends it depends and it said if I mean

Andrew:

and I agree with this for the most part though I think again the portability for me

Andrew:

yeah being being able to understand everything modify it to that extent I

Andrew:

think there's so much I don't understand it's nice to know what parts I

Andrew:

do understand I I totally agree and and there's interestingly a lot of uh

Ajay:

trade-offs when it comes to doing something like this it's completely reasonable to

Ajay:

ask is like, hey, the cost of me redoing my tooling and completely giving up on what

Ajay:

I've built, all the learnings that I had is also expensive, right?

Ajay:

And then I think the alternative trade-off is like, but me not adopting this tool

Ajay:

that has a level of refinement in areas that I didn't even know I needed could also

Ajay:

be a cost and opportunity cost lost in terms of like efficiencies that could benefit

Ajay:

me a lot.

Ajay:

The usual thing where I'll say is like your workflow is built around what you know

Ajay:

the workflows that they're offering are maybe a combination of what you've probably

Ajay:

experienced but also a bunch of stuff that they've experienced everybody around them

Ajay:

has experienced so maybe the real value in my opinion for something like that is

Ajay:

when if you start working with more people if you work with more engineers if you

Ajay:

work with more people who are doing coding they don't have to be engineers but

Ajay:

people building the software with you in the same space consistency provided by a

Ajay:

single common tool that has that makes a lot of sense again again contracts and

Ajay:

consistency yes that makes a lot of sense scaling horizontally to more people is

Andrew:

when I would imagine it sounds like I need to self-publish my workflow make that the

Andrew:

new standard yes and then then I'll be like okay well we're all on the same page in

Ajay:

this in this stage the real thing and I've and I've thought about this a lot uh

Ajay:

recently is um and I'm gonna come back into this whole like engineering loop thing

Ajay:

right like I think this observability thing was uh a valuable uh thing for me to

Ajay:

recognize what I I'm trying to target and what I'm trying to improve I like the idea

Ajay:

of something like that having observability so that I can focus on my things where I

Ajay:

press enter a lot the number of times are I am the human in the loop and actually

Ajay:

slowing the thing throwing slowing the process down cases where I need I want to

Ajay:

figure out like saying things like copy files from one location to the other in the

Ajay:

same folder I'm like go for it right like don't ever ask me again please but then if

Ajay:

it's like oh I want to make this api call that's going to delete your whole stack.

Ajay:

I'm like, yeah, don't ever do that on your own.

Ajay:

I think that is what I want to discover.

Ajay:

I want to discover which are high risk scenarios that I'm not willing to give up

Ajay:

control for, and which are the low risk scenarios that I'm totally comfortable

Ajay:

giving up control for.

Ajay:

Observability will help me refine my own workflow.

Ajay:

Again, coming back to this, I do not think there is a one size fits all shoe

Ajay:

available for these workflows.

Ajay:

So I do agree.

Ajay:

At the same time, what's stopping you from stealing some ideas from somebody else?

Ajay:

Not literally, like these are open, this is open source software.

Ajay:

I am not condoning actually stealing ideas.

Ajay:

I'm saying that this is open source software.

Ajay:

The licenses are very permissive.

Ajay:

They're asking you to go ahead and experiment because a lot of open source is built

Ajay:

on the idea that, hey, maybe you'll come back and contribute when you find something

Ajay:

that other people can also benefit.

Ajay:

I will also encourage that.

Ajay:

If you feel like there are some things where you're like, this has got to be

Ajay:

something everybody's feeling pain with, I would recommend you go and contribute

Ajay:

into their solution um and when you have a team a wider team that experiences the

Ajay:

same pain that makes a lot more of a sense of like uh okay five individuals are all

Ajay:

seeing the same issue that is not available in their repo that's not available in

Ajay:

our repo let's go contribute I would prefer we go contribute it back to their repo

Ajay:

so that everybody benefits from that because then you know you're genuinely probably

Ajay:

running into something that's going to be beyond just five people sample size is

Ajay:

small but not one individual it's five times that sample size right like when you

Ajay:

have five people it's something that's well beyond an individual coming back

Ajay:

into this whole thing about the loop I believe that you should be modifying it for

Ajay:

your own benefit but taking ideas and trying your best to keep up with other things

Ajay:

because there may be areas where you can then also work with other people it becomes

Ajay:

a lot easier for you to collaborate in the future so that is where in my loop I do

Ajay:

feel like whatever loop if I do create the software engineering loop idea right I

Ajay:

would say that observability is a non-negotiable because the improvements to the

Ajay:

loop that I can make for my own benefit will not be possible or would be much more

Ajay:

difficult without that then I think regardless of what tools I'm putting in the loop

Ajay:

I need to have hooks I need to have some kind of a plug-in that I can modify so I

Ajay:

need enough control that regardless of what the opinionated solution is that was

Ajay:

built before I joined or I came should be modifiable easily so that is the second

Ajay:

real tenet that I'm going to go and say like in my mind that I believe is important

Ajay:

is flexibility having a hook that you can easily plug into and provide your own

Ajay:

configurations your own things where you're like hey these are the permissions I

Ajay:

want these are permissions I'm not allowing even if there's an opinionated start I

Ajay:

want to make sure that it meets my expectations so those are the two top level

Ajay:

tenants that I come across on my side so I do feel like you're living up to those

Ajay:

already um albeit in different ways and I want to get your take on do you feel like

Ajay:

there's any tenants that come to your mind when it comes to like the software

Ajay:

engineering loop now that you've spent a good amount of time oh absolutely so so I

Andrew:

was just thinking about it my mind like where can I kind of like provide my

Andrew:

perspective here and so as I've been learning so much more about you know the

Andrew:

software development life cycle integrating more formal concepts into my project you

Andrew:

know having an interest in actually learning the lingo the culture um and all that

Andrew:

the ritual no don't don't the rituals are not necessarily a positive thing yeah

Andrew:

but there are some things right um having an appreciation for at

Andrew:

least some of the older rituals that have occurred in the past having an

Andrew:

appreciation that's a good way to put it right Like maybe I'll change it to my need,

Andrew:

which is great, easy to do, but, but knowing about it, I think is the first

Andrew:

thing.

Andrew:

And so what you were mentioning with the, um, requirement of observability,

Andrew:

right?

Andrew:

You only know that because that's not a blind spot, right?

Andrew:

And so as I've been learning more and more, I, it's not that I don't have the

Andrew:

capability of learning it.

Andrew:

It's just, I didn't know.

Andrew:

right and so just like we don't know you don't know exactly and so and that's a true

Andrew:

blind spot so as I've been learning and going through development of the project

Andrew:

I've been slowly discovering where all my blind spots are and then slowly

Andrew:

integrating all those formal concepts or macgyver concepts macgyver

Andrew:

concepts just like the whole workflow a lot of what I have gotten from gsd is

Andrew:

already in my workflow and that's why it's very hard for me to switch like coming

Andrew:

from something that I've built from hand right with clock code yeah I mean yes

Andrew:

yes absolutely but to something that was well refined and obviously used by a lot of

Andrew:

people criticized by a lot of people right um that shows me and gives me confidence

Andrew:

like okay there must be a reason why they're doing these things right and so the

Andrew:

when I mentioned earlier that it had more coverage those were other

Andrew:

things that I still had a blind spot to that now that looking at this

Andrew:

project I was like oh there's more to consider that's that's that's really a

Ajay:

great point because I think once you start learning more things you can adapt

Ajay:

towards them that you can change your behaviors uh but ideally also you can figure

Ajay:

out which ones really suit you versus don't I am curious though like can you maybe

Ajay:

do what is a more recent example of this particular thing is it like the glm 5.2

Ajay:

versus a codex conversation that you're like okay sure like about reading things and

Ajay:

then being like uh my experience may not be exactly the same but I can see where

Andrew:

people are coming from kind of idea yes there there was one thing in particular that

Andrew:

I'm not sure if I remember quite enough to explain it well but I'll just try

Andrew:

um so there is one function or skill or something in um the

Andrew:

skills yes gsd yeah that when it does review round or review

Andrew:

something right yeah it will you know check whether the oh

Andrew:

I understand that so so when my tests were running it was

Andrew:

checking to make sure the test was there and it was wired up all right yeah yeah

Andrew:

yeah but gsd would go one step further which is sounds obvious now in

Ajay:

hindsight um it would actually see where it's wired to ah of course of

Ajay:

course yes so so yes it's for the certain module I'm reviewing a module

Andrew:

right it's built this test is built but some of the data that it needed for the

Andrew:

contract from somewhere else I guess is the contract yes yes thank

Andrew:

you the the um other module wasn't going to emit this at this time it wasn't it that

Andrew:

wasn't done yet so does that mean that this test is accepted oh or

Andrew:

not okay I was gonna say right because because if you it depends on

Andrew:

your definition well if the module is done yeah but the other module but the other

Andrew:

module is not done so it's not going to work okay what was his answer so the answer

Andrew:

so the answer is gsd would go a step further and say it's not going to work because

Andrew:

then it now we know for sure that this part needs something else for

Andrew:

it to yeah absolutely it's found a linkage that was originally

Ajay:

missed yeah by asking simply and one more question yeah and uh and originally

Andrew:

like I was gonna say it depends I was gonna say because it followed the spec sheet

Ajay:

exactly right because it actually kept track it built it built the test because it

Ajay:

mapped out the code base it followed spec sheet to build the test and then it was

Ajay:

like wait a minute that's not done yet I need to wait it's not gonna work it's not

Ajay:

gonna work so so but did that actually call call it failed or so so so I didn't

Andrew:

actually use I it this was part of my review of gsd from fable it was saying like oh

Andrew:

you're missing this part okay right right it's like okay this particular part is

Andrew:

something you're not covering right so you have to wait till that part well no no I

Andrew:

mean like um the part of the review oh yes right it's so it's like okay so what the

Andrew:

basic I asked fable I was like what does gsd cover that we aren't or where

Andrew:

does it perform better you'd ask that yes essentially like what is it doing

Andrew:

that has you know that we're missing or you know there I asked some sort of

Andrew:

question like that and that's your comparison yeah yeah like compare like compare

Andrew:

the two workflows tell me what's different tell me what's the same tell me what's

Andrew:

missing right and that was one of the things that popped up and it kind of was

Andrew:

obvious this is one of those things that there's just so many things going on it's

Andrew:

easy for something that simple to slip by I you know again and I think and at the

Andrew:

end of the day it seems obvious like oh yeah maybe like yes it is built but

Andrew:

it's not gonna work right so it shouldn't be

Ajay:

classified and gsd was telling you that basically yeah well fable told you that

Andrew:

gsd would tell you that yeah it would it wouldn't be necessarily it being a blocker

Andrew:

right it would just be like oh this needs to be like considered yeah deferred right

Andrew:

it needs another module before it's going to be fully accepted you know this I'm

Andrew:

getting because it can't be validated until it's actually been able to hook that so

Ajay:

so interestingly I am I'm getting really excited uh excited about this particular

Ajay:

conversation because we've had two independent data points today to hear that the

Ajay:

latest model from anthropic fable yeah um has not only given a totally reasonable

Ajay:

answer of it depends on certain things, and also gotten better in terms of its

Ajay:

ability to execute things on the first run.

Ajay:

And now you're telling me it's able to do a reasonably good comparative study and

Ajay:

actually identify what the interpreted value of something could be.

Ajay:

So, you know, just Andrew, I have to ask this, and it's not just because we're

Ajay:

getting up into that time space right now.

Ajay:

I do have to ask this is, so do you feel like it's an act or do you feel like

Ajay:

it's showing intelligence?

Andrew:

based on what I've read recently I do think it's starting the show intelligence wow

Andrew:

wow yes and and it's not just from what I've read from the screen it's

Andrew:

also from what anthropic directly published recently I'm not sure if you heard about

Andrew:

it they also had advertisements for some stuff recently is that no no they they

Andrew:

published like um or like a scientific like research ah yes yes yes okay okay did

Andrew:

you read that no no so so basically the idea I haven't read the whole thing so this

Andrew:

is kind of like my paraphrased interpretation um I'm also not a neuroscientist but

Andrew:

from what I understand and what they've published they are finding

Andrew:

evidence um that uh clod code or clod in general these models

Andrew:

are creating spaces in their memory

Andrew:

or like as they're processing things that is very similar to how

Andrew:

theoretically humans process information okay I I will I will slow us down there a

Ajay:

little bit I I also know that there is a level of uh marketing that goes into some

Ajay:

of these things yes and I'll wait for a more peer-reviewed approach towards some of

Ajay:

that conversation I do I am very excited about it and I think uh just I think in the

Ajay:

interest of time I'll start wrapping up and I'll come back to this though it sounds

Ajay:

like at the end of episode four we may have our first indication of the it's not not

Ajay:

being an act as much as a potential real you know signal towards intelligence there

Ajay:

are some papers that are talking about it and we may have to wait for a little more

Ajay:

evidence for those papers maybe we should read that paper maybe we should read that

Ajay:

paper uh but at least for the end of episode four andrew so I guess I'll reframe and

Ajay:

I'll ask again is uh so rv today is this is this a uh one scenario for it is not

Ajay:

acting so it's a yes it's not acting it's actually intelligent I think it's getting

Ajay:

there I think it's getting wow yeah so that's that's okay that's a big one yeah all

Ajay:

right at the end of episode

All Episodes Previous Episode

Listen for free

Show artwork for Act of Intelligence

About the Podcast

Act of Intelligence
Act of Intelligence is a podcast where software engineer Ajay Medury and systems engineer Andrew Sierota share in-the-trenches notes on building with AI—tools, workflows, and philosophy—to test old engineering wisdom against new instincts.
Act of Intelligence is a podcast where a software engineer (Ajay Medury) and systems engineer (Andrew Sierota) trade honest, in-the-trenches notes on building real things with AI — the tools, the workflows, and the philosophy — to figure out which old engineering wisdom still holds and which new instincts to trust.

About your hosts

Ajay Medury

Profile picture for Ajay Medury

Andrew Sierota

Profile picture for Andrew Sierota