but also annoying AF for humans to be waiting hours for a staging deployment
Shit Punk Says
Oceans of Wisdom
19,517
matching drops
#1510653
2026-10-11 21:49
it is very linear time and token inefficient to keep the bots waiting around for hours/days for their deployments to go in
regardless of how many PRs are merged
so that there is some deterministic amount of time before the deployments get into staging
1/ you deploy one at a time but with some rule like
2/ every 15 minutes you just bundle everything and deploy it
and then you run into other problems where the PRs outrun the staging deployments and you get an infinite queue, so you need to build a mixed formula that is something like this
yes you need queue mgmt
i think the only way how it could work currently is not by every pr pipeline activating staging redeploy by itself. this could be a redeploy conflict hell by running 10 builds in parallel against the same target with different versions. i think the automation just merges to staging and then some cron job checks every five minutes if there are new commits in staging and if there are then redeploys and the dev pipeline just checks if its development is in staging by comparing what is in staging hash to its head hash. and as soon as it discover its deployed, it runs its tests and reports..
maybe something like that
at GPT 5.5 xhigh class (and soon I guess we are getting better), the net effect of this approach (with the proper workflows and practices) is an extremely material increase in overall system productivity
there is no free lunch in life. you can't say "I am going to 3x our cadence but everyone is going to work the same amount of work" - somethign has to change, "someone" has to work more, so you can decide to let the bots grind it out, even if it is less efficient per bot/thread than a human doing it, it is ok because it is not a human doing it
this does not matter so much, if the bots have app and AWS CLI access to the staging environment/vpc/etc they grind it out in the logs and figure it out
and i imagine if constant staging merges and redeploys auto-happen then it's a bit hard to report which actual version "caused a problem". but this might not be as big of an issue as it first seems
and anything we can do to make the deployments faster is important
yes for this to work staging need to stay up during deployments
the bots have spent all weekend fixing at least the obvious things they broke on staging
at least before I handed over my work to the professionals
is working on a common code base
if they are at least merged, then anyone starting work on something new on Monday
so we need to solve each step of the process at a time
it is going to be Conflict Central
we will collapse into chaos, we won't know what branch to work on
and now we are waiting for you too look at each one before even merging
there are 200 PRs unmerged
we can't have a situation where you come in on Monday
this is weeks away not months away here
from trivial (sure, some), but also normal and some serious ones
across multiple parts of the app
between Friday night and Monday morning, I pushed 290 PRs
I had a weekend where I wanted to push some things forward materially
still a bit dangerous, but less dangerous
and I know the repo much better so I am less dangerous
two weekends ago, on the other repo, where all my tooling is more mature
yes this is exactly the way
so the middle ground is "release branches". we merge the pr's we deem safe-to-deploy to release branches and then merge those from time to time.
I want to get you all in the right mood of what the "very near future looks like"
as I had mentioned a few weeks ago, I speak from painful experience, I have gone through this whole cycle on a different repo starting in January and have 100% certainty how it will play out
because then we are creating new future issues from ourselves
and even with staging in place, it is important that production does not fall 420 commits behind
so it is neither going to be 1 at a time nor 'all of staging'
but 8 was a big change in how we handle wallets, let's give it another look tomorrow, all of us with a fresh mind
a lot of them will be relatively straightforward and harmless and with tools to help review them, a human review will allow you to say: yup, deployments 1 to 7, 9, 12, 14,15,16 seem ok, lets get them to prod
what this looks like at "today's scale" not "tomorrow's scale" is that we might have 10-30 deployments per day to staging
i think we are going to have to cherry pick what goes from staging to prod
so the plan is that bots constantly deploy to staging and then at some point we merge staging to production? or we still merge dev branch to production after bots have finished testing it to staging?
the exact same reasons apply here
could you @[simo] do a good job developing if you could never deploy any of your work to staging to see what you did? would it improve or worsen your coding?
the way to maximize their performance is to treat them like a human
in time though (and I have done it, so I am speaking from experience), they are good enough even to handle db migrations and test them (on staging)
we can start with the FE, where it less sensitive
and so we need tooling to avoid that
bc once we make this change, you will proceed to get lost in the deployments
this is going to need significantly more sophisticated workflows and tooling than you guys have now
human review comes in part 5 and is the gating factor for now to go to production
1/ bots E2E test their work on staging
2/ once bot says "we are good"
3/ we have some other specialized bots that also check repeating issues e.g. security, performance, etc (all this means that staging has to become a realistic environment with actual data)
4/ #3 reviewers may be send them back to #2
5/ once everyone is happy, then the bots keep a list that shows the various commits/deployments, rank them by blast radius, risk, what was touched, testing strategy