In 1968 Douglas Engelbart used the wonderfully space aged term "brain amplification" in his mother of all demos.
In practice, i'm not sure how it actually works. Systems work can be automated away. And it becomes brittle. Something like emacs or vim absolutely feel like iron man armor. I can't really see what vim for systems work would look like.
My vague thoughts are, maybe build systems that give lots of information about what's happening right now, and how it's different than the past. Lots of commands for introspection. some commands for intervention. At scale, picking a specific machine to remove from load balancing pool or restart or destroy and reallocate at a keystroke might be handy.
It's an interesting perspective, but the difference seems subtle in most practical contexts.
There are many MySQL tools which are able to automate the process of failing over between a master and a slave. Github initially chose a solution which would trigger this failover automatically, when a master was unresponsive for a particular amount of time. One bad query came in, and the tool did it's job by failing over from the master to the slave. Here's the problem - the query was automatically retried against the slave, causing it to fail over back to the master. Repeatedly.
The solution? Page when MySQL became unresponsive, let a human initiate the failover, and let the tools handle the failover itself (which is a very complex dance to perform manually). I think they eventually moved to a model where it would fail over once per few hours, and alert if the condition occurred again.
Here's how I built our deployment system at work for our various interdependent packages and services, each of which is a git repository (and where libraries are symlinked into those git repositories as needed).
First, I built a tool called checkout_package which clones/updates a package and then recursively installs and symlinks all of its dependencies. Devs used this for local installations as well as on production servers.
After that had its problems ironed out and everything was working, I built a system called command_runner, which was a simple JSON-based RPC daemon. This had the ability, among other things, to call out to command_runner and return the result back to the callee. Now our developers could run command_runner, which we knew worked (and knew how it worked), on remote machines without having to log in. A simple shell script was all that was required to loop over every machine to update them all.
After that, we built a web interface that could let you choose which packages you needed to update, and it would run command_runner in X number of threads to do the updates concurrently. It also returns the output from command_runner (which is extremely terse if everything went fine) back to the user of the web interface so they could tell when something went wrong, and retry if necessary.
We also use monit. I added functionality to command_runner to run and parse 'monit summary' to see what was installed on and running/not running on a given server, and to run monit commands (start, stop, or 'restart', i.e. stop and then start) on a given server. From there, we followed the same pattern: shell script to iterate, integrate into web interface.
Eventually, we tied the two together. After you deploy, it takes you to a version of the services management page that only shows (and already has selected) the services which are, or depend on, the packages you deployed.
None of our system abstracts away any of the details of what's happening; it merely provides an easier way to do what you would be doing anyway, and provides more information than you would normally have (for example, the package deployment page can show you the git diff between what's live and what's HEAD, so that you know whose changes you're about to deploy). Because the devs still use checkout_package at the command line in their local environments, it requires that tool to remain both usable and sufficiently user-friendly, and for nothing to be wrapped up into any other layers of automation that is necessary for doing the fundamentals.
This way, we have managed to prevent our deployment process from becoming impossible to reproduce by hand if necessary (e.g. when deploying/rolling back changes to our deployment system).
The system provides extra information that you wouldn't have had, does some tasks for you that you don't have to do manually anymore, and does stuff for you all at once instead of having to do the same task over and over again per server, but fundamentally it's more like cruise control than a self-driving car.
In practice, i'm not sure how it actually works. Systems work can be automated away. And it becomes brittle. Something like emacs or vim absolutely feel like iron man armor. I can't really see what vim for systems work would look like.
My vague thoughts are, maybe build systems that give lots of information about what's happening right now, and how it's different than the past. Lots of commands for introspection. some commands for intervention. At scale, picking a specific machine to remove from load balancing pool or restart or destroy and reallocate at a keystroke might be handy.
It's an interesting perspective, but the difference seems subtle in most practical contexts.