I'd love to use a real debugger but as someone who has only ever worked at large companies, this was just never an option. In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. Print debugging is all you have. If there's a problem with the logging system itself or something that crashes the program before the logs can flush, then not even that.
This is basically it. When I started programming in C, I used a debugger all the time. Even a bit later doing Java monoliths I could spin up the whole app on my local and debug in the IDE. But nowadays running a dozen processes and containers and whatnot, it's just hopeless. The individual developer experience has gone very much backwards in the microservice era so the best thing to do is embrace observability, feature toggles etc and test in prod or a staging environment somewhere outside of your local machine.
A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.
Service is provided by people. You, for example, discover a problem with OpenAI's system that you integrate with and the only way you can address it is to employ the services of the people who work for OpenAI. While that is an example of a macroservice (or what we usually just call a service), it playing out in the macro economy, microservice is the same concept except applied in the micro scale.
But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?
Or are you just struggling to say that your service utilizes multiple applications?
We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
I certainly know how to debug each of the services in my environment, but how do you step-through debug a single request across services? Like, if service A make a gRPC call to service B, are you saying you can “step into” the call from A and your debugger is able to break on the corresponding call in B? And frames from the call in A are there in a logical “stack” from the breakpoint in B?
(Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)
No I can’t debug multiple services at once, unfortunately. But I will switch between them as I track the request over multiple runs. Also extensive logging in available in grafana helps me know which service is having the issue before I start debugging.
This is usually enough for me, too. Use tracing, figure out where things fell apart in the traces, isolate those service(s), and debug from there. It's definitely more work. When we start new projects, I encourage people not to use services until proven necessary because this added layer of friction is a real drag. For a lot of us that isn't a choice, though.
In my experience you just slap minikube or k3s on your dev machine, and treat it as any other environment. Argo, helm, kustomize, whatever can all work against a local single-node cluster just fine. It takes some effort to make sure your configs are overridable per-environment, but it’s worth doing. (And something you’re hopefully doing anyway if you’re doing any kind of integration/test environment.)
It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
Curious to learn more about why it is difficult to debug. I'm not familiar with service mesh. I also work at a large corp, but we use gateways and most things are event driven with kafka across domain boundaries. I spend most of my time debugging each service locally by injecting mock messages or objects. I do this one at a time if the problem is upstream. Usually, our logging helps us pinpoint the exact service at to target for debugging. It's really easy. Our devops infrastructure has built out patterns and libraries when teams need to mint a new service. Everything is standardized with terraform. Everything has the same standard swagger pages, everything is using okta, etc.. Seems a bit boring (which is good)
It's easy if Someone(tm) has already set up a system where you can whip up the relevant bits with something like localstack.
But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
I suppose in the description of "large corp" and service mesh, Someone would already exist and this would already have been worked out. It would be a nightmare dealing with hundreds of microservices without this kind of game plan.
It's not a realtime system kind of thing where the debugger would change the behavior too much... It's possible with enough engineering work, but nobody has put that work in, in fact they had a debugger for some staging envs that they deleted. Lately they keep adding more red tape making it hard to even run something locally, let alone attach a debugger.
I guess you can attach a debugger for unit tests, but that's not very useful.
> In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger.
I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
