Watching the video I think turning each tensor into packets is quite clever as you get some ability to send them around a network and organise layers/data manipulation/transforming the layers/compression all as part of the stack.

I’m pretty surprised no-one has actually exposed the actor model for parallelising neural networks, it seems it would work quite well and allow you to have a layer per node (or actually many split configurations). Maybe data locality would be an issue with actor based approaches. They seem to be solving this at a lower level but with less knowledge of the actual parallelism in software.

Also Jim was almost certainly the one doing the interviewing, so there’s likely something interesting in there that he felt was worth his time.