Zero Trust & Go
source link: https://changelog.com/gotime/292
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Transcript
Changelog
Play the audio to listen along while you enjoy the transcript. š§
So today we have Michael Quiqley from NetFoundry joining to talk about zero trust. Michael, how are you doing?
Iām good. How are you, Natalie?
I donāt trust you. Are you sure? Are you sure that youāre good? How do we establish trust? [laughs]
Thatās a tricky one. Thatās the heart of the matter, unfortunately. [laughter]
Yeah, Iām doing good as well. Iām trying to do good as well. What is zero trust?
So zero trust started with this ā I mean, Iām exactly sure when it started, but it sort of became popular with this concept that came out at Google called BeyondCorp. So they started this big initiative, and I donāt remember the exact timeframe. I want to say it might have been maybe ā letās call it a decade ago, or something like thatā¦ Where they had this BeyondCorp initiative, and it was about not considering the difference between LANs and WANs. Kind of like where you are in terms of physical security being related to how your network security is postured. So a lot of times a while back we would have configured access to things by āIām in a corporate officeā or āIām in some physical location where Iāve got a specific network access to something.ā Zero trust was sort of about getting rid of that concept, and going to āIt doesnāt really matter where youāre physically located and what your underlying network isā, and treating the security posture differently than that. So using certificate authentication, and crypto, and things to strongly identify whoeverās using your network resources; like, actually securing them that way. So that was sort of the genesis of the zero trust concept.
Was there anything specific that kind of created that? Was there some big DevCon incident? Was it just a growing pressure?
I think it was more of a growing pressure for like remote workforces, and that kind of thing. This was pre-COVID, and all that sort of thing, so I think people were just starting to really get into ā not having to be tied to an office, and that sort of thing. It made sense, and I think it was sort of a āHow can we make this work better than it had worked previously?ā Because in the old days youād have VPNs, and youād have facilities to sort of make you act like youāre on a local network, and some of those sorts of things, instead of actually solving the real problem. So I think zero trust is sort of a response to that.
Yeah. And how did the adoption go in the early days?
I think the adoption is still ā I think in the early days it was sort of a pipe dream, and a little bit of aā¦ It probably worked for a lot of folks at Google, or certain segments of the Google workforce might have been using that sort of BeyondCorp approach. And actually no, I wasnāt there, but I think itās been a slow adoption over time. I think a lot of the VPNs and some of the other styles of securing things are still very commonly used, and I think there are progressive organizations and people that are really starting to get into zero trust and start using those tools. Thereās also better tooling coming out, thereās a lot of infrastructure thatās better than it was a while back.
I want to say that the White House might have had some sort of initiative around zero trust. Thatās actually a talking point, in terms of like ā yeah, I think thatās a thing that people are sort of being exposed to more now, because itās sort of a commonly used talking point, about cybersecurity and some of those kinds of buzzwords.
Okay, thatās nice to be up to date with the timesā¦
I think thatās reasonably accurate.
[laughs] And then how would you describe the world of zero trust these days?
Thereās a lot of really interesting products out there. I mean, Iām obviously biased to the things that I work on. I think we solved some of those problems in especially clever and innovative waysā¦ But thereās a lot of really great tools out there now. Thereās all kinds of ways to solve the zero trust problem. Some of them look more like traditional VPNs, and some of that sort of thing, and then on the other end of the spectrum we have tools like the stuff Iām working on, which is OpenZiti, and zrok, and some of those things that take zero trust in this other different direction. But the tooling in general is getting a lot better. I think itās one of those things that ā people understand it more now than they did a while backā¦ So as people understand things more, the tooling gets better, and people start iterating on tools, and conceptually they improve.
And so tell us about what is different in your approach.
So we started working on OpenZiti, I want to say it was maybe six or seven years ago at this pointā¦ Probably about six years. We started with a sort of whole cloth solution to this. We actually started with a proprietary product that we sort of built a wrapper around. It was this closed source thing that was used by the Department of Defense and some of those kinds of people to do spread spectrum-like security and some of those kinds of things around securing data in motion. We kind of built that and added some additional layers of security on top of it. And then we started looking at the problem from sort of a holistic perspective, like āHow do we want this to work?ā And what we ended up with was sort of a network overlay, that doesnāt sort of piggyback on top of IP in terms of like IPsec, or WireGuard or anything like that, where itās actually reencapsulating and encrypting the data. Conceptually, what we built is closer to something like Tor, where we have a mesh network that you we call the overlay, and then all of the underlying IP networking, whether itās WAN links, or whatever it might be, we refer to that as underlay. So all of the overlay is basically running in user space, and weāve got this mesh network of routers that has smart routing, has its own sort of addressing protocol built on top of it, and layered into that from the very beginning is strong cryptography and strong security around identity. So it truly is zero trust from that perspective, in that you canāt really do anything on the network without having strong identity in the loop.
[00:06:14.00] But that let us do some interesting stuff where we can sort of tailor the addressing model, and the naming model, and the policies and all those sorts of things at a user space level. Even the routing protocols across the mesh, they basically are running in user space. And you can reroute traffic, you can say āI only want this traffic to go through these linksā, you can control costs to process things however you wantā¦ So in the early days of a lot of this stuff I was working on the underlying fabric, the mesh networking and the architecture around that. And then on top of that weāve layered various edge protocols, and higher-level constructs that allow you to embed the stuff in the applications. And if weāre talking about like an elevator pitch for this, I think the main thing that we like to talk about in terms of like how we would explain OpenZiti to people is that itās more of a programmable network. So we really come at it from the perspective of -we have SDKs and weāve got infrastructure thatās designed for programmers to embed this stuff into their applications.
So rather than sort of being a network layer thing that you use like a VPN, or you use as like some sort of bolt-on your application to sort of try and secure it and bolt zero trust to it, I think our ideal use case is we want people to take our SDKs and embed this stuff directly into their applications. So thatās the main difference, I think.
You talked about meshes, and Iām thinking of Istio and similar meshes as wellā¦
Yeah, thereās a lot of overlapping concepts. In fact, weāve got people on our team who are working with OpenZiti in Kubernetes and that kind of stuff, and thereās a lot of things we can do with our mesh that I think are interesting in that kind of a managed, containerized sort of environmentā¦ So yeah, thereās definitely overlaps there.
Does it also do things like logging and home monitoring while edit, or tracing?
Yeah, weāve got all those things. Metrics, all the things youād expect to have in a network like that. Yeah, we do those things.
So itās a sort of a full stack mesh.
Yeah, yeah. And the nice thing about the design too is you can take whatever layers make the most sense for what youāre trying to do ā like, we generally ship the entire stack to customers, and they use the whole thing, but you could use just the mesh if you wanted to, you canā¦ And then going up even higher, now that Ziti is starting to become mature, Iāve sort of pivoted my work and I work on an open source project called zrok now. And zrok was sort of a ā the name, you can kind of tell what it was inspired by just the name. Most people are familiar with ngrok; that kind of had theā¦ I was basically like āGo figure out what something like ngrok would look like on top of OpenZiti.ā So I built zrok. And one of the cool things about zrok is that you look at tools like ngrok, and itās awesome that you can share a private resource on a public URL, and let anyone use it, and you donāt have to open any security holesā¦ All those things are great. But one of the cool things about Ziti is because we have this overlay network, and we control the addressing, and itās all zero trust, we can do peer-to-peer across that network without requiring any of the peers to expose themselves publicly, or anything like that. All the peers can remain ādarkā, but we can still have peer-to-peer communication. So that lets us do things like private sharing, where we can say āI want to share a network connection between two completely dark peersā, and the traffic will go across the overlay, across the internet, but none of thatās exposed to any of those things. All that data is endĀ-to-end encrypted, itās all zero trust under the coversā¦ So thatās one of the things thatās really cool about it. So zrok ends up becoming like a really good use case for showing people how you can use something like Ziti to build things that you couldnāt otherwise easily build.
So thatās an interesting story. So when we started building Ziti, it was almost accidental that we ended up on Go. And I couldnāt be happier that we chose ā I mean, Goā¦ Iāve been doing this a long time, and I would say that Go has replaced a lot of other programming languages for me.
Oh, anything that I would probably have done in Java 15 years ago, I would do in Go now, without even missing a beat.
How did you come to hear about Go? When did you first try it, and why?
Thatās part of this story. So we started building OpenZitiā¦ One of the things we were looking at was weāve got this mesh network, this overlay, and we want to obviously make it as performant as we can. So one of the things we were taking a look at was what kinds of protocols could we use to implement this mesh. And at the time, we were big into Quic. We were taking a look at Quic. Quic, itās an alternative HTTP style protocol that Google put out. And the main implementation that you could find on the internet of Quic was written in Go. Itās quic.go, itās the library. Itās still widely usedā¦ So we accidentally started building Ziti in Go because of quic.go.
Interesting. So that was the reason. Were there other languages that you were considering, and you said āNah, quic.go is more importantā?
I want to say an early, early version of some parts of Ziti that I had worked on back then were written in JavaScript. So it was Node. We actually started parts of it with Node. And as soon as we started messing around with quic.go, and started getting into ā there were a couple of us working on this stuff at the timeā¦ As soon as I started working in Go, I was just like āThis is it.ā
Iām sure a lot of people have very similar stories with Go adoption; it took me not a lot of time to understand the language, but then some of the idiomsā¦ When youāre building a mesh router, or anything like that, thereās a lot of complicated state in that, and thereās a lot of complicated concurrency in that. And some of the patterns in Go took me a little ā you know, Iāve been mutexes, and semaphores, and things like that for yearsā¦ Some of the chan, and some of the concurrency patterns in Go took a minute to get used toā¦
Mm-hm. Until you stop writing JavaScript in Go.
Yeah, stuff like that. And I wouldnāt say that Iām still perfect at itā¦ Occasionally I do slap a lock around something, because it just seems like the right way to do it, butā¦
Yeah, it feels safer, exactly. So yeah, thatās how we ended up in Go. And in retrospect, I wouldnāt change a thing.
Nice. And then how was the community adoption? Did you start speaking with Go developers, or how did you ā how was your interaction with that, with the community?
Weāre actively working on that stuff. I think, like a lot of open source projects, thatās always a challenge, to get the word out and to get people to understand what youāre doing. I think one of the things thatās tricky for us too is that Ziti is fairly different in terms of like what weāre doing philosophically, and like how we think about overlays, and security, and that sort of developer-first approach. It can be a little hard to explain to people. I think when people think overlay networks, and zero trust, they think like VPNs, and theyāve sort of got a bunch of concepts already in their head, and theyāre used to thinking about these things in certain ways. So we kind of have to say āWell, like that, but different in these ways.ā And it can be a little bit of a challenge.
So I think thatās one of the things that weāre actively working on, is trying to figure out how to get the message right. And like a lot of people that work in tech, weāre not always awesome at marketing; sometimes we struggle with these thingsā¦ Just like weāre not awesome with documentation all the time. It can be a little tricky to get that going. But definitely, things are picking up, weāve got a lot of great customers, thereās a lot of people that are ā weāre starting to get a lot more attention, basically.
I guess you can always try to speak to the heart of Go developers saying that you wouldnāt write Go syntax the Java way, or the JavaScript way. Same logic.
Yeah, absolutely. And I think if you look at our Go SDKs, weāve got our ā I kind of think of it as thereās OpenZiti and then thereās zrok layered on top of it. And zrok is kind of a really good way to sort of understand at a high level whatās going on with some of this stuff. You can start with that perspective and sort of drill into it.
[00:14:14.09] So thereās an OpenZiti Go SDK, but thereās also a zrok SDK. And the zrok SDK, like if youāre a Go network developer, itās net.Conn and net.Listener at the end of the day. It couldnāt be any more simple than that. Thereās a couple of extra lines of code to sort of establish identity and that sort of thing, but beyond that, like, dial and bind, and youāre there. Itās basically traditional net.Conn and net.Listener and youāve got a working network. And everythingās zero trust, and itās secure, and peer-to-peer, and all those kinds of nice things.
Okay. So you say that what you do is different from some of the traditional zero trustā¦ So letās talk about what is a traditional zero trust architecture, what are some components there, and what is different with you.
Sure. I could be wrong in this perspective, but I feel like when I think like traditional zero trustā¦
Itās not the best word, but in a lack of a better word, we go with thatā¦
Yeah. I would think of things like Zscalerā¦ So Zscaler is a commercial product that you can buy, that I believe fairly closely mimicked the sort of BeyondCorp, the Google approach. And I havenāt spent a ton of time working with Zscaler, but my sort of naive understanding of it is that itās more or less sort of a network proxy. So a lot of these traditional zero trust products kind of want to be a proxy that mediates security for you, and that deals with identity, and deals with validating that your users are who they say they are, and those kinds of things. Weāre different than that in that we establish an overlay network, a true overlay network, that mesh network, and then endpoints can sort of sit wherever they are. So you donāt go through a proxy, youāre basically connecting to that mesh network. And that lets us do all kinds of things with citing resources wherever they need to be on that mesh, and weāve got all kinds of fancy load balancing, all that kind of stuff, but it can be dealt with at the right level of abstraction for the applications youāre working with. So your applications - you can put resources wherever they belong. You can do the same thing without a proxy, but it feels different and acts different at the end of the day. Does that make sense?
I donāt want to sound like a ChatGPT, but if you were to summarize the core parts of your average zero trust architecture, what would be those components?
I would think ā I mean, again, these things all look a little different these days, but I would say the average zero trust implementation is probably some form of a proxy. So thatās what that looks like.
And you can move that responsibility around, basically.
Yeah. Yeah, you can put that proxy and that border wherever you think it makes the most sense.
And what are some misconceptions or misunderstanding, or like wrong usage patterns of zero trust, if that exists at all?
Honestly, I feel like one of the anti-patterns of zero trust is trying to treat it almost like a VPN. And I think thatās one of the things that we try very hard to sort of steer users away from, is rather than ā you can take our components, and weāve got a really powerful tunneler, weāve got resources that will let you get down to layer two and layer three, and those kinds of thingsā¦ And let you treat what we do almost like a VPN. But I kind of see that almost as an anti-pattern. The right way to do this is to embed that zero trust capability, that border directly into your application code. The less optimal way of doing it would be to sort of try and bolt it on through some sort of gateway, or something like that. That to me feels a little bit clunkier, and a little bit more disjointed.
Does the use of zero trust components in general make at all ISO certification for security more complicated, simpler, or have no effect at all?
[00:18:02.29] I donāt actually know. I havenāt been through a certification process like that, so I donāt really know. Yeah, I donāt have a good answer for that. So I donāt know.
I can definitely see the value of using that on the technical side, but usually you do security ā if you spend a lot of focus on security, youāre probably going to display that for others to know, and know that this is a checklist that they mark, and are able to use, soā¦
Yeah. I would assume, like most certification processes, youāve got to sort of go through and vet all the components that are involved, and that kind of thing. So yeah.
And letās say you have a system that uses whatever security measures, and you want to integrate/introduce zero trust into thatā¦ How do you go about that?
It depends. If these are things where you donāt have access to source code, or willingness to change source code, thereās all kinds of ways to add zero trust layers on top of things. For Ziti weāve got what we refer to as a desktop edge, or a mobile edge, which is something that sits ā itās a piece of software, itās more or less sort of an end user-facing daemon, or in the case of like a headless Linux system itās just sort of an actual daemon, that mediates connectivity into the overlay. And from there - I mean, your application just treats the local Ziti endpoint as if it was the remote resource. So itās just a matter of configuring things in a way that makes sense. So really, the non-zero trust, non-secure traffic doesnāt leave your local host. When it leaves your local host, itās going through a security layer; through Ziti, more or less.
Does this affect in some way latency in the system?
Yeah, I mean, I think itās physicsā¦ If you add extra machinery in the middle, itās definitely going to impact performance. The interesting thing about that though is - you know, throughput latency, and those kinds of things, you would be surprised at how little impact that actually has in practice. Itās actually ā weāre not fully there yet, but weāve worked very hard to sort of make these things perform well and not be overly lateā¦ And yeah, thereās extra connection setup involved, and connecting an overlay; youāre gonna have to talk to a control plane that you wouldnāt otherwise talk to. But youād be surprised that in practice it doesnāt really make that big of a difference.
Good thing youāre using Go.
Yeahā¦ Yes. The performance of our Go code is very high, for a relatively low cost. So to get good-quality performance out of code, Go is a great choice, because the overhead of getting there to me feels lower than C++, or something like that. Soā¦
Did generics affect your codebase at all?
One of the developers whoās currently working on a lot of the low-level stuff in the mesh layer now has done a lot of work with generics. But when weāve built most of OpenZiti, generics wasnāt a thing. So I think there are layers that were sort of adding some generics to, because it does actually make a difference; I think some of that stuff is some of the persistence layers, and those kinds of things. Weāre using a Bolt for a lot of storage. Bolt is like in Etcd, and some of the ā itās part of Kubernetes, basically. So we use that as a persistence layer. And weāve got a whole bunch of infrastructure around that, that does use a bunch of generics and things. But I donāt think thereās a ton of generics in OpenZiti as a whole.
Is there any prediction that you do or plan to do something with usage of AI in recognizing potential good, bad, recommended behaviors, or any other use of AI?
So one of the things that we talked about a lot, that I think I mentioned earlier, is the smart routing capabilities. So we have this overlay network, and thereās connections in all directions, and weāve got this mesh. And we have sort of a basic implementation of what we call smart routing. And thatās about rerouting traffic to a different path if thereās a better path available, or rerouting around damage, and some of those kinds of things. Thereās a couple of things that I think could pull AI into the mix. Weāre talking about multi-tenancy, and weāve got this big data plane, and we want to build to allow multiple customers to share large amounts of resources on the overlay network. So weāre having a big, giant, well-provisioned mesh, and then we let lots of customers use that mesh. Itās one of the things thatās a discussion point.
[00:22:21.22] And then our smart routing implementation could very easily use statistical learning or ML or something along those lines to sort of make smarter decisions around optimization, potentially. So I think there is an aspect of what weāre doing with that overlay that could benefit from that kind of an approach.
Have you seen that at all? Is that some trend that might be coming?
Itās mostly just something weāve talked about in discussions about the product. I donāt think thereās anything necessarily ā thereās no mandates or anything like that coming down the pipe. Itās not a huge area of focus, but itās something that weāve talked about a bit as a team.
And generally ā itās a question about the ecosystem, and itās obviously a very big question on something that moves way too fast, but is this a trend, or a direction (direction is a good word) that the zero trust ecosystem is going towards?
I donāt think Iāve seen much about zero trust architectures in AI at this point. I think most zero trust implementations are mostly trying to get people to understand a) the need, and b) the architecture and how you kind of adopt these things, and getting a good foothold for what they do. I donāt know that, by nature, a lot of these things are going to be tools to help you optimize a lot of whatās going on. I mean, I do have some interesting stories about that in terms of optimization stuff that weāve done along the way, butā¦
Thatās very interesting.
Yeah. As a side note, one of the things that I worked on a couple years ago was we ā one of the things that our previous closed source product did was it did a really good job of sort of being a WAN optimization layer. So you could send traffic through it, and it would in some cases outperform a generic TCP connection by a good bit. Sometimes two, to four or five times what a TCP would do, especially on like a poor-performing link, the other side of the planet, that kind of a thing. Big, long link, with a lot of packet loss. So one of the things I worked on was a library I called Dilithium. And the overall product, I just called it ā I apparently was on like some sort of Star Trek kick at the timeā¦ I called it Transwarp. And it was to build a user space protocol that acted like TCP. So the goal was to have the same reliability guarantee, to have the same sort of look and feel as an end user, as something like TCP, but provide a better performance than TCP.
So I actually was able to get to a point where on lossy links especially, and certain ā it didnāt work in every case, it wasnāt always faster than TCP, but in a good handful of cases we saw two to five times faster performance. The main thing that I think helps in those cases, itās all about - if you look under the covers, TCP is doing packet loss mitigation; itās trying to look and see āHey, I sent all these UDP datagrams, and things didnāt arrive; things have gone missing.ā And thereās certain fairness heuristics built into TCP, where if youāve got 100 TCP connections that are all sharing an oversaturated link, theyāre all going to try and back off and be generally fair to try and make sure that all of the TCP connections or whoever else might be using that link is getting a reasonable slice of the available bandwidth. Well, the protocol I built didnāt care about that. It assumed that the only thing that was going across that link was this Dilithium connection. And it didnāt try and be fair, or do anything like that. It just tried to saturate the link as much as it possibly could.
And because it worked best for the edge cases where it was, for example, far, and then it also means it was not very common, so it was not competing too much, so itās okay, kind ofā¦ Itās okay greedy.
[00:26:02.14] Yeah, exactly. It was designed to be used in cases where it was okay to be that kind of greedy. I believe thatās still available as a transport option in Ziti. So I think itās still ā I donāt know, I havenāt looked to see if anyoneās pulled it out, but I want to say you can still set up a transport mesh and have that protocolā¦ And again, thereās certain cases where I think it might perform less well than TCP. I think local hosts ā like, thereās a couple places where the fact that it runs in user space, thereās some buffering issues and things where it doesnāt always perform as well as it couldā¦ But for certain kinds of links itās definitely ā youāll get a pretty good performance boost over TCP. And that repository is in our ā if you go to GitHub.com/openziti/testkitchen, weāve got a separate sort of GitHub group. Thereās a ā I believe itās there; thereās a Dilithium project. Itās all in open source, so people can check that out.
Taking notes of all the things youāre mentioning - and this will all be part of the show notes, with linksā¦ So if anybody wants to go and check this out, you will find this in the show notes.
Yeah. If thereās any links that you donāt find or anything, let me know, Iām happy to help out with that.
Yeah, itās interestingā¦ When you say that, Iām thinking of a project that I did as part of my graduation project in university; or it was a somewhat similar, but I like finding comparisons between two fields. So that one, when you do sampling of signals, you have to follow the Nyquist rule, or theorem.
And then that was specialized ā the by-focus was on the subnyquist, when itās a very dead and empty ā I forget all the terminology already, but when it was in subnyquist, then you can change the behavior also to be more greedy. So itās nice to see the same pattern goes over everywhere.
Yeah, absolutely. Iām also a musician, and I do much audio engineering, so Iām familiar with Nyquistā¦ Yeah, I get that.
Yeah, what is your favorite theorem as a musician?
My favorite theorem? I donāt know that I have a favorite theorem. I probably donāt have a favorite theorem, butā¦ Yeah, Iām familiar with Nyquist, and [unintelligible 00:28:03.20]
Yeah. Identity and access management, and audit log, and so on - is this something that is also common in zero trust?
I think different products approach it in different ways. I know for us, we ā so, like any business, we have to find a way to make moneyā¦ So we kind of break our universe internally into what we ā all of our products are open source projects. So we donāt really have special things that we sell for more money, or anything like that. We do ā primarily, our main bread and butter is hosting OpenZiti for people, and running it in a professional way. We call that CloudZiti. So weāve got OpenZiti and CloudZiti. And I know that CloudZiti does have full ā thereās audit logs and all that kind of stuff for all those things. And the way that access management works in Ziti - we have a policy system. So identity ā [unintelligible 00:28:58.16] thereās identities, thereās services, and those things are controlled; like, who can access what is controlled by policies. So itās very much a rules-based access control sort of thing.
In your cloud solution are you using Ziti?
Oh, yeah. We run the exact same code that we ship as open source. We havenāt added anything new to it, or anything. Thereās no special sauce, or anything like that. Itās literally the open source project.
But are you also users of it yourself?
Like, are you trying it once from the user side, and once from the ā so you experience all the pain points.
One of the people on the team ā I always use the old dogfooding analogy, like eating your own dogfoodā¦ Itās like, no, no, no; we drink our own champagne. Thatās a better ā so our champagne drinking is thatā¦ We used Slack in the old days; we switched to Mattermost. So we use Mattermost, and our Mattermost is zitified. So the only way itās completely dark, and the only way you can access our Mattermost is through Ziti. So every day, all day, our internal communications are all through Ziti.
So every time there is onboarding, something in the documentation gets better.
[00:30:10.11] Yeah, thatās the hope. Documentation is hard, though.
Yeah, for sure. Although I guess itās one of those things that ChatGPT can also help with āHereās what we do. Explain this better.ā
Yeah. Iāve actually been ā not just ChatGPT, but Iāve been running a bunch of local LLMs for various writing tasks and stuff, and that stuff can be very useful.
What is your favorite LLM these days?
One of my favorite LLMs is Karen the Editor. Thereās an LLM called Karen the Editor, and itās basically an LLM that was trained to edit text. So I kind of almost use it like a Grammarly kind of thingā¦
Yeah. And I donāt often take the text right out of it, but Iāll look at what it suggests and Iāll sort of rewrite thatā¦ Because I very much like to sort of do my own editing as much as possible. But itās good to sort of diff what I wrote and what it thinks. Iām sure I could just use something like Grammarly and have it be a little slicker for me, but I kind of like the DIY ethos of doing that myself.
Yeah, yeah. Interesting. Iād never heard of that one, but itās also in the show notes.
Yeah, okay. Yeah, Karen the Editor. Yeah. On Hugging Face thereās a user called TheBloke, that anyone that uses open source LLMs is probably familiar with. He has a 4bit quantization of that model that I use all the time.
Whatās a good practice of integrating that with DevOps pipelines?
A good practice of integrating that with DevOps pipelinesā¦
I guess you have internally some DevOps pipelines. You mentioned youāre using generally Ziti internally, so maybe something that you can share that you are doing internallyā¦
I mean, so internally weāve got a pretty extensive smoke test. As a side note to that, thereās another open source project that we kicked off a while ago called FabLab. And it originally stood for the Fabulous Laboratory, or something like that. I forgot ā itās taken a life of its own, but itās basically DevOps tooling for Ziti. And weāve got a number of smoke test components and things like that that run as part of our CI. So we do a lot of CI stuff with Ziti internally to test it and make sure that ā you know, smoke testing and those sorts of things.
So zrok as a project sits on top of Ziti. So itās what we call a Ziti native application. And the thing that makes that interesting - itās about automating the control plane in addition to just using Ziti as a framework for communicationā¦ But thereās a bunch of CI-related bits that we very much run Ziti, and host zrok on top of it. And thatās a service that users can actually go to zrok.io and sign up for. Just like you can sign up for ngrok, you can sign up for zrok; itās the same sort of idea. But thatās all running in production, itās running OpenZiti, and basically thereās some CI components to make sure that weāre always up to date, and that sort of thing.
How do you measure if this is doing a good job with zero trust?
How do we measure?
Yeah. What would be even a KPI for zero trust, other than āWeāre not getting hackedā?
Right?! [laughter] Probably, I would imagine usage. If you were an organization and you kind of wanted to measure zero trust uptake, how much uptake there is for zero trust, the number of services, the number of endpoints, the amount of traffic, things like that would be good indicators of like the success of that, versus how much traffic is not going through a zero trust network or through zero trust endpoint. That differential might be a useful thing to measure to know how youāre doing with that.
And maybe also delay or latency before and afterā¦
Sure. I would think of that as more like an operational metric, like how well is the overlay performing. And I would imagine ā there are cases, like with the whole Transwarp/Dilithium thing, thereās cases where an overlay could improve the performance of what youāre doing, but I think in general most people arenāt using them to try and make things faster or better. I think itās more about ā itās physics. If youāre going to put your bits through more CPU, itās gonna take longer.
[00:34:13.27] Yeah, yeah. Itās a trade-off that probably is okay to have alsoā¦
This is a question in the context of what would you measure and not. Like, is it good or badā¦? Sometimes we have to do that.
Yeah, absolutely. I mean, operationally, CPU utilization, all those normal things are all good things to measure in terms of performance.
Iām trying to understand now if I want to do tracing, and all the tunneling, and like finding smart paths happening. What would tracing look like? Would you be able to see the lifespan and location span of such a request?
I believe weāve got the tooling to do all of that. I know that thatās one of those things thatās looked different throughout the life of Ziti. Things have gotten better, and responsibilities have moved around; that is a tough problem, because thereās a lot of layers to something like this. Itās going through an overlay, thereās various underlays where youāre going from overlay concepts to underlay conceptsā¦ Overlay concepts being like āThis identity, in this serviceā, and underlay concepts being like āThese pairs of IP addresses, and these ports.ā Thereās a lot of translation that needs to happen at various layers, and I do believe weāre at a point now where having run this stuff for production customers for a number of years now weāve had to solve enough problems and drank our own champagne enough that we can ā
I see what you did there.
Yeahā¦ Those tools are reasonably mature at this point. You know, Ziti in general is still pre 1.0. I think weāre pretty close to getting a 1.0 out the door, but weāre still at like 0.30, or something.
There will be champagne at 1.0.
Yeah. I know back in the early days I had built some tooling that would let you look at over ā just like you would do like a TCP dump of an underlay network, like taking a look at what packets were there. There was ā it probably is still there, but there was tooling to do the same thing for overlay concepts, like looking at what flow control is doing, and routing, and some of that kind of stuff, and how things are packetized. But just like most traceability things, Iām pretty sure we can assign a UID to a connection and trace it all the way through.
Yeah. Itās always interesting.
Yeah. Those are tough problems to solve.
Yeah, yeah, for sure. Those are very interesting challenges that youāre havingā¦ And I guess youāre accepting contributions to the open source repos, alwaysā¦
Absolutely. We love external contributions. Thatās one of the things Iām very happy about with all of this, is that weāre very committed to open source; weāre very much an open source-first company. Itās one of my favorite things about the stuff that I get to work on. I feel very fortunate that Iām able to work on open source full-time. Thatās amazing.
Weāre a bunch of real people who are sitting around and working on this stuff, and we do keep track of stars on repos, and stuff. So every time a flurry of stars comes in, thereās a lot of us that get pretty happy about that. So we always appreciate those things. Anytime anyone submits a PR or something, weāre like āOh, my God, thereās somebody from the outside world submitting a PR right now.ā Itās always really nice.
Can anyone submit a PR? Are there some guidelines that youāre asking to follow, or is this pretty easygoing?
I think itās pretty easygoing. We do have contribution guidelines. I think you have to ā I think there might be a little form, or somethingā¦ Like a lot of projects. I think there might be a form or something you have to fill in, but itās relatively straightforward. Thereās not too much to it.
And Hacktoberfest is around the corner.
[00:37:44.13] Yeah, yeah. That would be amazing. One of the things that are part of my world right now is we do have that zrok SDK, and thatās sort of the easiest possible way to sort of get yourself programming on top of something like OpenZiti. You can sign up for a zrok account, download it, take the SDK for zrok as a dependency in a Go project and have a working application in five minutes. Itās super, super-easy. We are rolling out SDKs for other languages and stuff, too. I know for a fact weāre working on Python and Node right now. And Iām sure weāll probably roll out something in Rust, and Iām sure weāll hit some of the high points there, butā¦ I mean, itās actually a really good time to get involved with some of that stuff. Itās really easy to build very powerful peer-to-peer stuff.
The example I built for zrok is sort of a ā I called it a paste bin, because when I first started coming up with this, I was thinking of it kind of like a paste bin sort of thing, but distributed. Itās probably not the best name for it, but itās basically a buffer that you can work with between any pairs of machines. Any enabled zrok machine, you can shove data through it like itās a copy and paste buffer. And itās also the SDK example. The core of it is like three lines of code on both the server and listener side. Like, itās trivial to get your head around some of that stuff.
And if I start with the Go SDK, what is kind of the out of the box, out of the simplest box things that I can look at?
In terms of like the Go SDK?
I mean, at the end of the day itās pretty straightforward. A lot of the SDK itself sort of wraps the identity concepts. So all of the things that you would have to deal with - theyāre complicated; like the identity, and the cryptography around it, and all those things are sort of already wrapped up with a convenience wrapper in the SDK. So all you have to do is call the right API calls, and basically, youāve got all your identity and all those things working. And then from there, it literally walks and talks just like a regular Go server or client. Itās net.Conn, net.Listener, and those things act just like they would act if it was just plain TCP, or something along those lines. So if youāre familiar with writing a network server or writing a network client, itās pretty much the same thing, but it goes over and overlay and itās all secure and zero trust, and those kinds of things.
Lots of questions and lots of interesting learnings for me here.
What is your favorite protocol? You keep saying TCP. Is it TCP?
No. Whatās my favorite protocolā¦?
I donāt know, some of the more interesting things Iāve done in my career have involved building custom protocols on top of UDP.
What was the customization doing?
So in a previous life I worked on a product that was telepresence-oriented. So I actually have a patent for embedding haptic data into live telepresence streams.
So like controlling devices, and things like that, things that involve touch, and then synchronizing that with audio and video, so that itās an immersive experience. Actually, I was the author of a patent along those lines.
Wait, Iām trying to imagine what does it do. You touch the screen and then it makes sounds, basically?
The device on the other end, basicallyā¦
Yeah, so it was used in the entertainment space. But itās the low-level implementation of it I built, and at the time - this was pre WebRTC. So this was before WebRTC was a thing, which is built in every browser now, and itās a good C++ library for doing that sort of stuff. I had to build my own engine that was basically audio/video, more or less like a Skype or a Zoom. I had to actually build that from the ground upā¦ Which sounds like a lot, but ā
ā¦the nice thing about doing that was I was able to cut corners, because I didnāt have to worry about interoperability. So I didnāt have to worry about like conforming to standards, or anything. I could just say āIām going to use VPA, and Iām going to use this audio codecā, and not worry about negotiations, or anything like that. So I was able to build something that let me save some time, because I didnāt care about interoperability. But that did allow us to sort of ā one of the things you have to deal with when youāre writing an audio/video client like this is you have to make sure that the lips move at the same time that the words come out, like when you hear the audio.
[00:42:05.11] So thereās a lip sync thing that has to happen, because generally, the audio and video streams show up independently; theyāre separate UDP streams. So you have to synchronize those in the client; you actually have to play them out at the same time. So synchronizing arbitrary streams of data is just a matter of building a synchronization primitive that lets you synchronize more things.
So you were building this type of synchronization on top of the UDP, basically.
Yeah, yeah. So again, telepresence is always about UDP, because you need the lowest possible latency, and also, versus something like TCP, you canāt ā TCP will hide the fact that thereās packet loss from you, whereas UDP, youāre dealing with low-level data grams, so you know if you lose packets. So the way that you tune ā if youāre building like a telepresence client, the way you react to packet loss is youāre going to tune a codec, or change your transmission rate; youāre going to change things in a different way than you would for like a network protocol, because youāre concerned with latency.
Yeah. And you can interpolate, and thatās okay.
Yeah, exactly. Thereās things you would do in that domain that you wouldnāt do in other domains. Itās one of the things Iād like to work on at some point. Iād like to actually do some telepresence stuff on top of overlays in things like Ziti. One of the cool things about zrok is it lets you share things really easily without having to worry about āDo I have to open ports?ā or any of that sort of stuff. Itād be really cool to do a self-hostable peer-to-peer telepresence setup on top of something like Ziti. Maybe itās something Iāll get to work on in zrok at some point, I donāt knowā¦ But I would probably enjoy that.
Interesting, yeah. There must be some interesting usage in here also for interpolation, and just using UDP to make it fast [unintelligible 00:43:43.21] some data. We have to brainstorm later.
Itās still a thing in our roadmap to talk about ā you know, the mesh network that we build now is all aboutā¦ It uses TCP connections primarily. TLS-encrypted TCP connections. But itās still very much on our roadmap at some point to get around to renewing the work I started with Dilithium, and talking about UDP-based transports and things for Zitiā¦ Because you can get some performance out of those things.
For sure. How reasonable is it to plan some UDP for zero trust? How bad would be packet loss in the context of zero trust?
I donāt know that those are necessarily super-related. On top of the overlay itself - so we actually do TCP connections between routers, but the data gets repacketized on top of the overlay. So itās not like a connection between endpoints goes across the overlay; itās basically taking all of the local loops from each end of the ā so you have the overlay, youāve got the endpoints, they each have their own connection to the overlay, and then the data that goes across the overlay is packetized again at a higher level. So it actually acts like UDP across the overlayā¦ Which is part of how we avoid ā one of the things people often talk about is youāre going to sort of tunnel TCP through another layer thatās more or less TCP, you end up with weird effects with flow control, and those kinds of things. We avoid that by repacketizing the data.
And is data loss that can happen with UDP a problem?
No, because all of these things have the normal kinds of flow control and retransmission and all the things that you would expect to have in any sort of packetized network.
Mm-hm. So basically itās solved on another layer, making sure that this is consistent. I see.
Yeah, yeah. So the overlay guarantee ā well, if you send a packet across the overlay, itās guaranteed to arrive just like a TCP connection. But that may actually incur packet loss that needs retransmission, and some of that sort of stuff.
All those trade-offs.
Okay, so UDP is your favorite protocol.
I would think so, yeah. To me it feels very pure, andā¦
Yeah. To me, UDP is sort ofā¦ You know, datagrams - thatās the internet.
Okay. Then I will ask one more question as we are moving into the unpopular opinions.
Jingle: [00:46:11.17]
So do you have an unpopular opinion? ā¦on any topic at all. It doesnāt have to be on packets or on protocols.
I have a lot of unpopular opinionsā¦ Yes, I have an unpopular opinion. My unpopular opinion is I think weāre going to look back at some point in the future - letās call it 10 years, 15 years - and this whole model of content and things being shared, or things being hosted by somebody else for the sake of convenience, streaming, whatever you want to call it, will flip back around. I think weāre going to move back to a world where I can own a piece of content and curate it and manage it myself. I feel like itās sort of a fundamental human thing to want to collect things, and curate collections, and build collectionsā¦ And I think the world we live in right now, where everything is sort of streaming doesnāt really support that very well. I mean, itās very common you might fall in love with some TV show and all of a sudden itās just gone. I think weāre gonna move back to a place at some point where the ā I donāt know how that will look technically, or how thatāll work socially, but I think weāll move back to a place where you can sort of say āI own this thing.ā And I think thatās an unpopular opinion.
I saw an interesting startup that does something a little bit similarā¦ In some way, letās say it reminds me of that - as an artist, you upload images, or something that you do, and then people can generate images in your style, and you get the royalties for that. So that sounds like a ā
ā¦relative of this concept.
Yeah. I mean, personally, I kind of wish there was a ā if youāre familiar with Bandcampā¦ I walk this walk. I have huge collections of media that I curate, and I own. I still buy CDs, and I still buy a lot of music from Bandcamp, because I like to download it and own a copy of it. I kind of wish there was something like Bandcamp, but for movies. I donāt actually care about owning a BluRay, but Iād love to own the highest possible quality copy of like my favorite film, and be able to legally download a good copy of it. That would be fantastic. Iād be a huge customer or something like that.
Interesting. I think I will disagree with you here. I donāt mind having this just as a on-demand. And Iām also not the person who watches the same movie twice, soā¦ Maybe Iām not even in the focus group anyway, but itās a good start for an unpopular opinion, yeah. [laughs]
Yeah. I figured that would be a very unpopular opinion.
I have an unpopular opinion that Iām still trying to phrase on this fieldā¦ So Iām mostly agreeing with it, but maybe Iām still not fully agreeing with it myselfā¦ But zero trust is a little bit like a trust issue. So if you want to have practical trust issues, maybe you can learn something from the fundamentals of zero trustā¦ But maybe itās a bad idea. Maybe I will vote against this when this becomes a poll on Twitter as well.
Well, that was educational and interesting. Thank you for answering all the many questions, and thanks for a great conversation on networks as well while at it.
I really enjoyed it. Thank you for having me.
And all the many, many tools and links and repos that you mentioned will all be in the show notes.
And weāre all real people, so feel free to reach out. Weāre on GitHub, and weāre very, very reachable.
Thank you very much, Michael.
Changelog
Our transcripts are open source on GitHub. Improvements are welcome. š
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK