Automation and Transformation in IT Infrastructure with Jordan Lowe

Hi, and welcome everybody. This week on network a f. I'm talking to my friend, former collaborator, and, former boss, Jordan Low, Jordan has, runs a company called Defft, and, we're talking about what's fun in, and, frustrating in networking and nowadays. Jordan, could you Give a little bit of a brief intro?

Yeah. Thank you, Abby, for having me. I'm excited to to join you on here. So as you mentioned, I'm the founder and CEO of DEft, which was, previously known as Server Central.

Before that, I was working with a company called N Lair, which got acquired by what is now GTT, which is a lot of the networking stuff we'll be talking about today. I'm sure. But that's it. You know, I was with, you know, sergeant Venge.

We've as we rebranded to DefNow, it's been, so over twenty two years now. So I've been mostly doing the same thing for quite a while. It's been it's been a lot of fun. Well, it's a little different every year over year both on the tech and and business side.

Where where did where did service central start?

That's a great question. Start it started in the dorm room. I mean, truthfully, it's, it's one of those stories that, somehow, I didn't success. You know, I, you know, I, Yeah. We started with an old courier modem, at my college department. You know, I still remember updating the firmware on that thing, to, you know, v dot x dot fast or whatever the latest My latest thing is to get that extra two or three k off the the dial up speed, but, you know, twenty two liters, here we are.

And it was you're also doing DNS and and things like that, you know, overlay issues. Yeah. So, yeah, then when the company started, we were doing hosting. We were one of the first people to kinda combined domain names and web hosting into one plan.

So we're basically giving away domain names, which was unheard of at the time. This is back when, you know, network solutions was charging thirty five a year for a domain name. And, we started bundling it in with, you know, a five or ten dollar month hosting account. People didn't believe it was true.

But, that ended, you know, pretty well. A lot of people, we, you know, got thousands and thousands of customers there. And that's, kinda, what kicked us off into what we are today. We, you know, we kinda wrap that business up sold it off in, I think, two thousand eight or two thousand nine.

But that's, you know, that basic customers and those, that, that experience look kinda got us where we are today. Cool. So you benefited from another of, my guests and friends Elliot Noss and open SRS and two cows and what they were up to. Absolutely.

Yeah. No. That's cool. So you've been running networking for quite some time. I I won't ask you.

What year you first got enabled.

I think, yeah, but for me, it was way too early. It was actually, like, you know, in in in college, really AVs, and I tried not to do too much damage.

But, you know, things are different, but things are same. I mean, what what's what's exciting about, running a network, what's cool about running a network nowadays? Lots of stuff. I I was just thinking back to when I got, you know, when I first started playing, actually, like, What was that?

I was, you know, it was a little bit earlier. I mean, I was, you know, I think my first computer was like a Lisa Which of what I mean, you know, what are the latest or running Lisa? I think it was probably Lisa too, because I'm a little younger than you. It was probably like eighty, eighty nine.

Uh-huh. So it would have been the end of the first time I used a Lisa, I demoed a Lisa using Lisa. The first time I used a Lisa was when it they, you know, made a Mac clone so they could run like Mac OS. You know?

Yeah. I it's still remember that. There was, like, a phone cord connecting the mouse. It was, it was pretty classic.

Yeah. Then I got, like, a Mac LLC, I think, in the early nineties, my first real computer, and, you know, all remember the first bone, the first, you know, I guess, I guess it wasn't around my first business at Service Central. When we started that, it was actually, I guess I ran a BBS for a little way back in the day. Remember I having a couple phone lines in my parents' house?

Yeah. I mean, that was, that was some some classes. I always looked at at home. That was how I got into b side because I had pent up frustration from not running a multi line VBS, but my, you know, we weren't into it.

My parents weren't into it, so when I was growing up, That's fine. Sorry. I apologize. You're asking about, you're asking about, you know, what what challenges networking has today or, you know, what Or let's look at the positive side.

I mean, For me, it's, like, I turned I plugged in, you know, hundred gig connections, and I was expecting that the heavens depart and or there to be tons of problems. And instead, it was just like, choop choop. Wake up. You know?

I was like, wow. That was pretty anticlimactic. I mean, you know, it's it's some ways, it's pretty cool what what's going on with networking today. So I thought I'd It's awesome.

Yeah. I mean, you know, I guess I'll open up with us, you know, with us being old. You know, like, when we started, things were a lot different. Right?

Like, you know, to get a t one or a DS three was a huge deal. It was very complicated.

You know, and it, you know, wasn't that fast, you know, even back then. But, I mean, yeah, I mean, having hundred gigs left just work today. I mean, four hundred gig is, you know, there's putty carriers out there that can you can get four hundred gig ports today, which is which is, which is crazy. And, yeah, it it does just work now.

It's a lot, you know, stuff, you know, what's dicing over time is stuff isn't as complicated. It does generally just work Mhmm. Which has taken a lot of pain and a lot of, you know, I'm sure bugs you know, multiple people, including us have found over the years to keep this stuff working a little bit better. But it is, it is pretty nice.

I I will say things are much easier when it comes to, you know, adding capacity and and just general things at the moment. It's, makes a lot of things like DDoS, less less painful. You don't have to worry as much about office you did back then when anybody could kinda take out. Now it takes a little bit a little bit more energy than it than it did before slightly.

Slightly a little bit more energy to throw more than a couple hundred gigs around. Who was I talking with last week? We were talking about the truck that people used to do who had IRC servers of, like, buying a, buying a connection for someone you hated and single homing, you know, to IP space from them, your IRC server, you know, so that, so that they would get the attacks instead of you. So it was like, early DDoS defense.

Yeah. I mean, that that's actually one of the things that kinda trained us and keeps on our toes. You know, we've we've had an IRC server entire time, basically. Oh, wow. So, you know, we we still run it. It is still up. And, you know, it it it, obviously, these days, it doesn't get too many attacks anymore as RRC too exciting after things like, you know, Slack copied and and and Maybe we should show IRC through a lawn mosque, you know, maybe he wants to, you know, bring back IRC and, UsNet, you know, for decentralized future deliveries.

It would work. Still still out there. I mean, it is and, you know, still decentralized.

So what about, you know, automation? There's a lot of hype about it, but I was, talking to some folks that are customers of yours. And, you know, they said that, you know, it's a big part of of doing the design and getting automation and, you know, having stuff up, you know, how does that differ from twenty years ago when you, when you turn someone up with a minute, you know, that has multiple switches and routers and surfers and and stuff.

Oh, yeah. I mean, it's changed pretty wildly. I mean, you know, back in the day, you know, the best you could do with some scripts But then a lot of the router vendors, you know, we've always been preferential with Juniper. You know, Juniper allowed us to do have a little more flexibility in the early days and kinda load more of the, you know, customer side fake into the router as a template.

So we were able to kinda make it a lot easier to turn customers up, not having to repeat, you know, seventeen commands every time you're adding a new BGP neighbor. But, I mean, these days, it's a completely different world. You know, it's all, you know, and what, you know, what would be like an ansible setup or, you know, these Python scripts that really allow us to really do a much more modern job and really not have to lock on the switches at all.

Not saying it doesn't have its own kind of drawbacks and then problems, you know, the you know, the the generation of people that are used to being on those routers and and checking things and the process they do is a little bit different than, what it is today. It is easy to miss a couple things. You know, we've we've definitely had our our own challenges as well. You know, with the automated side, you know, you might not be monitoring the same type of stuff that you were before.

You know, so the same kind of thing is it would kick off a, you know, CLI error. You know, you can write that a CLI error happen. It might not actually work, and then you don't know about it. So, obviously, every time we make a change like that, and we learn and script around, you know, scripted and and add an air condition, it's it's programming the big challenges is, you know, like, I remember when I was trying to use Brocade for the first time, and I was cutting and pasting because it was so close to Cisco.

And, like, a halt they had a thing where every time you made a change, it would say, like, yes, I made a change. So I didn't notice that it wasn't actually taking the route and then I turned on BGP with no route maps and became make clueless because, you know, so whether it's CLI or, you know, when you try to automate and do things too fast, And then, you know, hopefully the vendors don't change, you know, API response codes. And, look, we're we're we have this as SaaS company, we have the same challenge, right, which is people build to your APIs. That becomes a promise, you know, in in the router world, CLI is a promise.

APIs is a promise.

We could move to API if people would just make everything only available for available via API also. And some vendors do better at that, you know, than others. So Yeah. But, I mean, mean, realistically, you know, all of our new deployments, you know, have have all been able to be automated.

And it's it's a very different way to work. And, you know, it's go ahead. So are you at the point where for, you know, I I know this one deployment wasn't cookie cutter, but, you know, for for a reasonable deployment that you're really just getting in via CLI, either to to bootstrap, you know, an IP address or to just check that it feels right at the end. Yep.

Yeah. Yeah. Generally. And if if even that so again, that's only the newest stuff. I mean, a lot of our, you know, I'll I'll say legacy equipment, but, you know, it's still all over.

A lot of our older stuff is not accepted that spec yet. Well, you know, our team is working really hard to get there and they're doing an awesome job. But, I mean, we have, you know, being around for twenty two years gives us a lot of, you know, what, I guess, what we'd call a cruft We have, you know, lots of devices, lots of things that are do not, you know, perfectly align with everything else. So, you know, you can't really automate everything working our, you know, as far as we can, they get all the kind of standard stuff moved off in into databases and where we can, you know, really have it all automated.

It's great. You know, because, you know, what, you know, idea for us, obviously, a lot of customers to do a lot more. You know, we've always had a couple fun things customers could do that were, you know, groundbreaking. We did it, you know, adding letting customers add their own, like, no routes or add their own firewall rules you know, that was cool ten years ago, but now it just kind of expected, to be able to do stuff like that.

But but realistically, you know, a lot of people have not been exposed to this stuff. I mean, even our customers seeing your product, you know, seeing KentIC, for example, it's, it's life changing. You know, we, we have customers every time, you know, we show it in our portal demos. You know, it's not something a lot people offer.

It's super appreciated. It gives them insights and something that they never think about. And it's actually, you know, it's a huge win for something that, you know, we've obviously to that when you started the company years ago. People love it.

And, you know, there's a difference.

We have to talk with through it with some companies because they're like, wait. But if the customer can see why their links are full, then maybe they won't buy as much. It's like, well, but maybe they'll just be happier. And by more from you.

So sort of like the old QOS discussions. If we rate limit the, you know, sell them sell them as much as we can and hope they use none of And that's how we're gonna make money. You know, it's just very different from the modern business philosophy of, let me make this customer, like, super enthused, and they will go somewhere else and bring us in, and they will tell their friends, and they will adopt it widely. You know, it's it's, it's sort of doing well by doing good, but know, brought to the business side, which is pretty cool to see.

Yeah. Definitely. It's, yeah, it's it's been fun, man. I I, you know, network stuff is is definitely close to my heart, and I love playing.

I I love getting the latest stuff out there, and, you know, it's, you know, it's it's a very gentle kind of prodding way to do to to move stuff along because most of our customers don't wanna take an outage to kinda shift platforms or shift deployments because it doesn't change anything for them in. Like, they're not really getting a big win. We're getting a win, obviously. But, you know, we have to say, hey, this is long term.

This is the first ability. This is to get you on the latest, everything. And, well, you do make sure the greenfield stuff, the new deployments, you know, are are run that way. It is I I really would like, when I was learning TCP IP, There was still a question in the late eighties about the ISO protocols.

And I think the only I don't know if there's anything left, but is is, you know, from Aisalend in sort of modern networking, but I really, in the late eighties, again, early nineties would have thought that by now, the way the internet would be built would be different. Instead, we have we have IPV six.

But we also have IPV four. We have BGP version four. We have, you know, we still have RIPV two up through. And so it's that's a testament. That's really awesome, especially as someone that doesn't enjoy doing the, you know, endless protocol meetings and RFC type stuff. You know, I'm I'm thrilled that there's people that love ITF. It's just not for me with my ADD.

But, no, I mean, that's that's pretty cool when I look and see, well, you know, we're communicating over Zoom. And the internet is mostly working. And so it's sort of a mystery to me that it hasn't moved, but also pretty cool that it that that it's been able to work just going from, you know, a megabit to a gigabit to pretty soon we'll have terabit interfaces. So I mean, it was Pretty amazing when when COVID started and everyone started doing Zoom and and, you know, all the different meeting platforms and it generally worked.

Obviously, a lot of, you know, people running around crazy upgrading stuff with, you know, the back of all the consumer networks, and I'm sure everyone tell you stories. All that works is like, oh, we gotta go do all this stuff and all this splits and all this stuff. But, you know, in the end, it got done. It worked.

You know? Zoom was able to spawn up, spin up a ton of servers. They made a couple small, you know, they did do some things to limit bandwidth obviously initially because they, you know, nothing all, you know, that's a, that was a huge increase. You know, they wanna limit bandwidth for both their, you know, their their equipment, Amazon equipment.

I'm not gonna say not their bills as well, but Right. You know, they can they it kicks some defaults down to four eighty p or whatever the the lower bandwidth rate is. And but they already had a pretty good platform there. You know, they already did a good job of making all of the speakers' video a full speed.

When they were speaking, everyone else kinda slowed down to a lower bit rate, which is, which is, you know, already existing, which is hard. So customer base. Just people had a bunch of over provisioning in place. So, you know, that was that was really good.

But, so sounds like life cycle automation is something that you guys have.

That is really where we see most of our customers, you know, working towards.

And, of course, if you can't see it, you know, then that's not really that's not really helpful.

I don't we don't really have any customers doing the magic streaming telemetry, you know, stream four hundred gigabits of state from the thing and have magic ML, you know, show you everything that's wrong. But there's a lot that you can do just with, you know, device metrics and, of course, traffic performance and all that.

So what are the challenges of, you know, running a network? It's much more efficient to run network than it used to be, you know, certainly per bit, but, you know, what are the challenges?

Yeah. I mean, I mean, just off of that, you know, lately, it's been hardware availability. I mean, that's kind of the the big thing both on the, you know, the, the W. D. W. M. An optical side, and top r x, which has been a bit of a challenge.

We were lucky enough, you know, our our team did suggest we placed a lot of large orders for both network equipment and and compute and storage right when this loss happened. So that's all been kind of flowing in over the last, you know, year or two, which has been working pretty well. So we haven't had any kind of major issues, but we've definitely a couple of upgrades here and there. We definitely have a couple locations on our network that are a little bit behind. You know, we have some sites running older routers and then we're planned.

Nothing kind of traffic. But, I mean, you know, hardware wise, you know, it's, that's been the biggest challenge, really.

But, I mean, people has been is has been a challenge, you know, know, I'm lucky to have an awesome team working with. We have a bunch of really great guys, you know, between operations, automation, you know, platform teams that all kinda, you know, network side It's really great. So, you know, we we've been lucky we've got a great team.

You know, what else? I mean, stability is obviously always a big challenge today. With using all these kind of newer protocols we've talked about, like, you know, the newer deployments are all, like, EVPN and know, v x, you know, v x LAN, which are, you know, while they're not that new, they're still new, you know, first things that are twenty years old. So they're not as stable as all, you know, it they're not as stable as the old stuff.

Now that's a pretty small percentage of issues, but there's still more issues and bugs on the new stuff, and then there ever was force. We've gotta be very careful about testing, very careful about routing the right, couldn't, you know, purchase a code, all the right stuff because there are issues There's still some interoperability issues. There's still, you know, just a host of things that we've gotta watch out for all time. So it takes a lot of documentation.

I'll review a lot of testing especially on the the stacks that run the latest and greatest stuff. Which ties back to the people, right, which is I love it that networking is like I mean, I hate it and I love it that networking is like physics where you have to, like, know the first principles and be able to building your head a model of what the thing is doing in a Einsteinian thought experiment to say, well, is this what it should be doing or is this a bug? Like, sometimes just things break and sessions die, but sometimes, like, routes just don't get advertised. You're like, well, maybe I used it wrong or maybe my script to automate it is is wrong.

It's it's not as bad as the nineties, is it when, like, you know, you'd stick the fitting card in and sparks would spit out the hissing port. And you're like, how could they make software do that. And it was all one, like, a one operating system program loader in the old, big, bad old Cisco days. I I'm just I was actually thinking back to my dorm room where we had we had a local talk.

So I don't wanna get called there, but, yeah, we have those local talk things. X n s s n a local talk. Yes. Yep.

Yeah. That was before that I went to, you know, before, you know, I just remember doing all this stuff over, what was the original coax ten base two or something? Or, What was the original coax network? After thick.

So you weren't doing vampire taps and all that. It was just it was just the net. I mean, it's just ethernet the net stuff. I was I was trying to tell me to but back back to the networking challenges, you know, as you're saying that, you know, we we have a different and we've had a experience than people coming up, you know, today.

You know, all, you know, the the the the kids as we'd call them today, coming up with networking, you know, they never got the chance to do any of that stuff. A lot of these guys have, you know, they've grown up writing network as code. They've grown up, you know, starting all, you know, playing with Amazon as their, you know, original platform. They didn't build the server.

They didn't build switch, they didn't program it, set the IPs and get the serial console working with, you know, back in the day, you know, you know, trying fourteen different USB adapters, like we've had to do over the last ten years.

But, I mean, that that's been a little bit of a challenge, you know, just people, you know, learning the basics when problems do happen. You know, you need you need to kinda thick back to how how it does work. So that can always kinda be fun. You know, obviously, as as I said, you know, most of our guys are, are awesome and then have some of that experience, but new guys, it's always fun to kinda get them in the lab and and break something and and learn how to fix it the first time. When it when it's not something that's running, you know, a lot of people come into a running network. So do you have a lab both for training and for, you know, testing new testing new configs and software and all that. And is it virtual, physical, both?

Both. So we have, both the physical app for, like, like, even from, like, the DWDM side, you know, we have, like, WDM labs with, you know, like, we have, like, for testing and training for both our data center operations guys, the guys that are going to patched off so they can learn how to do loops in light level testing and cleaning fibers, things like that. And then that's what goes all through. The network labs, where we have, you know, basically one of each type of router and switch that we deploy if not multiple, then we can go ahead and, you know, stage code on there, run upgrades and, you know, link it into the network.

You know, you know, do traffic tests, stuff like that to make sure everything properly. And that's that stuff is, as I mentioned, is more and more important because not, you know, before you can kinda get away with just, you know, doing a doing a shotgun upgrade, these days, you gotta be very sure working correctly and then test everything very carefully because it's not working. There's some gals among the guys, you know, at the f done, as well. But, yeah, I I the early nineties, I I think I still have some people that work for me who have trauma.

We used to have a bat light in the office that would go off when I enabled. And, you know, when I was coming to a pop, I would just pull the UPS, you know, and make sure that it worked. Cause if it was gonna die, it would be faster if I was there, because we were very lean and you know, a very small org at the time. And so, yeah, I mean, things are things have definitely, changed hugely in terms of how we think about that.

Yeah. You know, customer's always on. So, networking doesn't work. And even the cloud outages, you know, most storage outages in cloud are really network code or configs or, I mean, everything is sort of layered down.

It's not like it isn't the same technology. It's just, you know, weird and different names. So Yeah. Scaling issues, deployment issues, or code upgrade issues that break seventeen, you know, dependencies to make it all work.

Learn learn hard lessons. Yeah. Yeah.

Yeah. Which do you see as more challenging?

Taking someone that is cloud native in their thinking. So they're using peering and tunnels and things like that, but maybe not thinking about them as that. And then getting into networking and primitives and, you know, how to live in that hybrid world or sort of the reverse and taking people that have come up on networking and computers and storage and know that stuff, but don't know the names and the bugs and the APIs and you know, don't think program first or get ops first.

You know, which, which, which is more what's your approach and which is more challenging? I mean, I I would say, you know, most of our customers are not super technical. You know, we've got a lot of awesome tech forward companies with, you know, great staffs. But, you know, a lot of our, you know, I wanna say more enterprise customers or or just normal customers are not, you know, don't get into the weeds. Right? They just expect them to work. And the downside of that is when it works perfectly for years and there is an issue, they're like, you know, they get upset because they got away with not having redundancy, it just has to recovery things like that, where if you have a little blip or a little problem, you know, the the world is sending.

So, you know, luckily that doesn't happen very often.

But, you know, versus the, you know, the technical customers, I think, understand if, you know, if there's issues or problems a lot more, and it's we really enjoy talking to them. I mean, one of the best things we can do is when our staff enjoy talking to our customers, you know, when our guys can talk about and geek about stuff all the time and and get into it and talk about challenges and issues and build new cool stuff. I mean, that's really fun for the staff too. So truthfully a good a good mix of that stuff is the best.

You know, like, we have, you know, a bunch of people. They're just happy being stable and and running. And then we have the fun customers we get to talk tech with and build all kinds of new solutions for. And so do you have different teams for sort of cloud networking migration versus network ops versus some of the automation and tool side, or, you know, we do.

We do. We do. We do. We do merge and not So there's a couple different groups.

You know, we what we separated out, I would say, maybe a year or two ago was kind of the the projects and tooling team from the operations. And support guys. So that way, you know, before, you know, it would be, like, the same guys would be kinda trying to do everything, you know, take the support calls, work on the automation, you know, test and accept grade. The problem is, you know, customer stuff always comes first and they'd get backlogged and we wouldn't get to all the fun projects.

We wouldn't get to get everything tested and upgraded in time. We separated all that out, which was a good strategy. Now we have kinda three different groups that kinda handle all those different different items. And that way, that the automation guys and tech guys are focused on, you know, new customer turn ups, automation, going back and fixing stuff, making sure, you know, for example, Some of the automation requires all report descriptions to be the same or exact.

Right? Stuff like that. You know, going through devices that might have been, you know, sitting there untouched for a long time other than software upgrades, just making sure everything's perfect, making sure it's all gonna work. About a little stuff and then, you know, tuning all the monitoring.

You know, we've got all kinds of different groups for the stuff now, which helps us, helps us kind of get through everything a lot better versus just hoping, you know, pushing someone to make time for it. Now it's just their job is to do that part.

Yeah. The just to come back to it about automation, that does seem to be the the real core that everyone needs to have, which is the life cycle when stuff is turned up, turned down, when we onboard You know, it's almost like we have customers that ask us if we can do PS, and we're looking for good partners there, you know, when someone comes on board and you know, they've they've got they wanna they wanna sort of automate monitoring to follow applications, and we can do that on the Kubernetes side or on the cloud side. But when you have a network full of interface descriptions that are wrong, that's, you know, definitely a heavy lift, to get into.

And And, but it it isn't always it isn't always possible to just say, well, just redo it all, you know, greenfield and then figure it out. And There's yeah. I mean, yeah. I mean, there's thousands of thousands of, you know, tens of thousands of interfaces in our case.

And, you know, we we've made mistakes. You know, like, you know, for example, like, one of the easy mistakes to make is one of those typos that means customers might not get built, for example, or the customer not get, you know, are learning my network, monitoring my network. Right. So we've, you know, it's takes a lot of energy and time to make sure that we have we we put checks in for each of these changes to make sure that there's a port up without matching description.

You know, it does, you know, does something like that. So all those kinda you know, side cases and corner cases that we catch.

That's a it's a it's a lot of work.

Yeah. And it's it's a big jab. So how many about how many people is deaf right now?

Hundred and thirty something, I believe. Mhmm.

So you mentioned you you stone n layer. I was there for that.

And, You still so you you sold in there, but you still do peering and and run a Is it global or is it national? You're you're sort of peering in transit network, you know? Global. Yeah.

So is that Or why why do you do that? Why not just buy transit? You know, what's what's your what's your view on that? Is it, you know, automated? It's not a big deal? Is it is it is it for cost performance, security, availability?

All of it. I mean, first of all, you know, a lot of our customers, So the back point the back point portion allows us to do a lot of things. I mean, everything from, like, on the peering side, obviously peering with, you know, our big traffic it'd be people who sell out traffic, you know, in the US, for example, like Comcast. You know, so we have, you know, multiple hundred peaks with Comcast in each market.

If there is an issue with Comcast and one marker, they have a router maintenance, you know, or some sort of issue, you know, having the backbone allows us to send that traffic to another city. Kind of rat around the issue, you know, pull that traffic from Chicago at Ashburn, send it out in Ashburn so everything kinda stays smooth for our customers.

The big part of this disaster recovery and and, you know, DR between the locations, you're having a backbone allows us to move IP addresses around the network. So if we have a customer that's got, you know, an active active setup, we can actually automatically move IPs between markets.

So, you know, as part of, you know, well, for example, we normally do like lead DR test for our customers. We can, you know, flip from active to passive and from Chicago to California, for example, and move the IPs across which for some legacy's people, and for some systems, there's a little bit, you know, it's it's I don't wanna say it's the the most modern way of doing failover an application stuff, but it's a really good way for a lot of, applications that are a little bit older to have just, you know, the everything keeps running on the VMs or you just point, you know, the public, the public IPs change.

Try the public IP stay the same. We just get moved across the market. Right. Yeah.

So that's, that's a fun thing. And, also, allows us to do MPLS networks, things like that for our customers. A lot of customers that we interconnect offices around the world. You know, so they might have an office in London, an office in Ashburn, an office in Dallas, and we'll be able to link all those back up to the same backbone.

And us running around back mode allows us to kind of fully control that. You know, it's like control your own destiny stuff where we see the backbone we can tell if there's an issue or an outage, we can do the routing ourselves, we can do the quality control to make sure stuff is scaled properly.

You know, just allows us to see everything and mix the, you know, the, any issues a lot faster to troubleshoot diverse trying to call into one of the carriers and wait on hold, and hopefully stuff gets fixed.

So, yeah, I use the word control there, you know, to be able to could be able to control the packet density. You know, I guess that we see that also, you know, it's a big thing. Is is peering set up mostly automated now in your life cycle stuff? Or I'm not sure.

Actually, that's that's a good question. I know. We don't do a ton of public peering at the moment. And it's, you know, the the peering world goes through kind of a there's two parts of it as as you surely know is is both the price and the size of the ports on the peering exchanges.

So the our cash fly brand, you know, the CDN side does appear in all the markets, obviously extensively and we interconnect with them in some markets as well. But you know, on, on the theft side, we do peer in certain markets, but most of our peering is kind of private peering with, like, the amazon's or, or people like that. You know, obviously, we're not supposed to talk about who we interconnect with. But, most are appearing at the moment is private caring. We are on some of the exchanges.

But it's one of those things where there's a couple business cases behind it, both the cost of the ports and the cost of transit.

Depending on those two things, it kind of incents you to either do pairing or not do pairing. And, you know, right now on the cycle of things, we're currently in the doesn't make a ton of sense to do a huge amount of peering right now. Traditionally, I would be on the other side of that. Right? Like, traditionally, the transit expensive, and it was faster and better to do well appearing. So right now, we do appearing where we need to, but most of our stuff goes out over, you know, over we have multiple obviously different transit environments in each market.

And, you know, it's it's a little bit easier. Right. Or if the c n CDN was a core in house, you know, core in house product, then it might be different. So So people don't yell at you to stop playing with the routers anymore.

Okay. I am not in the router too much anymore. You know, it's it's pretty unusual. I'm in there, which is, which everyone will probably tell you is a good But I definitely can get in there and, I know enough to be dangerous still, but, you know, all the latest, you know, MPLS automation stuff is people people better than me are handling that now.

I've got to, go down. So I'm going to ITW in a few weeks, and then I probably won't get to appearing for him after that. But, I I threw in a couple of arrestas, and it's time to get, the twenty year old cat tube you know, sub seven twenty three B XL, out of there with filtered slash twenty fours. So I break anything.

That is not our production network. That is our lab network, by the way. They do not it's not they don't let me use the production network, but they're really again, there's there's there's no need for me to be on our production network. It's automated push, everything.

There's no, you know, CLI is only for super debug. So don't know if you can see the shock in my face for it to even say you still have a supe seven twenty running, but that is Yes. Impressive, sir. Five years.

No. It works okay. I mean, then again, the arista's I'm doing. They have something called selective selective advertisement.

Basically, a route map between the the fib and the ribs. So, I am we have a bunch of customers that use that. So I'm looking forward to playing with some performance based and, you know, selective, selective advertisements for people that wanna use especially given the equipment shortages, older stuff as edge as edge devices, you know, that's a specialty that they've had for some time. So Yeah.

Yeah. I mean, that's what I mean. We've, you know, in my in my days, I've definitely done that on multiple platforms, you know, having to kinda say, okay. We're running out of fib space.

We gotta start start limiting a few things. Yeah.

You know, especially these days with, you know, again, we don't have anything in our core that, obviously, we need that. But there's definitely devices out there that can't, you know, the amount of routes out there now is insane. You know, everyone's had to find every twenty four. They can find under every carpet, and, you know, like, the number of routes is just We'll have where we're gonna see it as unbelievable. I don't wanna call it a debate, but maybe a real talk on v four versus v six. At some point, I'll invite you on, and we can talks about that. I have been surprised that the routing table hasn't blown up more.

You know, until there's someone that Someone major that says I'm only gonna be reachable by v six or, you know, eyeballs or or other or content you know, we're just gonna be living this, continue living this dual home world, which, you know, seems like it's gonna continue for some time. So, you know, I think that would be a great show, and you know, I don't wanna you know Yeah. I mean, that would be entertaining. Yeah.

I actually agree. You know, like, I don't know anyone that could give anybody a business case to say turn off before, unfortunately, I mean, it would obviously be great, but realistically we're still as we've all been saying, we're years and years away from it's just topical because the Danog thread about using two forty and reminded me of when I was on the Aaron Advisory Council, you know, p v eight and Jim Fleming and jumping for a wormhole and peering with urine. And, you know, it's like, I don't know. But, yeah, problems.

So, so what what is new and emerging and hot and, you know, interesting, you know, as you think about the next few years, what are you looking forward looking forward to see coming down the road on the networking and the tech side and the business side also. Yeah. I mean, networking and tech, but, you know, there's a lot of cool stuff coming out networking wise, especially on you know, on the optical side. There's a couple new things that'll make it a little bit easier to do what we do on kinda what what they've branded DCI data center interconnect.

Because you can imagine, you know, we, we have a lot of, circuits between data centers and between Was this like robots or lemur glass and and and electronical tune electronic tuno tunable prisms? Or what is it? No. I mean, just just the integration between platforms I think will be a lot easier and also the the platforms themselves. There's always kinda you know, I don't wanna get too far into the optical side. You know, there there's all different systems that that should work together that sort of work together, but don't really work together in between all the major vendors.

We've been asked if that work about together, work better. And there's a couple of proposals out there and things that seem to be happening that will make that a lot easier, which should be nice. And you can just kinda, you know, right now, most systems, if you want, like, a ten gig and a one gig and a, you know, forty gig and a hundred gig, it's not that easy just to plug all those things in the same box. Mhmm.

And there's some new stuff coming out that's gonna make that a lot easier, hopefully. So we'll see when that comes out. I mean, you got still a couple years out. That'll be that'll be nice. You know, if we can, you know, that way we can just plug in a, you know, ten gig and a hundred gig and forty eight for a customer, and we don't have to have a different type of card, different everything.

You know, we're using more not things that are not just passive prisms muxes.

Correct. Yeah. Yeah. So, almost all the stuff we do now is active active, you know, carrier grade optical here.

Mhmm. And a lot of that stuff is kinda split up into, you know, hundred gig. I see. So that's more fixed.

Because on the on the routers and switches, they don't really care, and the the passive stuff doesn't care that much. Yeah, there are there are some change on the passive side, just the sizes of channels.

For stuff, you know, the traditional stuff is all the, you know, the hundred gigahertz band. Obviously, there's people that are much more expert at me at the stuff that would be happy to chat about it. Yeah. There's some changes coming there where there'll be a little bit more.

Mhmm. You know, I've always liked the idea of, of, I guess, what you call Italian wavelengths or or buying you know, a spectrum instead of waves. Mhmm. Some of the stuff, I think, will be enabling a lot more as long as the carriers go along with it.

In that way, you can have less things to break in between everybody. Which makes things more reliable, less cards, especially right now, it would be easier.

Yeah. But, yeah, I mean, on the network side, I mean, You know, I'm excited to, you know, as as time goes on, more people gonna get into the more, you know, more of the, I guess, what I call SDN, you know, that more, more of that becomes reality. Which is, which is great. You know, when it's working, when it's stable, it's awesome. You know, as as most things, when it breaks, it's it can be a little bit harder to troubleshoot.

You know, so I'm hoping that gets a little bit better as well. I don't know who originally said it, but it was Russ White who quoted it to me, he said, automation does not mean simplicity.

I mean, it's it's it can be simple to consume, but when it breaks, yeah, it's it's tough. What about, what about hype? I have my list, but what do you see that you just sort of groan and you go, architecture? You know, I I grown a lot about, I guess I got more, like, What's the SD?

What are they calling SD? SD WAN type stuff? I complain about that for a while, but I think it's it's getting to the point where it's pretty cool and working, you know, now that the underlying network can support it more. Mhmm.

So I would normally grow in about some of the STN stuff going on there. I'm sorry, not Estan, but SD WAN. Mhmm. I know it says five different names depending on who you're talking to about it. Sassy. No. You know, it's Yep.

Yep. I mean, it is, you know, it is. It is useful in the right spot now. I don't know.

There's there's a lot. There's a lot going on. Is this always tough for me to figure out, like, what's a box? What's a service?

Well, like, it's all cloud provision. Okay. But, like, who runs a backbone? Who doesn't? What requires Like, what is basically like a mushroom multi homing box versus whatever?

And so, you know, obviously, we can't take we get to see or some all our customer. Across the customer base, we have some of everything, and we bias towards the things that have great telemetry, especially if they can peek into the the performance side But, you know, we'll work with anything that isn't closed. But, we still see a lot of confusion on the customer side about what is the actual difference and you have to, like, go talk to actual people because the marketing talks about value as as one as it does in enterprise, it talks about value and problem solved and not the technology, so it can be hard.

Yeah. I mean, what what happens is they go out there to these offices that have, you know, on let's just say on the on the SD WAN side. They go out to offices that have, you know, they have a gig AT and T, it gets took up from Verizon, both private MPLS that cost five thousand dollars each. And they say, hey, you just gotta, you know, you get a one gig from AT and T and a one gig internet from both these guys.

And, you know, my mind, your mind So, okay. How's that gonna work? Is it gonna be, you know, or it's MT, you're gonna be too small? How's the performance?

How's it gonna fail over? You know? Magic cloud. And then it just works. Yeah. And and what and I guess what I didn't think about some of the stuff, you know, is is now that the the you can do it now.

Like, you know, like, not it's it's not the way it was you know, I think with the encoding and the the way they can there's enough CPU now in these boxes to to shove it all through and if it could work properly. At there were definitely some limitations when I first came out and we would run into them pretty hard. But these days, I'm, you know, it's kinda cool. And I think it's I think it's neat, especially, you know, there's been this round of, pretty major price drops in kind of the corporate, you know, business internet circuits from the major players these days.

So, you know, you can get internet and a lot of buildings now for not what you can get it for at home, but but close. And it makes that make a lot of sense, especially with all these offices.

Are mostly empty these days. So it's a it's a nice cost it's a nice cost savings option, and it makes it easier to manage as well. So something that, something that I've been But I think it's it's definitely a practical application of, you know, automation just as number one SDN thing is probably VMware. You know?

So, deployed works pretty well, but it's not trying to do anything super complicated.

So Yeah. It took them a lot of tries. I mean, it took them a lot of tries to get it working, right, with VX line and all the, you know, all the networking and.

I'm sure, you know, I mean, they've had many versions of their SDN on the on the horizon. But, yeah, I mean, these days, it does work quite well. We do a fair amount of it as well.

Yeah. It's just, you know, Google started the idea of program everything and just like with streaming telemetry and just like with automation, it can be easy to read the marketing and think that everybody is fully automated, no network people anywhere. You know, the network self heals, everything is self driving or closed loop or whatever. And in reality, what you're talking about life cycle, you know, some of the SD WAN stuff, some of the in the data center stuff, that's where, you know, we see the real automation.

And, yeah, you still need you still need people who are involved in design and and setting it up and designing it with observability in mind. And, yeah, life is better. But, you know, still work to do, which, to me, that's exciting. Yeah.

Because, yeah, maybe if that's enough, you pop that box in all these voices and it shoots back to a biblical place like you guys to make sure their performance is there. Everybody's happy. I mean, there's a lot of there's a lot of cool. The only cool thing I've been, I've been monitoring is, you know, on the, on the hardware side and the server side for us and server deployments, you know, so we do a fair amount of, you know, medium size, you know, server deployments for our customers, you know, that, that like to build these I guess, what we call large, large private clouds, to either, you know, the a lot of our customers, you know, they basically kinda run their steady state loans to hundreds of machines.

Yeah. Yeah. So they run a kind of their studies they load in the data center. They put the burst stuff all in the clouds, which is absolutely a great way to do it.

What, you know, the challenge some of that stuff is always kind of the deployment, management, honoring of all those hardware boxes. There's lots of solutions out there that are good, but there's nothing that's that's great. And and what's happened recently is, you know, some of these hardware based solutions. There's, you know, there's now some PCR cards.

You can pop into a lot of machines that handle a lot of the security and deployment and monitoring and back end stuff It's the same type of stuff that that's in AWS and Azure. The way, you know, they do it, you know, they've done, obviously, you guys, I'm sure you've seen the the presentations on them, but they have the cards that handle all the encryption security storage, you know, just a PCI card that pops in the server, having that available to the rest of the, to the rest of us, I think it's gonna be pretty cool in the coming year. Does that add to the complexity of the system that there's abstraction from the computer perspective?

Does it make it harder to bug, or does it make it more efficient, or does it actually make it easier, you know, to design and operate? I mean, I I would hope in the end that makes it easier. I mean, yeah, right now it's a new thing, a new layer, a new system, another platform you've gotta deal with, but it does replace enough other platforms. So I think it can make it easier, you know, long term.

I've been following it too, and and we definitely have we have a few customers.

Even among our our big CDN customers, most of them don't use, you know, smart next, but, yeah, there's definitely a lot of chatter about disaggregating, aggregating hyperconverging.

Hyperconverging versus disaggregating as an architecture.

And, you know, for us, we get pitched and it's like, Computers are pretty damn fast nowadays. And, again, we're a SaaS company. We're not, you know, we have big data on the back end, but Ethan, that's fine for that. And We try to, you know, do the compute on the, near the storage at least.

So it's not on. So, You're making me think about something. So, you know, what what we've we've got two customers now doing some pretty, you know, innovative GPU stuff. Uh-huh.

Which is always entertaining. You know, for us to be in.

No. Just just high end, you know, just racks full of GPUs that are power draws in the, you know, in, in the ridiculous numbers.

Yeah, we got servers, you know, doing, you know, eight KW each just kinda entertaining. Wait. Wait. Wait.

Wait. A server doing at KW. So you're you're doing immersive cooling? Like, what's your maximum density for those cabinets?

That's a that's a good question. I think we've got we definitely have a couple in the forties that are still doing air, but they, you know, we're we're we're probably limited. I think around forty is, you know, without and we can put the the backs on the cabin to the water cooling is the easiest. Kinda handle that.

That way, you're just kinda cool to wreck as the air comes out. They're they're loud. I'm not gonna lie. I mean, there's there's some there's some turbo fans going on in there.

Man, I mean, these are all mostly available, mostly commercially available servers. You can get out there with a, but a bunch of those new, they went under wherever their GPUs are in there. But you get these boxes that have, I think, eight of those. And, yeah, they're they're they're up there.

They're they're cranking the power. But what I was getting at, though, the cool thing a lot of people are doing now, we have another, one of those guys is testing something right now that kinda disaggregates all that stuff. So disaggregating the GPUs from the storage, from the RAM event. So you can basically go and allocate, you know, this GPU and this much RAM and this much storage all over, you know, some it's probably Infiniband or something.

Some funky network It's super fast.

But that's actually really interesting to, quite a few of our customers. So I'm excited to see that stuff working. Yeah. I've seen, people been talking about it for a long time, but, you know, our peers are more SaaS companies. We don't see them using it. The infiniband, you know, there's been this promise of Infiniband switching, not not Sam per se, but all oriented at this disaggregation and virtual computers in stuff. And I guess you could put GPUs on it.

You know, but one of the issues with GPUs is always how do you get enough data in fast enough? So Yeah. Well, it's another area I look forward to learning more about. It's it's it's cool to see, cool to see the innovation.

What about Edge? Is that is that is that hot pipe or both?

You know, are your customers taking advantage?

So done right, you know, in the right place, with the right people, with the right location. It it depends how how far to the edge you're talking about, you know, so I will, tell you that I think there's plenty of applications that can take advantage of edges and metro, you know, metro areas, you know, similar, you know, following a pretty standard CDN deployment.

I I don't think going past that makes any sense anytime soon. You know, I'll I'll I will let's talk about five g. Per minute, you know, five g is supposed to take the, you know, the, the internet from being at, you know, one data center in this in each city. Or one data center in another city, you know, that Internet edge moving from that data center, that core, you know, almost out to the tower.

I I don't know if there's a huge amount of apps and stuff and, and, and, and use for that at this point. Yeah. This this could be a this obviously will be another call. And the, you know, this is one of those things where I'd look back at this video in five years, be, man, I was an idiot.

I mean, right now, I I can't think of a ton of apps that would make a huge difference being computed at, you know, that would take care of that, you know, that would that last millisecond or two But even if there's one having ten thousand cities and two hundred places per city is something that would be hard to coordinate even for our most sophisticated, you know, our most sophisticated customers. But, yeah, we'll see. We'll see. We'll definitely see where that goes.

The good part about that is cloud native architecture, being able to deploy it. One, it's a coordination and telemetry and management monitoring and state That's hard. Yep. That's nice.

At least we've got the other stuff, you know, which is running it in a standard way. It's just openstack and OVFs and all the ways that people were, you know, OVA's, OBS, whatever. You know, like, you yes. You could package up a VM, but how do you do you turn it on, turn it off, hook it to stuff.

Kubernetes is a massively complex beast, but at least with the other Kubernetes, Whether it's one location or four hundred locations in a place, we we're starting to have a link with Franco there, which I think will be beneficial, whether it's just someone running, you know, off stuff in their network at the edge or, you know, eventually letting letting people deploy third party stuff, you know, out near their near the towers. So that's pretty cool. Yeah. I mean, you can say, I mean, the one thing that's that's been doing better is kind of, like, the cloud gaming, you know, cloud gaming.

I'm not saying there's a huge amount of adoption, but, like, that's something that that's been working better. Right? You know, the with the advent of a lot of people big major areas having fiber to their home, and they they are a couple milliseconds away from these data centers now. That's starting to work.

But I don't know if that scale is ever gonna make sense to install servers compute for that in the middle of Well, it's like it's like IPV six, I think. And again, I don't wanna imply that I'm down on an edge and rip things are going again. It's it's driving architecture in a great way. But when IPV six was initially sold, it was, oh, well, it gives you security.

You don't get with IP three four. It gives you routing scalability. You don't get with IP three four. It gives you QOS.

None of those were true. And so people were like, well, what's going on here? And I think some of the first applications were like, oh, well, self driving cars need it. It's like, well, not really, but maybe something does.

And again, I I like, do I'm not going in it? I was like, ready player one or, you know, and now we've got meta. Right? You know, I mean, if we really wanted to do that, again, hard computer science problem still to solve, but, you know, we will see.

Yeah. I mean, I I guess, I mean, backing up into what we'll take advantage of it. I mean, I think some people right now are deploying code. Like, right now, I mean, what percent of sites are really optimized for Edge at all right now?

Right? Most sites are still have there's still some database somewhere in some city that you're waiting for that database to confirm something before it comes back to you no matter what it is. That's a lot to go from home. You know, EdgeML is a case because you know, people are trying to build very lightweight, composable, meaning that you can do the work distributed and and add it up.

And then, you know, you got some stuff where power is cheap at the edge and more clean. So there are some applications where maybe you don't need it, but maybe it's better for the planet to run it distributed. Yeah. You know, there again, lots of to me, it's awesome that things are changing because, you know, who wants to get bored.

Creates a lot of opportunities to, you know, start different companies. Yeah. It just application wise. Like, I mean, right now, people need to take advantage of what's even available at the moment.

Right? Like, you know, we've got, you know, a great people, you know, fly dot IO, for example, you know, they're one of the people that are doing. Enable, a lot of people do kind of stuff on the edge without doing too much work on the application, but, like, allowing that stuff to work in these new data, you know, like cockroach T. V.

For example, like, these there's a lot of technology out there that people are just starting to adopt now that allows you to do kinda creative stuff, you know, localized database that will that'll work on the edge.

And that stuff is that stuff is pretty cool. And I think just using that's all that's all available today. Right? You can go from being in one day or two day centers like a primary NDR to actually having much better application performance at the edge.

Yeah. Which, you know, with the advent today of everyone being on mobile all day, it does make a big difference. Yeah. And just getting just getting in the right city is is half the battle, I think.

The last where, you know, where I will argue is, you know, going from you know, Chicago, for example, like being at one, you know, you know, five miles away, you know, it doesn't really need does it really matter if it's three fifty of Cermac or if it's you know, ten miles closer to me. You know, that's gonna be a hard argument to win at the moment.

Right. Or it's just how would you be the management challenge of fifty per city versus just having fifty cities or two times fifty cities, you know.

So speaking about what else is cool, You're in Chicago.

And, I assume you're a promoter for fair for your fair city.

Any restaurant recommendations for people that, come to town?

Oh, that's a that's a great question. Let's start with what's your favorite place in Greek Town?

You know, I'm a traditionalist in Greektown. I mean, I really like the Greek aisles, just because it's it's big. It's been forever. They're very reliable. And I really appreciate, a place you can go to, and it's gonna be good every time.

You know, a lot of stuff has changed with COVID, you know, we lost a lot of great restaurants. We, you know, a lot of things of have, I wonder if my niece and Golly is still there at Fogo, you know, that Yeah. I mean, truthfully, you know, Chicago being, you know, the, the very publicity that we are. It's like, we're, we're definitely we're, we're just kinda getting going now. Right? Like, you know, I I was just telling you, I've only, you know, I've only been out to a couple restaurants here.

In the last few months, you know, me having a couple small kids doesn't doesn't help the situation. Yeah. But, you know, I I'm, I'm excited to get back out on the time to tell you the truth. So I don't I I think I have to do a re a retail thing. Basically, I think I need to go back out to all my favorites and address new ones.

Because I don't I don't I don't think I have a great answer for that question yet. I don't even know if queue is still there. We need to check. Well, for any listeners that are into science fiction, the world science fiction convention, is in Chicago this year, September, I think, first to the fifth.

And, we will be sampling food for any science fiction or network nerds, that come to town then. Yeah. Come join us. Yeah.

So how can people find you, Jordan, and and and, depth? Thanks. Jordan at deaf dot com. Okay.

That's that's the easiest.

Okay. Well, I am Avi friedman.

I am Avi friedman on Twitter and LinkedIn and Avi at kentech dot com. And kentech is k e n t I k.

And, Jordan is Jordan at dev dot dot com, and thanks everybody for joining for this episode of NetworkF, and Maybe, we'll, we'll have a a network community, dinner when, World Con is on this summer.

Yeah. Wrong.