Building a successful networking team in 2023

Building and operating a network has definitely changed over the last few years. It's no longer a matter of just knowing how spanning tree works or all about OSPF or how to configure a VPC on your data center network switches. Those things were all still important, but expectations have changed.

Just looking over the landscape of the industry today, we can see that network engineers very much need to be involved in public cloud and programming and developing an understanding of the nuance of how applications actually work over the network. And this makes sense to me at least because When it comes down to it, the network is the actual delivery mechanism underlying what we all care about accessing our applications and getting to our data. So the network touches everything. The need to know networking fundamentals is is still there, but so is this other host of technical knowledge just to figure out why something isn't working right in today's very complex environment.

So with me today is Tony Avantis CCIE and Principal Network Security Engineer, working very much hands on and in the trenches on a variety of networking projects. And we're gonna be talking about this in detail. What do we expect from network engineers today? And what does it mean to build a successful networking team?

My name is Philip Travasse, and this is telemetry Act.

Tony, welcome to the podcast. It's good to see you. And, well, I can see you. Most our audience cannot because this is an audio only podcast, but it is good to see you again. Last time I saw you, I think, was at Cisco Live a few years ago, no, last year. When was it? Yeah.

Yeah. No. That was, I'm looking at my calendar now. That was twenty twenty two. Right? That was in Vegas. Okay.

Right. You were there. You gave a you gave a a great, what was the name of that section of the, the World of Solutions area? You gave a talk over there. A little private thing in front of the, it's like a little projector and stuff, a little side room.

That's right. That's right. I don't remember the name of the area, but I a talk on network observability. We got into some, you know, data science stuff.

I love that stuff. It's really interesting for me to talk about right now. But now before we get into anything, I mean, I'm really excited to talk about this because it's so top of mind to talk about careers and transitions especially because I made a big transition from, you know, the traditional network engineer to what I do now, not even two years ago. So it's top of mind for me, but I'd like to hear a little bit about your background from technical perspective, what you've been doing as an engineer, what kind of engineer you are, what you're doing right now?

Yeah. I, first of all, I wanna tell you telemetry now. Every time I hear that, I can't help but thank, you know, Jerry Stiller on serenity now on on the gosh, what is the name of that episode? I think that's called the, the strike.

Is it called the Is that the name of the episode?

Or no. I'm sorry, gosh. The strike is the, is the festivus episode. Anyhow, I I just every time I hear telemetry now, I hear it in Jerry Stiller's voice.

So rest reston That's exactly what we were intending.

So I'm really glad to hear that.

Yeah. Yeah. So anyhow for me, For me, I'm actually coming up on ten years in IT.

Ten years in IT. I, I made a career change when twenty eight. And I went and got my CCNA. And that's when I left the construction field behind and started, basically, a help desk job.

And I just loved networking. So even though I was doing help desk and kinda like mundane windows management tasks, I I kept studying. I pursued my CCMP on my own and and did that and finally I got the opportunity to actually be a network engineer for, for a company as a contractor. And that was a lot of fun, and I just kept pushing and doing the the grind.

I think, you know, most people that I've met, if they if they know me, it's from my my CCI studies and and how I sort of broadcast that on YouTube and And, so so I worked my way up all the way to the to CCIE achieving that. And, in a fairly short amount time, but but it was fun. You know, I didn't feel rushed at all. And I enjoyed it.

It was stressful at times, but it was cool. But sort of at that time in my career because I got my CCI. I I become this principal in my company, and my my company is a small business. You know, so it's not like a, you know, I don't have hundreds of people underneath me or anything.

I was, you know, the lead network engineer out of, a couple dozen people who do a little bit of networking and who who have expertise in other domains.

And at that time, for about the next year or two, I kinda was I don't wanna say promoted, but I I kinda say this like jokingly when I'm when I'm having a sidebar conversation with people, but I got promoted off the keyboard.

There was actually a period of time when it was like a year and a half before I saw the command line of a router.

I just I I because I was now the principal in this role, I had to become like a mentor and a guide and I had to handle proposals. I had to be part of proposal writing teams and, like, like, all this stuff to sort of help grow the company and, and show off, you know, the best networking face the company had and network security, you know, to our clients and things. And it was like a year and a half before I touched, command line interface.

And, you know, and then COVID hit. And I had I had a fun project then, but I wasn't doing a lot of hands on sort of command line jockey. Stuff. That's what I call them dot command line, Jockeys. Comftee, you know, getting in there. Show IP interface brief.

I wasn't doing a lot of that. I was actually doing a migration of physical firewalls to virtual firewalls. So there's a lot of VMware work. We weren't doing like NSX or thing.

It was basically a one for one. We called it a P to V. Basically, a physical to virtual migration, moving the rules one to one right over. And, So that is not your traditional networking, right?

I mean, I went through the entire Cisco NetACAD all the way to CCIE, and I don't ever remember getting in VMware and configuring Palo Alto firewalls. It it wasn't part of their portfolio, but the expectation was there that, like, hey, this is our project. You need to be able to handle that. And was no problem for me.

I had some basic VMware skills. I think as you progress through your career, you're You may focus on something, but just the the the more work you do, you always get a little bit more, experience that's out of your wheelhouse. And, you know, for me, a lot of labbing for networking is done like on a VM, on VMware or something. Sure.

And so I had that experience, and it wasn't complicated, but I needed to know that. But but now, you know, now that now we're talking, I'm I'm getting up on my ten year mark. I recently started a a contract, I'm supporting agency of the DOD again, and we've built an awesome networking team, and I am surrounded by command line Jockeys, and I love it. I love sitting here.

We got an excellent team of about fifteen people varying expertise, varying backgrounds, and you look on every single one of the monitors, and there is, you know, more than one command line open.

And it's been a long time since I've seen that. You know, and that was the environment I thought I was going to be in, you know, that I was, that I was, you know, doing the certification grind to be part of. And, and finally I'm in it.

Albeit I have a little bit of a a different role, but I'm in it. I'm in the middle of it. And, And it's it's very cool. It's very cool now.

Do you think that the help desk job that you started off with many years ago now set you up to have a broader understanding of how these technologies all work together as opposed to having a very narrow focus on networking only with the blinders on, so to speak, the tunnel vision so that way you don't really see how other technologies and networking interact with each other and how they're interrelated. That make sense?

Yeah. I I'm actually I'm gonna tell you straight up.

No. That's not what I expected to hear.

I'm gonna tell you no.

I'm gonna tell you what the help desk did teach me.

I actually think no matter what you do, you're going to get those basic fundamental technological skills in your career, which is, you know, how to plug in the printer, how to ping the printer, you know, how to ping your gateway, you know, the internet's not working. Do you have DNS? Can you resolve DNS? You know, these these fundamental components of a working network whether it's a home or a business or an enterprise, there are some fundamentals.

Can you ping the gateway? Can you resolve DNS? Can you externally resolve DNS? Is there a traceroute?

These are simple. I think anyone is going to pick that up. Whether they're in help desk or anything else, but what working the help desk did prepare me for was the end user experience.

I was actually taking calls, for end users. End users that would say the we we used to joke and say, we used to take calls all the time and the the call was a a single sentence or the ticket that would come in was a single sentence. Internets broke.

That could be thousands or or, you know, I don't know, innumerable amount of things, but they they just boil it down to internet's broke. And, but but that taught me what the end user experience was. Right? Because from, from an IT professional, you know, even though I was new in my career at the time as a working help desk, you learn those fundamentals of, can you ping your gateway?

You know, and they're like, I don't know what that is. So you walk them through it on the command line, or if you have the ability to remote into their machine, you know, sometimes you can say, Hey, do you mind taking your hands off the keyboard? It okay if I remote into your machine? I'm gonna do a couple of tests.

You get to the point where, like, you're pinging Google, you know, and you're like, That that that's always the test. Right? You're you're pinging Quadate. You're pinging Google's DNS and you're like, internet's not broke.

You know? But it doesn't matter because that's not the perception of what the problem is to the end user. The perception is when they opened their web browser and they went to whatever domain, it didn't work.

You know, and so and so that's really what the help desk trained me for. So now I'm in, and since then I've been in lots of organizations and lots of teams where I am miles and many levels removed from an end user, but I still have to be aware of what that those impacts are, you know, how that end user is impacted and how, The issues that they see are reported back to us. You know, in fact, today, today, I'll give you a little behind the scenes on on my work.

We had we suffered a a a minor outage.

It wasn't a wasn't a big thing. It was a it was a minor blip. It went down for a minute. And, but a lot of things you know, a lot of messages were sent via text via email, you know, all these sort of, you know, different ways to communicate when your network is down. And And what the the network wasn't down. It was actually just a functional component of our network.

That enables, remote workers.

And because the workforce today, has to be enabled to work remote. If that in a bit if that if you're unable to do that I don't know if it's unable or enabled, but if you're If you're unable to work remotely, then that it's it's a complete outage. You know, it's considered the network is down, you know, and it's and it's what's going on. And that wasn't the case this time.

But that's the perception, you know, and so getting the rest of the network team I have. You know, they're all there saying like, I can ping out to Google and I can trace route across our network and, you know, all these things are up and and monitoring shows all these services up.

But the user's perception is the network's down because they can't get online. And so, so understanding that and and how we can build resiliency, into those components is is sort of how I'm looking at things now.

Yeah. And that makes sense. I think it's important for us in the networking industry to remember that What we do in essence is construct and maintain the delivery mechanism for the services usually in the form of applications that actual human beings are consuming on their phone, on their tablet, on their on their computer.

And so, nobody cares about the network. The network is the delivery mechanism, and it's it's only important until it doesn't work. And at the same time, it's profoundly important because it is the delivery mechanism. So here we are.

Now you Tony, you you mentioned that you just recently built a team of engineers, where you work, can you explain a little bit about what you were looking for when you were interviewing candidates?

A little bit. And first of all, I don't wanna take hundred percent of the credit or the majority of the credit for building the team. I was I was part of establishing this, this networking, this networking team. And, but You know, I I I think it I think it ranges.

It ranges a lot. So you want a well rounded team is is what you want. And and I'm so fortunate now today. That's exactly what we have.

And what I mean by that is you want a couple of seniors a couple of, I call him journeyman, you know, and and then a couple of juniors.

Okay. What's a what's a journeyman? What do you mean by that?

A journeyman would be mean, if you wanted to put like a certification equivalent, you know, I would compare it to like a CCMP. Okay. You know, someone that someone that you know, can, I did the CCMP? It was called, was it called route switch?

It was the three part test, whatever it was. Yep. And, you know, so someone who's, who's well rounded, who knows a pretty good amount of what's going on in the network, and how the network supports the other business functions. I think that's really what they try to get across at that level.

And, so I consider that sort of a journeyman. You know, you're a professional. You're getting paid to do your job. You are expected to know the fundamentals at that level.

There should be no question. You're not expected to know everything. You know, no one's touting you as the expert.

So, you know, so it's okay to keep studying. It's okay to ask questions. It's okay to learn. You know, it's in fact it's okay to do that at at all phases and levels.

Sure. Yeah.

But it, but it's expected, you know. And the junior people, they're expected to have those to be exposed to the fundamentals.

They're they should be learning those fundamentals.

Some of them might already have them. Maybe they they have a good grasp of fundamentally. It's always a a weird thing getting into the the start of someone's career because they may have like really good, a book knowledge and the ability to pass a certification exam, but if they don't have any experience, even a year or two at a help desk on there, they might not cut it as a junior engineer for for some teams.

But that's generally sort of how we break it down.

And, and it's not just in networking. We also need the network team to have a wide range, of domain experience across networking. Right? We gotta have, a UC person.

We actually have someone on the team. I'm gonna give them a call out right now as UC Rob. We have many robs And, on day one, when he showed up, when I met him, he realized there were multiple robs. And he was like, I'm UC Rob.

There you go.

And so he named himself, and now he wishes he didn't.

But but there's, almost everything we do has a component of security.

And I love let me give you a little peek inside my mind. I love network security. I have always had one foot in networking and one foot in security. Now security is a huge domain. It covers hosts, it covers containers, it covers all kinds of stuff. And and the kind of security that I like is network security. I really enjoy proxies, firewalls, deep packet inspection.

I, I like all the cool things that are coming out or that that have been out now with, Sassy products and things that really love being able to leverage the network as part of the security, architecture across the entire thing. So anyhow, so many network engineers that I've seen in my professional career might be great at routing and switching or VoIP or something else. But they don't sort of understand, how like stateful firewalls work. Sure. Or or how a particular brand of firewall treats packets differently than another particular brand.

Which is especially important if you're in UC.

Yeah.

So while we're a networking shop, we expect most people to have some fundamental understanding of the firewalls, across the network. And Maybe that's just a team specific thing. And and I think it very much is. I've I've talked with other departments and and, other places and other team members who have worked elsewhere where sometimes there's a firewall team.

Yeah. And that's what they do. They they manage either the rules or the devices themselves or or whatever it is. And for us, maybe we hope to spin off this firewall rule responsibility onto someone else But at the moment, there's no better team that understands the flow of the network better than the network team. So we sort of take that responsibility on our team because we want to make sure when the firewall rules are entered in the device that they're accounting for directional flow, they're accounting for what kind of traffic, dynamic ports, you know, there's a lot that really goes into, to something as simple as a firewall rule. And I know that the network team that I help support has a good grasp of that. And when we bring people into the team, when we hire new people, we're looking for that blend, you know, you gotta be a good command line jockey.

As well as, as have not just a routing and switching background or, you know, or if so, we're gonna expose you to that, and I want you to be perceptive and, and, you know, and enjoy that as well, you know, enjoy having multiple domains under your responsibility.

So, So, yeah, I mean, just a well blended team, a blended team of experience and a blended team of skills and, and, and knowledge domains. Know, sort of going like vertically and horizontally. Yeah.

So can you explain why you organized your team in this way? Is it, to have a more efficient workflow based on skill level or specialty areas?

You know, I I'm not sure.

I'm not sure, but it certainly has its advantages. Right? There are definitely some tasks that, that as a, as a team or as an organization, you wanna give to your senior people because, maybe they've seen it before, maybe maybe you just have that trust in them, and you, and you know that the task will get carried through from a to z, or or the expectation is that someone, you know, wouldn't have that that domain of knowledge at a junior level. And so, and so it misses them.

But by having the team sort of spread out again, sort of, you know, vertically in in experience and and horizontally across knowledge domains, you know, allows us to to move those tasks around the team.

Mhmm.

You know, we gotta do a bunch of firmware upgrades and stuff like that. You know, hey, almost anyone can do that. We can show you how. We'll give you a five minute demonstration and upload this thing, you know, schedule it for for for an outage and and hit the reboot button. And, while other times we're doing, you know, circuit engineering and working with the ISPs and, and, and and and different things like that. So, yeah, so we do pass those tasks around and sort of assign them that way. And it just works out great.

I mean, mean, you have your subject matter experts that are gonna do some very advanced troubleshooting that a CCNA would likely have never seen before or doesn't understand how a BGP adjacency even is formed in the first place.

Right? So you're gonna assign that, by default to a more senior person, whether they have a CCI or not, by the way. I know that the certification may not be there officially.

But, likewise, you probably wouldn't say, alright, we have to rename, all of these interfaces. And then you have, like, the most senior CCIE owned staff, multiple CCIEs. Let's do it that way. And this is your job for the next three weeks just to manually maybe maybe even make a script and make it a little bit easier, but nevertheless super mundane. I can't imagine that you did assign that to to that person.

Or maybe maybe maybe you're all that humble. I'm willing to get the job done.

Yeah.

If you if you wanna ask me, like, my my absolute most humble opinion is like I'll take the interface renaming job because I wanna script it out. There you go.

Make it a project.

Because You know what? I don't care if it takes me all day to write the script to cover every interface in the entire enterprise.

I'll do it because, because I find that challenging. Yeah. For me, it's just a personal growth thing. I've really been embracing, scripting and automation lately. I've really done a lot more than I would say dip my toes in. I I'd say I'm in the swimming pool when I'm I'm treading in the deep end. I'm not an expert.

By no means, I'm learning every day, but there has been nothing I can't accomplish yet.

With some Python chops. Like, there's no task. I haven't been able to, to sort out, with some rudimentary Python chops. I'm I think I'm doing okay.

Yeah.

I I might be at that, at that peak, where where I come down to reality here soon, but Yeah.

But, but I feel pretty good. And so what I'm trying to do at my role, at least within the team and the organization, I always wanted to be surrounded by command line jockeys. Right? People just like, configs on every screen and, you know, everyone working together and collabbing at whiteboards everywhere with stables and and all that kinds of that was that was what I thought I was I was going to be handed when I became a CCIE, and instead I got taken off of the keyboard.

And, it's so exciting to be back in that environment because now at least where I am in my career, maybe it's just my personal age or the experience that I've gathered in my career, but I'm starting to mature, in my, in my job. And so some of the things that I'm trying to help the networking team with isn't isn't all the time solving the most complex problem. Okay. You know, don't, they don't bring all their most complex problems to me.

They'll bring some. And some I will take. And I'll say, I'm working that. Yeah. And and it's just for fun, you know, not for fun.

It's because, you know, we we have a job to do.

No. It's totally for fun, Tony.

But I do have a fire like in one thing. I've seen this thread, being kinda woven through a lot of what you said that you enjoy the puzzle. And I think a lot of us engineers in this field, network engineers, previous network engineer now really enjoyed the puzzle of figuring something out. And then the satisfaction of having figured it out.

Right? One hundred percent. Yeah. Yeah.

It's it's getting to walk away, with the trophy that you gave yourself because you chose the task.

But, but yeah, but sort of where I am now is a little bit more mature, in my professional career and in my personal life. And so I have a great team of network engineers around me of great, varying levels of, experience and across a wide, domain of of knowledge. And so what I'm trying to do is bring innovation.

I'm, almost everyone that we have on the team isn't excellent command line, Jackie, whether they like that term or not.

A lot of them will take on a task like, hey, I gotta rename these interfaces, and they'll just copy and paste everything in the notepad, do a find and replace and blah blah blah and paste it back in. But I I'm trying to look at things a little bit differently and say like, Like, I I know you can do that. That's great. If we wanna get the job done, go ahead.

But but let's figure out how we can script this. You know, let's, and let's not just develop a, a single shot script that we put in a folder and go, this is the rename interface script for this device. Right? That can't really be reused. We can check the code, but then we have to change it for another device. Like, let's not just write the scripts, but let's sort of develop things in the most modular way and the most reusable way possible so that once we have the boiler plate done, everything else just goes on top, like whatever your job is goes on top. And there's lots of frameworks out there that sort of do that already.

But just getting people's mind set up for that. You know, gosh, there was a thing today. I I I think I told you about a little network little network outage and One of the things was, a device that needs to be rebooted.

And we were kinda like joking in our in our network chat channel.

That Tony is probably thinking about how to script this reboot process.

I'm like, I have a cron job ready for you.

I'm ready.

You know what I mean? And but that's just that's that's what I get to, that's how I get to give back.

Is, is trying to broaden everybody's, way of thinking, way of thinking about these as different problems. Yeah.

Something recently is we have a new team and a new contract in a in a in an existing organization. So we got a lot of people that are coming in blind. Come I say coming in blind, but I mean, they haven't seen this network before. And, and the the organization doesn't have like a ready tool set, like a networking tool set that's ready for us, you know. So we have to kind of figure out how to make the things that are existing on our provisioned laptops work for us.

And Let me start with the most simplest thing as like SSH access to a network device.

That that is basic. Yeah.

This is the fundamental way to access a networking device.

I don't even think books today even mentioned Telnet. I think it's some devices. It's not even an SSH is the bottom line default for some devices.

My point is is is this is even challenging because we don't have things like secure CRT. You know, like, that needs to be bought as part of the organization, and that was never purchased. So Windows ten comes with open SSH if you include that module in your, in your baseline image. And so we're having to do everything through the Windows command line. Via SSH.

And so one of the things I started doing was creating these SSH config files that allow me to automatically jump through jump servers using key based authentication. Right? So I'm trying to remove as much friction as possible so that the team of network engineers, the good people I have can do their jobs more easily. And that's sort of how I look at this problem now.

Right? I'm not just solving some network engineering problems. I am. That's fun, but I'm also solving some process problems.

Yeah. Some team based problems, some organization problems, and I love coming up with those solutions because That stuff they don't teach you in CC and A, CC and P, CCI. You know, that that is really just just from being around. Just from being around shops like this and and sort of understanding how these systems interwork and what can we do to make it easier for ourselves.

And I'm enjoying the hell out of it, man. I'm enjoying the hell out of it.

Yeah. Yeah.

Well, you're you're enjoying figuring out how to solve the problem. And, that's I in our at the very beginning, you mentioned how you changed careers in your late twenties. I did as well. Maybe for different reasons, but I changed careers looking among other things for a larger paycheck, but looking for also a career that I can sink my teeth into, and that was a never ending challenge, a never ending project.

There may be other ones out there that would have fit me, but falling into technology and then very shortly thereafter networking, I've perfectly fit the bill because it truly is a never ending project. But I think, you know, what you're talking about there is the maturity of having worked in environments, having dealt with these devices but also just being a more mature human being and understanding how people work together and how, what can I do to remove the friction between my coworker and the devices that they support that makes that makes their job easier? And and I I assume they'll be more successful then. Right? And the the network will be more stable. You can even take it to that extent. Right?

Yeah. I mean, it's it's not, that example I laid out is not the end goal, is not the the perfect picture of how we want to do things, but it it it is reducing enough friction that we don't have to get frustrated while we wait for the, what do you call it? The the perch of a of a proper, you know, multiple SSH manager. Yeah.

So and that was the thing.

I remember Oh, actually this was in a job interview, where they asked me some question about managing some devices and what would you do here? And I gave them this this spiel about how I believe that, you know, we're engineers. And so we're trying to find a solution to the problem. Like I've been saying for the past few minutes, and how you don't necessarily need to just rip and replace all your gear.

How do how do the protocols work? How do these devices interact with each other? How can I make this work for the organization in the constraints, the business constraints that I have? Maybe the business constraint is we can't replace this gear.

Or I can't shift traffic from here to here for whatever reason. Security possibly. And so therein lies the puzzle, but they're in all lies the, the, the, the, the engineering component that I really enjoyed, just figuring out how do I make this work with what I have. Sounds like that that's kind of an important thing for any engineer applying to your team to have if they're showing up to a job interview.

I mean, they don't necessarily have to have the the terminal degrees from MIT, from PhD and computer science from MIT or or multiple CCIEs.

But they need, they need to have that drive to wanna solve the problem. Maybe the curiosity I don't know. And that's not that's probably not right.

Yeah. Yeah. Definitely. Definitely. That I mean, just what you described, I'm just like smiling over here as you were describing it because, like, that is the best part of some of the network challenges that I have. It is not, what is some what is some nerd knob that I can turn on some protocol, you know, hey, hey, come over here, check this thing out, look what I can make OSPF do. It's not that.

It's because in a in a wide open network, you know, any solution can be used for anything, but it's these constrained environments. These environments where things either can't change or won't change or or or, you know, these are these are the sort of roadblocks and still making the work, excuse me, the network work.

Yeah.

Work with, resiliency.

Work with high availability, you know, being able to provide, a redundancy plan for the network in all of these constraints.

That's the engineering challenge. Yeah. It's it's not a greenfield solution.

So, you know, you know, one of the things that I'm getting ready to face in the next couple of days is I have to talk about the IPV six deployment plan Okay.

For this agency. And I am so excited to be doing this.

To be honest, if someone asked me to do it two years ago, I would I would have been extremely scared. I'm like, I am so ready for it because I realize, it it's not actually going to be that difficult.

I've gotten over the monster. That is the IPV six addressing. Right? Just just the fact of of seeing letters and numbers together. Just was like, you know, I I'm used to seeing things, like, in in, you know, numeric characters only.

But I'm, but I'm so excited about this because even though the underlying plumbing of the network is not changing at all, it is almost like a greenfield deployment. You know, because the IPV six and IPV four are like ships in the night.

They don't have to interact with each other. They can occupy the same cables. And so we can actually build kind of a, a new network if we want to. And, and it's just so exciting to have that flexibility because I can't move the IPV four space that's there now find the network and these buildings and these lands and everything else, you know, but we can do whatever we want with IPB six. And, and it's just super cool to kind of you know, again, there are business constraints. There's a lot of constraints as well as there's an existing network that we have to make it work, but I get to lay down an IPV six addressing schema across the country.

It's not greenfield, but I'm calling it greenfield, you know, because there is no existing IPV six There isn't one.

So we get to do it, you know, for the first time, and I'm so excited about it.

So That's awesome.

It's been a while since I've been, I use the term field engineer sometimes, but just an engineer outworking at the command line and building networks and fixing networks in a while.

It hasn't been it's been more than the two years that I've been working in, network observability because prior to that, I was a solutions architect for a few years. And my experience in the field, quote unquote, air quotes was in, you know, a lab learning Viptella when they were purchased by Cisco or SD Access when that came out and and that kind of stuff. So still very cool, but it's been a while since I was in a cutover that went south.

And I'm literally sweating from my, you know, on my forehead and ignoring the customer's phone calls because they're just every ten minutes like, when will this be up? I'm like, well, if I knew that, then I would that implies I know the answer and it would be up. And then eventually you figure out what's going and the pings go through and you see all the exclamation marks. And I I remember even calling my wife and saying, it's working.

I also remember calling my wife and saying, I'm quitting, you know, Both of those things were true on the same night, and during cutovers. I'm I missed those days a lot on one hand. I don't miss doing cutovers at two in the morning. I've worked bankers hours now.

But I missed that a lot because I missed that puzzle, and figuring stuff out. Do you think that if somebody came to you looking for a job, right? You have a job opening. You got a wreck open and you're interviewing people, and you could smell that they you could kind of sniff out that if you had an IPV six project coming down the road, and it's not on their resume, super familiar.

And you could tell that they weren't ambitious enough to go out and learn it because there was an upcoming project. They probably wouldn't be a good candidate. Right?

You could say no. It's okay.

Yeah.

You know, uh-uh, there isn't one I'm gonna tell you there isn't one thing that makes any candidate a good candidate or a bad candidate. Okay.

That's fair. Yeah. Maybe I'll look at things a little bit too.

Yeah. I'm I'm sorry I have to be that way, but but I'm not sure. They could be they could be dynamite in so many other ways. But, but you know what, I I I have a I have a really good team, and I'm I'm pretty sure most of them are not hundred percent comfortable with IPV six.

And in fact, I'm pretty sure some of them, might be a little bit scared.

I'm not gonna name names, but Right?

Yeah. But I, on the other hand, I, you know, and maybe it's like false confidence You know, sometimes it's hard to tell the difference, but, but I'm excited for the challenges that we will uncover. Because that does I'm not implying that I have the answer to everything and I'm gonna know I'm gonna do this right and I'm gonna be like a sniper and get everything in one shot.

Yeah.

No. But what I am confident in is we are gonna uncover some challenges, and we're gonna work through them, and we're gonna solve them. And, and I think that's what that's what makes us as a team. Phil, I I wanted to I wanted to to tell you when you were just talking about that cut over. I just told you that once I got my CCI, I didn't touch a command line for, for about a year and a half. I was doing proposals and, business development kind of stuff, and, you know, doing road shows and things.

And, two weeks ago, I did my first, like, command line cut over.

It was so exciting. Dude, it was so exciting.

How did it go? It was a disaster. Oh, my god.

No, listen. I I was doing this cut over and, the team lead now for the team, the network team that, that I'm a part of. He is actually someone that I brought in as an intern, like seven years ago, He was an intern to me to to my team.

And, and now he's the networking team lead. Very experienced and knows this environment and and just it couldn't be a better person for the job. But because he knew I was doing the cut over, he was like, Alright. I'm gonna stay tonight, you know, and so it was just gonna be me. I was gonna do it by myself.

Yep.

And, and it was a router swap. We were going ASR one thousand six an ASR one thousand and six x. Yep. So a little small router, not a lot of interfaces, but, but still, you know, I Yeah.

So he tells, so I'm gonna start it at six pm eastern. And, and he's like, yeah, I'll hang out tonight. I'll give you a hand, you know, whatever you need. And I'll be, hey, that'll be awesome.

He and I just had a blast because when he was an intern for me when I brought him in as a junior engineer, we did a hundred cut overs.

We did I was the lead at the time, and we did a hundred cut overs. And there's just some Some like there's just some camaraderie that happens when you stay up all night with somebody. And and you two are just working a problem. Or two people or what however many how many is on your team, but where you and your team, you become so in sync with what the other people are doing that you're like you're just talking over cubicles because the rest of the office is empty because it's nighttime.

So you're just talking over cubicles and you guys know exactly what the other person needs as far as information. I know exactly what you need. I'm gonna show you this. It's on this VLAN.

It's this subnet here. I'll send it right over blah blah blah. And it was just like we were just like falling into an old rhythm. And we got those pulse checks from the customer.

Any update guys? Mhmm. You know, I was I need thirty more minutes, please.

Right. And And then finally, you know, you know, it would we're we're into it for a few hours and I I was I was coming up on the the the boundary of my change window. And, and we got the pulse check. You know, what's the how much more time guys?

It's getting pretty late over here. And I'm like, I'm not answering I can and I I I'm gonna answer him. And he's like, I'll answer him on my hands. We need some more time guys.

Yeah. Anyhow, it was successful. We had a blast. I mean, when it was done, it was just like falling into old times.

And and I think that's like it's I think it might be just unique to, to network engineering. I mean, I I'm sure other industries have it, but when I talk about it with another network engineer, there's some camaraderie there even amongst strangers that you just you know what the other people have gone through, you know, what they're dealing with. And, And and and really I love it. I mean, I'm a little bit older now.

Staying up late is sometimes painful.

And I'm glad I don't have to do it all the time, but, But I'm glad that when I do, I'm not doing it because, like, I was assigned this. I was doing this to, as a proof of concept for this kind of architecture change that we were doing so that I can map it out and hand it down to that networking team and those other engineers and sort of, here's the playbook. You know what I mean? This is what you need to do.

Go ahead and run this. And and it was so much fun. So so, yeah, man. I don't I don't know, guys.

I I I'm glad I don't have to do a lot of them, but but I'm glad I have a long history of cut overs there so much fun.

Oh, absolutely. I think that for me, the most meaningful times in my career as a network engineer out in the field were those cut overs when Sometimes things went south and I had to figure it out and then, of course, the sense of accomplishment that you did that the technical lessons that you learned when you figure out a technical issue you take that with you into the rest of your the rest of your career for sure. But also when things go south and and you can't figure it and the slice of humble pie that you have to eat to roll things back, you really don't want to. And your customer doesn't want to. Believe me. And, you know, you have to explain to your customer what happened.

But I think I was in that position as of our engineer, working solo almost the entire time in my career because I had an understanding about their technology whether that's VMware or navigating a Linux file system or troubleshooting wireless or how active directory structure works. You know, we're pulling that into the wireless system. Right? So if you're troubleshooting, a core cut over not going well, you might need to know all those other technologies as well. And so it gave me a lot of advantage and and I think propelled me in my career.

Even though I was focused on a particular network cut over, having that other knowledge and those skills, I think really gave me an edge, very much. So I have to imagine then for an entire network team, you wanna have that variety of skill set because of the reality of what it means to do networking tasks today and then touching all these other technologies. In order to be successful, at least, And so, you know, for me, it was I was always alone because I was of our engineer, but if I had an entire team, I'd have my VM and my cloud specialist and my security specialist, all, you know, networking, but with all those special skills as well because of the reality of what networking does. It touches everything.

Yeah. Yeah. Touching everything. I mean, like, like, mentioned about the the team I'm working with now, we have, a couple of cloud projects, and it's more than just like, standing up a tunnel to a cloud environment. It's it's way more than that.

You know, it's it's orchestrating, whole, network, address management deployments in cloud environments, and they don't even really exist. You know what I mean? They're they don't exist yet. They're they're deployable, or or BGP BGP, I was gonna say BGP routes.

BGP peers that aren't routers. They're just like software you turn on, and now you get BGP routes. And wait a second. What what routes are these?

And there's a whole different sort of cloud networking component, and I'm not an expert in that. I'm I'm very much just have one ear turn to my coworkers who are in deep to that. So so I just hear some of those struggles, but that's all within our team. Right?

The cloud networking team.

Securities within our team.

So much we have to touch. And I don't know if that's the right answer. What I will tell you is we knew a lot of these challenges going in, and that's why we built the team the way we did, with, with a wide domain of, of different skill sets and different experience levels.

But I know there's a, there's like a common saying, which is a lot of times the network has to do what the applications can't.

And one of those things, right? Like you just mentioned, know, the developers, they write an application, and then then they're done, you know, and and that's right because a lot of the development teams. Well, I'm just painting with a broad brush. Maybe I shouldn't, but I have heard some developers, firsthand, that when they write an application, when they wanna make a network call, they're just calling the operating system and just saying, Hey, open a socket.

You know, or, hey, I need to make a network call. Just let me know when it's ready, you know, sort of from a software developer standpoint.

You know, and they're not maybe taking into account the firewall rules that need to happen and the things like that. So Again, they're they're developing a tool for something. I don't know for whatever accounting or something. Sure. But they're leaving the security part to the network so that so they're they're putting that on to us. And and I've heard so many discussions recently on, you know, in different, webinars and in different companies and different things about doing like devsec ops and and trying to bring the security back into the actual development of the application.

And about how challenging that can be, especially for things that are already written.

You know, you can't, like, unwrite the code to put security in Right?

So we're still having to enforce it at the network layer. And I'm not saying that network secure security at the network layer. Network security needs to go away.

But but it should be changed, right? And and and the developers need to do maybe more than just, hey, telling the operating system, hey, open a sock me, let me know when it's ready. I'll give you some data, you know, and just letting it, pipe it over the network and leaving network security to something else.

But that's the way it is today.

And so, so the, the team of people I have, they're all really well rounded, each with their own sort of, expertise. And, it's been really awesome.

Do you care about, seeing certifications or college degrees or anything like that on resumes when you're when you're talking to folks?

I'd say in in my industry. Mhmm.

And I'm getting if I call networking my industry.

I don't see a ton of value in college degrees. Okay? But let me back that up. I'm not gonna say the networking industry as a whole. What I will say is network operations, network administration, applied network security in college degrees because they don't teach the kind of challenges that you're going to see on the battlefield. So to speak. Experiences came.

Right? And, you know, everyone knows you gotta have a personal brand. If you got a blog, if you gotta get you get it whatever, and and you got stuff worth sharing. Share it. That's your personal brand. Put that on your resume. I wanna see the blog about your, the BGP changes you made on your last cutover or, or whatever, you know.

That, that adds a ton of value. That lets me know more about your experience. So experience is king.

Certifications are, I would say, second in line of importance, and this is just my opinion. But because I've been through the CERT program, I know what it can do for you when you apply yourself.

You can learn a lot.

Yeah.

I have learned a lot. I've probably forgotten more than I've learned. But there is a ton of value in, in actually getting the certifications, as someone who's gonna do the interviewing or the hiring, there are a lot of, there are a lot of people out there who try to game the certification system.

Either through cheating or dumps or or whatever, whatever that case whatever the case may be. And it's it's your your job to suss that out. So don't put all of your, you know, don't put all of your weight into someone's, certifications they present on their resume or that they talk about, right? It's your job to suss out whether or not they have the experience to back it up. And I think that's the bottom line.

Experiences King, certifications help, but ultimately experience needs to back up what you have.

So then if if one is trying to build a successful networking team, if you are trying to build a successful networking team, are you looking for candidates that have that broader knowledge outside of networking as well that know something about public cloud that know something about Kubernetes that know something about, something deeper than the lay person of network security?

Yeah. Absolutely. Although it's it's harder to find those people than it is your, your focused network engineer.

Is it, you know, someone that knows network It's much harder to know some to find someone who knows Kubernetes and, Cisco and Juniper, you know, normally, they they sort of play in two different playgrounds.

But you would get the best of both words. I mean, if you find those people, absolutely. One of the things that I'm pushing for now is I want all of the engineers on our team to learn some automation and to start to understand what automation can do for you personally, us as a team, and the organization as a whole, which is how, how us automating the network can move the organization and the other teams that support the organization forward.

Yeah.

And, that's That's challenging right now as well. I know there's a huge, support through, network engineers getting into answerable in Python and everything. And actually when I started my CCI, at that time, I saw a split in the industry. Oh, yeah. About half the industry went to automation and half stayed on the CC. The the the certification route.

I went on the grind.

And so I abandoned abandoned automation, and I didn't pick up automation seriously until the last two years. And now I'm like, I don't even want to touch the command line anymore. I just want to do everything through some sort of API. So if you have those skills you are a threat.

You are an amazing threat. If you are a a decent network engineer with experience to back it up, and you can code or script even just a little bit, you know, but enough to make it, to make it useful in your field. That's amazing. That's amazing.

I think I'm just getting to that point where like, I can start to make useful products for other people in my team to use pushing out simple scripts checking different things. Yeah. Go do the config backups for all the devices in the network. You know, just just those simple things are starting to become valuable or are valuable and we're just starting to to make use of them.

Phil, I wanted to I I don't mean to derail you, but I I wanted to I just had a thought about something that I mentioned earlier, which was maturity.

Mhmm. Sure.

And I mentioned I mentioned I had a little bit of professional maturity going on.

And how I always sort of envisioned myself as this, network engineer and this awesome team with all these screens around and all these command line prompts and the terminals blinking and whiteboards with markers everywhere and everything. And and and that's all very citing, and and it still is. You know, I I talked about how I enjoyed my cut over a couple of weeks ago.

But I'm also looking at the network Well, I guess let's just start out by saying, at that time, when I was coming up and and even now, even sometimes now, you you wanna show off the network like it's this pet that you built. You know, I know there's an argument of like pet versus cattle, you know, in the in the Kubernetes world. Sure.

Yeah.

But this pet that you built, and you wanna show people how, you know, all your access lists are all capital letters with underscores, no hyphen. You know, and how you adhere to all these interface description standards and and how all these things and, you know, your your protocols are written just this way and it's an advantage for for your topology because it makes this easier and and you wanna be able to show people that and show off the network and like come over here and look at this and this is cool and this is why it cool and blah blah blah.

And I'm mature now and I realized no one gives a shit.

Nobody cares.

No one. No one. And and here and here's what I here's what I just covered through my my sort of maturity, my personal maturity professional maturity is we should actually be building networks that are invisible.

The more invisible the network, the better. And what I mean is I think you started our talk by saying this like, you're a user, you're using an application, or you're accessing an application through the net you actually don't give a crap about the network. And you know what?

It shouldn't matter.

It should be the cleanest, fastest pipes possible to get the job done, and it shouldn't matter. It shouldn't matter to the user, and it shouldn't matter to the application.

Yet a lot of times, I see us sort of compensating for different network designs or different applications and sort of doing so with the network and, and trying to achieve this, this beautiful topology, this topology that's like a perfect tree, you know, of, of having some sort of root and branching out and branching out, you know, and you, and you get to the edges of the network in.

I used to think that I wanted to build these perfect networks that that the configs were pristine. You know, I think I always hear Greg and, talk about the artisanal command line, you know, your prompt that you, you know, made by my artisanal fingers. And You know, I used to think that was a thing, you know, and that, you know, when you were a mature good network engineer, you'll get to that point, and I realized That's that's so stupid. That's totally not the case. The the realism is Simple is king. Mhmm.

Yeah.

In network design. Simple is king, and the network doesn't matter.

My point is is I have a team of fifteen people now to operate a network. The goal would be in a number of years to get that down to Three?

Four people. Really?

You know what I mean? Yeah. If the if we can simplify the network, the architecture Mhmm. Simplify the touch points and the interfaces.

And I don't mean the interfaces like you do interface gigabit zero slash zero. I mean, the interfaces between devices and applications in the network. The the parts were too unlike devices touch. We simplify that.

They need less maintenance.

Sure. Right.

Less administration, less operational changes simple is king, and the network shouldn't be seen.

That's where I'm trying to get to now. So I have a very complex network that me and my team members work on, and it's a lot of fun. And we have a blast.

But ultimately, I I'm trying to use the knowledge I've gained through other networks and some things I've learned from other people. And, also through certifications and things like that, and using everything to my advantage not to build complexity into the network. Sure. But to remove it. Yeah. And And and that's like when I had that realization, I realized, like, oh, crap. I'm a different engineer now.

Because there was a time when I was trying to put complexity into the network to show off what the network can do. Yeah.

And that doesn't matter. So, yes, it was a big growing point and, and I'm happy to say that making things simple is easier than making them complex.

So it's the right decision.

At the application's expense, Right? It's true. At at or maybe I'll make it broader, making the network more complex at the expensive service delivery. So that's the point is to deliver that application or whatever service it is.

I, you know, when I was a when I was a engineer, I would go into sometimes I'd go into a customer's network, and you can tell that the person running the network, they're studying for the CCI. Because they are implementing or like all these tunneling protocols and you're like, why why are you doing this stuff? To be fair, I was also studying for the CCIE. So I'm like, alright, we're gonna get our hands dirty here.

But I mean, we know better now. We know that that adding complexity actually causes problems. Not only is it a waste of time and adding complexity that you have to sift through later on, it actually increases the likelihood of of a failure in the whole point of the network, which is delivering the service or the application to a human being.

And, you know, the reality is it's not even It's not even the the application that we care about. Right? Like when I go to my bank's app on my phone, I don't actually care about the app either. Not only do I not care about the network I don't also care about the app.

I care about managing my money, the actual task that the application is enabling me to do. And so in building a a successful network team, I have to imagine that that's a very important component that understanding that, hey, what we're doing here is is all about delivering an application to a person. And so finding ways to simplify the network, not adding complexity, at the cost of the cool factor, of course, should be paramount in any engineer's mind as their as their constructing a network and also managing the day to day finding solutions to problem. Or how can we continue over the course of time to simplify the system that we manage and thereby increase its reliability.

Yeah. Another way, another way to phrase it to sort of what you were saying and and what I was just talking about previously was, we shouldn't build networks just to build networks.

We build networks to connect people to data.

I mean, that's as simple as I can put it where I I think sometimes and my maybe my younger years, and I think I'm guilty of probably practicing my CCIE on a few production networks. I'm guilty. Okay? Sorry. But I was building a network to build a network.

Yeah. Yeah.

You know, I was doing it to do it. And, and and now I realized that that network should be transparent. Sure. It's about delivering data to people, or data, or data to other data, you know, a system to system. That doesn't have to be a human in it, but, but, but yeah, that's That's what it's really about.

Yeah. Absolutely. But to be fair, sometimes, there is inherent complexity in achieving a particular goal.

So, you know, when we're trying to make managing the network, easier like our WAN, for example. And then so we start looking at technologies like some some vendor of SD WAN. And under the hood, there's complexity. There's overlays, maybe they're using VXLAN as a control plane or something else. Who knows?

I I do believe that there is these are tools. Right? But it again, it's it's using the it's using the correct tool to solve a specific problem and not just throwing my entire tool back at the problem and saying, I got this new saw. What can I cut? You know? Or I got this new hammer. What I'm looking for nails everywhere.

But sometimes you do need a saw and sometimes you need a hammer and sometimes you do need to implement a complex technology to solve a problem. So it's not it's not that we don't like complex I think it's that we don't like complexity for complexity's sake. Right?

I mean, our when it's unnecessary.

Yeah. Our networks are complex by virtue of the fact that they're distributed and we have public cloud and containers, networks we own, networks we don't own, networks that we don't manage, that we do, and then, of course, how do we secure all of that? There's a lot of stuff going on inherently complex whether we want it or not?

I I forget what the what the the the saying is, or or maybe it's a quote, and I'm not sure who it's from. But it's where, when there's nothing left to remove, It's it's not when there's not nothing left to add. It's when there's nothing left to remove.

You know, and what that means is is that's you have whatever whatever you're building is as simple as it can be. It has only the components that are necessary.

You know, Tony, I was, I was googling that while you were speaking. The, the direct quote is perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

And, yes, I agree with that. Now, Tony, I do have to say before we close out that I was following you when you were looking at the results from your CCIE exam. And I remember even now as I'm talking to you, I remember very clearly the look on your face, the the hand gestures, and the facial expression that you had when you opened up the results when you passed. Think you were streaming it, live, I think. Right? Mhmm.

I remember that, and you were just like you can't see this. This is audio only at which the audio could see it, but you were you were shocked in awe. I were your kids there? Were your kids around?

Yep. Yep. My kids I my kids, I I always brought them in for the reveal. I don't know why I chose to do that, but, you know, through throughout the CCIE studies, it's it's kind of common knowledge, or I don't know, a commonly talked about thing that, you know, your entire family pays the price, for you study for you to achieve your CCI.

And so in doing so, I wanted my entire family to be part of the the moment, the the of success. You know, of the joy of success. And, so I brought them in even on the first attempt when I was pretty confident I was a zero, but but I ended up walking away with, I think one w there is where they call it pass. So so you need four passes to pass, and I had one on that first one.

But, yeah, the last one, yep, my kids were there.

I remember them. Very cool. It was great. Yeah. So, Tony, I think this is a good place for us to end here.

And, I I just wanna say thank you for joining me today. It's it really has been a pleasure. Talk to you engineer to engineer and swap war stories about cutovers and and the lessons learned from over the years about about being a network engineer. And and also to get your thoughts about what it means to be a a network engineer moving into twenty twenty three and and what it is to build a successful net working team.

That's that's really been the the underlying theme of today's podcast that we wandered all over the place, and I I love that. So again, Tony, thank you much appreciated. Yeah.

Thanks thanks for having me.

So, Tony, if, folks wanna reach out to you with a question or comment, how can they find you online?

Yep. You can find me on Twitter. I'm sure IP interface brief. And that's probably the best way. Look for me there.

Great. And you can find me on Twitter as well. Network underscore fill. Still very active there. And you can also search my name in LinkedIn, which I also am pretty active in these days.

Now, if you have an idea for an episode or if you'd like to be a guest on the show, please reach out to us at telemetry now at kentick dot com. And you can also follow telemetry now on both Twitter and, and on LinkedIn. So until next time, thanks very much. Bye bye.