Summary
After 20+ years in the network management sector, spanning both enterprises and service providers, you like to think that you’ve seen it all. So by the time I was first exposed to what Kentik was doing, I was pretty sure that I understood every angle and approach that is or could be taken to establish network/traffic visibility. Frankly, my first impression was that Kentik didn’t sound all that new or different. But the more I learned, the more I realized that Kentik truly was unique.
Network management tools and technologies have evolved over time, but always revolve around two key objectives: visibility and control. While control is a worthy topic in itself (I’ll come back to it in future posts), there is a long, varied, many-shaded story to tell around network visibility. There are many, many ways to establish visibility into the network, and a lot of compelling benefits in terms of operational resilience.
As an analyst for six years at EMA, I conducted periodic research studies on Megatrends in Network Management, poking and probing to see if and how network management tools, technologies, and practices were evolving. And while almost everyone with whom I talked used monitoring tools — ranging from basic SNMP to logs to NetFlow/xFlow to real-time packet inspection — there was a glaring gap in the adoption of application/traffic visibility by front line network operations. The data is there in their tools, but NetOps either didn’t want to be distracted with that next layer of detail or they were afraid to inundate their operators with what can be a fire hose of data, including performance-related alerts/alarms.
As an executive at NetScout, I was a big advocate of using the company’s network performance management technology for sustained real-time monitoring. But only a small slice of the installed based embraced the Operations use case; most were focused on troubleshooting or planning & engineering. And despite successes in large enterprises and wireless service providers, it never gained traction in web-scale commercial organizations. Those shops typically preferred to build their own tools rather than make the requisite (and substantial) investments in 3rd party packet monitoring probes/appliances.
Late in 2014, I decided to research another cool/emerging trend: the use of big data technologies in network and infrastructure management. I had always been of the belief that network visibility was a big data problem, particularly for live traffic and performance monitoring. On the one hand you have multiple sources and types of valuable data. On the other you have relentlessly increasing network size, bandwidth, and complexity. Combine the two and you meet the traditional “three Vs” definition of big data: volume, velocity, and variability. What wasn’t clear was how and if the existing tools-vendor community would use big data technology to address network visibility.
I went out in search of solutions, and found plenty who were willing to talk about big data or to send their own data into big data back ends. But precious few were actually using big data tech directly. I ran across Kentik (then called CloudHelix) and immediately thought I had found a match. “What exactly is the solution?” I asked. “Network visibility using NetFlow at scale,” was the response.
At first I was pretty disappointed. Hadn’t this been done already? Weren’t there already plenty of tools out there that could handle millions of flows per second of NetFlow/sFlow/IPFIX? The answer was yes, but not in the way that Kentik was approaching the problem. No other NetFlow/xFlow solution in the mainstream market was using a big data architecture, nor delivering a massively scalable approach as SaaS. Further, no one else retained full raw NetFlow, which is a hard and fast prerequisite for troubleshooting and security forensics.
There were some parallels using high-volume NetFlow/xFlow for security monitoring, but none of those were designed to address NetOps. The Kentik solution, on the other hand, was built from the ground up to specifically address NetOps use cases. The founders had all been directly involved in running huge networks in places such as Akamai, NetFlix, and Cloudflare, and they drew on their own experiences to design the Kentik solution. These guys knew what was most important for the NetOps environment: visibility, quick access to data, and fast answers to important questions.
While this all had me intrigued, what put me over the top was this: Kentik, just 14 months from founding, already had paying customers and a boatload of prospects in trial. And these were not a bunch of struggling startups themselves — they were big household names. While I’m not allowed to share all of the big names here, having companies like Yelp, Box, and OpenDNS on the list confirmed that Kentik had found a real and viable gap and was successfully filling it.
What Kentik is doing aims directly at the heart of what I’ve been working towards for over two decades: network visibility that is practical, useful, and cost effective, built by operators for operators, using leading edge technologies and backed by a team with a passion for success. When they asked me to be part of that team, how could I say no? I’ve been a visibility geek for a long time, and this was a company where I could channel that passion into better answers for all.
As VP Product, I will be helping to organize and prioritize product roadmaps and requirements as well as to establish alliances with a range of potential industry partners. We’re just at the beginning of the Kentik story, and I look forward to sharing it with you as it unfolds, shade by shade.