Episode 157: Inside the Cisco TAC

Just a Call Center? 

Initially, I was not too excited about doing a show on TAC, Cisco’s Technical Assistance Center.  This topic has been requested in various forms throughout the years and for whatever reason, we just never took the time to do it right.  Until now.  

Cool thing about TAC as a subject for us? It is TECH Reality.  This really can’t be a marketing show as they deal with deployments.  As much as I could now make an argument for why they should be a part of any pre-sale process - I get it. They truly live at that golden moment…when someone has a problem and they are either going to become a customer for life…or be lost forever.   (Watch the entire Show Right Now)

Jimmy Ray never reads directions

Jimmy Ray never reads directions

‘Service after the sale’ is something we all enjoy complaining about.  While often the last thing we consider when making a purchase…it’s the FIRST thing we call out when it fails us.

Sure TAC does a great job, they have also developed some great models for managing high volume, high complexity change...a worthy goal for any  customer service environment.   I honestly did not realize just how much they do until we really dug into our research for this.  

Humble Hero

TAC has a great reputation inside of Cisco and each person we met struck me as that classic ‘unsung hero’ archetype. 

The Humble Hero: Despite the great power they possess, they always defer to their friends or allies, and rarely want to take the spotlight, even when their friends encourage them to do so. This isn’t caused by low self-esteem nor by guilt. The character honestly and truly doesn’t think of himself as any better than anyone else, despite the great deeds they’ve done, or the inspiration they’ve become. In fact, such praise usually just embarrasses them.
— tvtropes.org

To fully understand and appreciate the heroic acts couched as everyday activities - we needed to poke around. Ask some questions. 

  • How does such a large operation really operate? 
  • How does this translate to value for customers and for Cisco? 
  • Can we (as customers) access TAC resources more efficiently?
  • What tools do they offer you may have overlooked. 

Cisco TAC employs and grows the best of the best.  Outstanding individuals abound in a place like this.  In getting this one started I reached out to Jay Johnston, one of the hosts of the TAC Security Podcast.  I am so glad I did.  He and his co-hosts went overboard introducing us to the people you get to meet within the show plus many others.  

TAC may, at first glance, look like a Call Center, but as you will soon see, its more of an ‘Engineering Center’

Steve Yager, Vice President of Technical Support

Steve Yager, Vice President of Technical Support

Our first interview in the show was with Steve Yager, Vice President of Technical Support.  We actually did this interview on the very last day of the week we spent in RTP.  (We were also filming a show on the new ISR 4K that will be out soon).  I bring this up because I apparently wore myself down…or the lack of diet/lack of exercise finally caught up with me…I was sick. I had to fake my energy level with Steve…and not only was he pretty high energy, turned out he wanted to mess with me too.  Watch the show, you will see.  I loved it. 

The idea here at the top of the show was to cover the high level stuff:

  • What differentiates TAC from the competition? 
  • What are the core TAC values? (General 10000ft view) 

I wanted to set the stage for TAC as an engineering center, not a call center.  When you start meeting ANY of the [over 3,500] TAC experts, you do see great examples of the well-rounded network engineer:  RFCs, industry accomplishments, Books, CCIE and Cisco Certifications galore, Cisco Live Presentations, charity stuff, environmental focus through various lab practices, integration with schools at all grade levels, networking competitions.  Yeah. These are not call center agents.

  • TAC engineers have
    • Authored over 22 RFCs with 38 Active Drafts in the IETF
    • Been issued 323 patents issued with 472 pending as of this show

They perform like the geek version of a NASCAR pit crew opening over 38,000 cases a week and closing them all in less than 11 days on average.  (Think how big some of these cases can get…Cisco is easily in every one of the biggest networks in the world). 

This is a an operation that never closes of course and they are manning people and labs around the world with locations in RTP (North Carolina), San Jose (California),  Brussels Belgium, Bangalore India, Krakow Poland and Sydney Australia. 

Those are just the big operations, there are also regional TAC’s in Korea, China, Japan, Mexico, Brazil, Egypt, 

When we asked them the most important metrics they track?  Customer Satisfaction.  This is always big everywhere at Cisco…no exception here.  They were currently at a 4.67 out of 5…which is incredibly good.

Dave Katz

Dave Katz

The World’s Networks get diagnosed & fixed in these labs
— Dave Katz

In the interest of seeing what an ‘engineering center’ focused on solving customer problems looked and ran like.  We got to meet and learn from Dave Katz.  Dave has been operating and growing TAC Labs for many many years. He has personally overseen incredible increases in efficiency, response time and industry leading workflows directly contributing to TAC’s industry leadership. 

The labs exist to help solve problems. The ability to run the same code on the same hardware, Cisco or otherwise, is key. Speed, scale, flexibility, & replication are hallmarks of these labs with a consistent target for re-creation of any customer problem in 4 hours, day or night.  

Cisco is a big company and we have a lot of labs.  I remember how every sales office even had its own lab, often multiple, that the local SE’s would use for all kinds of obvious reasons.   The SE’s would have to coordinate with the lab manager to ensure that equipment and configuration was accounted for and available for any customer oriented project they had in progress. It was tedious, prone to error and was easily the opposite of anything you would call efficient.  This is a necessary evil to a certain extent on the pre-sale side of things…but for TAC to be successful, this was not a model to be sustained. 

Dave oversaw a move to globalize the equipment and facilities centralizing capabilities in a more scalable shared fashion.  The mega-lab we toured in RTP is over 21,000 sq. feet of networking prowess with staff doing 40% of all global work in this one facility. 

The development of COE’s (Centers of Excellence) eliminated redundant equipment purchases that historically suffered from low utilization. Virtualization has had a positive impact on capabilities with TAC labs now operating over 17,000 virtual machines.  A heavy focus on using power more efficiently is now saving over $2.6 million per year. 

Growing new talent and providing opportunities that benefit employees and Cisco now feature a system of college interns doing the hands on system builds in the labs so that master, certified engineers (the veterans) can come in and work with them on solving the bigger problems.  These same interns get experience and exposure that would be difficult to find anywhere else.  Cisco benefits from continuing to have the best talent anywhere in the world. 

 (Watch the entire Show Right Now)

Knowledge sharing is also one of the best ways TAC labs continues to improve upon their abilities.  Intelligent case tools, integrated case systems, checkout systems, procurement and ordering, power, asset and build automation. 

The integrated lab systems (checkout + case system + power automation + ordering + automation) enable these labs to be extremely fast & efficient. 

World-class checkout and reservation system for all lab gear (appliances, cards, VMs and IPs).  Automated notifications when items are about to check-in.  

Automated power off during non-bus hours and ensuring that all gear has the ability to be power cycled remotely. 
        
We also asked about what happens when TAC discovers Cisco bugs.  Many of the lab re-creates center on bug related issues.  They have refined their systems for not just filing them but also working with product development to understand what they are and how they can be fixed.  TAC Labs even goes so far as to validate these within the lab before being released back to the customer.  

Product development teams interface with TAC for input on service-ability on new products and product categories as well. TAC may be one of the best customer advocates one could hope to have. 

Networking technology is a rapidly evolving area of course and on-going training is a measured component of life as an engineer. CSE’s are trained with CCIE pods and many have access to the labs so they can study and push their own knowledge at any time. 

David White showcases his work on 'Packet Tracer' for the ASA

David White showcases his work on 'Packet Tracer' for the ASA

Tools and Resources

We had quite a few tools worth sharing. It turns out that TAC creates a lot of tools they both use and share. A few internal ones we wanted to share were deemed too early for public consumption…so suffice to say that will happen later.  We narrowed down to two interesting tools and the creators that made them.  (You should be able to find all the tools available on this Tools and Resources page)

David White has four patents on Packet Tracer for the ASA. It basically records every action a packet takes traveling through the ASA shows them and the corresponding lines of the configuration. 

Andy Gossett, Engineer and Author of NetDR Parser

Andy Gossett, Engineer and Author of NetDR Parser

Andy Gossett is a TAC Engineer and author of the *NetDR parser tool*.  Andy designed this one to help  customers understand their ‘top talkers’ reported by netflow information. TAC uses this to solve customer issues all the time. 

It parses the netdr output from Cisco 7600 and 6500 series routers and presents Top Talkers statistics. The output includes a Detailed Counters section that lists the total number of unique entries for a particular field (source IP, destination IP, TTL, etc...). Expanding the field will list the total number of hits for each entry that can be used as a filter in the Flow Results section. 

Robb, Jimmy Ray and the TAC Security Podcast Panel: David White, Magnus Mortensen, Jay Johnston

Robb, Jimmy Ray and the TAC Security Podcast Panel: David White, Magnus Mortensen, Jay Johnston

Engineering Panel

We gathered the entire TAC Security Podcast team for a little chat we thought would be fun in rounding out the show.  Favorite stories, bits of advice and a few snippets about ‘life in the TAC’ from some very well recognized veterans.

  • David White
  • Magnus Mortensen
  • Jay Johnston

This show was fun to make.  Incredible group of people we got to meet and some of the most amazing labs, processes and pride of operation we have seen in a great long while.  We need to find another excuse to visit again.

Robb

Don't forget to watch the entire Show Right Now)