Archive for My Internship at Wizards of the Coast

My Internship at Wizards of the Coast

Wizards of the Coast

During the summer of 2013, I was the Performance Engineering intern at Wizards of the Coast, the West coast branch of Hasbro and maker of Magic: the Gathering and Dungeons & Dragons.  I performed in this internship under the mentorship of Ernie Nelson (Senior SDET, Customer Engagement Systems) and Sheryl Shulman (Computer Science Professor at The Evergreen State College).

According to the job description:

Performance Engineering is an internship designed to expose the student to real-world experience as a Software Development Engineer in Test (SDET).  The student will work alongside an experienced SDET at Wizards of the Coast (a subsidiary of Hasbro) to build new performance test automation, maintain existing automation, execute performance tests and analyze test results.”

This blog chronicles my experiences.  See the following links to check out what I did in any particular week:

I learned a ton about all aspects of software testing, focusing on engineering automation and performance tests within a .NET-heavy environment.  Just seeing how real-world software teams work was invaluable, and will positively shape the start of my career moving forward.

During the course of my internship, in addition to software & performance testing, I also received experience & training in:

  • Using Visual Studio
  • Basic C# programming
  • Writing & executing simple SQL queries for data retrieval
  • Agile/Scrum methodologies
  • Source control (Perforce)
  • Installing new builds
  • Service Oriented Architecture principles
  • Software troubleshooting

I supported & reinforced my learning using the following texts:

  • Scrum: a Breathtakingly Brief and Agile Introduction (Johnson & Sims)
  • How Google Tests Software (Whittaker, Arbon and Carollo)
  • Professional Application Lifecycle Management with Visual Studio 2010 (Gousset)
  • Service Oriented Architecture for Dummies (Hurwitz)
  • .NET Performance Testing and Optimization: The Complete Guide (Glavich & Farrell)
  • Practical Load Balancing: Ride the Performance Tiger (Membrey, Hows & Plugge)
  • Advanced Web Metrics with Google Analytics (Clifton)

Overall, it was a fantastic summer and I am grateful to Ernie Nelson, Sherri Shulman, and Wizards of the Coast for the opportunity.

Week 12 – Fond Farewells… and Cake!

This week marked my last week as an intern at Wizards of the Coast.

Knowing the end was near, I got a bit manic about wrapping up my work and ended up staying late almost every day this week.  I managed to tune up quite a bit of test automation and finally resolved a persistent issue with data file imports that had been plaguing me.

I released a new version of a C# tool which assists in debugging a particular database issue that was previously onerous to troubleshoot.  This tool has already proven quite popular within my department, and is my parting legacy.

I also wrote up a “QA Intern New Hire Resource Guide”, which was basically a knowledge dump of all the best links to helpful resources & tools I’d found, both inter-company and from the web at large.  Hopefully this will help out the next batch of interns to work at Wizards.

My team held a little farewell party for me:

Goodbye Cake

I count myself blessed that it didn’t say “Get Out Justin”.

It was a bit sad to pack up my office, but I am grateful that I was able to network with so many fantastic people in my time there, and I can check “Work at Wizards” off of my bucket list.

Week 11 – Back to Baselines

My internship at Wizards is winding down, with just one more week to go. Hard to believe the summer has flown by so fast!

This week, I worked with Hazera (fellow performance engineer) to create some new baseline analytics for our load testing. The baseline is basically a load scenario which represents a typical day of virtual traffic for all of our web applications, which gives us a constant to run comparison runs against. I found myself grateful that I’d just taken a Statistics class, as that is what a lot of this boils down to.

I also did some troubleshooting for a web test plugin we use in our automation which extracts strings from our test email inbox. It wasn’t cleaning up after itself property, causing the inbox to periodically fill up. I ended up doing a little IMAP programming in C#, and I ended up additionally creating a load test plugin to clear the mailbox as needed. I was surprised at how easy the built-in C# classes make this.

One of the applications I’ve been helping do the performance testing on is called Planeswalker Points, and they had a major new feature release launch this week which went very smoothly. I couldn’t be prouder of the team!

This week marked my supervisor’s birthday. He is a big fan of Batman, so we literally covered his cube in Batman decorations and brought in a heap of desserts.

Next week, I’ll be going through my exit paperwork & interview, and doing my best to dump all that I’ve learned into documentation to help the next batch of interns along.

Week 10 – Loads More Load Testing

Load testing and performance analysis began this week for a major new release, which occupied most of my time.  I have been concurrently reading “.NET Performance Testing and Optimization: The Complete Guide” (Glavich & Farrell), which has cemented my understanding of performance testing.  I feel pretty confident that I could now walk into a new company and implement a performance testing program for web applications and services, from the initial hardware stages to the software side.  Basically, the rough process would go something like this:

  • Work with the Operations department to requisition some hardware to use for a load controller and load agents.  This doesn’t have to be anything fancy; it could even be old extra computers the company has laying around to start.  Ideally, though, you want your load setup to be as close in specification to the production servers as possible.
  • Set up the load hardware in it’s own network subnet with the system under test, to ensure that normal network traffic doesn’t color the load.
  • Install load controller & agent software on the machines.  The load agents are basically slave machines that actually spin up the simulated user load (this could potentially be thousands of processes), while the load controller simply manages and stores the resulting performance data in a central location.
  • Work with the business to determine desired performance metrics for the System Under Test, specifically:
    •  Desired user load
    •  Acceptance criteria for Response Times and other metrics
    •  Typical usage scenarios
  • Create data-driven test automation which reflects the various product usage scenarios, with validation in place to ensure that the requirements of the business are being tested.
  • Compose the test automation into a load scenario for the desired user load.
  • Execute the load scenario using the controller & agents.
  • Perform analysis on the results and determine if the System Under Test has attained acceptable performance.
  • If desired, move on to perform endurance, stress, and failover testing.

Obviously each step is actually a lot more intricate and detailed than that, but that’s a basic outline.  Note that the above is all textbook academic process, not necessarily Wizards’ process.

One of the most surprising things I have learned about performance engineering in my time in here is just how interdisciplinary it can be.  There are elements of psychology, statistics, and even philosophy in trying to simulate how the user will use the system.  It also really requires a broad knowledge of computer networking, hardware, databases, queueing theory, software development practices– you name it, and a performance tester can probably benefit from knowing it.  It seems like a career in which there is always more to learn.  Futhermore, since a performance tester is constantly analyzing the entire system from a bird’s eye view, he gains an instinct and insight for the System Under Test that others in the business do not, which leads to the performance engineer being a very valuable asset beyond just the fulfillment of his immediate duties.

I also tore through “Scrum: A Breathtakingly Brief and Agile Introduction” (Johnson & Sims) to deepen my understanding of Scrum and Agile methodologies.  It’s pretty interesting stuff, and is definitely superior to the old waterfall method for most kinds of software… but I had to chuckle a bit when I read that Scrum meetings are sometimes called “ceremonies”.

I have been training a new-hire SDET on what I know of web performance tests.  There is nothing like teaching someone else to build confidence in your own knowledge of an area.

I also got some more SQL practice this week, as I had to write a query to generate an answer key for the manual testers who are testing a Magic the Gathering exam site.  This was more work than it sounded like, as it required a 5-way table join and some casting.  I’m now feeling pretty good about my abilities to do basic CRUD operations in SQL, but hope to get yet more practice in the database realm in my remaining time here.

On an unrelated note– can I just say that this is possibly my favorite email I have ever gotten?

Best. Email. Ever.

Best. Email. Ever.

It couldn’t have come at a better time!

Week 9 – Saving The Environment

Most of this week was spent installing a new build of a major Wizards web service to our performance testing environment in preparation to do some load testing analysis.  I also worked to ensure that it functioned correctly and integrated successfully.  This required some remote troubleshooting along the way as well as some database work.I also did some independent research into load balancing, and reviewed some Architectural Design Documents (ADD) and Technical Design Documents (TDD).  It was pretty interesting to see how these are written in the real world.  Being a software architect would be a fascinating job.Sadly, this week marked the end of my remaining fellow interns’ time here.  We celebrated their departure with cake. I will miss the daily card games against them in the lunch room.

I am now the lone intern standing, and until such a time as they move someone into the old interns’ spaces, I have a huge office…

My workstation at WotC.

My workstation at WotC.

I just have three more weeks to try to work my way up to a corner office with a window!

Week 8 – WCF Heavyweight Championship

This week, I began to feel like I have finally hit my stride and begun to transition from a knowledge sponge into a competent, contributing member of the team.  I spent most of this week creating test automation for a Windows Communication Foundation service.  It was a bit tricky, as the particular service in question does not have a GUI but rather is a pure network messaging component, but fortunately there are tools out there (*cough*WCF Test Client*cough*) that ease the pain of figuring out the message schema.  By the end of the week, I had created a suite of component-based, data-driven automation that invokes the various API methods of a WCF service.  This automation will be used in the future for both performance and functional testing.

I also spent some time updating existing project files to use a newer version of the .NET runtime.

Company culture-wise, the department I’m in had a team outing this week.  We all went to a nearby park, where the heads of the project teams had rented out a space.  There was a virtual cornucopia of food available– pizza, snacks, and various other treats abounded.  We spent all day at the park, playing games and doing a photo scavenger hunt.  There was a bit of a competitive element, as there was a big pool of prizes up for grabs… mostly though, the day consisted of a lot of lounging in the sun and visiting with my coworkers.  It is always nice to get to spend time with people outside the office, and I’m grateful that I just so happened to be interning when this event took place.  Wizards definitely looks out for the happiness of it’s worker bees.

Week 7 – Web Test Whack-a-Mole

This week was largely spent revising some existing web performance tests, and expanding their coverage on a particular business application.  At this point, I feel pretty confident in my ability to create quality component-based and data-driven web tests that can be used for both acceptance and performance testing.

I have begun studying more about Windows Communication Foundation testing.  WCF is frequently used for service-oriented applications, which seems to be the way the industry is moving at the moment.  I will continue working more on WCF testing next week.  I also had some exposure to writing SQL scripts for the purpose of data generation.

Sadly, this was the last week for my co-intern Kunal.  Though he may have left the building, his spirit lives on within the team… as does his image, pasted to the resident office mummy and mounted to the back of Lunk:

Kunal's presence lives on, especially for Lunk!

Kunal’s presence lives on, especially for Lunk!

This week was national “Crack a Pack” day for Magic the Gathering, and the office celebrated with a literal mountain of booster packs and some pretty cool swag:

A literal swag bag.

A literal swag bag.

Week 6 – Loads of Load Testing

I focused on the troubleshooting & performance analysis sides of my internship this week.  I worked with another performance engineer to do some load test analysis and completed introductory work on performance monitoring.  I also spent a good deal of time shadowing my supervisor to learn some system troubleshooting and build processes.  It’s amazing how valuable good log files can be… a lesson I’ll be taking forward into my own programming projects.

Continuing along with my database research from last week, this week I learned about MDX, a query language for OLAP databases.  OLAP cubes are n-dimensional (as opposed to relational databases, which are 2-d) and are often used in analytics and business intelligence.  At a high-concept level, MDX queries essentially slice a geographic region from a hypercube– this is a similar technique to clustering methods I learned in Data Mining last quarter, and is really cool stuff if you geek out on information theory like I do.

I also experimented with some of Visual Studio 2012’s new features, played around with the ASP.Net profiler, and made further revisions to my previous suite of web performance tests which will make them more useful in load testing.

Oh, and I went undefeated for my first four matches in the employee Magic league– not bad for an intern!

Week 5 – Databasics

Most of this week was spent doing independent research into database topics.  As Evergreen does not regularly offer database coursework, I was grateful for this opportunity.  I focused my studies on SQL concepts and ADO.NET, which is a C# framework for interacting with relational databases.  I ended up writing a small C# console program that uses ADO.NET to interface with a local database instance, performing basic tasks such as selecting/inserting/deleting rows and calling stored procedures.  This was a great exercise and really gave me a solid basic foundation for coded database interaction.  Soon, I will begin applying those concepts to writing some C# code related to gathering database performance metrics.

I also completed a number of other small projects.  I finished my research into Silverlight web testing and wrote up a document presenting my findings.  I also fixed a couple of issues with the suite of web tests I had written previously, and have begun writing an internal troubleshooting document for web performance tests.

I had a moment of excitement when the tests I wrote turned up what looked like a bug, but it turned out to be a known issue… so the search continues for my career-first bugfind.

Culture-wise, I picked up my cards for the employee Magic: the Gathering league and made my deck.  I will be playing approximately 15 games over the next five weeks against various coworkers; this will not only be fun, but be a great chance to network.

Week 4 – Swag Bag Fever

I spent Monday and Tuesday making a third pass at my suite of web performance tests for an internal database tool.  This time around, I wrote up a design document which detailed my proposed changes to the existing codebase and cleared it with my supervisor, to ensure that I wouldn’t overlook anything major again.

I am fairly happy with the results.  I ended up with a package of standard tests which run through various functionalities of the tool, looking for errors and unacceptable response times.  These will be executed nightly, so that if any daily development changes induce a bug, it will be caught right away rather than later (when it is much more expensive to fix). These standard tests are mostly component driven– I created a couple dozen components and updated existing components to be dynamic and data driven, whilst preserving compatability with legacy code.  These components each execute simple tasks such as logging in, looking up a certain piece of data, making changes to various fields, and so on… the idea is that they are modular building blocks that can be used to create further tests.

I checked in my new tests to the source control and they are now being executed nightly, although there is still a bug with one of the tests that Ernie is going to take a look at.  As you can tell, though, I am learning quite a bit about web performance test automation!

On Wednesday, I spent the afternoon shadowing Ernie as he troubleshooted some technical issues.  Witnessing him winnowing down the various possibilities was a great chance for me to get a sample of exactly what a Senior SDET does.  I also started reading a book on Service Oriented Architecture (SOA), in order to broaden my understanding of the relationships between the various software components the company uses.  This will be invaluable moving forward in my career, as so many companies either already use SOA or are moving towards it.

I ended my week researching the topic of web test creation for Microsoft Silverlight applications.  Silverlight is a web application framework with capabilities analogous to Flash or Java applets, and which uses an encoded binary body messaging protocol called msbin1. Unfortunately, official support for Silverlight is weak, and there is no built-in way to edit msbin1 messages in Visual Studio.  So in order to make a quality & dynamic web performance test, you need some kind of third-party plugin to handle decoding & encoding of msbin1 objects.

At first, I tried to write my own plugin by following a tutorial on MSDN, but it didn’t really work that well.  I did some research and found a third-party solution that I am pretty happy with, and presented it to my supervisor.  The only problem with this third-party plugin is that documentation is very poor, so I have begun drafting a document which explains it’s use and capabilities.  I will also create a few sample tests to demonstrate the plugin’s capabilities.

Friday, I came in to see a bag full of goodies sitting in my office chair.  It was chock-full of Magic cards, Kaijudo cards, deck boxes, and a D&D soda cozy.

WotC Swag

A few of the goodies from the swag bag.

On the company culture side of things, I attended the employee pre-release party for Magic 2014.  A giant booster-draft card tournament was held… I ended up winning two of my four matches, which I was relatively happy with, given how good at the game many of the employees there are.  I also met several different people from around the company and learned about what they do.  Oh, and pizza was provided… it seemed like there were around a hundred pizzas, although that could have been my imagination.

I walked away with a ton of booster packs, exclusive promotional cards, and good memories.  Being an intern does pay!