A new framework for thinking about products
When I was handed over the products, as the director of product design, following the basic design heuristics alone was enough to significantly improve the products but that's not where I set the bar.
In our market positioning efforts, the marketing team had came up with "Fast", "Real", "Fair", and "Loved" to describe customers' sentiments toward CoderPad. To set up a truly customer-focused organization, I asked myself if basic design principles could be augmented with those sentiments to help us find new ways to improve the products. Those principles became the following:
Real
How to close the gap between work skills and assessment methods?
Fair
How to ensure candidate can show their best in the process?
Loved
How to enable users to extract more value from our product?
The following sections show some of the things I did, using these principles.
Making it Real
Making it fair
Making it Loved
CoderPad 2.0 - a complete overhaul of the pad experience
A simple web-based collaborative editor on the web felt so magical in 2014 that it grabbed the attention of many companies including Facebook and Apple and became their go-to solution for assessing developers. But with the rise of more sophisticated and complex technologies, the magic was fading. Testing developers in a single file was not realistic and effective anymore!
I started my job at CoderPad as a hands-on design director to help them fix this problem.
For Coderpad's privacy, confidential information in this case study have been omitted or obfuscated confidential.
Discovery
Demand & usage discrepancy
More sophisticated web frameworks were not a new phenomenon but there was another factor in play, a trend shift in the hiring process from assessing theoretical to practical knowledge.
Our single file editor was perfect for algorithm and simple tests but not very well suited to mimic the actual environments developers work in.
A survey we had done in that year had helped us know what type of skills were in demand. To a great extent, the result of the survey and the usage data matched our pad usage but there were two anomalies.
Data discovery
Javascript had the highest market demand but was ranked 5th in pad usage.
Typescript was ranked 5th in market demand but was 15th in our system.
What do the two languages have in common? They are both used heavily for front-end development. That was a strong signal and a good starting angle to go about understanding the problem.
The problem
Why despite the demand, Javascript numbers were low?
Who in the company might have any insight? Talking to customer support and sales has never disappointed me in the past so that's where I started again.
Key insights and answers from the call.
Knowledge of Javascript was assumed
JS were the necessary skills for certain frameworks but it was not the subject of the assessmnet.
We were loosing customers/usage for lack of framework support
Teams were fragmented in companies and each team was assessing our product separately for their needs.
We needed more than a script compiler
Assessment of framework knowledge was nuanced and enabling the environment to run react wasn't enough.
The session helped us better understand the survey numbers and learn that JS was not a substitute for testing other JS-based frameworks. Supporting them was not a light project but the cost of not supporting them was becoming more obvious to us.
We had heard "React" enough to know we should support that but the work couldn't be only about React. An additional survey helped us understand the full demands on frameworks.
It was becoming obvious that we needed to have a full strategy around Front-end interviewing.
Design Strategy
A band-aid before taking on the long-term project
Understanding the scope of work, I knew that our team and customers could not wait for a solution months down the road. With the inputs from engineers, I came up with an improved version for front-end interviews.
The temporary fix provided a very basic structure that allowed users to better focus on different components of the code. Users were also able to type React in the JS panel.
While I knew that this was far from what users needed, this short-term remedy gave something to our sales team breathing room so we could focus on the ultimate solution.
The solution space
How did the pad environment need to evolve?
We didn't need to go far to learn about the front-end interviewing process and what we needed to consider. We sat with our great internal engineering team and spoke about their past interviews and actual work process.
CSS/JSS/SASS
Writing test cases / Testing frameworks
Logic Layer
JS/TS
Result rendering
Knowledge of libraries (Flex, Redux, Apollo, ect.)
Debugging (less in interviews)
System design vs. generic tests
Webpack
Package management
We needed the following to accommodate the general front-end tasks:
- File-tree to allow candidates navigate and create structures.
- A way to show the server status.
- A way for users to see the existing packages and install new ones.
Design Exploration
Designing for the most stressful professional context
It's not news that everyone, regardless of their confidence and competence, feels nervous during an interview. It was my job to make sure candidates felt right at home when they started their interview session. They should know exactly where things are and how they work. So I started to review different code editors and understand their design and affordances to come up with a design that everyone felt familiar with.
A question I needed to ask throughout the design was whether to follow the exact pattern I found in other tools or whether the subject should be optimized for the context of an interview. These are a couple of examples:
Package management: From a design perspective, a custom design for package management made perfect sense but we didn't carry that idea in favor of aligning the interaction with real-life method.
Open files tab system: Jumping easily between files is very important in the context of work. However, during an interview, the priority was to have participants on the same file. The tabs were simply complicating the matter and therefore they did not end up in the final solution.
And after exploring tens of different design variations, the general structure of our IDE was born. Drag the circle to see the before and after versions.
Making the design real
Nice vs. practical
In technical interviews, some companies allow candidates to pick their preferred language. In the old version, picking a new language was simply commenting out the old code and allowing you to write code in the new language. This was no longer possible in the multi-file environment. We had to come up with a new structure to lay down the pad options. I added a tab bar to the pad and came up with different ways to display the options.
I love designers to be creative but when it comes to navigation, I am conservative…There I said it! I always fall for the most clear design.
Ask me about the design considerations while designing the tab options.
Optimizing the design
Getting ready for the prime-time
We performed different types of research to:
Picking the right design options and ensuring the general usability of the design
First-click tests to ensure the placement of the items match users' expectations.
Doing all those precautious, was I fully confident that no one would be lost and confused at the very sensitive task of conducting an interview? NO. I gave interviewers an escape hatch.
The research and all the preparation paid off. Very few people felt the need to go back to the old UI.
Go to Market
CoderPad 2.0 was born. Did customers go crazy on it?
Not so quickly. Teams design their interview process for each role. Breaking their routine is not easy. So like any new product, we needed to advertise it to our customers.
We promoted the new features in our e-mails, within product contextual ads, created new landing pages, and more to make users adopt and use the big product investment we did. And eventually, all the hard work paid off.
The IMpact
What was the result of this 6-month-long project?
The team set some targets and within a year after the framework's public launch, we achieved some impressive results.
27% of usage on new accounts was on frameworks.
46% of usage existing accounts started to use front-end frameworks.
Total Front-end usage doubled and jumped from 8% to 15.4% of total pad usage
Later on, additional frameworks were supported which helped the usage to jump to 28% of the total usage