CoderPad 2.0 - a complete overhaul of the pad experience
A simple web-based collaborative editor on the web felt so magical in 2014 that it grabbed the attention of many companies including Facebook and Apple and became their go-to solution for assessing developers. But with the rise of more sophisticated and complex technologies, the magic was fading. Testing developers in a single file was not realistic and effective anymore!
I started my job at CoderPad as a hands-on design director to help them fix this problem.
For Coderpad's privacy, confidential information in this case study have been omitted or obfuscated.
Discovery
Demand & usage discrepancy
More sophisticated web frameworks were not a new phenomenon but there was another factor in play, a trend shift in the hiring process from assessing theoretical to practical knowledge.
Our single file editor was perfect for algorithm and simple tests but not very well suited to mimic the actual environments developers work in.
A market study had helped us know what type of skills were in demand. To a great extent, the market trends and the usage data matched our pad usage but there were two anomalies.
Data discovery
Javascript had the highest market demand but was ranked 5th in pad usage.
Typescript was ranked 5th in market demand but was 15th in our system.
What do the two languages have in common? They are both used heavily for front-end development. That was a strong signal and a good starting angle to go about understanding the problem.
The problem
Why despite the demand, Javascript numbers were low?
Who in the company might have any insight? Talking to customer success team and sales has never disappointed me in the past so that's where I started again. We shared our questions with them and met with the team after two weeks.
Key insights and answers from the call.
Knowledge of Javascript was assumed
JS were the necessary skills for certain frameworks but it was not the subject of the assessmnet.
We were loosing customers/usage for lack of framework support
Customers explicitly mentioned lack of support for certain frameworks to be the problem for them.
We needed more than a script compiler
Assessment of framework knowledge was nuanced and enabling the environment to run react wasn't enough.
The session helped us better understand the survey numbers and learn that JS was not a substitute for testing other JS-based frameworks. Supporting them was not a light project but the cost of not supporting them was becoming more obvious to us.
We had heard about "React" enough to know we should support that but the work couldn't be only about React. We went back to research the market to understand the full demands on frameworks.
It was becoming obvious that we needed to have a full strategy around Front-end interviewing.
Design Strategy
A band-aid before taking on the
long-term project
Understanding the scope of work, we knew that the cost of waiting for a perfect solution was very high. With the inputs from engineers, we launched an improved version for front-end interviews.
The temporary fix provided a very basic structure that allowed users to better focus on different components of the code. Users were also able to type React in the JS panel.
While I knew that this was far from what users needed, this short-term remedy allowed our sales team to stop the bleeding so we could focus on the ultimate solution.
The solution space
How did the pad environment need to evolve?
We didn't need to go far to learn about the front-end interviewing process and what we needed to consider. We sat with our great internal engineering team to learn about their past interviews and their actual work process.
CSS/JSS/SASS
Writing test cases / Testing frameworks
Logic Layer
JS/TS
Result rendering
Knowledge of libraries (Flex, Redux, Apollo, ect.)
Debugging (less in interviews)
System design vs. generic tests
Webpack
Package management
To allow more complex assessment tasks, I boiled down the needs to the following:
- File-tree to allow candidates navigate and create structures.
- A way for users to see the existing packages and install new ones.
- Allow users to compile code and monitor server status.
Design Exploration
Designing for the most stressful professional context
It's not news that everyone, regardless of their confidence and competence, feels nervous during an interview. It was my job to make sure candidates felt right at home when they started their interview session. They should feel completely familiar with their interview environment and be able to focus on their tasks. To achieve that, I started reveiwing different code editors and understand their design and affordances to extract the common patterns among them.
I needed to find the balance and know where to stick with the patterns and when to deviate from them. Here are some examples:
Open files tab system: Jumping easily between files is crucial in a work context. However, during an interview, the priority was to keep participants on the same file. Tabs complicated the collaborative interactions, so they were not included in the final solution.
Package management: From a design perspective, a custom solution for package management made perfect sense. However, we chose not to pursue this idea in favor of aligning the interaction with real-life methods.
And after exploring tens of different design variations, the general structure of our IDE was born. Drag the circle to see the before and after versions.
Making the design real
Sacrificing nice for practical
In technical interviews, some companies allow candidates to pick their preferred language. In the old version, picking a new language was simply commenting out the old code and allowing you to write code in the new language. In the multi-file environment this was no longer possible. We had to come up with a new structure to lay down the pad options. I added a tab bar to the pad and came up with different ways to display the options. That's how I added the tabs to separate the environments.
I love creative design but when it comes to navigation, I am conservative… There I said it! I always fall for the most clear and intuitive design.
So between the nice + and hidden options behind that or hiddle labels, I picked all visible options and lables.
Optimizing the design
Getting ready for the prime-time
With the numerous changes and additional features added, we needed to ensure that both existing and new users could navigate the environment seamlessly. For example, usability research helped us fine-tune our language selector. While search was the more optimal method for language selection, we discovered that the visual appeal of the language icons deterred users from using the search. With some simple design tweaks, we were able to increase search usage.
First-click tests helped us ensure that the placement of certain options matched users' expectations.
Despite all those steps, given the sensitivity of the interview session, I wanted the existing users to feel fully in control. I gave interviewers an escape hatch.
The research and all the preparation paid off. Very few people felt the need to go back to the old UI. In roughly 3000 sessions, only in 98 sessions, users decided to switch backed to the old UI.
Go to Market
CoderPad 2.0 was born. Did customers go crazy on it?
Not so quickly. While we knew the demand was there, teams design their interview process for each role and breaking their routine is not easy. So like any new product, we needed to promote and educate our customers about the changes.
We promoted the new features in our e-mails, within product contextual ads, created new landing pages, and more to make users adopt and use the big product investment we did. And eventually, all the hard work paid off.
The IMpact
What was the result of this 6-month-long project?
The team set some targets and within a year after the framework's public launch, we achieved some impressive results.
27% of usage on new accounts was on frameworks.
46% of existing accounts started to use front-end frameworks.
Total Front-end usage doubled and jumped from 8% to 15.4% of total pad usage
Later on, additional frameworks were supported which helped the usage to jump to 28% of the total usage