The Joel Test is a very quick way of measuring the quality of a software engineering team by asking them 12 questions, which can only yield a yes or no answer. A score of 12 is great, 11 is tolerable and 10 or less is a fail. It’s harsh, but fair — a combination of two or more fails could result in larger problems.
When trying to size up a development team I usually ask them “Have you heard of The Joel Test?”. If they’ve heard of it I ask them if they know their score, we then usually have a discussion about each question. If not, I introduce them to it and do the same thing. Anecdotally, I’ve found that being aware of the Joel test increases the likelihood of a tolerable or passing score.
When Joel Spolsky, a former program manager on Microsoft Excel and CEO at Stack Exchange, first published the blog post outlining this test in 2000 questions such as “Do you make daily builds?” were probably a lot more relevant than they are now in a world where a lot of development is for the web. A lot of modern web development doesn’t require a build stage, and even when it does it’s usually quick enough to take place on every commit rather than once a day. Whilst big software packages such as Microsoft Excel probably do still require daily builds the vast majority of software teams I talk to aren’t making things like that.
Because the teams I usually speak to are doing different things I end up altering some of the questions and I thought it might be worth noting down my changes; even if it’s only myself that ever refers back to them.
- Do you use source control and follow a workflow?
- Are your test and local environments highly similar to your production environment?
- Do you use a continuous integration server to build and test every change (be that at the granularity of per branch or per commit?)
- Do you have a bug database?
- Do you fix bugs and make time for refactoring before writing new code?
- Are developers involved early on in design and product decisions?
- Do you have a roadmap?
- Do programmers have quiet working conditions?
- Do you use the best tools regardless of cost or license?
- Do you make testing everyones concern?
- Do you use Code Reviews?
- Do new candidates write code in their interviews?
- Do developers have access to stats and metrics for the live product?
As you can see, I’ve got 13 questions rather than 12 — therefore a perfect score is 13; 12 is a tolerable score and anything less means you should be looking to make improvements.
1. Do you use source control and follow a workflow?
In the Joel Test, the question just asks if source control is used. I’ve not come across a commercial team — so far at least — that doesn’t utilise source control. However, merely using git doesn’t mean you’re using it in the most optimal way. Following a known workflow such as GitFlow or the GitHub Flow, makes it easier to maintain multiple versions of a product and work on multiple different new features at the same time.
2. Are your test and local environments highly similar to your production environment?
There have been a number of times where I’ve had a bug that only appears in production environments, serving live traffic; this sucks because you can end up testing fixes in production and affecting your real users. Whilst having an identical system locally can be difficult — by design most large scale web systems are distributed so couldn’t be fully emulated on a single machine — it should be possible to have an environment that is highly similar at a component level. Tools like Docker make this easier.
3. Do you use a continuous integration server to build and test every change?
Joel speaks about utilising a daily build to catch mistakes programmers routinely make, such as not checking in a new file resulting in broken builds. Shortening the feedback look from once every 24 hours to a few minutes after every change by following the principles of continuous integration is a more modern approach. TravisCI, Jenkins and CodeShip are popular tools for achieving this.
4. Do you have a bug database?
This question remains as relevant now as when Joel asked it back in 2000.
5. Do you fix bugs and make time for refactoring before writing new code?
Joels test focused just on fixing bugs before writing new code. However, I think making time for refactoring and the reduction of technical debt in small amounts over a period of time is also crucial to the continued effectiveness of a team.
6. Are developers involved early on in design and product decisions?
I’ve experienced development teams where web developers were given a final pixel-perfect design by a graphic designer and asked to reproduce it — this rarely works well because a single PSD rarely shows the complex interaction a user can have with, for example, a web page. Having developers collaborate with a graphic designer and other product stakeholders from early on the process can result in better, more complete, specifications and a better understanding of the business and user needs by the people developing the software for those people — that can be no bad thing.
7. Do you have a roadmap?
Joels question asked if the software team had access to an up-to-date schedule. Most teams I’ve worked with take a more agile approach to development and therefore don’t have a schedule set in stone, or often at all. However, it is important to know the direction the ship is sailing in still. What projects are on your roadmap?
8. Do programmers have quiet working conditions?
This one is more important than people give it credit for. I like the approach to working conditions taken by Stack Exchange.
9. Do you use the best tools regardless of cost or license?
Joels question originally asked if a company uses the best tools money can buy, however in many cases now the best tools don’t need to be bought — they’re free as in beer or open source. So I’ve simply clarified that with my version of the question.
10. Do you make testing everyones concern?
Some developers like having a team whose sole purpose is to test other peoples code; and Joel’s original question marks you down if you don’t have dedicated testers. However, I am of the opinion that testing should be every individuals concern. Given that code that is easy to test exhibits different attributes to code that is difficult to test, removing the responsibility of testing from developers increases the chance that they may produce code that the testers subsequently struggle with.
However, only having develops test their own code would result in poorly tested code due to code being tested with the same set of assumptions as it was developed with.
Therefore, everyone needs to be concerned with testing and quality in general. Developers in the first instance, a second pair of eyes — be it a dedicated test engineer or another developer — and a product stakeholder should be the minimum number of people involved in the quality assurance of any change.
11. Do you use Code Reviews?
Code Reviews serve a few purposes;
1) They improve the quality of code by allowing for input and discussion with developers not initially involved in its development.
2) They help mistakes to be spotted before they cause any problems.
3) Code reviews are, in my opinion, the single best way to disseminate knowledge within a team — you can’t have a part of the system only one person understands if other people have been involved in regular code reviews
I personally like to have a code review as part of accepting any pull request in the Github flow, but this isn’t necessary to pass the question. They should, however, be regular.
12. Do new candidates write code in their interviews?
This question is unchanged from The Joel Test.
13. Do developers have access to stats and metrics for the live product?
Its important that developers can see how their work is managing in the real world; whether this be performance metrics (RAM usage, number of concurrent connections, etc.) or business statistics such as step conversion. Allowing the people who are building the product to be able to see the results of their work means that they can see if they need to take a new approach or if their work is paying dividends; therefore this aids both motivation as well as early detection of possible problems. Note: Having a business analyst or similar between the developer and the metrics doesn’t count, the developers should be able to access them directly and in real time if possible.
This test has the same caveats as The Joel Test; you can get 0/10 and by some divine intervention have a team that is constantly delivering, conversely you can ace the test and still be working in a dysfunctional way and obviously you shouldn’t be using this as a checklist to see if your team is capable of working on nuclear power plant control software.
However, what this test should do is let you know how much a team has thought about quality and developer experience, and open up a dialogue which allows you to investigate further their ideals around development.