A little while ago I was curious about how software that uses Transaction Logs works, so I built a toy to-do command line application to get some hands on experience.
Transaction Logs are used when you want to have metadata detailing each action that has taken place on a system but then actually use those logs as the data for the system. Therefore to build up the state of the system at any time you can simply “replay” each action that took place before that time. This enables you to time travel as well as know exactly what happened for user 54 to go into his overdraft.
Above you can see every feature of
todo-log (I did say it was a toy application!) being used from the macOS terminal.
add command you can store a todo item with both a title and some additional body text. This todo item is then assigned a short code — shown in red above — which can be used as the unique identifier to manipulate it in other commands. The transaction detailing the addition of the todo item is saved to a JSON Lines (
ls command, without parameters, displays a list of all the todo items which are outstanding at the current time. Some basic formatting is applied including showing a friendly date (e.g. “two hours ago”) for the creation time of a todo-item.
Deleting a todo item is simple, just pass the shortcode of the item to the
rm command. The command doesn’t then delete the add transaction from the
jsonl file, instead it adds its own
remove transaction detailing the time of deletion and the todo item involved.
log command just prints out the raw
.jsonl to the terminal for inspection. In the above screenshot you can see the format of both the
Finally, passing a positive integer parameter to the
ls command goes back in time by that number of transactions. In this example you can see what the state of the todo list was before the previous
rm command was issued. Everytime
ls is called it simply builds up the current state of the system by rerunning each of the actions detailed in the
.jsonl file — to go back in time we just ignore the last
n lines, where
n is the number provided to the command.
You can have a look at this fun little project over on my Github.
Though I only spent a couple of hours on it, and it doesn’t do much, I feel like working on this gave me a greater appreciation of what transaction logs can be used for and the challenegs around them. Systems like RDBMS transaction logs and Apache Kafka are much more interesting to me as a result.
Sometimes the easiest way to learn about something and become more interested is just to jump in and build something
Mark Zuckerberg recently unveiled Jarvis, a simple AI to control various facets of his home including his lighting and more amusingly his toaster, which was a result of a challenge he set himself late last year to be completed by the end of 2016. The Facebook CEO is well-known for setting himself a “personal challenge” each year — a phrase which I think much better describes what we aim to achieve when we set ourselves a “New Years Resolution”. I thought that it might be nice to try to follow in Marks footprints, though not aiming for something quite as grand as building my own AI.
It’s well-known that Gyms are very popular in January as many people set themselves a New Years Resolution of losing weight or getting fitter but, by early February the same gyms are usually back to normal levels of use. Though there are undoubtely many reasons why this happens I think one of the big ones is that “losing weight” isn’t a great objective — losing 1lb technically completes it though thats probably not what the resolver intended.
Thinking back to my Business Studies A-Level I remember learning about S.M.A.R.T objectives. S.M.A.R.T is a mnemonic describing the properties which make an objective a useful measure.
Using these properties we could make our “lose weight” objective into “Lose 10lbs by December 31st”. That way we at least have something we can measure our progess against and a time set for when we want to have it done by! It’s also not so difficult as to be unachievable. Much easier to maintain motivation for.
With that in mind here are my Personal Challenges for 2017:
- Read 24 books by December 31st. That’s one every two weeks. Up from my current year score of half a book (Code Complete in case you’re wondering). Split between fiction and non-fiction.
- Contribute and have accepted some code to Chromium, Servo and Node by December 31st
- Build and deploy the “Organisations” feature of CS Blogs by March 31st
- Finish and deploy the new Universal React version worldwidelighthouses.com by July 31st
- Write at least 12 blogs by December 31st
- Write and have accepted a talk for a conference held in London by December 31st
- Lose 10lbs by December 31st
I think all of these objectives are S.M.A.R.T. so I should be able to keep the blog up-to-date with my progress.
In early September I decided I wanted to find a new role in which I could make more of an impact than at my previous jobs. After having spoken to the very enthusiastic CTO of PepperHQ, Andrew Hawkins, about a role as the Senior Software Engineer of the Pepper Platform I decided it would be the perfect place for me to make a mark.
Pepper build a series of iPhone and Android applications for resturants, retail and hospitality — primarily Coffee Shops at the moment — which allow users to pay for products and recieve awards for being a loyal customer.
In my mind the coolest use case of the Pepper Apps is CheckIn/Pay by balance. Imagine you work in Canary Wharf and visit the same Coffee Shop every morning to get your caffeine hit. Without Pepper you would have to go in, order your drink, wait for it to be made and then pay for it using cash or a credit card and, if you wanted to earn loyalty rewards, you would have to carry a flimsy bit of paper with you and get it stamped every morning — assuming you don’t lose it before you’ve collected enough stamps for a drink.
With Pepper you can automatically be “Checked in” to a location as soon as you are within a given distance of the store, perhaps just as you come out of the tube station. You can then make your order from your phone and have it ready for you as you get to the counter. Here’s the cool bit, you can just pick up your coffee and walk off. Checking in to the location earlier made your profile picture show up on the till in the store so the Baristas know that it’s your coffee. The payment is taken from your in app wallet (which can, optionally, be auto-topped up from your credit card, meaning you never have to think about it again). Your loyalty is also managed in-app.
Pepper is really one of those applications that makes the most sense when you see it in action and realise just how much time it would save someone who buys two or three coffees a week.
My role at the company is to be in charge of the pepper platform — all of the backend services, primarily Node.js, that manage the interaction of the applications and point of sales systems.
I’ve been at the company for 3 months now and am really enjoying my time here. It’s pretty neat to build a product people can see the value in, and that is available for use with companies that are household names.
So far in my time at Pepper I’ve added “Pseudo Currency” as a type of loyalty scheme, improved the development process by introducing Continuous Integreation, Linting and a Pull Request merge model using protected branches and started work on a series of improvements to the loyalty reward process.
I plan to keep the blog up-to-date with any developments at the new job.
I first created this blog in September 2011 in preparation for the start of my undergraduate degree — the idea being that I would chronicle my time at university both in terms of what I learnt and the fun I was having.
Not being able to get a wordpress.com domain using my name (there are 1,574 people in the US called Danny Brown according to the US Census Bureau) I stuck on the name “dannycomputerscientist”, perhaps somewhat presumptuously considering I hadn’t even started my Computer Science degree.
256 blog posts later I’m still here blogging about Computer Science but also about my hobbies and holidays. A change of domain name to something shorter, more memorable and easier to communicate made sense — so when I discovered that DannyBrown.blog was still available to purchase I jumped at the chance.
So for the time being I — reluctantly — remain on wordpress.com but now the blog is known as, and accessible by, dannybrown.blog.
I hope you continue to enjoy reading as much as I enjoy writing.
The Joel Test is a very quick way of measuring the quality of a software engineering team by asking them 12 questions, which can only yield a yes or no answer. A score of 12 is great, 11 is tolerable and 10 or less is a fail. It’s harsh, but fair — a combination of two or more fails could result in larger problems.
When trying to size up a development team I usually ask them “Have you heard of The Joel Test?”. If they’ve heard of it I ask them if they know their score, we then usually have a discussion about each question. If not, I introduce them to it and do the same thing. Anecdotally, I’ve found that being aware of the Joel test increases the likelihood of a tolerable or passing score.
When Joel Spolsky, a former program manager on Microsoft Excel and CEO at Stack Exchange, first published the blog post outlining this test in 2000 questions such as “Do you make daily builds?” were probably a lot more relevant than they are now in a world where a lot of development is for the web. A lot of modern web development doesn’t require a build stage, and even when it does it’s usually quick enough to take place on every commit rather than once a day. Whilst big software packages such as Microsoft Excel probably do still require daily builds the vast majority of software teams I talk to aren’t making things like that.
Because the teams I usually speak to are doing different things I end up altering some of the questions and I thought it might be worth noting down my changes; even if it’s only myself that ever refers back to them.
- Do you use source control and follow a workflow?
- Are your test and local environments highly similar to your production environment?
- Do you use a continuous integration server to build and test every change (be that at the granularity of per branch or per commit?)
- Do you have a bug database?
- Do you fix bugs and make time for refactoring before writing new code?
- Are developers involved early on in design and product decisions?
- Do you have a roadmap?
- Do programmers have quiet working conditions?
- Do you use the best tools regardless of cost or license?
- Do you make testing everyones concern?
- Do you use Code Reviews?
- Do new candidates write code in their interviews?
- Do developers have access to stats and metrics for the live product?
As you can see, I’ve got 13 questions rather than 12 — therefore a perfect score is 13; 12 is a tolerable score and anything less means you should be looking to make improvements.
1. Do you use source control and follow a workflow?
In the Joel Test, the question just asks if source control is used. I’ve not come across a commercial team — so far at least — that doesn’t utilise source control. However, merely using git doesn’t mean you’re using it in the most optimal way. Following a known workflow such as GitFlow or the GitHub Flow, makes it easier to maintain multiple versions of a product and work on multiple different new features at the same time.
2. Are your test and local environments highly similar to your production environment?
There have been a number of times where I’ve had a bug that only appears in production environments, serving live traffic; this sucks because you can end up testing fixes in production and affecting your real users. Whilst having an identical system locally can be difficult — by design most large scale web systems are distributed so couldn’t be fully emulated on a single machine — it should be possible to have an environment that is highly similar at a component level. Tools like Docker make this easier.
3. Do you use a continuous integration server to build and test every change?
Joel speaks about utilising a daily build to catch mistakes programmers routinely make, such as not checking in a new file resulting in broken builds. Shortening the feedback look from once every 24 hours to a few minutes after every change by following the principles of continuous integration is a more modern approach. TravisCI, Jenkins and CodeShip are popular tools for achieving this.
4. Do you have a bug database?
This question remains as relevant now as when Joel asked it back in 2000.
5. Do you fix bugs and make time for refactoring before writing new code?
Joels test focused just on fixing bugs before writing new code. However, I think making time for refactoring and the reduction of technical debt in small amounts over a period of time is also crucial to the continued effectiveness of a team.
6. Are developers involved early on in design and product decisions?
I’ve experienced development teams where web developers were given a final pixel-perfect design by a graphic designer and asked to reproduce it — this rarely works well because a single PSD rarely shows the complex interaction a user can have with, for example, a web page. Having developers collaborate with a graphic designer and other product stakeholders from early on the process can result in better, more complete, specifications and a better understanding of the business and user needs by the people developing the software for those people — that can be no bad thing.
7. Do you have a roadmap?
Joels question asked if the software team had access to an up-to-date schedule. Most teams I’ve worked with take a more agile approach to development and therefore don’t have a schedule set in stone, or often at all. However, it is important to know the direction the ship is sailing in still. What projects are on your roadmap?
8. Do programmers have quiet working conditions?
This one is more important than people give it credit for. I like the approach to working conditions taken by Stack Exchange.
9. Do you use the best tools regardless of cost or license?
Joels question originally asked if a company uses the best tools money can buy, however in many cases now the best tools don’t need to be bought — they’re free as in beer or open source. So I’ve simply clarified that with my version of the question.
10. Do you make testing everyones concern?
Some developers like having a team whose sole purpose is to test other peoples code; and Joel’s original question marks you down if you don’t have dedicated testers. However, I am of the opinion that testing should be every individuals concern. Given that code that is easy to test exhibits different attributes to code that is difficult to test, removing the responsibility of testing from developers increases the chance that they may produce code that the testers subsequently struggle with.
However, only having develops test their own code would result in poorly tested code due to code being tested with the same set of assumptions as it was developed with.
Therefore, everyone needs to be concerned with testing and quality in general. Developers in the first instance, a second pair of eyes — be it a dedicated test engineer or another developer — and a product stakeholder should be the minimum number of people involved in the quality assurance of any change.
11. Do you use Code Reviews?
Code Reviews serve a few purposes;
1) They improve the quality of code by allowing for input and discussion with developers not initially involved in its development.
2) They help mistakes to be spotted before they cause any problems.
3) Code reviews are, in my opinion, the single best way to disseminate knowledge within a team — you can’t have a part of the system only one person understands if other people have been involved in regular code reviews
I personally like to have a code review as part of accepting any pull request in the Github flow, but this isn’t necessary to pass the question. They should, however, be regular.
12. Do new candidates write code in their interviews?
This question is unchanged from The Joel Test.
13. Do developers have access to stats and metrics for the live product?
Its important that developers can see how their work is managing in the real world; whether this be performance metrics (RAM usage, number of concurrent connections, etc.) or business statistics such as step conversion. Allowing the people who are building the product to be able to see the results of their work means that they can see if they need to take a new approach or if their work is paying dividends; therefore this aids both motivation as well as early detection of possible problems. Note: Having a business analyst or similar between the developer and the metrics doesn’t count, the developers should be able to access them directly and in real time if possible.
This test has the same caveats as The Joel Test; you can get 0/10 and by some divine intervention have a team that is constantly delivering, conversely you can ace the test and still be working in a dysfunctional way and obviously you shouldn’t be using this as a checklist to see if your team is capable of working on nuclear power plant control software.
However, what this test should do is let you know how much a team has thought about quality and developer experience, and open up a dialogue which allows you to investigate further their ideals around development.
There has been a lot of discussion recently about the advantages and disadvantages of both native mobile applications — colloquially referred to as just “apps” — and Progressive Web Apps, which allow people to use website functionality whilst offline.
One of the big advantages of PWAs is that you don’t have to install them; they reside in your browsers cache, no user interaction required. However, I think a bigger advantage in that area is that you don’t have to update them, just refresh the page!
Updating apps is a chore — yes, it’s better than the experience on a desktop machine where every application has its own update daemon or requires complete re-installation — but it’s still not something that as a user I particularly look forward to doing.
It’s made all the worse when something like the below screenshot is what awaits me behind the App Store update badge.
You wouldn’t let someone borrow your phone without them telling you why they want access to it (ringing international numbers or randomly deleting my contacts is a big no-no) so I don’t see why we should allow developers who won’t even tell us their intentions install things to our devices.
Yes, you’ll never truely know whats going on unless you have a copy of their source repository and a corresponding hash to test the binary you’re going to install against (and even then, what if someone hacked their compiler… etc.) but it would at least be nice to know what, in their own words, they’re intending to get your device to do. Especially seen as data caps and storage limitations are still a big deal on mobile.
Normally applications say something along the lines of “We update every week, make sure to keep updating us”, but I felt that Twitter really won the battle for most absurd update description this week:
Not too much has changed. But enough to warrent an update. Happy Tweeting!
Even if we ignore that fact that the first full-stop in that paragraph should be a comma, it still doesn’t make much sense. What size does a change have to be in order to warrent sending a 78.4MB package to millions of users? How do they quantify not much? What are you doing to my phone?
Now, I’m not for a moment suggesting we get a list of git commits which have entered an apps master branch and display those to the user — the average user is certainly not technical enough to appreciate that, however, it would be nice to at least list features or things that have been fixed.
Credit where credit is due, Spotify release fantastic update descriptions for their applications, where they use this exact approach. New features are highlighted, followed by bugs which have been resolved and finally, to make it a little bit fun, they add a description of a “Fictitious” improvement such as “This app is now available in three new fruit flavours. (Berry Surprise is still quite buggy.)”. That bit of humour makes people more likely to check out app updates and question what is being ran on their device — which I think is no bad thing.
So developers, please write better update descriptions. It’s exciting to release new features or fix a bug that has been haunting someone for a week — let the people know you’ve done just that!