Stabilising an existing Arduino or embedded product

Sometimes the situation arises where a product is built (or gets close to being built), before any concerns about it’s stability are discussed or proper planning arranged. Often this leads to code being written without any proper test plan in place. Combined with very tight deadlines there’s often even no plan to go back and fix things up. Once this situation occurs, it’s probable that the product release will be compromised.

In Agile / Lean speak this is referred to as technical debt. Think of it somewhat like a credit card, where a project sets out with a zero balance on the card. Every time a sub-optimal decision is taken to rush something, a bit more technical debt is created. It’s not necessarily a problem and is quite usual to have a little of this hanging around, but the problem starts when it gets out of hand, and you can no longer ship the product.

In the embedded world, this is particularly dangerous because broken code in this domain:

• is many times harder to track down than it’s desktop equivalent.
• takes much longer to sort out by putting testing and good practise in place.
• could actually cause a hardware failure or brick the device.
• may not be easily updated or only fixable at a high cost point.
• can get your company a very bad name, very quickly.

The cost of unstable embedded projects on companies

As I look around crowd funding web sites, there’s various projects offering to build an embedded devices, I’m sure that many have gone through all the appropriate planning, but in some cases the technical expertise is not on hand to understand how difficult it is to get such a product right. It gets even more complex with things like thermal management and ensuring a product will AT THE VERY LEAST survive the guarantee period (and hopefully many years after). My suspicion is that most of the funding difficulties seen seem to come from protracted development cycles, trying to over-deliver and trying to address problems far too late in the day.

Sometimes, I’ve been asked how do we get to a deliverable product by companies where things have gone wrong, the answer is generally to try and get some kind of testing in place, and move toward a more Agile process. Very rarely is the answer a complete re-write, if many man years have been spent to get somewhere, there’s no guarantee a re-write would work either.

Building production ready systems, that work day in, and day out is something I’ve done for years. It requires a focused mindset on all aspects of the project, knowing what the minimum viable product is, keeping technical debt under control and testing early and often.

Also, think about how long you’ll need to support the product with updates. These can be very costly if there’s no new money coming in. You want an example - look at Android phones, many are out of date and no longer maintained within two years of buying them. Trying to write software for Android is a case of working out how many non-standard, out of date installs you’re prepared to support. I’m not knocking Android, but it’s a fact that writing for that platform is made harder by this. Maintaining software can be expensive, especially if testing and delivery is not somewhat automated.

Moving toward a more agile process

Although the solution is quite complex, here are a few pointers.

The most important steps to move toward a working project is simply to take back control. At this point a few key things need to be determined:

• Who speaks directly for your customers (the product owner)?
• Who are the customers and how are they involved in R&D?
• What is the current budget / expenditure, is it sustainable?
• Who’s currently looking after the development efforts?

Once this is determined, stop all development for a few hours, get in the pizza’s (or favourite food of choice) and get everyone from the project in a room. First off, make clear it’s not about blame and then make sure it’s not about blame. Work out what the state of play is and make a list covering:

• Process issues: EG the code does not easily build, there are no tests, library X can’t be updated.
• Problems with the system as it stands. EG crashes during function X, too slow updating display.

Next, try to get back to the original project remit, and work out how the project has deviated from this. Get everyone’s opinion at this stage, chances are that staff morale will be low and this must be addressed too. Add anything discussed at this point to the list.

At this point there’s a known list of issues and required features. They need to be prioritised, and I’d recommend putting testing up there near the top. Once everything is prioritised, make a new rule that anything new must be tested in an automated way, much earlier in the cycle.

Also, ensure the code is being stored in a proper source code repository such as git and is properly backed up at least weekly. I’d recommend external source code hosting such as AWS, GitHub private, GitLab private etc.

Break work up into two to four week cycles and try to have a viable demonstration to the product owner (the one we set up earlier) at the end of each cycle. In every cycle (let’s call it a sprint) deliver some automated testing and try to fix some of the issues. It may be for the first few sprints, nothing but technical debt is solved.

We normally recommend moving slowly into an Agile process. There are many books and courses available on this subject.

As an aside, I find this video from Spotify quite interesting, where they were already following Agile, but needed to scale things up when they started to hit problems:

• Spotify Agile culture video - Vimeo