The title is meant to be ironic. Solving problems are not easy especially when you are 2000 miles from your App Developers, 6000 miles from your Chinese employee and camera producer and neither of you have a common language except different levels of English. However, that is not the reason things are difficult – the reasons things are difficult are because there’s no standardized way to solve the problems. There are ways to do things to narrow the problems down, but sadly, it’s a more of a process of going with it whilst at the same time working ahead of yourself so you can roughly direct what is going on. I’m going to give three examples of some problems we solved, each of varying difficulty. The first, a microphone placement problem, the second, a white balance balance problem, the third, implementing APIs from the the producer into the app with incomplete documentation.
First problem: a microphone placement issue
We knew this problem existed as soon as we got the first beta samples – it was a mis-translation and mis-understanding that lead to the problem. But no point dwelling on that. So, we had a problem where the microphones were too close, and whilst we had in built software to minimize the static the microphones proximity meant for about 30 seconds or so they would create odd noises then the software would realise what was going on and sort it out. So whilst we had a win on the software working, we had a problem on the actual positions of the microphone.
This is probably the simplest and easiest problem we have faced and yet my guess is 90% of people will not realise the work that went on to solve this problem.
We found the problem, and documented it, photographed everything, and wrote a description with video footage of the problem. We did this on four separate cameras. – this then went immediately to the producer. This was made easier as we caught this problem in China. The producer then gets a camera dissects it and finds the problem themselves and confirms it again. They find the problem and let us know exactly what the problem is: *insert a lot of technical Chinglish jargon here*. As the problem stemmed from the PCBA layout it went to our PCBA design team. They then confirm it again. Re-arranging a PCBA is not easy, it’s not like lego where you just take something out and place it back it where you want, everything is highly balanced and optimized. You change something here, it means for example the wifi module has to move, but because that needs x amount of space and power, it can only go here, which because of that means the microsd card slot has to move, which because that can only go about in 4 places the buttons have to move slightly, and now because the PCBA has changed, we now need to update the camera case. So off to the camera case manufacturer we go. They now look at the PCBA design, the ventilation, durability, and integrity of the case. Well there’s a problem, if they do what we need them to do, the case cannot find everything in without changing the case structure…soooo back to the PCBA team. This happens about 3 times before we nail it. Yes, this was a simple problem and by all accounts, the simplest we have solved. Now the microphone will be on opposite sides, one at the front, one at the back, giving stereo sound.
Second problem: white balance changing (plus improving image quality)
This is still an on-going problem (though greatly improved and almost solved) and highlights how finding a problem, testing, altering, and repeating, does not necessarily result in a problem being solved. As I have stated before, there’s a white balanace issue, going through a tunnel and then coming out into bright sunlight, it used to take 1-2 seconds to adjust but now it’s down to 0.5 second. It’s getting better with each improvement but it’s still not where we want it to be. So lets walk through how we are solving this. First we look at the problem, this is not a simple, oh the colours are wrong this is a: which colours are wrong, when are they wrong, where are they wrong, when do transition, how long does it, where does it transition first, last, what is the time it takes, plus more information. So we watch the video and take notes. Next step, get the beta tester to take frames of where they think the worst problem is – why you ask? We want to see if it matches up with what we think – if it does not, we submit both, if it does, we just submit ours.
This goes off to the firmware people who take a look at this issue. They then the necessary adjustments in what we call ‘baseline’ firmware (firmware is changeable etc especially as we are incorporating exposure, sharpness etc, but this is the baseline for the firmware). This baseline is optimized and refined heavily so you can get the camera out of the box and start recording with fantastic results. Some people will obviously want to tweak but the baseline will be of the best quality. They then test – this is where it gets hard – what works in one situation does not work in another – it’s very hard to replicate real life situations, even situations like tunnels which are quite standard, enter in light, dark tunnel, exit into light, are quite hard to replicate and as such results vary. So we go back to the testers and ask them to repeat. The whole process starts again.
We follow a very similar procedure for improving image quality – with a lot of users commenting on how the good pre-production quality we have plus how rapidly it has improved, this is by no means a mistake. A Netflix producer recently said of deciding to not purchase Top Gear ‘”Our buying decisions tend to be somewhat data-driven. We have a lot of data to get the deals we want’ (http://www.bbc.co.uk/news/entertainment-arts-34167864). Whilst we are not buying Top Gear, we have a similar methodology which is data driven. We look at everything, take note what’s not right, why it is not, and so forth. This allows changes to be precise and timely, rather than us going ‘oh uhm, maybe we should get it sharper?’ and then going ‘actually I don’t think sharp works’, we have a wealth of knowledge and data that we can call upon which is, this colour green here is this shade, but actually in this lighting it should be closer to this shade, as it is underexposed, we give this information to the firmware people and they can easily (most of the time) change what we need.
Third problem: implementing APIs from the the producer into the app with incomplete documentation.
This is quite possibly the hardest and most time consuming problem. In classic Ambarella fashion they have left out about 50% of the APIs needed. This has made our life just that bit more fun. We’ve changed how we have operated on the firmware/app relationship, from being firmware led, to being app led. This is for numerous reasons but one because the firmware team has a direct line of communication to Ambarella and have been working with them for far longer than the app people have. So I’ll walk you through how we deal with a problem here, I’m going to make a function and call it Function X. Function X is amazing and whilst has never been done before, is not that hard.
1) App company make sure all buttons, call functions, variables are in place so the app behaves correctly when Function X is used. They then look at the documentation and implement the supposed API. There’s a pretty good rule of thumb if command returns -1 the error is app based, if it is -15 it is firmware based, 1 is success. App company test and see if it works, if it does, yay, usually it does not. The thing Ambarella have not thought of is that a camera can have more than one command and as such more than one command may need to be sent. This I personally would have thought would have been pretty basic but apparently not.
2) If the app returns -15 value this goes straight to firmware who look at it and decide the next course of action. If it is a -1 error we debug. We had a major problem at one point at commands being sent instantly (as soon as they are updated so is the camera), where the app would send the command and essentially the firmware would go ‘nah cant really be bothered, why don’t you just get in a queue and I may update…though I probably won’t’. This was caused by neither App nor firmware and as stated was Ambarella who did not seem to count on commands working with other commands. The real problem emerges when a -1 error is given but the command cannot be debugged – ie, there’s no clear reason for it to be wrong. This could be because of the firmware struction, of an API varaible that is silently conflicting, or because well it’s just not a good day. This usually means we have to peel each function back bit bit by bit until we get the result we expect and we can find. This of it like you have an onion and only one layer is bad. You peel each layer off until you find the bad bit, remove it, and then rebuild your onion. Once this has been done we test again. This then goes off with a message to the firmware people who refine everything and tidy the code up. Whilst most of the time this is straight forward the amount of time we spend looking at about 150 pages of API documents is staggering. Luckily our head firmware engineer is incredibly talented and so sometimes writes the APIs we need exactly – this makes life easier. This is one of the reasons the app has moved slower, every function is tested meticulously and data collected, all so we know it ‘works’. Most of the time we correct a problem and the testers go….yeah you didn’t…or yeah a new one has emerged, and off we go again. It’s time consuming, but ask any of the testers, from when we first released the app to now, the app is unrecognizeable.
I hope this has given you a bit of an insight into solving problems and progressing a camera. If not just sound off in the comments with any questions and I’ll be glad to answer them.