Designing a camera, a lot of time, creating a camera, a lot of money, having incredible beta testers – priceless.
Beta testing is NOT easy. A lot of people when we were advertising for beta testers would send us emails (even though we stated we were picking people at random), stating what they would do for us, and this is almost always how it went:
‘Hi JooVuu, my name is *******. I would love to be on your beta testing program as I am an avid driver. I drive to work each day and as such can give constant feedback about the camera.’
If only testing a camera was that easy. When beta testing you have to test the camera 100 ways from here to the sun. Each function has to be tested hundreds of times, the camera has to go through every scenario you can imagine testing each function as you go. Then when you have done that – do it again. Done in twice? Do it a third time. Done it three times? Well done, we need it done again. You get the picture. It’s unrelenting. They feedback, we go away and look at it. We update, the firmware, they start all over again. They feedback, we update, they test again. So far to date we have released 17 firmware updates. This does not sound like a lot but each firmware update will usually have about 100 changes. Some of them obvious – X Y Z feature is enabled, a ton of bug fixes, and a lot of tidying up of code making it more efficient.
This happens a lot, but not only that they then help each other, with suggestions when problems arise, reducing mine and Emma’s workload whilst increasing their own. This has helped the development an immeasurable amount. We’ve added features, changed features, moved features, altered features, and the same for hardware based on their feedback, and the problems they have brought to our attention. When this camera is released a large part of it will be down to the current beta testers.
A perfect example is that of a current white balance problem we are solving, where when this problem was first brought to our attention; if the car goes into a tunnel the white balance auto-corrects itself but when you come out of the tunnel it takes about 1-2 seconds to re-adjust and get back to what it should be. This problem has been on-going and one of the testers has essentially been driving through a tunnel whenever he can in order for us to see what has changed and then make alterations. This has happened now well over 30 times and we’re almost there with solving it. Sounds simple, in that process two other members have helped, we’ve been in constant contact with the production team, and we’ve re-written a large chunk of code. Whilst sometimes it seemed like nothing was happening it was behind the scenes, we’ve now brought the change in white balance closer to 1 seconds and believe we can bring this further down. This is horrendously repetitive and most times seems like nothing is changing, but each time, we’re taking measurements, looking at different aspects – what colour is changing first, is it the top of bottom of the screen, what colour does it change to first etc – all the time getting them to retest it so we can simply see if the tweaks have worked. In man hours this has probably taken close to 70.
Some people have PM’d me on DashCamTalk asking if they we would like them to take over testing as some people are not posting anything. Let me rewind a bit, when I first started the program, I had this grand idea that beta testers would do a lot of testing, write a blog each week, and upload a lot of footage to various forums – well I was crazy and delusional for thinking that was possible. As everyone has other things going on it would be almost impossible if they wanted to do anything else. We already feel horrendously guilty about the work they are putting in, but this work has helped us to such a degree we believe our product has improved drastically just in the beta phase because of their input.
So, when you think of beta testing, please do not think of people who just get early access and get to play with something, think of the hundreds of hours, tediuous repitition, and sometimes snail pace progress that each tester goes through. This bunch of testers have been so good we’ll be keeping them on for all future beta testing of firmware and new cameras we create. That’s putting our money where our mouth is.