Introduction
It goes without saying that great consumer electronics products require extensive testing. In this post, we detail some of the unique product testing we do at Keen Home and dive into our use of an entire test home.
To start, it’s helpful to discuss the basic forms of product design testing a team can undertake along with their limitations. It’s helpful to frame this discussion around the following framework:
The scope of this blog post does not include the meaty topics of manufacturing (production) testing and the details around software testing paradigms. These are great topics that deserve unique attention and are in the hopper for future blog posts.
Laboratory Testing
Benefits: The testing is controlled
You may or may not recall learning about the scientific method and the importance of running controlled experiments in elementary school. Not to worry if you don’t, a controlled test environment is one in which we are able to dial in and keep tabs on the test conditions or factors that may affect how a product performs. This greatly aids in our ability to ensure the product satisfies specific quantifiable design requirements like operating temperature, motor torque, battery voltage, etc. If something broke when we shook it, we should be able to measure how hard it was shaken, the temperature it was shaken at, the orientation in which it was held and many other variables that could affect that outcome.
Limitations: May Miss Real World Conditions and Human Factors
In order to successfully execute testing in the lab, the engineering team must brainstorm the types of conditions that may be experienced in the real world and try to reproduce them in the lab. A risk is that the engineering team may fail to consider all real world variables. Laboratory testing is also limited to testing the product itself. It is equally, if not more important, to also observe the user and how they interact with the product, a process sometimes referred to as “human factors” testing. Finally, laboratory testing generally must be repeated on every iteration of the design, build and test cycle.
Beta Testing
Benefits: Real world variables, including human factors are discovered
The Smart Vent System may have worked the way we designed it to in the lab, but there is a lot to be gained by putting pre-production product in the hands of real-world users. It is human nature to make different assumptions about how a product should work from how it actually does work and this is especially true of a product in a new category where control interfaces have not existed before, let alone been standardized. The solution is generally two-fold; educate the user on how the product works and evolve the user interface and user experience (UI/UX) to be intuitive for a wide user base.
Limitations: Beta testing occurs in the ‘wild’ where our understanding of the environment is only as good as the beta tester’s account of what occurred. Perhaps a Smart Vent closed unexpectedly when the beta tester wasn’t in the room, they may be unsure of whether someone else in their household adjusted it manually when they weren’t there or whether the design or assembly of the vent and its software had failed.
Meet the Keen Home Test Home
Like every hardware company that attempts to follow best practices, we utilize a combination of lab and beta testing. At Keen Home, we also identified early on that our first product required a testing environment that would combine the control of a lab with the variability of a beta test; not only for the purposes of testing the product as a system, but also to study the science of HVAC and air flow in a typical home.
Renting an entire house was the obvious choice as it allowed us to fill in the gaps between traditional testing and to dig deeper into the science that would allow us to better connect our customers’ heating and cooling problems with unique solutions. In addition to offering a real world environment, unpredictable conditions, and a place to house an amazing intern; the test home gave us the opportunity to run tests that would have made a beta tester’s life miserable--or at the very least made their home quite uncomfortable.
We instrumented our test home with an independent sensor network to test how a Smart Vent System affects temperature and airflow throughout a home. In the early stages, it would have been premature to trust any of the data supplied by prototype hardware, hence the use of an independent sensor network as a “control” for testing the system (see the reference to the scientific method early in this blog post). Unlike a product destined for your home, the test home’s sensor network is not limited by cost, aesthetic or simplicity constraints but instead was designed to provide as much useful data to the development team as possible.
What has been achieved at the test home?
Over the past year of renting the test home we have run a number of critical experiments and validation tests. For example, early on, our engineers used the sensor instrumentation they wired the house up with to model HVAC performance both before and after installing a Smart Vent system in the home. This gave us incredible insights into HVAC safety, air flow and temperature modulation.
One of the most important experiments in the test home investigated what happens to HVAC system pressure, when a vent is closed. As you might imagine, this is a specific example of a test that we would not want to run at a beta user’s home.
System safety was a primary concern when we started investigating whether a robotic zoning would work. We started by asking experts at the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE). They have a standard rule of thumb: as long as the equivalent of one vent is open per ducting branch, closing vents can be safe.
We then decided to prove this mantra to ourselves. We started closing vents while measuring everything from power consumption to motor and duct temperature. Our Smart Vents have a fail-safe that releases air when the pressure behind them is too high - so we sealed up that mechanism with gaffer’s tape. Only at the highest pressures did we see signs of system stress; instead of pressure causing the problem, duct temperatures began to rise. Long term, high duct temperatures can strain a furnace. Our built in temperature sensors, pressure relief and intelligent algorithms watch for this symptom. Eventually, our Smart Vent System will be used as an HVAC diagnostic system to help homeowners and HVAC pros diagnose system issues.
We also looked at how we might be able to fit our four Smart Vent sizes to slightly larger or less common vent sizes. We altered the home’s ductwork and found that flow rates don’t change when fitting a 6”x12” vent to a 6”x14” duct with our new “fit-kits.”
In addition to our own testing, we have hosted independent third party testing organizations in the test home to validate Keen Home’s marketing claims; namely that Smart Vents do redirect airflow such that they can significantly improve comfort and/or save energy. One such test was administered by New York State Pollution Prevention Institute (NYSP2I), an institute based at RIT, a premier engineering school in New York State.
The rigorous study produced positive results in which the Smart Vent System was able to “increase comfort” with “no detectable increase in energy consumption.” A paper summarizing the institute's findings can be found here.
What’s on deck at the test home?
Looking at the picture above, you may have noticed the dual thermostats on the wall, including the EcoBee3. We have not so secretly been working behind the scenes to build a stellar integration with this lauded smart thermostat, and the test home has been ground zero for internal testing and feature development. The extra hardware connected to the thermostats pictured above allows Keen Home to switch between thermostats or emulate HVAC system operation without having to rewire the home. More news relating to EcoBee3 is coming later this summer.
Summary
At the end of the day, it is easier said than done for a small company to execute rigorous testing on complex hardware, software and human factors. As much as innovation in the tools and processes used in practice has attempted to reduce the cost and time required for testing, these remain highly resource intensive activities that must be balanced with cost and time to market. Even the largest brands in consumer electronics have seen intermittent quality issues that have made it into the hands of a consumer. Keen Home maintains a last line of defense in that our customer service and technical support responsibilities are operated 100% in-house allowing our development team to capture feedback from customers well beyond the beta stage.
Stay tuned for more blog posts on the topics of testing and customer service or feel free to reach out with suggestions on future content.