Solutions Looking for Problems

Have you ever wondered why people create solutions looking for a problem? In my case, it’s because solutions in my domain (embedded systems development) now cost 1/1000 (0.1%) of what those solutions cost during the peak of my career around the year 2000. I remain so amazed at what I can do for $20, that I do it over, and over, again.

Embedded Development

Embedded devices use small micro-controllers, with software programs burned into them, to perform real world functions. Examples include WiFi thermostats with automatic setback functions, dishwasher controllers and switches (one embedded device) that you can turn on and off with a wireless control (yet another embedded device).

Here’s an unscientific, un-annotated, comparison showing the effect of 20 years on the costs of embedded product development:

Then (2000)Now
Microprocessor emulator: $20,000Not required
C compiler: $8,000$0, open source or “community” version
Development board: $500$10 (Raspberry Pi Zero W is my favorite)
Version control archive system: $8000$0, open source server and client SVN
Electronic CAD program: $15,000$0, open source kiCAD
Mechanical CAD program: $15,000$0, open source FreeCAD
Alpha prototype PC boards: $2,000, including “tooling”$20 for mail order boards, batched with others’ prototypes
Beta prototype PC board: $2,000Often not needed for simple designs, as the CAD tools are very good
Hand made enclosure prototype: $10,000$1 material using a $200 printer
High volume tooling and production expenseHigh volume tooling (25% of what it used to cost) and production expense (sadly, higher)
Costs for Embedded Development, 2000 and Now

You can add up the costs (I didn’t, because my estimates are totally unreliable) but clearly it’s gotten a lot easier to set yourself up at home to develop embedded systems. You no longer need to be shackled to a corporate job to have really fun technology toys to play with. Things have, um, changed.

A Solution and a Problem

So I developed a solution. Just because I can. For amazingly little investment other than quite a few of my limited hours left on this earth. My solution implements monitoring for household mechanical systems. In our house, the systems I’ve opted to watch include:

  • Four zone forced air HVAC system
  • Five zone hot water heating system
  • Recirculating domestic hot water system
  • Well pump

After some searching, the problem I opted to solve was that sometimes these mechanical systems go awry when we’re not watching, with very occasionally disastrous results. The practical solution to that problem is good insurance, which I recommend and have already purchased myself. But I assure you, building this “HouseWatch” monitoring system was far more entertaining than the purchasing of insurance ever will be.

Don’t Cause Failures So You Can Monitor Them

In setting this up, I ensured that the monitoring system remained fully independent of the monitored systems. I certainly did not do FMEA on my system, and the hardware and software is not particularly robust. So, for example, I’m not willing to have the heating system for my Minnesota house dependent on my minimally-validated software running on one-of-a-kind hardware. And my dear wife doesn’t need to log in over a SSH link to restart the well pump. None of that. Switch off HouseWatch, screw up a firmware update or have any HouseWatch hardware fail, and the household systems continue about their business, undisturbed (and unmonitored, like they were in the first place).

Beaglebone Black Core

The core of HouseWatch is a Beaglebone Black module with a USB WiFi dongle,left over from another system that was upgraded to a Raspberry Pi Zero W.

It’s supported with an AB Electronics UK 1-wire Pi Zero interfaceand a custom-designed optically-isolated digital I/O board.

I was surprised that there were no cost-effective isolated input modules available off-the-shelf for Beaglebone (or Raspberry Pi). The one I designed takes in 5-30V AC or DC, and gives a 3.3V logic output for the microcontroller. If I want to monitor a 120 VAC signal, I use a tiny transformer, stuffed in the junction box, to give a 24 VAC output I can run with low voltage wire. I put resistance in the transformer output so a short in the field wiring won’t cook anything. I actually did hear of two separate cases where fire monitoring systems caused fires themselves. Not on my watch. And someday, this sentence may become a link to details of the input board.

The Black runs a custom Python script which implements:

  • monitoring of on/off state of inputs
  • polling and temperature monitoring of remote one-wire sensors
  • WiFi connection to local network
  • periodic and change-based logging of the condition of inputs
  • on-demand status reports on the local network
  • origination of warning messages via text messages

Data Logging

Logging is to an offsite server that runs a custom .NET server program I wrote, called SwitchBoard. (MQTT was an option, but I did this.) Devices log their data, tagged with date, time, sensor ID, sensor value, and a destination device ID. Logging occurs hourly, as well as when a temperature changes by over 2 degrees. A client can connect periodically to retrieve these logged values for display and analysis. The use of an offsite server ensures logged data is not lost while the client is unavailable, and also that the data is accessible off site. The links to SwitchBoard are protected by SSL, a password and “other sophisticated measures” I won’t describe here. Someday, this sentence may become a link to details of the SwitchBoard system.

On Demand Status Reports

HouseWatch UI on WiFi Terminal Program

On-demand status reports are minimal, but effective. Using an off-the-shelf WiFi terminal program, on a phone or a PC, one can connect to HouseWatch and open a command line user interface. Enter “H”, and a list of single letter commands is printed. Enter a command, and the requested data is shown. This link has no security whatsoever, and is limited to access on the local network.

Warning Messages

Sending warnings as text messages is done simply (crudely?) as well, by connecting to a mail server included with our web hosting package to send an email to the text message portal of my mobile carrier. The logic to decide what is an alarm and what is not, and avoiding high warning traffic levels in response to unanticipated conditions, is not simple. Discriminating a burst pipe from a water softener cycle, or lawn watering, is impossible with the information that is available to HouseWatch. So I do get quite a few nuisance reports, to which I can combine the additional information I have (such as, is the washer running?) to decide which few I should respond to. This month, the only one that drew a response from me was a report that was delayed some hours by my phone carrier (above), putting the alarm report into a different (incorrect) interpretation context. One of the first alarms the system reported after installation was high water use after we had left the house for the day. Ignored as a new system malfunction, it turned out to be a correct report of a (thankfully harmless) stuck toilet flush valve.

Log Data Client

A .NET PC client (Switch Client) I wrote in C# retrieves the log data from SwitchBoard whenever I get interested in it. The system generates quite a bit of log data, as might be expected with these 19 monitored parameters:

  • Well pump runtime and duty cycle
  • Boiler runtime and duty cycle
  • Furnace and air conditioner runtime and duty cycle
  • 5 hydronic loop return temperatures: garage, office, east basement, west basement and master bath floor
  • Boiler output and return temperatures
  • Furnace/AC plenum and return temperatures
  • Domestic hot water recirculation return temperature
  • Garage air temperature

Crucially, Switch Client cleans up and massages the raw data, nominally XML-formatted, into a CSV format for analysis and display.

Display and Analysis

It took several iterations to get a display utility I like using Excel. Now, when I open the data viewer spreadsheet, Excel queries the CSV file and imports a fresh raw data table. Then a sheet for each of the monitored parameters is updated from the raw data, with its own query and graph. As this process got easier and more automated, I found myself interested in the data more often. People are so lazy…

Reflections

This same infrastructure supports a similar system that monitors the bilge water level, bilge pump cycle rate, main battery voltage and cabin temperature on our boat. The data collector there is a Raspberry Pi Zero. And I’ve been making temperature sensors from Raspberry Pi Pico W modules stuffed into a 3D-printed polyurethane case.

When I think what a classic car hobby, or traveling even more than we do, might cost – it’s clear that embedded development activity has kept me out of trouble and saved some serious money. And with the tools being so efficient, I still have plenty of time for volunteering in the community – which should be the real work for engaged retirees.