Software on Tom's Journal

Fledging the Nest

Wed, 24 Jun 2026 15:43:27 +0000

When we moved in to our house just over a year ago, it came with a Nest Thermostat. It worked well enough, and after paying the 3rd Party integration tax to Google, it worked with Home Assistant, so we left it at that – we had more impactful improvements to make. When we upgraded the 30+ year old HVAC system a few months later, the Air Handler that replaced our furnace had support for 24V thermostats (for backwards compatibility), so the Nest survived that change as well. However, as the months went on and I made more automations to coordinate the thermostats across the house (the HVAC upgrade added two additional heads to the mix), the reliance on Google’s cloud meant that updates to the Nest thermostat from Home Assistant were both slower than the local thermostats, and crucially, relied on the internet for them to work.

I had looked around for other 24V thermostats that integrate directly with Home Assistant, but found nothing that stood out to me. There was a Honeywell Z-Wave thermostat that had good reviews, but was difficult to find as it had been discontinued in favor of the version that had WiFi built-in (and consequently used their cloud). EcoBee’s thermostats have HomeKit support, and therefore local control in Home Assistant, but the reviews were mixed, and the thermostats themselves are also rather expensive; so the Nest got to keep its spot on the wall a while longer.

Meanwhile, the two new mini-split heads had been integrated into Home Assistant a few weeks after they were installed. This was trivial thanks to the Smlite Ductless HVAC Wi-Fi Module. Unlike the air handler, these units were designed for a WiFi module to be plugged directly in to them to enable remote control. As for the air handler, the wired remote accessory is the closest thing to a thermostat that is not a 24V thermostat. While it has built-in WiFi, it only integrates with the brand’s cloud, with mixed Home Assistant compatibly, and again relies on a working internet connection. A bit more searching and staring at the air handler’s control board revealed a path forward, the central communications bus otherwise used to network multiple units together. The community had reverse engineered the protocol used (XYE), and had built an ESPHome custom component to integrate Home Assistant with XYE.

The Requirements

Before we go too much further, I should clarify what I was looking for in a thermostat. This should not be too much to ask for, but in the era of the Internet of Shit, every smart home device wants to connect to their proprietary cloud. This would not be so bad if these proprietary clouds supported third party integrations (like Home Assistant), but where is the money in that. Add to that the inevitability of being left with a very nice paperweight (or empty kibble bowl) when the cloud service goes out of business (or is deprecated).

Fully Local – No Cloud Dependency
Self Contained – Everything needed to fit within the controls area of the Air Handler
Same capabilities as we had with the 24V thermostat – set desired temperature, set desired mode (heating / cooling), and read the temperature of the rooms affected by the unit.

Since I could not find any thermostats off the shelf that addressed all of these desires, without introducing yet another cloud dependency, I decided to build my own.

Hardware

It had been a while since I worked on hardware, but the various GitHub repos I landed on helped me to assemble a rough bill of materials. We needed only 3 main things: an ESP32 based micro-controller, a RS485 transceiver, and an AC to DC converter. The AC to DC converter was result of the self-containment requirement as the only power source available inside the control cabinet that was not 120VAC was the 24VAC terminals (R & C) that would otherwise power a traditional thermostat. I went to school for Computer Science where we all, for the most part, gladly abstracted away the hardware our precious programs ran on to the Computer Engineering and Electrical Engineering departments in the buildings down the road. Consequently, converting between AC and DC was not something I was particularly looking forward to. I bickered with Claude for a while about this problem and ended up with a simple enough 2 stage AC to DC conversion that while not super cost effective, was simple enough for me to build and understand.

Bill of Materials

QT PY ESP32-S3 NO PSRAM - The microprocessor driving the whole show
MAX485CPA+ - RS485 transceiver, since XYE runs over RS485
DB107-BP - A full bridge rectifier, converting AC to choppy DC
ECA-1HM102B - A 1000UF soothing capacitor
TSR 1-2450 - DC to DC Converter to give us a smooth 5V DC bus
Adafruit 1/2 PermaProto - My favorite way to go from breadboard to perf-board
OSTVN02A150, OSTVN03A150 & A 08-LC-TT - Terminal Blocks & IC Sockets
1 10kΩ and 1 20kΩ ¼W resistor, and some wire

Software

ESPHome is an absolute delight and makes integrating the real-world into Home Assistant a joy. I got my start in hardware over 18 years ago with an Arduino NG thanks to my Dad and things have come a long way since then. ESPHome makes it super simple to utilize ESP32, ESP8266, and RP2040 microcontrollers into build custom smart-home devices. You write YAML files that declare the various components and how they should interact with one another, and ESPHome generates and compiles the relevant C++ code before uploading it to your device. Could you do everything ESPHome does yourself? Sure, but I don’t feel like writing support for OTA, WiFi fallback, and all the other nice things that ESPHome brings. When ESPHome does not have support for something, you can extend it via External Components.

Another wonderful feature of ESPHome is that you can pull data out of Home Assistant from your embedded device. This is particularly helpful when you want to supply the Air Handler with the room temperature which comes from a few Zigbee Temperature Sensors. All that leads us to a config file that looks something like this…

esphome:
 name: air-handler
 friendly_name: Air Handler

esp32:
 variant: esp32s3
 framework:
 type: arduino # Climate is not supported in esp-idf

logger:
 logs:
 midea_xye: INFO
 uart: WARN

api:
 encryption:
 key: !secret air_handler__encryption_key

ota:
 - platform: esphome
 password: !secret ota_password

wifi:
 ssid: !secret wifi_ssid
 password: !secret wifi_password
 ap:
 ssid: Air Handler Fallback Hotspot
 password: !secret fallback_password

captive_portal:

web_server:

external_components:
 - source:
 type: git
 url: https://github.com/tpaulus/ESPHome-Midea-XYE
 ref: main
 refresh: 1min
 components: [midea_xye]

#UART settings for RS-485 converter
uart:
 # UART1 on QtPy
 tx_pin: 17
 rx_pin: 18
 baud_rate: 4800
 flow_control_pin:
 number: GPIO9

sensor:
 - platform: homeassistant
 id: ha_room_temp
 entity_id: sensor.air_handler_room_temperature

climate:
 - platform: midea_xye
 name: Air Handler
 period: 3s
 timeout: 500ms
 use_fahrenheit: true
 follow_me_sensor: ha_room_temp
 sync_fan_mode_from_device: true

Integration

Anyone who has worked in engineering long enough knows that building the individual components is only half the battle, the true fun comes from trying to integrate all of the disparate pieces, especially when the interface between those pieces is fuzzy. That was especially true in this case. I assembled all of the hardware, but instead of using the resistor-divider for the RX line, I had a TXB0140 level-shifter in its place. I hooked it up to the air handler control board and got 00 out. I hooked it up to the Air Handler and got back a slew of error messages: Bad response length (0 bytes, expected 32) for Command C0; resyncing. Time to start troubleshooting.

Is it the hardware?

The easiest answer here would be to grab a logic analyzer or a fancy digital oscilloscope and probe the X and Y (A and B for RS485) and see what was going on. Only one problem, I do not own an oscilloscope, let alone a logic analyzer. I have a multimeter, tenacity, and that’s it. More bickering with AI followed, the TXB0140 went out, in its place came a simple resistor divider – one less variable. The breadboard made its way in and out of the unit, up and down the stairs at least 5 times. Together with AI, I verified each part of the hardware to ensure that the basic behaviors were as expected. I had reached the point where there was either a software problem, or a comm bus issue.

Is it the software?

I had started with fork of HomeOps’s ESPHome-Midea-XYE repo that applied a bug fix Seerenity had made which supposedly fixed a dead-lock issue. I had initially sent AI into the mines to add support for Flow-control, as the MAX485 does not have automatic flow control, but during the hardware debugging phase, I had discovered the ESP32 has built-in support for Flow Control, so the custom flow-control changes also went out the window. However, much like with the hardware, there were too many variables. So, with the help of AI (once again), we took a much simpler example from wtahler and manually constructed frames to send down the wire. Crickets…

Back to the hardware, AI suggested to permanently ground the Receive Enable pin on the MAX485. This would mean that we get an echo, since we would receive our own transmission, but it would eliminate yet another variable. This lead to a key insight - we could hear our own messages. This proved that the hardware was working correctly, and we could send bytes from the ESP32 and parse them back again. That left only a communications issue.

Is it the Air Handler?

If I had one piece of feedback for the folks at Carrier about the 45MBAA Air Handler, it would be that the hardware documentation sucks. There are at least 10 dip-switch blocks across the unit and 3 different rotary encoders with tiny labels and close to no documentation on their purpose. The diagrams that are there are confusing and leave the reader to make assumptions. The unit had been configured for a 24V thermostat, this means setting SW1 (A block of 4 dip-switches) to On On On Off. The sticker on the front door says for Wired Control, set them to Off On On On. This was a lie - in tiny letters in the corner of the control board layout was the key “All DIP Switches default to Off”. So, I turned off the unit again, and set all of SW1 to Off, and turned it back on. This time, the seven-segment display on the unit was blank. The immediate first thought was did I just release the magic (expensive) blue smoke from the control board?

The panic was thankfully short lived. The power board (right behind the control board), had a green LED light, so it wasn’t completely dead. The control board has only a single input, a tactile button next to the seven-segment display. So, like a curious Chile (or cat), I pushed the button and the display flashed on, EF. I pushed the button again, 75. I pushed the button a third time and the display extinguished. This was different, before, it always read 00 on the display, which I had learned to mean idle, since it went to 01 for first stage cooling, 02 for second stage, and 04 meant heat mode.

I went back upstairs to my desk and sent a new set of serial commands to the control board, and this time it got a response. Not just the echo of the command, but a response from the unit. It was alive and working! I changed the ESPHome configuration back to use the custom component and made a few more fixes to get support for the various modes into the Home Assistant climate integration and to fix the representation of the Follow-Me temperature, but it was working!

My fork of the Custom Component is on GitHub for those also looking to convert their Carrier (Midea) Air Handler to direct control.

tpaulus/ESPHome-Midea-XYE

ESPHome external component for controlling Midea HVAC systems over the XYE/CCM RS‑485 bus. Provides a native Home Assistant climate entity with full mode, fan, and setpoint support.

tpaulus GitHub

Keeping it dumb

So having the air handler in Home Assistant is fantastic, but as a general rule I try to preserve physical controls for all of the things we make smart across the house. This means that at 3am when you want a glass of water, you don’t need to get your phone from the room next door and stare into the sun, just to nudge on the lights.

For lighting, we use the Lutron Caseta family of switches which preserve the ability to press a button and have the lights turn on/off. When we added smart shades, a similar button went on the wall, which is powered by Home Assistant, but still obviates the need for a phone, or shouting at Siri to “please lower the god-damn shades in the living room”. When it came to the air handler I wanted something similar, and preferably a control mechanism that did not rely on Home Assistant, just in case – I still want my wife to be able to turn on the heater when the server is down. The answer in this case was actually right in front of us, the wired remote accessory from earlier. Notably, it did not rely on the 24V interface that we had disabled to enable the ESPHome device to integrate with the air handler, and could re-use the existing thermostat wires in the wall. eBay to the rescue, and a few days later, the control was on the wall and the best part – it reflects the changes made by Home Assistant, and vis-a-versa. Make a change on the control, Home Assistant reflects it moments later.

Backing up Unifi Protect

Wed, 24 May 2023 01:16:32 +0000

A few weeks ago, I was scrolling through the r/Ubiquiti Subreddit when I came across a post where someone had their NVR stolen during a break in. We have a number of Protect cameras across the house and the post got me thinking… I have offsite backups of data stored on the NAS, but not the NVR. A lot of the recordings are useless, but it would be really nice to have the events stored offsite for a few days, just in case.

Luckily, the comments of the original post (which I can’t find anymore) had a bunch of helpful suggestions, one of which being to run unifi-protect-backup on a Raspberry Pi. However, because Pis are in short supply right now, and because I don’t love running Pis for production workloads, I wanted to run the backups from the NVR itself. I thought that the NVR had enough spare CPU and Memory to do the work, and had all the data available locally, but boy was I mistaken.

Since I wanted to both encrypt and compress the video files before uploading them, this turned out to be a computationally expensive operation and when run on the NVR, would result in a lock-up that would only be resolved by a power-cycle.

Since I had recently started using Nomad for my Container Orchestration, I opted create a Nomad Job for this task and have the job run from one of my servers, where significantly more CPU was available, and I did not risk locking up the NVR. The Nomad Job configuration ended up being much simpler than the work needed to get the backups running on the NVR (especially since the NVR likes to reset things after updates).

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Weaving a New Solution to Container Networking

Fri, 19 May 2023 19:48:40 +0000

As I described in my recent post setting up Nomad, I restored to using macvlan to get containers their own network addresses, but this had the unfortunate side effect of container being unable to talk to their host due to intricacies of how the Linux kernel does packet routing. I wasn’t happy with the solution, both because containers would sometimes get stuck and not get cleaned up, resulting in more than one container with the same IP address running on the network, leading to all sorts of strange issues; and because it meant that there was a single point of failure in my router, serving both Consul DNS and hosting cloudflared for the applications which used macvlan. So I went looking for options.

Looking for a Solution

I was weary of CNIs and their support in Nomad from the many hours I spent cursing the last time I tried to deal with this issue, so I was looking for solutions that had already been successfully integrated with Nomad in the past. I landed on three options: Calico, Flannel, and Weave, thanks to a post by SUSE’s Rancher project. Each had their upsides and downsides.

Calico - We use this at work, so I am somewhat familiar with its operations, but the use of BGP would cause issues as my router, a Unifi Dream Machine Pro, doesn’t support BGP out of the box. It also had a bunch of enterprise features, like network policies, I do not need, so I kept looking.

Flannel - This seemed like a great option, until I got to the part where I read that it relies on an etcd cluster. While I don’t have anything against etcd, I did not want to spin up another state management cluster if I didn’t need to. So Flannel went in my back pocket, as a last resort.

Weave - Weave uses a proprietary routing fabric, rather than linux primitives (like iptables), it seemed like a good fit for my application, given the low barrier to entry and the simple configuration options when using it alongside Docker.

Implementing Weavenet

I used Ansible to install Weave and its associated systemd unit.

- name: Install Weave
 hosts: nomad
 tasks:
 - name: Install Weave
 uri:
 dest: /usr/local/bin/weave
 status_code:
 - 200
 - 304
 url: https://git.io/weave
 mode: 0755

 - name: Create Systemd Service
 copy:
 dest: /etc/systemd/system/weave.service
 mode: 0755
 content: |
 [Unit]
 Description=Weave Network
 Documentation=http://docs.weave.works/weave/latest_release/
 Requires=docker.service
 After=docker.service
 [Service]
 EnvironmentFile=-/etc/sysconfig/weave
 ExecStartPre=/usr/local/bin/weave launch --no-restart --no-dns --ipalloc-range $IP_CIDR --metrics-addr=0.0.0.0:21049 $PEERS
 ExecStart=/usr/bin/docker attach weave
 ExecStop=/usr/local/bin/weave stop
 [Install]
 WantedBy=multi-user.target

 - name: Ensure sysconfig Directory Exists
 ansible.builtin.file:
 path: /etc/sysconfig
 state: directory

 - name: Create Weave Peers File
 copy:
 dest: /etc/sysconfig/weave
 content: |
 PEERS="10.0.10.48 10.0.10.64 10.0.10.80"
 IP_CIDR="172.30.0.0/16"

 - name: Start Weave Service
 systemd:
 state: restarted
 daemon_reload: true
 name: weave

I ended up overriding the default CIDR (10.0.0/8), as it conflicted with the parent network and explicitly listing all of the peer servers to avoid needing to join them manually.

I then ran weave expose on each of my nodes and added the /24 returned to my router’s static routes table, with the target being the node that returned the IP block. This enabled clients not part of the weave network to be able to access weave clients. From there, I just needed to explicitly set the network_mode in my Nomad Job Docker Configuration and ensure that the address mode was set to driver for my service definitions.

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Example Configuration using WeaveNet

Hey Siri

We have a handful of Apple’s HomePod Mini’s across the house and my partner and I both have iPhones, so it is handy to be able to ask Siri to turn on/off the lights and what not. The actual work is done via the Home Kit Integration in Home Assistant.

One of my usual tests after I change something with either Home Assistant, or its network is to ask my office HomePod to turn on or off my Desk Lamp. Unsurprisingly, after I configured Weave on Home Assistant this stopped working. A bit of pondering lead to a potential issue. mDNS. When you pair a HomeKit device, you never put in an IP Address or other configuration, it is all handled via multicast DNS (mDNS), and now that Home Assistant was on a different subnet than the Home Pods, and iPhones, this connection was broken, despite the networks being accessible from one-another.

Conveniently this is not an uncommon problem, and it’s already been solved via tools like mDNS Reflector. Deploying a container that bridged the weave network and the server host network (which is the same one as the network our phones, HomePods, etc are connected to) instantly restored the connection between HomeKit on my phone and Home Assistant. And just like that, we were back in business.

Resilient Infrastructure for the Homelab

Thu, 18 May 2023 23:25:52 +0000

As I’ve alluded to in the past, Home Assistant plays a major role in making my home work, from managing the HVAC system, to automatically turning on the lights in the utility closet when the door opens. For many years, I ran Home Assistant on an i5 Intel NUC within a Docker Container. This worked great, but when the NUC went down (be it for updates, or some unscheduled maintenance), so did Home Assistant and my ability to control the lights. While my partner is very forgiving and understanding, it’s rather annoying to not be able to turn on some of the lights for a few hours while I troubleshoot an issue.

Single-points of failure are scary, and at work, we try to eliminate them wherever possible, so I thought it was time to do the same at home. There are a lot of existing solutions to this problem in the form of orchestration engines, like Docker Swarm, Kubernetes, K3s, etc. We use Kubernetes at work, but it was rather overkill for what I needed to do at home, which was schedule Docker Containers (and ideally Shell Scripts) to run across various nodes in my home-lab (and ideally on a specific schedule). I landed on using Hashicorp’s Nomad to get this done. Nomad met all of my needs, all without being too complex.

From the beginning of this project, I had in mind to eventually get 2-3 new servers to enable the decommissioning of the NUC that had been working its little heart out for me for the last 4 years. As such, I wanted to automate as much of the provisioning and management of the new servers as much as possible. I wasn’t quite ready yet (or sure of what I wanted to get), so I used the NUC and a Gaming Server I had built a while back to start experimenting.

Ansible for Provisioning

I love using Terraform for config management, but it doesn’t support configuring machines well (there are other tools in the Hashicorp offering that do this, but are heavily focused on the cloud). That left tools like Ansible, Chef, Puppet, and a few others. A little time spent on Google revealed that Ansible was the easiest to get started with, used one of my favorite languages (Python) for extensions, and didn’t require complex provisioning workflows.

Ansible has the concept of sharing Roles and Tasks via the Ansible Galaxy. Conveniently for me, there was already a Galaxy Collection for configuring Consul (we’ll get to that in a bit) and Nomad. I had to make some adjustments to fit my needs, notably, running both the Nomad Server and Client on the same node.

Consul for Service Discovery

Before we can talk about Nomad, and container orchestration, we need to talk about Service Discovery. Service Discovery allows us to answer the question, where does X service live and how do I communicate with it. When running containers on a single node, this isn’t a significant issue, since the question of where is THE server, and the question of what port can be answered by the container orchestrator (Docker in my case). While we can pin specific container to specific nodes, that doesn’t allow us to build a resilient system, which is where tools like Consul come in.

Consul provides a single source of truth for the answer to the what and where question. It adds some extra nice features, like health checks. Nomad more recently started offering a built-in, simple, service discovery option, but Consul still wins out for me because of 1 key feature - DNS. Consul provides a DNS interface for service discovery, which enables non-Consul aware tools, like Home Assistant, to use Consul. For example, by Home Assistant instance depends on an MQTT server, which may be on a different node than the one Home Assistant is running on. While I can’t use dynamic ports, I can use the DNS interface for Consul as the host name for the MQTT server in Home Assistant. In my case, this means that in the eyes of Home Assistant, it can reach the MQTT Server at mqtt.service.seaview.consul.

Nomad for Orchestration

Nomad turned out to be the right balance complexity and resiliency, while not getting in my way too much with its relatively straightforward configuration DSL for jobs. Together with its support for CSI and CNIs it checked all the boxes, and was relatively easy to install and get going, given I had an already operational Consul Cluster.

CSIs

A lot less cool than Crime Scene Investigators, Container Storage Interfaces, provide a mechanism by which volumes can be mounted to containers. This is another case where having jobs potentially move across nodes presents an issue, as many containers are not stateless and need somewhere to persist their state.

In preparation for the move to Nomad, I replaced the OS on my NAS with TrueNAS Scale, both because of some reliability and connectivity issues I was having with Unraid, but also because of the first-class support TrueNas had for CSIs. In my case, I ended up using the democratic plugin to enable automatic NFS share creation and management. By connecting the plugin with the TrueNAS API, I am able to create, update, and delete shares from within Nomad, super convenient.

CNIs

CNI (Container Network Interfaces) allow for adding custom networking interfaces to Docker Containers. This is super convenient if the default bridge mode doesn’t work for your use case. For me, I like running Home Assistant on its own IP address since it has a bunch of different ports it uses for all the various integrations and those can clash with ports used by other containers. The macvlan plugin allows for this (assuming your NIC allows for advertising multiple MAC Addresses). However, because of some fun Linux networking issues that I don’t fully understand, containers using ipvlan or macvlan cannot access the host via its IP address. This posed a problem since I needed to communicate with the Consul Agent on the host to resolve Service Discovery DNS Requests. Not a problem, I would use a plugin called Multus to connect both a macvlan and bridge interface to the container.

This didn’t work. I spent hours troubleshooting issues, ranging from macvlan IP address not being released, to no requests making it to/from the container at all. Some more googling showed that there were a number of open issues with CNI support and Nomad dating back to 2021 that had clearly not been fixed. While not ideal, I could work around this problem. Rather than having the containers talk to their host for DNS resolution, I deployed a container to my router that augmented a DNS Server with a Consul agent to enable the containers to get their answers that way.

tpaulus/consul-core-dns

CoreDNS Container also running a Consul Agent to serve Consul DNS Requests

tpaulus GitHub

Terraform for Config Management

Since the goal of this project was to reduce the manual configuration, I opted to manage my Nomad Job configuration via Terraform. This not only allows my to quickly re-provision all of my jobs, it allows for automated image updates, which we will discuss in a bit.

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Cloudflare Tunnels

While my partner may disagree, I don’t run a data center in our garage and we have a single non-static public IP address for the whole house. The solution here comes in the form of a tunnel, specifically Cloudflare Tunnels. I created a system job in Nomad to run a Cloudflare Tunnel on every server. This gave me 3 different tunnels, pointing to the same services. While I could use DNS round-robin to route traffic between the servers, that would cause issues if any single server went down, since 1/3 of requests would fail. Using a Cloudflare Load Balancer solves this and allows for more intelligent routing of requests. In my case, I am simply using the health-check option to conditionally send traffic to each of the servers, ensuring that only healthy servers receive requests.

Foreshadowing

One afternoon, I came into the garage to hear an unusual clicking noise coming from the rack. Some investigation yielded that the CPU Fan on the NUC had died. It was time to take advantage of the high-availability infrastructure I had spent so many hours configuring.

I shut down the NUC and moved the Zigbee Dongle to another server and that was that. Nomad moved the allocations (jobs) from the NUC and scheduled them on another server in the rack. It was wonderful to see how quick and easy it was, especially given all of the headaches this failure would have caused before my Nomad adventure.

Optimizing Docker Image Upgrades

Back when I was hand managing container configurations on a single node, I would use Watchtower to automatically upgrade my Docker Containers to use the latest images. This approach doesn’t work when using Nomad though since it expects the docker configuration to be a certain way, and if the allocation were to get rescheduled to a different node, the updated image tag would not stick, resulting in an older image version being used until Watchtower updated it again (not ideal).

Renovate to the Rescue

My first thought was to build a tool to do this myself, but that would be a lot of work. My second thought was to clone the GitHub action that bumps the Image Tag in Dockerfiles, but again, that was going to be fair bit of work. We recently started using Renovate to update our dependencies (including Docker Images) at work, and got to thinking if I could adapt Renovate to do this for me. Some digging through the docs later, I found the Regex Manager, which enabled using Regex Expressions to parse files with a specified extension type and pass those to different dependency managers. A bit of cursing at Regex later, I ended up with the following rennovate.json which would look through all my .hcl files, find the docker images, and upgrade them if needed.

{
 "$schema": "https://docs.renovatebot.com/renovate-schema.json",
 "extends": [
 "config:base",
 ],
 "regexManagers": [
 {
 "fileMatch": ["[^.]+\\.hcl"],
 "matchStrings": [
 "image\\s?=\\s?\"(?<depName>.*?):(?<currentValue>.*?)(?:@(?<currentDigest>sha256:[a-f0-9]+))?\""
 ],
 "datasourceTemplate": "docker",
 "depTypeTemplate": "docker_image",
 "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
 }
 ]
}

And because I configured the integration with the Terraform Cloud, whenever I update the repository (like merging the Renovate PR), the changes are applied to the Nomad Cluster and the running container is replaced.

Conclusion

I’ve been running Nomad as my container orchestration system for the last few months and have been rather pleased with the performance and resilience. Servers reboot for updates and I never notice, what more could I ask for. I’ve still got some more work ahead of me, especially on the container networking front, as I’m not happy having a single point of failure on the DNS side of things, but that’s work for another day. For now, I can sleep better knowing my home services are resilient should a server fail again (or just need to reboot for a kernel update).

Replacing PiHole with Cloudflare Gateway

Sun, 30 Oct 2022 23:05:01 +0000

Before I can dive into how I configured Cloudflare Teams Gateway to work for my home, I need to give a bit of context and what lead up to this. I began using PiHole a number of years ago, running on a Raspberry Pi Model 1B. That worked fine for a while until my partner moved-in. She would report that the internet would occasionally stop working for her, and most of the time, this turned out to be a DNS issue. Combined with the occasional hick-ups caused by running on an older, low-power SBC, the need for occasional reboots to install updates meant that it was time to setup a second PiHole to run along side the first. Those Raspberry Pis were eventually replaced with Docker Containers running on much beefier machines, but the occasional outages persisted. Either due to fun intricacies of macvlan which is how the Docker Containers get their IPs on the network, or an over aggressive block list, it lead to both of us turning off WiFi when a page wouldn’t load. Don’t get me wrong, I love hosting my own infrastructure for all the things we need at home, but DNS is a fickle thing and issues with DNS can manifest themselves in the most peculiar, unsuspecting ways.

In a different thread, I started working for Cloudflare in April and quickly learned that a lot had changed since I became a Cloudflare User in 2014. More recently, Cloudflare has been expanding in the Zero-Trust space with tools and service aimed at making securing corporate infrastructure easier, while enabling work-from-home. One of these products is Gateway, which is part of the larger Zero Trust offering (and has a very generous free plan, which is what I use). Gateway DNS, which is what we’ll be discussing for the rest of this post, brings firewall style polices to DNS resolution, all while being wicked fast, which is critical for a DNS service.

So, how did I go about moving from PiHole to Gateway for my at home DNS resolution. Well, making the actual switch is as easy as updating your router’s DHCP DNS settings to inform clients of the Gateway DNS IP address, instead of the IP addresses of the PiHole containers. While that will get your DNS request off of PiHole, there is a bit of extra work needed to get similar levels of privacy and security from Gateway DNS.

First, a bit of house-keeping. Because there is a limited number of IPv4 addresses available (and they are in short supply), Gateway DNS re-uses the same IP addresses across many different customers (customizing the IP Address is an enterprise level feature), so Gateway DNS uses the source IP of the request to associate the incoming request with your account. For those with a static public IP address, it’s a set it and forget it affair, but for those who can’t get a static IP (or don’t want to pay $15/month for the privilege) we need to get a bit creative. I ended up writing this simple script which runs on one of my servers every 15 minutes to check my public IP and update it if necessary.

So, at this point we have Gateway DNS configured to always have the correct Public IP, but we haven’t established any rules yet. I’ve been exploring Terraform in my free time, in part because of how much we use it internally at Cloudflare to manage our configuration. While updating the Public IP via Terraform seems a bit overly complex since it can change often, using Terraform to define the Gateway Policies seemed like a perfect fit. That’s a long, round about way of saying, I defined the Gateway Policies in Terraform.

tpaulus/terraform-cloudflare

tpaulus GitHub

You’ll want to configure the rules for your Gateway based off of your needs, but for me, I blocked content categories identified as security risks (i.e., New Domains, Parked Domains, Malware, Phishing) as well as any traffic destine for the TOR Network. Additionally, I maintain a manual list of Trackers and Advertisers I don’t want to see and block them by domain name. Cloudflare’s domain classification is pretty good, but the manual list allows me to quickly block a domain without updating the policies to include that specific hostname.

With all that configuration complete and the logging tuned, we can actually update the router configuration to route DNS requests to Gateway. That covers requests coming from you home, but the awesome part of all of this is these protections and configurations can move with me via the Cloudflare 1.1.1.1 app (a.k.a. WARP), Cloudflare’s take on a VPN. By enrolling the device in my Zero Trust account, these same Gateway policies will apply to requests made from those devices, regardless of where they are in the world, as long as WARP is enabled. WARP also enables some really slick split tunneling that leverages Cloudflare Tunnels (f.k.a Argo Tunnels) which allow me to access my home network from anywhere, without using port-forwarding on my home router.

Cloudflare Gateway is not designed to be an ad and tracker blocker like PiHole and it would take significant effort to configure Cloudflare Gateway to block 158,614 domains sourced from the known ad-lists. But so far, this replacement does an all right job, while ensuring DNS is highly available for my home network without too much hassle, with the bonus of being able to take it with me, wherever I go.

Breaking up with Logi Pop

Sun, 01 Nov 2020 20:48:59 +0000

I got my first set of Logitech Pop buttons (Small BT LE Buttons that interface with a WiFi Hub) in December 2017 and they have moved with me from college dorm room, to dorm room, finally making their way on the wall next to my bed in my apartment now where they control the various lights not only in my bedroom, but in the entire apartment. Ever since the beginning, I have leveraged the convenience of these tiny, reliable (I have yet to need to change the batteries in over 3 years of clicking) via IFTTT. However, with the recent announcement of IFTTT moving to a subscription model, I needed to look elsewhere. While all of the lights in my bedroom are Philips Hue bulbs, many of the other smart devices I have in the rest of my place are not, leading me to use a central automation tool (Home Assistant in my case) to manage everything.

Logitech Pop Button (Left) next to a Xiomi Aquara Button (Right).

The Logitech Pop bridge supports a number of integrations, including Philips Hue, IFTTT, Smart Things, etc.; but what they do not support is local access, meaning that while getting bridge to talk to Home Assistant is theoretically possible using an emulated-hue bridge and a lot of time, it just is not feasible in my case. As such, I am parting ways with my beloved Pop buttons (oh, how beautiful they are), and replacing them.

There are a number of IoT clicky things out there, one of the most notable being Flic. However, not only is a set of Flic buttons incredibly pricy (a starter kit with 2 buttons costs $159.99 at the time of writing), they have many of the same pitfalls, although offering a Home Assistant Integration, it relies on bypassing the hub and using the bluetooth module on the host running Home Assistant. For some context, I run Home Assistant on an Intel NUC via docker, so Bluetooth device access is possible, but again not practice, plus I have some concerns about the Bluetooth range from where the NUC is, to where the buttons would be. Some googling lead me to this Reddit Post on r/homebridge. This post and its comments, along with a lot more googling (free time is in great abundance this year), I landed on the Xiomi Aquara Buttons, which run about $18 on Amazon. These are ZigBee buttons, which means that they require a compatible hub of some sorts. I didn’t think anything of this though, as I already had a working VeraPlus Hub, which among other things supports both ZigBee and Z-Wave devices. So I ordered 3 to replace my small family of Pop buttons and waited.

When the buttons arrived, I was quite pleased with their design and build. Yes, nowhere close to as thin and sexy as the Logitech Pop buttons, but for the price, I was more than happy to put up with the slightly more clunky build, if I could get them to work. Keyword being if. After a fair bit of fiddling, I got them to associate to the Vera, resetting them, and then pushing the “Effective Range Test Button” every few seconds while the Vera was in search mode eventually got them to associate. However, Vera does not officially support Aquara devices, so they showed up as Generic IO devices.

Vera Dashboard showing “Generic ZigBee device”

There were a fair number of people trying to get Aquara devices to work on Vera’s Community, but the post has been inactive for 9 months no, with only some success getting the Temperature sensor to work. Xiomi has a hub that you can plug in and add to your network, but given the news on Xiomi’s data collection practices, I was hesitant to add one of their devices to my home network, even if it was on an isolated VLAN. On top of that, I generally try to avoid vendor lock-in wherever possible, especially in the IoT space, where the fall of a company can lead to a very hungry cat, as we saw with Petnet. So, I did some more googling (again, free time is not hard to come by in the midst of a pandemic), and stumbled upon ZigBee2MQTT.

I had used MQTT before, when I built a ESP8266-powered, internet-connected heart light, so I was aware of how quick and powerful it was. So, I did some reading and ended up ordering a zig-a-zig-ah from Electrolama on Tindie, wanting to avoid the frustrations of flashing a CC2531 USB sniffer, and wait a short eternity for some very reliable parts to arrive from AliExpress. So, I waited.

I had some hardware issues with the adapter that I originally received, but Omer from Electrolama was very quick to respond to my concerns and before I knew it I had a new adapter in my hands. From there, it was almost plug and play. For compliance reasons, the adapter ships with a simple blink program, but there is some excellent documentation on the Electrolama website that details how to flash the correct firmware for ZigBee2MQTT on to the device, which is accomplished with some python libraries which are installed via pip, and a python script to handle the flashing. Once flashed, the zig-a-zig-ah shows up as a USB serial device, so no extra drivers were needed (I am using Ubuntu Server 20.04 LTS, your milage may very). From there, I was on the home stretch. Because I already use Docker for most of my needs on my home server, I went with the Container version of ZigBee2MQTT.

docker run \
 -d \
 --name="zigbee2mqtt" \
 -v /etc/docker/zigbee2mqtt/data:/app/data \
 --device=/dev/ttyUSB0 \
 -e TZ=America/Los_Angeles \
 --restart=always \
 --network host \
 koenkk/zigbee2mqtt

On first run, ZigBee2MQTT creates the default configuration file, which requires a slight bit of modification before ZigBee2MQTT can be started successfully. Namely, I needed to change the Serial device it was trying to use, and I reconfigured the Network Key while I was here and hadn’t paired any devices.

homeassistant: true
permit_join: false
mqtt:
 base_topic: zigbee2mqtt
 server: 'mqtt://127.0.0.1'
serial:
 port: /dev/ttyUSB0
 adapter: zstack
advanced:
 rtscts: false
 network_key: GENERATE

I was already running a MQTT server that I had already wired in with Home Assistant, but that too is relativity trivial. Home Assistant makes everything pretty easy, especially with their Discovery Functionality. ZigBee2MQTT includes some very helpful configuration cards that you can include in your dashboard, which I included via the Package Method described on the documentation. I also added the Network Map Card, which all together looks something like this once I got all of my devices joined.

ZigBee Network Information in Home Assistant Lovelace

I found the provided card to do everything I needed it to do. All that was left for me to do was to configure the automations that I used to have configured in the Pop software for interfacing directly with Hue and the Automations that I had in IFTTT. On each side of my bed, I have a button that I use to control the Hue bulb that is directly above it, tapping once for a night light, twice for a reading light, and holding to turn off all the lights in the apartment. The last one was the easiest; however the previous two were much harder due to some intricacies of how Home Assistant handles light.toggle actions. Specifically, you can set the color and brightness of a light with light.toggle, but you call light.toggle again with a different color or brightness, rather than updating the color, it turns off the light. So after some searching and thinking, I came up with this automation.

- id: '1601945836861'
 alias: Bedroom - Button - Right - Reading Light
 description: ''
 trigger:
 - platform: device
 domain: mqtt
 device_id: 98de24c1076711eb9eb06778c10b3ef9
 type: action
 subtype: double
 discovery_id: 0x00158d00027bf559 action_double
 condition: []
 action:
 - choose:
 - conditions:
 - condition: template
 value_template: '{{ state_attr(''light.hue_bedroom_bed_right_light'', ''color'')
 == {''x'': 0.447, ''y'': 0.406} and is_state(''light.hue_bedroom_bed_right_light'',
 ''on'')}}'
 sequence:
 - service: light.turn_off
 data:
 transition: 1
 entity_id: light.hue_bedroom_bed_right_light
 default:
 - service: light.turn_on
 data:
 profile: reading
 brightness: 254
 transition: 1
 entity_id: light.hue_bedroom_bed_right_light
 mode: single

I have 4 similar instances of this automation, one for the night light and one for the reading light, for each side of the bed. The differences being the colors/brightnesses and the devices respectively. Specifically what this automation does, is it checks if the light is in the desired state already (both in terms of color and state) and if it is, turns off the light, if it isn’t, it means that the light is in a different state and the user is requesting a state change, which is achieved via a call to light.turn_on with the desired parameters.

All in all, once I got everything working (which wasn’t too difficult honestly), things have worked great. The buttons are much more responsive, especially for the automations that used to leverage IFTTT. I have also already made some more additions to my ZigBee network, like moving my Hue Bulbs and remotes over to ZigBee2Mqtt and getting a SmartThings Motion Sensor and Door Contact Sensor to replace a similar ZWave set. From my small sample size, ZigBee is much more responsive than ZWave, although that may also be because of the setup I was using for ZWave (using a VeraPlus connected via Home Assistant) was less than ideal. I must give credit to the ZigBee2MQTT team and community for making a reliable, easy to use, and well supported piece of software that makes integrating these physical devices into our Home Automation Systems so easy (and dare I say it, fun).

A Xiomi Aquara button mounted on my bed side.

Project Heart

Sat, 04 Aug 2018 02:35:57 +0000

Long Distance Relationships are tough, there is no denying that. But this isn’t a post about the woes of love, rather a little project to help bridge the distance. Project Heart is a light that is controlled by your partner, regardless of where they are in the world, as long as they have an internet connection, they can light up your light.

Making use of the Adafruit ESP8266 HUZZAH breakout, we can connect a series of NeoPixel LEDs to the network without too much hassle. The basic premise involves the ESP8266 running a simple Web Server that is listening to request on port 80, specifically POST requests with a plain text body of either pulse or chase to run an animation sequence on the LEDs.

Software

Programming the ESP8266 with the Arduino software is relatively straightforward after you have set it up. However, keep in mind that you will need a USB to Serial Adapter if you are using the specified Breakout. The Adafruit Feather HUZZAH has this built in, but it a little more expensive. Instructions for setting up the Arduino IDE to program the ESP8266 can be found on the Adafruit Learning System.

Allowing the lights to be controlled from anywhere, not just inside of your home network, presents another fun, but easily solved challenge. All of this is done in your router’s settings (typically found in the Advanced Section). First we will want to assign a static IP to the HUZZAH to allow ensure that the device’s internal IP remains the same. Next, you will want to set up Port Forwarding to allow the outside world to talk to your device.

Note: Most ISPs block common ports like 80, 443, 8080, so some trial and error may be necessary to find an open port that will work for you. Any open port will suffice. Most routers allow you to map a different external and internal port, which allows us to run the HUZZAH on port 80 for convenience.

Finally, since most ISPs use DHCP themselves to assign your home an IP address, we will want to use a service like NoIP to ensure that we can always reach our home network, even if the ISP decides to issue our Modem a new IP address. How this is setup depends on your router, and the NoIP site has setup instructions for all major manufactures. So, at this point, we should have a URL which we can use to access our HUZZAH from anywhere in the world. Mine looks something like this: http://example.hopto.org:5000

You can test out your configuration using an HTTP request tool like Postman, but that’s not very convenient. That’s where we can use a service like IFTTT to make our life easier. By combining the button trigger and the Webhook service we can use a widget in the iOS notification center to trigger the light in a matter of seconds.

IFTTT Widget Settings

You can then add the IFTTT Widget to your Today View to access the animation controls quickly and easily.

Hardware

The project is pretty simple from a hardware perspective, with a short bill of materials, especially if you have some of the tools already lying around.

The capacitor is optional and provides some power smoothing, which is especially useful for very flashy (high current draw) animations like the pulse. When things are all set and done, mount the board and leds how you see fit, Hot Glue is one of my go to favorites for things like this.

Bill of Materials

Adafruit HUZZAH ESP8266 Breakout
FTDI Serial TTL-232 USB Cable (Required for programming - although any USB to Serial adapter will due)
USB Micro-B Breakout Board
Side Light NeoPixel LED PCB Bar
Stranded-Core wire
Micro USB Cable
USB Power Supply (1A should be sufficient for the default animations)

All of the code for this project is on GitHub and is free for you to use and modify to your heart’s content: https://github.com/tpaulus/Project-Heart

Daily White Card

Mon, 11 Sep 2017 05:53:50 +0000

Cards Against Humanity is a College Favorite and often leads to some wonderful, even though somewhat disturbing, conversation. Through a strange series of events, a few of the staff of Maya Olmeca at the Front Desk ended up talking about the various White Cards¹ and how some of them would be quite funny to see in real life, like “Vigiorius Jazz Hands”.

Aaron (RA for the Visual and Performing Arts Residential Learning Community in Maya) had the awesome idea of making a competition to find White Cards in real life. The rules would be simple, take a selfie with showing what every the White Card described.

With a little bit of Python (~65 lines), GroupMe’s super simple API, and the work of Chris Hallberg, I put together a simple bot that posts in a GroupMe² group every morning at 8 am with a Random White Card.

How it Works

Using JSON Against Humanity I generated a JSON object with all of the cards from the Original CAH Decks (The Original Set, The 6 expansions, and the Green Box), which totaled 1159 white cards. These cards are loaded into a python dict object, from which one random card is selected whenever the program is executed. This card is then given some context, like “Today you are looking for…” and the text of the White Card. This is then posted to the GroupMe Group via the Bot that was created specifically for this purpose on GroupMe’s Developer portal.

# Message Context
MESSAGES = ['Today you are looking for... %s',
 'Can you find... %s',
 'How about... %s',
 '%s Find it!',
 '%s DO YOUR WORST!',
 '%s']

# Pick a Random White Card & Message Context
secure_random = random.SystemRandom()
card = secure_random.choice(Cards.get_white_cards())
message = secure_random.choice(MESSAGES)

# Post the Card text to GroupMe
GroupMe.post_message(message % card)

Scheduling is accomplished via Heroku’s Scheduler Add-on and is run for free on one of their Dynos.

Taking only an evening to put together, I’m excited to see what “interesting” results this challenge will have. The Full Source Code is available on GitHub and is licensed under MIT.

https://github.com/tpaulus/daily-white-card

Cover Photo: By tombullock (Cards Against Humanity) [CC BY 2.0], via Wikimedia Commons

White Cards are typically used to fill in the blanks for Black Cards and contain a multitude of various phrases, ranging from the simple (“God”) to the complex and disturbing. ↩︎
We use GroupMe across the department for communication, and it allows for super simple and easy photo sharing, which is why we used it for this task. Plus, everyone on our staff already has the app installed. ↩︎

Backing Up with Amazon Glacier

Thu, 05 Jan 2017 01:13:00 +0000

Amazon Glacier is a long-term, low-cost data archival and backup service offered by AWS. It is surprisingly cheap at only $0.004¹ per GB per month, it has some caveats though. Mainly, retrievals can take upto 5 hours, and archives must be stored for a minimum of 3 months².

Glacier is available in most, if not all, of their data centers, but I will be using the US East (N. Virginia) datacenter, since that is where my Lightsail instances are hosted, and inter-datacenter transmissions are free. If you are using Lightsail, make sure to turn on VPC Peering; this should facilitate communication between your LightSail VPS and the Glacier system.

Since I want to use my storage space most efficiently, I will only be backing up the “non-recoverable” information I have saved on my server. This includes, database dumps, images, configuration files; however, does not include Docker Images, etc. My logic behind this is, that in the event of a failure, I can easily pull a docker image from the repository, or reinstall a tool via apt, so I do not need to save a copy. Also, since archives will be saved for an extended period of time, depending on your desired backup frequency, saving a few hundred megabytes can add up over a few months of backups.

Background on Glacier

While the AWS management console allows you to do some work with Glacier, most interactions will occur via a CLI or API requests. Thankfully, Glacier is a relatively popular service, so other developers have already written scripts and tools to help us work with Glacier. AWS has also written a well documented SDK for Glacier as well.

Glacier treats every file that is uploaded as an non-mutable archive, be it a photo, video, or in our case a ZIP file. Each file is assigned a unique ID which can be used to identify the archive in Glacier.

Since our Archives will likely be larger than 100MB, we will be using a Multipart upload, which allows the upload of a single larger archive to be chunked and parallelized. It also allows for easy recovery if something goes wrong while uploading a chunk, as only that chunk needs to be reuploaded, not the entire archive. The Multipart Upload process is discussed in detail on the Developer Guide for Glacier: http://docs.aws.amazon.com/amazonglacier/latest/dev/uploading-archive-mpu.html.

Configure Glacier

Before we get started on our backup system, we have to do some work in the AWS Management console. Before you start, make sure that you have set the correct region in AWS for where you want your Glacier Vault to be.

IAM

We will first create an IAM user with Programmatic access which we will use to authenticate our backup agent with AWS. Next, we will attach the AmazonGlacierFullAccess policy to our user, this will allow our user to upload archives to our Vault. Take note of the Access Key and Secret key for the user you just created, as you will need to save them in a config file a little later. This is the only time you will see these, however you will be able to create additional keys later if necessary.

Glacier

Navigate to the Glacier service in the management console, and click get started. The region that you chose earlier should be listed, as well as a prompt for the nave of your vault. The name of your vault must be unique across the region! You can setup SES notifications if you like about Vault Statuses, but you can skip that safely if you like.

Configure System to be Backedup

Clone the GitHub Repository, which contains the uploader.jar, backup script, and aws.properties template via git clone https://github.com/tpaulus/glacier-backup.git. You may need to make the backup script and the uploader jar executable, which can be achieved by cding into the folder and executing chmod +x *.sh *.jar.

Configure Backup Script

Once you have cloned the repo, modify aws.properties with the accessKey and secretKey that you took note of when you crated your IAM user. Next update the fields at the top of in backup.sh with the correct values. The correct endpoint url can be found at: http://docs.aws.amazon.com/general/latest/gr/rande.html#glacier_region. You should use HTTPS wherever possible.

The script assumes that you have your MySQL running in a docker container, whose name is a variable defined at the top of the script. Modify this section, or the other sections to suite your needs. If you are trying things out, set SANDBOX to true to disable the uploader and cleanup portions of the script. This prevents the compressed archive and backup folder from being deleted, so you can check if everything worked as desired, without having to deal with Glacier.

Running the Script

Before you can run the script, you may need to install Java, you can check if Java is installed by running java -version in the command line. If an error is returned, you need to install Java. On Ubuntu, this can be achieved by running sudo apt-get install openjdk-8-jre-headless.

Once you have verified that the script works as desired, you will want to add it as a cron job to run regularly.

Cron Job

Lastly, we will want to have our backup run automatically on a schedule, which can be easily achieved with the help of CRON. CRON allows you to input when you want a certain command or task run, and it will take care of the rest for you automatically. To help generate your crontab line, tools like Crontab Generator can take out most of the guess work and frustration. For the command, you will want to cd into the directory to which you cloned the repository, and then execute the script. My finished crontab looks something like this:

0 12 * * * cd /usr/share/glacier-backup; ./backup.sh >> /var/log/glacier-backup.log 2>&1

I decided to have my script run at 12PM (UTC) everyday, since that is a time when I expect not to make any changes to the server. If you are unsure of the timezone of your server, run date to find out the current date, time, and timezone of your system.

Now run sudo crontab -e to add the line you got from the Crontab Generator to your local crontab file. The sudo is necessary, as we want our script to be run by the root user to bypass any permissions issues.

Save and exit, and you should see crontab: installing new crontab. This means that your line was added successfully to the crontab for the root user.

That’s it, now you can rest assured that if something should go wrong, you have a recent backup of your essential files and databases that are safely stored offsite, ready for your retrieval within 5 hours³.

Recovery Procedure

In the event that you should need to use one of the Backups from your Glacier Vault, you will first need to retrieve the Inventory from the Glacier System. Setting the environment variables as shown below will make our lives a little easier as we complete the steps necessary to recover an archive.

Additionally, the steps listed below use the same Command Line tool as we used above to send our Backup to Glacier, so ensure that the commands below are being executed in the same directory as the backup.sh file. This is done to make sure that the aws.properties file is located in the correct place in relation to the JAR file being executed to interface with Glacier.

Set Common Environment Variables

The values for most of these variables will be the same as those found at the top of the backup.sh file. They are not in a shell file, however, because this process can take some time to execute⁴.

export CREDENTIALS="aws.properties"
export ENDPOINT="https://glacier.us-east-1.amazonaws.com"
export VAULT="myvault"

List Inventory

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --list-inventory

This will give you a job ID for the inventory listing, make note of this, as you will need it to access the result.

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --list-inventory yourjobidfromthepreviousstep

Be sure to include the jobid from the previous step in the above command. You will receive an error massage (status 400) until the inventory has been generated for you to retrieve. This can take about 4 hours, depending on Vault Size and Demand, according to the Amazon Developer Documentation.

Download Archive

Once you have found the desired archive that you would like to retrieve from your inventory, making note of its Archive ID, you can request it via:

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --download myarchiveid --target path/to/filename.zip

Make sure to change the Archive ID that you want to download to one in the inventory, and the target to the location you want the archive saved.

Note: In order for you to retrieve the Archive, your IAM User that you have configured (see above) will need access to the Amazon SQS and SNS Service. This is a separate permission than the one granted for Write Access to Glacier.

If you are getting errors in regard to the “current data retrieval policy,” have a look in your Vault General Settings, accessed via the Management Console, and make sure that you allow for enough data to be downloaded per hour. You should set the cap a little above the estimated size of the archive that you want to retrieve. The higher you set your retrieval cap, the more you can be charged. You are only charged for retrieval, this is not a monthly fixed charge.

That’s It. While retrievals can be time intensive, they significantly lower cost makes it worth it for most users. If you find yourself making a significant number of retrievals, you may want to consider configuring Expedited Retrievals via provisioned capacity, or moving to S3 for Hot-Storage, rather than Glacier, which is considered Cold-Storage.

At time of writing, each GB of Glacier storage in Region: US-East-1 (N. Virginia) cost $0.004. Updated pricing can be found at https://aws.amazon.com/glacier/pricing/ ↩︎
While archives can be deleted at anytime, the documentation states that you will be charged a prorated fee for the remaining duration. https://aws.amazon.com/glacier/faqs/ ↩︎
Archive retrieval times depend on the archive size and retrieval option selected. More information can be found in the Glacier Developer documentation: https://docs.aws.amazon.com/amazonglacier/latest/dev/getting-started-download-archive.html. ↩︎
While this process can be automated with the help of Amazon SNS and an Amazon SQS queue, that can be a bit more complex to setup and configure. If you will be frequently be retrieving archives (at which point you should re-consider your use of Glacier) it may be advantageous to look into this: https://docs.aws.amazon.com/amazonglacier/latest/dev/retrieving-vault-inventory-java.html ↩︎

Installing Fish Terminal on MacOS

Fri, 23 Sep 2016 20:23:00 +0000

Fish is a great alternative to the default bash or shell that is installed on your computer by default, while Fish is available for a variety of platforms, even Windows (with Cygwin); but this guide will focus on macOS, because that is what I use on a daily basis. It is very easily to install, since they provide a package that you simply download and run and it installs fish to /usr/local/bin. However, if you are a bit more savvy and already have Homebrew installed, you can install fish with a single command in your currently very boring shell.

brew install fish

After the install completes, you will need to allow/whitelist the fish shell to be used, and make it your default shell, unless you want to type fish every time you start a session.

sudo echo '/usr/local/bin/fish' >> /etc/shells chsh -s /usr/local/bin/fish

chsh -s /usr/local/bin/fish

Now that you have installed, whitelisted, and made it your default shell, you are probably wondering why fish is so cool. Fish’s power lies in its expansibility, especially in the awesome things that other fish enthusiasts have already created. One of these, that is likely already installed is Tackle. To use it, you will likely need to add a few extra dependencies, which can be easily done via another brew install command, and an install script. This script can be run implicitly, but you should understand what the script does before doing so, else bad things could happen. Justin has documented the Tackelbox framework and the code very thoroughly, and I recommend checking out the framework and the install code: https://github.com/justinmayer/tacklebox

brew install vcprompt grc

curl -L https://raw.githubusercontent.com/justinmayer/tacklebox/master/tools/install.fish | fish

Tackle and Tacklebox requires some configuration to work, but the config is easily modified as it is stored in a hidden file in your home folder, under ~/.config/fish/config.fish. I have included my config file, adds extra colors and functionality with tools like git, pip, etc.

# Paths to your tackle
set tacklebox_path ~/.tackle ~/.tacklebox
# Theme
set tacklebox_theme entropy
# Which modules would you like to load? (modules can be found in ~/.tackle/modules/*)
# Custom modules may be added to ~/.tacklebox/modules/
# Example format: set tacklebox_modules virtualfish virtualhooks
set tacklebox_modules virtualfish virtualhooks
set tacklebox_plugins extract grc pip python up
# Which plugins would you like to enable? (plugins can be found in ~/.tackle/plugins/*)
# Custom plugins may be added to ~/.tacklebox/plugins/
# Example format: set tacklebox_plugins python extract
# Load Tacklebox configuration
. ~/.tacklebox/tacklebox.fish

If you have issues, the Fish Shell Website has some great examples and documentation: https://fishshell.com/.

Quiver, My New Favorite Note Taking App

Mon, 04 Jul 2016 21:40:53 +0000

Evernote was the note taking application was the most popular option for a number of years and was the least bad of the options out there. However, recently Evernote has been promoting their paid plans in a way the hurts their free users. Removing popular features, like emailing in notes (1), and imposing extra restrictions, like limiting you to only 2 devices (2), made me reconsider if Evernote was really the best option out there. Popular alternatives to Evernote at the time of writing include: Simplenote, and Microsoft’s OneNote. I like to avoid Microsoft wherever possible, and Simplenotes offerings were average at best, and the reviews did not help sell me on their product.

I have quite a few notes in Evernote. Thankfully, Quiver can import notes in the Evernote .enex format.

The advantage to products like OneNote, Evernote, and Simplenote is that they offer Mobile apps and Syncing across devices. I rarely find myself editing, or creating notes on my device, but I would like to be able to access notes on the go, just in case (usually for reference purposes). The Apple App Store features Quiver as one of their “Apps for Developers” and since starting college, I have used Evernote primarily to take notes in class (many of which are Computer Science Courses). If you have ever tried formatting a note in Evernote, you know the nightmare that it is. And while apps like Alternote make it a little easier, they still don’t make it easy. Quiver fixes all of their with their innovative approach, which they call cells. Cells are different areas of your note that can be formated as text, code (with syntax highlighting for a bunch of languages), Markdown, LaTeX, or Diagram; all of which are rendered beautifully by the App. The app even supports importing Evernote Notebooks! The only downside for me is that there is no sync engine, or way to view my notes on the go (either on a mobile device or in the cloud).

In order to take my notes on the go, I took advantage of their export script that converts entire notebooks of notes into HTML folders. The files themselves are hosted on my webserver and are displayed via a beautiful jQuery, PHP file explorer from Tutorialzine. While the Quiver Notebook Library is saved in Google Drive (mostly for backups and safe keeping), I am using the very powerful rsync command to sync the library between my mac and the web server.

Server Setup

Set up a folder (with password protection if you like) with your web server of choice (mine is NGINX) and copy the files from the repo into the folder. Take note of the path of the folder since you will need to enter it into the sync script that we will install on the mac.

Download the File Explorer and Assets from GitHub.

Install the quiver2html node module. npm install -g quiver2html
You will want to set up Apache, NGINX, or your web server of choice to serve the PHP content in the folder you cloned the server files into. Google is your friend here.

When you are done, navigate to the URL for your folder and you should see something like this.

Mac Setup

We need to configure the computer to sync the Notebook Library automatically and request that the server convert the library into HTML. This is achieved with a simple shell script that is run every 5 minutes by launchctl, Apple’s version of CRON on newer OSs. Launchctl uses specialized plist files (which can be created with launched) to tell it when to run what command.

Download the Shell Script and Plist File from GitHub.

Update the script’s variables to match your installation.

QUIVERLIB_LOC="Path/to/Quiver.qvlibrary"
SERVER="lorien" # Host Name of your server, as set in .ssh/config
SRV_DEST="/srv/notes/" # Public HTTP Directory where Notebooks folder is

Copy the shell script to your local/bin folder.

sudo cp quiver_sync.sh /usr/local/bin/quiverSync
sudo chmod +x /usr/local/bin/quiverSync

The advantage with placing the file in your local bin folder is that you can call “quiverSync” at anytime from your command line to toggle a sync. Now would be a good time to test the script to see if it works before continuing. Take a look at the website and you should see something like this.

If everything is working the way it should, place the launchctl script into the Launch Agents folder. If the ~/Library/LaunchAgents directory does not yet exists, you can create it with “mkdir”. There are several Launch Agent folders on your computer, the folder in your user library only runs when you are logged in, which is what we want, since you will likely not be making changes to your quiver notes while you are not logged in.

sudo cp com.whitestarsystems.quiver_sync.plist \
~/Library/LaunchAgents/com.whitestarsystems.quiver_sync.plist
launchctl load -w ~/Library/LaunchAgents/com.whitestarsystems.quiver_sync.plist

Conclusion

I think that Quiver, with the addition of the sync tool we implemented, will replace Evernote for me. It is an extremely powerful note-taking tool that won’t be charging you monthly per-device. It may not have as many features as the competition, but it serves its purpose for me very well.

Trinket Timer

Mon, 25 Aug 2014 07:15:15 +0000

On my last school free Sunday afternoon of the summer, I built a little project to help my Frugal Father.
My dad had bought a small timer for his DJI Phantom to monitor the flight time, a $6 investment straight from Mainland China. It quickly lived up to the stereotypical expectation of all Chinese products, it failed. The seven segment display began to malfunction, only displaying the tops of the numbers, rendering the timer useless. I felt bad for my dad, I use a Turnigy 9x with ER9X firmware which allows me to run a timer, which is triggered when I move the throttle stick.

I decided to put my last day off to good use; I pulled out my box of electronics parts, filled with mostly Adafruit parts. I found my favorite Seven Segment display, a .56″ White display with an I2C backpack. I had planned to use a Trinket, but quickly realized that I would need to use an Arduino to do the initial prototyping. The Arduino allows me to use the Serial Monitor, a key tool in debugging sketches. The Uno, my favorite of the Arduino boards, also has an auto-reset function which makes uploading sketches a breeze.

I used 2 buttons, I like Adafruit’s Colorful Square Tactile Button Switch, because they fit nicely in a breadboard and they add a dash of color to an otherwise colorless project. It was a relatively simple project to wire up, 10 jumper wires and 2 resistors. Things got a bit more interesting when it got to the code though.

I wanted to stay away from the delay function to increment the clock as it pauses the code entirely for the allowed time. I found a library by Simon Monk on the Arduino Playground. It was just what I needed. I then began to code. I wrote my code objectively, using the things that I had learned from Python to make my code more understandable and easier to implement and maintain. The Timer library allowed my to focus my attention to reading the buttons and updating the display with the correct time, not worrying that I was slowly getting out of sync. The Timer Library is like a timer, you tell it to do something every X milliseconds, or you can tell it to hold pin 10 high for X milliseconds, while your program does something completely different.

After working out the major bugs, I migrated to the Trinket and faced a few more problems. Specifically, I found that pin 3 constantly output 3.3 volts, which made reading the button that was connected to pin 3 impossible. The source of this problem is very clear, pins 3 and 4 are shared with the USB port which is used during programming. The problem was easily solved my moving the button to pin 1. To make your experience a bit smoother, I have rendered the breadboard with Frizzing, and the code is on GitHub.

View the Code on GitHub

Happy Timing!

The Morning Mailer

Fri, 01 Nov 2013 17:03:59 +0000

Every morning, I use the same 3 apps: Todoist -to find out what I have to do today, Weather Underground – will it be warm, hot, cold, and a news app – what’s happened while I was sleeping. I needed some app that did all 3 things for me, but after searching Google’s Play store, nothing suited my needs. I decided that I would design my own app.

The Morning Mailer is a Python script that runs on my server, tompaulus.com, and sends me an email, every week day at 6 in the morning. First, greeting me in a foreign language, and then telling me what the weather will be like. Utilizing Weather Underground‘s API I gain access to a concise text forecast for the day and a high-resolution, animated Radar map of my area. The greetings are stored on the server in a “.csv” file, and I can add/remove greetings simply, without having to modify the Python code.

I use Todoist to keep track on my schoolwork, like homework and upcoming tests and other events. Todoist’s API allows me to access all the pertinent data, making it simple to stay on-track.

The rest of the email contains the four top stories from the New York Times, including a picture and summary.

The information is parsed by Python, then injected into a beautiful HTML email, which was downloaded from MailChimp‘s Email Blueprint Github repository. Clone their repo here.

The documentation is available on the GitHub Page for this repo: http://tpaulus.github.io/MorningMailer/

Clone the repo: http://github.com/tpaulus/MorningMailer

Morning Mailer did its job well!