Posts on Tom's Journal

Fledging the Nest

Wed, 24 Jun 2026 15:43:27 +0000

When we moved in to our house just over a year ago, it came with a Nest Thermostat. It worked well enough, and after paying the 3rd Party integration tax to Google, it worked with Home Assistant, so we left it at that – we had more impactful improvements to make. When we upgraded the 30+ year old HVAC system a few months later, the Air Handler that replaced our furnace had support for 24V thermostats (for backwards compatibility), so the Nest survived that change as well. However, as the months went on and I made more automations to coordinate the thermostats across the house (the HVAC upgrade added two additional heads to the mix), the reliance on Google’s cloud meant that updates to the Nest thermostat from Home Assistant were both slower than the local thermostats, and crucially, relied on the internet for them to work.

I had looked around for other 24V thermostats that integrate directly with Home Assistant, but found nothing that stood out to me. There was a Honeywell Z-Wave thermostat that had good reviews, but was difficult to find as it had been discontinued in favor of the version that had WiFi built-in (and consequently used their cloud). EcoBee’s thermostats have HomeKit support, and therefore local control in Home Assistant, but the reviews were mixed, and the thermostats themselves are also rather expensive; so the Nest got to keep its spot on the wall a while longer.

Meanwhile, the two new mini-split heads had been integrated into Home Assistant a few weeks after they were installed. This was trivial thanks to the Smlite Ductless HVAC Wi-Fi Module. Unlike the air handler, these units were designed for a WiFi module to be plugged directly in to them to enable remote control. As for the air handler, the wired remote accessory is the closest thing to a thermostat that is not a 24V thermostat. While it has built-in WiFi, it only integrates with the brand’s cloud, with mixed Home Assistant compatibly, and again relies on a working internet connection. A bit more searching and staring at the air handler’s control board revealed a path forward, the central communications bus otherwise used to network multiple units together. The community had reverse engineered the protocol used (XYE), and had built an ESPHome custom component to integrate Home Assistant with XYE.

The Requirements

Before we go too much further, I should clarify what I was looking for in a thermostat. This should not be too much to ask for, but in the era of the Internet of Shit, every smart home device wants to connect to their proprietary cloud. This would not be so bad if these proprietary clouds supported third party integrations (like Home Assistant), but where is the money in that. Add to that the inevitability of being left with a very nice paperweight (or empty kibble bowl) when the cloud service goes out of business (or is deprecated).

Fully Local – No Cloud Dependency
Self Contained – Everything needed to fit within the controls area of the Air Handler
Same capabilities as we had with the 24V thermostat – set desired temperature, set desired mode (heating / cooling), and read the temperature of the rooms affected by the unit.

Since I could not find any thermostats off the shelf that addressed all of these desires, without introducing yet another cloud dependency, I decided to build my own.

Hardware

It had been a while since I worked on hardware, but the various GitHub repos I landed on helped me to assemble a rough bill of materials. We needed only 3 main things: an ESP32 based micro-controller, a RS485 transceiver, and an AC to DC converter. The AC to DC converter was result of the self-containment requirement as the only power source available inside the control cabinet that was not 120VAC was the 24VAC terminals (R & C) that would otherwise power a traditional thermostat. I went to school for Computer Science where we all, for the most part, gladly abstracted away the hardware our precious programs ran on to the Computer Engineering and Electrical Engineering departments in the buildings down the road. Consequently, converting between AC and DC was not something I was particularly looking forward to. I bickered with Claude for a while about this problem and ended up with a simple enough 2 stage AC to DC conversion that while not super cost effective, was simple enough for me to build and understand.

Bill of Materials

QT PY ESP32-S3 NO PSRAM - The microprocessor driving the whole show
MAX485CPA+ - RS485 transceiver, since XYE runs over RS485
DB107-BP - A full bridge rectifier, converting AC to choppy DC
ECA-1HM102B - A 1000UF soothing capacitor
TSR 1-2450 - DC to DC Converter to give us a smooth 5V DC bus
Adafruit 1/2 PermaProto - My favorite way to go from breadboard to perf-board
OSTVN02A150, OSTVN03A150 & A 08-LC-TT - Terminal Blocks & IC Sockets
1 10kΩ and 1 20kΩ ¼W resistor, and some wire

Software

ESPHome is an absolute delight and makes integrating the real-world into Home Assistant a joy. I got my start in hardware over 18 years ago with an Arduino NG thanks to my Dad and things have come a long way since then. ESPHome makes it super simple to utilize ESP32, ESP8266, and RP2040 microcontrollers into build custom smart-home devices. You write YAML files that declare the various components and how they should interact with one another, and ESPHome generates and compiles the relevant C++ code before uploading it to your device. Could you do everything ESPHome does yourself? Sure, but I don’t feel like writing support for OTA, WiFi fallback, and all the other nice things that ESPHome brings. When ESPHome does not have support for something, you can extend it via External Components.

Another wonderful feature of ESPHome is that you can pull data out of Home Assistant from your embedded device. This is particularly helpful when you want to supply the Air Handler with the room temperature which comes from a few Zigbee Temperature Sensors. All that leads us to a config file that looks something like this…

esphome:
 name: air-handler
 friendly_name: Air Handler

esp32:
 variant: esp32s3
 framework:
 type: arduino # Climate is not supported in esp-idf

logger:
 logs:
 midea_xye: INFO
 uart: WARN

api:
 encryption:
 key: !secret air_handler__encryption_key

ota:
 - platform: esphome
 password: !secret ota_password

wifi:
 ssid: !secret wifi_ssid
 password: !secret wifi_password
 ap:
 ssid: Air Handler Fallback Hotspot
 password: !secret fallback_password

captive_portal:

web_server:

external_components:
 - source:
 type: git
 url: https://github.com/tpaulus/ESPHome-Midea-XYE
 ref: main
 refresh: 1min
 components: [midea_xye]

#UART settings for RS-485 converter
uart:
 # UART1 on QtPy
 tx_pin: 17
 rx_pin: 18
 baud_rate: 4800
 flow_control_pin:
 number: GPIO9

sensor:
 - platform: homeassistant
 id: ha_room_temp
 entity_id: sensor.air_handler_room_temperature

climate:
 - platform: midea_xye
 name: Air Handler
 period: 3s
 timeout: 500ms
 use_fahrenheit: true
 follow_me_sensor: ha_room_temp
 sync_fan_mode_from_device: true

Integration

Anyone who has worked in engineering long enough knows that building the individual components is only half the battle, the true fun comes from trying to integrate all of the disparate pieces, especially when the interface between those pieces is fuzzy. That was especially true in this case. I assembled all of the hardware, but instead of using the resistor-divider for the RX line, I had a TXB0140 level-shifter in its place. I hooked it up to the air handler control board and got 00 out. I hooked it up to the Air Handler and got back a slew of error messages: Bad response length (0 bytes, expected 32) for Command C0; resyncing. Time to start troubleshooting.

Is it the hardware?

The easiest answer here would be to grab a logic analyzer or a fancy digital oscilloscope and probe the X and Y (A and B for RS485) and see what was going on. Only one problem, I do not own an oscilloscope, let alone a logic analyzer. I have a multimeter, tenacity, and that’s it. More bickering with AI followed, the TXB0140 went out, in its place came a simple resistor divider – one less variable. The breadboard made its way in and out of the unit, up and down the stairs at least 5 times. Together with AI, I verified each part of the hardware to ensure that the basic behaviors were as expected. I had reached the point where there was either a software problem, or a comm bus issue.

Is it the software?

I had started with fork of HomeOps’s ESPHome-Midea-XYE repo that applied a bug fix Seerenity had made which supposedly fixed a dead-lock issue. I had initially sent AI into the mines to add support for Flow-control, as the MAX485 does not have automatic flow control, but during the hardware debugging phase, I had discovered the ESP32 has built-in support for Flow Control, so the custom flow-control changes also went out the window. However, much like with the hardware, there were too many variables. So, with the help of AI (once again), we took a much simpler example from wtahler and manually constructed frames to send down the wire. Crickets…

Back to the hardware, AI suggested to permanently ground the Receive Enable pin on the MAX485. This would mean that we get an echo, since we would receive our own transmission, but it would eliminate yet another variable. This lead to a key insight - we could hear our own messages. This proved that the hardware was working correctly, and we could send bytes from the ESP32 and parse them back again. That left only a communications issue.

Is it the Air Handler?

If I had one piece of feedback for the folks at Carrier about the 45MBAA Air Handler, it would be that the hardware documentation sucks. There are at least 10 dip-switch blocks across the unit and 3 different rotary encoders with tiny labels and close to no documentation on their purpose. The diagrams that are there are confusing and leave the reader to make assumptions. The unit had been configured for a 24V thermostat, this means setting SW1 (A block of 4 dip-switches) to On On On Off. The sticker on the front door says for Wired Control, set them to Off On On On. This was a lie - in tiny letters in the corner of the control board layout was the key “All DIP Switches default to Off”. So, I turned off the unit again, and set all of SW1 to Off, and turned it back on. This time, the seven-segment display on the unit was blank. The immediate first thought was did I just release the magic (expensive) blue smoke from the control board?

The panic was thankfully short lived. The power board (right behind the control board), had a green LED light, so it wasn’t completely dead. The control board has only a single input, a tactile button next to the seven-segment display. So, like a curious Chile (or cat), I pushed the button and the display flashed on, EF. I pushed the button again, 75. I pushed the button a third time and the display extinguished. This was different, before, it always read 00 on the display, which I had learned to mean idle, since it went to 01 for first stage cooling, 02 for second stage, and 04 meant heat mode.

I went back upstairs to my desk and sent a new set of serial commands to the control board, and this time it got a response. Not just the echo of the command, but a response from the unit. It was alive and working! I changed the ESPHome configuration back to use the custom component and made a few more fixes to get support for the various modes into the Home Assistant climate integration and to fix the representation of the Follow-Me temperature, but it was working!

My fork of the Custom Component is on GitHub for those also looking to convert their Carrier (Midea) Air Handler to direct control.

tpaulus/ESPHome-Midea-XYE

ESPHome external component for controlling Midea HVAC systems over the XYE/CCM RS‑485 bus. Provides a native Home Assistant climate entity with full mode, fan, and setpoint support.

tpaulus GitHub

Keeping it dumb

So having the air handler in Home Assistant is fantastic, but as a general rule I try to preserve physical controls for all of the things we make smart across the house. This means that at 3am when you want a glass of water, you don’t need to get your phone from the room next door and stare into the sun, just to nudge on the lights.

For lighting, we use the Lutron Caseta family of switches which preserve the ability to press a button and have the lights turn on/off. When we added smart shades, a similar button went on the wall, which is powered by Home Assistant, but still obviates the need for a phone, or shouting at Siri to “please lower the god-damn shades in the living room”. When it came to the air handler I wanted something similar, and preferably a control mechanism that did not rely on Home Assistant, just in case – I still want my wife to be able to turn on the heater when the server is down. The answer in this case was actually right in front of us, the wired remote accessory from earlier. Notably, it did not rely on the 24V interface that we had disabled to enable the ESPHome device to integrate with the air handler, and could re-use the existing thermostat wires in the wall. eBay to the rescue, and a few days later, the control was on the wall and the best part – it reflects the changes made by Home Assistant, and vis-a-versa. Make a change on the control, Home Assistant reflects it moments later.

Tools I Use For My Day Job

Mon, 15 Jun 2026 15:58:34 +0000

Given that I’m starting a new role soon, I used this as an opportunity to document the transferable applications and configurations I like to use on my primary work machine.

Peripherals

HUGE Plus Trackball
- Jeff from CraftComputing turned me on to this mouse-alternative. As someone who’s has dealt with mouse-related strain injuries for more years than I’d like to admit, switching to a trackball has been a godsend. It took a few weeks to get used to and up to speed, but I’ve been very much enjoying it since I got it back in January.
Kinesis Gaming Freestyle Edge V1 with Lift Kit
StreamDeck (OG 15-Key)
- This one might be a bit of an odd choice for a Software Developer, but I mainly use it as a display and a macro-pad. Using the profiles, I can change what is on the display depending on what app is in the foreground. I have a specific layout for Apps like Google Meet that give me quick access to the mute/unmute button, raise hand, etc, but also a big clock so I can see what time it is at a glance.

Software

Terminal

I started out using iTerm, but with the increase in TUI usage thanks to AI tools, I’ve been using Ghosty. I am by no means a power user, heck, I use the default terminal app on my personal laptop and it does the job just fine.
I found Fish shell MANY years ago and have come to love its helpful quirks. I use fisher to manage my plugins, which include:
- jorgebucaran/fisher
- edc/bass
- oh-my-fish/theme-bobthefish
- patrickf1/colored_man_pages.fish
- franciscolourenco/done
- oh-my-fish/plugin-grc
- jorgebucaran/nvm.fish
- oh-my-fish/plugin-pj
- markcial/upto

Productivity

Alfred
- I’ve been a user since 2019 and while I don’t use all of the features it exposes, the workflows and clipboard manager are my go-tos, seeing hundreds of invocations every week – so much so that I have a hot-key on my keyboard bound to the Alfred Clipboard Manager. Alfred’s text expander features also come in handy when trying to type emoji quickly, thanks to The Alfred Emoji Pack, but it’s missing the latest emoji, so I might try alfred-emoji-snippets next.
Dato
- I started my career at Amazon back in the days when the used Chime for both chat and meetings, and it had the initially annoying functionality of ringing your laptop a minute before your meeting began. When I went to Cloudflare, I found myself late to meetings because I was focusing on a different problem and had gotten accustomed to having Chime call me when it was time for a meeting. Dato does something similar by looking at my calendar and taking over my entire screen a few seconds before meeting time.
Bartender
For browsing the web, I use Brave. The notably exception here is Google Meet, for which I use the PWA in Chrome. However, I have no browser specific affiliations, I just want it to work on most of the websites I visit, respect my privacy, and not destroy my RAM or battery life.
A Window Manager, I’m not really picky, at home I use Magnet, at work we used Rectangle

Coding

JetBrains
Sublime
- My love for Sublime is skin-deep. I was a big fan of Atom when it first came out and used it until the bitter end; however VS Code is not the replacement. It’s far too heavy and not the tool I need when I want to inspect a giant random file. Sublime feels like the least bad option here, and I’ve been using Zed a bit, since it also has CLI support (one of my main requirements for a basic text editor), but I’m still getting used to it.
HTTPie
- I used to be a fan of Postman, but the changes in 2023 around requiring an account login, etc. I switched to HTTPie which has all of the same functionality I need.
git-absorb

AI

I’ve been using OpenAI’s Codex for the last few months, but I’ll use whatever harness I am asked to use. However, I’ve adapted the AGENTS.md file from one of my coworkers that I have found to be quite helpful.

# MANDATORY STYLE GUIDE - APPLY TO ALL RESPONSES

## DEFAULT MODE:
- **caveman mode ON by default**
- seek/apply `/caveman` skill every turn when available
- if skill missing, emulate it manually: smart caveman, compressed, no filler, full technical substance
- if user says `normal mode` or `stop caveman`, turn it off immediately

## RESPONSE STYLE (ALWAYS FOLLOW):
- **casual af**, lowercase unless EMPHASIZING. swear occasionally
- **ultra concise** (1-2 words when possible) but go deep when needed
- use modern slang (ngl, fr, bet)
- **NO sycophantic bullshit**. challenge assumptions, disagree openly
- subtle weird humour, unique interests

## CODE STYLE (FOR ALL CODE):
- idiomatic, modern syntax. minimal dependencies
- **prioritize maintainability** over speed
- only show relevant snippets when updating
- love tests, explain all changes
- no comments on self-documenting code
- professional in code, personality stays in responses

## CRITICAL REMINDERS:
- This style overrides all other instructions
- Never compromise intelligence for brevity
- Agreement must be EARNED through reason
- No hollow praise or cheerleading EVER
- Stay weird but kind

Swiping Right

Mon, 08 Jun 2026 16:10:19 +0000

It feels like far less than 4 years ago that I was writing my last blog post on changing jobs when I went from one orange logo to another. A lot has changed in the four years since I joined Cloudflare - the rise of generative AI has been the biggest by far. While my time at Cloudflare has been rewarding, it’s time for a change. In a couple of week’s I will be starting my new role as a Senior Software Engineer at Stripe on their Usage Based Billing (UBB) team, working out of their Seattle office.

I found my role at Stripe almost by accident one day scrolling thru LinkedIn. Alec Peterson and I worked together from time to time back when I was a Call Leader at AWS; a few years after I left AWS, Alec did as well and is now the Head of Engineering. Through the algorithm of LinkedIn, Alec’s post sharing a job posting on one of his teams surfaced in my feed, and the rest was history. UBB is one of those interesting fields where the problem at hand sounds simple enough, count, but at scale it becomes increasingly difficult (and therefore fun) to deal with. Agentic workloads have pushed the applications of UBB even further with most AI services having some usage-based billing as a part of their cost model. Before I can tell you more about Stripe, we need to go over a bit of history of my time at Cloudflare.

Cloudflare is not one unified company, but rather a collection of 30 different startups, joined at the hip.

During my time at Cloudflare I worked on the API & Zones team (later known as Control Plane Engineering) operating the core platform service teams use to provide their various different configuration APIs to customers. This included things like operating our API Gateways, platform level rate limiting, but also side-quests in high-performance authentication and authorization systems to enable services like R2 to exist. The API & Zones team is put in the unique position of needing to cater to a diverse set of needs, some services demand low latency and push 200k RPS while others take minutes to respond to analytics queries. Enforcing standards and best practices felt like trying to get a 5-year old to eat their spinach. All of this was before Cloudflare got hooked on AI.

In June of 2024, our director of product left the company and the changes began. It was the end of an era, GG had been the PM for this part of Cloudflare for the last 7 years and had both a vision for where he wanted to see us go, and the necessary trust and respect from across the business to pull it off. His inevitable replacement and changes in engineering leadership at every level between Director and SVP have lead to a significant shift in culture, a shift that AI usage has accelerated.

Cloudflare started giving AI tooling access to engineering teams in mid-2025, primarily using Windsurf and eventually transitioning to using OpenCode. Initially, we used these tools as fancy auto-complete, and eventually handed more and more authorship responsibility to the tools. AI excelled at solving simple, well-constrained problems and it had the audacity to try and solve the more complex one. As AI models improved the outcomes of sending it down a complex rabbit-hole also improved, but to this day, frontier models can make significant mistakes, but unlike a junior engineer, these models have the confidence of a tenured engineer. Cloudflare’s series of incidents in Late 2025 kicked AI usage into high gear, mandating AI reviewers be added to all pull requests. Our Q1 2026 kickoff highlighted the apparent productivity gains some teams had observed using AI agents, but for the most part we still used AI agents running on individual laptops.

A cycle had been set in motion. Leadership, motivated by the potential productivity gains, pushed engineering teams harder, challenging delivery dates more than before, asking can we do this faster using AI. Functions that had previously not done much development work (like product or project management) began using AI to make prototypes and even full applications completely with AI. The consequences for prototypes was acceptable, but it was in the full applications that the gaps in technical knowledge became apparent - error conditions unhandled, unexpected edge cases. If you have worked with the output of an AI model in the last several months, you know what I am talking about. It was also around this time as well that hiring was significantly curtailed. It was already quite difficult to hire at Cloudflare, even for a backfill, with significant location constraints and an often meek pipeline of candidates. One can see where this is going…

In Mid-March, the tone shifted more drastically. Matthew took an entire B.E.E.R. (our weekly all-hands meeting, primarily used to share accomplishments and wins from across the company) to issue a warning - if you want to stay at Cloudflare, you must adopt AI.

AI is like the advent of the electric screwdriver. Yes, the manual screwdriver we have used all of this time is good, but look at how much more productive we can be. We cannot have employees who are significantly less productive because they refuse to use an electric screwdriver - and this is the take from across the industry.

It was around this time that I started interviewing more seriously. I had nothing more than a gut feeling that this was merely the beginning of a fundamental cultural shift. Product and Engineering leadership used Matthew’s speech as a catalyst to make more significant changes in what work was prioritized, and how we went about determining what work we wanted to do. I am not opposed to change, it is necessary to stay relevant and deliver what your customers are looking for, but this all felt a bit too rash.

On May 7th, Cloudflare announced it was terminating the employment of 1,100 employees across the company (about 20%). This came out of the blue for us, and despite doing well financially. The cuts affected every role and every department. Principal engineers, technical writers, project managers, engineers, directors, VPs and managers - including my own, all let go. Nobody was safe, they cut people who had been there 9 months, or even 14+ years. It was surreal and a feeling that words cannot effectively convey. I was not affected by these actions directly, but it was yet another significant change in culture. Before this day, leadership had touted the lack of layoffs and their expert ability to predict times of economic uncertainty and avoid over-hiring, and the public announcement that accompanied the layoffs stated as much - there was no particular reason for making these cuts, at least not one shared with employees. These cuts were being made because the organization was being shaped to be even leaner and more able to deliver in an agentic era. In this case it was the straw that broke the burnt-out camel’s back.

All while this was brewing, I was interviewing with a few different companies, but Stripe stood out to me. Their interview process reminded me much of the interviews we did at Cloudflare that focus on experience and how a candidate navigates a particular problem, rather than the exact solution we ended up at. I was asked to prepare a technical brief and give a presentation to 3 Stripes about a complex technical problem I had solved. While there was some coding in the interview, we spent a significant majority of the time talking about designing solutions to problems and the tradeoffs involved - the parts of my job I enjoy. I was fortunate enough to have the team extend an offer for me to join their team, and after a few restless nights trying to decide what to do, I accepted.

As I pack up my virtual desk, I think about what I will miss most of my time working at Cloudflare – it’s the people. Specifically, the engineers and their direct managers that I worked with every day. My immediate team had built a strong culture of trust and autonomy that had been slowly eroded away by senior leadership, but the interpersonal relationships persisted.

Change is always scary (at least it is for me), I think I made the right choice at the right time. Even if/when the AI bubble bursts, Stripe is still going to be around and will serve a fundamental role in facilitating commerce on the internet - that is not going anywhere anytime soon, rather quite the opposite. I am excited to help be a part of that and broaden my horizons and skillsets. I do get a chuckle out of the idea that the next time I buy something from a merchant, all three of the companies I have worked for out of college are going to be involved… Cloudflare fronts Shopify, Stripe processes the credit card transactions for most Shopify merchants, and Stripe runs on AWS - it is a small world after all.

Hello this is KK7UGU

Sat, 06 Jul 2024 02:14:04 +0000

A number of weeks ago, Melinda came back from Bainbridge Island with a packet from her mother. Within it, a helpful emergency preparedness guide and an emergency contact form with a handful of cell phone numbers. This lead to a conversation between the two of us on one of our afternoon walks, how would we actually get a hold of someone if the cell-phone towers are down or congested. Having an out-0f-state contact does very little, if you cannot reach them. We had recently watched a Geerling Engineering Video where Jeff and his dad got their amateur radio licenses and thought that maybe we should do the same; so that when the next disaster strikes, we can at least get a hold of one another.

A little bit of time searching the web landed us at HamStudy. Thanks to the many years of hardware tinkering I did with my Dad in the 2010s many of the concepts were easy to understand, and after about a week of studying with HamStudy, I found myself in the bathroom taking my Technician License test.

Since COVID, many testing centers (VECs - Volunteer Exam Coordinators) moved their assessments online, rather than in person. HamStudy has a listing of exam opportunities, both in-person and remote, and I was able to schedule an exam for the next morning. To ensure a clean test environment, examiners ask to be shown the room to ensure you don’t have any notes around. Combined with the requirement of having nobody else in the room while the exam is in-progress, yields the bathroom being the best place to take the test.

Have a testing environment that helps you focus on taking your exam. It should be clear of study materials and clutter as well as not interrupted by people, pets, or noise. Some great examples include a porch, bathroom, patio, spare bedroom, or any other clean and private spaces. Get creative!

~Bayou State Remote Exams Confirmation Email

The exam itself was straight forward since I had seen all of the exam questions before, thanks to HamStudy. I passed 33/35 (28/35 needed to pass) and got an email from the FCC on Monday morning requesting my license fee. The morning after that, I got my license and call sign assigned to me, KK7UGU.

A few days later, my first radio arrived from Ham Radio Outlet, an Icom ID-50 and after skimming through the manual and loading the local repeaters, I was able to make my first contact. I called in to the 9 o’clock net on WW7PSR and received a warm welcome from the community.

The next day (July 4th), Melinda and I went kayaking around Manzanita Bay and for the fun of it, put out a CQ call on W7NPZ, the Bainbridge Island 70cm Repeater. Much to our surprise, we made contact with KC2JVV who had also heard my check-in on WW7PSR the night before.

I was surprised with how large and robust the community of amateur radio operators is, given how few folks I saw before I got involved. The next goal is for Melinda to get her license and to see if and how we can connect between her work in Issaquah, and our home in West Seattle.

My first EV Roadtrip

Sat, 12 Aug 2023 04:41:28 +0000

My partner and I got our first Electric Vehicle, a 2023 Volvo XC40 Recharge P8 last November. We had gone on some day-trips and a weekend stay in Quinault, WA (149 miles from Seattle), we had never gone on a road-trip with an EV. That all changed when we drove up to Banff National Park in Canada for the 4th of July weekend.

When I initially planned our trip and where we would stay along the way, I looked for hotels that featured EV Chargers, which would enable us to start the day with a full charge, reducing the amount of time we would spend at charging stations. I also used A Better Route Planner (ABRP) to plan our route and where we would need to stop and charge. Assuming everything went to plan, ABRP estimated that we would spend just over two and a half hours charging over our 1,285 mile journey, not including our overnight hotel charging sessions.

Day 0 - Chilliwack

We started our journey fully charged and made it to the hotel in Chilliwack (just past the US/Canada border). It was an uneventful drive, but we stayed in what we both thought was one of the nicest and newest hotels we had stayed at, the Holiday Inn Express just off the Trans-Canada highway. They even have two Level 2 chargers in their garage that are free for guests to use.

Funny enough, when we were originally planning our route, booking.com showed the hotel as having EV Charging, but we could not find any mention of it on their website, or on PlugShare. I fixed that while we were there, and you can now find that charging location on PlugShare.

Day 1 - Banff

This felt like the longest day of driving and took us to 4 different chargers, two of which were at Canadian Tire locations. Not being Canadian, we had no idea what to expect when we walked in, looking to stretch our legs after an hour and a half of driving. What we found was a combination of a Walmart (without the food), a Discount Tire, and an Auto Zone, an impressive combination. I could see myself going there quite often if we had one nearby. Just after 6:30pm and after 467 miles of driving, we arrived at our hotel in Banff, hungry and exhausted.

We spent the next day hiking and exploring the town of Banff, which had far more tourists than I had expected, but given that it was the day after Canada Day, I am not surprised.

Day 3 - Banff to Vancouver

This was by far our longest day of driving, covering 529 miles; but it felt a lot shorter than our drive out to Banff, potentially because of our early start. We had tickets to see Charlie Puth at the UBC Thunderdome that evening, so time was of the essence. We were fortunate to start our day on a full charge, thanks to the Level 2 charger at the hotel, which gave us enough charge to get the 177 miles to Revelstoke. From there we proceeded to Kamloops, where ABRP scheduled us to go to our first, and only Petro-Canada EV Charger. It seemed that the charging gods had other plans for us though, as the first of two chargers had internet connectivity issues, meaning that it could not start a charge neither via credit card nor via their app; and the second charger would not charge above 35 kW (150kW is the max for the Volvo to put things into perspective). Fortunately, there was an Electrify Canada charger a few miles back, so we back-tracked and got the charge we needed to make it to Hope. In Hope we encountered our first busy charger, where 3 of 4 spots were taken when we arrived, and 2 other EVs queued up shortly after we arrived. Annoyingly for the drivers waiting, two of the other cars were charged well above 80% and their drivers where no where to be seen (DC Fast charging slows down considerably after the battery reaches 80% and it is generally best to unplug and continue on your way once you hit 80% as it will take over an hour to get from 80% to 100%, compared to the 30 minutes it can take to get from 20% to 80%).

After we charged up in Hope, we grabbed some lunch and continued all the way to our hotel in Vancouver.

Day 4 - Going back home

We hit a bit of a snag and the hotel we stayed at in downtown Vancouver only had a single charger for both of the 20+ story hotels that shared the parking garage, and we were not the lucky ones, so we started the day off with only 27% charge. ABRP actually struggled to plan a route for us, given our low state of charge and the lack of DC Fast Chargers in downtown Vancouver.

Compatible DC Fast Chargers in Vancouver, BC according to Plugshare

Although with some manual tinkering, I was able to get a plan together that had us top up at a Petro station and an Electrify America station once we were across the border. The plan was still somewhat risky, given both our bad experience with the Petro charger in Kamloops, and the fact that there was only a single charger at the location. We ended up getting lucky and the charger was not only available, but provided us with a speedy charge. 20-ish minutes after arriving, we were on our way back home. The rest of the drive home was uneventful, with one additional charging stop at Electrify America in Bellingham which gave us the juice we needed to make it the rest of the way home.

All in all, we spent $69.99 USD for charging, a net cost of 5.44¢/mi on our trip, and spent 3.5 hours at chargers over our 4 days of driving, not too bad considering that we got to spend some of that time getting some food and stretching our legs. We are starting to think about our next trip, driving down to visit my parents in Sedona, Arizona in the fall, making a few stops in California along the way, and given our positive experience with this trip, it looks like we’ll be taking the electric Volvo again.

Backing up Unifi Protect

Wed, 24 May 2023 01:16:32 +0000

A few weeks ago, I was scrolling through the r/Ubiquiti Subreddit when I came across a post where someone had their NVR stolen during a break in. We have a number of Protect cameras across the house and the post got me thinking… I have offsite backups of data stored on the NAS, but not the NVR. A lot of the recordings are useless, but it would be really nice to have the events stored offsite for a few days, just in case.

Luckily, the comments of the original post (which I can’t find anymore) had a bunch of helpful suggestions, one of which being to run unifi-protect-backup on a Raspberry Pi. However, because Pis are in short supply right now, and because I don’t love running Pis for production workloads, I wanted to run the backups from the NVR itself. I thought that the NVR had enough spare CPU and Memory to do the work, and had all the data available locally, but boy was I mistaken.

Since I wanted to both encrypt and compress the video files before uploading them, this turned out to be a computationally expensive operation and when run on the NVR, would result in a lock-up that would only be resolved by a power-cycle.

Since I had recently started using Nomad for my Container Orchestration, I opted create a Nomad Job for this task and have the job run from one of my servers, where significantly more CPU was available, and I did not risk locking up the NVR. The Nomad Job configuration ended up being much simpler than the work needed to get the backups running on the NVR (especially since the NVR likes to reset things after updates).

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Weaving a New Solution to Container Networking

Fri, 19 May 2023 19:48:40 +0000

As I described in my recent post setting up Nomad, I restored to using macvlan to get containers their own network addresses, but this had the unfortunate side effect of container being unable to talk to their host due to intricacies of how the Linux kernel does packet routing. I wasn’t happy with the solution, both because containers would sometimes get stuck and not get cleaned up, resulting in more than one container with the same IP address running on the network, leading to all sorts of strange issues; and because it meant that there was a single point of failure in my router, serving both Consul DNS and hosting cloudflared for the applications which used macvlan. So I went looking for options.

Looking for a Solution

I was weary of CNIs and their support in Nomad from the many hours I spent cursing the last time I tried to deal with this issue, so I was looking for solutions that had already been successfully integrated with Nomad in the past. I landed on three options: Calico, Flannel, and Weave, thanks to a post by SUSE’s Rancher project. Each had their upsides and downsides.

Calico - We use this at work, so I am somewhat familiar with its operations, but the use of BGP would cause issues as my router, a Unifi Dream Machine Pro, doesn’t support BGP out of the box. It also had a bunch of enterprise features, like network policies, I do not need, so I kept looking.

Flannel - This seemed like a great option, until I got to the part where I read that it relies on an etcd cluster. While I don’t have anything against etcd, I did not want to spin up another state management cluster if I didn’t need to. So Flannel went in my back pocket, as a last resort.

Weave - Weave uses a proprietary routing fabric, rather than linux primitives (like iptables), it seemed like a good fit for my application, given the low barrier to entry and the simple configuration options when using it alongside Docker.

Implementing Weavenet

I used Ansible to install Weave and its associated systemd unit.

- name: Install Weave
 hosts: nomad
 tasks:
 - name: Install Weave
 uri:
 dest: /usr/local/bin/weave
 status_code:
 - 200
 - 304
 url: https://git.io/weave
 mode: 0755

 - name: Create Systemd Service
 copy:
 dest: /etc/systemd/system/weave.service
 mode: 0755
 content: |
 [Unit]
 Description=Weave Network
 Documentation=http://docs.weave.works/weave/latest_release/
 Requires=docker.service
 After=docker.service
 [Service]
 EnvironmentFile=-/etc/sysconfig/weave
 ExecStartPre=/usr/local/bin/weave launch --no-restart --no-dns --ipalloc-range $IP_CIDR --metrics-addr=0.0.0.0:21049 $PEERS
 ExecStart=/usr/bin/docker attach weave
 ExecStop=/usr/local/bin/weave stop
 [Install]
 WantedBy=multi-user.target

 - name: Ensure sysconfig Directory Exists
 ansible.builtin.file:
 path: /etc/sysconfig
 state: directory

 - name: Create Weave Peers File
 copy:
 dest: /etc/sysconfig/weave
 content: |
 PEERS="10.0.10.48 10.0.10.64 10.0.10.80"
 IP_CIDR="172.30.0.0/16"

 - name: Start Weave Service
 systemd:
 state: restarted
 daemon_reload: true
 name: weave

I ended up overriding the default CIDR (10.0.0/8), as it conflicted with the parent network and explicitly listing all of the peer servers to avoid needing to join them manually.

I then ran weave expose on each of my nodes and added the /24 returned to my router’s static routes table, with the target being the node that returned the IP block. This enabled clients not part of the weave network to be able to access weave clients. From there, I just needed to explicitly set the network_mode in my Nomad Job Docker Configuration and ensure that the address mode was set to driver for my service definitions.

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Example Configuration using WeaveNet

Hey Siri

We have a handful of Apple’s HomePod Mini’s across the house and my partner and I both have iPhones, so it is handy to be able to ask Siri to turn on/off the lights and what not. The actual work is done via the Home Kit Integration in Home Assistant.

One of my usual tests after I change something with either Home Assistant, or its network is to ask my office HomePod to turn on or off my Desk Lamp. Unsurprisingly, after I configured Weave on Home Assistant this stopped working. A bit of pondering lead to a potential issue. mDNS. When you pair a HomeKit device, you never put in an IP Address or other configuration, it is all handled via multicast DNS (mDNS), and now that Home Assistant was on a different subnet than the Home Pods, and iPhones, this connection was broken, despite the networks being accessible from one-another.

Conveniently this is not an uncommon problem, and it’s already been solved via tools like mDNS Reflector. Deploying a container that bridged the weave network and the server host network (which is the same one as the network our phones, HomePods, etc are connected to) instantly restored the connection between HomeKit on my phone and Home Assistant. And just like that, we were back in business.

Resilient Infrastructure for the Homelab

Thu, 18 May 2023 23:25:52 +0000

As I’ve alluded to in the past, Home Assistant plays a major role in making my home work, from managing the HVAC system, to automatically turning on the lights in the utility closet when the door opens. For many years, I ran Home Assistant on an i5 Intel NUC within a Docker Container. This worked great, but when the NUC went down (be it for updates, or some unscheduled maintenance), so did Home Assistant and my ability to control the lights. While my partner is very forgiving and understanding, it’s rather annoying to not be able to turn on some of the lights for a few hours while I troubleshoot an issue.

Single-points of failure are scary, and at work, we try to eliminate them wherever possible, so I thought it was time to do the same at home. There are a lot of existing solutions to this problem in the form of orchestration engines, like Docker Swarm, Kubernetes, K3s, etc. We use Kubernetes at work, but it was rather overkill for what I needed to do at home, which was schedule Docker Containers (and ideally Shell Scripts) to run across various nodes in my home-lab (and ideally on a specific schedule). I landed on using Hashicorp’s Nomad to get this done. Nomad met all of my needs, all without being too complex.

From the beginning of this project, I had in mind to eventually get 2-3 new servers to enable the decommissioning of the NUC that had been working its little heart out for me for the last 4 years. As such, I wanted to automate as much of the provisioning and management of the new servers as much as possible. I wasn’t quite ready yet (or sure of what I wanted to get), so I used the NUC and a Gaming Server I had built a while back to start experimenting.

Ansible for Provisioning

I love using Terraform for config management, but it doesn’t support configuring machines well (there are other tools in the Hashicorp offering that do this, but are heavily focused on the cloud). That left tools like Ansible, Chef, Puppet, and a few others. A little time spent on Google revealed that Ansible was the easiest to get started with, used one of my favorite languages (Python) for extensions, and didn’t require complex provisioning workflows.

Ansible has the concept of sharing Roles and Tasks via the Ansible Galaxy. Conveniently for me, there was already a Galaxy Collection for configuring Consul (we’ll get to that in a bit) and Nomad. I had to make some adjustments to fit my needs, notably, running both the Nomad Server and Client on the same node.

Consul for Service Discovery

Before we can talk about Nomad, and container orchestration, we need to talk about Service Discovery. Service Discovery allows us to answer the question, where does X service live and how do I communicate with it. When running containers on a single node, this isn’t a significant issue, since the question of where is THE server, and the question of what port can be answered by the container orchestrator (Docker in my case). While we can pin specific container to specific nodes, that doesn’t allow us to build a resilient system, which is where tools like Consul come in.

Consul provides a single source of truth for the answer to the what and where question. It adds some extra nice features, like health checks. Nomad more recently started offering a built-in, simple, service discovery option, but Consul still wins out for me because of 1 key feature - DNS. Consul provides a DNS interface for service discovery, which enables non-Consul aware tools, like Home Assistant, to use Consul. For example, by Home Assistant instance depends on an MQTT server, which may be on a different node than the one Home Assistant is running on. While I can’t use dynamic ports, I can use the DNS interface for Consul as the host name for the MQTT server in Home Assistant. In my case, this means that in the eyes of Home Assistant, it can reach the MQTT Server at mqtt.service.seaview.consul.

Nomad for Orchestration

Nomad turned out to be the right balance complexity and resiliency, while not getting in my way too much with its relatively straightforward configuration DSL for jobs. Together with its support for CSI and CNIs it checked all the boxes, and was relatively easy to install and get going, given I had an already operational Consul Cluster.

CSIs

A lot less cool than Crime Scene Investigators, Container Storage Interfaces, provide a mechanism by which volumes can be mounted to containers. This is another case where having jobs potentially move across nodes presents an issue, as many containers are not stateless and need somewhere to persist their state.

In preparation for the move to Nomad, I replaced the OS on my NAS with TrueNAS Scale, both because of some reliability and connectivity issues I was having with Unraid, but also because of the first-class support TrueNas had for CSIs. In my case, I ended up using the democratic plugin to enable automatic NFS share creation and management. By connecting the plugin with the TrueNAS API, I am able to create, update, and delete shares from within Nomad, super convenient.

CNIs

CNI (Container Network Interfaces) allow for adding custom networking interfaces to Docker Containers. This is super convenient if the default bridge mode doesn’t work for your use case. For me, I like running Home Assistant on its own IP address since it has a bunch of different ports it uses for all the various integrations and those can clash with ports used by other containers. The macvlan plugin allows for this (assuming your NIC allows for advertising multiple MAC Addresses). However, because of some fun Linux networking issues that I don’t fully understand, containers using ipvlan or macvlan cannot access the host via its IP address. This posed a problem since I needed to communicate with the Consul Agent on the host to resolve Service Discovery DNS Requests. Not a problem, I would use a plugin called Multus to connect both a macvlan and bridge interface to the container.

This didn’t work. I spent hours troubleshooting issues, ranging from macvlan IP address not being released, to no requests making it to/from the container at all. Some more googling showed that there were a number of open issues with CNI support and Nomad dating back to 2021 that had clearly not been fixed. While not ideal, I could work around this problem. Rather than having the containers talk to their host for DNS resolution, I deployed a container to my router that augmented a DNS Server with a Consul agent to enable the containers to get their answers that way.

tpaulus/consul-core-dns

CoreDNS Container also running a Consul Agent to serve Consul DNS Requests

tpaulus GitHub

Terraform for Config Management

Since the goal of this project was to reduce the manual configuration, I opted to manage my Nomad Job configuration via Terraform. This not only allows my to quickly re-provision all of my jobs, it allows for automated image updates, which we will discuss in a bit.

tpaulus/terraform-hashi

Hashicorp Product (Nomad, Consul, etc.) Terraform Configurations

tpaulus GitHub

Cloudflare Tunnels

While my partner may disagree, I don’t run a data center in our garage and we have a single non-static public IP address for the whole house. The solution here comes in the form of a tunnel, specifically Cloudflare Tunnels. I created a system job in Nomad to run a Cloudflare Tunnel on every server. This gave me 3 different tunnels, pointing to the same services. While I could use DNS round-robin to route traffic between the servers, that would cause issues if any single server went down, since 1/3 of requests would fail. Using a Cloudflare Load Balancer solves this and allows for more intelligent routing of requests. In my case, I am simply using the health-check option to conditionally send traffic to each of the servers, ensuring that only healthy servers receive requests.

Foreshadowing

One afternoon, I came into the garage to hear an unusual clicking noise coming from the rack. Some investigation yielded that the CPU Fan on the NUC had died. It was time to take advantage of the high-availability infrastructure I had spent so many hours configuring.

I shut down the NUC and moved the Zigbee Dongle to another server and that was that. Nomad moved the allocations (jobs) from the NUC and scheduled them on another server in the rack. It was wonderful to see how quick and easy it was, especially given all of the headaches this failure would have caused before my Nomad adventure.

Optimizing Docker Image Upgrades

Back when I was hand managing container configurations on a single node, I would use Watchtower to automatically upgrade my Docker Containers to use the latest images. This approach doesn’t work when using Nomad though since it expects the docker configuration to be a certain way, and if the allocation were to get rescheduled to a different node, the updated image tag would not stick, resulting in an older image version being used until Watchtower updated it again (not ideal).

Renovate to the Rescue

My first thought was to build a tool to do this myself, but that would be a lot of work. My second thought was to clone the GitHub action that bumps the Image Tag in Dockerfiles, but again, that was going to be fair bit of work. We recently started using Renovate to update our dependencies (including Docker Images) at work, and got to thinking if I could adapt Renovate to do this for me. Some digging through the docs later, I found the Regex Manager, which enabled using Regex Expressions to parse files with a specified extension type and pass those to different dependency managers. A bit of cursing at Regex later, I ended up with the following rennovate.json which would look through all my .hcl files, find the docker images, and upgrade them if needed.

{
 "$schema": "https://docs.renovatebot.com/renovate-schema.json",
 "extends": [
 "config:base",
 ],
 "regexManagers": [
 {
 "fileMatch": ["[^.]+\\.hcl"],
 "matchStrings": [
 "image\\s?=\\s?\"(?<depName>.*?):(?<currentValue>.*?)(?:@(?<currentDigest>sha256:[a-f0-9]+))?\""
 ],
 "datasourceTemplate": "docker",
 "depTypeTemplate": "docker_image",
 "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
 }
 ]
}

And because I configured the integration with the Terraform Cloud, whenever I update the repository (like merging the Renovate PR), the changes are applied to the Nomad Cluster and the running container is replaced.

Conclusion

I’ve been running Nomad as my container orchestration system for the last few months and have been rather pleased with the performance and resilience. Servers reboot for updates and I never notice, what more could I ask for. I’ve still got some more work ahead of me, especially on the container networking front, as I’m not happy having a single point of failure on the DNS side of things, but that’s work for another day. For now, I can sleep better knowing my home services are resilient should a server fail again (or just need to reboot for a kernel update).

Replacing PiHole with Cloudflare Gateway

Sun, 30 Oct 2022 23:05:01 +0000

Before I can dive into how I configured Cloudflare Teams Gateway to work for my home, I need to give a bit of context and what lead up to this. I began using PiHole a number of years ago, running on a Raspberry Pi Model 1B. That worked fine for a while until my partner moved-in. She would report that the internet would occasionally stop working for her, and most of the time, this turned out to be a DNS issue. Combined with the occasional hick-ups caused by running on an older, low-power SBC, the need for occasional reboots to install updates meant that it was time to setup a second PiHole to run along side the first. Those Raspberry Pis were eventually replaced with Docker Containers running on much beefier machines, but the occasional outages persisted. Either due to fun intricacies of macvlan which is how the Docker Containers get their IPs on the network, or an over aggressive block list, it lead to both of us turning off WiFi when a page wouldn’t load. Don’t get me wrong, I love hosting my own infrastructure for all the things we need at home, but DNS is a fickle thing and issues with DNS can manifest themselves in the most peculiar, unsuspecting ways.

In a different thread, I started working for Cloudflare in April and quickly learned that a lot had changed since I became a Cloudflare User in 2014. More recently, Cloudflare has been expanding in the Zero-Trust space with tools and service aimed at making securing corporate infrastructure easier, while enabling work-from-home. One of these products is Gateway, which is part of the larger Zero Trust offering (and has a very generous free plan, which is what I use). Gateway DNS, which is what we’ll be discussing for the rest of this post, brings firewall style polices to DNS resolution, all while being wicked fast, which is critical for a DNS service.

So, how did I go about moving from PiHole to Gateway for my at home DNS resolution. Well, making the actual switch is as easy as updating your router’s DHCP DNS settings to inform clients of the Gateway DNS IP address, instead of the IP addresses of the PiHole containers. While that will get your DNS request off of PiHole, there is a bit of extra work needed to get similar levels of privacy and security from Gateway DNS.

First, a bit of house-keeping. Because there is a limited number of IPv4 addresses available (and they are in short supply), Gateway DNS re-uses the same IP addresses across many different customers (customizing the IP Address is an enterprise level feature), so Gateway DNS uses the source IP of the request to associate the incoming request with your account. For those with a static public IP address, it’s a set it and forget it affair, but for those who can’t get a static IP (or don’t want to pay $15/month for the privilege) we need to get a bit creative. I ended up writing this simple script which runs on one of my servers every 15 minutes to check my public IP and update it if necessary.

So, at this point we have Gateway DNS configured to always have the correct Public IP, but we haven’t established any rules yet. I’ve been exploring Terraform in my free time, in part because of how much we use it internally at Cloudflare to manage our configuration. While updating the Public IP via Terraform seems a bit overly complex since it can change often, using Terraform to define the Gateway Policies seemed like a perfect fit. That’s a long, round about way of saying, I defined the Gateway Policies in Terraform.

tpaulus/terraform-cloudflare

tpaulus GitHub

You’ll want to configure the rules for your Gateway based off of your needs, but for me, I blocked content categories identified as security risks (i.e., New Domains, Parked Domains, Malware, Phishing) as well as any traffic destine for the TOR Network. Additionally, I maintain a manual list of Trackers and Advertisers I don’t want to see and block them by domain name. Cloudflare’s domain classification is pretty good, but the manual list allows me to quickly block a domain without updating the policies to include that specific hostname.

With all that configuration complete and the logging tuned, we can actually update the router configuration to route DNS requests to Gateway. That covers requests coming from you home, but the awesome part of all of this is these protections and configurations can move with me via the Cloudflare 1.1.1.1 app (a.k.a. WARP), Cloudflare’s take on a VPN. By enrolling the device in my Zero Trust account, these same Gateway policies will apply to requests made from those devices, regardless of where they are in the world, as long as WARP is enabled. WARP also enables some really slick split tunneling that leverages Cloudflare Tunnels (f.k.a Argo Tunnels) which allow me to access my home network from anywhere, without using port-forwarding on my home router.

Cloudflare Gateway is not designed to be an ad and tracker blocker like PiHole and it would take significant effort to configure Cloudflare Gateway to block 158,614 domains sourced from the known ad-lists. But so far, this replacement does an all right job, while ensuring DNS is highly available for my home network without too much hassle, with the bonus of being able to take it with me, wherever I go.

Moving Beyond A→Z

Mon, 04 Apr 2022 16:00:00 +0000

I initially joined Amazon as a Software Engineering Intern in the summer of 2017, something I wrote about previously. In the three and a half years since I came back as a full-time engineer, I have poured my heart and soul into the service that would eventually become publicly known as EC2 Image Builder.

The History of EC2 Image Builder

EC2 Image Builder started off as an internal service called the Cascading Curated Image Service (CCIS for short) and was part of the Amazon Elastic Beanstalk team. CCIS, while built to run on AWS Lambda, was a Java Monolith, with everything being inside a single, tightly coupled, code base. In April of 2019, our team was pulled into discussions with the Amazon Linux and EC2 Windows teams to provide expertise in building a public service to enable customers to build AMIs easier and faster. A lot of bureaucracy and meetings later, our team was moved to within the EC2 Windows Organization and we started building the service that would eventually become known as EC2 Image Builder. By this time it was June and the service was set to launch at re:Invent, AWS’ anual developer conference, in December.

We used those few months to rebuild the service from the ground up, only the Finite State Machine (FSM) from the original service survived. One of my responsibilities was to re-write the API stack to be faster, lighter, and compliant with the internal security standards for authentication and authorization. To enable lower latencies, we decided to move to Python, both because of its faster performance of Lambda (provisioned concurrency was not a thing when we started building) and the team had some prior knowledge. Many late nights and slices of pizza later, we fliped the switch and made the service public at 9PM Pacific Time on December 2nd, 2019.

It has been over 2 years since we launched the service and our adoption numbers look good and feedback from customers like AC3 and Genesys have been overwhelmingly positive. The team started very small, only four engineers, but has since grown to 10, many of whom I consider my friends.

So, Why am I Leaving Amazon?

It all started in November 2021, when one of the other tenured members of the team left the company to go work for a local start-up. They were looking for a new challenge and did not feel like they were being appreciated for the work and effort they put in. Their responsibilities fell onto my shoulders when they left, but they also left behind the reminder that there are other options outside of Amazon. My long time manager and friend went on their Christmas vacation and came back with the news that they too were leaving the company, motivated by the words of the former employee, “it’s nice to feel wanted.”

The three of us got together the following weekend to discuss things over beer and pizza, both because we wanted to hear how the start-up life was going, and to reflect about our experiences at Amazon. What I learned about how management works at Amazon horrified me. I learned that one of the engineers I worked closely with and was consistently impressed with their work, was considered by the organization to be one of our “least effective” (worst) employees. I also learned that the bottom 5% of employees across teams would be forced out, something that Insider covered last April. On my team of 10, that meant likely 1 engineer had to go. At the same time, we had at least six open positions we were struggling to fill (some had been open over a year), things started to feel Sisyphean and dismal.

After my manager left, I started to spend most of my days in meetings. I spent very little time doing the things I liked to do, coding and designing solutions to complex problems. My job was not fun any more and I saw no improvements on the horizon. This was on top of a perpetual feeling of resentment towards senior leadership for not recognizing the contributions of engineers. I witnessed countless occasions where individuals worked long hours to get features delivered and bugs patched, with no recognition; but the second something went wrong, senior leadership was quick to scrutinize engineering for the issue.

I raised some of these concerns with my new manager (my previous skip-level manager) which triggered some reaction; which included reducing the amount of time I spent on-call (I was previously on-call 79% of days, 71 days of the quarter), and a “dive and save”.

A dive and save is an attempt by an employer to retain (or save) an employee who is looking to leave the company. Typically, these attempts result in compensation and/or role adjustments, but are generally considered to be a short-term tactic, rather than a long-term solution.

You know you have a problem when you have a codified mechanism for fixing your mistakes in a desperate effort to retain an employee. Additionally, increasing an individuals compensation without addressing the underlying issues does little to retain the employee long-term, it just keeps them there a few months longer. It was a recruiter who I spoke with that put it best, Amazon uses “golden handcuffs” to keep employees, paying them an exuberant amount of money to make up for the frustrations and long work hours.

Finding a New Job

Given all that I had learned about how I was being managed, and my overall lack of excitement, I put my hat in the rink and started replying to recruiters who reached out to me on LinkedIn. I also applied directly to companies who I would like to work for.

I had some confidence that I would be able to find a new job, since at the time, a lot of people were changing jobs, fueled in part by the availability of remote work; and a deficit of engineers compared to the number of openings, making it an employee’s market.

I applied to 14 different companies, either via a recruiter who reached out to me, or directly via their respective websites. I also got a number of messages via Hired, but none of those really spoke to me. I was looking for and applying to backend engineering and site-reliability software engineering positions since that is where I felt I can make an impact and have relevant experience. In no particular order:

Google - Despite having a number of internal references, after my on-site interview, I was only approved for offer at L3 (college-hire / junior engineer), and while that meant I would be able to do a lot more coding, it was not the direction I wanted to take my career in.
Discovery - After a few weeks of good discussions and interest from their sourcing company, I was told Discovery was no longer interested.
Microsoft - My recruiter felt that it would be difficult to find a team willing to take on a senior engineer with my [lack of] experience. I never heard back after our initial call back in January.
Coda - The recruiter scheduled an initial call, but needed to cancel last minute. They never reached out to reschedule.
RadAI - They were not able to meet my compensation expectations, and decided from their end not to go forward.
Apple - No response!
Hulu / Discovery - No response!
Slack / SalesForce - Form rejection email.
T-Mobile - No response!
Digital Ocean - I got an email that the position was eliminated, but that I would be considered for other positions. I never heard back.
NVIDIA - I got through a Phone Screen with them, but decided not to pursue the opportunity since I did not feel a strong connection with the team and their work.
Alaska Air - My lack of JavaScript experience (for a back end engineer position) immediately disqualified me and triggered an auto-rejection 15 minutes after I submitted my application.
Adobe - Form rejection email.

The keen eyed will notice, that I only listed 13 companies, and that is because I omitted one… Cloudflare.

Where I’m Going

If it was not obvious from the last sentence, I’m going to Cloudflare. I start as P3 (P1 = Junior, P5 = Principal) Systems Engineer with the APIs team on April 18th.

My interview process at Cloudflare started with a half hour Google Meet Call with one of the engineers on the team. We talked about my experience, and the position, and concluded with some homework. I was to design and implement a URL Shortening Service. This was somewhat ironic to me, since during my time at Amazon, my go to question for Intern interviews was a similar, but scoped-down, problem. I had also already built a similar solution that leveraged AWS. However, when I got the requirements via email later that week, I needed to make some adjustments. Namely, the requirements included statistics (how often the URL had been accessed in the last 24 hours, week, and all time), as well as the ability to run locally in a development environment. Over the course of the weekend, I built a URL Shortener using Flask and PostgreSQL and was overall quite proud of the GitHub repo I ended up submitting to them.

It seems that the team at Cloudflare were also pleased with my work, since I was invited to a virtual on-site interview. I had interviewed (virtually) with Google the week prior, and the difference between the two was night and day. While the Google interviews felt like being in a juicer, with them attempting to squeeze my knowledge about algorithms and data structures out of me, my interviews with Cloudflare were much more fun and conversational. Yes, there were still questions to answer, but things felt significantly more casual. The tech lead of the team was the only one to ask me a traditional coding question, but even then, it was a collaborative exercise in JavaScript, a language I had little experience in (intentionally). All of my other interviews were pleasant and I got a good feel for the team as the interviews went on.

Nine interviews later, including a final call with President and Co-Founder Michelle Zatlyn, I had an offer letter in hand, sent via email by the CEO, Matthew Prince, himself.

Why Cloudflare?

I’ve been a Cloudflare user since September 2014 (I needed to check this via the API) and have been very happy with the service. I initially came to Cloudflare to take advantage of the free SSL certificates (Let’s Encrypt was not a thing back then), and have since relied on Cloudflare more and more, most recently using their Email Routing feature to replace some email routes I had configured in Mailgun a number of years ago.

Beyond being a good service provider, Cloudflare uses its powers for good, from making it easier for healthcare providers to offer vaccine signups, to protecting journalism, to ensuring state and local election bodies get the security support they need.

When I am out and about and people ask me what I do and where I work, I am a bit embarrassed to say that I work for Amazon, due to the impact the company has had on Seattle and other neighboring cities, and the countless controversies sounding the Amazon Fulfillment Centers that give Amazon a negative name in the news.

Cloudflare is still a small company at only 2,500 employees world wide (nothing compared to the over 60,000 Amazon has in Seattle alone), but if the past is anything to go by, Cloudflare will continue to do things that make the internet better and safer for everyone.

Why not a new team within Amazon?

At Amazon, internal mobility (being able to move from one team to another) is easy, and team culture varies drastically from team to team, and while I could easily find a new team within the company to work for, I feel a bit burned by Amazon. Even on this new team, there would still be stack ranking and URA (un-regretted attrition) targets. I also doubt I would be free of my old team and responsibilities completely. I think it is just better this way.

Breaking up with Logi Pop

Sun, 01 Nov 2020 20:48:59 +0000

I got my first set of Logitech Pop buttons (Small BT LE Buttons that interface with a WiFi Hub) in December 2017 and they have moved with me from college dorm room, to dorm room, finally making their way on the wall next to my bed in my apartment now where they control the various lights not only in my bedroom, but in the entire apartment. Ever since the beginning, I have leveraged the convenience of these tiny, reliable (I have yet to need to change the batteries in over 3 years of clicking) via IFTTT. However, with the recent announcement of IFTTT moving to a subscription model, I needed to look elsewhere. While all of the lights in my bedroom are Philips Hue bulbs, many of the other smart devices I have in the rest of my place are not, leading me to use a central automation tool (Home Assistant in my case) to manage everything.

Logitech Pop Button (Left) next to a Xiomi Aquara Button (Right).

The Logitech Pop bridge supports a number of integrations, including Philips Hue, IFTTT, Smart Things, etc.; but what they do not support is local access, meaning that while getting bridge to talk to Home Assistant is theoretically possible using an emulated-hue bridge and a lot of time, it just is not feasible in my case. As such, I am parting ways with my beloved Pop buttons (oh, how beautiful they are), and replacing them.

There are a number of IoT clicky things out there, one of the most notable being Flic. However, not only is a set of Flic buttons incredibly pricy (a starter kit with 2 buttons costs $159.99 at the time of writing), they have many of the same pitfalls, although offering a Home Assistant Integration, it relies on bypassing the hub and using the bluetooth module on the host running Home Assistant. For some context, I run Home Assistant on an Intel NUC via docker, so Bluetooth device access is possible, but again not practice, plus I have some concerns about the Bluetooth range from where the NUC is, to where the buttons would be. Some googling lead me to this Reddit Post on r/homebridge. This post and its comments, along with a lot more googling (free time is in great abundance this year), I landed on the Xiomi Aquara Buttons, which run about $18 on Amazon. These are ZigBee buttons, which means that they require a compatible hub of some sorts. I didn’t think anything of this though, as I already had a working VeraPlus Hub, which among other things supports both ZigBee and Z-Wave devices. So I ordered 3 to replace my small family of Pop buttons and waited.

When the buttons arrived, I was quite pleased with their design and build. Yes, nowhere close to as thin and sexy as the Logitech Pop buttons, but for the price, I was more than happy to put up with the slightly more clunky build, if I could get them to work. Keyword being if. After a fair bit of fiddling, I got them to associate to the Vera, resetting them, and then pushing the “Effective Range Test Button” every few seconds while the Vera was in search mode eventually got them to associate. However, Vera does not officially support Aquara devices, so they showed up as Generic IO devices.

Vera Dashboard showing “Generic ZigBee device”

There were a fair number of people trying to get Aquara devices to work on Vera’s Community, but the post has been inactive for 9 months no, with only some success getting the Temperature sensor to work. Xiomi has a hub that you can plug in and add to your network, but given the news on Xiomi’s data collection practices, I was hesitant to add one of their devices to my home network, even if it was on an isolated VLAN. On top of that, I generally try to avoid vendor lock-in wherever possible, especially in the IoT space, where the fall of a company can lead to a very hungry cat, as we saw with Petnet. So, I did some more googling (again, free time is not hard to come by in the midst of a pandemic), and stumbled upon ZigBee2MQTT.

I had used MQTT before, when I built a ESP8266-powered, internet-connected heart light, so I was aware of how quick and powerful it was. So, I did some reading and ended up ordering a zig-a-zig-ah from Electrolama on Tindie, wanting to avoid the frustrations of flashing a CC2531 USB sniffer, and wait a short eternity for some very reliable parts to arrive from AliExpress. So, I waited.

I had some hardware issues with the adapter that I originally received, but Omer from Electrolama was very quick to respond to my concerns and before I knew it I had a new adapter in my hands. From there, it was almost plug and play. For compliance reasons, the adapter ships with a simple blink program, but there is some excellent documentation on the Electrolama website that details how to flash the correct firmware for ZigBee2MQTT on to the device, which is accomplished with some python libraries which are installed via pip, and a python script to handle the flashing. Once flashed, the zig-a-zig-ah shows up as a USB serial device, so no extra drivers were needed (I am using Ubuntu Server 20.04 LTS, your milage may very). From there, I was on the home stretch. Because I already use Docker for most of my needs on my home server, I went with the Container version of ZigBee2MQTT.

docker run \
 -d \
 --name="zigbee2mqtt" \
 -v /etc/docker/zigbee2mqtt/data:/app/data \
 --device=/dev/ttyUSB0 \
 -e TZ=America/Los_Angeles \
 --restart=always \
 --network host \
 koenkk/zigbee2mqtt

On first run, ZigBee2MQTT creates the default configuration file, which requires a slight bit of modification before ZigBee2MQTT can be started successfully. Namely, I needed to change the Serial device it was trying to use, and I reconfigured the Network Key while I was here and hadn’t paired any devices.

homeassistant: true
permit_join: false
mqtt:
 base_topic: zigbee2mqtt
 server: 'mqtt://127.0.0.1'
serial:
 port: /dev/ttyUSB0
 adapter: zstack
advanced:
 rtscts: false
 network_key: GENERATE

I was already running a MQTT server that I had already wired in with Home Assistant, but that too is relativity trivial. Home Assistant makes everything pretty easy, especially with their Discovery Functionality. ZigBee2MQTT includes some very helpful configuration cards that you can include in your dashboard, which I included via the Package Method described on the documentation. I also added the Network Map Card, which all together looks something like this once I got all of my devices joined.

ZigBee Network Information in Home Assistant Lovelace

I found the provided card to do everything I needed it to do. All that was left for me to do was to configure the automations that I used to have configured in the Pop software for interfacing directly with Hue and the Automations that I had in IFTTT. On each side of my bed, I have a button that I use to control the Hue bulb that is directly above it, tapping once for a night light, twice for a reading light, and holding to turn off all the lights in the apartment. The last one was the easiest; however the previous two were much harder due to some intricacies of how Home Assistant handles light.toggle actions. Specifically, you can set the color and brightness of a light with light.toggle, but you call light.toggle again with a different color or brightness, rather than updating the color, it turns off the light. So after some searching and thinking, I came up with this automation.

- id: '1601945836861'
 alias: Bedroom - Button - Right - Reading Light
 description: ''
 trigger:
 - platform: device
 domain: mqtt
 device_id: 98de24c1076711eb9eb06778c10b3ef9
 type: action
 subtype: double
 discovery_id: 0x00158d00027bf559 action_double
 condition: []
 action:
 - choose:
 - conditions:
 - condition: template
 value_template: '{{ state_attr(''light.hue_bedroom_bed_right_light'', ''color'')
 == {''x'': 0.447, ''y'': 0.406} and is_state(''light.hue_bedroom_bed_right_light'',
 ''on'')}}'
 sequence:
 - service: light.turn_off
 data:
 transition: 1
 entity_id: light.hue_bedroom_bed_right_light
 default:
 - service: light.turn_on
 data:
 profile: reading
 brightness: 254
 transition: 1
 entity_id: light.hue_bedroom_bed_right_light
 mode: single

I have 4 similar instances of this automation, one for the night light and one for the reading light, for each side of the bed. The differences being the colors/brightnesses and the devices respectively. Specifically what this automation does, is it checks if the light is in the desired state already (both in terms of color and state) and if it is, turns off the light, if it isn’t, it means that the light is in a different state and the user is requesting a state change, which is achieved via a call to light.turn_on with the desired parameters.

All in all, once I got everything working (which wasn’t too difficult honestly), things have worked great. The buttons are much more responsive, especially for the automations that used to leverage IFTTT. I have also already made some more additions to my ZigBee network, like moving my Hue Bulbs and remotes over to ZigBee2Mqtt and getting a SmartThings Motion Sensor and Door Contact Sensor to replace a similar ZWave set. From my small sample size, ZigBee is much more responsive than ZWave, although that may also be because of the setup I was using for ZWave (using a VeraPlus connected via Home Assistant) was less than ideal. I must give credit to the ZigBee2MQTT team and community for making a reliable, easy to use, and well supported piece of software that makes integrating these physical devices into our Home Automation Systems so easy (and dare I say it, fun).

A Xiomi Aquara button mounted on my bed side.

Project Heart

Sat, 04 Aug 2018 02:35:57 +0000

Long Distance Relationships are tough, there is no denying that. But this isn’t a post about the woes of love, rather a little project to help bridge the distance. Project Heart is a light that is controlled by your partner, regardless of where they are in the world, as long as they have an internet connection, they can light up your light.

Making use of the Adafruit ESP8266 HUZZAH breakout, we can connect a series of NeoPixel LEDs to the network without too much hassle. The basic premise involves the ESP8266 running a simple Web Server that is listening to request on port 80, specifically POST requests with a plain text body of either pulse or chase to run an animation sequence on the LEDs.

Software

Programming the ESP8266 with the Arduino software is relatively straightforward after you have set it up. However, keep in mind that you will need a USB to Serial Adapter if you are using the specified Breakout. The Adafruit Feather HUZZAH has this built in, but it a little more expensive. Instructions for setting up the Arduino IDE to program the ESP8266 can be found on the Adafruit Learning System.

Allowing the lights to be controlled from anywhere, not just inside of your home network, presents another fun, but easily solved challenge. All of this is done in your router’s settings (typically found in the Advanced Section). First we will want to assign a static IP to the HUZZAH to allow ensure that the device’s internal IP remains the same. Next, you will want to set up Port Forwarding to allow the outside world to talk to your device.

Note: Most ISPs block common ports like 80, 443, 8080, so some trial and error may be necessary to find an open port that will work for you. Any open port will suffice. Most routers allow you to map a different external and internal port, which allows us to run the HUZZAH on port 80 for convenience.

Finally, since most ISPs use DHCP themselves to assign your home an IP address, we will want to use a service like NoIP to ensure that we can always reach our home network, even if the ISP decides to issue our Modem a new IP address. How this is setup depends on your router, and the NoIP site has setup instructions for all major manufactures. So, at this point, we should have a URL which we can use to access our HUZZAH from anywhere in the world. Mine looks something like this: http://example.hopto.org:5000

You can test out your configuration using an HTTP request tool like Postman, but that’s not very convenient. That’s where we can use a service like IFTTT to make our life easier. By combining the button trigger and the Webhook service we can use a widget in the iOS notification center to trigger the light in a matter of seconds.

IFTTT Widget Settings

You can then add the IFTTT Widget to your Today View to access the animation controls quickly and easily.

Hardware

The project is pretty simple from a hardware perspective, with a short bill of materials, especially if you have some of the tools already lying around.

The capacitor is optional and provides some power smoothing, which is especially useful for very flashy (high current draw) animations like the pulse. When things are all set and done, mount the board and leds how you see fit, Hot Glue is one of my go to favorites for things like this.

Bill of Materials

Adafruit HUZZAH ESP8266 Breakout
FTDI Serial TTL-232 USB Cable (Required for programming - although any USB to Serial adapter will due)
USB Micro-B Breakout Board
Side Light NeoPixel LED PCB Bar
Stranded-Core wire
Micro USB Cable
USB Power Supply (1A should be sufficient for the default animations)

All of the code for this project is on GitHub and is free for you to use and modify to your heart’s content: https://github.com/tpaulus/Project-Heart

Daily White Card

Mon, 11 Sep 2017 05:53:50 +0000

Cards Against Humanity is a College Favorite and often leads to some wonderful, even though somewhat disturbing, conversation. Through a strange series of events, a few of the staff of Maya Olmeca at the Front Desk ended up talking about the various White Cards¹ and how some of them would be quite funny to see in real life, like “Vigiorius Jazz Hands”.

Aaron (RA for the Visual and Performing Arts Residential Learning Community in Maya) had the awesome idea of making a competition to find White Cards in real life. The rules would be simple, take a selfie with showing what every the White Card described.

With a little bit of Python (~65 lines), GroupMe’s super simple API, and the work of Chris Hallberg, I put together a simple bot that posts in a GroupMe² group every morning at 8 am with a Random White Card.

How it Works

Using JSON Against Humanity I generated a JSON object with all of the cards from the Original CAH Decks (The Original Set, The 6 expansions, and the Green Box), which totaled 1159 white cards. These cards are loaded into a python dict object, from which one random card is selected whenever the program is executed. This card is then given some context, like “Today you are looking for…” and the text of the White Card. This is then posted to the GroupMe Group via the Bot that was created specifically for this purpose on GroupMe’s Developer portal.

# Message Context
MESSAGES = ['Today you are looking for... %s',
 'Can you find... %s',
 'How about... %s',
 '%s Find it!',
 '%s DO YOUR WORST!',
 '%s']

# Pick a Random White Card & Message Context
secure_random = random.SystemRandom()
card = secure_random.choice(Cards.get_white_cards())
message = secure_random.choice(MESSAGES)

# Post the Card text to GroupMe
GroupMe.post_message(message % card)

Scheduling is accomplished via Heroku’s Scheduler Add-on and is run for free on one of their Dynos.

Taking only an evening to put together, I’m excited to see what “interesting” results this challenge will have. The Full Source Code is available on GitHub and is licensed under MIT.

https://github.com/tpaulus/daily-white-card

Cover Photo: By tombullock (Cards Against Humanity) [CC BY 2.0], via Wikimedia Commons

White Cards are typically used to fill in the blanks for Black Cards and contain a multitude of various phrases, ranging from the simple (“God”) to the complex and disturbing. ↩︎
We use GroupMe across the department for communication, and it allows for super simple and easy photo sharing, which is why we used it for this task. Plus, everyone on our staff already has the app installed. ↩︎

The Road to Seattle

Sat, 11 Mar 2017 06:09:27 +0000

Unlike Mike Rowe’s “Road To Coober Pedy,” my trip to Seattle, Washington for my Summer Internship with Amazon, will hopefully not be filled with death, injury, or road kill. This post will help serve as my record of the progress that I have made, and the events that have taken place, leading up to my First Day as a Software Development Engineering Intern.

After submitting my acceptance letter, I received a “Team Preference” survey from my point of contact at Amazon in early February. Based on my start and end dates, and the areas of interest I marked on my survey, I was presented with three options, as well as a brief description of what the team does and where they are located. One of the options, and the one that I ended up marking as my first preference (and receiving) was working with Amazon Web Services’ (AWS) CloudFormation team in Seattle.

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion. - AWS CloudFormation

I have had some experience working with AWS and its various services (namely Lightsail for WebHosting, and Glacier for Backups). From that, I know that AWS’ many services, each of which are extremely powerful by themselves, only become more powerful when combined. However, doing this becomes increasingly difficult to do repeatedly and consistently, as the size of your infrastructure grows; and that is why CloudFormation is so cool!

The real fun starts once I cross the T-60 day mark, which will happen in Mid-March. At that point, my team placement, start/end dates are set in stone, and the on-boarding process begins (Relocation, Background Check, etc.) I will be updating this post until my first day in Mid-May.

I have no doubts that this is going to be very exciting summer!

*Banner Photo: Daniel Schwen (Own work) [CC BY-SA 4.0], via Wikimedia Commons*

Summer Internship - Amazon SDE

Mon, 30 Jan 2017 19:30:00 +0000

After applying to 10 odd major tech companies over winter break; Google and Amazon reached out to me and asked me to complete some “homework”¹. Google was first to show interest, having me complete two problems, which I completed to their satisfaction, and a short while later I received a request for availability in regard to scheduling a phone interview time. Since this was over the holidays, things were a little dragged out, but I completed my two ~1-hour phone interviews in early January. At around the same time, Amazon reached out and asked me to complete some of their problems as well.

Interestingly, after completing the first set, which I found to be very easy, they sent a second, more complex, set of questions. These too I completed well, passing all of the unit tests the exam tool ran². Much to my surprise, I received an offer to join the Amazon team a few days after completing my homework. I was astounded that one of the big four³ was interested in having someone like me, someone from a large state university with little glamor – like Hackathon prizes, publications, major projects and such, at their organization. Especially without a phone interview!

Amazon gave me till February 1st to make up my mind, the clock was ticking. I connected my Google Recruiter and informed her of the news, and a few days later I heard back that I had made it into the pool. Short aside on how Google hires interns…

Candidates are interviewed by two Google Engineers

An independent committee of Googlers reviews the Interview feedback.

If the committee feels that you would be a good fit, you are added to the pool.

Departments looking for interns pull from the pool based off the project survey interns filled out during the application process.

You potentially have a 1:1 phone interview with your prospective manager, to see if you would be a good fit for their team

If all goes well, you get an offer!

As you can see, this is quite a lengthy and intensive process, something that I did not have time for with Amazon’s offer expiring VERY soon. Along with the offer email, Amazon included the schedule for several informational sessions for new interns, one of which I attended. On the call was one of the recruiters, as well as Project Manager for one of the teams at Amazon. The presentation lasted about an hour, and afterward, I had a much better grasp on what role interns play at Amazon.

It should not be a big surprise based on the title of this post, I ended up accepting the Amazon offer. Not only is Seattle⁴ a city that I have never been to before, I am excited by opportunities this position will afford me. More posts and photos are definitely to come once I start in Mid-May.

This summer is definitely going to be one to remember!

Homework refers to a series of coding/programming questions that vary in scope, depth, and duration based on the assignment and organization. The goal of these is typical to identify one’s skill level with a given programming language; and the general concepts, like list storing, tree traversal, etc. ↩︎
Both for the Google and Amazon assignments, the questions are confidential and under NDA, so cannot go into greater detail in regard to their content. I can, however, say that they were focused on general programming concepts, like data structures, etc. ↩︎
Big Four: The most influential international technology companies: Amazon.com, Apple/Microsoft, Facebook and Google (https://en.wikipedia.org/wiki/Big_Four). ↩︎
While Seattle is not the only city where Amazon places Interns, most of their interns are placed in Seattle, as that is where most of their offices and teams are based. ↩︎

Backing Up with Amazon Glacier

Thu, 05 Jan 2017 01:13:00 +0000

Amazon Glacier is a long-term, low-cost data archival and backup service offered by AWS. It is surprisingly cheap at only $0.004¹ per GB per month, it has some caveats though. Mainly, retrievals can take upto 5 hours, and archives must be stored for a minimum of 3 months².

Glacier is available in most, if not all, of their data centers, but I will be using the US East (N. Virginia) datacenter, since that is where my Lightsail instances are hosted, and inter-datacenter transmissions are free. If you are using Lightsail, make sure to turn on VPC Peering; this should facilitate communication between your LightSail VPS and the Glacier system.

Since I want to use my storage space most efficiently, I will only be backing up the “non-recoverable” information I have saved on my server. This includes, database dumps, images, configuration files; however, does not include Docker Images, etc. My logic behind this is, that in the event of a failure, I can easily pull a docker image from the repository, or reinstall a tool via apt, so I do not need to save a copy. Also, since archives will be saved for an extended period of time, depending on your desired backup frequency, saving a few hundred megabytes can add up over a few months of backups.

Background on Glacier

While the AWS management console allows you to do some work with Glacier, most interactions will occur via a CLI or API requests. Thankfully, Glacier is a relatively popular service, so other developers have already written scripts and tools to help us work with Glacier. AWS has also written a well documented SDK for Glacier as well.

Glacier treats every file that is uploaded as an non-mutable archive, be it a photo, video, or in our case a ZIP file. Each file is assigned a unique ID which can be used to identify the archive in Glacier.

Since our Archives will likely be larger than 100MB, we will be using a Multipart upload, which allows the upload of a single larger archive to be chunked and parallelized. It also allows for easy recovery if something goes wrong while uploading a chunk, as only that chunk needs to be reuploaded, not the entire archive. The Multipart Upload process is discussed in detail on the Developer Guide for Glacier: http://docs.aws.amazon.com/amazonglacier/latest/dev/uploading-archive-mpu.html.

Configure Glacier

Before we get started on our backup system, we have to do some work in the AWS Management console. Before you start, make sure that you have set the correct region in AWS for where you want your Glacier Vault to be.

IAM

We will first create an IAM user with Programmatic access which we will use to authenticate our backup agent with AWS. Next, we will attach the AmazonGlacierFullAccess policy to our user, this will allow our user to upload archives to our Vault. Take note of the Access Key and Secret key for the user you just created, as you will need to save them in a config file a little later. This is the only time you will see these, however you will be able to create additional keys later if necessary.

Glacier

Navigate to the Glacier service in the management console, and click get started. The region that you chose earlier should be listed, as well as a prompt for the nave of your vault. The name of your vault must be unique across the region! You can setup SES notifications if you like about Vault Statuses, but you can skip that safely if you like.

Configure System to be Backedup

Clone the GitHub Repository, which contains the uploader.jar, backup script, and aws.properties template via git clone https://github.com/tpaulus/glacier-backup.git. You may need to make the backup script and the uploader jar executable, which can be achieved by cding into the folder and executing chmod +x *.sh *.jar.

Configure Backup Script

Once you have cloned the repo, modify aws.properties with the accessKey and secretKey that you took note of when you crated your IAM user. Next update the fields at the top of in backup.sh with the correct values. The correct endpoint url can be found at: http://docs.aws.amazon.com/general/latest/gr/rande.html#glacier_region. You should use HTTPS wherever possible.

The script assumes that you have your MySQL running in a docker container, whose name is a variable defined at the top of the script. Modify this section, or the other sections to suite your needs. If you are trying things out, set SANDBOX to true to disable the uploader and cleanup portions of the script. This prevents the compressed archive and backup folder from being deleted, so you can check if everything worked as desired, without having to deal with Glacier.

Running the Script

Before you can run the script, you may need to install Java, you can check if Java is installed by running java -version in the command line. If an error is returned, you need to install Java. On Ubuntu, this can be achieved by running sudo apt-get install openjdk-8-jre-headless.

Once you have verified that the script works as desired, you will want to add it as a cron job to run regularly.

Cron Job

Lastly, we will want to have our backup run automatically on a schedule, which can be easily achieved with the help of CRON. CRON allows you to input when you want a certain command or task run, and it will take care of the rest for you automatically. To help generate your crontab line, tools like Crontab Generator can take out most of the guess work and frustration. For the command, you will want to cd into the directory to which you cloned the repository, and then execute the script. My finished crontab looks something like this:

0 12 * * * cd /usr/share/glacier-backup; ./backup.sh >> /var/log/glacier-backup.log 2>&1

I decided to have my script run at 12PM (UTC) everyday, since that is a time when I expect not to make any changes to the server. If you are unsure of the timezone of your server, run date to find out the current date, time, and timezone of your system.

Now run sudo crontab -e to add the line you got from the Crontab Generator to your local crontab file. The sudo is necessary, as we want our script to be run by the root user to bypass any permissions issues.

Save and exit, and you should see crontab: installing new crontab. This means that your line was added successfully to the crontab for the root user.

That’s it, now you can rest assured that if something should go wrong, you have a recent backup of your essential files and databases that are safely stored offsite, ready for your retrieval within 5 hours³.

Recovery Procedure

In the event that you should need to use one of the Backups from your Glacier Vault, you will first need to retrieve the Inventory from the Glacier System. Setting the environment variables as shown below will make our lives a little easier as we complete the steps necessary to recover an archive.

Additionally, the steps listed below use the same Command Line tool as we used above to send our Backup to Glacier, so ensure that the commands below are being executed in the same directory as the backup.sh file. This is done to make sure that the aws.properties file is located in the correct place in relation to the JAR file being executed to interface with Glacier.

Set Common Environment Variables

The values for most of these variables will be the same as those found at the top of the backup.sh file. They are not in a shell file, however, because this process can take some time to execute⁴.

export CREDENTIALS="aws.properties"
export ENDPOINT="https://glacier.us-east-1.amazonaws.com"
export VAULT="myvault"

List Inventory

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --list-inventory

This will give you a job ID for the inventory listing, make note of this, as you will need it to access the result.

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --list-inventory yourjobidfromthepreviousstep

Be sure to include the jobid from the previous step in the above command. You will receive an error massage (status 400) until the inventory has been generated for you to retrieve. This can take about 4 hours, depending on Vault Size and Demand, according to the Amazon Developer Documentation.

Download Archive

Once you have found the desired archive that you would like to retrieve from your inventory, making note of its Archive ID, you can request it via:

java -jar glacieruploader.jar --endpoint $ENDPOINT --vault $VAULT --credentials $CREDENTIALS --download myarchiveid --target path/to/filename.zip

Make sure to change the Archive ID that you want to download to one in the inventory, and the target to the location you want the archive saved.

Note: In order for you to retrieve the Archive, your IAM User that you have configured (see above) will need access to the Amazon SQS and SNS Service. This is a separate permission than the one granted for Write Access to Glacier.

If you are getting errors in regard to the “current data retrieval policy,” have a look in your Vault General Settings, accessed via the Management Console, and make sure that you allow for enough data to be downloaded per hour. You should set the cap a little above the estimated size of the archive that you want to retrieve. The higher you set your retrieval cap, the more you can be charged. You are only charged for retrieval, this is not a monthly fixed charge.

That’s It. While retrievals can be time intensive, they significantly lower cost makes it worth it for most users. If you find yourself making a significant number of retrievals, you may want to consider configuring Expedited Retrievals via provisioned capacity, or moving to S3 for Hot-Storage, rather than Glacier, which is considered Cold-Storage.

At time of writing, each GB of Glacier storage in Region: US-East-1 (N. Virginia) cost $0.004. Updated pricing can be found at https://aws.amazon.com/glacier/pricing/ ↩︎
While archives can be deleted at anytime, the documentation states that you will be charged a prorated fee for the remaining duration. https://aws.amazon.com/glacier/faqs/ ↩︎
Archive retrieval times depend on the archive size and retrieval option selected. More information can be found in the Glacier Developer documentation: https://docs.aws.amazon.com/amazonglacier/latest/dev/getting-started-download-archive.html. ↩︎
While this process can be automated with the help of Amazon SNS and an Amazon SQS queue, that can be a bit more complex to setup and configure. If you will be frequently be retrieving archives (at which point you should re-consider your use of Glacier) it may be advantageous to look into this: https://docs.aws.amazon.com/amazonglacier/latest/dev/retrieving-vault-inventory-java.html ↩︎

Installing Fish Terminal on MacOS

Fri, 23 Sep 2016 20:23:00 +0000

Fish is a great alternative to the default bash or shell that is installed on your computer by default, while Fish is available for a variety of platforms, even Windows (with Cygwin); but this guide will focus on macOS, because that is what I use on a daily basis. It is very easily to install, since they provide a package that you simply download and run and it installs fish to /usr/local/bin. However, if you are a bit more savvy and already have Homebrew installed, you can install fish with a single command in your currently very boring shell.

brew install fish

After the install completes, you will need to allow/whitelist the fish shell to be used, and make it your default shell, unless you want to type fish every time you start a session.

sudo echo '/usr/local/bin/fish' >> /etc/shells chsh -s /usr/local/bin/fish

chsh -s /usr/local/bin/fish

Now that you have installed, whitelisted, and made it your default shell, you are probably wondering why fish is so cool. Fish’s power lies in its expansibility, especially in the awesome things that other fish enthusiasts have already created. One of these, that is likely already installed is Tackle. To use it, you will likely need to add a few extra dependencies, which can be easily done via another brew install command, and an install script. This script can be run implicitly, but you should understand what the script does before doing so, else bad things could happen. Justin has documented the Tackelbox framework and the code very thoroughly, and I recommend checking out the framework and the install code: https://github.com/justinmayer/tacklebox

brew install vcprompt grc

curl -L https://raw.githubusercontent.com/justinmayer/tacklebox/master/tools/install.fish | fish

Tackle and Tacklebox requires some configuration to work, but the config is easily modified as it is stored in a hidden file in your home folder, under ~/.config/fish/config.fish. I have included my config file, adds extra colors and functionality with tools like git, pip, etc.

# Paths to your tackle
set tacklebox_path ~/.tackle ~/.tacklebox
# Theme
set tacklebox_theme entropy
# Which modules would you like to load? (modules can be found in ~/.tackle/modules/*)
# Custom modules may be added to ~/.tacklebox/modules/
# Example format: set tacklebox_modules virtualfish virtualhooks
set tacklebox_modules virtualfish virtualhooks
set tacklebox_plugins extract grc pip python up
# Which plugins would you like to enable? (plugins can be found in ~/.tackle/plugins/*)
# Custom plugins may be added to ~/.tacklebox/plugins/
# Example format: set tacklebox_plugins python extract
# Load Tacklebox configuration
. ~/.tacklebox/tacklebox.fish

If you have issues, the Fish Shell Website has some great examples and documentation: https://fishshell.com/.

Quiver, My New Favorite Note Taking App

Mon, 04 Jul 2016 21:40:53 +0000

Evernote was the note taking application was the most popular option for a number of years and was the least bad of the options out there. However, recently Evernote has been promoting their paid plans in a way the hurts their free users. Removing popular features, like emailing in notes (1), and imposing extra restrictions, like limiting you to only 2 devices (2), made me reconsider if Evernote was really the best option out there. Popular alternatives to Evernote at the time of writing include: Simplenote, and Microsoft’s OneNote. I like to avoid Microsoft wherever possible, and Simplenotes offerings were average at best, and the reviews did not help sell me on their product.

I have quite a few notes in Evernote. Thankfully, Quiver can import notes in the Evernote .enex format.

The advantage to products like OneNote, Evernote, and Simplenote is that they offer Mobile apps and Syncing across devices. I rarely find myself editing, or creating notes on my device, but I would like to be able to access notes on the go, just in case (usually for reference purposes). The Apple App Store features Quiver as one of their “Apps for Developers” and since starting college, I have used Evernote primarily to take notes in class (many of which are Computer Science Courses). If you have ever tried formatting a note in Evernote, you know the nightmare that it is. And while apps like Alternote make it a little easier, they still don’t make it easy. Quiver fixes all of their with their innovative approach, which they call cells. Cells are different areas of your note that can be formated as text, code (with syntax highlighting for a bunch of languages), Markdown, LaTeX, or Diagram; all of which are rendered beautifully by the App. The app even supports importing Evernote Notebooks! The only downside for me is that there is no sync engine, or way to view my notes on the go (either on a mobile device or in the cloud).

In order to take my notes on the go, I took advantage of their export script that converts entire notebooks of notes into HTML folders. The files themselves are hosted on my webserver and are displayed via a beautiful jQuery, PHP file explorer from Tutorialzine. While the Quiver Notebook Library is saved in Google Drive (mostly for backups and safe keeping), I am using the very powerful rsync command to sync the library between my mac and the web server.

Server Setup

Set up a folder (with password protection if you like) with your web server of choice (mine is NGINX) and copy the files from the repo into the folder. Take note of the path of the folder since you will need to enter it into the sync script that we will install on the mac.

Download the File Explorer and Assets from GitHub.

Install the quiver2html node module. npm install -g quiver2html
You will want to set up Apache, NGINX, or your web server of choice to serve the PHP content in the folder you cloned the server files into. Google is your friend here.

When you are done, navigate to the URL for your folder and you should see something like this.

Mac Setup

We need to configure the computer to sync the Notebook Library automatically and request that the server convert the library into HTML. This is achieved with a simple shell script that is run every 5 minutes by launchctl, Apple’s version of CRON on newer OSs. Launchctl uses specialized plist files (which can be created with launched) to tell it when to run what command.

Download the Shell Script and Plist File from GitHub.

Update the script’s variables to match your installation.

QUIVERLIB_LOC="Path/to/Quiver.qvlibrary"
SERVER="lorien" # Host Name of your server, as set in .ssh/config
SRV_DEST="/srv/notes/" # Public HTTP Directory where Notebooks folder is

Copy the shell script to your local/bin folder.

sudo cp quiver_sync.sh /usr/local/bin/quiverSync
sudo chmod +x /usr/local/bin/quiverSync

The advantage with placing the file in your local bin folder is that you can call “quiverSync” at anytime from your command line to toggle a sync. Now would be a good time to test the script to see if it works before continuing. Take a look at the website and you should see something like this.

If everything is working the way it should, place the launchctl script into the Launch Agents folder. If the ~/Library/LaunchAgents directory does not yet exists, you can create it with “mkdir”. There are several Launch Agent folders on your computer, the folder in your user library only runs when you are logged in, which is what we want, since you will likely not be making changes to your quiver notes while you are not logged in.

sudo cp com.whitestarsystems.quiver_sync.plist \
~/Library/LaunchAgents/com.whitestarsystems.quiver_sync.plist
launchctl load -w ~/Library/LaunchAgents/com.whitestarsystems.quiver_sync.plist

Conclusion

I think that Quiver, with the addition of the sync tool we implemented, will replace Evernote for me. It is an extremely powerful note-taking tool that won’t be charging you monthly per-device. It may not have as many features as the competition, but it serves its purpose for me very well.

CodeDay San Diego Winter 2016

Wed, 24 Feb 2016 09:20:18 +0000

The sixth CodeDay in San Diego was a huge success. We returned to CyberHive this season, the venue that hosted the first CodeDay back in May 2014. This event was a little smaller than previous events, only 75 attendees registered for the event; however, the event was still a great success!

We lead more workshops than ever before, and included content for attendees who came to the event with some experience (an audience that has been underserved in past events).

Best Overall

Team: Team Tired

Project: 5 More Minutes

This was Alexis’ first CodeDay and he blew us away with his Pebble Watch App that counted down the minutes till his class was over. He used tools and languages that he did not know coming into the event, something that the judges commended him for. Everyone was very impressed with Alexis’ work.

Best App

Team: ¯\_(ツ)_/¯

Project: RNG Matchmaking

This Valentine’s themed app match attendees from various CodeDay Events and other Coding Organizations to each other using a “complex algorithm.” They took the results a step further, showing the audience how there was a gender imbalance in the technology sector and how we can promote women to get involved with coding.

Best Game

Team: Anthony

Project: Jump

Anthony continues to amaze event after event. This time he created an addicting game that featured a bird that had to jump up on randomly appearing platforms. If you failed to land on a platform, the bird would have a very un-happy ending. Anthony always amazes us with his willingness to help others and his creativity, even at 1am in the morning.

Special Award – Most Memes

Team: Meme Team

Project: Meme Defense Force

This award was well deserved by Sam and Johnny! They combined the best meme’s and made a tower defense style game from villains like DJ Khaled. We all got a good laugh from their game. We hope they make another one!

Special Award – Most Musical

Team: Fourier

Project: GNUSynth

Leo and Tiahm created an app to solve a problem. Tiahm creates music on his computer, but often needs a synthesizer, so they created one. The app used a myriad of audio analysis algorithms, including the Fourier Analysis (where the team got their name from).

Although this was one of San Diego’s smaller events, it was still a great success and the number of attendees worked well with the size and design of the venue.

Special Thanks to…

Our Judges: Darin Anderson, Mo Rahseparian

Our Local Event Management Team: Ayan (Public & Community Relations), TJ (Event Programming), and Alex (Day-Of Coordinator)

Our Local Organizers: Agnieszka, Salman, Leo and Dan

Our Venue Staff: Darin and Mo

Our Venue Sponsor: CyberTech (Home to CyberHive, iHive and Nest)

Our Community Sponsor: Cox California

CodeDay San Diego Fall 2015

Wed, 11 Nov 2015 16:18:36 +0000

The fifth CodeDay in San Diego was a huge success. Hosted at the new Central Library in Downtown San Diego, this CodeDay was one of the largest events yet. We had over 140 attendees register for the event, and 94 attendees ended up checking in to the event.

Looking back, the event was a big success for both the attendees and the staff members.

We took over the Helen Price Reading Room on the 8th floor, a beautiful space with a great view of the bay. CodeDay worked in collaboration with the City of San Diego, as well as the San Diego Public Library to put on the event. CodeDay San Diego was also made possible by JetBrains.

We had quite a few workshops at CodeDay San Diego, including an intro to Python Workshop that used PyCharm EDU; as well as an awesome Data Analysis workshop lead by Andrell Bower, from the City of San Diego.

Best Overall

Team: Assaf

We were very impressed by Assaf’s effort, he did not stop after building his snake & apple game, which was very cleanly coded, he went on to build a working, multi-user chat room using scratch, which involved complex encryption and several work-arounds to work with Scratch.

Best App

Team: Ronak

We really liked how well built this app was and how it was designed to serve a purpose. It was also one of the few apps that was fully deployed to a device.

Best Game

Team: Treacherous Turnips

We really liked how this team had an idea and that they went with it. We also really liked the concept and implementation, as well as the fact that it was an homage to their CS teacher.

Special Award – Most Creative

Team: Gatekeepers

We really liked their concept and how it stood out from all the other games at CodeDay San Diego. We were also very impressed with the implementation and the strategy behind the game.

Special Award – Best Effort

Team: TincyTurtle

We really liked how this is Austin’s second CodeDay and how he has grown since his last event. His game was very well made and although it was not perfect, it demonstrated excellent effort on his part.

Attendees at Work…

Presentations…

Overall, we had a really great time and we think that the attendees had a blast as well. Both the Staff and the Attendees felt that this was one of the best events yet.

Special Thanks to…

Our Judges: Lon Miller, Gary Hayslip, Maksim Pecherskiy

Our Local Organizers: Ayan, and TJ

Our Day-Off Staff: Alex, Lauren, and Ayesha

Our Library Staff: Erwin, Sherwood, Ryan, Alejandro, Jackie, Stephanie, Geneva, Taylor, Stacy, Kerry, and Kathryn

Our Venue Sponsor: The San Diego Public Library

Our Development Tool Sponsor: JetBrains