cdegroot.com

Tackling software complexity with the CELP stack

Thu, 14 Jan 2021 20:02:03 -0500

The last couple of months I’ve been deeply immersed in the combination of Elixir with Phoenix LiveView for the Web UI, Commanded as a CQRS/ES implementation, and PostgreSQL for persistence - I think I heard someone call it the CELP stack, so let’s call it the CELP stack. I think this stack is a great way to tackle software complexity.

[Note: the timestamps in this post are off, but given that they are part of this post’s URL, not simply unfixable. The actual date this got published was 23 May 2021

Some may disagree with Fred Brooks’ distinction between accidental and essential complexity, but my experience indicates that he is correct. I have worked in what is now called SaaS for almost 25 years (and “regular” software another decade before that), I’ve seen the gamut between horrible contraptions and trivially simple systems, and the interesting thing is that hardly ever, the extra complexity was introduced by any sort of requirements: it just happened, people thought it was normal, “this is what Gartner adviced”, and so on. In my experience, accidental complexity is one of the biggest sources of waste in our industry, and tackling software complexity therefore logically means reducing that as much as possible. We need simpler systems with less layers that human beings can reason about, not more.

Simple software scales. It scales in both ways of scaling that are important: you can put more load on it in production and you can grow it (both in terms of LOC and team size).

If you talk about software scaling, most will think you’re talking about the first kind of scaling, but, frankly, that’s a solved problem. I’ve worked with pretty terrible systems that scaled by just throwing hardware at it. Is it “elegant” to run 300 webservers because your PHP app cannot deal with more than a couple of dozen requests per second per server? Nope, but it works. And the sort of companies I’ve worked at invariably saw the idiotically high hosting bills that resulted as rounding errors. If your revenue is €100million and your growth rate is 20-30%, it’s hard as CFO to get excited about some techie that tells you there’s a savings of €2m in the architecture if we just can stop building new product and to a rewrite.

No, business wakes up when the other kind of scalability starts hurting, is my experience. When they cannot onboard more developers for all the new great plans that product has for even more growth, or “velocity” slows down because the software and the infrastructure around it just got too complex: these are the costly things that will make your CFO take the executive elevator down into the coding dungeons and ask what’s up. Tackling this is invariably very costly and very disruptive to the business, so I’ve learned to do a little less of “You Ain’t Gonna Need It” and a little more of thinking about how to prevent this kind of pain in the future. The best solution? Ruthlessly hunt down and erase any kind of accidental complexity. Make “KISS” your religion. Funnily enough, that’ll often also help you scale in the other way: simpler systems are easier to reason about and therefore scaling bottlenecks are likely simpler to find and resolve.

Everything Should Be Made as Simple as Possible, But Not Simpler

– Maybe Albert Einstein

As simple as possible would be - in the Elixir nook of this world - a straight Phoenix app. Phoenix comes with a very powerful set of generators that make whipping up a quick form trivial. But it also comes with the known weaknesses of “old school” CRUD apps: read/write mixing makes it hard to scale your RDBMS - often done by adding read replicas and sending read traffic there - and there’s some additional complexity because Phoenix, with its server-side templates, does not address a lot of the modern expectations of web applications around interactivity, liveliness, and so on. I think that’s why a lot of teams find themselves these days writing a Vue or React application and then using Phoenix to power the API. Now you have two languages, two deployment pipelines, two production environments. Often, two teams. It is modern. KISS, it is not.

Read/write scaling has been a focus for me since 2005. The typical scenario is: you buy/lease/cloud-provision a database server and all is well. You are successful and the DBA comes complaining. You buy/lease/cloud-provision a bigger database server and all is well. That goes fine for a well - at one company we “joked” that all we would need to do is survive until the next AWS re:Invent conference where the next bigger piece of kit would be announced, and I heard rumours that Paypal at one time had a dedicated team hunting for the more powerful hardware for their central Oracle RAC db. Of course, then the horizontal scaling people start yelling that this is nonsense and that you should have read replicas because - usually - for every write transaction you have ten reads and half of them with overly slow queries. Vertical scaling isn’t nonsense, of course: it is super cheap, compared to pausing half your development work (which we assume powers your growth) and splitting reads and writes. Trust me, I’ve done it. The more magic your CRUD framework is (looking at you, Rails), the more painful it is.

Split your reads and writes early on, even if you don’t need it. It is one of these things that is very simple to do when you start, very simple to keep up as you grow, and will make future you very happy when the “how do we scale the DB” question comes up. It’s an insurance premium, where in this case the thing you’re insured against is your own success :).

CQRS/ES is one way of forcing you to split your reads and your writes, while getting extra benefits along the ride. The extra benefits are quite likely to make business people happy and thus will make it easier to sell doing it early:

You force yourself to think about stuff that happens in your system in terms of behaviours, not data. That alone will make things more understandable;
You have this event stream that lets you trivially walk through history, links events together, and is a veritable gold mine for the data scientists that management hired without knowing what to do with them.
It is trivially easy to create new projections, replay the past, and create whatever dashboards management wants without having to tell them that they’ll be useful in 6 months because you weren’t collecting the data.

Greg Young advices not to use Event Sourcing unless there’s a proven need, but he’s a C#/.Net person and I get it that when you start with an overly complex and baroque system, you want to be careful adding more ;-). However, over here in Elixirland it’s a somewhat different story: we start with a simple system and Event Sourcing is a natural extension to a system that is already driven by functional thinking and data pipelines. I am very close here to inverting Greg’s advice: use it unless you have reasons to not do it. With Commanded, Elixir has top-notch support for the collection of patterns that go under this label and adopting it is simple.

That leaves the other thing: our two systems/teams for front-end and back-end. Phoenix LiveView is the answer. It’s an answer that will make a lot of “I only know Javascript” type of front-end developers unhappy, but if you’re truly “full stack” then adopting to it should be pretty easy. I can be short about it: I’ve used pretty much everything under the sun since my first forays on the web (through X.25 from the DECUS Germany server to CERN) and this, so far, is the only thing that has truly worked both in terms of productivy and long-term cleanliness. It is not perfect yet, I’m hoping that some component system will get added, but as far as web application frameworks go it is very close to optimal. One language, and one running server on your system, seamless refactoring of logic from your UI down to your business logic when you want it: it is all there and thanks to Elixir’s excellent incremental compilation system and Phoenix’ pushes, your page is refreshed in the time it takes you to blink after you save an IDE buffer. Pro tip: add TailwindCSS for even more fun, but that is strictly speaking not part of this story.

The true magic comes when you combine these two concepts. Because Phoenix LiveView is also all about read/write spitting the two fit perfectly: user writes get turned into messages to the server process (every active LiveView has an active OTP “gen_server” process), and view updates/reads - after the initial load - are pushed down by messages from the server process back to the view in the browser. This is what you end up with:

User presses “submit” to, say, add something to their shopping cart;
the active LiveView server receives event, and converts it into an “AddItemToCart” command which it dispatches to the Commanded application;
Commanded takes the command and routes it to your “ShoppingCart” aggregate root, hydrating it if necessary;
Your aggregate root has an execute method that checks whether the item wasn’t already in the cart, etc, and if all is well, emits an “ItemAddedToCart” event;
Commanded persists the event;
PostgreSQL fires a trigger which prompts a listener to wake up;
That listener, another Commanded building block, gets the event and sends them out to anyone who is interested;
One of the interested parties is your Ecto projector, which takes the “ItemAddedToCart” event and translates it into a row add in a database that has current shopping carts (let’s assume that shopping carts must be persistent here);
After your listener is done, Commanded calls the module’s after_update handler which puts the events on a Phoenix PubSub topic;
That view that received the event subscribed when the cart was created/retrieved, so receives the event as a handle_info standard OTP message;
Your event handler converts the event into a simple structure and assigns it to the socket;
LiveView notices the updated assigns, calculates the deltas with the current assigns, and pushes a diff to the browser;
The browser patches the DOM, the user sees the extra cart item.

Complex, not? Well… not really. Mostly a lot of typing :). But look at the finegrained separation of concerns here - that is what will matter as you grow. Not how many LOC you have, but how little spaghetti.

Your Command cleanly defines what data is required to perform an action. It also is the entry point if you ever want to create an alternate UI - just send the already implemented commands. It is a specification:

typedstruct module: AddItemToCart do
  field :id, String.t()
  field :product_id, String.t()
end

Your execute handler does contract checking: is all the data there? Is it possible?

def execute(cart, c = %AddItemToCart{}) do
  if has_item(cart, c.product_id) do
    raise "You cannot add the same item more than once"
  else
    %ItemAdded{id: cart.id, product_id: c.product_id}
  end
end

Your apply is responsible for effecting the state change (if any) on the aggregate root. And that completes the model side of things - notice the complete absence of external dependencies here, making for trivial testing.

def apply(cart, %ItemAdded{product_id: product_id}) do
  %{cart | items: [product_id | cart.items]}
end

Your projection (normally you’ll have an Ecto projection - you could argue that a shopping cart may not belong in the database and that is fine, with projections you can always change your mind later) converts the data into something that is easy to read on the initial load of a page.

project(e = %ItemAdded{}, _metadata, fn multi ->
    Ecto.Multi.insert(multi, :project, %Projections.CartItem{
      id: e.id,
      product_id: e.product_id,
    })
  end)

Finally, on the UI side, your event handler is just concerned with creating a command in reaction to user actions…

def handle_event("add-item", %{product_id: product_id}, socket) do
  App.dispatch(%AddItemToCart(id: socket.assigns.cart_id, product_id: product_id))
  {:noreply, socket}

…and in the PubSub handler with updating the view state by updating the socket state (LiveView will then figure out the differences and send out a push).

def handle_info({:ecto_projection_completed, %ItemAdded{id: cart_id, product_id: product_id}}, socket) do
  socket = if cart_id == socket.assigns.cart_id do
    assign(socket, items, [product_id | socket.assigns.items])
  else
    socket
  end
  {:noreply, socket}

Whereas with a normal CRUD app, a lot of these concerns get mashed up in a single controller method, the CELP stack forces you to separate everything, which will force you to write short, clean, and reusable code. Absorbing new changes is just as trivial. Say that product says that you can have more than one item of a product in a cart.

Your command handler checks whether there’s already an item with that product id in the cart. If so, it will emit an, say, “ItemCountIncreased” event instead of “ItemAdded”.
A new event handler just increases the item count for an existing item (the cart state will now probably be a map or keyword list containing (product_id, count) elements).
A second projection method handles that event, just emitting an update. The insert and update are nicely separated without conditional logic.
You can now add a command to set the item to a certain count and hook it up to the UI, reusing a lot of the pipeline already created for adding an item.

In essence, CELP forces you to build small, reusable blocks. It’s more typing (although macro libraries are already being brewed in this space), but the extra code is clean, concise, and small enough to form a bucket of reusable blocks. In the meantime, you’re gathering a history of everything that happened in your web app for when management decides that Data Science is the next big thing ;-). Yes, you can achieve the same end result using TDD and refactoring, but sometimes it is nice to know when to skip ahead.

“You Ain’t Gonna Need It” is very good advice. Every time you pull in more than the bare minimum to get things done, you need to see it as an actuarial thing: are you insuring against something that could conceivably happen? Is the premium small enough to not get in the way of the important stuff? If so, you should consider pulling in the little extra thing. In my experience, it is exceptional that such a move is warranted. In my experience, pulling in the CELP stack is such an exception.

A heretic's guide to testing

Thu, 14 Jan 2021 20:02:03 -0500

Our industry is largely driven by fads: someone comes up with a great idea, and people then lift that idea out of context, run with it, and declare it “the true and only way.” In this post, I want to talk about testing and especially about the dogma that tests should be written first and they always should be there. In other words, about Test Driven Development (TDD) being the true and only way.

Let’s start with what I think our goals should be.

Quality, accessibility, pick two.

“Quality” is to me mostly captured by the notion of code being fit for purpose. It is contextual, if the purpose of your code is to impress an interviewer your idea about quality is going to be different than if you, say, write a system for a country’s air traffic control. Your code should do what it is supposed to do, and do it well.

“Accessibility” is a term I introduce here to capture a bunch of human factors aspects of code that I feel are often somewhat neglected. To me, the primary interaction with code is the interaction that people have with it, not the interaction that machines have with it. I write code for humans, whether it is my collaborators or my later self, and pretty much on equal footing with “it should be fit for purpose” is “it should be accessible.” Code needs to be readable, navigable, and malleable. The latter aspect cannot be stressed enough: if I cannot go back to my code and change it in a safe way, knowing that neither quality nor accessibility will be impacted, it is to me quite useless code.

TDD, as I understand it

I was taught TDD in Java. I learned TDD in Smalltalk. The idea of writing, compiling, and running a test that talks to non-existent classes or methods might be strange but is totally natural in Smalltalk - if you refer to a class that does not exist, you get a polite ask whether you want to define it, if you refer to a method that does not exist, you get dropped in the debugger and get a chance to create it while having the actual arguments that resulted from your test run right there.

It is very interactive and all but guarantees that you will only write code that is covered by a test. It is so efficient, that the running joke-with-a-truth is that you can recognize a seasoned Smalltalker by the fact that they spend most of their time in the debugger. In other languages, the feedback cycle is slower, sometimes much slower, as expensive VMs need to be started up for every test run. Still, even in these cumbersome languages it is a very nice way to develop software. Write a test, make it green, commit, rinse, repeat.

It is also limited in scope, and that is up next

Where TDD breaks down

TDD is extremely good at two things: teaching about unit testing and developing algorithms, like tricky calculations or somewhat complex data traversal.

The interesting thing is: at least in my industry that is a small part of my job. A lot of my work consists of talking to APIs. Take in some data, do some very simple logic work, and send it off again. Sometimes, the logic is so simple, it is what I call “too simple to test.” TDD breaks down here, because it inevitably would take you down the path of integration testing. The complexity lives in the APIs and your code does, well, not much. A much heard critique of Smalltalk is “everything happens elsewhere”, which addresses the issue that your logic is often smeared out over lots of collaborating classes; well, we now have the distributed version of that where your logic is smeared out over lots of collaborating services, partially owned by your employer, partially owned by third parties.

Less handoffs: reviewer merges

Thu, 14 Jan 2021 20:02:03 -0500

When developing software in teams, there are always seemingly conflicting forces at work. You want to move as quickly as possible, but to do that in the long term, you need to make sure that the full team knows what is going on. A lot of people accept that this tiny slow down in the short term has a huge payoff in the long term, and one of the practices that seems to have become entrenched is the practice of code reviews.

When I started, code reviews were rarely done, or at best very informally done. You committed your work and went on to the next item on the list, and sometimes, while making changes, you happened to see code that was written in a way to make you angry, sad or very happy and you gave feedback. Since then, through a lot of intermediate steps, most teams seem to have settled on what is now the de facto standard workflow on Github: you branch, create a pull request, and before the pull request is merged back into the main branch a colleague reviews.

There’s some divergence here, though. Often, “a colleague reviews” is translated as “a senior colleague reviews” which I don’t think is a good thing for various reasons: I believe in teams that are flexible and where everybody can do anything that the team needs to be done to at least a certain extent. The best way of teaching junior members how to do code reviews is to let them do it. In a similar vein, “a colleague reviews” often means “a colleague does the whole review and the rest of the team does a drive-by review”, essentially having the whole team pile on. It’s not nice to the coder, it is inefficient, and it breaks my rule where any two people in a team should feel empowered to make decisions. The rule is there to make sure that when there is work to be done, any two people can jump in and you don’t need to wait for “the right person” (read: the lead developer or the manager) to be available. Tickets stalling in the pipeline is enemy number one of the state of flow you are hopefully trying to achive.

So, let’s establish that “a colleague reviews” means: one, and only one, team member reviews and decides whether the code meets the team’s definition of done, whatever it may be. A reviewed PR is a PR that is up to par and can be pushed out to production (I am assuming continuous delivery here, which really should be your default in 2021). What happens next might surprise you….

The reviewer slaps a thumbs up on the pull request, moves the ticket to the “Ready to Deploy” lane, and goes back to their own work.

What? We have a ticket stalled. Why? Because the “original developer” should do the merge and pamper it through the deploy pipeline? I’ve seen this behaviour regularly, and it seems to be the default until I bring it up, and I cannot understand why. Let’s pull the virtual andon cord and discuss this.

Why is the ticket stalled? There is a double hand-off: first, it stalled waiting for a reviewer. That is a necessary wait, because we agreed to do reviews and this is the price we pay for it. Then, there is a second hand-off: it stalls waiting for the original coder complete whatever is currently in progress, and then go back to the “old ticket” that is waiting to be merged. This hand-off does not add anything to the team, it is just a delay, so purely by the rules of kanban/lean/whatyoucallit, it needs to be removed. Removing the hand-off means that the reviewer gets the ticket to “Done.”

And why would the original developer need to merge in the first place? Is the reviewer unsure that the review was thorough enough? Is it to shift blame when deployment breaks something on whoever write the code and not share it between coder and reviewer? My stance has been for quite some time now that if you, as a reviewer, do not feel completely up to speed with what the ticket was about, how the coder implemented it, and what the potential risks of a merge/deploy are, your review has not finished yet. It is not just a check to see whether the curly braces are in the right position - a review is a full share of knowledge of, and responsibility for the work. It is essentially some form of asynchronous pair programming.

As such, a reviewer should also feel empowered to make small fixes, remove that extra whitespace, and so on. No need to do a hand-off back - just make sure that the coder gets feedback at some point but please write that test you feel should be added. A hand-back should be an exception, a flag to the team that something went not according to plan, and probably a item worth of debate with the whole team. If reviewers constantly need to ask questions then the definition of Done is not robust enough to deal with this virtual hand-off, all the knowledge needed is not embodied in code and docs, and therefore the team should think on how to improve. That is the exception, not the rule.

Therefore, the reviewer merges and, if necessary, checks that deployment to production went fine. No need for another context switch with the original coder, and a good way to ensure that knowledge is actually getting shared: you’re not just the person that tells the coder that a trailing comma is missing, you’re part of the pair that got the ticket from conception to completion. Feel proud to be able to move that card to “Done”. High five. Next task.

Back to the '70s with Serverless

Thu, 17 Dec 2020 20:02:03 -0500

I’ve now worked with “The Cloud” for long enough to see that there’s still a long way to go before it becomes materially better than, say, the oldschool method of renting a couple of servers with a co-location company and running your software there. The latest fad is Serverless, which makes me feel a lot like we’ve arrived in 1970.

A long, long time ago, I was pretending to study business administration while teaching myself to code in C. The university was in two worlds: we had a lab with “personal computers” but also terminals to the mini computer and assignments had to be made on either one depending on the professors’ preferences. But both systems were at least “on-line” systems–interactive, with immediate feedback. A friend of mine was not so lucky: he had an assignment for one of his courses in civil engineering that had to be completed in Pascal and had to be handed in as a print-out from the university’s mainframe.

He called me in for help, because he never wrote a line of code before. So I hopped over to his place, a Turbo Pascal floppy disk in hand, and assumed that we would make short work of the assignment. And we did, even though I never coded in Pascal before: the IDE, even back then, was marvellous and a quick succession of trial-and-error got us the results we wanted (they were some simple engineering calculations, much quicker done on even the most primitive of calculators, but such are course assignments. At least it was easy for us to verify the program was correct).

We hopped into the car and drove over the the university’s computer center, a ’70s brutalist concrete bunker. We sat behind a terminal, typed in the code, surrounded it with the required IBM Job Control Language that the teaching assistant helpfully added to the assignment, and hit “Enter” (not “Return”, actually “Enter” - Enter The Job). A couple of minutes later, the printer started and spit out a couple of pages. Between gibberish in all caps, there it was: a compiler error. Back to square one - Turbo Pascal and IBM Mainframe Pascal clearly had some differences. A couple of hours and reams of wasted paper later, we had what my friend wanted: the results of a good run. The numbers matched our notes, and we could retreat to the nearest bar for a well-earned beer.

Ever since that experience, I have stressed the value of feedback and especially of quick feedback. Even in this minimal example, we spent more time tweaking the code for the dialect than we spent on writing it in the first place. Slow feedback loops kill performance, and if you don’t believe me, find an online version of The Beer Distribution Game and play it. You’ll be surprised.

The pinnacle of interactivity was, and still is, Smalltalk. I worked in that language for a couple of years and having your system compile and run most of your tests in less than a second is addictive. It is incredible how much your performance increases on a fully interactive programming system but it takes first-hand experience to fully appreciate it. It’s a strange system, a strange language, a tough sell; therefore Smalltalk is still in a very undeserved state of limbo.

After my Smalltalk years, I met a lot of large Java systems. They started out as horrible hairballs, but when the idea of “dependency injection” got a foothold, things started to get better. Except for the return of the Job Control Language, this time disguised in XML and carrying friendlier names like “Spring”. Internally, the code was reasonably clean and modular, but telling the computer how to run it took pretty much the same amount of typing, this time not in a programming language but in a structured markup language. That language lacked all the facilities that apply to writing good code, so principles like “Don’t Repeat Yourself” went overboard and copy/paste programming ensued. New business logic? New controller, twenty lines of Java boilerplate, 10 lines of Java business logic, 50 lines of XML to wire it to the rest of the system.

At least, in hindsight, XML was a blessing: your editor would tell you whether your “this is how the code is wired up” description had the right structure, and later on even whether you used names that existed in your Java codebase.

These systems took ages to compile, hours to run all the unit tests, and clearly interactivity went down the drain. So “split it all up” was not a bad idea. It was an unnecessary idea, mostly driven by the shortfalls of mainstream programming languages, but necessity is the mother of invention - even though your average Java business app was much simpler than the Smalltalk IDE before you added a single line of code to write your app, it was already too complex to maintain and “divide and conquer” was the answer. Service Oriented Architecture, and later microservices, was born. Split your codebase, split your teams, create a lot of opportunities for mediocre coders to grow into mediocre engineering managers, everybody was happy. Including your hardware vendor, because suddenly you needed much more hardware to run the same workloads. Because networks are slow, and whereas it can be argued whether a three tier system is actually a distributed computing system, a microservices system certainly qualifies for that label.

The return of the Job Control Language this time was in the form of, again, configuration data on how to run your microservice. Microservices were somewhat fatter than the very fine-grained objects of the old days, so there was less of it, but still - it was there. The feedback cycle became worse as well: in the monolith-with-XML days, your XML editor would get you mostly there and a quick local compile and run would leave you all but certain that your configuration was working. XML, however, was universally rejected in favour of things like JSON, Yaml, HCL, Toml - all free of structure, with zero indication whether a computer would find your prose gibberish or the next Shakespeare play until you actually pushed your code to some test cluster. It suddenly felt a lot like being back at that university interacting with a mainframe, but at least you still owned the hardware and could introspect all the way down, especially if you were doing “DevOps” meaning mostly that you had administrative access to your hardware.

Microservices have trouble scaling, and they are very complex. Most companies that employ them have no need for them, but the systems and programming languages they employ are sufficiently lacking that this stacking of complexity on top of complexity becomes a necessity. It seems that, contrary to what everybody is saying, software developers are in plentiful and cheap supply so wasting a lot of their talent makes economic sense over the perceived cost of adapting more powerful programming systems. C’est la vie. The feedback cycle is truly broken - testing a microservice is merely testing a cog in a machine and no guarantee that the cog will fit the machine - but we just throw more bodies at the problem because Gartner tells us this is the future.

So, we’re now at the next phase of this game: maintaining these very complex systems is hard and not your core business, so outsource it (simplifying it would cost too many managers’ jobs, so that is not an option). The Cloud was born, first as a marketing label for an old business model (offering “virtual private servers” to the public), but more and more as a marketing label for an even older business model (the mainframe - we run it, we own it, you lease capacity).

Apparently, Worse Is Better and you can do worse than Virtual Private Servers, so through a short-lived detour through containerizing microservices and deploying them on a distributed scheduler like Mesos, Nomad, or Kubernetes, we have arrived at “Serverless”. You deploy individual stateless functions. But not inside a Java monolith, that is old, but on top of a distributed system. You would have been laughed out of the door if you had proposed that in 2000, and you should be laughed out of the door right now, but such is the power of marketing. So what have we now? A “mono repo” codebase, because clearly a Git repository per function in your system would be too much, a large deployment descriptor per fine-grained component, which Spring maybe called “Controller” but is now called “Function”, and instead of combining them all on your desktop, you send them off to someone else’s mega-mainframe. You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow. At least we’re not having to walk to the printer on every try, but that pretty much sums up the progress of the last half century. Oh, and a “Function” will be able to handle a single request concurrently, so here’s your AWS hosting bill, we needed a lot of instances, I hope you won’t have a heart attack. Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise.

History will repeat itself, of course. Conceptually, hiding a lot of the scaling and coordination machinery is not a bad idea; programming systems like Erlang and OTP have shown for decades how well that can work and Elixir is giving the platform a well-deserved surge in popularity. But there’s a big difference here: a platform like OTP handles pretty much everything that a platform like AWS Lambda handles, but it does it in a single programming language. Tools of the trade are available: you can refactor Erlang code, you can write Elixir macros, all to keep the system clean, malleable, and free of accidental complexity.

It is called “Configuration as Code”, and it truly is a great idea (but hardly a new one). But XML is not code, and neither is JSON, nor YAML, nor HCL. There is an essential quality lacking in all of these markup languages, and that causes all the copy/paste programming I am currently seeing everywhere, whether a team deploys on Nomad or uses Terraform or CloudFormation to whip up mega complex clusters to run a marketing site. This level of complexity is not sustainable, and my fear is that it will be solved the way our industry likes to solve the problems they have created themselves: by adding more complexity on top if it.

Not by going back through our history and figuring out what caused the Personal Computer Revolution. Which is a shame.

Building a chicken coop dusk timer with NodeMCU and Lua

Mon, 02 Nov 2020 20:02:03 -0500

One of the more fun bits of hardware I have acquired in the last couple of years is the NodeMCU development board.

This little board contains the Espressif ESP8266 WiFi-enabled microcontroller which packs a punch: a 32bit RISC processor clocked at up to 160MHz, on board WiFi, very power-efficient “deep sleep” modes, and the NodeMCU board adds power management, 4MB flash, and some odds and ends without increasing the footprint much beyond a small Arduino. The best thing is that your local store will have it for around $5 and if you source if from China for around a dollar and change. I’m a big fan of Elixir and especially Nerves, but I’m also Dutch and therefore a cheapskate - I’ll use a cheaper part if I can and the ESP8266 displaces a lot of stuff where years ago, you would only have considered a Raspberry Pi.

NodeMCU is a development board and an SDK, and the SDK is in one of my favorite little languages, Lua. Lua was made for embedding, it is hugely popular in a lot of industries, and rightfully so - the whole official language and run-time environment weigh in at ~20k lines of C and ~10k lines of Lua code, there are multiple highly compatible implementations to choose from (including Luerl which implements Lua from scratch in Erlang), and given its size, it is not hard to run it on really small machines, like the ESP8266. NodeMCU also comes with excellent documentation and a library containing all the things you will need for your embedded project.

So why am I writing about this? We have chickens. That’s why.

Chickens are awesome creatures. They hatch and then they come preprogrammed with everything they need to know to survive - we know, because we got ours as day-olds and we certainly did not teach them how to swallow a young frog whole. We let ours free roam and being excitable and curious creatures they often forget to check back into the coop in time. So they enter a dark coop around dusk, cannot see the roost, and then sleep somewhere else with the risk of being pooped on by the ones that did make it back in time. Every night, we check them on the coop cam (a Wyze 2, never owned a very small cube that runs Linux and talks to me before), notice it happened again, get up, and rectify the situation manually.

Therefore, what’s needed is some extra light around dusk.

The simplest definition of “dusk” is “when it gets darker”, and I started hooking up a photoresistor to the dev board’s A/D converter, which worked fine, but then I thought about how to calculate, with certainty, that it gets darker because of dusk and not because of a thunderstorm. Yes, I could have just pivoted and turned the “dusk light” into a “it is getting somewhat dark for any ol’ reason” light, but I was not ready to do that yet. Therefore, I needed a way to know what time it was, probably. I said that NodeMCU comes with a rich standard library so I was very happy to find the sntp module, a simple NTP client.

Hmm… so if I know, from NTP, what date it is can’t I just calculate dusk instead of relying on a photoresistor? Less hardware, more software, just up my alley. Not a lot of stuff from Lua land came up, but Wikipedia has an entry on the Sunrise Equation that felt like something I could translate into Lua. So, I started out with a test:

function test_sunrise_sunset()
   -- Toronto
   local lat = 43.740
   local lng = -79.370

   -- Somewhere on 1 Nov 2020
   -- According to https://nrc.canada.ca/en/research-development/products-services/software-applications/sun-calculator/:
   -- Sun rise: 6:54 LT / 11:54 UTC
   -- Solar noon: 12:01 LT / 17:01 UTC
   -- Sun set: 17:07 LT / 22:07 UTC
   local jd = posix2julian(1604250457)
   local rise, noon, set = sunrise_sunset(lat, lng, jd)
   assert_equal("2459154.9966005", ""..rise)
   assert_equal("2459155.2098313", ""..noon)
   assert_equal("2459155.4230621", ""..set)
end

Note the string compares - pure laziness, because lunit does not have something to compare floats with a delta and I was not going to let me be sidetracked in finding something else/better or even writing a small test framework from scratch. The code was a relatively simple transcription from Wikipedia, with some thinking over where to put degree-to-radian conversions as Wikipedia worked in degrees but most math libraries want, for good reasons, radians. It disagreed with the National Research Council of Canada’s calculations by one or two minutes, which could simply be because of altitude corrections. I decided that for my application, this was immaterial so I manually verified the calculated results and punched them back into the test assertions as a future safety net, hence the very precise numbers the test uses to compare.

Without further ado, I landed on this:

local sunrise_sunset = function(lat, lng, jd)
   local julian_date = math.floor(jd) -- note, by the way, that ".0" equals noon,
                                      -- not midnight, as the Julian date numbering
                                      -- started at 1200 somewhere in 4713 BC

   local n = julian_date - 2451545.0 + 0.0008 -- julian day
   local j_star = n - (lng / 360.0) -- mean solar noon
   local m = math.rad((357.5291 + 0.98560028 * j_star) % 360) -- solar mean anomaly
   local c = 1.9148 * math.sin(m) + 0.0200 * math.sin(2 * m) + 0.0003 * math.sin(3 * m) -- equation of the center
   local lambda = math.rad((math.deg(m) + c + 180 + 102.9372) % 360) -- ecliptic longitude
   local j_transit = 2451545.0 + j_star + 0.0053 * math.sin(m) - 0.0069 * math.sin(2 * lambda)
   local earth_tilt = math.rad(23.44)
   local delta = math.asin(math.sin(lambda) * math.sin(earth_tilt)) -- declination of the sun
   local correction_angle = math.rad(-0.83)
   local lat_rad = math.rad(lat)
   local omega_0_divident = math.sin(correction_angle) - math.sin(lat_rad) * math.sin(delta)
   local omega_0_divisor = math.cos(lat_rad) * math.cos(delta)
   local omega_0 = math.acos(omega_0_divident / omega_0_divisor) -- hour angle
   omega_0 = math.deg(omega_0)

   local j_rise = j_transit - (omega_0 / 360.0)
   local j_set = j_transit + (omega_0 / 360.0)

   return j_rise, j_transit, j_set
end

Note that this works with Julian dates which I think is a concept that deserves more attention in the software world. Given that the POSIX timestamp and the Julian date are both offsets from a given epoch, the conversion is trivial:

local posix2julian = function(posix_timestamp)
   return (posix_timestamp / 86400.0) + 2440587.5
end

Where the 2440587.5 represents the difference between 1970 and 4713BC. One odd thing happened with my code which was a headscratcher until I realized that the Julian epoch starts at noon (UTC) and not midnight, and that cost me a lot of time. I could not understand why the outcomes where half a day off :).

I wrote the code and tests on my Linux box, but the same code worked on the ESP8266 once I rebuilt the firmware with two options:

Use the included Lua 5.3 instead of the default Lua 5.1; the 5.3 version includes the math standard libraries;
Realize just how bad single precision floating point works for this sort of stuff and switch Lua to double precision.

Once that was done, everything worked fine. I now had a microcontroller that could, theoretically, ask the Net for the exact date and calculate sunset from there. My first computer was a TRS-80 so yeah, mind blown.

NodeMCU’s “Node” comes from its borrowing Node’s concurrency model, heavily relying on callbacks. I really do not like that programming model, because you invariably end up mixing up concurrency and business level code, but for this it worked just perfectly. We boot, some housekeeping is done (like attaching to WiFi), and then we ask what time it is:

function ntp_sync()
   sntp.sync(nil, ntp_callback, ntp_err_callback, nil)
end

gpios_off()
ntp_sync()

gpios_off switches off the LEDs (we’ll get to the hardware later). The sntp module has a sync command that does two things: it sets the time in the ESP8266’s RTC (so that subsequent calls to the rtctime module give the right answer) and it calls a callback function with some info on the NTP response (the error callback I pass in just reboots the MCU). The callback then calculates sunset, calculates the time that the lamp should be on (I put that half an hour before sunset), and depending on whether we’re in the lamp on interval or not, does the right thing. The whole thing is in Github so I’m not going to copy/paste, but for example if the lamp needs to be switched on, then this gets called:

function turn_lamp_on(time_to_lamp_off)
   log.info("Turning lamp on for "..time_to_lamp_off.." seconds")
   gpios_on()
   if not tmr.create():alarm(time_to_lamp_off * 1000, tmr.ALARM_SINGLE, function()
                                log.info("Lamp has been on enough, turning off")
                                gpios_off()
                                sleep_until_next_round()
                            end)
   then
      log.error("ERROR Could not create timer until lamp off, rebooting.")
      node.restart()
   end
end

Like most functions there, it ends up doing something and then setting up a timer with a callback to do the next thing. As far as state machines with delays go, I found it a very clean and simple to reason about way to write code, and it was pretty much one of the first times that the callback model for multithreading looked like it found its sweet spot. The state machine breaks down in four functions that do something and then call or schedule each other.

You would expect me to write something about testing now. I should. I won’t. But it’d be extremely trivial in Lua to have dummy implementations of tmr and sntp and friends. Instead, I decided that there are just a couple of use cases, it was around dusk anyway, so I just rebooted the MCU at set times to see whether it made the right decisions. It did :). Time for hardware.

Breadboard first, of course. While the target hardware uses left over LED strips, these take 12V so I verified that things worked using regular LEDs. The transistors (I used 2N3904 for the very good reason that I had them) are triggered through two GPIO pins with a 1KΩ resistor as a current limiter between GPIO and base, and the LEDs are between 5V and the transistors’ collectors. The emitters are wired to ground. Needless to say, this is 101 electronics, if it even makes that bar and everything just worked. Still, had to verify with the blinkenlights, not?

Why two transistors, I hear you ask. Well, I only had low power transistors and the end result would use a 12V LED strip that was left over; it has six LEDs on it and on measuring it on my bench power supply, it drew 50mA. 50mA times 13.6V is 680mW which is higher than the transistor’s 625mW max power dissipation. Instead of calculating and measuring more to see how much the transistor would actually dissipate, I decided to use the LED strip’s hint and cut it in half (these strips have a cut mark every three LEDs) and just trigger two GPIOs. Shorter strips are simpler to work with anyway. For those that, like me, are not electronics wizzards, the schematic:

and, for a good laugh, the end result, mounted dustproof in an old pickle jar:

The jar is dustproof, fireproof, and transparent so it is the perfect enclosure for a chicken coop. The jar is fed by 12V from a solar+battery combination, and I added a DC-DC voltage regulator; I once bought a handful of these; they are based on an LM259 switching regulator and they’ll eat anything up to 40V and spit out whatever you need. The LEDs get the 12V directly and the NodeMCU receives 5V through its USB port (I could have soldered the 5V directly on the NodeMCU but this way the device either gets 5V or is connected to my laptop, never both at the same time which I think is safer). Next up is install in the coop, as soon as the snow stops :). Because it is in the coop, and I never trust my code, I added remote logging as well so that a PC that is always on now shows exactly what is happening in its /var/log/syslog:

Jan  1 00:00:00 ESP-E1B27B Waiting for IP address...
Jan  1 00:00:00 ESP-E1B27B Wifi connection is ready! IP address is: 192.168.17.219
Jan  1 00:00:00 ESP-E1B27B Startup will resume in five seconds, in case of panic loop execute `file.remove("init.lua")` on console
Jan  1 00:00:00 ESP-E1B27B Initializing CoopLight™ (v1.0)
Nov  3 11:13:48 ESP-E1B27B Synchronized NTP with 199.182.221.110
Nov  3 11:13:48 ESP-E1B27B Current POSIX time is 1604402028
Nov  3 11:13:48 ESP-E1B27B Current Julian time is 2459156.9679167
Nov  3 11:13:48 ESP-E1B27B Sunset in: -45935 secs
Nov  3 11:13:48 ESP-E1B27B Sleeping for an hour...

I hope I got your attention for this wonderful and cheap IoT device and development environment. The whole build took me maybe two days, and for much less than a Raspbery Pi I now have something that does the trick, is network connected, easy to upgrade (although not yet over-the-air), and will tell me what it is doing through syslog to a PC that is always on and has remote logging enabled. Using ESPlorer development is highly interactive and Lua seems to me like a language tailor-made for this kind of device.

Also, I hope you had fun reading this, of course :)

Code can be had from Github.

Fun with macros: anaphoric lambda

Mon, 01 Jun 2020 21:02:03 -0400

In “On Lisp”, Paul Graham introduced the concept of “anaphoric macros”, a term he borrowed from linguistics. We can have them in Elixir as well, and while they’re dangerous because code written in them might confuse people, they are fun to play with.

So what is an anaphoric macro exactly? Let’s start by example: Common Lisp has the loop macro which is pretty much for with everything but the kitchen sink. I think books have been written about it. From the Wikipedia article on anaphoric macros:

(loop for element in '(nil 1 nil 2 nil nil 3 4 6)
       when element sum it)
 ;; ⇒ 16

Like an anaphora in linguistics, it refers to something that came before, in this case the result of the conditional clause. It just magically appears, without warning, poking up its head like a jack-in-the-box and you need to know about the loop macro to know that you can use it here. But, for its expensive and posh sounding name, it’s a simple concept: a macro that on expansion introduces variables the user didn’t specify.

A more interesting example is when you make an anaphoric lambda macro:

(defmacro alambda (parms &body body)
   `(labels ((self ,parms ,@body))
      #'self))

This macro packs a punch: it introduces the lambda expression as an anaphoric variable in the macro, which means that you can refer to it from the lambda expression. The result is that you can write:

(alambda (n)
   (if (= n 0)
     1
     (* n (self (1- n)))))

Look - an anymous function that can call itself recursively supported by just three lines! Much nicer than the Y combinator version which is too long to list here. On the same page, the Elixir Y combinator solution is much shorter:

fib = fn f -> (
      fn x -> if x == 0, do: 0, else: (if x == 1, do: 1, else: f.(x - 1) + f.(x - 2))	end
	)
end

y = fn x -> (
    fn f -> f.(f)
  end).(
    fn g -> x.(fn z ->(g.(g)).(z) end)
  end)
end

IO.inspect y.(&(fib.(&1))).(40)

It works, but it is hardly elegant. Can we have an anaphoric lambda in Elixir?

Well, yes, but it is somewhat harder. While Elixir has a pretty powerful macro system it has a very clear and compile time distinction between “data” and “code” and the sort of on-the-fly function definition thing that the Lisp version leans on using labels won’t cut it. The simplest I got to work is this:

defmodule AnaphoricLambda do
  defmacro alambda(fun) do
    r = Macro.var(:self, nil)

    quote do
      unquote(r) = unquote(fun)
      fn (x) -> unquote(fun).(x, unquote(r)) end
    end
  end
end

defmodule Test do
  import AnaphoricLambda
  def test do
    fib = alambda fn
      x, _ when x < 0 -> raise "Stay positive!"
      0, _ -> 0
      1, _ -> 1
      n, self -> self.(n - 1, self) + self.(n - 2, self)
    end
  end
end

IO.inspect Test.test()

This is short but not very elegant. We inject the function into itself to enable the recursion, but that means that we need to write the single-argument function as a two-argument one where the second argument is the anaphoric variable.

Now, I’m not sure that this is an optimal solution but I puzzled over it for a couple of hours and this is the best I could come up with (I’m writing this post partially in the hope that someone will correct me and come up with something nice - I’m hardly a macro specialist). Note the

r = Macro.var(:self, nil)

which creates a variable self in an empty context. If you start digging through the ASTs during macro hacking then you’ll see that you need this nil context for such local variables. It expands to just {:self, [], nil} but writing it this way shows intent better.

What we want is a simple one-argument fn and the macro then has to tweak things. Without further ado:

defmodule I do
  defmacro alambda(block) do
    r = Macro.var(:self, nil)
    block = Macro.prewalk(block, fn
      {:when, meta, elems} ->
	{:when, meta, [r | elems]}
      b = {:->, meta, [lhs, rhs]} ->
        case lhs do
	  [{:when, _, _} | _] ->
	    b
	  _ ->
	    {:->, meta, [[r | lhs], rhs]}
	end
      { {:., meta, [{:self, _, _}]}, _, args} ->
        { {:., meta, [r]}, [], [{:self, [], nil} | args]}
      other ->
	other
    end)
    block = quote do
      r = unquote(block)
      fn x -> r.(r, x) end
    end
    IO.puts Macro.to_string(block)
    block
  end
end

defmodule Test do
  import I
  def test() do
    fib = alambda fn
      x when x == 1 or x == 0 ->
        x
      x ->
        self.(x - 2) + self.(x - 1)
    end
    IO.inspect fib.(20)
  end
end

Test.test()

We made the invocation quite nice: the alambda expression now has just a single argument and the self is magically injected. But that macro is enough to give you a headache (and it is hardcoded to single argument functions, although that shouldn’t be too hard to fix). We use Macro.prewalk/2 to walk through the function body and we match for three clauses in the AST:

{:when, meta, elems} matches a when clause. elems are the arguments to when and this is a flat list of parameters and then the when expression as the last element. We simply prepend self to the parameters so that x when x == 1 now reads self, x when x == 1.
{:->, meta, args} matches a regular function head. We need to exclude a when clause in therei because it gets processed in the first clause but otherwise we do the same trick: x -> is changed to read self, x.
Finally, { {:., meta, .... matches a function call. When we recurse from the function body, we need to rewrite self.(x - 2) into self.(self, x - 2) and that completes the expansion.

There is debugging code in the macro code to print the expansion, and this is what we create:

 r = fn
    (self, x) when x == 1 or x == 0 ->
      x
    self, x ->
      self.(self, x - 2) + self.(self, x - 1)
  end
  fn x -> r.(r, x) end

Pretty much what we wrote manually before, and this compiles and runs. The first clause causes an “unused” warning because the self there really should be a _self but as the author here I can leave that as an exercise to the reader.

Anaphoric lambda, yay! Now, are we loosing a lot of performance? The reason I switched from factorial to fibonacci is that the trivial recursive implementation is bad. It is even O(2ⁿ) bad, people on StackOverflow tell me (and it is true because that answer got upvotes). Using :timer.tc/1 and a trivial regular function definition, we can compare:

def fib(x) when x == 0 or x == 1, do: x
def fib(x), do: fib(x - 1) + fib(x - 2)
...
IO.inspect :timer.tc(fn ->
  fib.(20)
end)
IO.inspect :timer.tc(fn ->
  fib(20)
end)

This very scientific benchmark shows 1.4ms for the anaphoric lambda version and 1.2ms for the regular function. Not bad, the overhead for the anaphoric lambda doesn’t seem to be very high, but O(2ⁿ) starts showing. Let’s push it a bit and calculate the 40th number instead of the 29th (the tc/1 function returns a tuple with the time in microseconds and the function return value):

{21719438, 102334155}
{18201677, 102334155}

Ouch. 2ⁿ hits hard. And I wanted to know the 400th Fibonacci number! Can we do better? Yes we can, with an all-time favorite macro writers tool, “memoization”. We sneak in some time-for-memory tradeoff by expanding the macro so that we only calculate any given fibonacci number once. The whole variable injection stays the same, but we now introduce a unique key for this particular macro and then return code that stores results in ETS so they only need to be created once:

    ...
    memo_key = :erlang.unique_integer()
    retval = quote do
      r = unquote(block)
      m_r = fn self, x ->
	case :ets.lookup(Memoize, {unquote(memo_key), x}) do
	  [] ->
	    v = r.(self, x)
            :ets.insert(Memoize, { {unquote(memo_key), x}, v})
`	    v
	  [{_key, val}] ->
	    val
	end
      end
      fn x -> m_r.(m_r, x) end # TODO more than one arg
    end
	...

This now results in the following expansion:

r = fn
    (self, x) when x == 1 or x == 0 ->
      x
    self, x ->
      self.(self, x - 2) + self.(self, x - 1)
  end
  m_r = fn self, x -> case(:ets.lookup(Memoize, {-576460752303423327, x})) do
    [] ->
      v = r.(self, x)
      :ets.insert(Memoize, { {-576460752303423327, x}, v})
      v
    [{_key, val}] ->
      val
  end end
  fn x -> m_r.(m_r, x) end

We first lookup in ETS, and if there’s a value we return that. We smash O(2ⁿ) to O(n) and that shows:

{14, 102334155}

Note that the really nice thing here is that we did not have to touch our code. The alambda fn ... is untouched but we sped it up by, well, a lot of orders of magnitude. Memoization truly can be a miracle!

If you want to use memoization for more serious purposes than this code diversion, there are multiple packages on Hex that will help you out. The version here is lacking because there is no way to clean out unused values (you probably want something with a TTL, for example).

Also, if you want to generate fibonacci numbers quickly, here’s an O(1) version:

def Fibonacci do
  @φ (1 + :math.sqrt(5)) / 2
  @sqrt_5 :math.sqrt(5)
  def fib(x) do
    round(:math.pow(@φ, x) / @sqrt_5)
  end
end

It probably shows that math trumps recursion and certainly shows that tricks like anaphoric macros and memoization are best left in the toolbox until you’re sure you need them. But I had fun with them, and that counts for something :)

Erlang stuff that Elixir developers should know

Mon, 23 Sep 2019 21:02:03 -0400

A lot of people are learning Elixir these days. Just at my employer, more than 150 people have taken Dave Thomas’ excellent video course. Learning Elixir is simple, but you’ll also be embracing a new ecosystem, the Erlang/OTP ecosystem; treating it as a black-box is just going to make life harder, so let’s open it.

There’s a lot to be said to learning Elixir, even if you don’t need it for work:

Given how hard it is to make anything mutable, it’s a great way to force yourself into good functional programming habits;
There’s something in how the OTP actor system makes distribution and concurrency easy to reason about;
Phoenix, and especially LiveView is a great way to build websites that scale;
Nerves is mindblowing if “large embeded”, like Raspberry Pi, is your thing. Trust me—try it and then over-the-air update some firmware.

The ecosystem is not yet as rich as, say, Python’s or Ruby’s, but mostly what you want is there. On top of that, the language is much cleaner, development is probably faster, and the end result will actually run at speed and scale well. Furthermore, as I said, it is simple to learn; there are some harder areas like macros (I think that Saša Jurić’ introduction is an excellent one), but overall it is mostly “nothing to see, no magic, no sleigh of hand”.

However, the way Elixir achieves all that goodness is by literally standing on the shoulders of a giant: the Erlang/OTP ecosystem. Elixir is an extremely well-designed language, but José Valim had a relatively easy job of creating a run-time library given that most of it is a thin wrapper around the already excellent code that Erlang and OTP provide. Needless to say, there is a point where you have to look down, notice the giant, and interact with it.

Therefore, given my proclivity for advocating “machine affinity”, here’s a list of stuff you should do to learn more about the system you’re working on. Elixir does, in most places, an excellent job of hiding Erlang and OTP from you, but there are times where it makes sense to remove the veneer and look at the raw material underneath. The list here is roughly in order of importance/complexity and all but complete.

Learn some Erlang

Erlang is a language from a different time. Whereas languages these days let themselves be inspired by Python or Ruby, Erlang comes from a time where things like Prolog where what you looked at. As such, it’s a bit “funny”: variable names start with an uppercase, statements are separated by commas, atoms are just lowercase words and thus don’t stand out like :atom does, and so on. Learn you some Erlang and other introductions abound; work through one until you are at least comfortable reading and maybe patching Erlang code. A lot of libraries rely on Erlang libraries and there will be a time you will have to dive in.

Learn some OTP

A lot of the power of Elixir directly derives from the power of the famous Open Telecom Platform, whether you write telco software or not. The Elixir docs mostly tell you how to use OTP elements like GenServers and Supervisors and Applications, but are thin on details. For good reason: why replicate documentation that already exists? The OTP Design Principles User’s Guide goes into details and I consider it mandatory reading for everybody who is serious about Elixir.

Learn about releases

Since Elixir 1.9, mix release is a freebie. Before that, Distillery did a great job. However, here is one of the spots where Elixir and Mix muddy the waters a bit too much: by going out of their way to be user friendly, a lot of the underlying details are hidden and that often results in confusion and black-box prodding (try something and see whether it works in production, say).

Here’s a small exercise. Open up a command line, and check that elixir and mix are version 1.9. If not, make it so (I use asdf-vm for that). Then:

cd /tmp
mix new releases
cd releases
mix release
cd _build/dev/rel
find . -print

Questions you now want to answer:

What files are here? What are their roles? What are, specifically, the ‘.script’ and ‘.boot’ files?
What happens if you bump the mix.exs version number and run mix release again?
What happens if you add a dependency and run mix release again?
What happens if you add :observer to extra_applications and run mix release again?

You just read the OTP guide, including the chapter on Releases, so what is in there does make some sense. Note especially how files you would manually create and maintain when coding in Erlang are mostly generated for you with Elixir.

For bonus points, create or copy a simple Erlang project and build a release. Compare with an Elixir release.

Learn about the Erlang standard libraries

People joke that Erlang is not a language, but an operating system. To some extent, that is true because the system comes with an enormously broad list of libraries. All these are pretty much free to use (they’re already installed) and rock solid (they run a good chunk of the world’s telecom operations), so should be a first option to use. Some pointers:

The ERTS documentation contains the basic library documentation, including distribution (the chapters on building custom distribution and discovery protocols are very interesting) and the :erlang module which has a pretty random collection of sometimes very handy utilities.
Browse through the reference documentation of the bundled applications; there is a lot there, and what you want to skim through is most likely going to be influenced by your interests. I’m sure there is someone out there who is going to jump with enthusiasm when discovering that there is a bundled ASN.1 to Erlang compiler in there.

Learn about ETS and Tracing

ETS is the main way to share state in the Erlang/Elixir ecosystem, and it is well-documented. Tracing is something completely different, a way to inspect a running VM. It is documented here, here, and probably some pages that lead from there.

Now why would I bunch these two together?

The commonality is called “match specifications” and it is an important topic to pick up when you go deeper; it enables programmatic tracing and advanced ETS operations and it is in fact a small, primitive mini language you write using ASTs. Erlang has a compiler available to compile functions to match specifications, and a neat exercise would be to create one for Elixir too.

Dive into the OTP source code

I’m out of ideas here but if you completed the above, you’re well underway on the path towards expertise; what’s more, you can probably find where things are now. There’s always a point, though, where you want or need to know exactly what is happening. To that end, I always have a checked out copy of the OTP source code lying around, built and ready to go if I want to add some debugging printf() statements or just want to lookup what is happening (I also have the Elixir source code always checked out, but it is simple enough to jump from documentation to source code on Hexdocs so there’s a bit less need for that).

Want to know what happens if you write Kernel.send/2? Grab the OTP code, and dive right in. There are two main directories:

erts/ has the “emulator” (the BEAM VM and associated stuff) and the base libraries (so you’ll find the send call there, most likely as a built-in function (BIF));
lib/ has the bundled Erlang libraries, mostly a mixture of Erlang code and native interface functions (NIFs)

I’ve found the code well-readable and reasonably commented, so to me it has been a fallback that, while not needed often, has proven an invaluable resource during that one or two times I really needed to know what was going on.

Updates

I’m happy that some people came back with excellent suggestions of more material to read:

Erlang in Anger. Yes, I slapped my head when someone mentioned that one. I really, really should have included it. It’s a great resource about debugging.
Efficiency User’s Guide. The name is a bit odd, but most of the material in here is a good read. Start maybe with the chapter on performance myths.

The guide also has a chapter on binaries and that one is really important to read. You will end up in a situation where having some knowledge about binaries and their allocation and garbage collection strategies are going to matter, and a very timely example was just published by my colleague Tyler Pachal on Medium. It’s a great read on why all of the above actually really matters.

The Programming Language Conundrum

Wed, 27 Mar 2019 21:02:03 -0400

A lot has been written about “hyperproductive teams”, and a lot has been written how programming languages help here.

Beating the Averages is probably the most famous one. I myself worked in Smalltalk full-time, and can share the same kind of story. So why aren’t we all using either of these two languages?

When I created my own startup, an Internet hosting company, I went down the same path as Paul Graham: two coders, me and a full time one, a VisualWorks Smalltalk license, and we managed to develop something that was utterly impressive given that we had 1.5FTE on the job. Hosting provisioning, accounting, all sorts of really advanced stuff around whole-sale packaging, multiple currencies, and so on. Later on, I worked at a larger Smalltalk shop and between 5-6 developers we completely beat large companies employing ten times as many people. So why aren’t we all using either of these two languages?

I can recite many of such stories; of people keeping up with their ten person team by coding whatever the team did during the day in 2-3 hours at night using Squeak Smalltalk; about an old Lisp hacker working in a research setup with mostly Java colleagues and always estimating to need 10% of the time (and then actually making the estimate). So why aren’t we all using either of these two languages?

The answer is multifaceted, of course. There’s a fear, often more with management than actual techies, of choosing a non mainstream language because hiring for that language will be too hard (wrong). Tooling for niche languages often is not as good as for mainstream stuff (quick, find a way to deploy a Pharo Smalltalk image on EC2 or in a container). The list goes on and on and usually every point can receive the same reaction: yes, that might be true, but if the payoff is that big, why not make the investment?

However, there’s one thing in common between all the stories I’ve heard and the successes experienced first hand: small teams. Every single time, you are looking at a software development shop with a grand total of, say, less than a dozen developers in a pretty stable setup.

And that, I think, is the only way to use these very powerful languages (and do not forget: they are that powerful). Because, each in its own way, these languages encourage you to solve your problem in two steps:

Create a language to solve your problem;
Solve your problem.

Lisp is more explicit about this, but I opened an old Smalltalk image and the “application” language ended up being very high level, very domain specific, and just as hard to grok as your average macro-ridden Lisp system.

These languages are very powerful, but they don’t scale. So even when you get ten times the productivity, the mostly tribal complexity will not scale beyond your “pizza-sized team” and therefore you now have a hard upper limit on your total software development capacity.

Harsh, isn’t it? But I cannot find counterexamples. Whereas I can find plenty of examples of teams using “normal” programming languages (Java, PHP, Python, Ruby, or newer entrants like Elixir or even Golang) that breeze through the 100, 200, 500 developer marks without much trouble. So I must conclude that there seems to be a scaling limit.

Why? My theory, completely made up and untested, is that these powerful Lisp/Smalltalk systems weave code and brains together in a way that amplifies both. It’s very organic, very hard to document, and thus it becomes very hard to absorb new brains into the system. When I joined the small Smalltalk group it was very hard to get started in that code base. It was complex, and completely different from the previous thing I worked on. Both were using the same VM (VisualWorks) but both had their IDEs adapted to a point that they became new and very well-honed tools and new things in their own right. In Smalltalk, there’s no difference between writing “business” code and writing “tooling” code, and even no difference between that and changing your IDE, so everything happens interchangeably and fluidly and you end up with an IDE that’s adapted for the problem at hand, not for everybody’s generic problems. It’s hard to ramp-up in these environments and certainly in the light of the short retention times these days, likely won’t work.

And on that terrible disappointment… I think I’ll end while hoping that someone out there on planet Earth reads this and proves me wrong :)

Love thy product owner...

Tue, 18 Dec 2018 20:24:00 -0500

People regularly find me quoting a Goethe poem called “Art and Nature”. It is the only Goethe poem I’ve read and remembered (except probably during German classes in highschool, but that’s repressed memory), and the bit I quote is at the far end; I like to quote it because it contains such a cardinal truth in such a succinct statement that I’m jealous at the power of poetry.

Here’s the poem translated by David Luke:

Nature and Art, they go their separate ways,

It seems; yet all at once they find each other.

Even I no longer am a foe to either;

Both equally attract me nowadays.

Some honest toil’s required; then, phase by phase,

When diligence and wit have worked together

To tie us fast to Art with their good tether,

Nature again may set our hearts ablaze.

All culture is like this; the unfettered mind,

The boundless spirit’s mere imagination,

For pure perfection’s heights will strive in vain.

To achieve great things, we must be self-confined:

Mastery is revealed in limitation

And law alone can set us free again.

The last three lines are the interesting ones, and I keep coming up at situations where they apply, especially at work.

A lot of people, especially those newer to the field, are awed by the fast pace of development and the constant stream of new toys. And people, techies first, want to play with these new toys: new languages, new databases, this whole cloud thing, surely there must be a problem that’s just dying to be solved with a new toy.

However, if you’ve been around for long enough, you notice patterns and repeat cycles. After some thirty years of doing paid coding work, there is the realization that these new toys are invariably pushed by very skilled sales people and that they never present a silver bullet, and that the actual hard parts of our work are more mundane and less prone to change:

Write really good code, which is made hard because good code wants to be simple and business requirements usually don’t have that characteristic;
Work with Peter L Deutsch Fallacies of Distributed Computing and know your systems’ limitations. Hint: a website is a distributed system, so there’s a fair chance that you’re working on a distributed system right now.
Know design patterns and when and where to apply them (I’m a card-carrying Hillside Group member because I think that patterns are a very powerful tool to distill and share knowledge).

There’s probably more to that list, but this is what pops into my mind right now as things that haven’t really changed since I started typing away on my first paid gig. As the French say, “plus ça change, plus c’est la même chose”. As I progressed from deployment through a 5.25” floppy disk handed out to my first customer the tools changed, but the business problems I was solving and the way in which we went about it did not change a lot. And the shinies that sales people and enthusiasts told me to use sometimes helped, sometimes not. As I progressed, I started focusing more and more on becoming really good at these core skills and less on a language, a library, or the newest way to send messages from A to B reliably (something solved in the ’80s with store and forward style mail delivery systems, by the way).

What does this have to do with “Love thy Product Owner?”

A PO will let you focus. A PO has a list of business problems, ideas, and whatnot that is good for four years of work for your team, everything must be done yesterday because a competitor popped up, and can we please stop talking and start typing?

As the prime representative of the business you’re working in, your product owner will rub your nose in the dirty every day reality of doing business and will make it clear that playing in the sandbox with new toys is not something that will make them, or by extension the business, happy.

Your product owner restricts you. They are the “limitation” in Goethe’s poem. Your company’s architecture, chosen stack, tools, and production platform is the law. They work together to allow you to shine, set you free, and show your mastery. Mastery in solving business problems by focusing at the domains I listed above, and becoming really good at them. Not mastery by getting a new tool from the shelf and applying it because a) it’s more fun that way and b) maybe it will actually solve your problem this time. There’s a time and space for that, of course, but make “innovation” the exception, not the norm.

The norm is good, honest, writing code. The best code you can deliver and that makes your business happy now and in the future. Sure, learn new languages, toy with new tech - it’s important to see what’s out there because some day, your PO will come with a new idea and one of the things you toyed with 8 months ago seems to fit the bill exactly. But it is a side gig, and frankly mostly an optimization; your real leverage sits in your core skills. Writing good code is hard, but delivering good code is extremely fulfilling. And it is more satisfying than chasing new shinies, toying with tools that come and go over time, and - worst case - being in a development team that has to spend half its time on maintaining three language stacks, two relational databases, four messaging systems, and knowledge about half AWS’ product catalog just because your predecessors tried to solve relatively simple business problems by just constantly hunting for new tools.

Chose a stack, stick with it. Embrace the law it lays down, as it lets you focus; love thy product owner, because a decent one will put the pressure on your team that will help you excel.

(If you made it here: thanks for reading, and big thanks to @dileshnij for proofreading multiple times!)

Integration testing considered harmful

Wed, 12 Dec 2018 20:24:00 -0500

I just watched J.B. Rainsberger’s talk “Integrated Tests Are a Scam” and I urge you to do so as well (I’m gonna refer to him as JBR down below in line with the good old “name people by their TLA” tradition).

It is a pattern that hits me over and over again, and I feel like groundhog day already because this is the fourth gig in a row where I enthusiastically start coding away, then ask “what is this?”, get the answer “oh, our legacy monolith” and commence work on getting rid of it. I guess I’m getting old.

Integration tests. Or, as JBR probably correctly calls them, integrated tests. Tests that don’t test whether the 10 lines of code you just wrote are correct, but also the 10,000 lines below that and a couple of million lines in, say, MySQL. These tests are a scourge.

Just yesterday I changed some code which essentially added two lines to an Elixir “application” (in Elixir/Erlang/OTP, an “application” is basically a coarse-grained component and I’m gonna use the word “component” going forward to not make this language/platform specific). One line to grab a number from a new component I wrote, and one line to actually insert that number in an outgoing Kafka message.

make test showed three components that failed. The component I just added the single line to (I was expecting that, of course), and two components that had that one as a dependency. Graphically:

{C,D}
  \-- B
      \-- A

I added A, B depended on that and I expected some test breakage there, but C and D?

It turns out that C and D had “integration” tests, just like B. Basically test that treat everything as a black box, make real connections to Kafka and MySQL (but arbitrarily stub out others because “too hard to get running in Circle”, like our Rails monolith that this service also talks to), and it is easy to see that if you add behaviour in A, use that to change how B operates, then C and D will suddenly see a differently operating black box and will have failing tests.

I tried to make the tests run which was actually trickier than I thought. After an hour of trying, I filed a pull request instead to just remove these tests. Tests that play dominoes and are hard to fix have no place in good software. Then I remembered some blog posts by JBR, found the video, and watched that instead. Way better use of my time than trying to fix tests that shouldn’t be there in the first place.

Integration tests are usually harmful and deliver no positive value. That is, they do deliver some value, but it is offset by the high cost of maintaining them and therefore they don’t make sense, economically. They are just too darn expensive.

So, what is the solution? Starting with your test suite, if you have a language that so clearly and easily componentizes like Elixir, stub liberally. So, in the above dependency tree:

A tests itself. It tests whether its external API abides with its contract, it may use the stacked testing that JBR proposes, or actually have integrated tests, I don’t care. I have my preferences, but I don’t care that much. One nice thing of componentizing your software is that you can stop caring at component boundaries.
B tests itself, but stubs out A. Always. Again, I’m more than happy to treat B as a black box, but it is not allowed to instantiate A. I also don’t care whether A provides a stub or B creates one, as long as the code in A doesn’t get invoked during B’s test suite.
C and D, likewise, never invoke B but stub it out. Given that B has two modules using it and we’re looking at a single service/repo here, DRY says that B’s stub might end up with B but it is not clear cut - for all I know C and D use B in non-overlapping ways and thus can each have a partial stub.

Oh, and I’m not a linguist either, so I don’t really care whether you call it a “stub”, or a “mock”, or a “test double”. To quote the great Richard Feynman:

“You can know the name of that bird in all the languages of the world, but when you’re finished, you’ll know absolutely nothing whatever about the bird. You’ll only know about humans in different places, and what they call the bird. … I learned very early the difference between knowing the name of something and knowing something.”

I care about what it is, not about what you call it. Yes, one helps identifying the other but the terminology is so tangled up that I’ll look past your code’s name and into its structure to see what it is :).

JBR spends quite some time about explaining about how testing two sides of the contract should prevent issues when you actually integrate code. I think he knows that there will be gaps here, and I hope that he’ll have the same opinion as I do: “this is fine”. The goal of your test suite is not to eliminate all errors, it is to give a good return on investment on writing test code versus fixing production issues. Test suites are economically defined, not by some absolutes. If the cost of failure is low, risk more. Besides, there is more you can - and probably should - do besides testing:

Deploy your branch to a staging environment and fire a “tracer bullet” through your code. Preferably automated, one button. “Integration” tests in your CI environment typically don’t test the actual database, the actual message queue, the actual monolith-you-don’t-want-to-talk-to-but-have-to, and so on. Your staging environment probably has these and thus deploying and testing there is a way better method of checking whether things properly fit together.
Have components emit health check data and have an integrated “all good” health check API endpoint in your service. It will tell you whether indeed it can talk to Kafka, MySQL, that API you depend on, and so on. Again, not in some synthetic CI environment but in the real deal, which is way more useful information.
Deploy your final code to production as a canary; the aforementioned health check will tell you whether all the fiddly (and largely untestable) bits around configuration are correct before any traffic is sent to it. Then, when that is all green, you trickle some production traffic through it for a couple of minutes and when that is all green, there’s a good chance that you can ramp up the new version and all will be good; things you would classically catch with integration tests usually aren’t the sort of subtle things that fly under the radar but more likely the big “oh, well, we didn’t look at that!” stuff that explodes right in your face the first time you hit the service.
Be able to roll back in a minute. Sometimes, a thing does fly under the radar, you roll back, figure out a test for the issue that happened, and try again. In most industries, the one or two minutes of irregular behaviour are not very costly; normally, this is less costly than carrying around a 1000-test integration suite that breaks all the time.

Note that this requires a pretty flexible environment, where you can run multiple versions of a service side by side, have automated health checks, and so on. All this newfangled stuff around “DevOps” is not there to make deployment simple. Deployment was simple enough when I built Win16 installers to be burned on a “Golden Master” CD in the ’90s when working for an ISV. It was one script and a cup of coffee. I don’t need no stinkin’ buzzwords to accomplish that.

No, all this stuff is there to make deployment low risk.

You reduce risk by lowering impact and cost of bad deployments and you can use that reduction to offset some of the heavy-handed testing that people still thinks are needed. There’s no test like actual production traffic.

“Testing” is a means, not an end. And it is part of a whole chain: writing code, reviewing it, having automated tests and a CI environment, deploying it through a CD environment with automated canary testing, even manual production testing after deployment: they are all small steps in a large chain that just has one goal: add value to your business. Locally optimizing one bit of that chain by focusing on “we have to integration test everything” while not seeing that other parts of the chain are so good they allow you to do less testing is folly. Optimize the whole system and know how to make trade-offs to that the whole chain adds maximum value at minimum cost. _That is the art of software development, if you’d ask me.