Passion narrative considered harmful

I’ve gotten tired of seeing articles on programming blogs mentioning the word “passion” in the opening paragraph so I’m going to explain why you should ignore all such articles and focus on the mechanics of programming. To become good at programming you just need to practice. Programming is a skill and passion is not a prerequisite for practicing a skill. Continue reading

Book notes

Designing Data-Intensive Applications: Ch 6. – Partitioning

After a certain point your data will become a bottleneck either because of size or access patterns and at that point you’ll need to partition the data. Although there are a few ways to do that and several open source solutions for partitioning massive data sets each has its own downsides that will need to be carefully considered for specific use cases. Continue reading

cobbling together code

A while back I wrote an implementation of SHA256 in Dart using generators and byte buffer views. Going back and looking at the implementation I can’t really make sense of it. It is not obvious why I index and shuffle the byte buffers one way instead of some other way.

So what is the lesson here? Writing code is always easier than understanding it. If after putting it aside for a while I can’t figure out why I made certain decisions then what hope is there of someone else figuring it out?

software engineering principles to live by

  1. There are three ways to solve problems: bring the problem closer to an existing solution, bring an existing solution closer to the problem, invalidate the problem by changing all your underlying assumptions.
  2. It is impossible to maintain large codebases in dynamic languages. Anyone that thinks otherwise hasn’t yet written enough dynamic code.
  3. Be proud of your work but do not be your work. Be critical of the code but not the person that wrote the code.
  4. The tools must always be subordinate to human intentions and not the other way around.
  5. Capabilities are always better than features. Capabilities can be composed. Features can only be contextualized.

encrypt and check your secrets into git

When it comes to managing secret tokens whatever you do someone will come out and say you’re doing it wrong and propose an alternate solution that on the surface seems to be better but upon closer scrutiny is susceptible to the same attack vectors. The secure solutions is always some kind of custom solution for storing secrets that provides an audit trail. The audit trail means whenever a secret is accessed you have a log entry of it somewhere. Which is nice because when stuff leaks you can in theory trace it back to some specific person. Unless that person was hacked and they were used as a patsy. So today let’s compare storing encrypted secrets in git or some custom solution that is not git. Continue reading

stuff for 2017

Be kinder to your fellow human beings. If you’re a nerd then like Joscha Bach you probably have an extra degree of freedom when it comes to normative belief systems and the signals regular people use to gauge how good/bad they are is probably lost on you. I know it is lost on me because the hyper-parameters in my neural nets are configured slightly differently. My empathy networks are more subdued than the average. Fortunately if you’re not a psychopath there is plenty of time to re-adjust those parameters.

Godspeed friends.

P.S.: I’m an atheist. I just think “godpseed” sounds cool.

optimizing spot instance allocation

There is surprisingly little information on how to optimize costs using the AWS spot instance market. There are several services that provide management on top of the spot market if you have an architecture that supports an interruptible workload but very little in the way of how to go about doing it yourself other than surface level advice on setting up autoscaling groups. To remedy the situation here’s an outline of how I’ve solved part of the problem for CI (continuous integration) type of workload Continue reading

learn some math

This is in response to the math myth.

The author couldn’t be more wrong even if he tried. My current project is using GLPK to do some basic mixed integer programming to optimize AWS spot instance allocation. If I had not taken linear algebra, calculus, and a few courses in linear programming the idea would not even have crossed my mind that I could use mixed integer programming to solve the spot allocation problem and that’s just the first half.

The second half can be considered a problem in control theory because it requires taking the new allocations and gracefully transitioning from the old set of allocations. You can go even further and say that the whole thing would be even better if I understood more about stochastic processes and could potentially model the spot market and make predictions ahead of time to simplify the control problem and get ahead of the price fluctuations.

Saying all you need is Excel and 8th grade is in the words of one famous physicist “not even wrong”. If you’re in an engineering discipline then the more math you know the better.