Allen Pike 2016-04-01T08:35:30-07:00 http://www.allenpike.com/ Allen Pike http://www.allenpike.com/2016/systems-design Diagrammatically Correct 2016-03-31T18:00:00-07:00 Allen Pike http://www.allenpike.com/ <p>Once upon a time, back when Netscape roamed the earth, software development was hard. Desktop apps shipped in cardboard boxes, web apps ran on Solaris boxes, and video games came on little plastic boxes. In this box-crazed world, making sure software was right the first time was Super Important™.</p> <p>Back then, writing an app – or “application” as it was referred to in the archaic English of the time – was an arduous process. Good design patterns were still evolving, and system libraries were of little help. Much of the code in any given app was, effectively, boilerplate. Collection classes, sync frameworks, windowing systems, memory management schemes – a wretched hive of tricky computing science problems.</p> <p>Meanwhile, it had become well understood that software projects, especially large ones, had a disturbing tendency to fail spectacularly. While every field has its blunders, research found that software developers as a discipline were hilariously bad at actually achieving the goals they set out to achieve.</p> <p>In response, software developers armed themselves. They increasingly adopted tools and systems for planning, communicating, and documenting their systems. They formalized requirements before designing, diagrammed objects before coding, and debated architectures before ordering two million dollars of servers. “If you fail to plan, plan to fail,” they would say, and then resume architecting their system’s UML to achieve optimal domain synergy. It was a strange time.</p> <h2 id="years-later">Years later</h2> <p>Recently a <a href="https://warpedvisions.org/">good friend</a>, a survivor of this gilded era of software engineering, asked me about a hiring problem. You see, he was interviewing to hire a senior software engineer, but many of his candidates were struggling with the systems design portion of the interview. These are supposed to be senior engineers, yet their diagramming skills seemed totally absent. How much systems diagramming do you do at Steamclock?</p> <p>“Uh, well… systems diagramming hey? Well, we don’t exactly… I wouldn’t say we formally… I mean… hm.”</p> <p>After I quelled a wave of guilt, I admitted that we don’t do very much. Of course we’ll diagram out proposed AWS setups, graph gnarly retain cycle issues, and whiteboard the hell out of UI flows, but we haven’t produced a UML document in the history of the company. While we tirelessly design and refine our user interfaces, our technical designs are rarely formalized before we start prototyping.</p> <p>While this is not what we were taught in school, this kind of modern ad-hoc process is now incredibly common in the industry. For every blog post or podcast about diagramming and systems architecture, there are ten thousand on choosing frameworks or hiring. Why design a system up front when it’s just going to change? Why make a document when it’s just going to get out of date?</p> <p>Well, because sometimes that’s what’s necessary to scale software, that’s why. We know how valuable diagramming can be as a tool for thinking through hard problems. We also know that thoughtful systems design is important for building complex, novel systems. Up-front systems design put us on the moon. So, why don’t we do any?</p> <h2 id="meet-the-prototyper">Meet the Prototyper</h2> <p>From a technical perspective, most software projects today just aren’t that special. If you want to explore just how not-special most software projects are, subject yourself to a project review for the Canadian <a href="https://en.wikipedia.org/wiki/Scientific_Research_and_Experimental_Development_Tax_Credit_Program">SR&amp;ED Tax Credit</a>. This program provides tax credits for technically novel software development, which when you dig into it is very little of what happens in a typical software project. Review a year’s worth of time entries on a challenging project, and you’ll often find that nine hours out of ten, you were just picking libraries, adapting to new requirements, and improving the UI.</p> <p>Modern software projects are often just the cherry on top of a wheelbarrow-sized sundae of opaque 3rd party code. Why write a blogging engine from scratch when you can make one that fits your needs perfectly in a few hundred lines, by just importing <a href="https://www.caseyliss.com/2016/3/27/node-is-weird">200,000 lines of dependencies</a>? Why reinvent the wheel when you can <code>gem install wheel</code> or add <code>wheel 0.1.x</code> to your package.json?</p> <p>In this environment, more and more software developers now spend their careers as prototypers. Startups, product consulting shops, and indies typically spend a lot of time starting stuff, and relatively little time scaling it. In these kinds of businesses, prototypers’ shipping skills are invaluable. Any given project may be relatively small, and may or may not turn into something big, but if you can ship it, you’ve overcome the biggest threat to most new software: never shipping in the first place.</p> <p><a href="http://cube-drone.com/comics/c/missing-the-point"><img style='max-width: 100%' src="/images/2016/cube-priority.jpg" /></a></p> <p>Our culture has never been better focused on shipping. Agile – <a href="http://agilemanifesto.org/principles.html">the manifesto</a>, not <a href="http://cube-drone.com/comics/c/dont-go-chasing-waterfall">the catchphrase</a> – makes no bones about the need to deliver sooner rather than later:</p> <blockquote> <p>Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. … Working software is the primary measure of progress.</p> </blockquote> <p>Say what you want about agile the catchphrase, but the increased focus on iteration and delivery has been a huge positive force in the industry. Fifteen years later, we have a whole new generation of developers for whom this mentality has always been the norm.</p> <p>While this has been a net win, there are some interesting consequences to this mindset. Folks who spend their careers working on a series of small iterative projects tend to think less in terms of traditional software engineering, and more in terms of immediate user goals and short term improvements. Besides leading to a lower prioritization of systems design and planning up front, this mentality can also lead to laissez-faire attitudes about documentation, unit testing, and refactoring. The OG software engineers would not approve.</p> <p>For better or worse though, the iterate-now-plan-later philosophy has come to dominate the popular discussion around software development. This outlook is an easy sell at lean startups, UX-focused development shops, and any studio more focused on solving user problems than technical ones: who enjoys writing specifications anyway? As a result, we now have an entire generation of software developers raised on the “move fast and break things” approach to solving problems.</p> <h2 id="dude-wheres-my-composite-structure-diagram">Dude, where’s my Composite Structure Diagram?</h2> <p>Unfortunately, at a certain scale breaking things is not an option. Or, more accurately, breaking things becomes staggeringly costly. Moving fast is great, but if you’re serving <a href="http://www.bizjournals.com/sanjose/news/2013/06/11/14-eye-popping-apple-statistics-from.html">billions of push notifications per day</a>, it becomes more important that everything doesn’t go to shit.</p> <p>Projects at scale have an outsized impact, and early mistakes can become hard to undo. The larger and more technically novel a project gets, the more you suffer the pain of “not maintainable here”, and the less bananas it is to design and build your own custom frameworks and other components.</p> <p>Naturally, folks that spend their careers building and maintaining large systems at scale typically spend a lot more time thinking about systems design and software engineering. At a certain point, generalized “best practices” that work for the typical system stop working, and speculatively iterating towards your servers not bursting into flames can be untenable.</p> <p>All this necessitates serious-business systems planning, diagramming, and projections. Triply so for backend systems, which get more complicated and have longer lives than our beloved clients.</p> <p>So, where are the senior systems designers? Well, they’re doing what they do best: building massively scaleable backends at Amazon, Apple, and Google. They’re launching spacecraft to Mars and running nuclear power plants. They’re grappling with the face-meltingly complex and fragile enterprise systems that keep airlines, banks, and telecoms from collapsing under decades of technical debt. Basically, they’re saving our bacon.</p> <p>Still, these experienced systems designers make up a smaller and smaller proportion of software developers over time. Unfortunately, all too often their respective corporate bureaucracies prevent them from blogging, podcasting, or even tweeting about the work they do. This reinforces the echo chamber of startups and indie product developers, who like to celebrate a lack of process, and think of rapid iteration and thoughtful software engineering as mutually exclusive. The pendulum has swung so far that the very idea of up-front design is thought of as old and busted.</p> <h2 id="one-drop-does-it">One drop does it</h2> <p>I’ll readily admit that I’m most comfortable wearing my “move fast” cap. We work on a lot of new apps, put a big focus on UX, and do a lot of work for startups. Like many studios, getting bogged down on exhaustive up-front planning isn’t a great tradeoff for most of our projects.</p> <p>That said, it’s crucial that we keep in mind the benefits of the tools in our software engineering toolbox, and make thoughtful choices of when to use them and when not to. Rather than debating whether all projects should adopt this process or that process, we should debate how to tell when a certain project justifies more formal engineering.</p> <p>For example, projects for larger companies typically have a higher likelihood of hitting scale, so that’s a flag that their projects justify spending more resources on meticulous planning, testing, and preparation. In contrast, an experimental product from a startup would more likely benefit from moving as quickly and efficiently as possible.</p> <p>So, next time you’re starting a new project, keep an eye out. A diagram here, a short planning document there – where there are technical risks, there are opportunities to use system design. It might not be sexy or futuristic, and it sure as hell doesn’t need to be UML, but sometimes, a little planning can go a long way.</p> http://www.allenpike.com/2016/cofounding-variables Cofounding variables 2016-02-29T18:00:00-08:00 Allen Pike http://www.allenpike.com/ <p>Finding a great technical co-founder is an incredibly difficult yet critical part of starting a business. Given that, folks often ask how I met Nigel, my co-founder at Steamclock. As it happens, we met in a relatively unusual way.</p> <p>You see, in high school I was a theatre kid. I acted, I ran lights, I did improv – I even did a bit of writing and directing. In retrospect, it was the perfect crash course in public speaking and teamwork, and should be required coursework for shy nerds everywhere.</p> <p>Anyhow, at our school every student needed a certain number of official work experience hours to graduate. Our theatre department was in the business of organizing those work placements, specifically for performing arts. Sometimes it was commercials, sometimes we were stage extras. Strangely enough, I ended up playing a soldier in a couple operas. As fun as that was, my career goals were more technical than theatrical. Fortunately this got Patricia, my theatre teacher, thinking outside the box.</p> <h2 id="a-radical-idea">A radical idea</h2> <p>As luck would have it, her husband was a senior game developer at a local game studio called Radical Entertainment. Instead of setting me up for another acting gig, she asked him if I could do a work placement at Radical.</p> <p>Obviously, this was the most exciting thing to happen since ever. By that time I’d concluded that I would one day be a game developer, and as such I’d spent years <a href="https://www.allenpike.com/2006/fantasytech-3-goto-fun/">writing terrible games</a> in cutting-edge tools like QBasic, C++Builder, and PHP. All this hard work was finally going to pay off. I was going to see how a <strong>real</strong> game company worked!</p> <p>Before you know it, I was signed up, given an access badge, and set up with a PC, Visual Studio, and a menacing mountain of source code. At that time they were working on a AAA game called Simpsons Hit &amp; Run – far too important a project to let some high school kid touch.</p> <p><a href="http://www.upup.fm/show/nels-firewatch/"><img style='max-width: 100%' src="/images/2016/henry3d.jpg" width="250px" /></a> Luckily, one of their internal tools had some nagging intern-sized issues just waiting to be scratched. For each game the artists would build their assets, then use a previewing tool to see them in 3D before loading them into the full game. As expedient as this was, the previewer had a really rudimentary camera system and was difficult to use. My first task was to simply add a dot where the virtual camera was pointing in 3D space. Just a little dot.</p> <p>At first it was daunting. Then it was fascinating. A couple days later, it was done. I had a dot – <em>my</em> dot – displaying in 3D space. I rotated it around, moving the camera, and observed what I’d created. Seeing this little dot come to life was unbelievably, irrationally exciting.</p> <p>The rest of the placement flew by – making other improvements, adding color-coded axis markers, improving the mouse controls, and learning the joys of source control. It was short and sweet, but left a permanent impression on me as I started applying to universities and planning out my career.</p> <h2 id="and-as-good-luck-would-have-it">And as good luck would have it</h2> <p>A couple years later, as I was working through my Computing Science degree, Patricia got back in touch. She was starting a community theatre company and was wondering if I could play a part in their first production: Midsummer Night’s Dream. I agreed, and through those shows I got back in touch with her husband. When he wasn’t putting on Shakespeares, he was still toiling away at Radical, which had since been acquired and subsumed into the <a href="https://en.wikipedia.org/wiki/Activision_Blizzard">Activision Blizzard</a> juggernaut. We worked on a half-dozen shows together, chatting between scenes about making software and the trials of working at a giant corporation.</p> <p>One fateful day as we chatted backstage, he mused that he was thinking of starting his own company. Funnily enough, I was planning to do the same thing. Within a few days we were sketching out what would become <a href="http://www.steamclock.com/">Steamclock</a> – and that’s how Nigel became my co-founder.</p> <p>So when people ask me how to find a co-founder, I first ask if they happen to be in a high school theatre program run by somebody married to a software engineer. If not, I generalize my advice somewhat: start with somebody you’ve worked with before.</p> <p>A business is in many ways like a marriage, and the only way you can be sure somebody will make a good teammate is to try working with them. Whether you work with them on tiny 3D viewers, Shakespeare productions, or maybe even useful software – that part is up to you.</p> http://www.allenpike.com/2016/parse-joy-of-shortcuts The Joy of Shortcuts 2016-01-31T18:00:00-08:00 Allen Pike http://www.allenpike.com/ <p>Some time ago, we had a client come to us with a problem. Their app was a mess. It consisted of roughly 400 kilograms of copy-pasted Objective-C, written by a departed team member in some kind of over-caffeinated fever dream. There was no server logic, or anything else that would qualify as logical for that matter. It was simply a large, quivering mass of queries to Parse.</p> <p>As a whiz-bang NoSQL Backend as a Service, Parse let people quickly build mobile apps. You just start coding, they said, and we’ll worry about the rest. Your fever dream will only last so long, and any moment you spend worrying about server infrastructure is a moment you’re not furiously typing square brackets or whatever else makes your app special. And you know what, it kinda worked.</p> <h2 id="a-rocket-ship-to-somewhere">A rocket ship to somewhere</h2> <p>Like many other tools, Parse was a shortcut, a temporary simplification, an abstraction. Shortcuts let you move faster from idea to product, helping you focus more on shipping and less on diversions. Just like Interface Builder helps you rapidly prototype your UI and JavaScript frameworks help you get a web front-end up faster, Parse let you defer writing and maintaining a custom server backend until another day. Judicious use of shortcuts like these can make or break a fragile new venture.</p> <p>As nasty as our client’s mass of Parse and Objective-C was, their working prototype allowed them to raise a serious round of funding and collect a waiting list of eager customers. With that funding they hired us to write a new, high quality version of the app, and roll it out to their newfound fans. This is an age-old story in technology: a quick and dirty prototype is a stepping stone to something good. Hell, Objective-C itself started out as just a bunch of macros on top of C. Sometimes, shortcuts are the best first step.</p> <p>Perhaps more importantly though, shortcuts help you fail cheaply. In 2014, the wonderful folks at Panic <a href="https://panic.com/blog/the-2015-panic-report/">built and tested an app</a> to share and discover music. Instead of building a whole custom backend, they put it up on Parse. The app was beautiful, and it worked, but they rightly questioned its revenue potential. They deliberated, and in the end they decided to shelve the project. With that, whatever money they saved by building it on Parse was cash in the bank. When an idea doesn’t pan out, every shortcut taken on the way there is a blessing.</p> <h2 id="i-have-an-idea-for-an-app-please-advise">I have an idea for an app, please advise</h2> <p>The world of consumer software is a sea of uncertainty. Most app ideas simply don’t work out. Still, when you’re excited about your idea’s potential to take over the world, it’s easy to get sucked in to building out a sophisticated scaling infrastructure, loading up on administration and monitoring, and deeply documenting and unit testing your soon-to-be masterpiece.</p> <p>Once that’s done, though, once your miracle of modern software engineering wades out into the red ocean of social apps and flapping poultry, you’re going to be fighting the same battle as everybody else. If you’re lucky, you’ll be clamouring for attention, iterating, and working your ass off looking for success, hoping you don’t run out of time or money before you get there. If you’re unlucky, <a href="http://mashable.com/2012/10/17/color-shuts-down/#N75AVQyT6qqu">you’ll hit a brick wall</a>. Only if you get traction will the price of your shortcuts come due.</p> <p>Of course, as useful as shortcuts can be for proving or disproving an app idea, they’re much more dangerous when your project is more than an idea. While most apps will never get 1,000 active users, others are likely to get hundreds of thousands, if not millions. If your product is highly anticipated, is bringing a popular app to a new platform, or is packing the power of a big brand, you don’t have the luxury of quiet experimentation. That’s when the rigorous work of instrumenting, testing, and building for scalability needs to happen up front. It sounds scary, but <a href="https://marco.org/2016/01/30/mjtsai-sunsetting-parse">it’s easier than it sounds</a>. The hard part is knowing when enough is enough.</p> <h2 id="moving-on">Moving on</h2> <p>Next January, <a href="http://blog.parse.com/announcements/moving-on/">Parse is shutting down</a>. The successful Parse apps will get moved to a custom backend like ours was, perhaps using <a href="http://blog.parse.com/announcements/introducing-parse-server-and-the-database-migration-tool/">Parse’s excellent open-source server and migration tool</a>. The unsuccessful Parse apps will die. Hundreds of thousands of unsuccessful Parse apps will perish. Like links to long-dead Geocities pages, dead mobile apps that relied on Parse will linger in the App Stores for years, slowly accumulating one-star reviews.</p> <p><img style='max-width: 100%' src="/images/2016/parse-trust.jpg" width="100%" /></p> <p>As much as Parse will try to get the word out that they’re shutting down, many apps’ owners don’t even know that they’re reliant on Parse. Parse’s overly generous free plan made them popular with freelancers and consultants building quick app backends for their clients. Many of those clients don’t know what Parse is, let alone that the little app they commissioned a couple years ago is a ticking time bomb.</p> <p>While the death of these apps is sad, it’s unclear who to blame. The majority of them would have died anyway, regardless of what they were built on. The lack of a monthly bill from their server provider kept them shuffling along for a while, but sooner or later, unmaintained software dies.</p> <p>Our industry, the software industry, builds things that are so ephemeral, so fragile. Sometimes software grows, it changes, and it has a long life. Other times it doesn’t, and it’s swept away. Either way, though, these minor calamities help us sort out what kind of things we want to build. Do you want to build bold experiments, take shortcuts, and see where your customers take you? Or do you want to build solid foundations, sturdily engineered, and hunker down for the long haul?</p> <p>Me, I’m still not entirely sure. Either way, I know one thing: I won’t be building them on Parse.</p>