Marko Anastasov

Product Friction Starts As Unclear State

Fri, 27 Mar 2026 12:37:00 GMT

Looking through this week’s issues at SuperPlane, one pattern stood out.

A lot of UX friction comes from the product leaving people unsure what state they are in.

A simple CLI issue was a good example. A canvas update command exited with code 0 and printed nothing. Exit code 0 indicates success, but as a user, I am still not sure what happened. I have to run another check just to verify whether anything changed.

Another example came from sample event payloads. One issue pointed out that the sample data behind a component was contrived, and because it also fed configuration autocomplete, it taught users the wrong mental model before real data even arrived.

We identified similar cases across CLI, API, and UI. Different surfaces, same problem. The system was not making its state clear enough.

The Don’t Make Me Think book was published in 2000 and that is the lesson I am relearning this week. Users should be able to tell what happened, what state they are in now, and what to do next, without doubt.

A good product does not just do the right thing. It makes the right thing obvious.

Stop Prompting, Start Plugging In Your Context

Fri, 27 Feb 2026 12:00:00 GMT

If you’re still using AI by writing prompts, you’re missing 80% of the value.

The real gains are made when you give models access to your context: your docs, your tickets, your code, your metrics, and your decisions.

Which leads to a fundamental truth: AI is an amplifier of what you already have in your organization.

That’s the good news and the bad news.

If your company playbook is stale, AI will confidently repeat outdated information.

If your decisions aren’t recorded, AI will “align” people around fiction.

If your engineering team isn’t doing continuous delivery with fast CI, frequent deploys, and easy rollbacks, AI will generate more code than you can responsibly move into production.

The greatest returns come from investing in the foundations: your data, your platform, your engineering discipline, and your feedback loops with users.

The good news is that the amplifier works both ways.

When AI gives you a bad answer, it’s rarely an AI problem. It’s a spotlight on what needs to become explicit, searchable, and true. Fix that, feed it back into the system, and the next answer gets better, along with everything else.

Backseat Software →

Tue, 03 Feb 2026 10:50:00 GMT

A historical perspective on the evolution of software by Mike Swanson:

What if your car worked like so many apps? You’re driving somewhere important…maybe running a little bit late. A few minutes into the drive, your car pulls over to the side of the road and asks:

“How are you enjoying your drive so far?”

Annoyed by the interruption, and even more behind schedule, you dismiss the prompt and merge back into traffic.

A minute later it does it again.

“Did you know I have a new feature? Tap here to learn more.” […]

I’ve started to think of this as backseat software: the slow shift from software as a tool you operate to software as a channel that operates on you.

It’s a long post but worth reading in its entirety.

Key points I want to remember:

When software collects data and sends updates, usage analytics (DAU, funnels, etc.) can become central to product decisions. The focus then shifts from helping users achieve goals to optimizing engagement metrics.
Metrics can be correct but still mislead you. For example, a feature seems unused because it was badly designed and buried in the UI, not because it isn’t valuable.
Testing can produce insights but also turns products into continuous experiments on users, where decisions favor what moves metrics fastest, not necessarily what aligns with product vision.

The paradox is that this approach is practiced at companies who believe that this is what they should be doing to be successful. But the visionaries who built enduring companies kept their focus on the users.

Steve Jobs:

You’ve got to start with the customer experience and work back toward the technology — not the other way around.

Jeff Bezos:

We start with the customer and work backwards.

So instead of backseat software, we should build frontseat software. Software that stays out of your way, helps you finish the job, and shuts up when it’s done.

via Daring Fireball

Engineering, Product, and Design Are Now Like Video Game Skills

Mon, 02 Feb 2026 08:33:00 GMT

Engineering, Product, and Design used to be separate skills, but in the age of AI agents, they are like video game skills. You can clear the early levels by maxing one, but you won’t progress if the other two are at zero.

It’s now normal to see product managers and designers writing code and implementing UI with agents in the loop.

Two examples from the SuperPlane team:

Aleksandar is a Product Manager. He spends most of his time curating the product not by writing tiny issues for engineers but by acting as a full-stack developer, writing code to build new features or tweak the existing ones. He’s also orchestrating agents to create issues based on product requirements, review pull requests, and generate new ideas.

Petar is a UI/UX Designer. He’s creating the UI/UX for the app and implementing behaviors that he wants to see in the app as a full-fledged frontend developer. Since picking up Cursor, he has not opened Figma.

The best thing about AI is that it makes leveling up universally available. It’s fun to see how people are using it to evolve and do more than they ever thought possible.

SuperPlane Is Open Source

Fri, 30 Jan 2026 10:29:00 GMT

This week we open-sourced SuperPlane, a DevOps control plane for running event-driven workflows across the tools you already use. You can think of it as ‘n8n/Zapier for DevOps’ that sits on top of Git, CI/CD, observability, infrastructure, and the rest of your production system.

Why does this matter? To evolve, DevOps needs to move on from one-off scripts, tribal knowledge, and untraceable workflows.

For years, the best automation tool at hand was the humble CI job. Pipelines are great at building and testing code, but they are not great at running workflows that span multiple tools and repositories, pause for human intervention, and run for hours or days.

What we need is a vendor-neutral interoperability layer for DevOps automation; a way for tools, humans, and agents to talk to each other, with domain-specific channels and components. This launch is the alpha release of that layer, starting with 40+ core components and integrations.

A few links if you want to kick the tires:

Read the release announcement on SuperPlane blog
Star the repo / browse the code: superplanehq/superplane
See concrete example use cases
Try the app locally in two minutes

Ultimately, the goal is to make it safe to describe what you want in natural language and have the system execute it in a safe, auditable way. It’s a journey, and we’re committed to keeping it open and transparent.

You Need To Change The Whole Assembly Line

Wed, 21 Jan 2026 10:55:00 GMT

In the DevOps Paradox episode 2026 — The Year of Discovery, Viktor and Darin talk about how software work is changing, but the bottlenecks aren’t where people think.

A few quotes are basically a version of why we started SuperPlane. My comments below.

You cannot just give developers, here’s AI. Everything else stays the same… you need to change the whole assembly line.

Most organizations are currently applying AI to optimize the coding process. The constraint will simply move downstream: releases, verification, coordination, incident handling. If you don’t address the system, you get faster queues, not faster outcomes.

You have a platform that you can say, hey, give me a database, and you get that database. And that database is done based on your company’s best practices, rules, policies […] ask me questions that I need to answer and don’t ask anything more.

This is what engineers are actually asking for: outcomes, and systems that hide incidental complexity without hiding state.

People simply don’t want to… relinquish part of their power… [and] don’t want to introduce randomness in production.

This is the hard part. It’s not so much a capability problem as much it’s a trust problem. People will only use automation in production when it’s controllable, auditable, and fails in predictable ways.

To get there, I think we need a control plane that makes cross-tool operational workflows explicit and inspectable: what triggered, what ran, what it touched, and what evidence it produced. If you can’t explain a change, you can’t safely automate it.

Awesome Password Generator

Thu, 08 Jan 2026 16:38:00 GMT

I got tired of password generator websites bloated with ads and trackers. So I built a small side project to fix that:

fully client-side
no tracking, no popups
open source

awesomepasswordgenerator.com

Starting SuperPlane

Mon, 22 Dec 2025 08:47:00 GMT

Darko Fabijan and I started a new company called SuperPlane.

The best engineers are creating and reviewing an order of magnitude more code with AI. But the way they manage and operate their production systems is still manual and fragile.

We think there’s a better way to do DevOps in the age of AI. It’s our most ambitious project to date, and we’ll share more as soon as we have something to show.

We’re hiring engineers. If you’re into building developer tools and looking to do career-defining work, let’s talk.

Cloudflare's November 18 Outage – A Continuous Delivery Perspective

Thu, 27 Nov 2025 12:26:00 GMT

On November 18th, 2025, Cloudflare had what they describe as their worst outage since 2019.

It didn’t start with a cyber attack or massive hardware failure. It started with a database permissions change.

This is exactly the sort of thing Continuous Delivery is supposed to make safe. So since the post-mortem came out I wanted to analyze it through a CI/CD lens.

Summary of what happened

Around 11:20 UTC, Cloudflare’s network started returning a huge spike of HTTP 5xx errors for traffic flowing through their core network. Users all over the Internet started seeing Cloudflare error pages instead of the sites they were trying to visit.

The root cause was a change to how one of their ClickHouse database clusters handled permissions and metadata.

Here’s the rough chain of events:

Cloudflare deployed a change to ClickHouse permissions and metadata visibility. That change meant queries against system metadata suddenly started returning more rows than before — including underlying tables in another schema (r0) where previously they only saw the default database.
One of the systems that depended on that metadata was Cloudflare’s Bot Management feature file generator. It queried system.columns to assemble the list of features used by a machine learning model to score bots. That query didn’t filter by database name, so after the change it suddenly saw roughly double the number of columns, and produced a much larger feature file.
That feature file is shipped out to Cloudflare’s edge network every few minutes. The bots module in their core proxy expects the number of features to be below a hard limit (200) and preallocates memory accordingly. The new, larger file exceeded that limit. The Rust code hit the limit and panicked, leading to 5xx responses from the core proxy.
Because this file is generated every five minutes from a distributed ClickHouse cluster that was being gradually updated, the system actually went up and down for a bit: sometimes a good config, sometimes a bad one, depending on which node produced it. Eventually all nodes produced the bad file, and the failure became stable.
Downstream systems that depend on the core proxy – things like Workers KV and Access – were also impacted until they were temporarily rerouted or bypassed.

Main impact started around 11:28, substantial mitigation began around 14:30, and everything was fully resolved by 17:06 UTC.

So: a change to database permissions → bigger config file → hard limit exceeded → proxy panics → global outage.

That’s a change management story.

This is why Continuous Delivery exists

The primary cause of production failures is change. Continuous delivery is about making change boring.

If your system is updated frequently then you must assume that any change is guilty until proven innocent.

From a CD perspective, a few big themes jump out:

1. Config is code

The feature file that broke things isn’t some harmless data; it is executable behaviour. It changes what the bot-scoring model does and it clearly has operational consequences. That should be treated with the same discipline as application code:

Stored and versioned in source control
Validated in pipelines
Subject to contract tests with the consumers (the proxy module)

2. Contracts between systems

There is an implicit contract here:

Producer: “I will generate a feature file.”
Consumer: “I can safely handle up to 200 features.”

That contract wasn’t explicitly enforced at integration boundaries. The producer didn’t know that exceeding 200 features would crash the consumer, and the consumer effectively said: “If this assumption is broken, I will simply panic.”

In a good CD setup, we encode these contracts as tests:

“If the feature file has > 200 features, the pipeline fails.
Or, at runtime: “If it’s too big, log, drop extra features, and degrade gracefully instead of crashing.”

3. Fast feedback tied to change events

When production starts returning loads of 5xxs right after a change, your first hypothesis should almost always be: “We broke it.”

Cloudflare initially suspected a large DDoS attack, partly because their off-platform status page also went down by coincidence, which complicated the diagnosis.

A strong CD/operational posture ties monitoring directly to deployments:

“We deployed change X to ClickHouse at 11:05.”
“At 11:20, 5xxs spike globally.”
The system should scream: “Roll back that change first, then keep investigating.”

To their credit, they did eventually correct course, stop propagation of the bad file, push out a good one, and restart the proxies.

Where CI/CD could have helped more

Let’s talk about specific points where mature CI/CD practices can help avoid or limit this sort of outage.

1. Test the config generator like production code

The change that triggered this was essentially a behaviour change in the query used to build the feature file.

In a CD world, we’d treat that as an ordinary code change and ask:

Do we have automated tests for the query behaviour?
Do we have tests that assert the range of output sizes we consider safe?
Do we have a test environment where a realistic ClickHouse cluster – including these permission changes – is exercised before we touch production?

A property test like:

Given realistic metadata, the resulting feature file must have ≤ 200 features, or the build fails

…would have caught this before it ever reached live traffic.

2. Practice progressive delivery for everything

Cloudflare already had a gradual rollout of permissions across the ClickHouse cluster, but the effect of that was system going up and down: sometimes a good config, sometimes a bad one.

From a CD perspective, you want controlled blast radius:

Ship new behaviour to a tiny slice of traffic or a subset of regions first.
Observe: “Did the new feature file cause any spike in 5xxs, latency, or resource usage?”
Only then ramp up.

Instead of having the feature file immediately pushed to the entire network every five minutes, imagine:

A canary group of edge nodes is updated first.
If they see the bot module panic or error rates spike, an automated system:
- Rolls back to the last known good config file.
- Blocks further rollout.
- Raises an incident with a clear “config rollout blocked” signal.

That’s progressive delivery applied not just to code, but to ML feature sets and configuration.

3. Design for graceful degradation, not panic

The Rust code in the affected FL2 proxy is essentially designed to “panic” (as per post-mortem) if it gets more than 200 features.

From a resilience standpoint, that’s exactly what we don’t want. In a world of continuous delivery and constant change, you assume your assumptions will be broken.

Better options might include:

Drop any features over the limit and log loudly.
Disable the Bot Management module temporarily and continue forwarding traffic, maybe treating everything as “unknown bot score” rather than bringing down the proxy.
Trip a feature kill-switch that turns off Bot Management globally while keeping the core CDN and proxy path alive.

Cloudflare’s own remediation list mentions:

Hardening ingestion of internal configuration files like user input.
More global kill switches for features.
Reviewing error handling across core modules.

That’s exactly the direction you’d expect from a team embracing CD and resilience engineering.

4. CI as safety net for “platform-ish” changes

One subtle thing here: this wasn’t a direct change to Bot Management. It was a change to how the database platform handled permissions and metadata.

In many organisations, platform changes are treated as “infra stuff”, not subject to the same product-level tests. But in a CD culture:

Platform changes go through pipelines that also run representative workloads and integration tests for the services that depend on them.
That ClickHouse permission change should have run:
- A compatibility test suite for all consumers that query system.columns.
- Specific tests for the Bot Management feature generator pipeline.

If you can’t do that comprehensively, you at least start with the critical systems: anything that can bring down your main proxy path should have extremely strong automated protection.

Good engineering is about turning failure into fast, safe learning

Cloudflare did the right thing by publishing a detailed post-mortem, while expressing engineering humility and laying out concrete follow-up steps.

Everyone running large-scale systems has failures. What distinguishes good engineering organizations is:

They treat incidents as opportunities to improve the system, not to blame individuals.
They share what they learned.
They make structural changes – to code, to pipelines, to architecture, to daily practices.

From a continuous delivery perspective, the lessons I’d highlight are:

Every change is potentially dangerous. CD is about making lots of small changes safe, not moving fast and ignoring risk.
Config, queries, and ML features are code. They need the same CI/CD discipline: tests, contracts, and progressive rollout.
Design for graceful failure. When – not if! – your assumptions are broken, the system should bend, not snap.
Tie observability tightly to deployments. If you’ve just changed something and the world is on fire, suspect your change first.

Cloudflare’s outage is painful for them and for a lot of the internet. But it’s also a rich example of why we do continuous delivery in the first place.

If It Hurts, Do It More Often

Mon, 13 Oct 2025 08:42:00 GMT

A common saying from the culture of Continuous Delivery is “If it hurts, do it more often”.

When something hurts, lean into it. Discomfort is the teacher that forces you to adapt until it disappears.

Let’s say deploying is painful because:

The process is manual
Only one person can do it
If it goes wrong, nobody else could fix it

You can solve it by:

Automating the deployment
Defining a task on Semaphore to make deploying and rolling back a matter of pushing a button
Giving other engineers access to Semaphore workflows and production logs

This is of course a simplified example. “Automating the deployment” may alone be a gargantuan task. Good. You decompose big problems into small problems. Go step by step.

The same logic applies when the pain comes from the world having changed. Today development teams are going through a radical transformation with the use of AI tools. The discomfort is real. As Tom Blomfield recently tweeted (emphasis mine):

Hearing from a lot of good founders that AI tools are writing most of their code now. Software engineers orchestrate the AI.

They are also finding it extremely hard to hire because most experienced engineers have their heads in the sand and refuse to learn the latest tools.

I get it. You need to shift from writing code by hand—a big part of your identity—to curating and reviewing what the AI agent produces. And you need new tooling and habits so agents can run without breaking anything.

So lean into it:

Start with bug fixes and chores that bore you to death
Write prompts, specs, and checks the agent must pass before it lands in review
Pair with teammates so you spot patterns together
Update your AGENTS.md so your prompts stay DRY and everyone shares the same playbook
Share lessons with the team in an open chat room and weekly meetings
Celebrate small wins, then ramp up the tasks you hand to agents

Whatever hurts is a signal that you have a mission to complete to level up.

Operately 1.0 →

Wed, 30 Jul 2025 11:50:00 GMT

Today we’re releasing Operately 1.0. Most open source products have a bad UX. At Operately we have set a higher standard.

For example, the Work Map is our third design solution for visualizing goals and projects. Sometimes you need a few bad takes before you nail it.

Would love to hear your thoughts if you’re looking to replace duct-taped Notion documents or legacy project management tools at your company. Drop me a DM on any of the socials linked below with any feedback or ideas as you try it out.

This is just the beginning. The best is yet to come.

AI-Generated Personal Message Equals Trash

Tue, 03 Jun 2025 09:48:00 GMT

A paper from Harvard Business School (PDF) provides data evidence on something we all knew: people dismiss AI-generated personal messages at best, and consider them offensive at worst.

People could barely tell AI from human messages (59% accuracy), but when they thought something was AI-generated, they rated it as less helpful even when a human actually wrote it.

What’s also interesting is that the harder the AI tried to sound human, the stronger people’s aversion became when they detected it.

I believe that most managers will eventually have an AI avatar that people can talk to when they’re not around. But using AI to generate a message to send to someone else is a different matter.

There’s plenty of ways to leverage AI for communication and decision making. Use AI to brainstorm counter-arguments, detect blind spots, research background, or polish your draft - but the ideas and final message must be yours. You must stand 100% behind them.

People don’t just read your words. The basic currency between humans is trust. So even when they’re not consciously thinking about it, every time you talk to someone they’re deciding whether to trust you.

Use AI to think better, not to think for you. Because the moment you stop owning your words, you stop being worth listening to.

Dream in Years, Plan in Months, Evaluate in Weeks, Ship Daily

Wed, 21 May 2025 10:33:00 GMT

DJ Patil, former U.S. Chief Data Scientist kept this as a daily reminder card:

Dream in years
Plan in months
Evaluate in weeks
Ship daily

Prototype for 1x
Build for 10x
Engineer for 100x

What’s required to cut the timeline in half?
What needs to be done to double the impact?

Easter walk

Mon, 21 Apr 2025 16:29:00 GMT

Peaks calling

Thu, 17 Apr 2025 16:23:00 GMT

A Blue Dot Underneath The Rings of Saturn

Mon, 14 Apr 2025 08:37:00 GMT

The Day the Earth Smiled is a composite photograph taken by the NASA spacecraft Cassini on July 19, 2013.

How Channable Made Deploying Often Easy →

Tue, 08 Apr 2025 09:34:00 GMT

This 250+ person tech company deploys 50+ times daily with zero drama:

At various times, CI was not fast enough, in response to which we did various optimizations. Very aggressive caching played a large role. We also made sure we only tested code that actually changed. In the early days of our CI adventure we switched CI providers twice before settling on Semaphore for performance and ergonomics reasons.

It’s incredibly rewarding to see how companies like Channable use Semaphore to power their development workflow. Their journey shows exactly why we built Semaphore with performance and developer experience in mind.

Developers can get new features to production with just a few simple steps
The entire deployment process requires minimal human intervention
Code changes move from branch to production safely and confidently

The business impact is clear - faster innovation, happier developers, and a better product for customers. This is what happens when technical excellence meets thoughtful process design.

A Checklist for Cofounder Happiness

Thu, 03 Apr 2025 09:58:00 GMT

Successful companies are built on strong cofounder relationships. These partnerships don’t happen by accident - they require these foundations:

100% mutual trust
0% ego - leave it at the door
Shared vision for the company’s future
Full transparency
Honest conversations
Having uncomfortable conversations early
Recognizing and valuing your differences
Mutually agreed expectations
Written agreements for clarity, not just handshakes
Consistently delivering on your commitments
Clear decision-making process for disagreements
Flexible roles that evolve based on company needs
Acknowledgment of each other’s wins and contributions
Aligned personal timelines
Aligned financial needs
Time apart to recharge and maintain perspective
Remember why you started this journey together

Just Do Things

Wed, 26 Mar 2025 23:08:00 GMT

You can just do things is the best startup meme in a long time.

Action produces information. When you do things it makes things happen. Or not.

Doing things shows you things you didn’t know. Motion creates clarity. So you can learn things to do more of the right things and less of the wrong things.

The world doesn’t belong to people who keep planning things because they rarely accomplish things.

It certainly doesn’t belong to people who have opinions about things.

The world belongs to those who do things.

Don't Guessbuild

Fri, 21 Mar 2025 15:22:00 GMT

One of the principles in product design that we’ve established at Operately is to avoid guessbuilding.

We stop the design of a new feature at the edge where certainty ends and speculation begins.

The pattern we’ve noticed is, we start by solving specific problems that either we or our users are experiencing. As we analyze the problems, we start seeing patterns that shape the general solution.

This is where things get dangerous. The general solution opens up possibilities for handling hypothetical use cases that we haven’t actually encountered yet.

This isn’t new. In software development, people who practiced Extreme Programming in the mid 90s coined an acronym YAGNI – you ain’t gonna need it.

Don’t be a fool and spend time programming capabilities that you presume your software will need in the future — because it most likely won’t.

Likewise, we aim to draw a clear line between:

What we know our users need (backed by direct feedback or usage data)
What we think our users might need (speculative features)

When these phrases start appearing in our discussions, it’s usually a sign that we’re guessbuilding:

“This could be useful if…”
“Users might want to…”
“What if someone needs to…”
“We should make this flexible enough to handle…”

So we stop there to build confidently for the known needs, ship that solution, and move on to another area of our product. The new solution needs to simmer long enough to have new evidence emerge that clearly indicates what we should build next.

An example

Recently we were discussing how to approach removing goals. Currently, you can close a goal and mark it as accomplished or not. But this action didn’t feel appropriate for situations when you want to stop working on a wrong goal. Or when as a new user you’re just exploring with imaginary data.

So we started with an idea that we need to support both archiving and deleting. Archiving would hide dropped initiatives. Deleting would completely get rid of stuff. But who actually asked for archiving? Just because archiving is a common concept doesn’t mean we have to apply it. A goal isn’t stateless like a document or a workspace. It has a natural lifecycle—it can be closed. It doesn’t need archiving. So we decided to just do deleting.

Looking more closely, deleting can be implemented in several ways. We can just wipe all the data. But in a collaborative business app, the content you create belongs to the team as much as it does to you. Is wiping the data too destructive? We could nullify the data and leave traces that something existed. Also, what about the associated sub-activities, like projects and sub-goals? What should they point to? Or should they be wiped out too?

As we addressed these questions, we realized we went too far and said—let’s not guessbuild. Let’s simply do the thing that we’re sure is needed: users who make a mistake or fool around and want to start fresh should be able to delete a goal and not see it again. So we’re going to allow just that—with a clear warning of implications. And if a goal has any sub-activities, we’re going to politely ask the user to delete those first. We’ll be happy to make it more nuanced and complicated if and when we see strong feedback pointing in another direction.

Semaphore's Open Source Repo →

Thu, 27 Feb 2025 10:12:00 GMT

As announced, Semaphore has open sourced its CI/CD platform under Apache 2.0 in a new GitHub repository: semaphoreio/semaphore.

Here’s what you can do with it:

Star the repo as a way to support us (2 seconds)
Self-host a webscale CI/CD platform (less than 30 minutes)
Contribute to the codebase and help make it even better (infinite fun)

We also have a new website domain: semaphore.io.

This is the biggest news we’ve released since launching Semaphore 2.0 in 2018. We’re super excited to continue building the best CI/CD platform in the world with the community. Marathon, not a sprint.

How to Come Up With New Features That People Want

Tue, 18 Feb 2025 17:25:00 GMT

I opened r/founder while waiting for my Magic Mouse to charge to 2%, started writing a reply and a few minutes later realized I was writing a blog post. So here goes a slightly extended version.

A founder who built and launched products before but never hit product-market fit asked:

How do you come up with new features? Purely from user requests, or do you pitch ideas and let users upvote?

Being data-driven sounds compelling, but you won’t build a great product through feature requests or metrics. So what works? Getting super deep into your space, actually watching users struggle (painful but worth it), and having the wisdom and guts to decide which problems are actually worth solving vs which ones are just noise.

First of all, build in a domain you deeply understand. Start with a clear vision of what you want your product to accomplish for its users. You need to believe in your bones that this is worth spending your next 5-10 years on.

The first versions of your product will cover only a small percentage of that vision at best. From there, you are responsible for determining the optimal path to filling the gaps.

You don’t need users to explicitly request features (although they will) — you need to learn from them about what issues they’re running into, what’s stopping them from accomplishing their goals, and what workarounds they’re creating.

Spend time actually watching people use your product in their natural environment, not just in artificial hypothetical scenarios. Video calls will be fine, as long as you don’t tell people where to click and what to do. You’ll see firsthand what frustrates them and what part of the product they totally did not understand the way you intended it to work.

Features aren’t solutions, they’re responses to problems. The best features come from asking “why?” repeatedly when users tell you what they want. For example, they might ask for a dashboard, but what they really need is confidence their work is on track. In reality you can rarely ask humans “why?” five times. Instead, again, it is your job to think deeply enough and figure it out.

You don’t need formal user voting systems. Simply track how frequently something is requested and multiply by the impact it would have. But be careful with using this as the foundation for what you do—sometimes the most valuable improvements are ones nobody asks for because users don’t know they’re possible. This doesn’t mean they think it’s technically impossible, just very unlikely that you will actually implement it and so never mention it.

Always, always, listen to your users. Sometimes you need to do exactly what they say. But often, what they say is a second derivative of a true problem at best.

You can’t automate or outsource judgment.

Making a Full-Content RSS Feed in Astro

Mon, 17 Feb 2025 18:07:00 GMT

I spent more time than I’d like to admit implementing the RSS feed for this blog. Creating a pure text RSS feed with Astro, my favorite website framework, for a Markdown content collection is straightforward. However, creating a feed that includes arbitrary MDX and renders URLs of build-time optimized images proved challenging. Having a full-content feed is free internet 101, so I couldn’t let it go. After opening an issue on GitHub and talking to some kind people on Astro Discord, I figured it out.

This was also a textbook example of a junior-level challenge that you can’t just brute force your way through with AI prompting. The solution is non-linear: it starts from the out-of-the-box idea that instead of messing around with pre-processing, you need to headlessly render posts before further adjustments, which is exactly what the experimental Astro Container API is for.

A Graveyard of Incomplete Execution Loops →

Mon, 10 Feb 2025 12:07:00 GMT

Cedric Chin writing on Commoncog:

it is easy to come up with plans and then start executing on them. The problem is that:

Either you get distracted, mid-execution, by something else that’s shinier. Or you get distracted mid-execution by something that blows up (and in business there’s always something that’s blowing up).

Or you get distracted by something else that pops up at the end of the current execution loop,

Or you forget to do the ‘study’ bit at the end of a loop, which informs your next cycle.

It’s easy to proclaim focus, but it’s a whole other game to practice the sheer discipline of seeing things through when your big customer cancels, your head of engineering leaves, the backlog gets boring, and the next shiny idea comes knocking while you’re trying to execute a multi-month strategy.

Kudos to Cedric for addressing what many of us avoid: how deeply distractions and forgetfulness affect our work, despite it seeming unprofessional to admit.

The whole Commoncog website feels like a breath of fresh air in the world of fluffy business writing.

Semaphore Summit 2025 →

Thu, 06 Feb 2025 10:42:00 GMT

We’re hosting a virtual Semaphore Summit on February 26-28 to kick off our open source journey.

Each session is short and focused: Darko, our CTO, will open the Summit with a keynote on why we made this move and what it means for the future of CI/CD. Other engineers will share tips on running Semaphore and optimizing CI/CD workflows. We’re also excited to host developers from companies like Confluent and SimplePractice sharing their experiences.

The event is free to attend and the sessions will take place during NYC noon time. The best part is that every talk will have a live Q&A session. So no long dull videos, just real conversations about continuous delivery.

Serbia's Student Revolution

Tue, 04 Feb 2025 12:45:00 GMT

Serbia is experiencing an unprecedented political protest movement.

On November 1, 2024, a 300-tonne concrete canopy collapsed at a railway station in Novi Sad, killing 15 people and severely injuring two. The root cause: terminal-stage corruption.

The railway station reconstruction project involved over 60 subcontractors—many without relevant work experience—and cost five times more than projected. The country’s president, minister of construction, and city mayor—all from the ruling party—had opened the station to the public. Documents later revealed that at the time, the station was still a construction site without a usage permit.

What followed was a unique form of political protest led by university students, which developed into Europe’s largest student-led movement since 1968.

When hooligans sponsored by the ruling party attempted to sabotage the first protests with violence, students responded by locking down the universities. Their demands were simple: publish all documentation related to the railway station reconstruction and ensure complete criminal accountability for the collapse.

For a systematically corrupt government, these demands are like Schrödinger’s cat—fulfilling them would likely put the party leadership in jail.

Waiting for students marching 80 km from Belgrade to arrive, Novi Sad, Serbia, January 31, 2025

As of this writing, hundreds of thousands of people have participated in 24-hour blockades of key intersections and bridges in Novi Sad and Belgrade. Universities remain closed, with professors joining their students. Many high schools, some elementary schools, and public kindergartens have shut down. Attorneys are on general strike. Daily protests have spread to over 200 towns and villages. When students march, they are welcomed like liberators with tears of joy and hope.

The unified message is clear: we stand with our children and support their demands.

Each day at 11:52—the exact moment when the canopy collapsed—people across the country pause their daily activities. They stand in silence for fifteen minutes, one minute for each life lost in the tragedy.

When it’s a part of a street protest, the gathering is now secured by farmers with tractors and bikers. This precaution was implemented after several incidents where people attempted to drive through the crowds and injured protesters. These attackers were presumably influenced by inflammatory government rhetoric that labeled protesters as being backed by invisible foreign powers aiming to destabilize the country.

It’s hard to describe the experience of standing in silence with twenty thousand others on a city street. The collective grief, unity, and resolve for change are palpable.

Perhaps most remarkable is the students’ organizational discipline. They have rejected all outsiders from their movement—no politicians, activists, opposition parties, or NGOs. They operate without a single leader, practicing direct democracy through plenary sessions. When speaking to media, they rotate spokespersons to prevent any individual from becoming prominent.

They cannot be provoked, frightened, defamed, or bribed.

Students marching with a banner ‘DIALOGUE’ in response to a violent incident. On the same day the government invited students to dialogue, a 23-year-old woman was attacked with a baseball bat by someone who emerged out of the ruling party’s office. Her jaw was broken. The attacker was arrested and the prime minister resigned.

The students consistently reject unconstitutional calls for “dialogue” from the country’s president—legally a ceremonial position—to negotiate their demands. They dismiss as irrelevant this man who typically dominates daily media coverage and controls everything. Instead, they insist that institutions like prosecutors and police simply do their taxpayer-funded jobs. They call for the restoration of checks and balances and the rule of law. In these dark times created by weak leadership on both sides of the political landscape, such basic demands feel revolutionary.

Eventually, this movement must culminate in both justice and political change. The timing and method remain uncertain, but consensus is building around establishing a transitional expert government with a limited term to unblock captured institutions and ensure fair elections. One thing is clear: there’s no returning to the status quo that existed before the canopy collapse.

Mantras

Thu, 30 Jan 2025 14:30:00 GMT

Therapy is expensive but saying this is free:

Ship it.
Fail fast.
Just do it.
Locked in.
Skill issue.
It’s always day one.
You can just do things.
Fuck around and find out.
Move fast and break things.
Done is better than perfect.
Discipline equals freedom.
Don’t ask for permission.
Open source everything.
Fuck it we ball.
Zero to one.
Amp it up.
Don’t die.
Kaizen.
LFG.

Semaphore Is Going Open Source →

Wed, 29 Jan 2025 17:05:00 GMT

After more than a decade of building Semaphore as a commercial CI/CD platform, we’re open sourcing the entire core under Apache 2.0. This is a moment I’ve been waiting for for a long time. The stars have finally aligned — the tools for self-hosting cloud native applications have matured, developer tools have found their natural home in open source, and we’ve built something worth sharing with the world.

What excites me most isn’t just opening up our codebase — it’s the transformation into building in public. The best developer tools are built in the open, shaped by the collective wisdom of their users. It’s time for Semaphore to join that tradition.

Update: we’re kicking off our open source journey during the Semaphore Summit on February 26th.

Foggy Weekend

Mon, 27 Jan 2025 15:55:00 GMT

Semaphore is now SOC 2 Type 2 Certified →

Fri, 24 Jan 2025 15:12:00 GMT

We just got our SOC 2 Type 2 certification at Semaphore. This means we’ve proven our security practices work consistently over time, not just on paper. Protecting our customers’ code and data has always been a top priority for us, and now we have the audit to back it up.

The difference between Type 1 and Type 2 matters here. Type 1, like ISO 27001 which Semaphore has been certified to since 20201, is a snapshot – it shows your security controls look good on a specific day. Type 2 proves you’re actually following these practices over months of real operations. It’s the difference between having a gym membership and showing you actually go regularly.

Getting here wasn’t quick or easy. Unlike typical product work, there’s no definitive specification for SOC 2. No clear manual that says “do exactly these things and you’ll pass.” Instead, you’re working with broad principles about security, availability, and confidentiality that you have to interpret and implement in the context of your specific business.

I’m very proud of our small security team who turned these vague compliance requirements into real, practical security improvements across the organization.

To our customers: this certification confirms what we’ve been doing all along – treating your code and data with the care it deserves. To anyone considering Semaphore: this is what we mean when we say security is built into how we operate, not bolted on later.

AI Can Build Apps Like Calculators Can Do Math

Mon, 06 Jan 2025 12:17:00 GMT

Naval Ravikant on X:

AI won’t replace programmers, but rather make it easier for programmers to replace everyone else.

Someone jumps in with the “but Jensen Huang says AI writes code!” argument. Naval’s response packs as much wit and insight per character as ever:

Calculators can do math per CEO of Texas Instruments.

Exactly. Now is the best time in history to be a programmer.

AI expands your capabilities into domains where your knowledge was insufficient. If all you ever wanted was to create, and you’re looking at AI the right way, chances are you feel reborn.

Designers who dreamed of breaking new ground in human-computer interaction but got stuck styling forms and churning out YouTube thumbnails should be excited.

The industry is resetting to where it was 18 years ago, before iPhone and when “social” web was just emerging. The soul-crushing parts of the job are being automated away. What’s left? The work that actually matters: original thinking backed with good taste.

Technology creates demand for more technology. There will always be new programs to write.

In the new year, fly free

Thu, 02 Jan 2025 11:21:00 GMT

Enable Dictation

Fri, 27 Dec 2024 15:49:00 GMT

You can save yourself hours of typing by talking to your Mac.

With AI, better context means better results. But there’s a gap between knowing this and doing it. Even as someone who writes constantly (running remote companies made me a pretty effective writer), I still hesitate before diving into those long, context-rich prompts that yield the best outputs.

The solution is simpler than you might think: macOS dictation. I completely slept on this feature, assuming I needed diction of a US president. However, it’s surprisingly accurate for non-native speakers too - just hit F5 and talk.

Now instead of carefully typing out detailed prompts, I can talk through my thought process naturally. Same goes for drafting documents or emails (with some cleanup before sending). This post started as pure dictation.

Here’s why it works: verbal expression of complex ideas is fundamentally easier than typing. We do it effortlessly in meetings and conversations. But for many people, writing is an order of magnitude harder than speaking. Remove the typing friction, and you’re more likely to tap into the full potential of AI.

Try hitting F5 next time you’re staring at that blank prompt. Let your Mac do the typing while you focus on the thinking.

10 Ambient Music Albums of 2024

Fri, 20 Dec 2024 12:23:00 GMT

Ambient music is my workday companion. The albums collected here — ranging from spectral drones to late-night meditative jazz to minimalist classical pieces — have a peculiar effect: they help me carve out a space where focus is sustained and time seems to suspend itself. These are the records that have colored my working hours throughout 2024.

Rafael Anton Irisarri - Façadisms

Endurance (Original Motion Picture Soundtrack)

Earthen Sea - Recollection

Jakob Bro - Taking Turns

Malibu - Palaces of Pity

Jon Hopkins - Ritual

Alva Noto - Xerrox, Vol. 5

Roger Eno - The Skies: Rarities

Not Waving & Romance - Wings of Desire

Suso Saiz - Distorted Clamor

Musk Operates Like a 19th Century Industrialist →

Wed, 18 Dec 2024 12:05:00 GMT

Marc Andreessen on Chris Williamson’s podcast:

I’m not aware of another current CEO who operates the way that he does. But if you go back in history, you find characters more like him, especially the industrialists of the late 1800s, early 1900s, people like Henry Ford or Andrew Carnegie, or Thomas Watson, who built IBM.

The top line thing is just this incredible devotion from the leader of the company to fully, deeply understand what the company does and to be completely knowledgeable about every aspect of it and to be in the trenches and talking directly to the people who do the work, deeply understanding the issues and being the lead problem solver in the organization.

Basically what Elon does is he shows up every week at each of his companies, identifies the biggest problem that the company is having that week and fixes it. Then he does that every week for 52 weeks in a row. And then each of his companies has solved the 52 biggest problems that year. Most other large companies are still having the planning meeting for the pre-planning meeting, for the board meeting, for the presentation, for the compliance review and the legal review.

He delegates almost everything… [except] the biggest problem right now until that thing is fixed.

The best founders focus on fixing their biggest problems instead of managing them - but doing this week after week, year after year, is a brutal standard that most can’t meet.

Blind by Default →

Mon, 16 Dec 2024 16:06:00 GMT

Sebastian Bensusan, writing about how digital work is invisible by default:

Imagine you work at a paper store:

One day, you observe there are a lot more customers in the store, mostly Chinese, and they are all buying red paper. Some of them ask for red envelopes, which you don’t have.

Eventually you ask them, what are the envelopes for? “Honboa?! Chinese New Year gifts of course!”. Trying to orient yourself, you read the Wikipedia page and it all clicks together.

You decide that you should be ready for the next Chinese New Year.

You act by ordering the envelopes which 10x sales. Success!

This wasn’t a tough mystery to solve… if you work at the store.

Yet if you work at an e-commerce store selling the exact same items, it’s almost impossible to get this depth of information at comparable speed:

By default you know nothing, not even if you had sales that day. To know that, you have to:

Decide that you want to know the daily revenue.

Calculate it (and debug it!) with a SQL query, then put it on a dashboard.

Look at the dashboard daily.

But are you breaking the revenue by ethnicity / nationality? Are you breaking it down by paper color? Did you notice Chinese New Year?

Most likely not. In a software business, you are default blind.

The concept of organizational blindness is one of the most under-discussed challenges in digital business.

What makes this article powerful is how it shows that blindness isn’t just about lacking data or tools. It’s about constantly working with an unknown number of unknown unknowns.

Looking back, most of my career has been about building tools that make the invisible parts of work visible: first with Semaphore and CI/CD workflows in software development, and now with business operations in Operately.

There’s no silver bullet except developing a habit of actively observing and questioning what’s happening around you, and sharing what you learn with others.

Semaphore Pulls Back the Curtain →

Fri, 13 Dec 2024 10:29:00 GMT

Semaphore launched a new YouTube channel called Semaphore Backstage to document the engineering journey building the next-generation CI/CD features. The first video takes you inside a product meeting where the team tackles real challenges in continuous delivery – from multi-region deployments to version queuing. Smash the like button and subscribe!

Cudis Smart Ring →

Wed, 11 Dec 2024 09:31:00 GMT

For decades, we’ve been presented with a choice in consumer tech: either surrender your personal data to the surveillance economy or pay a high premium for privacy-focused alternatives, often sold by companies that don’t sustain over time. This dichotomy has become so ingrained that we barely question it anymore.

The Cudis smart ring appears to be attempting something different, storing user health data using IPFS and blockchain verification, with users maintaining ownership. The company claims users could monetize their own health data—up to $5,000 per year. While that specific figure warrants skepticism, the model itself is intriguing. When crypto enthusiasts talk about use cases this is the kind of practical application that’s been missing—not speculative tokens, but fundamental restructuring of data ownership. Whether it works remains to be seen, but it’s refreshing to see someone challenging the assumption that privacy must come at a premium.

Orchid Synth →

Tue, 10 Dec 2024 08:22:00 GMT

Conceived by Tame Impala and created by Telepathic Instruments, Orchid is

an advanced chord generating hardware synthesizer developed for songwriters, producers and musicians to expand the possibilities of songwriting and musical exploration.

What a fascinating design. One of those rare things that bring out the inner child in me.

Windsurf →

Mon, 09 Dec 2024 23:22:00 GMT

I haven’t hit “subscribe” this quickly in ages. While I’ve been using Claude and its artifacts as my programming sidekick for months, Windsurf editor takes the AI coding experience to new heights.

What makes it stand out:

Full codebase awareness with ability to mention specific files and folders
Direct command execution
Intelligent reasoning about manual (is this how we will call it now?) code changes
Seamless multi-file workflows

My number one advice to anyone struggling to get results from products like this: you need to improve at explaining what you mean.

Public By Default →

Mon, 09 Dec 2024 12:26:00 GMT

Still fascinated by the effects of genuinely building Operately in public - from meetings on YouTube to public Discord discussions. As a tiny challenger in the market, every breadcrumb we leave in public becomes both a trail for the right people to find us and a record of our journey. The supposed risk of openness pales in comparison to the risk of never being found at all.

Starting a Personal Blog in 2024 (Yes, Really)

Sun, 08 Dec 2024 20:53:00 GMT

There’s something wonderfully absurd about starting a personal blog in 2024. The whole concept feels outdated – like buying a typewriter or developing your own film. And yet, here we are.

In recent years, I was convinced that maintaining a personal blog would be self-indulgent at best and a waste of time at worst. After all, I have enough obligations, and we have social media.

But lately I’ve been thinking about what we lost when we all moved our thoughts to social platforms. There’s a peculiar kind of clarity that comes from writing—and reflecting back—on your own website. It’s just you and your thoughts, without the constant background radiation of likes, trending topics, and engagement farming.

Every social platform wants us to be performance artists now. They want us to create “content” exclusively for their walled gardens, each with their own arbitrary rules and algorithms. Post an external link? Algorithm penalty. Share something not aligned with your designated topic? Sorry, we’ll have to limit your reach going forward. It’s exhausting, and worse, it shapes how we think.

I miss the weird personal websites of the early web. They were genuinely personal. People shared what they liked with actual commentary, posted random thoughts without worrying about engagement metrics, and generally treated their websites like digital gardens rather than content farms.

So here’s what this space is going to be: a place for notes, quotes, pretty images, and bookmarks about business, software, open source, art, and whatever else feels worth sharing. Will anyone read it? Maybe, maybe not. But if nothing else, I’ve found that the simple act of curating and publishing things publicly helps me clarify thoughts and nourish creativity.

Think of this as a quiet corner of the internet where the wifi is decent and the algorithms can’t find us. No pop-ups, no ads, no growth hacking – just some notes from someone who believes the internet works best when people have their own spaces to think out loud and connect with others.

Welcome to my corner.

Edit: After a while I imported some posts I originally wrote for the Rendered Text, Semaphore, and Operately blogs. The archives now go back to 2010.

Operately v0.1: Open-source first, SaaS next

Thu, 05 Sep 2024 12:45:51 GMT

We just dropped Operately v0.1. You can install it in 5 minutes on a cheap cloud server.

This release marks a significant milestone for us: it’s the first time users can install and run Operately on their own servers, bringing our open-source commitment to life.

Wait, I thought you guys are building a SaaS?!

We are! But we’re also dead serious about open-source. Self-hosting needs to be easy and cheap from day one.

Our north star is WordPress-level simplicity for business software.

SaaS is in our DNA, self-hosting isn’t. So if we don’t nail self-hosting now, we never will.

OK, makes sense. What’s in the release?

Operately v0.1 is your company’s command center that unifies goals, projects, activities, and areas of ownership.

Watch this quick video:

Nice! When can I try it online?

Soon. Join the waitlist to be first in line.

Behind the scenes

It took us 2 months to ship v0.1 after deciding we were ready. There’s always a slew of not-so-fun stuff to take care of the first time. To give you an idea, here’s the brief from our development project:

Shipping builds momentum. It’s not new to us but the feeling is exhilarating and never gets old. We’re committing to monthly releases of self-hostable Operately and continuous updates once we launch the SaaS.

We want to bridge the gap between open source flexibility and SaaS convenience. Stay tuned for what’s next.

The Proactive Manager

Thu, 25 Jul 2024 15:32:01 GMT

Your startup is running a marathon, and it’s your job as a leader to clear the road ahead so your team can keep running.

But how often do you find yourself so deep in the trenches, working on current projects and reacting to problems, that by the time you finally catch a breather, you have no idea what to focus on… until the next fire erupts?

Being a proactive manager means spending an appropriate amount of time up front to minimize the number of fires you’ll need to put out later.

On the far end of the spectrum of mastery, we have people like Jeff Bezos, who claims he’s always working two or three years into the future.

Replace years with months, and you have a good target to aim for as a startup founder.

Here are a few things you can do to get there:

Start sketching and writing down a detailed plan of what you’d like to see happen at your company over the next 3-6 months.
Think about upcoming work and identify any potential issues. Remember, it’s your job to clear the road. Your team will feel more supported and less likely to burn out.
Draft wireframes and scope out projects for 1-2 iterations ahead.
Set up logistics. Something as trivial as not booking a venue or scheduling work with your external partner up front may delay the schedule.
Align the appropriate people to be ready when work comes their way.
Know everyone’s holiday schedules.
Be ready to pivot when new information, opportunities or challenges arise.
Deeply understand each ongoing project and the issues they’re running into. Ivory tower mode is for corporate types making a career at EnterpriseGlobal.

Look, unexpected stuff will always pop up - that’s startup life. But by carving out time to think beyond the current fires, you’re giving yourself a fighting chance.

Sketch out that new feature. Prototype that crazy idea. Or even just close your eyes and imagine where you want to be in six months.

You’ll see how things will flow from there.

The future you will be a lot less stressed — and a lot more excited about what’s next.

How to Start a Year with a 10x Month

Thu, 18 Jan 2024 13:38:27 GMT

Let’s start with something simple. Sketch out your goals for this year or maybe just the next six months.

Now, challenge yourself: how could you accomplish all of it in just one month?

Let’s be clear, though. It’s not about burning yourself out with an impossible todo list. There are only so many projects you can handle at once, and sure, some things inherently take time. But what if you pushed the boundaries… just a bit?

Give it a shot. Craft three pieces of content, even if they’re not perfect, before lunchtime. That job description you’re mulling over for next week? Finish it today. Planning to reach out to prospects “later in Q1”? Send emails to two people right now; the perfect CRM and a comprehensive playbook can wait.

You might just be surprised by the results.

At the very least, you’ll be energized by the momentum you’ve created. This energy is contagious and your colleagues are likely to catch the vibe. You’ll realize that many roadblocks are internal, not external. And once you start, you won’t want to stop.

There are enough voices out there telling you what can’t be done. Psyop yourself for a change. Shake things up. You might be amazed at where you end up.

You Don't Need a Grand Vision

Wed, 20 Dec 2023 13:08:47 GMT

When Semaphore hit the seven-figure mark as a bootstrapped business, my cofounder and I sat down to rethink our strategy. Spread out in front of us was a business framework canvas, with ‘Big Hairy Audacious Goal’ (a concept introduced by Jim Collins in his 1994 book Built to Last) right in the middle. Our gut reaction? “Bro, we just want more folks using and benefiting from our product!”

The guiding principle of ours—making it possible for all developers to test & deliver cool products quickly—didn’t seem grand or hairy enough. It felt like we were missing some secret ingredient that successful leaders supposedly have to spell out their success.

But as time went by, Semaphore just kept growing. What drove this growth? It was a mix of steady, continuous learning and iterative improvements, peppered with a few bold moves. Like when we decided to overhaul the product for a new era of container-based cloud computing workflows. Or launching an enterprise offering.

You hear all the time that you need some grand, sweeping vision to make it in business. Here’s the thing: you don’t. If your product solves a real problem that people are willing to pay for, just keep at it:

Stick to the core reason why you started in the first place.
Maintain close contact with your customers.
Pay attention to how customers get creative with your product or where they hit walls. Let these insights guide your product’s evolution.
Know when it’s time to step up from iteration to a bold leap. Pursuing a high risk/reward opportunity falls squarely on the shoulders of a founder.

Along the way, your experience might just lead you to grander, wider-reaching ideas. You’ll figure out what resonates with your team and you. Maybe one day you’ll pen down that big, bold vision statement that feels just right.

For now, there’s plenty to tackle with what’s in front of you. Let your vision grow and change with your business – that’s how it should be.

The Pitfalls of Over-Optimizing for One KPI

Wed, 18 Oct 2023 11:45:12 GMT

We all act differently when we’re alone vs when there’s someone else in the room. In the same vein, the mere act of measurement influences our actions.

Ever noticed an engineering team, when judged by ticket counts, suddenly breaking tasks into smaller tickets? Or a call center team, gauged by call numbers, wrapping up calls quicker, often leaving customers furious? What about that social media intern boosting engagement metrics with divisive content that jeopardizes brand reputation?

In our age of information deluge, there’s an undeniable appeal in finding a singular guiding metric or “north star”. But there lies the trap.

Goodhart’s Law warns, “When a measure becomes a target, it ceases to be a good measure.” This translates to: if you single-mindedly chase one goal, you’re likely to pursue it, even if it spells disaster elsewhere. Achieving one goal by neglecting other pivotal aspects is hardly a win.

Consider a marketing team zeroed in on ramping up blog traffic. They could easily drum up content and campaigns luring pageviews, neglecting to ensure these visitors align with the brand’s target audience. The result? A surge in irrelevant leads unlikely to ever convert.

Goodhart’s Law offers us a clear lesson: balance is crucial. Rather than fixating on a lone KPI, a set of varied metrics offers a more holistic, and less short-sighted, approach to reaching our goals.

A reality-challenging physics experiment shows that particles act differently while being observed which led some people to think there are parallel universes

Introducing An Open Library of Business KPIs

Fri, 01 Sep 2023 13:06:07 GMT

As we’re building the alpha version of Operately, we’re equally committed to creating content that helps businesses work smarter. And in that spirit, I’m excited to share with you our first curated content project: KPI Examples.

KPI Examples homepage on Sep 1, 2023

Does the world really need a website about KPIs?

It’s a common adage: You can’t manage what you don’t measure. A proven method for efficiently getting things done is by assigning each team member a specific metric for accountability.

A few years back, while establishing new departments at Semaphore in areas I wasn’t familiar with, my initial step was often hunting for a comprehensive list of KPIs for that specific function. Because a robust KPI list essentially outlines what excellence looks like in that domain.

In that job, I kept returning to kpilibrary.com—a comprehensive crowdsourced database that, regrettably, is no longer online. Nowadays, if you search for example KPIs, much of what surfaces is content that lacks depth—aimed more at SEO or product promotion rather than providing clear-cut insights.

KPI Examples is open source and a work in progress

We want the content we publish—especially since KPI Examples is open for contribution—to be around for a long time. That’s why the website is open source on GitHub, and the KPI database itself is a public Google Sheet.

You’ll observe that some categories on the site are currently empty; however, you can leave your email to be notified once they’re populated. We’re committed to expanding the database of KPIs and categories on a weekly basis.

If there’s any feedback you’d like to share or if you’re interested in contributing, please don’t hesitate to reach out. For our newsletter subscribers, simply hitting ‘reply’ will direct your message straight to me.

When and How to Move to a New Customer Segment

Tue, 22 Aug 2023 11:45:39 GMT

Chatting with an early-stage founder recently, he posed a question that I’m sure rattles around in the heads of many: “We’re selling our consumer software for $10/month. How do we get some of those big enterprise customers? Ads? Outbound?”

My response was straightforward: You can’t force with some paid ads or clever outreach what you don’t see already organically happening inbound.

Products don’t just leap from one customer segment to another on a whim. There’s always a little wiggle room; you can, of course, cater to those slightly below or above your core market. But go too far below, and your product feels like a behemoth. Aim just a touch too high, and suddenly you’re the lightweight with frustrating shortcomings.

Still, if you’ve made a strategic decision to move upmarket, there’s a path forward: cozy up to your most demanding customers. Understand their pain points, figure out what would make your product indispensable to their entire organization, and craft a roadmap around those needs. Balance this with nurturing the needs of your existing customers. And if all goes well, you’ll find a new breed of clients beating a path to your door.

This isn’t a sprint; it’s a marathon. You can make some progress refining your current features, but sometimes it takes an executive decision to make some audacious bets.

Semaphore’s two big bets

Here’s a case in point. Semaphore, initially tailored for small dev teams crafting web apps, underwent a full overhaul five years post-launch.

While it’s almost taboo to even whisper about a product rewrite—most product gurus would advise against it—it was our ticket out of a cutthroat competition in our primary segment. Fast-forward to today, and we’ve doubled down on that strategy, building Semaphore On-Premise catered specifically for the enterprise giants.

So, if you’re standing at that crossroad, deciding whether to make that leap to another segment, remember this: the journey isn’t easy. But with customer understanding, bold moves, and commitment, the leap can redefine your product’s destiny.

Before a Product, Build Financial Stability

Tue, 25 Jul 2023 11:45:03 GMT

Many aspiring founders dive headfirst into product creation, carried away by the fascination of the process. But let’s not mince words—it’s an uphill battle.

The pressure to monetize quickly—to validate your worth to investors or merely keep the lights on—can trigger premature product launches, pivot you away from your original vision, or worse.

Funding is a privilege, not a given

Startup culture often portrays funding as being always within reach: craft a compelling pitch, tirelessly network, and if your idea is solid, you’ll be rewarded with a cash infusion to bring it to life.

But the reality is starkly different. Despite the advent of remote work, securing funding remains a privilege accessible to a select few, concentrated in global startup hotspots. For most of the world, bootstrapping presents a path of independence, offering an option to either remain entirely self-reliant or leverage future funding on more favorable terms.

Consider starting with services

An established business acts as a financial safety net, offering stability as you venture into product creation. It’s a bit like having a secure day job while pursuing a passion project—you have the freedom to experiment, iterate, and innovate without the constant worry of keeping the roof over your head.

Reflecting on my journey, I recognize how running a web development consultancy offered a prism to critically evaluate potential product ideas. It also served as a hands-on training ground, introducing me to business development, hiring, and management, thereby preparing me for the complexities of running a product-based company.

To all the aspiring product creators out there: financial stability isn’t just a safety net—it’s a launchpad. Take the time to establish a solid foundation before you leap into the exciting world of product creation. It might make all the difference between a graceful takeoff and a crash landing.

How to Know What Customers Would Pay For

Tue, 18 Jul 2023 11:45:04 GMT

Hobbyists create what they wish to see in the world. Successful entrepreneurs discover the needs and problems that customers are eager to pay to resolve.

Sure, being your own first customer makes things easier. You can “scratch your own itch” and find innovative solutions to your problems. But remember, the world doesn’t revolve around you. Your “itch” might not be a viable business opportunity.

Nobody will pay for a nice-to-have

To clarify whether your product is a major pain killer or simply a nice-to-have, ask yourself the following questions:

1) How significant is the problem you’re trying to solve? Successful businesses aren’t built on minor annoyances. Your solution should address a critical and/or urgent pain point. After all, your greatest competitor is doing nothing.

2) Who is dealing with this problem? Is it a niche group or a broad market segment with potential for sizable returns? How much purchasing power do your product champions have? Ensure your target audience not only wants, but is also allowed to pay for your solution.

3) What alternatives exist? Are there competitors in the market, and how will your product compare? Are there established solutions who could add competing features in the short or mid term? Your product needs to provide a significant, long-lasting improvement.

4) How much would your customers save by using your product? Current economic conditions favor products that enhance productivity and efficiency. If your product can save customers time, money, or stress, they’ll be more likely to invest in it.

A weak example: single-purpose AI tools

Right now there’s a flurry of AI tool products that are essentially thin layers over ChatGPT or another AI language model. You’ve probably seen many ideas or MVPs that follow this pattern:

An AI tool that (takes some input) in (domain) and (provides some output).

But think about this: what if every company with a stronghold in (domain) is already building this? Or OpenAI itself? Whoops.

To avoid overcrowded spaces, build what isn’t obvious

If you are interviewing prospects in search of problems to solve, keep in mind they might not be able to articulate what they’d pay for. Their pain points could be so ingrained and “normal” to them that they fail to identify them as such.

This is where your deep expertise and observation come in. By truly understanding your customers and their needs, you can unearth problems they may not even know they have—until they see your solution, that is.

The inception of Operately wasn’t a result of random inspiration or systematic research—it came from over a decade of experience running software businesses. We’ve come to realize that disciplined and efficient processes are critical for successful execution. Yet, until now, most organizations have been piecing together makeshift solutions. So, even without direct competitors sharing our vision yet, we’re confident that we’re addressing a real, substantial problem.

In conclusion, understanding what customers want is just the start. Digging deeper to unearth overlooked but critical problems they’d willingly pay for—that’s what sets successful products apart.

Deep Expertise: The Foundation of Successful Product Creation

Tue, 11 Jul 2023 14:02:03 GMT

A recent conversation with an aspiring founder led to an intriguing question: “How did you know that you were ready to start building your products?”

For both Semaphore, twelve years ago, and Operately, more recently, the simplest answer is—it felt inevitable.

Of course, creating products isn’t about one eureka moment or a quick sprint. It’s a meticulous day-by-day process that unfolds over years. So, setting the final product aside, how do you reach this point of inevitability?

It all begins with developing deep, first-hand expertise in a specific area.

The birth of Semaphore

Having first-hand expertise gives you the power to intimately understand the market’s pain points, gaps, and opportunities. It allows you to design solutions that are genuinely useful and desirable.

Rewind to the early 2010s, when our company, Rendered Text, was a modest web development consultancy. Every single one of us was on a mission—to become the best developer we could be. Automated code testing and deployment were the backbones of our workflow. As our team and the projects we worked on expanded, we realized the need to formalize the continuous integration process with a dedicated tool.

Our best option at the time was Jenkins—an antiquated software that required end-users to install and maintain both Jenkins itself and the infrastructure to run it—a far cry from user-friendly cloud products like GitHub and Heroku that we were already used to!

This glaring gap presented an opportunity. We envisioned a SaaS solution that required just minutes instead of days for setup, was inexpensive and fast, not expensive and slow, and offered one-click scaling over manual labor. We were confident we weren’t alone—after all, we didn’t invent the product concept or the core customer needs.

Our hands-on experience enabled us to design Semaphore’s MVP with features that directly addressed the problems our very first users were willing to pay for.

In a nutshell, our journey with Semaphore wasn’t sparked by a random inspiration or a casual idea—it began with an intense immersion in our domain and industry. That deep expertise gave us the lens to identify opportunities and create something genuinely helpful for our customers.

So if you’re wondering when to kickstart your product journey, my advice is this—immerse yourself in an area where you have innate abilities and interests. Develop your unique expertise and let it guide you to the point of inevitability.

Your Biggest Strengths Are Also Your Biggest Weaknesses

Tue, 04 Jul 2023 11:45:02 GMT

During a recent indie founders’ meet-up, one question sparked a lively discussion: “What are your biggest weaknesses as a founder that are holding you back?”. The common thread that emerged had little to do with gaps in skill, knowledge, or drive. It was about our inherent belief that we, as founders, could do everything ourselves.

Now, let’s unpack that a bit. Founders are known for their ability to wear multiple hats. They’re often polymaths, possessing the knack to navigate the early days from idea to launch—handling everything from coding to sales, from fundraising to customer support.

But, as the company grows, the founder’s role needs to pivot too. They must transition from being the jack-of-all-trades to leading a team, and later, an entire organization.

Here’s the catch: when circumstances change, our biggest strengths can quickly become our biggest weaknesses.

The world-class expertise that got the company off the ground can become a hindrance in hiring someone else to take over. It may be tough for first-time founders to realize that 80% quality delivered by an autonomous, fault-tolerant team is far better than a distracted genius’s scattered efforts.

If you’re highly independent and self-sufficient, you may find it challenging to delegate, causing burnout and opportunity cost before finally making the leap. Similarly, while founder’s natural optimism keeps morale high and promotes perseverance, if left unchecked it can lead to underestimating risks and ignoring negative outcomes.

As founders, we need to recognize that in parallel to our startup’s journey, our own journey unfolds too — one of self-awareness and maturity. With great power, there comes the need for great adaptability.

Loving to be Wrong

Tue, 27 Jun 2023 11:45:11 GMT

You might have heard some of the most effective leaders and innovators proudly declare, “I love being proven wrong!” At first glance, this might seem counterintuitive. Who would enjoy making mistakes or facing failure?

Well, let’s clarify what they’re really expressing.

Take the example of Thomas Edison, the famous inventor known for his dogged pursuit of innovation. When asked about his repeated failures while inventing the light bulb, Edison famously remarked:

I have not failed. I’ve just found 10,000 ways that won’t work.

For him, and many like-minded innovators, being proven wrong is not about the failure itself. Instead, they relish the clarity that each “failure” brings, taking them one step closer to the truth they’re pursuing.

Concept inspired by Visualize Value

In school, we’re frequently rewarded for right answers and penalized for wrong ones. This binary perspective often fosters a mindset that equates being wrong with failure, embarrassment, or incompetence. As a result, many people might strive to avoid being wrong or view it as a negative experience.

People who break free from fear of being wrong open up new possibilities for innovation and problem-solving. For these progress-oriented individuals, mistakes are not roadblocks, but signals: “Good. Now we know. We can move forward.”

The journey to success is not a straight line, but a series of trials, errors, and constant forward movement.

So, the next time you hear someone say that they love being wrong, remember that it’s not about finding some twisted joy in failure. It’s a declaration of commitment to learning, growing, and relentlessly pushing forward. It’s a profoundly effective approach to personal and professional growth—one that entrepreneurs and startup founders would do well to embrace.

How Startup Founders Should Spend Their Time

Tue, 20 Jun 2023 11:46:00 GMT

Like any executive, a startup founder’s primary task is to concentrate their energy where it will be most impactful. Easier said than done, of course, but the following trifecta can serve as your guiding star:

What your team, customers, and company need
The big bets and most critical areas, in other words, high-leverage work
Where your personal strengths lie

Where these three factors overlap, that’s most likely to be your sweet spot—the perfect work to invest your time in.

Maker or Manager?

Once you’ve identified your sweet spot, your schedule will likely fall into either a maker’s or manager’s rhythm, as explained by Paul Graham.

If you’re in ‘maker’ mode, focusing on creation, building, or strategic planning, you’ll require uninterrupted blocks of focus time, ideally during your peak energy hours. For example, you may want to dedicate your mornings until lunch (or gym!) time to this. Even better, try to set aside whole days for your top priority, using only short afternoon slots for asynchronous communication and meetings unrelated to your key focus.

Conversely, if you’re in ‘manager’ mode, your goal is to stay disciplined and concentrate on your most crucial work for the majority of the day, despite other tasks vying for your attention. And remember, it’s crucial to carve out some quiet time for reflection every day.

The Peril of Wearing Too Many Hats

I’ll share a personal insight: the most hair-pulling times in my career were when I tried to juggle both maker and manager modes, bouncing between different business areas within a single day.

At one point I even kept a detailed spreadsheet of my daily activities, categorized by work area, type, and purpose. Yet, I was the least productive person in the company. I was always wrestling too much unfinished business, and results suffered.

Forge Your Own Path

For hired execs, the ‘what’ to focus on is pretty much set in their job description. But for founders, it’s a whole other story—it varies big time depending on where the company’s at.

In the early days, you might be immersed in building the product, getting the word out, or fundraising. Later, your focus could shift towards hiring or setting up processes. This variability is why you’ll get a different answer each time you ask a CEO how they spend their time. It doesn’t make much sense to try and mimic the daily routines of famous founders or CEOs, as there’s no one-size-fits-all answer here.

There’s no silver bullet. Instead, use the trifecta we talked about to reason from first principles and figure out where to put your energy. It’s your journey, after all. Forge your own path.

You Might Be an Executive and Not Even Know It

Tue, 13 Jun 2023 11:45:49 GMT

Have you ever thought of yourself as an executive?

I know, I know, you’re thinking: “Executive? Me? You’ve got the wrong person.” But stick with me here. Peter Drucker, the grandfather of modern management thinking, had a definition of “executive” that’s broader—and more applicable—than you’d think.

For Drucker, anyone making decisions that affect the performance and direction of their organization, no matter how small that effect and organization may be, is an executive.

That Social Media Manager who plans and publishes the content that influences a company’s reputation and customer engagement? Executive. The HR Manager who vets job candidates and crafts policies shaping a company’s culture? Executive. The solopreneur who’s juggling sales, marketing, and maybe even a bit of coding? Definitely an executive. Even a Team Lead, who ensures projects get delivered on time while juggling team dynamics, is an executive in their own right.

Surprised? You shouldn’t be. In the knowledge economy based on intellectual capital, more of us are “executives” than we realize. The kicker is, many of us are flying blind, not knowing the essential skills we need to rock these roles.

Drucker outlined five practices for business effectiveness - real, tangible skills to level up your work performance. And is it always goes, the oldest advice is the best advice:

1) Know your time: Master time management, strip out the fluff, and focus on what really moves the needle.

2) Know your contribution: Stop asking “What do I want to do?” Start asking “What would be the best thing I do?” Get obsessed with how you bring value to your team, your clients, your company.

3) Play to your strengths: Don’t sweat on weaknesses. Understand your strengths, and put them to work. Do the same for your team. Let everyone shine where they’re best. Square pegs don’t fit in round holes.

4) Make big bets: Drucker was raving about the 80/20 rule long before fancy TED talks. Focus on the most critical areas instead of trying to do everything. Most results come from a fraction of the effort.

5) Make effective decisions: You’ve got to make calls – some tough, some easy. But that’s the job. Gather the facts, weigh the options, decide, move on, and most importantly, learn from the outcome.

Drucker published The Effective Executive in 1966. But the playbook isn’t just for the suits in the Mad Men-style boardroom. It’s for all of us. Understanding that you’re playing an “executive” role and having the right tools to play it well - that’s how you win.

What's the Real ROI of a Company Retreat?

Tue, 06 Jun 2023 11:45:47 GMT

Just a week after my post about the ins and outs of establishing KPIs, I was heading to a company retreat with the teams from Semaphore and Operately.

Semaphore team at company retreat in Turkey, 2023.

It got me thinking… It’s easy to get sucked into the notion that a business is just a bundle of inputs and outputs, a mix of processes and metrics. Like if we just zero in on the right ones, we could steer the whole ship based on pure, data-driven feedback loops. But as critical as systems and figures are to keeping a business running, there’s a whole lot that matters that we simply can’t boil down to numbers.

Take the company retreat, for example. Semaphore operates remotely, our 35-strong team scattered across Europe and the Americas. With a few exceptions, most folks who’ve joined us post-Covid haven’t met their colleagues face-to-face. Sure, we’ve got our virtual watercooler chats as part of the regular work rhythm, but the shift in team energy before and after spending actual time together? That’s something else.

Actually a super productive engineering meeting

A startup’s culture — its shared values, behaviors, and beliefs — is a secret sauce in its recipe for success. A startup is, at heart, on a mission to bend reality against all odds, and it needs a strong team spirit, motivation, and trust to make it happen. But here’s the question: how do you measure the vibrancy of a culture and the cohesion of a team? It’s safe to say you won’t get far with quarterly NPS surveys and tallying up employee turnover.

Queueing up for breakfast. Getting turned around in an unfamiliar town in the pouring rain. Convincing the hotel manager to bend the rules for some after-hours pool games. A late-night beach gathering. It’s only after such moments that we can truly realize that, as my colleague Ivana has put it, “Each person in our team is so unique, yet our energies match perfectly.” It’s truly something to be part of, and there’s nothing like working with the team battery fully charged to 100%.

Beyond Outcomes and Outputs

Tue, 30 May 2023 11:45:54 GMT

“Outcome vs. output” is the wrong way to frame business performance.

Startups go through two stages of frustration before they figure it out:

Stage 1: You aim for goals like “Get the new product/campaign to X people by the end of Q2”. Your team pulls it off, but bottom line doesn’t budge. You feel disappointed. Then you realize — “We should focus on outcomes, not just output!”

Stage 2: So, you shift gears. You set goals like “Bring in XYZ new trials each month.” You run a bunch of related projects, or maybe you execute by a playbook, but you only get about 8% closer to your goal. You feel stuck.

So what’s stage 3? Let’s dig into that.

There’s no doubt that outcomes are what ultimately matters and must drive the plans. But here’s the rub: they aren’t quite in our control.

When it comes to planning and checking up on ourselves, we need something more within our control.

In his 2009 Letter to Shareholders, Jeff Bezos described Amazon’s focus on business inputs (emphasis mine):

Senior leaders that are new to Amazon are often surprised by how little time we spend discussing actual financial results or debating projected financial outputs. To be clear, we take these financial outputs seriously, but we believe that focusing our energy on the controllable inputs to our business is the most effective way to maximize financial outputs over time.

So, where do these ‘inputs’ fit in our “outcome vs. output” puzzle? Here’s a few definitions to help clarify things:

Inputs are the stuff we need for our work — and that includes us.

We’re talking things like your team, your workspace (be it an office or you remote setup), your hardware and software, what your customers need, your business partners and vendors, funding, etc.

Activities are the tasks and processes needed to get work done.

This can be anything from internal communication, researching, designing, building, testing & deploying, creating & publishing, and closing deals — all of which you can examine more closely for improvement.

Outputs are what you get when an activity is done.

These could be anything from launched products and features, services you’ve provided, to your team being trained on essential security practices. This is where our direct influence ends and we yield control to the market — and luck.

Outcomes are the immediate, primary effects you get from delivering a work product — usually the actual added value that comes from an output.

Think things like a new product feature being successfully used by all targeted users with uptime of 99.99%, an increase in signups for a free trial, or customers raving about your amazing customer service.

Impact are the long-term ripple effects of the work you’ve completed.

This can show up as strong customer retention, a healthier financial bottom-line, a competitive edge, or a reinvigorated company culture.

Once you’ve got these ideas clear in your mind, it’s much easier to put a continuous improvement cycle into action.

Of course, if you’re a startup and you’re trying something new, there’s no way to know for sure if what you’re doing will work. But with this kind of thinking, you can approach your work like a series of hypothesis-driven experiments, not like a shot in the dark. Your odds of success will surely improve.

5 Pitfalls to Avoid When Establishing KPIs

Tue, 09 May 2023 12:02:55 GMT

Have you ever seen Key Performance Indicators (KPIs) get set and then fade away? This may actually be fine!

You set KPIs to measure your progress and maintain focus on your goals. As things get done and goals change, you shouldn’t feel anxious about fading KPIs associated with objectives that have been accomplished.

But KPIs can also turn into energy-drainers that yield zero results. Here’s a rundown of common pitfalls to avoid.

Too many KPIs

Remember, they’re called “key” for a reason.

Having too many KPIs dilutes their impact and makes it difficult to zero in on what counts.

Instead, pick a handful of essential KPIs aligned with your current objectives. Don’t hesitate to let go of or replace them with more fitting KPIs as goals evolve.

Consider this: an early-stage startup can go a long way by tracking just these four KPIs:

Finance: Revenue growth
Marketing: Leads per week
Product: Free-to-paid conversion rate (or Close rate, if your business is sales-driven)
Customer service: Churn rate

Not setting clear targets

Sometimes you are measuring something for the first time and you don’t even know what to expect. That’s fine.

But after a while, establish a specific target or benchmark to provide a clear indication of whether you’re making progress or falling behind.

Put this into action with a KPI tracker that color-codes values in red or green (check out conditional formatting rules if you’re using a spreadsheet). Thanks for reading the Operately blog! Subscribe to receive new posts on building and scaling businesses.

Not reviewing KPIs regularly

Regular reviews and upkeep are the bedrock of any system, whether it’s a personal to-do list or an entire business:

Review and regularly update KPIs within a standard cadence of company’s habits.
Default to reviewing KPIs once a month in each group or department and adjust according to circumstances. Make this a standard agenda item, accompanied by progress updates on goals.

Overlooking the human element

Numbers say a lot but can never tell the whole story. Qualitative metrics can offer valuable insights into customer happiness and team morale.

Take NPS, for example. It has its fair share of critics, but I believe its true worth lies in customers’ qualitative feedback and conversations that follow from the process.

Misaligned team

Founders should take the time to ensure that the team understands the KPIs and their importance. Engage them in the KPI-setting process and encourage their feedback to foster a sense of ownership and accountability.

Keep in mind, this takes time. Founders may need to reiterate the point of KPIs at staff meetings for months until everyone’s on the same page about the company’s performance and goals.

FAQ

How to spot irrelevant KPIs:

Your team can’t influence them.
A major swing in either direction doesn’t make you jump out of your chair.
It’s an absolute number that always goes up—prefer a ratio or percentage instead.
There’s no initiative in the company to affect them.

How to handle less critical metrics? Track them, just don’t call them or treat them as KPIs. For instance, conversion rate of a new landing page is a metric that a marketing team will closely track and act on after launch, but is not something that the Head of Marketing will discuss at a monthly leadership meeting.

When to add a new metric? Add a metric to your KPI tracker only when you believe that you need to know that metric in order to make an informed decision.

Selecting and maintaining KPIs that are relevant, actionable, and aligned with your company’s goals can be a tall order. We’re tackling this challenge head-on with Operately. Stay tuned as we share more on this subject in the near future.

Don't Get Bogged Down in Too Much Unfinished Business

Tue, 02 May 2023 11:45:07 GMT

Small startups lose their edge over incumbents when they fail to manage projects effectively as they grow. This is a common issue for startups and it severely hampers their momentum.

One of the key advantages startups have over established companies is speed. They’re nimble, unburdened by bureaucracy, and able to make decisions quickly. This agility allows them to innovate and run circles around incumbents, who are primarily focused on maintaining their market position.

But velocity can be fleeting. After product-market fit, the founding team expands, and new groups form. This growth brings with it a communication and alignment tax—a demand for more structure and discipline. If this tax goes unpaid, operational debt builds up and execution suffers.

It’s a problem that’s difficult to see from within. Projects drag on, excitement wanes, and resources are spread too thin. The result? A slew of half-finished projects that never reach completion—or cancellation.

Looking back on the early years at Semaphore, I wish we’d hired our first project manager sooner. It took us eight(!) years. We believed that teams should self-manage, but we didn’t consider the value in having someone solely focused on managing projects.

Makers aren’t always inclined to ask the same questions week after week, take stock, document progress, or address performance issues. That’s where operators come in. They thrive in these roles, and their presence actually empowers a team of makers by freeing them from the burden of project management.

That’s why we’re developing Operately—to help startups implement just the right amount of cross-functional process so they can focus their resources on building truly unique and valuable products. As we work on Operately, we’ll continue to share ideas and techniques right here on our blog. Stay tuned!

Why “Ignore the Competition” Is Terrible Advice

Tue, 25 Apr 2023 11:45:12 GMT

Here’s the latest idea that’s gone too far on the startup scene: ignore your competition. Sure, obsessing over every move your rivals make leaves you incapable of having your own goals and ideas, but completely disregarding them is a rookie mistake.

Customers aren’t clueless. They’re always comparing products and services. If you’re out of the loop, you’re leaving your business vulnerable. And it’s not just about having a good answer to when a prospect asks “How do you compare against X?” during a call. Your job is to know what they’re seeing, so that you can decide what to do to make your offer compelling.

Instead of dismissing your competitors, learn from them. Studying what they do can provide valuable insights into customer needs and marketing strategies. As the popular book says, “steal like an artist”: adapt and improve upon their good ideas without becoming a copycat.

Competitive insights is valuable information for customer-centric companies. Use that information as you focus your energy on doing what helps you differentiate and provide better value.

Lessons Learned From Fixed and Usage-Based SaaS Pricing Models

Wed, 19 Apr 2023 12:00:00 GMT

Startups are often in search of that perfect pricing model for their products. The one that would be predictable, simple, and provide steady growth. But the road to finding it can be a bumpy one due to unforeseen second-order effects. In this post, I’ll share some of the lessons we’ve learned from implementing two different pricing models at Semaphore.

When Simple Pricing Meets Complex Reality

We initially went for a fixed-cost tiered pricing model: $29/mo, $99/mo, etc. The rationale was straightforward: our customers could predict their costs, and we could predict our revenue. We bet on simple. “When you need more, you just upgrade,” we said. Unfortunately, the complex reality of B2B threw a wrench in our plans.

The second-order effect we encountered was limited growth from existing customers. The disconnect between those who needed more resources and those who made purchasing decisions created a situation where upgrading became a major (and rare) event. The people who used the product would often invest significant engineering effort to bypass plan limits, leading to frustration and a not-so-happy relationship with our product.

Additionally, both our customers and ourselves recognized that teams typically require minimal to moderate capacity for the majority of the time, with a substantial increase during periods of peak activity.

Consumption-Based Pricing: A Cloudy Solution

Having learned from our first experience and driven to address the need for variable capacity, we moved to a consumption-based, pay-as-you-go model. It seemed like a great solution – customers who used more would pay more without any extra effort from us.

Since then, Semaphore revenue grew multifold. We’re still happy with this strategy. But again, there were unforeseen second-order effects to deal with, such as:

Complexity spillovers: Everything related to usage and billing became more complex and slow. For example, it became difficult to get clear insights into how customers are trialing and utilizing the product over time, resulting in a painfully long feedback loop for improving the onboarding experience.
Difficulty in upselling 2.0: Customers grew accustomed to having the entire product available all the time. But continuing to ship new, advanced features requires funding. There’s little choice but to figure out how to segment and sell additional plans and services. This requires a strong value proposition to avoid alienating the loyal customer base.

How to minimize risks

First-time founders often lack the experience to anticipate all relevant second-order effects when making major strategic decisions. Some lessons have to be learned through own experience, but there are ways to minimize the risks:

Resist the urge to move fast: Making decisions that will have long-lasting effects requires careful consideration, not just quick action.
Be vigilant: If a solution sounds perfect, you’re missing a catch.
Do thorough due diligence: Seek examples of different outcomes and discuss the pros and cons in painful detail. This can help you identify potential pitfalls and hidden effects in your own decisions. Start by interviewing ChatGPT and asking who in your network might have gone through a similar journey.

Decisions about pricing models may seem deceptively simple, but they can have far-reaching effects on your business. By understanding the second-order effects, doing your homework, and being open to change, you can find the solution that best fits your company and leads to sustainable growth.

Solve The Biggest Bottleneck First

Tue, 11 Apr 2023 12:00:00 GMT

It’s a familiar scene: a passionate first-time founder, eyes wide with ambition, lays out an intricate plan to change the world. Ask them how they’re really doing, and they recount an exhaustive list of problems they need to solve and a sense of urgency to fix everything right away.

It’s natural for a founder to be overwhelmed by the magnitude of what they’re trying to accomplish. But there’s good news, and it lies in a simple, yet profound truth that I’ve learned over the years of running Semaphore.

Here it is: at any given moment, your business needs you to solve exactly ONE constraint that’s holding it back. Not ten or twenty, just one. Everything else, no matter how pressing, can take a back seat.

Think of your business like a chain, with each link representing a different challenge. The weakest link – that’s the one you need to repair first. The others can wait. Because no matter how shiny, strong, and polished the rest of the chain is, it’s only as strong as that weakest link.

Once you have tackled the biggest bottleneck, you’ll find that progress begins to unfold more smoothly. You can then turn your attention to the next most significant constraint, and so on.

Now, I’m not saying that you should ignore all other problems – that’s just unrealistic. But prioritizing your efforts and tackling one constraint at a time allows you to make meaningful progress without stretching yourself too thin.

Developers tackle software bottlenecks by using profiling tools to identify slow application screens. They sort these by load time, creating a prioritized to-do list with the most critical issue at the top.

Similarly, product managers sort feature requests by combining factors like number of requests in different customer segments, effort required, and revenue impact.

The good news is that the Pareto Principle applies. In most cases, you’ll find that addressing the top 20% of constraints will render the remaining issues significantly less urgent or even irrelevant.

So, next time you’re feeling overwhelmed by the sheer number of growth challenges on your plate, remember to slow down, identify that single biggest constraint, and work on it until it’s resolved. It’s a game-changer, and it just might be the key to unlocking the full potential of your business.

Unlocking Thoughtful Decisions: Rigorous vs. Lazy Thinking

Tue, 04 Apr 2023 12:00:00 GMT

Sally is a new manager, overseeing a team that’s brimming with enthusiasm and ideas. But, more often than not, their suggestions seem impulsive and lack consideration for the bigger picture, so Sally keeps rejecting them. After a while, the team gets the impression that the company is not so welcoming to new ideas as it claimed to be. The overall progress is stagnating. It’s time to introduce the concept of rigorous thinking.

Rigorous thinking involves taking the time to ponder and examine a decision systematically, asking critical questions every step of the way.

Its counterpart, lazy thinking, is a haphazard approach – tossing ideas around to see if any of them catch, without considering the implications or how they might work in practice. Ultimately, lazy thinking is a surefire path to failure.

Practicing rigorous thinking means being prepared to advocate for and defend your ideas, understanding the upsides, downsides, trade-offs, and potential second-order effects.

How not to pitch an idea

“Hey boss, why don’t we try [insert random tactic that competitors are doing]?”

👎 Result: You’re left feeling discouraged by your manager’s negativity, while your manager is burdened by decision fatigue from having to evaluate every half-baked idea.

In such scenarios, managers are constantly forced to correct their team’s flawed logic and find diplomatic ways to say “this isn’t a good idea” without crushing their spirit.

Let’s see a better approach.

So good they can’t ignore you

“Hey boss, I propose we ____. The benefits include ____, while the downside is ____. However, we can mitigate risk with a small-scale experiment by doing ____.”

👍 Result: You feel empowered and your role becomes more influential, while your manager experiences less decision fatigue.

Here, individual contributors and managers collaborate to test assumptions, fostering a culture of idea validation rather than binary yes or no decisions.

The manager’s role

As a manager, your role is to ask insightful questions, encouraging your team to think deeply about their ideas instead of rushing to provide answers. Some questions to kick off the conversation could be:

“That’s a great start. What would be the next steps?”
“What does success look like?”
“What’s the challenging part?”
“What’s your take? I’d like to hear your perspective.”

Adopting a curious and supportive attitude demonstrates that you’re right there alongside your team, believing in their capabilities. By posing difficult questions, you empower them to find solutions, refine their ideas, and take ownership of their work.

Build a Company Culture That Gets Things Done

Tue, 28 Mar 2023 12:00:00 GMT

As startups grow, they face an onslaught of information, problems, and uncertainty. The kind of company culture they build makes a difference in how they navigate these challenges.

Sociologist Ron Westrum’s model of organizational culture, developed in his 1984 book Complex Organizations offers some guidance.

Westrum identifies three types of cultures: Power-Oriented, Rule-Oriented, and Performance-Oriented. Startups, especially when they’re small, naturally lean towards a performance-oriented culture. This type of culture focuses on getting things done and promotes collaboration, learning, and transparency.

As a founder, it’s crucial to remember the importance of maintaining this performance-oriented culture as your company scales.

But don’t mistake the means for the ends.

Collaboration, communication, and learning are valuable because they help your team achieve results. They are not the goal in themselves. Transparency supports trust and removes barriers that might impede progress. But sharing information without context and in an untimely manner is useless at best and dangerous at worst.

A great recent example of this comes from Ryan Denehy’s story about a co-founder who prioritized getting things done above all else. They didn’t bother with management books or fancy frameworks. Instead, they succeeded through sheer drive and determination. While it’s unrealistic to expect employees to match a founder’s level of urgency, performance should always be non-negotiable.

Embrace failure as a learning opportunity, but also recognize that perpetual learning without action is pointless. Some decision-making will need to be centralized, and innovation isn’t desirable in areas like recruiting and project management. Instead, focus on fostering a company culture that prioritizes results and adapts to the ever-changing landscape of the startup world.

Behavior-Driven Development

Thu, 31 Mar 2022 09:42:00 GMT

Behavior-Driven Development (BDD) is about minimizing the feedback loop. It is a logical step forward in the evolution of software development practices. This article explains the concept and its origins.

Waterfall

If you are a software developer or an engineering manager, you are probably familiar with the Waterfall Model, shown in the following diagram:

What was later named “Waterfall” was first formally described by Winston Royce in his 1970 paper Managing the development of large software systems (ACM). I recommend reading the whole paper to understand the idea. Most people learn about it secondhand and assume that this process was presented as the ultimate solution at the time. However, Royce recognized that having a testing phase at the end of the development process was a major problem:

I believe in this concept, but the implementation described above is risky and invites failure. […] The testing phase, which occurs at the end of the development cycle, is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed. […] The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect, the development process has returned to the origin, and one can expect up to a 100-percent overrun in schedule and/or costs.

This model is still used to develop software in many companies worldwide for various reasons. Waterfall implies flow, but in practice, there are always feedback loops between phases. All major improvements to the model over time have been made by minimizing the feedback loops and making them as predictable as possible.

For example, if we write a program, we want to know how long it will take us to find out if it works. On the other hand, if we design a part of a system, we want to learn if it is actually programmable and verifiable, and at what cost.

So, when we look at a feedback loop, we look for methods we can use to minimize it. At first, our goal is to remove obviously wasteful work. Later, we realize that we can optimize and do things faster and better than we could have ever imagined back when we were doing things the old way.

The first optimization: Test-First Programming

The first optimization emerged from the Coding and Testing phases. In the traditional quality assurance-based (QA-based) development model, a programmer wrote code and submitted it to the QA team. It took a day, a few, or weeks to get a report on whether the code worked and if the rest of the program worked as well. There were often bugs, so we would have to go back to programming and fix any issues, even though we thought the work was finished.

To cut down the feedback loop, developers started coding and verifying simultaneously, i.e. writing some code, and then writing some tests for it. Tests produced an excellent side effect–the automated test suite–which we can run at any time to verify every part of the system for which we have written a test. From this emerged the desire to have a test suite that covers the entire system in order to be able to work as safely as possible.

The feedback loop of coding followed by testing still takes some time, so the next step was to invert it: writing tests before writing a single line of code. In this way, the feedback loop shrinks, and it doesn’t take long to realize that developers are writing only the code needed in order to pass the existing tests.

This is called Test-First Programming. When working test-first, tests are used to help “fill in” the implementation correctly. This reduces the number of bugs, increases programmer productivity, and positively affects the tempo of the whole team.

The problem with writing tests later

The Waterfall Model puts tests at the end of the development cycle, which, as we’ve seen, is very inefficient. But that is not the only problem. Code written in isolation is difficult to test because developers are focused on solving a problem rather than writing testable code.

Later, when we begin writing tests, the mindset is yet again in the wrong place: we’re concentrating on testing that the code we wrote is indeed the code we wrote, instead of testing the code’s behavior.

The result of this is that code and tests become highly coupled. This results in even more problems when we refactor. Changing tightly-coupled code makes the related test obsolete — we have to rewrite a bunch of tests every time we change something in the code. The feedback loop suffers and gets longer as we continually deal with the consequences of past decisions.

The solution is to have a test-first mindset. When tests come first, no pre-existing code influences how we write them and we can write tests that check what the code is actually supposed to do. We also get another benefit: the resulting code is mode modular and testable, making tests smaller and more readable.

Test-Driven Development

Once we have a continuous loop of testing and coding, we’re still doing all our program design upfront. We’re using Test-First Programming to make sure that our code works, but there’s a feedback loop where we may find out (disturbingly late) that a design is difficult to test, impossible to code, performs poorly, or just doesn’t fit together with the rest of the system as we are trying to implement it.

To minimize this loop, we apply the same technique. We invert it again by doing Test-First Programming before we start designing. Or rather, we do the Testing, Coding, and Program Design steps all at the same time. A test influences code, which in turn influences design, which influences our next test.

It quickly becomes clear that this cycle organically drives design ideas, and we start to implement only the parts of the design that we need in a way that can easily evolve. Design now includes a substantial refactoring step, which gives us the confidence to under-design instead of over-engineer. That is, we end up with just enough design and appropriate code which meets our current requirements.

This is Test-Driven Development (TDD). It combines Test-First Programming with design thinking by continuously applying refactoring principles and patterns. The positive side-effects are now amplified: we have not only reduced the number of bugs, but we are also not writing any code that doesn’t help us implement a feature. This further increases the team’s productivity by helping avoid design mistakes which are more costly to fix down the road.

TDD is the crystallization of an old idea that says that design and testing should be interlaced in a continuous iteration loop. From the 1968 First NATO Software Engineering Conference:

A software system can best be designed if the testing is interlaced with the designing, instead of being used after the design. [..] A simulation which matches the requirements contains the control which organizes the design of the system. [..] Through successive repetitions of this process of interlaced testing and design, the model ultimately becomes the software system itself.

Making the next step with Behavior-Driven Development

Now that we are designing, coding, and testing in one loop, it’s time to revisit at the Analysis step. By analysis, I assume “understanding what we need to build”. Again, we’re interested in optimizing the loop in the name of efficiency. In practice, this would involve preparing a list of a dozen or more features and passing it on to developers, who would complete them all before moving forward. This way, we often end up implementing features that we don’t need. Sometimes we also discover new features we didn’t expect or discover something new about the features that we know we need.

We can apply the same technique and bring Analysis into our loop. Now we test-drive a feature before we try to implement another. It is worth illustrating that the duration of such cycles for a developer is measured in hours, sometimes even minutes, not in days or weeks.

After applying this technique consistently for a while, we notice that we tend to break down all features in the smallest units and consistently deliver them one by one. Our understanding of how features affect one another improves and we find ourselves able to respond to changes faster. This allows us to identify and discard unwanted features quickly and prioritize important features.

By test-driving our analysis, we better understand the system’s behavior and how to appropriately design and implement it. At the same time, all that we are doing from day one is producing a test suite, which keeps our entire system constantly verifiable.

This is called Behavior-Driven Development (BDD). It saves time for both the stakeholders (business owners) and the development team. By asking questions early, developers help both themselves and the stakeholders gain a deep understanding of what they are building. Stakeholders get results at a predictable pace, and since the features are worked on in small chunks, estimates can be done more accurately and new features can be planned and prioritized accordingly.

The BDD Lifecycle

The BDD lifecycle begins with the discovery phase. The team discusses and refines user stories in this phase by clarifying acceptance criteria. Stakeholders, product owners, and developers collaborate to create illustrative examples that describe a feature’s intended behavior, usually in Gherkin.

Examples, also known as scenarios, serve as living documentation of the system’s requirements. Once the examples are defined, the team moves on to writing and automating them. This involves associating the Gherkin steps with test code (step definitions) that can be executed against the product. By converting examples into automated tests, everyone gains immediate visibility into whether the system behaves as expected.

Next comes the implementation and continuous feedback loop. Developers write the application code necessary to make the failing BDD tests pass. After the new or modified features are implemented, the automated BDD scenarios run to validate correctness.

If tests fail, the team refines the code or clarifies the requirements, ensuring alignment with the expected behavior. As development progresses, the BDD scenarios remain a source of truth, continuously tested to confirm that the software still meets its defined requirements.

Over time, as features evolve, these scenarios can be updated or added to, maintaining clear documentation and preventing regressions. This cycle of discovery, specification, and validation drives consistent collaboration and results in a product that closely matches the user’s and stakeholders’ desired outcomes.

The Three Amigos

Within the context of BDD, the “three amigos” (three friends) is a practice that involves three key roles:

the business or product owner: representing requirements and business value
the developer: responsible for implementing the solution)
the tester or quality assurance specialist: (ensuring quality and correct functionality).

This trio meets to discuss, refine, and confirm the acceptance criteria of a user story or feature. By bringing these perspectives together early and regularly, the team aims to arrive at a clear, shared understanding of what needs to be built and why it matters.

The primary benefit of this approach is its emphasis on preventing misunderstandings and defects before coding begins. Teams reduce costly rework by catching ambiguities and potential pitfalls early. The result is not only more robust software but also a more cohesive team that consistently delivers features that meet stakeholders’ expectations.

BDD in Agile

Behavior-Driven Development (BDD) aligns with the values of Agile. After all, Agile is an iterative, incremental approach to software development that emphasizes collaboration, customer feedback, and rapid delivery of valuable software.

BDD fits neatly into Agile sprints by guiding how user stories are discovered, refined, and validated. During backlog refinement or sprint planning sessions, teams may use the “three amigos” model to uncover hidden assumptions and clarify expected behavior.

The automated tests derived from BDD scenarios provide continuous feedback throughout the Agile development cycle. Each time code is integrated, these tests run automatically, confirming whether the system continues to behave according to the established specifications.

BDD: what, not how

To practice BDD, we must first write a specification that describes the behavior of the system. We do this by asking questions such as: “how should the system respond when a user does X?” or “does X follow the user’s expectations?” This puts the focus on the problem we are working on without getting mired in implementation details. We think about what the system does, rather than how it does it.

We really don’t care how the system looks at this stage, if it has round or square buttons, or even what devices it runs on. Thus, we decouple the problem we want to solve from the technical details. The specification presents the case in a clear and concise form, as shown in the example below:

Scenario: Blog Search
    Given I visit the blog page
    When I search for “BDD”
    Then I get posts related to BDD

The example is written in Gherkin, a language used by frameworks like Cucumber to define test cases in the BDD-style. Under the hood, the scenario drives a series of automated tests that ensure that the specification is followed.

BDD allows us to interweave tests, analysis, and design, so they feed into each other, creating a feedback loop that guides development and enables us to track our progress.

BDD is not UI testing

Inevitably, as an idea becomes popular, some of the important nuances get lost. One of the biggest misconceptions about BDD is that it is a synonym for UI testing.

Take this example:

Scenario: user logs in to application
    Given authorized user “John”
    When I enter “John” in the username field
    And I enter “sekret1” in the password field
    And I click the login button
    Then the homepage should open

This Gherkin scenario has several problems. For one thing, it’s too focused on the UI, making it brittle and too tied to the implementation — if any of the field names change, the test breaks.

Another problem is that the logic of the test is limiting because it forces developers to follow a recipe. They need to implement classic username/password authentication even when better authentication mechanisms such as fingerprints or face recognition are available. Scenarios that are too detailed make the lives of developers harder because they hide the problem to solve under a long list of steps to follow.

A better version of the scenario is shown below. In the example, we establish the specification (what the system should do) without going into details (how it should do it). Developers are free from the shackles of following a predetermined solution and are free to innovate.

Scenario: user logs into application
    Given authorized user “John”
    When “John” logs in correctly
    Then “John” can access their items

BDD works at every level

Another big misconception about BDD is that it does not work for integration and unit testing. The reason for this is that Gherkin introduced an additional layer of abstraction which has led people to believe that BDD is synonymous with end-to-end testing. While BDD is usually paired with acceptance tests, nothing is preventing us from using it at any level of the testing pyramid.

To see how that BDD can be used at every level, here’s a unit test written in Gherkin:

Feature: Sum a Pair
  It sums a pair of numbers

  Scenario: adding numbers
    Given a 1
    When add a 2
    Then the sum is 3

The example is so simple that you could even throw away Gherkin and write the test directly in any framework supporting behavior-driven DSLs such as Jest, as shown here:

import { sum } from './maths';

test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toBe(3);
});

Thinking that BDD only works on high-level tests leads to the inverted test pyramid (also known as the ice-cream cone). While in some cases, the inverted pyramid is an acceptable solution, more often, it only leads to hard-to-maintain, complex test suites that run very slowly.

Going even faster

If you look at the remaining phases listed in the Waterfall diagram, which all need to happen regardless of the methodology, you may wonder if the same feedback loop minimization can be applied to them. The answer is, of course, yes. However, such loops are of a scope that is broader than just design and development, and they involve people working across very different fields, which means that they are out of the scope of this article. However, I will mention them briefly.

Lean Startup would be the closest concept that brings together requirement gathering, feature development, and marketing as a way to close the loop on learning what a startup needs to build. Of course, the process goes somewhat differently in enterprises, although they are learning to apply lean startup principles in many projects as well.

Merging BDD with deployment and operations brings us to the broad concept of continuous delivery. The most important processes are continuous integration (CI) and continuous deployment, which you can easily configure for any project on Semaphore.

Conclusion

Behavior-Driven Development evolved from optimization of various phases in the software development process. We can produce better software by analyzing, testing, coding, and designing our system in one short feedback loop, which helps us avoid mistakes and wasteful work.

It is a common misconception that TDD is about testing and that BDD is just another way of approaching software testing, since it has its origins in TDD. This is not the case, although tests are a nice byproduct. It is a holistic approach to software development, derived from one simple idea: the desire to optimize the feedback loops in our work.

BDD is a powerful tool for effective software development, one that takes time to learn and apply well. It helps you write tests that add value, have meaning, and help you in your designs. As long as you stay focused on the ‘what’ and avoid the ‘how’, you’ll be fine.

New Book: CI/CD with Docker and Kubernetes

Wed, 02 Dec 2020 14:29:00 GMT

A gift for you: a free, 90-page ebook to learn CI/CD with Docker and Kubernetes. The book is open source, and you can download it today.

Why we wrote this book

Containers have become the standard executable format for cloud applications. But, changing how you build, test, and deploy code is a big deal for any engineering team.

Semaphore has been around since 2012, and our Customer Success team has worked with hundreds of organizations as they moved to a container-based architecture. Many more are yet to migrate.

We’ve seen it time and time again: adopting containers takes time.

There are many moving parts, and it’s not clear how to make the timeless practice of Continuous Integration and Delivery (CI/CD) more productive, not less.

Eventually, it does get easier to onboard new team members, run tests reliably, be cloud-agnostic, and build new services. But to get there, you have to build internal know-how from several books and dozens of blog posts.

We wrote this book to help you build reliable and fast CI/CD pipelines for containers — in days, not weeks or months.

Who is it for

We believe the book will be useful to:

Senior developers who are beginning their journey into microservices or cloud-native architecture, or need to migrate an existing system to Kubernetes.
CTOs and Directors of Engineering who need a clear vision of what a reliable, high-velocity CI/CD pipeline looks like and what it takes to build one.

Download it today

I want to thank the book co-authors:

Jerome Petazzoni, a leading instructor on containers, creator of container.training, and a former Docker core engineer.
Tomas Fernandez, whom you might already know from Semaphore blog as an author of highly valuable CI/CD tutorials.

We combined our knowledge in CI/CD and containers to bring you a practical guide to shipping high-quality cloud apps rapidly and consistently.

Get a copy today.

CI/CD: Continuous Integration & Delivery Explained

Tue, 24 Sep 2019 09:14:00 GMT

CI/CD enables the best tech companies to improve their products many times per day. Here’s what you need to know to do the same.

Summary

CI/CD stands for the combined practices of Continuous Integration and Continuous Delivery. This approach automates the software release process, ensuring new changes are frequently tested, integrated, and delivered.

Key terms

Continuous Integration (CI): Developers merge code changes into a shared repository daily. Automated tests run to quickly catch and fix errors.

Continuous Delivery (CD): Pipelines test and deliver code changes but stop for manual approval before pushing them to production.

Continuous Deployment: Extends CD by automatically releasing updates into the production environment.

What is CI/CD?

CI/CD is a way of developing software in which you’re able to release updates at any time in a sustainable way. When changing code is routine, development cycles are more frequent, meaningful and faster.

Continuous Integration is a prerequisite for CI/CD, and requires:

Developers to merge their changes to the main code branch many times per day.
Each code merge to trigger an automated code build and test sequence. Developers ideally receive results in less than 10 minutes, so that they can stay focused on their work.

The job of Continuous Integration is to produce an artifact that can be deployed. The role of automated tests in CI is to verify that the artifact for the given version of code is safe to be deployed.

In the practice of Continuous Delivery, code changes are also continuously deployed, although the deployments are triggered manually. If the entire process of moving code from source repository to production is fully automated, the process is called Continuous Deployment.

A litmus test for doing CI/CD

If any developer in your team can stop what they’re doing right now and ship the current development version of code to production in 20 minutes or less without anyone stressing about what could happen — congratulations, you’re doing CI/CD!

CI/CD principles

Continuous Delivery practices take CI further by describing principles for successful production deployments:

Architect the system in a way that supports iterative releases. Avoid tight coupling between components. Implement metrics that help detect issues in real-time.
Practice test-driven development to always keep the code in a deployable state. Maintain a comprehensive and healthy automated test suite. Build in monitoring, logging, and fault-tolerance by design.
Work in small iterations. For example, if you develop in feature branches, they should live no longer than a day. When you need more time to develop new features, use feature flags.
Developers can push the code into production-like staging environments. This ensures that the new version of the software will work when it gets in the hands of users.
Anyone can deploy any version of the software to any environment on demand, at a push of a button. If you need to consult a wiki on how to deploy, it’s game over.
If you build it, you run it. Autonomous engineering teams should be responsible for the quality and stability of the software they build. This breaks down the silos between traditional developers and operations groups, as they work together to achieve high-level goals.

To make CI/CD a reality, you need to automate everything that you can in the software delivery process and run it in a CI/CD pipeline.

A simple example of a CI/CD pipeline

Here’s a simple example of fully automated continuous deployment pipeline:

CI/CD shouldn’t be more complicated than necessary

In this example, each change on the main Git branch performs the following steps on Semaphore:

Build code and web assets, while reusing dependency cache.
Run an automated test suite. This is a JavaScript / Node.js application, so the tests are written with Jest.
If the tests pass, a Deploy block updates the production code that runs in the cloud.

An example of CI/CD with manual steps

Here’s a CI/CD workflow for Kubernetes which includes more steps:

Multi-stage CI/CD with Docker and Kubernetes, with manual steps to strategically deploy to staging or production environments (click to view full size)

In this example, each change automatically performs these steps:

Build application from source code and dependencies.
Run an automated test suite.
If tests pass, automatically build a Docker container image and push it to a private registry.

At the end of the Docker build pipeline, we have a working artifact, a container image. The developer, or more formally, release manager, can decide to manually trigger:

Deployment to staging or production, which may include smoke tests to verify no major problems have been introduced.
Tagging of the container image as an artifact that was introduced to production, to enable audits and rollbacks.

What are the benefits of CI/CD?

CI/CD is much more than the automation of tasks to avoid human error. It lets us get new solutions into the hands of users as quickly, efficiently and cheaply as possible.

As Dave Farley, co-author of the Continuous Delivery book and a software engineering expert points out in an episode of Semaphore Uncut podcast:

The trillion-dollar mistake is miscategorizing software development as a production problem, in the sense of being able to scale it up in order to be able to produce things more reliably.

Software development is always an exercise in learning, discovery and experimentation. […] If you read the Accelerate book, State of DevOps report, the numbers tell us that organizations that practice continuous delivery produce higher-quality software more quickly. The people working on it enjoy it more and the organizations that practice it make more money.

If your company practices CI/CD, it will have a competitive advantage which enables you to:

Deliver software with less risk. CI/CD pipelines standardize release processes across projects. By testing every change in source code, we reduce the chances of introducing bugs.

Release new features more frequently. A CI/CD pipeline can visualize your entire path from commit to production in a single screen. You can navigate across stages, spot inefficiencies, and optimize your process. By removing the roadblocks to productivity, you enable your company to succeed.

Deliver the product that users need. Delivering updates often leads to more user feedback. You can take advantage of that by A/B testing features, or testing early versions of products with real customers. This way you avoid investing too much in features that your customers don’t need, and focus on those that matter.

Improve developer productivity. Engineering teams that don’t practice CI/CD often work under stress. There are constant fires for bad deploys and hard-to-fix outages. Developers write a lot of code that never gets used. Long-lived feature branches are too big to get proper peer review, so code degrades in quality. On the other hand, CI/CD guides product management to optimize for user impact. Developers deploy code while it’s fresh in their minds. The result is a happy engineering team.

The benefits of CI/CD compound: fast deploy cycles lead to less risky and more frequent updates, which lead to faster learning and more user feedback, which leads to happy developers building better products.

Addressing Security Risks in Pipelines

While CI/CD accelerates delivery, it also introduces its own set of security risks. Without proper safeguards, CI/CD pipelines can become an attack surface for malicious actors, risking the integrity of code, the confidentiality of sensitive data, and the stability of the production environment.

Common security vulnerabilities in CI/CD pipelines include:

Credential and Secret Exposure: Hardcoded credentials, tokens, or keys in pipelines or configuration files can be compromised if not stored and managed securely.
Untrusted Dependencies: Vulnerable libraries and third-party packages can introduce backdoors or exploited code paths into your software.
Insufficient Access Controls: Overly broad permissions for developers, service accounts, or automation tools increase the risk of unauthorized actions.
Incomplete Logging and Auditing: Limited visibility into pipeline activities makes it challenging to detect and investigate security incidents promptly.

To mitigate these risks, consider implementing the following strategies:

Secure secrets management: Use vaults or encrypted secret management systems to ensure that tokens, keys, and passwords are never exposed in plaintext.
Dependency scanning and supply chain security: Integrate automated security scanning tools into your pipeline to identify vulnerable dependencies early. Adopt a “trust but verify” approach with third-party components.
Least privilege access controls: Apply the principle of least privilege to all pipeline services and user accounts. Restrict access to only what is necessary for each role or process.
Robust logging and monitoring: Enable comprehensive logging and integrate alerts for abnormal pipeline activities. Monitoring pipeline logs and events helps detect intrusions and quickly respond to security incidents.
Continuous security testing: Incorporate static security testing and vulnerability scans into the CI/CD pipeline. Regular security tests ensure new code doesn’t introduce new attack vectors.

By integrating security considerations into every stage of the CI/CD process, you ensure that as you move faster, you’re not sacrificing the safety and integrity of your systems.

Continuous Integration in the Broader DevOps Framework

CI/CD is not a standalone practice; it’s a foundational element within the broader DevOps culture and methodology.CI/CD acts as a cornerstone, enabling faster feedback loops, smoother releases, and a more responsive development cycle.

Key ways CI/CD enhances DevOps initiatives include:

Encouraging collaboration: automated code integration, testing, and deployments removes bottlenecks and silos. Development, QA, and Ops teams work more closely, communicate more openly, and share accountability for delivering high-quality software.
Accelerating feedback and iteration: DevOps is all about short feedback loops. CI/CD pipelines ensure that code changes are integrated, tested, and deployed quickly, enabling teams to learn from each iteration, respond to user needs, and continuously refine their products.
Aligning with Infrastructure as Code (IaC) and automation: DevOps principles often promote Infrastructure as Code (IaC, think Terraform or AWS Cloudformation).
Improving quality and reliability: By embedding testing, security checks, and compliance validations directly into the pipeline, we get safer software.
Driving cultural change: DevOps goes beyond technical implementation. It encourages teams to adopt a culture of continuous learning, experimentation, and improvement. Over time, this mindset leads to higher efficiency, better product quality, and increased customer satisfaction.

CI/CD and DevOps are closely intertwined. While DevOps defines the cultural and organizational context, CI/CD provides the technical backbone to achieve DevOps objectives.

When is CI/CD not feasible?

“Continuous Delivery is great but won’t work with my project.” There are indeed some cases when CD may not be suitable:

Your customers don’t want continuous updates to their systems.
Regulations restrict how software can be updated. For example, continuously updating software used in aerospace, telecom, and medical industries is not an option.

Even in a CD-averse environment, teams can enjoy the benefits of easy deployment and keeping the system in a deployable state. They can also practice and reap the benefits of CI to the full extent.

A typical journey to CI/CD

If you’re used to long development cycles, you will need to change your mindset.

But if you are considering to adopt CI/CD, you’ve made an important first step!

Think of anything that gets in the way of practicing CI/CD as technical debt, and treat it as such. Estimate work that needs to be done. Set clear expectations with all stakeholders, and schedule a time to work on it.

There are two major milestones on your journey to CI/CD success. The first is to implement CI: fast, reliable and frequent integrations. The second is to implement CD: automate deployment, make it a push-button operation, and make it easy to test new code in production-like environments.

Getting to more frequent integrations

1/ The first thing is to start treating master as if you would deploy it at any time. Since you haven’t done this before, it will be hard. But it’s a start of doing something painful often until it stops being painful.

2/ Treat every test failure as a bug. This includes flaky tests. Extract the log, investigate and fix it. Regardless of what the test was for, you can’t close the issue until it’s fixed.

3/ Continuously improve your test suite. You may need to rewrite some code to make tests more reliable. Have all developers commit to writing tests as carefully as they write production code. After a while, you’ll reach a point where a failed test means there’s a real bug. Eventually, you’ll hit 100% green CI builds. That’s your first mountain. Celebrate it!

4/ Stop working in long-lived feature branches and start branching by abstraction using feature flags. This way everyone’s pushing to master and it’s easy to test features in development with the rest of the system. So you can detect issues right away instead of after merging months later.

5/ Optimize the feedback loop. Investigate slow tests and improve or rewrite them to be faster. Invest in a CI/CD tool that can separate your CI pipeline in stages and run tests in parallel. Semaphore customers often report drastic improvements such as reduced build time from an hour or even two to around ten minutes. This way you fully test every pull request before it’s merged.

Developing a fast and comprehensive automated test suite is a big upfront investment. However, the benefits are huge. Testers and QA can hunt for bugs and do much more useful checking. Developers are confident that they can push a change and CI will catch unforeseen issues. This confidence drives development velocity.

6/ The final step is to automate the deployment.

If your deployment process is manual, start by documenting it as a series of tasks. One by one, automate each task with a script (Bash, Makefile), or a configuration management tool such as Ansible, Chef or Puppet. Once you’re done, put all the tasks together in a CI/CD pipeline. You can start by setting up the transition in each stage to be triggered manually.

The details of release automation will depend a lot on your technology stack. The implementation when using Python on Docker will be different from using VMs to run Java application servers.

7/ If you’re deploying in the cloud, you can automate the creation of infrastructure by using “infrastructure as code” tools such as AWS Cloudformation and Terraform. If you’re deploying on-premises, you will likely not be able to create and destroy production-like environments on-demand. But you can still allocate them to be always available.

The final result is that after all this effort your team will start releasing monthly, then weekly, then daily. You can be Agile in the original sense of the term and deliver value incrementally, even for enterprise customers.

Rails Testing Handbook

Mon, 28 May 2018 12:06:00 GMT

At Rendered Text and Semaphore, we have a long history with Ruby on Rails. Checking the blog archive reminds me that we published first posts about working with Rails way back in 2009.

Many years and thousands of lines of code later, I’m proud to announce that we’ve published our first ebook. It’s called Rails Testing Handbook and it distills the essence of building Rails apps with tests, doing continuous integration and collaborating via pull requests. It’s the workflow which has served us well over the years, most notably in creating and scaling Semaphore without losing our minds.

Early reader feedback has been overwhelmingly positive. Some lead developers have shared that they got a few new tricks from it. Newcomers said that the book has helped them finally understand behavior-driven development (BDD).

So if you’re working with Rails, I encourage you to download the book — it’s free! — check it out, share it with your friends and send us any feedback you may have. Enjoy!

Framing Risk

Fri, 13 Nov 2015 12:12:00 GMT

Risk has a bad reputation. In both everyday and business context, when we say that something is “risky”, we usually mean “dangerous”, “maybe you shouldn’t be doing that”, “you don’t know what you’re doing”, and so on. I’d like to reset that thinking in myself.

In fact, “a risky business move” is more of an oxymoron. A company puts its resources, means of production and employees’ time and effort into use today for an expected return in the future. The future is unknown and unpredictable by definition. Therefore, risk is the basis of all economic activity. It’s just that we tend to forget that. But why?

Entrepreneurs out there hiring their first employee right now are, through that action, taking a risk. They don’t know what the outcome is going to be. Hiring that person might turn out to be a bad choice for all sorts of reasons, which is ultimately relatively easy to detect and fix. But the potential upside is much larger than the downside: together, the team may do things that are much larger than the sum of their individual efforts, and the hiring decision may be the moment a successful company is born. Entrepreneurs know this, as much as they know that taking that “risk” is the only way to move towards the goal they are trying to reach. No “risk”, no fulfillment.

We hear so often that big companies are “risk-averse”. This is exactly why a lot of them die. There is a quote that’s been passed around recently stating that “in 10 years, 40% of Fortune 500 companies will no longer exist.” Probably as much it is clear to an entrepreneur that making change (also known as “disruption” and “taking risk”) is necessary for success, to the people in a big company it is not. An entrepreneur’s antennae are tuned directly to signals of environment, while the people in an enterprise may not realize that the bigger the organization, the thicker are the layers of isolation from the outside world and its constant changes.

So I’d say the next time you find yourself pondering doing something risky, go ahead. Just make sure you’re picking the good kind of risk and prefer the action with a much bigger upside than downside.

Enjoying The Conversations

Thu, 10 Sep 2015 12:15:00 GMT

One of the less obvious things that I enjoy about running a product company are the diverse conversations that I get to have with our customers. It starts with a formal correspondence, initiated by a support request, customer interview or feedback discussion. And it often ends there. But sometimes after you’ve exchanged enough words or spent enough time talking, something clicks. People relax and tell you more about what it is that they do, what’s happening, what have they learned recently, or what’s bothering them. It’s just a common human trait to enjoy sharing and hearing stories, I suppose. Count me in.

I’ve noticed that it is an often repeated startup lesson to go out and talk to your current or potential users. People get a little obsessed with the ideas that they have and start ignoring the rest of the world. And I can relate to that. New ideas can be really exciting. But if you’re building a product and not talking to your users on a regular basis, it’s not only the product-market fit that you’re likely going to miss. It’s the timeless pleasure of establishing human contact too.

Become an exceptional programmer by learning to ship

Thu, 30 Jan 2014 12:20:00 GMT

It seems like there is always a shortage of good programmers; that is why programming is one of the highest paid professions today. Note that there is no shortage of programmers in general. Companies that need a high body count in order to be formally capable of negotiating certain kinds of deals do not have a problem finding people for that. But I am not interested in that kind of a programmer. I am interested in those who love their craft and want to create something meaningful.

Such programmers need to stand out of the crowd when starting their career in order to work in a company that is a satisfying place to be in. These companies typically say something like, we hire through open source. When we at Rendered Text post a job ad, we always mention as a requirement that the person applying should have shipped at least one project to end users, be it open source or otherwise. Because sometimes cause and effect get confused, I want to explore a bit what that means exactly.

What does it mean to ship?

Young programmers, due to lack of experience, sometimes don’t see that writing code is only a small part in producing software. To borrow from the terminology presented in the famous book The Mythical Man-Month by Frederick Brooks, at the beginning we have a program. It is standing alone on a drive somewhere, not used by anyone and not interacting with other components in some kind of a production system. This is the easiest thing to accomplish, because generally we work alone and can declare the code as “finished” at any point in time.

It is more expensive to produce a programming product. This now involves shipping your code to the public. Desktop apps need to be easily installable. If the program is a library you need to polish the API, hopefully refactor the code to some degree, provide some tests, write documentation, write and publish a public announcement. The moment all that is completed, you enter a phase of maintenance: you begin to communicate with your users, perhaps contributors as well. You also need to repeat the process of developing and releasing new versions.

So for example, publishing source code of an implementation of an audio compression algorithm on GitHub is not shipping. On the other hand, a library with multiple versions, downloaded and used by hundreds of developers is a clear indicator of shipping behavior.

Most software needs to interact with other parts of a system in order to become alive and useful. You need to make sure that your program is stable and works efficiently with other components. For example, a web application needs to use a database and system memory. Writing code for mining data from a remote API is a good first step; making sure that it’s a runnable application that can scale to many users over a long period of time is a different matter.

Useful software then is a product that works well in a production system, and shipping means developing useful software. Any software company’s best interest is to avoid hiring programmers who are not capable of persisting through all phases of development over a long period of time. People do drop off for whatever reason, some of who unfortunately have brilliant coding skills. There are more aspects to hiring of course, but demonstrating that you’ve been capable of shipping something in the past is definitely a requirement in any good software company.

First thoughts after mentoring startups

Tue, 22 Jan 2013 12:44:00 GMT

Last week I was invited by Vukašin Stojkov to mentor attendees of the Startup Academy in Belgrade. The Academy is a unique two-month long program for people and teams that are starting a new tech business. The course is made of talks by regional entrepreneurs who’ve “been there and done that”, hands on workshops (eg on front-end or Android development) and mentoring sessions. It is an extremely valuable initiative for the local tech startup scene in Serbia, which is in early development.

I was positively surprised that most teams have begun implementing their idea already. The stage of course varies: while some are still working on a viable strategy based on an interesting concept, some teams are not far from launching the full MVP.

While discussing their MVP, most of the people I have spoken with have had an a-ha moment, usually after a suggestion on how to do less. Wanting to do many things is a common fallacy for first-time product makers, caused by underestimation of the scope of the execution that is required.

Even properly shipping a tiny open source library - including readable code, clear and extensive documentation, announcements - and iterating more versions after that, while talking to its users, is a big undertaking, and a nice lesson that is easily accessible for programmers to take.

When working on the MVP, there is not just the question of limiting the scope of an idea, but also figuring out if the idea is right for you in the first place. If the product is such that it cannot grow organically, but requires a kind of a sales process that you don’t have yet a remote idea how to conduct effectively, succeeding is going to be tough.

Your product cannot be bigger than you; it can grow only as you do. The less experience your team has in shipping products, the smaller your first steps should be. If you are describing your business idea in eight sentences, pick a detail that fits in one sentence. Start with a micro app to test the waters, and yourself. Overnight success takes years. It’s a long and wonderful journey.

On learning non-default Rails stacks

Tue, 15 Jan 2013 12:47:00 GMT

Steve Klabnik has recently posted an argument that Rails now has de facto two default stacks: the “37signals” stack and “Prime” stack, and that this as a result:

puts a pretty heavy burden on a new developer: they have to learn the 37signals Stack and the delta. At the moment when someone is new, and barely knows anything, we throw them into making more choices.

When you declare that there is another “default” stack, not by officially shipping it but in a blog post, there’s going to be some disagreement. For example, our practice at Rendered Text is actually 75% aligned with “Prime”, except for Haml. Any non-default stack is just a one man’s or a company’s current opinion.

I think DHH partly addressed the concern for newcomers last year (won’t somebody please think of the noobs!). But I’ll try to explain from another angle why I think that newcomers are fine, as long as they learn wisely.

As a beginner, simply don’t pay attention to those who “tell [you] to do a bunch of configuration”. You should have one teacher and not question him or her too much.

If you’re learning on your own, the plan is simple:

find out who created a language or framework you’re interested in find out if (s)he has (co)written a book about it; if yes: read it and practice Darko and I created Plakatt using the default Rails stack that the Rails book has taught us. The core of the application was unchanged for a long time and was serving thousands of users for almost four years, until we took some time last year to upgrade it to all the latest bits.

It can be tempting to first investigate all tools people are talking about on Twitter or blogs, but that just steers you off course with a false sense of progress. Once you’ve mastered the basics and defaults, preparing your own omakase will come naturally.

If you’re entering a company with a plan to work with Rails, you have even less reason to listen to the chatter on the internets in the beginning. Your new colleagues have probably been working with Rails daily for some time, and will introduce you to a solid set of introductory material, tools and practices.

Prerequisites for continuous integration

Mon, 12 Nov 2012 12:00:00 GMT

The core points come from the Continuous Delivery book. My comments come from a background of working with a dynamic language and based on issues that we’ve seen some Semaphore users share.

Version control

You have to keep everything related to your project in version control. We recommend git. Not just code and tests, but database scripts and any scripts required for building and deployment.

If you’re writing a Rails app, you have a suite of default database scripts (rake db:*) that should serve you in almost all cases. Regardless of the framework, a common mistake is to inject additional behaviour to these scripts that introduce dependency on a certain order of code loading, assume that a database already exists or certain records are present. You can do that, but double check if your framework offers a predefined place for your extensions, and make sure that your scripts work on a machine that has just checked out the source code and does not have a database yet created.

Some things should not be in version control. Usually that means results of your scripts. For example, if you’re using Bundler to manage dependencies, put the bundle directory in .gitignore or equivalent. Your project may depend on some native extension which will not run if another environment, for example, has a different version of the native library.

Automated Build

The Ruby community understands this - most of us run our rake spec or similar test suites from the command line. However, some non-obvious pitfalls still exist and end up as a case of “it works on my machine but fails in CI”. Just like you need to manage and automate configuration of your application, you need to do so for your tests too. That means not having a suite that depends on undocumented or non-scripted OS settings, external services or unstable tools (most problems arise with browser automation).

A robust test suite does not contain tests with a random outcome or side-effects that affect other tests (eg a change of timezone). A test suite that finishes quickly encourages frequent commits and granular changes that keep the development team in control of what they do.

Of course, it is assumed that your test suite covers the functionality of the application to a degree that, when the build is green, gives everyone sufficient confidence that everything is working and that code can be deployed to production.

Agreement of the team

Continuous integration is a practice, not a tool.

Therefore it takes discipline. Everybody on the team must be on the same page about the development and deployment process. When the build is broken, the responsible developer must do everything to fix it, or revert the changes as soon as possible.

A typically broken process involves one developer working in small, iterative changes, pushing a dozen times a day and keeping the build green, while another works in silence for days, then drops a bomb in form of a single commit containing multiple changes and - if the build goes red - ignores the results.

Working continuous integration gives all stakeholders immediate feedback on the state of the application. When this practice is taking place, every project we’ve seen benefits from huge a productivity boost for developers and higher quality to the end users.

Good and bad organizations

Mon, 20 Aug 2012 12:00:00 GMT

Ben Horowitz in A Good Place to Work

In good organizations, people can focus on their work and have confidence that if they get their work done, good things will happen for both the company and them personally. […] In a poor organization, on the other hand, people spend much of their time fighting organizational boundaries, infighting and broken processes.

It may sound like a large company’s problem, but it can also be a small group’s problem. A group of people is not a team, and making a functional team is a lot of work.

Why we moved away from Heroku

Fri, 03 Aug 2012 12:00:00 GMT

Heroku recently tweeted that they’ve had 99.99% uptime during July. It looks like a fact, a number measured vigorously by a publicly traded company. I wish it was true.

Partly it is subjective - it was just one day earlier, the last day of June when they had a massive outage, concluding many, sometimes consecutive days of infrastructure issues. Statements would be made as if it was a nuclear reactor (“the supervisor process was itself crashing when this cooldown was reached”). But sometimes the understandable words in those statements would not match what we saw in reality. For example, last downtime caused by SSL Hostname issues lasted, by our internal reports and Pingdom, for almost ten hours, not two.

All this is terrifying when your bootstrapped product, serving developers all over the world 24/7, goes offline. With each new user, each happy customer it was getting more painful to be so powerless about your application’s multi-hour downtime. We made a distributed system for continuous integration where each build server is a stateless, replaceable unit, yet we had a single point of failure in the front-end, which receives post-commit hooks and schedules jobs.

An idea about migrating away would come to our minds, we’d discuss it for a while, but in the end come to a conclusion pretty much in tune with what Michael Bleigh recently posted. We had a very young product, we thought, and they’re going to work it out.

Until it was too much.

Other issues

Besides possible downtime which will make you feel miserable, there are a few other things to deal with which you might not expect. It turns out that you have to do maintainance and upgrades involving downtime even when your app is running on the most managed cloud infrastructure.

For example, some of those Heroku post-mortems hinted that if our app wasn’t still on the Bamboo stack, it wouldn’t suffer as much. It was also using a deprecated SSL add-on, and so we basically needed to kill our app, create a new one on Cedar, and setup everything from scratch. The problem is that, because Heroku has such a complex infrastructure, a thing like having an app under SSL on a root domain is more difficult to configure than on a VPS or dedicated server. What you don’t fully understand you fear a little, and postpone, especially if it means that you must also schedule some downtime.

Other things we were disappointed with are:

Not having influence over how the asset pipeline is done. The docs say that caching and proper last-modified timestamps will be in place, yet in practice they weren’t and we are still not sure why.
Slow deployment. The performance of “git push heroku” has deteriorated comparing to the early days, usually taking a couple of minutes.

New setup

Since over time we’ve learned quite a bit about automatic provisioning, a migration to a dedicated server was mostly straightforward. Once you grow tired of repeating the same setup steps on new servers, I highly recommend that you learn to use Chef. Even really understanding Capistrano, combined with a few Bash scripts, will take you a long way. Also, each new version of Ubuntu has more user-friendly sysadmin tools, one recent example being introductuon of ufw to hide the complexity of iptables.

Now, with a failover IP address and some straightforward backup, we’re able to recover in case of true emergency very quickly. It is a responsibility, but it is a level of responsibility and flexibility that we want.

Note that when you boot your app on a dedicated server, you might be impressed with its new speed, but your users in Brazil, India or Australia might have the opposite experience. That problem of accumulating latency is solved by putting your assets on a CDN. In the end what we got is 5x faster page load for 2.5x less money.

Is Heroku really that bad?

No. We’ve been happy users of Heroku for about three years and sent a lot of our and our clients’ money to them. Almost all apps that we’re working on are on it. We’re sure that they’re working on avoiding all these issues in the future (perhaps letting out a chaos monkey more would help?), and they remain our default choice for new apps. Heroku has advanced the state of the art in web development tremendously with such ease of deployment and add-on system.

If you lack the sysadmin experience, or don’t expect to be able to able to fully devote your team to infrastructure development and maintenance, in the long term you’re much better off in the cloud. However, with Semaphore, our own product that is already critical to many people, we feel better now when we’re fully in control.

The meaning of pain with mature tools

Fri, 11 May 2012 12:00:00 GMT

There are moments in work when I’m trying to do something with a mature, well-established tool (eg a framework or library), but it gets very difficult. I’m spending a lot of time doing silly things to make it work, but it’s not happening.

There’s ever only one reason, assuming I generally know how to use the tool: it is not made for that purpose. Recent examples include trying to serve XML files from the asset pipeline and testing a JSON API with Cucumber. Yes, some people do it, but the hacks it takes is a sign that it’s not the right thing to do. Learn to recognize those signs quickly and take another route. It’s the finer part of knowing to pick the right tool for the job, the timeless engineering skill.

Static code analysis will do you good

Mon, 23 Apr 2012 12:00:00 GMT

My interest in static code analysis has been renewed after I came across John Carmack’s blog post:

The most important thing I have done as a programmer in recent years is to aggressively pursue static code analysis.

For Ruby, I found reek. I ran it against the source code of the project I’ve been working on. Complains about long methods, referring to a variable more than self. Does it really matter if a method has six statements, I thought?

I decided to blindly follow the warnings and reduce them to zero. During the process, I found that thinking harder about refactoring makes me discover nasty bugs that I was not aware of. Working out seemingly “small” and “annoying” warnings leads to improved code design: even more lightweight objects, with easy to follow communication and very short methods.

If you’re following the BDD practice, chances are big that your code is pretty good. But it can be made even better, and to have a program issue cold warnings when you get carried away in your little algorithms is critical in that next step.

How do you maintain that good practice? It needs to part of your workflow, so make your specs fail whenever there’s a smell. That’s now one new step after you satisfy your acceptance and unit tests.

I did this while the project was relatively new (about 2 months in) and of course it’s a different story if the project has been going on for years. In that case I suppose you could do it on a file-by-file basis, as your work brings you to them.

Agile and change

Tue, 29 Nov 2011 12:00:00 GMT

It happens that people identify themselves with a piece of work that they did. From a purely logical perspective, this does not make any sense, and generally it is a guaranteed way to lose your personal tranquility. Yet it’s sometimes difficult to control, especially if you don’t practice it somehow.

To a programmer it may occur on both code and feature level. You think that something is such a great piece of code that it’s a pity to remove it. The more time you’ve spent on it, the greater the feeling. It influences your thinking in a situation when you’re asked to discuss the real user value of it.

That’s one aspect where agile shines. When you’re writing a spec first, that flow has framed the implementation code for you differently. It is more like a means to an end - to make the nice test pass. And that’s good because your code is a means to a user’s end. So you’re not personally attached to it.

And when you’re collaborating with your client daily, he gets to see the direction where you’re going with a larger feature when it’s 20% done and not 99% done. It’s a great framework of collaboration where nobody is afraid or opposed to change for any reason. Indeed, you will embrace change, as change is natural and feel a boost every time you refine your creation to better match the product.

Making change

Tue, 18 Oct 2011 12:00:00 GMT

From Individuals and Interactions: An Agile Guide:

When building a new system, the software developers’ livelihood is based on creating change. At the same time, the world of many users of the new software may resist having to learn new tools.

While we, hackers, are all proud to be makers, it’s easy to forget that in the end anything we make is a change for everyone who’s exposed to our work. It is probably an underestimated reason why most software projects fail - they simply fail to negotiate enough attention with users.

Every time you decide to use a piece of software, you enter an unwritten contract, where the application promises to be useful for certain tasks that are important to you, while you promise to give it your time, attention, and a spot in your living habits.

It is no wonder that everyone sooner or later optimizes to work with less software. That is actually the starting point for most people outside the technology world - and they never move from it. We often need a number of years to enter that zone, and we may even glorify that realization, as shown by the popularity of software blogs that talk about “minimalism”, “simplicity” and “zen”.

Similarly, on the web, “tools” are replaced with “web sites” and the concept remains. We have just enough space in our brain to visit a few sites when we’re online, and it why most people get most of their information via recommendations off very few social web sites.

That would mean that most web sites should be optimized for casual, short interaction and a short attention span. Be useful, and provide for quick exit. For a user it’s a happy loop, and it’s a promise for an eventual return.

Book review: The Clean Coder

Fri, 19 Aug 2011 12:00:00 GMT

This book could be called Timeless advice for working programmers and it would be appropriate. In 200 pages it packs succinct advice on professionalism, managing expectations, negotiation, coding practice, self-management, collaboration and basically everything that makes a software developer’s professional life.

The chapter “Saying No” delighted me, as it completely uncovered the psychological background of feature negotiations. That moment when the manager or client comes up and says “I’m hoping you can do the login page tomorrow”. It’s crucial to realize that it’s up to you assess the situation and hold on to your professional word of what can be done and when. Also the “Typical Project Proposal” guest essay is brilliant, it’s everything that can go wrong from the moment when you accept to do the impossible.

Estimation is something that we at Rendered Text have been learning completely from our own experience and mistakes. We no longer commit to a time estimate for an iteration longer that a month, but it took us a lot of failed optimism to reach that. And thanks to a chapter in Clean Coder, we now use the PERT scheme coupled with Planning Poker.

Here’s a great quote on testing:

How can you consider yourself to be a professional if you do not know that all your code works? How can you know all your code works if you don’t test it every time you make a change? How can you test it every time you make a change if you don’t have automated unit tests with very high coverage? How can you get automated unit tests with very high coverage without practicing TDD?

Chapters are usually illustrated with anecdotes from Uncle Bob’s younger days. In hindsight, it’s a little surprising what sort of mistakes he and his teams would make, but also amazing what kind of work and effort had to be made in the old days for what we now take for granted (debugging from tape anyone?). Everything written here is so real. Thanks, Uncle Bob.

Test now

Tue, 19 Jul 2011 12:00:00 GMT

A common statement among the people who don’t do TDD or BDD goes something like this:

We don’t write tests in this project because we don’t have time, but if we’d start a new project we’d write tests.

Obviously “we don’t have time” is another way of saying “we’re not good at writing tests”. Time-wise, there is no real difference in being at the beginning and in the middle of a project - you’re equally inexperienced. So practice! It’s the only way you’ll get there, and your workplace is not where you should be doing it. Learn more about code katas, or find coding buddies and do some coding dojos.

Some people think that their code is too complicated to even begin writing tests. Anyone doing BDD will tell you that only good code is easily testable, but to those who find it difficult, start small. When a bug is reported, write a test case first. Define the behaviour, mock external services if that’s an issue. Then refactor, write smaller methods, repeat.

Developers in some companies do not write tests because there is a “QA team” doing all the testing. It’s a shame that in this day and age, when RSpec and Cucumber can do the entire QA of a complex web application in a couple of minutes, there are beaurocracies of people clicking around for hours as per instructions in a handbook.

Do not be afraid to acknowledge that you need to learn something new and improve your skills. In fact, we should all find great pleasure in that. And few things feel better in our job than automation.

Book review: Rework

Thu, 23 Jun 2011 12:00:00 GMT

Common sense is most objective. We, on the other hand, get carried away easily. The depths of our ego, desires and passions lead us to believe that we need to implement features to serve thousands of customers on day one, treat employees as children and overrreact with ASAP notices.

Easy is a word that’s used to describe other people’s jobs.

That’s why we need something to periodically reiterate common sense for us. But it’s not only about correcting mistakes and reassurance of what matters in a company. This book is a catalyst, and after reading it I am even more excited about my work as my ideas are augmented with a realization that I can simplify even more.

Predictability of (open) platforms

Mon, 14 Mar 2011 12:00:00 GMT

Chris Dixon wrote a nice and concise article on the economics of software platforms, which reflect how seasoned software developers approach them.

The final paragraph explains, in a nutshell, the core reasons why we’ve based our company on open platforms:

The least risky platforms to develop on are successful open platforms like the web, email, and Linux. These platforms tend to change slowly and have very public development roadmaps. In the rare case where a technology is subsumed by an open platform, it is usually apparent far in advance. […] The predictability of open platforms is the main reason that vast amounts of wealth have been created on top of them and investment around them continues unabated.

HTML5 in a nutshell

Fri, 14 Jan 2011 12:00:00 GMT

This post is for our clients and anyone who needs a high-level overview of HTML5, focusing on use cases and examples. As always, we welcome your comments in the discussion section below.

HTML5 is the newest edition of the HTML standard, the markup language that is used to write web pages. While HTML5 does bring some new HTML tags to the table, that’s not the whole story as it also includes browser APIs that allow us to create animated graphics or offline apps. Together with a new version of HTML comes a new version of CSS — CSS3, which has its own novelties, all the while both are in fact a work in progress and not fully supported in all browsers.

So when usually when you hear about HTML5, it an umbrella term for a collection of features that allow us to create better web apps.

The structure

Most web pages share a common structure of headers, navigation, main section, sidebar and footer. Articles usually have a timestamp somewhere. All these now have corresponding HTML tags. They will not just make the implementation more meaningful, but provide more meaning to assistive technology and search engines as well.

Search engines support another set of extensions called microdata. Google has a Rich snippets program where they read special markup elements for people, organizations, reviews, products, recipes and events and show them as a nicely formatted search results.

Smarter forms

Forms are everywhere on the web. Over time, they’ve come to share a lot of enhancing behaviour. It’s nice to have placeholder text that goes away once you start typing in an input field. It’s convenient to automatically focus on an input field once a signup form shows up so that users can get started quickly. This form usually needs to validate that the email address entered is in a valid form. We love our mobile devices more when they show specialized keyboard layouts, such as a “.com” button when we’re entering a web address, or an ”@” button when we’re in an email field.

HTML5 defines easy ways to accomplish all that, and more, without any programming.

The great thing about all these features is that we can use them today without breaking anything. If we use a special search input type for example, users with Apple devices will see a special keyboard, Chrome and Safari users will see a little “x” button to clear text, while the browsers that don’t support it, they’ll just show it as a regular input field.

The looks

The age-old limitation of having to use a handful of fonts for displaying text on a web page goes away with a new CSS rule that allows you to download a particular font from your server to render a webpage if the user hasn’t got that font installed (source).

Embedding video on a web page used to mean that we must use Flash, resulting in cooler fans going wild on laptops across the globe. Same if you wanted a simple music player. That is no longer the case with new HTML elements for video and audio, including the APIs to play and pause a video or audio track and more.

Apple has been the forerunner in this area. Its devices do not include Flash, and the company has decided to strategically support HTML5. They have a nice showcase of its features, including animations that can be accomplished with CSS3, and a list of big sites that support HTML5 video.

This has driven video sharing sites such as YouTube to implement a HTML5 version of their players for browsers that support it. If your site has a tour video done in Flash, it will not be visible to people with an iPad. The easiest way to reach everyone is to upload it on a video sharing site, who will take care of showing the most efficient player to everyone.

A new thing called canvas, with a corresponding new JavaScript API, allows to draw pretty much anything on a web page, each piece of the drawing being fully interactive. A common use case are charts (we like Highcharts), which now anyone can quickly develop without paying a 3rd party vendor for expensive authoring tools. There are even full blown games made of this stuff.

Offline and location-aware

A few years ago Google developed a thing called Google Gears which allowed their applications to work offline. Now that HTML5 includes ways to store application cache and offline data, we can all make use of it in a standardized way. This is especially important for mobile apps where we cannot assume the Internet being always available, while users’ desire to see their data we can.

Another nice thing is called geolocation API, which lets you share your location with trusted web sites that can then do fancy location-aware things like finding local businesses or showing your location on a map (source).

Browser support

In an attempt to sum up, we can say that most capable browsers nowadays are on new mobile devices (they can be seen as drivers of innovation in fact), IE as always is a disappointment, while computer literate people who use Firefox, Chrome or Safari can benefit from most of these features today.

SchnitzelConf wrap up

Mon, 13 Sep 2010 12:00:00 GMT

If you do something, your business will do something.

The quote from Thomas Fuchs, one of the organizers of the awesome SchnitzelConf in Vienna, is not quite epic (less than a dinner with dinosaurs at least) but it really is the basic assumption of success: keep moving.

The point of the conference was to encourage European web developers with an entrepreneurial spirit to build a product, charge for it and be independent. Definitely a good thing, as in the slightly socialist Europe the system usually trains people to look for “safer” opportunities.

The talks were fantastic, as the speakers were very open and shared a lot from their experience. They all spoke with a a slightly different focus but with the same background – that of a tech person with creativity, curiosity and individualism that could not be idle, to paraphrase Michael Buffington.

Andy Croll wrote a very nice overview, and has documented the presentations as well. Jarkko Laine’s mind maps are a good reminder in handwritten form. Buffington had a slide with random interesting quotes that he heard before his talk.

Using Flickr API with Ruby and r2flickr

Thu, 13 May 2010 12:00:00 GMT

If you want to use Flickr API with Ruby, there are plenty of options. The one we’ve been using for a long time was rflickr by Trevor Schroeder, which is an implementation that matches the Flickr API as it is officially documented. However, it was unmaintained in RubyForge’s svn repo, and it needed some fixes, so we eventually published r2flickr. You can install it as a regular gem:

sudo gem install r2flickr

Here’s a hint on how you can search for photos with a certain tag and get some information about each of them:

flickr = Flickr.new(FLICKR_CACHE_PATH, FLICKR_API_KEY, FLICKR_SHARED_SECRET)

photos = flickr.photos.search(
  :tags => "sunrise",
  :per_page => 20, :page => 1,
  :sort => "date-posted-desc"
)

photos.each do |photo|
  info = photo.flickr.photos.getInfo(photo.id)
  puts "#{info.title} from #{info.ownername}"
  puts photo.url('t')
end

For more details, check out the project on GitHub.

Rendering Textpattern

Mon, 15 Mar 2010 12:00:00 GMT

Recently we needed to deploy a site made with Textpattern CMS by our friends from Sprawsm, along with a forum app that we built with Rails. The scenario is that a designer develops a web site locally in his Coda or similar environment, and hands it to you archived with a filled config.php a database that is ready to be imported on the server. We wanted to set the site up as a virtual host.

It wasn’t that quick and easy, as deploying a Textpattern site is basically a nonexistent term. Instead, it is usually just uploaded via FTP to a shared host. Searching for a sample VirtualHost configuration led me to the Textpattern Multi-Site Howto, which is quite noisy and suggests that you point your DocumentRoot to a subdirectory in sites/. It includes pieces such as [Textpattern will tell you to place your config.php file in /textpattern/. This is INCORRECT.[ and [it will present a link to the [main interface.[ This link will be INCORRECT, so do not click it[. Ignore that guide. In fact, remove the sites directory, unless you do have multiple Textpattern sites and you’re absolutely sure that you want to save a couple of kilobytes of PHP source code at the cost of getting lost in configuration.

This is what works for us:

/var/tpapp$ rm -rf sites

<VirtualHost *:80>
	ServerName tpapp.com
	ServerAlias www.tpapp.com
	DocumentRoot /var/tpapp
	<Directory /var/tpapp>
		Options +FollowSymLinks
	</Directory>
</VirtualHost>

Then in /var/tpapp/textpattern/config.php:

<?php
$txpcfg['db'] = 'tpapp_db_name';
$txpcfg['user'] = 'root';
$txpcfg['pass'] = 'rootpassword';
$txpcfg['host'] = 'localhost';
$txpcfg['table_prefix'] = '';
$txpcfg['txpath'] = '/var/tpapp/textpattern';
$txpcfg['dbcharset'] = 'utf8';
define('txpath', $txpcfg['txpath']);
?>

By the way, on Ubuntu the easiest way to get all the dependencies quickly is to install the Textpattern package, and then manually create the target database:

apt-get install textpattern

Textpattern has a screen to diagnose possible issues which you can access at /textpattern/index.php?event=diag. For example, you might’ve forgotten to enable mod_rewrite (via sudo a2enmod) on a fresh server image, or fix some simple permission errors.

New event explore interface in Plakatt

Wed, 03 Feb 2010 12:00:00 GMT

This video shows the new event explore page on Plakatt that we’ve been working on recently.

It’s probably hard to understand before we make it available in English and you do not speak Serbian, however to sum up, in that big input box you can type in a city, event category or a time defining term such as ‘today’ or ‘tomorrow’. We call these filters. Filters can be combined to form queries that match your interest (except for the those that define a time, which are exclusive). Registered users can save them for later as shortcuts in the dashboard sidebar, handy when you want to quickly check which gigs will be happenning today and which art exhibitions are planned in current month, for example.

As you see the page is making Ajax calls only. However each change to the current filter set updates the URL which can also be accessed directly, and your browser’s Back button still works as it should. We make heavy use of jquery-history to achieve that.

We store each explore query, so that when when you return back to the explore page your last state is restored (guests are tracked by session). This is where we use Tokyo Cabinet (over Tokyo Tyrant in fact), since it is hundreds of times faster than MySQL in write operations. In some of our future posts we’ll highlight how exactly we set up and use our Tyrant instances.