One of the hardest refactorings to get right is the balance between extract and inlining of functions.

Often people struggle to understand why sometimes when they “do the right thing” and extract a function, their code quality feels like it decreases - this is actually one of my biggest criticisms of Clean Code as a book - it’s rife with low quality method extraction.

I think that feeling comes from the weight of “context transference”.

Context transference is the amount of information you have to pass between boundaries, and method extraction where 100% of the callers context needs to be transferred for the new function to be meaningful is a poor refactor. You don’t reduce any cognitive load - it’s a failure of encapsulation.

In object oriented languages this is a more common problem because the context is often scoped to a class, so it’s implicit which parts of state are used in any given function, and as a result this transfer of context is equally implicit.

While it might sound like object scoped state is actually a solution to this problem, it isn’t, because the transfer of context is actually the amount of information the programmer has to mentally track during code reading, not the information a computer has to track (because functions exist purely for the programmer) - it’s a design time concern and rarely a significant runtime one.

Extraction refactorings should operate on a strict subset of the context of the parent function to be meaningful and not detrimental to reading code, as a general rule.

(Standard disclosure: generalisations include exceptions by default, I’m sure there are many cases where this doesn’t apply)

Before we get started - I’m working on a large paper on the current hype-state of AI, what’s actually real, and what’s hubris along with the slow steady march of progress that is actually happening. This is an extract from that work.

It’s also a bit of a “Star Wars: Episode 4” moment - I’m publishing the middle first - 7 predictions (that might age badly), but I think they’re interesting enough to publish before I finished the rest of the paper. The full thing takes a deep and introspective look at both the state, and the ethical problems we have with the current wave of AI. So if you’re about to “well actually” about some ethics thing or another, it’s cool, hold your breath, that’s part 3.

AI people, we really need to talk

I am not an “AI builder”, I’m a systems builder, and a programmer first and foremost – and I’m explicitly an “AI moderate”.

What I mean by “AI moderate” is that I think AI is simultaneously one of the most interesting and exciting things that’s happened in my technical career, but equally one of the most overhyped, misdescribed, poorly marketed and generally misunderstood pieces of technology.

AI is difficult to talk about because it’s become such a poorly debated, thoroughly misunderstood and quickly changing space that it makes understanding what is real and what is hubris hard to grasp.

It’s in the best interest of the people trying to sell you AI to over-hype it’s real capabilities, but dismissing it outright is a foolish thing because despite the ghouls that chase the tail of technology trends (Blockchain, NFTs, et al) being a large cohort of the people that are chasing this trend it’s truly not the same thing.

That’s easy to see when you look at the people really involved in building out and betting big on this technology. Neural Nets are not new, ML models are not new, transformer models are a little bit newer. Exceptionally smart people have been working to this point for a long time – this isn’t a get rich quick scheme – it’s decades of research and a huge amount of investment that’s been slowly rolling onwards for the last two decades in its current form.

This is all set against the controversial backdrop of the training data that’s processed by large organisations to produce vast general-purpose models that pushes against the edges of existing laws around fair use, people’s personal ethics and challenges human exceptionalism. I’m explicitly omitting the ethical discussion from the first half of this piece so I can dedicate the whole second half to it. Discussion of AI cannot exist in a social vacuum, but it also shouldn’t overshadow factual discussion.

This is my attempt to try summarising where we are, where we’re going next, and hopefully sift through what is real and what isn’t about the AI-hype.

There’s plenty of bad faith critique and evangelism in this space, so I’m going to try and neatly side-step both of those things in the following ways:

1. I have nothing to sell you.
2. I have no vested interest in AI other than having to hold the pen on a platform strategy that has to exist in the same universe as it.

But here’s what I think is going on in the industry and why.

7 Predictions - What’s next?

Let’s start with the big claim – there’s not going to be “AGI” – artificial general intelligence – in my lifetime.

The Star Trek myth of the sentient computer with the personality, the ghost in the machine, the AI of sci-fi which we have ethical quandaries about because it might be alive? That doesn’t exist.

It’s science fiction given what we’re currently working with, and each time you see another click-baiting post towards that thing its people clowning themselves. Obviously “never say never”, but that’s not where we are, and not what we’re building right now.

On the other hand, it’s exceptionally likely that we’ll have a collection of technologies and systems that integrate in such a way that if you squint at them might look, to the amateur, like we’re progressing towards that thing. People will absolutely be selling you “AGI” sooner than you think, but it’s going to be far more traditional than you expect.

I want to share my 7 predictions about what the next 5 years of “AI” looks like in practice:

1. The Future of AI is LLMs on the Edge, blended with traditional systems integration
2. The Future of Language Models is “Small Language Model Expert Systems”
3. This approach will lead to a renaissance in standards-based RESTful online services and model integration technologies
4. Websites and apps will decline in lieu of “Assistant Computing”
5. The building of those traditional systems will be AI assisted
6. Large models either have, or will soon plateau and won’t get drastically better
7. Software development jobs will change, but aren’t going anywhere

I think that until such a point where we start to have something “more” than three LLMs in a trench-coat, that a more honest name for what’s currently happening is “Model-Assisted Computing” and we should have probably used that rather than the hubristic “AI” and “AGI” naming. Hell, we could have even called it “MAC” for short.

I think if you start to draw a through line from Web 2.0 through smartphones and to the current rising tide of model-assisted computing, then you’ll realise that this is where we’ve been going the entire time and it’s mostly just a continuation of the vision of the web.

Here’s how it’ll happen

The existing wave of LLMs have shown that they’re exceptionally good at fuzzy matching human input. They’re statistical transformer models that predict output given an input, making them pretty good at what you’d traditionally associate with “Q&A” based on a training set. They’re really the next evolution of Google Assistant, Cortana, Siri and Alexa – the thing on the edge that can turn natural language questions into commands that need to be fulfilled.

As an industry, assistant-lead compute has been a thing since 2011 when Siri launched but was popularised arguably by Amazon’s Alexa in 2014. We’re 15-years into this and LLMs have given us a fuzz-matching technique that’s just plain better than defining platform-specific “skills” (to use Amazons terminology) for systems integration.

These large models will marginally improve over time, but they won’t be good at doing any hard or detailed work at all because they are purely statistically models. Most of the ignorant critique of LLMs focuses heavily on this point – that the models are “wrong” – because they’re not even trying to be right. Over the past 16 months we’ve seen the rise of Retrieval-Augmented Generation – a technique that interpolates data from data sources (frequently vector databases) into the outputs of LLMs so that they can source factual data.

RAG was the first step, followed swiftly by plugin models in GPT, but both of those things are rapidly giving way to two standard protocols – MCP – the Model Context Protocol, and A2A – the Agent-to-Agent protocol. Both protocols go some way to systemising RAG, exposing tools and resources for models to call out to, and wrapping models in web-standards for authentication and discovery.

When we get this right, the accuracy problem is solved – language models are used to interface with humans, and protocols revert to traditional systems integration techniques to perform operations and source facts, effectively giving us the best of both worlds. This isn’t speculative, both protocols have been in rapid development and adoption over the last 6 months and are probably the future of “agentic computing on the open web”.

What does this mean for builders? Back to web standards we go. The easier it is for the thing you do to be described as APIs, and metadata, and commands, the easier it will be to context shift into MCP and A2A workflows that interact with large models that by default will be enabled on everyone’s pocket devices. We’re going to be here in the next 6 months.

Layered on top of that, A2A offers some /.well-known style service discovery for agents – a place for you to define the operations that your top-level domains can provide. It’s an incredibly small leap from here to realise that this will eventually evolve into something close to a DNS registry of things that the runtimes that host large models have access to, which in the most open-minded place we can be is a great thing for services on the web (“hey Siri, check my bank balance for me”) and in the darkest places provides the platform operators of those large models a vehicle to deeply integrate, but also probably levy an app-store style tax on your systems from the outer edge. Still, a globally discoverable, automatically integrate-able set of commands across the whole internet is a wonderful enabling technology.

When we get there, people will start trying to sell you this as “AGI”. Probably. Because to the amateur eye, it might kind-of look like it.

The second order effects? The read-only portion of most webapps will sink under the substrate of agent-computing. Within a few years, it’s unlikely people will be opening your app or webpage to check on data. They’ll obviously still come to rich experiences for content (the web or apps aren’t going anywhere) but purely transactional things – “buy me that cinema ticket”, “check my bank balance”, “do the simple X” – anything that can be wrapped in a one-time step up auth flow, probably will diminish in importance and only rich interactive content will survive on the screens.

That’s going to be the consumer experience. It’s easy to doubt this now because plenty of the existing implementations of these things are rough approximations that absolutely suck (everyone can make fun of googles bad AI search, for example), but this isn’t that thing. This isn’t “can you work out how to sift through this data”, this is “I’ve told you exactly how to do this, follow this well-known protocol to do this well-known integration”. It’s how all your apps work today, wrapped in a thin veneer of language models to protocol shift your requests.

But it’s coming, because the technology mostly works now and the user-interaction is the place we’ve been trying to get for decades. Frankly, also, once you step back, it’s also good computing – computing that slips under the substrate of all the technology and returns to a place of magic. It’s good UI design – no UI.

What does this mean for technology vendors?

Just keep on keeping on, in a sense. Since Web 2.0 and the mobile app revolution, services that don’t provide APIs to deeply integrate have been F-tier ghetto services that people hate using. Unless you get better at systemising your… systems and meeting the market where it’s at with good machine-to-machine APIs, your business will eventually die.

But the reality is that most technology businesses are already living that experience in real-time. This isn’t new. What is new is that we’re going to see a cottage industry of language models and agents trained on proprietary in-house data sets wrapped up in APIs and sold as agents that the assistance-driven compute services can interact with like a marketplace (you can hear the mega corps salivating at taking their 15% already).

Organisations rich in data will realise that the same expertise they used to sell with humans can be synthesized and sold as commands and tools for these models to interact with, scaling their business.

From an engineering perspective? There are tonnes of APIs to be built. You’ll probably be using Copilots to help build them quickly, but you’ll still require more engineers than ever to operationalise them and make them work. This mirrors the last decade of real-world challenges in operationalising data engineering and machine learning. It’s not easy, it’s buggy and requires a lot of focus. The AI revolution won’t replace programming jobs, but it might take some of the repetitive work around the edges away.

We’ll see a rise in organisations operationalising “Small Models”, trained on their own data, and exposed as agents. We’re also going to see the platform vendors of LLMs training specialised small language models that can be run on local compute for domain specific tasks. This will partly be in response to the market asks (more secure, domain specific), but also as a loss leader into their platforms.

This is based on the fact that today, large language model vendors have working proof that they can train smaller targeted models from synthesised data that are as effective as LLMs in a lot of domains.

The large models will plateau – we’re already seeing this now. Compared to traditional software development, model development is a lot less deterministic. Successor models aren’t always strictly “what we had before but more”, and the steeper the climb becomes, the more we will rely on LLMs connecting to constellations of specialised models and tools to do detailed work. There’s a good chance we’ve already hit near the ceiling here given the current struggles to get the next-generation models out of the door. The cynics all seem to think this is “the end of GenAI”, but what it actually is the general point of utility where the model we have today become operationalised in more interesting ways.

Finally?

Well, engineering jobs will change. People are going to get over the hiring malaise (“surely we don’t need these noisy nerds anymore!”) and realise that the biggest challenge in software isn’t writing it, it’s operating and maintaining it. Software organisations will reach the conclusion that using the models to help maintain, remove, reduce and optimise code is a saner and more sustainable path than just pouring more code onto the tyre fire.

Just having more code doesn’t help anyone.

I think this is the obvious through-line, the predictable end of the path that draws together what we dreamt of for the internet (connected services exchanging structured data), the smartphone era (everything is an app), and the frontier AI fever dreams (we’ll make a sentient machine!) into something that’s both obvious and almost real today.

And everyone will say we’ve got AGI, and it’ll still be bullshit.

Footnote

For those about to reach for the comments section, the full paper looks like this:

The State of “AI”

• I am an AI Moderate
• Bad Faith Critique
• The Precision Problem
• Current AI tools are a better hammer
• 7 AI Predictions – what’s next?
• What does this mean for programming?

Ethics and the Adoption of AI

• Technology is a labour concern
• AI is incapable of art
• Reactions to your understanding of “value”
• Capitalism and AI
• Open-Source and AI
• The real-world costs of training and operating models
• A model trained on the web should be given to the web

Thinking Machines

• What if intelligence is a latent quality of data?
• Discovering a general-purpose computing approach

So do wait for the rest :)

Form is one of the hardest things to understand in software, mostly because it gets conflated with style and formatting. Style and formatting influence how people read and understand your code - you can bury good design in bad style, and you can make bad design look good with good style, but form is fundamentally an expression of design.

We struggle with this because there are many decent working programmers that have never actively engaged with the design of their software, so mistake cosmetic decisions for design decisions. The reason that form is so important in software is that it’s the thing that directly influences how people understand your code, and the tool you have to articulate your design goals, and the trade-offs you’ve made to achieve them.

From is the way you name your variables, the interaction style you outline with your APIs, the kinds of data structures you pass around your software. The intent of your software design is encoded in its form.

One of the more interesting aspects of form is the tension between regular and irregular form. Regular form is the default form of a language - it’s collection of idioms, defaults, “standard ways of doing things”, compared to an irregular form, where you diverge from form to achieve some other goal.

While people might not realise they’re responding to the form of a piece of software sometimes I’ll see people talk about things being idiomatic / non-idiomatic, or “feeling right” / “feeling wrong” - this is a response to the form of the software, and how it’s articulated. You see this everywhere in different programming language - python developers often talk about “pythonic” code, golang has a philosophy of emphasising “simple and minimal” code that trends towards repetitive, and multi-paradigm languages like C# will often face backlash when language features are introduced that bring in a new form of doing something that exists elsewhere in the ecosystem. All of these behaviours are people internalising regular form and responding to it.

Regular form, and regular style are a good way of helping people feel comfortable and oriented in software, but some of the most interesting software design comes from subverting regular form to achieve some other goal.

For example, ASP.NET middleware looks like this - taken from the official documentation:

public class RequestSetOptionsMiddleware
{
    private readonly RequestDelegate _next;

    public RequestSetOptionsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext httpContext)
    {
        var option = httpContext.Request.Query["option"];

        if (!string.IsNullOrWhiteSpace(option))
        {
            httpContext.Items["option"] = WebUtility.HtmlEncode(option);
        }

        await _next(httpContext);
    }
}

This class can be discovered or passed to the framework at startup, and implements the “middleware” pattern common in web frameworks. If you’ve used ASP.NET Core, you’ll be familiar with these classes, but as a class, it’s actually pretty weird. The default, conventional way of thinking about classes in C# is that they should be stateful, and have methods that operate on that state. The middleware construct in ASP.NET Core neither follows that pattern nor is used in that manner - really just acting as a container for a function that gets invoked as part of the request pipeline. This is a subversion of regular form in C# to achieve a different goal - to make it easier to compose and reason about the request pipeline in a web application. ASP.NET Core and MVC in general are full of these subversions of regular form - so much so that they’ve become expected, regular forms of their own (Controllers are another example of a class behaving in a way you would not traditionally expect a class to behave, with lifecycles managed invisibly by the framework).

Framework authors often subvert regular form to express the design and intent of the frameworks they’re building and their features - despite using the same syntactic constructs as the code that lives in the “application space”.

There’s a wrinkle in this though - subverting regular form is a high-risk, high-reward strategy. It’s easy to subvert regular form and make your code harder to understand, or to subvert regular form and make your code harder to maintain. It’s a tool that should be used sparingly, and with intent. It also, by definition, requires a mastery of regular form to correctly subvert.

I went to see a great exhibition of famous film director and animator Tim Burton’s work recently, and one of the more interesting things was looking at the sketches that he did while training as an animator. Not because they exemplified his signature style (his own subversion of form) but because they were so textbook and quality. He had to learn what correct form was in traditional illustration, and why it existed, to be able to develop his own style that explicitly had something to say.

Burton is famous for esoteric stylised characters and settings, but he had to learn the rules before he could break them. All of the best people at a discipline occasionally subvert conventional form in design because regular form is the “best worst” and least contentious path. But to subvert form, you must understand what the most common form communicates, and find a way to do it better.

This approach to learning mastery before reform reminds me of the thought experiment of “Chestertons’ fence”:

    "In the matter of reforming things, as distinct from deforming them,
    there is one plain and simple principle;a principle which will probably
    be called a paradox. 

    There exists in such a case a certain institution or law; 
    let us say, for the sake of simplicity, a fence or gate erected
    across a road. The more modern type of reformer goes gaily up to it and
    says, "I don't see the use of this; let us clear it away."

    To which the more intelligent type of reformer will do well to answer:
    "If you don't see the use of it, I certainly won't let you clear it away.
    Go away and think. Then, when you can come back and tell me that you 
    do see the use of it, I may allow you to destroy it."

You shouldn’t change or subvert something until you know why it is.

Regular form is what drives conversations (that are mostly boring) around design patterns. Design patterns, popularised by the “Gang of Four” book of the same name, were intended as model answers to well known questions in software design of the time. They gave language and shapes to regular problem spaces. Unfortunately, as a side effect, people ended up fixated on their regular form, and “best practice” to their detriment. Many of the lessons dissolved into cross-generational knowledge held by people that never read the original works, who didn’t realise that they were presented as “things that we have observed working several times in specific contexts” and instead ended up as accidentally canonised designs.

It’s easy to think that I am advocating for “do whatever” design, but I’m not. Regular form - patterns and defaults, are valuable because they’re explicable to all. You are absolutely allowed, and should, with maturity as a designer, subvert regular form, but only when in your context you have something that achieves the same goals as the regular form, with better outcomes that says something about the design of your software, APIs, or modules. Making random non-default decisions is probably not smart, unless you’re an expert doing it knowingly and making an explicit trade-off that elevates or better solves for part of your design.

Footnote

Notes on the Synthesis of Form is a book by Christopher Alexander, a famous architect and urban planner. It’s a book about design, and how to think about design in a way that’s not just about making things look good, but about making things that work well. It was the original inspiration for the term “design patterns” in software. I invoke its name half in homage, and half as a knowing nod.

“Test coverage is not correctness” and “test coverage is not a quality metric” are two of the lesser understood common phrases you’ll here people parrot. I agree with the first, but strongly disagree with the second.

I love test coverage because it’s a strong, negative, leading indicator towards quality.

The presence of test coverage “we ran this code”, is not the same as the test being correct “I verified the correct behaviours”, but a lack of it tells me a lot.

It tells me to expect low quality and poor automatic verification. It tells me where the worst parts of your codebase probably are.

This is because coverage comes for free in a well-tested system - it’s a measure, not an attribute, of a codebase. It’s a torch in the dark of working effectively with legacy code.

What else does a lack of coverage say?

It tells me that a particular codepath is either so well trusted its owners perceive it cannot fail (or will fail another signal immediately), or it’s so complicated to exercise that it’s not executed often at all. All code infrequently executed is bound to eventual failure as the world changes around you.

Coverage conveys a lot of information, don’t ignore it. It might not be a proxy for correctness but that doesn’t make it useless.

Footnote: Can we fix the “coverage is not correctness” problem? Actually, yes. Mutation testing is a technique where you “automatically test your tests” - notably implemented across languages by the tool Stryker, mutation testing creates subtly modified versions of your codebase tens to hundreds of times, and executes your tests over these “mutants”.

If your tests don’t fail after they’ve been run against mutants? That proves you’re not correctly verifying some behaviour. Mutation testing is awesome and infrequently seen in the wild.

The vast majority of systems that you build will inevitably call a HTTP API at some point. Whether it’s a microservice, a third party API, or a legacy system. Because of this, it’s not uncommon to see applications with reams of configuration variables defining where their downstream dependencies live.

This configuration is frequently a source of pain and duplication, especially in larger systems where tens or hundreds of components need to keep track of location of downstream dependencies, many of which are shared, and almost all of which change depending on deployment environment.

These configuration values get everywhere in your codebases, and often are very difficult to coordinate changes to when something changes in your deployed infrastructure.

Service discovery to the rescue

Service Discovery is a pattern that aims to solve this problem by providing a centralised location for services to register themselves, and for clients to query to find out where they are. This is a common pattern in distributed systems, and is used by many large scale systems, including Netflix, Google, and Amazon.

Service registries are often implemented as a HTTP API, or via DNS records on platforms like Kubernetes.

Service discovery is a very simple pattern consisting of:

• A service registry, which is a database of services and their locations
• A client, which queries the registry to find out where a service is
• Optionally, a push mechanism, which allows services to notify clients of changes

In most distributed systems, teams tend to use infrastructure as code to manage their deployments. This gives us a useful hook, because we can use the same infrastructure as code to register services with the registry as we deploy the infrastructure to run them.

Service discovery in .NET8 and .NET Aspire

.NET 8 introduces a new extensions package - Microsoft.Extensions.ServiceDiscovery - which is designed to interoperate with .NET Aspire, Kubernetes DNS, and App Config driven service discovery.

This package provider a hook to load service URIs from App Configuration json files, and subsequently to auto-configure HttpClient instances to use these service URIs. This allows you to use service names in the HTTP calls in your code, and have them automatically resolved to the correct URI at runtime.

This means that if you’re trying to call your foo API, that instead of calling

var response = await client.GetAsync("http://192.168.0.45/some-api");

You can call

var response = await client.GetAsync("http://foo/some-api");

And the runtime will automatically resolve the service name foo to the correct IP address and port.

This runtime resolution is designed to work with the new Aspire stack, which manages references between different running applications to make them easier to debug, but because it has fallback hooks to App Configuration which means it can be used with anything that can load configuration settings.

Here’s an example of a console application in C# 8 that uses these new service discovery features:

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

// Register your appsettings.json config file
var configuration = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
    .Build();

// Create a service provider registering the service discovery and HttpClient extensions
var provider = new ServiceCollection()
    .AddServiceDiscovery()
    .AddHttpClient()
    .AddSingleton<IConfiguration>(configuration)
    .ConfigureHttpClientDefaults(static http =>
    {
        // Configure the HttpClient to use service discovery
        http.UseServiceDiscovery();
    })
    .BuildServiceProvider();

// Grab a new client from the service provider
var client = provider.GetService<HttpClient>()!;

// Call an API called `foo` using service discovery
var response = await client.GetAsync("http://foo/some-api");
var body = await response.Content.ReadAsStringAsync();

Console.WriteLine(body);

If we pair this with a configuration file that looks like this:

{
  "Services": {
    "foo": [
      "127.0.0.1:8080"
    ]
  }
}

At runtime, when we make our API call to http://foo/some-api, the HttpClient will automatically resolve the service name foo to 127.0.0.1:8080. For the sake of this example, we’ve stood up a Node/Express API on port 8080. It’s code looks like this:

const express = require('express');
const app = express();
const port = 8080;

app.get('/some-api', (req, res) => res.send('Hello API World!'));
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

So now, when we run our application, we get the following output:

$ dotnet run
Hello API World!

That alone is pretty neat - it gives us a single well known location to keep track of our services, and allows us to use service names in our code, rather than having to hard code IP addresses and ports. But this gets even more powerful when we combine it with a mechanism to update the configuration settings the application reads from at runtime.

Using Azure App Configuration Services as a service registry

Azure App Configuration Services provides a centralised location for configuration data. It’s a fully managed service, and consists of Containers - a key/value stores that can be used to store configuration data.

App Configuration provides a REST API that can be used to read and write configuration data, along with SDKs and command line tools to update values in the store.

When you’re using .NET to build services, you can use the Microsoft.Extensions.Configuration.AzureAppConfiguration package to read configuration data from App Configuration. This package provides a way to read configuration data from App Configuration Services, integrating neatly with the IConfiguration API and ConfigurationManager class.

If you’re following the thread, this means that if we enable service discovery using the new Microsoft.Extensions.ServiceDiscovery package, we can use our app config files as a service registry. If we combine this extension with Azure App Configuration Services and it’s SDK, we can change one centralised configuration store and push updates to all of our services whenever changes are made.

This is really awesome, because it means if you’re running large distributed teams, so long as all the applications have access to the configuration container, they can address each other by service name, and the service discovery will automatically resolve the correct IP address and port, regardless of environment.

Setting up Azure App Configuration Services

You’ll need to create an App Configuration Service. You can do this by going to the Azure Portal, and clicking the “Create a resource” button. Search for “App Configuration” and click “Create”.

For the sake of this example, we’re going to grab a connection string from the portal, and use it to connect to the service. You can do this by clicking on the “Access Keys” button in the left hand menu, and copying the “Primary Connection String”. You’d want to use RBAC in a real system.

We’re going to add an override by clicking “Configuration Explorer” in the left hand menu, and adding a new key called Services:foo with a value of:

[
	"value-from-app-config:8080"
]

and a content type of application/json.

Setting up the Azure App Configuration SDK

We need to add a reference to the Microsoft.Extensions.Configuration.AzureAppConfiguration package to access this new override. You can do this by running the following command in your project directory:

dotnet add package Microsoft.Extensions.Configuration.AzureAppConfiguration

Next, we modify the configuration bootstrapping code in our command line app.

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var appConfigConnectionString = "YOUR APP CONFIG CONNECTION STRING HERE";

var configuration = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
    .AddAzureAppConfiguration(appConfigConnectionString, false) // THIS LINE HAS BEEN ADDED
    .Build();

This adds our Azure App Configuration as a configuration provider.

Nothing else in our calling code needs to change - so when we execute our application, you’ll notice that the call now fails:

$ dotnet run
Unhandled exception. System.Net.Http.HttpRequestException: No such host is known. (value-from-app-config:8080)
 

The surface area of software is often very complicated to comprehend, and a direct result of that is that we’re often subjected to amateur discourse about how teams, or organisations should have worked on a thing, based purely on outside observation. It’s almost always wrong, and I want to spend some time talking about how software is produced, and how it scales.

For people not close to code these examples can kind of look like similar situations with somehow opposite suggested “outcomes”, so let’s talk about the impact size has on code. We’ll start with the most difficult thing for non-technical folks to get their heads around:

1. The observed surface area of a system often has very little in common with its actual size, either in staff count, or lines of code.

The obvious example of this is Google and it’s “one text box” UI, covering a massive indexing (+more) operation.

2. More code is frequently not a better scenario to find yourself in. Code atrophies and rots over time naturally. Every line of code you write increases your total maintenance cost.

Doing things with less is most commonly a better outcome.

3. There’s only a certain amount of people that can “fit around” codebases of certain sizes - once the size of your teams outgrows the code, work slows rather than speeds up, because the rates of conflicts, points of contention and hotspots, and coordination increases.

4. To fit more people around a codebase, systems are often decomposed to different libraries, subsystems or services. This expands the footprint of the teams you can surround your code with (to increase parallelism of work) but at least doubles the amount of coordination needed.

5. Microservice architectures largely help because they allow organisations to define boundaries around pieces of their software (“bounded contexts”) and parallise work - at the cost of expensive coordination and runtime latency.

6. Equally, software is sometimes decomposed to help it scale in technical terms, rather than to match it’s human needs (fitting more folks around a codebase) - this what most people often think of as “scalability” first but the former is more common.

Few have technical scaling requirements or limitations.

7. Evey subdivision of software tends to increase total complexity of comprehending a system - it makes it less likely anyone can keep it all in their heads, it increases possible failure points (both technical and human) and increases the total cost of ownership of code.

8. Why? Simple - each time you divide up your software, you have to also create new supporting structures (teams, work tracking, CI+CD pipelines, infrastructure) in order to allow the now bigger teams to be self sufficient and have enough autonomy.

9. Lots of software does this too readily (see the knee jerk reactions in the form of monorepos and back-to-monolith trends) - but dividing your software is good, but requires thoughtfulness and intention to accept the cost.

10. This isn’t news, @martinfowler was talking about monolith-first designs about a decade ago. I like to think of it as “fix things that hurt”. A lot of backlash against microservice arch. is really just folks getting lost in premature complexity too soon.

11. In both kinds of scale - human and technical, you should build for a reasonable amount of growth. Some folks say an order of magnitude, others say 10x traffic, but it should only be one big leap.

Don’t copy what big tech do before you are big tech.

12. The important rule is this - in modern software, the best software is software that is malleable, software that can be added to without buckling, with foundations made for growth later, but not now.

This is why basically all layperson takes on software are wrong.

Folks look at software and presume they can predict its form from its external surface area (false), it’s complexity from its ease of use (actually often the inverse), and fall into the mythical man month trap (RIP Fred!) of “if one woman can make a baby in 9 months, get 9 women!”.

The size of code is a function of your team size, the subdivisions in your software and your cost and plans.

It’s all a compromise. And it’s never “just go faster” or “just spend more” - even if those things can sometimes help, they can just as often hinder and bury a project.

Making software, design and architecture “just the right size” is really very difficult. Many, many systems have fallen into the nanoservices/distributed monoliths trap, even more into premature internal package management hell.

Remember every subdivision of a system has a cost.

Monorepos (meaning “a singular repository”) is a term coined by Facebook to describe a single repository that contains all the code for a project.

It is a pattern that has been used by many large companies, including Google, Facebook, Twitter, and Microsoft. It is also used by many smaller companies, including GitHub, and by many open-source projects, including the Linux kernel.

It is frequently misinterpreted to mean “all of the software that we build”, and I want to share some notes that clarify where monorepos succeed, and fail, in organisations of various sizes.

Where do monorepos work?

Monorepos work well in an inverse bell curve if productivity related to the size of the software and teams that you have:

• when your repo is just really one app and a “component library” (…just the bits of the app in some bad directory layout)
• when you have a very low number of apps you are coupling together via source control
• when you have apps that either change super infrequently, or are all sharing dependencies that churn all the time that must be in lockstep.
• when you’ve really just got “your app and a few associated tools” - that’s very “same as it ever was” because so few repos ever had “just one tiny piece of a system” in them to start with.

Unfortunately, the zone of productivity for these organisational patterns - in my opinion - is a trap that folks fall into.

Most software doesn’t fit those three categories mentioned above.

Software tends to moves at medium speed, with SME shaped teams - and in those situations monorepos are hell fraught with problems that only occur once you’ve opted in, wholesale, to that organisational structure.

Alternatives that match those probem spaces

In most of those cases:

• when the software is really just one app - you should use directories instead of complicated build tools
• when it’s all for some shared libraries - you’re going to reach a point where you want to version in distinctly because the blast radius of change is going to start hard coupling your teams together over time

It’s trivially easy to end up in the bad place where teams end up with tightly coupled deployments that get extremely slow and have to be resolved with tools like nx that frequently take over your entire development workflow (bad!)

But the biggest red flag with them is obvious - we’ve been here before and it sucked!

Just an old solution

The first decade of my career before DVCS (distributed version control systems) was all effectively big monorepo source trees and it was absolutely horrible and fraught with the same coupling risks. So we changed!

Git is designed for narrower slices, and doing the monorepo dance in medium to large orgs with all your software will leave you inevitably fighting your tools, both in build, deployment, and source control scenarios.

The sane approach is this:

Software that versions together, deploys together and changes together, should be collocated.

In the case of the thin end of the wedge with web apps, this is often just “the app, a few shared libraries, and a backend admin thing, perhaps a few tools”.

Monorepos are fine here! At least until you need to split ownership of those things between team boundaries where things creek.

TL;DR - This is all about Conway’s Law and change frequency that charts the success of software organisation - and hard team coupling is more dangerous than software coupling.

Monorepos in massive organisations

Let’s briefly talk about the other end of the spectrum - the massive organisations that have a lot of software and a lot of teams, and all claim to use monorepos. There are notable examples - Google, Facebook, Twitter, Microsoft, GitHub.

Firstly, none of those organisations use a monorepo as it is frequetly interpreted by smaller orgs and the community. It’s easy to verify this, because they all operate open-source repositories that are public, and distinct from any internal monorepos they may have. What they do tend to have, is application centric repositories where a single application, and it’s associated tools and libraries are colocated.

This makes absolute sense, and is no different from your existing non-monorepo.

In fact, the majority of the “famous monorepos” - Windows, the Linux kernel (which of course, isn’t the same as “Linux”), and Facebook - all have entire tooling teams dedicated to making collaborating on them work, at scale, with the communities they serve. It’s very important that you don’t apply logic from organisations of a scale that you aren’t, with resources that you do not have, to your own problem space without strong consideration.

If you don’t have the budget for entire teams working on source control and collaboration, nor tens of thousands of developers to fit around your codebase - perhaps don’t mimic the patterns of those who do.

Should I use a monorepo?

Application centric repositories with associated tools and libraries?

Yeah! Knock yourself out, makes lots of sense.

Putting all your applications, spread across multiple teams and ownership boundaries, into a single repository?

Absolutely not, this way leads to madness and coupling hell.

Modern software development makes use of hundreds of thousands of open-source software libraries and in some cases full applications. This page is written primarily for a non-technical audience and will presume no prior knowledge of the landscape - so questions, however small, are welcome.

This piece was originally written to help non-technical internal audiences better understand how open-source interacts with traditional procurement processes in the Enterprise. If you’re a technical person looking for some literature to help your organisation understand the impact open-source has on it or are a non-technical person looking to gain an understanding of the same - then this piece is hopefully for you.

What is Open-Source Software?

From Wikipedia:

“Open-source software (OSS) is computer software that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose. Open-source software may be developed in a collaborative public manner. Open-source software is a prominent example of open collaboration, meaning any capable user can participate online in development, making the number of potential contributors’ indefinite. The ability to examine the code facilitates public trust in the software.”

Open-source software is an exceptionally prominent type of software in the world today, with many languages, frameworks, and even programming tools being open source.

Open-source projects are mostly maintained by groups of volunteers often with corporate sponsorship or managed by companies, who provide the software to sell either consulting or support contracts.

In its most common form, open-source software is shipped as software libraries that can be incorporated into your own applications, by your own programmers.

Open-Source Software and Your Applications

Even if you’re not aware of it, most organisations use tens to hundreds of thousands of open-source libraries in the software that they build. This is the default position in technology - to labour the point - the world’s most popular web framework, React - which many organisations build all this webapps in - is both open-source, and depends on 1391 additional open-source packages. So, to build any of those web applications, just to start, would be using 1392 open-source packages. For every website.

The proliferation of open-source in this manner can appear uncontrolled if you’re not familiar with the ecosystem, and in this document, we’ll cover some of the strategies organisations use to adopt open-source safely. Most of these processes should be pleasantly unremarkable and are standard across the technology industry.

Open-Source and Licensing

By default, it’s common for teams to be permitted to use open-source software with permissive licenses. This means, licenses where there is no expectation of the organisation sharing their own software or modifications - common examples of these licenses include, but are not limited to the MIT license, and the Apache2 license. It’s also common to see organisations license open-source software that is available to businesses under commercial terms, in lieu of contributing code back.

These dual-licensed pieces of open-source effectively allow organisations to pay, instead of contributing, and this model is used to help open-source projects become sustainable.

Because authoring open-source software in a sustainable manner is often a challenge - the projects maintainers or core contributors are frequently working for free - a culture of commercial licensing, consulting, and support contracts has grown around open-source software.

Commercial Support for Open-Source Software

If this sounds confusing, it’s because it often is!

It’s possible, and common, for multiple commercial organisations to exist around supporting open-source software. For the most part, because of the nature of open source - people don’t opt for support. This is because the prevailing attitude is that if an issue occurs in an open-source library, then the consumers can fix it and contribute those changes back to the community.

However, for certain categories of open-source - commonly when it’s entire open-source applications instead of libraries, it’s prudent to make sure you have a support relationship in place - and even more important where the open-source application occupies a strategic place in your architectural landscape. Furthermore, it’s often the correct ethical position to take, and helps ensure the longevity of projects that organisations depend on.

Quality Assurance of Open-Source Software

As a result of the complex nature of the development of open-source software - that it’s code, developed by unknown persons, and provided as-is for consumption, with no legal assurance whatsoever - it’s important that you surround your use of open-source software with the same software development practices that you’d expect from software you author yourself.

What this means is, that where you consume open-source libraries:

• Surround them with unit and integration level automated testing to verify their behaviour
• Execute static and dynamic security scanning over the libraries (often referred to as SAST and DAST)
• Use security scanning tools that track vulnerabilities and patch versions of the libraries to make sure they’re kept up to date.

The fact that you’re doing these activities makes our use of open-source libraries safe at point of consumption - augmented by the fact that of course, you’ll have full control of what and when you update, and you can read all the code if we so choose.

Procurement and Open-Source Software

With traditional, paid for software, a procurement process is required to license software, and subsequently start using and developing with the application or library. With open-source, the software just… exists. It can be included from public package management sources (the places that you download the software to use), and development can start with it immediately.

However, this inadvertently circumvents and entire category of assurance that would exist in traditional procurement processes around software. With most of the open-source software that folks use, the burden of quality and assurance is on your own teams at the point of integration. This means that by wrapping the libraries you use in your own software development lifecycle activities like automated testing and scanning, you treat those libraries exactly as if they are code that you author.

But there exists three kinds of open-source as oft consumed:

• Open source for which you do not require a support contract and can test/integrate yourself
• Open source that you desire support for from a maintainer or organisation providing those services
• Open source that is available commercially under dual-licensing - requiring payment for commercial use

For the first category of open-source, you are on your own, and your own internal development processes must be enough. For the two subsequent categories of open-source, in Enterprise, you will inevitably require a procurement and assurance process.

Important Questions for Open-Source Support

Traditional assurance process often centres on the organisation providing the support - their policies, their processes, their workplace environment, and their security. In open-source projects these questions are often nonsensical; there are no offices, there are no staff, the work is done at the behest of contributors. But as you evaluate support contracts with vendors, it’s important to understand their relationship with the product and what kind of support and assurance that they can provide for the payments you make to them.

Here are some sample questions that might replace more traditional assurance questions in open-source procurement processes.

“What relationship do you have with the project?”

Ideally, you would purchase support from the core authors of the project, or the organisation formed around it.

Good answers: “we are the core maintenance team”, “we are core contributors”, “we have commit rights”

Bad answers: “None”, “we are a consultancy providing extra services with no relationship with the project”

“Do you have a process in place to vet contributors?”

Code changes should be reviewed by maintainers.

Good answers: “Changes go through our pull request and testing process, and are accepted by maintainers”

Bad answers: “None”, “we take any contributions without inspection”

“Do your contributors sign any legal agreements formally granting ownership of their contributions to the project?”

Larger open-source projects, or corporate sponsored projects, often have paperwork to ensure that there can be no copyright claims made against the software by employers of contributors. For example, Microsoft use the Contributor License Agreement to ensure this.

While not exceptionally common, it’s a good question to ask.

“Do you have the ability to integrate any bug fixes required back to the main project source?”

You would ideally buy a support license where any bug fixes were integrated into the main project, and not a fork for your own benefit.

Whilst this might sound counter-productive, this ensures that you don’t end up using a “forked”, or modified version of the main project that can drop out of active development.

“Are there any special feature request processes or privileges that come with this support contract?”

Ideally, if you are buying a support contract, it would give us priority in feature requests.

“Do you, and if so, how do you, provide security scanning and exploit prevention in the software?”

Even open-source software is expected to be tested, have build pipelines, and be reliable - doubly so if you’re paying for support licenses.

“What is the SLA on response to bug reports?”

Equally, it’s important to know the service level you’re expecting from support contracts - to manage expectations between both your internal teams and the support organisation.

“Do you have a process in place to support us in the case of a 0-day vulnerability?”

0-day vulnerabilities are “live, fresh bugs, just discovered” and there should be an expectation that if a support body becomes aware of a bug in the software they are supporting, there would be a process of notification and guidance towards a fix.

Conclusion

The use of open-source software is a huge positive for commercial organisations (who effectively benefit from free labour), and to use it both sustainably and ethically, it’s important to be aware of both your responsibilities around assurance, and those of any partner you choose to engage with.

Paying for software, and open-source, is one of the most ethical ways organisations can interact with the open-source community (short of contributing back!) and should always be considered where possible.

There are several categories of problems that require data to be append only, sequentially stored, and able to expand to arbitrary sizes. You’d need this type of “append only” file for building out a logging platform or building the data storage backend for an Event Sourcing system.

In distributed systems where there can be multiple writers and often when your files are stored in some cloud provider the “traditional” approach to managing these kinds of data structures often don’t work well. You could acquire a lock, download the file, append to it and re-upload – but this will take an increasing amount of time as your files grow, or you could use a database system that implements distributed locking and queuing – which is often more expensive than just manipulating raw files.

Azure blob storage offers Append Blobs, which go some way to solving this problem, but we’ll need to write some code around the storage to help us read the data once it’s written.

What is an Append Blob?

An Append Blob is one of the blob types you can create in Azure Blob Storage – Azure’s general purpose file storage. Append Blobs, as the name indicates, can only be appended to – you append blocks to append blobs.

From Azure’s Blob Storage documentation:

An append blob is composed of blocks and is optimized for append operations. When you modify an append blob, blocks are added to the end of the blob only, via the Append Block operation. Updating or deleting of existing blocks is not supported. Unlike a block blob, an append blob does not expose its block IDs.

Each block in an append blob can be a different size, up to a maximum of 4 MiB, and an append blob can include up to 50,000 blocks. The maximum size of an append blob is therefore slightly more than 195 GiB (4 MiB X 50,000 blocks).

There are clearly some constraints there that we must be mindful of, using this technique:

• Our total file size must be less than 195GiB
• Each block can be no bigger than 4MiB
• There’s a hard cap on 50,000 blocks, so if our block size is less than 4MiB, the maximum size of our file will be less.

Still, even with small blocks, 50,000 blocks should give us a lot of space for entire categories of application storage. The Blob Storage SDKs allow us to read our stored files as one contiguous file or read ranges of bytes from any given offset in that file.

Interestingly, we can’t read the file block by block – only by byte offset, and this poses and interesting problem – if we store data which has any kind of data format that isn’t just plain text (e.g., JSON, XML, literally any data format) and we want to seek through our file, there is no way we can ensure we read valid data from our stored file, even if it was written as a valid block when first saved.

Possible Solutions to the Read Problem

It’s no good having data if you can’t meaningfully read it – especially when we’re using a storage mechanism specifically optimised for storing large files. There are a few things we could try to make reading from our Append Only Blocks easier.

• We could maintain an index of byte-offset to block numbers
• We could pad our data to make sure block sizes were always consistent
• We could devise a read strategy that understands it can read partial or seemingly malformed data

The first solution – maintaining a distinct index, may seem appealing at first – but it takes a non-trivial amount of effort to maintain that index and make sure that it’s keep both in track and up to data with our blob files. This introduces the possibility of a category of errors where those files drift apart, and we may well be in a situation where data appears to get lost, even if it’s in the original data file, because our index loses track of it.

The second solution is the “easiest” – as it gives us a fixed block size that we can use to page back through our blocks – but storing our data becomes needlessly more expensive.

Which really leaves us with our final option – making sure the code that reads arbitrary data from our file understands how to interpret malformed data and interpret where the original write-blocks were.

Scenario: A Chat Application

One of the more obvious examples of infinite-append-only logs are chat applications – where messages arrive in a forwards-only sequence, contain metadata, and to add a little bit of spice, must be read tail-first to be useful to their consumers.

We’ll use this example to work through a solution, but a chat log could be an event log, or a list of business events and metadata, or really, anything at all that happens in a linear fashion over time.

We’ll design our fictional chat application like this:

• A Blob will be created for every chat channel.
• We’ll accept that a maximum of 50,000 messages can be present in a channel. In a real-world application, we’d create a subsequent blob once we hit this limit.
• We’ll accept that a single message can’t be more than 4MiB in size (because that’d be silly).
• In fact, we’re going to limit every single chat message to be a maximum of 512KiB – this means that we know that we’ll never exceed the maximum block size, and each block will only contain a single chat message.
• Each chat message will be written as its own distinct block, including its metadata.
• Our messages will be stored as JSON, so we can also embed the sender, timestamps, and other metadata in the individual messages.

Our message could look something like this:

{
  “senderId”: "foo",
  “messageId”: "some-guid",
  “timestamp”: "2020-01-01T00:00:00.000Z",
  “messageType”: "chat-message",
  “data”: {
    “text”: "hello"
  }
}

This is a mundane and predictable data structure for our message data. Each of our blocks will contain data that looks roughly like this.

There is a side effect of us using structured data for our messages like this – which is that if we read the entire file from the start, it would not be valid JSON at all. It would be a text file with some JSON items inside of it, but it wouldn’t be “a valid JSON array of messages” – there’s no surrounding square bracket array declaration [ ], and there are no separators between entries in the file.

Because we’re not ever going to load our whole file into memory at once, and because our file isn’t actually valid JSON, we’re going to need to do something to indicate our individual message boundaries so we can parse the file later. It’d be really nice if we could just use an open curly bracket { and that’d just be fine, but there’s no guarantee that we won’t embed complicated object structures in our messages at a later point that might break our parsing.

Making our saved messages work like a chat history

Chat applications are interesting as an example of this pattern, because while the data is always append-only, and written linearly, it’s always read in reverse, from the tail of the file first.

We’ll start with the easy problem – adding data to our file. The code and examples here will exist outside of an application as a whole – but we’ll be using TypeScript and the @azure/storage-blob Blob Storage client throughout – and you can presume this code is running in a modern Node environment, the samples here have been executed in Azure Functions.

Writing to our file, thankfully, is easy.

We’re going to generate a Blob filename from our channel name, suffixing it with “.json” (which is a lie, it’s “mostly JSON”, but it’ll do), and we’re going to add a separator character to the start of our blob.

Once we have our filename, we’ll prefix a serialized version of our message object with our separator character, create an Append Blob Client, and call appendBlob with our serialized data.

import { BlobServiceClient } from "@azure/storage-blob";

export default async function (channelName: string, message: any) {
    const fileName = channelName + ".json";
    const separator = String.fromCharCode(30);
    const data = separator + JSON.stringify(message);

    const blobServiceClient = BlobServiceClient.fromConnectionString(process.env.AZURE_STORAGE_CONNECTION_STRING);

    const containerName = process.env.ARCHIVE_CONTAINER || "archive";
    const containerClient = blobServiceClient.getContainerClient(containerName);
    await containerClient.createIfNotExists();

    const blockBlobClient = containerClient.getAppendBlobClient(fileName);
    await blockBlobClient.createIfNotExists();

    await blockBlobClient.appendBlock(data, data.length);
};

This is exceptionally simple code, and it looks almost like any “Hello World!” Azure Blob Storage example you could thing of. The interesting thing we’re doing in here is using a separator character to indicate the start of our block.

What is our Separator Character?

It’s a nothing! So, the wonderful thing about ASCII, as a standard, is that it has a bunch of control characters that exist from a different era to do things like send control codes to printers in the 1980s, and they’ve been enshrined in the standard ever since.

What this means is there’s a whole raft of character that exist as control codes, that you never see, never use, and are almost fathomably unlikely to occur in your general data structures.

ASCII 30 is my one.

According to ASCII – the 30th character code, which you can see in the above sample being loaded using String.fromCharCode(30), is “RECORD SEPARATOR” (C0 and C1 control codes - Wikipedia).

“Can be used as delimiters to mark fields of data structures. If used for hierarchical levels, US is the lowest level (dividing plain-text data items), while Record Separator, Group Separator, and File Separator are of increasing level to divide groups made up of items of the level beneath it.”

That’ll do. Let’s use it.

By prefixing each of our stored blocks with this invisible separator character, we know that when it comes time to read our file, we can identify where we’ve appended blocks, and re-convert our “kind of JSON file” into a real array of JSON objects.

Whilst these odd control codes from the 1980s aren’t exactly seen every day, this is a legitimate use for them, and we’re not doing anything unnatural or strange here with our data.

Reading the Chat History

We’re not going to go into detail of the web applications and APIs that’d be required above all of this to present a chat history to the user – but I want to explore how we can read our Append Only Blocks into memory in a way that our application can make sense of it.

The Azure Blob Client allows us to return the metadata for our stored file:

    public async sizeof(fileName: string): Promise<number> {
        const blockBlobClient = await this.blobClientFor(fileName);
        const metadata = await blockBlobClient.getProperties();
        return metadata.contentLength;
    }

    private async blobClientFor(fileName: string): Promise<AppendBlobClient> {
        this.ensureStorageExists();
        const blockBlobClient = this._containerClient.getAppendBlobClient(fileName);
        await blockBlobClient.createIfNotExists();
        return blockBlobClient;
    }

    private async ensureStorageExists() {
        // TODO: Only run this once.
        await this._containerClient.createIfNotExists();
    }

It’s been exposed as a sizeof function in the sample code above – by calling getProperties() on a blobClient, you can get the total content length of a file.

Reading our whole file is easy enough – but we’re almost never going to want to do that, sort of downloading the file for backups. We can read the whole file like this:

    public async get(fileName: string, offset: number, count: number): Promise<Buffer> {
        const blockBlobClient = await this.blobClientFor(fileName);
        return await blockBlobClient.downloadToBuffer(offset, count);
    }

If we pass 0 as our offset, and the content length as our count, we’ll download our entire file into memory. This is a terrible idea because that file might be 195GiB in size and nobody wants those cloud vendor bills.

Instead of loading the whole file, we’re going to use this same function to parse, backwards through our file, to find the last batch of messages to display to our users in the chat app.

Remember

• We know our messages are a maximum of 512KiB in size
• We know our blocks can store up to 4MiB of data
• We know the records in our file are split up by Record Separator characters

What we’re going to do is read chunks of our file, from the very last byte backwards, in batches of 512KiB, to get our chat history.

Worst case scenario? We might just get one message before having to make another call to read more data – but it’s far more likely that by reading 512KiB chunks, we’ll get a whole collection of messages, because in text terms 512KiB is quite a lot of data.

This read amount really could be anything you like, but it makes sense to make it the size of a single data record to prevent errors and prevent your app servers from loading a lot of data into memory that they might not need.

    /// <summary>
    /// Reads the archive in reverse, returning an array of messages and a seek `position` to continue reading from.
    /// </summary>
    public async getTail(channelName: string, offset: number = 0, maxReadChunk: number = _512kb) {
        const blobName = this.blobNameFor(channelName);
        const blobSize = await this._repository.sizeof(blobName);

        let position = blobSize - offset - maxReadChunk;
        let reduceReadBy = 0;
        if (position < 0) {
            reduceReadBy = position;
            position = 0;
        }

        const amountToRead = maxReadChunk + reduceReadBy;
        const buffer = await this._repository.get(blobName, position, amountToRead);

	  ...

In this getTail function, we’re calculating the name of our blob file, and then calculating a couple of values before we fetch a range of bytes from Azure.

The code calculates the start position by taking the total blob size, subtracting the offset provided to the function, and then again subtracting the maximum length of the chunk of the file to read.

After the read position has been calculated, data is loaded into an ArrayBuffer in memory.

 ...

 const buffer = await this._repository.get(blobName, position, amountToRead);

        const firstRecordSeparator = buffer.indexOf(String.fromCharCode(30)) + 1;
        const wholeRecords = buffer.slice(firstRecordSeparator);
        const nextReadPosition = position + firstRecordSeparator;

        const messages = this.bufferToMessageArray(wholeRecords);
        return { messages: messages, position: nextReadPosition, done: position <= 0 };
    }

Once we have 512KiB of data in memory, we’re going to scan forwards to work out where the first record separator in that chunk of data is, discarding any data before it in our Buffer – because we know that from that point onwards, because we are strictly parsing backwards through the file, we will only have complete records.

As the data before that point has been discarded, the updated “nextReadPosition” is returned as part of the response to the consuming client, which can use that value on subsequent requests to get the history block before the one returned. This is similar to how a cursor would work in a RDBMS.

The bufferToMessageArray function splits our data chunk on our record separator, and parses each individual piece of text as if it were JSON:

    private bufferToMessageArray(buffer: Buffer) {
        const messages = buffer.toString("utf8");
        return messages.split(String.fromCharCode(30))
            .filter(data => data.length > 0)
            .map(m => JSON.parse(m));
    }

Using this approach, it’s possible to “page backwards” thought our message history, without having to deal with locking, file downloads, or concurrency in our application – and it’s a really great fit for storing archive data, messages and events, where the entire stream is infrequently read due to it’s raw size, but users often want to “seek upwards”.

Conclusion

This is a fun problem to solve and shows how you could go about building your own archive services using commodity cloud infrastructure in Azure for storing files that could otherwise be “eye wateringly huge” without relying on third party services to do this kind of thing for you.

It’s a great fit for chat apps, event stores, or otherwise massive stores of business events because blob storage is very, very, cheap. In production systems, you’d likely want to implement log rotation for when the blobs inevitably reach their 50,000 block limits, but that should be a simple problem to solve.

It’d be nice if Microsoft extended their block storage SDKs to iterate block by block through stored data, as presumably that metadata exists under the hood in the platform.

Software development transforms human requirements, repeatedly, until software is eventually produced.

We transform things we’d like into feature requests. Which are subsequently decomposed into designs. Which are eventually transformed into working software.

At each step in this process the information becomes denser, more concrete, more specific.

In agile software development, user stories are a brief statement of what a user wants a piece of software to do. User stories are meant to represent a small, atomic, valuable change to a software system.

Sounds simple right?

But they’re more than that – user stories are artifacts in agile planning games, they’re triggers that start conversations, tools used to track progress, and often the place that a lot of “product thinking” ends up distilled. User stories end up as the single source of truth of pending changes to software.

Because they’re so critically important to getting work done, it’s important to understand them – so we’re going to walk through what exactly user stories are, where they came from, and why we use them.

The time before user stories

Before agile reached critical mass, the source of change for software systems was often a large specification that was often the result of a lengthy requirements engineering process.

In traditional waterfall processes, the requirements gathering portion of software development generally happened at the start of the process and resulted in a set of designs for software that would be written at a later point.

Over time, weaknesses in this very linear “think -> plan -> do” approach to change became obvious. The specifications that were created ended up in systems that often took a long time to build, didn’t finish, and full of defects that were only discovered way, way too late.

The truth was that the systems as they were specified were often not actually what people wanted. By disconnecting the design and development of complicated pieces of software, frequently design decisions were misinterpreted as requirements, and user feedback was hardly ever solicited until the very end of the process.

This is about as perfect a storm as can exist for requirements – long, laborious requirement capturing processes resulting in the wrong thing being built.

To make matters worse, because so much thought-work was put into crafting the specifications at the beginning of the process, they often brought out the worst in people; specs became unchangeable, locked down, binding things, where so much work was done to them that if that work was ever invalidated, the authors would often fall foul of the sunk cost fallacy and just continue down the path anyway because it was “part of the design”.

The specifications never met their goals. They isolated software development from it’s users both with layers of people and management. They bound developers to decisions made during times of speculation. And they charmed people with the security of “having done some work” when no software was being produced.

They provided a feedback-less illusion of progress.

“But not my specifications!” I hear you cry.

No, not all specifications, but most of them.

There had to be a better way to capture requirements that:

• Was open to change to match the changing nature of software
• Could operate at the pace of the internet
• Didn’t divorce the authors of work from the users of the systems they were designing
• Were based in real, measurable, progress.

The humble user story emerged as the format to tackle this problem.

What is a user story

A user story is a short, structured statement of a change to a system. They should be outcome focused , precise, and non-exhaustive.

Stories originated as part of physical work-tracking systems in early agile methods – they were handwritten on the front of index cards, with acceptance criteria written on the reverse of the card. The physical format added constraints to user stories that are still useful today.

Their job is to describe an outcome , and not an implementation. They’re used as artefacts in planning activities, and they’re specifically designed to be non-exhaustive – containing only the information absolutely required as part of a change to a product.

It’s the responsibility of the whole team to make sure our stories are high enough quality to work from, and to verify the outcomes of our work.

Furthermore, user stories are an exercise in restraint. They do not exist to replace documentation. They do not exist to replace conversation and collaboration. The job is to decompose large, tough, intractable problems into small, articulated, well considered changes.

User stories are meant to represent a small, atomic, valuable change to a software system and have mostly replaced traditional requirements engineering from the mid-2000s onwards.

The user story contents

The most common user story format, and generally the one that should be followed by default, was popularised by the XP team at Connextra in 2001. It looks like this:

As a <persona>

I want <business focused outcome>

So that <reason driving the change>

Accept:

• List of…
• Acceptance criteria…

Notes: Any notes

This particular format is popular because it considers both the desired outcome from a user’s perspective (the persona), and also includes the product thinking or justification for the change as part of the “So that” clause.

By adhering to the constraint of being concise, the story format forces us to decompose our work into small, deliverable chunks. It doesn’t prevent us from writing “build the whole solution”, but it illuminates poorly written stories very quickly.

Finally, the user story contains a concise, non-exhaustive list of acceptance criteria. Acceptance criteria list the essential qualities of the implemented work. Until all of them are met, the work isn’t finished.

Acceptance criteria aren’t an excuse to write a specification by stealth. They are not the output format of response documents when you’re building APIs, or snippets of HTML for web interfaces. They’re conversation points to verify and later accept the user story as completed.

Good acceptance criteria are precise and unambiguous – anything else isn’t an acceptance criteria. As an example – “must work in IE6” is better than “must work in legacy browsers”, equally “must be accessible” is worse than “must adhere to all WCAG 2.0 recommendations”.

Who and what is a valid persona?

Personas represent the users of the software that you are building.

This is often mistaken to mean “the customers of the business” and this fundamental misunderstanding leads to lots of unnatural user stories being rendered into reality.

Your software has multiple different types of users – even users you don’t expect. If you’re writing a web application, you might have personas that represent “your end user”, “business to business customers”, or other customer architypes. In addition to this, however, you’ll often have personas like “the on call engineer supporting this application”, “first line support” or “the back-office user who configures this application”.

While they might not be your paying customers, they’re all valid user personas and users of your software.

API teams often fall into the trap of trying to write user stories from the perspective of the customer of the software that is making use of their API. This is a mistake, and it’s important that if you’re building APIs, you write user stories from the perspective of your customers – the developers and clients that make use of your APIs to build consumer facing functionality.

What makes a good user story?

While the vast majority of teams use digital tracking systems today, we should pay mind to the constraints placed upon user stories by physical cards and not over-write our stories. It’s important to remember that user stories are meant to contain distilled information for people to work from.

As the author of a user story, you need to be the world’s most aggressive editor – removing words that introduce ambiguity, removing any and all repetition and making sure the content is precise. Every single word you write in your user story should be vital and convey new and distinct information to the reader.

It’s easy to misinterpret this as “user stories must be exhaustive”, but that isn’t the case. Keep it tight, don’t waffle, but don’t try and reproduce every piece of auxiliary documentation about the feature or the context inside every story.

For example:

As a Back-Office Manager
I want business events to be created that describe changes to, or events happening to, customer accounts that are of relevance to back-office management
So that those events may be used to prompt automated decisions on changing the treatment of accounts based on back-office strategies that I have configured.

Could be re-written:

As a Back-Office Manager
I want an event published when a customer account is changed
So that downstream systems can subscribe to make decisions

Accept:
- Event contains kind of change
- Event contains account identifiers
- External systems can subscribe

In this example, edited, precise language makes the content of the story easier to read , and moving some of the nuance to clearly articulated acceptance criteria prevent the reader having to guess what is expected.

Bill West put together the mnemonic device INVEST , standing for Independent, Negotiable, Verifiable, Estimable, Small and Testable – to describe characteristics of a good user story – but in most cases these qualities can be met by remembering the constraints of physical cards.

If in doubt, remember the words of Ernest Hemingway:

“If I started to write elaborately, or like someone introducing or presenting something, I found that I could cut that scrollwork or ornament out and throw it away and start with the first true simple declarative sentence I had written.”

Write less.

The joy of physical limitations

Despite the inevitability of a digital, and remote-first world, it’s easy to be wistful for the days of user stories in their physical form, with their associated physical constraints and limitations.

Stories written on physical index cards are constrained by the size of the cards – this provides the wonderful side effect of keeping stories succinct – they cannot possibly bloat or become secret specifications because the cards literally are not big enough.

The scrappy nature of index cards and handwritten stories also comes with the additional psychological benefit of making them feel like impermanent, transitory artefacts that can be torn up and rewritten at will, re-negotiated, and refined, without ceremony or loss. By contrast, teams can often become attached to tickets in digital systems, valuing the audit log of stories moved back and forth and back and forth from column to column as if it’s more important than the work it’s meant to inspire and represent.

Subtasks attached to the index-card stories on post-it notes become heavy and start falling apart, items get lost, and the cards sag, prompting and encouraging teams to divide bloated stories into smaller, more granular increments. Again, the physicality of the artefact bringing its own benefit.

Physical walls of stories are ever present, tactile, and real. Surrounding your teams with their progress helps build a kind of total immersion that digital tools struggle to replicate. Columns on a wall can be physically constrained, reconfigured in the space, and visual workspaces built around the way work and tasks flow, rather than how a developer at a work tracking firm models how they presume you work.

There’s a joy in physical, real, artefacts of production that we have entirely struggled to replicate digitally. But the world has changed, and our digital workflows can be enough, but it takes work to not become so enamoured and obsessed with the instrumentation, the progress reports, and the roll-up statistics and lose sight of the fact that user stories and work tracking systems were meant to help you complete some work, to remember that they are the map and not the destination.

All the best digital workflows succeed by following the same kinds of disciplines and following the same constraints as physical boards have. Digital workflows where team members feel empowered to delete and reform stories and tickets at any point. Where team members can move, refine, and relabel the work as they learn. And where teams do what’s right for their project and worry about how to report on it afterwards, find the most success with digital tools.

It’s always worth acknowledging that those constraints helped give teams focus and are worth replicating.

What needs to be expressed as a user story?

Lots of teams get lost in the weeds when they try to understand “what’s a user story” vs “what’s a technical task” vs “what’s a technical debt card”. Looking backwards towards the original physical origin of these artefacts it’s obvious – all these things are the same thing.

Expressing changes as user stories with personas and articulated outcomes is valuable whatever the kind of change. It’s a way to communicate with your team that everyone understands, and it’s a good way to keep your work honest.

However, don’t fall into the trap of user story theatre for small pieces of work that need to happen anyway.

I’d not expect a programmer to see a missing unit test and write a user story to fix it - I’d expect them to fix it. I’d not expect a developer to write a “user story” to fix a build they just watched break. This is essential, non-negotiable work.

As a rule of thumb, technical things that take less time to solve than write up should just be fixed rather than fudging language to artificially legitimise the work – it’s already legitimate work.

Every functional change should be expressed as a user story – just make sure you know who the change is for. If you can’t articulate who you’re doing some work for, it is often a symptom of not understanding the audience of your changes (at best) or at worst, trying to do work that needn’t be done at all.

The relationship between user stories, commits, and pull requests

Pull request driven workflows can suffer from the unfortunate side-effect of encouraging deferred integration and driving folks towards “one user story, one pull request” working patterns. While this may work fine for some categories of change, it can be problematic for larger user stories.

It’s worth remembering when you establish your own working patterns that there is absolutely nothing wrong with multiple sets of changes contributing to the completion of a single user story. Committing the smallest pieces of work that doesn’t break your system is safer by default.

The sooner you’re integrating your code, the better, regardless of story writing technique.

What makes a bad user story?

There are plenty of ways to write poor quality user stories, but here are a few favourites:

Decomposed specifications / Design-by-stealth – prescriptive user stories that exhaustively list outputs or specifications as their acceptance criteria are low quality. They constrain your teams to one fixed solution and in most cases don’t result in high quality work from teams.

Word Salad – user stories that grow longer than a paragraph or two almost always lead to repetition or interpretation of their intent. They create work, rather than remove it.

Repetition or boiler-plate copy/paste – Obvious repetition and copy/paste content in user stories invents work and burdens the readers with interpretation. It’s the exact opposite of the intention of a user story, which is to enhance clarity. The moment you reach for CTRL+V/C while writing a story, you’re making a mistake.

Given / Then / When or test script syntax in stories – user stories do not have to be all things to all people. Test scripts, specifications or context documents have no place in stories – they don’t add clarity, they increase the time it takes to comprehend requirements. While valuable, those assets should live in wikis, and test tools, respectively.

Help! All my stories are too big! Sequencing and splitting stories.

Driving changes through user stories becomes trickier when the stories require design exercises , or the solution in mind has some pre-requirements (standing up new infrastructure for the first time etc. It’s useful to split and sequence stories to make larger pieces of technical work easier while still being deliverable in small chunks.

Imagine, for example, a user story that looked like this:

As a customer I want to call a customer API To retrieve the data stored about me, my order history, and my account expiry date

On the surface the story might sound reasonable, but if this were a story for a brand new API, your development team would soon start to spiral out asking questions like “how does the customer authenticate”, “what data should we return by default”, “how do we handle pagination of the order history” and lots of other valid questions that soon represent quite a lot of hidden complexity in the work.

In the above example, you’d probably split that work down into several smaller stories – starting with the smallest possible story you can that forms a tracer bullet through the process that you can build on top of.

Perhaps it’d be this list of stories:

• A story to retrieve the user’s public data over an API. (Create the API)
• A story to add their account expiry to that response if they authenticate. (Introduce auth)
• A story to add the top-level order summary (totals, number of previous orders)
• A story to add pagination and past orders to the response

This is just illustrative, and the exact way you slice your stories depends heavily on context – but the themes are clear – split your larger stories into smaller useful shippable parts that prove and add functionality piece by piece. Slicing like this removes risk from your delivery , allows you to introduce technical work carried by the story that needs it first, and keeps progress visible.

Occasionally you’ll stumble up against a story that feels intractable and inestimable. First, don’t panic, it happens to everyone, breathe. Then, write down the questions you have on a card. These questions form the basis of a spike – a small Q&A focused time-boxed story that doesn’t deliver user-facing value. Spikes exist to help you remove ambiguity, to do some quick prototyping, to learn whatever you need to learn so that you can come back and work on the story that got blocked.

Spikes should always pose a question and have a defined outcome – be it example code, or documentation explaining what was learnt. They’re the trick to help you when you don’t seem to be able to split and sequence your work because there are too many unknowns.

Getting it right

You won’t get your user stories right first time – but much in the spirit of other agile processes you’ll get better at writing and refining user stories by doing it. Hopefully this primer will help you avoid trying to boil the ocean and lead to you building small things, safely.

If you’re still feeling nervous about writing high quality user stories with your teams Henrik Kniberg and Alistair Cockburn published a workshop they called “The Elephant Carpaccio Exercise” in 2013 which will help you practice in a safe environment. You can download the worksheet here - Elephant Carpaccio facilitation guide (google.com)