HiDef Software

The Single Source of Truth - Really?

Mon, 26 Aug 2019 00:00:00 +0000

In software engineering, there’s often talk of the Source of Truth. Which database is source of truth? Yours or mine? When things get complicated, does it still make sense to think about databases or single systems as the source of truth?

These conversations have all made the assumption that they can decide who has authority over reality. Unfortunately a database is never a source of truth. The word “source” means the point of origin. When thinking about the source of truth we need to think about where the data originates. Let’s look at a few examples.

An eCommerce Website

The website has an Orders database in the website, and you have a call centre platform with it’s own database. It seems like the source of truth would be the website’s Orders database, and the call centre should accept that. But what happens when the call centre wants to take orders too. Does it have to send them through the website so the website can remain the source of truth? That’s obviously coupling and creating the website as a single point of failure.

Stock Management

Your business starts with 1 warehouse. You get stock levels from the warehouse database and import them in to your purchasing database (for admin staff) and sales database (for your customers). Clearly that warehouse is the source of truth.

Now you expand and add another warehouse. The first warehouse cannot be the source of truth any more because it does not know about stock in the new warehouse, so you create a new stock levels database to hold a view of stock across multiple locations. Is THIS database the source of truth? If it becomes out of sync with the warehouse, it can’t be said to be truthful anymore. Saying any of these systems is the source of truth is clearly lieing to yourself.

The Reality

The reality? None of these systems is the source of truth. To really find the source of truth we have to expand our idea of a ‘system’ to include 3rd parties, software and importantly people, who make decisions and measure reality.

For your eCommerce business, the customer is the source of truth for orders. They make the decision. Your website, mobile app or call centre is just the channel by which they place their order.

For the stock management system, the source of truth is the staff in the warehouse, actually counting stock and recording it on their tablets. Both warehouse databases are equally the source of their respective truths.

So the phrase Source of Truth doesn’t refer to which database I have built, it refers to the decision maker and the path by which data about that decision arrives with me.

Thinking Differently

If we stop to think about databases as an authority we can open ourselves up to other opportunities. Instead of deciding where data should live, we can start thinking about how it should get there. This means we can think in terms of integration and extension.

Now, a more useful toolset to use is that of event streams. The events may come from multiple places or go to multiple places, but the stream is where you can find that truth.

Databases become storage utilities, and the event stream becomes the centre of your system. Databases can be populated by reading the stream; notifications and other side effects can be triggered by things happening on the stream.

What does that mean for our examples?

Well, the eCommerce website and the call centre are both sources of orders. Your fulfilment centres, your business intelligence, even your accounts are all reacting to these orders when they happen.

In your stock management system, you might still have a database that shows a view of stock in both warehouses, but it is nothing special. It is merely a projection of the stock keeping events which happened in your warehouses. And any side effects, sending notifications to your purchasing systems or updating business intelligence are triggered not by reading a single central database, bit by the stream.

So what, I hear you say

Changing the way we think is challenging, and there is always more to it than can be written in a single blog post.

But thinking about the events that happened in our businesses, and how data about then flows allows us to build software that model our businesses, not make our business adhere to the software.

Things are more extendable, more decoupled and easier to explain to non-technical colleagues.

So next time you are thinking “where is the source of truth?”, try asking yourself “How does the truth get here?”, “Who cares about it?” and importantly, what event has occurred in your business that we care about?”

You might just find a simpler way to build your software that better represents the reality you are working with.

Be A Grafana/Graphite Power User

Thu, 28 Jun 2018 00:00:00 +0000

Graphite (the time series stats database) and Grafana (the brilliant visualisation tool) both have excellent documentation that tells us how to operate the software, but simply knowing how to use a tool does not mean you truly grok it.

This is a collection of tips and suggestions to help your use of these tools be easier and more meaningful.

Grafana

1. Summarise all graphs by a templated interval to allow easy control of data granularity

Grafana is pretty good at trying to guess the interval at which you want to summarise your data, but it doesn’t always get it right.

Creating a template variable to control the interval of a summarize(…) function on all graphs on your dashboard will let you control the granularity of an entire dashboard with ease.

The default interval (1 minute)

An overridden interval of 1 hour

Trends in the data can be made more obvious by viewing it at a different level of detail.

2. Use Points Draw Mode to make sparse data more visible

On sparse data, data points can appear as lines of zero length and be completely invisible. Enable the Points draw mode to make isolated stats stand out.

I recommend a width of 1 to ensure that the points merely point the data out and do not blend together, hiding the true data.

3. Repeat panels and graphs for similar data

Standardise your stats, then use repeated Panels for repeated stat types. e.g. for each dependency, show timings, counts and errors.

You can use template variables to create a list of options (even discover the available stats from your graphs), then configure a graph or panel to repeat for each item in the list.

Maintaining duplicate graphs will just give you a headache.

4. Share Snapshots

Dashboards and their data are changing all the time. If you share a link to a dashboard, the graphs may be changed or the data may be archived or reaggregated in future, meaning that your link may no longer show the information you wanted to share.

Share a snapshot instead. Grafana will store your data and protect it from change so that it can be referenced from documentation or incident management with confidence.

5. Use non-linear scale for timings to make highly variable data understandable

Linear scale on the Y axis can often hide data. Outliers can cause the fine detail of the data to be invisible.

Using Log2 as the scale for the same data shows that the different data series are not exactly the same, and that smaller, but still significant spikes are more common.

Graphite

1. Never transform null timings to zero

In a sparse data set, it’s tempting to guarantee that your data does not have huge gaps in it. When counting a stat, e.g. requests per second, a gap in the data means that you got no requests. In this case applying transformNull(0) makes sense, filling in the gaps in the visualisation.

Applying this function to timings, however is positively misleading. It would give the impression that, for periods of time, the timings were 0.

It now appears that our timings are very spiky and unreliable, which might not be the case.

A gap in latency stats does NOT mean that the latency was 0. It means that no data was available. The gaps in the visualisation tell us that we are not able to answer any questions during that period.

Omitting transformNull(0) from timing graphs highlights the present and absence of data much more clearly.

Lack of data should not be hidden. Think carefully before transforming data to ensure that the transformation does not obscure the meaning of the data. Graphs are about making information visible, not hiding it.

2. Applying transformNull(0) before aggregation changes the meaning of the aggregation

If you do choose to use transformNull(0), do so before movingAvg(), movingMedian(), or other aggregate functions, otherwise the aggregations will not include the transformed data.

3. Always specify your units

Always specify the unit of time, e.g. summarise(), movingAverage(), the default is the number of data points included.

On sparse data, the last N data points may be 100 seconds or 100 minutes, you won’t know unless you double check the graphs generated against the source data.

Useful links:

Kubernetes February Round Up

Sat, 03 Mar 2018 00:00:00 +0000

Another month is passed, and it seems like activity around Kubernetes is accelerating exponentially.

The New Stack

https://thenewstack.io/ebooks/

The New Stack has a collection of ebooks about adopting Kubernetes in an enterprise environment, covering topics including Monitoring, Security, Orchestration and several others.

What makes a cluster a cluster

https://coreos.com/blog/cluster-osi-model.html

We deal with so many levels of abstraction that it can be hard to see how things should fit together. This interesting comparison with the OSI model can help get things in line.

Developing with Kubernetes

Debugging Crash Loop Backoffs

https://medium.com/kokster/debugging-crashloopbackoffs-with-init-containers-26f79e9fb5bf

It can be infuriating when your container doesn’t start, and doesn’t even produce any logs. Here are some tips to help you figure out what’s going on.

Speed up development by mapping volumes

https://github.com/vapor-ware/ksync

Sometimes the whole Code, Build, Test, Code cycle can be too onerous, especially when you’re trying to get something working by trial an error. ksync helps you sync your local file system with a remote container to make live-changes.

Operating Kubernetes

GKE vs AKS vs EKS

https://blog.hasura.io/gke-vs-aks-vs-eks-411f080640dc

Managed Kubernetes from 3 of the biggest cloud providers. Information on EKS is sadly lacking.

Paul Maddox - Amazon EKS

https://blog.uprightvinyl.co.uk/2018/02/08/aws-builders-day-part-3-kubernetes-on-aws-with-amazon-eks/

Addressing the lack of information about Amazon’s Elastic Kubernetes System, this write up of a talk by Paul Maddox probably covers 90% of what I have seen about EKS.

Kured

https://github.com/weaveworks/kured

Manages node reboots safely, to facilitate things like host OS updates

Providing IAM Roles to Containers

https://github.com/jtblin/kube2iam

When coming from AWS services like EC2, where you choose the permissions you want and the relevant credentials are given to you, kube2iam brings this same pattern down in to your Kubernetes cluster.

KubeAPI AWS Authentication

https://github.com/heptio/authenticator

Initially Kubernetes only grants admin access to a single set of SSH keys. If you want to share API access with your team, and want to use AWS IAM users to manage it, this can help you out.

Kubernetes January Round Up

Fri, 02 Feb 2018 00:00:00 +0000

The first month of the year is gone and I have seen some interesting things come across my plate this year. So, I’ve pulled them together in case anyone else is interested.

Operating Kubernetes

Provisioning

kops

https://github.com/kubernetes/kops

Part of the Kubernetes project itself, kops targets AWS and GCE, allowing you to spin up a production grade cluster in minutes. It is well documented and well supported, which makes it one of the best tools I have seen. Unfortunately it stores configuration and keys in S3 and requires access to them at runtime, which would make it hard to create push-button deployments for test clusters. That said, if you want to run your own cluster in AWS, this should probably be your first port of call.

Kubicorn

https://github.com/kris-nova/kubicorn

Although not an official part of the Kubernetes project, Kubicorn aims to make it easier to provision clusters on more providers than just AWS and GCE, while giving your more flexibility. If you want to spin up a test cluster on A.N.Other cloud provider, this is probably what you want. Be aware though, it is not yet rated production ready, as it is lacking some key aspects, including depth of documentation.

Friendly DNS

https://github.com/kubernetes-incubator/external-dns

Kubernetes provides it’s own DNS service, but this is intended entirely for service discovery, with meaningful but ugly DNS names like my-svc.my-namespace.svc.cluster.local or pod-ip-address.my-namespace.pod.cluster.local. If you want to expose a site or API to users, use external-dns to provision vanity URLs for services or ingress. It supports a number of providers including Route53, AzureDNS, CloudFlare and, once the controller has been deployed, you only need to decorate your services to make us of it.

Custom DNS

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/

As mentioned, Kubernetes’ DNS service is used for internal service discovery. If you have your own private DNS that you want to integrate with, you can easily extend Kubernetes DNS with your own DNS as a fallback.

Developing For Kubernetes

Service Meshes

Linkerd

https://linkerd.io/

Tried and tested by the likes of Monzo, Linkerd can transparently handle retries, time outs, rate-limiting, circuit breaking and the like, removing the need for tangled resilience implementations with Polly.net or custom code. It even has multiple service discovery plugins, for example using consul to resolve service hosts directly, removing the need for internal load balancers.

Istio

https://istio.io/

More than a single service, Istio is an entire service mesh project. Combining resilience, service discovery, load balancing etc. with a TLS secured mesh, capable of being extended outside of your cluster. It’s not as mature as Linkerd yet, but it has lofty goals and broad community support. One to watch.

AWS ELB Config Options

https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

We live in an AWS world. Most AWS configurations provision ELBs to expose services outside the cluster. Here is a handy cheat sheet for configuring your load balancer, e.g. internal only, SSL certs, time outs.

Docker for Mac /w Kubernetes

https://blog.alexellis.io/docker-for-mac-with-kubernetes/

Now Docker for Mac has Kubernetes support built in Alex Ellis, author of OpenFaas, compares it to some of the other options for Kubernetes developer tooling, and a quick view of running OpenFaas (a kubernetes based, vendor agnostic, alternative to Lambda)

Docker for Windows /w Kubernetes

https://goo.gl/LUTFjS

Thanks to Mr. Hanselman for running through the new release of Docker for Windows with one-click Kubernetes built in, as well as a flying example of the Kubernetes Dashboard, and deploying ASP.NET Core in to Kubernetes.

If you want to contribute anything to this list, or my next round up, drop me an email at roundup@hidefsoftware.co.uk.

Getting Started With .NET Core on Mac or Linux

Tue, 03 Jan 2017 00:00:00 +0000

I’ve been writing C# for years and that has meant that I have always been constrained to working with Windows.

Not any more! With the advent of .NET Core and Visual Studio Code I am finally free to move completely to a Unix environment and take advantage of all the tools and technologies that are available in Mac OS or Linux environments.

Here are the steps I took to get my new .NET Core projects up and running.

(I usually work on a Mac, but for the purposes of this article, everything I write here has been tested on a clean Debian installation.)

Install .NET Core

Getting .NET Core environment installed is actually quite simple. Unfortunately each platform has a slightly different approach so you’re best heading over to the .NET Core Download Page and following the instructions for your environment. At the time of writing, the latest version (and the version I will be referencing later on is 1.1.0).

The IDE

Visual Studio Code is a god send. Unlike it’s namesake Visual Studio, Code is fully cross platform (being implemented on top of Github’s Electron platform). It’s also very light weight.

Download Visual Studio Code and install it. This will take minutes as the download package is only 51meg for mac.

There are three points about Visual Studio Code that make it a winner for me:

It’s light weight
It’s got support for many different languages/environments
It’s a fully fledged IDE with debugger and intellisense support.

Once installed, it’s helpful to install the code command so you can launch Code and open files from the command line, e.g. code myfile.cs.

Cmd/Ctrl + P > Install 'code' command

Extensions

There are many extensions but we only need one right now. Hit Cmd/Ctrl + Shift + X to open the Extensions manager and install the C# extension. Once you’ve found the extension you want, it’s a click to install it, and then it’ll ask you to restart the window. You’ll be ready in seconds.

Creating A New Project

Yeoman

I use Yeoman to do alot of the boiler plate when creating a new project. the aspnet-generator is published by the Omnisharp team which, although not endorsed by Microsoft, are closely connected to the Visual Studio Code team, and the C# extension (a core extension for Code) is driven by Omnisharp.

Install Node

Mac with Home Brew

brew install node

Ubuntu/Debian

curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
sudo apt-get install -y nodejs

Install yeoman

npm i -g yo

Install the omnisharp dotnet generator

npm i -g generator-aspnet

Create the Library

yo aspnet
> Class Library
Name: CoreBootstrapLibrary

Create the test Library

yo aspnet
> Unit test project (xUnit.net)
Name: CoreBootstrapLibrary.Tests

Update Yeoman’s .NET version

Yeoman is quite helpful, but it does make a few assumptions that don’t quite fit.

Firstly, the Yeoman generated code references NetCore 1.0.1, but the currently latest version is 1.1.0, so we need to upgrade the test project. In /CoreBootstrapLibrary.Tests/project.json we must change Microsoft.NetCore.App version from 1.0.1 to 1.1.0 so file contains this section:

"frameworks": {
    "netcoreapp1.0": {
        "dependencies": {
            "Microsoft.NETCore.App": {
                "type": "platform",
                "version": "1.1.0"
            }
        }
    }
},

Tidy up Yeoman’s .gitignore files

Secondly let’s remove the .gitignore files from the project folders and put them at the root of the project.

mv CoreBootstrapLibrary/.gitignore ./.gitignore
rm CoreBootstrapLibrary.Tests/.gitignore

Initiate our directory as a git repo

git init
git add .
git commit -m "Initial Commit"

Restore initial dependencies

Finally we need to retrieve all the referenced dependencies, again through the CLI. This will restore all dependencies on all projects in child directories. The first time you run this, it has to download the whole of .NET which, althought not as big as in the old days, still takes a few minutes.

dotnet restore

Add Some Code and Tests

I’m not going to walk you through writing code or tests, but obviously we need to make the test project reference the code project.

In CoreBootstrapLibrary.Tests/project.json:

{
    ...
    "dependencies": {
        ...
        "CoreBootstrapLibrary": {
            "target": "project"
        }
    },
    ...

We can run the tests using the test command:

cd tests
dotnet test

Publishing Nuget Packages with AppVeyor

There are many cloud CI services out there, and finding one which is right for you is like buying a new pair of shoes. However I don’t think I need to explain the meaning of Agile, so I’m going to recommend that you dive in with AppVeyor and then you can change your mind later, if you see fit.

AppVeyor has a nice interface for adding new projects, is configurable using an appveyor.yml file in the root of your project, supports .NET Core natively, and it’s all round pretty good. It even has a free tier for public projects and is quite reasonably priced for private ones too, and mighty fine documentation to boot.

Follow the Documentation for how to set up a new project in AppVeyor, and you’ll have it hooked up to your git repo in no time. Then we just need to get it configured.

By default AppVeyor will discover traditional .Net solutions, build them and run tests. They’re not that slick for .NET Core yet though.

Dropping this script in the root of your repository will tell AppVeyor how to build, test and publish our project as a NuGet package.

appveyor.yml

version: 0.0.{build}
build_script:
- cmd: |
    dotnet restore
    dotnet build CoreBootstrapLibrary
    dotnet pack CoreBootstrapLibrary
test_script:
- cmd: >
    dotnet test CoreBootstrapLibrary.Tests
artifacts:
- path: '**/*.nupkg'

The last bit of AppVeyor goodness is that any *.nupkg files published as artifacts (in this case, our library and it’s symbols) are published to a private NuGet feed for your account. Access control is a bit of a blunt instrument, but for internal teams, or your own private projects, it’s a great starting point.

Consuming Private Nuget Packages

One of the differences of working in a Unix environment, rather than Windows, is that we no longer configure Visual Studio with our NuGet sources. Instead we configure them in our environment. Our per-user NuGet.config file is located at ` ~/.nuget/NuGet/NuGet.config`, so to add our private NuGet source with our newly built packages, we just have to add a PackageSource and a set of credentials.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        ...
        <add key="Appveyor" value="https://ci.appveyor.com/nuget/<my account>" />
    </packageSources>
    <packageSourceCredentials>
        <Appveyor>
            <add key="Username" value="<appveyor email>" />
            <add key="ClearTextPassword" value="<appveyor password>" />
        </Appveyor>
    </packageSourceCredentials>
</configuration>  

Unfortunately I wasn’t able to find any documentation on how to encrypt the password stored in this file, other than that it is possible to do so.

Now we can create an application to consume this package.

Creating a Service

One of the beauties of a Unix environment is that you don’t need to create complicated Windows Services with an installer and a framework like TopShelf (much respect to TopShelf, but it’s filling a gap that should never have existed).

A service can be a simple console application. You can run it on the console yourself to test it, or you can run it using upstart in Ubuntu or inside a docker container in exactly the same way.

So in a new directory for a new project:

yo aspnet
> Console application
CoreBootstrapApp

The new console application can by run using the run command:

cd CoreBootstrapApp
dotnet run

Next we add the dependency for the NuGet package we created in CoreBootstrapApp/project.json:

{
    ...
    "dependencies": {
        ...
        "CoreBootstrapLibrary": "*",
    },

And the rest is up to you. You can now write library code, publish it to a private NuGet feed using AppVeyor and consume it from your own application.

Another time I will write about hosting your new .NET Core application in Docker courtesy of Heroku, and implementing a 12 Factor app using .NET Core.

You can find the code for these example projects here:

And CI for the Library over at AppVeyor

The Caching Antipattern

Sun, 25 Dec 2016 00:00:00 +0000

TL;DR - Caching done badly has bad implications. Try your hardest not to cache data; but if you really do have to, make sure you do it right.

There are only two hard things in Computer Science: cache invalidation and naming things.

– Phil Karlton

Caching?

To make sure we’re on the same page, when I say caching, I am talking about the practice of speeding up your own application by masking slow dependencies by remembering previous responses and using them instead of making another slow call to the dependency.

As mentioned briefly by Phil Karlton in his well known sound byte, caching is a tricky problem. This is compounded by a number of common mistakes which I have seen over the years which have resulted in unnecessary confusion delay.

Common Mistakes

Here are some of the mistakes I have seen and what we should to differently.

Cache on startup

When you know for a fact that your dependencies are so slow they are unusable so you don’t even try at run time. Pre-populating a cache as your application starts up rather than querying your dependent service is just admitting that your dependency is not fit for purpose. If it’s a 3rd party service then maybe there is nothing you can do about that, but all too often this technique is used to avoid having to make your services actually fit for purpose.

The problem with caching like this is that it extends application startup times, making scaling and failure recovery hard or even non-viable.

And even if you are okay with having a service that is slow to start or restart (which you shouldn’t be) it results in a cache which bares no relation to the nature of the data or your service’s usage patterns. A simple cache expiry policy would be meaningless here, as this pattern explicitly exists to avoid ever hitting your dependencies again.

Caching too early

I don’t mean too early in the request life cycle, I mean too early in the development cycle. Many times I have seen developers write code, decide that it is too slow, and stick cache in front of it.

From that point on, the fact that their service is slow is hidden. There is no reason to optimise or improve the solution as the cache will ensure that the second request is much quicker; so why worry, right?

Integrated Cache

What’s the S in SOLID? Single Responsibility. If your caching capability is integrated directly into your service layer and you can’t run without it you’re definitely in breach of this principle. It’s not my place to extol the virtues of this principle here.

Caching everything

Blindly applying caching to every external call to ensure responsiveness without a thought for the implications. Even worse, this approach can result in developers and operators not even knowing that caching is occurring and making assumptions about the reliability of the underlying services that are simply not true.

Recaching

Caching everything, or even just caching a lot, can result in caches that cache caches which cache caches.

On one end of the spectrum, this might result in all internal caches expiring shortly before the very front cache, resulting in a huge waste of time and resources to operate layers and layers of caching which are never used.

At the other end, it can result in effectively summing the cache expiry time of all caches involved, so 10 layers of 30 minutes caches can result in a system serving data which is 5 hours old. How’s that for counterintuitive?

Unflushable Cache

Occasionally cache implementations may back on to data stores like Redis which has management tooling which can be used to flush the cache on demand.

Other implementations, such as hand cranks in memory caches or even caches provided by mainstream frameworks will not expose any cache management tools. This leaves ops with the only option, to restart the service to flush the memory. (Or worse, know enough about the cache implementation to find it’s location on file system and clear it out manually.)

I have seen releases that have taken hours longer than necessary while different members of the team tried to track down cache after cache, flush them with a restart or wait for expiry before moving on to the next layer. All the time with a system taken offline because the system cannot be said to consist within itself.

The implications

Caching data has implications which these mistakes can accentuate as well as new issues we didn’t have to think of before.

Deploying to a heavily cached system can be extremely time consuming while you wait for caches to expire or have to bust every cache you can find. Even in systems such as some CDNs I have worked with which serve traffic to tens of percent of the Internet and are thought to be the leaders in content delivery, flushing content and configuration caches globally can take up to two hours. This doesn’t need to be the case (Fastly can purge their cache in 150ms), and it results in confusion (is the new data being served yet?).

The natural response is to come up with a workaround, usually some cache busting. Let’s just think about that. A work around for a feature that you just implemented. Think about the time, effort and cognitive load just to compensate for time, effort and cognitive load you just spent on caching in the first place.

Debugging a cached system also becomes a challenge as being up to your neck in a tricky debugging session will end up with unrelated things being missed or forgotten about. 3 hours of confusion later you realise that you haven’t actually been testing any of the changes you have been making.

What we should do instead

Don’t cache!

Okay, sometimes caching is your only option. You’re on the web and it’s happening whether you like it or not. But even in this case there are options other than simply slapping on Cache-Control: max-age=xxx.

Know your data

You should know when your data was last modified at the very least. Now you can make use of the If-Modified-Since header. Return 304-not-modified if the data hasn’t changed. Now you can intelligently utilise the the client’s caching capability without sacrificing visibility and control. Using this header will let you serve new content instantly and also cache indefinitely. The best of both worlds. Taking it one step further, if you’re able to version your data (or just generate a hash of your response), you can make use of etags and still interact with the client and apply appropriate logic without the latency related to data transfer.

Optimise for performance, don’t hide bad performance

Invest in profiler tooling. Find out why your application is slow and fix it. Reduce duplicate execution paths. Sort out bad query execution plans. Utilise indexes properly. If you’re using S3 or blob storage for your data you can always build your own index using Redis or the like. Redis is more than a cache. You can use it intelligently and get many benefits without the problems introduced by caching.

Finale

Caching is a useful tool, but can be easily abused without giving any signs of the abuse.

Don’t get involved with caching till the last minute; find any other way you can first. Optimise your application before you use the blunt tool of caching.

If you’ve come across any fundamental problems caused by caching and bad discipline there of, let me know and I can add them to the list.

Can we do better than database migrations?

Sat, 24 Dec 2016 00:00:00 +0000

TL;DR - Database migrations are an antipattern, Model your database and generate diff scripts instead.

What are database migrations?

Migration is about getting from one place to another. Database migrations are about getting our database from one state to another. As we maintain and develop a database application we will want to add and remove new tables and columns, create new stored procedures and modify reference data.

SQL database engines facilitate all of these operations with imperative statements so when playing with the database in your Dev environment we issue a series of these statements to get the database in to the state we want. When we’re happy that we know what we want, we write all of these statements down together (either directly in SQL or using a DSL like ruby or asp migrations) and declare them as a migration from the previous state to the new.

Why do we use them?

The natural parity between making the changes ourselves and writing a migration to make them for us is obvious.

It also fits neatly in to any change control process that demands a list of all changes prior to the change being deployed.

Thirdly in an environment where databases are maintained by developers but owned by DBAs, it gives the DBAs a perfect opportunity to review the exact changes that are being applied.

Similarities

Managing your production environment by managing the transition between different states is reminiscent of version control, where a Version Control System keeps a record of the changes between versions, rather than of the individual states.

Alternatives

In the opening section I mentioned that SQL database engines provide an imperative language for modifying a database’s state (in contrast to the declarative language used to retrieve data). Declarative approaches are useful as they make the primary purpose of the code to communicate the desired behaviour or state, rather than how this should be achieved. Think HTML, CSS or even XML.

The technique of database modelling (using tools such as Microsoft’s database projects, Redgate’s SQL Source Control or Toad Data Modeller) let us work in this way. By defining how you want your database to look rather than how it should be changed, your source code will always tell you what your database looks like at a glance. While these tools provide a UI and maybe even a visual language for describing your database, their real power is in implementing your database changes.

If you have a version controlled model of your database it becomes a complex but conceptually neat task for generating the difference between any database at any version and any source code at any other version. This means that, using one of these tools will allow you to move a database easily from any one state, to any other state.

Disadvantages of Migrations

Migrations only allow you to move one step at a time and (due to the developer mindset of always looking forwards) can be irreversible if you are not vigilant.

Working with a long established project results in a need to apply hundred, if not thousands, of migrations to construct a new database. This can be mitigated by taking a snapshot from your production, but this means you are replicating production back in to test and dev, rather than your source code being authoritative.

Your source code does not describe your database, only how to construct your database. Drupal patches, for example are code which is distributed this way; however, I have seen this cause confusion as looking at the code and the patches can never give you a full picture. Applying your migrations is mandatory to get an idea of the true nature of your database.

Because migrations must necessarily have an order, this order is usually defined by file name convention. This makes managing migrations in a very active project with Pull Requests a tiresome task. Every time a PR is merged ahead of you, you must rename your migration to ensure that it is applied after the newly merged migration. This is essentially duplicating the synchronisation work done inherently by your Version Control System.

In Conclusion

While migrations appear to offer a solution which matches our ways of working and sometimes our change process; they could be considered counterproductive when it comes to communicating intent in our code.

There are a great many tools available to work with migrations but it seems as if these tools are merely automating a process which becomes too unwieldy when we try to do it at any scale. They are not changing the way we work for the better but making an antipattern continue to be viable for longer, while the adoption of these tools can make it harder to change our approach in future.

While simple migration tools which work directly with SQL files or other raw data files offer us little, other migrations tools, e.g. Rails migrations, offer a layer of abstraction away from the dataset underneath. This can be used to provide portability and improve readability, so these tools can help us progress.

Fundamentally though, it appears that we need to think about describing our desired state and using tools to help us get there, as we do in many other areas of our discipline.

We should think long and hard before adopting migrations to manage a database, and we should put serious thought in to modelling as an alternative approach.

Management as Multithreading - Part 2

Thu, 03 Nov 2016 00:00:00 +0000

Previously I wrote about how management of a team and multiple work flows can be viewed as similar to the multithreading problem in software development. I also explored some key multithreading concepts and compared them to work management.

There are more concepts that are worth exploring.

Thread Pooling

When you have a number of long running tasks in software it makes sense to spin up a thread for a task, and when the task is complete, tear that task down. If you are managing an organisation you might do the same; hire a contractor for 6 months to do a task and when the task is complete, let the contractor go.

If you have many different tasks cropping up all the time, the effort of finding a contactor for each task, like the time involved in starting a new thread, can become prohibitively expensive. This is why organisations employ permanent staff which are kept on retainer for when they are needed, and why a thread pool can be used to have threads ready and waiting to perform tasks.

By maintaining a pool of workers or threads, which will will perform tasks on demand we can remove the overhead of beginning and ending threads/employment if we are willing to take on the cost of maintaining these while there is no work available.

In practice an organisation is always able to find work for us to do.

Task Scheduling

In computing the term ‘scheduling’ can mean more than one thing:

Choosing ‘when’ a task runs. Fairly obvious.
Choosing ‘where’ a task runs. If there are a number of environments with different capabilities, a memory set of tasks may be routinely scheduled to run in an environment with a lot of memory, or highly parallel work on an environment with a GPU.

As humans we don’t have RAM or SSDs or GPUs, but we do all have different skills and capabilities.

It is a managers role, or that of a self managing team, to act like the scheduler in a software system, to allocate tasks to the relevant individuals or teams.

I hope that thinking about these familiar concepts in different ways may help us appreciate the nuances of both, and may be also gain a bit of understanding for responsibilities that might be outside our own realm.

Management As Multithreading

Wed, 02 Nov 2016 00:00:00 +0000

What if developers and managers can be models using the same concepts as in multi-threaded software?

As developers we often talk about limiting work in progress. The reasons for this are well understood thanks to books like [The Phoenix Project](https://en.wikipedia.org/wiki/The_Phoenix_Project_(novel%29) and [The Goal](https://en.wikipedia.org/wiki/The_Goal_(novel%29) before it. As a developer I notice that this is not entirely unlike a single thread in my software doing a single piece of work at any one time.

Obviously, in an organisation of any size, you are going to have many individuals in many teams all working towards different goals, and so you must have multiple work streams, even a single work stream is only working on one task at a time. This is a parallel for multithreading; increasing through put by increasing the number of channels available to do work.

Some of the roles of management in an organisation are to ensure that different work streams:

Have work to do
Have access to the resources they need
Timelines align with those of other related teams

If we apply the idea of multi-threading to the space of work stream management, we might find some insight into how teams might work better, and work better together.

Concepts

Forking/Joining

When a program reaches a point where work can be executed concurrently, multiple threads can be created to perform this work concurrently. After work has been done on multiple threads, the program will reach a point where all concurrent work must have been completed before the program may continue.

When a stream of work reaches a point where multiple tasks may be conducted in parallel, e.g. multiple clients must consume the same service, and completion of these tasks must be synchronised, e.g. for a product launch, the manager is essentially forking threads of working, waiting for them to complete and joining these threads again before continuing execution of the piece of work.

From this analogy it seems that it might be worth an experiment in visualising work streams as threads complete with forks and joins so that dependencies might be made visible to the streams involved,and so that the limited number of works streams available (as in a limited number of available threads) can be used to show when work is able to be executed concurrently but cannot be due to other constraints, such as lack of a team/individual to conduct the work until other work has been completed.

Locks

When two threads are trying to operate on the same resource, we often use locks to ensure that they do not interfere with each other. This means that the act of waiting for a resource to become available, be it a response from a request made to another thread or simply the right to access that same resource is a blocking operation. This mirrors the problem of workstreams being blocked when waiting for dependent tasks and resources to become available. Obviously steps are taken to ensure that the team does not remain blocked. Parking a task and continuing on another is a form of asynchronous work.

Preemptive Multi tasking

This is probably the most common form of multitasking as an individual worker. You work on a task until someone interrupts you and asks you to do something else. This does allow you to be highly responsive as other parties do not have to wait for you to finish your task before interacting, but both humans and computer must context switch every time they are interrupted. For computers this overhead is so minor it’s barely even considered, but for humans it is a huge inconveinence and frustration. One of the biggest complaints I hear around the office is “I keep being interrupted, I can’t get any work done”. While this model works well in software for greatly increased flexibility, the trade off of context switching makes this a very painful way of managing concurrent tasks.

Cooperative Multi tasking

In contrast to preemptive multi-tasking, cooperative multi-tasking allows the code or the worker to break multiple long running tasks down into smaller chunks, so that when an individual chunk has been completed, a chunk from a different task can be conducted. This has several parallel switch agile software delivery. Small work items allow management (or the coordinating code) to choose its work dynamically rather than commiting to long periods without the ability to react and change direction. In our work, this may be a change of business or technical direction, in software this may be the decision to stop enumerating a complex data source. This model sacrifices the ability to interrupt a task, also reduces the obstacle of context switching. As humans we are also inclined to be happier about context switching when we have got to a natural break in the work.

The lack of any ability to interrupt tasks can be a problem in software and in our work. Runaway threads/tasks or tasks that have become blocked and never unblocked must be managed. Cooperating multitasking offers you nothing in this area; so while it is a much easier way for us humans to work it is useful to keep preemptive multi-tasking as a weapon on your arsenal in the unfortunate instance that a task or thread gets out of control.

I have explored more of these ideas in Part 2.

Event Handling In Games

Thu, 29 Mar 2012 00:00:00 +0000

Being predominantly an applications programmer, I have a different take on things that people in the games development area might. Event handling is something that one might do quite differently in many different situations. Here are the ideas I’ve had and the pros and cons.

Callbacks

Provided by the language. C#, nowadays, makes extensive use of Funcs, Actions and provides Lambdas to let you do it. Lots of other languages do this too. Even though this is a new concept in some of the higher level languages, only arriving in C# in 2003, C has had this for donkeys years. With C++ probably being the biggest gamedev language this option is widely available for most gamedevs for a long long time, however I have rarely seen it used.

Messages

A more application oriented than game oriented way of handling the problem would be to send messages around the game, be they objects or strings or hash tables which contain the relevant information. This gives us the option to trigger an event and then not care about when or where or how it happens. The advantage in application programming is that you don’t need to worry about handling the event, other code will. Great for chopping things up, but I have never seen this approach used in a game, EVER.

Ad Hoc Handling

The most common use of event handling that I have seen in games, is just handling it as it comes. I understand that this is nice and simple and logical to have your update method identify that some work needs doing and doing it. But:

This buries your events deep in your update method
It can make your update method a huge and unwieldy mass of code
It forces you to process events AS they happen

In a simple game this is probably the easiest option as you don’t have to think about it to go down this route. However as your game grows, you will be tied to doing ALL event work in any given update method, regardless of whether it requires immediate attention or not.

I would probably put myself in favour of a Messaging approach, especially if you are doing any networking. If you can make your game a system which passes messages around from component to component, then passing messages to a component on a remote system via networking is nothing new.

As such, the next game that I work on I intend to approach it as a messaging exercise to see if it as easy as I suspect to transition from a message based game engine to a remote-messaging based game engine (networking).

Does anybody have a different view on it? Does anybody do anything else?

Comments to: http://www.reddit.com/r/gamedev/comments/rj9mf/event_handling_in_games/ please.