Matt Sweetman

Latest articles

New version of dummy-json

I'm proud to announce the latest version of dummy-json, my mock JSON generator for Node. It's now officially at version 1.0.0!

This is a significant update with lots of new features. You can now generate all kinds of data from names and addresses to GUIDs and colors. It's also possible to set a random seed, allowing the same results to be generated every time. If you're using dummy-json in your own projects this means you can now write tests against the output.

Under the hood there's been a major refactor, with the entire core being rewritten to take advantage of the latest version of Handlebars. Implementing the random seed also required changes to the underlying architecture, but the final result is a more modular and extensible version of dummy-json.

New helpers

There's over 15 new helpers, many of which have been suggested by people who have used the library. I've also updated some of the existing helpers to make them more powerful. Below is a quick glance at a few of the new additions, and you can take a look at the readme for a complete list.

Numbers

{{number}} has been separated into {{interger}} and {{float}} and now supports an optional formatting string. This allows you to generate currency-like values.

{{int 0 100}} // 43
{{float 10000 50000 '0,0.00'}} // 23,462.88

Date and time

Both these helpers now support a formatting string:

{{date '2010' '2015' 'DD/MM/YYYY'}} // 16/06/2012
{{time '09:00' '17:30' 'h:mm a'}} // 2:08 pm

Addresses

There are a collection of over 100 street names and cities allowing you to generate full addresses. You can provide your own address data if you wish to use different street or city names.

{{int 100 300}} {{street}}
{{city}} {{postcode}}
{{country}}

// 221 Baker Street
// London NW1 6XE
// England

Phone numbers

The new phone number helper allows you to provide a mask to generate numbers in any format:

{{phone}} // 445-295-1044
{{phone "+64 (x) xxx xxxx"}} // +64 (5) 883 4711

Lorem ipsum

This generates a specific number of words of lorem ipsum. It automatically inserts punctuation to create sentences.

{{lorem 6}} //  Orci nisi laoreet maximus, dictum om.

Random seed

The original dummy-json generated random values every time it was run. It turned out there were situations where it was preferable to repeat the results. One of these situations was my ability to write tests against the core dummy-json code. I couldn't do it if the results were different every time. So I replaced the default Math.random generator with a third party library that supports seeds.

By default dummy-json is still random, but now you can set an optional seed:

// The seed can be any arbitrary string value
var result = dummyjson.parse(string, {seed: 'abc123'});

Using the same seed every time will generate the same results. Omitting the seed will generate different results.

Upgrading

There are a few breaking changes to the API and built-in helpers which might require you to update your code. For more information see the upgrading section of the readme.

If you find any bugs or have any ideas for additional features then let me know!


Indeterminate checkboxes in React

I've been working in React recently and once again found myself needing an indeterminate checkbox. I previously mentioned how to do this in Angular. Here's a quick recap: the indeterminate checkbox state can't be set via element attributes (eg: <input indeterminate="false">), but it can be set via JavaScript (eg: inputNode.indeterminate = true. In order to do this you need to get hold of the input DOM node and modify it directly.

If you're using React 0.14+ then this is very simple, all you need to do is use the ref callback attribute:

<input
  type="checkbox"
  checked={props.allItemsSelected}
  ref={input => {
    if (input) {
      input.indeterminate = props.someItemsSelected;
    }
  }}
/>

The ref attribute takes a function that gets called whenever the component is mounted and updated. It passes the attached DOM node as the first argument. The callback also gets called during unmounting, in which case the input argument will be undefined, hence the conditional check. The great thing about the ref callback is it can be provided as an anonymous function, meaning you can keep the surrounding component stateless.

(For older versions of React you'll need to use the ref string attribute and create a stateful component to hook into the lifecycle methods.)

Wrapping it up in a component

The ref callback code is so minimal that I haven't wrapped the checkbox into a component in my own project, but if you'd like a reusable component then this should be all you need:

const Checkbox = props => (
  <input
    {...props}
    type="checkbox"
    ref={input => {
      if (input) {
        input.indeterminate = props.indeterminate;
      }
    }}
  />
);

The component ensures that any other props you pass through will be applied to the input element (using spread attributes). You can use it throughout your code like this:

<div>
  <Checkbox checked={props.allItemsSelected} indeterminate={props.someItemsSelected} />
</div>

Thoughts on architecting a large Angular app

For the last year I've been working on an Angular app for a financial trading platform. It's one of the largest projects I've been involved in and I've learnt a lot about how to develop a single-page app in the process. There are many complex aspects to developing an app of this size, from handling large volumes of data to performance and memory management, but more than anything else it's the architecture that proved to be the biggest challenge.

Angular doesn't officially give much guidance on large-scale app architecture. There's some vague mentions of MVC and a rudimentary Todo app, but scaling these basic ideas doesn't always work, especially when you're dealing dozens of developers trying to write and integrate features simultaneously. While the particular app I worked on started off highly decoupled and encapsulated it soon became apparent that the features were far more interwoven and interrelated than they first appeared. This quickly led to a lot of spaghetti code that tried to connect all the pieces together.

The following article looks at some of the things that were done in order to simplify and make sense of this large-scale architectural problem, and deal with it in a way that allowed the app to scale gracefully in size and complexity.

More directives

As the app progressed I found myself using more and more directives and fewer controllers. By the end of the project the only controllers in the app were the ones attached to routes. These controllers actually became incredibly important, acting as sort of linchpins for the entire app, which I'll explain in more detail later.

The great thing about directives is that they can be very explicit in their API. The data they need and the events they emit can be specified in the bindable HTML attributes. This allows other developers to see what goes into and comes out of them. When carefully encapsulated they can be reused across an app by simply dropping them into another view and binding new data. Controllers on the other hand a less explicit, leading to you to infer knowledge about the surrounding context and assume you know what the controller is affecting when it changes stuff.

When building directives it's worth using the internal controller() and link() functions to separate concerns. The controller function should responsible for business logic, eg: preparing data for display or defining click handlers; and the link function should only contain code that directly manipulates the DOM. If the code doesn't need access the DOM then try to keep it in the controller. This makes it easier for people to reason about the roles of the functions inside your directive. After applying these rules I generally found the majority of directives held their code in the controller.

Managing state

Coordinating state changes across a large app can be extremely complex and is commonly regarded as the source of most UI bugs. An app usually contains deeply nested hierarchies of components, each requiring their own state and perhaps some shared state from further up the tree. If you're not careful in coordinating changes you can be left with stale data lying around in components or partial updates not filtering down the tree. I've seen (and even written myself) complex sequences of events to try and connect disparate parts of the app together and coordinate changes, but it inevitably ends up creating a mess that's hard to debug.

It's important to note that when I'm talking about state I'm talking about two things: the model data, such as a list of products; and stateful non-model data, such as whether a panel is open in the UI. While technically they're slightly different they both end up describing the state of your app when it's rendered on the page.

In a lot of cases you'll need both of these types of state spread throughout your app, and many components will need the same state. For example the logged-in state of the user might be needed by several components so they can render the right label or button. In Angular there's a great temptation to keep state in Services and then inject them into your controllers and directives. It's quick, convenient, and at first seems to like a good solution.

But there's two problems with this: firstly it creates a direct dependency between directives and services, reducing the reusability of your components in other situations; secondly, and more importantly, it gives you the temptation to mutate the state directly from your directives.

In order to understand why the second point is a problem you need to imagine your app on a large scale. As it grows you'll have more and more components trying to mutate state. Sometimes several components will want to update the same state at once, or need another piece of state updated before updating it's own. If all your components are making direct mutations to the same state then you've spread the logic across the app in lots of little bits. This makes it hard to coordinate changes.

Moving state into a centralised location

There's a simple rule that alleviates this problem: code that mutates state needs to happens in as few places as possible. It needs to be predictable, coordinated, and have the ability to modify several related pieces of state at once.

The solution is to move state out of your directives and services and up into strategic points in the component hierarchy. I chose to create controllers at these points, and they were responsible for creating and managing a tree of directive components beneath them.

In order to choose the best point in your component hierarchy to put a controller you need to identify where you can hold state data that affects all child components. To do this find all the components that rely on the same state then go up the tree till you find their common parent (this is usually a lot higher than you think.) This is the point at where you want to place a controller. It will be responsible for holding state for an entire sub-tree of components.

The next point is very important: you should only ever mutate state in this same controller, never in your nested components. Remember our aim is to ensure state changes happen in as few places as possible. But how do we achieve this?

Push state down, bubble events up

All components beneath the stateful component should have data pushed into them. You do this via bindable attributes on directives. You can keep passing down the entire state, or portions of the state, into child components. If one of the child components wants to modify the state it shouldn't do so itself - instead it should signal the intent back up to the controller that owns the state, passing along any necessary information about the modification. The controller will listen to this event and make the change. Then, due to the magic of two-way binding, the hierarchy of components will automatically update and re-render.

There are a couple of ways of signalling back up the component tree, you can use $scope.emit(), which will send an event up the scope tree, or you can open up callbacks on the component's bindable attributes. Both will work, but the second is more explicit and gives you the option of whether to attach the event or not.

Pushing data all the way down and bubbling events all the way up might seem like a laborious task, and will undoubtably mean you have to write lots of bindings at each level, but I can't stress enough how much this will help you in the long run. It means you have a very explicit API for every component, minimising the amount of 'magic' that goes on behind the scenes, and state mutations happen in a predictable, centralized location.

Pure functions

This is something you should be striving to use across your entire codebase. A pure function has two characteristics: it always returns the same output when given the same input, and it doesn't produce any other side-effects other than on the input itself. The biggest benefit of pure functions is they are predictable. Calling them a hundred times with the same input will always return the same output. This makes testing them easy.

Try and use pure functions wherever possible. Services are a good place to start, they can simply contain a collection of functions that are responsible for mutating data. A good portion of directives can also use pure functions, especially those put into the controller section. The stateful controllers are a little harder, as their main responsibility is to update the state held within them, so you're likely to have less pure functions there. But by doing all of this you're helping move the complex and error-prone stateful code to just a few places in your app.

Putting it all together

So to recap here's how the various aspects of the architecture are used and how they communicate with each other:

Services don't hold state, but contain a collection of side-effect free functions that load or mutate state. The state mutating functions should be pure and side-effect free and return the result of the mutation, this makes writing tests for them very simple.

Controllers exist only at strategic points in the component hierarchy, ideally where the state for various child components meet at a common parent. These controllers hold the state and model data in their scope then pass it down the tree of directives beneath. Each directive passes bits of the state down to their children, and so on. Controllers also have handlers for events that bubble up from the child directives, these handlers modify the state according to whatever new data was passed up in the event payload. State modification actually happens through services (as described above) and the result is saved back into the controller again.

Directives accept state and model data through their attributes, never implicitly through the scope chain. They also accept callback handlers in the attributes for events they might want to emit (again never through the scope chain). Internally directives are split into separate controller and link functions. The controller function is responsible for business logic, eg: preparing state or model data for display, or preparing events before emitting them. The link function only ever contains code that manipulates the DOM.

Where possible use pure side-effect free functions. You can use them liberally throughout your services, and frequently inside directives.

When using this architecture it can feel like you're writing a lot of duplicated code - you end up passing down data and bubbling up events through many layers of components - but that's OK. In the long run this explicitness will help with debugging. Knowing that state mutation only occurs in a few keys areas of your app means you can quickly limit the scope of your search.

Sometimes you might find the only logical place to hold state is at the very top of the component hierarchy - and that's OK too. You'd be surprised at how many components rely on the same pieces of state in a large app. In fact I'd go as far to say that in a small app you can have just one state object at the very top.

This particular architecture took many iterations and refactors before it became clear it was a more manageable and scalable approach. But it's by no means a definitive approach. The important thing to take away are some of the core concepts, not necessarily the specific implementation. The two concepts I'd say are worth remembering are: 1) centralize your state mutations, 2) use pure functions. Together they will help make your codebase a more predictable place.


Indeterminate checkboxes in Angular

Checkboxes

By default Angular doesn't provide a binding mechanism for the checkbox indeterminate state, meaning you can't do something like this:

<input type="checkbox" indeterminate="true"> <!-- Doesn't work :( -->

There's actually a good reason for this, and that's because it's not actually settable via regular element attributes. You can however set the indeterminate state via JavaScript:

var checkbox = document.getElementById('my-checkbox');
checkbox.indeterminate = true; // Does work :)

This means wherever you need an indeterminate checkbox you'll have to make sure the surrounding component has access to the DOM node and keeps it up to date. Repeatedly writing this wherever you have a checkbox is not ideal, we want to keep things DRY, we want to abstract it away and expose it just like a regular attribute. Then we can bind to it in our template just like any other attribute.

In order to do this we're going to use an Angular directive, but a special kind of directive that can only be applied to element attributes. This can be achieved by setting restrict: 'A' in the directive config. These type of directives allow you to bolt-on functionality to an existing elements, or even other directives, while retaining access to everything the element already has.

The indeterminate attribute directive

This is a complete example so you can copy the directive below into your own application and use it straight away (you might want to change the module namespace first).

angular.module('myModule').directive('indeterminate', function() {
  return {
    // Restrict the directive so it can only be used as an attribute
    restrict: 'A',

    link(scope, elem, attr) {
      // Whenever the bound value of the attribute changes we update
      // the internal 'indeterminate' flag on the attached dom element
      var watcher = scope.$watch(attr.indeterminate, function(value) {
        elem[0].indeterminate = value;
      });

      // Remove the watcher when the directive is destroyed
      scope.$on('$destroy', function() {
        watcher();
      });
    }
  };
});

An important thing to note is we don't define a custom scope in the directive. This means the directive shares whatever scope is currently available on the element. This is important for two reasons: firstly an element can only have one scope, so this directive won't create a new scope and overwrite the existing one; secondly it means we have access to the existing scope and all the properties available on it. While the second point isn't useful in this example it can be quite powerful in other situations.

To use the directive just bind to the indeterminate attribute (remember that the checked attribute is bound by ng-model):

<!-- The value of indeterminate="" can be bound to any angular expression -->
<input type="checkbox" ng-model="data.isChecked" indeterminate="data.isIndeterminate">

Where is this useful?

The most common use for an indeterminate checkbox is a "select all" at the top of a list of selectable items. If all of the items are selected then the checkbox is ticked, but if only some of the items are selected then the checkbox is indeterminate.

To set the state of a "select all" checkbox you need just two boolean variables. In the example below we're going to call them allItemsSelected and someItemsSelected. Let's also assume you have an array of item objects called $scope.myItems and each item inside has a selected boolean. A couple of simple array methods can help us calculate the selection states:

angular.module('myModule').controller('MyController', function($scope) {

  // We can use the ES5 'every' function to check whether each item in our array
  // has a selected property that equals true
  $scope.allItemsSelected = $scope.myItems.every(function(item) {
    return item.selected === true;
  });

  // We use the same technique to determine if only some of the items are selected
  $scope.someItemsSelected = !$scope.allItemsSelected && $scope.myItems.some(function(item) {
    return item.selected === true;
  });

});

And now we can use them in our corresponding template:

<div ng-controller="MyController">
  <input
    type="checkbox"
    class="select-all"
    ng-model="allItemsSelected"
    indeterminate="someItemsSelected">
</div>

Creating mock JSON data with dummy-json

Dummy JSON is a little Node utility I wrote over a year ago but have used countless times since. I thought I'd write about here in the hope that other people would take a look and perhaps help contribute. The github page for the project explains it best:

Dummy JSON is a Node utility that allows you to generate dynamic JSON data using Handlebars templates. It returns a JSON compatible string you can use in your app.

The following example explains it further, showing the input template and the generated output:

{
  "people": [
    {{#repeat 2}}
    {
      "id": {{index}},
      "firstName": "{{firstName}}",
      "lastName": "{{lastName}}",
      "email": "{{email}}",
      "work": "{{company}}",
      "age": {{number 20 50}},
      "optedin": {{boolean}}
    }
    {{/repeat}}
  ],
  "images": [
    {{#repeat 3 6}}
    'img{{index}}.png'
    {{/repeat}}
  ],
  "revision": {{uniqueIndex}},
  "tolerance": {{number '0' '2'}},
}
{
  "people": [
    {
      "id": 0,
      "firstName": "Leanne",
      "lastName": "Flinn",
      "email": "lflinn@unilogic.com",
      "work": "Unilogic",
      "age": 26,
      "optedin": true
    },
    {
      "id": 1,
      "firstName": "Edward",
      "lastName": "Young",
      "email": "eyoung@solexis.com",
      "work": "Solexis",
      "age": 31,
      "optedin": false
    }
  ],
  "images": [
    'img0.png',
    'img1.png',
    'img2.png',
    'img3.png'
  ],
  "revision": 0,
  "tolerance": 1.7508240924216807,
}

The template on the left is written using Handlebars. The keywords enclosed in curly braces are custom Handlebars helpers written specifically for dummy-json. They generate random numbers, names, emails, and so on. Because they're just regular Handlebars helpers it means they're easily extensible, and adding functionality to dummy-json is extremely easy.

Using as a mock data source

This is perhaps the best way to use dummy-json, and something I've used several times to good effect. The idea is to set up a mock server that simulates your real API and use dummy-json to return mock JSON data. You can easily set up a local server using something like express. The following example shows you how easy this is to do:

var fs = require('fs');
var express = require('express');
var dummyjson = require('./dummy-json');

var template = fs.readFileSync('template.hbs', {encoding: 'utf8'});
var app = express();

app.get('/people', function(req, res) {
  res.set('Content-Type', 'application/json');
  res.send(dummyjson.parse(template));
});

app.listen(3000);

Future additions

There are many helpers I'd like to add, such as dates, addresses, and lists of countries. I'd also like to add a random seed so that the generated values can be consistent. If anyone is interested in contributing new ideas I'd be extremely happy! Take a look at the github page and check it out https://github.com/webroo/dummy-json