Matt Sweetman

Creating dynamic mock data with dummy-json

Dummy JSON is a little Node utility I wrote over a year ago but have used countless times since. I thought I'd write about here in the hope that other people would take a look and perhaps help contribute. The github page for the project explains it best, so I'll just quote it here:

Dummy JSON is a Node utility that allows you to generate dynamic JSON data using Handlebars templates. It returns a JSON compatible string you can use in your app.

The following example helps explain it further, showing you the template input and the generated output:

{
  "people": [
    {{#repeat 2}}
    {
      "id": {{index}},
      "firstName": "{{firstName}}",
      "lastName": "{{lastName}}",
      "email": "{{email}}",
      "work": "{{company}}",
      "age": {{number 20 50}},
      "optedin": {{boolean}}
    }
    {{/repeat}}
  ],
  "images": [
    {{#repeat 3 6}}
    'img{{index}}.png'
    {{/repeat}}
  ],
  "revision": {{uniqueIndex}},
  "tolerance": {{number '0' '2'}},
}
{
  "people": [
    {
      "id": 0,
      "firstName": "Leanne",
      "lastName": "Flinn",
      "email": "leanne.flinn@unilogic.com",
      "work": "Unilogic",
      "age": 26,
      "optedin": true
    },
    {
      "id": 1,
      "firstName": "Edward",
      "lastName": "Young",
      "email": "edward.young@solexis.com",
      "work": "Solexis",
      "age": 31,
      "optedin": false
    }
  ],
  "images": [
    'img0.png',
    'img1.png',
    'img2.png',
    'img3.png'
  ],
  "revision": 0,
  "tolerance": 1.7508240924216807,
}

The template is written using Handlebars. The keywords enclosed in curly braces are custom Handlebars helpers written specifically for dummy-json. They repeat blocks of text and generate random numbers, names, emails, and so on. Because they're just regular Handlebars helpers it means they're easily extensible, and adding functionality to dumm-json is extremely easy. You can find more information here: https://github.com/webroo/dummy-json#advanced-usage

Using as a mock data source

This is perhaps the best way to use the utility, and something I've used several times to good effect. The idea is to set up a mock server that simulates your real API and then use dummy-json to return your JSON data. You can easily set up a local server that returns mock data using something like express. The following example shows you how easy this is to do:

var fs = require('fs');
var express = require('express');
var dummyjson = require('./dummy-json');

var template = fs.readFileSync('template.hbs', {encoding: 'utf8'});
var app = express();

app.get('/people', function(req, res) {
  res.set('Content-Type', 'application/json');
  res.send(dummyjson.parse(template));
});

app.listen(3000);

Future additions

There are many helpers I'd like to add, but unfortunately I don't have time to write them at the moment. I'd love to see something that generates dates, addresses and lists of countries. I'd also like to add a random seed, so that the generated values can be consistent if necessary. I'm happy to accept pull-requests if anyone is willing to write these and contribute!

Take a look at the github page and check it out! https://github.com/webroo/dummy-json


Using the new RequireJS bundles feature

A couple of months ago a new feature was added to RequireJS that allows you load modules from something called 'bundles'. Bundles are collections of modules, usually optimized and minified. RequireJS will only load a specific bundle the moment it needs a module from inside. This might not sound like a very exciting addition, but personally it solves a problem I've been having with a several large and javascript-heavy SPAs (single page apps).

Optimizing and minifying your modules down into a single .js file is considered best practice, and it's very useful - it reduces HTTP requests, something of a precious commodity on mobile devices. However if your project grows in size you could find your optimized js file getting larger and larger. I know of a couple of instances where it's been over 0.5MB. That's a lot of minified javascript. Not only is it a lot to load over a mobile connection it's also a lot for the browser to parse in one go. I've seen older devices choke and lock-up temporarily when trying to deal with it.

One approach to this problem is to structure the site differently, perhaps split it into several smaller SPAs, each served on it's own page. But RequieJS bundles offer us another way.

Using bundles

Bundles allow you to group together your modules into a collection of optimized script files. RequireJS then only loads each bundle when the app needs a module inside it. For example, you might have a bundle for each complex page of the site - taking this website as an example: 'homepage', 'articleslist', 'projects', etc. and a 'shared' bundle for the site wrapper and any utilities used across the others pages.

Bundles can be defined using the requirejs config object:

requirejs.config({
  bundles: {
    'shared': ['shared/main', 'shared/util', 'shared/site', 'text!shared/templates/site.html'],
    'homepage': ['homepage/home', 'text!homepage/templates/home.html'],
    'articleslist': ['articleslist/articles', 'text!articleslist/templates/articles.html']
    // etc...
  }
});

This is a very simplistic setup, but it tells RequireJS which modules are found in which bundle. When the site is running and it tries to load a module it will look in this config to figure out which bundle to load it from. If the bundle isn't already loaded requirejs will load it. This means your site only loads the minimum bundles it needs to show a page while still benefiting from using optimized and minified scripts.

If you've got a SPA with just one entry page you'll want to load the modules at runtime to get the most out of this. In the following example the module 'site' contains the site-wide menu, and will respond by loading the relevant section when it's clicked:

define(['site'], function(site) {
  // On clicking to navigate to the articleslist load the module at runtime:
  require(['articles'], function(articles) {
    // RequireJS has now loaded the articles module from the articleslist bundle
  });
});

Note: if you're relying on runtime loading of modules you'll want to use require.js, not almond, as almond deliberately can't load modules across the network (to keep the plugin small and simple).

See the RequireJS docs for more info on how to configure bundles.

Creating the bundles

This is the more complex part of the process, and I recommened knowing a little bit about how the optimizer works before continuing.

You'll need to create each bundle using the RequieJS optimizer, ensuring that you only include the modules you need in each. For this process I used the Grunt requirejs optimizer task grunt-contrib-requirejs.

To make your life easier it's worth separating the individual modules into their own folders, one for each bundle (articleslist, homepage, shared):

src/
|-- app/
    |-- articleslist/
    |   |-- articles.js
    |   |-- templates/
    |       |-- articles.html
    |-- homepage/
    |   |-- home.js
    |   |-- templates/
    |       |-- home.html
    |-- shared/
        |-- main.js
        |-- site.js
        |-- util.js
        |-- templates/
            |-- site.html

You then need to create multiple requirejs grunt tasks to optimize each of the bundles into a minified file. The following is a quick example of how to generate one of these bundles, the 'homepage':

requirejs: {
  homepageBundle: {
    options: {
      insertRequire: false,
      baseUrl: src/app/',
      mainConfigFile: 'src/app/homepage/main.js',
      out: outputFolder + 'out/app/homepage.min.js',
      include: [
        'home',
        'text!templates/home.html'
      ],
      exclude: [
        'shared/main',
        'shared/site',
        'shared/util',
        'text!shared/templates/site.html'
      ]
    }
  }
}

You'll notice that you need to explicitly exclude the shared modules for each bundle. This is because your dependency chains will inevitably pull in shared modules across the codebase, and there's no point including them in each bundle. (See the following section for more info on this.) Also note that you obviously don't want to exclude the shared modules from the shared bundle, just leave the exclude option out in this case.

The only problem with this is the duplicated list of excluded files. It isn't very DRY. And if you have large bundles with dozens of file it isn't exactly clean either. So here's a little function that creates a list of files based on a given folder, allowing you to dynamically generate the include and exclude options.

function generateIncludes(baseUrl, glob) {
  // Join together the path and glob so we can use grunt.file.expand to get an array of files
  var fullGlob = path.join(baseUrl, glob);
  var files = grunt.file.expand({filter: 'isFile'}, fullGlob);

  var includes = files.map(function(url) {
    // Strip off the baseUrl, catering for the fact there may or may not be a trailing slash
    url = url.replace(new RegExp(baseUrl + '/?'), '');
    var ext = path.extname(url);

    if (ext === '.html') {
      // If the extension is .html we need to prefix the include with 'text!'
      url = 'text!' + url;
    } else if (ext === '.js') {
      // If the extension is .js then we need to strip the extension off
      url = url.replace(new RegExp('\\' + ext + '$'), '');
    }

    return url;
  });

  return includes;
}

Now you can simplify the grunt task like so, keeping things neat and tidy:

requirejs: {
  homepageBundle: {
    options: {
      insertRequire: false,
      baseUrl: src/app/',
      mainConfigFile: 'src/app/homepage/main.js',
      out: outputFolder + 'out/app/homepage.min.js',
      include: generateIncludes('src/app/homepage', '**/*'),
      exclude: generateIncludes('src/app/shared', '**/*')
    }
  }
}

Being careful with dependencies

The last thing to be aware of is dependency chains. The RequireJS optimizer follows your dependencies and includes every module it finds along the way. If you're not careful it's easy to create dependencies between different bundles. The simple rules are: The 'shared' bundle should only ever depend on other modules within 'shared'. Other bundles should only ever depend on modules within 'shared' and themselves.


New website built with Haggerston

The latest version of this site was built using Haggerston, a static site generator developed in Node.js by myself and Kelvin Luck. It was named after the place it was built in.

For the past few years this site has been powered by Wordpress, but after several hacking attempts and an endless battle with spam I decided it was time to change. The new site is served as flat HTML - no PHP or databases in sight. I write the articles in markdown, run Haggerston, then sync the files to the web server. It's fast, easy to update, and security issues are kept to a minimum.

Leveraging the power of Grunt

Haggerston is built as a Grunt task. The reason we chose Grunt is because it can already do a lot of what is needed to generate a static site. It can copy files and folders, compile CSS, rsync to remote hosts, start local servers, and watch for file changes. The only task that's missing is one to generate HTML pages. This is where Haggerston comes in.

Haggerston is a relatively simple task, it's job is to turn a folder of .json files into .html files. Each json file specifies a template to use when rendering the html. If you've used javascript templating libraries before then the process should be familiar to you. All Haggerston is doing is turning templates into html files, and using the json files as data for the templates. Haggerston uses Swig as its templating language.

A basic example

Using Haggerston is as simple as dropping in an empty task into your Grunfile.js:

module.exports = function(grunt) {
  grunt.initConfig({
    haggerston: {}
  });

  grunt.loadNpmTasks('grunt-haggerston');
};

This makes the following assumptions: source files exist in a subfolder called src/content, templates exist in src/templates, and the generated site will be created in a subfolder called out. Each of these properties can be overridden in the config if necessary.

Let's take the url for this article as an example. The page exists at /articles/new-website-built-with-haggerston/, which started life in a source folder on my local machine like so:

src
 |-- content/
 |-- articles/
 |  `-- new-website-built-with-haggerston/
 |     `-- index.json
 `-- templates/
    `-- article.html

Running grunt haggerston from the command line will take each .json file from the content folder, render it against a chosen Swig template from the templates folder and export the .html into a corresponding structure in the output folder:

out
 `-- articles/
    `-- new-website-built-with-haggerston/
       `-- index.html

You choose which template to render the data against by specifying it in each json file. Let's take a look at the index.json file of this page as an example:

{
  "template": "article.swig",
  "templateData": {
    "title": "New website built with Haggerston",
    "date": "2013-04-27",
    "blurb": "How I built my new website with Haggerston and Grunt",
    "content": "index.md"
  }
}
  • "template" specifies which Swig template will be used to render the .html file. This is the only mandatory property of a Haggerston .json file.
  • "templateData" is the data that will be sent to the Swig template when it's rendered. You can put anything you like inside it, or omit it entirely.

This process is extremely simple, but the key to making a good static site is knowing how to use templates. Swig supports something called template inheritance which allows templates to inherit and override blocks of markup from their parent. I won't go into detail here, but check out the Swig documentation for an example of how to use it.

Using the concept of middleware

Each step in the Haggerston process is designed as middleware, an idea borrowed from connect that allows developers to intercept the flow of a process and manipulate the data being worked upon.

You may have noticed a property in the index.json example above:

"content": "index.md"

Haggerston is designed to find properties with values that end in '.md'. Once it finds them it tries to locate the referenced markdown file, parse it into html, and replace the original property with the result. This means that when you use {{content}} in your template you're actually getting the html generated from the index.md file, not the string 'index.md'.

The part of Haggerston that does this is called the Markdown middleware. It's simply a function that gets called during the overall Haggerston Grunt task. Each middleware acts the same, and the function is given access to all the json data and rendered templates, meaning it can manipulate the pages in whatever way it likes before they're written to disk.

The basis for a piece of middleware looks like this:

module.exports = function() {
  return function (pages, next, options) {
    // Here you can manipulate the pages array in whatever way you want.
    // Afterwards just call next(pages) to continue with the Haggerston process.
    next(pages);
  };
};

Haggerston comes with a default set of middleware modules that performs common tasks. Each middleware is executed sequentially, and is top-and-tailed with core tasks that cannot be moved. Here's an example of how the overall process works:

  1. Core process: Read .json files into javascript objects
  2. Middleware: Read markdown files and parse them into html markup
  3. Middleware: Render the template into html markup
  4. Middleware: Highlight any code blocks found in the rendered markup
  5. Core process: Write the rendered markup to .html files.

You can write your own middleware and insert it at any point between the opening and closing core processes.

If you want to know more check out the github repo, or take a look at the source of this website.


Managing nested views in Backbone

Dealing with nested views is an aspect of Backbone that causes more confusion than anything else. It's usually the first problem new users face when trying to build an application, and usually the first thing they'll abstract into some sort of reusable code. Many of these solutions eventually get turned into plugins, which is exactly what I've tried to do with Backbone.Baseview.

The problems

It's worth taking some time to understand the problems associated with managing nested views in Backbone. Even if you don't end up using BaseView then hopefully the following will give you a good idea of how to tackle the problem yourself.

It's extremely easy to unbind DOM events by accident

Watch out for jquery's html(), using it will unbind the events of any attached subviews. This means events inside the subview will cease to work.

initialize: function() {
  this.welcomeView = new WelcomeView();
},
render: function() {
  // On the second time round this will remove the subview and unbind its events...
  this.$el.html(template(templateData));

  // ...so we need to attach the subview and re-bind events every time
  this.welcomeView.$el.appendTo(this.$('.container'));
  this.welcomeView.delegateEvents();
  this.welcomeView.render(); // You may also want to call this, to cascade render
}

Internally .html() calls .empty() which unbinds events on all child nodes. The solution above simply uses the built-in delegateEvents() to re-bind the events. You can also try temporarily detaching the subview from the DOM while you render.

The order in which you attach/render your subviews can make a difference

Sometimes a subview will size itself proportionally based on the dimensions of it's parent. If the subview isn't attached to the parent at the point you call render then you won't be able to read any meaningful dimensions from it. While not a common problem it's certainly one that's very frustrating if you've ever encountered it. It can be especially tricky if you've perpetuated the problem down a deep view hierarchy.

The following example shows how to fix this:

// Common approach, however the view won't be attached to the parent during render
this.$el.append(this.mySubview.render().$el);

// Easily fixed by just attaching the view before rendering
this.$el.append(this.mySubview.$el);
this.mySubview.render();

Remembering to dispose of your subview references

Clearing up your subview references isn't hard, but it's easy to forget, and if you don't remove all the references then you might run into some memory management issues down the road.

Backbone does a good job of automatically cleaning up events in your views when you call remove() - wouldn't it be nice if it did the same for subviews?

The solution

BaseView attempts to break these issues down into two separate solutions:

  1. Attaching subviews - Solves event binding and the attach-then-render problem
  2. TODO: Storing subviews - Solves keeping track of subviews and automatically disposing of them

Attaching subviews

BaseView provides several methods to help you attach views to the DOM:

  • myView.appendTo(element) - Appends myView.$el to the given jquery element
  • myView.prependTo(element) - Prepends myView.$el to the given jquery element
  • myView.replace(element) - Replaces element with myView.$el

Internally each of these functions also re-delegates events and ensures the view is attached before it's rendered. This solves two of the previously discussed problems.

Here's an example of their usage:

var SiteView = Backbone.BaseView.extend({
  initialize: function() {
    // WelcomeView is a subclass of BaseView
    this.welcomeView = new WelcomeView();
  },
  template: function() {
    return '<h1>My Site</h1><div class=".container"></div>';
  },
  afterRender: function() {
    // Attach the subview to the container div, this will also render it
    this.welcomeView.appendTo(this.$('.container'));
  }
});

var siteView = new SiteView();
siteView.appendTo('body'); // Also use appendTo to attach and render the top-most view

Note: You should try to use appendTo() for every view, even the top-most one. This ensures you're taking advantage of the attach-then-render functionality for the entire view hierarchy.

It's worth mentioning that these methods don't store any subview references, you'll still be responsible for keeping track and disposing of them manually.

These methods are designed to maintain consistency with the Backbone API. As you may know Backbone already provides remove(), which is a wrapper around this.$el.remove() that also performs some additional tasks (such as cleaning up events). BaseView simply adds wrappers around other jquery methods (eg: appendTo) and performs it's own tasks.

The plugin is available on github at https://github.com/webroo/backbone-baseview.


Flashkana for iPhone

Flashkana screenshot

Flashkana is now available on the App Store for iPhone. This is the iOS version of my original Android app.

After finishing the Android version I spent some time trying to figure out the best way of porting the app to iOS. Eventually I decided to write it from scratch using the native toolset. I was keen on learning Objective-C and I knew using Cocoa-Touch for the UI would give me the best performance. I even found time to brush up on C while learning about pointers.

My late arrival to iOS development has given the framework plenty of time to mature. Building apps is quite an effortless process, the XCode toolset is rich and powerful, documentation is detailed, and community support is plentiful. It took a fraction of the time to develop compared to the Android version. It was obvious that the Android framework was still finding it's feet when I started. I remember I had to write a custom component to achieve the card-swiping mechanism. In iOS the UIScrollView gave me exactly what I needed, right out of the box, instantly solving a large chunk of the build. (It's worth noting the same type of component is now available in Android.)

The only slight hiccup was the certificate and provisioning process, which is a little byzantine (at least compared to Android and the Play store).

The app is free, so head over to the App Store to check it out.