Adopting Modular JavaScript

Writing applications that are both maintainable and scalable is a challenge, but one we can more easily overcome with modularisation. The emergence of new module types and powerful tools has furnished JavaScript developers with the freedom and control they need to deliver easy-to-manage and larger scale web products. In this talk, Richard Bultitude explains what modular development is, why you should use it and how you can implement it as a solo dev or within your organisation.

This talk was part of the Front-end London (FEL) March 2016 event.

Introduction [00:00]

Maybe aren’t familiar with Modular JavaScript at all or what even that means, so I’ll cover a bit of that, but also I’ve hopefully got some slightly meatier, juicy stuff for you guys that actually already know about, so that I’m not entirely preaching to the converted, pointlessly.

A little bit about myself; this is me, I think I’m aged 11 there but this is before you had time-coded cameras and stuff. Something around that age. As you can see, I was big into my tech, I’ve got a Casio keyboard. That’s how I became what I am now.

These are some of the things that I’m interested in or I do in my job. My job definitely focuses a lot on best practice because half of my role is about leadership and understanding what the industry does and making sense of that and feeding that into my team. I’ve also got a little personal obsession with web accessibility. I think we should all be making products that can be used by anyone. For the keen eyed of you out there, you might notice I’ve actually got gardening in there. It’s true. Amongst all those other amazing esoteric things, I do actually have an allotment.

Yes, I work at a company called, “Zone.” We are a full service agency, we have offices in London, Bristol, and now Cologne. Weirdly enough, as of tomorrow, I will have been there for four years. This is something of an anniversary.

Without further a due, what I want to explain is a little bit about what it is, what Modular JavaScript is and why you would switch to it if you don’t use it already. A little bit about what the actual APIs are that are out there that you can use. What tools you can actually use to make it happen. Some of the ways we actually go about, where I work, implementing it because it can be a bit of a minefield.

What and why [02:10]

What exactly is it? Let’s maybe start by looking at the actual goals. All the APIs that have been produced over the last whatever, five or six years have kind of got a similar goal. Yes, to encapsulate code, there was a good talk last time about CSS modules and one of the big pluses of that was capsulation and having logically organised files in a similar way to other languages. Other languages obviously had this approach for quite some time. Also, defining dependencies that must load before a module is executed. That’s really, really important and one of the massive persuading factors for switching to this. Essentially, you are ensuring that you don’t have that ridiculous situation where you have script tag somewhere on the page and jQuery loading as the JavaScript run time or the browser sees fit and then another bit of code that relies on it maybe not loading and actually having access to that. This whole structure is to avoid that silliness. Finally, to load smaller, more manageable files and only load them when needed, which some of the things that I’m going to talk about later in the talk do better than others.

What it actually really is, is a design pattern. I suppose in lots of industries where you make things, you have different patterns, different approaches to it. Imagine in engineering and some of the industries, you have different ways of organising things so that you can reuse them. A quick example of the things that I’m going to dip into are having multiple files that are maybe, if you decide so, organised in such a way that you control them with one single file. There are other patterns, more complex ones, but the whole idea of it in a way is to reduce complexity. As nice old Brian Kernighan said, “Controlling complexity is the essence of computer programming.” If you wrote Unix, you probably know what you’re talking about.

This is how it used to be for us. When I started at Zone it was very much like this. We had one awful, horrendous, single, ginormous file of thousands of lines of code in it and no dependency management, poor ordering. If you could be arsed, you could move some functions around to group them with their buddies, but generally no one could be arsed or didn’t have the time. You had really weird close coupling and you had loads of merge conflicts because obviously if you have lots of big files with lots of lines of code, Git or whatever it is you use. The version control is going to have a nightmare actually making sense of all of that, if you have lots of devs working on the same file. It was kind of like this for us. You’d look at a file and what? You couldn’t make any sense of it.

Now, it’s considerably better. We might have on maybe a medium sized CMS driven project 30 smaller files. We have complete dependency control, so any third party plugins, any frameworks, any libraries we’re using are only required when necessary. We have loads less repetition. I used to look through, when I was tech leading on a couple of projects, I used to look through the code that we’d produced in our ginormous single files and actually sometimes find the same function doing the same thing, because one dev would have written it and happily gotten it working and another dev wouldn’t know that it existed and happily got it working, and so it goes on. Consequently, fewer bugs and fewer merge conflicts, but just to point out, with all these approaches, you have a little overhead and I’ll come to why that is later.

Now, we’ve got a nice warm feeling when we work with each other. You know? That’s me. Just to back it up with some evidence, and you should always have a slide like this in a presentation that’s about really full on vertical tech stuff, numbers. Yay. Numbers sell stuff. Stack overflow. Absolutely tons of results for people having loads of headaches with Modular JavaScript. Loads of Git head repos, loads of people have tried to solve the same problem, which is great, and loads of Google results if you Google it. That’s an astonishing number of results, half a million. I guess maybe some of it is porn, spam, or stuff, but you know… it’s a good number. As a consequence, thousands have happy devs. We’ve used this approach on loads of projects, so it’s just a handful of the clients that we work with. It definitely saves our bacon in a couple of circumstances though. The work we did for Tesco and RSBB, more recently, has really benefitted from us being really organised and taking this approach.

Why adopt it [07:10]

For those of you that haven’t adopted it already and I’m guessing some of you have. In fact, let’s do a quick poll. Hands up if you use a Modular JavaScript approach. Okay, so yes, just over half. I’ll skip this bit then. No. Joking. This is why you would do it, I would say. You’re going to get more manageable code. You’re going to be able to separate concerns more easily. One advantage that is probably a bit underrated but we’ve all had headaches with is you protect the name space, so not everything in the whole world is hanging off the window and possibly fighting with other things and you can leverage open source packages more easily. I’ll come to that in a bit but essentially it’s about using things like NPM or other ecosystems that exist out there, and you can do lazy loading. I’ll come to that shortly as well, too.

If you had to sell it to your colleagues for any reason or your boss. Maybe they’re particularly stubborn – I don’t know – but you can definitely make the case that you’re going to have a lot easier on-boarding and handovers with other team members. More easily have parallel work streams because one person could work on one module or one feature which could be a module and another person on another; therefore, you get easier application maintenance. Yes, you get less errors and debugging is easier because you’re able to focus on the problems that exist in one small area.

In one phrase that would be the sale. It’s basically an investment. If you haven’t got a boilerplate or something that is configured like this, put a bit of time in, make it work, and reap the benefits.

The Landscape [08:57]

The landscape, this is the bit that fried my head when I was putting it together, I have to say, because there is quite a big landscape out there. However, I’m going to start off gently. The first thing I think is worth pointing out before we go into any of the major APIs, there is a pattern that’s been around for ages that Addy Osmani has done a great job of documenting, which is called, “The Revealing Module pattern.” It’s not necessarily quintessentially Modular JavaScript but it’s just a way of thinking about JavaScript. We’ll have a quick look at some code. I know that’s a bit scary but essentially you can have a module that’s just a self-calling function and some things in it, they could be objects, they could be functions, they could be properties. You decide what it is you want to return. When you actually want to run it, you’ve essentially decided what you’ve revealed or not and you can call it. One of them isn’t going to work and one of them is. You can have things that run privately, things that happen when you’ve loaded that module, but it’s just a really, really simple idea. It’s a really elegant idea that you just encapsulate something and you decide what you make private and what you make public.

Early on in the thinking about JavaScript patterns, a group formed who were previously called, “Sever JS” and have since become “Common JS”. Now, they created a pattern. They actually wanted this to be the tonic for lots of JavaScript programming woes and they were hoping, I think, that the industry, maybe even JavaScript itself, would adopt their ideas and maybe in a funny kind of way it did. They came up with this idea and it was born out of service side JavaScript, so Node essentially. It’s really, really nice and clean to look at and simple to use. It doesn’t have any wrapping function or anything. You can just decide what dependencies you list and decide what things you export as a module. Funnily enough, it does actually work synchronously so if, in a weird situation you were loading a module which it could get but it was going to take time to acquire, it would actually completely stop the browser from working, temporarily, while it was thinking about that. The whole premise of Common JS is it’s about having a short distance, so all the files that you’re using are in your application and are easily accessible and therefore, you don’t really have a problem with that – you won’t notice it. There is a little bit of an example of Common JS. As you can see, you don’t need anything wrapping anything else, you can just decide what it is that you attach to this module object which it gives you. It could just be a calculation; it could be another function. Yes, it’s really, really clean and straightforward and very nice.

Then shortly after that came AMD, which at the time seemed like the most wonderful thing in the world, I guess. We certainly thought it was. The whole idea of this was rather than the Common JS approach, which is kind of about the server, this was about giving something to front-end devs, people who work in the client, in the browser and designing a system that’s not blocking. When you request access to another module it goes and gets it asynchronously, so it basically puts the phone down and walks away and goes and watches the TV for a bit and comes back and says, “Your module is ready now but it hasn’t broken anything in your app” which is really nice, or can be really nice. But, speaking of wrapping functions, it uses this define wrapper and yes, that has its pluses and minuses. It’s a bit ugly maybe and it’s, yes, it’s more of a BOS, so you have to list the things you want to load and then you have to list the things that you might get back if those things return something. Obviously, I’m using jQuery here and I get back a dollar and then I can use it. It can get pretty messy. We had some really, really fun times with the AMD approach. Even though this talk isn’t supposed to have too much opinion in it, you’re probably going to get some, you know? I’m over this now. I’m over it. I had to shoehorn this in because I think this is a really interesting example of how weird the landscape is. Someone cooked up this idea that, okay, we want to be able to publish modules that use AMD and CJS and actually jQuery plugins and it goes on and on. They made this amazing thing called, “The UMD Wrapper” which you can throw in the top of your module. It’s only really, really small and concise, like this, and really easy to configure.

Wow, I actually contributed to a library recently, where I had to add this and it so wasn’t fun. The real got you is that if it uses a Node style thing, like a Common JS kind of thing, then all these paths will probably work because things like Moment JS or ES6 Promises may exist on NPM and that’s great because it would go and find it or look it up by its name, but if it’s AMD it probably won’t know where it is unless you’ve told it so, so you’ve to configure these paths yourselves. Anyway, it served a purpose, it still serves a purpose but as Scott Andrews says, “It’s not a lie” I mean, no one said it was a lie, right? Anyway, “It’s not a lie, it works but it is a compromise.” He is actually the author of, “Curl JS” which is a library a bit like Require, which I’m sadly not going to feature in this talk. Apologies, Scott.

Yes, here we go. It’s the slide everyone was waiting for, probably. It’s the one I’m so excited to deliver. ES6 modules. ES6 is the panacea for everything, right? Yay ES6. Someone is shaking their head. Okay. Well, I’m going to go with it anyway. It’s good for certain things but it’s got a way to go. Yes, because of the proliferation of these APIs and because they’re managed by different groups and non-standard, yes, JavaScrip adopted it. It’s actually an official part of the spec now, even though almost no browser actually fully can implement it yet. It’s designed for the browser on the server which is really, really nice, so it caters for both of those worlds. The wrapping that I mentioned, it’s implicit so you don’t need to do anything too fancy. It’s really, really concise, I suppose, and pretty to look at. There are lots and lots of really lovely ways that you can import and export. You can do multiple things at once. You can rename modules just using one line. I won’t go too much into it because I’m probably eating into my own time enthusing about it, but yes, it’s pretty smart.

Just to summarise this bit, AMD uses this require and define paradigm and it actually does actually support module exports and exports. You can actually, with requiring with AMD environments, you can actually also write in the CJS style. I don’t know if anyone’s done that or had much fun doing that but I can’t imagine it’s particularly great. CJS uses Require and its exports, well, this module object. ES6 just has a very simple import and export. Interestingly, it does also have a system import, which again, I don’t think it’s implemented that well but there are poly-fills for it and it basically means using promises. You can load other modules, so you can do nice asynchronous stuff. Yes, conversion. Yes, it really, really pains me to think of anyone who’s had to do this. Convert from AMD to CJS, you can do it. There’s a couple of decent libraries on Github. I can’t imagine it’s much fun. You can go the other way as well. You can do from CJS to AMD, which I believe Sound Cloud had to do. They had loads of legacy code and they had to convert it, so they used a library called, “Sweet JS” to do it. You can convert from ES6 to AMD and CJS. Confused? Okay, good. Which is getting to the number of the talking away because this approach is what we actually take and is working really well for us, using the lovely Babel Transpiler.

Module loaders [18:11]

Let’s have a quick look at the loaders. First up, Require, which largely uses the AMD pattern, but as I say you can also do CJS. The nice thing about Require is that it runs in the browser so you basically can write a module and save it and it will manage it for you. There’s no build step and it can lazy load things. Let’s say you have an application where you’ve got loads of libraries on one page and you’ve got D3 and P5 and loads of canvas stuff, loads of fancy stuff going on. You can just load the modules that you want for that page without having to do any fancy bundling. It’s not got some pretty decent community support because it’s been around for a while. The cons for using it, the error logging is terrible, if for some reason you’ve got one single name wrong for a module it’s not going to tell you that. Basically, if you use it for long enough you get used to these weird errors and you eventually figure out what they mean or you make educated guesses, but that’s not fun. The caching problems. This is a really weird one for me. Even if you’ve got dev tools open and you refresh, it still doesn’t necessarily give you your new code, which is horrible. The documentation is so awful. In fact, I’m going to stop there. I’m just moaning. Yes, one got you. One thing that you’ve got to watch out for if you’re using it is this order, the order of the things that you’re loading and the objects you’re getting back in your call back, they have to be absolutely correct. If this structured data was before base, game over. Just one to watch out for there.

Next up Browserify, which is basically born out of Node. Check out that logo. What a stinker. Horrible. I don’t know what they were thinking. It’s just nothing to do with Node, is it? Anyway. Yes, with Common JS, someone saw the need to bring all that Node loveliness into the browser and so they came up with Browserify, which basically allows you to take advantage of NPM and all the Node modules that are out there in that ecosystem. Pros for Browserify, as I said earlier, simple, syntax, you do get some sensible errors, it’s got really, really nice documentation if you can overlook the branding and with the help of Babelify, which is like a nice little plugin, you can write in ES6 and it will spit out CJS for you. Yes, one more downside, I suppose, is you need a build step but then I suppose lots of front-end or devs in general are using task managers and stuff to do build step, so it’s maybe not a big deal. Yes, if you want to output multiple bundles, it’s a bit more of a headache. It only supports CJS and as I said earlier, it’s synchronous which could be a downside but I’ve not had a problem with it. Yes, jut to reiterate, awful. Got yous; I suppose the only got you, really, is, and this is just a general Node got you, is that what an actual NPM module returns varies wildly. If you’re using this nice handy ES6 Promise Polly Fill, you’ve actually got to attach to that particular property. Somethings don’t actually return anything at all, which I suppose happens in other situations but it definitely has some NPM weirdness or Node Module weirdness.

Next up, Webpack, which is by far the most powerful of all of them, I would say. It supports AMD and CJS. It’s really easy to create multiple bundles which kind of ameliorates the problem of not being about to do lazy loading so well. It’s got a massive set of features, so you can process images in it and do all sorts of fancy stuff. Documentation is really, really good. Logo is okay. Ships with its own server which is pretty nifty. It’s basically trying to take over the world, really. It’s completely comprehensive, it’s amazing. I sort of struggled with some cons. I suppose it’s quite a lot of faff to configure and there’s a bit of a steep learning curve because you’ve got so much power. It’s not very prescriptive. Browserify does a very single purposed thing; whereas, Webpack, yes, as I say, tries to do too much.

Just quickly, JSPM, which is a new kid on the block. This has its own registry. I’m going to leap to that point because it’s a really interesting one. You can’t just say, “JSPM install jQuery” and it will go and get it. It will look it up. It supports all the different patterns there are but you have to pick one, so that’s one of the cons I’ve included. That you actually have to pick an API. In Webpack you can, which is really interesting. The community is quite small and some of the plugins aren’t very mature. It’s a really nice idea JSPM but I’m not sold on it, personally. As I say, no opinion whatsoever.

Implementation [29:15]

How do we actually implement this? I think the most important things to access first are the performance of the actual product that you’re making. The maintainability and the scale. If it’s a really small project, then I suppose there’s no reason to throw all the big tools at it. You could just go for the simplest and straightforward, scaffolded project. That’s not to say you shouldn’t use Modular JavaScript, I’m just saying you might not need Webpack or something as heavy as that, but it depends entirely on what you want to achieve. The shape of your team is important as well. Okay, if you’re a freelancer then perhaps this doesn’t matter so much. If you’re not going to have to hand the code over to someone else, perhaps it doesn’t matter so much. What I’ve found is, every time we’ve made a new step forward, or it feels like forward, where we’ve adopted a new strategy to building websites and making our code more maintainable that we’ve obviously got to educate the less experienced people in the team. Some of the concepts in here and some of the products like Require, for example, which have great documentation, they’re going to take more of your time if you’re the person who’s in charge of educating other people. Yes, you’re going to have to really put the hours in and put the effort in to make sure everyone’s up to scratch. You’re probably going to have to sit with them occasionally and do some really grim debugging. Yes, make the time for the education.

The nature of the app itself. If you’re using a framework like React or something, then it starts to decouple lots of the script and the views and things anyway. It’s got its own way of tacking some of the goals which this subject does. Yes, if the application shares code on the server and on the client, then it’s maybe just worth picking the Common JS approach because it’s designed for the server and can work in the browser. If the project is going to be around for a long, long time, I could argue that you should use ES6, not just for Modular JavaScript, but for everything. Any shaking heads out there this time? Maybe not. Yes, if you want to future proof it, then that’s the direction that everyone is seemingly going in; although, I appreciate that there is some effort that goes with that because you have to transpile down to something that browsers can understand.

Here’s just a quick diagram, simple diagram of how you might manage a site, how you might do bundling. You have a bunch of things that go into two different modules, which are then only rendered on the home page and you can decide on the architecture yourself. As I say, the easiest thing I found for doing this is either Require or Webpack. Browserify does do it but it’s a bit more faff. The tools that you can use to actually compile and concatenate all these things together and they do need to compile because even with CJS, you need something which is going to tell the browser how to actually work with it. Gulp and Grunt are the two obvious task managers but you can if you want to use some of your colleague’s server side applications. I personally wouldn’t, but no opinion.

The way we do it at Zone is all modules are discreet pieces of functionality. They are sometimes a trackable feature in a project management sense. Sometimes there’s more than one module that belongs to that particular feature. In the browser that is, they always run based on the presence of a data attribute, so we don’t use CSS classes or anything like that. We keep it to just using data attributes for decoupling. We use source maps, so with Gulp and Grunt you can create a map of all the modules that you are writing in so that when it actually renders one bundle or one big file, you still know what you’re looking at. If there was a bit more time I could have shown a nice little screen grab of that, but you just have to use your imaginations. Sorry.

Headaches. I mean, of course, with these things there are always headaches. Lots of times we’ve had incorporating third-party scripts, so you might need a Shim or you might need to actually wrap it yourself in that UMD wrapper or something to make it work properly, because not everything that’s published on the internet, funnily enough, works with everything else. Keeping a large project organised takes a bit of effort. Consider things like naming conventions and how you structure your folders. Yes, one last little tip. Handling data, each module is obviously encapsulated so you need to think about how you’re going to share things that happened, user actions, and we use Pub Sub, which is a way of listening for particular events, you can use jQuery to do that too. We use Promises for asynchronous stuff. We completely avoid using call backs as much as possible.

This is our Stack. We write in ES6 as much as possible, we use Gulp to manage most of our tasks, we use Browserify/Babelify to convert from ES6 to Common JS and it works in the browser. This is what we’re moving to, so we’re going to adopt Webpack. I’m sorry, that was almost half an hour. Big subject though. Thanks a lot.