Skip to content

Document your Javascript: interview with Michael Mathews of JSDoc Toolkit

Documenting source code is something that doesn’t always get the priority it deserves. Especially Javascript which can be implemented rather ad-hoc. But there are enough tools around that can make documentation a lot easier.

Michael Mathews

Michael Mathews

I recently discovered JSDoc Toolkit, a tool that generates documentation of your Javascript code in the form of a website. That was dearly needed because I wanted to give me and other team members a better overview of my growing JavaScript codebase.

The history of JSDoc

The JSDoc-toolkit and its predecessor JSDoc have been around for nearly 10 years now and in both cases the bulk of the code is programmed by Michael Mathews, often shortened to “micmath”. JSDoc started as a solution for the growing Javascript code at the company he worked for: “I couldn’t find any existing tool that did what I wanted, so I just naturally started building my own. I didn’t want to totally reinvent the wheel of course. I basically tried to port Sun’s JavaDoc idea over to JavaScript.” So he wrote a documentation tool in Perl. “Once it was done and we’d used it a bit, I decided to release it as open source on SourceForge. I figured if I found it useful then maybe someone else would as well.”

Although JSDoc is a very useful documentation tool and probably the first free tool as well, it wasn’t received that well back in 2001 according to Michael Mathews: “It seemed the general JavaScript community was not yet ready for JSDoc. Back then the majority of folks doing JavaScript were not in the habit of practicing what you would call ‘Software Engineering’ best practice! It was, more often than not, a field dominated scripting cowboys. The idea of automatic documentation generation, I think, was seen as something unneeded by most. According to the stats from SourceForge, the JSDoc tool was only downloaded a few hundred times in the first several years – and frankly that already seemed amazing enough to me.”

The renaissance of JavaScript

But as the web got more mature, so did the developers. Javascript gained respect in the last few years and the emerge of large libraries such as YUI, jQuery and Prototype showed that Javascript can be used in a complex but constructive way. New ways of coding were used to structure Javascript. Michael noticed the trend: “Since about 2005 JSDoc really started to take off. Since then it’s been downloaded over 30,000 times. It still seems amazing to me.” The latest version of JSDoc Toolkit is downloaded almost 5000 times since it was published in September 2009.

JSDoc in the wild

The term JSDoc by the way is a bit confusing. It refers both to the syntax and the program. The syntax has been relatively successful as well: “I know that a few of Google’s projects use JsDoc comments to extend their functionality. Closure Compiler is a good example of that, though it can be a pain when other projects start redefining the JsDoc syntax for their own purposes in ways that are incompatible with the original project. Of course there is no official standard for JsDoc comments, so I can’t blame them, but that is something I’d really like to see: an official JsDoc standard, I mean.”

“The idea of JSDoc has been picked up by many large projects over the years. However, my application was originally written in Perl and wasn’t very flexible, consequently it got rewritten by others a few times for internal use only. I don’t want to name any names because I personally think that, while perfectly legal to do that, it violates the open source spirit I intended the work to be used in.”

Document your code...

Document your code...

...run the tool

...run the tool...

... and publish the generated documentation website

...and publish the generated documentation website.

The Toolkit of JSDoc

A year after JSDoc was released, Michael resigned as an active contributor to the project. Gabriel Reid took care of JSDoc after that. He maintained and extended it. But around 2006 Michael decided to do a major upgrade for JSDoc. “Big frameworks were the norm and along with that was both the need for API documentation as well as the complexity of writing documentation for all the new and interesting things people were doing with JavaScript. I was beginning to think that, in order to get JSDoc up to speed with this new world, it would need to be rewritten from scratch. So I started a discussion about that on the users’ list. That eventually led to the current incarnation of the project: JsDoc Toolkit.”

The rewrite was written in Javascript itself. That was possible with the open source Rhino platform that parses the Javascript. That wasn’t based on any technical issue: “I just liked the idea of a program parsing itself. As a bonus the users were more likely to be able to modify the source code if it was written in a language they work in.”

Rhino runs on Java and was initially developed by Netscape. Rhino would have been a part of a larger project to port Netscape Navigator to the Java platform, but that never happened. Rhino is still being used and is maintained by the Mozilla Foundation.

How JSDoc works

JSDoc reads the code and the comments. The comments need to be formatted according to the JSDoc convention. It’s a notation that’s used in other tools like Google’s Closure as well and is similar to the JavaDoc convention.

The comments give hints to JSDoc Toolkit how the JavaScript is structured. Unfortunately not all notations are recognized by the tool. In fact: most of the really exotic notations are not recognized. Michael: “People don’t realize how difficult it can be to do static code analysis on a language as dynamic as JavaScript. Well, at least I didn’t. It is provably impossible in most cases to know what the various states of a JavaScript program will be at runtime, just from looking at the source code. I tell people, if you can get a program to know what you want have programmed, just from the plain source code, consider yourself lucky!”

“But for the rest of us JsDoc Toolkit provides a way to document your code in such a way that the tool never has to even see your code. In fact, you can even use JsDoc Toolkit in files with just comments, no code whatsoever! So in that sense there isn’t any code you can’t document with JsDoc Toolkit, as long as you don’t rely on the static code analysis features.”

The word “Toolkit” in the name “JSDoc Toolkit”, by the way refers to the template engine JSPlate. Michael: “I eventually went with the ‘Toolkit’ name mostly because I liked the sound of it, but also as a little nod to a favourite templating system of mine, Andy Wardley’s Perl Template Toolkit.”

JsDoc Toolkit Version 3

Right now Michael is in the middle of programming version 3 of JSDoc of which the first beta should be released somewhere this summer. “Basically there are a few feature requests from JsDoc Toolkit 2 that seem reasonable enough -things like documenting the same symbol more than once- which are utterly impossible due to the core architecture of the tool itself. All those feature requests are now on the table and I’ll try to include them in the new version.”

“One feature that will definitely be supported is the idea of overloaded functions. A typical example would be a function named ‘width()’ that returns the width when it is called with no arguments, but will set the width when called with a numeric argument.”

“This is fundamentally not possible to document in JsDoc Toolkit Version 2. And that really is the point of rewriting the tool for Version 3: it will include, in it’s core, all the lessons learned from the past several years of supporting JsDoc Toolkit. I believe it will provide a much more flexible foundation to build on.”

In addition, Michael will make more use of Rhino than he already did. Until now Rhino was only used to parse the server-side JavaScript. The parser that walks over the source code looking for doc comments and things like function declarations was written by Michael himself. Rhino has a JavaScript parser built in, but it would took “some serious Java hacking” to get it right.

“Fast forward a few years and some important things have changed: namely, Norris Boyd has made accessing Rhino’s internal Abstract Syntax Tree much easier. In fact, it is now as easy to walk over your source code as it is to use, say, DOM nodes to walk over HTML. I’d be crazy not to take advantage of all that free power just sitting there waiting to be used!”

“So Version 3 will have none of my own parser left in it. This means I get to shed about 40% of the project’s code, which was dedicated to just that one task, and can now focus on the more important bits: parsing the doc comments.”

“It also means that, in theory, if Rhino can understand it, you’ll be able to get JsDoc Toolkit to understand it. Of course this is all taking time to get up to speed with, but I’ve decided to use build my tool on top of the new RingoJS framework and that’s made my life much easier, and the process much faster.”

The first beta of version 3 is estimated to be released in early summer.

Continuation

One of the weak spots of the JSDoc project is that most of the time there’s only one active developer working on it. Although the tool is still alive and kicking, it naturally needs some nurturing to prevent it from getting stale. “To be honest it can feel like a real burden sometimes.” Michael admits. “It’s essentially just me these days, and I mostly can’t give the project all the time it deserves.” So developers are welcomed to join in the project: “I’m ready to sign them up! The fact is that I think of JSDoc as my baby. Not that I invented anything new or amazing, I most emphatically didn’t. But I’d lived with it long enough and I’d become associated with it, so I wanted to see it do well in life.”

And so the development continues even though there are now some other tools around like JSDoc. He even used some of them. “I used to write a lot of Perl and JavaScript at the same time and found NaturalDocs to be a great tool because it required a single format that could work in any language.”

Server side JavaScript

Although JavaScript as a program language is appreciated much more than a few years ago, it is still a quirky language. But that’s no problem for Michael: “Yeh, it’s a weird language. But I come from a background of Perl, so I like weird languages! But actually, JavaScript on the server side is a totally refreshing experience. Rhino, for example, allows you to use all the latest cutting-edge JavaScript syntax and features, and you never have to worry about running in an ancient or buggy web browser. I hope and expect that server side JavaScript will really start catching on in the next few years.”

A bit about Michael Mathews

It was hard to find a bit of information about New Yorker Michael Mathews. Although JSDoc has been around for almost 10 years and is downloaded a lot, he hasn’t been interviewed before. He has been asked many times for some assistance though. It can create funny situations: “I was doing some freelance work and the fellow sitting next to me asked if I’d ever heard anything about this ‘JSDoc’ stuff he was trying to use. I said I had heard of it and helped him sort it out. I had to eventually admit that I was the creator of the tool. He seemed a little surprised by that.”

Mathews started as a zookeeper, which he had done for a few years. But it wasn’t that much fun for him. “I basically cleaned up after lots of unusual animals.” When he wasn’t working, he played with his computer. “By the late nineties I was had my own homepage, complete with the large, requisite, ‘Under Construction’ message.” His work behind the computer payed off eventually: “in the fall of 1998 I was employed as a Web Developer for MTV Online, working in Times Square in New York City. It still seems amazing to me.

He now works since 2008 for the BBC in London where he works on the Glow JavaScript library. And yes, everything of it is documented with the help of JSDoc Toolkit.

Glow is a JavaScript library that just, like Prototype and jQuery, makes programming in JavaScript easier. The difference is that Glow supports whatever standard is currently defined by the BBC Standards and Guidelines for browsers, accessibility and usability. This could mean it has to support older browsers that other major JavaScript libraries have already dropped.

Links

Bookmark and Share

Leave a response