14 September 2014
Dave Anderson • @eymiha
A Meteor Dissection - The Package API Object

Meteor is not just about the code you might happen to put in your project to make the lights blink and flash in your browser or on your phone or tablet. No, Meteor is also about creating packages that will let other software developers re-use your work and make the lights blink and flash for them as well. To a large extent packages are the life blood of Meteor; as you dig in, you find that much of the Meteor system itself is expressed in packages.

“Gosh,” you say. “Packages sound incredibly versatile. I’d love to learn more about them, and maybe write one myself.”

Hopefully you’ll learn a bit more about packages here. I’m not going to show you an example of one, but I’m going to take a dive into the code underlying the Package api object and look at what’s going on behind the scenes.

The code we’re going to be looking at is necessarily a moving target, and quite probably by the time you read this, it will be terribly out of date. Meteor is just coming up on it’s 1.0 release (soon, oh soon, we hope) but it’s not there yet, and everything I’m about to talk about could change. (In fact, in the short time I was writing this, it did.) But perhaps not so radically. Hopefully this snapshot will still be worthwhile tomorrow.

So if we look at the Meteor documentation at this moment in time, The package interface is:

# Package.js

Package.describe
Package.onUse
  api.versionsFrom
  api.use
  api.imply
  api.export
  api.addFiles
Package.onTest
Npm.depends
Npm.require
Cordova.depends

First a little context. A package pulls code into your Meteor application, and this interface details how that happens. The definition of a package comes in three parts: a description (describe), what to do when you use it (onUse), and what to do when you test it (onTest). When you write a package, you include a package.js file which contains these three functions. When someone adds your package, Meteor pulls in the package.js file, calls the functions and, voila, the package is integrated into the application. The workhorse of the mechanism for practical purposes is the onUse function. (The Npm.depends and Npm.require are also fairly practical, and the Cordova stuff is insanely great, but again, for now we’re going to focus on onUse.)

In case you’re interested, the code for all this is in the Meteor hierarchy under

   meteor-tool/
      major.minor.revision/
         meteor-tool-os.operating_system.machine_architecture/
            tools/
               package-source.js

The hierarchy’s a bit deep in there, so tread carefully.

Package.onUse

Your onUse function call will look like this:

Package.onUse(
  function(api) {
    // lots of calls on the api object
  }
);

in other words, in your call to the Package.onUse function you’re passing a function that Meteor will call with the application’s api object. The function calls you then make on the api object add the data that describes the files your package depends on and what your package will export.

Don’t give onUse a lot of attention by itself, really. It’s JavaScript plumbing that needs to be there to give you access to the api object. But it is really ingenious plumbing that pulls the files in the packages into your application.

The api Object

It’s the api object that’s passed to the function you define and pass to Package.onUse that’s key to pulling code into your application. We’ll go through each of its functions and see what’s going on.

api.versionFrom

The versionsFrom function indicates the particular release track and version that you’re targeting with your package. Most of the time you’ll just be working in the Meteor release track, so you can just specify the Meteor version you’re targeting.

api.versionFrom(release)

The release argument of the form track@version, which lets you specify an arbitrary Meteor track, but again, unless you’re actually developing Meteor itself, you’ll probably just care about the @version part.

What Really Happens

The code first looks to make sure you haven’t called this function more than once on the api object. You only can do it once - it doesn’t make sense to target more than one version with a package. If you have, an error is thrown and the call returns. If you really need to target more than one version, you need to define another package.

Next the code checks the catalog is to make sure your package isn’t part of the already-injected Meteor core packages. If so, an error is thrown and it returns. (And if your package is part of the core, why are you reading this? You already know this stuff!)

If the track wasn’t specified, the code next pulls in and uses the Meteor default. It looks to the catalog to get the default track and sticks that into the release string that you passed in.

Now we split the incoming release string. If it had more than one @ character in it, an error gets thrown and it returns. Otherwise it checks it against the catalog’s official released versions and complains if it can’t be found. If it was found, it’s recorded in the internal package object.

The catalog?

We just made a few references to a catalog object. There are actually several that get created, and they keep track of the packages in the system. There’s one that tracks what’s on the server, another that tracks local packages, and still another that knows what’s part of the Meteor core. Basically, the catalog can be searched for what’s been loaded and what hasn’t, can figure out what depends on what and load everything in the right order, and can refresh the packages that need refreshing.

If you think of catalogs as the keepers of the packages in the application, you’re pretty safe. If you’re interested in taking a closer look, they’re in the catalog.js and catalog-base.js files in the same directory as the package-source.js file.

api.use

Packages don’t exist in a vacuum, of course. Just as our applications use packages to do their work, your package may need other packages to work as well. The use function is used to indicate what other packages that need to be loaded for your package to function.

api.use(packagename, architecture, options)

The packagename has to be there, and it should be suffixed with a @version to specify the lowest compatible version that can be used with this package. A higher version may be substituted, so stay aware; when things you depend on change, you may need to upgrade your package. You can pass an array of package names, by the way.

The architecture is either ‘client’ or ‘server’, or an array of architectures, and is optional if you want it them all.

The options ‘weak’ and ‘unordered’ are handled in the code: weak means load the used package before this package, but it’s not actually needed; unordered means loading the used package is required, but it can be done anytime.

In typical use, you may see only an

api.use(packagename)

letting the system default all the additional values.

What Really Happens

The first thing that use does is to adjust the incoming arguments. packagename is converted to an array of package names, the architecture is filled if it was missing and then normalized and validated to contain only recognized architectures, and the options are initialized to an empty hash if they’re missing.

Note that the architecture is validated into different values than what you may pass in. server is mapped to os and client is mapped to web, which in turn maps to web.browser and web.cordova currently. If you know what you really want, you can pass them in rather than having the mapping normalize the values.

The options next have a constraint applied to them. If both unordered and weak are true, an error is thrown and the call returns. There both being true is contradictory.

Finally, for each packagename (you can pass in multiple package names even though the api docs only indicate one), it is checked to make sure that the @version part works with the package name, and it throws an error and the call returns if the combination is bogus - if the version is left off, it’s unconstrained and the check should succeed. Once it gets this far, the used package is added to the uses hash - the packagename, version constraint and it’s weak and unordered option values.

The uses hash?

Each package that gets used is added to a uses hash. The uses hash keeps track of all the uses and after being built up using the api interface, the uses for each architecture are extended to include unspecified dependencies from the Meteor release. What this ends up being is something digestible into an ordered set of packages that need to be loaded to get your package to work.

In can get ugly. The weak and unordered option values can be adjusted to handle circular references when they occur, allowing otherwise seemingly gridlocked package configurations to work.

api.addFiles

You’ve indicated which Meteor you’re targeting and what other packages you’re using. Now you’re set to get to the crux of your package: the files that make it up.

api.addFiles(filename, architecture, fileoptions)

Here you specify a filename (or array of file names) and optionally an architecture (or array of architectures). Filenames are assumed to be relative. The files contain the code that make up the innards of your package for the architecture in question.

The optional fileoptions argument… well, most of the time you would leave it off. The idea is that certain files should be ‘bare’ (not wrapped in a closure the way CoffeeScript is treated, for instance) or that a file specified may be an asset whose extension is used to trigger a particular handler for that type of file, or etc, and etc. The compile phase considers these options (the code is loosely considered to be compiled as translations and loading take place) and builds up the application that actually runs in Meteor. Unless you’re working in the compiler area (see compiler.js) just look askance at the fileoptions argument and leave it out.

What Really Happens

First the filename argument is wrapped up into an array, in case it wasn’t one already, and the architecture is wrapped up into an array, validated and expanded if you passed in ‘client’ or ‘server’. All architectures are included if you left architectures off.

Next, for each of the architectures, the filename and any file options you specified are wrapped into a source object and added to the sources hash.

And we’re done. addFiles is the shortest of all the api methods because it doesn’t do any work with the files except pack their names away for the compiler to smash into. Maybe I’ll dig into that in another dissection, but not until Meteor has passed into the after-1.0-release territory.

The sources hash?

Each source file that gets added is pushed onto a sources hash, keyed by architecture. Once the overarching onUse function call returns, it’s going to try to chew into all the source files in sources hash. If something’s not found, or a dependency is wrong, or for some other reason something looks amiss, error messages are going to be generated and the package is going to be ignored.

So, just know that the files you add to your package through the api aren’t looked at as you’re adding them, but only after they’ve all been added. That’s just how it works.

api.imply

Sometimes you want your package to pull in other packages that should also be used. Your package acts as an umbrella covering the other packages and they’re added to the application.

api.imply(packagename, architecture)

The imply function does that, taking a packagename (or an array of package names) with the @version optionally included, if needed. An architecture (or array of architectures) may optionally be specified (‘client’, ‘server’ - like in the use or addFiles functions.)

What Really Happens

A lot of the same stuff that happens in the use function happens here, but without the options rigamarole. First packagename is converted to an array of package names, and the architecture is filled if it was missing and then normalized and validated to contain only recognized architectures.

Then each packagename is checked to make sure that the @version part works with the package part, throwing an error and returning if it’s bogus. If the version is left off, the package name is unconstrained and shouldn’t be held up. Finally, the implied package is added to the implies hash.

the implies hash?

In the same spirit as the uses hash, there’s the implies hash, which keeps track of what packages are implied by this one. Also like uses, after being built up using the api interface, the implies for each architecture is extended to include unspecified dependencies from the Meteor release.

Implied packages are just more packages to be pulled in. It’s an easy way to expand the packages available for developers to use without forcing them to be fastidious, instead you’re figuring out the things that developers might need for them.

There really seem to be a lot of hashes that the api object is filling up, and that’s pretty much how it goes. That’s the pattern that’s getting exploited. But wait, there’s still one more.

api.export

When you want to expose an object into the Meteor namespace for all to use directly, you use the export function.

api.export(symbol, architecture, options)

The symbol (or array of symbols) holds the name of the objects that you’d like to export into the specified architecture (or array of architectures). The only option in the options hash that’s currently considered is ‘testOnly’, which make the symbols available only for the architectures in the context of the onTest function.

The onTest function?

You don’t think the Meteor folks would go to all this trouble to create the api object and not re-use it, do you? Alongside the onUse function is the onTest function that indicates what packages are available for testing. Depending on how you set up your packages, you may want different sets of symbols available for testing which might make additional symbols available to let you evaluate internal states, specially calculated values, or whatever. The ‘testOnly’ option is your backdoor to let you re-use the same api setup for both using and testing your package. The api object is no one-trick pony.

What Really Happens

The first thing that happens is a little of the old shell-game: assuming that if only two arguments are passed in and the second isn’t an array, then it’s really not architecture but an options hash instead. The value of the second argument is assigned to options and the architecture is set to null, which will key it to be expanded to all architectures a few steps ahead.

options is assigned to be an empty hash if it’s missing, just to let the subsequent code flow smoothly without checks.

Next, symbol is expanded to an array of symbols if it wasn’t already, and architecture to a validated expanded array of architectures in the same way it’s been done in each of the other functions.

For each symbol, the name is then validated - it must only contain letters, underscores and numbers, must start with a letter or underscore, and letters are both upper and lower case. If it has an invalid character in it, a error message is generated and the code returns. If it passes, it’s pushed into the exports hash for each architecture with a testOnly flag if it was marked as test only in the options.

the exports hash?

The last of the hashes is the exports hash, which keeps track of the symbols being exported by this package. The symbols in the exports hash are used by the linker (if there’s loosely a compiler, then it stands to reason that there’s also loosely a linker, in linker.js) to pull in as needed when the application is being readied.

And That’s All They Wrote

…so far anyway.

Remember, at the time of this writing, Meteor is still pre-release 1.0, and this could all change.

Takeaways

I wrote this because I was stymied adding some files to a package directory that include a CoffeeScript class. The error I got was that the class couldn’t be found. It took a lot of head scratching and questions to my Meteor friends before I found the problem. I hadn’t included the file containing the class into the package using an addFiles call on the api object passed to the package’s onUse function. You live and you learn.