Ecstasy modules

We talk about modularity and reusability as core tenets of the design of the Ecstasy language, but what does it actually mean that a language supports modularity? What does effective software reusability actually require? What are the aspects of a language that make modularity possible, and reusability simple?

It should be self-evident that no language starts off with a design plan that says: "Minimize reusability. Encourage monolithic architecture. Prevent modularity." To the contrary, most languages make claims about enabling reusability, and many languages claim to support modularity. And yet, when it comes to modularity and reusability in existing languages, the actual results seem to fall woefully short.

The Ecstasy language is built on an explicit notion of modularity, and reuse is an explicit goal of its design. Let's talk about what that translates to in the real world.

Big and Little

The first requirement that we had for modules in Ecstasy is that they had to work really well "in the small", supporting a module containing as little as a few lines of code (perhaps a single function), but also "in the large", supporting massive applications with tens of thousands of classes, and tens of millions of lines of code. From experience, attempting to solve such dramatically different extreme points on a single scale results in significant trade-offs at best, or turns out to be an altogether quixotic quest at worst.

Let's start with a minimalist example:

  module MyLittleModule {}

Strangely enough, that is an entire module. Admittedly, it doesn't do much, but it can be compiled, and it can be loaded as part of an application. Perhaps a better example would include something of use:

  module TaxCalculation.example.com
    {
    static Dec calcTax(Dec amount)
      {
      return amount * 0.05;
      }
    }

In this second example, the module TaxCalculation contains a single function, which unsurprisingly calculates some tax. Why would someone write such a module?

• By splitting an application into more than one module, work can proceed in parallel across these multiple modules, naturally separated by module boundaries.
• Organizational separation of responsibilities can map well to module-based development, where each module is the responsibility of a specific part of an organization.
• Modules may be produced by different organizations altogether; by separating functionality into multiple modules, it is possible to cleanly delineate work from one organization from that of another.
• In Ecstasy, modules represent both units-of-compilation and units-of-deployment; by separating functionality into multiple modules, updates to functionality located in a single module can be written, built, versioned, and deployed independently of other modules.
• By allowing a module to be as absolutely small as necessary, it reduces the natural risks associated with version changes, by allowing a module to fulfill a deliberately minimum set of responsibilities.

Many of the other aspects of the design of Ecstasy modules are designed to support the complexities inherent in building, maintaining, co-existing with, and consuming large modules, but it is important to remember that a module can be as simple as a single source file, and as simple a single line of code.

A Module: A Class of its Own

In the minimalist example above, we introduced the keyword "module". In Ecstasy, a module is a form of a class. As a class, it can contain methods, properties, typedefs, functions, constants, and other classes. Additionally:

• A module is a const class, so once its construction completes, it cannot be modified.
• A module is a singleton, so its singleton instance can be accessed from anywhere within the module by using the simple (unqualified) name of the module.
• A module automatically provides an implementation of the Ecstasy Module interface.
• A module also has the ability to contain another special form of class, called a package; only modules and packages may contain packages.

In many ways, a module is just like a package, except that a module's name can be qualified to include a domain name corresponding to the organization that provides the module. By allowing a qualified name, Ecstasy naturally supports distributed module repository systems, cryptographically-secure module signatures, and name-spacing that mirrors the Internet itself.

Module Name-Spacing

Code within a module can only refer to and use names within the same module. In other words, while modules themselves may have qualified names, code within a module never uses qualified names to reference other modules.

Instead, module dependencies are mounted within the module that needs them, just like network drives can be mounted within a local file system. To mount a module within another module, the package keyword is used. For example, using the example module that we introduced above:

  module MyApp.example.com
    {
    // mount the entire module "TaxCalculation.example.com"
    // (and its hierarchical namespace) as package "taxrate"
    package taxrate import TaxCalculation.example.com;

    const Invoice
      {
      // ...      Dec subtotal;
      Dec tax.get()
        {
        return taxrate.calcTax(subtotal);
        }
      Dec total.get()
        {
        return subtotal + tax;
        }
      }
    }

(Note that while this example conveniently shows the Invoice class contained directly within the module, the Invoice class would typically be split out as its own source file. A file may contain as little as a single class, or at the other extreme, a single file would contain the entire module, even if the module is enormous. As a rule of thumb, multiple classes should only be placed into a single file if doing so improves readability and maintainability.)

Module Embedding

If a module is only being used in one application, there may be no reason to use it as a separate unit of deployment. In this case, it is easy to physically include it in the resulting module as part of the compilation process:

  package taxrate import:embedded TaxCalculation.example.com;

Module embedding allows multiple modules developed independently to be delivered as a single module.

Module Optionality

When a module is mounted as a package, it is assumed to be required; however, in some cases, the module may be optional. There are two two flavors of optional. The first indicates that the runtime should use its best efforts to obtain the module:

  package helpers import:desired HandyHelpers.example.com;

The second allows one module to indicate a supported (but not required) module; in this case, the runtime will only load that module if it is desired or required by another module that is being loaded:

  // we only support version 3 or later for this module
  package fmts import:optional Formats.example.com v:3;

By allowing module dependencies to be optional, Ecstasy modules can include support for third-party modules that may or may not be used in a particular environment. In short, while coupling itself may be unavoidable, a coupling at the source code level can exist without creating a runtime dependency.

Module Versioning

A module's version is not part of its source code; the version can be assigned to (or "stamped onto") a module after it has been built. The version information is intended to support Continuous Integration (CI) models, Software Development Life-Cycle (SDLC), and the long-term maintenance requirements of complex applications:

• A build can be marked as a development, CI, alpha / beta pre-release, release candidate (RC), or GA release build.
• A build can be marked as a next logical version of another version; for example, version 2 is the next logical version after version 1.
• A build can be marked as the revision of another version; for example, version 1.1 is a revision of version 1.
• The revision tree supports any arbitrary depth, making it possible to patch any existing version; for example, after releasing version 1.0 and version 1.1, it would be possible to release  version 1.0.1 to fix a problem with version 1 for a customer who is unwilling to adopt version 1.1, and then subsequently to release 1.0.0.1 to fix a more specific problem with version 1 for a customer who is unwilling to adopt version 1.0.1.
• Ecstasy module versioning explicitly supports Semantic Versioning 2.0.0, and adherence to that specification is encouraged. However, Ecstasy does not require conformance to the rules of Semantic Versioning; organizations and developers are free to support a versioning model of their choice.

Module versioning allows an application to specify which versions of other modules that it requires or is supported with, in order, to any arbitrary level of complexity; consider the example:

  package json import JsonUtils.example.com v:2
      avoid v:2.0.1
      prefer v:3, v:2.1;

The syntax supports a list of clauses using the keywords allow, avoid, and prefer. The result is a rich yet comprehensible means to declare a dependency on a separately-versioned module, while retaining a level of control over the versions of the other module to use if and when necessary.

Module Packaging

Multiple versions of a module can be combined into a single module file, allowing any number of versions (including patches and pre-release versions) to be delivered in a single file. This allows an organization to provide support for older versions of a module, while continuing to deliver newer versions of the same. It also dramatically reduces the entropy related to managing repositories full of different supported versions, and is designed to enable CI automation for testing against all available supported versions of a module.

The packaging of multiple versions of a module together into a single file is very space-efficient; only the "deltas" require additional space.

All of the version metadata is included in the module file, and is available for use by build, CI, testing, repository, and deployment tools.

Conditionality

One cannot support the notions of module optionality and module versioning in a type system that employs static typing (and type safety with transitive closure), unless there exists a means to adapt to the presence or absence of a module, a version thereof, or particular components contained therein. In other words, when considering the above optionality and versioning examples, it must be possible to build a module in such a way that it still works correctly in the absence of the desired HandyHelpers.example.com module, and that it integrates with the Formats.example.com module when it is present, and that it can gracefully handle the absence of the same.

To accomplish this, Ecstasy supports link-time conditionality that can be easily expressed using existing and obvious language constructs:

• It is possible to declare data types such that they exist conditionally;
• It is possible to declare aspects of a data type such that they exist conditionally; and
• It is possible to define logic that exists conditionally.

Using the Formats.example.com example from above, if that module were to provide a Formatter class, logic can conditionally take advantage of that optional component:

  String to<String>()

    {

    if (fmts.Formatter.present)
      {

      return Formatter.format(this);
      }

    return super();

    }

Unlike a pre-processor approach to conditional logic, Ecstasy compiles and checks each and every combination of conditions, ensuring that all of the language rules are always enforced, and compiling all of the potential combinations into the same compiled module file. As a result, the conditions can be evaluated at link-time, based on the exact information that is available when a set of modules of specific versions are selected to be used together.

The keyword used to test conditionality is the "if" statement, and includes:

• "name".defined is used to test if a particular named option, such as "debug" or "test", is specified for the modules being loaded;
• identity.present is used to test if a particular module, package, class, property, method (etc.) is available;
• module.versionMatches( ver )  is used to test if a version of a module (or a revision thereof) is available.

A benefit of this design is that unit and functional tests, debug builds, production builds, patches, and unreleased versions may all be included in the same compiled module file. That same file can have dependencies, some of which are optional, on any number of other modules, and it can even contain any necessary work-arounds for dealing with inconsistencies in specific versions of those same modules. These are the types of things that, until now, had to be done by hand; while the inherent complexity remains, the complexity has been encapsulated in a manner that allows solutions to be automated, and for the remaining non-automatable challenges to be solvable.

(Conditionality is a powerful capability, but as a rule of thumb, its use should be minimized to the extent possible. Even with the powerful organizational and automation capabilities that have been described here, the combination of variability and complexity still exists, and will inflict tremendous pain on those who underestimate the cost of entropy.)

The Ecstasy Core Module

Ecstasy's own type system is provided as the module Ecstasy.xtclang.org, which is automatically imported into every module as the ecstasy package. The Ecstasy core module has no dependencies on other modules, but all modules have a dependency on it. It contains all of the primordial types, which represent both the basic building blocks from which new types can be formed, and the means by which the runtime interacts with application code, such as exceptions.

(The Ecstasy core module is also dependent on the Ecstasy core module, which introduces a recursive dependency. As a result, not only is the class ecstasy.text.String available inside of every module, but so is ecstasy.ecstasy.text.String and ecstasy.ecstasy.ecstasy.ecstasy.ecstasy.text.String. Not surprisingly, the type system is referred to as the Turtle Type System, because it is turtles, all the way down.)

Transitive Closure

The module is the unit of transitive closure for the type system. Everything that a module references must be present within that module, either by importing another module, or by placing the necessary components within the module itself.

When a module is built, it contains a detailed fingerprint of all of its dependencies on other modules. The fingerprint defines the exact set of modules, packages, classes, properties, methods, and so on that are used by the dependent module.

When a module is loaded, each of the modules that it depends upon are loaded along with it. The loader is responsible for selecting the version of each module that will be loaded, for selecting the combination of conditions that will apply to the modules, for resolving all of the dependencies among the various modules, and for verifying that the result is correct according to the rules defined by the XVM.

Summary

Modules can be as small as an individual component, as reusable as a library, and as complete as an entire application. Modules support a wide range of granularity, and promote both composition and reuse. Modules contain extensive information in order to support versioning, in order to define dependencies in a non-ambiguous manner, and in order to provide supporting information to various development and management tools.