Signs of Spring

Here in the New England, we are getting our first taste of spring, with temperatures reaching almost 70F (21C). The birds are returning, and the grass is starting to wake from its winter nap, with a beautiful green color beginning to peek out everywhere you look, and with the wildlife busy building nests and popping out the next generation. The plants have put their winter coats away for the year, and are rushing to get their flowers out and busy, making this a gorgeous time of the year for one and all.

We've been busy, too. For the past several years, we've been designing, prototyping, validating, and documenting the first programming language designed for the cloud: the Ecstasy programming language. We're not yet ready to make it generally available, but we're getting closer every day, and now seems like the right time to start introducing it.

I'd like to start by talking about the why: Specifically, why would anyone in their right mind build a new programming language from scratch? Or, asked differently: What are the problems that Ecstasy is designed to solve that are inefficient or impossible to solve with existing languages?

This, in turn, is quite a hard question to answer in a short space, because -- as computer science explains -- that which can be performed by any one Turing complete language can also be performed using any other Turing complete language. However, we can begin to describe which challenges that Ecstasy is designed to address, and how it was designed to solve them.

Scalable

There are a lot of meanings of the term scale as it is used in computer science and software. In the general sense, a scalable solution is one that works well in the relatively-small, and it continues to work well in the relatively-large. For a language to be considered scalable, it must work well for a small throw-away project -- perhaps even only a few lines of code! -- yet it must also be able to support the needs and processes of a large team building and maintaining a huge code base over a period of years or decades. From a runtime perspective, a scalable language must work well in a constrained device, yet it must also be able to efficiently utilize the capabilities of a large server cluster.

Ecstasy is a strongly typed language, and uses a modular type system with verified transitive closure. What this means in practice is that Ecstasy's design minimizes the number of potential surprises from code changes, and thus the number of flaws that can only be detected as runtime failures. This topic is expansive enough to require its own volume, but in general, Ecstasy's design emphasizes predictability above all other goals, and that predictability helps the language scale to large teams and to large projects.

Another design emphasis is on readability, meaning that when a choice is required between making the language easier-to-write versus easier-to-read, the choice has almost always been to make the language easier to read. This means that the Ecstasy language is not intended to win any terseness awards, and its style encourages the use of vertical space, explicit constructs, and rich documentation integrated as part of the code.

The Ecstasy language also includes fundamental support for the entire software development life cycle, including versioning and upgrades, mocking and testing, and dependency management -- including conditional dependencies.

From a runtime perspective, the Ecstasy language is designed to produce libraries and applications that can be (a) compiled to native executable formats, (b) executed in JIT/dynamically-optimizing runtime environments, or even (c) completely embedded into other languages, applications, and platforms! The language does not prescribe an explicit threading model, nor does it directly expose any of the raw capabilities of the underlying platform, allowing the language to scale from a single core, low-power CPU, all the way up to large SMP servers.

Memory management is fully automated by the language runtime, but it is not bound to a particular data structure (such as a heap), nor to a particular management approach (such as garbage collection or reference counting). The Ecstasy virtual machine (XVM) architecture was designed explicitly to efficiently support terabytes and even petabytes of application memory, without requiring any "stop the world" pauses.

So, why does cloud computing require a scalable language? First, because the cloud is connected, and the clients connecting to applications in the cloud range from low-power IoT devices, to phones and tablets, to PCs and web browsers, and even to other applications running in the cloud; a language built for the cloud needs to scale well across that range of requirements. Second, because quality is important: A language needs to be able to support the organic growth of applications over time, from prototype to MVP to production and beyond, all while supporting a growing development team and the accompanying development process automation. Lastly, because demand is bursty: A language for the cloud needs to scale because there is a potential for millions and even billions of connected clients.

Secure

The Ecstasy language is secure by design, and not through the use of layers of complex runtime logic and security checks. Security is a topic far more complex than can be contained in this article, let alone in a tome, but the Ecstasy principle is exquisitely simple: It must be possible to fully defend an environment from Ecstasy code running within that environment, even (or perhaps especially!) when the environment is itself built in the Ecstasy language.

To accomplish this, Ecstasy uses a container model, but unlike an operating system container, an Ecstasy container has absolutely no surface area from within the container -- not even an API! Code running within an Ecstasy container has no access to any operating system or hardware resources -- it doesn't even have visibility to the fact that an operating system or any hardware exists!

The resources that an Ecstasy library or application requires are enumerated within the compiled form of the Ecstasy code, and when the code is loaded, the host environment can decide precisely what to provide for each of those requested resources. All resources are provided via resource injection, and from the application's point of view, the resources are provided as if out of thin air! The resources (and their injection) are completely opaque to the application, even if the resources are implemented entirely in the Ecstasy language. Furthermore, resources are automatically limited in their surface area to the exact programming interfaces defined by the host.

Lastly, the Ecstasy runtime can only host Ecstasy code; Ecstasy cannot host native code. The only way to have "native" capabilities in an Ecstasy runtime is to inject a resource from outside of the Ecstasy runtime that in turn makes use of native code.

So, why does cloud computing require a secure language? First, because client devices and the servers in the cloud itself must never trust the code that they are hosting. Second, because a developer should never trust a language that doesn't assume the first.

Portable

The requirement for portability is pretty much a given in the cloud, but like so many of these terms, portability has multiple meanings associated with it. Like many modern languages, Ecstasy is a Turing-complete language with a well-defined intermediate compilation form that can be targeted to almost any operating system and hardware environment. It is compilable to a machine-dependent form ahead-of-time (AOT), it is just-in-time (JIT) compilable, and it is adaptively re-compilable (runtime profile-driven optimizations).

Code written in Ecstasy is also portable to the client. Ecstasy was designed to support compilation to Web Assembly in order to support browser deployment, ARM and x86 to support client device deployment, as well as the obvious cloud hosted deployment model.

Ecstasy is designed to support another form of portability: Application portability. By designing around a strict container model and an event-driven programming model, and by designing for the possibility of a fully managed runtime, Ecstasy applications can conceptually be lifted from one server or virtual machine and moved to another.

So why does cloud computing require a portable language? First, because the code needs to run across a variety of development environments and cloud provider platforms. Second, because applications themselves need to be able to migrate within a cloud and across cloud boundaries. Lastly, because the same code and libraries should be easily re-usable on client devices, in IoT applications, in a web browser, and on any other computer.

Reusable

It should be obvious by this point that reusability was high on the requirements list in the design of the Ecstasy programming language. Ecstasy approaches the challenge of reusability in several ways, but the most obvious is through its support for modular programming.

A module is the unit of reuse in Ecstasy. In Ecstasy, a module can be as simple as a few lines of code, or as complex as the world's biggest applications. When one module uses another, it creates a dependency that can be easily defined as optional, desired, or required. Each module contains all of the information to explain what module dependencies it has, down to the individual specific parts of those modules that it uses! Modules can be developed and versioned completely independently, but the dependencies can be specified in terms of versions -- including which versions to avoid.

Modules each have their own identity, including information about where they come from. Modules can be signed, and -- for execution in some environments -- even fully encrypted! Module repositories allow modules to be managed and made available online or offline, supporting capabilities such as caching and download-on-demand from the module provider or a module repository service. For the purpose of prepackaging, a module can even completely embed all of the modules that it depends on -- all within a single file!

Because a module is intended for wide reuse, it can carry all of its versions -- including patches and even pre-release versions -- inside a single, compact file. To manage the complexity of interactions among multiple third-party modules, each module carries all of its integration logic that allows it to support any of its optional dependencies, yet without the use of a language preprocessor, runtime reflection, or other convoluted and error-prone approaches to avoiding hard dependencies.

When working with an untrusted module, the module can be loaded and used in its own isolated container, relying on the secure design of the Ecstasy language and runtime. Not surprisingly, this is how Ecstasy cloud applications are intended to be hosted by default.

So why does cloud computing require reusability, and specifically with the type of solution offered in Ecstasy? First, because developers need to be able to safely reuse their own work. Second, because developers more and more rely on components, libraries, and services developed by others, and must be able to consume those capabilities in a secure, reliable, and predictable manner. Lastly, because without an explicit design for reusability, things get messy as the number of reused components and the complexity of their interactions grow.

Other

In the interest of not making this introductory article into its own book, many of the design elements that make Ecstasy the obvious language choice for the cloud must be unfairly glossed over, but we'll come back to them in later articles:

• Nestable - The Ecstasy container model is recursive, allowing any Ecstasy application running in a container to create its own containers, and so on.
• Simple - The Ecstasy language was designed to be easy to pick up by anyone who knows C++, Java, C#, or Python. Most of the time, you won't even realize that you're using a different language -- except that some of those little things that bothered you have been fixed.
• Compact - The compiled form (IR) of an Ecstasy module is amazingly compact, even when carrying multiple versions and dependencies.
• Service-Based and Event-Driven - Instead of exposing operating system threads, mutexes, semaphores, and other inflexible, error-prone constructs, Ecstasy supports a simple, well-defined service model, using an event-driven programming model with full support for both lambdas and continuations, and full support for both futures/promises and async/await.
• Managed - All of the Ecstasy capabilities are designed to be hostable in a fully managed environment, whether in the cloud or in the palm of your hand. Resources like CPU, memory, storage, and network can be completely metered and managed. Operating system services can be carefully managed as well, preventing unauthorized and undesirable behavior.

Free and Open Source

Undoubtedly, the largest change in software in the past few decades has been cultural and not technological. The term “open” has long been used for marketing commercial software that had some slight yet often-only-theoretical potential for interoperability with other commercial software. Today, most core software components, libraries, operating systems and applications are available in complete source code form for use under open source or software libré license, and many of the specifications and standards – including languages and execution systems – that enable interoperability are similarly open and available. From an economic standpoint, it would appear that the demand for a fundamental set of software standards and components being available as a public good eventually out-weighed the cost of creating and managing that public good (even in some cases lacking any consistent centralized authority!), and the cost of reverting to private goods for that fundamental set of software standards and components is unacceptable for all but the most especial of requirements.

It is in this spirit that the Ecstasy specification (XTC/XVM) and source code will be made available, with its ideas and concepts often inspired from others’ open work, and – if any prove worthwhile – its own ideas and concepts freely available for re-use and recycling as the reader sees fit. To accomplish this, the source code will be made available under both a permissive open source license (Apache v2) and a software libré license (GPL), and the specifications will be made available under the permissive Creative Commons CC-BY-4.0 license.