Binary Interfaces and Language Interoperability – xlang

How does a program call a function in the operating system? If so many languages can all call operating system functions, why can’t any programming language call functions in any other programming languages more easily? How can we make this work?

Usually for any given one language, interoperability is defined by the language or runtime, and the code that you need to write is unique for that toolset. In almost all cases, the one “other language” that you can call into is “C”, although that can sometimes be used as a bridge to other languages.

Why are language interop solutions all unique?

Every language has it’s own object management, data management, and feature sets. Enabling full-fidelity access to these features requires domain-specific design for that language. For instance, if a language uses garbage collection, object implementations may need to be able to trace references correctly to participate in cleanup. String data may need to be stored in a global string table, or allocated using the same allocator the runtime uses. Dynamic languages may need the ability to enumerate functionality on the native objects. The models used for language extensibility can be faster and more expressive by being specifically tuned to the design characteristics of the language and runtime.

However, even in this short description, you can see emerging patterns that span many languages and scenarios. xlang doesn’t attempt to replace the native interface layer for any one language, but instead provides a framework for mapping common concepts across many languages.

Why is C so magical?

As a “low level” language, C function definitions can specify their calling convention and exact data in memory. A calling convention specifies how control and data is passed from one routine to another as defined by machine instructions and memory layout, rather than by programming language semantics. While it’s easy to see how this might allow different programming languages to work together, it also can be critical to allow two pieces of code written in *the same* programming language to work together when using separate toolchains.

Operating systems typically define their functions in C for this reason. A C (or C++) function that specifies its calling convention and uses size-specific types effectively defines the instructions, registers and memory layout needed to make a function call. We call this type of specification a binary-stable function definition.

xlang’s predecessor, WinRT

Windows took this approach a step further in the 90’s, and created a set of conventions that also specified a binary layout for a virtual function dispatch table (often referred to in C++ as a v-table) and a few fundamental operations that enabled developers to use object-oriented programming through binary-stable interfaces to make calls across programming language boundaries. This system was known as COM, and was a fundamental building block that enabled Visual Basic to call objects implemented in C++ and shipped as binary code by the operating system or by other developers. It also provided the foundation for OLE, which allowed applications to embed controls implemented in other applications.

Around 2010, Windows introduced the Windows Runtime type system. Building on the COM model of binary stable interfaces, the Windows runtime added language-independent metadata for describing types and well-defined interface call patterns that could be expressed as common object oriented constructs like static methods, constructors, properties and inheritance. Today this is used to support calling directly between a wide range of languages including C, C++, C# (and other .NET languages), Rust, JavaScript, Python, and others.

The Windows Runtime model has been very effective for exposing operating system features, but it also has certain characteristics that limit how & where it can be used. It builds on COM, which requires the COM runtime to be initialized & present. It also relies on specialized support built into the operating system. This means that the latest features are only available on the latest Windows OS’s, and it can’t easily be taken to other platforms.

Introducing xlang

A project that my team is developing called xlang seeks to solve this problem using the same approach that Windows uses to make functionality written in one language callable by a code written in a wide variety of programming languages.

In simplest terms, the xlang project is a refresh of the Windows Runtime type system without the hard coupling to the Windows operating system. It defines a binary-stable interoperability model for languages that isn’t tied to a particular platform or toolchain. It also represents an evolution of the Windows Runtime, improving scalability & performance based on our experience living with the system for a decade. Because it borrows heavily from Windows Runtime concepts, we’re continuing to move & recreate some of our Windows Runtime tooling in the xlang repo as well so that we can contribute support for new languages & features back to the Windows Runtime as well.

As a recent example of that progress in action, the latest release of C++ /WinRT is now built from the xlang repository.

I’m looking forward to writing more about this project, it’s inspirations, progress, and goals over the coming months.

Thanks for reading along.

Ben Kuhn

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s