Ralph – History
Here is the long overdue story of how I ended up creating Ralph.
After I got my bachelor’s degree in the fall of 2010, I finally had some time to explore mobile application development on iOS and Android. I had a few rough ideas and started implementing them, but soon realized it could not be done as quickly as I had initially hoped.
Mobile application development is a pain
First, I assumed it would be possible to write applications in a common language and share a single code base among both platforms by abstracting away the framework-specific code. However, iOS and Android applications differ significantly in their framework’s language (Java vs. Objective-C), architecture (e.g., lifecycle of screens) and features.
That means code can basically only be shared across platforms by moving it to a native, shared library. As Objective-C is a C superset, calling into a native library is easy, but on Android, calling native functions and exposing Java methods to a native library requires writing additional bridging code using the JNI, which is known to be error-prone.
Second, just implementing simple user interfaces required writing large amounts of Java and Objective-C, and structuring the code in complex and unmaintainable ways. There was just too much boilerplate code that could be abstracted away, and patterns that could be automated (e.g., serialization on Android).
Third, the long edit-compile-and-run cycle reduced productivity significantly. On both platforms, modifying the applications at runtime is not supported directly, which is very unfortunate, as user interface development usually involves quickly prototyping new ideas and iteratively adjusting details.
Interactive development in Lisp is a joy
So an investigation into the possibilities of using a dynamic language which could solve those problems followed. I mainly focused on Lisp dialects, as they commonly have powerful metaprogramming facilities (e.g., macros), and allow interactive development (e.g., through hot code replacement or a REPL).
In the Dylan world, OpenDylan is the only remaining maintained compiler. It does not have a backend that generates native ARM code, but one that generates C, which can be ported to new architectures/platforms with less effort than creating a completely new backend. However, nobody had ported it to iOS or Android yet and my embedded development and compiler knowledge was fairly limited, so porting it would have taken a significant amount of time and effort, and success was uncertain.
In the Common Lisp world, ECL was already ported to iOS and Android, but these ports were unmaintained and it was unclear how to perform interactive development.
In the Scheme world, Gambit-C looked very promising. It could be easily compiled on Android and was used for game development on iOS – interactively using a REPL! However, it only provides a low-level C FFI and was still missing a high-level bridge to each platform’s framework, which again would have required a lot of effort to get working and correct.
JavaScript as a compilation target
Ironically, web development suffers from the opposite problem: JavaScript is the only language natively supported by web browsers, and it has many problematic features, which makes developing web applications directly in JavaScript quite difficult. On the other side, it also has many great features, such as support for first-class functions, closures, lexical scoping, higher-order programming and dynamic typing. This makes it a great compilation target for higher-level languages. As a result, many compilers targeting JavaScript were created.
Using JavaScript as a shared language turned out to be a great solution to my problem. Executing JavaScript and using each framework’s API was still painful, but it was much easier than from a native library, and libraries exist to simplify the bridging. Also, this would allow sharing code beyond native mobile applications to web browsers and even the server-side.
At the time, the list of compilers translating a Lisp dialect to JavaScript was still fairly small, with ParenScript being the only active and stable project. Today, the list of compilers for Lisp dialects is a lot longer, and ClojureScript is probably the most widely used one.
ParenScript’s compiler initially worked well for small programs, but I soon ran into problems while working on larger applications. For example, writing macros was prone to errors, as they need to be written in Common Lisp, but generate ParenScript code. Also, I noticed performance problems testing the resulting apps on actual mobile devices.
After looking at the code generated by ParenScript, I came to the conclusion that it performed a very naive translation to inefficient JavaScript, which had to be optimized further to be usable on low-powered devices. Using Google’s Closure Compiler helped a bit, but often optimizations could not be performed automatically. So I delved into ParenScript’s internals to see how the emitted code could be tweaked for the Closure compiler.
While studying ParenScript I realized that many forms could be translated directly to efficient JavaScript, and figured it could be possible to create a simple compiler for my favorite Lisp dialect, Dylan. This would of course require keeping the number of features supported by the subset small. For example, it would only support single method dispatch, instead of multiple dispatch, and have a simple S-expression based syntax, instead of an ALGOL-like infix syntax, which would require implementing a complex grammar.
Ralph to the rescue
After some research, I discovered that an early version of Dylan was much more Scheme-like and its syntax was still based on S-expressions – a perfect fit. As I’m not good at naming things, I simply adopted Apple’s internal code-name for Dylan: Ralph.
I started implementing a bootstrapping compiler in JavaScript, which only took a couple of weeks, and soon reimplemented it in Ralph itself. A few weeks later I got it to compile itself and thus had a self-hosting compiler.
With the compiler now being written in Ralph, I could stop worrying about JavaScript-related problems, make use of Ralph’s features and focus on improving the generated code. While fixing many issues, I also learned more about proper compiler design and compilation techniques for Lisp-like languages. A complete rewrite of the compiler followed, with proper compilation stages that lower the complex input language into a simpler intermediate representation, and I also evolved the language by introducing a hygienic macro system, inspired by Clojure’s syntax-quoting.
Eventually, Hannes convinced me to write up my findings in a paper and submit it to the International Lisp Conference. Luckily, it was accepted and I got to present a couple of slides about it. Last year I didn’t work much on Ralph, as I was busy with my Master’s thesis, but I’m currently trying to revive the project .