As Rich Hickey says, the secret sauce of Lisp languages is the ability to directly manipulate the Abstract Syntax Tree through macros. Can this be achieved in any non-Lisp dialect languages?
Being able to "directly manipulate the abstract syntax tree" by itself is nothing new, though it's something that very few languages have. For example, many languages these days have some kind of an
To give an example of a language that does have that feature (but not something that would be considered macros proper), see OCaml. It has a syntactic extension system, CamlP4, which is essentially a compiler extension toolkit, and it revolves around the OCaml abstract syntax as its most important purpose. But this is still not what makes the corresponding feature in Lisps so great.
The important feature of Lisps is that the extensions that you get using macros are part of the language in the same way that any other syntactic form is. To put this differently, when you use something like
Now, viewed in this light, something like the D facility is very similar in nature. But the fact that it deals with raw text rather than ASTs limit its utility. If you look at the example on that page,
you can see how this doesn't look like part of the language -- to make it more like Lisp, you'd use it in a natural way:
with no need for a
One important thing to note here is that since D is a statically typed language, ASTs have crept into this mental exercise in an explicit way -- as the
In Lisp, you're essentially getting something very close to this functionality, rather than the poor string solution. But you get even more -- Lisp intentionally puns over the basic list type, and unifies the ASTs with the runtime language in a very simple way: the AST is made of symbols and lists and other basic literals (numbers, strings, booleans), and those are all part of the runtime language. In fact, for those literals, Lisp takes another step forward, and uses the literals as their own syntax -- for example, the number
The reasons LISP is "special" are...
The built-in functionality is very economical:
It supports functions in such a way that new function definitions are indistinguishable from built-in functions:
It supports macros in such a way that arbitrary Lisp code can always be defined in terms of a domain-specific language:
With the above features, you can:
E.g. you can easily implement systems for namespaces, any data structure, classes, polymorphism, and multiple-dispatch on top of Lisp, and such features will work like they were built into Lisp.
But it all depends on your definition. Some levels of "syntactic abstraction" are supported in other languages in quite varied ways. Some of these ways are more powerful than others, and nearly match Lisp's flexibility.
In Boo, you can use syntactic macros to define new DSLs that will automatically be handled by the compiler. With this, you can implement any language feature on top of existing features. The limitation compared to Lisp is that these are evaluated at compile time, so run-time code generation isn't directly supported.
Because of this, you can replace or shim the existing system functionality with your own functionality. This is often quite useful in shimming in your own runtime debugging features, or for sand-boxing (by un-defining system calls you don't want isolated code to access).
The C++ pre-processor allows you to define your own DSL with a somewhat similar syntax to existing function calls. It does not let you control evaluation (which is the source of a lot of bugs, and why most people say
The code generation support in C/C++ macros is limited because macros are evaluated before your code is compiled, and can't be controlled via C code. It is nearly completely limited to textual substitution. This greatly limits the type of code that can be generated.
The C++ template feature is quite powerful (WRT to C/C++ macros) for syntactical additions to the language. It can turn a lot of runtime code evaluation into compile-time code evalution, and can do static assertions on your existing code. It can reference existing C++ code, in a limited way.
But template meta-programming (TMP) is very unwieldy because it has a terrible syntax, is a very strictly limited subset of C++, has quite limited code generation ability, and can't be evaluated at runtime. C++ templates also arguably output the most difficult error messages you will ever encounter in programming :)
Note that this hasn't kept template meta-programming from being an active area of research in many communities. See the boost project, of which a good portion is devoted to TMP-support libraries, and TMP-implemented libraries.