Exploring how refl-cpp — a compile-time reflection library for C++, works under the hood.

Disclaimer: I am the main developer behind refl-cpp.

Table of Contents

  • Preface
  • Starting with the Basics
  • Building the compile-time member chain
  • Putting it all together
  • Overloaded member functions
  • Enumerating the member metadata at compile-time
  • Applying an operation to a list of types
  • Associated constant objects — Attributes
  • Proxies- Changing the behavior while preserving the interface
  • refl-ht — the refl-cpp preprocessor (optional)

Preface

Let’s start with the basics, by covering what refl-cpp aims to do and what it does not aim to do.

What refl-cpp aims to do:

  • Provide a means of compile-time reflection in C++17 & up.
  • Support compile-time enumeration and introspection of a type’s members
  • Support type templates and member templates
  • Support compile-time value-based decorations of types and members (called attributes in refl-cpp).
  • Be easy to use while having a type-preserving interface, without requiring the user to resort to metaprogramming

What refl-cpp does NOT aim to do:

  • Not use macros since they are bad and ugly and stupid (I am a firm believer that macros are rarely the solution to a problem, but also that there is a place and time for them even in C++)
  • Support reflection of private members (please, comment below if you have a use-case that requires that).
  • Support querying type information by name at runtime.

With these points in mind, let’s look at some of the things one should have in mind when developing such a solution.

The Basics

First and foremost, refl-cpp is a compile-time reflection library. That means that instead of maintaining a run-time structure that resembles something like this:

refl-cpp instead works by storing the metadata in a type-dependent manner, through template specializations:

Nothing new or different about that. The type information for a type can now be accessed with TypeInfo<MyType>. The first benefit with this approach is that now the members of the particular TypeInfo specialization are not inter-dependent anymore. There are some downsides to this approach too, such as that now types cannot be discovered at runtime by their name, but that would generally be of very little value in a language like C++ (just imagine passing untyped data everywhere). refl-cpp provides the tools one needs to take the compile-time metadata to run-time though.

Building the compile-time member chain

Now we have a means to “store” information about a type, but what about its members? Member information is stored in quite a similar way to type information. refl-cpp uses a novel (to my knowledge) strategy to build a list of types, which represent that member information.

The list of members is based on template specialization on a size_t template parameter under the hood. It’s just that in refl-cpp, this MemberInfo template is enclosed in it’s declaring TypeInfo.

You may have noticed that additional typename Dummy parameter, it’s only needed due to the fact that in C++, full member template specializations are disallowed, but partial member template specialization are fine.

Putting it all together

OK, we can now use specializations to specify member reflection data and we can also address them with the common name MemberInfo<N, void>. Before continuing with some of the other features of refl-cpp, let’s see how this structure is actually created by the user. As you probably already have guessed, it is done through the (prudent) use of macros.

An example of the resulting code after preprocessing can be seen here.

There’s more to that in refl-cpp, though — every TypeInfo and MemberInfo has at least the following:

  • static constexpr char name[] = …
  • static constexpr std::tuple<…> attributes = {…}
  • static constexpr auto* pointer = &Type::MemberName

Now, about these overloaded (possibly template) member functions

refl-cpp also allows reflection of member functions, of course. And to distinguish between member types, each MemberInfo has a public typedef, that is equal to one of refl::members::field or refl::members::function (tag types). With function MemberInfo specializations there is a bit more trickery involved to pull everything in place. That is due to the fact that member functions can be either or both overloaded and/or templates.

refl-cpp again uses a different approach to solve the problem of taking a pointer to a member function here. It is based on requiring the user to pass the parameter types they want to call the function pointer later with and looks like this (not accurate as of v0.6):

That might be a lot to go through at once, so let’s break it down. What is the problem this code is trying to solve? Imagine having a type A with two overloads of a function — f(int) and f(const std::string&). When trying to take a pointer to the function f (by &A::f), how is the compiler going to know how to resolve that expression to a specific definition?

What refl-cpp does is aid the compiler in deducing the proper overload in the same way it does when directly invoking the function — by passing the &A::f to another function as a parameter, which is one of the cases when the c can be concrete type can be resolved (when the context it is passed in has a specified type). resolve has no definition, just a prototype — that is simply because it is never needed. It just acts as a hint to the compiler. The nice thing about passing the functional arguments (resulting from std::declval) is that all the usual argument conversions apply, meaning that one can use MemberInfo<?>::pointer<int> and get a pointer of type void(*)(long) as a result (which would be fine to invoke with an int).

Enumerating the member metadata at compile-time

Now, we’ve covered more of how member metadata is created and stored. What about enumerating the members of a type? Well, they are already represented as types. We just need to build an abstract list of them to allow for common operations to be easily defined. We can do that with a variadic template. We just need a template to parametrize over the unknown number of types.

But in order to create a TypeList<…> of the members of a type a few more steps are needed. That TypeInfo::MemberCount field is going to be of use as valid indices for the MemberInfo type of a TypeInfo are the values 0..MemberCount. To create a list of discrete values from 0 to N one can use std::make_integer_sequence<T, N>, in particular, its partial specialization — std::make_index_sequence which returns std::index_sequence<0…N>. It’s just that for us to use the generated indices, we need to pass that specialization in a way that can be generalized over.

The template non-member function EnumerateMembers takes care of that, we don’t even need to define an implementation for it as it will never be needed at runtime. Now MemberList<T> will be equivalent to TypeList<TypeInfo<T>::MemberInfo<Idx>…>, where Idx is a template list of valid indices, which is exactly what we needed.

Applying an operation to a list of types

Now that we have a list of a type’s members, we might want to be able to apply some common types of operations to it. The type-preserving nature of refl-cpp, allows for two types of implementations of these operations which I like to call trait-based and function-based, and which are implemented in refl::trait and refl::util respectively.

A trait-based operation looks like this:

The operation is based on a trait-like type mapper/predicate. The results are aggregated into a TypeList<…>.

Whereas a function-based operation looks like this:

Essentially, function-based operations serve as a bridge between type-based and value-based data. In refl-cpp they are most-often used for applying operations to all members of a type or filtering the list of members according to a predicate based on runtime-only data.

The implementation of each has its peculiarities, so I won’t be going into the nitty-gritty details in this article. You can see the original implementation for the trait-based map operation here and the function-based map operation here.

refl-cpp also provides a few other familiar operations such as for_each, filter, count_if, accumulate, contains, etc…

Associated constant objects — Attributes

Associated constant objects — called attributes in refl-cpp, bring a whole new spin on the reflection game in C++. With them, one can not only introspect the members of a type, but also provide and request additional information about the reflected item. Here is an example of attributes that is compatible with present-day refl-cpp.

Attributes are passed to the appropriate macro as variadic parameters and are passed to std::make_tuple under the hood! This allows for arbitrary values to be associated with any member or type as long as they are constexpr-constructible. refl-cpp also has a system for detecting duplicate attributes and verifying proper attribute usage. A user attribute class simply needs to derive from one of refl::attr::usage::{any, type, member, field, function}, which intuitively specifies the target requirements for the attribute. Attributes can aid in designing serialization systems, DAO systems and many more.

refl-cpp comes with a slim set of built-in attributes that are accessible in macro context without prefix (but are not part of the global namespace). Those attributes are the debug attribute (used to provide custom behaviour for the refl::runtime::debug function), the bases<Ts…> attribute (use for explicitly specifying the base types of the target), and the property attribute (included due to how ubiquitous the notion of properties — values exposed through accessor methods is). All built-in attributes exist in the refl::attr namespace.

Proxies— Changing the behavior while preserving the interface

All reflected members in refl-cpp also have an internal associated template called remap<T> that has a member named the same way as the original member, but delegates the implementation of that member to T. Here is an example of how the generated code looks.

You might notice that static_cast<Proxy&>(*this) expression. That, of course, relies on the Empty Base Optimization (required for StandardLayoutTypes since C++11) to kick in in the case of inheriting multiple disjoint remap implementations. That little gotcha is why this template is not directly available for users. Instead, all that functionality is wrapped in the refl::runtime::proxy type. It automatically derives from the remap<T> template of the member descriptors of the target types and sets up some workarounds for older compilers to enable EBO in that multiple-inheritance scenario. Users can then derive from refl::runtime::proxy and implement a static invoke_impl function that can seamlessly provide the implementation for the member. You can refer to this usage example.

refl-ht — the refl-cpp preprocessor

And finally, the tool that makes using refl-cpp a lot more enjoyable — refl-ht (refl-cpp header tool). Whilst still in preview, refl-ht is meant to be the yin to refl-cpp’s yan. refl-ht is an unobtrusive, fast, portable (WIP) utility that currently uses clang under the hood for parsing the user’s source code. refl-ht searches for the markers left by the unified REFL(…) macro and generates the appropriate metadata, then emits the equivalent REFL_TYPE/FIELD/FUNC macros in a reflmd/<input_file> file. The input file is then decorated with a #define REFL_METADATA_FILE that points to that metadata file and an #include REFL_METADATA_FILE directive is added at the end of the file (if missing).

As you can see, refl-ht is really quite unobtrusive. It does not severely modify the input files nor does it change or dictate the project structure. A simple ./refl-ht --input=main.cpp invocation is all that is needed for the metadata for a project with that main file to be generated (included headers are automatically scanned for refl-cpp usage). All generated source is conveniently located in a separate folder called “reflmd”.

refl-ht only depends on clang and can be extremely portable. Furthermore, should you decide to ditch refl-ht and only use hand-written metadata, that scenario is completely supported and hassle-free. The generated header files by refl-ht do not depend on the tool being present and are compilable with any compiler supporting refl-cpp.

I hope you liked this deep dive in the refl-cpp compile-time reflection library. Comments are also very welcome. You can also join the discussion on reddit.

GitHub repo: github.com/veselink1/refl-cpp