November 21, 2014

C++ - Generative Programming

Nikon D80 - Alcatraz from Twin Peaks
San Francisco Bay Area from Twin Peaks - Nikon D80

C++ IOStreams are a powerful mechanism for transforming input into output.  Most programmers are at least familiar with C++ IOStreams in the context of reading and writing bytes to a terminal or file.

When a file or terminal is opened for reading or writing by a process, the operating system returns a numerical identifier to the process.  This numerical identifier is known as a file descriptor.
In turn, the file or terminal can be written to by the process via this file descriptor.  The read and write system calls, which are implemented as wrappers in libc, are passed this numerical file descriptor.

Many layers of abstraction reside on top of the read and write system calls.  These layers of abstraction are implemented in both C and C++. Examples of C based layers of abstraction are fprintf and printf. Internally, these functions call the write system call.   An example of a C++ based layer of abstraction is the IOStreams hierarchy.  Out of the box, most C++ compiler toolchains provide an implementation of IOStreams.  IOStreams are an abstraction on top of the read and write system calls. When data is written to a terminal via an IOStream, the IOStream implementation calls the write system call.  Lastly, these layers of abstraction handle things such as buffering and file synchronization.

In UNIX, everything is a file. Consequently, network devices, virtual terminals, files, block devices, etc. can all be written to via a numerical file descriptor - this in turn is why UNIX is referred to as having a uniform descriptor space. With this being said, the basic IOStreams and printf abstractions I mentioned above are not designed to used with network sockets, pipes, and the sort.  The lower layer read and write system calls can be used but there are a number of functions that must be called before writing raw bytes to an open file descriptor that points to a network socket.

The additional functionality that is needed for communicating with network sockets, shared memory, and the like, can be implemented in classes that are derived from the C++ iostream class.  It is for this reason that the IOStreams classes are extended via inheritance.

Over the years, several popular C++ libraries have implemented classes that are derived from the base classes in the iostreams hierarchy.  The C++ Boost library is a popular example.  However; this has not always been the case.  Going back to 1999, the Boost library did not exist and there were one or two examples on the entire Internet as to how to properly extend the C++ IOStreams classes.  

In 1999, I pulled the source code for the GNU compiler toolchain that is available on and derived a class hierarchy to support sockets, pipes, and shared memory. The methods in the classes that I derived from the base classes in the iostreams library were designed to be reentrant and easy to use.  I used generative programming techniques and template metaprogramming to create objects that could be instantiated using familiar C++ iostreams syntax and semantics. The library that I created is called mls and it is licensed under version 2 of the GPL

Since 1999, Boost has come a long way.  It provides support for cryptographic IOStreams, sockets, and all kinds of other fancy stuff. It uses generative programming techniques. It is very clean and I highly recommend it.

If you would prefer to roll your own, then I would suggest downloading the gnu compiler toolchain source code from  From there, you can run ctags over the source tree and begin to dig into the internals of the iostreams hierarchy. I would also recommend the following book 
Generative Programming - Methods, Tools, and Applications.  Last but not least, you'll need a Linux host with a reasonable distribution running on it, such as Fedora.

namespace mls 
  template<class BufType, int direction, class BaseType=mlbuf> class mlstreamimpl;
  template<class Parent, class BaseType=mlbuf> class mloutputimpl;
  template<class Parent, class BaseType=mlbuf> class mlinputimpl;
  template<class BufType, int direction, class BaseType=BufType>
  struct StreamConfig;
  template<class BufType, int direction, class BaseType>
  struct StreamConfig
    typedef typename SWITCH<(direction),
    CASE<0,mlinputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
    CASE<1,mloutputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
    CASE<10,mlinputimpl<mloutputimpl<mlstreamimpl<BufType, direction, BaseType>, 
         BufType>, BufType >,
    CASE<DEFAULT,mlinputimpl<mlstreamimpl<BufType, 10, BaseType>, 
         BufType > > > > > >::RET Base;