I’ve spoken to many people about this project, including a presentation on it at ADC24 in Bristol, and it’s notable that the issues I had with ValueTrees are not mine alone; many of those people have discussed their own versions of code to route around these problems. Before digging into the actual design and implementation of cello, it’s worth looking more deeply into the problems I hoped to address, some other libraries or frameworks that I looked to as inspirations and models, and my general design philosophies when working on the project.
Ludwig Boltzmann, who spent much of his life studying statistical mechanics, died in 1906, by his own hand. Paul Ehrenfest, carrying on his work, died similarly in 1933. Now it is our turn to study statistical mechanics.
– David L. Goodstein, “States of Matter”
ValueTree Concerns/Annoyances
To reiterate some of the problematic areas when working with ValueTrees:
Type Safety
Because all properties are stored in the var variant type, the burden of ensuring that data obtained from a property is of the same type as the data originally stored in that property is the responsiblity of the developer. You can either expect yourself and all future developers to work carefully or require that all accesses take place through get/set functions, which would need to be created and maintained for each property in use (see also ‘Boilerplate,’ below).
Thread Safety
ValueTrees have no built-in mechanism for enforcing any kind of thread hygiene, and because the implementation of the storage takes place in a separate shared object, there’s not a good place to insert e.g. a critical section object that could take care of this for you.
Initialization
It’s impossible to know for sure that all the properties you might expect to be present in a particular ValueTree are actually there without inspecting it before using it; it’s completely valid to have a ValueTree that is completely empty except for its type, and all ValueTrees begin this way, by instantiating a typed tree with no contents, and then adding individual properties programatically.
While there are cases where this behavior is a powerful way of working, in most of the code that I’ve worked on, the content and behavior of data types is completely defined at compile time, and it’s frustrating to continually need to look before you leap when accessing data inside of a ValueTree.
Boilerplate
Typically, there’s a great deal of boilerplate code to be written when working with ValueTrees:
- Define a set of
juce::Identifiersto name the trees and properties in your project. - Figure out a way to initialize all the properties of a tree, or to wrap an existing tree.
- If you want to add some level of type safety, add a pair of
get/setfunctions for each property in all your trees. - Any class in your application that wants to be notified of changes in any of your trees need to:
- be derived from
juce::ValueTree::Listener - override the appropriate listener methods, typically at least
valueTreePropertyChanged() - manage adding and removing itself as a listener to the tree as appropriate.
- be derived from
Opportunities for Errors
All of the above issues provide many opportunities for bugs of varying severity and obviousness to creep into your code:
- Adding properties to the tree, but failing to initialize them property.
- Since identifiers are handled manually, copy/paste errors when writing
get/setmethods easily create situations where a methodgetFoo()is internally accessing a property namedbar. - The
valueTreePropertyChanged()listener method is called whenever a property changes in a watched tree or any of its subtrees. Failing to check that you’re responding to the change you are interested in is a common error.
Models
I had two models of existing systems or projects in mind when designing cello; each of them focused on the same kind of ease-of-use improvement that I was interested in adding to ValueTrees:
Object/Relational Mappers
Libraries like SQLAlchemy (Python), Hibernate (Java) and ActiveRecord (Ruby) provide the capability to expose data contained in relational databases through an object API that a developer can work with directly, hiding the mechanics of interacting with the database.
Python Requests
The popular Python “Requests” package is an API wrapper that sits over the standard library’s urllib modules; it’s billed as “HTTP for Humans.” I was especially interested in the ways that Requests makes common HTTP operations significantly more straightforward than the corresponding urllib code would have been. Significant amounts of boilerplate code are completely unnecessary when using Requests, and this was something that I very much wanted to emulate in cello.
High-Level Design Goals
Once a list of issues to be addressed was enumerated and the example of some other libraries/frameworks was in mind, I wrote a list of guidelines and goals for the project:
Simplify Use of ValueTrees
Creating a new class or struct that uses ValueTrees to handle its storage should be of roughly the same complexity as creating a class that uses plain old data.
Two core classes provide this functionality:
cello::Valueencapsulates an individual ValueTree property.cello::Objectencapsulates a ValueTree and contains members of typecello::Value.
In practice, creating a new cello::Object class/struct just requires that you:
- Derive the class/struct from
cello::Object. - Make sure your constructor(s) correctly initialize that base class.
- Add one or more members of type
cello::Value.
By restricting it to act as simple storage, you’re encouraged as a developer to keep business logic partitioned from UI display or input from the outside world. The disparate pieces of your application will remain decoupled and unaware of each others’ existence, just communicating anonymously through this data backplane.
Access Properties Using Dot/Arrow Syntax
This was a crucial goal in the project: working with ValueTrees feels like calling an API. I wanted it to feel like working with a regular C++ data structure, accessing properties naturally:
MyCelloObject thing1;
thing1.x = 100;
int yValue = thing1.y;
// arrow syntax works as well
MyCelloObject* pThing { &thing1 };
int xValue { pThing->x };
Note that because our ValueTree properties are represented using real member variables in a class, the issue of knowing whether or not a property is present goes away; attempting to access a property you haven’t defined a cello::Value for will be code that refuses to compile.
Type Safety
The cello::Value objects need to enforce type safety for us, and should be able to support any type of data that can be round-tripped losslessly through a JUCE var. Any attempt to store data of any type other than what the Value is expecting should result in a warning or an error from the compiler.
By writing a suitable template specialization of the juce::VariantConverter structure, we can define a method for serializing arbitrary data types including complex data structures through a var for storage in a ValueTree. Once this VariantConverter exists for a type, we can use that type transparently in a cello::Value, and the var conversion will be handled automatically as the data is stored into and retrieved from that Value; there’s no casting or other manipulation needed by your code.
Data Validation
While we’re at it, these Value objects should be able to validate data being stored in their property, both on read and write. This way we can do things like restrict data to valid ranges, or make sure that class invariants aren’t broken, or a bunch of other useful things.
A (perhaps non-obvious) side effect of this is that this mechanism means that there’s no reason for your Value members to be private any more, and it ends up being cleaner to avoid classes and define your cello::Object-derived types to be structs instead.
Thread Safety and Interprocess Communication
We’ll develop a facility to work safely across thread boundaries (using the already extant juce::ValueTreeSynchroniser class and some other clever bits), and then extend that design so that we can just as easily send and receive data between processes or physical machines using named pipes or sockets.
Explore Reactive Programming
One of the common techniques in JUCE applications for notifying different parts of your code that there’s been a change in state is to use the juce::ChangeBroadcaster and ChangeListener base/mix-in classes, which are a fairly low-effort way to send and receive these notifications, but they’re extremely coarse, typically sending a notification that some unspecified value in an object has changed. This is fine when the intended semantics are “hey, UI component: repaint yourself because I changed.”
cello is designed so that you can be notified of changes at the granularity of individual Value objects or changes to the Object itself (adding/removing/re-ordering children).
Connecting a UI component directly to the Value that contains its state is trivial with this design.
The less obvious way to think about this is that under this system, changing a Value is a mechanism to cause the execution of code that your logic making this change is completely decoupled from. Changing a Value in some piece of logic can end up writing a line in a log file, sending a message to an analytics server, generating and sending a MIDI message, and updating one or more UI components.
This can be a difficult thing for developers to wrap their minds around. Spend some time on the JUCE forums and you’ll see developers going through a few stages:
- Application logic written directly inside UI component code. These developers eventually run into the problems that this approach leads to–brittleness, inflexibility, difficulty extending the code, etc.
- Logic and presentation separated, but still tightly coupled. These developers end up needing to do something like display lists of things that come and go and end up needing to build mechanisms to respond to changes in the list or other dynamic changes to the data model.
- Logic and presentation separated and decoupled. This is where
celloleaves you; you still need all parties interested in a piece of data to have access to a shared state, but we give you an easy way to explore that state and work with it dynamically.
Theses
- The best way to understand something is to use it in earnest. Something that I’ve learned about myself it that I frequently have initial encounters with a new piece of tech that rub me the wrong way, typically in some trivial way, but then I do my best to avoid any more contact with that thing. I also frequently, when pushed to actually use that thing, discover that either my initial feelings were wrong to some extent, or that I still don’t care for the thing, but can express a list of reasons instead of just a vague unease.
- The best way to critique something is to make something you like better. As a composer, I’ve always tried to follow this rule, and frequently do so as a developer as well. One of the best parts of being a software developer is having the ability to craft tools that fit your hands perfectly, and that was very much how I approached this project.
- All problems in computer science (except one) are solved by adding another layer of abstraction. Proof of this left as an exercise for the reader, including pondering what the exception case is.