Make of It What You Will

Put a photograph of an empty road next to one of a packed suitcase, and a story begins to emerge through the act of putting them together. Add a third photograph, maybe a beach or a hospital or an old rickety house, and the story changes. Most of us are quite good at projecting a story through these types of associations. Whether it’s the story someone intended is another question. That’s where the exchange becomes interesting: they might elaborate, challenge, or offer something we hadn’t considered. We come away understanding both the arrangement and a little more about the person who made it.

I’ve been thinking about how that kind of empathy could show up in a user experience. Could individual interface elements be encoded with meaning, so a system could interpret something about the point of view a person constructs with them? And could a dialogue between the person and the system reshape that interpretation, changing how the system reads what the person makes next?

There’s something about the limited vocabulary of magnetic poetry that makes it a particularly interesting place to explore this. Individual words carry their own associations, but put a few together and they can evoke something entirely different. The possibilities may be constrained, but the meaning isn’t.

I built a little experiment to see what could happen.

The words are ordinary, but the interface elements are hiding some interesting math. Each carries a numerical representation generated by MiniLM, a small language model, that describes its relationships to other concepts. Not a dictionary definition, but a kind of map of semantic proximity. I’ve also given the system six moods—joy, calm, wonder, melancholy, unease, and defiance—to use as reference points. As you assemble a line, the interface compares the encoded meanings of its constituent words against those moods and combines the results into a reading. Each piece carries something of its own meaning; their composition gives the interface something to interpret.

Of course, a collection of words isn’t the same thing as a sentence, and a sentence isn’t the same thing as an intention. Sunlight, Storm, After, and The might suggest a mood quite different from the one any of those words carries alone, and can change based on their arrangement. So I included also an optional whole-line reading analyzer, using an active version of MiniLM to consider the phrase as one object. Flip the switch (and download a 46 meg file) and you’ll see a more holistic interpretation of the words against the sentiment map. The two approaches don’t always agree. I like that they don’t. They expose the distance between the interpretation of the parts and the interpretation of the whole.

Then comes the experiment in empathy, which begins when the system has to confront someone else’s interpretation of what they’ve made. Before the machine reveals its judgment, you’re asked what you meant. There’s a reason for the order: I wanted your interpretation to exist before the interface offers you its view, which you might feel obliged to agree with, or game. In those scenarios, I’ve added a function you can call with the Read it my way button. The system adjusts the stored representations of the words in your line, calculating how it should nudge them toward the mood you intended. Related words not used inherit some of that adjustment. Those changes are kept in the browser and can affect subsequent readings.

It’s a modest mechanism that can’t claim to acquire insight into the user’s inner life. Within its little world, though, the system can revise some of the relationships it uses to interpret what you make next. Not quite empathy, but not unempathetic either.

One could imagine a different interface assembled from products, pictures, priorities, places, or ideas, each having its own embedded semantic relationships. What interests me is the possibility of meaning emerging from composition, without turning every encounter into a form about what the user wants.

There are plenty of ways to push this further. Corrections could belong to a particular phrase rather than to its ingredients. A person might choose two moods, reject all six, or explain what was misread. I’m also curious about transparency: whether showing the scores changes what people choose to say. If I start adding words to make the machine register melancholy, whose point of view am I constructing now? It seems entirely possible that, in teaching the system to understand me, I’m also learning to make myself understood by it. Can an interaction have empathetic qualities even when neither the mechanism nor the intelligence behind it possesses empathy? Whether that’s the beginning of a more empathetic relationship or a rather sophisticated form of accommodation is something I plan to explore further.