Toddlers vs Twombly: Structured Thinking and Wasted Intelligence in ML Models

It is hard to convince most people why their toddler’s artwork on the fridge is worth zero dollars, while Twombly’s ‘Leda and the Swan’ is worth a small fortune, yet I’ll give it a go. Look up at the corner of the room you’re reading this from and find a straight line. Keep looking at it. Focus, blurring out everything else – is it still straight? The answer is no. You see, a straight line is impossible. Now your eyes might wander trying to invalidate the claim, but you’ll find that if you look at any line long enough it ceases to be straight. If this surprises you: why have you never noticed this? Well, your brain does a lot for you and one of its most important functions is reconciling the abstract world of ideas with the physical world. The marriage between wooden block and chair happens in your mind first. This is hardly breaking news, and in a big way the basis of simple but evocative visual effects in art. Using the fact that our mind fills in the gaps so to speak, interrupting this process can produce real emotions in the viewer. In a way this is what Twombly does, he takes things that you know to be true, things that you recognize, and he twists them and malforms them enough that you feel as though you shouldn’t, so that in the back and forth along the line between nonsense and perfect sense you get vertigo.

I am hardly qualified to speak to Twombly’s genius, but more and more, the classic layman’s critique of Twombly’s work finds its analog in the contrast between fully AI generated essays, papers, and ideas and human ones. To the layman they look increasingly identical, but to the well-informed, structure (or lack thereof) invalidates the former and gives the latter its character. In relation to works of art, structure goes beyond line, form, and color or the motifs composed of the three across the canvas, but how each of these elements contributes to the final product. Put simply: what they add and how they add it. The way these modular components are combined for maximal effect characterizes a kind of creativity on the part of the artist, and defines the nature/extent of the achievement that is a work of art. Organizing those elements to maximal effect, in maximally efficient service to the final product is a skill no toddler can boast of. Similarly, written works are made up of fragments – words, sentences – that may be combined to higher order motifs – ideas, metaphors, etc – and the effect each has in service of the entirety of the written work and its purpose again characterizes a kind of creativity or literary brilliance. Moreover, service in this context means something very specific: Fragments in service of the whole narrative are part of a dependency structure on which the main idea/purpose rests. LLMs, however, don’t operate within this paradigm. Although dependency structures underlying narrative throughlines increasingly can be retrofitted to LLM outputs, the causal relationship between fragments is not guaranteed. Indeed, at this present moment, there is no means of guarantee. That is, we do not yet have the ability to outline the rhetorical aims of an LLM output and fragment for fragment the structure used to support it with certainty. So as LLM outputs have become increasingly identical on the surface to the real thing, their integration into critical systems, where certainty in that causal relationship between fragments is paramount, has also stayed surface level.

In industries where dependability is paramount, where human lives are at stake, LLMs serve merely as an appendix to facilitate human endeavor at human discretion. As models continue to get more intelligent, capable of more and more complex ‘reasoning’ this leads to a double jeopardy of sorts, where the problem of undependable LLM outputs and analysis is also one of wasted intelligence, if the uncertainty of the causal relationships stays unaddressed and they continue to fall short of full integration. That is, thinking of the fragments and input data as a resource, and the higher order concepts and ideas that result from analysis are valuable products from said resources, more of which are made available at higher levels of intelligence: a toddler’s ability to aim towards organizing and marshalling motifs and fundamentals towards a larger cohesive idea pales in comparison to an adult lesser artist whose abilities would in turn pale to Twombly’s. Thus real derivable products from unprecedented levels of investment and intelligence are left on the table so to speak. Whether the causal relationships between fragments can be made certain — not retrofitted, but guaranteed by construction — is, I think, the most valuable open question in applied AI. It is the one I am working on.