Stage 1.X—Meta-Design with Machine Learning is Coming, and That’s a Good Thing

An Example of How Inserting a Meta Design Component into a More Traditionally Formulated Design Process Could Work

Here’s how meta-design would work to affect the creative composition of, say, a catalog or book. Just for fun, let’s say the meta-design program is called “Milton.” A design director would assemble the contents—the text, the images and any ancillary graphic elements.[c] This assembly of contents constitutes stage 1 of the design process. But now, instead of immediately beginning iterative manual sketches, a process of layout and format conceptualization that is stage 2, stage 1.X—Milton—would be called upon. The goals of the project as delineated in the project brief (including such information as audience profile, cultural or stylistic milieu, and possible associative alliances) would be input into Milton. The design director might request 10 or 20 sets of layouts. Rather than the design director producing sketches, Milton would then set up or facilitate stage 2 by running out the requested number of iterations, in a variety of compositions, each taking into account the parameters of the brief. Now working within conventional stage 2, the design director would review the layouts, make decisions based on these candidate solutions, editing and manually revising as required.

Just as happens upon completion of a conventional design process, feedback is received that informs future similar iterations. The feedback would proceed through two channels, as depicted in Figure 3.

Figure 3: As Stage 1.X, or “Milton,” is repeatedly incorporated into a designer’s process, over time, Milton begins to learn the designer’s formal preferences. Two feedback flows are created— one guided by analog input and the other through artificial intelligence channels— that combine to inform the decision-making of the designer.

The first, a conventional channel, would be used to allow the designer and client to note the apparent success or weaknesses of the results of the design once it is produced and distributed into the world. The second feedback path would be directed to Milton. Part of the machine learning training for Milton is this continual feeding back that informs Milton about which edits and changes were favored by the design director (expertise). In much the way as speech recognition programs get better at learning a speaker’s dialect and diction with time and utilization, so too will Milton begin to learn the design director’s formal preferences and come closer to aligning with them as similar projects are undertaken in the future. Milton, over time, acquires your taste. Also—by the way—over time and with repeated usage, Milton influences your taste. This mutual reactivity, an inherent component of meta-design informed by machine learning, can be expected to lead to innovations, some of which will be mentioned later.

Examining the Critical Modules of Stage 1.X: Expertise, Semantic Profiles, Inference Engine

Now let’s look more closely at what constitutes the three essential modules—expertise, semantic profiles, and inference engine—that comprise stage 1.X.

E: Expertise Module

Expertise is the most difficult of the modules to produce and effectively operate, and why there have been no eloquent meta-design programs to date.[d] Digital technology is very good at quickly making yes/no, on/off decisions, but understanding what a goat represents in Tanzanian culture, or the aesthetic feel of vernacular-versus-modernist genres has almost always been thought to be the special domain of actual people, not machines. I grant to those that say meta-design with AI is impossible that this is a really difficult problem. It’s more difficult than faking a face-on-body transposition, or making an AI image of a bird, cat, or Princess Leia. It’s difficult because it does not rely on iconicity (pictorial resemblance), but on appropriate metaphorical and symbolic discourse within a subculture.

It’s a difficult problem, but, at least at basic levels, such as those that address formal layout and composition, it is not insurmountable. For instance, similar problems arise in areas such as mineralogy, where a particular rock must be defined as a type. Very few species of rock are “pure” examples of their type. So how does one decide how to classify a vague or ambiguous sample? It requires years of experience for a geologist to acquire the expertise to consistently make that judgment with accuracy. But recently, using fuzzy set theory[e] and applying it to digital systems, programs have been developed that can begin to approach the expertise of the finest geologist.[5]

Designers, especially those who have many years of experience within the industry, have absorbed a tremendous amount of cultural and implicit knowledge. Understanding how to employ both grids and classical compositional systems, the links between stylistic forms and cultures, looks and styles preferred by various demographic groups, a judicious use of typographic choices—all of these are encyclopedic, semiotic domains of knowledge. Being able to take this expertise and fold it into an algorithm is key to employing fuzzy logic within the AI system. To what extent this expertise can be encoded in AI is yet unclear. Using a grid to guide composition is one thing, but at some point we humans pass from understanding how to use a grid to understanding the cultural relevance of a grid. Can we expect that level of cultural expertise to be translatable to an algorithm? We should be aware that on the immense spectrum from our current dumb templates to sophisticated cultural knowledge, there are many hash marks. It’s reasonable to assume that at least to some degree, the knowledge of the linkages between cultural connotations and compositional form can find its way into Milton.

Sp: Analyzing the Semantic Profile Module: Presence, Expression, Denotation, Connotation

The Expertise component would then be filtered through a Semantic Profile module. Semantic profiles are a kind of “personality profile type” for a display.[6] Semantics, taken in its broadest interpretation, involves all of the effects, or impacts, that a visual display elicits. There are four basic classes of these effects: presence, expression, denotation and connotation.[f] The semantic effects of presence and expression constitute the “affective register,” while denotation and connotation constitute the “conceptual register.”

Presence, the most context-dependent of the four semantic effects, refers to the degree of attention-attraction a display might be expected to elicit, solely on the basis of its color, contrast, size and other qualities of its syntax.

Expression, also within the affective register, is “feeling from form.” It is the degree and particular kind of feeling engendered within the receiver from a display, due solely to the display’s form and syntax. It does not refer to feelings or emotions that may ensue based on the “content” or subject matter of the display. For example, a photograph of a political icon whom you vehemently loathe might yet be very low expression if, for example, his portrait was taken in a completely unremarkable, mundane manner and printed in faded gray ink on a white page. On the other hand, the “Obama Hope Poster” was high in expressivity, no matter what one’s political leanings, because of the deployment of color, contrast and cropping within this display.

Denotation is within the conceptual register, and it refers to the nuts-and-bolts informational content of the display. If, in order to pay a visit to your dentist who has recently moved, you are looking at a building’s directory, it is the denotative semantics that gets you to the right office. Legibility is a denotative variable. In the famous Josef Müller-Brockmann Musica Viva[g] poster series, regardless of the presence or expression various typography clusters might possess, it was their denotative success that got the audience in their seats on the right day and at the correct time. (Note, however, that what makes these posters famous is precisely the other three semantic vehicles: i.e. everything that is not denotative about them.)

Connotation, the most personally variable and experience-dependent of the semantic effects, has to do with all of the associations and connections that a receiver makes, both consciously and sub-consciously, upon seeing the display.

A semantic profile assigns, for a display, a value of importance, called valence, to each of these four classes of effects. The simplest semantic profile uses a bi-valent value system, ascribing a target of either high or low valency to each semantic category.[h]

The Semantic Profile (SP) module provides a way for the program to make decisions about emphasis, hierarchy, and tone. The SP module probably will not specify the particular quality of expression, only that the display is expressive. It doesn’t specify what connotations, only the degree of connotative richness; likewise, it doesn’t pinpoint what the specific denotative content is, only how important denotative clarity is to the overall message.

I: Inference Engine

Meta-design procedures must, as the final stage, be run through an Inference Engine module. The Inference Engine Module essentially involves running a fuzzy logic algorithm off of the input from the e and SP modules. It is a means to facilitate the making of guesses, and dumb guesses at that. In a composition that incorporates typography, heads and subheads need to be identified, captions need to be linked to their images, and textual information needs to be provided with hierarchical labels (much as we do now with style sheets). Any physical parameters, such as size and colors, need to be specified. Given this framework, the inference engine will begin to produce compositions—to be played out at full size or reduced to thumbnails, as preferred.

Exploring the Benefits of Meta-Design

Imagine that you are the design director of a company that publishes on-line and print catalogs. What if once you receive the photos and the copy intended for use in a given issue, you could turn to a staff of twenty layout artists and have each one of them lay out the entire catalog. The twenty designers come back in two weeks, and, from those twenty layout design concepts, you choose the one you think is most exciting and best solves the visual communication problem. Now, of course, working with twenty design assistants in parallel on a single project would be an inefficient and impossibly costly thing to do! No one does this; instead, the job goes to a single specialist (often the design director herself) who sketches out some potential “looks.” Once these have been determined, possibly a team of people work—each on a single section—and adhere to a fairly rigid, pre-determined layout format. But imagine that you had those twenty designers, and each of them were good typographers and adept at composing catalogs—you would have the extreme pleasure of choosing the best from a set of twenty very fine layouts. Of course, you could still revise the design, and if you really had a large budget at your disposal, you could even add some freshly-shot master photography and repeat the entire procedure, making variations on that winning compositional theme, refining and adapting the chosen look until you were completely satisfied in every detail.

Meta-design with machine learning takes those twenty designers and places them at your service. Instead of requiring two weeks to produce the twenty layouts, they give you the twenty layouts in two minutes. You can still tweak, revise, and do variations on the theme. In the time it takes you to do some rough conceptual sketches of a few possible compositions, you are given as many compositions to study as you wish.

Not only that, those twenty iterations that you are provided with in a couple of minutes are completely “live” in terms of content. In other words, they are not created using so-called “Greeking,” or other forms of simulated content, unless you ask the meta-design module to do so. Any tweaks you make immediately reflow the live information throughout the design. If you change typeface or type size, the entire layout, or array of layouts, accommodates it just as it does now in InDesign. The format doesn’t change unless you ask to see additional formats and repeat the procedure.

Finally, just like the design office with twenty designers in your employ, and, maybe to an even greater degree, your designbot assistant learns your taste. Over time, the iterations you are offered grow closer to what you consider refined, workable, functional and eloquent. The Expertise module is noticing your preferences and associating certain modes of visual style with particular content areas or audience groups. You can feed the program samples of compositions you like from the award pages of design magazines. The machine will not plagiarize them, but it will use the visual characteristics to expand its library of potential influences. As you work with it, the baseline Expertise module is learning and adapting to an increasingly clearer universe of discourse. Within a year or two, you may find that your particular AI meta-design assistant is surprising you with unexpected, but uncannily appropriate compositions. The meta-design program begins to be a creative extension of yourself as an individual designer. You and it are blending in a peculiar way, you begin anticipating what it will make, and it is improvising off of what it “senses” you like.

Design and culture critic Kenneth Fitzgerald has written, “Economic forces play the crucial role in determining graphic design’s capability.”[7] If that is so, then the speed advantages of meta-design, once translated into costs of production, will inevitably lead studios to meta-design solutions.

Missing Link 1: Expertise? And Who’s Expertise?

It’s easy to draw a diagram with an “Expertise Module”—it’s a lot more difficult to actually encode that expertise in a functional algorithm. Certainly, the procedure of programming and training will, at least on the surface, be vastly different than what happens in a university studio where students, over the course of several years, begin to acquire the ability to compose complex information. Even the most raw and naive Freshman design student comes into a classroom armed with a lifetime of cultural associations, aesthetic preferences, and the ability to make visual choices. The burgeoning AI algorithm has none of that. It must “learn” things such as the gestalt principles of visual perception without actually having visual perception. That the technical challenges are severe is one reason we don’t already have advanced AI bundled into the Adobe Creative Suite. And yet...

In some ways what happens when a student learns graphic design is precisely the same as developing an AI meta-design agent. They progress from being decoders of visual messages, to being encoders of them. First, students are provided a vocabulary so they can conceptually subdivide what had previously been undifferentiated visual material into discrete parts. That conceptual and verbal definition allows us to talk to each other about the variety of formal features inherent in a given typeface, photograph, or composition. Then, art and design principles are classified, broken down, experimented with. Students are next encouraged, when confronted with a variety of choices, to make progressively more acute discriminations in judgment, so that those choices become increasingly targeted toward a desired semantic outcome. Finally, students are saturated with explicit and implicit cultural linkages, so that (to give one example) a Swiss grid is associated with modernism while the violation of the grid is treated as a postmodern strategy. Students are made conscious of those kinds of deeply historical and cultural connotations only through lessons where the connections are pointed out, consciously experimented with, and eventually assimilated into the unconscious working memory of the graduate designer.

Every lesson that can be made explicit through words can be translated into code for a machine. Unlike Milton Glaser, meta-design “Milton” may not have an aesthetic soul, but it will not, in the end, need one for the work (of a certain kind[i]) it is asked to do: with the design director’s oversight, it simply needs to be suggestive and, like in horseshoes, come close; the design director can tweak the composition into a ringer. Over time, by observing and remembering those adjustments, the meta-design program continues to learn, developing a more uncanny ability to arrive at the ringer the first time.

But wait—isn’t all of this terribly elitist? If a design program “memorizes” all the best design compositional practices of a certain strata of privileged design professionals, where is there room for “the street”?

As I imagine it, the basic meta-design program would come equipped with a “best practices” library of core concepts—gestalt principles, grid systems, traditional symmetrical book typography, and so on. Overlaid on that core could be extensions that would represent stylistic off-shoots, culturally linked sub-styles, that would allow a diversity of outputs. A second source of diversity would be the trainability of the program. It must be possible with meta-design to have machine learning, so that it is trained over time by being shown new images. A category—“tattoo art” for example—would permit thousands of images of tattoos to be entered and the program could begin to discover the formal compositional relationships that they share.[j] This process would be also guided by the design director who would assist the program, through labeling and pairing strategies, in categorizing and classifying the input. We already see such procedures in dictation programs and in early image-production AI programs.

Missing Link 2: Whence, Creativity?

All of this sounds as if it takes the one thing that is by far most satisfying about being a graphic designer and gives that job to a machine. Where is the creativity, the discovery, the surprise of working intuitively?

First of all, let’s recognize that there are different kinds of creativity. The creativity that is responsible for inventing a humorous concept for an ad campaign is the kind of creativity that is most rare and the hardest to achieve with meta-design. Maybe no program will ever be able to concoct a series of fifteen-second spots like Dos Equis’s “Most Interesting Man in the World.” But would it be able to compose a successful ad combining the photograph, the typography, the logos and the product shots? Why not? That’s an assertion of a different kind of creativity—something that relies somewhat less on cultural understanding, double entendres, the mysterious core of metaphor and humor, and more on the classification schemes of formal structures.

For this kind of compositional creativity—a deeply aesthetic, almost haptic feel for what is right in terms of pacing, emphasis, contrast and balance—meta-design, ironically, would seem ideally suited. When we use optical margin alignment in setting text, we don’t cede authority to InDesign, we allow InDesign to suggest something that it “has been told” will look better to our eyes.[k] This is actually a nascent example of meta-design. We, as design directors, allow the optical margin alignment at a certain value, we review it, make tweaks, and never feel our compositional creativity is being impinged. Our human aesthetics are predicted by the program on the basis of our expertise, our input into the program.

In meta-design compositions, we will maintain final design control. We will be the ones setting the values, determining the stylistic universe of the iterations, and ultimately revising until we are satisfied. Meanwhile, our meta-Milton will be learning what we changed, each revision being recorded and which, in time, will provide it with the information it needs to guess closer to the mark.

In another important way, our own human creative capacity will be increased. Currently, if we have a complex or lengthy project, we don’t have the time to do a great number of conceptual studies, and we are pressed to move fairly quickly toward a solution. We use our intuition and experience (expertise) to do this, but usually this means doing things that are not a huge departure from things we have done before. We stay fairly close to the tried-and-true, modifying a few peripheral details. But, using stage 1.X, we will be able to instruct Milton to give us, for example, three International Style looks, three Traditional Symmetrical looks, and three Random Choice looks. In looking at the variations, we might be expected to get ideas that we would not be able to invent with the time pressures we are working under.

This leads us to the related issue of...

Missing Link 3: Convergence to Mundanity

Since the program always stays within its parameters, so, over time, as the “taste parameters” become more clearly articulated via feedback to the program, fewer digressions can be expected. But it is precisely those digressions, and even mistakes, that often lead us to unexpected new insights and novel directions. Especially with the most competent meta-design, in which the bot has unerringly mastered your look, how can anything be truly new?

It’s certainly the case that any program that can only observe, imitate, and repeat must always enter a death spiral of stasis. But in a smart meta-design program that allows fresh—even random—input, this cycle can be broken. Exposing the program to new compositional stimuli that is foreign and hitherto not available to the feedback loop, injects new strains of possibility. Even if the program is essentially in an imitating mode, it can be shaken out of it by crossing two disjunctive compositional strategies (for instance what would combining symmetry with a grid produce?). Three techniques can be employed to avoid the death spiral: first, expose the program to a variety of novel solutions within a given category; second, expose the program to new categories of composition (i.e. an outside category); third, ask for occasional random settings which then suggest compositions that could not have been foreseen and which, after revision, can be reinserted either within a given category or as a new category.

Indeed, the more interesting question here may be what keeps us human designers from converging to the safe and redundant composition now, in the absence of meta-design? Only our fatigue at seeing the utterly familiar? We purposefully seek out new stimuli by going to galleries, looking at award-winning work, and traveling. Even so, we are often in an imitative stance as is shown by design’s own “me too” movement—I speak of the “me toos” that relay redundant and plagiarized visual tropes all over the Behanced and Pinterested internet to many of the redundant results found within every Google Image Search. And so, if the convergence to the utterly familiar is so seemingly deeply ingrained in all of us, it must be actively thwarted in meta-design by creating a negative parameter in the programming of the meta-design algorithm; it must be proactively avoided.

Missing Link 4: Employment Ethics

It has been estimated that at the height of the pre-digital age there were over 4,000 typesetting shops in North America.[8] With the developments in 1984-5 of the 300 dpi Apple Laserwriter, the 1200 dpi Linotronic, and the incorporation the PostScript Linotype type library in new Macs, tens of thousands of typesetters lost their jobs. The broad adoption of page layout programs such as Aldus PageMaker and Quark XPress meant the designer could handle the entire composition of text as well as the positioning of all graphic elements without leaving the office—or paying for galley repros. A 400-hundred-year-old profession disappeared in about eighteen months.

Meta-design will, for the first time, automate and personalize not just the typesetting, but the entire compositional design of screens, books, magazines, and catalogs. Won’t it be expected to cost a lot of jobs? If so, is it ethical to develop a technology that threatens so many livelihoods?[l]

The answer may depend somewhat on what activity defines the real essence of the field of graphic design. If I made my living laying out catalog issues month after month—not as the art director but as a production artist—I would certainly be concerned. But if I were the art director for that catalog company I might welcome the new world we’re moving into. Who is the graphic designer in that case: production artist, art director, both?

It is the repetitive and routinized operations that will be replaced. The design director will still be tasked with overseeing the aesthetic and qualitative feel of the project. The AI support will remain just that—a supporting technique. It will be a trainable technique, but it is a long way from being a competing freelancing entity.

The key to what will make the meta-design profession vital will be the cultural awareness, the ability to think in metaphors, symbols, to understand the symbolic message within the compositional choices. This is a level of expertise that will remain in constant flux and therefore a set of perceptions that cannot—with the required acuity—be built into the Expertise module.

Indeed, designers (as opposed to production assistants) will be in more demand than ever, because one of our tasks will be the training of Expertise modules within a meta-design system. No one will be better positioned to make the judgments necessary to input for, edit, and select from the vast number of candidate compositions. Design is continuing to move further from its legacy as an intermediary step in the technological process of printing. Increasingly, we are the mediators of messages, semiotic professionals, links not between one technology and another, but between the form, semantics and function of the communication. If that is our role, meta-design will only facilitate the process of making the graphic designer indispensable.

All of this alleviating of fears, of making sure the missing links are tended to, however, is dependent upon one very important ingredient: the general culture has to learn that graphic design is not about merely making stuff with type and images. Design is not making, but moving. We are not makers, we are movers. Graphic design is about causing people to be moved: moving them in their heads, in their hearts, in their actions and beliefs, and ultimately in their knowledge of the world. That is the core expertise that we alone have as designers. It’s not completely transferable to a bot because that which moves a particular person or group at any time is fluid, semiotically inflected, constantly in flux. The design director imparts into the Expertise Module, and the design director edits, revises, extends. But this position of the designer, as someone who understands the semiotics of visual communication—and who stands in some awe of it—must be broadcast to the society at large. Otherwise, “economics alone will play the crucial role,” and Milton will work directly under the CEO. That would not be a utopian visual world to live in, nor a utopian world for a creative person to work in.