| |
|
The Age of Intelligent Machines, Chapter Nine: The Science of Art
From Ray Kurzweil's revolutionary book The Age of Intelligent Machines, published in 1990.
The great discovery of the twentieth century in art and physics alike, is a recoil from and transformation of the impersonal assembly-line of nineteenth century art and science.
Marshall McLuhan, The Gutenberg Galaxy
The Musical ArtsAt a time like ours, in which mechanical skill has attained unsuspected perfection, the most famous works may be heard as easily as one may drink a glass of beer, and it only costs ten centimes, like the automatic weighing machines. Should we not fear this domestication of sound, this magic that anyone can bring from a disk at his will? Will it not bring to waste the mysterious force of an art which one might have thought indestructible?
Claude Debussy, La Revue S.I.M. (1913)
Collaboration with machines! What is the difference between manipulation of the machine and collaboration with it? I have sometimes experienced a state of dynamic tension rising in me out of what would seem to be a state of mutual responsiveness between the machine and myself. Such a state could require hours of concentrated preparatory exploration, coaxing of machines, connecting, so to say, one's own sensibilities, one's own nerve endings to the totality of the tuned-up controls. And, suddenly, a window would open into a vast field of possibilities; the time limits would vanish, and the machines would seem to become humanized components of the interactive network now consisting of oneself and the machine, still obedient but full of suggestions to the master controls of the imagination. Everything seemed possible: one leaned on the horizon and pushed it away and forward until utter exhaustion would set in and, one by one, the nerve endings ceased to connect, the possibilities contracted, and an automatic reversal to routine solutions was a sure danger signal to quit. An affectionate pat on a control here and there was not to be resisted. Switches and lights off! If there is an unfinished bit of conversation between you and the machines, either take note of all the controls or leave them alone until tomorrow. Recapturing the exact circumstances of such periods as just described is not easy. Tomorrow it may seem all cold steel, copper and colored plastic. The coaxing may have to start all over again.
Vladimir Ussachevsky (composer and early pioneer of electronic music), Electronic Tape Music
Computer technology is now having a major impact on all fields, including the creative arts. While the visual arts are just beginning to feel the impact of advances in computer graphics and imaging technologies, the computer revolution in music is already well under way.1 One reason that the transformation of music is further along has to do with the substantially greater "bandwidth" (communication and memory capacity) required for images as compared to sounds. With digital technology we can already store, analyze, modify, and recreate sounds with an accuracy equal to that of the human auditory system.2 To do the same with visual images requires an amount of memory and processing power that is still quite challenging for today's computers. Another reason has to do with the fact that music theory is more highly developed and quantitative than theory in the visual arts.
As discussed earlier, a historic transformation is taking place in the musical instrument industry away from the acoustic technology of the piano and violin and toward the computer-based technology of the synthesizer.3 The advent and now enthusiastic acceptance of digital instruments follows a long tradition. Music has always used the most advanced technologies available: the cabinet-making crafts of the eighteenth century, the metalworking industries of the nineteenth century, and the analog electronics of the 1960s. This latest wave-the digital electronics and artificial intelligence of the 1980s-is again making historic changes in the way music is created.
Up until recently, instrument-playing technique was inextricably linked to the sounds created. If you wanted flute sounds, you had to use flute-playing technique. The playing techniques derived from the physical requirements of creating the sounds. Now that link has been broken. If you like flute-playing technique (or just happened to have learned it), you can now use an electronic wind controller that plays very much like an acoustic flute yet creates the sounds not only of the flute but also of virtually any other instrument, acoustic or electronic. In fact, there are now controllers that emulate the playing technique of most popular acoustic instruments, including piano, violin, guitar, drums, and a variety of wind instruments. Since we are no longer linked to the physics of creating sounds acoustically, a new generation of controllers is emerging that bears no resemblance to any conventional acoustic instrument but instead attempts to optimize the human factors of creating music with our fingers, arms, feet, mouth, and head.
Music controllers and sound generators (or synthesizers) can be linked together by an industry-standard electronic interface called MIDI (Musical Instrument Digital Interface).4 MIDI, which employs an inexpensive communication link, has allowed independently developed synthesizers, controllers, sequencers (computerbased devices that can remember and replay sequences of notes), and other sound modification devices to communicate and interact with each other. MIDI has even been used to control stage lighting and other visual effects. Another industry protocol called SMPTE Time Code (Society of Motion Picture and Television Engineers) allows interfacing sound-manipulating devices to video. MIDI and SMPTE Time Code together are facilitating the development of computer-based systems that are revolutionizing video and audio production in the same way that electronic publishing software, personal computers, and laser printers have transformed the way printed documents are created.
In addition to new music controllers, new sounds to be controlled are being created at an accelerating pace. While the sounds of the piano and other acoustic instruments continue to be sounds of choice, they are routinely mixed with new sounds that approach the richness and complexity of acoustic sounds but have never been heard before.5 The creation of a musical work such as a pop song used to
A home music studio that uses Musical Instrument Digital Interfaces (MIDIs).
involve the creation of a few fixed elements: melody, rhythm, harmony, lyrics, and orchestration, say. Now an entirely new element, the invention of new timbres, has been added. This has led to the appearance of a new type of musician, one who specializes in sound design. Rare these days is the successful pop song that does not introduce some new timbre to the music world's palette of sounds.
Sound modification techniques are being developed as rapidly as new sounds and control methods. It is now possible with personal computers and appropriate software to take a musical note, break it into all of its frequency components, reshape the amplitude (or loudness) envelope of each such component, and then reassemble the sound.6 Like most experiments, many such attempts fail to produce anything worthwhile, but the experiments that do succeed are creating an ever expanding repertoire of new, musically relevant timbres. A wide variety of sound modification techniques can be used off-line when editing a multi-instrumental composition and controlled in real-time during performance.
Digital technology has overcome many of the conventional limitations of acoustic instruments. All sounds can now be played polyphonically and be layered (i.e., played simultaneously) or sequenced with one another.7 Also, it is no longer necessary to play music in real time.8 Traditionally, music performance has often depended on nearly superhuman feats of finger acrobatics. While technical playing skills are still being used to good effect by the virtuosos, absence of such skills no longer represents a barrier to the creation of music. Music can be performed at one speed and played back at another, without changing the pitch or other characteristics of the notes. Further, it is possible to edit a recorded sequence by inserting, changing, and deleting notes in much the same way that one edits a text document with a word processor. In this way, music can be created that would be impossible to perform in real-time.9
Expert systems and other AI techniques are also providing new methods for composing music. A musician can provide his own original ideas for a composition and allow the computer to do the rest of the work using systems that are programmed with extensive knowledge about the music composition process.10 One such system that can compose music automatically, the Cybernetic Composer, was developed by Charles Ames and Michael Domino. The Cybernetic Composer composes entire pieces automatically, while similar systems allow the musician to contribute his own musical ideas. For example, one can write the melody and rhythm and allow the expert-system software to generate the harmonic progression, the walking bass line, the drum accompaniment, or any combination of these parts. Such systems will be of growing value in teaching music composition and theory, as well as in allowing composers to concentrate on those aspects of the composition process where they can add the most creativity.11 Also increasing in popularity are systems that allow the musician to interactively change parameters of a piece in real time essentially, computer-assisted improvisation. For example, for the Macintosh there is Jam Factory, M, Music Mouse, and Ovaltine.12 Other computerized aids for composers include systems that can automatically generate high-quality music notation. Examples include Finale, Professional Composer, and an advanced system now under development by Don Byrd.13
AI systems are likely to have a major impact on music education. In place of the tedious and simpleminded auto-play type of features found on home organs, future musical instruments for the home will contain intelligent music accompanists that can help teach music and provide both children and adults with a richer musical experience at early stages of keyboard skill development.14
Mirroring the broadening choices in sounds and musical control is a broadening appreciation in popular music for diverse musical traditions. Initiated in part by the Beatles almost two decades ago and fueled by many other musical innovators, popular music has expanded from its Afro-American roots to include today the rhythms, styles, melodies, and harmonies of folk and classical traditions from around the world.
Thus today's musicians are confronted by a staggering array of choices: an ever expanding set of sounds, music controllers, sound-modification techniques, sequencing methods, composition tools, and even music traditions. All of this does not necessarily make the musician's job easier: it is generally not desirable to use a plethora of sounds and processing techniques in a single work. Indeed, much of popular music has been criticized for the overuse of electronic techniques.
The goal of music, however, remains the same: the communication of emotions and ideas from musician to listener through sound, using the elements of melody, rhythm, harmony, timbre, and expression. The challenge for the musical artist is the same as for all artists: to make choices, to select the right timbres and melodies to express their musical ideas. The Visual ArtsAn image can be placed somewhere between these two antipodes:
Realism = Abstraction
Abstraction = Realism
Wassily Kandinski, The Problem of Form (1912)
As I mentioned above, the computer revolution in the visual arts lags well behind the major transformation already taking place in the musical arts. In music, computers are no longer a novelty but are now the strongest driving force in the rapidly changing face of music creation. In the visual arts there is still a widespread tendency to view the computer as an interloper, as something alien to the creative process.15 This inclination is reinforced by the fact that computer art has until recently been constrained by the limited resolution of most computer display screens and hard-copy output devices.16 For several years now computer-based devices have been able to create, modify, and manage sounds without compromising the quality and accuracy of the result. The comparable situation in the visual arts is only now becoming possible. The result is that computer art, while a creative and rapidly changing field, is not yet in the mainstream. Recognition that the computer is a viable and powerful tool for artistic expression with unique capabilities should become more widespread in the early 1990s now that the latest generation of personal computers is providing an ability to create graphics comparable to other artistic media.17
Some of the advantages of computer-based art techniques over manual methods apply to any domain--e.g., ease of revising, of making backup and archival copies, of making many types of global transformations (in music, transposing; in art, changing size or color). The computer also permits the use of techniques that would otherwise be impossible to realize.
Methods for using the computer to create visual art vary. The most straightforward approach is to simply use the display screen as a window onto a simulated canvas. The Macintosh MacPaint program and many more-recent applications popularized the computer as a sketchpad but also made obvious some of its limitations, including an inability to represent fine detail, the jagged appearance of curved lines, and for many of these systems, the lack of color. More recent high-resolution color graphics displays are now beginning to provide image quality comparable to that of good 35-mm slides.18 Foremost among the advantages of using a computer screen as a canvas is the availability of image-processing techniques that would be impossible using ordinary paints and pencils. Among the dozens of techniques available, users can alter shadings, reconstruct surfaces, synthesize natural and artificial backgrounds, create reflections, and distort shapes.19
One simple technique is the automatic repetition and rotation of images. If the original image is complex, this would require enormous skill, not to mention time, to perform manually. Using a computer, artists can easily view any object from any orientation, complete with accurate perspectives if so desired.20 But few artists would be happy with these exotic techniques if the more ordinary effects they have always used were not available. Just as each musical instrument provides its own unique set of methods for sound modification (vibrato, pizzicato, and others on a violin, for example), each method of visual art creation provides techniques for creating and modifying color and shape. Oil paints provide an ability to blend and fuse colors, pastels provide certain types of shadings, and so on. With improvements in display technology and image processing techniques, artists at a computer canvas will soon be able to simulate all of these techniques in a single medium, while taking advantage of methods that would be impossible with real paints and solvents.21
Photo by Lou Jones www.fotojones.com
Computer graphics from Tokyo.
Perhaps the greatest advantage the computer gives to the artist is the ability to experiment and change one's mind. Doing this is quite difficult with real paints, but it is one of the inherent benefits of the computer canvas. An unusual way of applying computers to visual art is called artificial life.22 This technique uses the computer to create a simulated environment with simulated "organisms" controlled by a "genetic" code.23 The artist provides the genetic code and the rules for procreation and survival. The computer then simulates dozens or even thousands of generations of simulated evolution. The resulting "creatures" and environment can be quite beautiful. The artist is thus like a deist god who creates a starting point and then unleashes a recursive process of repetitive re-creation.24 The artistic value of this technique should not be surprising. After all, real evolution has certainly created a myriad of beautiful forms. In fact, aesthetics itself is often considered to be rooted in the beauty of the natural creation. The artificial-life approach to creating art is to imitate this ultimate design force. Fractals
Artificial life is considered a recursive technique because each new generation of artificial life begets the next.25 The ultimate application of recursion to design involves the mathematics of fractals. Fractal images are derived from a branch of mathematics called fractal geometry, devised in the 1970s by IBM Research Fellow Benoit Mandelbrot.26 One method for generating a fractal image is through a recursive procedure in which specific parts of a picture (e.g., every straight line) are repeatedly replaced with more complex parts. By using various starting figures and transformations, a great variety of patterns can be generated.
There are actually several types of fractals. I have been discussing geometric fractals. Instead of using exactly the same transformation at each stage, we can allow a random process to control the transformations (randomly skipping some of the steps, for example). In these chaotic fractals, the patterns are less regular and, oddly enough, more natural.27 This observation led Mandelbrot to realize the potential application of fractals to describing a wide variety of natural phenomena, including economic trends, the progression of epidemics, the organization of proteins, and the structure of music.28
Fractals have a number of interesting properties. Unlike most mathematically generated shapes, the nature of a fractal is the same at all scales of magnification. Magnify a fractal a thousand times, and it still looks the same. Magnify it another thousand times, and again, no essential change. This paradoxical quality describes a surprisingly large number of natural phenomena quite well. The most famous example is that of coastlines. Maps of coastlines have the same jagged appearance regardless of their scale.29 Another important example is cloud formation. Pictures of clouds look the same whether one looks at a cloud of 10 feet or a cloud of 100 miles.30
Indeed, perfect fractals keep the same appearance at all scales. But fractal like shapes in the real world have this property, called self-similarity, only to a point.31 In a graphic display the limit is the granularity of the display, which is known as the pixel size.32 In the universe as a whole, which turns out to be a spongelike fractal, the pixel corresponds to the galaxy. Within galaxies, a different type of fractal known as a fractal whorl governs the organization of stars.33
The word "fractal" derives from the phrase "fractional dimension."34 Because of its infinitely fine detail, a perfect fractal shape actually has infinite length even though it resides on a finite two-dimensional surface. According to fractal geometry, it therefore has a dimensionality somewhere in between that of an ordinary line segment, which has a dimensionality of 1, and a surface, which has a dimensionality of 2.35 For example, the Koch snowflake has a dimensionality of 1.26, just slightly greater than the coastline of Great Britain, which is considered to have 1.25 dimensions. The galactic sponge has been estimated to have about 2.2 dimensions.36
For the sciences and engineering disciplines, fractal geometry provides a powerful tool for understanding the dynamics of turbulence, chaos, and other unpredictable patterns.37 For the visual arts, fractal geometry provides an equally powerful methodology for creating images that can be natural, spectacular, or both. Fractals are capable of generating realistic clouds, trees, bushes, lakes, mountains, shorelines, landscapes, and other similar images.38 For example, George Lucas used the technique to create the moons of Endor in his film The Return of the Jedi. It can create environments that, while very alien-looking, appear to be the result of natural processes.39
Another use of the computer to create artistic images is to provide a computer with a set of rules and other knowledge structures that explain the processes of drawing, composition, and layout and to let it be the artist. Probably the leading exponent of this approach is Harold Cohen. Cohen has programmed over 1,000 rules for the creation of complex drawings; his computer uses these to direct a robotic drawing machine to create the drawings.40 These can be of people, plants, or abstract objects. Cohen clearly intends the pictures as art, and indeed, his work has been highly acclaimed. Cohen, who is considered a master colorist, often colors in his robot's drawings manually. His latest ambition is to build a robot colorist programmed with his own theory of coloring.
The computer as artist raises a number of interesting questions. First, who is the artist? Cohen claims that he is, and his computer has not been programmed to complain. Second, why bother? Why not simply make the drawings directly? Both of these questions also pertain to the computer as composer.
These are important questions, and there are, in my view, several answers. First, computer art is a perfectly legitimate means of creating art. The computer is simply a tool, a medium through which the artist (or musician) is expressing himself.
Second, the work of art is, in a sense, more interesting. We can regard the program itself as the work of art that manifests itself differently every time we choose to look at it (or listen to it). It is a work of art that can change its appearance as quickly as the program can generate a new drawing. Since the style is unmistakably similar in each such manifestation, all manifestations can reasonably be regarded as a single work. It can be pointed out, however, that it is not unusual for multiple works of one artist to exhibit such a degree of similarity. We can, therefore, consider a program such as Cohen's to be either an artist itself, or a work of art, or both.
Third, the rules themselves give us an objective and rigorous statement of the theory of the artwork and thus an unprecedented degree of insight into its structure. Understanding the boundary between the rules themselves and the expressive aspects of art that we might feel cannot (yet) be expressed with such precision also provides a valuable perspective.
The rules also have educational value. In art, a great deal is known about perspective, composition, the drawing of different types of shapes, and other facets of visual art. These facets can be expressed in precise terms. Similarly, a great deal can be said with mathematical precision regarding the nature of rhythm, harmonic progressions, the structures of different genres of melodies, and the other elements of music. Creating rules that can in turn create entire satisfactory works of art (or music) helps us to understand, and therefore to teach, the objective aspects of artistic creation.
Finally, such systems can have practical value. In music they can perform compositional chores by automatically creating walking bass lines, rhythmic accompaniments, and so on. In the visual arts, after the artist indicated his intentions, such cybernetic assistants could finish a drawing, performing chores of shading, coloring, creating perspective, balancing the sizes of objects for proper compositional balance, and other tasks. In this mode of operation, such computer-based assistants would always be working under the careful eye (or ear) of the artist.
Such automatic assistants have already been actively harnessed in the creation of animation, where the chores of creating hundreds of thousands of images (over a thousand per minute of film) is formidable. Computer-based animation systems have substantially boosted the productivity of human animators.41 Another area of commercial art where computer design stations are already of substantial practical value is in fashion design and layout.
As in the world of music, the role of the computer is not to displace human creativity but rather to amplify it. It is a tool, like a paintbrush, but one of unique and virtually unlimited potential. Clearly, the great artists of old must have had many ideas beyond the ones they had the time to actually express. By reducing the many chores involved, computers can give artists the opportunity to realize more of their artistic visions.
Also, many computer techniques allow the expression of forms that are simply impossible to realize in any other way. Because the results of some of these methods, such as recursive fractal generation, are unpredictable until they are tried, the computer can be a partner with the artist in exploring the artist's imagination.
Finally, the computer can open the world of artistic expression to more participants. Many of us have rich visual imaginations and a good sense of aesthetic judgement but are lacking in the technical skills to express our imaginations using conventional artistic materials and techniques. The inability to draw a straight line (as the ads used to say) may no longer be a barrier to becoming an effective and successful artist.42
Photo by Lou Jones www.fotojones.com
Automated animation in Kawasaki.
The Literary ArtsAt all events my own essays and dissertations about love and its endless pain and perpetual pleasure will be known and understood by all of you who read this and talk or sing or chant about it to your worried friends or nervous enemies. Love is the question and the subject of this essay. We will commence with a question: does steak love lettuce? This question is implacably hard and inevitably difficult to answer. Here is a question: does an electron love a proton, or does it love a neutron? Here is a question: does a man love a woman or, to be specific and to be precise, does Bill love Diane? The interesting and critical response to this question is: no! He is obsessed and infatuated with her. He is loony and crazy about her. That is not the love of steak and lettuce, of electron and proton and neutron. This dissertation will show that the love of a man and a woman is not the love of steak and lettuce. Love is interesting to me and fascinating to you but it is painful to Bill and Diane. That is love!
A poem from The Policeman's Beard Is Half Constructed (the first book ever written entirely by a computer) by RACTER (a program by William Chamberlain and Thomas Etter)
Bill sings to Sarah. Sarah sings to Bill. Perhaps they will do other dangerous things together. They may eat lamb or stroke each other. They may chant of their difficulties and their happiness. They have love but they also have typewriters. That is interesting.
RACTER, The Policeman's Beard Is Half Constructed
eons deep in the ice
I paint all time in a whorl
bang the sludge has cracked
eons deep in the ice
I see gelled time in a whorl
pffftt the sludge has cracked
all green in the leaves
I smell dark pools in the trees
crash the moon has fled
all white in the buds
I flash snow peaks in the spring
bang the sun has fogged
Haiku poems written by a program by Margaret Masterman and Robin McKinnon Wood
In the literary arts, computers are already of substantial practical benefit. Of greatest impact is the simple word processor. Not an AI technology per se, word processing was derived from the text editors developed during the 1960s at AI labs at MIT and elsewhere.43 Also assisting the craft of writing text are increasingly sophisticated
Photo by Lou Jones www.fotojones.com
David Boucher, Harry George, and the Interleaf Desktop Publishing System.
Photo by Lou Jones www.fotojones.com
spelling checkers, intelligent syntax checkers, style checkers, on-line dictionaries, thesauri, and other linguistic data bases.44 Of practical value to poets is the on-line rhyming dictionary that provides rhymes and half-rhymes with particular rhythmic patterns.
Writers of nonfiction are beginning to utilize the rapidly growing array of data bases available on almost every subject. As the world's knowledge bases gradually shift from paper to on-line information utilities that can be intelligently accessed through telecommunications, the productivity of research will be greatly improved. Ultimately, computers will be able to conduct entire research projects on verbal requests from their human bosses.45
Magazine and newspaper writing has benefited for the past decade from on-line systems, such as those provided by Atex and other companies, that facilitate all aspects of material creation, including text creation and editing, graphics preparation, and layout. The new industry of desktop publishing, pioneered by Xerox,
Photo by Lou Jones www.fotojones.com
Mitch Kapor, founder of Lotus Development Corp. and developer of Lotus 1-2-3 and Agenda.
Interleaf, Apple, and others, provides writers with their own personal publishing capability through the use of an ordinary personal computer and a laser printer.46
A new genre of expert systems for personal computers are now appearing that provide assistance in the organization of ideas. These systems allow users to explore a set of thoughts, notice patterns and similarities, and develop outlines that can later be fleshed out into text. Some of these systems are intended to be used in real time as thoughts develop and include abilities to organize a variety of activities. A prime example of this type of software is Lotus's Agenda, developed by Mitch Kapor.47
Expert systems can also be applied to tracking the complex histories, characterizations, and interactions of characters in such extended works of fiction as long novels, series of novels, and television drama series. (A system that can assist with the development of characters and plots for soap operas is described in Michael Lebowitz, "All Work and No Play Makes HAL a Dull Program" in this chapter.)
Most challenging of all would be a system that could actually write or substantively assist with the writing of prose or verse in the same way that expert systems are now beginning to assist the composer and the graphic artist. Efforts have been made to accomplish this, and some are represented in the epigraphs at the beginning of this section and in the article "A (Kind of) Turing Test" in this chapter. The computer undoubtedly has a role to play in the generation of text, and computer generated prose and verse is a small but flourishing field. The results, however, are significantly less successful to date than in the musical and visual arts. RACTER's prose has its charm, but is somewhat demented-sounding, due primarily to its rather limited understanding of what it is talking about.48 Deeply imbedded in written language is knowledge, extensive, diverse, and intricately organized knowledge, the vast bulk of which has yet to be captured. With language translation systems currently providing only 80 to 90 percent accuracy on technical texts, it is not yet realistic (until well into the first half of the next century) to expect computers to have sufficient facility with language to generate literary quality text without assistance.49 The objection might be raised that there is a comparable depth of knowledge in works of music and of visual arts. This is certainly true. The reason why computers have had greater success in assisting with the creative process in these areas is that we have a better understanding of the surface structures of music and visual art (particularly the former) than we do for literature. But even in these fields, the ultimate responsibility for making an artistic statement is still the province of the human artist.
The role of the computer is the same for all of the arts. First is to provide an effective tool for improving the productivity of the artist, to enable him to realize more of his imagination. Second, the computer can provide forms of expression that were previously unavailable. Finally, to whatever extent we are able to capture even a portion of the creative process in a program, we learn something about the art form as well as the process of creating it. The computer can be a powerful partner in exploring our thoughts and emotions and finding new ways of expressing them.
| | Join the discussion about this article on Mind·X! | |
