October 27, 1997. Informative review of Rhapsody in c't magazine.
August 22, 1997. Apple published a Rhapsody FAQ (link broken 3/98) that supplied many reasuring details. Same for August 29 interview (link broken 5/00) with Apple Game Sprockets technical lead, Chris De Salvo.
[Sent to Apple on] August 14, 1997
Re: Yellow Box API
To: Dr. Avie Tevanian, Sr. VP of Software Engineering
cc: Jon Rubenstein, Sr. VP of Hardware Engineering
cc: James Gable, VP of Platform & Technology Marketing
cc: Nano Urbina
Dear Avie Tevanian
We are vision scientists, and a few hearing scientists, since our needs overlap. (Our 97 signatures appear below.) [+6 added, at end, after letter was sent.] Macs have offered significant advantages for vision research, and are used by more than half of the vision labs we know. Our labs have many Macs and we train many graduate and undergraduate students, who all learn firsthand how handy a machine this is for generating visual stimuli and doing science in general.
We are excited about the unprecendented portability that the Yellow Box API promises: allowing us to run the same source code on Mac OS, Rhapsody, Windows, and NT. This would be a great boon, allowing us to share our software with all our colleagues. However, there are some features that are essential to our work that we'd like to alert you to while you are writing the API. Most of these features are probably already present in your draft documents. The demands of vision experiments are very similar to those of games, which we know Apple wants to encourage.
This isn't a wish list. It's what we really need, based on our experience of writing and sharing vision-research software to run on Macs:
http://vision.nyu.edu/VideoToolbox/
http://psychtoolbox.org/
http://vision.arc.nasa.gov/mathematica/Cinematica/Cinematica.html
http://vision.nyu.edu/VideoToolbox/Faith%27sSoftwareReview.html
We need:
1. WINDOWS. An unlimited number of on- and off-screen windows, on multiple screens, and fast drawing commands, like QuickDraw or PostScript. We need to create a window whose content region fills the WHOLE screen, with no visible controls or menu bar. (Presumably Apple's declared intent of supporting DrawSprocket in Yellow Box implies that we needn't worry on this score.)
2. FAST COPYING. To copy a rect, unchanged, from an offscreen window to an onscreen window at absolutely the highest speed that the machine hardware is capable of. (CopyBits meets this criterion, e.g. 48 MB/s on a PowerMac 7500/100. Its replacement must too.) Copying from onscreen to offscreen needn't be so fast. In Mac OS it was dauntingly difficult to figure out all the little tricks necessary to convince CopyBits to copy with no color translation. We hope that for Yellow Box, Apple will provide complete instructions for how to achieve direct image copying with no color translation.
3. RAW PIXELS. In order to interface to other environments (e.g. MATLAB arrays), fast low-level read and write access to the raw pixels in the window (like GetPixBaseAddress), preferably for all windows, but at least for offscreen windows.
4. CLUT. Fast read-write access to the color lookup table (CLUT) of the graphics display hardware. At present this is provided by the cscSetEntries, cscGetEntries, cscSetGamma, and cscGetGamma graphics driver calls. This mechanism must be fast enough to allow us to load a new CLUT for each frame of the attached monitor. We only read the CLUT occasionally, but the ability to read is essential in confirming that the writes are working correctly. Cards with more-than-8-bit DACs are important to us; their CLUTs need more bits. (In Mac OS this requires making two calls, cscSetGamma and cscSetEntries, each time.)
5. SYNCH to VBL. (VBL stands for Vertical Blanking Level, which is an electronic pulse that's sent to the video monitor at the end of each frame.) In order to show real-time movies (with no dropped frames, unlike QuickTime) we need to synchronize our software to the graphics hardware. We need to know when VBL happens. Mac OS provides two less-than-ideal mechanisms. (A.) VBL interrupts allow us to set up a flag bit that our programs can monitor, but Mac OS interrupt latency is not great, and we often find that achieving real-time performance requires temporarily blocking interrupts, making this mechanism unavailable. (B.) cscSetEntries has the side effect of waiting for VBL. (This was turned off in Mac OS 7.6 for several built-in drivers, but fortunately Apple provided a way to turn it back on: cscSetTimeDelays.) (C.) Although the Mac OS does not provide access to it, we have the impression that every graphics device (built-in and card) has an undocumented VBL bit, somewhere, that is monitored by its graphics driver. Apple's Yellow Box should provide a standard API to access that bit. Even better, would be to allow us, when possible, to read the hardware line counter.
6. HOG THE MACHINE. Supposedly the point of multitasking operating systems is to prevent user programs from hogging the machine. However, scientific experiments and games place a very very high premium on getting the visual stimulus right at all costs. We can't afford to miss frames. Thus we need a way of blocking all but a few essential and brief interruptions for the second or so that it takes to show a visual stimulus to an observer. It's ok for this kind of control to be available only through a driver (still part of the Yellow Box API), just make sure that we can do it. Hogging the machine is needed by the larger community of scientists and engineers who do real-time data acquisition. E.g. for physiology experiments, data acquisition and serial communications need to be fast and precisely timed. In the past, Apple has claimed that the Mac is not a "real time" machine, but we beg to differ, and we hope that future developments will continue to support the large community of users with real time needs.
7. SCREEN RESOLUTION & FRAME RATE. We routinely get and set each screen's pixel depth. Fifty of us wrote to Apple in March 1995 (link below) to express our need for an API to set a graphics device to arbitrary resolution and frame rate (e.g. over 100 Hz). We understand from Apple Engineer Nano Urbina that substantial progress has been made toward this goal. This is still very important to us, and would be an important reason, all by itself, to adopt the Yellow Box API.
<http://vision.nyu.edu/Tips/Tips/cscSetModeTiming.html>
[THANKS: As an interim measure, in October 1995, Nano produced for the vision community a modified version of the 7500/8500 graphics driver supporting a 120 Hz frame rate. (It now needs updating for compatibility with Mac OS 7.6 and 8. Many labs are buying PowerMacs and running 7.5.3 just to use this driver.) This has been a great boon to labs doing stereo or motion or physiology experiments that demand very high frame rates.]
8. ASYNCH SOUND. Continued support for playing sounds asynchronously (i.e. while we're showing movies).
9. GET USER INPUT IMMEDIATELY. Vision experiments, like games, need to get user responses (keyboard and mouse) immediately, without the delays of an event manager.
(10.) WISHES. We said "needs, not wishes", but in the process of collecting our thoughts a few wishes did surface that we thought you'd want to hear. "Genlock for stereo work." "We hope multi-processor support will let us devote individual processors to video, neural spike recognition, eye movement recording, etc." "To collect accurate reaction times, the ability to poll ADB devices and the microphone at millisecond rates would be great."
Where necessary, we are willing to write a Yellow Box driver to access hardware. And we recognize that the lack of standards for graphic devices in general may mean that we'll have to write some machine-dependent code to achieve some of these things. We can live with that, as long as there is always some way of doing it. In those cases, we'd be very grateful for guidance, e.g. technotes with code snippets.
While ours is a user perspective, we suspect that a former OS-kernel-writer like yourself can relate to these needs--certain tasks demand uninterrupted absolute control. The following Apple Engineers know a lot about these issues and could advise you on what's involved in achieving these goals: Nano Urbina (graphics drivers) and Bo3b Johnson (minimizing interrupts, i.e. hogging the machine). Other relevant experts, whom we know less well, include Guillermo Ortiz (graphics), Cary Farrier (DrawSprocket), and Tim Carroll (GameSprocket). We'd be delighted to discuss this with you and other Apple folks.
We're excited about the portable future offered by Yellow Box. We wish Apple every success.
Sincerely yours
Denis Pelli
Psychology Dept. & Center for Neural Science, New York University
Apple Partner
http://vision.nyu.edu/VideoToolbox/
David Brainard
Psychology Dept., UC Santa Barbara
http://psychtoolbox.org/
Hiro Akutsu
Ophthalmology Dept., University of Illinois
David Alais
Department of Psychology, Vanderbilt University
Duane Albrecht
Psychology Dept., University of Texas
Kenneth R. Alexander
Dept. of Ophthalmology & Visual Sciences, University of Illinois at Chicago
Karl Arrington
Visual Research Lab, USAF Armstrong Laboratory
David A. Atchison
School of Optometry, Queensland University of Technology, Australia
Michael Bach
Dept. of Ophthalmology, University of Freiburg, Germany
http://www.ukl.uni-freiburg.de/aug/bach.html
Benjamin T. Backus
School of Optometry, UC Berkeley
Curtis Baker
Dept. of Ophthalmology, McGill University
http://www.psych.mcgill.ca/labs/mvr/Curtis/cbaker_home.html
Timothy Bates
School of Behavioural Sciences, Macquarie University, Sydney, Australia
http://teach.bhs.mq.edu.au/~tbates/
Patrick Bennett
Department of Psychology, University of Toronto, Canada
Peter Bex
Center for Visual Science, University of Rochester
Margaret Bradley
Psychology Dept., University of Florida
Nicola Bruno
Dipartimento di Psicologia, Universita' di Trieste, Italy
http://www.psico.univ.trieste.it/users/nick/
Axel Buchner
Psychologie, Universitaet Trier, Germany
Steve Buck
Psychology Dept., University of Washington
Tom Busey
Psychology Dept., Indiana University
Marisa Carrasco
Psychology Dept., New York University
Patrick Cavanagh
Department of Psychology, Harvard University
http://visionlab.harvard.edu
Bob Cent
Regional Primate Research Center, University of Washington
Russell Clarke
rwc@kagi.com
Max Coltheart
Department of Psychology, Macquarie University, Sydney NSW, Australia
http://teach.bhs.mq.edu.au/psy/staff/
Lawrence Cormack
Psychology Dept., University of Texas
Frans W. Cornelissen
Laboratory of Experimental Ophthalmology, University of Groningen, The Netherlands
http://cornelis.med.rug.nl/
Jim Crowell
Division of Biology, Caltech
Andrew Derrington
Psychology Dept, University of Nottingham, UK
Richard A. Eagle
Dept. of Experimental Psychology, University of Oxford, UK
Michael Eckert
University of Technology, Sydney, Australia
Raymond van Ee
Vision Science Department, UC Berkeley
Edgar Erdfelder
Psychologisches Institut, Universitaet Bonn, Germany
Rhea Eskew
Dept. of Psychology, Northeastern University
Chris Evans
Department of Psychology, Macquarie University, Sydney NSW, Australia
http://teach.bhs.mq.edu.au/psy/staff/
Bart Farell
Institute for Sensory Research, Syracuse Univiersity
Jocelyn Faubert
Ecole d'optometrie, Universite de Montreal, CANADA
http://mistral.ere.umontreal.ca/~faubert/jocelyn_labo
Ione Fine
Center for Visual Science, University of Rochester
Eric Fredericksen
Cognitive Science Dept., UC Irvine
Patti Fuhr
School of Optometry, University of Alabama at Birmingham
Joachim Funke
Department of Psychology, Heidelberg University, Germany
http://www.psychologie.uni-bonn.de/~funke_j/
Wilson S. Geisler
Psychology Dept., University of Texas
David Gilden
Psychology Dept., University of Texas
Michael E. Goldberg
Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health
Pete Gowdy
Department of Psychology, Harvard University
Kevin Haggerty
Optometry School, Indiana University
Lewis O. Harvey, Jr.
Department of Psychology, University of Colorado
http://psych.colorado.edu/~lharvey/
James M Hillis
School of Optometry, UC Berkeley
Don Hood
Psychology Dept., Columbia University
Hans Irtel
Psychologie, Universitaet Mannheim, Germany
http://www.uni-mannheim.de/fakul/psycho/irtel/
Alun Johns
Psychology Dept., Oxford University, UK
http://users.ox.ac.uk/~xpsy0020/
David Keeble
Naito Group, Information Science Research Laboratory, NTT Basic Research Laboratories, Kanagawa, Japan
Phil Kellman
Psychology Dept., UCLA
Stan Klein
Optometry School, UC Berkeley
Joel Lachter
Psychology Dept., University of Arizona
Dennis Levi
School of Optometry, University of Houston
Mats Lind
Center for Human Computer Studies, Uppsala University, Sweden
Walt Makous
Center for Visual Science, University of Rochester
Will McClintock
Department of Ecology Evolution & Marine Biology, UC Santa Barbara
Pascal Mamassian
Psychology Dept., University of Glasgow, Scotland
Fran Miezin
Dept of Neurology, Washington University Medical School
Ethan D. Montag
Center for Imaging Science, Rochester Institute of Technology
Sacha Nelson
Dept. of Biology, Brandeis University
Koichi Oda
Dept. of Communication, Tokyo Woman's Christian Univ., Japan
http://www.twcu.ac.jp/~k-oda/
Izumi Ohzawa
School of Optometry, UC Berkeley
Apple Associate
http://totoro.berkeley.edu/~izumi/
Elizabeth Olds
Dept. of Psychology, University of Waterloo, Canada
http://www.cgl.uwaterloo.ca/~esolds
[now http://www.wlu.ca/~wwwpsych/eolds/]
Joel Pokorny
Visual Sciences Center, University of Chicago
Bertram O. Ploog
Dept. of PSA, College of Staten Island, CUNY
Zoran Popovic
Dept. of Ophthalmology, Sahlgrenska University Hospital, Goteborg, Sweden
http://www.oftalmologi.gu.se
Dave Rose
Psychology Dept., Cambridge University, UK
Nava Rubin
Center for Neural Science and Psychology, New York University
Evan Relkin
Bioengineering and Neuroscience, Syracuse University
Dan Sanes
Center for Neural Science, New York University
Takao Sato
Dept. Psychology, Univ. of Tokyo, Japan
Jim Scileppi
JGS3d@aol.com
David Schloerb
Research Laboratory of Electronics, MIT
Allison Sekuler
Arbeitsgruppe Hirnforschung, Institut fuer Biophysik und Strahlenbiologie, Universitaet Freiburg, Germany
Department of Psychology, University of Toronto, Canada
Michael N. Shadlen
Dept. of Physiology and Biophysics, University of Washington
Alex Shepherd
Medical Research Council Applied Psychology Unit, Cambridge, England
Steven Shevell
Psychology Dept., University of Chicago
Satoshi Shioiri
Department of Image Science, Chiba University, Japan
Vivianne C. Smith
Visual Sciences Center, University of Chicago
Joshua A. Solomon
Institute of Ophthalmology, London, UK
http://vision.arc.nasa.gov/personnel/jsolomon
Lothar Spillmann
Arbeitsgruppe Hirnforschung, Institut fuer Biophysik und Strahlenbiologie, Universitaet Freiburg, Germany
Larry Stern
Center for Vision and Image Sciences, University of Texas
Jim Stone
Psychology Department, Sheffield University, England
http://www.shef.ac.uk/~pc1jvs/
Hans Strasburger
Inst. f. Medizinische Psychologie der Universitaet Magdeburg, Germany
Inst. f. Medizinische Psychologie der Universitaet Muenchen, Germany
http://WWW.med.uni-muenchen.de/medpsy/staff/hans/homepage.html
Anne Treisman
Department of Psychology, Princeton University
Stefan Treue
Cognitive Neuroscience Laboratory, University of Tuebingen, Germany
John Trueswell
Psychology Dept., University of Pennsylvania
Christopher W. Tyler
Smith-Kettlewell Eye Research Institute
(The Smith-Kettlewell Eye Research Institute is an Apple Developer.)
http://www.ski.org/cwt.html
David Van Valkenburg
Psychology Dept., University of Virginia
Brian A. Wandell
Psychology, Neuroscience, and Electrical Engineering, Stanford University
http://white.stanford.edu/wandell.html
Scott Watamaniuk
Psychology Dept., Wright State University
Peter Wenderoth
Department of Psychology, Macquarie University, Sydney NSW, Australia
http://teach.bhs.mq.edu.au/psy/staff/
Laurie Wilcox
Psychology Dept., York University, Canada
Pepper Williams
Psychology, University of Massacusetts-Boston
Joongnam Yang
Psychology Dept., New York University
Signatures added after letter was sent:
Frans Verstraten
Psychology Dept., Harvard University
Mike Tarr
Department of Cognitive and Linguistic Sciences, Brown University
http://www.cog.brown.edu/tarr.html
Gay Snodgrass
Psychology Department, New York University
Mal Semple
Center for Neural Science, New York University
Donald Gudehus
3D Image Technology
Peter April
Psychology Department, Concordia University, Canada