I joined Apple in the summer of 2013, as part of Graphics, with a mandate of improving game related technologies, and professional production workflows.
ModelIO was started early 2014, and introduced at WWDC 2015. It's purpose was to introduce 3d asset IO, analogously to ImageIO providing import and export of a large variety of 3d file formats. Due to various concerns including ongoing negotiations with Autodesk, FBX was not part of the release.
ModelIO also contained a suite of pipeline related tools, in-memory scene organization tools, geometry transformations, and bakers of various kind such as automated distortion minimizing UV mapping, occlusion baking, and so on.
What ModelIO did not have was collaborative editing, large scale scene management, and notions of asset composition such as template assets, attribute overrides, asset variations, and the like.
Having architected the content creation pipeine at LucasArts for a decade, and from having worked as a developer and architect on ILM's proprietary Zeno application and early versions of ILM's virtual production technologies, I knew where I wanted ModelIO to evolve into, and I also knew that the investment to get over the finish line would be a factor of ten or more over that of the initial ModelIO effort.
I worked up a plan to develop and maintain a proprietary stack. I also knew that once such a stack exists, it takes time for it to mature, so I also worked up a maturation and development plan. Once I looked at the numbers I realized that even with such an investment, we would be forever in a catch up position with more mature pipelines and engine technologies such as Unity and Unreal which were coming to the fore at the time.
My conclusion therefore was that it made strategic and economic sense to partner with somebody.
I reached out to several large studios, of which Dreamworks, Industrial Light + Magic, and Pixar made the second round of discussions. In addition we also considered Unity and Unreal, but those did not have the maturity at scale, nor the mission focus, for the kind of content production we wanted to enable, which was large scale, collaborative, simultaneous, real time, production by large teams into a single scene database.
I also concluded that on-going collaboration made more sense for such technology, in order that the systems be driven by real production needs, not only indirect and often conflicting feedback from developers. Without a production need, a certain amount of interpretation and second guessing would be unavoidable.
Dreamworks Apollo platform (https://3dtotal.com/news/interviews/dreamworks-new-apollo-platform-studio-visit-by-evan-shamoon-interview) exhibited the characteristics we were after, and late in 2014, during intense management turmoil at Dreamworks, it seemed that they were interested in partnership or outright sale of the technology. During negotiations, Dreamworks shut down the Redwood City offices of PDI where the Apollo technology was developed, and amid increasing turmoil, this opportunity slipped away.
Industrial Light + Magic's Zeno, and Virtual Studio architecture, were the model I had in mind when architecting our own systems, so the fit was close. What was less clear to ILM was whether they would have the attention or resources to engage in a collaborative effort, and whether making their toolsets publicly available would affect their technical competitive advantage, and their ability to innovate independently and secretly. Negotiations and discussions continued for some time.
Pixar's Presto technology also shared many of the touchstones exhibited by Apollo, Zeno, and VStudio, but it wasn't clear at first if the technology could be readily extended to encompass the other domains Apple wanted to support, which included live action workflows, episodic development, and games. Apple was keen however to explore a foundational technology relationship with Pixar, as such efforts had been a cornerstone of the relationship between the companies under Steve Job's tenure, and the possibility of rekindling that magic was very attractive.
After exploring and laying out all of the above, and meeting with Apple's C-suite many times to educate them on why legacy formats such as fbx were insufficient to modern needs, the benefits of partnership, and so on, I got the greenlight from Craig Frederighi to begin discussions with Pixar.
Early in 2015, I met with Guido Quaroni who told me that my overture was well-timed. He said there was interest in open sourcing Pixar's core technology base. Open Sourcing was a best case opportunity from my perspective, much more compelling than collaboration on a proprietary stack such as Apollo, Zeno, or VStudio, because the code would be available to the development community for both audit and enhancement.
Audit was crucial, as the Achille's heel of a proprietary stack is that if things don't work as expected, and end user's only recourse is black-box testing to discover the roots of a behavior, or, conversations moderated through forums and what the owner of the stack is willing to disclose. Audit would also provide third party adopters of the technology to give themselves confidence around the provenance and intent of embodied behaviors.
So, I was sitting there with Guido talking all this through, and he mentioned that approval hinged on the effort having a significant industry partner as an anchor, so that it wouldn't just be Pixar supporting everyone. I asked if Apple would be a significant enough partner; Guido picked up his phone, grinned, and put it down, and said "Disney says yes."
Apple formalized its commitment, and the relationships with other candidate partners went in other directions.
Apple immediately hit the practical realities of the effort, it was Linux only, had untenable dependencies such as berkeley db, and file access patterns that were unsuitable to the kind of storage on iPhones and iPads. Early in 2015 therefore, I started working frequently on site with George and Sunya on porting the code to Mac, iOS, and Windows, the core team on architectural work, and the GPU team in a growing effort to fit the technology on Apple's hardware and prepare for a release.
Some missing bits were added to USD in cooperation with Apple, UsdSkel was one collaboration, and was foundational to Apple Animojis. The focus on memmapping that underlies USDC was another development that reflected a need Apple had. USDZ, an archival format was another.
In May 2017 during WWDC keynote, Tim Cook announced the partnership, and Apple's formal adoption and support of USD as the preferred pipeline technology, and thus it became plan of record and Apple has continued to increase its investment since then.
Also, in addition to people mentioned above, crucial key support to the initial effort at Apple: Jacques Gasselin de Richebourg, Travis Brown, Rob Partington, Greg Joswiak. At Pixar: Michael B Johnson ("wave") and Richard Guo.
In University, I was in the last class that had to do a Fortran program on punch cards. I was so mad that I had do it that I took the cards to the roof of a tall building and then fanned them out into the sky. Now I wish I had kept them to wave at whippersnappers like you guys
Now that I'm old and gray I agree with you completely on appreciating history. On littering, you may not have a clear idea of what an active scofflaw I was in those days, and the frequency and severity of my transgressions upon our shared social contract. I recently bought myself the pair of tall lace up Doc Martens that I could not afford in those days to express my attitude, and it definitely feels nice to be back in those shoes, so to speak, minus the attendant law breaking OF COURSE. On recycling, it rains, a lot, in Victoria, and coarsely fibered paper such as a punch card is formed of dissolves into mush and disappears quickly. I was at least conscious of impact. I don't recall the particulars of the program but I'm quite sure it was exceedingly useless, as I really, really, really, wanted to be programming on the Sun machines that were being installed for next years' class - they had color bitplanes, and one could do "graphics" on them.
When we wound LucasArts down, I worked with Daryl Jacobson at ILM to perform the inventory and archiving. He's still at ILM, so he could be a good resource for insight.
I think we did several things right and several things wrong.
The Good
We gathered "build shrines", PCs with all necessary software, tools, code, and assets, that could reproduce a game without being connected to a network. We wrote a "build bible" for each game. The build shrines were stored in a physically secure location. Tapes stored at Iron Mountain
The Not So Good
We didn't anticipate that the shrines wouldn't be touched for close to a decade. Can you still spark up, reliably, a machine that's been off for that long? Do we want to risk power cycling the machines more than a few times? I didn't prepare redundant build shrines, just one per game. I don't know if Daryl implemented redundancy, I certainly didn't think of it at the time. The code side of stuff is on the order of a gigabyte for newer games, 10's of megabytes for older. No problem. The art side is on the order of a terabyte for newer, a gigabyte for older. If you open a box and find a terabyte of art assets, what even are you going to do with that? I lost track of the physical archive. In the Indiana Jones Warehouse, perhaps Daryl is the only one who knew which crates had the shrines? There's only a few LucasArts programmers still in Disney, and not all games are represented any more. So the institutional knowledge on how to deal with those archives is effectively evaporated, after only a decade. Technical design documents and wiki materials were already lost to a large degree at the time we made the shrines. I didn't think to archive Sharepoint, for example, not sure if Daryl did either.
Oops
No online archives, or digital indexing of materials. Tapes stored at Iron Mountain are not 100% retrievable. For example, the tapes for Super Bombad Racing were not retrievable ten years ago, due to being corrupted. We didn't archive version control history, so the only source of truth is the shrines. LucasArts' P4 servers failed frequently. Even if we had stored P4 archives, I'm not confident they'd have survived the decade.
back in my day we had EIGHT BITS and WE LIKED THEM
All 8 of them, there was Least Significant Bit at one end, and Most Significant at the other. We could roll them left and arithmetic shift them right.
Kids these days, with their "pipelined instructions" and "speculative execution". Give me a box with eight toggle switches and I'll show you how programming is really done
I was chatting with George Lucas one day about the history of Avid, and he lamented that the best thing about EditDroid was the custom controller they'd built, with knobs and sliders that fell naturally under the operator's hands. When he sold it to Avid his greatest regret was that they felt that the value was in the editorial software, and they ditched the control. His opinion was that software is ephemeral, but physical interfaces are a more direct link between your mind and the operations you are performing than endless checkboxes and menus
I used to work on classic adventure games. In those days, we had this notion of hot spots, which were areas on the screen that when clicked, were bound to an action. A hot spot might be hidden, for example in the Riddle of Master Lu, we had a diamond-pattern wallpaper, and one of the diamonds was a playing card with a clue written on the obverse. Finding that card was a maddening process of hovering the mouse over every one of hundreds of diamonds until the mouse changed into an action cursor. We called it the pixel hunt, and it was possibly the worst thing about those games. I regret to say that I find that all of the modern interface patterns, Metro, Modern, Fluent, UIKit, are egregious modern examples of the pixel hunt. Just now, I was trying to find the active interface elements in Slack, and had to run my mouse over the screen to find out what was clickable, as if I was stuck in an adventure game hedge row maze from 1995 where all the images are repetitions of the same leafy sprite. We've regressed so far into minimalism that UX is an experience all right, but I hesitate to go so far as to say it provides an effective UI. I can't say the enshrinement of Maya style interfaces, in kits such as ImGui, where we can iterate properties and display them in a panel is much better. The affordances are there, yes, I can tell where I must click, but these provide a programmers-eye view of tasks and operations, the ability to directly reach in with a surgeon's scalpel and forceps to adjust the most minute underpinnings, where the artistic intent is obscured. The artist didn't intend to delicately adjust 152 subtle values, the artist intended to depict a character smiling.
Nick Porcino 2 days ago @iangarrett that "wait to talk" patent is thought provoking, thanks for posting it. The concept is at some level clearly doing a desirable thing; for example blind people might want notifications spoken for them in a non-disruptive manner. The system however, is always listening (at least when there is a notification to be spoken), which rings the surveillance bell. Consider: A closed cybernetic system is a motion sensor that activates a light when movement is detected. Surveillance is when that movement is an input to another system and that system does not reciprocate an input to the first system. If the wait to talk occurs on device, without reporting either the delay or audio to the mothership, it would be a closed cybernetic system rather than surveillance.
At the moment, we are collectively waking up to the mechanisms that run the world, and there's a tendency to lump everything into a counter reaction against the clear dystopia inflicted by these systems we've invented. To get things on a constructive track, that can yield specific actions and policies, I think the discourse on surveillance needs to advance a little, to draw a distinction between cybernetics and surveillance. :heart: 2
New
Heidi Boisvert 3 hours ago @Nick Porcino I appreciate your attempt to distinguish btw a closed cybernetic system & surveillance, however, the intention established during the Macy Conference’s (distinct from British Cyberneticists) focused on “command & control.” Surveillance, I would argue stems from a cybernetic ethic whose legacy informs the design of our current technological systems. Core drivers of American cybernetics were social control, quantification & prediction…
Nick Porcino 1 minute ago It's true, Norbert Weiner started with self-regulating mechanisms in "Control and Communication", and quickly moved on to the "Human Use of Humans". I completely agree with your point that surveillance is exactly related to cybernetics. Perhaps the word cybernetics is too fraught for what I'm trying to express; which is that we need a mental framework and therefore language to better understand the consequences of cybernetics; a way to distinguish between a self-regulating mechanism, such as "wait to talk", from a self-regulating system that incorporates surveillance; the non-reciprocal information transfer, such as "someone was talking, therefore the system waited to talk." Of course that language exists, I just used it. I'd like to see, as a common point of discourse, an explicit check whether a system involves "non-reciprocal surveillant information transfer", or if the system is a "self-contained self-regulating mechanism". Perhaps a badge, ♻️ versus :sleuth_or_spy: ! (edited)
To be honest, he has gone down a bit of a metaphorical black hole with his paragraphs on pointers. (I’m looking at https://pages.gseis.ucla.edu/faculty/agre/ni.html - hopefully not to different than whichever version you are looking at.)
Imagine a computer memory to be nothing more than a huge row of bricks, and from start to end, we number the bricks sequentially from some starting value, say, 1000, to some ending value, say 2000. Further imagine that some number might be written on each brick.
We might decide that we are interested in counting apples and oranges. We then might decide that the number of apples is written upon the brick 1100, and that oranges are recorded upon brick 1899.
We might then declare that “pointer to apple is 1100, and pointer to orange is 1899”.
Now, if we wish to record the number of apples, instead of stating “record the number of apples on 1100", we might instead state “record the number of apples where the pointer to apples tells us.”
So a pointer is a tiny bit of indirection to help us work with large collections of bricks.
In a possibly ironic twist, we will need designate a brick as the brick that records the pointer value. So we might say “I have recorded the pointer to apples on brick 7.” So I would go look at brick 7, discover that 1100 is written upon it, and then go write the number of apples on brick 1100. We could get even more elaborate, and designate a brick to tell us where the brick is that tells us where the apples are.
The Vogue article is well considered. I went into it thinking that despite expectations, photography didn’t kill illustration, motion pictures didn’t kill photography, and digital hallucination won’t replace any of them. I still believe that, but it’s much more nuanced; and she points out “we’ve fought to change the perception that we are just a sample size or a prop for clothes” ~ that struggle will continue, and need to continue through this digital shift. The sample size/clothes prop role seems most likely to fall to digital, whereas I don’t expect the need for real people to diminish due to a rise of Miquella bots, any more than the existence of Magilla Gorilla means there’s no need for real gorillas.
I asked a super bright junior to implement A* one time, and he insisted that he didn’t need to look at textbooks because he was convinced he’d thought of a better algorithm. I didn’t have the time to explore new solutions, but it was a pattern that he automatically dismissed every suggestion as stupid/old. The light kind of went on for me at that moment. I let him work through the problem at his own speed, and took the schedule hit. He independently reinvented A*, because he’s super bright. After it was done, I walked him through actual A*, and his reaction was “goddammit if you’d told me about this before, I would’ve saved weeks,,,,,,, oooooh”. He still kept inventing amazing things, but he started checking with me to make sure I wasn’t holding out on the good stuff.
you know what the monorepo is genius for? Giant distributed systems like Facebook and Google.
I don’t mean that facetiously. The fundamental difference between a product such as a DCC, and a product such as a social networking site, is that a DCC is monolithic. In other words, it is a bundled distribution of a single aggregate artifact.
Facebook on the other hand is a single view on literally thousands of independent programs, which are each updated on an independent cadence.
The paradox here, is that the monorepo cadence is monolithic. It is a massive single distribution of a single artifact, but the product is thousands of independent pieces that do not advance on the same cadence. A software product like a DCC, is the opposite. It is a single artifact that must be distributed, but it is itself composed of independent pieces that do not advance on the same cadence.
In other words, advance an image filter on the Facebook timeline: the entire repo advances monolithically, but the product itself is distributed with atomic changes. Change an image filter in the DCC: the repo advances atomically, but the DCC will itself advance as a monolithic bundle when released to users.
A long time ago I worked with Mark Ferrari on games with a lot of palette effects. Palettes make those kinds of special effects very easy. There's a trend today in sprite tools to work with RGB images and smart remapping, but honestly I think it's not a great direction.
We had neat things like waterfalls, flickering torches, night and day color sets, and of course character costume changes. We organized the palettes into a 16xN RGB image, and the artist could organize it as they like. The first part of the image was the palette as applied to indices, and the second was the remappings.
There was a table that assigned semantics to the image, like this:
0: system
1: UI
2: player1
3: player2
4: fire
5: water
6-15: sprites
16-20: fire
20-29: water
30-39: sprites
40-49: sprites
The game programmer was responsible for doing things like cycle 20-29 onto slot 4. The table was for human reference, rather than in-game usage. The API was just
palette_set_row(int target, int source)
game recording/playback
I always built that into my engines at LucasArts. you can have the game play itself on another monitor while you ponder behavior or code. you can build QA automation around it, like “go to second platform, verify coin obtained” you can build ML training around it. Super Bombad Racing relied on recordings of testers to generate the training set for the racers you can make a fake second player for testing multiplayer by your self
sydneyskybetter 4:02 PM PDF Kleist-On the Marionette Theater.pdf 132 kB PDF132 kB — Click to view
4:02 As ever, because puppets.
Nick Porcino 4:25 PM That is an astonishing journey from the grace of a puppet to fencing with bears. I must correct the mathematic, that described as elliptical being in fact parabolic. The notion that a dancer could be replaced prosthetically piece by piece from an artificial limb, begging the role and location of heart in the performance; how ever did you find this piece?
sydneyskybetter 4:25 PM lol I very much love @Nick Porcino you correcting the math on this one. 4:26 I was consulting on a new course here at brown about the history of AI 4:26 a historian colleague suggested I check out this piece as a way of introducing a few themes of embodiment, performance and animus 4:27 It's thanks to Holly Case! (Who i'm hoping to drag on to the slack eventually)
Nick Porcino 4:28 PM "embodiment, performance, and animus" Three great tastes that go great together! The peanut butter cup of Prior Ghosts Custom response
Slackbot 4:28 PM jelly time
Nick Porcino 4:28 PM I bet that course is going to be very interesting, I envy the students 🙂
sydneyskybetter 4:29 PM LOL 4:29 Won't be boring, that's for sure.
Nick Porcino 4:32 PM Now I want a coat of arms, so I can put e, p, a on as a motto!