`Metamuse Episode 37 — August 19, 2021 Visual programming with Maggie Appleton, Mark and Adam. From https://museapp.com/podcast/37-visual-programming/ and transcribed with https://replicate.com/thomasmol/whisper-diarization

visual_programming_podcast_maggie_appleton_transcript.txt

Maggie: Maybe the past generation of programmers were sort of subjecteded to a really awful version of visual programming, but it would be really sad to let that frustration that other people have felt ruin what visualprogramming could be in a much better design format, or just done in very different ways, done in ways that combine graphics and text.
Adam: Hello, and welcome to Metamuse. Muse is a tool for thought on iPad. But this podcast isn't about Muse the product, it's about Muse the company and the small team behind it. I'm Adam Wiggins, joined by my colleague Mark McGranigan. Hey, Adam. And our guest, Maggie Appleton from Egghead.
Maggie: Hello.
Adam: Thanks for joining us, Maggie. Now, listeners of the podcast will know that Mark and I are fans of computing history. We think sometimes the technology industry is a little unaware of its own history. And I understand you recently visited a museum on that topic in your area.
Maggie: I did. I got last weekend to go up to Cambridge, which is about an hour away from London, and visited the Computer History Museum of Cambridge, which was mostly a warehouse that some people had put an old N64 in and an old Mac and got to play a bit of Pong and Space Invaders, which is terribly educational, but probably not a comprehensive history of computing. Yeah.
Adam: Now, depending on what generation you grew up with, you know, so now I'm old enough that I go to a computer history museum or sometimes there's, I don't know, like the Pinball Museum in Alameda there in San Francisco or some of these arcade ones. And I remember this stuff when it was new, maybe as a kid. So in that case, it can be as much a sense of nostalgia or a comparison with your sort of adult perspective on something that loomed large as a child. Were there things there you recognized or was this all like...
Maggie: I was definitely of the N64 generation and definitely the, I don't know which version of the Mac it was, but I have a very clear memory of going into a friend's house who had the Mac. Windows 95 was maybe what we had had at home. I mean, I was, grew up sort of in the noughties, so I'm that generation.
Adam: Is that what we're calling it now? I've heard the noughts.
Maggie: I know, the noughts. There's no good word for it, I mean.
Adam: Yeah. The zeros. Turn of the century, maybe we could go with. Yeah. Yeah. So before we get into our topic, love to hear a little bit about your background, Maggie, as well as a little bit about what you do at Egghead.
Maggie: Sure. So yeah, Egghead is where I currently work and it's a web development education platform. So we kind of will joke it's Netflix for developers. It's mostly video tutorials, teaching anything, mostly on JavaScript, lots of react, angular view, that sort of thing. And I'm the art director, designer, illustrator there. So essentially I handle most of anything that is visual design. And I also do quite a bit of UX design there. It's sort of a small company, so you get to be a bit of an all rounder. And because it's a very developer focused company, I also develop mostly working in react. I like to call myself a mediocre developer, so I'm not intense, you know, I don't really like dig into, you know, the deep code, but I know my way around, you know, front end database. So yeah, that's kind of me.
Adam: And it seems like you came maybe from the, I don't know, illustration or design side, and then that led you to. Kind of the development side, or was it the other way around?
Maggie: Yes, it was definitely design and illustration first. So I had grown up in a very technological household. My parents were both programmers. I had a lot of access to technology quite young. So I learned, you know, HTML and CSS, but I wasn't really introduced to proper programming young. And, you know, there wasn't any like IT education in school. So I went more into design and illustration in my early twenties and then learned programming mostly out of employment necessity. You know, I was, I was designing websites and, you know, starting to learn JavaScript a bit and figuring out how to get a modal to work. And then of course, once I started working at Egghead, got a lot more into programming and development. So I very much come at it from, I struggled to learn it. I'm maybe not, I don't want to get into natural types of people, but of course, some people have an affinity for being able to do abstract reasoning and they're very good programmers because they can hold a ton of cognitive context in their head about what component is connected to what database, you know, where messages being passed. And I am not strong on a lot of those fronts. So I do development as someone who needs things visually displayed in front of me. I need a lot of explicit feedback from the machine telling me what's happening and what things are connected. So my perspective I kind of bring to programming is I'm the person that struggles with it. So I really, really want tools that actually help you program if you're not naturally inclined to abstract reasoning.
Adam: I think it's interesting that you saw basic programming, through HTML and JavaScript, as being almost sounds like a table stakes for your career. And I think that's the nature of the computing medium. You can imagine doing visual illustration design work in print, for example, and there you probably need to learn some things about printers. But for the most part, you're fine just doing everything in pixels on the screen. You hand that over and there you go. And that's probably true in other realms as well. But once you're on the computer, now you're working in a dynamic medium and your ability to express things is very limited. And if you're stuck with what's static, I think that's going to become more and more true here in the
Maggie: future. Yeah, I think it was good. I was ignorant of the fact at the time that I was in the generation where people were designing websites in Photoshop. And I didn't know that was an option. So I went and learned CSS and JavaScript because I couldn't figure out any other way to build a website. I didn't know I could just mock up pixels and hand it to a developer. So I'm very glad I was ignorant
Adam: of that. So our topic today is visual programming, and also to some degree, its relationship to the learnability of programming. And since you suggested this one, Maggie, maybe you can explain a little bit for the audience with the basics. What is visual
Maggie: programming? Why is it an interesting area? Sure. So visual programming is sort of a catchall name for a whole bunch of approaches to trying to do programming differently, I want to say. I think the name itself is problematic, which we can get into a bit. But it's essentially trying to explore graphical interfaces for programming in various ways. And there's been a long history of people trying to do this, and a fair amount of push back to versions of it that have gone quite badly, which we can also get into. But yes, it's essentially trying to explore ways of doing programming that is not just based in text, you know, linear text in a text editor, which is kind of the default nowadays.
Adam: Certainly, when people think of programming, they think of it as being synonymous with coding. I think of that Hollywood hacker view where, of course, there are a lot of the things that you're sitting in front of the terminal. The text is probably green on black, and it's a bunch of text in this computer language. Now, is there a well-known example of what would count as visual
Maggie: programming? The two that jumped to my mind, I'd say most famous is probably MIT has a programming language called Scratch that is technically designed for children in order to teach kids programming, sort of as a first programming language. And I think this gets referenced a lot in visual programming circles because it gets rid of needing to use sort of arbitrary syntax symbols to define things like objects and functions. Instead, it uses drag and drop blocks with colors that correlate to the type of each element. So it's a very interesting example of how you can kind of lower the bar to people coming in and doing programming since it is designed for children. So that's one that gets famous. But others that I would count as being some form of visual programming that will be very familiar to the audience are tools like Zapier, If This Then That, Integromat now, the sort of the whole low-code, no-code movement where people are essentially building node-based logic flows where they're saying, you know, if this happens, then do this, else do that. They're doing, you know, programmatic logic, but they're doing it in a very graphical interface. And I count that as visual programming, but I think there's also a wide diversity of, we can talk about, you know, the amount of power each of these platforms might have, the amount of flexibility and control they give to the user. But yeah, those would count, I would say, as visual programming.
Adam: One that comes from my childhood thinking of kids learning programming is there was a game called Rocky's Boots. It was like a 1980s, ran on the Apple II or something. And it showed you how to put together logic gates, the same kind that you use in electrical engineering to make circuits and solve problems. And that was one of the things that I thought was really cool. And I think that's maybe introduced me to this sort of form of logic that eventually led to my interest in programming. But one thing I think is interesting about that is, in fact, electrical engineers designing circuits work with a form of visual programming. You make a diagram that shows where the resistors and the capacitors and the LEDs and whatever else go in this form that is very much, I guess it's less of a flow diagram and more shows where the electricity flows, which may or may not exactly represent sort of the abstract logic. But I think that's a, a professional context where there's kind of a visual programming, at least representation.
Mark: And we can also talk about examples of visual augmentation. So with Scratch, kind of the whole environment is inherently very visual. Other examples we could look at are spreadsheets where you have the cells themselves are like text formulas, more like a typical programming language, but it has this visual element of layout and spatial-based variable naming. Another example from when I was learning programming is the I think it was called Dr. Scheme. I think it has a new name now, or maybe a change to that from something else. I forget, but it was the Scheme development environment, which first of all, was all-in-one, which is great. You just got this like one executable you could run and do everything. But also when you hit errors, which is the dreaded case of a new programmer, it just drew lines to everything you needed to look at, which sounds so basic, but just connecting the dots between this is your call site and this is your function definition. And there's something wrong, you know, check where these two lines terminate and see what's up. We could talk about all kinds of examples of visual augmentation as well.
Maggie: Yeah. And I know you guys are also familiar with a few maybe more powerful visual languages like origami. And I think Xcode is quite visual in the way that it allows people to program. So there are fully-fledged IDEs that have visual elements to them and might not be fully-fledged visual programming environments, but the hybrid is actually much more interesting than this idea of pure visual programming, but we can get there.
Adam: Our colleague Leonard uses origami for some design prototyping. We even put some screenshots into one of the beta. We did a link that, but it's interesting because he's doing design work. And typically he can do that in a tool like Figma and pretty static mock-ups. And then we go from there working with Yulia interface engineers. They connect the static screens to behavior, but there are certain things, for example, when we were working on this whole infinite canvas flex boards feature, where just how it felt and how it behaved was the whole design. And you just couldn't evaluate that by looking at a static screen. And so I think that's a really good point. And I think that's a really good point. And I think that's a really good point. And I think that's a really good point. And I think point. And furthermore, just the standard click prototypes like you get in a lot of modern design tools just won't cut it. And so he ended up doing some fairly complex programs that you could then in turn run on the iPad and really feel how it worked. It's quite impressive.
Mark: It's going to be very tempting to market this episode with a D&D style alignment chart for visual programming tools. It's like lawful good is scratch. And what's the other corner? Chaotic evil is a meme generator as a programming tool. Something like that. We'll see if we can pull it off.
Maggie: Yeah, I'm thinking of other ones I've seen. I think there's a category of them that I think was very popular in the 90s. Like I'm specifically thinking of LabVIEW gets a lot of negative press, I'll say, is maybe not being chaotic evil, but it's one of the infamous ones that I think was proprietary. So you kind of got locked into the system. And then there's a whole Tumblr. We can put it in the show notes when I find the link to it, but I can't remember the name now that is just screenshots of awful literal spaghetti code because it's a node based visual programming language. So it would just be thousands of boxes and lines running between all of them and literal just a visual spaghetti that you can't tell down from up. But yeah,
Adam: that would be somewhere in the evil dimension. Yeah, the boxes connected by lines also reminds me of another big category that I've come across in my, I guess not my day job would be the way to put it, but music production is something that's been of great interest to me, particularly earlier in my life. And so I worked a lot with software like Logic Audio. For example, sequencers or something like Reason, which is a kind of an all-in-one synthesizer plus sequencer. And these typically have various ways to essentially wire stuff together. But I think one reason that the visual thing works well there is you have this clear kind of input output flow. The output is always going to be an audio stream, or in some cases, there's video tools as well where there's a video stream, but it's a single sort of time-based flow. And so like a music track or something like that, you've just got, seven minutes of music output. And you can, in Logic, for example, go to the quote-unquote environment, and there you can just string together, okay, I want to take the source material, I don't know, someone playing a guitar, and I'm going to put it through these filters, and I want to fork it off this way. And then the end output just goes out through, you know, your master volume. And obviously that is very much designed for professional, very serious use cases, but these are non-programmers.
Maggie: Yeah, I've heard the visual effects industry has a lot of those same sort of interfaces. So the node model box in there, I was just connecting different things together and really just showing data transformation over time, which is when a flow diagram actually comes in useful.
Adam: And then another one I think I'd be remiss not to mention is iOS has a automation system built in called Shortcuts, which was originally a very cool indie app that Apple acquired called Workflow. And this essentially gives you sort of a scratch style model, but probably even simpler in the sense that you don't have a lot of nesting and what have you. You just sort of assemble these blocks top to bottom. You can do it on a phone, which actually is pretty interesting that it's possible to do reasonable programming on a phone. And again, I think it does rely on a fairly linear input and output. You're going to apply your shortcut to this image. You're going to do some stuff to it. Something comes out the other side. Maybe it's a transformed image. Maybe it's something else, but that's a good example of a place where visual programming maybe is actually the mainstream in a way that a lot of folks don't even acknowledge.
Maggie: Yeah, it kind of transitions us into the idea that visual programming ends up being this really large umbrella that I think a lot of the debates around it get into people just not having an agreed definition of what we're really talking about. And so I mentioned that we would come back to it being that it's kind of a terrible name because to call something visual programming suggests that it is something visible with human eyes, which would most certainly include text or a screen. So that would include all programming. So when people say visual programming, I think what it's really getting at is, this is my sort of opinion on it, I suppose, is it's trying to take things that would otherwise be implicit in programming. So parts of the program we don't see. So right data transforming over time or a variable being passed through a function and making it visible in some way. And this is kind of where it gets into the balance that visual programming platforms or languages have to kind of grapple with. How much do you make visible and in what context? Because you can't make everything visible because the point of programming is abstraction and working with higher level primitives and sort of taking away things we don't want to always deal with, right? Like where memory is allocated. You don't want that visible in a higher level programming language. So I always see it as how we take programming and we make different aspects of it visible in different contexts, depending on what we're trying to do.
Adam: Is there a better name? Or if you were in charge of the rebrand on this some time?
Maggie: I feel like graphical programming would at least more accurately describe what a lot of these platforms are trying to do. I mean, a lot of them are simply just trying to apply better UX and interaction design to programming, right? They're just trying to make it a more enjoyable experience that gives the user or in this case, you know, the developer, what they need at the right time without overloading them with information, you know, really just being a good cognitive tool for them to get their work done. So yeah, but graphical programming is a good I think would more accurately describe node-based interfaces, or drag and drop, you know, ones like Scratch, or even debuggers, really, I think are a form of visual programming in a way, ones that really walk you through how the program is executing over time. Although I think it's kind of wild that you wait till something goes wrong until those tools come into the picture. Like, why isn't there an inline debugger in every IDE? But you know.
Adam: Actually, one of the first sort of one of my mentors, when I was getting started with the professional programming, he strongly felt that you would go and put breakpoints at every interesting point in the code the first time you ran it, and you wouldn't run it and just see what happens. You're going to step through, inspect the variables, watch every single point, and only once you've been through it all and watched it all execute from the inside, and everything's what you expect, then you can look and see what lands on the screen, if that's what you expected or not. But that's kind of the final step. That's more your victory lap, not your first test of whether what you did worked.
Maggie: I think you mentioned that console logs, right, are one method of sort of putting out or making the program more visible at certain points in time. I was trying to think of a good metaphor for what console logs are. It's like the stream of the program, and it's like you're shooting up a small bit of information at a point in time.
Mark: Yeah, it's sort of like if you have a map of a territory, and you take a journey along the territory, like we go in loops, and you cross back over your own path, and you make jumps and stuff. It kind of captures all of that in a linearized form. And I see it as a complement to the map. You know, the map gives you the nature of the territory. Now people are going to get mad at me on the internet for saying that, but you know, the map has some correlation to the territory, hopefully, but also the linear view of everything that you encountered is also very useful.
Adam: One idea I would raise for the rebrand question I asked earlier was on the ink and switch piece we did on end-user computing. And we can talk about the relationship of visual programming to end-user computing, but we talked about this. Property of embodiment. And it's exactly what you said, the idea of taking something implicit and potentially very abstract and making it visible. And I think a lot of times in discussions I've seen about this, which are often among people who are already professional software engineers, and that includes both people who are very enthusiastic about end-user programming or just increasing the accessibility of programming, but they themselves are still professional programmers. I have that background, Mark does as well. Yeah. Yeah. Yeah. Whether or not they're pro or con, they're coming from that perspective of someone who, for whatever reason, found it, was drawn to or was good at holding all of that abstraction in their minds. And so in some ways, I wonder when I see the objection of, I don't know, there's so much visual spaghetti here, but that spaghetti all exists in symbolic code as well. You just can't see it and you have to mentally model it. And some people are unbelievably good at that. And a lot of those people are people who go on to be really great at it. And I think that's a really good example of some of the things that we can do to be really good at programming. I'm probably kind of in the middle on that. My complexity threshold is not that great, and I tend to always want to factor my code in a way to make it more comprehensible and give myself more visibility and that sort of thing. So I wonder how much the needs of visual programming or this umbrella we're calling visual programming, those serve sort of hypothetical people that today are not programmers, but maybe would have that capability, which I think is the other dimension of this, right, which is the learnability of the programming.
Maggie: side of it. Yeah, I definitely agree with that, that I do like to be the representative for mediocre programmers or people who aren't naturally inclined to it. And that I have heard a lot of the debates around it of people saying, well, I love them. Don't take them away from me. You know, if you try to make it visual, I won't be able to work in my crazy text editor with all my customized hotkeys. And I'll say like, textual programming is not going anywhere. You know, it's like, this is all going to be additional affordances built on top of a textual paradigm. You know, text is not leaving. But people say they can't understand why we would ever need it. Whereas my perspective coming into programming, you know, starting when I was 20 or so, when I was first shown VS Code and the JavaScript tool chain that I was supposed to work in, and NPM and the command line, I just went, you can't possibly be serious. Like, this can't
Adam: possibly be the way that I'm going to do this. They're pranking you.
Maggie: Yeah, why? My computer looks like it's just stepped back into the 80s. Like, I don't understand. You know, I've grown up in the rich graphical user interface that gives me what I need visually. And it was just shocking. So I do think that people who are programmers, or professional programmers, are maybe ones that I don't know if they want to say they're more willing to step into that world of invisible connections and implicit logic. And then it is everyone that maybe would shy away from that or would find that more stressful or just less intuitive. They end up being like, okay, well, programming is not for me.
Mark: That really resonates with my experience. And I think there's a few things going on here. One is this issue of accidental complexity, where you have aspects of the experience that are just not really essential to doing the actual programming. Basically, the tools are designed poorly and unnecessarily. So there's too many things going on. Some of them don't work. They're unnecessarily confusing. I had a similar experience when I first wanted to do a web app. At a time, the state of the art was these Java servlet things. This is full-on enterprise Java wildness. And when I encountered that, I was like, there's no way I'm going to do this. Sorry. And so I went back to my doctor scheme where everything is in a nice little bubble. Eventually, you got Ruby on Rails and everything was fine and so forth. But there's this other issue of communication and information theory. So by that, I mean, a lesson that I learned from management is whenever you're communicating something, something is said, and then something else is received on the other end. It's like a lossy process. And I think the reality is that it's a lossy process. And I think the reason a lot of programmers like text-based interfaces is that they have very low loss there. They kind of understand what all those symbols mean. And furthermore, they're very dense. Whereas a new programmer, and I was once one of these, basically it all gets lost. It just looks like a garble of alien symbols. So one of the reasons that visual programming environments can be so appealing is they have lower underlying information density, but they're constructed in a way such that the optical clarity is higher, right? More information gets through to the other side. And I think ultimately, that suggests that there's a lot of loss there. There's this endgame of you get both the very high bandwidth potential of text-style interfaces with the augmentation that you can get from other visual aids. I don't know exactly what that looks
Maggie: like, right? But I think there's something there. I definitely, I think, discovered that when as part of my work for Egghead, a lot of my work is illustrating programming concepts, right? So it's very much just educational material. We're talking like articles or animations that kind of go along with video lessons, where, you know, I'm visually explaining things like, you know, how GraphQL works, and how it works, and how it works, and how it works, and how it works, and how it works, or how React's hooks work. So it's just taking concepts. And what I believe I'm doing is making visible the mental models. I tend to use the word metaphors, because I'm from more of a cultural anthropology background. And I've read a lot of thinkers like George Lakoff and Mark Johnson, who talk about embodied metaphors, and sort of the fact that almost everything that we talk about in the abstract realm is based in our embodied experience of the world. So we have this inherent understanding that, like, up is good, and is more, and is happy. We'll say the stock market went up, and the stock market didn't physically move upwards, right? It, like, had more money move into it. But we use the term up to think more, good, better, right? And we'll say, like, you know, I'm feeling down today means I'm feeling sad. You're not physically moving downwards. But the point is that we have these metaphors that are based in being humans with bodies that have, like, fronts and backs and ups and downs, and we map all our abstract reasoning on top of that. And when you have to draw, like, a programming function, you really have to look at the language we use around programming functions and really figure out, like, okay, there is a physical metaphor baked in here, because all things are based off physical understanding. And then drawing it, you know, you draw, okay, there's a component that's a higher order component on top of a lower order component, and it's passing data. So it's, you know, mimicking this idea of physical objects moving through space. So it was very obvious to me in trying to draw programming concepts, that it's based in physical knowledge. And visual programming languages or visual interfaces, just make that explicit, right? They like actually draw a box that is above another box, or it is to the right of another box, which suggests it's in the future, because we think the future is to the right. And the past is to the left, at least in English, Western speaking countries, that's how we map time. So I've always just been kind of fascinated by how we can sort of use these embodied metaphors to help people understand programming concepts, because everyone has a body. So you get these things for free, like we all get. The concept of front and back, and we can use those in interfaces to communicate to people, you know, the relationship of these otherwise very abstract lines of code that are just encoded things on a hard drive that would otherwise be incomprehensible.
Adam: That's super interesting. And also, it actually connects to our design ethos for Muse. A lot of our concept there is we want to tap more into, for example, your spatial reasoning, which is very strong for most humans, as well as things like the direct manipulation and touch, and so on. So we're really taking advantage more of our bodies and our minds, natural, might call them talents, or just built-in capabilities. Because, of course, so much what we do with computers, we're really bringing ourselves into an abstract intellectual realm. And we can learn to do it, but it is very much a learned and a learned over a long period of time skill.
Mark: Yeah, and I think this idea of understanding, whether that's through metaphors or physical embodiment or mental models, is a very important part of our design. And I think that's a very important part of our design, because ultimately, most programming errors or failures are ones of understanding and not ones of mistyping. And I think a lot of our tools, they focus on the mistyping issues, and it's much harder, yet more important to get at the ones of understanding. If you think about the most gnarly bugs you've dealt with, it's usually because you misunderstood something about the world you're modeling or how your program worked, not because you typoed something. If you did typo something, that kind of gets fixed quickly. So anything that can help us understand what actually is, is very powerful, which, by the way, is another reason why I'm a huge fan of logs, because they are a sort of ground truth with very high information density of what
Maggie: actually happened in your program. Yeah, I've actually recently been working on a project that really sort of emphasized to me how important this is, right, of the mismatch between the program creator's mental model and the person trying to write the programming language. So of course, I've mentioned JavaScript, because that's just kind of the world I work in. So this is a collaboration project that I worked on with a programmer named Dan Abramov, who's a really wonderful programmer. He's a really wonderful programmer. He's a really wonderful programmer. kind of well-known JavaScript developer who's on the React core team. And so we've been collaborating to help create a course that visually diagrams and animates the underlying mental models of JavaScript the way that it actually executes, which helps people get over a lot of the bugs that are written into JavaScript code simply because they don't understand what happens when you say, let banana equals fruit or whatever, and then you reference that later. But if it's been reassigned, or it's inside an object, and it's been passed through a function, they aren't tracking in their head exactly where the variable's gone and how it's changed over time, because it's not visible in the program. So anyway, this project is all about showing people literal, explicit diagrams, showing them what happens when certain types of code executes in JavaScript, and it corrects a lot of the mental models that are incorrectly taught in a lot of JavaScript textbooks. And this is where working with Dan and being able to help visualize this course, I completely reformed my understanding of how JavaScript code executes. But there's no IDE that was going to show me that. So it was just one where I went, it's great to make educational materials for this stuff, but it's not built into the tool.
Mark: Exactly. It reminds me, one of the earlier programming courses I took was in C, and it was kind of notorious because there was this one point in the course where a lot of people hit a wall, which was when you introduced pointers. It's this incredibly important and fundamental thing that totally changes how the variable behaves, but it's just a little star in front of the variable name or whatever. And it kind of resonates with what you were saying about the mismatch between how people are thinking about what's happening versus what is actually happening and the importance of correcting that.
Adam: How do you go about coming up with a metaphor when you're working on something like the Just JavaScript course or any of these egghead pieces? Presumably, you need to develop your own understanding of it, either from your own experience or from the course materials, but then you want to go to translate that to some kind of visual thinking or create a metaphor, hopefully one that's correct. How do you do that? Can you give us an example?
Maggie: Sure. I'll say there's sort of a spectrum. I mean, I like spectrums anyway, right? There's a spectrum of visual programming, and there's a spectrum of metaphors, I'll say. So the Just JavaScript project with Dan probably leaned a lot more on very direct spatial metaphors. It was very much like what is connected to what, what is above what. So it's sort of circles and arrows in a very classic diagrammatic way. But a lot of the other work I've done for egghead is illustrating programming concepts. And what I would say is, oh, I don't know, I don't know, I don't know, I don't know, I don't know, I don't know, I don't know, I don't know, I don't know. I don't know, I don't know. more towards the poetic spectrum of metaphor, where it's not technically like, you know, the most accurate, it doesn't exactly match to syntax showing what happens at each point. But I'll make a lot of illustrations that will be metaphors that give you an understanding of like, oh, okay, if it's a waterfall image, it's like, okay, this is something about data flow, that sort of level of poetic metaphor that gives you more of the gist of an idea more than the literal syntax interpretation. I'm thinking of one of like, did one for Redux, which is a state management system react. And I use like a little joystick with space aliens like shooting, because it has actions. So it's like you can like shoot out actions from Redux from like a single state source. So that's one way it's a bit more fun. Obviously, it's a lot more visually interesting than just doing a box narrow diagram. But it gets the concept across to people that you have a central place and it's shooting actions out from it. And that the process I kind of do for that is whenever I'm given something to design a metaphor for, I will read docs, you know, I'll read programming articles, I'll go see what people are talking about on Twitter related to the thing, stack overflow questions. And sometimes if the community is already speaking about it in certain metaphors, I just use that because they clearly already understand it. If they're trying to explain it to each other on stack overflow, and they're using these metaphors, that's usually what I'll base them on. But a lot of the time, I'm just kind of doing basic reasoning of going, okay, what are the functions of this thing? What are its qualities? And then you do a bit more creative lateral thinking to say, okay, what else in the world has those functions?
Adam: Those qualities. Yeah, I think that that points to another reason to have this kind of illustration in learning materials, which is just to make it fun and break things up. And yeah, visuals to break up, for example, long streams of text, or, you know, if it's more of a video format on strings of speaking, or you're just looking at a screen that kind of looks the same the whole time, you know, it's just a code editor window or something. I'm reminded a bit of Why's Poignant Guide to Ruby, which was a beloved, introduction to programming for the Ruby language and had these adorable cartoon characters and kind of a web comic style. And, you know, a lot of it was not specifically helpful for helping you learn it. It just made the thing just more fun to consume and more memorable, because you connect Chunky Bacon with, you know, some of these more practical concepts that are being taught to you.
Mark: Chunky Bacon, that's a throwback, man. Wow.
Adam: Now, I can't believe we got this far without mentioning Brett Vector's seminal piece, Learnable Programming. And maybe somewhat to the point you've both made in various ways, which is, it doesn't have to be either or. The idea of creating embodiment or visualization or showing more of what's happening, the abstract pieces, how things are connected, how things flow in time, can also go with symbolic code. And this piece from some years back now, but I think has been influential in terms of, well, people lamenting we don't have programming environments that are a bit more like this, but demonstrates lots of interesting and inspiring ways to potentially make, for example, control flow more visible.
Maggie: Yeah, the Brett Vector piece has kind of been a seminal launching point for this whole visual programming industry. Of course, it was alive and thriving well before. I mean, you can kind of trace it back to, of course, Alan Kay and Xerox PARC and Smalltalk. And there's a lot of precedence for this. But the Learnable Programming piece, I think, is the one I've seen referenced the most and has gotten the most attention. And I think it's a really good piece. People, you know, maybe in the last five to 10 years, more interested in it and to get them to go exploring and to go, okay, maybe the way we program now doesn't have to be the way we're always going to program. And the thing I really loved about Victor's piece, I mean, of course, it's a long article and there's many aspects to it that we could kind of dive into. But this idea of just that because code, when it comes down to it, it's something being executed in full-on machine code and byte code. And with so many layers of abstraction up from there, it's not like we only have to represent it in one way. When you get into the higher level programming languages, syntax is one way of displaying what that machine code is actually running. And the rendered UI is another way of displaying what that machine code is running. And I don't know why there's not a reason why we can't flexibly display the same set of machine code in different ways and turn it on and off and say, okay, I want to look at it in visual data flow setting, or I want to look at it in a way that shows me the component architecture, if it's like a component tree kind of context. So just flexibly displaying it in different visual formats, rather than saying, okay, syntax is the only way I'm ever going to interact with this code until it becomes the
Mark: final output. Yeah, interesting. And now that you mentioned that, I feel like, as is often the case, the gamers are ahead of us, in particular, the game designers. I'm not an expert in this, but my understanding is that the game engine environments, they're very sophisticated. There's like the code view, then there's like the object graph view. And you can like, you know, turn on different pieces of the visual environment. And there's all kinds of performance based views for looking at memory and CPU and GPU usage and so on. So there's perhaps something to learn from
Maggie: that industry. I also think the data visualization industry is one we could really draw a lot from like I know, Brett Victor had mentioned he had researched a lot in Edward Tufte's work, who's a information designer, I suppose is the best term for him. But he's written a number of books like Visual Explanations and Beautiful Evidence. That cover kind of the best practices for how you can represent data as visual charts in a way that it actually respects the original data and presents it in an accurate way that isn't bending it or being too manipulative about the way that you're presenting it. But he has a wonderful set of sort of principles for how to do it well. And Victor had talked about how he read him before writing learnable programming. But the dataverse community in general has done a ton of work on thinking of different ways to visualize.
Maggie: And that's sort of in my head, what we might be able to do with syntax in sort of maybe IDE plugins in the future. Of course, there's plenty of IDE plugins that do help you look at your code in different ways. But more of that is kind of what I would love to see is like more plugins that just like I'm just going to show you the architecture side of your code. I'm just going to show you how this variable changes over time in your code. I'm like one of those where you're kind of like, am I missing something? Like, why isn't this happening?
Mark: Yeah, I'd totally love to see more of it. And I think it can come both from the IDE language plugin layer, but also bottoms up from individual projects. I feel like I must have at some point done one of these things where you write a little script that like takes your code, your system and compiles a dot graph so you can visualize nodes and edges with one of these standard dot visualization tools. And I feel like there's a lot of opportunity to do that in individual projects, like you feel the need to have a different, more graphical view into your project. Well, you're a programmer, you can make it so.
Maggie: And it kind of to bring us back to what end user programming, I mean, the strange thing about programmers, right, is they are end user programming, like of all the industries, they're the only ones where they are both the users who have the ability to actually build their own tools, and to actually get in and change their own applications. And no other, I guess, community that uses computational tools is able to do that.
Adam: Here you're talking about a programmer writing their own VS code plugin, something like that?
Maggie: Yeah, exactly. Or even if you use a certain language or framework, you can build a library that adds on to that, right? Even if you're not on the core team, and you aren't going to PR into the main repo of that framework or tool, you can build extensions off it. You know, it's the beauty of being able to fork and change and collaborate on these things means that there's so much flexibility and in developers having their own authorship power.
Mark: Maybe we need more. There's a whole culture of jig making in software as there is in woodworking. Woodworking is interesting. It's one of the rare fields where maybe the only one I can think of where they have that ability to kind of build their own tools in wood. And there there's this whole culture of making jigs to help you build the furniture. In fact, you might spend most of your time making the jigs and the furniture is kind of this final, you know, culminating step. Perhaps we need more
Adam: of that in the software world. Some of the ways to visualize your code or your program differently that you both have mentioned, like in the game world, maybe something like Unity, where you tend to build your own software. You can build stuff into the game that lets you look at the scene graph in different ways. And I think we have that a little bit in the web world. Dev tools is one that we tend to give a lot of love to. I think we talked about that a good bit with Jeffrey recently here on the podcast. But there it is an example of what you said. You see the output, but there's also the kind of hover over elements and highlight which is where and what the extents are. And then there's also the margins and padding stuff that kind of goes down in the lower right corner. But then you've also got a expanding hierarchy. And then you've got a lot of stuff that goes down in the lower right corner. And you can see those all at the same time. And even as you're mousing over, it's highlighting. So you're looking at exactly the same thing in three different ways. And you can spot different problems or understand by visualizing it in these different ways. And I would further argue that we see that a little bit in more the native development world with toolkits like Flutter and SwiftUI, where there often is even a live side-by-side, here's your interface output, here's the code. And often you can change things on both sides. And it flows bidirectionally. But I think in all of those cases, it's already something where it's a visual output. And so you're interacting with some of the under the hood stuff around that visual output. It feels very natural to visualize that. When you come back to the more abstract things like program flow over time, that's where maybe it's less obvious to do that. Or maybe it's just less clear how you would do it in the first place. But yeah, I think that visual debugging or visual understanding of something that is already a visual output is a very natural thing. Put to begin with, is actually a pretty rich place for the stuff.
Maggie: And especially, I mean, you had mentioned the two-way or bidirectional aspect of it is super important. I mean, one thing I'm really liking is there's kind of a new rise in state charts and state machines, but visual ones that flow both ways. I'm specifically thinking of there's a tool called XState that works for React, where you can change either the diagram or the code and they're side by side and they mirror each other. Whereas if I want to debug some CSS thing in the browser, it's like, I'll go test it out in the browser, but then I still need to go back to my ID and actually change the actual code. And it's just that slight friction of you're not actually changing the thing itself. You're like mocking it up in one and then you go back to the actual source. We'll know in interaction design, any amount of friction you put into a tool or an
Adam: interface is just causing that much more cognitive load on the user. I'm looking at XState right now. I hadn't seen this before. It is very cool, like code and flowchart diagram side by side. We've done a bit of research on the bidirectional side. We've done a bit of research on the bidirectional side. We've done a bit of research on the bidirectional editing capability as part of Ink and Switch. And it can be very, very hard to do that well. If you have something like, maybe a tool like Apparatus would be a good example. I can link that in the show notes, but something where you have the ability to do direct manipulation with dragging something with the mouse or your finger in one kind of pane and the other, you have code that you can update, but having not one be the end state, the output. I don't know, the bad version of this is Dreamweaver, right? The output it writes is just completely incomprehensible and you can't kind of go back from there. So that bidirectional thing I think is sort of a holy grail in one way, but also it's very hard to do technically. And I think also depends a lot on the nature of the programming language itself and the tools you use to edit
Maggie: kind of the symbolic representation. Yeah, I usually take the long stream of maybe, I don't want to say failures, but disappointments in the WYSIWYG world. You know, everyone that's ever dreamed of, yeah, Dreamweaver, you draw the... website and you can change the size of the div and the code, you know, reacts appropriately. The fact that no one solved it after this long and that many, probably billions of dollars, I'm not quite sure. It shows that it must be an incredibly hard problem, you know?
Adam: Yeah, indeed. And I know in the Xcode world where it ships with this interface builder, which is essentially a drag and drop GUI system for setting up your interface, but very often I've seen iOS developers, they prefer to say, just don't use interface, they'll write everything in code because sooner or later you're going to be basically working in code anyways. And if you have some of your program is represented in this visual thing that's edited one way, some of your program is represented in code, the two interact with each other in a bunch of complex ways. You'd rather have just the one medium that's actually better long-term. Now, another great article that I came across by total coincidence, just as you and I, Maggie, were talking about possibly doing a podcast on this topic, is an article that is, called Hacker News Folk Wisdom on Visual Programming. And it essentially is a pretty large scale analysis of every single time something visual programming related has come up on Hacker News and sort of like distilling out the common themes and then starts with the really common and fairly shallow reactions. And it goes into some much more nuanced and interesting discussion. So there's lots of here, including some we've already talked about, but is there any pieces of this article that are especially interesting to you, Maggie?
Maggie: It's funny, I had been following a lot of those same threads, and there's a fair number of them, and they're the sort of Hacker News topics that go for hundreds of comments, right? People just go deep on this stuff. I had been reading a lot of those same ones and passing through them right when that article was published. And I tweeted the author, I said, you know, thank you so much for doing the hard work of distilling that all for us. But I think the thing I found most interesting about it, and just from reading the threads is, I mean, the strong opinions people have on it, or you really get into people have very emotional, I want to say, like, reactions to visual programming. And sort of the cultural anthropologist side of me just finds that quite an intriguing thing to look at, where many of them, I want to say, have almost been like emotionally scarred by past visual programming languages, like the stories they're telling in the comments of the time they had to work on the most horrendous proprietary visual programming platform. And no one could find anything. And the literal spaghetti code is just all over the place. And they just wanted to go back to text. So I think there's some very legitimate critiques that should be acknowledged that maybe the past generation of programmers was sort of subjected to a really awful version of visual programming. But it would be really sad to let that frustration that other people have felt ruin what visual programming could be in a much better design format, or just done in very different ways, done in ways that combine graphics and text. Yeah, so that was my main takeaway from it. It's almost like you have to do some historical healing before you can kind of move forward with this concept.
Adam: It is interesting that it can be such an emotional topic, because I think if we were to boil it down to a bit of a duality, you have both people like, I think those of us here talking right now, who really believe in some kind of end user programming future or programming being more accessible, or just computing as a medium is so important for humanity's future. And we need just more humans and a broader array of types of humans, different perspectives and different backgrounds to be able to... to access this medium. And so that's maybe something we feel strongly about. Certainly, I feel very passionately about. On the flip side is maybe, as you said, first of all, the bad experiences that have scarred people. But also, you do come back to this, again, the Hacker News comments are people who many and most are probably professional software engineers. They're people for whom the pure text model and classic text editors and IDEs and so on serve well. Or maybe it's less about being served well and more that they've had to go through the pain and suffering to get over that hump to learn all those abstractions and manage to hold all that stuff in their head. And then in a way, maybe they would feel robbed if someone else can just kind of skip over that. I actually had that feeling. I learned a little bit of electrical engineering many, many years ago doing Burning Man art projects. I wanted to make stuff that lit up in cool patterns with LEDs and so on. And I had to go through the whole kind of classic electrical engineering process. And I found it very challenging. It didn't really fit with how my brain worked. I thought it would be relatively easy because I'm a computer programmer, but circuits are a whole other thing. And just a few years later, we had the Raspberry Pi and the Arduino and increasing number of these kits and Adafruit and all these much more accessible things. I remembered talking to a friend who said, oh, you know, I'm going to set up my project that will have, you know, these LEDs or whatever. It'll color cycle like this. And I'm like, everything you're describing there, that's going to be a lot of work. I hope you've really budgeted some time. And he bought these kits, you know, off Adafruit. And a week later, he had this amazing thing working. And I went in and asked him about it. And I realized that he just didn't need to learn all this stuff about debouncing with capacitors or whatever that I had to fight through because they're these just more accessible platforms. And there's a knee jerk. I think I had that then, a knee jerk, kind of like, wait, I had to walk uphill both ways in the snow to learn to do this. Other people should have to, too, or I wasted my time or something like that. But of course, I think it's much better that there are much more people that can now, like, do things like make cool light up projects because these technologies are more accessible. I feel like there's probably a version of that for today's programmers versus people that might access it in the future if it was a little bit more learnable.
Maggie: Definitely. There's also another, I suppose, trend in people that are a bit anti-visual programming or I'll say resistant to it or have some, you know, skepticism and reservation around it is this, I have found this sort of a duality where people lump themselves into going, okay, I'm a mathologic person and art people are on the other side of campus and I am not an art person and everything visual is art. And this idea that they're not visual thinkers, I hear that all the time from people who are in programming. And I can't really fathom what that means in a lot of cases because I go, well, visuals about embodiment, you have a body. And I think, you know, you could probably draw a box and an arrow and communicate some visual stuff if you weren't trying to draw some detailed drawing. But yeah, I think the, I'll say, yeah, historical divide of art from science and visual. Visual from logic and this idea that math happens in textual syntax is a bit of a hangover that people just think, well, visuals don't belong in math and programming. But I mean, this is why I think cultural anthropology, one of the sort of points of it is to sort of show people the vast diversity of expressions that human life and culture can take and that everything we have was invented at one point in history. And we are one very specific, narrow manifestation of possible human culture. And there are many others. And we could switch to them or move towards them. And we don't have to live in some world where math and art live on alternate sides of a dichotomy.
Adam: That's a really powerful point, Maggie. I feel like we should wrap it up there. But for people who are wondering about examples, they want to see what this looks like, because indeed, it is a visual topic beyond the ones we've already mentioned. Are there some good resources you can name that allow someone to flip through and see the variety of different experiments folks have tried for visual programming?
Maggie: There's some people who have done a lot of wonderful labor compiling screenshots of I think every possible visual programming language or platform that's ever existed. And what I want to point to that we'll put in the show notes is a visual programming codex that Ivan Rees put together, which is a GitHub repo with just a wonderful array of examples. And there's another one called Whole Code Catalog. So a nice play on the whole earth catalog that has, again, a nice sort of categorized collection of different programming platforms and alternatives and the sort of the different ways that we can use them. And I think that's a really good point. Different qualities each of them has all nicely organized in a beautiful website.
Adam: Well, thanks, everyone, for listening. If you have feedback, you can write us on Twitter at MuseAppHQ, or on email, hello at MuseApp.com. You can help us out by leaving a review on Apple Podcasts. And Maggie, I'm really glad you're out advocating for making programming more visual, more embodied, more learnable. And I'm especially inspired by the idea of having a less bifurcated world between the art and the visual. And I think that's a really good point. I think we need that. That's a beautiful vision for me.
Maggie: Thank you so much for having me on. I really appreciated getting to come on the show. And I've mentioned to you, I adore the podcast. I think I've listened to every episode up until now. So it was really wonderful to get to finally come be a guest.