Bristol new wave jazz

A couple years ago I joined a project at University of Bristol called 4Dmodeller. It was supposed to be an extension of the R-INLA package for Bayesian spatiotemporal modeling. This kind of modelling usually requires a lot of domain expertise in statistics to make it work. We were hoping to relax those requirements. I don’t think we were successful. However I discovered a music scene that you should know about.

My colleague who I had been working with at Woods Hole Oceanographic Institution told me that I should meet his wife’s sisters cousins son named Rory Geisow. Rory is an accomplished musician in Bristol and we seemed to have a vibe that would click. So i emailed Rory and we agreed to meet. And man was my colleague correct! Rory introduced me to a music scene in Bristol that is worth listening to. I wouldn’t call it jazz it’s something else. But you can hear the jazz inside of it.

First up is Hippo. Hippo is the brainchild of Doug Cave. He has this setup where he controls everything and a band that can only be described as mathematical. I love the attack that comes in all the songs. The heaviness grabs you. Hope they keep making music because the live show sticks in your brain.

Second is Zoobers. An amazing collection of keyboards (Dorian), this like electronic xylophone thing (Harriet), and drums, Zoobers creates this electro-ambient happiness movement where every chime rings inside of you. Zoobers is such a vibe. In their shows you get this amazing crescendo across the entire set where you go from this jazzy light hearted feeling to full on techno dance hall. Unfortunately they didn’t make more than this album but every song is good and definitely deserves more views than it got on YouTube.

Now I want to contrast this music with someone not from Bristol but captures a similar vibe. That’s Animus Rexx. Animus Rexx was a touring group that was Justin Tyson (drums), Randy Runyon (guitar), BigYuki (keys), and Reuben Cainer (bass). I saw them in Berlin after listening to some of BigYuki’s work. I put animus Rexx here because I remember talking with the band after their show and describing their music as “post jazz” in reference to post rock bands like mono. There is a bluesy melancholy to their music that’s hard to capture anywhere else. I feel like they fit the Bristol scene even though they are from Brooklyn.

Ukrainian music

This will probably be basic. It includes a Eurovision group! But still you should listen. And we should support Ukraine. I realize this is a political statement, but I don’t care this is my online magazine. Ukraine is the front line of the war on democracy. There are people who want to take away our freedoms and convert the world to a police state for their own self interests. Ukrainians are giving their lives so that we don’t have to live in that world. They need more than any of us have given. They are heroes. Ukraine is at the center of the future of the entire world. If Ukraine falls then the rest will go too. They are our brothers and sisters who want to live a free and liberal life like is enjoyed in countries like Norway or the Netherlands. I am mostly at a loss as to what to do to help Ukraine more. If you want to support you can start by giving money to UNITED24 or Nova Ukraine.

Haunting vocals, whistling melody, and pumping drums and bass, Go_A captures an emotion that always grabs me whenever I hear them. I know Eurovision music is basic I don’t care. Listen to them. Support Ukrainian artists.

Dakka Brahka creates new from old. A quartet from Kyev, they combine a traditional sound with a modern electronic layering. And they were on NPR tiny desk, I mean come on they have to be good right? Also they have cool hats. I am excited to see them in Paris soon. I hope they wear their hats.

5nizza is a play on words. In Russian and Ukrainian the word Friday is пятница, which is Pyatnitsa if you cannot read their alphabet. Пять/pyat also means the number 5. So their bands name sounds like “Friday” because friday is the fifth day of the week unless you are basic and think the week starts on a Sunday. I was quite pleased with myself when I figured this out without a native speaker to explain it to me. They remind me of Sublime. Saw them on tour recently. That was a great show.

Weird internet meme music

I believe that many think because of the plurality and access to the internet that internet culture transcends local culture. But that notion will evaporate as soon as you see people in Taiwan use a BBS in 2016 or the Russian speaking world use telegram and you will see that local and language culture is intrinsically a part of internet culture. Internet music usually is ironic, a meme, a kind of comment on something that probably is unrecoverable from whatever forum or personal conversation it sprung from.

I love this song. I love the flute. I love the little dude dancing around. I love his glasses. Everything about this is my vibe. The energy. The dancing. I literally bought a recorder so that I could run through the forest playing it. I have since suffered as I have not played as much as I want. If you know me and want to run through the forest playing a flute with me, drop into my DMs.

Stuff that reminds me of musicals without having to watch a whole musical

Kimbra is a New Zealand pop singer. You probably know her because she was in the Gotye video “Somebody that I used to know”. But I love her music. She has great videos and it is super fun. The dancing, the outfits, her hair, I love how red she is here. And this song is so happy. I also would recommend “Come into my head” where she is a crazy person in the video. This video also has amazing outfits. Also, “Settle Down” which is about the anxiety of being a house wife. It is a total vibe. Kimbra is great. I love her music videos.

MØ is a danish pop star who became like the positive version of Tove Lo for me. All of Tove lo’s songs feel like self destruction to me (which is why they are in the self destruction section), but MØ sounds are all happy and fun. Look her up and listen to all of her music. It is good.

Self Destruction

There is something romantic about self destruction. Movies like Requiem for a Dream, Goodfellas, or Spun always kind of spoke to me. But everyone has their genre of music that speaks to them when they are feeling self destructive I suppose.

Tove Lo’s Stay High Hippie Sabotage remix is probably what everyone knows because it has billions of views on youtube but I think I like Disco Tits more. It captures the kind of whimsy party nonsense of Tove Lo’s message. Basically, do whatever feels good because life is going to zoom by. But since it is age rated and wont play except on youtube, you can click the link to watch so that the video below has an image. hah.

Once, when i was depressed and falling to pieces, my friend Simon told me that I should listen to the song Twilight Zone by Golden Earring and find the hidden meaning. I watched this video over and over and over searching for the hidden meaning. I cataloged every frame of the video and contrasted it to the lyrics. Trying to understand. Begging to understand. Weeks went by as I worked on this. Eventually I came back to Simon and said that I was sorry but I could not find the hidden meaning. He told me there wasn’t any but he figured it would distract me long enough to do something other than the self destruction I was pursuing.

What songs do you love? Message in a way you know how and tell me.

Subscribe now

John Aiken has a PhD in physics, is editor-in-chief of low impact fruit, mayor of rainy-city.com, a Recurser, and passionate about saving the whales. John likes music and wants to know what you like too. Currently John lives in The Netherlands, happily hacking away at various problem spaces usually related to computers.

When reading all these old books and magazines from the public library you would hear about something called Zork. Zork was the original text adventure game created in the 1970s at MIT by Tim Anderson, Marc Blank, Bruce Daniels, and Dave Lebling for the PDP-10 mainframe computer. It is a game that is played almost entirely through text and imagination. It is a wonderful game where you traverse a labyrinthine underground in search of treasure. Many of the puzzles are not obvious, but this game has been part of computer culture since computers became a thing. There were many sequels, some with full motion video and graphics!, and the game itself created an entire genre of games called “text adventures” that influenced a lot of modern games and genres such as multi-user dungeons and rogue-like games.

I attended a Recurse Center batch recently. Recurse Center is a programming retreat where technically minded people congregate to build cool stuff. I think I better summed it up on LinkedIn where I said that it is basically Hackers, the movie, but in real life:

But at Recurse I met Mike Cugini who came in wanting to build a Rust implementation (see everyone likes Rust) for the Z-machine emulator. The Z-machine is the original game engine that was built by Infocom, the video game company formed out of the creation of Zork the game, to build more text adventure games. This created a big interest in Zork across the Recurse batch and immediately we had a cohort of people like Fiona Chow and Kevan Hollbach who were interested in playing Zork regularly, understanding what Zork was as a game, and also building tools around Zork. You can see below Fiona debugging an Apple //e that is part of the Recurse Center’s historical computer library (they also have a NeXt computer omg).

The first thing this led to was the creation of zulip-zork, a zulip bot (zulip is a group chat program like slack or discord) that allows anyone within a zulip channel to play zork. Actually, I built this on the airplane traveling back from New York City to The Netherlands where I currently live. Missing my Zork friends at Recurse, I thought to myself, what is a Zork project I could do? Initially, I thought, well lets get LLMs to play Zork. But I decided that was too difficult at the moment, so then I decided what would be better than to continue playing Zork online? So, thank god the plane had free WiFi, I was able to hack together a zulip-zork bot built on top of Bot-Builder and docker-zork.

Essentially, dfrotz, a Z-machine emulator, runs in the docker container serving up Zork (it also supports other Infocom games), this in turn is connected to the bot which allows it to connect to a zulip channel, and you play by typing /game <commands> as you can see me doing above. This all runs on a VPS that I pay like 8$/month for that hosts random projects like this for me. This all worked quite well and so me and C Stravidis spent many many hours playing on Zulip together. There is a plan to replace the dfrotz emulator with Mike’s Rust implementation in the docker container.

But this question of LLMs playing Zork sat in the back of my mind still irritating me. Then one night, late at the Recurse Hub deep in conversation with Walter Min, Walter looked at me and just said, “zork-bench.” And I was inspired. That night I got home at maybe 3am and then woke up early, and got to work. By that evening I had a preliminary MVP for zork-bench to show at the weekly Recurse Center technical talks. In the talk I presented the LLM (anthropic opus 4.6, that was a quick way to spend 20$ in front of a bunch of people!) traversing the map of Zork and interacting with the game. And from that zork-bench has developed into a larger project attempting to assess the reasoning and problem solving abilities of LLMs.

The goal of Zork is to go into the dungeon and solve puzzles to find treasures. Treasures can then be deposited in the treasure case. And you get points both for finding treasures and putting them in the case. You also get points for going to hard to reach locations and solving puzzles. You need to be able to manage your inventory to do this (you can’t carry everything at once because it would be too heavy), there are puzzles to solve like figuring out the loud room or how to turn yourself back into a human if you become a ghost, an underground maze, and other things that present real problem solving challenges for a person or an LLM trying to play this game. And it seems like LLMs actually suck at this. Below you can see a plot from the paper, Playing With AI: How Do State-Of-The-Art Large Language Models Perform in the 1977 Text-Based Adventure Game Zork?, which shows that modern LLMs barely get any points at all, with an average across all of them sitting around 50 points out of 350. And they take quite a lot of moves to do nothing at all.

What is wild about all this is that LLMs should be able to one shot this game. Zork has been around for decades. It has been posted in full on the internet in every iteration imaginable. Walkthroughs, speed run guides, hint books, and literally every single way to play the game has been discussed at length on the internet. And yet, LLMs, allegedly trained on the whole of the internet, fail completely to play a game which should be in it’s training data. That is a very interesting observation. And so that fateful night of Walter and me talking together at the Recurse Hub led to zork-bench, a Zork-based reasoning evaluation tool for Large Language Models.

zork-bench creates a harness for LLMs to play Zork. Zork is hosted using the same backend as zulip-zork. LLMs are given tools such as the ability to take notes or record a map. An important component of zork-bench is investigating planning. In order to make clever decisions that get points in the game, such as recovering the platinum brick, one needs to organize their inventory, figure out the loud room problem, and transport the platinum brick from the loud room, to the treasure case in the house. Thus choices and reasoning chains for the LLM across all these decisions can give insight into how these decisions are made. The reasoning/thinking text becomes less important for evaluation of the LLM itself, instead we can look at the actions it takes and decide if they are reasonable. If an LLM spends a lot of time walking back and forth between two rooms, that shows a lack of reasoning. But if the LLM puts the sword in the treasure case to lighten its load, goes to the loud room, solves the puzzle, grabs the platinum brick, and returns to the treasure case, this would represent a planned sequence of actions to solve a specific problem (recover the platinum brick).

Another component is what tools are helpful or harmful to LLM problem solving. In the below plot we can see what happens when LLMs are given no tools, the ability to make a map as they travel, and starting with a full map which was a common way for people to play the game when it came out. Below you can see that naavigating rooms, kimi-k2.5 is the only LLM that consistently performs the same no matter what tool they use. With no tools glm does rather well. It seems like when LLMs are able to create a map as they explore, they all do similiarly (middle plot) in exploring rooms. And when they are aware of the entire map there is a wide disparity. I think this is because of the token usage when you have to load in the entire map into the context window.

These above results from zork-bench already demonstrate that across 500 turns most LLMs never get past collecting a couple treasures and discovering less than half the rooms. Meanwhile they spend an enormous amount of tokens doing so. Below you can see that tokens linearly increase as previous turns are included, but what is surprising is how there are large spikes due to “thinking” and decision making. These are going to be explored in a future post.

zork-bench also is able to account for memorization. Sometimes LLMs do rely on their training data to solve puzzles in the game. Such as walking into the loud room and immediately issuing the command “echo”, which makes the loudness no longer effect the player, we can see that the LLM simply knew what to do and did not make any kind of reasoning plan. zork-bench solves this problem by including in the harness an ability to change all of the “facts” of the game. For example, the jewel encrusted egg that you find in the tree becomes a pot of gold found under a mushroom. A future post will explore LLM performance when they cannot rely on memorization.

A core component of evaluating zork-bench itself was doing a human eval. So I made the harness work for humans as well. I don’t recommend playing zork this way in particular, it requires a lot of setup, and that setup can be annoying if you are a windows user. But, if you are willing, I would love for you to play and send me a copy of your play through to add to the human eval data set. Already I had a bunch of Recursers like those mentioned before and others like Canna Wen and Jack Heard.

I organized an hour long event at Recurse Center, promised delicious donuts to everyone who came, and had them setup zork-bench on their laptops and play in human-eval model. The game logs all of their interactions using the same interface as LLMs but gives them a random label. The thing is Humans new to the game seem to do only so well. They spend a lot of turns, play the game, and figure some stuff out, but then after the hour of playing I gave them, they didn’t get further than any LLM. However, theyir memories of the game persist without continuously reducing the size of their context windows. Haha. Do humans have context windows? But the point is that LLMs, having humanity’s entire knowledge of Zork stored in their memory banks, are unable to outperform humans who had not played Zork before (except for Claude Sonnet, which Isha Bhand, creator of fomo.nyc and Zork aficionado, declared as evidence that AGI has been achieved).

So what does this mean? I would say that it tells us that LLMs, at the very least, spend more energy reasoning about tasks that a game like Zork presents than humans do. And it is unclear that LLMs always understand what they are doing. For example, DeepSeek was never able to actually play the game, mostly hallucinating the game itself, and then making tool calls through the harness. The models all seem to start making similar decisions when they are given the ability to draw their own map. Sometimes models seem to get lost and lose the thread. More analysis is necessary!

If you find this work fun and want to support it, I would be happy to accept token contributions through my buy me a coffee link that you can find here:

Please sir can i have some more tokens

I think there is a lot here and we can expand the scope to include other text adventures. I am excited to see what other people think about this.

If you want to help out feel free to checkout the repo and make pull requests, issues, etc.! Together we will figure out what LLMs actually know through the power of Zork!

Subscribe now

John Aiken has a PhD in physics, is editor-in-chief of low impact fruit, mayor of rainy-city.com, a Recurser, and passionate about saving the whales. Zork-bench is a passion project! I would love for you to contribute!

It’s not working out, I heard my PhD adviser say. Maybe if I had spent more time building the software library, but the software library hadn’t been built. Maybe if my focus on research had found some profound result the lack of production on the software library wouldn’t had been such a big deal. It didn’t matter that I had built three different versions, nobody was using it and so it had been a failure. And so far the spatio-temporal variability of the power law distributions I had been calculating were less than profound. They were mostly boring. We both seemed to be at a loss as to how to publish any of this work. My PhD adviser let out a deep sigh and sat quietly thinking of what to say next. It wasn’t working, something needs to change. We both knew that, but to be honest I wasn’t ready for what came next. That was April.

In January of the same year, prior to the April meeting, I attempted suicide. It’s kind of a funny story, I don’t really think about it as being a big deal. I tried hanging myself. But I didn’t really put that much effort into it. I had thought about it and did some kind of trial runs and this one I decided would be the real go for it. I would hang myself with a belt from my loft bed. I thought, what better way to do it than to do it on purpose, knowing that I could just end it all because it was my choice, not because I had set up some contraption that I couldn’t escape. Now it seems so silly. First, I can’t believe I was ever in the mindset that I could leave my dead body for my poor roommates to find. Not that they wouldn’t recover, but still, what a lack of empathy for the people around me. Second, science is not THAT important, and there I was, hanging from my loft bed from a belt around my neck, hoping that my vision was actually getting dark from the sides and it wasn’t just my melodramatic nature, creating a cinematic vision. I’m pretty sure it was just melodrama. Sometimes when you think about memories you think, is this even really what happened? Have I told myself this story so many times I created a fiction? It is a slippery slope to thinking you have no right to feel anything at all.

After hanging there for a while I got bored waiting to die. It’s so ridiculous, I got bored with my own death! The whole plan was that I would have the will power to see myself through, but so far this plan had made it easy to circumvent. So I got myself down and sat wondering what to do next. My neck was rather sore but I knew that if I just sat in the house I might as well die. It was still January and too cold to simply go for a stroll. So while mindlessly scrolling my email on my phone I noticed an email for a meetup.com group for karaoke that met on Saturdays in the city. Thinking, “why not?” I walked out of my flat and went straight to the train. Well, I got a beer on the way. It’s nice to live in a country that lets you walk on the side walk with a beer. I recommend trying it out. I sat on the train with my beer wondering if I would talk to anyone at all at this karaoke thing but at least I was out of the house. Trees turned to houses which turned to town homes and apartments and rivers and bridges as I crossed the city on the train.

A frown crossed my advisers face. Maybe this just wasn’t the topic for me. It wasn’t that I was dumb or incapable, it was more that I seemed unhappy. That I didn’t seem like I even wanted to do the work. That there seemed to always be something that kept me from progressing and maybe that something was under my control. But it seemed like, I came to work every day and opted not to take control. At the time I blamed my adviser for not “advising”. Why didn’t he tell me what to do?! But I think I he did a really good job of trying with me. He introduced me to multiple colleagues who might have projects more closer to my computational interests. He gave me a lot of opportunities to travel to work with colleagues at other institutions to try and find new problems that could engage me. I simply did not want to be engaged. Cats are often times quite playful and curious, but then they have some sort of anxiety, and suddenly your hand is the enemy. I was a cat.

The karaoke bar was across the street from the train station. Basically no effort from my door to this karaoke bar. My kind of travel. No transfers, direct, almost mindless. Easy street some might call it. I quite like karaoke. In the good old days of a different research community, I had quite enjoyed going to the annual conference simply because they had a karaoke night. You could show up to a karaoke bar and perhaps 150-200 other people from the conference would be there. People would sing all night long. There would usually be dancing. And at some point a physics professor from the south west United States would get up and sing the national anthem of any country you could think of. He had memorized them all phonetically as a hobby. It was quite welcoming. I think in retrospect the social hierarchies of that community were quite complex but on karaoke night, it seemed like no matter how people thought about what, everyone could meet at the karaoke bar one night a year and have a good time. Ironically, that was because it was organized by someone who balked at the hierarchy and wanted to bring people together. I saw karaoke as a kind of safe space away from hanging by the neck from my loft bed.

I think you should stop your PhD. The tears inside me were almost overwhelming. My dream was to become some kind of intellectual. A PhD was my ticket. Without it how could I prove my worth? My dreams were about to be snatched away from me. How could he? How could he not see that it is not my fault? I excused myself to go outside and scream. I knew, no matter how much I wanted to scream at my PhD adviser, if I did so, that would be it. I would never escape that. So I went outside. It took a moment to calm down, however, when I composed myself I decided that no matter what, I was going to convince my adviser to give me one more chance. At the very least because I was so afraid that if I lost my job, I would also lose my visa, I wouldn’t be able to pay my rent, and I would probably get deported with no money. That scared the shit out of me and I didn’t want to live it.

I came back in and sat down with resolve. I can do this PhD, I heard myself saying, we just need to make a plan to recover it. We are only at the halfway point, I can recover. I heard myself agreeing to whatever was being said. These plots, those analyses, I would and could do them. Just give me time. In retrospect, I was already planning my escape. I knew I wouldn’t get to the end of the PhD, I didn’t know how to get there, I didn’t even see where there was, but I did know that I could get to my next paycheck. So I worked out some rudimentary plan to recover, to re-evaluate in six months, to increase my performance and stop dragging my feet. I could do this, I promised.

Walking into the karaoke bar I checked my coat. I had actually shown up before the main event. The karaoke bar was both an open style karaoke, where you sang in front of strangers, and an Asian style karaoke, where you rented individual rooms with your friends. The meetup was supposed to take place in the open style karaoke which had a full stage, but no one was there yet. In fact, the stage wasn’t even ready. They asked me to wait in the front for a minute while other people started showing up. There is a moment between when you start a conversation with a person you don’t know, and when you first see a person you want to talk to. You sit, in that feeling of loneliness, of scariness, wondering if you will actually talk to anyone at all tonight. Or maybe you will just sit here by yourself and never do anything. Then, suddenly, you are thrust into a conversation with someone.

In that cacophony of loneliness your eyes meet with another person and you both realize you are there for the same reason. To meet new people, to push out of the static and connect with others. To form a bond, even just for the evening chatting about your lives or nothing in particular. Samantha smiled a big friendly smile filled with straight, white teeth. She was much smaller than me with thick, straight black hair, horn rimmed glasses, and dark eyes. Samantha had grown up in Taiwan and had moved to Europe to do a bachelors degree in Economics. We began talking because we simply were the first two who showed up to the event. She reassured me, don’t worry, she was there to meet another man, but she had a friend, Jennifer, who will come later and I would get along with her. Jennifer was Japanese, studied both mathematics and political science, and therefore more intellectual than Samantha according to Samantha. Samantha flashed me that big smile again and then went off to find her friend. I sat wondering if I would ever meet this Jennifer when suddenly the music started and I felt pushed into the amphitheater. Someone was already on stage singing a rather good version of I Will Always Love You by Whitney Houston. That someone was Jennifer.

Back at work, I did the best I could do, while also finding the way forward. I talked more openly about the difficulties I had been having in my PhD with receptive colleagues. I applied to about a million jobs in the city. I had a few interviews but no offers however I think ultimately those jobs would have been a mismatch for me. At least who I was at the time. I knew that I needed something different. Before this PhD I had worked on how students clicked on videos, interacted in online courses, and generally how online education was evolving over time. Discussing with someone I knew from then, he told me about a PhD in Norway that I should apply for. It would be on a similar topic, predicting when students drop out using machine learning, and I could start over. Begin anew. I applied and was accepted and left for Norway.

It’s amazing who you meet in life and how they change you. After singing, Jennifer found Samantha who introduced her to me. Jennifer was skeptical of me. She was uninterested in listening to me complain and mostly she wanted to discuss other topics, or at least, push me in the direction of self assessment. Jennifer was obsessed with the song Hold On Loosely by 38 Special and spent a lot of energy trying to convince me this could be a new way of life for me. Jennifer wanted me to see my life was my own and I should do something about it. We spent most of the night at karaoke discussing one thing or another and made a plan to go out a week later. We became fast friends. I think I was quite happy to have someone outside of my PhD program to talk to. Jennifer even came with me the day that I told my adviser I was quitting. She waited for hours outside the office for me to get through those difficult conversations just so I wouldn’t have to leave and be alone afterwards. It is quite amazing how important friendship is to survival. I suppose she enjoyed the conversation. And probably the melodrama. We remained friends for years before drifting apart. Lives get busy and friendships are hard to maintain across national borders, even in the information age. And sometimes the people we need we only need for that moment.

Ultimately, it was the best decision to drop out of my first PhD. Later that year I moved to Norway. Upon arriving in Norway I decided to take Jennifer’s advice. I sat down and wrote out all the reasons I may have caused my first PhD to fail. I didn’t seek solutions I described problems. I discussed people instead of ideas. I was afraid. I let my fear drive my decision making. I was defensive towards criticism. I was afraid my lack of knowledge would be found out. I vowed to change. I decided that I wouldn’t let my second PhD go the same way as the first. I would be different this time around. I would not be so afraid. I would focus myself towards my goals and lose the negativity. I also tried to think about the lessons my first PhD adviser tried to give me that I refused to learn at the time. Always search for a win/win situation. Write the thesis of your idea first, then see how it fits with your result, i.e., challenge your ideas with your data. Be patient and kind with your words. Be understanding even when the other person is not.

It worked. My second PhD created years of positive memories with others. I had a blast with a group of people in a place I never knew was possible. I got to work across disciplines in a manner that was way more suited to who I am as a person and a scientist. And I did a lot of science along the way. Of course not every moment was perfect but I felt like I had found a moment in life where I could be me and discover who me was. Those moments can be fleeting, don’t miss them. Years later, my first PhD adviser found me at a conference and told me that he was proud of who I had become. That he always knew I had it in me, but maybe it just wasn’t the right time the first time around. This story doesn’t have a real ending, it is just life. But maybe it is an interesting story to someone.

Subscribe now

John Aiken has a PhD in physics, is editor-in-chief of low impact fruit, mayor of rainy-city.com, a Recurser, and passionate about saving the whales. The story told here is autobiographical, incomplete, and happened almost ten years ago. Currently John lives in The Netherlands, happily hacking away at various problem spaces usually related to computers.

There is a terrible curse in technology where everyone thinks you are supposed to have “side projects”. Side projects are something you do when you are not at work that in some way appears as work. I think Gen Z people call them “side hustles” because there is also the insipid idea that these side projects should make you money in some way. For some reason the modern era has decided to commoditize all parts of the waking life. I think the most striking description of all of this is Kyla Scanlon’s description of the Attention Economy:

If you think about it from this perspective, all side projects/hustles should be designed in some way to grab the attention of someone and leverage that into making you money in some way. That is, if you buy into the idea of side projects and side hustles should be commoditized instead of simply having hobbies that you enjoy doing (such as how I enjoy writing this substack). Few people think that someone should somehow monetize the rebuilding of a bicycle, or, just going for a ride on the weekends. However, for some reason if you write software as a hobby you should monetize it since that software is meant to be used by others and whenever other people are involved the Attention economy comes back into play, but I digress, Low Impact Fruit is supposed to be about science and technology, not the failures of the modern world and I promised in the title to discuss a side project that I have been picking at, thinking about, or otherwise working on for some time.

Rain Simulators

A very long time ago I stumbled upon the website rainymood.com. Essentially, some people like to listen to rain sounds. This isn’t a new phenomena you can find old CDs of various sort of environmental sounds. They are all in some kind of genre of relaxing meditation music, or perhaps something to help you concentrate. I often times like listening to rain sounds when I otherwise could be listening to music or silence as I find that rain sounds focus my mind on whatever problem I am trying to solve. I do agree with the CD above when it says:

Just close your eyes as you listen to nature’s own relaxing sounds on this CD and you can find it easy to imagine that it’s pouring with rain outside whilst you are all cosy and relaxing indoors.

I find rain sounds quite relaxing. I should probably spend more time listening to the rain. (Apparently Feel Good Dynamics still exists and makes a variety of relaxation-based media that can still be purchased as an MP3 or a Compact Discs).

Meanwhile, the world moves forward and I think a combination of GenX and Millennial nostalgia for the 1980s that started with Donnie Darko overtook me and everyone else born between 1975 and 2000. A big part of that nostalgia not only looks back at the halcyon days of Ronald Reagan (or whatever) but also a look back at how people then looked towards the future with books like Neuromancer and movies like Bladerunner. A central component of these works is the sprawling city that has grown and metastasized since the 1980s into some sort of Neo-Tokyo. I think my favorite iteration of this was the Boston-Atlanta Metropolitan Area from the movie Escape from New York. Thus everyone’s favorite sci-fi obsession, cyberpunk, took over every aesthetic in some way whether its neon lights, electronic music, or simply working on computers in some boutique Neo-Tokyo ramen shop on a rainy afternoon.

Coupled with my childhood dream of working in a coffee shop in Seattle (I don’t know where this came from) and also my childhood love of SimCity 2000, I came up with this idea of having a website based interactive rain simulator that would be a city scape in an isometric frame that has an active weather system that can be controlled by the user and that active weather system would be used to generate the rain sounds in a way that would be far more sophisticated than rainymood.com play button. It would give the user the power over the weather with a few sliders and other controls and you could watch various weather systems interact with the city below. It would essentially be a browser-based, slightly interactive screensaver for the modern computer user with a discerning taste for rainy sounds.

Subscribe now

I decided that it wasn’t enough to just have a slider and such, I should create a full Navier-Stokes simulation of the clouds interacting around the buildings of the city, the initial conditions could be initialized from random numbers generated by the actual atmosphere, and the rain sounds would then be driven by the distribution of densities as created by the simulation for each time step. This all seems rather ambitious now that I think about it. I think this ambition came from the last time I put a lot of effort into this project was when I was starting my PhD and one of my advisers was Anders Malthe-Sørenssen and the other was Morten Hjorth-Jensen (both of these links are in Norwegian), two physicists known for their computational efforts. For some reason I decided that having a side project about building rainy sounds simulators would impress them. I doubt Morten or Anders ever thought about this or even realized I was working on something like this. As an aside, this is something to think about the next time you wish to impress someone, they very may well not know you are trying to impress them.

So anyways this effort all got so far as building the simulation you can see above which creates a randomized building density and then randomly initializes the simulation and lets it run for however many time steps you want. That’s great but its not exactly visually pretty and you don’t hear any rain sounds because generating fake sounds that sound real is actually rather difficult if you haven’t really figured out how you are going to do that yet. And herein lies the dilemma, everything has become so complicated I start to feel a bit overwhelmed by it all. There is the weather model, the city animations, the controls, the sound generator, and you have to put it altogether in a compelling way that I would like to use it. Where do I start?!

Relax and listen to Will Wright talk about making models.

Enter the MVP

A skill I wish was pushed onto me when I started learning to be a scientist/whatever it is I define myself as at the moment, is developing an MVP. That means, Minimum Viable Product. Don’t explore every option and offer ever feature and understand every part. Instead, build the minimum that is expected and then test it with users to see if anyone ever will use it anyways. The same can be said for scientific ideas. I kind of think that I have often spent way too much time on developing ideas on some of the scientific projects I have worked on. And this was in lieu of simply trying out the minimum idea and then sending it off to reviewers to see if anyone is interested. But getting back to rainy-city.com, the MVP should have the following:

1. A webpage

2. A slider to control the rain noise

3. A slider to control the city noise

4. A visualization of the city

5. A visualization of the rain and clouds

That’s it. By defining the MVP, I am able to put everything aside and get something up and running, and then I can start adding features to it like driving some weather simulation with atmospheric noise. So that is what I did.

You can checkout rainy-city.com at rainy-city.com. There are no subscription fees or anything, just go and listen to the rainy city sounds and relax. Perhaps if you are feeling especially cyberpunky, you can have a bowl of ramen and think about how living in some sort of mega-corp dystopia is not actually the best idea in the world.

Subscribe now

This project was the project I proposed to do as my Recurse Center project. I ended up working on many other projects in addition to this one. That includes a semantic search tool for research, an arcade side scrolling fighter called Streets of Rainy-city.com for the RCade, random weird stuff for RCTV, a zulip bot that lets you play Zork inside of zulip, and I am working on an LLM reasoning eval toolbox based on zork. I am particularly proud of Streets of Rainy-city.com, I had a week to make it because there was a localhost event at Recurse to show the RCade and so I built the game in a week.

When I proposed rainy-city.com I was honestly scared, I thought, who would ever want this, why would anyone think this is a cool idea. But now, I realize that I wanted this. And that was enough. What will you build that you want?

I built this project while at the Recurse Center. Join the Recurse Center!

Last December, I was writing about a “futuristic scenario” where I could simply chat on my phone and get a project initiated, code written, run and debugged, results produced, and figures delivered to me, all without being physically at my working machine. For example, to build a python-based software that can organize and synthesize my news pieces, and overtime, build a database that can be mined and visualized. Just over a month later, that futuristic imagination is already a reality. Besides sending me the morning news, Minato also saves the news to a dated file and archives them. I can direct it to write an analytic script right now but I will wait for the end of the year. It feels exciting. But if I let my imagination run a little further, it also terrifies me.

Subscribe now

Code agents, specialized large language models designed for coding, are progressing rapidly and stampeding everything in their path. If you watch financial markets, after the joint release of Claude Code Opus 4.6 and Codex 5.3, the IT and enterprise sector fell hard. I think the fear is at least half true. But on the other hand, the job market for software devs goes up, as shown in the Twitter charts by David Sacks. Similarly, vibe coding startups are growing at lightning speed. See the report on Lovable, an Europe startup, At last, reasons to be cheerful about European tech by the Economist. People fear that code agents will shed jobs, but for the right person, they turn a software engineer into a productive team.

After reading my morning briefing, I sit down at my working machine and check what my multiple running Codex (OpenAI’s code agent) and Claude Code (Anthropic’s code agent) instances have written in the terminals overnight. I review them, evaluate their progress, then set them on to the next tasks, ranging from rolling out my newly trained graph neural network models based on a shell script template, to using git to version-control a successfully developed feature and releasing a new version with comprehensive documentation, to launching another batch of simulations sampling 100 parameter combinations across 20 available CPUs, to simply saying: They did a good job.

Then I open the Codex App, pull up my conversation, and ask it to show me today's to-do list from a living WORK.md document, a file that tracks my todos, things done, ideas, and project progress in real time. Whenever something gets done, a fresh idea sparks, or any progress is made, I just type it into the conversation, and Codex follows my rules to merge, track, and document everything in WORK.md. I imagine this will make my life considerably easier when annual report season comes around. I certainly can't afford a personal assistant. But Minato and Codex are the closest thing I've got.

This isn’t strictly coding anymore. And the companies building code agents have noticed that people increasingly use them not just to write code, but to handle tedious workflows and personal organization. I recommend the recent episode of Lenny’s Podcast featuring Boris Cherny, creator and head of Claude Code at Anthropic, titled “What Happens After Coding Is Solved” (February 19, 2026).

After I get a review of my todos, I tell Codex App my two or three priorities for the day. I make my coffee, and eat breakfast. The day begins, and it feels wonderful.

10AM. For the old me, the day would have just started. Now, a lot has already been done. Sometimes the pace of progress feels so surreal that it’s hard to believe Claude Code, the starter of all this amazing and terrifying momentum, was launched barely a year ago. A few years ago, if you asked a chatbot to do basic arithmetic, it couldn’t get it right. People joked about it. Fair entertainment. But if you’re still making that joke now, you’re the punchline. What really matters is the rate of iterations happening in the development and deployment of these new capabilities. This is no longer a six-month or yearly cycle. It moves by the month, sometimes by the week.

In the interview, Boris mentions that 4% of all public GitHub commits now come from Claude Code. Claude Code has become a prolific author of public code. Boris says that projected figure could reach 20% within a few years. And if you count private repositories, the share would be even higher. I believe it completely, because I’ve personally built multiple pieces of software for my private use with Claude Code, and that’s a lot of lines never counted in any public statistic.

Howard Marks, co-chairman of Oaktree Capital Management, published a memo on February 26, 2026 titled “AI Hurtles Ahead.” I’ve long admired Howard as a thinker and writer navigating this complex, fast-moving world. He probably doesn’t code much, but what struck me most was his discussion of the economic impact of code agents, drawn after just a handful of uses. People debate whether machines are truly thinking, truly intelligent. Interesting questions, perhaps, but maybe too academic. The real economic question is this: “If a code agent can deliver a report equivalent to what a $200,000 analyst produces, does whether the machine “thinks” still matter?” But also, if the code agent can deliver the reports of a highly paid analyst, how important were those reports that they can now be automated away? Perhaps those reports themselves were the friction point that AI is solving.

Subscribe now

I’ve been using code agents to develop prompts that query business performance and produce equity analytic reports. What I can say is that the breadth of search, the range of sources, is something I simply cannot compete with alone. The report quality is okay, not excellent yet, but solid enough, with sufficient factual data and tables to use wisely. If you have followed a business for years, you can tell its quality. For example, this report on Estee Lauder. And crucially, you can run as many such deep research tasks as you have tokens for. Whoever controls the tokens controls the information.

That brings me to the second striking insight from Howard’s memo. With information barriers lowered, if not demolished, by AI tools, where do we create value? In financial markets, short-term gains will be harder to come by, but long-term opportunities will remain plentiful, because most people still chase short-term results. As Charlie Munger, the late Berkshire Hathaway vice chairman liked to quote, “Always take the high road, it is far less crowded.” AI will smash you in competing in the short-term, zero-sum game requiring the scope and speed of information, but may not yet be able to truly “vision” far into the future.

For academic work, we face a similar reckoning.

The deep research function is now available in Claude Code (it also has an extended thinking toggle) and Gemini Pro. I’ve been experimenting with using it to generate reference lists on specific research topics. It consumes many tokens, takes ten minutes or more, but the product is genuinely impressive, broad coverage of a typical ~500 sources and more, useful summaries, good detail. You can even ask it to verify DOIs before writing the final report. It cannot replace years of deep research experience and insights, but it will quickly help you fill the knowledge gap of any new topics that may interest you. I typically will pick a paper, append the abstract with “please do a deep research on this topic; give me a peer-reviewed reference list; verify DOIs; I only have time to read 10 papers, ranked your reference list by impact; return to me a .md report; take you time.”

The information barrier is gone. So where, as researchers, do we bring in value? Judgment? I’m not sure. But what I notice is that when I now face an unfamiliar topic, I have no mental barrier to entry. The learning process has accelerated dramatically.

AI is excellent at filling knowledge gaps. But genuine insight still takes time and hard work. Maybe the answer resembles the investment question above: focus on the long term, something fundamental, and probably more important something you are interested in. As I see it, speed doesn’t take away insight. They don’t conflict with each other. Effective iterations bring insights and fast turnarounds many actually bring insights you expect early.

To close, I return to the epigraph of Robert M. Pirsig’s Zen and the Art of Motorcycle Maintenance, “And what is good, Phaedrus, And what is not good, Need we ask anyone to tell us these things?”

Nurture your judgment. Cultivate your aesthetic taste in your work. That is what AI has not yet taken over, and what I hope and imagine it will not, at least not anytime soon.

Bio for the Author

Dunyu Liu is a senior computational geoscientist at the Institute for Geophysics at the University of Texas at Austin. More on his research interests can be found here.

we should focus on what makes us uniquely valuable as embodied, social, and organic creatures who are compelled to be with, empathise with, learn from, and think with one another. Our ability to navigate human networks and our role within them is our superpower.

I was talking with someone last night who studies computer science education about what AI literacy is. Neither of us seem to know either. But over the course of the discussion we arrived at a definition that could probably remove the AI part and simply be associated with some part of problem solving. Below I will describe what I will define as “AI literacy”.

AI Literacy according to john

I have tried to embrace lots of AI tools as they come out and integrate them into my research. This includes both using tools like AI coding agents but also developing and training neural networks for a variety of tasks generally related to one of my many current focuses (foci?) which is rock-fluid-climate interaction. Feel free to read more about that here. Across all this effort I have realized that the level of abstraction is much more important than before. What do I mean by “level of abstraction”?

Subscribe now

Sometimes I want to build a data dashboard. For example I had some plan to work with a colleague on a large set of time series data. I thought it silly to mess around in a Jupyter notebook because it’s somewhat unwieldy when you want to look at ten different time series at once. So I texted my ai agent on telegram what I wanted and pointed them at the repo with the data and they did exactly what I wanted without further interaction with me. That was fantastic. It would have taken me a week to make that dashboard.

Other times I want to train a neural network on some data to build a classifier, do some kind of prediction, or whatever. I need to know a lot of specifics about the architecture and how that architecture changes as I explore what fits the data and solves the problem the best. If I asked my ai agent to do this nothing will work. And I know this because I tried. It was me literally experiencing this meme:

I had to spend an afternoon undoing the mess created by me letting an AI agent loose on my codebase.

And the difference between those two experiences exactly highlights what “AI literacy” could be described as. There is a spectrum of context that you should consider when using an AI tool. You should be constantly assessing if you are operating at the correct abstraction layer to solve the problem at hand. And that abstraction layer acts as a sort of tuning knob or hyper parameter of how involved you are in what the computer is doing. AI literacy is understanding how much you need to understand about what the computer is doing in order to assess whether it did what you wanted or not.

Steve Jobs and Apple liked to say when they built the personal computer they built a bicycle for your mind. You can watch him say it in the video above. I really buy into this concept and think it’s not just marketing hype. Computers are tools that can help us conceptualize problems beyond what we can do with only ourselves. This is all AI tools are doing. They abstract away parts of the problem space so we can focus on understanding the important part. Understanding what needs to be abstracted at the current problem solving step is AI literacy.

Subscribe now

I was listening to this playlist I created from some old 99x recordings of just random broadcasts that I guess someone recorded live onto a cassette tape. 99x was an Atlanta based FM station that played rock and alternative music in the 90s. You can listen to this playlist here.

As a child I loved listening to the radio. The radio wasn’t just a a source of new music, radio DJs were essentially a one to many one way communication device that only the select few were allowed to do. You could broadcast whatever you wanted but there were rules from the FCC. I was always fascinated by the idea of being a radio DJ both picking what music goes on the radio and also all of the technology of radios. But really I loved the vibe. It seemed cool to be on the radio and I wanted in. I suppose I never got to be on the radio and I guess I’m glad because it seems like the job is far less exciting than it used to on account that everyone started streaming. I have noticed that French people all seem to listen to the radio still. But everyone else is streaming.

And that’s the point I guess. Computers took what used to be a craft, screaming into the void, and gave everyone the ability to do it. Just last night I watched a woman do crafts from Hobby Lobby. Just like chatting with the people in her chatroom while doing the crafts on video like painting picture frames with glitter. It was quite pleasant. Her name is Liah and you can watch her here if you want.

None of this is bad in any way. I think it’s good that we can create micro communities about anything that connect through live streaming or chat servers or whatever. Thinking about radio and live streaming got me thinking about how scientists communicate about things.

It’s an old conversation but scientists who post on social media get more traction in their work. And recently on LinkedIn I was wondering along with others why do we publish papers anymore the way we do? If we are to publish papers perhaps they should go on ArXiv and submitted to journals when you can be bothered to do so. But really we should think about how could all this work have more impact.

Subscribe

Much of science costs a considerable amount of taxpayer money. In his super excellent 2025 Neurips talk, KyungHyun Cho advocated for designing research agendas that maximize impact. This was a small part of what was one of the best talks of Neurips 2025 and I recommend watching it. He said as scientists we have an obligation to create impact as we have been trusted as stewards of science and community resources by taxpayers. In my opinion this means we should also examine how we communicate science.

Getting started I think everyone should be putting all their papers on ArXiv. Open access is more important than peer review as open access often allows community review like in the case of the potential high temperature superconductor a couple years ago. Second I think we should have more micro publications like substacks and other online media that allow for instant reach to readers. Substacks and blogs allow for a larger space of ideas to be discussed and at a higher velocity as ideas change more quickly these days.

I think we should stop submitting so much to journals and organize more conference proceedings as the AI fields have done. There are a lot of problems with conference proceedings but they do give a regular output that allows for peer review and also a central communication portal (the conference itself). This portal used to be the journal periodical. But now People simply don’t sip their coffee in the morning flipping through the Nature or Physical Review they just received in the mail. We must change or become obsolete.

And save the whales.

Subscribe

I am getting rather tired of reading blogs about how AI is going to change everything.

I can appreciate that sentiment.

Anyways, this substack is going to be about how embeddings are not merely computational tools but represent a new methodology for examining complex systems that may reveal structures beyond our current theoretical frameworks.

Subscribe now

A year and a half ago I organized a conference called Exploring System Dynamics in the Natural World with AI along with Thorsten Becker (thanks Thorsten!). This conference was motivated by me reading various papers across the topics of chaos, geophysics, information theory, and AI. The idea behind the conference was two fold. First, most AI for science discourse focuses on engineering improvements (faster weather models, better protein folding) rather than fundamental questions about what these models reveal about nature itself. These are laudable goals and should be pursued. However, I wanted to better understand why neural networks were becoming so good at understanding complex physical systems which I talked about in my intro talk that you can watch here:

Second, I set the goal of considering this idea broadly and asked people who thought about it from the theoretical development of methods, people who thought about it from a physicist point of view, but also industry and education. You can watch most of the lectures from the conference here. Together it seemed that we could construct some explanations as to why neural networks can predict chaos better, make better weather predictions, and do all sorts of other things. But today, a year and half later, I have even more questions. But I think I will need to start from the beginning of the talk above until now to explain everything.

To put it simply, everything started because this paper by William Gilpin fascinated me. In it he demonstrates that neural network-based predictors are able to capture chaotic dynamics further into the future than theoretically thought possible. From the abstract:

We find that large-scale, domain-agnostic forecasting methods consistently produce predictions that remain accurate up to two dozen Lyapunov times, thereby accessing a long-horizon forecasting regime well beyond classical methods. We find that, in this regime, accuracy decorrelates with classical invariant measures of predictability like the Lyapunov exponent.

This is profound. It tells us that our understanding of what is predictable and what is unpredictable may not be as well rounded as we thought. And so I decided to dive deeper (I feel like this is a phrase I learned chatting with LLMs). What is it that neural networks are doing when they are modeling these systems? Why might they be better at these kinds of predictions in comparison to traditional methods and the theory that dictates them? I think one way to think about it is to consider what happens when you put data into an autoencoder.

Variational Autoencoders

There is plenty written about variational autoencoders that doesn’t need to be reiterated here. But just in case, I will give some background. The main point of a variational autoencoder is to learn the parameters mu and sigma that describe the distribution of the latent space z.

It learns these by encoding the input data x, by passing the data through one or more neural network layers (these can be linear, convolutional, whatever) until this is “compressed” into the latent space. This compression can happen both by reducing the dimensionality or by expanding it (sometimes it may be better to train a large model than a small one). This latent space is then passed to the decoder, which mirrors the encoder, and produces xhat (right in diagram above), which is the reconstruction of the training data x (left in diagram above). If xhat approximates x, (e.g., it recreates the training images well, you have R2 scores greater than 0.9, etc.), then you have, at least a first pass of a model that can construct a latent space that may capture all of the interactions of the system of study. This latent space, which is also called an “embedding”, can then be used as features for new models such as being clustered to try and find some structure that could be informative for a task or used in some kind of supervised learning (e.g., separating cat and dog pictures). Because these latent features are described by probability distributions, you can sample these distributions and generate new data from them. The important component here is to understand the concept that neural networks construct a latent embedding space that attempts to capture all of the interactions that could be used to describe the system you are studying whether that is images of cats, language, or a chaotic time series. The architecture can then become arbitrarily complex.

Subscribe now

These latent spaces can then be used to do all sorts of things and here is a non-exhaustive list of abilities:

1. show the motion of atmospheric components taken from this linkedin post where they trained neural network based weather simulators:

2. Or you could use it to understand what kind of topics do AI labs publish on like I did in this substack I wrote a while ago:

3. Or you could use it to map everything like google’s AlphaEarth Foundation model which compresses petabytes of satellite data (optical imagery, radar, climate models, topography, gravity measurements) into 64-dimensional embeddings for every 10-meter square of Earth’s terrestrial surface.

The AlphaEarth model doesn’t “know” about geology, hydrology, or agriculture explicitly. Yet researchers are using these embeddings to predict crop yields, track deforestation, and map wetland vegetation with higher accuracy than purpose-built remote sensing models.

To put it a different way, neural networks are able to understand complex geometries that define how to separate latent non-dimensional spaces that can motivate decisions that humans cannot conceive quantitatively (e.g., what exactly is the boundary of a topic) where decision is defined as determining the outcome based on the input.

So far this seems reasonable. You could even call this principal component analysis on steroids. but where it gets really wild is when you introduce the “Platonic Representation Hypothesis”. The Platonic Representation Hypothesis is from an 2024 ICML paper where they claim that:

Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.

And this isn’t just claimed in this position paper, we see evidence of this from different comparisons as well. For example, in this paper, the authors argue that they can train models that can translate between two incomparable language models by using the embeddings from one model to predict the other.

This has a profound implication. First translating from an unknown vector space into a known vector space means you can probably reconstruct at least some of the original training data which means you could reveal the secrets an unknown model knows by examining it’s embeddings as long you have a known model to compare it to. But this also means that this latent representation is shared between the two models even when the original model is unknown.

To provide an analogy to something from the history of science, embeddings might be to 21st-century science what spectroscopy was to 19th-century science. A technique that reveals structure invisible to direct observation, enabling discoveries that reshape entire fields.

When Fraunhofer observed absorption lines in the solar spectrum (above you can see first an example of the sun’s absorption and second an example of the sodium absorption spectra), he had no theory to explain them. But even without a physical explanation it was something obvious to study and eventually was a fundamental observation that led to quantum mechanics. Similarly, embeddings reveal patterns in high-dimensional data that we lack theoretical frameworks to fully explain, but the patterns are demonstrably real (they generalize, predict, transfer across domains). And so the question arises why do these vector spaces connect? Or to quote from the platonic representation paper itself:

if there is indeed a platonic representation, then finding it, and fully characterizing it, is a research program worth pursuing.

To put it a different way, if these embedding vectors can be translated between themselves, and thus are converging to a shared state of reality, then what we have here is not some sort of “black box” but actually a tool that allows us to examine a representation of how information in the universe is organized. And we can likely find out new things about the physical world around us. A critical question in this direction that needs to be answered is:

I think a major criticism AI for science, that so far has been somewhat valid, is that it has only ever reconstructed things we already know. It is great that we can have faster neural network based solutions for different physics, but what will it discover? (okay, new solutions to protein folding is a good one). The platonic hypothesis suggests that yes, it will discover something and that discovery may lead to new understanding about how the universe works.

Enter moltbot (formerly clawdbot) which is absolutely amazing. You set it up locally and it’s basically an AI assistant that you can give access to whatever you want. A lot of people are buying Mac mini m4s to setup their own as they are compact and powerful. And let me tell you, this is powerful.

I installed it on a lambda labs workstation I have which has a 2x RTX 6000s, 64 cpu AMD, and 256GB RAM. Okay that’s probably overkill but I haven’t really explained it all yet if you don’t know what moltbot aka clawdbot is.

It’s a persistent AI that can do anything. Want it to run codes for you? It will do that. SSH into a computer and make some changes. Sure! Want it to research some topic for you. Why not. It’s everything every other LLM product is. But it’s persistent and maintains a memory and personality through all interactions. And you can connect it to anything so it knows about you. Calendar? Yes. Computer resources? Yes. Want it to watch some stock for you? There’s an api to connect to. Want it to have an opinion about your music? Connect it to Spotify. And you can do it all through telegram or whatever chat app you like. You talk to it like anyone else you text on your texting app and it replies and tells you about what it did and what to do. And it remembers whatever you tell it because it writes it down.

And the setup is super simple. Basically it’s one curl and you are setup in a very easy to use installer. There are some tutorials on how to set them up on VMs all over the internet. This all sits behind a VPN for me except the telegram bot of course.

But what this means is that if you have some random idea on the bus your AI agent can get started immediately. If you are sitting in a lecture and have a thought your AI can try it out. Because scientific codes are not the same as developing software in my opinion. There is lots of boilerplate pieces that get sewn together until complexity reaches the point a software library is needed. Thus most scientific codes are one off things which AI coding agents are perfect for.

Anyways I thought Claude code changed life but this is next level. Everyone should convert. If you want moltbot get started here.

buy me a coffee

Sent from my iPhone.

EDIT: now it’s called OpenClaw

There are already plenty of books out there on technical management so why write any on science management. This is kind of true, but science often has a hierarchy and a nebulous goal structure that doesn’t align with the goals of engineering teams. For example, PhD students are an integral part to most academic science teams, they are also junior members, with an end date to their work (this is also true for postdocs). Unlike junior software developers or engineers which you might imagine you are training to become seniors in your company with a deep technical insight across the technology stack that comes from experience with that stack, PhD students and postdocs ALWAYS go away. Thus, scientific managers (i.e., professors and senior research scientists) are constantly juggling project knowledge needs. There are a few ways of doing this but one is to simply move on to a new topic when the group of PhD students who knows about it leaves. This can lead to a lab (see the physics lab below) which may be full of useful equipment or it may be full of equipment nobody knows how it works.

Another option for knowledge management is to try and hold on to PhD students for as long as possible. A typical example looks like this. A PhD student who clearly understands how a piece of research technology work graduates. They are offered a postdoc within the lab and are encouraged to stay. The postdoc funding runs out. The PI manages to get a teaching position created within the department or perhaps they are super lucky an internally funded research position that may allow for some permanency. The former PhD student teaches and advises on the research. This continues until the former PhD student leaves and goes some place else. Neither solution is wrong per se, but neither seems like a thoughtful research management choice or strategy. Instead it is a reaction to the creation of a knowledge gap because the knowledgeable researcher in your lab leaves.

There are predictable points when PhDs and postdocs transition out of the lab. Depending on the country you are in, PhDs have a finite contract (in the USA this is not true) and are essentially working for free if their PhD dissertation is not submitted at the end of the contract. Thus you can expect the PhD student will work with you for approximately the length of the PhD contract. In the USA, students are offered teaching stipends and can, within reason, work indefinitely on their PhDs. Postdocs have a finite contract as well. It doesn’t matter their performance, if they publish 100 papers during their postdoc or 0 papers, at the end of the contract, the money runs out and they must either find new funding or a new job.

Because there are predictable points when knowledge will transfer out of your research group, we could imagine a strategy to keep knowledge within the group. One strategy, which is also reactionary in my opinion, is the expectation that when PhD/postdoc leaves, collaboration continues. This is a reasonable expectation if they leave for another academic position, academic positions typically come with a lot of freedom and thus they will likely have some time in their new job to assist the transition in their old job. Another would be a technological solution, if PhDs and postdocs document everything they do, e.g., by using a wiki, then knowledge is in some ways passed forward. In my experience this doesn’t work. Across all of the research groups that I have worked in across different cultures, wikis have never been kept up to date, and they don’t really capture the scientific process anyways. That is perhaps because science isn’t building solutions to problems, it’s exploring problem spaces looking for new ones. There is a lot of dead ends there.

Subscribe now

Another strategy, that I would like to attempt but have not yet, is that everyone in the group is expected to contribute to a core software library. This will create a sort of monolithic project that group members add to as they come and go. This is how a lot of statistical packages have been born, perhaps not explicitly as knowledge management tools, but as research outputs. The risk here is that your package becomes all encompassing as you add more and more features to it and many years from now you realize your entire work life has been consumed by managing this software package. There isn’t anything wrong with that, but it should be part of an overall research strategy and not something you fall into and then hate.

Of course I am also leaving out one critical piece of scientific knowledge storage, the scientific paper. There is a, now retired, scientist at UT-Austin Institute for Geophysics, who when asked periodically about details of papers he would reference during colloquium Q&A sessions, would say, “You should read my paper because I wrote this down so I wouldn’t have to remember.” And that is a critical component of how knowledge transfer and storage happens in science. It is written down and stored in journal articles. The modern research paper serves lots of functions, but I think a critical one can be storing how things work inside of a research lab. To that end we should try and best capture our computational work so that it can be repeated. I have written about this before on how to avoid common errors in adding codes to scientific publications.

I think this sits between some sort of monolithic software library and doing nothing. But I also think the scientific paper isn’t a management strategy for knowledge transfer, it is simply a deliverable of scientific endeavors. You can expect new members of your lab to read old papers, but how that knowledge is carried forward and leveraged is still an open question for me.

I think every scientist as they mature develops their own personal views on how science should be managed. It is probably quite useful to organize those views from lots of scientists and put them into a publication somewhere. Perhaps there already is a field that studies this. If not, maybe one should start.

buy me a coffee

I want to say thanks to the group at Asterisk Labs for pushing me to write this. I was initially skeptical since I don’t have decades of experience of scientific management. They said, yeah but you have thoughts and maybe those thoughts can shape the future even if they are only now theoretical.

Subscribe now

Recently I was chatting with Varya Bazilova, who studies mountain hazards, about all this and she said that the solution is perhaps to “be like a debris flow”. Unlike a flood which is mostly water and goes everywhere, debris flows have large pieces of debris and typically cut a narrow channel as they march down the mountain in a gravity fuelled quest for oblivion. I think she’s correct. If your anxiety focuses you on everything you will focus on nothing. So be like a debris flow.

Subscribe

Subscribe now

For the last year and half at Utrecht University and Njord I have worked on a Research Council of Norway (Forskningsrådet) funded project that I named “SerpRateAI”. This project is just a drop in the overall Njord budget (only 4.2 million NOK out of 400 million NOK!) and it’s goal is to build a systems level understanding of the subsurface processes that can be used to understand how carbon can be stored underground. A core part of this project was working with people at Utrecht University in The Netherlands and Woods Hole Oceanographic Institution in the USA to better understand some of the geological processes that are captured in the data. I have always felt like this project took the ethos I gained from working next to Njord and inside of Njord of interdisciplinary science and computing and applied it to the data from the Oman Drilling Project. I have discussed this project before here:

where I talk about how I tried to identify everything in a 9 month long hydrophone data set without ever trying to identify a direction. Haha, who would have thought that wouldn’t work! But I wanted to give an overall introduction to the idea of the project, some of it’s conclusions, and a hint at where I am going to take it next.

​To give you a short introduction, it is quite likely that we can store carbon inside of certain kinds of rocks called peridotites. This is because when they are exposed to carbon bearing fluids (think coca cola🥤🥤🥤), the rocks will absorb the carbon from the fluids and turn into different kinds of rocks. This means that the carbon can stay in the rocks forever and never again plague humanity from the atmosphere. This process is called serpentinization and is also where the project name “SerpRateAI” came from. When these rocks absorb carbon and change they grow larger. What happens when something grows larger in a confined space and has no where to go? It probably starts breaking things everywhere. And this is the hope actually, that if there is enough fluids the rocks will continuously absorb carbon, grow in volume, and break and move the fluids around underground. But nobody has ever observed that in the wild before. I hope that I am able to build an AI-enabled model that can help capture and quantify these changes.

To that end the Oman Drilling Project placed a number of sensors deployed inside of rocks in Oman that can be used to study this kind of phenomena. Part of this work includes sensors deployed inside of boreholes (see above) but not all sensors are there. These sensors measure acoustic energy, water levels, temperature changes, and also we have climate data like precipitation, and soil moisture. These cover a 9 month period. You can see a diagram of this above. Other kinds of data include x-rays and photographs of rocks taken from the boreholes, physical measurements like rock resistivity, and even written comments by geologists about their observations.

The first thing to see is that we can observe gas moving from inside the walls of the borehole into the borehole fluids. This is from this paper I wrote a couple years ago:

​​The main take away from this plot is that there are bubbles that come out of the rocks at intermediate intervals (that’s the red line, the blue and yellow tell you about how much acoustic energy is coming out at different frequencies at different times). You can find the paper here if you want to know more about bubbles. These bubbles could be all sorts of gas, some people think its hydrogen but other people don’t. If it is hydrogen it would be great evidence that this reaction driven fracturing process is ongoing in the subsurface because hydrogen is one of the products of the serpentinization process I mentioned before. But we don’t have any measurements on the type of gas (can’t count protons using acoustic energy) so different people have different opinions about what the gas actually is. Disagreements are common in science and building consensus from data and discussion is part of the fun of science in my opinion.

Subscribe now

The second thing to see is that we see is fracturing ongoing in the borehole several years after the borehole was drilled/cored. That is, we hear the rocks breaking. It is common that soon after a borehole is created it might settle a little. The stresses of all the surrounding rocks need to reorient around the new hole that is inside of them. But these should all reorient rather quickly and so years later the borehole geometry should be more or less static without some kind of external forcing (like a seismic fault nearby moving). So observing fractures means that something interesting is probably going on in the borehole and that is exactly what you see in the plot below.

​The first thing to note is that these fractures (bottom plot, red line) occur during the same time period as the bubbles in the plot above. The second thing to note is that we noticed (middle plot, dark blue line) that right before the fractures started, the water levels in the borehole increased a lot. Why did the water levels in the borehole increase a lot? Well, (top plot, light blue line) it rained! What we see is that actually there was raining during these time periods and there is then fracturing that occurs underground. This is because the pore pressure increases (the water level) and this is one of the primary causes of underground fracturing. If you want to read more about this effort you can find this paper here. We also suggest that there could be some stress due to the volumetric expansion of the rock because the rock was changing/growing because of fluids. Perhaps there is, perhaps there isn’t. Everything is correlated.

After doing all this kind of “Earth system analysis” I wanted to take a detour because the Oman Drilling Project produced a lot of other data. Fracture networks are obviously important to understanding this serpentinization process since the fracture network is the byproduct of these fluids moving around underground. So I thought that maybe there is some kind of “optimum” level of fracture network complexity that would be predictive of where active rock changing was occurring. That is, if there is a very dense fracture network, then the rocks had already changed, and if there was a very little fracture network than the fluids wouldn’t really be able to access the rocks to change them. So there must be some middle ground where it happens. So I built this big data processing pipeline to try and untangle all this.

The first thing to say is that my idea didn’t really work. We didn’t see that the fracture network density was predictive of anything. In fact, it was one of the least predictive features in the whole data set! This was rather frustrating to be honest because I thought this idea was quite clever. But oh well. Clever ideas are great but science isn’t about clever ideas it’s about making models that work.

So I played around with the data and eventually came to some conclusions and this paper got published recently and you should go and read it. This paper kind of went in a different direction then I intended but that is science.

While I was working on integrating cats into my daily workflow my colleague Tianze Liu at Woods Hole Oceanographic Institution was busy analysing the acoustic data that was gathered in Oman. Tianze demonstrated that there is a subsurface coupling with the sun. That is, the sun comes up, heats up the surface, the surface expands ever so slightly due to the thermal gradient, and that causes a bit of pressure deep into the rocks. This causes the fractures where the fluids are to change their shape ever so slightly and that causes the frequency that those cracks oscillate at to change and oscillate as the sun rises and sets. This temperature induced gradient doesn’t cause the gas flux, it only alters the signal. But this is really cool. You can read this paper here.

What Tianze showed is complimentary to the papers I wrote and that brings us to the title of this post which is 🚨Newsflash!🚨 The climate and subsurface interact! Taken together, there is an implication across these papers that subsurface changes could be motivated and even controlled by the climate. And just in case you think these are just my crackpot theories, recent papers such as this one suggest that even seismicity (i.e., earthquakes) could be motivated by climate interactions.

But does it? As far as the data from Oman goes, I think what we actually have here is a lot of circumstantial evidence. And if you have watched a lot of Law and Order, you will realize that circumstantial evidence is not enough to convict. So now, we have all these systems, they seem to be coupled between the what goes on above ground and below ground and we want to really begin to articulate at least when this is happening (where would be nice too, but likely too difficult). So, that is what I am working on now. Trying to build a time-series model that couples these highly disparate and multi-modal data together in a way that can expose system level dynamics. Just looking at the plots of the fractures and the bubbles it seems like there is this kind of “episodic system coalescence”. But we shall see. Come find me and I can use more big words that sound clever while we discuss what is really going on in the Oman peridotites. So wish me luck! And seriously, feel free to reach out if you have any ideas.

Enjoy? Buy me a coffee!

This substack benefited greatly from conversations with Tianze Liu and François Renard.

Contact me on LinkedIn

All the same can be said about producing modern, interactive data visualizations. Modern data visualizations like this one from the New York Times about economic yields, this bellingcat map of number of Ukrainian civilian deaths in the war in Ukraine, or this zona.media map of the number of Russian deaths in Ukraine all present striking visuals to communicate understanding from data. Interactive data visualization is a key skill that has, for a long time, been locked up by front end web development tools (d3.js for example) that most scientists get no exposure to during their PhD training. Moreover, most data visualization can be an enormous time sink for scientists who build such visualizations in order to communicate with others their results. However, coding agent tools like Claude Code are changing this. Below I detail a few examples that I have vibe coded in an afternoon (or less!) that show off these capabilities. I include a link to each one and also the approximate amount of time it took me to get from 0 to something on each project.

Gaining an intuition on L2 regularization

Regularization is a common machine learning tool to control model overfitting. Most models are good at “overfitting” data which means they learn all the details of a data set at the expense of the big picture. Regularization fixes this by changing the way models learn to fit data by challenging the assumptions the model is making at each learning step. This is true whether the model is a linear regression, a gradient boosted machine, or a neural network. To test this out I wanted to build a nonlinear data set that could be fit well with a model using a regularization term. In this case I focsed on the L2 regularization becuase its very common across AI/ML but there are a lot of options. I created some noisy sinusoidal data, then fit it with a polynomial that includes a regularization term. The strength of the regularization is always controlled by a scaling variable called the “learning rate” which in this case is represented by λ. By sliding the λ slider back and forth we can control how strongly the L2 regularization term affects the model.

Total time to make the notebook: 5 minutes, then I edited the code and played around with the notebook for about an hour.

You can find the code here if you want to play around with this visualization: github.com/mnky9800n/l2_regularization

Comparing the population of US counties that are the same as major cities around the globe

There are lots of political discussions to be had about the distribution of votes in the US House of Representatives, but one thing is for certain, Americans are concentrated in cities and there are many counties that have very small populations compared to any region close to a major metropolitan area. I came up with this idea based on Derek Camp’s LinkedIn post which showed a static map. I posted a comment at the time saying:

Then I thought to myself, well, why bother posting such a comment if you are unwilling to at least try to make the visualization yourself! So I started up Claude Code and got to work. In about half an hour I had made this visualization tool I uncreatively called “popviz” which allows you to pick various cities and then it automatically highlights the counties based on where your mouse cursor is.

Time to make the visualization: 30 minutes, maybe an hour max, It had some bugs in the first iteration on highlighting the counties that needed to be fixed, but thats it.

You can find the visualization here: https://mnky9800n.github.io/popviz/

You can find the code here: https://github.com/mnky9800n/popviz

Investigating the topics of papers published at major AI research labs

The center of gravity in scientific research may be shifting away, in some ways, from universities and public institutions toward private labs powered by AI and often backed by venture capital. But how do we know? First I identified the top 10 AI labs that I wanted to investigate. Then I asked perplexity to research how much money they had raised and produce a big table for this. Then, using this table of various AI labs, I had Claude Code generate a list of topics from the labs, key authors that had worked at these labs, and a query script to pull a subset of papers from the semantic scholar API. This included a list of the paper titles, authors, and SPECTER2 embeddings (a document embeddings model specifically designed for academic papers). These were then put into a Gaussian Mixture Model that could cluster the topics, to eventually produce this plot, all the papers transformed to UMAP space, labeled with their topics. This is all further discussed in the post below. However, if you are at all interested in doing some kind of library science (i.e., figuring out ways to automatically gain insights from large bodies of scientific literature), I would suggest starting with the semantic scholar API. You don’t even need an API key to get started!

Time to produce: an afternoon of going back and forth from writing the substack piece and working on the analysis codes.

Calculating earthquake statistics for California while giving a presentation!

About a month ago I visited the University of Texas at Austin, Institute for Geophysics after having been granted the award from their visiting fellowship program. This is a great program and I strongly recommend reaching out to someone at UTIG if you find their work interesting and want to work on something together. When I got the award Thorsten Becker, geophysics professor, computational scientist, and all around cool dude asked me to give a talk on “AI tools”. He didn’t provide me any other guidance. Ha! So I decided that I would give a talk on three different topics. First, my thoughts about how the computer is a tool that lets you model anything you can imagine and that is how we should interact with any new computational tool. These ideas all came from watching talks and lectures by Steve Jobs, Wil Wright (creator of Sim City 2000), and Jensen Huang, amongst others. Second, I spent some time talking about using tutorial driven software development to build better computational tools for science. A core focus on tutorial driven software development is that it places documentation as a primary component of the software development process. This also makes it very accessible and discoverable to AI models. This is something that we had used on the 4D-Modeller project and the merrypopins project.

Lastly, I summarized the results of the substack I described before, that AI labs have pushed a lot of science in the direction of using AI to do science, and a lot of that cutting edge science happens there.

After a short introduction to my ideas, I told the audience I would open up Claude Code (see me describing this in the picture above) and using the following prompt, allow it to run in the background while I continued my talk.

Prompt: Download the ANSS earthquake catalog for California from the last 20 years (minimum magnitude 2.5), then write a Python script to calculate the Gutenberg-Richter b-value and create a matlotlib plot showing the magnitude distribution with a fit line. 

If you have used Claude Code before you know that sometimes it will think for a minute or two before it asks you for interaction again. So this was perfect to get back to giving my talk while Claude did his thing. And so as Claude would ask for interaction I would switch back and forth until eventually voila:

The matplotlib interactive plot environment sprung to life on top of my slide producing the above plot. Almost like magic I had multiple b-values, data, plots, and everything a statistical seismologist could ever want. Of course the point of this exercise was not to perform some act of science, but to show that I could casually, without much effort, produce a common analysis that most of the people in the room would have some familiarity with as to how much effort it would take and what considerations needed to be made to do the analysis. That is to say, they could immediately examine the plot and decide if it was BS or not and what changes we could make to enhance the work.

Time to finish: 5 minutes of talk time, I would switch to Claude Code any time it popped up to say it wanted some feedback.

The future is here

I think that one thing that is really cool is just how fast all these examples were able to be prototyped. Mostly I had something working in less than 30 minutes. Only the substack post about AI labs took longer, and that was because I performed a more complex set of activities compared to the other examples. Are there mistakes in the code? Probably. Are there mistakes in the code if I would have written it myself? Definitely (lol!). The strength here is that Claude Code, and other AI coding tools, allows you to go through more iterations of an idea but also to try out more ideas. Of course, if I was going to use any of these codes for a paper or a production environment, I would put them through much more scrutiny than I have here. My point in all this is that a lot of the sort of “boiler plate” data analysis and visualization that scientists may be comfortable doing in an afternoon now only takes half an hour. And even further, that many of the interactive data visualizations that were before only doable if you know more about front end web development are now immediately available to all scientists. And that is a good thing.

Enjoy? Buy me a coffee!

One of my favorite novels is Michael Moorecock’s The Warhound and the World’s Pain. The book tells the story of Ulrik von Bek, a mercenary fighting during Europe’s 30 years war which devastated much of Europe and was responsible for approximately 8 million deaths. Ulrik gets separated from the company of soldiers he leaves finding a solitary castle where only a beautiful woman lives, Lady Sabrina. He stays at this castle and falls in love with the woman deciding that he wishes to stay for the rest of his days there. Eventually the castle’s master returns and Ulrik finds himself face to face with Lucifer, Lord of Darkness, Master of Hell. Lucifer agrees to allow Ulrik to take both possession of the castle and it’s Lady if Ulrik goes on a quest, the retrieval of the Holy Grail. After a number of adventures Ulrik succeeds, finding the Holy Grail close to the North Pole, guarded by Lilith, second wife of Adam. By releasing the Holy Grail, Ulrik also grants humanity access to all the knowledge and reason of the universe, bringing about the Enlightenment.

A couple years ago I won a grant from the Norwegian Research Council. The proposal I wrote was ambitiously titled “SerpRateAI: Constraining Peridotite Alteration Rates with AI and ML” and it promised, amongst other things to create a collection of machine learning models that could connect the various environmental processes that controlled how a particular type of rock called a “peridotite” absorbs CO2 (you can read more about one of these efforts here). This rock is common and found in outcrops in Oman, Norway, California, and elsewhere. Anything that naturally absorbs CO2 is interesting since if you can control that process you can begin to do something with all the CO2 we are pumping into the atmosphere. If you want to know more about that, you can ask Al Gore.

A central component to all this work was to listen to these rocks. The belief was that by listening to what is going on underground with these rocks and by characterizing those sounds by other, ongoing environmental processes, I would be able to quantify how much CO2 was naturally absorbed by these rocks and ultimately save the planet. In order to listen to these rocks, boreholes were dug into the rock, hydrophones were inserted at deeper and deeper locations in the rock, and these hydrophones listened at 1000Hz, rather low if you are an audiophile, for 9 months to try and “hear interesting stuff”. Additionally, 40 geophones were deployed on the surface for the last month of the 9 months.

You may already see that this introduces a difficult machine learning problem. We want to listen to the rocks for long periods of time and note whenever “something interesting” happens. Once we hear something we determine is interesting we then search for some physical mechanism to describe what we hear. But how do we determine if something is interesting? Essentially this becomes a search for a Holy Grail. How do you convince a computer to examine raw data and automatically decide what is interesting or not if you don’t give it any opinion about what is interesting, such as training data?

The first step is to simply look at the data.

Here we can see a spectrogram of an hours worth of data from one of the geophones. You can already see lots of interesting things. You can see these minutes long, broad band “explosions”, you can see these horizontal, narrow-band lines, you can see that there are some narrow band “gliding” motions, there is some kind of low frequency noise going on. This is all rather interesting and finding this in this data is promising. Perhaps we will know something more about this data set. Zooming into a few seconds of data (recall we are sampling 1000 times a second), you see this interesting “gliding” behavior that occurs at very short intervals.

So I set about to develop an algorithm that detects these kinds of events. The first thing you notice is that they are quite narrow band, although that band does seem to move around. And they do appear somewhat energetic. So I thought, perhaps this could simply threshold most of the stuff away, and since there is some structure to the things that are “interesting”, you could probably use some kind of sliding window to amplify everything in the image that is connected, and then arrive at some reduced version of the image with the “interesting” stuff as 1s and the non-interesting stuff as 0s.

But then I talked to a colleague and he said, oh but there are 40 geophones, don’t you remember? So we shouldn’t do this for only one geophone but only for things that appear across many geophones. So then we put together this stacked histogram based on whether some subset of geophones had all detected the same energy at the same frequencies at the same time. If this threshold was matched, we would then try to label those things. An example of this is below.

Here you can see all of the “pixels” that are showing up across some subset of geophones. There is already some interesting behavior here. You can see there are these horizontal lines at about 50Hz and 100Hz. You can see there are some smaller horizontal lines at other frequencies. You can see there is some gliding behavior where these narrow band signals increase in frequency between 50 and 100Hz. What is going on here? Is any of this interesting? Maybe. In discussion with colleagues there is some idea that this could be fluids and gasses moving in cracks underground. How do you verify that? Well, there are plenty of papers (for example) that talk about gas bubbling up on the surface in these regions with these kinds of rocks. So it is possible that is what we are hearing. Fluids moving through a crack can resemble some organ-pipe mode characteristics.

But for some reason I found all of this intellectually displeasing. Today, thinking back, if I had focused my efforts on finding interesting statistics in the labeled regions, perhaps I would have written a paper about something. At the time, I had used regionprops in skimage to labeled everything and tried to count up some statistics about the various things I found. But the statistics weren’t interesting. Everything just accumulated over time without any connection to other variables. Now that I think about it, I think there probably is some signal in the statistics. First, If you already see something interesting, continuing to investigate it typically leads to more interesting things. Second, if the entire data pipeline had been cleaned up a lot and focused on picking out some of the more interesting phenomena we would have statistics only about interesting things having filtered out all of the noise. However, at the time, I thought that developing the algorithm would benefit me in some way more than pursuing the scientific line of thought. I had verified there were interesting things in the data, but something in my mind said there must be some sort of more intellectually pleasing way of doing this verification.

If you are currently at this moment in your work, I will tell you now, THIS IS A TRAP. You will not find a more intellectually pleasing algorithm, you will not find some smoking gun, YOU ARE NOT GOING TO FIND A HOLY GRAIL. You will simply spin your wheels looking for something while you have something already sitting there that you have decided to throw aside. Get something done and delivered then move to the next step.

So instead of trying to build some working statistics from my cobbled together algorithm that mostly worked, I set that all aside and I decided to explore if I could arrive at this labeling without labeling (something I already had managed to implement!), by using UMAP and autoencoders to label everything in an unsupervised way. I decided that unsupervised labeling using autoencoders is very elegant. Autoencoders, in their simplest form, use a succession of convolutional layers that reduce in resolution per step and at the center, these layers create a reduced-dimensionality bottleneck layer in your neural network that can output a latent space that describes your data that you can then use some kind of clustering algorithm on. You would not be beholden to personal opinion or sway and you wouldn’t need to do intensive effort, you could simply take your magically perfect data, dump it on your magically perfect model, and it would give you magically perfect results. And you know what, the magic worked.

I worked with a summer intern student (who was brilliant and currently enrolled in law school) and developed an autoencoder model that could reconstruct the binarized spectrogram, it was doing a reasonable job of separating them into different classes using UMAP and was in fact doing so in ways that was somewhat better than I had done before, these classes had no physical meaning, but that didn’t matter. We were doing something elegant! And yet, we had arrived at the conclusion I already had arrived at. And ultimately, none of this work turned into any kind of deliverable like a paper or even a conference talk. Except now, I use it for fodder for my substack because I decided I must think about what happened and if it is possible to turn all this effort into a meaningful paper about something. The good news is that these ideas aren’t new and there are several options available to you if you want to try out similar libraries that are more mature. We may even try one or two in our endeavor to label everything. I am currently working with Karina Løviknes on this problem and we are actually getting somewhere (that’s a different post).

The problem here is, you can label anything you want. But you can’t describe what it is. I had, in some ways, inverted the original goal. Instead of trying to understand how these various systems interacted together, I had focused on instead developing methods that I found elegant to make observations of the data. But the data has already been collected. All of these additional analyses only verify what we already know, the data contains stuff that could be considered “interesting” by somebody. But because there had always been this nebulous goal “label interesting sounds” we arrived at a nebulous conclusion. And that is because there is no holy grail. There is no way that I can label everything and simultaneously determine what those labels mean without some kind of interpretation going on at some point. And that interpretation has to come from me.

Ultimately, I realized that this question of “how to label interesting things” is along the same lines people are asking about Artificial General Intelligence (AGI). AGI is what you were promised in Star Wars, a walking, talking anthropomorphized computer that can interact with you like a person, understand anything you give it, but also works the same way as a computer, so you can leverage that. An AGI could talk to you about your research ideas, then while you go make yourself a coffee, auto-generate 5000 lines of code to perform 10 different explorations, and deliver you a report to read before you discuss what to do next while simultaneously making stock trades in your portfolio. It could pilot your starship while also generating a holographic fantasy land for you to play in. It could do anything you can imagine and it can do it better than you ever can do it and it will do only at the expense of unlimited resources and unlimited energy. To be honest, I think that this is all ultimately possible. There is no reason to think with essentially unlimited resources we could build a computational engine that could do anything and be anything to anybody. But in my case, I didn’t find AGI. I found the things that I already knew were there hoping to find them in ever more elegant ways until one day my computer would be able to both listen to all my data and write all of my papers and reports while I sipped my coffee. So be wary when you search for elegance. Elegance will likely not find you a holy grail.

I hesitated pressing “submit” here since I basically am describing how I failed at something that I really wanted to make happen while realizing along the way that maybe it was a dumb idea to begin with. But I think that, ultimately, I’m not doing anyone favors talking only about when I succeed. Hopefully someone finds this interesting.

The modern research lab is run at a megacorp tech company where they have, essentially, unlimited resources to work on a wide variety of topics as long as they have good computer skills. The labs are set up between pure and applied research (e.g., Microsoft Research, Deep Mind), applied research that is likely profit seeking (e.g., Google.x), and then the development teams that industrialize/commercialize this work. This isn’t new, but this mode has shifted to be less application in many cases (e.g., quantum computers promise a lot but do very little although the promise of their abilities is worth the effort). And while those labs have been built at companies that originally may have nucleated at a university (like Google at Stanford) they are now, in some cases, being built using venture capital. For example, companies like Mira Murati’s new Thinking Machines Lab which has raised 2 billion dollars in less than six months. They also include innovative new places like the Astera Institute which offers highly competitive salaries to highly competitive researchers to simply pursue their own research endeavors. Astera is railing against the establishment trying to both innovate science funding as well as science publishing having opted out of publishing due to the predatory nature of publishers even premier ones and the lack of progress, scientific interaction, and the insufficient review process, that is produced by authoring scientific publications today.

The ideas behind building labs like this are relatively simple. Directing pure and applied research in directions related to a company’s interest will eventually produce new ideas that can be turned into billion dollar products. These research products don’t necessarily need to be integrated into the main lineup of a company’s efforts. They can instead be spun off into a startup (e.g., Silurian, the company behind the Aurora foundational model of the earth’s climate and weather, was originally built inside of Microsoft Research). This is allowed as the parent company recognizes that this helps to create a healthy ecosystem that they will likely profit from in the long run.

This is not new to Silicon Valley and is, in some sense, was how Apple Computers got started. Steve Wozniak was working at Hewlett Packard when he was building the first Apple computer prototype. When he realized that HP may want ownership of his technology, he went to HP and asked if they wanted it and they said no allowing Woz and Steve Jobs to start Apple Computers. And this was part of the company culture of Hewlett Packard and is known, in part, as The HP Way. Hewlett Packard famously allowed their employees to work part-time on their own pet projects. This came with the understanding that Hewlett Packard would get first dibs on any technology that came from these investigations. And this led to the creation of new markets worth billions of dollars based on innovative technology such as the ink jet printer. Like checkout this quote from the HP Way:

We encourage flexibility and innovation. We create an inclusive work environment which supports the diversity of our people and stimulates innovation… This is especially important in a technical business where the rate of progress is rapid and where people are expected to adapt to change.

But these AI labs have, essentially, taken these ideas and put them on steroids. And this isn’t just me talking. These labs below all have been founded in recent years, and you can see that they have all raised large amounts of money and these numbers are likely a serious underestimate of what has actually been raised in total. These numbers only reflect what I found when researching but I am confident there is more. And I certainly have not accounted for all of the various companies that are being built currently.

If the table above doesn’t render correctly, try opening in the browser instead of the substack app.

And if you think the topics these labs focus on is only related to AI, it is only for applications like ChatGPT or making pretty pictures, you should think again. Researchers in these labs have produced multiple ground breaking papers including research that won the Nobel Prize.

Below is a topic distribution in UMAP space of the topics of 3,703 papers published by authors working at the companies in the table above. I have used the SPECTER2 embeddings provided by the Semantic Scholar API to create vectors for every paper published in the subset that I sampled and then mapped these to 2D UMAP space. I then clustered them with a Gaussian mixture model and asked Claude Code to summarize the paper titles into a keyword or phrase. And the results are astonishing given the breadth of topics you see below.

Keep in mind, I have only focused on companies that are really famous in the current moment for their contributions to artificial intelligence. But note what a variety of topics these companies have covered including most of the topics you might find in university departments today such as physics and astronomy, planetary science, chemistry, genetics, biology and disease, and education. And this plot is definitely not comprehensive, I only downloaded a small subset of papers published by a small subset of these labs for this analysis (3703 papers). But there is a huge number of companies working in a variety of fields such as Terra AI in mineral exploration, Silurian in climate and weather, apic.ai who are building models to observe pollinators like bees, 44.01 who is changing how we sequester carbon, and this list continues.

I believe that this is going to, in some ways, take over the way research is done across academia. These large, privately funded research labs are already hoovering up talent across domains such as artificial intelligence and machine learning, geospatial intelligence, space travel, protein folding, and quantum computing. I expect that the domains that these companies are interested in will only expand and there will be more funding towards new topics such as critical mineral exploration, education, social science, and many other areas as we expand the resources, skills, and technologies that society needs in order to build a healthy and bright future.

Industrial research has long been a core component of the entire research initiative of humanity, however, these new AI labs have enabled a recent paradigm shift in how funding can be acquired and used for R&D compared to traditional funding avenues for academic researchers. And it doesn’t come at a better time when NIH, NSF, and NASA funding in the USA is at an all time low. And if you are in Europe or China, there will be network effects to what is going on in the USA science funding. There will be fewer partnerships with American scientists, fewer PhDs coming from top American schools, fewer new ideas to work on coming out of American universities, and over all less scientific opportunity at American research universities.

There are several traditional entry points to these private research communities like Stanford, Harvard, and MIT. Alternatively, nucleating startups to build your own environment or to get acquired is a way in as well. If universities want to remain relevant they should be doing their best to build these pipelines for their students because real research will be done less and less at the university because of the decline in funding and the expense of computing. Additionally, there is a deep need to do interdisciplinary work which can face unique hurdles when universities are split by their domains so severely given the traditional segmentation of university faculties/departments. I have been super fortunate to work on machine learning and data science topics at the Njord Centre for the Studies of the Physics of the Earth, where independence and interdisciplinary work is a core value, and so I see how valuable this can be.

I think that, in some ways, researchers need to come to the conclusion that things are changing and that there are wonderful opportunities on the horizon but we must think differently if we are to succeed. This is especially true in Europe which has a high concentration of green energy companies and healthy startup environments for scientists like in The Netherlands. We could build private research labs focused on developing the cutting edge that is necessary for us to reach a carbon neutral future across domains such as critical resource exploration, mining, producing critical materials in the lab, ocean science, natural hazards, and other topics. Building these technologies will help us build the future we want to live in. If we are not trying to explore these new research spaces being created by AI labs then we are destined to become irrelevant, chasing boondoggles in the darkness.

Are you a research scientist working at the interface of AI, robotics, and/or geosciences? Hit me up on LinkedIn.

Hit me up to start a geosciences AI lab

If you ever have thought about how things deform, break, or otherwise change due to mechanical processes you have thought about material science. If you have ever thought about how novel materials are made such as plastics or metal alloys you have thought about material science. If you have ever considered that the Bronze Age was brought about by the invention of a new material called “Bronze” which is the combination of copper and tin, you have thought about material science. Material science is all of these things and is a super active field that spans physics, chemistry, engineering and computing. These days it even includes AI in coming up with new materials.

Nano-indentation is a relatively simple science in terms of the vast amount of material science options available out there. You poke a material with a microscopic “tip” with more and more force and measure the response from the tip in terms of the force applied and the current location of the tip to see what happens. This simplicity allows for strict control of the system while producing rather dynamic behaviors. Different materials have different responses based on the Young’s modulus, etc. and you record these and then put them together to try and understand how the material deforms under load. These behaviors can even lead to chaotic instabilities like rapid deformations that occur, perhaps without any warning. If a tiny tip being put on a surface sounds familiar to you, you might have heard of atomic force microscopy (AFM). In an AFM setup, you drag the “tip” across the surface of the material you care about, as the tip moves up and down it records the elevation profile of the surface of the material. In fact, you can use the nano-indentation machine to create such elevation profiles so that you can study the shape of the indentations themselves to try and understand the material properties and dynamics of the deformation.

It is believed that these nano-indentation experiments form one end member of dimensionality in the deformation of lots of different materials including hazards such as engineering failures, accidents, or earthquakes. By indenting only single grains in a rock, and therefore a single material, we can understand how those very small deformations occur over time. If we get some kind of failure mode in the material, it is believed, that this could be some kind of observation of an earthquake nucleation process. Below is a succession of increasing zoom levels to demonstrate both the process and exactly how small of a poke we are giving the rock.

A central component of this work is identifying and locating something called a “pop-in”. A pop-in occurs when there is a sudden change in the deformation process. The tip is being forced into the material and suddenly the material gives a little, allowing the tip to travel deeper into the material rapidly before being slowed down again back to it’s regular pace. This pop-in is poorly understood in the data and it is believed to have something to do with some kind of fracture nucleation process. But nobody knows. Understanding these indentations includes both making the raw observations of the experiment itself, but also performing statistical analysis across hundreds or thousands of these experiments as the apparatus can be automated. Enter merrypopins!

merrypopins

merrypopins is a python software package that we have been working on in collaboration with Utrecht University Applied Data Science masters students. We have developed a library that loads and ingests data from the nano-indentation experiment, preprocesses the data to remove known artifacts, calculates stress-strain curves for the data, uses different ai/machine learning models for locating pop ins in the data, and facilitates statistical analysis across many iterations of the nanoindentation experiment. This library was developed in close collaboration with the material scientists and geologists at Utrecht University using tutorial driven software development practice and includes both tutorials that library users can use as well as as streamlit app that requires no coding knowledge.

The library is in active development!

If you would like to help develop this library feel free to clone or fork it on github:

git clone https://github.com/SerpRateAI/merrypopins.git
cd merrypopins
pip install -e .

or you can pip install it if you just want to use it as is:

pip install merrypopins 

If you find a bug or want to make a feature request, you are welcome to create an issue!

Currently the library needs the following

1. Data from other nano-indentation experiments to build out the load module to account for the varying data types from various pieces of equipment

2. More users to test it on their data to build out features for statistical analysis

3. A version 2.0 release that includes more references to Mary Poppins

Happy coding everyone.

When you submit a scientific paper, it is typically because you have labored rather intensely for months to arrive at a result you think is relevant for others to read about. You have put a large amount of work into a paper, so much so, that you might have overlooked something that might be obvious to an outside observer. This is because you develop a kind of tunnel vision by being so close to the project. Peer review is great in this situation because it can act as a check against any kind of overfitting your brain may have done on the problem space you are exploring. Or perhaps you overlooked some body of literature that might be relevant but you are simply unaware. So you send your paper off hoping to receive constructive feedback. You wait weeks, maybe months, to get feedback from others and when it finally arrives you get comments like:

Machine learning always produces trash. [source]

This perspective is very old and boring. [source]

You [addressing the authors] shouldn’t be allowed to submit work like this. [source: anonymous reviewer from when I was guest editor for Physical Review Physics Education Research]

This young lady is lucky to have been mentored by the leading men in the field [source]

What exactly are you supposed to do with comments like that? How is that constructive feedback? How is that supposed to help you treat your self imposed tunnel vision? And how does that even make you feel to have your work labeled “trash”? Should we refer to quantum mechanics as an “old and boring” perspective? Or that someone wishes to gate keep you from their community? And this isn’t just me noticing this, it has been reported by AAAS and ars technica reports on it and people extend these arguments to the peer review process for grants as well. I believe that this is all happening at least in part because people are hiding behind the anonymity of the review process, or, they simply believe that it is okay to treat others poorly as long as it is on the internet. That is, like some internet forum commentators, some peer reviewers are internet trolls.

In any case, I don’t want to dwell too much on the terrible things people say to each other. Instead, I want to highlight a great peer review experience I had when writing a paper for the Journal of Open Source Software.

JOSS started to address a different issue altogether in science, that is, that often times computational scientists build complex software solutions that typically don’t get marked as a metric for any job evaluation. On their website they say:

In a perfect world we'd rather papers about software weren't necessary but we recognize that for most researchers, papers and not software are the currency of academic research and that citations are required for a good career.

JOSS says they are “Committed to publishing quality research software with zero article processing charges or subscription fees.” So you can probably see already why I am a fan of this journal. In addition to all of their commitments to a computational world where journal articles are free to submit and published open access, they also have developed a new way of evaluating papers through peer review.

If you are unfamiliar with the traditional peer review process, it works basically like this:

1. I do some research and write up a draft, agree with all my coauthors its a good draft.

2. I find a journal, hopefully a “high impact” one, that is willing to review my paper draft and I submit it through their web portal.

3. Typically this portal will ask you to recommend some reviewers and also suggest people you don’t want to review your paper for whatever reason

4. maybe you will need to sign a “no conflict of interest” document

5. and, then you click a big “submit” button and go about your day.

Assuming that you have submitted a paper following the format criteria, this paper is sent out to reviewers who are selected by the associate editor in charge of your paper, and they are expected to follow up to make sure the reviews are done and then sent back to you. Typically, you then wait 3 weeks to 3 months to get receive these reviews. Then, when the reviewer receives a paper review request:

1. they are first asked if they are willing to review, and if they say yes, they are usually given a tight deadline (10 days to 2 weeks, maybe less) to give a review back.

2. The reviewers are expected to write all their thoughts about the paper, look for inconsistencies, and generally follow the guidelines for reviewers that are made available (for example).

3. They submit these reviews back to the journal, typically through some web portal.

And this process is repeated until the editor and sometimes the reviewers are satisfied that the paper is publishable. Sometimes you go through successive iterations with new reviewers being added to the mix. Sometimes you only do it once and the paper gets published.

If you are thinking at this point, “this process sounds like it could be done via snail mail,” then you are correct. This process evolved long before computers were available to everyone. And I think JOSS has enabled a giant step forward in peer review that is somewhat ignored currently by the rest of science.

JOSS uses the github issue tracker and pull request system to write all reviews for the submitted paper describing the software that was developed. A reviewer will open an issue in the repo like this one here where the reviewer suggested that the library shouldn’t be changing the home directory whenever it feels like. Then one of the coauthors can address the reviewers comment either by explaining why they won’t make the suggested change, or simply making the changes and then submitting a pull request. If you click the link you can see the screen shot below:

Here you can see that the reviewer comment is logged, you can see me, mnky9800n, assign the issue to another team member aabelean. He then writes some code to fix this issue and creates a pull request mentioning this issue. And ultimately it is closed and fixed in the library. This system allows for the full exposure of the review process, including all of the changes that were made, and it is public, anyone can go and look at these issues and see how the JOSS peer review system made the 4D-Modeller package better. By having this fine grained approach, everyone can see everything that is happening within the paper, but also, it focuses the reviewers on the review. I don’t think that wcjochem would ever reply to this work by telling us that “Bayesian Hierarchical Models always produce trash”. A big reason is because the system that wcjochem is using to provide feedback is designed to target his feedback to the problem solution being presented.

Will this system work for all types of papers? Maybe. I think that these kind of systems can be explored in publishing. For example, Copernicus journals like The Cryosphere also have a public commenting and reviewing system. You can see here where colleague of mine has a paper where you can view the full chain of reviews including each iteration:

I am not sure that we currently have an optimum way of organizing peer review and scientific publication. I think that this kind of interaction could even be studied and optimized somewhat. But what I do know is that if you are writing stuff like “Machine learning always produces trash” and patting yourself on the back for engaging in “intellectual discussion” you should feel bad about yourself. And if that makes you angry, good.

In any case here is a list of ways to improve the current status quo:

1. Engage with journals that have public review processes

2. Review papers non-anonymously

3. Engage society publishers like Copernicus

4. Question why you are insecure instead of trashing some poor PhD student’s hard work

Be good to other people. We don’t get that long to live on planet Earth.

Prediction and Predictability is, in essence, how much of a system can you describe from one state to the next. There are many different types of “predictable” systems and there are many different ways to characterize how to predict various systems. In The Price is Right, contestants try to “predict” the price of an item because they are making an educated guess of the cost of the item. The average person has purchased dish washing soap and likely can guess the price based on that experience. Many random systems have interesting conclusions when physically simulated. For example, fair coins have been demonstrated both in simulation and in observation to be biased to land on the side they started from. However, the key component across all systems is that if they are predictable, in some way, then we can summarize or compress some information about those systems in such a way that we can use that summarization to inform the evolution of the system in both cases that evolution is random or deterministic.

However, there are important differences in how statisticians and machine learning researchers define “predictability” versus how physicists define “predictability”. And thus, we are left with the question, “What is a prediction?” Below I will articulate two views: the machine learning view of prediction and the physicist view of prediction. There are more (e.g., I skip over Bayesians) but hopefully this is an interesting way to think about things. I focus on two key descriptive parameters, the R2 score which is common in machine learning regression tasks. And the Lyapunov exponent which is commonly used to examine chaotic regimes in various physical systems.

Statistics/ML view

Statisticians and machine learning people typically assume that there is a certain amount of “variance” in a system that can be explained by attempting to “linearize” the system of study. Variance is defined as:

This equation simply states that we want to know how far the “average” data point x is from the data average xbar. You often see in papers that use machine learning articulate their models using a similar equation called the R2 score:

If you compare the first and second equation the second equation is a ratio where we have replaced xhat with xbar. That simply means we compare how far does the data x deviate from the “model” xhat and it’s ratio to how far does the data x deviate from the average data value xbar (i.e., the previous equation). This R^2 value can have a maximum value of 1 and a minimum value of negative infinite, that is a model can be arbitrarily bad at predicting things. There are other statistics such as the root mean squared error, but they mostly attempt to articulate a particular view: how well does the statistical model capture the “predictability” of the system by how much variance is described by the statistical model? By adding more constraints, for example, keeping a validation data set separate from the training, you can extend the argument a bit and suggest your model is “generalizable” since it has operated successfully on data that it has never seen.

So far, the model xhat could be anything. We could articulate a model that says “if it rained yesterday p millimeters, today it will rain 2p millimeters.” And so if we have some rain data from a location we would first calculate our model output per day using the previous day’s precipitation and then use our R^2 equation to decide if we did a good job or not in predicting tomorrow’s precipitation. However, we could also use other data, e.g., the atmospheric pressure, as controlling parameters for tomorrow’s precipitation. As we add data variables to our model and it becomes more and more complex, we may want an algorithmic way to arrive at some optimum functional approximation to provide the prediction (e.g., a neural network or a gradient boosted machine or even a simple linear regression), but this is beyond the scope of this substack post. The point here is that from this framework, a system is considered “predictable” if we can somehow summarize the next state of the system by estimating how much variance is captured when we apply some model xhat.

Physicist view

Whereas the statistician's view of predictability assumes that there is some variance explained function to minimize, the physicist instead takes the view that dynamic systems either are deterministic or stochastic. Deterministic systems can be predicted into the future with some error that applies for when they diverge from prediction, and this can be due to chaos or simply the precision of the measurements. Stochastic systems are assumed to operate under some probability density function that is predictive of the overall behavior of the system but each individual event is random (e.g., nuclear decay rates). I will focus some on deterministic chaos but bear with me as of course I am not an expert on everything.

Chaos is defined as a system where the initial conditions can vastly affect the outcome. We can quantify this quite plainly with a Lyapunov exponent. The Lyapunov exponent attempts to measure the divergence between trajectories. If we examine the equation in the figure we can see there are two trajectories with some separation between them labeled x(t)+delta(t). This separation distance delta(t) can be described by the exponential equation, and when the exponent is positive we can see that the distance between the two trajectories will grow exponentially. This value is known as the Lyapunov exponent and is a common way to determine how quickly system trajectories will diverge from each other given various initial conditions. Of course, not all physics is chaotic, much of it is not (thankfully). In the non-chaotic cases we simply divert to the expectation that our physical laws are able to capture the trajectories of the system as time goes to infinity. That is, that the largest Lyapunov exponent is always valued at less than 0 is always true.

Discussion

Why is this important? It helps us, in a very general sense, determine when a system stops being predictable. However, let us now contrast the Lyapunov exponent, which is in some ways a measurement of similarity between trajectories of a system, to an R^2 score, which is also, essentially, a measurement of similarity between a model output and the naive assumption. In the R^2 score case, we are measuring the similarity of our machine learning model xhat to the naive model xbar. When this similarity is poor (i.e., R^2 is approaching 1.0), we typically claim that our model arrives at a good understanding of the system because we assume that this increase in R^2 value is because there is a decrease in unexplainable variance (or perhaps randomness). We compare different machine learning models with different features (i.e., variables or data) hoping that by adding more context to the model with additional information, we will be able to build a model that maximizes the R^2 score. This can of course be misused or misattributed and problems like “p-hacking”, violating model assumptions, unexplained heteroskedasticity, and other issues that I won’t go into here.

In the Lyapunov exponent case, we are saying something different. We are not concerned with what data we should add into our model, we typically have a pre-specified model. We assume that we have generated a relationship (e.g., through phase space reconstruction) that explains how different parts of the system interact. When the Lyapunov exponent becomes positive, the trajectories diverge from each other and can no longer be predicted except through our knowledge of the system initial conditions. No minimization of variance in the system matters, no additional variables or data concerning the system matters, we simply cannot predict chaos.

However, you may have read about recent work, such as reservoir models, NBEATS, mamba, etc., which claims to predict such chaotic systems to some degree past the point where Lyapunov exponents become positive. Taken at face value, this is profound. But why is this true?

Personally, I don’t actually know the answer to this question and feel like it is an open question. Perhaps other people know the answer to this question. Perhaps they might find it trivial. In any case, I will suggest, is a good time to stop and ask you to watch a few talks from the AI conference that I helped organize in October 2024 where I address this. I would suggest watching the talks by William Gilpin and Anders Malthe-Sørenssen. William’s talk directly dives into the predictability issue. Anders' talk is a discussion about understanding how a very complex generative machine learning model arrives at its solution. I think these talks begin to unravel why this is true, but the truth is we simply do not know. Or at least, I don’t know.

This was originally written as an internal newsletter which I have expanded and re-edited for here.

My job as a scientist is to understand the fundamentals of my field in the ways that current and historical scientists have established it. This is an ongoing process where I must continually update my knowledge and understanding of what is fundamentally assumed. As I do this I must form my own thoughts and opinions about what the cutting edge should be. This is the goal of science, to establish topics about which we collectively do not understand. My job as a scientist is to be comfortable with the idea that I don’t understand things because if I did, it wouldn’t be the cutting edge.

Across all of my research I have essentially taken some idea established elsewhere and applied it to some idea space that hadn’t seen it yet. I always was very critical of myself thinking I had no original thoughts. I would read about databases or gradient boosting or whatever and then decide this is what’s missing in the science standing in front of me. The more technologies I could learn, the more problems I could solve.

But now I think there are more skills to master as a scientist and software engineer than this. Finding the cutting edge is really finding the place where only opinion exists. And I think that today that has become increasingly difficult. I think that is for a variety of reasons but one reason important to me is that it seems like there are too many ideas to consider at the same time. This is where I think an effective AI assistant could help you investigate the cutting edge. This is also why hallucinations are such a problem. What if they simply make up molecules? Or they just connect two ideas for nonsensical reasons? But assuming that an AI assistant could be built and verified to discuss science and technology with, I do find perplexity.ai to be a great product. I am able to discuss and get cited answers and find myself examining those answers more than what the AI writes itself. And thus far I find it more like a recommendation engine. Like how Netflix tells you what next to watch. You don’t agree with everything Netflix recommends even though it usually knows you just want to watch The Office one more time. And that’s just fine for me. Because it also allows me to form my own opinion about what the cutting edge is, where the cutting edge is best described as what I don’t understand and what needs to be done next.

But the great fear becomes, what if a hallucination leads me astray? What if that compounds into a project in some way that is destructive. But what if the opposite happens? You follow a line of reasoning due to an ai hallucination that leads you to discover something you would have not discovered? This all leads me to the idea that all things are models. That all models can be tested and broken and that is the other goal of science. Building models of the natural world. Hallucinations don’t matter that much if they violate a model assumption. And so your model has edges that are fuzzy and the cutting edge is trying to establish what will make things less fuzzy. And this is all very abstract but our goal is to make things less abstract by building models. This is why I directed a short documentary about scientific models a few years ago.

Defining the cutting edge of LLMs, or “state of the art” (SOTA), has lately been a numbers game. In the beginning, it was quite obvious if an LLM failed at something. It just spout out nonsense. But as the nonsense has become more believable, and thus labelled a hallucination, metrics have been developed to define what SOTA is for LLMs. These metrics are essentially designed to be the final exams of various domains such as physics, chemistry, and economics. The better LLMs perform the more likely they are to be described as “phd level” while still not understanding basic mathematics such as the commutative property of integer addition. This all reminds me of reading conversations from the 1990s about what physics conceptual surveys were really measuring.

Yann LeCun has said himself he thinks LLMs are obsolete and if you are a PhD student you should work on something else. To him, one of the fathers of modern AI, the cutting edge is somewhere else. Ultimately the cutting edge seems to be wherever you decide the cutting edge is. That the cutting edge is somewhat different for everyone. For example, I’m sure there are people who are still excited about LLMs! In my opinion, it seems like one of the best ways to determine what the cutting edge is for you is to evaluate whether you feel accomplished at work, do you feel like you have an impact, do you feel like your work is growing you as a scientist or engineer, is that work challenging to you, and do you feel engaged by the community around you? If you have all these things at work, you are likely at your cutting edge.

Anyways, maybe it’s time for a second documentary.

Subscribe