There is a terrible curse in technology where everyone thinks you are supposed to have “side projects”. Side projects are something you do when you are not at work that in some way appears as work. I think Gen Z people call them “side hustles” because there is also the insipid idea that these side projects should make you money in some way. For some reason the modern era has decided to commoditize all parts of the waking life. I think the most striking description of all of this is Kyla Scanlon’s description of the Attention Economy:

If you think about it from this perspective, all side projects/hustles should be designed in some way to grab the attention of someone and leverage that into making you money in some way. That is, if you buy into the idea of side projects and side hustles should be commoditized instead of simply having hobbies that you enjoy doing (such as how I enjoy writing this substack). Few people think that someone should somehow monetize the rebuilding of a bicycle, or, just going for a ride on the weekends. However, for some reason if you write software as a hobby you should monetize it since that software is meant to be used by others and whenever other people are involved the Attention economy comes back into play, but I digress, Low Impact Fruit is supposed to be about science and technology, not the failures of the modern world and I promised in the title to discuss a side project that I have been picking at, thinking about, or otherwise working on for some time.

Rain Simulators

A very long time ago I stumbled upon the website rainymood.com. Essentially, some people like to listen to rain sounds. This isn’t a new phenomena you can find old CDs of various sort of environmental sounds. They are all in some kind of genre of relaxing meditation music, or perhaps something to help you concentrate. I often times like listening to rain sounds when I otherwise could be listening to music or silence as I find that rain sounds focus my mind on whatever problem I am trying to solve. I do agree with the CD above when it says:

Just close your eyes as you listen to nature’s own relaxing sounds on this CD and you can find it easy to imagine that it’s pouring with rain outside whilst you are all cosy and relaxing indoors.

I find rain sounds quite relaxing. I should probably spend more time listening to the rain. (Apparently Feel Good Dynamics still exists and makes a variety of relaxation-based media that can still be purchased as an MP3 or a Compact Discs).

Meanwhile, the world moves forward and I think a combination of GenX and Millennial nostalgia for the 1980s that started with Donnie Darko overtook me and everyone else born between 1975 and 2000. A big part of that nostalgia not only looks back at the halcyon days of Ronald Reagan (or whatever) but also a look back at how people then looked towards the future with books like Neuromancer and movies like Bladerunner. A central component of these works is the sprawling city that has grown and metastasized since the 1980s into some sort of Neo-Tokyo. I think my favorite iteration of this was the Boston-Atlanta Metropolitan Area from the movie Escape from New York. Thus everyone’s favorite sci-fi obsession, cyberpunk, took over every aesthetic in some way whether its neon lights, electronic music, or simply working on computers in some boutique Neo-Tokyo ramen shop on a rainy afternoon.

Coupled with my childhood dream of working in a coffee shop in Seattle (I don’t know where this came from) and also my childhood love of SimCity 2000, I came up with this idea of having a website based interactive rain simulator that would be a city scape in an isometric frame that has an active weather system that can be controlled by the user and that active weather system would be used to generate the rain sounds in a way that would be far more sophisticated than rainymood.com play button. It would give the user the power over the weather with a few sliders and other controls and you could watch various weather systems interact with the city below. It would essentially be a browser-based, slightly interactive screensaver for the modern computer user with a discerning taste for rainy sounds.

Subscribe now

I decided that it wasn’t enough to just have a slider and such, I should create a full Navier-Stokes simulation of the clouds interacting around the buildings of the city, the initial conditions could be initialized from random numbers generated by the actual atmosphere, and the rain sounds would then be driven by the distribution of densities as created by the simulation for each time step. This all seems rather ambitious now that I think about it. I think this ambition came from the last time I put a lot of effort into this project was when I was starting my PhD and one of my advisers was Anders Malthe-Sørenssen and the other was Morten Hjorth-Jensen (both of these links are in Norwegian), two physicists known for their computational efforts. For some reason I decided that having a side project about building rainy sounds simulators would impress them. I doubt Morten or Anders ever thought about this or even realized I was working on something like this. As an aside, this is something to think about the next time you wish to impress someone, they very may well not know you are trying to impress them.

So anyways this effort all got so far as building the simulation you can see above which creates a randomized building density and then randomly initializes the simulation and lets it run for however many time steps you want. That’s great but its not exactly visually pretty and you don’t hear any rain sounds because generating fake sounds that sound real is actually rather difficult if you haven’t really figured out how you are going to do that yet. And herein lies the dilemma, everything has become so complicated I start to feel a bit overwhelmed by it all. There is the weather model, the city animations, the controls, the sound generator, and you have to put it altogether in a compelling way that I would like to use it. Where do I start?!

Relax and listen to Will Wright talk about making models.

Enter the MVP

A skill I wish was pushed onto me when I started learning to be a scientist/whatever it is I define myself as at the moment, is developing an MVP. That means, Minimum Viable Product. Don’t explore every option and offer ever feature and understand every part. Instead, build the minimum that is expected and then test it with users to see if anyone ever will use it anyways. The same can be said for scientific ideas. I kind of think that I have often spent way too much time on developing ideas on some of the scientific projects I have worked on. And this was in lieu of simply trying out the minimum idea and then sending it off to reviewers to see if anyone is interested. But getting back to rainy-city.com, the MVP should have the following:

1. A webpage

2. A slider to control the rain noise

3. A slider to control the city noise

4. A visualization of the city

5. A visualization of the rain and clouds

That’s it. By defining the MVP, I am able to put everything aside and get something up and running, and then I can start adding features to it like driving some weather simulation with atmospheric noise. So that is what I did.

You can checkout rainy-city.com at rainy-city.com. There are no subscription fees or anything, just go and listen to the rainy city sounds and relax. Perhaps if you are feeling especially cyberpunky, you can have a bowl of ramen and think about how living in some sort of mega-corp dystopia is not actually the best idea in the world.

Subscribe now

This project was the project I proposed to do as my Recurse Center project. I ended up working on many other projects in addition to this one. That includes a semantic search tool for research, an arcade side scrolling fighter called Streets of Rainy-city.com for the RCade, random weird stuff for RCTV, a zulip bot that lets you play Zork inside of zulip, and I am working on an LLM reasoning eval toolbox based on zork. I am particularly proud of Streets of Rainy-city.com, I had a week to make it because there was a localhost event at Recurse to show the RCade and so I built the game in a week.

When I proposed rainy-city.com I was honestly scared, I thought, who would ever want this, why would anyone think this is a cool idea. But now, I realize that I wanted this. And that was enough. What will you build that you want?

I built this project while at the Recurse Center. Join the Recurse Center!

Last December, I was writing about a “futuristic scenario” where I could simply chat on my phone and get a project initiated, code written, run and debugged, results produced, and figures delivered to me, all without being physically at my working machine. For example, to build a python-based software that can organize and synthesize my news pieces, and overtime, build a database that can be mined and visualized. Just over a month later, that futuristic imagination is already a reality. Besides sending me the morning news, Minato also saves the news to a dated file and archives them. I can direct it to write an analytic script right now but I will wait for the end of the year. It feels exciting. But if I let my imagination run a little further, it also terrifies me.

Subscribe now

Code agents, specialized large language models designed for coding, are progressing rapidly and stampeding everything in their path. If you watch financial markets, after the joint release of Claude Code Opus 4.6 and Codex 5.3, the IT and enterprise sector fell hard. I think the fear is at least half true. But on the other hand, the job market for software devs goes up, as shown in the Twitter charts by David Sacks. Similarly, vibe coding startups are growing at lightning speed. See the report on Lovable, an Europe startup, At last, reasons to be cheerful about European tech by the Economist. People fear that code agents will shed jobs, but for the right person, they turn a software engineer into a productive team.

After reading my morning briefing, I sit down at my working machine and check what my multiple running Codex (OpenAI’s code agent) and Claude Code (Anthropic’s code agent) instances have written in the terminals overnight. I review them, evaluate their progress, then set them on to the next tasks, ranging from rolling out my newly trained graph neural network models based on a shell script template, to using git to version-control a successfully developed feature and releasing a new version with comprehensive documentation, to launching another batch of simulations sampling 100 parameter combinations across 20 available CPUs, to simply saying: They did a good job.

Then I open the Codex App, pull up my conversation, and ask it to show me today's to-do list from a living WORK.md document, a file that tracks my todos, things done, ideas, and project progress in real time. Whenever something gets done, a fresh idea sparks, or any progress is made, I just type it into the conversation, and Codex follows my rules to merge, track, and document everything in WORK.md. I imagine this will make my life considerably easier when annual report season comes around. I certainly can't afford a personal assistant. But Minato and Codex are the closest thing I've got.

This isn’t strictly coding anymore. And the companies building code agents have noticed that people increasingly use them not just to write code, but to handle tedious workflows and personal organization. I recommend the recent episode of Lenny’s Podcast featuring Boris Cherny, creator and head of Claude Code at Anthropic, titled “What Happens After Coding Is Solved” (February 19, 2026).

After I get a review of my todos, I tell Codex App my two or three priorities for the day. I make my coffee, and eat breakfast. The day begins, and it feels wonderful.

10AM. For the old me, the day would have just started. Now, a lot has already been done. Sometimes the pace of progress feels so surreal that it’s hard to believe Claude Code, the starter of all this amazing and terrifying momentum, was launched barely a year ago. A few years ago, if you asked a chatbot to do basic arithmetic, it couldn’t get it right. People joked about it. Fair entertainment. But if you’re still making that joke now, you’re the punchline. What really matters is the rate of iterations happening in the development and deployment of these new capabilities. This is no longer a six-month or yearly cycle. It moves by the month, sometimes by the week.

In the interview, Boris mentions that 4% of all public GitHub commits now come from Claude Code. Claude Code has become a prolific author of public code. Boris says that projected figure could reach 20% within a few years. And if you count private repositories, the share would be even higher. I believe it completely, because I’ve personally built multiple pieces of software for my private use with Claude Code, and that’s a lot of lines never counted in any public statistic.

Howard Marks, co-chairman of Oaktree Capital Management, published a memo on February 26, 2026 titled “AI Hurtles Ahead.” I’ve long admired Howard as a thinker and writer navigating this complex, fast-moving world. He probably doesn’t code much, but what struck me most was his discussion of the economic impact of code agents, drawn after just a handful of uses. People debate whether machines are truly thinking, truly intelligent. Interesting questions, perhaps, but maybe too academic. The real economic question is this: “If a code agent can deliver a report equivalent to what a $200,000 analyst produces, does whether the machine “thinks” still matter?” But also, if the code agent can deliver the reports of a highly paid analyst, how important were those reports that they can now be automated away? Perhaps those reports themselves were the friction point that AI is solving.

Subscribe now

I’ve been using code agents to develop prompts that query business performance and produce equity analytic reports. What I can say is that the breadth of search, the range of sources, is something I simply cannot compete with alone. The report quality is okay, not excellent yet, but solid enough, with sufficient factual data and tables to use wisely. If you have followed a business for years, you can tell its quality. For example, this report on Estee Lauder. And crucially, you can run as many such deep research tasks as you have tokens for. Whoever controls the tokens controls the information.

That brings me to the second striking insight from Howard’s memo. With information barriers lowered, if not demolished, by AI tools, where do we create value? In financial markets, short-term gains will be harder to come by, but long-term opportunities will remain plentiful, because most people still chase short-term results. As Charlie Munger, the late Berkshire Hathaway vice chairman liked to quote, “Always take the high road, it is far less crowded.” AI will smash you in competing in the short-term, zero-sum game requiring the scope and speed of information, but may not yet be able to truly “vision” far into the future.

For academic work, we face a similar reckoning.

The deep research function is now available in Claude Code (it also has an extended thinking toggle) and Gemini Pro. I’ve been experimenting with using it to generate reference lists on specific research topics. It consumes many tokens, takes ten minutes or more, but the product is genuinely impressive, broad coverage of a typical ~500 sources and more, useful summaries, good detail. You can even ask it to verify DOIs before writing the final report. It cannot replace years of deep research experience and insights, but it will quickly help you fill the knowledge gap of any new topics that may interest you. I typically will pick a paper, append the abstract with “please do a deep research on this topic; give me a peer-reviewed reference list; verify DOIs; I only have time to read 10 papers, ranked your reference list by impact; return to me a .md report; take you time.”

The information barrier is gone. So where, as researchers, do we bring in value? Judgment? I’m not sure. But what I notice is that when I now face an unfamiliar topic, I have no mental barrier to entry. The learning process has accelerated dramatically.

AI is excellent at filling knowledge gaps. But genuine insight still takes time and hard work. Maybe the answer resembles the investment question above: focus on the long term, something fundamental, and probably more important something you are interested in. As I see it, speed doesn’t take away insight. They don’t conflict with each other. Effective iterations bring insights and fast turnarounds many actually bring insights you expect early.

To close, I return to the epigraph of Robert M. Pirsig’s Zen and the Art of Motorcycle Maintenance, “And what is good, Phaedrus, And what is not good, Need we ask anyone to tell us these things?”

Nurture your judgment. Cultivate your aesthetic taste in your work. That is what AI has not yet taken over, and what I hope and imagine it will not, at least not anytime soon.

Bio for the Author

Dunyu Liu is a senior computational geoscientist at the Institute for Geophysics at the University of Texas at Austin. More on his research interests can be found here.

we should focus on what makes us uniquely valuable as embodied, social, and organic creatures who are compelled to be with, empathise with, learn from, and think with one another. Our ability to navigate human networks and our role within them is our superpower.

I was talking with someone last night who studies computer science education about what AI literacy is. Neither of us seem to know either. But over the course of the discussion we arrived at a definition that could probably remove the AI part and simply be associated with some part of problem solving. Below I will describe what I will define as “AI literacy”.

AI Literacy according to john

I have tried to embrace lots of AI tools as they come out and integrate them into my research. This includes both using tools like AI coding agents but also developing and training neural networks for a variety of tasks generally related to one of my many current focuses (foci?) which is rock-fluid-climate interaction. Feel free to read more about that here. Across all this effort I have realized that the level of abstraction is much more important than before. What do I mean by “level of abstraction”?

Subscribe now

Sometimes I want to build a data dashboard. For example I had some plan to work with a colleague on a large set of time series data. I thought it silly to mess around in a Jupyter notebook because it’s somewhat unwieldy when you want to look at ten different time series at once. So I texted my ai agent on telegram what I wanted and pointed them at the repo with the data and they did exactly what I wanted without further interaction with me. That was fantastic. It would have taken me a week to make that dashboard.

Other times I want to train a neural network on some data to build a classifier, do some kind of prediction, or whatever. I need to know a lot of specifics about the architecture and how that architecture changes as I explore what fits the data and solves the problem the best. If I asked my ai agent to do this nothing will work. And I know this because I tried. It was me literally experiencing this meme:

I had to spend an afternoon undoing the mess created by me letting an AI agent loose on my codebase.

And the difference between those two experiences exactly highlights what “AI literacy” could be described as. There is a spectrum of context that you should consider when using an AI tool. You should be constantly assessing if you are operating at the correct abstraction layer to solve the problem at hand. And that abstraction layer acts as a sort of tuning knob or hyper parameter of how involved you are in what the computer is doing. AI literacy is understanding how much you need to understand about what the computer is doing in order to assess whether it did what you wanted or not.

Steve Jobs and Apple liked to say when they built the personal computer they built a bicycle for your mind. You can watch him say it in the video above. I really buy into this concept and think it’s not just marketing hype. Computers are tools that can help us conceptualize problems beyond what we can do with only ourselves. This is all AI tools are doing. They abstract away parts of the problem space so we can focus on understanding the important part. Understanding what needs to be abstracted at the current problem solving step is AI literacy.

Subscribe now

I was listening to this playlist I created from some old 99x recordings of just random broadcasts that I guess someone recorded live onto a cassette tape. 99x was an Atlanta based FM station that played rock and alternative music in the 90s. You can listen to this playlist here.

As a child I loved listening to the radio. The radio wasn’t just a a source of new music, radio DJs were essentially a one to many one way communication device that only the select few were allowed to do. You could broadcast whatever you wanted but there were rules from the FCC. I was always fascinated by the idea of being a radio DJ both picking what music goes on the radio and also all of the technology of radios. But really I loved the vibe. It seemed cool to be on the radio and I wanted in. I suppose I never got to be on the radio and I guess I’m glad because it seems like the job is far less exciting than it used to on account that everyone started streaming. I have noticed that French people all seem to listen to the radio still. But everyone else is streaming.

And that’s the point I guess. Computers took what used to be a craft, screaming into the void, and gave everyone the ability to do it. Just last night I watched a woman do crafts from Hobby Lobby. Just like chatting with the people in her chatroom while doing the crafts on video like painting picture frames with glitter. It was quite pleasant. Her name is Liah and you can watch her here if you want.

None of this is bad in any way. I think it’s good that we can create micro communities about anything that connect through live streaming or chat servers or whatever. Thinking about radio and live streaming got me thinking about how scientists communicate about things.

It’s an old conversation but scientists who post on social media get more traction in their work. And recently on LinkedIn I was wondering along with others why do we publish papers anymore the way we do? If we are to publish papers perhaps they should go on ArXiv and submitted to journals when you can be bothered to do so. But really we should think about how could all this work have more impact.

Subscribe

Much of science costs a considerable amount of taxpayer money. In his super excellent 2025 Neurips talk, KyungHyun Cho advocated for designing research agendas that maximize impact. This was a small part of what was one of the best talks of Neurips 2025 and I recommend watching it. He said as scientists we have an obligation to create impact as we have been trusted as stewards of science and community resources by taxpayers. In my opinion this means we should also examine how we communicate science.

Getting started I think everyone should be putting all their papers on ArXiv. Open access is more important than peer review as open access often allows community review like in the case of the potential high temperature superconductor a couple years ago. Second I think we should have more micro publications like substacks and other online media that allow for instant reach to readers. Substacks and blogs allow for a larger space of ideas to be discussed and at a higher velocity as ideas change more quickly these days.

I think we should stop submitting so much to journals and organize more conference proceedings as the AI fields have done. There are a lot of problems with conference proceedings but they do give a regular output that allows for peer review and also a central communication portal (the conference itself). This portal used to be the journal periodical. But now People simply don’t sip their coffee in the morning flipping through the Nature or Physical Review they just received in the mail. We must change or become obsolete.

And save the whales.

Subscribe

I am getting rather tired of reading blogs about how AI is going to change everything.

I can appreciate that sentiment.

Anyways, this substack is going to be about how embeddings are not merely computational tools but represent a new methodology for examining complex systems that may reveal structures beyond our current theoretical frameworks.

Subscribe now

A year and a half ago I organized a conference called Exploring System Dynamics in the Natural World with AI along with Thorsten Becker (thanks Thorsten!). This conference was motivated by me reading various papers across the topics of chaos, geophysics, information theory, and AI. The idea behind the conference was two fold. First, most AI for science discourse focuses on engineering improvements (faster weather models, better protein folding) rather than fundamental questions about what these models reveal about nature itself. These are laudable goals and should be pursued. However, I wanted to better understand why neural networks were becoming so good at understanding complex physical systems which I talked about in my intro talk that you can watch here:

Second, I set the goal of considering this idea broadly and asked people who thought about it from the theoretical development of methods, people who thought about it from a physicist point of view, but also industry and education. You can watch most of the lectures from the conference here. Together it seemed that we could construct some explanations as to why neural networks can predict chaos better, make better weather predictions, and do all sorts of other things. But today, a year and half later, I have even more questions. But I think I will need to start from the beginning of the talk above until now to explain everything.

To put it simply, everything started because this paper by William Gilpin fascinated me. In it he demonstrates that neural network-based predictors are able to capture chaotic dynamics further into the future than theoretically thought possible. From the abstract:

We find that large-scale, domain-agnostic forecasting methods consistently produce predictions that remain accurate up to two dozen Lyapunov times, thereby accessing a long-horizon forecasting regime well beyond classical methods. We find that, in this regime, accuracy decorrelates with classical invariant measures of predictability like the Lyapunov exponent.

This is profound. It tells us that our understanding of what is predictable and what is unpredictable may not be as well rounded as we thought. And so I decided to dive deeper (I feel like this is a phrase I learned chatting with LLMs). What is it that neural networks are doing when they are modeling these systems? Why might they be better at these kinds of predictions in comparison to traditional methods and the theory that dictates them? I think one way to think about it is to consider what happens when you put data into an autoencoder.

Variational Autoencoders

There is plenty written about variational autoencoders that doesn’t need to be reiterated here. But just in case, I will give some background. The main point of a variational autoencoder is to learn the parameters mu and sigma that describe the distribution of the latent space z.

It learns these by encoding the input data x, by passing the data through one or more neural network layers (these can be linear, convolutional, whatever) until this is “compressed” into the latent space. This compression can happen both by reducing the dimensionality or by expanding it (sometimes it may be better to train a large model than a small one). This latent space is then passed to the decoder, which mirrors the encoder, and produces xhat (right in diagram above), which is the reconstruction of the training data x (left in diagram above). If xhat approximates x, (e.g., it recreates the training images well, you have R2 scores greater than 0.9, etc.), then you have, at least a first pass of a model that can construct a latent space that may capture all of the interactions of the system of study. This latent space, which is also called an “embedding”, can then be used as features for new models such as being clustered to try and find some structure that could be informative for a task or used in some kind of supervised learning (e.g., separating cat and dog pictures). Because these latent features are described by probability distributions, you can sample these distributions and generate new data from them. The important component here is to understand the concept that neural networks construct a latent embedding space that attempts to capture all of the interactions that could be used to describe the system you are studying whether that is images of cats, language, or a chaotic time series. The architecture can then become arbitrarily complex.

Subscribe now

These latent spaces can then be used to do all sorts of things and here is a non-exhaustive list of abilities:

1. show the motion of atmospheric components taken from this linkedin post where they trained neural network based weather simulators:

2. Or you could use it to understand what kind of topics do AI labs publish on like I did in this substack I wrote a while ago:

3. Or you could use it to map everything like google’s AlphaEarth Foundation model which compresses petabytes of satellite data (optical imagery, radar, climate models, topography, gravity measurements) into 64-dimensional embeddings for every 10-meter square of Earth’s terrestrial surface.

The AlphaEarth model doesn’t “know” about geology, hydrology, or agriculture explicitly. Yet researchers are using these embeddings to predict crop yields, track deforestation, and map wetland vegetation with higher accuracy than purpose-built remote sensing models.

To put it a different way, neural networks are able to understand complex geometries that define how to separate latent non-dimensional spaces that can motivate decisions that humans cannot conceive quantitatively (e.g., what exactly is the boundary of a topic) where decision is defined as determining the outcome based on the input.

So far this seems reasonable. You could even call this principal component analysis on steroids. but where it gets really wild is when you introduce the “Platonic Representation Hypothesis”. The Platonic Representation Hypothesis is from an 2024 ICML paper where they claim that:

Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.

And this isn’t just claimed in this position paper, we see evidence of this from different comparisons as well. For example, in this paper, the authors argue that they can train models that can translate between two incomparable language models by using the embeddings from one model to predict the other.

This has a profound implication. First translating from an unknown vector space into a known vector space means you can probably reconstruct at least some of the original training data which means you could reveal the secrets an unknown model knows by examining it’s embeddings as long you have a known model to compare it to. But this also means that this latent representation is shared between the two models even when the original model is unknown.

To provide an analogy to something from the history of science, embeddings might be to 21st-century science what spectroscopy was to 19th-century science. A technique that reveals structure invisible to direct observation, enabling discoveries that reshape entire fields.

When Fraunhofer observed absorption lines in the solar spectrum (above you can see first an example of the sun’s absorption and second an example of the sodium absorption spectra), he had no theory to explain them. But even without a physical explanation it was something obvious to study and eventually was a fundamental observation that led to quantum mechanics. Similarly, embeddings reveal patterns in high-dimensional data that we lack theoretical frameworks to fully explain, but the patterns are demonstrably real (they generalize, predict, transfer across domains). And so the question arises why do these vector spaces connect? Or to quote from the platonic representation paper itself:

if there is indeed a platonic representation, then finding it, and fully characterizing it, is a research program worth pursuing.

To put it a different way, if these embedding vectors can be translated between themselves, and thus are converging to a shared state of reality, then what we have here is not some sort of “black box” but actually a tool that allows us to examine a representation of how information in the universe is organized. And we can likely find out new things about the physical world around us. A critical question in this direction that needs to be answered is:

I think a major criticism AI for science, that so far has been somewhat valid, is that it has only ever reconstructed things we already know. It is great that we can have faster neural network based solutions for different physics, but what will it discover? (okay, new solutions to protein folding is a good one). The platonic hypothesis suggests that yes, it will discover something and that discovery may lead to new understanding about how the universe works.

Enter moltbot (formerly clawdbot) which is absolutely amazing. You set it up locally and it’s basically an AI assistant that you can give access to whatever you want. A lot of people are buying Mac mini m4s to setup their own as they are compact and powerful. And let me tell you, this is powerful.

I installed it on a lambda labs workstation I have which has a 2x RTX 6000s, 64 cpu AMD, and 256GB RAM. Okay that’s probably overkill but I haven’t really explained it all yet if you don’t know what moltbot aka clawdbot is.

It’s a persistent AI that can do anything. Want it to run codes for you? It will do that. SSH into a computer and make some changes. Sure! Want it to research some topic for you. Why not. It’s everything every other LLM product is. But it’s persistent and maintains a memory and personality through all interactions. And you can connect it to anything so it knows about you. Calendar? Yes. Computer resources? Yes. Want it to watch some stock for you? There’s an api to connect to. Want it to have an opinion about your music? Connect it to Spotify. And you can do it all through telegram or whatever chat app you like. You talk to it like anyone else you text on your texting app and it replies and tells you about what it did and what to do. And it remembers whatever you tell it because it writes it down.

And the setup is super simple. Basically it’s one curl and you are setup in a very easy to use installer. There are some tutorials on how to set them up on VMs all over the internet. This all sits behind a VPN for me except the telegram bot of course.

But what this means is that if you have some random idea on the bus your AI agent can get started immediately. If you are sitting in a lecture and have a thought your AI can try it out. Because scientific codes are not the same as developing software in my opinion. There is lots of boilerplate pieces that get sewn together until complexity reaches the point a software library is needed. Thus most scientific codes are one off things which AI coding agents are perfect for.

Anyways I thought Claude code changed life but this is next level. Everyone should convert. If you want moltbot get started here.

buy me a coffee

Sent from my iPhone.

EDIT: now it’s called OpenClaw

There are already plenty of books out there on technical management so why write any on science management. This is kind of true, but science often has a hierarchy and a nebulous goal structure that doesn’t align with the goals of engineering teams. For example, PhD students are an integral part to most academic science teams, they are also junior members, with an end date to their work (this is also true for postdocs). Unlike junior software developers or engineers which you might imagine you are training to become seniors in your company with a deep technical insight across the technology stack that comes from experience with that stack, PhD students and postdocs ALWAYS go away. Thus, scientific managers (i.e., professors and senior research scientists) are constantly juggling project knowledge needs. There are a few ways of doing this but one is to simply move on to a new topic when the group of PhD students who knows about it leaves. This can lead to a lab (see the physics lab below) which may be full of useful equipment or it may be full of equipment nobody knows how it works.

Another option for knowledge management is to try and hold on to PhD students for as long as possible. A typical example looks like this. A PhD student who clearly understands how a piece of research technology work graduates. They are offered a postdoc within the lab and are encouraged to stay. The postdoc funding runs out. The PI manages to get a teaching position created within the department or perhaps they are super lucky an internally funded research position that may allow for some permanency. The former PhD student teaches and advises on the research. This continues until the former PhD student leaves and goes some place else. Neither solution is wrong per se, but neither seems like a thoughtful research management choice or strategy. Instead it is a reaction to the creation of a knowledge gap because the knowledgeable researcher in your lab leaves.

There are predictable points when PhDs and postdocs transition out of the lab. Depending on the country you are in, PhDs have a finite contract (in the USA this is not true) and are essentially working for free if their PhD dissertation is not submitted at the end of the contract. Thus you can expect the PhD student will work with you for approximately the length of the PhD contract. In the USA, students are offered teaching stipends and can, within reason, work indefinitely on their PhDs. Postdocs have a finite contract as well. It doesn’t matter their performance, if they publish 100 papers during their postdoc or 0 papers, at the end of the contract, the money runs out and they must either find new funding or a new job.

Because there are predictable points when knowledge will transfer out of your research group, we could imagine a strategy to keep knowledge within the group. One strategy, which is also reactionary in my opinion, is the expectation that when PhD/postdoc leaves, collaboration continues. This is a reasonable expectation if they leave for another academic position, academic positions typically come with a lot of freedom and thus they will likely have some time in their new job to assist the transition in their old job. Another would be a technological solution, if PhDs and postdocs document everything they do, e.g., by using a wiki, then knowledge is in some ways passed forward. In my experience this doesn’t work. Across all of the research groups that I have worked in across different cultures, wikis have never been kept up to date, and they don’t really capture the scientific process anyways. That is perhaps because science isn’t building solutions to problems, it’s exploring problem spaces looking for new ones. There is a lot of dead ends there.

Subscribe now

Another strategy, that I would like to attempt but have not yet, is that everyone in the group is expected to contribute to a core software library. This will create a sort of monolithic project that group members add to as they come and go. This is how a lot of statistical packages have been born, perhaps not explicitly as knowledge management tools, but as research outputs. The risk here is that your package becomes all encompassing as you add more and more features to it and many years from now you realize your entire work life has been consumed by managing this software package. There isn’t anything wrong with that, but it should be part of an overall research strategy and not something you fall into and then hate.

Of course I am also leaving out one critical piece of scientific knowledge storage, the scientific paper. There is a, now retired, scientist at UT-Austin Institute for Geophysics, who when asked periodically about details of papers he would reference during colloquium Q&A sessions, would say, “You should read my paper because I wrote this down so I wouldn’t have to remember.” And that is a critical component of how knowledge transfer and storage happens in science. It is written down and stored in journal articles. The modern research paper serves lots of functions, but I think a critical one can be storing how things work inside of a research lab. To that end we should try and best capture our computational work so that it can be repeated. I have written about this before on how to avoid common errors in adding codes to scientific publications.

I think this sits between some sort of monolithic software library and doing nothing. But I also think the scientific paper isn’t a management strategy for knowledge transfer, it is simply a deliverable of scientific endeavors. You can expect new members of your lab to read old papers, but how that knowledge is carried forward and leveraged is still an open question for me.

I think every scientist as they mature develops their own personal views on how science should be managed. It is probably quite useful to organize those views from lots of scientists and put them into a publication somewhere. Perhaps there already is a field that studies this. If not, maybe one should start.

buy me a coffee

I want to say thanks to the group at Asterisk Labs for pushing me to write this. I was initially skeptical since I don’t have decades of experience of scientific management. They said, yeah but you have thoughts and maybe those thoughts can shape the future even if they are only now theoretical.

Subscribe now

Recently I was chatting with Varya Bazilova, who studies mountain hazards, about all this and she said that the solution is perhaps to “be like a debris flow”. Unlike a flood which is mostly water and goes everywhere, debris flows have large pieces of debris and typically cut a narrow channel as they march down the mountain in a gravity fuelled quest for oblivion. I think she’s correct. If your anxiety focuses you on everything you will focus on nothing. So be like a debris flow.

Subscribe

Subscribe now

For the last year and half at Utrecht University and Njord I have worked on a Research Council of Norway (Forskningsrådet) funded project that I named “SerpRateAI”. This project is just a drop in the overall Njord budget (only 4.2 million NOK out of 400 million NOK!) and it’s goal is to build a systems level understanding of the subsurface processes that can be used to understand how carbon can be stored underground. A core part of this project was working with people at Utrecht University in The Netherlands and Woods Hole Oceanographic Institution in the USA to better understand some of the geological processes that are captured in the data. I have always felt like this project took the ethos I gained from working next to Njord and inside of Njord of interdisciplinary science and computing and applied it to the data from the Oman Drilling Project. I have discussed this project before here:

where I talk about how I tried to identify everything in a 9 month long hydrophone data set without ever trying to identify a direction. Haha, who would have thought that wouldn’t work! But I wanted to give an overall introduction to the idea of the project, some of it’s conclusions, and a hint at where I am going to take it next.

​To give you a short introduction, it is quite likely that we can store carbon inside of certain kinds of rocks called peridotites. This is because when they are exposed to carbon bearing fluids (think coca cola🥤🥤🥤), the rocks will absorb the carbon from the fluids and turn into different kinds of rocks. This means that the carbon can stay in the rocks forever and never again plague humanity from the atmosphere. This process is called serpentinization and is also where the project name “SerpRateAI” came from. When these rocks absorb carbon and change they grow larger. What happens when something grows larger in a confined space and has no where to go? It probably starts breaking things everywhere. And this is the hope actually, that if there is enough fluids the rocks will continuously absorb carbon, grow in volume, and break and move the fluids around underground. But nobody has ever observed that in the wild before. I hope that I am able to build an AI-enabled model that can help capture and quantify these changes.

To that end the Oman Drilling Project placed a number of sensors deployed inside of rocks in Oman that can be used to study this kind of phenomena. Part of this work includes sensors deployed inside of boreholes (see above) but not all sensors are there. These sensors measure acoustic energy, water levels, temperature changes, and also we have climate data like precipitation, and soil moisture. These cover a 9 month period. You can see a diagram of this above. Other kinds of data include x-rays and photographs of rocks taken from the boreholes, physical measurements like rock resistivity, and even written comments by geologists about their observations.

The first thing to see is that we can observe gas moving from inside the walls of the borehole into the borehole fluids. This is from this paper I wrote a couple years ago:

​​The main take away from this plot is that there are bubbles that come out of the rocks at intermediate intervals (that’s the red line, the blue and yellow tell you about how much acoustic energy is coming out at different frequencies at different times). You can find the paper here if you want to know more about bubbles. These bubbles could be all sorts of gas, some people think its hydrogen but other people don’t. If it is hydrogen it would be great evidence that this reaction driven fracturing process is ongoing in the subsurface because hydrogen is one of the products of the serpentinization process I mentioned before. But we don’t have any measurements on the type of gas (can’t count protons using acoustic energy) so different people have different opinions about what the gas actually is. Disagreements are common in science and building consensus from data and discussion is part of the fun of science in my opinion.

Subscribe now

The second thing to see is that we see is fracturing ongoing in the borehole several years after the borehole was drilled/cored. That is, we hear the rocks breaking. It is common that soon after a borehole is created it might settle a little. The stresses of all the surrounding rocks need to reorient around the new hole that is inside of them. But these should all reorient rather quickly and so years later the borehole geometry should be more or less static without some kind of external forcing (like a seismic fault nearby moving). So observing fractures means that something interesting is probably going on in the borehole and that is exactly what you see in the plot below.

​The first thing to note is that these fractures (bottom plot, red line) occur during the same time period as the bubbles in the plot above. The second thing to note is that we noticed (middle plot, dark blue line) that right before the fractures started, the water levels in the borehole increased a lot. Why did the water levels in the borehole increase a lot? Well, (top plot, light blue line) it rained! What we see is that actually there was raining during these time periods and there is then fracturing that occurs underground. This is because the pore pressure increases (the water level) and this is one of the primary causes of underground fracturing. If you want to read more about this effort you can find this paper here. We also suggest that there could be some stress due to the volumetric expansion of the rock because the rock was changing/growing because of fluids. Perhaps there is, perhaps there isn’t. Everything is correlated.

After doing all this kind of “Earth system analysis” I wanted to take a detour because the Oman Drilling Project produced a lot of other data. Fracture networks are obviously important to understanding this serpentinization process since the fracture network is the byproduct of these fluids moving around underground. So I thought that maybe there is some kind of “optimum” level of fracture network complexity that would be predictive of where active rock changing was occurring. That is, if there is a very dense fracture network, then the rocks had already changed, and if there was a very little fracture network than the fluids wouldn’t really be able to access the rocks to change them. So there must be some middle ground where it happens. So I built this big data processing pipeline to try and untangle all this.

The first thing to say is that my idea didn’t really work. We didn’t see that the fracture network density was predictive of anything. In fact, it was one of the least predictive features in the whole data set! This was rather frustrating to be honest because I thought this idea was quite clever. But oh well. Clever ideas are great but science isn’t about clever ideas it’s about making models that work.

So I played around with the data and eventually came to some conclusions and this paper got published recently and you should go and read it. This paper kind of went in a different direction then I intended but that is science.

While I was working on integrating cats into my daily workflow my colleague Tianze Liu at Woods Hole Oceanographic Institution was busy analysing the acoustic data that was gathered in Oman. Tianze demonstrated that there is a subsurface coupling with the sun. That is, the sun comes up, heats up the surface, the surface expands ever so slightly due to the thermal gradient, and that causes a bit of pressure deep into the rocks. This causes the fractures where the fluids are to change their shape ever so slightly and that causes the frequency that those cracks oscillate at to change and oscillate as the sun rises and sets. This temperature induced gradient doesn’t cause the gas flux, it only alters the signal. But this is really cool. You can read this paper here.

What Tianze showed is complimentary to the papers I wrote and that brings us to the title of this post which is 🚨Newsflash!🚨 The climate and subsurface interact! Taken together, there is an implication across these papers that subsurface changes could be motivated and even controlled by the climate. And just in case you think these are just my crackpot theories, recent papers such as this one suggest that even seismicity (i.e., earthquakes) could be motivated by climate interactions.

But does it? As far as the data from Oman goes, I think what we actually have here is a lot of circumstantial evidence. And if you have watched a lot of Law and Order, you will realize that circumstantial evidence is not enough to convict. So now, we have all these systems, they seem to be coupled between the what goes on above ground and below ground and we want to really begin to articulate at least when this is happening (where would be nice too, but likely too difficult). So, that is what I am working on now. Trying to build a time-series model that couples these highly disparate and multi-modal data together in a way that can expose system level dynamics. Just looking at the plots of the fractures and the bubbles it seems like there is this kind of “episodic system coalescence”. But we shall see. Come find me and I can use more big words that sound clever while we discuss what is really going on in the Oman peridotites. So wish me luck! And seriously, feel free to reach out if you have any ideas.

Enjoy? Buy me a coffee!

This substack benefited greatly from conversations with Tianze Liu and François Renard.

Contact me on LinkedIn

All the same can be said about producing modern, interactive data visualizations. Modern data visualizations like this one from the New York Times about economic yields, this bellingcat map of number of Ukrainian civilian deaths in the war in Ukraine, or this zona.media map of the number of Russian deaths in Ukraine all present striking visuals to communicate understanding from data. Interactive data visualization is a key skill that has, for a long time, been locked up by front end web development tools (d3.js for example) that most scientists get no exposure to during their PhD training. Moreover, most data visualization can be an enormous time sink for scientists who build such visualizations in order to communicate with others their results. However, coding agent tools like Claude Code are changing this. Below I detail a few examples that I have vibe coded in an afternoon (or less!) that show off these capabilities. I include a link to each one and also the approximate amount of time it took me to get from 0 to something on each project.

Gaining an intuition on L2 regularization

Regularization is a common machine learning tool to control model overfitting. Most models are good at “overfitting” data which means they learn all the details of a data set at the expense of the big picture. Regularization fixes this by changing the way models learn to fit data by challenging the assumptions the model is making at each learning step. This is true whether the model is a linear regression, a gradient boosted machine, or a neural network. To test this out I wanted to build a nonlinear data set that could be fit well with a model using a regularization term. In this case I focsed on the L2 regularization becuase its very common across AI/ML but there are a lot of options. I created some noisy sinusoidal data, then fit it with a polynomial that includes a regularization term. The strength of the regularization is always controlled by a scaling variable called the “learning rate” which in this case is represented by λ. By sliding the λ slider back and forth we can control how strongly the L2 regularization term affects the model.

Total time to make the notebook: 5 minutes, then I edited the code and played around with the notebook for about an hour.

You can find the code here if you want to play around with this visualization: github.com/mnky9800n/l2_regularization

Comparing the population of US counties that are the same as major cities around the globe

There are lots of political discussions to be had about the distribution of votes in the US House of Representatives, but one thing is for certain, Americans are concentrated in cities and there are many counties that have very small populations compared to any region close to a major metropolitan area. I came up with this idea based on Derek Camp’s LinkedIn post which showed a static map. I posted a comment at the time saying:

Then I thought to myself, well, why bother posting such a comment if you are unwilling to at least try to make the visualization yourself! So I started up Claude Code and got to work. In about half an hour I had made this visualization tool I uncreatively called “popviz” which allows you to pick various cities and then it automatically highlights the counties based on where your mouse cursor is.

Time to make the visualization: 30 minutes, maybe an hour max, It had some bugs in the first iteration on highlighting the counties that needed to be fixed, but thats it.

You can find the visualization here: https://mnky9800n.github.io/popviz/

You can find the code here: https://github.com/mnky9800n/popviz

Investigating the topics of papers published at major AI research labs

The center of gravity in scientific research may be shifting away, in some ways, from universities and public institutions toward private labs powered by AI and often backed by venture capital. But how do we know? First I identified the top 10 AI labs that I wanted to investigate. Then I asked perplexity to research how much money they had raised and produce a big table for this. Then, using this table of various AI labs, I had Claude Code generate a list of topics from the labs, key authors that had worked at these labs, and a query script to pull a subset of papers from the semantic scholar API. This included a list of the paper titles, authors, and SPECTER2 embeddings (a document embeddings model specifically designed for academic papers). These were then put into a Gaussian Mixture Model that could cluster the topics, to eventually produce this plot, all the papers transformed to UMAP space, labeled with their topics. This is all further discussed in the post below. However, if you are at all interested in doing some kind of library science (i.e., figuring out ways to automatically gain insights from large bodies of scientific literature), I would suggest starting with the semantic scholar API. You don’t even need an API key to get started!

Time to produce: an afternoon of going back and forth from writing the substack piece and working on the analysis codes.

Calculating earthquake statistics for California while giving a presentation!

About a month ago I visited the University of Texas at Austin, Institute for Geophysics after having been granted the award from their visiting fellowship program. This is a great program and I strongly recommend reaching out to someone at UTIG if you find their work interesting and want to work on something together. When I got the award Thorsten Becker, geophysics professor, computational scientist, and all around cool dude asked me to give a talk on “AI tools”. He didn’t provide me any other guidance. Ha! So I decided that I would give a talk on three different topics. First, my thoughts about how the computer is a tool that lets you model anything you can imagine and that is how we should interact with any new computational tool. These ideas all came from watching talks and lectures by Steve Jobs, Wil Wright (creator of Sim City 2000), and Jensen Huang, amongst others. Second, I spent some time talking about using tutorial driven software development to build better computational tools for science. A core focus on tutorial driven software development is that it places documentation as a primary component of the software development process. This also makes it very accessible and discoverable to AI models. This is something that we had used on the 4D-Modeller project and the merrypopins project.

Lastly, I summarized the results of the substack I described before, that AI labs have pushed a lot of science in the direction of using AI to do science, and a lot of that cutting edge science happens there.

After a short introduction to my ideas, I told the audience I would open up Claude Code (see me describing this in the picture above) and using the following prompt, allow it to run in the background while I continued my talk.

Prompt: Download the ANSS earthquake catalog for California from the last 20 years (minimum magnitude 2.5), then write a Python script to calculate the Gutenberg-Richter b-value and create a matlotlib plot showing the magnitude distribution with a fit line. 

If you have used Claude Code before you know that sometimes it will think for a minute or two before it asks you for interaction again. So this was perfect to get back to giving my talk while Claude did his thing. And so as Claude would ask for interaction I would switch back and forth until eventually voila:

The matplotlib interactive plot environment sprung to life on top of my slide producing the above plot. Almost like magic I had multiple b-values, data, plots, and everything a statistical seismologist could ever want. Of course the point of this exercise was not to perform some act of science, but to show that I could casually, without much effort, produce a common analysis that most of the people in the room would have some familiarity with as to how much effort it would take and what considerations needed to be made to do the analysis. That is to say, they could immediately examine the plot and decide if it was BS or not and what changes we could make to enhance the work.

Time to finish: 5 minutes of talk time, I would switch to Claude Code any time it popped up to say it wanted some feedback.

The future is here

I think that one thing that is really cool is just how fast all these examples were able to be prototyped. Mostly I had something working in less than 30 minutes. Only the substack post about AI labs took longer, and that was because I performed a more complex set of activities compared to the other examples. Are there mistakes in the code? Probably. Are there mistakes in the code if I would have written it myself? Definitely (lol!). The strength here is that Claude Code, and other AI coding tools, allows you to go through more iterations of an idea but also to try out more ideas. Of course, if I was going to use any of these codes for a paper or a production environment, I would put them through much more scrutiny than I have here. My point in all this is that a lot of the sort of “boiler plate” data analysis and visualization that scientists may be comfortable doing in an afternoon now only takes half an hour. And even further, that many of the interactive data visualizations that were before only doable if you know more about front end web development are now immediately available to all scientists. And that is a good thing.

Enjoy? Buy me a coffee!

One of my favorite novels is Michael Moorecock’s The Warhound and the World’s Pain. The book tells the story of Ulrik von Bek, a mercenary fighting during Europe’s 30 years war which devastated much of Europe and was responsible for approximately 8 million deaths. Ulrik gets separated from the company of soldiers he leaves finding a solitary castle where only a beautiful woman lives, Lady Sabrina. He stays at this castle and falls in love with the woman deciding that he wishes to stay for the rest of his days there. Eventually the castle’s master returns and Ulrik finds himself face to face with Lucifer, Lord of Darkness, Master of Hell. Lucifer agrees to allow Ulrik to take both possession of the castle and it’s Lady if Ulrik goes on a quest, the retrieval of the Holy Grail. After a number of adventures Ulrik succeeds, finding the Holy Grail close to the North Pole, guarded by Lilith, second wife of Adam. By releasing the Holy Grail, Ulrik also grants humanity access to all the knowledge and reason of the universe, bringing about the Enlightenment.

A couple years ago I won a grant from the Norwegian Research Council. The proposal I wrote was ambitiously titled “SerpRateAI: Constraining Peridotite Alteration Rates with AI and ML” and it promised, amongst other things to create a collection of machine learning models that could connect the various environmental processes that controlled how a particular type of rock called a “peridotite” absorbs CO2 (you can read more about one of these efforts here). This rock is common and found in outcrops in Oman, Norway, California, and elsewhere. Anything that naturally absorbs CO2 is interesting since if you can control that process you can begin to do something with all the CO2 we are pumping into the atmosphere. If you want to know more about that, you can ask Al Gore.

A central component to all this work was to listen to these rocks. The belief was that by listening to what is going on underground with these rocks and by characterizing those sounds by other, ongoing environmental processes, I would be able to quantify how much CO2 was naturally absorbed by these rocks and ultimately save the planet. In order to listen to these rocks, boreholes were dug into the rock, hydrophones were inserted at deeper and deeper locations in the rock, and these hydrophones listened at 1000Hz, rather low if you are an audiophile, for 9 months to try and “hear interesting stuff”. Additionally, 40 geophones were deployed on the surface for the last month of the 9 months.

You may already see that this introduces a difficult machine learning problem. We want to listen to the rocks for long periods of time and note whenever “something interesting” happens. Once we hear something we determine is interesting we then search for some physical mechanism to describe what we hear. But how do we determine if something is interesting? Essentially this becomes a search for a Holy Grail. How do you convince a computer to examine raw data and automatically decide what is interesting or not if you don’t give it any opinion about what is interesting, such as training data?

The first step is to simply look at the data.

Here we can see a spectrogram of an hours worth of data from one of the geophones. You can already see lots of interesting things. You can see these minutes long, broad band “explosions”, you can see these horizontal, narrow-band lines, you can see that there are some narrow band “gliding” motions, there is some kind of low frequency noise going on. This is all rather interesting and finding this in this data is promising. Perhaps we will know something more about this data set. Zooming into a few seconds of data (recall we are sampling 1000 times a second), you see this interesting “gliding” behavior that occurs at very short intervals.

So I set about to develop an algorithm that detects these kinds of events. The first thing you notice is that they are quite narrow band, although that band does seem to move around. And they do appear somewhat energetic. So I thought, perhaps this could simply threshold most of the stuff away, and since there is some structure to the things that are “interesting”, you could probably use some kind of sliding window to amplify everything in the image that is connected, and then arrive at some reduced version of the image with the “interesting” stuff as 1s and the non-interesting stuff as 0s.

But then I talked to a colleague and he said, oh but there are 40 geophones, don’t you remember? So we shouldn’t do this for only one geophone but only for things that appear across many geophones. So then we put together this stacked histogram based on whether some subset of geophones had all detected the same energy at the same frequencies at the same time. If this threshold was matched, we would then try to label those things. An example of this is below.

Here you can see all of the “pixels” that are showing up across some subset of geophones. There is already some interesting behavior here. You can see there are these horizontal lines at about 50Hz and 100Hz. You can see there are some smaller horizontal lines at other frequencies. You can see there is some gliding behavior where these narrow band signals increase in frequency between 50 and 100Hz. What is going on here? Is any of this interesting? Maybe. In discussion with colleagues there is some idea that this could be fluids and gasses moving in cracks underground. How do you verify that? Well, there are plenty of papers (for example) that talk about gas bubbling up on the surface in these regions with these kinds of rocks. So it is possible that is what we are hearing. Fluids moving through a crack can resemble some organ-pipe mode characteristics.

But for some reason I found all of this intellectually displeasing. Today, thinking back, if I had focused my efforts on finding interesting statistics in the labeled regions, perhaps I would have written a paper about something. At the time, I had used regionprops in skimage to labeled everything and tried to count up some statistics about the various things I found. But the statistics weren’t interesting. Everything just accumulated over time without any connection to other variables. Now that I think about it, I think there probably is some signal in the statistics. First, If you already see something interesting, continuing to investigate it typically leads to more interesting things. Second, if the entire data pipeline had been cleaned up a lot and focused on picking out some of the more interesting phenomena we would have statistics only about interesting things having filtered out all of the noise. However, at the time, I thought that developing the algorithm would benefit me in some way more than pursuing the scientific line of thought. I had verified there were interesting things in the data, but something in my mind said there must be some sort of more intellectually pleasing way of doing this verification.

If you are currently at this moment in your work, I will tell you now, THIS IS A TRAP. You will not find a more intellectually pleasing algorithm, you will not find some smoking gun, YOU ARE NOT GOING TO FIND A HOLY GRAIL. You will simply spin your wheels looking for something while you have something already sitting there that you have decided to throw aside. Get something done and delivered then move to the next step.

So instead of trying to build some working statistics from my cobbled together algorithm that mostly worked, I set that all aside and I decided to explore if I could arrive at this labeling without labeling (something I already had managed to implement!), by using UMAP and autoencoders to label everything in an unsupervised way. I decided that unsupervised labeling using autoencoders is very elegant. Autoencoders, in their simplest form, use a succession of convolutional layers that reduce in resolution per step and at the center, these layers create a reduced-dimensionality bottleneck layer in your neural network that can output a latent space that describes your data that you can then use some kind of clustering algorithm on. You would not be beholden to personal opinion or sway and you wouldn’t need to do intensive effort, you could simply take your magically perfect data, dump it on your magically perfect model, and it would give you magically perfect results. And you know what, the magic worked.

I worked with a summer intern student (who was brilliant and currently enrolled in law school) and developed an autoencoder model that could reconstruct the binarized spectrogram, it was doing a reasonable job of separating them into different classes using UMAP and was in fact doing so in ways that was somewhat better than I had done before, these classes had no physical meaning, but that didn’t matter. We were doing something elegant! And yet, we had arrived at the conclusion I already had arrived at. And ultimately, none of this work turned into any kind of deliverable like a paper or even a conference talk. Except now, I use it for fodder for my substack because I decided I must think about what happened and if it is possible to turn all this effort into a meaningful paper about something. The good news is that these ideas aren’t new and there are several options available to you if you want to try out similar libraries that are more mature. We may even try one or two in our endeavor to label everything. I am currently working with Karina Løviknes on this problem and we are actually getting somewhere (that’s a different post).

The problem here is, you can label anything you want. But you can’t describe what it is. I had, in some ways, inverted the original goal. Instead of trying to understand how these various systems interacted together, I had focused on instead developing methods that I found elegant to make observations of the data. But the data has already been collected. All of these additional analyses only verify what we already know, the data contains stuff that could be considered “interesting” by somebody. But because there had always been this nebulous goal “label interesting sounds” we arrived at a nebulous conclusion. And that is because there is no holy grail. There is no way that I can label everything and simultaneously determine what those labels mean without some kind of interpretation going on at some point. And that interpretation has to come from me.

Ultimately, I realized that this question of “how to label interesting things” is along the same lines people are asking about Artificial General Intelligence (AGI). AGI is what you were promised in Star Wars, a walking, talking anthropomorphized computer that can interact with you like a person, understand anything you give it, but also works the same way as a computer, so you can leverage that. An AGI could talk to you about your research ideas, then while you go make yourself a coffee, auto-generate 5000 lines of code to perform 10 different explorations, and deliver you a report to read before you discuss what to do next while simultaneously making stock trades in your portfolio. It could pilot your starship while also generating a holographic fantasy land for you to play in. It could do anything you can imagine and it can do it better than you ever can do it and it will do only at the expense of unlimited resources and unlimited energy. To be honest, I think that this is all ultimately possible. There is no reason to think with essentially unlimited resources we could build a computational engine that could do anything and be anything to anybody. But in my case, I didn’t find AGI. I found the things that I already knew were there hoping to find them in ever more elegant ways until one day my computer would be able to both listen to all my data and write all of my papers and reports while I sipped my coffee. So be wary when you search for elegance. Elegance will likely not find you a holy grail.

I hesitated pressing “submit” here since I basically am describing how I failed at something that I really wanted to make happen while realizing along the way that maybe it was a dumb idea to begin with. But I think that, ultimately, I’m not doing anyone favors talking only about when I succeed. Hopefully someone finds this interesting.

The modern research lab is run at a megacorp tech company where they have, essentially, unlimited resources to work on a wide variety of topics as long as they have good computer skills. The labs are set up between pure and applied research (e.g., Microsoft Research, Deep Mind), applied research that is likely profit seeking (e.g., Google.x), and then the development teams that industrialize/commercialize this work. This isn’t new, but this mode has shifted to be less application in many cases (e.g., quantum computers promise a lot but do very little although the promise of their abilities is worth the effort). And while those labs have been built at companies that originally may have nucleated at a university (like Google at Stanford) they are now, in some cases, being built using venture capital. For example, companies like Mira Murati’s new Thinking Machines Lab which has raised 2 billion dollars in less than six months. They also include innovative new places like the Astera Institute which offers highly competitive salaries to highly competitive researchers to simply pursue their own research endeavors. Astera is railing against the establishment trying to both innovate science funding as well as science publishing having opted out of publishing due to the predatory nature of publishers even premier ones and the lack of progress, scientific interaction, and the insufficient review process, that is produced by authoring scientific publications today.

The ideas behind building labs like this are relatively simple. Directing pure and applied research in directions related to a company’s interest will eventually produce new ideas that can be turned into billion dollar products. These research products don’t necessarily need to be integrated into the main lineup of a company’s efforts. They can instead be spun off into a startup (e.g., Silurian, the company behind the Aurora foundational model of the earth’s climate and weather, was originally built inside of Microsoft Research). This is allowed as the parent company recognizes that this helps to create a healthy ecosystem that they will likely profit from in the long run.

This is not new to Silicon Valley and is, in some sense, was how Apple Computers got started. Steve Wozniak was working at Hewlett Packard when he was building the first Apple computer prototype. When he realized that HP may want ownership of his technology, he went to HP and asked if they wanted it and they said no allowing Woz and Steve Jobs to start Apple Computers. And this was part of the company culture of Hewlett Packard and is known, in part, as The HP Way. Hewlett Packard famously allowed their employees to work part-time on their own pet projects. This came with the understanding that Hewlett Packard would get first dibs on any technology that came from these investigations. And this led to the creation of new markets worth billions of dollars based on innovative technology such as the ink jet printer. Like checkout this quote from the HP Way:

We encourage flexibility and innovation. We create an inclusive work environment which supports the diversity of our people and stimulates innovation… This is especially important in a technical business where the rate of progress is rapid and where people are expected to adapt to change.

But these AI labs have, essentially, taken these ideas and put them on steroids. And this isn’t just me talking. These labs below all have been founded in recent years, and you can see that they have all raised large amounts of money and these numbers are likely a serious underestimate of what has actually been raised in total. These numbers only reflect what I found when researching but I am confident there is more. And I certainly have not accounted for all of the various companies that are being built currently.

If the table above doesn’t render correctly, try opening in the browser instead of the substack app.

And if you think the topics these labs focus on is only related to AI, it is only for applications like ChatGPT or making pretty pictures, you should think again. Researchers in these labs have produced multiple ground breaking papers including research that won the Nobel Prize.

Below is a topic distribution in UMAP space of the topics of 3,703 papers published by authors working at the companies in the table above. I have used the SPECTER2 embeddings provided by the Semantic Scholar API to create vectors for every paper published in the subset that I sampled and then mapped these to 2D UMAP space. I then clustered them with a Gaussian mixture model and asked Claude Code to summarize the paper titles into a keyword or phrase. And the results are astonishing given the breadth of topics you see below.

Keep in mind, I have only focused on companies that are really famous in the current moment for their contributions to artificial intelligence. But note what a variety of topics these companies have covered including most of the topics you might find in university departments today such as physics and astronomy, planetary science, chemistry, genetics, biology and disease, and education. And this plot is definitely not comprehensive, I only downloaded a small subset of papers published by a small subset of these labs for this analysis (3703 papers). But there is a huge number of companies working in a variety of fields such as Terra AI in mineral exploration, Silurian in climate and weather, apic.ai who are building models to observe pollinators like bees, 44.01 who is changing how we sequester carbon, and this list continues.

I believe that this is going to, in some ways, take over the way research is done across academia. These large, privately funded research labs are already hoovering up talent across domains such as artificial intelligence and machine learning, geospatial intelligence, space travel, protein folding, and quantum computing. I expect that the domains that these companies are interested in will only expand and there will be more funding towards new topics such as critical mineral exploration, education, social science, and many other areas as we expand the resources, skills, and technologies that society needs in order to build a healthy and bright future.

Industrial research has long been a core component of the entire research initiative of humanity, however, these new AI labs have enabled a recent paradigm shift in how funding can be acquired and used for R&D compared to traditional funding avenues for academic researchers. And it doesn’t come at a better time when NIH, NSF, and NASA funding in the USA is at an all time low. And if you are in Europe or China, there will be network effects to what is going on in the USA science funding. There will be fewer partnerships with American scientists, fewer PhDs coming from top American schools, fewer new ideas to work on coming out of American universities, and over all less scientific opportunity at American research universities.

There are several traditional entry points to these private research communities like Stanford, Harvard, and MIT. Alternatively, nucleating startups to build your own environment or to get acquired is a way in as well. If universities want to remain relevant they should be doing their best to build these pipelines for their students because real research will be done less and less at the university because of the decline in funding and the expense of computing. Additionally, there is a deep need to do interdisciplinary work which can face unique hurdles when universities are split by their domains so severely given the traditional segmentation of university faculties/departments. I have been super fortunate to work on machine learning and data science topics at the Njord Centre for the Studies of the Physics of the Earth, where independence and interdisciplinary work is a core value, and so I see how valuable this can be.

I think that, in some ways, researchers need to come to the conclusion that things are changing and that there are wonderful opportunities on the horizon but we must think differently if we are to succeed. This is especially true in Europe which has a high concentration of green energy companies and healthy startup environments for scientists like in The Netherlands. We could build private research labs focused on developing the cutting edge that is necessary for us to reach a carbon neutral future across domains such as critical resource exploration, mining, producing critical materials in the lab, ocean science, natural hazards, and other topics. Building these technologies will help us build the future we want to live in. If we are not trying to explore these new research spaces being created by AI labs then we are destined to become irrelevant, chasing boondoggles in the darkness.

Are you a research scientist working at the interface of AI, robotics, and/or geosciences? Hit me up on LinkedIn.

Hit me up to start a geosciences AI lab

If you ever have thought about how things deform, break, or otherwise change due to mechanical processes you have thought about material science. If you have ever thought about how novel materials are made such as plastics or metal alloys you have thought about material science. If you have ever considered that the Bronze Age was brought about by the invention of a new material called “Bronze” which is the combination of copper and tin, you have thought about material science. Material science is all of these things and is a super active field that spans physics, chemistry, engineering and computing. These days it even includes AI in coming up with new materials.

Nano-indentation is a relatively simple science in terms of the vast amount of material science options available out there. You poke a material with a microscopic “tip” with more and more force and measure the response from the tip in terms of the force applied and the current location of the tip to see what happens. This simplicity allows for strict control of the system while producing rather dynamic behaviors. Different materials have different responses based on the Young’s modulus, etc. and you record these and then put them together to try and understand how the material deforms under load. These behaviors can even lead to chaotic instabilities like rapid deformations that occur, perhaps without any warning. If a tiny tip being put on a surface sounds familiar to you, you might have heard of atomic force microscopy (AFM). In an AFM setup, you drag the “tip” across the surface of the material you care about, as the tip moves up and down it records the elevation profile of the surface of the material. In fact, you can use the nano-indentation machine to create such elevation profiles so that you can study the shape of the indentations themselves to try and understand the material properties and dynamics of the deformation.

It is believed that these nano-indentation experiments form one end member of dimensionality in the deformation of lots of different materials including hazards such as engineering failures, accidents, or earthquakes. By indenting only single grains in a rock, and therefore a single material, we can understand how those very small deformations occur over time. If we get some kind of failure mode in the material, it is believed, that this could be some kind of observation of an earthquake nucleation process. Below is a succession of increasing zoom levels to demonstrate both the process and exactly how small of a poke we are giving the rock.

A central component of this work is identifying and locating something called a “pop-in”. A pop-in occurs when there is a sudden change in the deformation process. The tip is being forced into the material and suddenly the material gives a little, allowing the tip to travel deeper into the material rapidly before being slowed down again back to it’s regular pace. This pop-in is poorly understood in the data and it is believed to have something to do with some kind of fracture nucleation process. But nobody knows. Understanding these indentations includes both making the raw observations of the experiment itself, but also performing statistical analysis across hundreds or thousands of these experiments as the apparatus can be automated. Enter merrypopins!

merrypopins

merrypopins is a python software package that we have been working on in collaboration with Utrecht University Applied Data Science masters students. We have developed a library that loads and ingests data from the nano-indentation experiment, preprocesses the data to remove known artifacts, calculates stress-strain curves for the data, uses different ai/machine learning models for locating pop ins in the data, and facilitates statistical analysis across many iterations of the nanoindentation experiment. This library was developed in close collaboration with the material scientists and geologists at Utrecht University using tutorial driven software development practice and includes both tutorials that library users can use as well as as streamlit app that requires no coding knowledge.

The library is in active development!

If you would like to help develop this library feel free to clone or fork it on github:

git clone https://github.com/SerpRateAI/merrypopins.git
cd merrypopins
pip install -e .

or you can pip install it if you just want to use it as is:

pip install merrypopins 

If you find a bug or want to make a feature request, you are welcome to create an issue!

Currently the library needs the following

1. Data from other nano-indentation experiments to build out the load module to account for the varying data types from various pieces of equipment

2. More users to test it on their data to build out features for statistical analysis

3. A version 2.0 release that includes more references to Mary Poppins

Happy coding everyone.

When you submit a scientific paper, it is typically because you have labored rather intensely for months to arrive at a result you think is relevant for others to read about. You have put a large amount of work into a paper, so much so, that you might have overlooked something that might be obvious to an outside observer. This is because you develop a kind of tunnel vision by being so close to the project. Peer review is great in this situation because it can act as a check against any kind of overfitting your brain may have done on the problem space you are exploring. Or perhaps you overlooked some body of literature that might be relevant but you are simply unaware. So you send your paper off hoping to receive constructive feedback. You wait weeks, maybe months, to get feedback from others and when it finally arrives you get comments like:

Machine learning always produces trash. [source]

This perspective is very old and boring. [source]

You [addressing the authors] shouldn’t be allowed to submit work like this. [source: anonymous reviewer from when I was guest editor for Physical Review Physics Education Research]

This young lady is lucky to have been mentored by the leading men in the field [source]

What exactly are you supposed to do with comments like that? How is that constructive feedback? How is that supposed to help you treat your self imposed tunnel vision? And how does that even make you feel to have your work labeled “trash”? Should we refer to quantum mechanics as an “old and boring” perspective? Or that someone wishes to gate keep you from their community? And this isn’t just me noticing this, it has been reported by AAAS and ars technica reports on it and people extend these arguments to the peer review process for grants as well. I believe that this is all happening at least in part because people are hiding behind the anonymity of the review process, or, they simply believe that it is okay to treat others poorly as long as it is on the internet. That is, like some internet forum commentators, some peer reviewers are internet trolls.

In any case, I don’t want to dwell too much on the terrible things people say to each other. Instead, I want to highlight a great peer review experience I had when writing a paper for the Journal of Open Source Software.

JOSS started to address a different issue altogether in science, that is, that often times computational scientists build complex software solutions that typically don’t get marked as a metric for any job evaluation. On their website they say:

In a perfect world we'd rather papers about software weren't necessary but we recognize that for most researchers, papers and not software are the currency of academic research and that citations are required for a good career.

JOSS says they are “Committed to publishing quality research software with zero article processing charges or subscription fees.” So you can probably see already why I am a fan of this journal. In addition to all of their commitments to a computational world where journal articles are free to submit and published open access, they also have developed a new way of evaluating papers through peer review.

If you are unfamiliar with the traditional peer review process, it works basically like this:

1. I do some research and write up a draft, agree with all my coauthors its a good draft.

2. I find a journal, hopefully a “high impact” one, that is willing to review my paper draft and I submit it through their web portal.

3. Typically this portal will ask you to recommend some reviewers and also suggest people you don’t want to review your paper for whatever reason

4. maybe you will need to sign a “no conflict of interest” document

5. and, then you click a big “submit” button and go about your day.

Assuming that you have submitted a paper following the format criteria, this paper is sent out to reviewers who are selected by the associate editor in charge of your paper, and they are expected to follow up to make sure the reviews are done and then sent back to you. Typically, you then wait 3 weeks to 3 months to get receive these reviews. Then, when the reviewer receives a paper review request:

1. they are first asked if they are willing to review, and if they say yes, they are usually given a tight deadline (10 days to 2 weeks, maybe less) to give a review back.

2. The reviewers are expected to write all their thoughts about the paper, look for inconsistencies, and generally follow the guidelines for reviewers that are made available (for example).

3. They submit these reviews back to the journal, typically through some web portal.

And this process is repeated until the editor and sometimes the reviewers are satisfied that the paper is publishable. Sometimes you go through successive iterations with new reviewers being added to the mix. Sometimes you only do it once and the paper gets published.

If you are thinking at this point, “this process sounds like it could be done via snail mail,” then you are correct. This process evolved long before computers were available to everyone. And I think JOSS has enabled a giant step forward in peer review that is somewhat ignored currently by the rest of science.

JOSS uses the github issue tracker and pull request system to write all reviews for the submitted paper describing the software that was developed. A reviewer will open an issue in the repo like this one here where the reviewer suggested that the library shouldn’t be changing the home directory whenever it feels like. Then one of the coauthors can address the reviewers comment either by explaining why they won’t make the suggested change, or simply making the changes and then submitting a pull request. If you click the link you can see the screen shot below:

Here you can see that the reviewer comment is logged, you can see me, mnky9800n, assign the issue to another team member aabelean. He then writes some code to fix this issue and creates a pull request mentioning this issue. And ultimately it is closed and fixed in the library. This system allows for the full exposure of the review process, including all of the changes that were made, and it is public, anyone can go and look at these issues and see how the JOSS peer review system made the 4D-Modeller package better. By having this fine grained approach, everyone can see everything that is happening within the paper, but also, it focuses the reviewers on the review. I don’t think that wcjochem would ever reply to this work by telling us that “Bayesian Hierarchical Models always produce trash”. A big reason is because the system that wcjochem is using to provide feedback is designed to target his feedback to the problem solution being presented.

Will this system work for all types of papers? Maybe. I think that these kind of systems can be explored in publishing. For example, Copernicus journals like The Cryosphere also have a public commenting and reviewing system. You can see here where colleague of mine has a paper where you can view the full chain of reviews including each iteration:

I am not sure that we currently have an optimum way of organizing peer review and scientific publication. I think that this kind of interaction could even be studied and optimized somewhat. But what I do know is that if you are writing stuff like “Machine learning always produces trash” and patting yourself on the back for engaging in “intellectual discussion” you should feel bad about yourself. And if that makes you angry, good.

In any case here is a list of ways to improve the current status quo:

1. Engage with journals that have public review processes

2. Review papers non-anonymously

3. Engage society publishers like Copernicus

4. Question why you are insecure instead of trashing some poor PhD student’s hard work

Be good to other people. We don’t get that long to live on planet Earth.

Prediction and Predictability is, in essence, how much of a system can you describe from one state to the next. There are many different types of “predictable” systems and there are many different ways to characterize how to predict various systems. In The Price is Right, contestants try to “predict” the price of an item because they are making an educated guess of the cost of the item. The average person has purchased dish washing soap and likely can guess the price based on that experience. Many random systems have interesting conclusions when physically simulated. For example, fair coins have been demonstrated both in simulation and in observation to be biased to land on the side they started from. However, the key component across all systems is that if they are predictable, in some way, then we can summarize or compress some information about those systems in such a way that we can use that summarization to inform the evolution of the system in both cases that evolution is random or deterministic.

However, there are important differences in how statisticians and machine learning researchers define “predictability” versus how physicists define “predictability”. And thus, we are left with the question, “What is a prediction?” Below I will articulate two views: the machine learning view of prediction and the physicist view of prediction. There are more (e.g., I skip over Bayesians) but hopefully this is an interesting way to think about things. I focus on two key descriptive parameters, the R2 score which is common in machine learning regression tasks. And the Lyapunov exponent which is commonly used to examine chaotic regimes in various physical systems.

Statistics/ML view

Statisticians and machine learning people typically assume that there is a certain amount of “variance” in a system that can be explained by attempting to “linearize” the system of study. Variance is defined as:

This equation simply states that we want to know how far the “average” data point x is from the data average xbar. You often see in papers that use machine learning articulate their models using a similar equation called the R2 score:

If you compare the first and second equation the second equation is a ratio where we have replaced xhat with xbar. That simply means we compare how far does the data x deviate from the “model” xhat and it’s ratio to how far does the data x deviate from the average data value xbar (i.e., the previous equation). This R^2 value can have a maximum value of 1 and a minimum value of negative infinite, that is a model can be arbitrarily bad at predicting things. There are other statistics such as the root mean squared error, but they mostly attempt to articulate a particular view: how well does the statistical model capture the “predictability” of the system by how much variance is described by the statistical model? By adding more constraints, for example, keeping a validation data set separate from the training, you can extend the argument a bit and suggest your model is “generalizable” since it has operated successfully on data that it has never seen.

So far, the model xhat could be anything. We could articulate a model that says “if it rained yesterday p millimeters, today it will rain 2p millimeters.” And so if we have some rain data from a location we would first calculate our model output per day using the previous day’s precipitation and then use our R^2 equation to decide if we did a good job or not in predicting tomorrow’s precipitation. However, we could also use other data, e.g., the atmospheric pressure, as controlling parameters for tomorrow’s precipitation. As we add data variables to our model and it becomes more and more complex, we may want an algorithmic way to arrive at some optimum functional approximation to provide the prediction (e.g., a neural network or a gradient boosted machine or even a simple linear regression), but this is beyond the scope of this substack post. The point here is that from this framework, a system is considered “predictable” if we can somehow summarize the next state of the system by estimating how much variance is captured when we apply some model xhat.

Physicist view

Whereas the statistician's view of predictability assumes that there is some variance explained function to minimize, the physicist instead takes the view that dynamic systems either are deterministic or stochastic. Deterministic systems can be predicted into the future with some error that applies for when they diverge from prediction, and this can be due to chaos or simply the precision of the measurements. Stochastic systems are assumed to operate under some probability density function that is predictive of the overall behavior of the system but each individual event is random (e.g., nuclear decay rates). I will focus some on deterministic chaos but bear with me as of course I am not an expert on everything.

Chaos is defined as a system where the initial conditions can vastly affect the outcome. We can quantify this quite plainly with a Lyapunov exponent. The Lyapunov exponent attempts to measure the divergence between trajectories. If we examine the equation in the figure we can see there are two trajectories with some separation between them labeled x(t)+delta(t). This separation distance delta(t) can be described by the exponential equation, and when the exponent is positive we can see that the distance between the two trajectories will grow exponentially. This value is known as the Lyapunov exponent and is a common way to determine how quickly system trajectories will diverge from each other given various initial conditions. Of course, not all physics is chaotic, much of it is not (thankfully). In the non-chaotic cases we simply divert to the expectation that our physical laws are able to capture the trajectories of the system as time goes to infinity. That is, that the largest Lyapunov exponent is always valued at less than 0 is always true.

Discussion

Why is this important? It helps us, in a very general sense, determine when a system stops being predictable. However, let us now contrast the Lyapunov exponent, which is in some ways a measurement of similarity between trajectories of a system, to an R^2 score, which is also, essentially, a measurement of similarity between a model output and the naive assumption. In the R^2 score case, we are measuring the similarity of our machine learning model xhat to the naive model xbar. When this similarity is poor (i.e., R^2 is approaching 1.0), we typically claim that our model arrives at a good understanding of the system because we assume that this increase in R^2 value is because there is a decrease in unexplainable variance (or perhaps randomness). We compare different machine learning models with different features (i.e., variables or data) hoping that by adding more context to the model with additional information, we will be able to build a model that maximizes the R^2 score. This can of course be misused or misattributed and problems like “p-hacking”, violating model assumptions, unexplained heteroskedasticity, and other issues that I won’t go into here.

In the Lyapunov exponent case, we are saying something different. We are not concerned with what data we should add into our model, we typically have a pre-specified model. We assume that we have generated a relationship (e.g., through phase space reconstruction) that explains how different parts of the system interact. When the Lyapunov exponent becomes positive, the trajectories diverge from each other and can no longer be predicted except through our knowledge of the system initial conditions. No minimization of variance in the system matters, no additional variables or data concerning the system matters, we simply cannot predict chaos.

However, you may have read about recent work, such as reservoir models, NBEATS, mamba, etc., which claims to predict such chaotic systems to some degree past the point where Lyapunov exponents become positive. Taken at face value, this is profound. But why is this true?

Personally, I don’t actually know the answer to this question and feel like it is an open question. Perhaps other people know the answer to this question. Perhaps they might find it trivial. In any case, I will suggest, is a good time to stop and ask you to watch a few talks from the AI conference that I helped organize in October 2024 where I address this. I would suggest watching the talks by William Gilpin and Anders Malthe-Sørenssen. William’s talk directly dives into the predictability issue. Anders' talk is a discussion about understanding how a very complex generative machine learning model arrives at its solution. I think these talks begin to unravel why this is true, but the truth is we simply do not know. Or at least, I don’t know.

This was originally written as an internal newsletter which I have expanded and re-edited for here.

My job as a scientist is to understand the fundamentals of my field in the ways that current and historical scientists have established it. This is an ongoing process where I must continually update my knowledge and understanding of what is fundamentally assumed. As I do this I must form my own thoughts and opinions about what the cutting edge should be. This is the goal of science, to establish topics about which we collectively do not understand. My job as a scientist is to be comfortable with the idea that I don’t understand things because if I did, it wouldn’t be the cutting edge.

Across all of my research I have essentially taken some idea established elsewhere and applied it to some idea space that hadn’t seen it yet. I always was very critical of myself thinking I had no original thoughts. I would read about databases or gradient boosting or whatever and then decide this is what’s missing in the science standing in front of me. The more technologies I could learn, the more problems I could solve.

But now I think there are more skills to master as a scientist and software engineer than this. Finding the cutting edge is really finding the place where only opinion exists. And I think that today that has become increasingly difficult. I think that is for a variety of reasons but one reason important to me is that it seems like there are too many ideas to consider at the same time. This is where I think an effective AI assistant could help you investigate the cutting edge. This is also why hallucinations are such a problem. What if they simply make up molecules? Or they just connect two ideas for nonsensical reasons? But assuming that an AI assistant could be built and verified to discuss science and technology with, I do find perplexity.ai to be a great product. I am able to discuss and get cited answers and find myself examining those answers more than what the AI writes itself. And thus far I find it more like a recommendation engine. Like how Netflix tells you what next to watch. You don’t agree with everything Netflix recommends even though it usually knows you just want to watch The Office one more time. And that’s just fine for me. Because it also allows me to form my own opinion about what the cutting edge is, where the cutting edge is best described as what I don’t understand and what needs to be done next.

But the great fear becomes, what if a hallucination leads me astray? What if that compounds into a project in some way that is destructive. But what if the opposite happens? You follow a line of reasoning due to an ai hallucination that leads you to discover something you would have not discovered? This all leads me to the idea that all things are models. That all models can be tested and broken and that is the other goal of science. Building models of the natural world. Hallucinations don’t matter that much if they violate a model assumption. And so your model has edges that are fuzzy and the cutting edge is trying to establish what will make things less fuzzy. And this is all very abstract but our goal is to make things less abstract by building models. This is why I directed a short documentary about scientific models a few years ago.

Defining the cutting edge of LLMs, or “state of the art” (SOTA), has lately been a numbers game. In the beginning, it was quite obvious if an LLM failed at something. It just spout out nonsense. But as the nonsense has become more believable, and thus labelled a hallucination, metrics have been developed to define what SOTA is for LLMs. These metrics are essentially designed to be the final exams of various domains such as physics, chemistry, and economics. The better LLMs perform the more likely they are to be described as “phd level” while still not understanding basic mathematics such as the commutative property of integer addition. This all reminds me of reading conversations from the 1990s about what physics conceptual surveys were really measuring.

Yann LeCun has said himself he thinks LLMs are obsolete and if you are a PhD student you should work on something else. To him, one of the fathers of modern AI, the cutting edge is somewhere else. Ultimately the cutting edge seems to be wherever you decide the cutting edge is. That the cutting edge is somewhat different for everyone. For example, I’m sure there are people who are still excited about LLMs! In my opinion, it seems like one of the best ways to determine what the cutting edge is for you is to evaluate whether you feel accomplished at work, do you feel like you have an impact, do you feel like your work is growing you as a scientist or engineer, is that work challenging to you, and do you feel engaged by the community around you? If you have all these things at work, you are likely at your cutting edge.

Anyways, maybe it’s time for a second documentary.

Subscribe

Around 2011 a software developer friend of mine convinced me that I needed to learn python and SQL at the same time. That python and SQL together would change my working life and that data science was becoming the next thing. And that I should definitely ignore the noSQL crowd for the time being because I needed to at least learn why SQL was so great before learning why it was bad (good logic for lots of solutions in search of a problem). So I took a course in database systems at Georgia State University taught by Raj Sunderraman. The course covered both theory and application of SQL databases. I had to read the original Codd papers describing the relational paradigm, we learned the underlying relational algebra, but most of the class was building software applications that respected a given database schema that we designed. You can see here a very embarassing CLI storefront I built for a class project, I guess the database itself is lost to time. At the time I had no idea how important SQL would become to me and all of the work that I would end up doing into the future, I only had my friend’s advice that learning SQL was great.

After finishing my masters degree I worked on a variety of projects that all used SQL databases including clickstream processing of Coursera data for video lectures, a data sharing model for physics education research that utilizes a lot of relational concepts, I organized an educational data mining internship where students learned SQL and machine learning and applied it to research questions, and my PhD where I rebuilt a 700+ table production database used by an American university into an analytics database used to predict what students would do at university like dropping out or changing their majors. The original production database was fascinating including information about all sorts topics from office assignments for professors to grass mowing schedules in addition to the more relevant student information.

Suffice to say, I quite enjoy building and using databases. I find that by building a database you create a structured opinion about how you plan to pursue a line of research questioning. By using SQL databases you also align yourself more closely to industry where they are used constantly. And I truly believe that using databases more frequently in our research we would accelerate results discovery. But maybe I should tell a bit about a database in order to make this post useful to all my readers.

Relational Databases

If you have ever used a JOIN or MERGE operation in python using pandas then you have used an idea that came from relational databases. Relational databases are software systems designed to store data. Essentially, all databases contain a set of tables, and the relationships between the contents of each table. This is separate from other ways of storing data such as a folder hierarchy of data files, as these database systems enforce control on their data using a user specified schema (the data held in the database is forced to conform to a specific structure), the data can be queried using a special query language called SQL (which stands for Structured Query Language not special query language), and the database will enforce things like ACID compliance. Traditionally a database insists on strict relationships instead of fuzzy ones, that is something either is or is not related to each other, but these days there is lots of technology to handle fuzzy relationships as well. But overall, I think the most important concepts for a scientist to understand about databases are schemas.

A schema is the description for how data connects to other data in a database. The schema can set dependencies, it defines the structure of each table (e.g., the type a variable that a column can store), how that table connects to other tables using primary/foreign key relationships, and what kind of data the table can store. Schemas insist on conforming your data to their structure, software must be written to take whatever data you have and transform it to this schema so that it can be queried. Once your data is inside of a database you can write powerful queries using SQL (the query language this is all named after).

SQL is an easy to read query language that is baked into any SQL database software system. It allows you to write queries against the data to build a dataset from the underlying database. In the example from my PhD, you could produce a dataset for all of the students ever registered as a physics major with a simple query like:

SELECT *
FROM Majors
JOIN Students
ON Majors.student_id_fk = Students.students_id_fk
WHERE Majors.Major_Code = 'PHYS'

This will produce an entire dataset describing these students. Coupled with a pandas.read_sql statement and you already have a very powerful analytics engine at your finger tips.

I do not think that a SQL database can solve every problem although modern SQL databases seem to be able to do anything. Postgres has both an excellent time series engine and geospatial engine. Understanding SQL gives you a powerful toolset for data mining. You can take piles of unstructured data, define a schema that you believe will enhance analytics of that pile, build data pipelines that transform that data into your schema, then do statistics, visualizations, and build models against your new analytics database.

But I do think the lessons of how data relates to other data is an excellent tool for any scientist to have in their belt. The only downsides is that can require time and you need a good amount of data. There is little reason to build a database if you only have 1 spreadsheet. However I encourage scientists to build more databases of the data they collect and make them more freely available. By “productionizing” your data storage, you make your data more accessible to others.

Subscribe now

I think the reason software specs change less frequently than data exploration is because there is an inherent difference in the goal of the scientist to the software developer. The scientist has a set of data that initially is diffuse and poorly understood. As they try out different ways of transforming and interacting with the data, they begin to find things in that data that was otherwise not apparent from the start. These data transformations are typically custom enough to the data set that it is easier to write a block of code to do the transformation and see if it shows something instead of developing a module that does something either as one line of code or as an application.

For example, transforming some kind of signal into information largely requires you to examine the signal both in amplitude space and in frequency space. In order to do this you use the fourier transform to calculate the frequency space. Windowed over time this produces a spectrogram. However there are all sorts of bits and pieces to consider when making such a calculation. What is the sample rate of the data? Do you want to do some kind of filtering prior to plotting a spectrogram? What time window size do you want? Do you want to focus only a subset of the frequency range or the entire range? Do you want to do this 1 million times to generate spectrograms to use for some kind of image-based machine learning signal finder? Ultimately, it ends up being easier to use a scripting language to do this instead of an application with knobs and buttons. But all this code may likely never be used again. Thus, it all falls into the “Exploring” and “Results” category above.

This conceptual relationship between how many lines of code you write versus how many of those lines of codes get reused ends up being a big sticking point and why you end of with questions about why are scientists so bad at writing code. Answers usually include that scientists have a different purpose than software devs. But what if you do want to build software for science? How do you get buy-in from highly technical scientists who may have poor software development skills? How do you create a conversation between the scientists and the software devs so everyone is on the same page? Enter tutorial driven software development.

Tutorial driven software development is a software development paradigm being developed by the brilliant Christopher Woods who formerly headed up the Research Software Engineering group at University of Bristol and now is heading up AI/supercomputing infrastructure group. Tutorial driven software development positions the scientific problem first by developing tutorials for using the software prior to the software library being written. This allows the software developers to sit with the scientists and work together to generate a set of tutorials that solves the problem of the scientist with software. This is done in a cycle. First you develop a draft of a tutorial. This tutorial is typically done via a jupyter notebook which allows scientists to write content and context via markdown along with function stubs that implement the library. These function stubs serve as the endpoints of the software library to be developed. The development is then pursued by software devs, and this is iterated on until the tutorial is able to be leveraged by the scientists and extended into their work. The tutorials also serve as the ideal documentation that faces scientists who are more likely to search for documentation that details their specific use cases via tutorials, than read documentation at the library level.

Christopher has used this software development paradigm to develop multiple packages including the 4d-modeller R package that we just wrapped up development at the end of last year. I really appreciate that a core part of this development paradigm respects the different coding and computational skills across a diverse group. It allows domain expert stakeholders to be an integral part of the development process leveraging their expertise while also respecting that they may not have the time nor background to develop a sophisticated piece of software as they have other project responsibilities.

Subscribe now

Model scale versus domain knowledge in statistical forecasting of chaotic systems

William Gilpin. Physical Review Research, 2024.

This paper, above most others, had a huge impact on me in 2024. Basically it presents a huge number of chaotic systems and demonstrates that machine learning methods (particularly NBEATS) are able to forecast further into the future than previously though possible. It also serves as the example in my previous post about writing good code for publications. Moreover, this paper suggests that there may be both methodological issues in how we characterize the predictability of systems, that is, we could be leaving some of what we know on the table when we are attempting to assess how chaotic a system is and when we should give up trying to predict it into the future. This paper was so influential that I invited William to give one of the plenary lectures at the 2024 Exploring System Dynamics with AI meeting at the University of Oslo Science Library. You can watch his lecture here.

Predictability: a way to characterize complexity

G. Boffetta, M. Cencini, M. Falcioni, A. Vulpiani. Physical Reports, 2002.

This paper is one of several of a collection of literature that discusses how chaos and complexity can also be described as data compression. That is, how chaos theory can be described using dynamical systems theory, entropy, and information theory.

A well ordered system such as

can be compressed exactly into a very small description. Why is this true? Because after one cycle we already can repeat all cycles as time goes to infinite. Therefore we need very little observation of the signal itself to predict it into the future. However, a chaotic system (i.e., has a positive Lyapunov exponent and therefore different initial conditions create exponentially diverging trajectories) cannot be treated this way. This is because as time progresses the uncertainties grow too fast for us to have a regular way to predict into the future. But what this means is that if we are to find efficient ways to predict the future we should find ways the efficiently compress the data we have so that it uncompresses without loss. Why should earth scientists care about data compression? Neural networks, in many cases, are a form of data compression. It is perhaps, trivially true that compression is an important component as to why neural networks are good at their jobs as they are able to efficiently organize data from latent spaces into meaning (i.e., prediction).

The Bitter Lesson

Rich Sutton

This is a blog post and not a paper, but I think it is as equally important as any paper. It basically sums up all of the current literature on AI/ML research as this: having generalized models, with modular structures that leverage computational power (i.e., more CPUs, GPUS, whatever-PUs) will lead to further development in any kind of statistical learning problem. This should be true for any kind of learning, not just for chatGPT type problems but also any geophysical problem. From a physics point of view this makes sense, we don’t have a huge selection of algorithms that describe electrodynamics, we have maxwell’s equations (as an example). What this lesson implies is that, all things being equal with access to data, having more computational power is the limiting factor on our ability to compute complex things. I like to couple this idea with a decades old paper from Microsoft Research which says, all things being equal, models of varying complexity arrive at the same result where accuracy of that result increases somewhat less than exponentially as model data increases exponentially.

Seismic events miss important kinematically governed grain scale mechanisms during shear failure of porous rock

Alexis Cartwright-Taylor, et al. Nature Communications

This paper isn’t about machine learning but I quite like it’s premise. Essentially, stress can elastically reorient within the rock through grain boundary rotations which could be why acoustic emission data (i.e., seismology) does not capture the entire stress history of faults. While this is an experimental paper, there is no reason to expect such conclusions do not apply to nature, and thus, we are left wondering what the true energy budget of a natural fault must look like if that energy budget is governed by a host of processes other than friction and fracturing.

Universal differential equations for glacier ice flow modelling

Jordi Bolibar, et al. Geoscientific Model Development

There are different paradigms when it comes to combining physics and machine learning (this is worthy of it’s own post I think). One paradigm is to treat your geophysical problem it as a statistical problem. This has a lot of advantages as it allows you to focus on data-driven discovery of complex relationships using data sets that have been gathered while not getting bogged down in trying to physically model a complex system. It also is great when these systems are so complex, they might not be able to be physically modeled. Another paradigm is a physics informed neural network. In this mode, you treat the neural network as the solver, and constrain it by training the network on initial and boundary conditions using a loss function represented by known physics. This allows for a lot of benefits to the modeler as it typically gives you a meshless estimator and can be trained in the fraction of a time of a traditional computational method. However, this can be tricky to escape a numerical setting and train on real world data depending on the topic. Universal differential equations instead use neural networks to replace the parameterization of a known system. In the case of this paper, they attempt to parameterize the spatial variability of glaciers from different climates. Each method has it’s positives and negatives. But added together, it makes me wonder if these methods sit on the spectrum between the two statistical cultures as noted by Leo Breiman. That is, do we assume that the mechanism that generates our data is known, i.e., it is some physical law, or do we assume that the mechanism is perhaps too complex and can only be estimated? This I think is a spectrum of what can be measured and is not an either/or kind of situation.

Conclusion

I am sure I have read other papers. There are perhaps, too many papers to read. Feel free to send me papers you think would be fun to discuss here. Also feel free to send me topics you would like to be discussed here.

Subscribe now

Code should work

Codes attached to a paper should run out of the box. The code represents some kind of result of your work, if reviewers (and subsequently readers) cannot even run the codes you have attached to your paper you have already failed to get buy-in to your ideas. In fact, in some cases I have wondered if authors are attempting to hide some result in their paper by attaching code that doesn’t work out of the box. Usually these problems are very simple to solve. I often assume that they occur because the authors have written their codes in one location and now are copy and pasting them into another location to prepare for publication in a journal that may require a code contribution.

Here are some (anonymized) examples from various papers I have reviewed:

Using functions that don’t exist

import numpy as np
x = np.random.random(200)
x.sums()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 x.sums()

AttributeError: 'numpy.ndarray' object has no attribute 'sums'

A function is used that doesn’t exist. In this particular case a numpy array has the function sum() but not sums(). This happens when authors write code but never run the code themselves before submitting to a journal.

Declaring variables without assignment

x = # typical values are between 5 and 10
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 x.sums()

AttributeError: 'numpy.ndarray' object has no attribute 'sums'

In [4]: x = # typical values are between 5 and 10
  Cell In[4], line 1
    x = # typical values are between 5 and 10
        ^
SyntaxError: invalid syntax

An assignment operator is used without anything being assigned to a variable with the user being prompted to input values themselves through an inline comment. This is a poor design pattern and essentially asks the reader of the code to understand what the scientist was thinking when they wrote the code to begin with. This also prevents reproducibility since we have no knowledge of what values the authors used in their original study. Finally, this code will not work and produces a syntax error.

Using imports that have not been imported

x = np.random.random(200)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 x = np.random.random(200)

NameError: name 'np' is not defined

In this example code was presented that uses an imported library but that library was not imported in the code provided. In this case there is the missing line:

import numpy as np

Code should be documented

When writing code, especially functions, the code should be documented. These comments are especially helpful for reviewers and readers in who are reading the code who may be unfamiliar with the particular data processing procedure but perhaps are familiar with the physics or geology that is being investigated in the paper. Your goal is to get acceptance of your ideas by other scientists. Well documented code helps you do this as it allows your codes to be readily interpreted by other scientists.

Python, and many other languages, allow you to create docstrings. These docstrings can then be converted automatically into documentation when building a larger library using tools like Sphinx or Mkdocs, or they simply can serve as descriptors of what code is doing in a script or notebook environment (you could also document the code using markdown in a juypter notebook).

Below is an example of a docstring that explains a simple function to calculate a spectrogram using pytorch. It also includes inline comments.

import numpy as np
import torch
import torchaudio

def calculate_spectrogram(waveform, sample_rate, win_length=None, hop_length=None):
    """
    Calculate and plot a spectrogram using PyTorch's torchaudio.

    Parameters:
        waveform (torch.Tensor): The raw audio waveform as a tensor.
        sample_rate (int): The sample rate of the audio waveform.
        win_length (int or None): Window size (default: None, uses n_fft).
        hop_length (int or None): Hop length between STFT windows (default: None, uses win_length // 2).

    Returns:
        torch.Tensor: Spectrogram tensor.
    """

    # Convert the data to a pytorch tensor if not currently a tensor
    waveform = torch.tensor(waveform)

    # Dynamically estimate n_fft based on sample rate
    n_fft = int(sample_rate //4)

    # Default win_length and hop_length if not provided
    if win_length is None:
        win_length = n_fft
    if hop_length is None:
        hop_length = int(win_length * 0.05)  # Typically 50% overlap

    # Create the Spectrogram transform
    transform = torchaudio.transforms.Spectrogram(
        n_fft=n_fft,
        win_length=win_length,
        hop_length=hop_length,
        power=2.0  # Use power=2 for power spectrogram
    )

    # Apply the transform to the waveform
    spectrogram = transform(waveform)
    return np.array(spectrogram)

Codes include arbitrarily long notebooks

Often times in science you are writing code to explore data, explore a concept with a new model, or to build a bespoke data pipeline that may only be used once. Across all of these activities you may simply not know what you will do next because you are testing and exploring ideas against your model of how you think your system of study evolves in time and space. In practice, this is often seen through monolithic notebooks that contain everything including data processing, function and class declarations, etc. I know this because I have done this myself! And it makes a lot of sense in some cases to have an ever growing notebook that is being developed quite differently from the way one would develop software.

Scientific coding is different from software development (this topic deserves its own post!). I like to think of science as trying to hold onto a hose that has a lot of water running through it. The hose represents your ability to explore the ideas and concepts that interest you, and the chaotic behavior of the end of the hose where the water comes out represents the high entropy of ideas that you are trying to control down into a publon. Your goal as a scientist is to control the hose to wash away the dirt and grime from your ideas and find a gem. This metaphor then lends itself directly, at least from my point of view, to building a monolithic jupyter notebook during your scientific exploration.

The solution here is to spend some time developing a module that sits separately from your notebook. This module should contain all of the functions and classes developed for data analysis. Then you can produce a separate and new notebook that presents the exact results associated with your paper. A good example of this comes from this paper, Model scale versus domain knowledge in statistical forecasting of chaotic systems and it’s associated github repo. Which leads to my next point.

4. Use github (or something like it)

I prefer github and use it across all of my projects. It both has tools to support scientific communication (e.g., notebook rendering in browser) and also allows you to create webpages associated with your work for free (e.g., a documentation website for the python library you have developed). It integrates with zenodo so you can attach codes directly from your repo to a paper which is required by many journals while still allowing you to develop your codes separately from the paper you are writing. Git and Github also is a professional skill that is necessary for most computationally related jobs you will get after completing your PhD. Github allows readers and reviewers to explore commit histories on codes, assuming the authors developed their codes using github, in the event that those codes may have failed. I cannot recommend to other scientists enough to learn to use git and github for their science. If people have objections to this, I would be happy to hear an alternative solution (e.g., Gitlab). But I have yet to see a system that offers so much for essentially free for scientists to track progress in work, advertise results, build webpages, etc.

Conclusion

This is not an exhaustive list. I think there are many things I could add to it. But if you write code that works and is well documented you already will be at the top of the stack of papers that I will want to review. Bad coding practices in science makes it harder for others to accept your ideas as it stands in the way between you and others by limiting your ability to communicate these ideas. This is a travesty for a scientist as a central goal of their day to day job is to gain acceptance in the scientific community of their ideas of how the universe works. Happy coding!