Short Thoughts

Quoting Unmesh Joshi via The Learning Loop and LLMs

An AI can generate a perfect solution in seconds, but it cannot give you the experience you gain from the struggle of creating it yourself. The small failures and the “aha!” moments are essential features of learning, not bugs to be automated away.

Relatedly, on the topic of learning, I was pondering today how popular AI chat apps are great at providing answers, but that is often opposed to learning.

This brings to mind the concept of designing a system so that users fall into the pit of success, instead of the pit of despair. When using the popular AI chat systems for learning they are designed so that by default users fall into the pit of despair, they are detrimental to learning. But it does not need to be that way. Chat systems can be designed so that users fall into the pit of success, so that by default they enrich the learning process.

A complex system that works is invariably found to have evolved from a simple system that worked

Gall’s Law

Young man, in mathematics you don’t understand things. You just get used to them.

John von Neumann (via)

To really understand a concept, you have to “invent” it yourself in some capacity Understanding doesn’t come from passive content consumption. It is always self-built. It is an active, high-agency, self-directed process of creating and debugging your own mental models.

François Chollet (via)

Quoting Andrej Karpathy regarding LLM intelligence Animals vs Ghosts

In my mind, animals are not an example of this at all - they are prepackaged with a ton of intelligence by evolution and the learning they do is quite minimal overall (example: Zebra at birth). Putting our engineering hats on, we’re not going to redo evolution. But with LLMs we have stumbled by an alternative approach to “prepackage” a ton of intelligence in a neural network - not by evolution, but by predicting the next token over the internet. This approach leads to a different kind of entity in the intelligence space. Distinct from animals, more like ghosts or spirits. But we can (and should) make them more animal like over time and in some ways that’s what a lot of frontier work is about.

That’s beautifully said. It paints a vivid picture of how LLM intelligence differs from biological intelligence. By training on the collective content of the internet, these models become a form of us, our past selves, our ghosts. We recognize an intelligence in them, but it’s a mistake to equate it with human or animal intelligence.

Is the current intelligence enough for AGI? Will the next AI winter come from trying to make models more like animals? Is that even a wise path to pursue?

I don’t think today’s intelligence is sufficient for true AGI. As Karpathy pointed out, it’s a fundamentally different kind of intelligence. I don’t see how this architecture evolves into something truly general. It can get closer, sure, but there will always be holes needing to be plugged. This will bring forth the next AI winter, until the next breakthrough is discovered and our capabilities reach the next level.

Still, I’m uneasy about that pursuit. There’s already so much potential in what we have now. Entire industries and creative fields haven’t even begun to fully explore it. And as a society, we’re not prepared for the intelligence we already face. However, it is in our nature to always be progressing. Perhaps by the time the next breakthrough occurs, society will have adjusted to the current level of intelligence, better preparing us us for the next level.

Responding to Has AI stolen the satisfaction from programming?

Questions similar to this have been brought up previously, and will continue to be for the foreseeable future. One point he mentioned sparked a thought I had:

The steering and judgment I apply to AI outputs is invisible. Nobody sees which suggestions I rejected, how I refined the prompts, or what decisions I made. So all credit flows to the AI by default.

Invisibility of effort is not new. People do not immediately arrive at an answer. This is true across domains. This is similar to the term “overnight success”. It may be the case something unexpectedly takes off, but all the work it took to get to the state where it could take off is forgotten.

When I write code I spend a lot of my time reworking it, massaging it, expressing it well. I may have gotten to a working solution quickly but it took much longer to get to a final solution. All of this effort is not seen by others, nor do I receive “credit”. Perhaps that is why the author does not recognize this similarity in other people’s end product, the hard work to get it to this state is not immediately obvious.

AI has changed where we receive satisfaction. The invisibility of the effort has always been true, but that doesn’t preclude satisfaction of the process and end result.

Quoting Toby Stuart on the EconTalk episode The Invisible Hierarchies that Rule Our World

The episode ended with a discussion on the impact of AI as it relates to people’s status, that it will reinforce the prestige hierarchy.

When you can’t judge quality that’s precisely the time in which you rely on pedigree.

He goes on to say:

So if you are a college admissions officer, take that problem at a place where a lot of people want to go, it’s really hard to read an essay and say “I’m admitting them because this is an outstanding essay,” if that ever happened. But what you can do is, I’ve heard of the high school, or there is some other status marker in the background, so I’m going to overweight that relative to information that formerly was a signal but it’s just noise.

Writing has been used just about everywhere to evaluate people. Now the capability of crafting well written content is available to everyone. Consequently, we become more reliant on other indicators for evaluation, which, unfortunately, are often characteristics over which people have limited control

This line of thought presents a sobering reality. What was once seen as an equalizer for those lacking inherited advantages potentially turns out to be detrimental to their advancement.

Responding to I Do Not Want to Be a Programmer Anymore (After Losing an Argument to AI and My Wife)

The article begins by sharing a story of attempting to use AI to resolve a difference of opinion with his wife, which convinced him he was wrong. His wife reaction:

It wasn’t the victory that stuck with her. It was how easily I surrendered my judgment to a machine.

He gives another example from work, from which he writes:

That’s the unsettling part. We don’t just listen to the machine; we believe it. We defer to it. And sometimes, we even prefer its certainty over the reasoning of the actual humans in front of us.

His concerning conclusion:

Wisdom has always come as the byproduct of experience. But if experience itself is outsourced to machines, where will the young earn theirs?

I also have experienced myself being resistant to the arguments of another only to be won over by consulting a LLM and reasoning through the arguments. In part this seems reasonable, the ideas of others which are contrary to our own are costly for us. Ideas which we arrive at, or we think we arrive at, on our own we believe we have already been through the work to vet.

Therefore, the question is whether we ask AI’s answer on the first take, or do we go back and forth with the AI examining the rationale. The first is concerning, to blindly accept the response without any further examination. But I suspect that is not what occurs in most use cases. Instead we become convinced by it because it is a nonthreatening way to explore the topic. I wonder if there is intimations of that when he says:

Clients, colleagues, even strangers are emboldened not because the machine gives them ideas, but because it gives them confidence.

When he provides the example at work the person sent him a “detailed breakdown” of how to improve the system. It sounds to me the person invested a lot of effort and thought into this, not quickly typed a question and forwarded on the AI response.

Circling back to his concern about wisdom, or lack of, I believe this highlights the need for relationship. If relationships continue to erode, lack of mentorship, and trust in AI continues to rise then is wisdom lost?

It feels this may be the case. But humans still accumulate experiences, from both our failures and triumphs. And from those experiences wisdom will still either be derived or ignored. It’s hard to imagine a complete loss of wisdom. Even the author gain wisdom from the experiencing of bringing AI into the conversation with his wife. There is precious wisdom humankind has obtained across our existence, which would be a tragedy to lose. But I have a hope in humanity, that we will continue to push forward and adapt, accumulating wisdom. It is in our nature, I don’t think we can do anything otherwise.

Replying to “You have 18 months” The real deadline isn’t when AI outsmarts us — it’s when we stop using our own minds.

And I am much more concerned about the decline of thinking people than I am about the rise of thinking machines.

I’m not precisely concerned about this. I don’t believe my thinking has declined since using these tools. Maybe they have in some trivial ways. But I believe my thinking has become more active as a result of these tools, because I am able to explore and ask questions, to investigate in ways I either was not able to before, or at least not as easily.

My concern is that there will be a division between people on how they use these tools. One side’s thinking will decline, while the other side’s thinking will be enhanced, which will lead to a further imbalance in society. It appears the statistics he references support this, the declines he reports are not coming from those who already reported as high.

The author later he answers the question about what kids should study:

While I don’t know what field any particular student should major in, I do feel strongly about what skill they should value: It’s the very same skill that I see in decline. It’s the patience to read long and complex texts; to hold conflicting ideas in our heads and enjoy their dissonance

While I do not entirely agree with his phrasing, or at least I am uncertain with how he phrased it, I do believe being able to work with conflicting ideas is an important skill. Perhaps if someone “enjoys” the dissonance they become energized and thrive in these situations. And so maybe the language is not too strong. But at the minimum I have found being able to wrestle with conflicting ideas to be an important life skill.

Quoting Richard Matthew Stallman in Reasons not to use ChatGPT:

It does not know what its output means. It has no idea that words can mean anything.… people should not trust systems that mindlessly play with words to be correct in what those words mean.

My initial reaction is agreement. But then what are the implications? Does it follow that machines will never have intelligence? What makes our intelligence different? A calculator doesn’t know what its output means. Should we trust it? Is the way the calculator works with numbers significantly different than the way the LLM works with words. It’s empirically different, but when attempting to consider each domain they operate in is it significantly different?

Did he think about all of this and then come to his conclusion?

These are all interesting questions and conversations we should be having but I wonder if his real complaint is included in his trailing thoughts:

Another reason to reject ChatGPT in particular is that users cannot get a copy of it.

This weekend I was listening to the re-release of the How to Have Great Conversations episode of the People I Mostly Admire podcast. Levitt and Duhigg discuss the three types of conversations: problem solving, emotional and relational. At one point Levitt says it took him until his 40’s to realize people have various goals in conversations, not just to find solutions.

It only seems natural to me we overly focus seeing things from our perspective, even when taking into account theory of mind. No area is excluded from this, including politics; especially politics.

Enter Arnold Kling’s The Three Languages of Politics. His thesis is that Progressives, conservatives, and libertarians each speak a different language, they each have a different way of expressing their point of view.

• Progressives
  • Oppressor and Oppressed
  • Concerned with the power dynamics between groups of people or organizations. They see government as a means to address this by ensuring justice for the oppressed.
• Conservatives
  • Civilization and Barbarism
  • Concerned about the order, values, and structure of society. They see government as a way to protect these from forces that could disrupt them, which include a breakdown of the rule of law.
• Libertarians
  • Freedom and Coercion
  • Concerned about people being able to make their own choices. They see government as a threat to this, believing its coercion (through taxation and regulation) is the main impediment to individual liberty and prosperity.

All three deal with a form of injustice, but the injustice is seen through a different lens, and therefore differ on what should be done.

In an effort to understand how these play out here is my attempt to think through the issue of immigration, documented and undocumented, in the United States. Each position could be fleshed out further, but for this discussion the context is limited to the direct concerns of the three languages.

Progressives see immigrants as a vulnerable people. It is government’s responsibility to welcome them and provide aid.

Conservatives see immigrants as a disruption to society. It is the government’s responsibility to protect society and foster an environment for citizens to thrive.

Libertarians see immigrants as groups of people who have been denied the ability to make their own choices. The current situation has been brought about by government’s hindrances to immigration as well as the assistance it provides.

This highlights the challenges of coming to agreement about immigration, which is not unique to this topic. It it helpful to keep in mind that these do not represent the entire perspective of people who hold these positions, instead it is the language they come to the issue with. Therefore, agreements can still be found, and they are more apt to be found when keeping in mind that people are likely to describe their position in one of these terms. It is also important to remember that people’s perspective is rich and nuanced.

I often think about the word Sonder:

The profound feeling of realizing that everyone, including strangers passing in the street, has a life as complex as one’s own, which they are constantly living despite one’s personal lack of awareness of it.

The thought of this word produces a sense of wonder and amazement. I hope to carry this sense with me into more relationships and interactions, including conversations on immigration.

I started reading Hannah Arendt. When she attended the trial of Adolf Eichmann, a Nazi and an organizer of the Holocaust, she observed what she called the “banality of evil”. Here’s a quote, emphasis mine.

The deeds were monstrous, but the doer—at least the very effective one now on trial—was quite ordinary, commonplace, and neither demonic nor monstrous. There was no sign in him of firm ideological convictions or of specific evil motives, and the only notable characteristic one could detect in his past behavior as well as in his behavior during the trial and throughout the pre-trial police examination was something entirely negative: it was not stupidity but thoughtlessness.

A few weeks ago I came across the opinion piece Will AI Destroy or Reinvent Education? I think this piece has lots of good thoughts.

The piece begins with talking through two articles about an MIT study on the impact of AI on our brain when using it in writing, Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task. The paper found LLM use led to weaker neural connectivity and less cognitive engagement.

Initially I thought, this is concerning and a good reason to have an awareness of how we use AI. But if people use it this way, it is to their detriment.

After reading Arendt, I’m even more concerned. Could AI use be fostering a new kind of “thoughtlessness”? If so, might horrific deeds be carried out by people who are “quite ordinary, commonplace, and neither demonic nor monstrous” simply because they stopped thinking critically?

Quoting Jessica Kerr, Austin Parker, Ken Rimple and Dr. Cat Hicks from AI/LLM in Software Teams: What’s Working and What’s Next

Empathy

People will do things for AI that they won’t do for each other. They’ll check the outcomes. They’ll make things explicit. They’ll document things. They’ll add tests.… And all of these things help people, but we weren’t willing to help the people. It’s almost like we have more empathy for that robot than for each other… we can imagine that the AI really doesn’t know this stuff and really needs this information. And at some level, we can’t actually imagine a human that doesn’t know what we do.

This comparison creates a visceral blow, because I feel it describes me. I consider myself a fairly empathic person, but I’m slow to create this information for other humans, yet find myself more willing to do so for AI.

Why do we behave this way? Here are some theories. Different expectation levels, ex: AI doesn’t have this background knowledge, but humans should, or at least can figure it out. Comparison and competition between ourselves and others. The impact is immediate when working with the AI, but unknown and in the future for humans. More self-serving when providing these to the AI, at least in the near term again.

Even with these plausible explanations, I can’t quite get myself off the hook. This nagging self-awareness, however, doesn’t diminish my fear that my behavior will remain unchanged.

Participation

Another topic of this interview deserves mention:

…have a training data problems, right? And we can question what we use it for, but it’s very difficult to do that if you sit outside of it. If you set yourself apart, you have to participate.

I do think that is incumbent upon us to grapple with, you know, the reality we’re faced with… We have the universal function approximator finally and there’s no putting that toothpaste back in the tube, so we can figure out how to build empathetic systems of people and technology that are humanistic in nature, or we can let the people whose moral compass orients slightly towards their bank account make those decisions, and I know which side of it I’m on.

AI is here and it will change a lot of things. It’s understandable to be worried about the negative impact of AI, but letting that prevent you from engaging is a way of sitting on the sidelines. Instead, we have a duty to participate and shape its future.

Quoting CasualPhysicsEnjoyer in You’re a Slow Thinker. Now what?

The broad thing that I’ve learnt so far is that it’s been better for me to lean into my thinking style, rather than fight against it. The funny part is that now, I come across as ‘quicker’. But that’s no accident - I’m just reciting things that I’ve worked on for a long time, slowly, behind the scenes!

This perfectly describes me, and probably a lot of other people too. When I appear “quicker” it’s because I’ve already had the conversation in my head many times, and covering many possible directions. I call it “pre-thoughts”.

My theory is because I am an introvert, both the characteristic of slow thinking as well as spending a lot of time in thought beforehand.

Quoting Robert P. George from the Pursuit: The Founders’ Guide to Happiness in the episode In Order to Be Happy.

Madison also understood that structural constraints by themselves would be insufficient. If we are to be self-governing people, that is, if we’re to be citizens of a republic, then we need to be citizens who are capable of self-control of self-mastery of holding in check, at least to some extent. Those powerful evil impulses in our nature, and that requires education.

So, Madison famously said only a well-instructed people can be permanently a free people. That is only a well-educated people can only be a people fit for self-government. They needed to be educated in their character. Capable of distinguishing good from bad, right from wrong, just for unjust.

Ethan Mollick writes that our relationship with AI is shifting, from being a partner we create with to the AI performing the work itself, On Working with Wizards. This aligns with the pursuit of AI agents, 2025 being dubbed the year of the agent. While Mollick doesn’t explicitly discuss agents, he uses the term “wizards” to describe a similar concept. He calls them wizards because:

Magic gets done, but we don’t always know what to do with the results

…wizards don’t want my help and work in secretive ways that even they can’t explain.

This presents two important challenges. First, we lose, or never develop, the skill to evaluate what was produced. We lose ground, and so we are forced to trust more. As Mollick states:

every time we hand work to a wizard, we lose a chance to develop our own expertise, to build the very judgment we need to evaluate the wizard’s work.

But what I found especially striking is that throughout time, when we did not understand something, it was considered magic. Science came along and brought a method of understanding of how things work. Technology replaced magic. Is this direction now reversing? Are we losing our understanding, is magic returning? Mollick writes:

The paradox of working with AI wizards is that competence and opacity rise together. We need these tools most for the tasks where we’re least able to verify them. It’s the old lesson from fairy tales: the better the magic, the deeper the mystery.

This is a shocking realization. While many would push back against this analogy, it seems to be at least partially accurate. It raises a pressing question: Why are we so willing to trade our understanding and accept the magic in this context?

A constant complaint I’ve heard from software developers is that there isn’t a product owner. No one is creating requirements, no one is curating the backlog. Instead the software delivery team attempts to suss out how applications and platforms are to be built. Fair enough, it’s not very efficient to be given a high level description of something and then have to determine what it means.

I’m not going to analyze or offer my thoughts on this predicament, but I was reflecting on it while considering my current software development workflow:

• Receive a high level feature request
• Use AI to create detailed requirements based on the request and the state of the current application
• Edit the generated requirements
• Provide the final version of the requirements to an AI agent to implement
• …

If the desires of software developers were fulfilled then pristine requirements would be created by a product owner. Software developers would then hand off the requirements to an AI agent for implementation, thus making software developers then new Tom, the product manager from Office Space.

What would you say… you do here?

I’ve read three different takes on hallucinations this week, and what struck me most was not how they agreed, but how differently they framed the problem. Each piece approaches hallucinations from a unique angle: technical, procedural, and philosophical. Taken together, they sketch a landscape of possibilities.

OpenAI’s Why language models hallucinate presents the view that not all questions have answers, and so models should be trained with an incentive to abstain rather than answer confidently.

One challenge remains stubbornly hard to fully solve: hallucinations. By this we mean instances where a model confidently generates an answer that isn’t true.

Most evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty.

Penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty.

Then there’s Is the LLM response wrong, or have you just failed to iterate it?, which suggests that inaccurate responses are often the result of receiving an answer too soon. If pushed further, by having the model iterate, it can examine the evidence and follow new lines of discovery, much like humans do.

But the initial response here isn’t a hallucination, it’s a mixture of conflation, incomplete discovery, and poor weighting of evidence. It looks a lot like what your average human would do when navigating a confusing information environment.

LLMs are no different. What often is deemed a “wrong” response is often merely a first pass at describing the beliefs out there. And the solution is the same: iterate the process.

Finally, there is Knowledge and memory, which suggests hallucinations will not go away because knowledge must be tied to memory. Humans feel the solidity of facts, while models lack the experiences required to ground their knowledge.

Language models don’t have memory at all, because they don’t have experiences that compound and inform each other.

Many engineers have pinned their hopes on the context window as a kind of memory, a place where “experiences” might accrue, leave useful traces. There’s certainly some utility there… but the analogy is waking up in a hotel room and finding a scratchpad full of notes that you don’t remember making… but the disorientation of that scenario should be clear.

The solid, structured memory that we use to understand what we know and don’t know — when and when not to guess — requires time, and probably also a sort of causal web, episodes and experiences all linked together.

Each of these pieces makes interesting points, and together they explain different facets of model hallucination. Models are too eager to provide an answer. There are many uncertainties and “it depends” in life. Incentivizing models to reflect this may irritate users, but it better mirrors reality.

However, these responses make clear that what we receive is only a first pass, one that should be refined by iterating, digging deeper, and pushing the model further. Perhaps this process of discovery is still not enough to create true memory, as the third author points out, but it does seem to edge closer to mimicking a brief experience.

Currently, model context is built by labeling messages as system, user, or agent. We’ve learned it would be better to create a hierarchy of significance for these categories, system messages should carry more weight and not be overridden by user messages. What if context were segmented by other dimensions, like time, so a model could build a clearer picture of what it has learned?

Humans also continue processing conversations or experiences outside the event itself. What if models pushed themselves to dig deeper without user prompting, allowing them to provide a more thoughtful answer after the interaction had ended?

There is still so much more to explore, we are far from exhausting what’s possible.

I’ve only watched one video but this seems like a promising course to get up and going in a productive way with Semantic Kernel agents.

11 Lessons to Get Started Building AI Agents

Quoting Lori Gottlieb in conversation with Andrew Huberman in How to Find & Be a Great Romantic Partner

Drama tends to be from all the assumptions. Like, I know that, my truth, is the right story… You were wrong. And I think it’s so interesting because people come to therapy with these faulty narratives…

We’re all storytellers. We all try to make sense of something, and we all believe that our story is the absolute accurate version of the story, and actually it’s so funny when you see couples and they experience the same experience, and they have wildly different versions.

And then there’s some part where the Venn diagram overlaps, and then finally, they can see, oh, that person’s not a bad person they were coming at it because they, in their story, they believe this. And that’s so important. So drama happens when assumptions are made. People characterize the other person’s story as inaccurate, their own story as accurate. And then there’s… no space for curiosity or connection. It’s all rupture, no repair. our-story-is-the-absolute-accurate-version.md