What AI gets wrong about human things

I practice Carnatic music with a tuning app nearby. It’s genuinely useful for what it can measure: it can tell me which note I’m on and whether I can hold it steady. But the beauty of Carnatic lives in the gamaka, the slide and oscillation between notes, and there the app goes quiet, or worse, reads the movement as a note that won’t settle. The thing that makes the music alive is the exact thing the software deletes.

I build software for a living, so I know why it happens. The model learned “in tune” from a tradition that holds a pitch still. Carnatic doesn’t hold the pitch still. The meaning lives in the movement between the notes, and the app was never taught that the movement is the music, not an error in it.

I want to be fair to the machines here, because I use them every day. LLMs, which is what most people mean now when they say AI, are genuinely good at a few things. They’re a great idea generator when you’re staring at a blank page, and an even better mimic. The mimicry has ruined em dashes for me: the models trained on decades of writing that loved them, and now original work has to leave them out. And as a practice partner they’re tireless, happy to hold your work against a standard a thousand times without getting bored. For an adult learner, that’s no small thing. Mastery is repetition with good feedback, and good feedback is the scarcest part of learning Carnatic music in America. There are only so many teachers, and mine can’t sit with me every day. The tools widen that door.

The catch is who defines the standard. Taste, deciding what good looks like, is the thing they don’t have. They can generate a thousand options without an opinion about which one matters. They can’t hear what a slide between two notes is carrying, or why one oscillation is rich and another is just wobbly. That knowledge was never written down; it passes from teacher to student, ear to ear. If it isn’t in the notation, it isn’t in the training data either. To be fair, none of this is unique to AI. My tuning app is ordinary software and it fails the same way. The models just fail more fluently.

I see the same failure at work, where the stakes are much higher. I build technology for hospital food, and a generic model will read an ingredient line like “Produce, Strawberry Fresh Clamshell, 6oz” and flag a shellfish allergy. The clamshell is the plastic box the strawberries come in. Off-the-shelf AI breaks on branded items, on the shorthand a specific kitchen uses, on a recipe nested three levels deep where it matches the right ingredient to the wrong step, on the knowledge sitting in an allergy note that one veteran dietitian understood and never wrote down. It’s hilariously confidently wrong, and that’s exactly where the human role is important.

That’s the pattern I keep running into. People like to split the world into corners, tech or art, AI or human. I don’t buy it. The interesting place is the seam, where the machine meets the irreducibly human thing, the gamaka, the allergy note, and flattens it while sounding certain. Someone has to notice, and hold the line.

People are scared that AI is stealing originality, that it’s coming for the future of everything creative. I understand the fear, but I don’t believe in purely original things. Everything human is remixed and built on something older; that’s how culture has always moved.

Remixing on its own doesn’t erase anyone. Erasure happens when the remix goes on without you. Chai becomes “chai tea” on menus written by people who never asked. An Asian tea house draws a jasmine and loses it in court to a French luxury house whose own monogram was drawn from Japanese crests. The conqueror’s oldest move is claiming what it took, and it works best on the people who aren’t in the room. So I’d rather participate in the future than sit it out. Showing up with everything particular about you is how your identities keep from being erased alongside everything else.

Holding the line is unglamorous. We start from the problem and work toward a solution, and sometimes the solution isn’t AI at all. Nothing a model says about an allergy reaches a patient’s tray without a person standing between them, and that stays true until it can pass evaluations we would stake a patient’s safety on.

As for the music, I still open the tuning app for the parts that hold still, the plain notes, the drone. For the gamakas, I sing to my teacher and wait for the nod. Some kinds of feedback haven’t made it into software yet. Gaps like that are why I build.