What's a Better Way to Search?

For the last two years, during my MBA, I’ve been circling a question: what’s a better way to search? Along the way I built ClipFind, a tool to find films by the feeling they leave you with. It didn’t work the way I hoped, but it pushed my thinking forward on the question I keep returning to: what does search become in the era of AI, especially for video?

01 — Passive feeds, active hunts

Most of the search we experience today is passive. The system already knows us based on what we clicked, what we watched, where we paused, what people like us tend to click, and it pushes information toward us before we ask for it. The feed is the purest version of the kind of search that happens to you. It has been refined relentlessly over the past decade, because passive discovery is what keeps people scrolling and staying on the platform.

But there is another kind of searching that has not really been upgraded since the early internet, and it is the kind I find myself doing constantly, which I call active search. It is the deliberate hunt that begins the moment something sparks. It is what happens after:

• you watch a show you felt emotionally connected to, and you want that feeling again;
• you stumble on an outfit style you love, and you want more of it;
• you get curious about a topic and want to go deeper;
• you find a place, and you want to find others like it.

This is intent-driven, not profile-driven. You are reaching for something specific. And until recently, this kind of search meant typing a few keywords and hoping the labels other people had attached to things happened to match the thing in your head. Large language models changed part of this. Active search got easier almost overnight, because for the first time you did not need to reduce yourself to keywords. You could describe a vague, half-formed intent in your own words and get something useful back. “Something like Zima Blue” could become “stories about existential wonder, isolation, and identity,” and the machine would understand. That felt like a real unlock.

But there is a constraint hiding inside that unlock: the LLM only helped when my intent could be turned into words. Language became the entry point. If I could say it, I could search it. And for a lot of what I actually wanted to find, the thing I reached for was not words at all. It was a picture.

02 — The picture I couldn’t say

This is where visual search comes in, and where I started to see an opportunity. I learn a huge amount from watching things while I get curious, and my instinct is to go looking for more: more clips, more scenes, more of whatever caught me. And what I most want to do, more often than any search bar will let me, is to take a screenshot of a single moment and say: “find me more things that feel like this.”

Here is what happens when I try. Even when a tool lets me upload that screenshot, the usual thing it does underneath is not look at my picture at all. It captions it. My neon-lit frame becomes “dim neon hallway, a lone figure, a feeling of melancholy,” and it is those words, not the image, that get matched against titles, transcripts, tags, and descriptions. I never see the caption and never get to correct it, and it has already thrown away almost everything that made me reach for the screenshot in the first place. The image was the query for a moment, and then it became words. The video itself is never the thing being searched; its text shadow is. And the feeling I captured has no good word-form.

03 — Searching the text shadow

Visual search comes in three quite different forms, and the differences are the whole point.

• The first, and most common, is caption-then-search: the image or clip is converted into tags, detected objects, and a transcript at the moment it is stored, and from then on you are matching text against text. The picture was real for a moment and then became words.
• The second is genuinely visual — the approach behind tools like Google Lens or Pinterest, where the pixels are turned directly into a numerical representation and compared against other pixels, with no words in between. That is real, and it works well for “find me a visually similar image.”
• The third is the newest: multimodal models that actually look at a frame and can describe or reason about what is in it. The catch is that even when I paste a screenshot into one of these and ask for “more like this,” the system often quietly falls back to the first form, and it writes a description of my image and runs a text search on that.

So the question is “what is the index actually made of, the pixels, or the words we wrote about them?” For most systems, over most video, it is still words.

And “visual understanding” is itself a ladder, not a single thing. At the bottom is perception — what is literally in the frame: objects, colors, faces. Above that is recognizing what is happening — the action, the event. Higher still, and unique to video, is grasping how things unfold over time: sequence, cause, pacing. And at the top sits the layer I care about most — mood, tone, cinematography, the feeling a scene leaves behind. Most working visual search reaches only the first rung or two. The feeling lives at the top of the ladder, and almost nothing climbs there reliably at scale.

It is worth naming the two failures separately, because they are different problems. One is the query: I cannot hand the system an image and have it stay an image — it gets flattened into words at the door. The other is the index: even if my visual query survived, the thing it searches against is still a pile of transcripts and metadata, not the visual and sonic substance of the video. Fix one and the other still defeats you. Both ends have to become visual, or neither does.

Here is why I think video is the place this finally breaks. For a text document, searching the words loses nothing — the words are the content; the text shadow and the thing are one and the same. But video is not like that. A huge part of what a scene means lives in things text can’t hold: the color and the light, the framing, a wordless performance, a music cue, the pacing, the way a silence lands. When you search video through its text shadow, you throw all of that away. For documents, routing search through text is free. For video, it is lossy — and it is lossy in exactly the dimension I care about most.

04 — Building ClipFind

This is the conviction I started from when I built my first version of scene-based search: that searching the content directly would be more powerful than matching the labels and texts we hang on it. I believed visual understanding could beat metadata matching. So I tried to build it.

The version in my head was a system that searched video by its own substance — frame by frame, image and sound together — so that the query could be a clip, a screenshot, or a described feeling, and the thing it matched against would be the footage itself, not the words anyone had written about it. The screenshot wish, made real: hand it a scene, get back scenes that genuinely look and move and feel like that one. No caption in the middle. No text shadow on either end.

What I actually shipped was smaller, and more honest about my limits. I called it ClipFind, and I built it alone in about three weeks. A viewer would type a loose, feeling-led query — “something like Love, Death + Robots” — and a language model would unpack the intent underneath it: the tone, the themes, the emotional thing the person was really chasing. It would match that against a small, hand-curated library of roughly thirty AI-generated short films — pulled from an MIT AI film hack, and chosen for one unglamorous reason, that they carried no licensing barriers — and hand back scene-level clips to watch.

But under the hood it was not yet the system I had described. Frame-by-frame understanding was beyond the skills and tools I had then. Instead of reading the videos, I matched against text proxies — transcripts, plot synopses, and short written “reflections” I composed about each clip — turned into embeddings. The thing I had built to escape the text shadow was, underneath, still searching the text shadow. I knew that, and I did it on purpose: the cheapest question to answer first was not “can I build the visual engine,” but “would anyone even want to search this way?”

On that question, the early signal was encouraging. I ran it past a handful of users, and the core behavior held up — given a way to search by feeling rather than by title, people did reach for it and explore intentionally, past the recommendation feed. What I had confirmed was an appetite for the experience: intentional, feeling-based discovery. What I had not confirmed — could not, with a text proxy — was the visual mechanism I actually believed in.

05 — The wrong wedge, the right question

But the same tests surfaced a harder truth, which is that entertainment consumption is predominantly passive. People come to film and television to lean back and be fed — the whole pleasure of the medium is not having to work for it. The active, I-need-to-find-the-thing-that-echoes-this searching is real, but it is a minority, episodic behavior — a flicker after a finale rather than a daily habit — and it runs against the dominant lean-back current. I had built an active-search tool for the one category where people most want to be passive. That caps the demand no matter how good the search becomes. And there was a second reason film made a weak starting point, one that compounds the first. Film and television are unusually well served by words. Every title has a synopsis, a script, reviews, a fan wiki, a thousand Reddit threads and video essays; we have been describing movies in language for a century. A film’s text shadow is rich and deep, so text-based search over it already works well enough, often enough, that the gap I was trying to exploit keeps closing on me. Even in those rare active moments, the frame-level visual search I believed in was least necessary exactly where I had chosen to prove it. Least demanded, and least needed — consumer film at the frame level was the wrong wedge on both axes.

The clean version of the idea lives where both conditions flip — where the search is forced rather than optional, and where text runs out. Try to find something in the clips on your own phone, the footage from a security camera, the hours a wearable or a robot quietly records, and there is no feed to lean back into and no synopsis, fan wiki, or helpful stranger who wrote down what happened. There, active search is not a minority behavior; it is the only way to find anything at all. And we are about to drown in exactly this kind of video — some of it AI-generated film, far more of it unlabelled and undescribed, the billionth clip no human will ever caption. In those places, text search does not work badly; it does not work at all. Visual-native search stops being a nicer way to do it and becomes the only door in.

So this is the question I’m still thinking about — what is a better way to search? My film application was the first and smallest answer. I chased active search into the one corner of the world that most wants to stay passive, and where words already do a passable job. But building it taught me where the idea actually points. The era of treating text as the universal middleman for all search is ending, and video is where it breaks first — on both ends, the query and the index. I started in film because that is where I could feel the problem myself. The answer, I think, lives somewhere I have only just begun to look.