Biswas

Building a face search engine end to end

Building a face search engine end to end

What I learned building a small educational face-retrieval system with detection, ArcFace embeddings, and cosine search.

Cover image for Building a face search engine end to end.
Field note cover.
Technical diagram for Building a face search engine end to end.
Technical flow described in this note.

The FaceCheck Pro README records a 252-face local demo. The repository does not ship that vector database, so treat the number as a build note, not a reproducible benchmark. It is nowhere near enough to claim production recognition.

I built it after an extractor I was reading stopped working. The extractor depended on a commercial service leaving source information in its response. When the response changed, the technique disappeared. Instead of chasing another response format, I built the smaller system I could inspect from end to end. It does one useful thing: traces an upload from detection to ranked matches.

The system I actually built

The query path has four steps:

uploaded bytes
  -> SCRFD detects a face
  -> ArcFace produces a 512-value embedding
  -> Chroma searches the demo index with cosine distance
  -> the API returns stored metadata and source links

The frontend is Next.js. The API is FastAPI. InsightFace runs the detection and embedding models through ONNX Runtime, and Chroma stores the embeddings and metadata.

The project also uses a lightweight browser detector to give immediate UI feedback. That browser result does not decide the search. The backend runs detection again so one model owns the actual retrieval path.

Detection and recognition are different jobs

A detector answers “where is the face?” A recognition model turns the cropped face into a numeric representation that can be compared with others.

The backend initializes InsightFace’s buffalo_l model pack and asks for detection and recognition:

self._model = insightface.app.FaceAnalysis(
    name="buffalo_l",
    allowed_modules=["detection", "recognition"],
    providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
)
self._model.prepare(ctx_id=0, det_size=(640, 640), det_thresh=0.5)

SCRFD is the detector in this setup. Its paper describes a method for redistributing training samples and computation across detection stages, with results reported on WIDER FACE. That is a narrower and more useful claim than calling it “accurate” in every setting. Lighting, pose, occlusion, image quality, and demographic performance still need evaluation for the population where a system will be used. See the SCRFD paper.

ArcFace is the recognition side. The original paper introduces an additive angular margin loss that encourages identities to separate on a normalized hypersphere. In the implementation I used, each detected face produces a 512-dimensional vector. See the ArcFace paper.

An embedding is not a name and it is not proof of identity. It is a representation optimized so that faces the model considers similar tend to be closer under a chosen distance function.

Bytes in, vector out

The application does not create its own temporary upload file before decoding the image with OpenCV. FastAPI’s UploadFile, however, uses a spooled temporary file and can move larger uploads from memory to disk. “RAM-only” was too broad a description for this route; the framework decides where the incoming upload is buffered. See the FastAPI file-upload documentation.

nparr = np.frombuffer(image_bytes, np.uint8)
image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
faces = self._model.get(image)

For every returned face, the code checks that the embedding contains 512 values. It does not perform an additional normalization step in this wrapper. If detection finds no face, the API returns HTTP 200 with an empty match list and a message.

“Processed in memory” is a property of this code path, not a complete privacy guarantee. Reverse proxies, hosting providers, crash reporting, request logs, and client behavior can still create copies. A production privacy claim would have to cover the entire deployment, including the Python function and its surrounding services.

Two other limits matter before deployment. The URL-search route fetches a supplied URL and follows redirects without blocking private-network destinations, which is an SSRF risk. The seed and clear-admin routes also have no authentication. This code should not be exposed as a public service in its current form.

The repository also omits insightface from backend/requirements.txt even though the embedder imports it. A clean installation therefore needs a dependency fix before the README setup can be called reproducible.

Cosine search in Chroma

A new collection is created with cosine space. An existing collection is loaded without checking which metric created it:

self._collection = self._client.get_or_create_collection(
    name="faces",
    metadata={"hnsw:space": "cosine"},
)

Chroma uses an HNSW approximate nearest-neighbor index for this configuration, and its collection documentation defines space as the distance function used by the index. Cosine is one supported option. See the Chroma collection configuration.

The query returns distances. The project converts cosine distance to a similarity value and filters weak results:

similarity = 1.0 - distance
if similarity < min_similarity:
    continue

This score is a ranking signal, not a calibrated probability that two images show the same person. A threshold chosen for a demo cannot be carried into a consequential use case without a labeled evaluation set, false-match measurements, and review of who bears the cost of an error.

What the index stores

Each demo record contains an embedding plus metadata such as the source URL, source name, title, category, and thumbnail URL. A 512-value float32 embedding occupies 2,048 bytes before database overhead:

512 values * 4 bytes = 2,048 bytes

That arithmetic explains why vector retrieval is practical. It does not tell us how a commercial service stores images, caches thumbnails, deduplicates people, or operates at a claimed scale. I do not have evidence for those implementation details, so the project no longer presents them as fact.

The repository’s query route does not save uploaded image bytes to disk. The vector index persists embeddings and metadata. The offline seeding code is less uniform: one path downloads images to a local directory and keeps successful files, while another processes bytes in memory. So I cannot honestly say the whole project stores no images. Search does not explicitly save the upload; one seeding path does.

The data source is the hard boundary

The working, documented source in the demo is the FBI Wanted API. The FBI publishes a REST endpoint that returns JSON records and accepts ordinary query parameters; its API documentation describes the interface.

The repository includes experimental crawler modules for other sources, but their presence is not evidence that those sources work or permit indexing. I kept the documented demo source to public FBI data and did not ship a private-person index. Any broader dataset would create consent, deletion, and false-match problems that the retrieval code does not solve.

What 252 faces can and cannot show

The small index let me test detection failures, ranked neighbors, and source metadata. The embedder can return several faces, but the current search endpoint queries only the first one. It also did not give me the data needed to measure broad recognition quality, fairness, production latency, or safe identification. That is what building the system showed me that chasing the extractor did not: retrieval code is the easy part; the dataset and the meaning of a match are the real system.

The code is public in the FaceCheck Pro repository. It shows the architecture. It does not prove that a match is correct.

This note also appears on Shellcat. Read that edition.

Biswas Workbench

Parable

Photos

Résumé

Books

Books I keep returning to

My favorites come first, followed by the Kafka and Dostoevsky collections and the rest of my reading shelf. Select any cover to open my note.

Cover images are served locally from Open Library and verified edition listings.

Music

Workbench TonesStopped

Three original synthesized tracks with a quiet office room bed. Sound starts only after you turn it on or press Play.

Contact Biswas

Send a note to my inbox, or use the direct links below.