How automatic background removal works, and why hair is the hard part

Ten years ago, removing a background meant an hour with the pen tool. Now it takes a second in a browser tab. The interesting question is what changed — and the answer explains, precisely, why these tools nail some photos and mangle others.

The problem is harder than "find the edges"

The obvious approach is to detect where the subject ends and cut there. It fails immediately, because for a great many pixels there is no "where it ends".

Consider a single pixel on the outer edge of someone's hair. Physically, that pixel covers an area containing a strand of hair and some background. The camera recorded one colour for it: a mix. There is no correct place to cut. Cut it out and you lose the strand; keep it and you keep a halo of the old background. The same is true of motion blur, of a wine glass, of a smoke wisp, of the soft edge of anything slightly out of focus.

So the real task is not classification but matting: for every pixel, estimate what fraction of it belongs to the subject. That fraction is the alpha value — 1 for fully subject, 0 for fully background, and everything between for the pixels that are honestly a bit of both.

The output of a good background remover is not a cut-out shape. It is a full-resolution greyscale map of those fractions, applied as the image's alpha channel. This is why the result has to be saved as PNG or WebP: JPG has no alpha channel, so exporting a cutout as JPG silently fills every transparent pixel with white.

What the model is doing

Modern removers run a segmentation network — ours uses RMBG-1.4 — that has been trained on a very large number of images paired with hand-made alpha maps. It is not looking for edges. It has learned what people, products, animals and plants look like, and it predicts the alpha map for an image it has never seen the way a language model predicts a word.

Two things follow directly from that.

First, it works best on what it was trained on. People, pets, cars, clothing and product shots are hugely over-represented in the training data, so those are near-flawless. An unusual object photographed unusually — machinery, lab equipment, an abstract sculpture — is guesswork, and it will look like guesswork.

Second, it has no concept of what you meant. Given a photo of two people it will keep both, because both are foreground. It cannot know you wanted one of them. Given a person holding a cup, whether the cup survives depends on whether the model reads it as part of the subject, and there is no setting for your intention.

Why it runs in your browser at all

A model like this is a few tens of megabytes of weights. That is a large download but an entirely feasible one, and once it is in the tab, WebGL or WebGPU runs the inference on your own graphics hardware. A typical photo takes a second or two.

The consequence worth caring about is that your photo is never transmitted. The model comes to the image rather than the image going to a server. For anything with a face, a document or a product you have not launched yet, that is not a minor detail.

Our background remover, background changer and background blur all work this way, on the same matte. Note that RMBG-1.4 is licensed for non-commercial use, which is stated on each of those pages.

Where it goes wrong, and why

Fine hair against a busy background. The classic failure. Each strand is thinner than a pixel, so the correct alpha for those pixels is some middling fraction. The model has to infer hundreds of them from context, and when the background behind them is high-contrast and detailed, it gets some wrong. The tell is a faint fringe of the old background colour through the hair.

Subject and background the same colour. A grey jacket against a grey wall gives the model almost nothing to work with. It is guessing from shape alone, and the boundary wanders.

Transparent and reflective things. A glass, a bottle, a pair of spectacles. Physically these pixels are partly background — you can see through them — so the correct answer is a partial alpha that also, ideally, keeps what is visible through the object. Almost every automatic tool either makes glass fully opaque or removes it entirely. Neither is right.

Low-resolution input. The model works at a fixed internal resolution. Give it a 300-pixel-wide image and there is simply not enough information at the boundary; the matte comes back soft and approximate.

Heavy JPG compression. Compression artefacts around high-contrast edges — the ringing you get from an over-compressed JPG — are precisely at the boundary the model is trying to read, and it reads them as real detail.

Getting a better result

• Feed it the largest original you have. Not the version already resized for Instagram. More pixels at the boundary is the single biggest improvement available.
• Shoot with separation in mind if you can. Contrast between subject and background, and a background that is out of focus, both make the matte dramatically cleaner.
• Save as PNG or WebP. Every time. A cutout saved as JPG is a cutout thrown away.
• Composite onto a similar colour when it is imperfect. A slight fringe of the old white background is invisible on a light new background and glaring on a dark one. If you know the cutout is going on black, be fussier about the matte.
• Do the removal first, then resize. Matting a large image and then scaling down averages small errors away. Scaling down first and then matting bakes them in.

What to expect, honestly

For a person or a product against a reasonably distinct background, an automatic remover now beats a rushed manual selection and takes a second instead of twenty minutes. For hair against foliage, for glass, for a subject that matches its background, it will get you 90% of the way and leave a boundary that a person still has to look at.

That is a genuinely large change from where this was a few years ago. It is not the same as "solved", and any tool that claims otherwise has not shown you a photo of someone with curly hair standing in front of a hedge.