The receipt scanner in Finly looks like one model call. You photograph a receipt, a vision model reads it, and you get a draft transaction. Most of the work is in what surrounds that call: what the model is told, what gets checked on the way back, and what the user sees when it goes wrong.
What the model is given
The app uploads the photo to our API, which accepts JPEG, PNG, GIF and WebP up to 20 MB and rejects everything else before any model is involved. The server then loads the user's own categories and income sources and puts them in the prompt, so the model is choosing from names the person already uses instead of inventing a fresh taxonomy for every receipt.
The model is asked for a bare JSON array and nothing else. It is also asked whether the document is money going out or coming in, because the prompt covers pay stubs and invoices people have received, not just purchases, and an income document needs a different shape from an expense.
What comes back, and what is checked
Models like to wrap JSON in a markdown fence even when told not to, so there is a small helper that strips the fence before parsing. After parsing, the only structural check is that the result is an array. If parsing fails, the user gets a plain message asking them to try again, and the failure is logged and counted.
Categories are matched to the user's list case-insensitively. If the model proposes a category that does not exist, the server creates it, one at a time rather than in parallel, to avoid races with simultaneous requests. If creation fails anyway, it logs a warning and carries on, so one bad category cannot sink the whole receipt. A transaction whose category cannot be matched comes back without one instead of failing.
Nothing is saved at this point. The scanner returns drafts. In the form the user edits and saves them; in the chat screen each line item becomes its own confirmation card.
The bug that was about dates
Receipts often print a partial date, or none that the model can read. Left alone, the model resolves the gap against its training cutoff, so a receipt from this week can end up filed in 2023. The fix is unglamorous: the server works out today's date in code and puts it in the prompt as the anchor, with an explicit instruction to use the current year when the year is missing and today's date when there is no date at all.
The wrong dates look plausible, which is what makes this kind of failure easy to miss.
What it does not do
As written, the extractor assumes the receipt is in the account's currency. It takes the user's currency from the app, falling back to a lookup, tells the model to read amounts as that currency, and stamps it on every result. A receipt printed in another currency is not detected. PDFs are rejected on this path for now, with a clear message; statements go through a separate import in a background queue.
When extraction fails in the chat screen, the user gets a message that suggests a clearer photo or typing the amount. When it finds nothing, it says so, instead of showing an empty card. Successes and failures are counted, which tells us when it breaks but not how accurate it is. There is no accuracy figure in this note because the code has none, and a proper evaluation set is the gap we would close first.