Somewhere on the internet I once read: "You can make almost anything, as long as you can turn raw data into numbers." Most of my projects are some version of that idea, and this one is no different. What makes it worth writing about is where it came from, and why the most important design decision was to keep a large language model out of it.
The problem: generative AI under pressure
I was sitting with my elder sister, an online teacher, while she took a live physics class for twenty students. She was close to the end of the lesson and needed to get through one last numerical before time ran out.
When we're in a hurry, our brains sometimes switch into lazy mode. She went blank halfway through the calculation. In a panic she started typing the problem into ChatGPT, then uploaded a photo of it, hoping for a quick answer.
It went the wrong way. The chatbot dug too deep: it over-explained the theory, hallucinated along the way, and produced a long, polished essay when all she needed was one hard number.
That's when it clicked for me. Sometimes we don't need a long, polished answer. We need a precise one. She wasn't trying to learn a new concept; she was confirming steps she already knew.
So I built the Neuro-Symbolic Math Solver: a lightweight web app where you draw an equation or upload a photo of one, and get back the exact answer. No essays. No hallucinations.
How it works, end to end
Before the details, here's the whole pipeline in one place:
draw or upload → base64 image → FastAPI
1. normalise polarity (white symbols on black, always)
2. segment characters (OpenCV contours, left to right)
3. classify each one (CNN trained from scratch, 14 classes)
4. evaluate exactly ("12+34" → 46)
5. return JSON → React shows the answer
The neural network handles the part that is genuinely fuzzy: reading messy handwriting. Plain, deterministic code handles the part that must be exact: the arithmetic. That split is what "neuro-symbolic" means here.
Phase 1: A frictionless frontend (React and Vite)
The goal was to capture an equation with as little friction as possible. I chose React with Vite for speed and had my coding agent (Antigravity) scaffold the interface from this prompt:
Design a frontend website interacting with the user. Features: the goal is a frictionless user experience. Tech stack: React JS, Google Fonts. Component 1: a drag-and-drop upload zone, plus an HTML5 <canvas> element allowing users to hand-draw an equation directly on the screen. Component 2: capture the image or canvas data as a Base64-encoded string and send it asynchronously to the backend via a POST request. Keep the application lightweight.
If you're wondering why I'm happy to admit an agent wrote my boilerplate: UI scaffolding is exactly the kind of work AI is good at, and the interesting engineering was elsewhere. I wrote about where that line sits in Stop Vibe Coding and in Should You Be Scared of Agentic AI?
Phase 2: The backend and image polarity (FastAPI)
This is the part where I don't hand anything to AI. A FastAPI server receives the image and does the heavy lifting.
The first problem is subtle. One person uploads a photo of dark ink on white paper; another draws white strokes on a dark digital canvas. The rest of the pipeline needs one consistent format, so convert_base64_to_matrix measures the image's mean brightness and picks the threshold accordingly:
brightness = cv2.mean(image)[0]
if brightness < 127:
# Dark background: the strokes are already the bright pixels.
_, binary = cv2.threshold(image, 127, 255, cv2.THRESH_BINARY)
else:
# Light background: invert, so dark ink becomes white foreground.
_, binary = cv2.threshold(image, 127, 255, cv2.THRESH_BINARY_INV)
Whatever comes in, what goes out is always white symbols on a black background, which is exactly what the next step expects.
Phase 3: Character segmentation (OpenCV)
Next, every symbol has to be isolated. cv2.findContours finds the connected white regions. I use RETR_EXTERNAL, which keeps only outer contours, so the hole inside a "0" or an "8" isn't counted as a separate symbol. Anything under 50 square pixels is treated as noise and dropped. The remaining bounding boxes are sorted by their x-coordinate, so the equation reads left to right:
contours, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
boxes = [cv2.boundingRect(c) for c in contours if cv2.contourArea(c) > 50]
boxes.sort(key=lambda box: box[0]) # left-to-right reading order
Each cropped symbol then gets a 20-pixel black border, so it sits centred like the training data, and is resized to 28 × 28 pixels. Why 28 × 28? That's where the deep learning comes in.
Phase 4: The "neuro" part (training a CNN in Colab)
I didn't want to call an external API or borrow someone else's pre-trained model, so I trained a convolutional neural network from scratch with TensorFlow and Keras.
It needs to recognise 14 classes: the digits 0 to 9 and the operators +, -, * and /. Digits are easy, because MNIST provides 60,000 labelled handwritten examples. Operators are not, and instead of hunting for a dataset I generated one. In a Colab notebook, a short script draws each operator with cv2.line, randomising stroke thickness and position so the samples look like messy handwriting rather than perfect glyphs:
thickness = np.random.randint(1, 4)
ox, oy = np.random.randint(-4, 5), np.random.randint(-4, 5) # random offset
plus = np.zeros((28, 28), dtype=np.uint8)
cv2.line(plus, (14 + ox, 6 + oy), (14 + ox, 22 + oy), 255, thickness)
cv2.line(plus, (6 + ox, 14 + oy), (22 + ox, 14 + oy), 255, thickness)
Six thousand samples per operator gave me 24,000 synthetic symbols to combine with the 60,000 MNIST digits. The network itself is deliberately small: two convolution and pooling stages, then a dense layer and a 14-way softmax.
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation="relu", input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation="relu"),
tf.keras.layers.Dense(14, activation="softmax"), # 0-9 and + - * /
])
After ten epochs in Colab it passed 99% training accuracy, and I exported it as mvp_math_solver.h5 for the live app.
Phase 5: Real-time inference (TensorFlow in the backend)
Training a model in a notebook and serving it in a web app are different problems. The backend loads the .h5 model once, when the server starts, so no request pays the cost of reading weights from disk.
When an equation arrives, OpenCV cuts it into those 28 × 28 boxes, and model.predict() classifies each one independently. The predicted class indices map back to characters, and the drawing becomes a plain string such as "12+34".
Phase 6: The "symbolic" part (exact evaluation)
This is where the architecture earns its name. Why not hand that string to an LLM? Because LLMs predict; they don't calculate.
Instead, the string is evaluated deterministically. Because the CNN can only ever output its 14 classes, the only strings that can reach the evaluator are made of digits and the four basic operators. That constraint is what makes it safe to pass the expression to Python's built-in eval(): there is no path for arbitrary text to get there. The day the vocabulary grows (brackets, variables, functions), that guarantee goes away, and eval() has to be replaced with a proper expression parser.
The result is exact, repeatable maths every time. That's the appeal of neuro-symbolic AI: the fuzzy pattern recognition of a neural network, combined with the strict, rule-based logic of mathematics.
Phase 7: Putting it together
Once the expression is evaluated, FastAPI returns a small JSON payload (the recognised expression and the answer) and the React frontend shows it.
No typing out formulas. No waiting for a chatbot to finish "thinking" and writing paragraphs. Draw it, send it, get the answer.
Limitations and what's next
This is an MVP, and I'd rather be clear about where it stops. It handles single-line arithmetic well. Multi-line problems, calculus or integrals are beyond it today, both because the vocabulary is limited to 14 symbols and because the segmentation assumes one line read left to right.
I'm building it in public, and extending it to multi-line problems and richer mathematical reasoning is the direction I'm working towards. The code is on GitHub, and a star there genuinely helps me keep going.
The takeaway
That's how a moment of live-class panic turned into a project. Building it reminded me that we don't always need a massive generative model to solve an everyday problem. Sometimes a small, targeted system built from scratch does the job faster, more safely and more reliably, precisely because it can't make things up.
If you have thoughts on neuro-symbolic architectures, or ideas for the next version, I'd love to hear them on LinkedIn.