Ruby

Support that can see

Customers buy sophisticated hardware and get a PDF. When something goes wrong, they scrub through a 40-minute video to find the 30 seconds they need, or email support and wait. The answer usually exists, but it is separated from the product, the customer's setup, and the moment they need it.

Ruby was AppliedMind's first product. A customer scans a QR code, points their phone at the product, and asks a question. Ruby identifies the model, the part, and its visible state, answers by voice, and plays a video clip for that exact step.

Because Ruby could see the product in context, the customer did not have to name an unfamiliar part, translate a generic diagram to the device in front of them, or describe what the camera already showed. Ruby could guide them from unboxing through troubleshooting in the phone browser, without an app or headset.

Support becomes product research

The moment a customer reaches for help is one of the clearest signals a product team gets from the field. Most support systems are built to resolve that moment, not preserve it. A ticket becomes a category, a call becomes a summary, and a design review gets a bar chart of complaints. The room the product sits in, the eight-second hesitation, and the workaround the customer invented are gone.

Ruby is present before those details disappear. Because the session runs through the customer's camera while they are stuck, it captures the environment, the question, the failed step, and whether the guidance worked. Each support session becomes a small field study with video evidence of where the design gives way.

Instead of receiving “users complain about pairing,” the design team can see something like this: 80% of the customers who asked for help could not enter pairing mode because nothing indicated that the button had to be held for five seconds. Here are the clips, grouped by firmware version and ranked by signs of frustration.

Later sessions show what happened after a fix shipped. If firmware v2.3 removes the pairing problem but creates a new one, the cluster moves rather than disappearing. The support interaction has now become the start of product research: the same lens that helps a customer use the current product also helps the team design the next one.

A product the brand can inspect

Ruby had to become useful without asking a brand to spend months preparing content. Its crawlers turn the material the brand already has — its website, manuals, support articles, and product videos — into structured how-to guides.

The guides appear in a dashboard where the brand can read, correct, and reuse them internally or publish them on its website. When guidance is wrong, the person who knows the product edits the guide rather than an agent prompt.

The live system is equally inspectable. For every answer, the brand can trace which product Ruby identified, which guide it used, what video it showed, and what the customer did next. Teams can inspect a bad answer, correct its source, and see whether the change worked in later sessions. That is what enterprise-grade observability and traceability meant in Ruby.

From AR-first to phone-first

Ruby began as an AR product, extending the situated displays and remote-expert overlays from my PhD work. The idea was to let customers troubleshoot hands-free, with virtual aids registered to the actual device in front of them.

Customer testing changed the premise. Pretty much no one has a headset at home, and using modern ARKit or ARCore capabilities would require people to install an app just to ask a question. Testing in the field and with people from Facebook groups made it clear that the lowest-friction version was something anyone could open on a smartphone. A modern web app was the answer.

Ruby became our answer to Project Astra for hardware brands: a camera-aware agent that could see a product, talk with its owner, and show the next step. A customer could scan a QR code, allow camera access, and start. Authoring moved to the brand's existing material rather than a library of hand-made AR scenes.

My role

I designed the customer flow, the operator dashboard, and the system that fed what Ruby learned in each camera session back into product design. I also bootstrapped the stack: a multimodal LLM for the conversation, a custom RF-DETR model trained to recognize each brand's product parts, WebRTC for the live video call, an iOS prototype with ARKit, and a Next.js / TypeScript / Node.js / Firebase dashboard.

For vision, I first expected to fine-tune a VLM for each brand. A smaller, purpose-trained RF-DETR model worked better for finding the specific parts of a product. It ran faster and cost less than a general vision-language model, and was easier to package for each customer.

The live VLMs I tested sampled video at roughly one frame per second, while a status LED, a hand entering the frame, or a quick change in position could happen between samples. Ruby ran the full call over WebRTC alongside the frames sent to the VLM. That preserved the missing moments for product-specific vision and for later review.

The first client drew guidance over an ARKit depth mesh. Testing showed that the install step lost people before the AR could help them, so the shipping path became QR-code-to-web: scan, allow the camera, and start.

My co-founder, Safwan Siddiqui, led outreach and customer development. Together, we brought Ruby to private beta with brands before AppliedMind wound down.

Related open-source work

Training that RF-DETR model meant labeling a lot of video. We built a video annotation tool on top of SAM 3 to label video data quickly, and released it open source.