Production LLM systems and the evals that keep them honest.
I'm Jonathan Reyes, a lead software engineer at Capital One. I build production LLM systems in regulated financial services, on 11 years of distributed-systems and platform work. Right now that means a document-intelligence pipeline on Python, a vision-language model (Llama) and S3, rebuilt from AWS Textract through Bedrock once phone-camera submissions started defeating OCR. I also run production MCP integrations across Jira, Confluence and GitHub, and set test strategy across five engineering teams. Off the clock I build boost, an open-source CLI whose retrieval engine is gated in CI on recall@k, MRR and nDCG@k.
LLM systems that run inside a regulated bank
Document intelligence on Python, a vision-language model and S3, plus the MCP integrations engineers on my teams use daily.
Document intelligence, from pilot to production
A pipeline that turns unstructured documents into structured findings.
- A fifth of submissions arrived as phone-camera photos, not PDFs, and OCR failed on them.
- I re-architected the OCR solution from AWS Textract through AWS Bedrock to a vision-language model (Llama). Bedrock handled more of the submissions than OCR could, but not the hardest ones — the VLM reads the image itself.
- The VLM reads images against the same extraction schemas, holding the accuracy analysts already relied on.
Agent tooling and MCP
Production MCP integrations across Jira, Confluence and GitHub, plus the team conventions for using them. boost ships its own MCP server:
- Hand-written JSON-RPC 2.0 over newline-delimited stdio, protocol version 2024-11-05.
- Six tools exposed to agents.
- A failing or unknown tool returns an error result instead of killing a long-lived session.
The platform underneath
Inference is one stage. The rest is durable orchestration and proof the work landed.
- EventBridge rules trigger Step Functions, which orchestrate AWS Batch jobs over S3 file drops.
- Companion state machines verify downstream delivery and alarm when a file never arrives.
- Instrumented with OpenTelemetry, and on my pager.
The hard part is knowing whether it works
A model swap, a prompt change or a schema change can make a system worse without anything failing.
Golden sets against a production pipeline
The document-intelligence pipeline's golden sets are built from analyst-provided reports, so every model or schema change is scored before it ships. A wrong extraction still comes back confident and well-formed. The same reviews cover the data: PCI, NPI and model-risk, including tokenization and masking.
Retrieval evaluation, enforced in CI
boost gates every merge on five retrieval metrics: recall@5, recall@10, MRR, nDCG@5 and nDCG@10.
- Absolute floors, calibrated below first measurement.
- A seeded 10,000-resample paired bootstrap, so a regression has to be both large and statistically significant to fail the build.
- A graded golden set of 36 queries and 107 relevance judgments; its generator refuses any judgment naming a skill missing from the corpus.
- Ragas faithfulness scoring runs in a weekly GitHub action to alert on generation-quality drift, rather than gating merges — a non-deterministic, token-costing metric makes a build flaky and expensive.
Cerebro — promotion gates for an autonomous system
An independent systematic-trading research platform on Nautilus Trader. It contains no ML. What transfers is the gate structure: no strategy reaches capital until it clears four of them.
- In-sample backtest.
- Walk-forward out-of-sample validation, with a Sharpe deflated for the number of configurations tried.
- Two weeks of paper trading.
- A human sign-off no code path can grant itself.
I rebuilt the performance layer from the account and fill streams after the engine's built-in analyzer reported wrong numbers.
Explore the architecture guide →boost — an agent-skills CLI and the harness that grades it
An independent, open-source CLI and MCP server that finds, installs and governs the skill files AI coding agents run on. Every merge is gated on what its retrieval core returns.
Its retrieval engine is BM25 written from scratch in pure standard-library Python: inverted postings, IDF, document-length normalization, and chunking on paragraph boundaries with overlap. A two-stage RAG pipeline retrieves a shortlist, then re-ranks it with an LLM.
- An optional dense backend embeds the same chunks through Voyage or OpenAI over plain urllib, no vendor client library, and drops back to BM25 when a key or model is missing.
- A hermetic CI gate scores retrieval over a generated 57-item corpus that needs no network access or API key.
- A second, independent CI gate runs over a pinned set of real registries with a recall floor.
boost scans installed Markdown for prompt injection and leaked credentials before an agent executes it as instructions. Findings are advisory by design. Every subsystem degrades instead of raising: without a model, key, index or the gh CLI, it still returns an answer.
boost pulls agent skills from Git-hosted registries into Claude Code, Windsurf and Cursor in one step. 70+ commands across 8 groups run over a curated catalog of 101 classified skill, rule and workflow registries. It ships on PyPI as boost-skill-cli.
$ python scripts/eval_gate.py --run corpus 57 entries · 36 queries · 107 graded judgments arm bm25 (shipped) catalog (reference) recall@5 0.507 floor 0.45 PASS 0.403 recall@10 0.602 floor 0.55 PASS 0.535 MRR 0.838 floor 0.78 PASS 0.733 nDCG@5 0.692 floor 0.64 PASS 0.550 nDCG@10 0.714 floor 0.66 PASS 0.583 paired bootstrap · 10,000 resamples · no significant regression GATE PASS # hermetic run · internal baseline, committed at evals/baseline.json # source · github.com/jonnyeclectic/boost
Eleven years across two companies
Eleven years of distribution and governance for the things engineers depend on, first binaries and now agent skills.
Capital One
Plano, TX June 2019 – PresentLead Software Engineer
Feb 2023 – PresentRemoteLead architecture and delivery for a document-intelligence pipeline built on Python, a vision-language model (Llama) and S3. I started it alone and grew it into a team effort.
- Re-architected its OCR solution from AWS Textract through AWS Bedrock to a vision-language model (Llama) when image-based submissions defeated OCR — Bedrock moved more of them than OCR could, but not the hardest ones — tuning extraction schemas, prompts and golden-set evaluations in production.
- Run production MCP integrations across Jira, Confluence and GitHub, and set the team conventions for agent tooling.
- Lead architecture for a high-throughput platform orchestrating communications with external dealership partners, including a zero-loss migration of a highly stateful distributed system between AWS accounts by traffic splitting live workloads.
- Support PCI, NPI and model-risk reviews, including tokenization and masking.
- Own production operations: on-call, incident triage, root-cause investigation and observability on OpenTelemetry, CloudWatch, New Relic, Splunk, PagerDuty and Prometheus.
- Serve as Testing Champion for the Sales organization, defining test strategy adopted across five engineering teams.
- Recognized with two ACE Awards for Technical Engineering Excellence (2024, 2025) and the Diamond Dealer Award (2024).
PythonJavaVLM (Llama)S3AWS BedrockMCPOpenTelemetryPrincipal Associate Software Engineer · Prequalified Applications
June 2021 – Feb 2023RemoteArchitected and operated external-facing services powering auto loan prequalification for public applicants, owning the API contracts, data models, AWS deployment and production support. Partnered with product and risk stakeholders to turn regulated lending requirements into service architecture.
API designAWSregulated lendingPrincipal Associate Software Engineer · Artifactory Platform & Metrics
June 2019 – June 2021Ran the platform around JFrog Artifactory, the company-wide binary repository, with services deployed via Terraform. Automated artifact promotion in production pipelines using AWS Lambda and DynamoDB, which removed a manual release step for engineering teams across the enterprise. Also delivered performance-analytics microservices and metrics pipelines feeding machine learning models. It is the problem boost now solves for agents: a registry, a promotion path and a lockfile.
JFrog ArtifactoryAWS LambdaDynamoDBTerraformIBM
Dallas, TX 2015 – 2019Software Engineer II
2017 – 2019Built OpenShift (Kubernetes) deployments for microservices. Migrated server provisioning to Netflix Conductor workflow orchestration: sequential job scheduling became parallel execution, and large-data processing dropped from hours to minutes.
OpenShiftNetflix ConductorSoftware Engineer I · SoftLayer (acquired by IBM)
2015 – 2017Developed APIs for globally integrated services. Built Splunk dashboards for service metrics and health checks, and contributed to the CI/CD functional test framework.
APIsSplunkCI/CDThe stack I work in
AI systems and the platform they run on.
AI & LLM systems
I run all of this inside a data-classification boundary.
Evaluation & retrieval quality
Five metrics, absolute floors and a seeded bootstrap, all of which have to agree before a merge lands.
Agent tooling
The protocol layer that agents call, and the scanners vetting what they execute.
Observability & operations
Where I look first when something breaks at 3am.
Cloud & distributed systems
Eleven years of distributed systems, on AWS since 2019.
Languages, APIs & data
External-facing contracts and the data behind them.
Technical guides I write
Written for myself first, then for the engineers I work with. Worked examples are reference designs unless a guide says otherwise. Walkthroughs of my own projects live with those projects: the boost visual guide and the Cerebro architecture docs.
pgvector, explained twice — vector search in PostgreSQL
Reference guide · retrieval and evaluationEvery section written twice: the plain-English version and the engineering version, side by side.
- Embeddings as a schema contract.
- HNSW versus IVFFlat.
- The operator-class mismatch that degrades to a sequential scan.
- Over-fetch-and-fuse ranking.
- recall@k, MRR and nDCG golden-set gates.
UNT computer science, AWS certified
BS, Computer Science — University of North Texas
2011 – 2015Minor in Mathematics, with certificates in Technical Communication and Game Programming.
AWS Certified Solutions Architect – Associate
2025Architecture, security and cost trade-offs across core AWS services.
If you want the detail behind any of this, ask me.