Operational Controls and Observability: Knowing the Right Numbers at All Times

Running an NLP system at national scale isn’t just about throughput or clever architecture. It’s about visibility. It’s about knowing what the system is doing, how it’s changing, and where it’s drifting — every hour of every day. And it’s about having the tools to act on that information immediately.

This article focuses on the operational controls that keep our system stable while processing 20M+ pages per day for 20,000+ users, and how we turn raw operational data into actionable decisions.

Nightly Reporting: The System’s Daily Diagnostic

Every night, the system generates a comprehensive report that captures the previous 24 hours of activity. It’s one of the most important operational tools we have.

What the nightly report includes

• All technical errors, grouped by subsystem
• Average and maximum timings for every major task type
• Total data processed, broken down by day and hour
• Top 5 slowest tasks per task type, with full performance metrics
• A two‑week history of data‑related metrics to detect provider drift

This isn’t a dashboard for executives — it’s a diagnostic instrument for engineers.

How we use it

• Slow tasks reveal bottlenecks
• Timing spikes reveal infrastructure pressure
• Error clusters reveal systemic issues
• Two‑week trends reveal data drift long before it becomes a production problem

The nightly report gives us a clear, quantitative picture of system health — and how that health is evolving.

Validation Console: Deep Debugging in a Production‑Secure Environment

When the nightly report flags anomalies, we investigate them using our Validation Console, a secure web application designed for deep debugging.

The console allows engineers to:

• Export cases with anomalies
• Replay them in a backup environment within the same VPC
• Debug them with production‑equivalent security controls
• Compare outputs, timings, and structured extraction
• Validate fixes before deployment

The backup environment uses different hardware, so it doesn’t mirror production performance — but it does mirror production security, which is the critical requirement for handling sensitive data.

Why it matters

• Engineers can safely inspect real cases
• Anomalies can be reproduced and analyzed without risk
• Fixes can be validated before they reach production
• Debugging is fast, controlled, and compliant

The Validation Console turns anomalies into actionable engineering work.

Real‑Time Management Console: Live Operational Awareness

Some issues can’t wait for a nightly report.  That’s where the Management Console comes in.

It provides real‑time visibility into:

• Current processing throughput
• Active tasks per server
• Queue depth
• Routing distribution
• Live performance metrics

But its most powerful capability is something few systems offer:

Real‑time stack traces from every thread on every server

At any moment, we can:

• Pull a full stack trace from every thread
• Identify bottlenecks instantly
• Detect deadlocks as they form
• See exactly what each server is doing

This capability has allowed us to quickly resolve:

• Performance regressions
• Rare concurrency issues
• Obscure deadlocks
• Unexpected library behavior

It’s the difference between guessing and knowing.


Acting on the Data: The Feedback Loop That Keeps the System Healthy

Observability only matters if it drives action. In our system, it does — every day.

When we see slow tasks…

We optimize the code path, adjust resource allocation, or refine data modeling.

When we see repeated user queries…

We promote those concepts into structured data.

When we see provider drift…

We update extraction logic, region detectors, or classification models.

When we see routing imbalance…

We adjust the PostgreSQL‑backed round‑robin distribution.

When we see latency spikes…

We scale web servers, tune analyzers, or rebalance SOLR load.

When we see anomalies…

We export cases to the Validation Console and debug them immediately.

This is how a national‑scale system stays stable:
observe → detect → investigate → act → validate → repeat.

The Bottom Line: White‑Box Transparency Is the Only Way to Scale

Most systems fail because they operate as black boxes.  At national scale, that’s not an option.

To run a robust NLP system that processes tens of millions of pages per day, you need:

• White‑box transparency
• Operational discipline
• Continuous measurement
• Actionable reporting
• Real‑time visibility
• Production‑secure debugging environments
• A culture that treats observability as a first‑class feature

You need to know the right numbers — and you need to know them all the time.

That’s how you keep a system this large predictable, trustworthy, and fast.