How to Monitor LangChain Agents in Production

The Challenge

LangChain agents in production are black boxes. You can see the final output, but you have no visibility into intermediate chain steps, retriever performance, tool call success rates, or token consumption trends. When an agent starts hallucinating or a retriever returns stale documents, there is no built-in mechanism to detect the degradation. Teams often discover issues through user complaints, not proactive monitoring. Without observability, you cannot set SLOs, track costs, or demonstrate to auditors that your AI agents are behaving as intended.

How NodeLoom Solves This

The NodeLoom Python SDK includes a native LangChain callback handler that integrates in two lines of code. The callback handler automatically captures every chain invocation, LLM call, tool execution, and retriever query as structured spans within a trace. All telemetry is sent asynchronously with less than 1ms of overhead. Once telemetry is flowing, NodeLoom automatically learns behavioral baselines and alerts you when drift is detected — whether that is a spike in token usage, a drop in tool call success rate, or a change in output sentiment.

Step-by-Step Implementation

1. 1

   Install the NodeLoom Python SDK

   Add the NodeLoom SDK to your Python project. The SDK includes built-in integrations for LangChain, CrewAI, and other popular frameworks.

   pip install nodeloom

2. 2

   Initialize the SDK and create the callback handler

   Import the NodeLoom client and the LangChain callback handler. Initialize with your API key and the agent ID from the NodeLoom dashboard. The callback handler wraps LangChain's BaseCallbackHandler and captures all chain events automatically.

   from nodeloom import NodeLoom
   from nodeloom.integrations.langchain import NodeLoomCallbackHandler
   
   nl = NodeLoom(api_key="nl_your_api_key")
   handler = NodeLoomCallbackHandler(client=nl, agent_id="agent_abc123")

3. 3

   Attach the callback handler to your LangChain agent

   Pass the callback handler to your LangChain agent, chain, or executor via the callbacks parameter. Every invocation will now be traced end-to-end, including LLM calls, tool executions, and retriever queries.

   from langchain.agents import AgentExecutor, create_openai_tools_agent
   from langchain_openai import ChatOpenAI
   from langchain.tools import DuckDuckGoSearchRun
   
   llm = ChatOpenAI(model="gpt-4o", temperature=0)
   tools = [DuckDuckGoSearchRun()]
   agent = create_openai_tools_agent(llm, tools, prompt)
   
   executor = AgentExecutor(agent=agent, tools=tools)
   result = executor.invoke(
       {"input": "What is the current price of AAPL?"},
       config={"callbacks": [handler]}
   )

4. 4

   Review captured telemetry in the NodeLoom dashboard

   Once the agent runs, NodeLoom captures a full trace with nested spans. Each trace includes: the total execution duration, every LLM call with input/output tokens and model name, every tool call with arguments and return values, retriever queries with document counts and relevance scores, and the final agent output. You can filter traces by agent ID, time range, status, and cost.

5. 5

   Set up drift detection and alerts

   NodeLoom automatically learns baseline behavior after 50 executions. Navigate to the agent's Drift Detection tab and configure thresholds for: token usage drift (e.g., alert when average tokens exceed 2x baseline), latency drift (alert when P95 latency exceeds 5 seconds), tool call failure rate (alert when failures exceed 10%), and output sentiment shift. When drift is detected, NodeLoom triggers your configured incident response playbook — which can send Slack notifications, create Jira tickets, or quarantine the agent.

Key Benefits

Frequently Asked Questions

Does the NodeLoom callback handler work with LangChain Expression Language (LCEL)?

Yes. The NodeLoomCallbackHandler works with both legacy LangChain chains and the newer LCEL pipe syntax. Pass it via the config parameter when invoking any runnable.

What LangChain components are traced?

The callback handler traces LLM calls (ChatOpenAI, ChatAnthropic, etc.), tool executions, retriever queries, chain invocations, and agent executor steps. Each component becomes a span within the parent trace.

Can I monitor multiple LangChain agents with one SDK instance?

Yes. Create one NodeLoom client and multiple callback handlers, each with a different agent_id. Each agent gets its own dashboard, baselines, and drift detection configuration.

Does this work with LangGraph?

Yes. LangGraph agents accept the same callbacks parameter. The NodeLoom handler captures graph node executions, state transitions, and conditional edges as spans within the trace.

What happens if the NodeLoom API is unavailable?

The SDK is designed to be non-blocking. If the NodeLoom API is unreachable, telemetry is buffered locally and retried. Your agent continues to function normally — monitoring is never in the critical path.