Tracing RAG Retrieval and Ingestion with OpenTelemetry

When an AI agent or RAG pipeline stalls in production, the final response rarely tells you why. A single retrieval request might generate query embeddings, execute hybrid vector searches across multiple memory spaces, pass candidate chunks through a reranker, and stream a synthesized summary from an LLM. Meanwhile, asynchronous ingestion workers are chunking and embedding new documents in the background.

When latency spikes or retrieval quality degrades, treating your memory layer as a black box makes root-cause analysis guesswork. You cannot tell whether a 4-second delay came from an overloaded embedding endpoint, a slow reranker, or an upstream LLM timeout, or whether a reranker failed and the pipeline fell back without saying so.

OpenTelemetry (OTel) is an open standard for collecting traces, metrics, and logs [1]. GoodMem exports OTel traces across both the retrieval and ingestion pipelines [2]. You can route the trace data into LLM observability platforms such as Langfuse and LangSmith, or into the infrastructure monitoring backends an enterprise platform team already runs, such as Jaeger and Grafana Tempo. How that support compares across memory systems is on the enterprise observability page. GoodMem’s existing Prometheus and Micrometer metrics stay on the Prometheus endpoint, and application logs stay where they are, carrying the trace and span IDs whenever a span is current.

Why trace a memory layer

Standard application performance monitoring tools tell you whether an HTTP or gRPC request succeeded, which is the outside of the request. GoodMem structures every API call and background job into an OpenTelemetry trace composed of individual operation spans.

A retrieval trace answers operational questions:

Where did the latency go? Break down wall-clock time across query embedding, vector search, reranking, and generation spans.
Did the pipeline run as configured? Verify which models were actually invoked at query time, and see immediately if a stage succeeded, timed out, or triggered a fallback.
Why did ingestion fail? Track asynchronous background processing attempts to pinpoint the exact chunking or embedding step where a batch of documents errored out.
What did the request consume? GoodMem records token usage on embedding, reranking, and LLM spans whenever the provider reports it. An absent counter means usage is unknown while an explicit zero means the provider reported zero. GoodMem does not estimate the missing ones.

Here is one request from the agent we run against our own company material, a question that searched its chat, chat_messages, and wiki spaces.

Trace b72b5ab6 · 39 observations · 9.17 s totalAgent (Python)GoodMem serverProvider call

company-memory9.17 s
llm2.33 s
search_chat693 ms
POST /v1/memories:retrieve687 ms
goodmem.retrieval685 ms
goodmem.retrieval.plan1 ms
goodmem.retrieval.query_embedding275 ms
gen_ai.embeddings274 ms
goodmem.embedding.provider_attempt274 ms
goodmem.retrieval.vector_search58 ms
goodmem.retrieval.fusion0 ms
goodmem.retrieval.hydration1 ms
goodmem.retrieval.postprocess350 ms
gen_ai.rerank349 ms
search_chat_messages529 ms
POST /v1/memories:retrieve528 ms
goodmem.retrieval526 ms
goodmem.retrieval.plan0 ms
goodmem.retrieval.query_embedding269 ms
gen_ai.embeddings269 ms
goodmem.embedding.provider_attempt269 ms
goodmem.retrieval.vector_search3 ms
goodmem.retrieval.fusion0 ms
goodmem.retrieval.hydration1 ms
goodmem.retrieval.postprocess252 ms
gen_ai.rerank251 ms
search_wiki533 ms
POST /v1/memories:retrieve533 ms
goodmem.retrieval531 ms
goodmem.retrieval.plan1 ms
goodmem.retrieval.query_embedding265 ms
gen_ai.embeddings265 ms
goodmem.embedding.provider_attempt265 ms
goodmem.retrieval.vector_search1 ms
goodmem.retrieval.fusion0 ms
goodmem.retrieval.hydration0 ms
goodmem.retrieval.postprocess262 ms
gen_ai.rerank261 ms
llm5.08 s

One question answered from three memory spaces. The agent calls the LLM once to plan the searches and once to write the answer; the three retrievals in between are the GoodMem server and its two providers.

Retrieval stages are named goodmem.retrieval.* and provider calls gen_ai.*. Ingestion, which this trace has none of, runs under goodmem.job.*. Every provider call has an attempt span beneath it, and a retry adds a second attempt with a backoff span between the two, so the wait appears as its own bar instead of as an embedding that took four seconds. The full inventory, roughly 300 attribute keys, is in the spans and attributes reference [3].

The two llm spans are 2.33 s and 5.08 s, which is 7.41 s of the 9.17 s. All three retrievals together are 1.755 s. Inside each one, the reranker is almost all of postprocess: 349 ms of 350, 251 of 252, 261 of 262. Embedding is around 270 ms a call and the vector search itself is 58 ms, 3 ms, and 1 ms. The memory layer was not the slow part of this request, and we would have guessed otherwise.

When a stage fails

On our demo deployment the summarizing LLM sits behind a proxy that injects HTTP errors. Here is a retrieval that went through it.

Trace 0ec1fbe0 · 19 observations · 6.68 s totalAgent (Python)GoodMem serverProvider call

company-memory6.68 s
llm2.83 s
search_projects1.81 s
POST /v1/memories:retrieve1.81 s
goodmem.retrieval1.80 s
goodmem.retrieval.plan4 ms
goodmem.retrieval.query_embedding623 ms
gen_ai.embeddings619 ms
goodmem.embedding.provider_attempt618 ms
goodmem.retrieval.vector_search11 ms
goodmem.retrieval.fusion1 ms
goodmem.retrieval.hydration4 ms
goodmem.retrieval.postprocess1.16 s
gen_ai.rerank335 ms
gen_ai.chat725 ms
goodmem.openai.http_attempt88 ms
goodmem.openai.retry_backoff482 ms
goodmem.openai.http_attempt3 ms
llm2.04 s

The two short bars under gen_ai.chat are the HTTP attempts. The long one between them is the backoff.

The embedder, the vector search, and the reranker all succeeded. The chat call failed after 725 ms, and under it are three spans: an attempt that came back 503 after 88 ms, a 482 ms backoff, and a second attempt that came back 400 after 3 ms.

The parent gen_ai.chat span carries the second attempt’s status, 400, and classifies it as invalid_request. Searching on the parent alone would send you looking for a malformed request. The 503 is on the first attempt.

The retrieval did not fail. It ends goodmem.outcome = partial with goodmem.postprocess.fallback = true, and returned its eight results without a summary.

Tracking token usage and cost

Aggregated across requests, the same spans give usage by model, the stages that are slow repeatedly rather than once, and token consumption over time. If you route traces to an LLM-native backend such as Langfuse [4], GoodMem’s telemetry maps into cost tracking [5]:

Embeddings and LLMs: When token counts and model identifiers match Langfuse’s pricing catalog, costs are calculated automatically. For self-hosted models (via vLLM, TEI, or Ollama) or custom endpoints, you can define custom pricing rates in Langfuse.
Rerankers: GoodMem exports reranker invocations as standard operational spans. Because Langfuse does not automatically price reranker spans, any reported token or billing units on those spans must be priced separately in your analytics pipeline.

Langfuse totalled our model costs at $0.66:

Model	Tokens	Cost (USD)
`gpt-5.5`	96.22 K	0.59
`gpt-4.1`	21.81 K	0.06
`voyage-3-large`	6.11 M	0.00

The embedder did 52 times the token volume of the two chat models combined and shows zero, because Langfuse has no catalog price for voyage-3-large and we had not entered one. Once sampling is on, the total covers the traces that were kept, and within those, only the calls whose usage a provider reported.

The trace counts behind those costs are mostly ingestion rather than queries:

Trace	Count
`BackgroundJob/document-processor`	20,240
`MemoryService/CreateMemory`	20,240
`MemoryService/ListMemories`	3,764
`MemoryService/DeleteMemory`	158
`SpaceService/ListSpaces`	35
`GET /readyz`	29

Keeping content out of traces

GoodMem runs on-premises or in a private VPC, which keeps documents, user histories, and model execution inside your own infrastructure. Exporting traces to a SaaS observability platform sends data out of it, and for most enterprise deployments that is the question the security review turns on.

GoodMem’s instrumentation excludes query text, memory content, prompts, generated text, and credentials, and there is no option to turn content capture on. In Langfuse, the Input and Output panels on a GoodMem span hold structural metadata rather than payloads: on the llm span above, Input was a role and a turn number, and Output was a finish reason, an empty tool-call list, and answer_chars: 1099. The answer itself, all 1,099 characters of it, is not there. The exclusion is by field rather than by scanning text for things that look sensitive, so what it covers is fixed and known in advance.

A span carries its own start and end time, and these attributes:

Recorded	Attributes
Models, providers, and operations	`gen_ai.request.model` `gen_ai.provider.name` `gen_ai.operation.name`
Reported token usage and result counts	`gen_ai.usage.input_tokens` `goodmem.result.count`
Outcome: success, failure, cancellation, retry, or fallback	`goodmem.outcome`
Error category and transport status	`goodmem.failure.category` `goodmem.rpc.status_code`
Correlation identifiers	`goodmem.request.id` `goodmem.space.id` `goodmem.principal.id` `goodmem.api_key.id`

Error categories arrive without the exception message or the stack trace. The correlation identifiers let you join a trace to an internal audit log without any payload crossing over, and the API key’s ID is recorded while its secret never is.

Here is the complete attribute map of one of the embedding calls in that trace.

gen_ai.embeddingsvoyage-3-large · 265 ms · trace b72b5ab637 attributes

gen_ai.operation.name: embeddings
gen_ai.provider.name: voyage
gen_ai.request.model: voyage-3-large
gen_ai.usage.input_tokens: 4
gen_ai.usage.total_tokens: 4
goodmem.audit_actor.id: 019cfcff-...
goodmem.embedder.id: 019cfd1c-...
goodmem.embedding.adaptive_attempt_count: 1
goodmem.embedding.batch_count: 1
goodmem.embedding.configured_dimension: 1024
goodmem.embedding.context: QUERY
goodmem.embedding.input_char_count: 40
goodmem.embedding.output_dimension: 1024
goodmem.embedding.provider_attempt_count: 1
goodmem.embedding.retry_backoff_count: 0
goodmem.embedding.retry_count: 0
goodmem.embedding.retry_elapsed_delay_ms: 0
goodmem.embedding.retry_requested_delay_ms: 0
goodmem.embedding.split_count: 0
goodmem.embedding.title_count: 0
goodmem.embedding.vector_type: dense
goodmem.inference.purpose: RETRIEVAL
goodmem.input_count: 1
goodmem.observation.kind: embedding
goodmem.outcome: success
goodmem.principal.id: 019cfcff-...
goodmem.principal.type: HUMAN
goodmem.request.operation: goodmem.v1.MemoryService/RetrieveMemory
goodmem.request.operation.category: retrieve
goodmem.result_count: 1
goodmem.retrieval.accessible_space_count: 1
goodmem.retrieval.space_ids: [01a13ed0-...]
goodmem.retrieval.space_ids_truncated: false
goodmem.space.id: 01a13ed0-...
goodmem.trace.name: MemoryService/RetrieveMemory
langfuse.observation.type: embedding
langfuse.trace.name: MemoryService/RetrieveMemory

Every attribute on the span, as exported. UUIDs are shortened here; nothing else is changed.

The query was 40 characters long and cost 4 tokens. Both numbers are on the span. The characters themselves are not.

Setting up trace export

Tracing is off by default and requires no changes to your client application or SDK code. You configure it at the GoodMem server level through environment variables or JVM system properties [6]. If someone else operates your deployment, the steps below are theirs to run.

1. Select your destination and profile

GoodMem exports over OTLP, grpc or http/protobuf. You can point the server at a backend directly or route through an OpenTelemetry Collector to fan out to several destinations. Which backends the other AI memory systems support is in the OpenTelemetry comparison.

GOODMEM_OTEL_EXPORT_PROFILE formats trace metadata for the target backend:

neutral (the default): Jaeger, Grafana Tempo, or an OTel Collector.
langfuse: metadata shaped for direct ingestion into Langfuse.
langsmith: metadata shaped for direct ingestion into LangSmith [7].

The profile only adjusts attribute mapping. You still supply the endpoint, protocol, and auth headers, and each backend documents its own. Langfuse builds its header as Basic auth from a project’s public and secret keys [4].

2. Configure the server environment

For installer-managed deployments, run goodmem system info to locate the active installation profile, then add the OTel configuration to the private .env file under ~/.goodmem/installs/local-docker/<profile-name>/. For containerized or custom deployments, inject the variables into the server environment.

A baseline configuration for http/protobuf with 100% sampling, for the initial check:

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic BASE64_CREDENTIALS
GOODMEM_OTEL_EXPORT_PROFILE=neutral
OTEL_TRACES_SAMPLER=always_on

The endpoint must include the full path your receiver expects, which for http/protobuf is usually /v1/traces. Keep the header value out of version control. The service name defaults to goodmem-server, and traces arrive under that name at the other end.

3. Restart the GoodMem server

GoodMem reads telemetry configuration at startup. Restart the service to initialize the OTel SDK:

goodmem system stop
goodmem system start

In production, drain first and stop the container only after the drain reports Quiesced: true:

goodmem system drain --wait

GoodMem fails open. If the OTel SDK does not initialize, the server disables export, emits a sanitized diagnostic, and goes on serving traffic.

4. Verify the export path

Trigger a lightweight authenticated request to check connectivity. With GOODMEM_URL and GOODMEM_API_KEY set, and a key that can list spaces, list a single space [8]:

curl -i \
  -H "x-api-key: $GOODMEM_API_KEY" \
  "$GOODMEM_URL/v1/spaces?maxResults=1"

Read goodmem-request-id from the response headers. Spans leave in batches, so give the export a few seconds. Then query your tracing backend for goodmem.request.id with that value, or request_id in Langfuse metadata, to jump straight to the corresponding trace.

5. Trace a full retrieval pipeline

Once connectivity is confirmed, run a retrieval against a populated memory space. With reranker_id in the post-processor config, the call goes through the embedder, vector search, the reranker, and the LLM, so every stage appears in the trace [9]:

curl -N -X POST "$GOODMEM_URL/v1/memories:retrieve" \
  -H "x-api-key: $GOODMEM_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-ndjson" \
  -d '{
    "message": "What do you know about AI?",
    "spaceKeys": [{"spaceId": "'"$SPACE_ID"'"}],
    "postProcessor": {
      "name": "com.goodmem.retrieval.postprocess.ChatPostProcessorFactory",
      "config": {
        "llm_id": "'"$LLM_ID"'",
        "reranker_id": "'"$RERANKER_ID"'"
      }
    }
  }'

Open the resulting trace in your observability tool. The parent request breaks down into child spans for query embedding, vector search, reranking, and LLM generation, with the latency of each and the token counts the providers reported.

Once that is verified, change OTEL_TRACES_SAMPLER to sample a fraction of production traffic, for example parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1.

Tracing in practice: an internal knowledge agent

The trace above came from an agent we run on our own material. It ingests company communications, architecture docs, whitepapers, and GitHub repositories into scoped memory spaces, which are listed on the profile line of each answer below.

A chat thread explains why a decision was made, a design doc records the requirements, and a repo shows the implementation, so the agent queries across several spaces to answer a question. Names in the two answers below are changed; the questions, the structure, and the source list are the agent’s.

Company Memoryprofile pair · gpt-5.5 · spaces: chat, chat_messages, docs, notes, people, projects, wiki

QWho should I speak to about integrations? Identify who implemented it, who contributed to its design, and who has worked on it recently.

Speak to Priya Raman first.

Implemented: Priya records finishing LangChain, DeepEval, AutoAgent and agent-framework. Marco Oliveira attributes most integrations to Priya, and sent the DeepEval PR.
Design input: Lena Haddad authored the integration overview and later gave product and design feedback. Sam Okonkwo left design tradeoff comments early on.
Recent work: Priya was active across the integration repos and E2E updates through late Sep 2026. Lena was working on goodmem-langchain4j in Sep.

Sources

Zulip message by Priya Raman on 2026-03-18 (#goodmem > Integrations)
Zulip message by Marco Oliveira on 2026-04-03 (#goodmem > Integrations)
Wiki: Popular Agent Frameworks
Zulip message by Lena Haddad on 2026-03-10 (#goodmem > Integrations)
GitHub activity of praman

That answer came from five sources in three systems: Zulip, the wiki, and GitHub.

Ensuring fresh context through ingestion tracing

When an engineer asks who currently maintains a component, stale memory returns the wrong answer, pointing to the original author from two years ago rather than the current maintainer. If recent GitHub commits or shared-drive updates are not showing up in retrieval results, filtering traces on the background job spans shows whether a document batch failed during chunking, hit a rate limit on the embedding provider, or succeeded as expected.

Ingestion is two traces rather than one. The write’s trace ends at the enqueue span, the work arrives later under its own trace id, and the job id is what joins them. In the 15 October re-ingest, the write returned 201 in 3 ms and recorded job 218922 on its enqueue span. Four seconds later the job’s own trace opened on goodmem.job.attempt: 321 ms, attempt 1, succeeded, with the embedding call under it taking 279 ms for 102 tokens and recording its context as INDEXING rather than QUERY.

Diagnosing multi-space retrieval latency

When an engineer asks a broad historical question, such as reconstructing a project’s timeline across discussions, docs, and code, the retrieval pipeline has to search several memory spaces, merge the candidates, rerank them, and pass the top chunks to an LLM. If that query takes 5 seconds, the trace waterfall shows where the time went: whether one memory space was slow to search, whether the reranker choked on a large candidate set, or whether the LLM was slow to generate the summary.

One question about a document and who wrote it took 14.4 seconds and six searches, of which three went to chat_messages and two to docs. The retrievals were not slow. The agent ran the same search three times.

Auditing access and scoped failures

In an enterprise knowledge base, not every user or agent is authorized to read every memory space, and GoodMem enforces role-based grants and scoped API keys before the search runs. If an agent fails to retrieve expected context from a restricted repository space, the trace separates an upstream model failure, such as an HTTP 502 from an embedder, from an expected authorization rejection, an HTTP 403. goodmem.failure.category and goodmem.failure.expected carry that distinction.

Checking the exclusion against a real answer

Company Memoryprofile pair · gpt-5.5 · spaces: chat, chat_messages, docs, notes, people, projects, wiki

QI keep getting disconnected from video calls. Has anyone else experienced this issue, and how did they resolve it?

Yes. Similar conference-room issues were reported, mostly with the office room system and the California work-day bridge:

Priya Raman was disconnected and resolved it by rejoining.
Tomas Nilsen reported the room was restarted and people could rejoin.
Dana Whitfield later said the office room was fixed after finding the room PC had a faulty CPU; the CPU was replaced.
Another issue involved the room controller becoming unpaired after a PC restart, but I found no recorded resolution for that thread.

Sources

Zulip message by Priya Raman on 2026-04-14 (#general > general chat)
Zulip message by Tomas Nilsen on 2026-04-15 (#mv-misc > general chat)
Zulip #general > general chat
Zulip message by Dana Whitfield on 2026-06-18 (#mv-misc > general chat)

The answer is assembled from four chat messages posted over two months in two different channels. None of that text appears in the trace of the request that produced it.

References

OpenTelemetry. https://opentelemetry.io/
OpenTelemetry. GoodMem documentation. https://docs.goodmem.ai/docs/reference/opentelemetry/
Spans and attributes. GoodMem documentation. https://docs.goodmem.ai/docs/reference/opentelemetry/spans-and-attributes/
Langfuse integration. GoodMem documentation. https://docs.goodmem.ai/docs/integrations/observability-eval/langfuse/
Token and cost tracking. Langfuse documentation. https://langfuse.com/docs/observability/features/token-and-cost-tracking
OpenTelemetry configuration. GoodMem documentation. https://docs.goodmem.ai/docs/reference/opentelemetry/configuration/
LangSmith integration. GoodMem documentation. https://docs.goodmem.ai/docs/integrations/observability-eval/langsmith/
Export traces. GoodMem documentation. https://docs.goodmem.ai/docs/how-to/export-traces/
Build a basic RAG agent. GoodMem documentation. https://docs.goodmem.ai/docs/how-to/basic-rag/