Profiling

If you are running into problems with Zendrite using excessive resources (e.g. CPU or RAM) then you can use the profiler to work out what is happening.

Zendrite contains an embedded profiler called pprof, which is a part of the standard Go toolchain.

Continuous profiling with Grafana Pyroscope

Zendrite can continuously push CPU and memory profiles to a Pyroscope server or Grafana Cloud Profiles. Profiling is disabled by default. To enable it, configure global.pyroscope in zendrite.yaml and restart Zendrite:

global:
  pyroscope:
    enabled: true
    application_name: zendrite
    server_address: http://localhost:4040
    basic_auth:
      username: ''
      password: ''
    tenant_id: ''
    tags:
      instance: my-homeserver

Set server_address to your Pyroscope ingestion URL, reachable from the Zendrite process or container. For Grafana Cloud Profiles, use the URL, username and access policy token from your stack’s Profiles configuration, placing the token in basic_auth.password. Use tenant_id only if your Pyroscope server requires multi-tenancy. The basic_auth, tenant_id and tags settings are optional, and application_name defaults to zendrite. See the Grafana Go SDK documentation for backend configuration details.

Zendrite collects CPU samples, allocated objects and bytes, and in-use objects and bytes. Profiles are uploaded in the background and remaining data is flushed during graceful shutdown. Upload failures are logged without stopping the homeserver. In Grafana, select the configured application name to explore its profiles and filter by your tags.

Pyroscope does not require PPROFLISTEN. While continuous profiling is enabled, avoid requesting a CPU profile from the pprof endpoint, since both use Go’s single CPU profiler. Disable Pyroscope and restart Zendrite before collecting a manual CPU profile.

Enable the pprof profiler

To enable the profiler, start Zendrite with the PPROFLISTEN environment variable. This variable specifies which address and port to listen on, e.g.

PPROFLISTEN=localhost:65432 ./bin/zendrite ...

If pprof has been enabled successfully, a log line at startup will show that pprof is listening:

WARN[2020-12-03T13:32:33.669405000Z] [/Users/neilalexander/Desktop/zendrite/internal/log.go:87] SetupPprof
  Starting pprof on localhost:65432

All examples from this point forward assume PPROFLISTEN=localhost:65432 but you may need to adjust as necessary for your setup.

Profiling CPU usage

To examine where CPU time is going, you can call the profile endpoint:

http://localhost:65432/debug/pprof/profile?seconds=30

The profile will run for the specified number of seconds and then will produce a result.

Examine a profile using the Go toolchain

If you have Go installed and want to explore the profile, you can invoke go tool pprof to start the profile directly. The -http= parameter will instruct go tool pprof to start a web server providing a view of the captured profile:

go tool pprof -http=localhost:23456 http://localhost:65432/debug/pprof/profile?seconds=30

You can then visit http://localhost:23456 in your web browser to see a visual representation of the profile. Particularly usefully, in the “View” menu, you can select “Flame Graph” to see a proportional interactive graph of CPU usage.

Download a profile to send to someone else

If you don’t have the Go tools installed but just want to capture the profile to send to someone else, you can instead use curl to download the profiler results:

curl -O http://localhost:65432/debug/pprof/profile?seconds=30

This will block for the specified number of seconds, capturing information about what Zendrite is doing, and then produces a profile file, which you can send onward.

Profiling memory usage

To examine where memory usage is going, you can call the heap endpoint:

http://localhost:65432/debug/pprof/heap

The profile will return almost instantly.

Examine a profile using the Go toolchain

If you have Go installed and want to explore the profile, you can invoke go tool pprof to start the profile directly. The -http= parameter will instruct go tool pprof to start a web server providing a view of the captured profile:

go tool pprof -http=localhost:23456 http://localhost:65432/debug/pprof/heap

You can then visit http://localhost:23456 in your web browser to see a visual representation of the profile. The “Sample” menu lets you select between four different memory profiles:

• inuse_space: Shows how much actual heap memory is allocated per function (this is generally the most useful profile when diagnosing high memory usage)
• inuse_objects: Shows how many heap objects are allocated per function
• alloc_space: Shows how much memory has been allocated per function (although that memory may have since been deallocated)
• alloc_objects: Shows how many allocations have been made per function (although that memory may have since been deallocated)

Also in the “View” menu, you can select “Flame Graph” to see a proportional interactive graph of the memory usage.

Download a profile to send to someone else

If you don’t have the Go tools installed but just want to capture the profile to send to someone else, you can instead use curl to download the profiler results:

curl -O http://localhost:65432/debug/pprof/heap

This will almost instantly produce a heap file, which you can send onward.