Skip to main content

How to Profile Databend

go pprof tool​

go tool pprof http://localhost:8080/debug/pprof/profile?seconds=20

Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile?seconds=20
Saved profile in /home/bohu/pprof/pprof.cpu.007.pb.gz
Type: cpu
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 5011, 100% of 5011 total
Dropped 248 nodes (cum <= 25)
Showing top 10 nodes out of 204
      flat  flat%   sum%        cum   cum%
      5011   100%   100%       5011   100%  backtrace::backtrace::libunwind::trace
         0     0%   100%        162  3.23%  <&alloc::vec::Vec<T,A> as core::iter::traits::collect::IntoIterator>::into_iter
         0     0%   100%         45   0.9%  <&mut I as core::iter::traits::iterator::Iterator>::next
         0     0%   100%         77  1.54%  <[A] as core::slice::cmp::SlicePartialEq<B>>::equal
         0     0%   100%         35   0.7%  <[u8; 8] as ahash::convert::Convert<u64>>::convert
         0     0%   100%        199  3.97%  <[u8] as ahash::convert::ReadFromSlice>::read_last_u64
         0     0%   100%         73  1.46%  <[u8] as ahash::convert::ReadFromSlice>::read_last_u64::as_array
         0     0%   100%        220  4.39%  <[u8] as ahash::convert::ReadFromSlice>::read_u64
         0     0%   100%        701 13.99%  <ahash::fallback_hash::AHasher as core::hash::Hasher>::write
         0     0%   100%         26  0.52%  <ahash::random_state::RandomState as core::hash::BuildHasher>::build_hash

Or

go tool pprof -http=0.0.0.0:8081 $HOME/pprof/pprof.cpu.007.pb.gz

memory profiling​

  • Build databend-query with memory-profiling feature enabled

    under the project root path:

    ~/workspace/fuse-query$ cargo build --features memory-profiling

  • Fire up databend, with proper MALLOC_CONF setting

    for example MALLOC_CONF=prof:true ./target/debug/databend-query

  • Dump memory prof

    NOTE: currently, periodical heap prof dump is NOT supported. A "snaphost" of the heap prof is dump instead.

    • by using jeprof
        ~/workspace/fuse-query$ jeprof ./target/debug/databend-query http://localhost:8080/debug/mem
        Using local file ./target/debug/databend-query.
        Gathering CPU profile from http://localhost:8080/debug/mem/pprof/profile?seconds=30 for 30 seconds to ~/jeprof/databend-query.1650949265.localhost
        Be patient...
        Wrote profile to /home/zhaobr/jeprof/databend-query.1650949265.localhost
        Welcome to jeprof!  For help, type 'help'.
        (jeprof) top
        Total: 16.2 MB
            10.2  62.7%  62.7%     10.2  62.7% ::alloc
             6.0  37.3% 100.0%      6.0  37.3% ::alloc_zeroed
             0.0   0.0% 100.0%     10.2  62.7% ::allocate
             0.0   0.0% 100.0%      0.5   3.3% ::call
             0.0   0.0% 100.0%      4.0  24.7% ::default
             0.0   0.0% 100.0%      1.2   7.2% ::deref
             0.0   0.0% 100.0%      1.2   7.2% ::deref::__stability (inline)
             0.0   0.0% 100.0%      1.2   7.2% ::deref::__static_ref_initialize (inline)
             0.0   0.0% 100.0%      0.5   3.1% ::from
             0.0   0.0% 100.0%      9.2  56.6% ::from_iter
        (jeprof)
    • or curl 
       ~/workspace/fuse-query$ curl -O  -J  http://localhost:8080/debug/mem/pprof/profile
           % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                          Dload  Upload   Total   Spent    Left  Speed
         100 22733  100 22733    0     0  21.6M      0 --:--:-- --:--:-- --:--:-- 21.6M
         curl: Saved to filename 'heap_dump_NIKcRe.prof'