googlecloudplatform / cloud-profiler-python Goto Github PK

View Code? Open in Web Editor NEW

27.0 33.0 25.0 115 KB

Stackdriver Profiler Python agent is a tool that continuously gathers CPU usage information from Python applications

License: Apache License 2.0

Python 47.70% C++ 35.93% C 1.89% Go 11.91% Shell 2.57%

cloud-profiler-python's Introduction

Google Cloud Python profiling agent

Python profiling agent for Google Cloud Profiler.

See Google Cloud Profiler profiling Python code for detailed documentation.

Supported OS

Linux. Profiling Python applications is supported for Linux kernels whose standard C library is implemented with glibc or with musl. For configuration information specific to Linux Alpine kernels, see Running on Linux Alpine.

Supported Python Versions

Python >= 3.7 and <= 3.11

Installation & usage

Install the profiler package using PyPI:
```
pip3 install google-cloud-profiler
```

Enable the profiler in your application:

import googlecloudprofiler

def main():
    # Profiler initialization. It starts a daemon thread which continuously
    # collects and uploads profiles. Best done as early as possible.
    try:
        googlecloudprofiler.start(
            service='hello-profiler',
            service_version='1.0.1',
            # verbose is the logging level. 0-error, 1-warning, 2-info,
            # 3-debug. It defaults to 0 (error) if not set.
            verbose=3,
            # project_id must be set if not running on GCP.
            # project_id='my-project-id',
        )
    except (ValueError, NotImplementedError) as exc:
        print(exc)  # Handle errors here

Installation on Linux Alpine

The Python profiling agent has a native component. The base Alpine image for Python does not have all dependencies required to build this native component installed. To build the Python profiling agent on Alpine, one must install the package build-base.

To use the Python profiling agent on Alpine without installing additional dependencies on to the final Alpine image, one can use a two-stage build and compile the Python profiling agent in the first stage.

Here is an example of a Docker image that uses a multi-stage build to compile and install the Python profiling agent:

FROM python:3.7-alpine as builder

# Install build-base to allow for compilation of the profiling agent.
RUN apk add --update --no-cache build-base

# Compile the profiling agent, generating wheels for it.
RUN pip3 wheel --wheel-dir=/tmp/wheels google-cloud-profiler


FROM python:3.7-alpine

# Copy over the directory containing wheels for the profiling agent.
COPY --from=builder /tmp/wheels /tmp/wheels

# Install the profiling agent.
RUN pip3 install --no-index --find-links=/tmp/wheels google-cloud-profiler

# Install any other required modules or dependencies, and copy an app which
# enables the profiler as described in "Enable the profiler in your
# application".
COPY ./bench.py .

# Run the application when the docker image is run, using either CMD (as is done
# here) or ENTRYPOINT.
CMD python3 -u bench.py

Troubleshooting

Resource temporarily unavailable errors with Python

If you see the following log entries after enabling the Profiler:

BlockingIOError: [Errno 11] Resource temporarily unavailable
Exception ignored when trying to write to the signal wakeup fd

see https://cloud.google.com/profiler/docs/troubleshooting#python-blocking for the cause and the workaround.

cloud-profiler-python's People

Contributors

Stargazers

Watchers

cloud-profiler-python's Issues

Error starting the profiler

When I try to start the profiler I see the following log message:

Failed to build the Discovery client for profiler (will retry after 634.737s): 'ClientOptions' object has no attribute 'credentials_file'

Environment details
GAE standard python37 runtime:
Python version: 3.7
google-cloud-profiler==2.0.4
gunicorn==20.0.4

Code example

 googlecloudprofiler.start(
            service=self._config.service_name,
            verbose=self._config.verbose,
            service_version=self._config.service_version,
            project_id=self._config.project_id,
            service_account_json_file=self._config.service_account_json_file)

I tried to fix it by forcing the version to google-cloud-core==1.3.0 but I was unlucky
I tried to call the start method without parameters as in the documentation, but it gives similar problems

Not compatible with Python 3.8

$ python -V
Python 3.8.0

$ pip -V
pip 19.3.1 from /env/lib/python3.8/site-packages/pip (python 3.8)

$ # pip install google-cloud-profiler
Collecting google-cloud-profiler
  Using cached https://files.pythonhosted.org/packages/40/61/e8c0f652ab61a1ec4832a03d11c84b54f536977b3e8e19762b5863762c62/google-cloud-profiler-1.0.9.tar.gz
Requirement already satisfied: google-api-python-client in /env/lib/python3.8/site-packages (from google-cloud-profiler) (1.7.11)
Requirement already satisfied: google-auth>=1.0.0 in /env/lib/python3.8/site-packages (from google-cloud-profiler) (1.7.1)
Requirement already satisfied: google-auth-httplib2 in /env/lib/python3.8/site-packages (from google-cloud-profiler) (0.0.3)
Requirement already satisfied: protobuf in /env/lib/python3.8/site-packages (from google-cloud-profiler) (3.10.0)
Requirement already satisfied: requests in /env/lib/python3.8/site-packages (from google-cloud-profiler) (2.22.0)
Requirement already satisfied: httplib2<1dev,>=0.9.2 in /env/lib/python3.8/site-packages (from google-api-python-client->google-cloud-profiler) (0.14.0)
Requirement already satisfied: six<2dev,>=1.6.1 in /env/lib/python3.8/site-packages (from google-api-python-client->google-cloud-profiler) (1.13.0)
Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /env/lib/python3.8/site-packages (from google-api-python-client->google-cloud-profiler) (3.0.0)
Requirement already satisfied: setuptools>=40.3.0 in /env/lib/python3.8/site-packages (from google-auth>=1.0.0->google-cloud-profiler) (41.2.0)
Requirement already satisfied: cachetools<3.2,>=2.0.0 in /env/lib/python3.8/site-packages (from google-auth>=1.0.0->google-cloud-profiler) (3.1.1)
Requirement already satisfied: rsa<4.1,>=3.1.4 in /env/lib/python3.8/site-packages (from google-auth>=1.0.0->google-cloud-profiler) (4.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /env/lib/python3.8/site-packages (from google-auth>=1.0.0->google-cloud-profiler) (0.2.7)
Requirement already satisfied: certifi>=2017.4.17 in /env/lib/python3.8/site-packages (from requests->google-cloud-profiler) (2019.9.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /env/lib/python3.8/site-packages (from requests->google-cloud-profiler) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /env/lib/python3.8/site-packages (from requests->google-cloud-profiler) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /env/lib/python3.8/site-packages (from requests->google-cloud-profiler) (1.25.7)
Requirement already satisfied: pyasn1>=0.1.3 in /env/lib/python3.8/site-packages (from rsa<4.1,>=3.1.4->google-auth>=1.0.0->google-cloud-profiler) (0.4.7)
Installing collected packages: google-cloud-profiler
    Running setup.py install for google-cloud-profiler ... error
    ERROR: Command errored out with exit status 1:
     command: /env/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zea9hnjz/google-cloud-profiler/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zea9hnjz/google-cloud-profiler/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bzrk0j0y/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.8/google-cloud-profiler
         cwd: /tmp/pip-install-zea9hnjz/google-cloud-profiler/
    Complete output (24 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/backoff.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/builder.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/__init__.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/__version__.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    copying googlecloudprofiler/client.py -> build/lib.linux-x86_64-3.8/googlecloudprofiler
    running build_ext
    building 'googlecloudprofiler._profiler' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/googlecloudprofiler
    creating build/temp.linux-x86_64-3.8/googlecloudprofiler/src
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Igooglecloudprofiler/src -I/env/include -I/usr/include/python3.8 -c googlecloudprofiler/src/clock.cc -o build/temp.linux-x86_64-3.8/googlecloudprofiler/src/clock.o -std=c++11
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Igooglecloudprofiler/src -I/env/include -I/usr/include/python3.8 -c googlecloudprofiler/src/_profiler.cc -o build/temp.linux-x86_64-3.8/googlecloudprofiler/src/_profiler.o -std=c++11
    googlecloudprofiler/src/_profiler.cc:15:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /env/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zea9hnjz/google-cloud-profiler/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zea9hnjz/google-cloud-profiler/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bzrk0j0y/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.8/google-cloud-profiler Check the logs for full command output.

googlecloudprofiler breaks other packages which depend on C++ code compiled against a different version of libstdc++

These lines here,

cloud-profiler-python/setup.py

Lines 46 to 53 in ea964fc

 '-static-libstdc++', 

 # While libgcc_s.so.1 is pretty much always installed by default 

 # for non-Alpine linux, it is not installed by default in Alpine. 

 # So, to support Alpine, we will always statically link "libgcc" 

 # package. We could alternatively require users to install the 

 # "libgcc" package, but the static linkage seems less 

 # invasive. 

 '-static-libgcc'

statically link this package's native code against the particular version of libstdc++ available at link time. This breaks other packages which have native code and depend on a different version of libstdc++.

It seems reasonable to me to ask your users to have a recent version of libstdc++ and libgcc available. Or perhaps you could publish two packages to PyPI, this one and one that uses a dynamic link?

Thanks!

PS: I discovered this because version 4.0.0 is incompatible with libsass.

Allow disabling of gce metadata lookup.

If this package is used outside of a GCE/GKE/CloudFunction/etc. environment it produces errors like
[WARNING] googlecloudprofiler.client client.py:retrieve_gce_metadata:88 | Failed to fetch instance/zone from GCE metadata server: HTTPConnectionPool(host='metadata', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/zone (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff9e40b9128>: Failed to establish a new connection: [Errno -2] Name does not resolve',))

Having an ability to disable this metadata lookup would avoid these errors.

Error installing in CI environment (Linux, Python 3.7), `fatal error: Python.h: No such file or directory`

Environment

a GitLab CI with a python:3.8/Linux docker image
I use tox as a testrunner. This trace below occurs when I install requirements in the py37 environment (but succeeds for py38)

Project requirements

cdsapi
ecmwf-api-client
apache-beam[gcp]
numpy>=1.19.1
pandas
xarray
requests>=2.24.0
firebase-admin>=5.0.0
google-cloud-datastore>=1.15.0,<2  # For compatability with apache-beam[gcp]
google-cloud-firestore
urllib3==1.25.11

Profiler version

Collecting google-cloud-profiler<4,>=3.0.4
  Downloading google-cloud-profiler-3.0.4.tar.gz (32 kB)

Stack trace

Building wheels for collected packages: google-cloud-profiler
  Building wheel for google-cloud-profiler (setup.py): started
  Building wheel for google-cloud-profiler (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /builds/XXXXXX/.tox/py37/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/setup.py'"'"'; __file__='"'"'/tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-yhglrqsm
       cwd: /tmp/pip-install-n8xc8_iv/google-cloud-profiler_67e67791160d41a2bb5cb70e302bb400/
  Complete output (26 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/client.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/__version__.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/__init__.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/builder.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/backoff.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-x86_64-3.7/googlecloudprofiler
  running build_ext
  building 'googlecloudprofiler._profiler' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/googlecloudprofiler
  creating build/temp.linux-x86_64-3.7/googlecloudprofiler/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Igooglecloudprofiler/src -I/builds/XXXXXXX/.tox/py37/include -I/usr/include/python3.7m -c googlecloudprofiler/src/stacktraces.cc -o build/temp.linux-x86_64-3.7/googlecloudprofiler/src/stacktraces.o -std=c++11
  In file included from googlecloudprofiler/src/stacktraces.cc:15:
  googlecloudprofiler/src/stacktraces.h:18:10: fatal error: Python.h: No such file or directory
   #include <Python.h>
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for google-cloud-profiler
  Running setup.py clean for google-cloud-profiler

Integration with Stackdriver Trace

What are the plans to tie together tracing (opencensus/opentelementry) and Stackdriver Profiler?

Cannot compile with Python 3.11

pip install fails using this Dockerfile (from https://cloud.google.com/profiler/docs/profiling-python#using-profiler ):

FROM python:3
RUN apt-get update && apt-get install -y build-essential python3-pip
RUN pip3 install google-cloud-profiler

Here is the build output:

% docker build --no-cache -t gcloud-profiler:local .
[+] Building 16.7s (6/6) FINISHED                                                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                                                       0.0s
 => => transferring dockerfile: 163B                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/python:3                                                                                                                                0.2s
 => CACHED [1/3] FROM docker.io/library/python:3@sha256:10fc14aa6ae69f69e4c953cffd9b0964843d8c163950491d2138af891377bc1d                                                                   0.0s
 => [2/3] RUN apt-get update && apt-get install -y build-essential python3-pip                                                                                                             6.1s
 => ERROR [3/3] RUN pip3 install google-cloud-profiler                                                                                                                                    10.3s 
------                                                                                                                                                                                          
 > [3/3] RUN pip3 install google-cloud-profiler:                                                                                                                                                
#6 1.591 Collecting google-cloud-profiler                                                                                                                                                       
#6 1.722   Downloading google-cloud-profiler-4.0.0.tar.gz (30 kB)                                                                                                                               
#6 1.737   Preparing metadata (setup.py): started                                                                                                                                               
#6 2.193   Preparing metadata (setup.py): finished with status 'done'                                                                                                                           
#6 2.295 Collecting google-api-python-client!=1.12.0,!=2.0.2
#6 2.315   Downloading google_api_python_client-2.66.0-py2.py3-none-any.whl (10.5 MB)
#6 3.372      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/10.5 MB 10.1 MB/s eta 0:00:00
#6 3.509 Collecting google-auth>=1.0.0
#6 3.528   Downloading google_auth-2.14.1-py2.py3-none-any.whl (175 kB)
#6 3.562      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.4/175.4 kB 5.3 MB/s eta 0:00:00
#6 3.598 Collecting google-auth-httplib2
#6 3.624   Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
#6 3.852 Collecting protobuf
#6 3.890   Downloading protobuf-4.21.9-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
#6 3.969      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 408.4/408.4 kB 5.3 MB/s eta 0:00:00
#6 4.030 Collecting requests
#6 4.049   Downloading requests-2.28.1-py3-none-any.whl (62 kB)
#6 4.060      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 11.9 MB/s eta 0:00:00
#6 4.125 Collecting httplib2<1dev,>=0.15.0
#6 4.147   Downloading httplib2-0.21.0-py3-none-any.whl (96 kB)
#6 4.158      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.8/96.8 kB 11.0 MB/s eta 0:00:00
#6 4.259 Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
#6 4.275   Downloading google_api_core-2.10.2-py3-none-any.whl (115 kB)
#6 4.288      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.6/115.6 kB 10.3 MB/s eta 0:00:00
#6 4.324 Collecting uritemplate<5,>=3.0.1
#6 4.344   Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
#6 4.443 Collecting cachetools<6.0,>=2.0.0
#6 4.473   Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
#6 4.527 Collecting pyasn1-modules>=0.2.1
#6 4.549   Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
#6 4.563      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 12.5 MB/s eta 0:00:00
#6 4.595 Collecting six>=1.9.0
#6 4.616   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
#6 4.657 Collecting rsa<5,>=3.1.4
#6 4.682   Downloading rsa-4.9-py3-none-any.whl (34 kB)
#6 4.780 Collecting charset-normalizer<3,>=2
#6 4.801   Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
#6 4.843 Collecting idna<4,>=2.5
#6 4.860   Downloading idna-3.4-py3-none-any.whl (61 kB)
#6 4.879      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 6.0 MB/s eta 0:00:00
#6 4.943 Collecting urllib3<1.27,>=1.21.1
#6 4.970   Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
#6 4.984      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 11.1 MB/s eta 0:00:00
#6 5.028 Collecting certifi>=2017.4.17
#6 5.055   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
#6 5.071      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 11.4 MB/s eta 0:00:00
#6 5.155 Collecting googleapis-common-protos<2.0dev,>=1.56.2
#6 5.177   Downloading googleapis_common_protos-1.57.0-py2.py3-none-any.whl (217 kB)
#6 5.204      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 218.0/218.0 kB 8.4 MB/s eta 0:00:00
#6 5.306 Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
#6 5.323   Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
#6 5.336      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 8.1 MB/s eta 0:00:00
#6 5.395 Collecting pyasn1<0.5.0,>=0.4.6
#6 5.429   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
#6 5.453      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 3.1 MB/s eta 0:00:00
#6 5.521 Building wheels for collected packages: google-cloud-profiler
#6 5.523   Building wheel for google-cloud-profiler (setup.py): started
#6 6.826   Building wheel for google-cloud-profiler (setup.py): finished with status 'error'
#6 6.833   error: subprocess-exited-with-error
#6 6.833   
#6 6.833   × python setup.py bdist_wheel did not run successfully.
#6 6.833   │ exit code: 1
#6 6.833   ╰─> [74 lines of output]
#6 6.833       running bdist_wheel
#6 6.833       running build
#6 6.833       running build_py
#6 6.833       creating build
#6 6.833       creating build/lib.linux-x86_64-cpython-311
#6 6.833       creating build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/__version__.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/__init__.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/backoff.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/builder.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       copying googlecloudprofiler/client.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       running build_ext
#6 6.833       building 'googlecloudprofiler._profiler' extension
#6 6.833       creating build/temp.linux-x86_64-cpython-311
#6 6.833       creating build/temp.linux-x86_64-cpython-311/googlecloudprofiler
#6 6.833       creating build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src
#6 6.833       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/_profiler.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/_profiler.o -std=c++11
#6 6.833       In file included from googlecloudprofiler/src/profiler.h:26,
#6 6.833                        from googlecloudprofiler/src/_profiler.cc:18:
#6 6.833       googlecloudprofiler/src/stacktraces.h: In member function ‘void AsyncSafeTraceMultiset::Reset()’:
#6 6.833       googlecloudprofiler/src/stacktraces.h:72:52: warning: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct AsyncSafeTraceMultiset::TraceData’ with no trivial copy-assignment; use value-initialization instead [-Wclass-memaccess]
#6 6.833          72 |   void Reset() { memset(traces_, 0, sizeof(traces_)); }
#6 6.833             |                                                    ^
#6 6.833       googlecloudprofiler/src/stacktraces.h:89:10: note: ‘struct AsyncSafeTraceMultiset::TraceData’ declared here
#6 6.833          89 |   struct TraceData {
#6 6.833             |          ^~~~~~~~~
#6 6.833       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/clock.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/clock.o -std=c++11
#6 6.833       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/log.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/log.o -std=c++11
#6 6.833       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/profiler.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/profiler.o -std=c++11
#6 6.833       In file included from googlecloudprofiler/src/profiler.h:26,
#6 6.833                        from googlecloudprofiler/src/profiler.cc:15:
#6 6.833       googlecloudprofiler/src/stacktraces.h: In member function ‘void AsyncSafeTraceMultiset::Reset()’:
#6 6.833       googlecloudprofiler/src/stacktraces.h:72:52: warning: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct AsyncSafeTraceMultiset::TraceData’ with no trivial copy-assignment; use value-initialization instead [-Wclass-memaccess]
#6 6.833          72 |   void Reset() { memset(traces_, 0, sizeof(traces_)); }
#6 6.833             |                                                    ^
#6 6.833       googlecloudprofiler/src/stacktraces.h:89:10: note: ‘struct AsyncSafeTraceMultiset::TraceData’ declared here
#6 6.833          89 |   struct TraceData {
#6 6.833             |          ^~~~~~~~~
#6 6.833       googlecloudprofiler/src/profiler.cc: In static member function ‘static void Profiler::Handle(int, siginfo_t*, void*)’:
#6 6.833       googlecloudprofiler/src/profiler.cc:160:32: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
#6 6.833         160 |     PyFrameObject *frame = ts->frame;
#6 6.833             |                                ^~~~~
#6 6.833             |                                cframe
#6 6.833       googlecloudprofiler/src/profiler.cc:163:40: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833         163 |       frames[num_frames].lineno = frame->f_lineno;
#6 6.833             |                                        ^~
#6 6.833       In file included from /usr/local/include/python3.11/Python.h:42,
#6 6.833                        from googlecloudprofiler/src/profiler.h:18,
#6 6.833                        from googlecloudprofiler/src/profiler.cc:15:
#6 6.833       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833          22 | typedef struct _frame PyFrameObject;
#6 6.833             |                ^~~~~~
#6 6.833       googlecloudprofiler/src/profiler.cc:164:41: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833         164 |       frames[num_frames].py_code = frame->f_code;
#6 6.833             |                                         ^~
#6 6.833       In file included from /usr/local/include/python3.11/Python.h:42,
#6 6.833                        from googlecloudprofiler/src/profiler.h:18,
#6 6.833                        from googlecloudprofiler/src/profiler.cc:15:
#6 6.833       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833          22 | typedef struct _frame PyFrameObject;
#6 6.833             |                ^~~~~~
#6 6.833       googlecloudprofiler/src/profiler.cc:166:20: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833         166 |       frame = frame->f_back;
#6 6.833             |                    ^~
#6 6.833       In file included from /usr/local/include/python3.11/Python.h:42,
#6 6.833                        from googlecloudprofiler/src/profiler.h:18,
#6 6.833                        from googlecloudprofiler/src/profiler.cc:15:
#6 6.833       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 6.833          22 | typedef struct _frame PyFrameObject;
#6 6.833             |                ^~~~~~
#6 6.833       error: command '/usr/bin/gcc' failed with exit code 1
#6 6.833       [end of output]
#6 6.833   
#6 6.833   note: This error originates from a subprocess, and is likely not a problem with pip.
#6 6.834   ERROR: Failed building wheel for google-cloud-profiler
#6 6.838   Running setup.py clean for google-cloud-profiler
#6 7.025 Failed to build google-cloud-profiler
#6 7.115 Installing collected packages: pyasn1, urllib3, uritemplate, six, rsa, pyparsing, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, requests, httplib2, googleapis-common-protos, google-auth, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-profiler
#6 8.314   Running setup.py install for google-cloud-profiler: started
#6 9.592   Running setup.py install for google-cloud-profiler: finished with status 'error'
#6 9.598   error: subprocess-exited-with-error
#6 9.598   
#6 9.598   × Running setup.py install for google-cloud-profiler did not run successfully.
#6 9.598   │ exit code: 1
#6 9.598   ╰─> [76 lines of output]
#6 9.598       running install
#6 9.598       /usr/local/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
#6 9.598         warnings.warn(
#6 9.598       running build
#6 9.598       running build_py
#6 9.598       creating build
#6 9.598       creating build/lib.linux-x86_64-cpython-311
#6 9.598       creating build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/__version__.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/__init__.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/backoff.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/builder.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       copying googlecloudprofiler/client.py -> build/lib.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       running build_ext
#6 9.598       building 'googlecloudprofiler._profiler' extension
#6 9.598       creating build/temp.linux-x86_64-cpython-311
#6 9.598       creating build/temp.linux-x86_64-cpython-311/googlecloudprofiler
#6 9.598       creating build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src
#6 9.598       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/_profiler.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/_profiler.o -std=c++11
#6 9.598       In file included from googlecloudprofiler/src/profiler.h:26,
#6 9.598                        from googlecloudprofiler/src/_profiler.cc:18:
#6 9.598       googlecloudprofiler/src/stacktraces.h: In member function ‘void AsyncSafeTraceMultiset::Reset()’:
#6 9.598       googlecloudprofiler/src/stacktraces.h:72:52: warning: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct AsyncSafeTraceMultiset::TraceData’ with no trivial copy-assignment; use value-initialization instead [-Wclass-memaccess]
#6 9.598          72 |   void Reset() { memset(traces_, 0, sizeof(traces_)); }
#6 9.598             |                                                    ^
#6 9.598       googlecloudprofiler/src/stacktraces.h:89:10: note: ‘struct AsyncSafeTraceMultiset::TraceData’ declared here
#6 9.598          89 |   struct TraceData {
#6 9.598             |          ^~~~~~~~~
#6 9.598       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/clock.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/clock.o -std=c++11
#6 9.598       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/log.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/log.o -std=c++11
#6 9.598       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Igooglecloudprofiler/src -I/usr/local/include/python3.11 -c googlecloudprofiler/src/profiler.cc -o build/temp.linux-x86_64-cpython-311/googlecloudprofiler/src/profiler.o -std=c++11
#6 9.598       In file included from googlecloudprofiler/src/profiler.h:26,
#6 9.598                        from googlecloudprofiler/src/profiler.cc:15:
#6 9.598       googlecloudprofiler/src/stacktraces.h: In member function ‘void AsyncSafeTraceMultiset::Reset()’:
#6 9.598       googlecloudprofiler/src/stacktraces.h:72:52: warning: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct AsyncSafeTraceMultiset::TraceData’ with no trivial copy-assignment; use value-initialization instead [-Wclass-memaccess]
#6 9.598          72 |   void Reset() { memset(traces_, 0, sizeof(traces_)); }
#6 9.598             |                                                    ^
#6 9.598       googlecloudprofiler/src/stacktraces.h:89:10: note: ‘struct AsyncSafeTraceMultiset::TraceData’ declared here
#6 9.598          89 |   struct TraceData {
#6 9.598             |          ^~~~~~~~~
#6 9.598       googlecloudprofiler/src/profiler.cc: In static member function ‘static void Profiler::Handle(int, siginfo_t*, void*)’:
#6 9.598       googlecloudprofiler/src/profiler.cc:160:32: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
#6 9.598         160 |     PyFrameObject *frame = ts->frame;
#6 9.598             |                                ^~~~~
#6 9.598             |                                cframe
#6 9.598       googlecloudprofiler/src/profiler.cc:163:40: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598         163 |       frames[num_frames].lineno = frame->f_lineno;
#6 9.598             |                                        ^~
#6 9.598       In file included from /usr/local/include/python3.11/Python.h:42,
#6 9.598                        from googlecloudprofiler/src/profiler.h:18,
#6 9.598                        from googlecloudprofiler/src/profiler.cc:15:
#6 9.598       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598          22 | typedef struct _frame PyFrameObject;
#6 9.598             |                ^~~~~~
#6 9.598       googlecloudprofiler/src/profiler.cc:164:41: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598         164 |       frames[num_frames].py_code = frame->f_code;
#6 9.598             |                                         ^~
#6 9.598       In file included from /usr/local/include/python3.11/Python.h:42,
#6 9.598                        from googlecloudprofiler/src/profiler.h:18,
#6 9.598                        from googlecloudprofiler/src/profiler.cc:15:
#6 9.598       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598          22 | typedef struct _frame PyFrameObject;
#6 9.598             |                ^~~~~~
#6 9.598       googlecloudprofiler/src/profiler.cc:166:20: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598         166 |       frame = frame->f_back;
#6 9.598             |                    ^~
#6 9.598       In file included from /usr/local/include/python3.11/Python.h:42,
#6 9.598                        from googlecloudprofiler/src/profiler.h:18,
#6 9.598                        from googlecloudprofiler/src/profiler.cc:15:
#6 9.598       /usr/local/include/python3.11/pytypedefs.h:22:16: note: forward declaration of ‘PyFrameObject’ {aka ‘struct _frame’}
#6 9.598          22 | typedef struct _frame PyFrameObject;
#6 9.598             |                ^~~~~~
#6 9.598       error: command '/usr/bin/gcc' failed with exit code 1
#6 9.598       [end of output]
#6 9.598   
#6 9.598   note: This error originates from a subprocess, and is likely not a problem with pip.
#6 9.601 error: legacy-install-failure
#6 9.601 
#6 9.601 × Encountered error while trying to install package.
#6 9.601 ╰─> google-cloud-profiler
#6 9.601 
#6 9.601 note: This is an issue with the package mentioned above, not pip.
#6 9.601 hint: See above for output from the failure.
#6 9.780 
#6 9.780 [notice] A new release of pip available: 22.3 -> 22.3.1
#6 9.780 [notice] To update, run: pip install --upgrade pip
------
executor failed running [/bin/sh -c pip3 install google-cloud-profiler]: exit code: 1

python:3 is equivalent to python:3.11.0
The install does succeed if I instead use python:3.10.8

Logging implementation could be better

What?
https://github.com/GoogleCloudPlatform/cloud-profiler-python/blob/master/googlecloudprofiler/__init__.py#L26
Defining the logging in the init() changes the configuration of the logger.

It makes the code looks pretty bad as before importing googlecloudprofiler. I have to initialize and add handler to it to the logging.

import logging

handler = logging.StreamHandler(stdout)
root_logger = logging.getLogger()
root_logger.addHandler(handler)

from googlecloudprofiler import start
...

Why?
First thing is that it took me a while to figure out that because of googlecloudprofiler the structlog is not showing the correct logs as it is not considering the format that I am defining later with logging.
I have used other python modules but never found such an implementation in the init.py file.
It is overshadowing the new logger implementation.

How?
Logger configuration can be defined somewhere and not in init.py such that just before using the googlecloudprofiler[NOT IMPORTING] the user has setup StreamHandler otherwise logging should be configured by your default method.

Syntax warning due to comparison of literals using is in Python 3.8

find . -iname '*.py'  | xargs -P 4 -I{} python -Walways -m py_compile {}

./googlecloudprofiler/client.py:167: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if len(self._profilers) is 0:

Use PyFrame_GetLineNumber to get line numbers

The CPU profiler currently uses f_lineno as the line associated with each sample. However, f_lineno points to the start of the frame. Looking at a source listing of these profiles in pprof results in all samples appearing to be from the function definition.

One can use PyFrame_GetLineNumber (available in Python 2.7 as well) instead to get the currently-executing line from a frame. This can help understand where within a function the program is executing.

pip install error

[root@plat-sh-infra-prod-jaeger-collector003 pypprof]# pip3 install google-cloud-profiler
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Collecting google-cloud-profiler
  Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/f5/b1/6a5f55bdcdc6d952268b91b4572b5ea31921f3e641cdbab464e6d68e5035/google-cloud-profiler-4.0.0.tar.gz (30 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wmo86p7q/google-cloud-profiler/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wmo86p7q/google-cloud-profiler/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info
         cwd: /tmp/pip-install-wmo86p7q/google-cloud-profiler/
    Complete output (50 lines):
    running egg_info
    creating /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info
    writing /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info/dependency_links.txt
    writing requirements to /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info/requires.txt
    writing top-level names to /tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info/top_level.txt
    writing manifest file '/tmp/pip-install-wmo86p7q/google-cloud-profiler/pip-egg-info/google_cloud_profiler.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-wmo86p7q/google-cloud-profiler/setup.py", line 112, in <module>
        'Programming Language :: Python :: 3.9',
      File "/usr/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/usr/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/usr/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/usr/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/usr/lib/python3.7/distutils/command/sdist.py", line 226, in add_defaults
        self._add_defaults_python()
      File "/usr/lib/python3.7/site-packages/setuptools/command/sdist.py", line 127, in _add_defaults_python
        build_py = self.get_finalized_command('build_py')
      File "/usr/lib/python3.7/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python3.7/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/usr/lib/python3.7/site-packages/setuptools/command/build_py.py", line 34, in finalize_options
        orig.build_py.finalize_options(self)
      File "/usr/lib/python3.7/distutils/command/build_py.py", line 45, in finalize_options
        ('force', 'force'))
      File "/usr/lib/python3.7/distutils/cmd.py", line 286, in set_undefined_options
        src_cmd_obj = self.distribution.get_command_obj(src_cmd)
      File "/usr/lib/python3.7/distutils/dist.py", line 857, in get_command_obj
        klass = self.get_command_class(command)
      File "/usr/lib/python3.7/site-packages/setuptools/dist.py", line 840, in get_command_class
        self.cmdclass[command] = cmdclass = ep.load()
      File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
        return self.resolve()
      File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
    ModuleNotFoundError: No module named 'setuptools.command.build'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.0.2; however, version 22.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

BlockingIOError on App Engine after some time

After adding google-cloud-profiler==1.1.1 to my Python 3.7 App Engine project, I'm getting lots of

BlockingIOError: [Errno 11] Resource temporarily unavailable
Exception ignored when trying to write to the signal wakeup fd:

in my service stderr logs

I'm having the same issue either with App Engine Standard python3.7 or with App Engine Flex over a custom runtime (Docker image)

Is this a known issue, is there something I can do to mitigate this issue ?

Please add a CHANGELOG

Hi,

I'm one of the maintainers of https://github.com/GoogleCloudPlatform/python-docs-samples/ which has some samples that use Cloud Profiler.

We use RenovateBot to keep our dependencies up to date. One of the nice things RenovateBot will do is pull in the contents of a repo Changelog. This makes it easier to determine significant changes between versions.

Could the maintainers add a CHANGELOG to this repository?

Thanks!

Add memory metrics support

I find the golang's SDK supports the memory metrics. https://codelabs.developers.google.com/codelabs/cloud-stackdriver-profiler/#3

Is there any idea to make python's SDK also support this?

Failed to collect and upload profile: dictionary changed size during iteration

this is the bug raised in production capture by sentry.

"Failed to collect and upload profile: dictionary changed size during iteration"

CPython, 3.8.2(3.8.2 (default, Apr 23 2020, 14:22:33) [GCC 8.3.0])

Please update dependency on google-api-python-client

I have a problem where I use google-api-python-client V2 which this library does not yet support :-)
Please update it so it can be used with V2 of google-api-python client

`
Updating dependencies
Resolving dependencies... (0.0s)

SolverProblemError

Because google-cloud-profiler (3.0.3) depends on google-api-python-client (<1.12.0 || >1.12.0,<2)
and no versions of google-cloud-profiler match >3.0.3,<4.0.0, google-cloud-profiler (>=3.0.3,<4.0.0) requires google-api-python-client (<1.12.0 || >1.12.0,<2).
So, because project-name depends on both google-api-python-client (^2.2.0) and google-cloud-profiler (^3.0.3), version solving failed.
`

Wall profile collection error due to line number being None on Python 3.10

Hi, we recently upgraded to Python 3.10 (from Python 3.9) in our service using cloud-profiler-python and are seeing the following error repeatedly in our logs:

INFO - warning: Failed to collect and upload profile whose profile type is WALL: Traceback (most recent call last):
INFO - warning:   File "/home/appuser/.local/lib/python3.10/site-packages/googlecloudprofiler/client.py", line 293, in _collect_and_upload_profile
INFO - warning:     profile_bytes = self._profilers[profile_type].profile(duration_ns)
INFO - warning:   File "/home/appuser/.local/lib/python3.10/site-packages/googlecloudprofiler/pythonprofiler.py", line 148, in profile
INFO - warning:     return self._serialize_and_clear_traces(duration_ns)
INFO - warning:   File "/home/appuser/.local/lib/python3.10/site-packages/googlecloudprofiler/pythonprofiler.py", line 243, in _serialize_and_clear_traces
INFO - warning:     profile_builder.populate_profile(self._traces, self._profile_type,
INFO - warning:   File "/home/appuser/.local/lib/python3.10/site-packages/googlecloudprofiler/builder.py", line 72, in populate_profile
INFO - warning:     location_id = self._location_id(func_id, frame[2])
INFO - warning:   File "/home/appuser/.local/lib/python3.10/site-packages/googlecloudprofiler/builder.py", line 131, in _location_id
INFO - warning:     line.line = line_number

After digging into it, the problem appears to be that the line number in the frame is None, which is described in this issue. The fix appears to have been backported to Python 3.11 but not 3.10, based on the issue comments.

I'm guessing the trace just needs to be discarded somehow if there's no line number but am not sure what the best way to handle this is.

Fortunately, it appears we are still collecting CPU and WALL profiles, but the repeated error messages are annoying.

Errors out on import

I receive the following error when I try to import the profiler. I tested several version of the profiler and the issue appears to start at version 3.0.0 of the profiler. I did not have the issue with 2.0.6

System Information:

Mac OS Big Sur 11.2.3
Python 3.8.7

Error:
ImportError: cannot import name '_profiler' from partially initialized module 'googlecloudprofiler' (most likely due to a circular import)

profile_pb2.py code is outdated and needs to be regenerated

With latest release of protobuf, the profile_pb2.py needs to be regenerated.
We are facing an issue in Apache Beam Python SDK importing the googlecloudprofiler which throws the protobuf version exception

pip install error on pypy3.9

Hello!
I'm developing this library for PyPy.

I get the following error when I run the minimal Dockerfile I wrote in the sample.

#Dockerfile
FROM pypy:3.9-7.3.12
RUN pip install --upgrade p
RUN pip install google-cloud-profiler

Do you know of a solution?
Or should I contact the PyPy administrator for this error?

~/Desktop/sample 
$ docker build ./
[+] Building 12.0s (6/6) FINISHED                                                                                                docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                             0.0s
 => => transferring dockerfile: 123B                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/pypy:3.9-7.3.12                                                                               0.8s
 => CACHED [1/3] FROM docker.io/library/pypy:3.9-7.3.12@sha256:a4420ce074f1105a586ec77f6c7c2d1a449077144187b423bc7669f527824440                  0.0s
 => [2/3] RUN pip install --upgrade p                                                                                                            3.2s
 => ERROR [3/3] RUN pip install google-cloud-profiler                                                                                            8.0s
------                                                                                                                                                
 > [3/3] RUN pip install google-cloud-profiler:                                                                                                       
0.951 Collecting google-cloud-profiler                                                                                                                
1.035   Downloading google-cloud-profiler-4.1.0.tar.gz (32 kB)                                                                                        
1.057   Preparing metadata (setup.py): started                                                                                                        
1.554   Preparing metadata (setup.py): finished with status 'done'                                                                                    
1.765 Collecting google-api-python-client!=1.12.0,!=2.0.2
1.781   Downloading google_api_python_client-2.97.0-py2.py3-none-any.whl (12.0 MB)
2.244      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 25.9 MB/s eta 0:00:00
2.411 Collecting google-auth>=1.0.0
2.425   Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
2.435      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 34.3 MB/s eta 0:00:00
2.457 Collecting google-auth-httplib2
2.473   Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
2.803 Collecting protobuf>=3.20
2.813   Downloading protobuf-4.24.1-py3-none-any.whl (175 kB)
2.823      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.7/175.7 kB 33.4 MB/s eta 0:00:00
2.875 Collecting requests
2.895   Downloading requests-2.31.0-py3-none-any.whl (62 kB)
2.903      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 14.7 MB/s eta 0:00:00
2.993 Collecting httplib2<1.dev0,>=0.15.0
3.006   Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
3.017      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 14.5 MB/s eta 0:00:00
3.142 Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0.dev0,>=1.31.5
3.159   Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
3.168      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 26.1 MB/s eta 0:00:00
3.190 Collecting uritemplate<5,>=3.0.1
3.203   Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
3.257 Collecting cachetools<6.0,>=2.0.0
3.275   Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
3.311 Collecting pyasn1-modules>=0.2.1
3.330   Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
3.339      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 36.3 MB/s eta 0:00:00
3.375 Collecting rsa<5,>=3.1.4
3.394   Downloading rsa-4.9-py3-none-any.whl (34 kB)
3.422 Collecting six>=1.9.0
3.434   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
3.477 Collecting urllib3<2.0
3.492   Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
3.500      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 37.1 MB/s eta 0:00:00
3.585 Collecting charset-normalizer<4,>=2
3.596   Downloading charset_normalizer-3.2.0-py3-none-any.whl (46 kB)
3.604      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.7/46.7 kB 11.0 MB/s eta 0:00:00
3.626 Collecting idna<4,>=2.5
3.637   Downloading idna-3.4-py3-none-any.whl (61 kB)
3.645      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 15.4 MB/s eta 0:00:00
3.680 Collecting certifi>=2017.4.17
3.698   Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
3.711      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 38.3 MB/s eta 0:00:00
4.006 Collecting googleapis-common-protos<2.0.dev0,>=1.56.2
4.019   Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
4.031      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 33.0 MB/s eta 0:00:00
4.140 Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
4.154   Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
4.162      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 28.3 MB/s eta 0:00:00
4.211 Collecting pyasn1<0.6.0,>=0.4.6
4.220   Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
4.227      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 24.3 MB/s eta 0:00:00
4.318 Building wheels for collected packages: google-cloud-profiler
4.319   Building wheel for google-cloud-profiler (setup.py): started
4.989   Building wheel for google-cloud-profiler (setup.py): finished with status 'error'
5.005   error: subprocess-exited-with-error
5.005   
5.005   × python setup.py bdist_wheel did not run successfully.
5.005   │ exit code: 1
5.005   ╰─> [32 lines of output]
5.005       running bdist_wheel
5.005       running build
5.005       running build_py
5.005       creating build
5.005       creating build/lib.linux-aarch64-3.9
5.005       creating build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/backoff.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/client.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/builder.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/__init__.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       copying googlecloudprofiler/__version__.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
5.005       running build_ext
5.005       building 'googlecloudprofiler._profiler' extension
5.005       creating build/temp.linux-aarch64-3.9
5.005       creating build/temp.linux-aarch64-3.9/googlecloudprofiler
5.005       creating build/temp.linux-aarch64-3.9/googlecloudprofiler/src
5.005       gcc -pthread -DNDEBUG -O2 -fPIC -Igooglecloudprofiler/src -I/opt/pypy/include/pypy3.9 -c googlecloudprofiler/src/_profiler.cc -o build/temp.linux-aarch64-3.9/googlecloudprofiler/src/_profiler.o -std=c++11
5.005       In file included from googlecloudprofiler/src/_profiler.cc:18:
5.005       googlecloudprofiler/src/profiler.h: In constructor ‘CodeDeallocHook::CodeDeallocHook()’:
5.005       googlecloudprofiler/src/profiler.h:59:25: error: ‘PyCode_Type’ was not declared in this scope; did you mean ‘PyCell_Type’?
5.005          59 |     old_code_dealloc_ = PyCode_Type.tp_dealloc;
5.005             |                         ^~~~~~~~~~~
5.005             |                         PyCell_Type
5.005       googlecloudprofiler/src/profiler.h: In destructor ‘CodeDeallocHook::~CodeDeallocHook()’:
5.005       googlecloudprofiler/src/profiler.h:67:24: error: ‘PyCode_Type’ was not declared in this scope; did you mean ‘PyCell_Type’?
5.005          67 |   ~CodeDeallocHook() { PyCode_Type.tp_dealloc = old_code_dealloc_; }
5.005             |                        ^~~~~~~~~~~
5.005             |                        PyCell_Type
5.005       error: command '/usr/bin/gcc' failed with exit code 1
5.005       [end of output]
5.005   
5.005   note: This error originates from a subprocess, and is likely not a problem with pip.
5.006   ERROR: Failed building wheel for google-cloud-profiler
5.007   Running setup.py clean for google-cloud-profiler
5.331 Failed to build google-cloud-profiler
5.612 Installing collected packages: urllib3, uritemplate, six, pyparsing, pyasn1, protobuf, idna, charset-normalizer, certifi, cachetools, rsa, requests, pyasn1-modules, httplib2, googleapis-common-protos, google-auth, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-profiler
7.269   Running setup.py install for google-cloud-profiler: started
7.837   Running setup.py install for google-cloud-profiler: finished with status 'error'
7.875   error: subprocess-exited-with-error
7.875   
7.875   × Running setup.py install for google-cloud-profiler did not run successfully.
7.875   │ exit code: 1
7.875   ╰─> [32 lines of output]
7.875       running install
7.875       running build
7.875       running build_py
7.875       creating build
7.875       creating build/lib.linux-aarch64-3.9
7.875       creating build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/backoff.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/client.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/builder.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/pythonprofiler.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/profile_pb2.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/cpu_profiler.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/__init__.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       copying googlecloudprofiler/__version__.py -> build/lib.linux-aarch64-3.9/googlecloudprofiler
7.875       running build_ext
7.875       building 'googlecloudprofiler._profiler' extension
7.875       creating build/temp.linux-aarch64-3.9
7.875       creating build/temp.linux-aarch64-3.9/googlecloudprofiler
7.875       creating build/temp.linux-aarch64-3.9/googlecloudprofiler/src
7.875       gcc -pthread -DNDEBUG -O2 -fPIC -Igooglecloudprofiler/src -I/opt/pypy/include/pypy3.9 -c googlecloudprofiler/src/_profiler.cc -o build/temp.linux-aarch64-3.9/googlecloudprofiler/src/_profiler.o -std=c++11
7.875       In file included from googlecloudprofiler/src/_profiler.cc:18:
7.875       googlecloudprofiler/src/profiler.h: In constructor ‘CodeDeallocHook::CodeDeallocHook()’:
7.875       googlecloudprofiler/src/profiler.h:59:25: error: ‘PyCode_Type’ was not declared in this scope; did you mean ‘PyCell_Type’?
7.875          59 |     old_code_dealloc_ = PyCode_Type.tp_dealloc;
7.875             |                         ^~~~~~~~~~~
7.875             |                         PyCell_Type
7.875       googlecloudprofiler/src/profiler.h: In destructor ‘CodeDeallocHook::~CodeDeallocHook()’:
7.875       googlecloudprofiler/src/profiler.h:67:24: error: ‘PyCode_Type’ was not declared in this scope; did you mean ‘PyCell_Type’?
7.875          67 |   ~CodeDeallocHook() { PyCode_Type.tp_dealloc = old_code_dealloc_; }
7.875             |                        ^~~~~~~~~~~
7.875             |                        PyCell_Type
7.875       error: command '/usr/bin/gcc' failed with exit code 1
7.875       [end of output]
7.875   
7.875   note: This error originates from a subprocess, and is likely not a problem with pip.
7.881 error: legacy-install-failure
7.881 
7.881 × Encountered error while trying to install package.
7.881 ╰─> google-cloud-profiler
7.881 
7.881 note: This is an issue with the package mentioned above, not pip.
7.881 hint: See above for output from the failure.
7.893 
7.893 [notice] A new release of pip is available: 23.0.1 -> 23.2.1
7.893 [notice] To update, run: pip install --upgrade pip
------
Dockerfile:3
--------------------
   1 |     FROM pypy:3.9-7.3.12
   2 |     RUN pip install --upgrade p
   3 | >>> RUN pip install google-cloud-profiler
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install google-cloud-profiler" did not complete successfully: exit code: 1

Failed to build the Discovery client for profiler, HttpError 429

When attempting to run the cloud profiler through the cloud run service, I encounter the following error:
ERROR:googlecloudprofiler.client:Failed to build the Discovery client for profiler (will retry after 19.609s): <HttpError 429 when requesting https://www.googleapis.com/discovery/v1/apis/cloudprofiler/v2/rest returned "Too Many Requests">

Any thoughts what am I missing based on the context files below?

Context
pyproject.toml file

python = "^3.10"
Flask = "^2.2.2"
Flask-API = "^3.0.post1"
gunicorn = "^20.1.0"
elasticsearch = "7.9.0"
confuse = "^2.0.0"
google-cloud-profiler = "^4.0.0"
google-auth = "^2.14.1"

App

import traceback
import json
import confuse
import googlecloudprofiler

from flask import jsonify
from flask import request
from flask_api import FlaskAPI

app = FlaskAPI("XXXXXX")

# Profiler initialization. It starts a daemon thread which continuously
# collects and uploads profiles. Best done as early as possible.
try:
    # service and service_version can be automatically inferred when
    # running on App Engine. project_id must be set if not running
    # on GCP.
    googlecloudprofiler.start(verbose=3)
except (ValueError, NotImplementedError) as exc:
    print(exc)  # Handle errors here

@app.route("/XXXXX", methods=["GET", "POST"])
def run():
    """
    Args:
        request (flask.Request): The request object.
        <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
    """
    request_json = request.get_json(silent=True)
    if request_json["text"]:
        return XXXXX.run(request=request_json, config=config)
    else:
        return {}
if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

Docker


RUN apt-get update && apt-get install -y gcc  \
    && apt-get install -y g++ \
    && rm -rf /var/lib/apt/lists/*

# Allow statements and log messages to immediately appear in the Knative logs
ENV PYTHONUNBUFFERED True

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

# Install production dependencies.
RUN pip install poetry \
    && poetry config virtualenvs.create false \
    && poetry install --no-dev --no-root

# Run the web service on container startup. Here we use the gunicorn
# webserver, with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
# Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

Python 3.12 `xgboost.core.XGBoostError: Invalid Parameter format for nthread expect int but value='-1'` when `DMatrix` used with `import googlecloudprofiler`.

This issue was originally posted in xgboost repo dmlc/xgboost#10224 .

I have a very peculiar error which happened when I've updated versions of Python and libs in project I'm working on.

Minimal example to reproduce the case is this:

# file.py
import googlecloudprofiler
from xgboost import DMatrix

DMatrix([[]])
print("works")

# requirements.txt
xgboost==2.0.3
google-cloud-profiler==4.1.0
#
numpy==1.26.4
scipy==1.13.0
google-api-python-client==2.125.0
google-auth==2.29.0
google-auth-httplib2==0.2.0
protobuf==4.25.3
requests==2.31.0
#
cachetools==5.3.3
certifi==2024.2.2
charset-normalizer==3.3.2
google-api-core==2.18.0
httplib2==0.22.0
idna==3.6
pyasn1==0.6.0
pyasn1_modules==0.4.0
pyparsing==3.1.2
rsa==4.9
uritemplate==4.1.1
urllib3==2.2.1

Python 3.12.2

Install with

pip install -r requirements.txt --no-deps

Run with

python file.py

Results in

Traceback (most recent call last):
  File "/project/path/file.py", line 4, in <module>
    DMatrix([[]])
  File "/venv/path/lib/python3.12/site-packages/xgboost/core.py", line 730, in inner_f
    return func(**kwargs)
           ^^^^^^^^^^^^^^
  File "/venv/path/lib/python3.12/site-packages/xgboost/core.py", line 857, in __init__
    handle, feature_names, feature_types = dispatch_data_backend(
                                           ^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/path/lib/python3.12/site-packages/xgboost/data.py", line 1081, in dispatch_data_backend
    return _from_list(data, missing, threads, feature_names, feature_types)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/path/lib/python3.12/site-packages/xgboost/data.py", line 1011, in _from_list
    return _from_numpy_array(array, missing, n_threads, feature_names, feature_types)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/path/lib/python3.12/site-packages/xgboost/data.py", line 207, in _from_numpy_array
    _check_call(
  File "/venv/path/lib/python3.12/site-packages/xgboost/core.py", line 282, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: Invalid Parameter format for nthread expect int but value='-1'

To "solve" the problem remove import googlecloudprofiler from file.py. I really have no idea why just importing the lib causes this problem; it would make more sense after googlecloudprofiler.start is called.

Moreover the code works for xgboost=1.7.6 and fails since xgboost=2.0.0.

Maintainer of xgboost mentioned

loading the _profiler.cpython-312-x86_64-linux-gnu.so inside google profiler extension causes the error

dmlc/xgboost#10224 (comment)

This is why I've opened issue here.

CPU profiling with gevent ?

Dear maintainers:
I have a celery application that processes asynchronous tasks.
I ran the worker with -P gevent option and it would hang for 10 seconds every chunk of time with the profiler enabled.
The profiler was initialized before the celery application comes up.
When I set verbose=3 on googlecloudprofiler.start(), I observed that the hiccup would happen between
Successfully created a CPU profile and Starting to upload profile.
My configuration works with any of the following actions conducted:

Set disable_cpu_profiling=True on googlecloudprofiler.start()
Omit -P gevent option on celery worker command.

Would it be related to the GIL ? Can some of you guys take a look at it ?
If there's not enough information please feel free to ask me for it.
Thanks.

add `protobuf` version requirement

google-cloud-profiler==4.0.0 seems to require at least protobuf>=3.20. it fails with protobuf-3.19.4:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install google-cloud-profiler==4.0.0 protobuf==3.19.4
$ python -c 'import googlecloudprofiler'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/j/venv/lib/python3.8/site-packages/googlecloudprofiler/__init__.py", line 19, in <module>
    from googlecloudprofiler import client
  File "/Users/j/venv/lib/python3.8/site-packages/googlecloudprofiler/client.py", line 41, in <module>
    from googlecloudprofiler import pythonprofiler
  File "/Users/j/venv/lib/python3.8/site-packages/googlecloudprofiler/pythonprofiler.py", line 23, in <module>
    from googlecloudprofiler import builder
  File "/Users/j/venv/lib/python3.8/site-packages/googlecloudprofiler/builder.py", line 19, in <module>
    from googlecloudprofiler import profile_pb2
  File "/Users/j/venv/lib/python3.8/site-packages/googlecloudprofiler/profile_pb2.py", line 19, in <module>
    from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/Users/j/venv/lib/python3.8/site-packages/google/protobuf/internal/__init__.py)

Cloud Run Build Error

I am running on FastAPI but get a build error on Cloud Run:

Step #0 - "Build": Failed to build google-cloud-profiler
Step #0 - "Build": ERROR: Could not build wheels for google-cloud-profiler, which is required to install pyproject.toml-based projects

Base Dockerfile:

FROM tiangolo/uvicorn-gunicorn:python3.11-slim

Profiler suddenly stopped working

Hello GCloud Team,

I have a problem with this library last month or so. It suddenly stopped working, nothing got changed in the console nor code. Could you please help me? We are using this in 3 environments (one of them is production), but we are unable to make it work properly.

How to reproduce
I don't have precise steps to reproduce this problem (because it happened all the sudden), so I cannot replicate this issue.

Logs

E 2019-10-30T12:13:46.389820758Z Failed to collect and upload profile: <built-in function profile_cpu> returned NULL without setting an error 
E 2019-10-30T12:13:46.187933854Z Failed to set ITIMER_PROF: � 
E 2019-10-30T12:13:17.687470023Z Failed to collect and upload profile: <built-in function profile_cpu> returned NULL without setting an error 
E 2019-10-30T12:13:17.495344141Z Failed to set ITIMER_PROF: � 
E 2019-10-30T12:12:34.063318304Z Failed to collect and upload profile: <built-in function profile_cpu> returned NULL without setting an error 
E 2019-10-30T12:12:33.852451918Z Failed to set ITIMER_PROF: � 
E 2019-10-30T12:12:05.084790490Z Failed to collect and upload profile: <built-in function profile_cpu> returned NULL without setting an error 
E 2019-10-30T12:12:04.843544227Z Failed to set ITIMER_PROF: �

Question: How often do new releases get shipped to PyPI?

I'm interested in integrating this library with one of my projects but there's a dependency conflict in version 2.0.3. I see a fix was checked in 9 days ago that will resolve the conflict and has bumped the version to 2.0.4, but it's not clear when it will go out. Thanks for your help!

Error when used with google-cloud-pubsub==1.5.0

When deploying a app engine flex app with the google-cloud-pubsub==1.5.0, the following error is seen in the logs:

INFO:googlecloudprofiler:Google Cloud Profiler Python agent version: 2.0.4
DEBUG:googlecloudprofiler.client:Profiler has started
ERROR:googlecloudprofiler.client:Failed to build the Discovery client for profiler (will retry after 40.744s): 'ClientOptions' object has no attribute 'credentials_file'

This is only fixed when google-cloud-pubsub is upgraded to 1.6.0 or greater.

[Wall time] Google Cloud Profiler Strange Behavior With Python time.sleep()

I have been experimenting with Google Cloud Profiler and Python.

I wanted to know if it was a reliable way to measure how long a function takes to run that contains sleep commands.

I have a gist with 3 sample scripts with slight variations. Mostly the same with differing wait times. The last is similar to the second except in the main loop it calls step1 multiple times at the end.
https://gist.github.com/crimsonaltima/402b41ee23be20491d73f97832c0c017

Running these, produce the following results.
Seq1

Seq2

Seq3

One thing I find odd about the profiler is that it shows a max duration of 10 seconds. Despite some functions sleeping for over 25 seconds. Is there an explanation for this?
While the bar sizes are accurate relative to their sleep time, it does not show exactly how long a function takes.

A second question, in mainseq3.py I call step 1 multiple times and noticed that all the steps1s were merged into a single bar and there is no distinction as for if this was the first call to step1 or a latter. Is that expected behavior?

A lesser question I have is does time.sleep(...) have odd effects on the profiler? I noticed the script with the really long sleep times took a very long time to begin uploading data to google cloud profiler.

segmentation fault under high load

We noticed that we get sporadic restarts in one of our apps, especially when under high load.

Our apps are running on gke:
k8s version: 1.26.7 machine image: cos_containerd

python version: 3.11
fastapi & uvicorn

google-cloud-profiler version: 4.1.0

I've enabled debug logging in profiler, but I don't get much more details except:

DEBUG:googlecloudprofiler.client:Starting to create profile DEBUG:googlecloudprofiler.client:Successfully created a CPU profile Segmentation fault

some additional logs:
kernel: [ 8685.460126] uvicorn[86674]: segfault at 48 ip 00007f78dc517b01 sp 00007f78dcff7020 error 4 in _profiler.cpython-311-x86_64-linux-gnu.so[7f78dc514000+a5000] likely on CPU 0 (core 0, socket 0) kernel: [ 8685.477270] Code: ff ff 0f 1f 80 00 00 00 00 41 54 55 48 89 fd 53 48 85 f6 0f 84 a8 00 00 00 48 8b 46 38 48 8b 58 08 48 85 db 0f 84 8c 00 00 00 <80> 7b 45 01 74 19 48 8b 43 20 48 63 90 a8 00 00 00 48 8d 84 50 b8

have you faced similar issues before and is there a possible solution/workaround?

AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

Starting today, when I import googlecloudprofiler, I get this traceback;

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/app/collection/__init__.py", line 16, in <module>
    import googlecloudprofiler
  File "/usr/local/lib/python3.7/site-packages/googlecloudprofiler/__init__.py", line 19, in <module>
    from googlecloudprofiler import client
  File "/usr/local/lib/python3.7/site-packages/googlecloudprofiler/client.py", line 29, in <module>
    import google_auth_httplib2
  File "/usr/local/lib/python3.7/site-packages/google_auth_httplib2.py", line 23, in <module>
    import httplib2
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 52, in <module>
    from . import auth
  File "/usr/local/lib/python3.7/site-packages/httplib2/auth.py", line 20, in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

Looks like the latest pyparsing release changed the name of these functions:
https://pyparsing-docs.readthedocs.io/en/latest/whats_new_in_3_0_0.html#other-discontinued-features

upcaseTokens and downcaseTokens - convert to using pyparsing_common.upcase_tokens and downcase_tokens

"Warning" is being published as "Error"

Summary

In commit (8147311#diff-6bc78e58322668fa0a7c4d34928f7647f75ac41852c8df1ee3e3aee2fd422966), you have changed log.error to log.warning. However, it seems that GCP Cloud Logging will take this as "Error" anyway since python's logging library is not fully respected[1]. There is an internal issue on this inside Google[2].

To heal this issue, we should import google.cloud.logging library. Please see the below reproducing steps.

Reproducing steps

I have tried to run following in Cloud Functions and "Error" came out anyway instead of "Warning" in Cloud Logging:

main.py:

import logging

logger = logging.getLogger(__name__)

def hello_world(request):
  logger.warning("this is warning")

requirements.txt:

google-cloud-logging>=2.7.0

After I added google.cloud.logging as follows, the warning has been published as "Warning" in cloud logging:

main.py:

import logging

import google.cloud.logging # Don't conflict with standard logging
client = google.cloud.logging.Client()
client.setup_logging()

logger = logging.getLogger(__name__)

def hello_world(request):
  logger.warning("this is warning")

Request to the team

Could you please check the above hypothesis and let me know whether I am right or not?
If so, could you fix this accordingly so that "Warning" in backoff file is published as "Warning" in Cloud Logging?

[1]. https://stackoverflow.com/questions/65776399/why-is-my-gcp-function-logging-errors-that-should-be-info-or-debug
[2]. https://issuetracker.google.com/124403972

Distribute wheels on PyPI?

It seems like the google-cloud-profiler PyPI package doesn't have any wheels, so it requires a compiler toolchain to install. It would be great if the package had pre-built wheels.

(host) $ docker run --rm -it amd64/python:3.8-slim-buster bash
(docker) $ pip install google-cloud-profiler
[...eventually fail due to missing gcc...]

(I'm making this request because Apache Beam added a dependency on Cloud Profiler recently, and this caused my Docker build to fail because we don't have a compiler available. To avoid adding bulk to our Docker image, we'd need to use multi-stage builds. As a Beam user, this seems like a fair amount of added complexity for a transitive/indirect dependency.)

Thanks!

Collect profiles from other threads.

Different serving environments have different thread utilizations -- only collecting from the main thread doesn't capture all workloads correctly.

Error message in backoff process

Every time the back-off process runs, an error message appears.
Code:

cloud-profiler-python/googlecloudprofiler/client.py

Lines 303 to 316 in bf96efa

 def _poll_profiler_service(self): 

 """Polls the profiler server stoplessly.""" 

 logger.debug('Profiler has started') 

 build_service_backoff = backoff.Backoff() 

 while self._profiler_service is None: 

 try: 

 self._profiler_service = self._build_service() 

 except BaseException as e: # pylint: disable=broad-except 

 # Exponential backoff. 

 backoff_duration = build_service_backoff.next_backoff() 

 logger.error( 

 'Failed to build the Discovery client for profiler ' 

 '(will retry after %.3fs): %s', backoff_duration, str(e)) 

 time.sleep(backoff_duration)

I often get this error, but this is a little bit noisy (because I don't have to worry about it if a retry succeeds).

Ideally, I think, the error message should appear only after a certain number of trials or a certain elapsed time.

Do you have any plan to fix this?

	'-static-libstdc++',
	# While libgcc_s.so.1 is pretty much always installed by default
	# for non-Alpine linux, it is not installed by default in Alpine.
	# So, to support Alpine, we will always statically link "libgcc"
	# package. We could alternatively require users to install the
	# "libgcc" package, but the static linkage seems less
	# invasive.
	'-static-libgcc'

	def _poll_profiler_service(self):
	"""Polls the profiler server stoplessly."""
	logger.debug('Profiler has started')
	build_service_backoff = backoff.Backoff()
	while self._profiler_service is None:
	try:
	self._profiler_service = self._build_service()
	except BaseException as e: # pylint: disable=broad-except
	# Exponential backoff.
	backoff_duration = build_service_backoff.next_backoff()
	logger.error(
	'Failed to build the Discovery client for profiler '
	'(will retry after %.3fs): %s', backoff_duration, str(e))
	time.sleep(backoff_duration)

googlecloudplatform / cloud-profiler-python Goto Github PK

cloud-profiler-python's Introduction

Google Cloud Python profiling agent

Supported OS

Supported Python Versions

Installation & usage

Installation on Linux Alpine

Troubleshooting

Resource temporarily unavailable errors with Python

cloud-profiler-python's People

Contributors

Stargazers

Watchers

Forkers

cloud-profiler-python's Issues

Summary

Reproducing steps

main.py:

requirements.txt:

main.py:

Request to the team

Recommend Projects

Recommend Topics

Recommend Org