joeyhendricks / python-benchmark-harness Goto Github PK

The imports of the example code is incorrect.

Will be adding the regression algorithms I have developed to detect relevant changes between software changes.
More info:

https://github.com/JoeyHendricks/automated-performance-test-result-analysis

I attached SonarCloud to QuickPotato this issue is to remediate all of the Python problems that SonarCloud found.

The better automate testing more unit tests need to be added to the library.

Got the following error in one of my project will fix this and update QuickPotato.

C:\Program Files\Python38\lib\copyreg.py:91: SADeprecationWarning: Calling URL() directly is deprecated and will be disabled in a future release.  The public constructor for URL is now the URL.create() method.
  return cls.__new__(cls, *args)


Ran 1 test in 1.273s

FAILED (errors=1)

Error
Traceback (most recent call last):
  File "C:\Users\user\OneDrive\My Documents\PycharmProject\venster\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "C:\Users\user\OneDrive \My Documents\PycharmProject\venster\venv\lib\site-packages\sqlalchemy\engine\default.py", line 719, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Hi Joey,

I have implemented my ideas on the observer_pattern branch of my fork. You can examine the diff here, although be warned that it is rather large.

I'm most confident in the changes I made to QuickPotato.profiling.intrusive and QuickPotato.profiling.interpreters. I'm least confident in any of the changes I made to the other test infrastructure.

If I run test_pt_boundary_testing, all tests pass.

If I run test_pt_visualizations, the test fails, although I'm not sure it was every supposed to succeed in the first place, as FlameGraph() does not receive a test_id.

I'm most confused about the test_pt_regression_testing. If I run the tests again and again and again without making any changes to the code, I get different results. Observe the following screenshots, and note that they were all taken successively with no changes to code.

I'm confused about why this happens. Is this expected behavior, or have I messed something up?

Edit: In case you are confused about how my PerformanceBreakpoint decorator should be used, see the following example:

# standard library
from time import sleep
from math import log10

# QuickPotato
from QuickPotato.profiling.intrusive import PerformanceBreakpoint
from QuickPotato.profiling.interpreters import SimpleInterpreter, StatisticsInterpreter

@PerformanceBreakpoint
def function_1():
    sleep(1)
    return len(str(log10(7**7**7)))

@PerformanceBreakpoint(observers=[StatisticsInterpreter])
def function_2():
    sleep(1)
    return len(str(log10(7**7**7)))

@PerformanceBreakpoint(observers=[SimpleInterpreter])
def function_3():
    sleep(1)
    return len(str(log10(7**7**7)))

# --- and now in console ---

>>> from QuickPotato import performance_test
... from ??? import function_1, function_2, function_3

# required in order to set performance_test.test_id
>>> performance_test.test_case_name = 'MyDemonstrativeTest'

>>> function_3()
# runs the SimpleInterpreter, which just prints to console, and returns the function's value
SimpleInterpreter
 ├─ self.method_name='function_3'
 ├─ self.test_id='GU98BK70CBI9'
 ├─ self.sample_id='BHVO2ZDN'
 ├─ self.database_name='QuickProfiling'
 ├─ subject.profiler.total_response_time=1.1437058448791504
17

>>> function_1()
# simply runs the function without profiling
17

>>> function_2()
# runs the StatisticsInterpreter, which logs to the database, and returns the function's value
17

I've not tested the execution_wrapper parameter yet, but I think I need to use that with Dagster.
When I used JoeyHendricks/QuickPotato:master with Dagster:

• profiling a Dagster pipeline only captured the pipeline setup, not its full run
• profiling a Dagster solid captured nothing and wrote None for every column of the performance_statistics database.

Based on that experience, I think I will need to respectively pass the following to execution_wrapper:

• execute_pipeline
• execute_solid

Edit 2: re Dagster Nope. I'm going to have to experiment a bit more with decorator voodoo.

I have noticed that the database connector in QuickPotato is not working as expected will improve this in next release.

The why:

When you're testing code performance outside of a "unit performance test" it is not possible to get all collected samples into one HTML report. This happens because the test id keeps changing everytime the function is executed not allowing you to rapidly capture information about your code.

Work around:

Use the unit performance test class to manually set a test case that will force QuickPotato to not generate a new one.

Solution:

Investigate options to allow multiple executions to appear in the HTML test report.

Test cases do not always flow nicely after one and another resulting in unnecessary failures.
I need to extend and improve the test cases.

This is a nice library. I had some time to play with it this afternoon (we previously exchanged a few messages on Reddit) and I am impressed.

If you don't mind, I am going to use this GitHub issue to maintain a dialogue with you since it appears that you have not enabled GitHub Discussions. If I develop any concrete feature requests or specific questions or bug reports, I will spin those out as separate and discrete GitHub issues.

I intend to use QuickPotato to find slow parts of my Dagster pipelines.

Dagster describes itself in the following way:

Dagster is a data orchestrator for machine learning, analytics, and ETL

Essentially, Dagster is a competitor/spiritual successor to Apache Airflow. It is used as follows:

• a data engineer defines discrete actions as Python functions. Each one of these Python functions is called a Solid.
• solids can be chained together to form a Pipeline, which is analogous to an Apache Airflow DAG ("directed acyclic graph").
• Dagster offers many other non-executable constructs, like Resource, Mode, Schedules, Sensor, IOManager, etc.; it really is a well thought out system, but for now, those aren't relevant to my questions.
• a Pipeline can be executed automatically via Schedule (think cron job), triggered by a Sensor ("run this pipeline when a new file appears in the S3 bucket"), run via the Dagster CLI, or run through the Dagit UI.

I was most attracted to this library because of the intrusive profiling this library offers. The other alternatives I looked at (py-spy, pyflame, etc.) require providing a specific Python script as an argument to the profiler. That approach simply doesn't work when integrating with Dagster! Being able to wrap a function and to know that the function will be profiled whenever it is called (and enabled=True) is really awesome. Congratulations 😀

I have provided the context above just so you understand my use cases.

I've watched your NeotysPAC demonstration. Please correct me if I am wrong, but it seems like your initial idea was to build the intrusive profiler, and since then you have subsequently added in the unit test workflows (for example, the pt.measure_method_performance(...) construct). At this point, it seems like the performance_breakpoint decorator is not feature complete.

Below are my thoughts after experimenting with QuickPotato for about an hour. Please note that I have not yet looked through the codebase extensively and I might hold some misconceptions.

• I would like to be able to provide more configuration arguments to performance_breakpoint. It looks like it inherits the performance_test.test_case_name from scope, but being able to pass this in, or dynamically set it, or generate it using some sort of rule (lambda?) would be great. Likewise with a database name/connection.

• It took me a while to figure out why QuickPotato wasn't writing to my PostgreSQL server (docker container). I finally saw that the database name is set by the test_case_name. This was not intuitive for me. It would be helpful if you could document how QuickPotato organizes its recordings, i.e. that each test case gets its own database table.

# QuickPotato/profiling/intrusive.py
database_name=performance_test.test_case_name

# QuickPotato/database/operations.py
return f"{self.URL}/{database_name}"

• I would like a way to link a specific execution of a Dagster Pipeline to its QuickPotato profile record. Each run of a Dagster Pipeline is automatically assigned a UUID, and its metadata, logs, etc. are stored in a database keyed by this UUID. It looks like each QuickPotato benchmark has both a test_id and a sample_id. It would be great to be able to get those from the StatisticsInterpreter before/during/after it writes to the database so I can then match the Dagster and QuickPotato IDs. I don't know how to do that though. Maybe modifying the decorator to work with the observer pattern could work? The test_id and sample_id could be given to some other function which handles matching the profile statistics with the Dagster run_id.

Anyway, great work so far!

Heatmap not including the code paths:

• Include the code paths
• better message when code paths are not included

Hi
i ve just discovered QuickPotato and for the first use i ve encountered strange behaviour.

this the very simple test i ve used

first point
On the method pt.measure_method_performance there is not named_arguments param, i looked in the source code and it figure than named parameters are not considered.

second point
data.average_response_time raise a ZeroDivisionError because data._response_times is an empty list.

best regards

QuickPotato can intrusively profile code this feature is overshadowed by the benchmarking object but this needs to be reworked and properly documented.

The why:

I saw that when I was not using the configuration class to define settings in the YAML file.
That QuickPotato would refuse to spawn a database.

Solution:

I need to look into why this happens and come up with a fix will update this issues as soon as possible.

Will be improving how databases are handled in QuickPotato for that the micro benchmarking core needs to be also reworked.
The idea behind this change is that QuickPotato works within one database using a test case name as an identifier.

Because of this change, QuickPotato can better integrate within a corporate setting and will become more friendly to database vendors like Oracle. This change also entails that the database system behind QuickPotato will become less complicated and relies more on SQLalchemy core than in the previous iteration.

Also, this change will help out in when and how data is stored in the benchmarking database because of this the benchmarking object will become less complicated in regards to saving data.

Summarized this issue will improve the following things:

• Reduce complexity around databases
• No more multiple databases by default (Can still be determined by the user by overwriting the database connection string.)
• Better save mechanisms around test evidence.
• Improvements to how it is possible to interact with statistical measurements from the benchmark object.
• Overall structural improvements to make room for the above changes.
• reworked parts contain a new or better doc string.

Commits will be pushed to this repo when the changes are complete.

The why:

When running in SQLite mode (not attached to a proper database server) QuickPotato will crash when the insert payload is over 999.
This crash can happen every time when turning on system resource collection or when your stacks are large.

Work around:

Turn off system resource collection in the options file and for now refrain from measuring large and complex piece of code when running in SQLite mode.
Or you can just connect to a database server like MySQL where to problem does not exist.

Reason:

SQLite cannot insert payload greater than 999 SQL variables therefore it would spit out the following error:

"too many SQL variables"

Link to problem description on stack overflow:

• https://stackoverflow.com/questions/7106016/too-many-sql-variables-error-in-django-with-sqlite3

Solution

Within the interpreter classes I need to make changes so the following action could be preformed:

• Check if SQLite is used as a database server within the interpreter class
• Split payload (if payload is over 999 and SQLite is used) in multiple chunks