bcg-x-official / artkit Goto Github PK
View Code? Open in Web Editor NEWAutomated prompt-based testing and evaluation of Gen AI applications
Home Page: https://bcg-x-official.github.io/artkit/_generated/home.html
License: Apache License 2.0
Automated prompt-based testing and evaluation of Gen AI applications
Home Page: https://bcg-x-official.github.io/artkit/_generated/home.html
License: Apache License 2.0
We plan to add an optional progress bar to the run
method in the dependency, fluxus
: BCG-X-Official/fluxus#23
It will then be available to the run
method in artkit
and we should introduce it in the "Building Your First ARTKIT Pipeline" tutorial, specifically in the section where we introduce the core ARTKIT functions.
Once the progress bar is added and we have updated the fluxus
version in artkit
, we will re-open this issue.
Google's Gemini supports text-to-image, but currently we only have a text-to-text connector. We should support text-to-image.
The new class will live here: src/artkit/model/diffusion/gemini/
Be sure to update the docs:
Connecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbPlease see the tutorial: Creating New Model Classes
Google's Gemini supports image-to-text, but currently we only have a text-to-text connector. We should support image-to-text.
The new class will live here: src/artkit/model/vision/gemini/
Be sure to update the docs:
Connecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbPlease see the tutorial: Creating New Model Classes
Currently, the README.rst
file does not contain any displayed badges that represent the repository. Some are in pypi_description.rst but don't have full list (see below)
Additional details on the .rst
file format are located here
Specifically, we'd like to add badges for the following:
Here's an example with all badges we want to replicate. Note: the example uses Azure DevOps to add the test coverage badge but this repo utilizes GitHub Actions so a different approach is required and will entail changes to the artkit-release-pipeline.yml. Given it's a more complex change, a separate issue documents the approach:
Readme.rst
and pypi_description.rst
with the relevant badges
.. Begin-Badges
|pypi| |conda| |python_versions| |code_style| |made_with_sphinx_doc| |license_badge| |Contributor_Convenant|
.. End-Badges
.. Begin-Badges
.. |pypi| image:: https://badge.fury.io/py/artkit.svg
:target: https://pypi.org/project/artkit/
.. |conda| image:: https://anaconda.org/bcg_gamma/gamma-facet/badges/version.svg
:target: https://anaconda.org/BCG_Gamma/artkit
.. |python_versions| image:: https://img.shields.io/badge/python-3.10|3.11|3.12-blue.svg
:target: https://www.python.org/downloads/release/python-3100/
.. |code_style| image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
.. |made_with_sphinx_doc| image:: https://img.shields.io/badge/Made%20with-Sphinx-1f425f.svg
:target: https://bcg-x-official.github.io/facet/index.html
.. |license_badge| image:: https://img.shields.io/badge/License-Apache%202.0-olivegreen.svg
:target: https://opensource.org/licenses/Apache-2.0
.. image:: https://github.com/BCG-X-Official/artkit/actions/workflows/artkit-release-pipeline.yml/badge.svg
:target: https://github.com/BCG-X-Official/artkit/actions/workflows/artkit-release-pipeline.yml
:alt: ARTKIT Release Pipeline
.. |Contributor_Convenant| image:: https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg
:target: CODE_OF_CONDUCT.md
.. End-Badges
RELEASE_NOTES.rst
file:
Problem
The tutorials in our Sphinx documentation are HTML pages built from Jupyter notebooks. Readers may want to download the Jupyter notebook so they can run the code, but it isn't clear where they can find the notebook.
Solution
Within each notebook tutorial, include a hyperlink to the Jupyter notebook on GitHub. From GitHub, the user can choose to download the notebook and explore related files (e.g., data or cache files which are used by the notebook).
Details
Place hyperlinks just beneath the main title of notebooks, e.g.:
# Building Your First ARTKIT Pipeline
[View notebook on GitHub](https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/building_your_first_artkit_pipeline.ipynb)
Add hyperlinks to all Jupyter notebooks which are included in the User Guide and Examples sections of the Sphinx documentation.
In the artkit
GitHub repo, you can find these files within:
sphinx/source/user_guide/
sphinx/source/examples/
Verifying your changes
Currently, CacheDB.clear
allows users to clear cached results which were accessed or created before a given datetime
:
However, it is not possible to clear results after a given datetime
. This would be useful when I make a mistake midway through an analysis and I want to clear only the most recent results from my cache.
CacheDB
class:
accessed_after
and created_after
optional arguments to the CacheDB.clear
methodCachedGenAIModel
class (uses CacheDB
):
accessed_after
and created_after
optional arguments to the CachedGenAIModel.clear_cache
methodOnce implemented, be sure to:
test_cache_clearing
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Amazon Bedrock provides an extensive suite of Gen AI models via a single API. Supporting Amazon Bedrock will significantly enhance ARTKIT's versatility.
Note that the Bedrock API supports multiple modalities, depending on the selected model. This issue is for LLMs. The new class will live in src/artkit/model/llm/aws/
Be sure to update the docs:
README
: https://github.com/BCG-X-Official/artkit?tab=readme-ov-file#supported-model-providersContributing
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbConnecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynb.env_example
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/.env_examplePlease see the tutorial: Creating New Model Classes
As the https://bcg-x-official.github.io/artkit/user_guide/introduction_to_artkit/connecting_to_genai_models.html#Google link is empty with no connection result.
Amazon Bedrock provides an extensive suite of Gen AI models via a single API. Supporting Amazon Bedrock will significantly enhance ARTKIT's versatility.
Note that the Bedrock API supports multiple modalities, depending on the selected model. This issue is for text-to-image diffusion models. The new class will live in src/artkit/model/diffusion/aws/
Be sure to update the docs:
README
: https://github.com/BCG-X-Official/artkit?tab=readme-ov-file#supported-model-providersContributing
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbConnecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynb.env_example
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/.env_examplePlease see the tutorial: Creating New Model Classes
Cohere focuses on providing low-latency LLMs. They have an async endpoint we can use and free API keys.
The new class will live in src/artkit/model/llm/cohere/
Be sure to update the docs:
README
: https://github.com/BCG-X-Official/artkit?tab=readme-ov-file#supported-model-providersContributing
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbConnecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynb.env_example
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/.env_examplePlease see the tutorial: Creating New Model Classes
The Azure OpenAI API is compatible with OpenAI's API. With Azure OpenAI, one can set up their own deployments of the common GPT and Codex models. When calling the API, they need to specify the deployment they want to use.
The new class will live in src/artkit/model/llm/azureopenai/
Be sure to update the docs:
README
: https://github.com/BCG-X-Official/artkit?tab=readme-ov-file#supported-model-providersContributing
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynbConnecting to Gen AI Models
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynb.env_example
: https://github.com/BCG-X-Official/artkit/blob/1.0.x/.env_examplePlease see the tutorial: Creating New Model Classes
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
Previously, classes in model.llm.base like ChatModel where reachable through artkit.api and its been popular to use ak.ChatModel for type notations in notebooks. However, this is no longer possible as from .model.llm.base import * is missing from api.py
Describe the solution you'd like
Selectively include classes from .base modules such as ChatModel : from .model.llm.base import ChatModel
Describe alternatives you've considered
Alternative is to import all packages from base, which might be too much
Describe the bug
artkit.api
does not import CachedDiffusionModel
and CachedVisionModel
.
To Reproduce
Steps to reproduce the behavior:
https://github.com/BCG-X-Official/artkit/blob/1.0.x/sphinx/source/user_guide/introduction_to_artkit/connecting_to_genai_models.ipynb
Expected behavior
CachedDiffusionModel
and CachedVisionModel
should be imported with artkit.api
Additional context
The fix is to update the following imports in src/artkit/api.py
, which currently do not seem to import anything:
from .model.diffusion import *
from .model.vision import *
to:
from .model.diffusion.base import *
from .model.vision.base import *
Please can you provide few documentation regarding the same and the gemini url that is being used.
Is your feature request related to a problem? Please describe.
The only connectors that exist currently are for existing models with clearly defined APIs. There is no way for a user to specify any HTTP endpoint to get a response.
Describe the solution you'd like
I want to be able to use ARTKIT with any endpoint that accepts a payload.
Describe alternatives you've considered
None
Additional context
It seems like a version of the existing HuggingFace connector could be leveraged
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.