clinicalml / co-llm Goto Github PK

View Code? Open in Web Editor NEW

89.0 89.0 7.0 7.7 MB

Co-LLM: Learning to Decode Collaboratively with Multiple Language Models

Home Page: https://arxiv.org/abs/2403.03870

Python 93.47% Shell 6.19% JavaScript 0.34%

co-llm's People

Contributors

Stargazers

Watchers

Forkers

souradip-umd lalomorales22 evdcush eltociear machinelearningsystem jipyeong-lee evelynmitchell

co-llm's Issues

About process_dataset

Hi, all! Thank you for your contribution. I am interested in your work.
I try to run your code. When I execute process_dataset.sh. I seem to find this file missing: python: can't open file 'open_instruct/process_dataset.py.
Looking for your help. Thanks~

Issue with Running deferral_generate.sh Script

When I run the bash scripts/evaluation/gsm8k/deferral_generate.sh script, I keep seeing the message "Checking again..." repeatedly. I am not sure what this means or how to resolve it. Could you please help me understand what is causing this and how to fix it?

Questions Regarding the Experimental Results

Hi,

I recently read your work in detail and found the idea of combining large language models with learning to defer to be quite creative. However, I have a few questions that I would like to discuss with you.

In Table 1, which shows Collaboration across domains, it seems that the "Co-LLM-7B + LLEMMA-34B" method does not consistently outperform the "PT (LLEMMA-34B + LLEMMA-7B)" method in several tasks, such as GSM, Factoid, List, and Yes/No. （I'll highlight the highest value in red）
In Table 2, which shows Collaboration across scales, it appears that the "Co-LLM-7B + LLAMA-70B" method does not outperform the "LLAMA-70B+7B (PT)" method across all tasks (except for the List task). Additionally, the collaborative approach does not seem to perform better than the standalone LLAMA-70B (QLoRA). （I'll highlight the highest value in red）
Given these observations, it seems that your method does not outperform the PT approach or a single larger language model in terms of the metrics. Could you please explain how your method demonstrates its effectiveness?
I look forward to your response and wonder if others have had similar thoughts. I hope to further communicate and learn from you.

experimental equipment

package version conflict

Hi, when I install packages via "pip install -r requirements.txt", it gives the following error. It seems caused by package dependency conflicts. When I try to install these packages one by one, it still suffers from the version conflicts. I notice that some packages lack the specific version, such as datasets, huggingface-hub. Could you give the specific version of all packages rather than >= or no version? Thank you so much!

Collecting git+https://github.com/yizhongw/transformers.git@left_padding (from -r requirements_new.txt (line 14))
  Cloning https://github.com/yizhongw/transformers.git (to revision left_padding) to /tmp/pip-req-build-6epb5ow_
  Running command git clone --filter=blob:none --quiet https://github.com/yizhongw/transformers.git /tmp/pip-req-build-6epb5ow_
  Running command git checkout -b left_padding --track origin/left_padding
  Switched to a new branch 'left_padding'
  Branch 'left_padding' set up to track remote branch 'left_padding' from 'origin'.
  Resolved https://github.com/yizhongw/transformers.git to commit fb2b3302c67915c40a2b4fb6711b7b5c5a8bb2a8
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: torch in /home/ubuntu/anaconda3/envs/co-llm-v2/lib/python3.11/site-packages (from -r requirements_new.txt (line 1)) (2.1.0)
Collecting sentencepiece (from -r requirements_new.txt (line 2))
  Using cached sentencepiece-0.2.0.tar.gz (2.6 MB)
  Preparing metadata (setup.py) ... done
Collecting datasets (from -r requirements_new.txt (line 3))
  Using cached datasets-2.19.2.tar.gz (2.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting deepspeed>=0.10.0 (from -r requirements_new.txt (line 4))
  Using cached deepspeed-0.14.2-py3-none-any.whl
Collecting accelerate<0.23.0,>=0.21.0 (from -r requirements_new.txt (line 5))
  Using cached accelerate-0.22.0.tar.gz (241 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting peft>=0.4.0 (from -r requirements_new.txt (line 6))
  Using cached peft-0.11.1.tar.gz (262 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting bitsandbytes>=0.41.1 (from -r requirements_new.txt (line 7))
  Using cached bitsandbytes-0.42.0.tar.gz (103.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting evaluate>=0.4.0 (from -r requirements_new.txt (line 8))
  Using cached evaluate-0.4.2.tar.gz (65 kB)
  Preparing metadata (setup.py) ... done
Collecting tokenizers>=0.13.3 (from -r requirements_new.txt (line 9))
  Using cached tokenizers-0.19.1.tar.gz (321 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      Collecting maturin<2.0,>=1.0
        Using cached maturin-1.6.0.tar.gz (187 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Building wheels for collected packages: maturin
        Building wheel for maturin (pyproject.toml): started
        Building wheel for maturin (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
      
        × Building wheel for maturin (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [35 lines of output]
            /tmp/pip-build-env-lt9x3nmx/overlay/lib/python3.11/site-packages/setuptools/config/_apply_pyprojecttoml.py:83: SetuptoolsWarning: `install_requires` overwritten in `pyproject.toml` (dependencies)
              corresp(dist, value, root_dir)
            running bdist_wheel
            running build
            running build_py
            creating build
            creating build/lib.linux-x86_64-cpython-311
            creating build/lib.linux-x86_64-cpython-311/maturin
            copying maturin/__main__.py -> build/lib.linux-x86_64-cpython-311/maturin
            copying maturin/import_hook.py -> build/lib.linux-x86_64-cpython-311/maturin
            copying maturin/__init__.py -> build/lib.linux-x86_64-cpython-311/maturin
            running egg_info
            creating maturin.egg-info
            writing maturin.egg-info/PKG-INFO
            writing dependency_links to maturin.egg-info/dependency_links.txt
            writing requirements to maturin.egg-info/requires.txt
            writing top-level names to maturin.egg-info/top_level.txt
            writing manifest file 'maturin.egg-info/SOURCES.txt'
            reading manifest file 'maturin.egg-info/SOURCES.txt'
            reading manifest template 'MANIFEST.in'
            warning: no files found matching '*.json' under directory 'src/python_interpreter'
            writing manifest file 'maturin.egg-info/SOURCES.txt'
            running build_ext
            running build_rust
            error: can't find Rust compiler
      
            If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
      
            To update pip, run:
      
                pip install --upgrade pip
      
            and then retry package installation.
      
            If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for maturin
      Failed to build maturin
      ERROR: Could not build wheels for maturin, which is required to install pyproject.toml-based projects
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.