i make. i break. computer witchcraft at asterisk.so.
reach out:
mufeed [at] asterisk [dot] so
pgp fingerprint:
49B7 4F49 C33A 02A9 7536 257F 45BE E76A 9562 CB5E
A CLI tool to convert your codebase into a single LLM prompt with source tree, prompt templating, and token counting.
License: MIT License
i make. i break. computer witchcraft at asterisk.so.
reach out:
mufeed [at] asterisk [dot] so
pgp fingerprint:
49B7 4F49 C33A 02A9 7536 257F 45BE E76A 9562 CB5E
Is it possible to add an argument so that any excluded files/folders/extensions are also excluded from the source tree?
I am currently working with some folders that have thousands of .npy files, or .wav, etc, and this means the source tree can get a little bit heavy!
Thanks for this great tool!
Hi, would it be possible to release a new version? Seems to have implemented quite a few features since 1.10. Looking at updating the package in nixpkgs, and this would be appreciated.
Hi, mufeedvh. Thank you for a very nice application.
I'm running into a problem right now when dealing with large code repositories. When I'm dealing with small code repositories, code2prompt works great. But when I'm dealing with large repositories, I have a token overflow problem when interacting with LLM.
So how should we deal with large code repositories? Sending only part of the source code will affect the context. Now it seems that only Gemini 1.5pro can handle about 200m tokens, which is the upper limit.
Can you perform tuning on a large code repository? Or do you have any good suggestions?
The --exclude-files and --exclude-folders flags, as documented in the code2prompt GitHub documentation, do not seem to function when used in the command-line interface (CLI). Attempting to use these flags results in an "unexpected argument" error, suggesting a possible discrepancy between the documentation and the tool's current CLI implementation.
Steps to Reproduce:
Attempt to use code2prompt with the --exclude-files flag:
This results in an error: error: unexpected argument '--exclude-files' found.
Attempt to use code2prompt with the --exclude-folders flag:
Similarly, this results in an error: error: unexpected argument '--exclude-folders' found.
Expected Behavior:
According to the documentation, users should be able to exclude specific files and folders from the prompt generation process by using the --exclude-files and --exclude-folders flags. The expected behavior is for code2prompt to accept these flags without error and to exclude the specified files and folders accordingly.
Actual Behavior:
The tool does not recognize the --exclude-files and --exclude-folders flags, resulting in an "unexpected argument" error message. This discrepancy between the documentation and the tool's behavior suggests a bug in the CLI implementation of these features.
Environment:
code2prompt version: 1.0.0
Operating System: (e.g., Ubuntu 20.04)
Shell: (e.g., bash, zsh)
this occurs in some setups when using SSH. Instead of panicking, it should still write the output to a file.
Thank you for a very nice application.
I am a beginner in programming, so I especially use "clean-up-code.hbs" and "document-the-code.hbs". Thanks to you, I am enjoying a very comfortable development experience.
Now to the main topic.
When '-json' is specified, json is output to the terminal.
What I am hoping for is that it will also be copied to the clipboard.
If I don't specify -json, I expect the same behaviour as if it were saved to the clipboard as text.
Am I wrong in my thinking? Ideas please.
▹▹▹▹▸ Done! [✓] Prompt copied to clipboard!
However, nothing had been copied to the clipboard. Has this been tested on a linux machine?
Basically, I am a Mac developer. However, I sometimes develop on Ubuntu with Windows WSL2 due to circumstances.
I can use Mac without any problem, but in WSL2 environment, the following commands do not work as expected.
1.code2prompt /home/USER/PJ --exclude="**/package-lock.json"
The purpose is to exclude package-lock.json existing in PJ/server, PJ/client, etc.
However, the output result includes the contents of package-lock.json.
Is there a way to properly use --exclude
?
code2prompt /home/USER/PJ
I am expecting the output results to be saved to the clipboard as a result of running this command, but sometimes they are not.
So, I use the following command every time.
code2prompt /home/USER/PJ --ouput /home/USER/PJ/output.txt
.
It accomplishes the purpose, but it is better to be saved to the clipboard.
Can you give me some advice on how to solve this problem?
P.S.
Very nice app, can't develop anymore without it! I look forward to more in the future.
Translated with DeepL.com (free version)
Hello,
thanks for creating this!
It looks super interesting.
With Gemini having 1 million tokens available. And the possibility to get 2 million soon, it will be amazing to see the amount of tokens for them.
https://ai.google.dev/gemini-api/docs/get-started/tutorial?lang=web#count-tokens
Thanks in advance for considering!
Greetings from Franconia (Germany)
Could a flag be added to skip copying the output to clipboard so code2prompt can be used on a remote without a clipboard.
@mufeedvh This is an awesome tool!
Feature request:
A --relative-paths
flag truncates paths to the directory passed as an argument so that parent directories are not included in the file paths of the output.
Today if I run code2prompt projectdir
when files are listed in the prompt payload the paths are displayed as $HOME/code/projectdir/somefile.txt
instead of /projectdir/somefile.txt
. Stripping the parent path saves tokens and keep details of my filesystem and username private.
.gitignore
contains /node_modules
, .env
, etc.
I want to exclude /node_modules
as is, but include .env
in the output.
For example, to ensure that Claude and ChatGPT understand when .env
is present, as shown below.
My rule of thumb is that strictly directing the location of files when interacting with AI makes development go very smoothly.
So, you need to read .env
only when running code2prompt
.
Is there a good way to do this?
Hello,
first of all, a cool project!
Larger codebases often significantly exceed even the largest context windows available these days, while offline LLMs are even more troublesome in this regard.
It could be useful to implement an indexing feature, that would not generate a single prompt from the codebase, but instead output multiple smaller prompts containing max. N tokens, with the purpose of creating some kind of code abstraction. This abstract could be afterwards used together with just a single file of code where modifications should be made.
I don't use LLMs for coding very frequently, but this seems like the only plausible approach for fitting large codebases into LLLMs. Have you made any considerations/experiments with this approach and possible implementation into code2prompt?
ahonnecke@antonym:~/src/wayward/src/wayward$ $ code2prompt . -o output.txt
$: command not found
Hey @mufeedvh,
Thanks for involving yourself into this nice project !
Could you please consider providing precompiled binaries through GitHub Actions ?
Cheers
I encountered a panic when running code2prompt
with the following command:
RUST_BACKTRACE=full code2prompt .
RUST_BACKTRACE=full code2prompt .
▹▹▹▹▸ Done! 2024-03-18 19:38:56.031 code2prompt[56653:3892444] -[NSPasteboard _setData:forType:index:usesPboardTypes:] returns false. Type: public.utf8-plain-text, index: 0, object class: __NSCFString.
thread 'main' panicked at /Users/slava/.cargo/registry/src/index.crates.io-6f17d22bba15001f/code2prompt-1.0.0/src/main.rs:142:50:
Failed to copy output to clipboard: "NSPasteboard#writeObjects: returned false"
stack backtrace:
0: 0x10ec08fed - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbd71adb7a72f4105
1: 0x10ea96323 - core::fmt::write::h30346430340bc336
2: 0x10ec06962 - std::io::Write::write_fmt::heb3d6316c565d5b1
3: 0x10ec08d79 - std::sys_common::backtrace::print::h67e51ff2e3d5cfbd
4: 0x10ec0a0a5 - std::panicking::default_hook::{{closure}}::hc666c9a55318d1f1
5: 0x10ec09e0c - std::panicking::default_hook::hf980b1da49948523
6: 0x10ec0a73f - std::panicking::rust_panic_with_hook::h683bce980186bbbe
7: 0x10ec0a4d4 - std::panicking::begin_panic_handler::{{closure}}::ha6dbd11ba0ec8af1
8: 0x10ec094e9 - std::sys_common::backtrace::__rust_end_short_backtrace::h889430bddc786c98
9: 0x10ec0a232 - _rust_begin_unwind
10: 0x10ec287c5 - core::panicking::panic_fmt::hff768cef35397791
11: 0x10ec28ca5 - core::result::unwrap_failed::h951d84d71b0bada2
12: 0x10ea8804a - code2prompt::main::h2d635b437be76536
13: 0x10ea75306 - std::sys_common::backtrace::__rust_begin_short_backtrace::hddd69f0893274fa7
14: 0x10ea7544c - std::rt::lang_start::{{closure}}::h0f70ef5c3efad0b4
15: 0x10ec000cd - std::rt::lang_start_internal::hba631f1493ca29ca
16: 0x10ea8a82c - _main
[1] 56653 abort RUST_BACKTRACE=full code2prompt
RUST_BACKTRACE=full code2prompt .
code2prompt
Version: v1.0.0The command should execute without panicking.
The command panics with the error message: "Failed to copy output to clipboard: "NSPasteboard#writeObjects: returned false""
I want to exclude files and directories listed in .gitignore
node_modules/
venv/
I've specified them in .gitignore as above, but code2prompt reads them without excluding them.
Should I specify them directly using --exclude
?
In that case, should I specify --exclude /**/node_modules/, /**/venv/
?
I look forward to your reply.
Can I just add specific files like code2prompt add file1,file2
? This feature would be great.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.