Giter Club home page Giter Club logo

rexit's Introduction

Rexit

Rexit - Liberate your Reddit Chats. This tool will export your Reddit chats into a plethora of formats

version license GitHub code size in bytes .github/workflows/cd.yml

Tool to export Reddit chats into a variety of open formats (CSV, JSON, TXT).

Export your Reddit Chats

Usage: rexit.exe [OPTIONS] --formats <FORMATS>

Options:
  -f, --formats <FORMATS>  The formats to export to. Options: csv,json,txt
  -t, --token              To use the bearer token flow, instead of username and password
      --debug              Allow debugging of Rexit
  -i, --images             Output images too (outputs to images folder)
  -o, --out <OUT>          What folder to output to [default: ./out]
  -h, --help               Print help
  -V, --version            Print version
      --no-usernames       Not Retrieve usernames (Is a lot faster)

Usage

Currently, you need to specify the formats, and it will ask for the username and password (or bearer token with that auth flow).

To get messages:

$ rexit messages --images
> Your Reddit Username: <USERNAME>
> Your Reddit Password: <PASSWORD>

To get saved posts:

$ rexit saved --images
> Your Reddit Username: <USERNAME>
> Your Reddit Password: <PASSWORD>

To download a Subreddit:

$ rexit subreddit r/redditDev --images
> Your Reddit Username: <USERNAME>
> Your Reddit Password: <PASSWORD>

It will save the files to the current directory. For CSV and TXT it is split by room. If an image (.jpg, .gif, .png, etc.) was sent the filename will be displayed as the message content, along with the prefix FILE.

Installation

You can use the files provided in the releases' page of this repository, install via cargo or brew or build from source.

Manual Install

  1. Download the build for your system (Windows or arm64-darwin)
  2. Use the terminal run Rexit with the arguments you want. (See Usage for details)

Cargo Install

$ cargo install rexit

Brew Install

To use brew you need to add my tap

$ brew tap mpult/mpult

Then install Rexit

$ brew install rexit

Building from source

  1. Install rust
  2. Clone the repository
  3. Run:
$ cargo install --path .

Contributing

To keep the docs focused on the user experience the contributing and technical docs were implemented through cargo doc.

To access these:

$ cargo doc --open

Common Errors

Q: Some images are not downloading, the console says Image was already downloaded; Skipping``.

A: Delete the imageLog.txt file from the out folder.


In general all contributions are welcome. I would appreciate if you'd create an issue beforehand, in order for me to plan things out nicely.

Note To Reddit: Care was taken to ensure as little API requests are made as possible.

  • Username requests are cached locally, and an option is provided to not gather the usernames
  • A log of already downloaded images is kept to prevent downloading images multiple times

License

GNU General Public License, Version 3

rexit's People

Contributors

codyps avatar dependabot[bot] avatar github-actions[bot] avatar mpult avatar nifoc avatar oneelectron avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

codyps nifoc

rexit's Issues

Add next batch feature

Currently only the first batch of sync data is analysied, it needs to be determined if relavent data is present in the next batches. if yes then that timeline is appended to the initial sync data (maybe save to temp file), same goes for the state > events attribute

MFA/2FA Support

Motivations

Please add support for accounts with MFA/2FA enabled.

  • Would you like to implement this feature? [y/n]
    I'm not sure I'd be able to (N).

Solution

Support for accounts with MFA/2FA enabled. Ideally, the CLI should ask for the 2FA code.

Alternatives

Using token based auth, but I was unable to figure out how this works.

Correlate messages and images

As far as I can tell, there currently is no way to correlate messages and downloaded images.

The message often just says "Image" or includes a filename that is different from the download image.

Would be nice to be able to view the downloaded images "in context".

This seems to be true for all export formats.

"No such file or directory" on image download attempt

Bug description

Hello again, @MPult ! Looks like we got ourselves another bug here ๐Ÿž
Running with image downloads, getting fatal error like this:

Rexit git:(main) /Users/***/.cargo/bin/rexit messages --images --redact --formats csv,json,txt

2024-02-07 17:49 - INFO: Getting image: [REDACTED]
thread 'main' panicked at src/ReAPI/images.rs:66:68:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0:        0x1005b5a60 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb1669d376cd16fb9
   1:        0x1005c3f30 - core::fmt::write::hfb70cbdb2260ac51
   2:        0x1005aaf30 - std::io::Write::write_fmt::h70ee144642a582c5
   3:        0x1005b589c - std::sys_common::backtrace::print::h404168d6289bebbe
   4:        0x1005a1c90 - std::panicking::default_hook::{{closure}}::hcad9841f2663dad9
   5:        0x1005a19e8 - std::panicking::default_hook::he7ca7c46dc216a28
   6:        0x1005a2154 - std::panicking::rust_panic_with_hook::h501076dcf33dacb0
   7:        0x1005b5f28 - std::panicking::begin_panic_handler::{{closure}}::h0665b0808da4d87c
   8:        0x1005b5c68 - std::sys_common::backtrace::__rust_end_short_backtrace::h6d7b13146df55e47
   9:        0x1005a1eec - _rust_begin_unwind
  10:        0x1005e8408 - core::panicking::panic_fmt::h5de4b603c189570c
  11:        0x1005e8750 - core::result::unwrap_failed::h6c2edae44e6d47ca
  12:        0x100289810 - rexit::ReAPI::images::get_image::{{closure}}::h71ed81371e6bd8e5
  13:        0x10028d634 - rexit::ReAPI::rooms::download_rooms::{{closure}}::hc29a397cde86e3d3
  14:        0x100290164 - rexit::main::{{closure}}::h5ce55bb2c365ebb1
  15:        0x100287928 - tokio::runtime::park::CachedParkThread::block_on::hf8573b3dd26c461a
  16:        0x1002ecfa8 - tokio::runtime::context::runtime::enter_runtime::he2113a63a4b0b715
  17:        0x1002d6ac8 - tokio::runtime::runtime::Runtime::block_on::h51c10d5419defe39
  18:        0x1002cbd94 - rexit::main::h22180b121616fe83
  19:        0x1002e8068 - std::sys_common::backtrace::__rust_begin_short_backtrace::h38bfbb5c4b2d170f
  20:        0x1002e56a4 - std::rt::lang_start::{{closure}}::h675f2af4ce139a66
  21:        0x1005a1ddc - std::panicking::try::h8fbeda15452e3045
  22:        0x100598700 - std::rt::lang_start_internal::h74e1b60c5b4153f8
  23:        0x1002cbe94 - _main

  • Would you like to work on a fix? [y/n]

To Reproduce

Steps to reproduce the behavior:

  1. ...
  2. ...
  3. ...
  4. ...

this is on the main branch, with RUST_BACKTRACE=full

Expected behavior

Should finish normally

Screenshots

Environment

  • OS: MacOS Sonoma 14.3
  • {{project-name}} version: [e.g. 0.1.0]

Additional context

Would greatly appreciate a fix!

Add Exporting

This is a essential part of this project, thus it has a high priority.

  • Implement method of internal data transfer to export function
  • Implement exporting
    • TXT
    • JSON
    • CSV

Error parsing array in rooms

Bug description

I'm new to the tool and just started using it. I successfully logged in using a bearer token because the username and password seemed to fail.

I do see the out folder after running but it is empty.

Maybe I'm doing something wrong that's obvious.

To Reproduce

Steps to reproduce the behavior:

  1. RUST_BACKTRACE="full" rexit messages -t
  2. entered the bearer token
  3. confirmed the bearer token

Next I tried the saved posts and that seems to fail too.

Backtrace

2024-04-21 04:30 - INFO: Login Successful
thread 'main' panicked at src/ReAPI/rooms.rs:51:10:
Error parsing array
stack backtrace:
0: 0x10538b496 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h39dedd9998475f7a
1: 0x10539a7a3 - core::fmt::write::h996a70950ef02a8b
2: 0x10538d928 - std::io::Write::write_fmt::hd6709b73e8721808
3: 0x10538b27f - std::sys_common::backtrace::print::h0c71c33392329482
4: 0x10537ba25 - std::panicking::default_hook::{{closure}}::h7c2dc6602e4afdfe
5: 0x10537b7ef - std::panicking::default_hook::hb8432d7d4b595a9c
6: 0x10537be68 - std::panicking::rust_panic_with_hook::h55cf7f98a5172bf4
7: 0x10538bad6 - std::panicking::begin_panic_handler::{{closure}}::hfb4dd0b5a9ef6466
8: 0x10538b6f9 - std::sys_common::backtrace::__rust_end_short_backtrace::hcbbcf4c6d355f082
9: 0x10537bbc6 - _rust_begin_unwind
10: 0x1053c1325 - core::panicking::panic_fmt::ha2d9edabf12b6300
11: 0x1053c19ba - core::option::expect_failed::ha1c277500abbfd3f
12: 0x1050072a7 - rexit::main::{{closure}}::hf89d91bb37411eec
13: 0x104fff366 - tokio::runtime::park::CachedParkThread::block_on::h446c42231b05c1eb
14: 0x10504418d - tokio::runtime::context::runtime::enter_runtime::h9d9d4d925d5f4bd5
15: 0x105068c50 - tokio::runtime::runtime::Runtime::block_on::h80285ab1ffaa2f2f
16: 0x105051ded - rexit::main::h191a9d02c00e6598
17: 0x1050641f6 - std::sys_common::backtrace::__rust_begin_short_backtrace::h712d1604905064ce
18: 0x10506a2bc - std::rt::lang_start::{{closure}}::h07c8b02b45aa39de
19: 0x10537bac4 - std::panicking::try::h3e54f477477c1d0e
20: 0x1053656ce - std::rt::lang_start_internal::h5ac0ebe39c2c25d0
21: 0x105051ecc - _main

Screenshots

Environment

  • OS: macOS 10.15.3
  • rexit version: 1.2.0

Additional context

Please let me know if you need me to provide any further context

Download Images

Add a CLI flag that will download the images.

Currently the mxc:// url is displayed as a message, but since this is a cumbersome thing to do Rexit should provide a function for this.

GET /_matrix/media/v3/download/{serverName}/{mediaId} is the URL SPEC

Polish main functionality

There are a few tidbits that need to be polished before it can be accepted as a release.

  • #5
  • Replace user ID with display name
  • Set default log level as INFO
  • Replace Image placeholder with link to image
  • Refactor
  • Add username & password login option

Automated test

tests should use password flow and compare export to a known good

libssl.so.3: cannot open shared object file

I downloaded the latest version of the linux build and ran it. I get

./rexit: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

I haven't done a lot of Rust stuff so I'm not sure where to continue.

Here's my system information

image

Login Failed

Login Failed

This might be user error, but I only get "Login Failed". I have changed my password to something short and easy to enter since I can't see what I am typing into cmd prompt. I'm sure I'm typing it correctly, but it still fails. Any advice?

  • Would you like to work on a fix? Not able.

To Reproduce

Steps to reproduce the behavior:

  1. ...
  2. ...
  3. ...
  4. ...

Expected behavior

Screenshots

Environment

Windows 10

Additional context

Improve docs

  • Readme should only include information relevant for USER
  • Secondary doc should explain the functioning of rexit with many links to the matrix spec. also info on developing etc.
  • readme should show output formats

Error - Missing field mimetype

C:\> rexit messages
> Your Reddit Username MyUserName
> Your Reddit Password ********
2023-09-16 12:03 - INFO: Login Successful
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("missing field `mimetype`", line: 1, column: 9469)', C:\Users\user.op\.cargo\registry\src\index.crates.io-6f17d22bba15001f\rexit-1.2.0\src\ReAPI\messages.rs:75:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Version: 1.2.0

Improvement Suggestions

Sorry, @MPult , the template of feature suggestions did not make sense to me, so starting from a blank.

As someone with A LOT of messages and images, may I suggest the following:

  1. It would be great to organize messages and images into subdirectories under /out with names corresponding to user names, such as /out/user_1 , /out/user_2 etc.
  2. The sorting of messages is not intuitive as you'd expect the oldest to be on top. Yes, they can be resorted by timestamp, but not the txt format
  3. Please add HTML format with links to images in the local folder (same folder as html file is suggestion No. 1 from this list is implemented

Thank you so much for all your hard work, @MPult , @oneElectron and whoever else has contributed!

Logging

Users should be let known what is happending using info messages. EG login suceeded etc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.