Giter Club home page Giter Club logo

rust-g's Introduction

rust-g

rust-g (pronounced rusty-g) is a library which offloads certain expensive or difficult tasks from BYOND.

This library is currently used in the /tg/station codebase, and is required for it to run. A pre-compiled DLL version can be found in the repo root of codebases that use it, but you can build your own from this repo (and you should if you're running a server).

Builds can also be found on the releases page but should only be used for Windows, as Linux has compatibility issues across distributions.

Dependencies

The Rust compiler:

  1. Install the Rust compiler's dependencies (primarily the system linker):

  2. Use the Rust installer, or another Rust installation method, or run the following:

    curl https://sh.rustup.rs -sSfo rustup-init.sh
    chmod +x rustup-init.sh
    ./rustup-init.sh
  3. Set the default compiler to 32-bit:

    # Clone the `rust-g` repository to a directory of your choice
    git clone https://github.com/tgstation/rust-g.git
    # in the `rust-g` directory...
    cd rust-g
    # Linux
    rustup target add i686-unknown-linux-gnu
    # Windows
    rustup target add i686-pc-windows-msvc

System libraries:

  • Ubuntu and Debian users run:

    sudo dpkg --add-architecture i386
    sudo apt-get update
    sudo apt-get install zlib1g-dev:i386
  • Other Linux distributions install the appropriate 32-bit development and 32-bit runtime packages.

If you want to use the pc-windows-gnu or similar other target ABI, do the following:

  1. Change the "rust-analyzer.cargo.target" setting in .cargo/config to i686-pc-windows-gnu.
  2. Run git update-index --assume-unchanged .cargo/config, which will tell git to 'ignore' the changes you made.
  3. If you find yourself ever wanting to change back, run git update-index --no-assume-unchanged .cargo/config.

Compiling

The Cargo tool handles compilation, as well as automatically downloading and compiling all Rust dependencies. The default configuration is suitable for use with the /tg/station codebase. To compile in release mode (recommended for speed):

Linux:

export PKG_CONFIG_ALLOW_CROSS=1
cargo build --release --target i686-unknown-linux-gnu
# output: target/i686-unknown-linux-gnu/release/librust_g.so

Windows:

cargo build --release --target i686-pc-windows-msvc
# output: target/i686-pc-windows-msvc/release/rust_g.dll

If you aren't sharing the binary with other people, consider compiling targeting your native cpu for potential performance improvements. You can do this by setting the RUSTFLAGS environment variable to -C target-cpu=native. For example, in Powershell you would use $Env:RUSTFLAGS="-C target-cpu=native".

To get additional features, pass a list to --features, for example --features hash,url. To get all features, pass --features all. To disable the default features, pass --no-default-features. You can't use --all-features because of conflicting native_tls and rustls_tls features to select the mysql backend.

The default features are:

  • acreplace: Aho-Corasick string matching and replacement.
  • cellularnoise: Function to generate cellular automata-based noise.
  • dmi: DMI manipulations which are impossible or degraded from within BYOND. Mostly used by the asset cache subsystem to improve load times.
  • file: Faster replacements for file2text and text2file, as well as reading or checking if files exist.
  • git: Functions for robustly checking the current git revision.
  • http: Asynchronous HTTP(s) client supporting most standard methods.
  • json: Function to check JSON validity.
  • log: Faster log output.
  • noise: 2d Perlin noise.
  • sql: Asynchronous MySQL/MariaDB client library.
  • time: High-accuracy time measuring.
  • toml: TOML parser.
  • url: Faster replacements for url_encode and url_decode.

Additional features are:

  • batchnoise: Discrete Batched Perlin-like Noise, fast and multi-threaded - sent over once instead of having to query for every tile.
  • hash: Faster replacement for md5, support for SHA-1, SHA-256, and SHA-512. Requires OpenSSL on Linux.
  • iconforge: A much faster replacement for the spritesheet generation system used by /tg/station.
  • pathfinder: An a* pathfinder used for finding the shortest path in a static node map. Not to be used for a non-static map.
  • redis_pubsub: Library for sending and receiving messages through Redis.
  • redis_reliablequeue: Library for using a reliable queue pattern through Redis.
  • unzip: Function to download a .zip from a URL and unzip it to a directory.
  • worleynoise: Function that generates a type of nice looking cellular noise, more expensive than cellularnoise

Regarding rust-analyzer: If you are using a feature set other than the default, you will need to adjust rust-analyzer.cargo.features.

Installing

The rust-g binary (rust_g.dll or librust_g.so) should be placed in the root of your repository next to your .dmb. There are alternative installation locations, but this one is best supported.

Compiling will also create the file target/rust_g.dm which contains the DM API of the enabled modules. To use rust-g, copy-paste this file into your project.

rust_g.dm can be configured by creating a rust_g.config.dm. See the comments at the top of rust_g.dm for details.

Troubleshooting

You must build a 32-bit version of the library for it to be compatible with BYOND. Attempting to build a 64-bit version will fail with an explanatory error.

Linux

On Linux systems ldd can be used to check that the relevant runtime libraries are installed, without which BYOND will fail to load rust-g. The following is sample output, but the most important thing is that nothing is listed as "missing".

$ ldd librust_g.so  # Linux
    linux-gate.so.1 (0xf7f8b000)
    libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf7957000)
    libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7935000)
    libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf7831000)
    libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf782b000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7643000)
    /lib/ld-linux.so.2 (0xf7f8d000)

If BYOND cannot find the shared library, ensure that the directory containing it is included in the LD_LIBRARY_PATH environment variable, or tweak the search logic in rust_g.dm:

$ export LD_LIBRARY_PATH=/path/to/tgstation

To examine what locations BYOND is searching for the shared library, use strace:

$ strace DreamDaemon tgstation.dmb 45000 -trusted -logself 2>&1 | grep 'rust_g'
# Early in output, the file will be listed when BYOND examines every file it can see:
open("rust_g", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOTDIR (Not a directory)
# BYOND will then search some common directories...
stat64("/home/game/.byond/bin/rust_g", 0xffef1110) = -1 ENOENT (No such file or directory)
stat64("/home/game/.byond/bin/rust_g", 0xffef1190) = -1 ENOENT (No such file or directory)
# Then anywhere in LD_LIBRARY_PATH...
open("/home/game/work/ss13/byond/bin/rust_g", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
# Then in several interesting places where ld-linux looks...
open("tls/i686/sse2/cmov/rust_g", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    ... snip ...
open("cmov/rust_g", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
# Until finding the library fails or succeeds (a value other than -1 indicates success):
open("rust_g", O_RDONLY|O_CLOEXEC)      = 4
# After that it goes back to the scanning from startup.
open("rust_g", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOTDIR (Not a directory)

If you're still having problems, ask in the Coderbus Discord's #tooling-questions channel.

License

This project is licensed under the MIT license.

See LICENSE for more details.

rust-g's People

Contributors

4dplanner avatar absolucy avatar actioninja avatar adamsong avatar affectedarc07 avatar anturk avatar bravemole avatar cadyn avatar crossedfall avatar cyberboss avatar edgelordexe avatar itsmeow avatar marksuckerberg avatar mchsl avatar mothblocks avatar neersighted avatar optimumtact avatar pali6 avatar pjb3005 avatar ralms avatar san7890 avatar shadowlarkens avatar silicons avatar sindorman avatar spacemaniac avatar tralezab avatar vuonojenmustaturska avatar vvvv-vvvv avatar zephyrtfa avatar zewaka avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rust-g's Issues

Clippy fails on windows build

{"reason":"compiler-message","package_id":"rust-g 0.5.0 (path+file:///D:/a/rust-g/rust-g)","manifest_path":"D:\a\rust-g\rust-g\Cargo.toml","target":{"kind":["cdylib"],"crate_types":["cdylib"],"name":"rust-g","src_path":"D:\a\rust-g\rust-g\src\lib.rs","edition":"2018","doc":true,"doctest":false,"test":true},"message":{"rendered":"error: statics have by default a 'static lifetime\n --> src\redis_pubsub.rs:7:24\n |\n7 | static ERROR_CHANNEL: &'static str = "RUSTG_REDIS_ERROR_CHANNEL";\n | -^^^^^^^---- help: consider removing 'static: &str\n |\n = note: -D clippy::redundant-static-lifetimes implied by -D warnings\n = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_static_lifetimes\n\n","children":[{"children":[],"code":null,"level":"note","message":"`-D clippy::redundant-static-lifetimesimplied by-D warnings","rendered":null,"spans":[]},{"children":[],"code":null,"level":"help","message":"for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_static_lifetimes","rendered":null,"spans":[]},{"children":[],"code":null,"level":"help","message":"consider removing 'static","rendered":null,"spans":[{"byte_end":181,"byte_start":169,"column_end":35,"column_start":23,"expansion":null,"file_name":"src\\redis_pubsub.rs","is_primary":true,"label":null,"line_end":7,"line_start":7,"suggested_replacement":"&str","suggestion_applicability":"MachineApplicable","text":[{"highlight_end":35,"highlight_start":23,"text":"static ERROR_CHANNEL: &'static str = \"RUSTG_REDIS_ERROR_CHANNEL\";"}]}]}],"code":{"code":"clippy::redundant_static_lifetimes","explanation":null},"level":"error","message":"statics have by default a 'static` lifetime","spans":[{"byte_end":177,"byte_start":170,"column_end":31,"column_start":24,"expansion":null,"file_name":"src\redis_pubsub.rs","is_primary":true,"label":null,"line_end":7,"line_start":7,"suggested_replacement":null,"suggestion_applicability":null,"text":[{"highlight_end":31,"highlight_start":24,"text":"static ERROR_CHANNEL: &'static str = "RUSTG_REDIS_ERROR_CHANNEL";"}]}]}}

Add Redis list operation support

Pub/sub is great for events if you have something listening, but not so great if you care about a historical record of the data if no one happens to be subscribed at the time. For such a use case, lists are better suited combined with a worker-style implementation via BRPOP.

Would be neat to have the ability to push to, and pop off, lists.

(LPUSH and LRANGE)

Move this repository to spacestation13 org

Given the wider use of this library by the ss13 community I thought it would be a nice thing to move the repo to spacestation13 where it can better represent it's shared nature.

It's much easier these days with a button to migrate and github maintaing redirections so it's much less likely to break CI across the world.

@tgstation/rust-g-dev-team thoughts?

rustg_read_toml_file is broken

As far as I can tell, #98 broke rustg_read_toml_file. We are using it in a tgstation branch, and when I happened to update rust_g, everything that depended on toml files broke.

the toml_file_to_json function in the library passes a json string containing "success" and "content" to byond, and the byond wrapper proc returns the value of "content" if it succeeds. From what I can tell, the value of content is escaped so that it is not actually parsed by json_decode() during this process, and returned as a json-formatted string (thankfully, with escape characters removed) instead of an associative array.

My current workaround is to json_decode the content again before you return it, and that appears to be working.

Test code (in an otherwise empty test environment, with rust_g.dm and a data.toml file to test):

world.New()
        world.log << "Starting"
        var/list/result = rustg_read_toml_file("data.toml")
        world.log << "Read done"
        if(istext(result))
                world.log << "It's just text"
                world.log << result
                var/list/result2 = json_decode(result)
                if(istext(result2))
                        world.log << "still just text"
                else
                        world.log << "converted?!?"

For me, this code returns the incredulous "converted?!?" text. The workaround is also functioning on a live server, providing the correct associative list for the data loaded from file.

Statically link vcredist

We faced issues with a few people where they couldn't use even old copies of rust-g because of obscure "The specific procedure could not be found" errors. Eventually they had to install vcredist AND restart their computer before it was fixed.

We should statically link vcredist to stop errors like these.

You just need this in .cargo/config.toml:

[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "target-feature=+crt-static"]

(Also reminder to fix #86)

text2file override is incorrect

This override in the DM API file is incorrect:

#define text2file(text, fname) rustg_file_write(text, fname)

rustg_file_write overwrites, while text2file appends.

Low priority because /tg/ does not currently use the file module or the RUSTG_OVERRIDE_BUILTINS feature.

rustg cannot create folders with rustg_file_write

Byond's text2file would automatically create the folders needed to make the file. rust_g doesn't seem to want to do this and does not create the folders required to make a successful write operation, meaning that the file is never created in the first place.

Function to download and unzip a .zip file from a given url.

e.g. https://github.com/tgstation/tgstation-server/releases/download/dmapi-v5.2.7/DMAPI.zip (Not in practice but good example)

Signature: rustg_download_and_unzip_async(url, unzip_directory)

Should use the current jobs system.
Should handle and report all errors on return.
On success, should unzip the contents of the .zip pointed to by url into uzip_directory (which may be a relative path), creating it if necessary and overwriting any conflicting contents.

Bounty revoked, I need to make myself learn rust.

Redis pubsub disconnects in V3.1.0

This now occurs when using 3.2.0, whereas in 3.0.0 it didn't

image-23-1.png

Reportedly occurs during high load procs such as round setup before start.

rustg_noise_get_at_coordinates returns a blank string regardless of inputs

Code:

/client/verb/noise_test()
    var/test = rustg_noise_get_at_coordinates(rand(1,1000),128,128)
    to_chat("The result is: [test]. Is it null? [isnull(test)]")
    var/test2 = md5("cock and ball torture")
    to_chat("The md5 is: [test2].")

Proof that I installed it right:

The result is: . Is it null? 0
The md5 is: 35f953af0a9f12c40b2da38f9b9df658.

Cargo install string used:

cargo build --release --features log,dmi,git,http,sql,noise,url,file,hash

hash module is now failing lint

Rust added a new clippy lint or something and it fails now:

 error: use of a fallible conversion when an infallible one could be used
   --> src\hash.rs:105:48
    |
105 |         let result = totp_generate(hex_seed, i.try_into().unwrap(), time_override)?;
    |                                                ^^^^^^^^^^^^^^^^^^^ help: use: `into()`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_fallible_conversions
    = note: `-D clippy::unnecessary-fallible-conversions` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::unnecessary_fallible_conversions)]`

error: could not compile `rust-g` (lib) due to previous error

Just switching to into() doesn't work as stated, since there is not a From conversion.

Clippy unhappy with compiling 3.1.0 on Windows

I am trying to update our (Aurorastation) rust_g version from 2.1.0 (+ some customized additions) to 3.10 (keeping the customized additions we have)

It seems on windows, the compile for the windows platform fails with:

error: initializer for `thread_local` value can be made `const`
  --> src\pathfinder.rs:11:48
   |
11 | static NODES: RefCell<Vec<Option<Rc<Node>>>> = RefCell::new(Vec::new());
   |                                                ^^^^^^^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(Vec::new()) }`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
   = note: `-D clippy::thread-local-initializer-can-be-made-const` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(clippy::thread_local_initializer_can_be_made_const)]`
error: initializer for `thread_local` value can be made `const`
  --> src\redis_pubsub.rs:10:76
   |
10 |     static REQUEST_SENDER: RefCell<Option<flume::Sender<PubSubRequest>>> = RefCell::new(None);
   |                                                                            ^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(None) }`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
error: initializer for `thread_local` value can be made `const`
  --> src\redis_pubsub.rs:11:82
   |
11 |     static RESPONSE_RECEIVER: RefCell<Option<flume::Receiver<PubSubResponse>>> = RefCell::new(None);
   |                                                                                  ^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(None) }`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
error: initializer for `thread_local` value can be made `const`
 --> src\redis_reliablequeue.rs:7:52
  |
7 |     static REDIS_CLIENT: RefCell<Option<Client>> = RefCell::new(None);
  |                                                    ^^^^^^^^^^^^^^^^^^ help: replace with: `const { RefCell::new(None) }`
  |
  = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#thread_local_initializer_can_be_made_const
error: file opened with `create`, but `truncate` behavior not defined
  --> src\unzip.rs:50:14
   |
50 |             .create(true)
   |              ^^^^^^^^^^^^- help: add: `.truncate(true)`
   |
   = help: if you intend to overwrite an existing file entirely, call `.truncate(true)`
   = help: if you instead know that you may want to keep some parts of the old file, call `.truncate(false)`
   = help: alternatively, use `.append(true)` to append to the file instead of overwriting it
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_open_options
   = note: `-D clippy::suspicious-open-options` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(clippy::suspicious_open_options)]`

I have also ported over the change made in #165 but without luck; I do not know much if anything about rust, but seeing the same error also happening in #160 it seems Clippy doesn't like some core rust_g code, hence opening this bug report

If I am wrong on the above assumption, please, do feel free to close this and let me know that I did something wrong on my side

Separate debug info from .so file

After #119, the release build of librust_g.so is 70 MB as opposed to the previous 10 MB.

This is extremely unideal. There are two ways I see to solve this.

One is that we make release w/ debug info a separate target, then ship both librust_g.so and librust_g_debug_info.so.

Another is that there might be a way to split it into its own file much like the .pdb. There's a few StackOverflow posts about this, something about objcopy? Not sure how doable it is but that would be ideal if it's feasible

dmi_strip_metadata leads to removing alpha channel from the icon

dmi_strip_metadata leads to removing alpha channel from the icon
Here is the code I used:

/world
	fps = 25
	icon_size = 32
	view = 6

/mob/Login()
 	var/icon/test = icon('twitcher.dmi', "preview")
	fcopy(test, "temp.png")
	client << browse_rsc(icon("temp.png"), "first.png")
	call("rust_g", "dmi_strip_metadata")("temp.png")
	сlient << browse_rsc(icon("temp.png"), "second.png")

twitcher.dmi: https://cdn.discordapp.com/attachments/693217263934898190/1111983453760651324/twitcher.dmi

Code result:
first.png:
first
second.png:
second

1.0.0

I know it doesn't mean much in a context like this, but rust-g has been battle tested in many major servers and likely will not receive many backwards incompatible changes for a bit. Would be nice to move to 1.0.0 to better communicate what is happening in the updates.

fexists() implementation

fexists("path") on Linux return 0 for symlinked files.

why

Who knows but it'd be useful for some /tg/ prod shared config stuff.

Can we get a rustg_fexists(path) call please?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.