Giter Club home page Giter Club logo

realistikgdps's Introduction

RealistikGDPS

The Python-based backend for RealistikGDPS, made as an expandable solution for a GDPS of any size.

For support and a public running instance, please visit our Discord!

What is this?

This is a modern Python implementation of the Geometry Dash server protocol meant to power my Geometry Dash Private server. It is written in asynchronous, modern Python and is meant as a replacement for our current PHP based infrastructure.

Interesting Features

  • Fully Dockerised, allowing for easy setup
  • MeiliSearch, allowing for typo tolerance
  • S3 support, allowing for flexible storage solutions
  • Proper ratelimiting
  • Logz.io logging support
  • Flexible command framework

How to set up?

  • Ensure Docker and Make are installed on your system.
  • Create a copy of the .env.example file named .env and adjust it to your liking.
  • Run make build to build the RealistikGDPS image.
  • Run make run to run everything.

TODO: Non-docker setup instructions.

Configuration

As previously mentioned, RealistikGDPS currently supports two ways of configuring the server. In both cases the configuration is done through the .env file.

  • Create a copy of .env.example named .env This creates a copy of the default config with all the field names ready for editing.

  • Edit the .env file to your liking Due to the all-in-one nature of Docker, the defaults are unlikely to require any changing with the exception SRV_NAME, representing the name of your GDPS.

Upgrading an existing server

Assuming your server is based off Cvolton's server implementation, there exists a migration utility for this built right into the codebase.

This is done by importing your old database into a database named old_gdps and running make converter (assuming you are using Docker). This will fill all empty tables with data from the old table.

realistikgdps's People

Contributors

7ez avatar cmyui avatar lenforiee avatar realistikdash avatar tsunyoku avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

cmyui 7ez

realistikgdps's Issues

Implement a filter for unwanted user content.

This issue suggests an implementation of a text filter that removes/blocks unwanted content in user text content (such as messages, comments, level descriptions, etc). Currently, user input of this nature is only verified on the surface level (such as length, character-set). This issue proposes analysing its content (such as censoring or blocking disturbing words and/or phrases).

This is meant to prevent undesirable text content from appearing on servers to create a better atmosphere for regular players.

Level model does not contain the level update timestamp.

As the title specifies, neither the level model nor the database table contain the timestamp at which the level was updated, which is crucial level metadata.

This fix should be reflected in both the database (using the migrations system) and Level model, alongside the level repository. Additionally, the level update and upload usecases should handle its updating the the current timestamp.

Validation of level and save data

Currently, a user is able to upload arbitrary data and have it uploaded and stored on the server. Additionally, in the case of levels, this arbitrary data is served directly to clients.

The solution to this would be to do some surface level validation on data received from the client (possibly on the Pydantic/FastAPI level). While these files may potentially be large (upwards of 200MB for some large save files), the usage of forms by Geometry Dash forces them to already be fully loaded into memory. The checks also don't have to be entirely thorough.

These proposed checks would only be present on 2 endpoints, both with a really low rate limit.

Base64 Validator fails on valid content.

Currently, the validator used to validate and decode base64 encoded content in FastAPI fails in some scenarios on valid input sent by the game client.

One reliable way I have found to trigger this bug was to append a ? to the end of a level description before uploading a level (level descriptions are an example of base64 encoded content).

Add a `comment_colour` field for users.

Users with a moderator badge can have text colours within the level and account (?) comments sections.
image
This is done through an RGB value sent along with the comment response for each user (these are completely ignored if the user does not have a moderator badge). Currently, this value is hard-coded to 0,0,0.

This issue suggests adding a comment_colour field to the user object (alongside the database table) which allows colours to be defined on a per-user basis. This would come with a corresponding command to allow the said colours to be edited on a per-user basis. This then can easily be referenced within the comment responses.

Design Issue: Assigning likes to a user ID

Currently, the like system has been designed with keeping track of specific users in mind. This however is an oversight, as the likeGJItem.php endpoint does not allow for authentication, complicating how assigning IDs to people is done.

One solution might be to use user IPs to work out who the user may be, but this may not always be correct (eg dynamic IPs) which may cause issues in the long run.

Another solution is to assign likes per IP, which may be the best alternative. This however means that any user will be able to like anything multiple times as long as their IP changes, which may happen naturally.

Open for discussion.

Implement level deletion endpoint.

The deleteGJLevelUser20.php endpoint needs implementation. It allows users to delete their own levels.

The backend for this already exists as logic wise, this is really similar to the /level delete command.

Issue parsing form values equal to zero.

Introduction

Placing this here as I was not able to replicate this in isolation (though still it is likely a Starlette/FastAPI bug).

After the first request, all body form values equal to zero will be parsed as "\x02" instead of "0". This does NOT happen on the first request even if it contains the same exact request body.

Behaviour Experienced (Console Output)

I have added a 3 print statements to the search endpoint (printing the body, headers and form data). I have then proceeded to start the server up and sent the same exact request twice. Looking at the form data logged, the first request's data is parsed correctly. The second's isn't. No external factors changed between the two requests.

realistikgdps-realistikgdps-1  | event=b'gameVersion=21&binaryVersion=34&gdw=0&type=4&str=&diff=-&len=-&page=0&total=1&uncompleted=0&onlyCompleted=0&featured=0&original=0&twoPlayer=0&coins=0&epic=0&secret=Wmfd2893gb7'
realistikgdps-realistikgdps-1  | event=Headers({'host': 'kirb.ussr.pl', 'accept': '*/*', 'content-length': '175', 'content-type': 'application/x-www-form-urlencoded'})
realistikgdps-realistikgdps-1  | event=FormData([('gameVersion', '21'), ('binaryVersion', '34'), ('gdw', '0'), ('type', '4'), ('str', ''), ('diff', '-'), ('len', '-'), ('page', '0'), ('total', '1'), ('uncompleted', '0'), ('onlyCompleted', '0'), ('featured', '0'), ('original', '0'), ('twoPlayer', '0'), ('coins', '0'), ('epic', '0'), ('secret', 'Wmfd2893gb7')])
realistikgdps-realistikgdps-1  | event="Successfully fulfilled the search for query '' with 1 results."
realistikgdps-realistikgdps-1  | event=b'gameVersion=21&binaryVersion=34&gdw=0&type=4&str=&diff=-&len=-&page=0&total=1&uncompleted=0&onlyCompleted=0&featured=0&original=0&twoPlayer=0&coins=0&epic=0&secret=Wmfd2893gb7'
realistikgdps-realistikgdps-1  | event=Headers({'host': 'kirb.ussr.pl', 'accept': '*/*', 'content-length': '175', 'content-type': 'application/x-www-form-urlencoded'})
realistikgdps-realistikgdps-1  | event=FormData([('gameVersion', '21'), ('binaryVersion', '34'), ('gdw', '\x02'), ('type', '4'), ('str', ''), ('diff', '-'), ('len', '-'), ('page', '\x02'), ('total', '1'), ('uncompleted', '\x02'), ('onlyCompleted', '\x02'), ('featured', '\x02'), ('original', '\x02'), ('twoPlayer', '\x02'), ('coins', '\x02'), ('epic', '\x02'), ('secret', 'Wmfd2893gb7')])

How to reproduce

Using the Geometry Dash client

  • Start the server
  • Open your Geometry Dash client and send the first search request
  • Refresh the page to resend the same exact request

Programmatically

Send a request with body below to the search endpoint.
b'gameVersion=21&binaryVersion=34&gdw=0&type=4&str=&diff=-&len=-&page=0&total=1&uncompleted=0&onlyCompleted=0&featured=0&original=0&twoPlayer=0&coins=0&epic=0&secret=Wmfd2893gb7'

Ensure your content type is set to application/x-www-form-urlencoded.

Improve S3 redundancy

Currently, if S3 is enabled, it is the only source that RealistikGDPS will fetch data from. This means that in cases where a connection to S3 is lost, saving data will completely fail and the user will be given an error. The S3 service is almost always ran by a third party and therefore it should not be trusted to have 100% uptime.

A solution to this might be to store files locally while S3 is unavailable, using default behaviour for when S3 is disabled.

This will require loading data to account for the possibility that some data may only be present locally.

However, storing data locally that should be in S3 isn't great in the long term. A potential solution to this is automatically uploading any local files to S3 upon retrieval and deleting the local copy. This could also help people gradually transition to S3.

Fix typo in in `hash_bcypt` function.

This is an introductory issue assigned by me as a means of introducing a few friends to Git/GitHub. As so, if you haven't been contacted by me about this issue, any contributions relating to this issue will be ignored/rejected.

Currently, the functions in rgdps/common/hashes.py relating to hashing passwords using the bcrypt hashing algorithm feature a typo. The typo is the missing r in the function name hash_bcypt (should be hash_bcrypt) and its async variant. The function should be correctly renamed and calls to it should be corrected as well.

Failure to load specific pages in search.

Sometimes, loading a level search page fails within the client. The server gives a valid response but the client is unable to display it.
image

Debugging progress

Here are things I found to reliably trigger this issue:

  • Level in the page has coins_verified set to True.

However, pages with levels that don't meet any of these conditions also fail to load.

Improve console logs.

When using the built-in logging provider support, console logs don't exist at all. This is further compounded with the delay between logs occuring and being processed by the logging provider. Using both methods of logging concurrently could be a useful config option to improve development workflow.

On top of this, even when using the console logging, the messages are often unhelpful because of the extra data missing. This could perhaps be addressed by a dict logger?

Level overwrite produces incosistent results.

Often times, trying to overwrite/update a level causes inconsistent results. Most of the times, it works completely correctly. However, sometimes it either completely errors or creates a new level instead.

Needs more research.

Move config to context

Moving the configuration to be part of the context rather than a global.

Rationale

  • Helps with testing
  • Some logic flow is dependent on the config (such as anything that stores files)

Add Discord webhook support for structured logging.

Add an optional, configurable logging handler that is able to post logs into a Discord chat via a webhook.

This is nice for errors to be quickly spotted and investigated. This will likely be always set at en extremely high logging level (warning/error).

Migrator missing comment colour conversion.

The current database converter is missing user comment colour migration. This is a minor detail that was previously attached on a per-role basis. However, RealistikGDPS handles this on a per-user basis. This shouldn't be too difficult as role logic is already implemented in the migrator.

Implement in-game leaderboards.

Geometry Dash features opt-in leaderboards available on a per-level basis.
image
These allow the user to view and submit their current progress on a level through an endpoint called by the client. These are sorted by:

  • The percentage completion
  • The number of coins

This also allows for a potential server-side anticheat, as these leaderboards often see basic level abuse:

  • Verifying coin counts match the level's max.
  • Verifying coins can only be obtained at 100%

Potential things to look into:

  • Might be a bit overkill but using Redis to store the order for quick lookups.
  • Wipe leaderboards on level update. I don't think the official game does this but it may be a good idea.

Textbox Character Set Validation

Geometry Dash textboxes feature a limited character space. This is usually only includes alphanumeric characters, but may occasionally allow extra characters (such as email allowing @ and passwords including more characters).

Enforcing this would be beneficial as the Geometry Dash response format does not support escaping characters. This means that currently, a level with a | in its name can genuinely completely prevent the game from reading the response.

Additionally, would help against botting.

An ideal solution would leverage pydantic's parsing, perhaps an implementation similar to the Base64String validator. Should be efficient as it will likely be used on many endpoints. https://github.com/RealistikDash/RealistikGDPS/blob/d767f4083241f9fbd40771bba4fbfa6c4a2cce62/rgdps/common/validators.py#L9-L30

Incorrect user displayed when downloading a level.

Currently, the incorrect user is displayed when opening a level created by another player.

image

This is likely a very simple issue related with response data as the data stored in the database and the object are all correct.

Successfully fetched level Level(id=4, name='Level by Test2', user_id=100, ...)

Improve transaction usage

Currently, there is no transaction usage when working with the MySQL database (or any for that matter). This means that if an error occurs during a request, data may only partially be written.

This problem extends to transactions between multiple services, with a possible example being a level being added to MySQL but not viewable due to a meili error.

Optimise bulk fetching of models from the database.

Currently, when multiple models have to be fetched at once, it is done through a look calling the corresponding repository function.
https://github.com/RealistikDash/RealistikGDPS/blob/85bd1a13ad78df1122001b2837b54fb996d14ea6/rgdps/usecases/user_relationships.py#L37-L38

This means that for each entry, a new individual query has to be made, which can add up very quickly. While the individual query is rather fast, it would be possible (and likely optimal) to fetch all of this data at once.

This issue proposes implementing a multiple_from_id function for all of the main resources (such as Level, User, etc). This option will take a list of object IDs and return all the possible results in a list. It should produce the same result as running the from_id function in a loop and ignoring the None entries. This means that the order of the entries should be presented in the same way as the input list.

This shouldn't add much complexity and should make working with large batches of objects easier.

Genericise the level search repository.

Currently, the API for searching levels has been very clearly influenced by the search endpoint.
https://github.com/RealistikDash/RealistikGDPS/blob/85bd1a13ad78df1122001b2837b54fb996d14ea6/rgdps/repositories/level.py#L567-L583
This is a design issue we wish to avoid to make code as reusable as possible.

Fields such as search_type and followed_list are very clearly logic related and necessitate game knowledge to understand fully. Instead, they should be handled through params such as (take these as examples and not requirements): ignored_levels, required_levels, ignored_user_ids etc. The rest should be handled by the corresponding usecases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.