By knowing the lines and start_level, line clears events can be detected beyond maxout even without a Game Genie code, and score can be counted in nestrischamps past 999,999.

Push down point past maxout can't be known obviously, but counting past maxout can be a nice addition.

This is already done to some extent since push down points are computed in the das trainer and classic layouts by comparing how much the score the players should have had after a clear with the actual score he has, and the difference are the push down points.

See the code here, copied below for reference:

onLine(event) {
	const
		num_lines    = event.lines - this.data.lines.count,
		lines_score  = this.getScore(event.level, num_lines),
		actual_score = event.score - this.data.score.current;

	// update total lines and points
	this.data.lines.count = event.lines;
	this.data.points.count = event.score;

	// update drop score
	this.data.points.drops.count += actual_score - lines_score;

Tools / Framework:

• App: Svelte
• CSS: Tailwind ? Bulma ?

Notify Yobi's Stuff server under Nestris/nestrischamps

There are many games in the DB with invalid or corrupted data. Games which were recorded when users were still calibrating, or recorded from calibration that need fine tuning.

This leads to weird entries like:

• Game with 2 frames
• Game with TRT greater than 1, (sometimes Infinity, sometimes NaN)
• Clear list having dashes when the line count is read incorrectly and therefore, lines - previous_lines can be negative 😑

We should not just blindly delete the game entries, since they can be linked to a game file in S3.

TODO:

1. Supply script that remove invalid game entries for known pattern, and delete both DB record AND game file
2. Supply script to crawl over the S3 bucket and delete any file found that is not listed in the DB.

At the moment, everything is in memory (room, users, sessions), so when the service restarts, everything dies, and while clients try to auto-reconnect, the websocket connections for producers and admin will fail because users need to refresh and relogin.

There are usual suspect things to do to make this better:

• Serve static assets from different places, so they can be released without affecting the service
• Store sessions in persistent store, so reconnections can succeed (although room states would still be gone)
• Store rooms in a redis, and use redis pubsub for message broker

We can assume that a given user of nestrischamps has a static setup: nes+some capture card, or emulator, and the colors within that setup are constants.

TODO:

• Allow a player to play a game from level 0 to 10
• Track block colors during a given level
• Compute colors from block average from color1 and color2 (remember to square then root - reference!)
• Store the palette (in local storage? Upload to server?)
• Use the palette in the user OCR.

Note:
Scanning color is actually fast, so there's not much to save on the OCR itself:

BUT capturing the colors grows the size of the overall capture (on the left, so that could save some of the time) since the source capture is the most costly thing JS does:

Reference palette generation script in NESTrisOCR:
https://github.com/alex-ong/NESTrisOCR/blob/master/scripts/compute_color_palette.py
That script works well, but it requires that a player has prepared field images for each of levels 0 to 9, and is therefore a little tedious)

• document frame format, and player class, and layout requirements.
• Create some upload system for custom layout and assets

This is to allow people to ad their own branding onto layouts. It's clear that restreamers do like to brand their layouts, and nestrischamps should allow it.

Technically, this is already possible in a way, non-hosted by nestrischamps.

Make a local html file (maybe start by taking one from this repo), load all the js assets like constants.js, player.js, etc., establish a webconnection to nestrischamps like this:

ws://nestrischamps.herokuapp.com/ws/view/ctjc_pace/<SECRET>

You would receive the player frames from the room you host, and you can do whatever you like with the data.

TODO: document this better

1. Rename current pace metrics from pace to runway
2. Use the name Projection for the Runway Efficiency Interpolation

Do the change both in variable name in code and in the layout labels

The App is currently tracking state inmemory:

• Rooms
• Users
• Sessions

This means deployment are very disruptive and kick users out, requiring a fresh login.

Ideally, while a deployment will break websockets, everything should resume transparently.

TODO:

1. Move sessions data into DB
2. Extract rooms and users state to DB (possibly to a redis DB?)

Note, with the current load, 1 server can handle all the traffic of nestrischamps. If this ever picks up and nestrischamps needs horizontal scaling, then using a redis instance will likely become a requirement, where rooms will be in redis, and game frame data will transit over redis pubsub.

It's a layout that assumes user streams the source stream (for cheating prevention) in the famous Stencil.

The Stencil has available space on the bottom-right corner, which can be filled with the Running Tetris graph and the drought counter.

Make it easier for people to pick what they want.

If cost and processing permits:

• Record frames for every single game
• Store games as individual files somewhere (S3 bucket?)
• store game file url in scores DB

The layouts classic and das_trainer have a chat box which is empty atm. The code to display message already exists, but nestrischamps isn't connecting to Twitch to read and forward chat messages.

TODO:

1. Augment permission scopes with chat:read when logging with Twitch
2. Use Twitch Chat Client to connect for the user specifically (single channel)
3. Forward chat messages to the user connections

The stats page is a little annoying, It return today's score, but today is defined at UTC timezone, it does not take the player's timezone into account.

TODO:
Change the TODAY stats to PAST 24H stats

This can be achieved purely as a SQL query in postgres without having to introduce timezone for players.

Let people track their score no matter what layouts they use, maybe even the competition layouts!

No need to incur encryption cost for this.

Actually should the whole site be http? 🤔

Nestrischamps competition layouts are not in use, partly due to some inertia of everybody already being on the Stencil view, both to produce their game and for restreamers having already setup their slice-n-shift restreaming layouts.

A user mentioned he doesn't like streaming o nStencil any more, because he had aa really good game in competition, but since it wasn't tracked on nestrischamps, it doens't all the extra stats he wants.

To break one barrier to entry, nestrischamps should offer a Stencil layout, players using it would have streams which are compatible with the slice-n-shift restreamer setups ,and if 2 players are on nestrischamps, a game host could switch to using a nestrischamps competition layout easily.

Note: To make this work nicely, issue #55 should be done as well.

Show parts is great to see when calibrating, but when the calibration is good, it's unecessary to waste resources drawing all the parts again.

TODO:

• Automatically turn off "show parts" after the last calibration update

On page load, "show parts" can be "on" and switch off after 30s or 60s. During calibration, any changes can push the automatic switch off into the future, until user is no longer tuning their calibration.

Scores are reported by the renderers, this is not a sane dev practice. Can easily be DDosed and malicious users can report whatever they want easily.

We can probably ignore fake scores, too bad if you want to cheat yourselves, but we should protect the platform and the DB.

A rate a limit of score submission by user should be enough, 1 score per minute maybe? TBD

At the moment, all match rooms belong to users, and the URL for a user's match room is deterministic.

So anyone can connect their producer to someone's room with or without their permission.

They can't force their way into the rendering of the room (since players are assigned by the room host), but they can still make life annoying for the host.

TODO:

• develop a concept of anonymous rooms.
  • anonymous rooms can be created by any users, they become the host and have access to the admin page
  • the room URL can be given to producers to connect to it
  • the room URL is not guessable, so there won't be undesirable visitors

The single player layout all show the pace metric atm, but for beginner player, the metric is not very interesting, since beginners (and many intermediate players), do not reach kill screen often (if at all).

On the other hand, most beginners and intermediate players do play at a level that let them transition (e.g. 15-start transitions at 100 lines, 18 start transitions at 130 lines).

For the single player layout therefore, it is more beneficial to show the transition potential for the early part of the game.

Once transition is reached, the same pace "box" can switch to show the game potential (to save screen real estate).

The layouts das_trainer and classic do show the transition scores, so the transition score is retained on screen even after the pace box switches from transition potential to game potential, but other layouts do not. We can consider whether the transition score is something we want to show in the other single player layouts.

Also TBD: should splitting the pace into 2 components be done for competition layouts too? Perhaps, where possible) the pace box should show data like:

• transition potential (stops changing at transition - duh)
• game potential (stops changing at killscreen OR keeps increasing to match score)
• game potential diff
• game potential tetris diff

Doing that would have the benefit of tracking the actual transition score on screen

OCR is currently done on either device capture or screen capture.

While this is fine to use, a test suite should be able able to play video files and verify the output.

There are plenty of material out-there of for CTM qualifier on Stencil.

TODO:

1. Find the CTM qualifier from someone Best if it was captured and published at 60fps
2. Extract 1 or 2 games from it, best if the games can go past maxout, 1.2M would be great, and level 30/31/32
3. Put the file as web assets somewhere
4. Find a way to play the file in the video tag
5. Run the OCR capture from the video file, verify that all is well

Make it easier for user to calibrate.

The current calibrator uses openCV, but that's a 8MB file to download and it's not very accurate.

Recommendation from @alex-ong in discord:

• pack assets
• use CDN
• serve static assets from a static site server (not from the app server)
• etc...

Context

In traditional restreaming competition, a player streams his own gameplay with webcam with the Stencil layout, while the restreamer captures the stream, cuts it to its own layout needs, and restreams.

With nestrischamps, there is no restreaming any more. There's only hosting a game. Game data from the player goes straight to the restreamer view, saving badnwidth and latency for everyone involved.

BUT this is limiting, player still want to have their own stream at the same time, and while they can stream their webcam, they cannot stream their gameplay on windows (because the input game video feed cannot be shared by both OBS and chrome).

TODO:

When a player is attached to a host's match room, the connection should additionally be attached to his/her own private room.

That way, while the restreamer can use whatever layout they like, the player may feed his own game frame into one of the single player layouts.

The node app's job should be to manage rooms and broker messages. Don't have nodejs waste its time serving static assets.

I dunno know how to do that in heroku. Needs some research and RTFM

Load the OCR page and send game frame anywhere you wish, so you can use the data youself!

A prettier config file would allow automatic formatting for all contributers, while sticking to the existing format.

Context

The field is read with a shine optimization. Basically, the algorithm for detecting blocks by their colors in the board is like this:

1. Check if there is shine on the top-left corner. If no shine it's black
2. Since there's a shine, it must be color 1, 2, or 3, compute color distance to find the best fit

Unfortunately, when some interlacing artefact is incorrectly detected as shine, then black (no block) can never be matched, and the darkest of color 1, 2, 3 will be picked. For example at 18, the darkest color is red, so a trail of red blocks may follow pieces as they fall on some graphic cards.

The shine detection implementation is out in place for 2 reasons:

1. It reduces the colors to compare against from 4 to 3 for each of the 200 blocks of the field. And in fact, since the first half of the screen is mostly black, it alows very quick identification of black tiles
2. It is necessary for retron capture, because levels 7, 17, 27, are display pretty much as black, and so at these leven, by comparing against 4 colors, black is often picked instead of the correct color. By removing testing for black for these levels, color 2 is always picked when there is a shine but the block is very very dark

Example where a white I piecce falling horizontally has a red "trail".

Sample of the interlace artifacts that causes the shine to be detected incorrectly:

TODO:

Options:

1. Add a toggle to activate or deactivate shine optimization overall. For people not on retron, we can restore comparig against 4 colors for every block.
2. Enable shine optimization only on level 7, 17, 27, 37 (x7!), such that things still work on retron.

In both cases, the shine optimization to detect black early can still be applied, the only thing that changes is: when there is a shine, do we compare against 3 or 4 colors.

uWebSockets seems to be the most performant websocket library for nodejs.

TODO:
replace ws by uws

Looks like uWebSocket cannot be installed with npm, so it might mean migrate the project to using yarn instead, which is fine I guess?

The Sanitizer logic right now is to delay reads from score on one end, and lines on the other. Level is read alongside changes to lines.

The base logic for this is that score can change independently (push down points), but level can only change when lines change.

There is a piece of code that sort of binds score and lines in case changes are not detected correctly, as it was seen in video processing in the project tetris_ctm_summary

So basically, when a line change is detected and there was already a pending score read, we should probably read the lines now anyway!

This makes for some clunky logic... and is probably not necessary at all.

Basically, the logic can be cascading with a single trigger. Just as level can never change without a line change, line can never change without a score change. So by inspecting only changes in score, we can have a delay for all of the field.

Just like some line changes have no level changes, some score changes, will have no lines or level changes (push down points), but again, there is a strict cascading order:
score -> lines -> level

And so by focusing only on score the delay logic for reads can be simplified.

The Sanitizer reads value after 1 frame delay to allow the values to stabilize (to fix the dreaded interlaced blurred transition frames) This works well for digits read, BUT field scanning itself should be delayed by one frame. That is because:

1. In Das trainer, field scanning works on palette, but to select the right color reference from the palette, the level must be known, but the level is OCRed in the same frame as the field scanning. So if the level is read incorrectly due to the blurred interlaced frame, then the color detection will be.

2. In Classic Tetris, the colors are read from reference location, but they too can be subject to transition corruption.

Note: There is a mitigating factors here: when reading from a capture device, the whole capture is halfed, which by itself, already eliminates many transition corruption. And when reading from emulator, we can expect the corruption is not present because the emulator doesn't render with interlaced lines (maybe? 🤷 )

So this issue might not be urgent (I have not seen colors problems in my own capture), but I thought I'd document the tought occured to me.

If we wanted to solve this, one way to go about it might be to store the source image with the delayed frame, and only after reading the level correctly from the "stable" frame, then scan the field and update the previous frame before brodcasting it.

Should be relatively easy because the class OCRSanitizer holds a reference to the instance of TetrisOCR, so it could call this.tetris_ocr.scanField(), provided it has the source image to pass along.

Same as a das trainer layout, but without the das data (duh!)

At the moment, games are reported from the UI, since UI computes all the game stats.

But because we have 2 different types of UI with different code (historical reason), it's a pain to maintain game reporting code.

Also, in match layouts, the layout is owned by one user (the game host), but may host 2 different players, and it's not right to have the game host report the game data from different users, from an security perspective, it implies there's an API where a user may report for another, and I don't like that.

We could add a column "reporter" in the DB, so we could still sort of clean bad entries if we wanted, but basically, I thin the solution is to track the game progression in the server, straight from the producer frames.

On the server, there's one ongoing game for a producer, tracking the minimum amount of stats we want to capture in DB. For reference, at the moment, the score table has this structure:

    Column    |            Type             | Collation | Nullable |              Default
--------------+-----------------------------+-----------+----------+------------------------------------
 id           | integer                     |           | not null | nextval('scores_id_seq'::regclass)
 datetime     | timestamp without time zone |           | not null |
 player_id    | bigint                      |           | not null |
 start_level  | smallint                    |           |          |
 end_level    | smallint                    |           |          |
 score        | integer                     |           |          |
 lines        | smallint                    |           |          |
 tetris_rate  | double precision            |           |          |
 num_droughts | smallint                    |           |          |
 max_drought  | smallint                    |           |          |
 das_avg      | double precision            |           |          |
 duration     | integer                     |           |          | 0
 clears       | character varying(400)      |           |          | ''::character varying
 pieces       | character varying(1200)     |           |          | ''::character varying
 transition   | integer                     |           |          |
Indexes:
    "scores_pkey" PRIMARY KEY, btree (id)

So basically, on the connection, a Game instance should run, and intercept game frames, compute basic stats (e.g. TRT and num droughts) like the UI currently does locally, and record the game into the DB when game over event is detected.

This will allo ALL games to be tracked, even the ones done in competition.

BONUS: the server could store all the frame entries into a binary file in S3 or digital ocean, such that any game could be replayed

At the moment, the layouts open a websocket, and the game data are sent over that websocket.

Anyone can open the web sockets to get the data, and then pipe that into whatever they want (no need to necessarily use a layout from nestrischamps).

So document how this works.

Alternatively, or in addition, allow people to submit custom layouts, so people can share their cool design and whatever other stats, and metrics they can think of.

The layouts currently show the twitch avatar in the field.

But for for competition, one might want to see the players video feed. nestrischamps should be about easy setup. a game host should not have to cut player streams, or have to start a side video call on skype or hangout or zoom and then screen cap the video call player feed in OBS.

Proposal:

• The producer window and the rendering layouts can be webRTC peers
• When any window starts, it registers itself as a peer and gets a peer ID and send that to the nestrischamps server
• When admin selects the player, the server instructs the layout it will be expecting a call from a given peer, and instructs the producer window to initiate a call to the rendering layout and submit its video feed

Complexities:

• Handle call drops, reconnection, admin changing players, etc.
• Producer page must allow player to select which webcam to pick
• How to integrate the player feed in the layouts? small areas? Full blown? etc.

Optional Bonus:

• The game frames themselves could be sent peer to peer (Nice for Say, Singapore players, whose game frame do a full roud trip to the US server even though both players and game host are in Singapore)

When OCR is not working all that well, or there are weird things happening in the combination of OCR and renderer, it can be hard to troubleshoot.

instead of requiring for specific details from users, it would be great to have a "submit setup details" button of sorts which would send a few things:

• The OCR config
• A raw frame from the source (emulator or capture)
• The precomputed area overlay
• The performance report

For competition layouts which support video feed, nestrischamps should not grab the video feed of webcam without user consent.

TODO:

• The producer window should have some ui control to allow/disallow player feed

Simple checkbox could do.

To check the health of the server, and plan capacity, an admin page showing monitoring information should be created.

Something to track:

• number of active private rooms
• number of active match rooms
• number of connections
• number of ongoing games

System metrics can be tracked on the infra provider (heroku or digital ocean)

Producers do only one thing now, send game frames over their web socket.

Many producer can connect to a single game room, where room owner can adjust who is player 1 and player 2 from the list of connected users, but at the moment, all producers are sending their game frame and the non-player 1/2 data is being dropped by the app.

Ideally, the app could instruct the producers to produce when needed. If you're a producer but not player 1 and 2, and are on standby, it's not necessary to send game frames at all.

At the very least, a connection-level command could be used to make the connection not send anything.

At a higher level, a producing-level command could let the producer not even do OCR if it's not needed to send anything since no-one is listening.

One more use case for producer to accept commands would be to synchronize a count down between the players.