Giter Club home page Giter Club logo

stash-empornium's Introduction

Docker Image Version GitHub release PyPI - Python Version GitHub

logo

stash-empornium

This fork of a script by user humbaba allows torrent files and associated presentations to be created for empornium based on scenes from a local stash instance.

Installation

The backend can be installed by cloning this repository or by running the Docker image bdbenim/stash-empornium.

For detailed instructions on installing the backend server, refer to the Installation page on the wiki.

Userscript

Dependencies

Currently, the script does not work with other userscript managers, though this may change in the future.

The userscript can be installed here.

Configuration

  1. Visit upload.php and open the Tampermonkey menu. Set the backend URL, stash URL, and API key (if you use authentication).

  2. Update config file located at config/config.ini:

[backend]
## name of a file in templates/ dir
default_template = "fakestash-v2"
## List of directories where torrents are placed
## Multiple directories can be specified in this format:
## torrent_directories = ["/torrents", "/downloads"]
torrent_directories = ["/torrents"]
## port that the backend listens on
port = 9932

[stash]
## url of your stash instance
url = "http://localhost:9999"
## only needed if you set up authentication for stash
#api_key = 123abc.xyz

The port above corresponds to the backend URL in step 1, so if you change one you must change the other.

Redis

The backend server can be configured to connect to an optional redis server. This is not required for any of the functionality of the script, but it allows image URLs to be cached even when restarting the backend, speeding up the upload process whenever an image is reused (e.g. performer images, studio logos). If redis is not used, these URLs will still be cached in memory for as long as the server is running.

Connection settings can be specified in the [redis] configuration section:

[redis]
host = "localhost"
port = 6379
username = "stash-empornium"
password = "stash-empornium"
ssl = false

Any unused options can simply be omitted.

Torrent Clients

The backend server can be configured to communicate with any of several different torrent clients, allowing generated .torrent files to be automatically added to the client. Path mappings can also be used to ensure the torrent points at the correct location of files on disk, allowing them to be started with minimal settings. Additionally, some clients support applying labels to torrents for more granular control.

Torrent client integrations are optional and are not required for the backend to work.

rTorrent

This software has been tested with rTorrent v0.9.6 with ruTorrent v3.10.

Example configuration:

[rtorrent]
# Hostname or IP address
host = "localhost"
# Port number
port = 8080
# Set to true for https
ssl = false
# API path, typically "XMLRPC" or "RPC2"
path = "RPC2"
# Username for XMLRPC if applicable (may be different from webui)
username = "user"
# Password for XMLRPC if applicable (may be different from webui)
password = "password"
label = "stash-empornium"

[rtorrent.pathmaps]
"/stash-empornium/path" = "/rtorrent/path"

Note

The path mappings for the torrent client are with respect to the paths on the backend server, not stash. If your client is reporting errors that files are missing, make sure you check this setting carefully. For example, if your files are stored in /media on your stash server, and that directory is mapped to /data on your backend and /downloads in your torrent client, then you will need something like this in your config:

["file.maps"]
"/media" = "/data"
...
[rtorrent.pathmaps]
"/data" = "/downloads"

Deluge

This software has been tested with Deluge v2.1.1. The same configuration options are supported as with rTorrent, with two exceptions:

  • Labels are not supported
  • No username is required for authentication

qBittorrent

This software has been tested with qBittorrent v4.6.0. The same configuration options are supported as with rTorrent.

Currently there is one limitation with the qBittorrent API integration which prevents the backend from triggering a recheck of downloaded files when adding a .torrent. This is planned for a future release.

Usage

  1. Run emp_stash_fill.py
  2. Get scene ID (it's in the url, e.g. for http://localhost:9999/scenes/4123 the scene id is 4123)
  3. Go to upload.php and enter the scene ID in the "Fill from stash" box
  4. Select the file you want if you have multiple files attached to that scene, tick/untick the generate screens box, pick template if you have defined others
  5. Click "fill from" and wait as the tedious parts of the upload process are done for you. Status messages should appear and instructions for final steps. Performer tags like pamela.anderson will be generated for you, along with resolution tags and url tags of the studio, e.g. 1080p and brazzers.com
  6. You still need to load the torrent file (the location on your filesystem will be given to you) into the form, set a category, optionally check for dupes if you didn't do so manually. Also load the torrent file into your client (you can configure the torrent output directory to be a watch dir for your torrent client) and make sure the media file is visible to your torrent client
  7. When you're satisfied everything is ready, upload

Within Stash

As of v0.17.0, a new button has been added to the scene page within Stash:

Screenshot of Stash upload button

Clicking this button will launch upload.php and automatically fill in the form with the current scene. This feature is still somewhat experimental, including the following issues:

  • The script needs to save your tracker announce URL before this feature can work, which is done simply by navigating to upload.php with the script enabled.
  • Clicking this button occasionally fails to fill in the form. If this happens, simply go back and try a second time.
    • This seems to happen more frequently when starting multiple uploads in quick succession from different tabs
  • Currently this only works with the default settings of generating screenshots and excluding associated galleries. In the future, these options will be configurable.
  • There may be other issues not mentioned above

Including Galleries

Uploads can optionally include a gallery associated with a scene by checking the box labeled "Include Gallery?" on the upload page. In order to generate a torrent with multiple files, they must be saved in a directory together, which requires some additional configuration options:

[backend]
## Where to save media for torrents with more than one file:
media_directory = "/torrents"
## How to move files to media_directory. Must be 'copy', 'hardlink', or 'symlink'
move_method = 'copy'

The media_directory option specifies the parent directory where media files will be saved. Each torrent will get an associated subdirectory here, based on the title of the scene.

move_method specifies how media files will be added to this new directory. The default is copy because it is the most likely to work across different setups, but the downside is that this will create a duplicate of your media. To avoid this, the hardlink or symlink options can be selected, but these have limitations. Symlinks point to the path of the original file, which means that if your torrent client sees a different path structure than your backend server then it won't be able to follow symlinks created by the backend. Hardlinks do not have this issue, but they can only be created on the same file system as the original file. If you're using Docker, locations from the same file system added via separate mount points will be treated as separate file systems and will not allow hardlinks between them. There are additional pros and cons that are beyond the scope of this readme.

Command Line Arguments

The script can be run with optional command line arguments, most of which override a corresponding configuration file option. These can be used to quickly change a setting without needing to modify the config file, such as for temporarily listening on a different port or saving torrent files in a different directory. Not all configuration options can currently be set via the command line. The available options are described in the script's help text below:

usage: emp_stash_fill.py [-h] [--configdir CONFIGDIR] [--version] [-q | -v | -l LEVEL] [--flush] [--no-cache | --overwrite]

backend server for EMP Stash upload helper userscript

options:
  -h, --help            show this help message and exit
  --configdir CONFIGDIR
                        specify the directory containing configuration files
  --version             show program's version number and exit

Output:
  options for setting the log level

  -q, --quiet           output less
  -v, --verbose, --debug
                        output more
  -l LEVEL, --log LEVEL
                        log level: [DEBUG | INFO | WARNING | ERROR | CRITICAL]

redis:
  options for connecting to a redis server

  --flush               flush redis cache
  --no-cache            do not retrieve cached values
  --overwrite           overwrite cached values

Templates

This repository includes default templates which can be used to fill in the presentation based on data from stash. Currently there are two, however more may be added in the future.

Adding Templates

To add a new template, save it in the templates directory alongside your config.ini file. Then add it to your configuration with the following format:

[templates]
filename = "description"

Templates are written using Jinja syntax. The available variables are:

  • audio_bitrate
  • audio_codec
  • bitrate
  • contact_sheet
  • container
  • cover
  • date
  • details
  • duration
  • framerate
  • gallery_contact
  • image_count
  • media_info (if mediainfo is installed)
  • performers
    • name
    • details
      • image_remote_url
      • tag
  • resolution
  • screens
  • sex_acts
  • studio
  • studio_logo
  • title
  • video_codec

Refer to the default templates for examples of how they are used.

Custom Lists

In addition to the template variables described above, additional tag lists may be added to the empornium config section by following the format of the sex_acts variable. These will automatically be parsed and made available to any custom templates as comma-separated lists. For instance, you may wish to add a section called performer_attributes to describe characteristics of performers in the scene.

Titles

Similarly to templates, the title has a few options for formatting. This uses python's builtin string formatter, so variable names are enclosed in braces ({}) within the string. The default title format is:

[{studio}]
{performers} - {title}({date})[{resolution}]

This would result in something like this:

[Blender Institute] Big Buck Bunny, Frank, Rinky, Gimera - Big Buck Bunny (2008-05-10)[1080p]

The available variables that can be used are:

  • codec
  • date
  • duration
  • framerate
  • performers
  • resolution
  • studio
  • title

Title Templates

Beginning with v0.7.0, the title_template config option has been added, which extends the title formatting capability using jinja templates. With this system, the equivalent to the earlier example is:

{ % if studio %}[{{studio}}]
{ % endif %}{{performers | join(', ')}}
{ % if performers %} - { % endif %}{{title}}
{ % if date %}({{date}})
{ % endif %}[{{resolution}}]

This system has the added advantage of builtin if statements, for loops, and many other features. The above example uses these to ensure that there are no empty square brackets if the scene's studio is not set, nor empty parentheses around a missing date. Since the resolution is determined by the script, this will always be available. The same variables are available to this setting as the title_default option, with some minor differences:

  • performers will be provided as a list rather than a single comma-separated string. This allows more control over how the list will be formatted, but the above example shows how to keep the same comma-separated list formatting.
  • framerate does not include "fps" in the string, again for more flexibility in the template

For more information on using jinja templates, refer to the documentation

stash-empornium's People

Contributors

asparghus avatar bdbenim avatar dependabot[bot] avatar dsrtusr88 avatar husariacakes avatar starnull avatar tamalesenmipinata avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

stash-empornium's Issues

Suggest tag mappings if not found in config

If a scene has a tag which is not found in the [empornium.tags] section of config.ini, the user should be presented with a suggested tag mapping, which they can accept, override, or ignore. Here's what I believe would need to go into that:

  • Generate suggested EMP tag (possibly in the same way performer tags are generated)
  • Return the suggested tag mappings to the userscript
  • Have the userscript present the suggested tag mappings to the user
    • Include form controls that allow the user to accept, override, or ignore the suggested tag
    • Ideally use EMP's autocomplete feature if the user chooses to override the tag
  • Save the user's decision in config.ini

stash port is hard-coded in .py file

`2023-09-24 13:18:48,432 - ERROR - Exception while serving /fill
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/waitress/channel.py", line 428, in service
task.service()
File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 168, in service
self.execute()
File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 456, in execute
for chunk in app_iter:
File "/usr/local/lib/python3.9/site-packages/werkzeug/wsgi.py", line 289, in next
return self._next()
File "/usr/local/lib/python3.9/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
for item in iterable:
File "/usr/local/lib/python3.9/site-packages/flask/helpers.py", line 118, in generator
yield from gen
File "/emp_stash_fill/emp_stash_fill.py", line 159, in generate
stash_response_body = stash_response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

is there a way to import your port/url to a variable instead of hardcoding in localhost:9999 ?

Enhancement: Previews/markers to cover/presentation gif/s

I don't know much about how folk get their really long run-time ultra-compressed gifs that don't look like garbage. But assuming this conversion is relatively straight forward and uses foss that can be included in the docker image (rather than another web api or something—though I suppose that would also be feasible), then:

"Easy"

Stash already generates a generic preview .mp4 video that's 10 seconds long, this seems like a convenient source for automatic conversion to a cover.gif. Unfortunately they sample from the very start of the video, so often the leading frames in these previews are garbage.

Less easy

Alternatively/in addition, and more challenging from a development point of view, markers conveniently generate a series of little .mp4s that are 20 seconds long. You could envision a few options here:

  • Dumping of each of these markers into their own .gif in the presentation along with their marker title. These might need shortening to 5 or 10 seconds each.
  • Stitching all the markers together to form a manual preview gif (but that actually includes the "best bits" rather than a random sampling). The compilation of markers would probably only use the first 2 or 3 seconds from each marker.

The use of markers auto-generated .mp4s might be a red herring given their length and the likely need to sub-sample them anyway. It might actually be better/easier to run ffmpeg on the scene's file directly using only the markers time-points.

Enhancement: Userscript support for a button "upload to emp" within stash

Title basically.

Add a button on a stash scene's panel . The edit tab makes most sense, as the workflow would be to scrape the scene, edit all the tags/metainfo etc, then upload. The button would basically open an empornium upload.php tab, and pre-populate that scenes id number into the "stash ID" field provided by the script. It could automatically activate the "fill form" button provided by the script. You'd have to assume that the user is already authenticated/logged in, so maybe/maybe not handle that "error", and you'd need to make .sx or .is domain configurable in the config I guess.

Workflow becomes:

  • Edit meta data in stash until you're happy with it
  • Click "upload to emp" button
  • New tab opens (ideally without even changing focus to it), and starts working on all the jobs.
  • User can keep working on the next scene or whatever, or upload a new one. These jobs can happen asynchronously I presume (I haven't actually tested).
  • At some point the user can come back to the emp tab(s) and review and click upload.

This enhancement would be improved by caching of image urls (which you've implemented but not released I believe), caching of the .torrent generation job being complete, etc, etc. So even if an upload tab is forgotten/closed by the user at some point they can simply click the button on the scene again and have it be "instantly" ready.

Potential issue:
If the user processes like 10 scenes at a time, it might be painful to associate the correct .torrent with each tab. It would be great if you could inject this as a part of the script. It's entirely possible to just make the .torrent in client-side js (see anthelion's upload.php, the js is all there). This should be it's own issue (lol). And this approach probably isn't great, because then caching the resulting .torrent becomes problematic should you need to re-run the script.

Edit:

Tasks

  • #128
  • #129
  • #130
  • Reliably know when to clear jobs from the list

Not getting on-page dialogues/No activity

2023-10-12 07:34:33,011 - INFO - Generating submission for scene ID 63102 including screens.
2023-10-12 07:34:33,054 - ERROR - Unknown studio logo file type: image/webp

When I click fill form I get no text generated on the browser page and the above is the only log notes shown.

Date as a string

Having issues when using the jinja template:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 456, in execute
    for chunk in app_iter:
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
    for item in iterable:
  File "/usr/local/lib/python3.10/site-packages/flask/helpers.py", line 115, in generator
    yield from gen
  File "/emp_stash_fill/emp_stash_fill.py", line 734, in generate
    title = render_template_string(TITLE_TEMPLATE, **{
  File "/usr/local/lib/python3.10/site-packages/flask/templating.py", line 164, in render_template_string
    return _render(app, template, context)
  File "/usr/local/lib/python3.10/site-packages/flask/templating.py", line 133, in _render
    rv = template.render(context)
  File "/usr/local/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/usr/local/lib/python3.10/site-packages/jinja2/utils.py", line 83, in from_obj
    if hasattr(obj, "jinja_pass_arg"):
jinja2.exceptions.UndefinedError: 'str object' has no attribute 'strftime'

Looks like the current data is being pulled only as a string in the .py code?

Default config values result in AttributeError

2023-10-07 10:08:33,824 - INFO - Reading config from /config/config.ini
2023-10-07 10:08:33,832 - INFO - Template "fakestash-v2" has a new version available in the default-templates directory
Traceback (most recent call last):
File "/emp_stash_fill/emp_stash_fill.py", line 103, in
TITLE_DEFAULT = conf["backend"].get("title_default", "[{studio}] {performers} - {title} ({date}){resolution}").value # type: ignore
AttributeError: 'str' object has no attribute 'value'

Enhancement: Torrent client integration

This will likely be broken into multiple issues, but since the goal for all will be the same I'm just creating one here for now.

Add the ability to integrate with torrent clients via their APIs so that .torrent files can be added automatically and pointed to the correct location of files on disk based on the paths returned by stash. A custom label could also be specified allowing rules to be created in the client, such as seed time or upload targets.

Major clients to be targeted are:

Enhancement: Advanced .torrent file output directory settings

Related to #64, a couple of proposed changes to how .torrent files are saved:

  • Allow more than one output directory
    • This will allow torrent client watch directories to be used while still preserving a copy of the file for the user to easily attach to the upload.
  • Detect when submission has been uploaded and send .torrent to torrent client
    • This also works in concert with watch directories, allowing the torrent to only be added to the client after it has been uploaded to EMP, ensuring that it will be registered and therefore able to begin uploading immediately without restarting the torrent in the client.

Enhancement: Support simultaneous upload processing

Currently most of the backend is written on the assumption that it will process only one upload at a time. This causes problems when more than one upload is being processed in parallel, due to issues with race conditions or simply global variables that should be local.

Hashlib was added in 0.8.0 but python 3.10.13 does not support hashlib.file_digest

Gettting this traceback:
2023-10-19 03:32:12,161 - INFO - Making torrent 2023-10-19 03:32:13,787 - INFO - Generating media info 2023-10-19 03:32:15,215 - INFO - Uploading cover 2023-10-19 03:32:15,215 - ERROR - Exception while serving /fill Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service task.service() File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service self.execute() File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 456, in execute for chunk in app_iter: File "/usr/local/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__ return self._next() File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded for item in iterable: File "/usr/local/lib/python3.10/site-packages/flask/helpers.py", line 115, in generator yield from gen File "/emp_stash_fill/emp_stash_fill.py", line 839, in generate cover_remote_url = img_host_upload(img_host_token, cookies, cover_file[1], cover_mime_type, cover_ext) File "/emp_stash_fill/emp_stash_fill.py", line 324, in img_host_upload digest = hashlib.file_digest(f, hashlib.md5).hexdigest() AttributeError: module 'hashlib' has no attribute 'file_digest'

Downgrading to 0.7.0 works fine.
hashlib.file_digest does not exist in python 3.10
It does exists in 3.11.

Can the required version of python in the dockerfile be bumped?

Enhancement: Userscript support for a button "upload to emp" within stash

Added on behalf of @preshow6237

Add a button on a stash scene's panel . The edit tab makes most sense, as the workflow would be to scrape the scene, edit all the tags/metainfo etc, then upload. The button would basically open an empornium upload.php tab, and pre-populate that scenes id number into the "stash ID" field provided by the script. It could automatically activate the "fill form" button provided by the script. You'd have to assume that the user is already authenticated/logged in, so maybe/maybe not handle that "error", and you'd need to make .sx or .is domain configurable in the config I guess.

** Workflow becomes:**

  • Edit meta data in stash until you're happy with it
  • Click "upload to emp" button
  • New tab opens (ideally without even changing focus to it), and starts working on all the jobs.
  • User can keep working on the next scene or whatever, or upload a new one. These jobs can happen asynchronously I presume (I haven't actually tested).
  • At some point the user can come back to the emp tab(s) and review and click upload.

This enhancement would be improved by caching of image urls (which you've implemented but not released I believe), caching of the .torrent generation job being complete, etc, etc. So even if an upload tab is forgotten/closed by the user at some point they can simply click the button on the scene again and have it be "instantly" ready.

** Potential issue:**

If the user processes like 10 scenes at a time, it might be painful to associate the correct .torrent with each tab. It would be great if you could inject this as a part of the script. It's entirely possible to just make the .torrent in client-side js (see anthelion's upload.php, the js is all there). This should be it's own issue (lol). And this approach probably isn't great, because then caching the resulting .torrent becomes problematic should you need to re-run the script.

Create a userscript to add an upload button to stash

Either create a second userscript or extend the functionality of the existing userscript to add an upload button to stash's UI. This button will trigger a job to start as in #128 before navigating the user to upload.php. If possible, pre-populate the upload form with the result of the job. If not, a menu can be displayed to the user allowing them to select the job to retrieve.

"this" undefined

Browser: Firefox v118.0.1
Violentmonkey version: 2.15.0
OS: Linux

Symptoms:
After pasting in a stash ID number from the URL, nothing happens. Inspect the page to read tho console and get the following...

Error:
Screenshot from 2023-10-03 18-01-08

Relevant code line:
Screenshot from 2023-10-03 18-00-26

Exception while serving /fill

`2023-10-07 13:17:42,585 - INFO - Generating submission for scened ID 59971 including screens.
2023-10-07 13:17:43,303 - ERROR - Exception while serving /fill
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 456, in execute
    for chunk in app_iter:
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
    for item in iterable:
  File "/usr/local/lib/python3.10/site-packages/flask/helpers.py", line 115, in generator
    yield from gen
  File "/emp_stash_fill/emp_stash_fill.py", line 344, in generate
    with Image.open(studio_img_file[1]) as img:
  File "/usr/local/lib/python3.10/site-packages/PIL/Image.py", line 3280, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/tmp/tmpdz7daizc.svg'
2023-10-07 13:18:06,218 - INFO - Generating submission for scened ID 59971 including screens.
2023-10-07 13:18:06,557 - ERROR - Exception while serving /fill
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/usr/local/lib/python3.10/site-packages/waitress/task.py", line 456, in execute
    for chunk in app_iter:
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
    for item in iterable:
  File "/usr/local/lib/python3.10/site-packages/flask/helpers.py", line 115, in generator
    yield from gen
  File "/emp_stash_fill/emp_stash_fill.py", line 344, in generate
    with Image.open(studio_img_file[1]) as img:
  File "/usr/local/lib/python3.10/site-packages/PIL/Image.py", line 3280, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file '/tmp/tmp8txhcgvf.svg'`

Option for/Intrinsic disablement of male performer auto-tagging

Male performers currently affect the presence of tags like "tattoo", "piercings", and "tattoos.and.piercings". They also affect the presence of tags relating to nationality (e.g. "american", "australian", etc).

I recall seeing some efforts in the code to discount male performers from certain tags when I read the PR adding this functionality, so I'm not sure if this is an oversight/bug or not. I'm not even sure that this needs to be an option/setting (i.e. males should just always be discounted from tagging, or if they're tagged then they should use the male tag variants). Emp either doesn't care about male performers when it comes to tags, or in the cases that they do, they use separate tags with 'male' in them—it's almost universally true that any genderless tag implies female on Emp.

E.g. "tattoo" is the popular tag, but it's pretty much only applied to the female talent, there's a separate tag "tattooed.male" for males. It's worth noting that "tattooed.female" also exists but only has like 7% of the popularity of the "tattoo" tag—this is often the case with the more specific "female" gendered tags, this specificity is just not as widely used because the genderless tags are assumed female.

I'd need to test more widely to see what other tags are inherited from the male performers—these might be the only ones, or there might be others that have been missed.

Enhancement: Add .torrent file to upload automatically

As mentioned in #63, this would allow the userscript to attach the generated .torrent file to the upload without requiring the user to navigate to it.

For security reasons, it is normally not possible for a script running on the page to attach a file without user interaction, as this would make it trivial for any malicious webpage to steal files from the user's computer without their knowledge. However, since we are actually generating the contents of this file ourselves, we don't actually care what file it comes from as long as we can somehow upload it to EMP. As such, here is how I see this feature eventually working:

  • After filling out the presentation on the page, the script downloads the .torrent file directly from the backend server and stores the data in memory
  • The script overrides the check for dupes and Upload torrent buttons to use this data stored in memory instead of the file referenced by the normal file chooser input
  • These buttons send requests that mimic what the buttons normally send to EMP, using this in-memory data

Although complex, this ought to succeed in circumventing the limitations of how scripts can interact with file inputs. The main issue will be maintainability, as this creates another possible breaking point of the script if the process of uploading to EMP ever changes. If the script is not updated at that time, it will cease to work entirely for uploading torrents. To prevent this, it should be easy for the user to disable this functionality if necessary. It also might be possible for the script to hijack existing code for submitting the upload instead of reverse engineering it. This would both save effort and reduce likelihood of the feature breaking in some future update to EMP.

enhancement: select multiple from stash, create pack

Would like the ability from the scenes view, select multiple scenes and toggle an option via Plug-in to Create Pack.
Pack would then generate a folder in a specified directory with the files hardlinked into it.

Enhancement: Append basic video description based on tags

What I've noticed is that most porn scenes follow a formulaic approach to how a scene unfolds. By the number of actors, gender, and basic tags, you can string together an EMP-passable text description.

For instance, if a scene has a male and a female actor:
"This is a scene involving a male and a female."

For tags associated (especially ones with AI generation (#6) as that can pick out the sequence and timing) you can almost certainly bet:
"The female begins with <blowjob> and moves to intercourse involving <positions>. The scene concludes with the man finishing <locations>."

Even that may be more than enough.

Issue uploading images

Normally no issues, but this one scene is causing problems.
2023-09-24 14:29:09,966 - INFO - Uploading cover
2023-09-24 14:29:11,293 - ERROR - Exception while serving /fill
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/waitress/channel.py", line 428, in service
task.service()
File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 168, in service
self.execute()
File "/usr/local/lib/python3.9/site-packages/waitress/task.py", line 456, in execute
for chunk in app_iter:
File "/usr/local/lib/python3.9/site-packages/werkzeug/wsgi.py", line 289, in next
return self._next()
File "/usr/local/lib/python3.9/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
for item in iterable:
File "/usr/local/lib/python3.9/site-packages/flask/helpers.py", line 118, in generator
yield from gen
File "/emp_stash_fill/emp_stash_fill.py", line 409, in generate
cover_remote_url = img_host_upload(img_host_token, cookies, cover_file[1], cover_mime_type, cover_ext)
File "/emp_stash_fill/emp_stash_fill.py", line 130, in img_host_upload
return response.json()["image"]["image"]["url"]
KeyError: 'image'

Final JSON result not being parsed

Hoping someone else can help with this since I'm not great with JS. Lately I'm getting a bug where the final JSON string containing the rendered template, title, tags, etc is not being parsed by the userscript.

For troubleshooting, I added a delay followed by an additional JSON response to emp_stash_fill.py:

    yield json.dumps(
        {
            "status": "success",
            "data": {
                "message": "Done",
                "fill": {
                    "title": title,
                    "cover": cover_remote_url,
                    "tags": " ".join(tags),
                    "description": description,
                    "torrent_path": torrent_path,
                    "file_path": stash_file["path"],
                },
            },
        }
    )

    time.sleep(1)
    return json.dumps({
        "status": "success",
        "data": {
            "message": "Finished"
        }
    })

With this code, the first object is parsed successfully, and the new last object is ignored. So the issue doesn't seem to be with the returned data, but something about being the last object in the stream.

Enhancement: Copy new config options from default.ini if not present in config.ini

Since configupdater is used to handle the config files instead of the builtin configparser, changes can be made to the config programmatically without deleting comments or otherwise changing formatting in a way the user would not expect. This means that if new config options are added in the future to default.ini, such as in #14, then these new default values can automatically be added to the user's config. This should be accompanied by a log message notifying them of the new option so that they can configure it.

This relates to #17 but doesn't entirely resolve it.

Enhancement: Start torrent after submitting upload

Related to #69 and #70. Add a listener to the upload button in the userscript and transfer the .torrent file to the client after the user submits the upload, whether by copying to a directory or transferring via an API.

Update:
Since #70 is now closed covering three major torrent clients, the goal of this enhancement is changed from adding the torrent file to the client after submitting the upload, to starting the torrent after submitting. Multiple output directories will remain supported, but (for now at least) complex behaviour involving copying to different directories at different times is not planned. Starting uploads on submission will be added for torrent client APIs, but not for watch directories.

Implementation will require a couple of other tasks:

Enhancement: Biological data to tags

Essentially the title, inherit the biographical data from any performer and convert them to tags. But to expand on the idea:

Simple (limited scope, straight forward mapping):

  • Eye colour mapping to tags: blue.eyes, brown.eyes, green.eyes, ...
  • Tit "status" mapping to tags: natural.tits, or fake.tits
  • Hair colour mapping to tags: blonde, brunette, black.hair, redhead, ...
  • Ethnicity mapping to tags: caucasian, asian, ebony, latina, ...
  • Tattoos mapping to tags: tattoo
  • Piercings mapping to tags: piercings

This obviously only applies to female performers, there's different tags for male performers. They often go untagged on EMP though, so not sure if they need to be implemented (sorry gaybros).

Complex (so therefore less important XD):

  • Nationality mapping to tags—this is pretty "easy", but the scope is large, so I've categorised it as "complex": australian, english, scottish, welsh, irish, russian, ukranian, croatian, latvian, czech, french, mexican, german, ...
  • Nationality "conglomerations" mapping to tags: british {welsh, english, scottish, ...}, european {french, german, czech, russian, ...}, ...
  • Breast size mapping to tags: tiny.tits, small.tits, medium.tits, big.tits huge.tits, ...
  • Breast "combo" mapping to tags: big.natural.tits, big.fake.tits, ...
  • Age (at time of performance) to tags—EMP sucks at age tags: teen, milf (when they're 20 I guess??,), grandmother (when they're 30 I guess??).

Tangentially related, scene composition tags (male performers are often poorly tagged, so this might not be a good idea. You also have "uninvolved performers", i.e. in cuck scenes or whatever...):

  • 1on1, threesome, foursome (just count the number of performers and spit these out, regardless of gender)
  • ffm, 3females.1male, 4females.1male, 5females.1male, ..., reverse.gangbang (anything that's 3female.1male or more)
  • mmf, 1female.3males, 1female.4males, 4female.5males, ..., gangbang (anything that's 1female.3male or more)
  • mmff, 3females.2males, 4females.2males, ... 2females.3males, 2females.4males, ..., orgy (anything that's 3females.2males, or 2females.3males or more)
  • probably a whole lot more tags for lesbian stuff, gay stuff, or trans stuff

Allow galleries associated with scenes to be included in upload

I'd like to be able to check a box on the upload page that tells the script to generate a torrent with both the scene and the associated gallery if one exists. This will require a few things to be done:

  • get gallery info with scene query
  • create/upload gallery preview images and contact sheet
    • unzip gallery if required
  • include additional info in presentation template
  • ensure scene & gallery files are located together for the torrent (maybe symlink?)
  • present this option to the user

Enhancement: Biological data to tags

Copied for tracking since #57 was deleted:

Essentially the title, inherit the biographical data from any performer and convert them to tags. But to expand on the idea:

Simple (limited scope, straight forward mapping):

  • Eye colour mapping to tags: blue.eyes, brown.eyes, green.eyes, ...
  • Tit "status" mapping to tags: natural.tits, or fake.tits
  • Hair colour mapping to tags: blonde, brunette, black.hair, redhead, ...
  • Ethnicity mapping to tags: caucasian, asian, ebony, latina, ...
  • Tattoos mapping to tags: tattoo
  • Piercings mapping to tags: piercings

This obviously only applies to female performers, there's different tags for male performers. They often go untagged on EMP though, so not sure if they need to be implemented (sorry gaybros).

Complex (so therefore less important XD):

  • Nationality mapping to tags—this is pretty "easy", but the scope is large, so I've categorised it as "complex": australian, english, scottish, welsh, irish, russian, ukranian, croatian, latvian, czech, french, mexican, german, ...
  • Nationality "conglomerations" mapping to tags: british {welsh, english, scottish, ...}, european {french, german, czech, russian, ...}, ...
  • Breast size mapping to tags: tiny.tits, small.tits, medium.tits, big.tits huge.tits, ...
  • Breast "combo" mapping to tags: big.natural.tits, big.fake.tits, ...
  • Age (at time of performance) to tags—EMP sucks at age tags: teen, milf (when they're 20 I guess??,), grandmother (when they're 30 I guess??).

Tangentially related, scene composition tags (male performers are often poorly tagged, so this might not be a good idea. You also have "uninvolved performers", i.e. in cuck scenes or whatever...):

  • 1on1, threesome, foursome (just count the number of performers and spit these out, regardless of gender)
    • ffm, 3females.1male, 4females.1male, 5females.1male, ..., reverse.gangbang (anything that's 3female.1male or more)
    • mmf, 1female.3males, 1female.4males, 4female.5males, ..., gangbang (anything that's 1female.3male or more)
    • mmff, 3females.2males, 4females.2males, ... 2females.3males, 2females.4males, ..., orgy (anything that's 3females.2males, or 2females.3males or more)
  • probably a whole lot more tags for lesbian stuff, gay stuff, or trans stuff

Enhancement: Change config format to TOML

TOML has better support than ini for more complex data such as arrays and data types such as integers, booleans, etc. As well, tomlkit appears to be a more mature parsing library than configupdater, which has some formatting bugs when writing config files.

Once implemented, the server should migrate the user's configuration file to the new format so that no user intervention is required and all settings are preserved.

I considered YAML as well due to its more visually obvious representation of nested hierarchies, but I don't think it's the best option for two reasons:

  1. pyyaml, which is recommended in case you need to write and not only read yaml files, does not appear to preserve comments or other formatting
  2. YAML is less user-friendly to write for users who don't use code editors because it relies on correct indentation
    • it is also less similar to ini and therefore might present a greater challenge either for the automated migration process or for user adjustment

Mod: Making release titles more compatible to parsing

The default title_default = [{studio}] {performers} - {title} ({date})[{resolution}] is a fairly non-standard release title format. Would like to propose a few tweaks.

New config option:
replace spaces with periods = false|true

New config variables:
titleslug (Site Title's becomes SiteTitles
2ddate (2023-10-14 becomes 23.10.14)

Change config option
title_default = {titleslug} {2ddate} {title} {performers} {codec} {resolution}
remove commas between multiple performers
[Blender Institute] Big Buck Bunny, Frank, Rinky, Gimera - Big Buck Bunny (2008-05-10)[1080p] becomes BlenderInstitute.08.05.10.Big.Buck.Bunny.Big.Buck.Bunny.Frank.Rinky.Gimera.x264.1080p

Image uploads fail after backend is idle

Only one session with the image host is established on startup, which eventually expires, causing all subsequent uploads to fail. This session should be periodically renewed or reestablished to prevent failures.

Docker won't start on latest

2023-10-05 08:14:23,736 - INFO - Template "fakestash-v2" has a new version available in the default-templates directory Traceback (most recent call last): File "/emp_stash_fill/emp_stash_fill.py", line 102, in <module> for k,v in conf["templates"].to_dict(): ValueError: too many values to unpack (expected 2) 2023-10-05 18:49:06,514 - INFO - Template "fakestash-v2" has a new version available in the default-templates directory Traceback (most recent call last): File "/emp_stash_fill/emp_stash_fill.py", line 102, in <module> for k,v in conf["templates"].to_dict(): ValueError: too many values to unpack (expected 2) 2023-10-06 15:14:37,431 - INFO - Template "fakestash-v2" has a new version available in the default-templates directory Traceback (most recent call last): File "/emp_stash_fill/emp_stash_fill.py", line 102, in <module> for k,v in conf["templates"].to_dict(): ValueError: too many values to unpack (expected 2)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.