Giter Club home page Giter Club logo

expanse's Introduction

expanse

selfhosted multi-user web app for externally storing Reddit items (saved, created, upvoted, downvoted, hidden) to bypass Reddit's 1000-item listing limits

  • features::
    • new items auto-sync
    • synced items not affected by Reddit deletion
    • search for items
    • filter by subreddit
    • unsave/delete/unvote/unhide items from Reddit directly on expanse
    • import csv data from Reddit data request
    • export data as json
  • demo
  • requirements::
  • setup::
    1. clone repo: git clone https://github.com/aplotor/expanse.git
    2. cd into repo: cd ./expanse/
    3. create environment variables file: cp ./backend/.env_example ./backend/.env_prod
    4. fill out the values in the ./backend/.env_prod file
  • usage::
    1. cd into repo
    2. start: sh ./run.sh prod up (stop: sh ./run.sh prod down)
    3. go to http://localhost:1301 / http://{host ip}:1301
  • updating::
    1. cd into repo
    2. update: sh ./run.sh prod update
  • (optional) systemd service
    1. cd into repo
    2. update repo path in service file: sed -i "s./opt/expanse.$PWD.g" ./expanse.service
    3. allow executing ./run.sh: chmod +x ./run.sh
    4. enable and start service: systemctl enable ./expanse.service --now

expanse's People

Contributors

aplotor avatar johschilleremnify avatar keagel avatar qtfuta avatar sugoidogo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

expanse's Issues

Unable to authenticate with Reddit

Super excited to try this out! Thanks for making this happen! I have, though, run into a setup issue...

  1. Visit https://local.domain.com/
  2. Click Login with Reddit
  3. Click allow from Reddit
  4. Returned to https://local.domain.com/logout#_

Some notes on my configuration

  • My user is in the approved list in the .env_prod file.
  • I am using a subdomain on a domain that I own that is proxied through Nginx Proxy Manager, which is also listed in the config file for the callback URL and is used on Reddit as the callback URL. This domain name is not accessible to the external network.

I am seeing the following in the logs

From the database:

2022-09-11 20:50:17.781 UTC [69] LOG:  incomplete startup packet,
2022-09-11 20:50:17.450 UTC [1] LOG:  database system is ready to accept connections,
2022-09-11 20:50:17.446 UTC [62] LOG:  database system was shut down at 2022-09-11 20:50:17 UTC,
2022-09-11 20:50:17.440 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2022-09-11 20:50:17.437 UTC [1] LOG:  listening on IPv6 address "::", port 5432,
2022-09-11 20:50:17.437 UTC [1] LOG:  starting PostgreSQL 14.5 (Debian 14.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit,
2022-09-11 20:50:17.437 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432

From the app:

server (expanse) started on (localhost:1301)
saved user (reddituser),
checking if user (reddituser) exists,
purged user (reddituser),
denied user (reddituser),
socket (WDZoBrdZVFDhSc-1AAAL) connected,
new user (reddituser),
socket (tk5NIOso-aftuQw4AAAJ) connected,

I've tried to figure out where localhost:1301 is being set, but can't find it. I have found that that can cause issues if it doesn't match the domain that I a using on the network. It is updated in the .env_prod file. This may not be related to this particular issue, but thought it wroth mentioning.

Let me know what other questions you have. At the moment, this is all the info that I can find on the issue. And I fully recognize that this very well could be a pebkac errror. Any help is greatly appreciated.

Expanse stuck at 57%

I've installed Expanse on Ubuntu and authorized the application. When I go to the Expanse webpage, I click on "log in with reddit" and I am taken to the page where I connect expanse to my reddit account.

The page proceeds back to the expanse webpage and then the percentage on the screen increases gradually till it gets to 57%. It stays stuck at 57%. I have tried this on 3 separate Ubuntu installations with the same result.

Of note, I have 2 other reddit accounts that I used with expanse and that worked perfectly on the 3 Ubuntu installations.

Is there a log that I could look at for errors?

Thanks!

Error in backend

Getting error while opening localhost:1301

expanse_frontend_1 exited with code 0
backend_1   |
backend_1   | > prod
backend_1   | > node ./controller/server.mjs
backend_1   |
backend_1   | update all completed
backend_1   | server (expanse) started on (localhost:1301)
backend_1   | [Error: ENOENT: no such file or directory, stat '/backend/build/index.html'] {
backend_1   |   errno: -2,
backend_1   |   code: 'ENOENT',
backend_1   |   syscall: 'stat',
backend_1   |   path: '/backend/build/index.html',
backend_1   |   expose: false,
backend_1   |   statusCode: 404,
backend_1   |   status: 404
backend_1   | }
backend_1   | TypeError: Cannot read properties of undefined (reading 'user')
backend_1   |     at file:///backend/controller/server.mjs:74:42
backend_1   |     at Layer.handle_error (/backend/node_modules/express/lib/router/layer.js:71:5)
backend_1   |     at trim_prefix (/backend/node_modules/express/lib/router/index.js:326:13)
backend_1   |     at /backend/node_modules/express/lib/router/index.js:286:9
backend_1   |     at Function.process_params (/backend/node_modules/express/lib/router/index.js:346:12)
backend_1   |     at next (/backend/node_modules/express/lib/router/index.js:280:10)
backend_1   |     at /backend/node_modules/express/lib/response.js:455:7
backend_1   |     at SendStream.onerror (/backend/node_modules/express/lib/response.js:1071:5)
backend_1   |     at SendStream.emit (node:events:513:28)
backend_1   |     at SendStream.error (/backend/node_modules/send/index.js:270:17)
backend_1   |     at SendStream.onStatError (/backend/node_modules/send/index.js:417:12)
backend_1   |     at onstat (/backend/node_modules/send/index.js:722:26)
backend_1   |     at FSReqCallback.oncomplete (node:fs:206:21)

Embedded database support

While browsing this repo I noticed from the compose.prod.yaml that expanse is a node.js application who's only dependency outside the NodeJS ecosystem is postgres, a database application. It looks to me like docker is used to run an instance of postgres as a dependency of expanse, but npm also has various embedded database libraries, and there's even embedded-postgres. Using one of these would allow the removal of docker as a requirement, which opens up many more options for packaging expanse, especially using existing automated tools for nodejs packaging. Using an embedded database by default and allowing the user to configure an external database at their preference provides a much higher degree of portability to the software that self-hosters like myself would much appreciate.

401 Error after trying authentication. Not sure what is the error.

image

here is the log of the expanse-app-1

> prod
> RUN=prod PORT=1301 node ./controller/server.mjs

update all started
update all completed
server (expanse) started on (localhost:1301)
socket (_lug_mR7qI5qoR3IAAAC) connected
socket (vYQ9_v4VIkiSWYdzAAAD) connected
socket (6_Zt4bbwk35zzZRuAAAF) connected
socket (SD8kRbeB_f_Hb7iUAAAH) connected
update all started
update all completed
socket (iy4uL8s43FAQKQcxAAAJ) connected
update all started
update all completed
update all started
update all completed
update all started
update all completed
update all started
update all completed
socket (TYvYWW6APesuVAPpAAAL) connected
update all started
update all completed
wait-for-it: waiting for db:5432 without a timeout
wait-for-it: db:5432 is available after 0 seconds

and here is the log of expanse-db-1

2023-07-25 19:39:26.843 UTC [1] LOG:  starting PostgreSQL 14.8 (Debian 14.8-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-07-25 19:39:26.844 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-07-25 19:39:26.844 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-07-25 19:39:26.848 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-07-25 19:39:26.854 UTC [27] LOG:  database system was shut down at 2023-07-25 19:38:32 UTC
2023-07-25 19:39:26.859 UTC [1] LOG:  database system is ready to accept connections
2023-07-25 19:39:27.227 UTC [34] LOG:  incomplete startup packet

PostgreSQL Database directory appears to contain a database; Skipping initialization

401 after OAuth

After I accept OAuth dialog, I get 401 returning to the app.
image

error getting credentials - err: exit status 1, out: ``

sh ./run.sh prod up

results in

[+] Running 0/2
 ⠏ db Pulling                                                              6.0s
[+] Running 0/2                                                            6.0s
 ⠋ db Pulling                                                              6.0st ⠋ app Pulling                                                             6.0s
error getting credentials - err: exit status 1, out: ``
./run.sh: line 28: return: can only `return' from a function or sourced script

Any idea what's going on?

Configure @db and @port

Hello,
Is it possible to make the connectionString configurable with environnement variable to specify db address and db port if we want to use an existing postgres database ?

Regards

Unable to run & .yaml files getting corrupted

Running VERSION=1.2.10
Used to run fine until got the latest version (hadn't updated in a while).
The moment I launch, the compose.prod.yaml gets corrupted with bash text (like linux permission, prompts, etc.)

Just curious, anyone else having issues?

[Feature] Store post content

Storing post content would protect against loosing access to deleted media, 3rd party sites taking down content etc.

Ideally it would be based on a plugin system where its quite easy to create a content extractor for new types of links/posts, with default ones for text and Reddit images/galleries.

Alternatively, or ideally as a plugin for the proposed content storage: Ability to Internet Archive/Wayback Machine links from posts and comments.

Awarded category no longer retrievable

I noticed that my content hadn't updated in a few days. When I looked at the logs for the application, I saw this error happening every time it went to update.

StatusCodeError: 404 - {"message":"Not Found","error":404}
    at new StatusCodeError (/app/backend/node_modules/request-promise-core/lib/errors.js:32:15)
    at /app/backend/node_modules/request-promise-core/lib/plumbing.js:97:41
    at tryCatcher (/app/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/app/backend/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/app/backend/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/app/backend/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/app/backend/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/app/backend/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/app/backend/node_modules/bluebird/js/release/async.js:102:5)
    at Async.drainQueues [as _onImmediate] (/app/backend/node_modules/bluebird/js/release/async.js:15:14)
    at process.processImmediate (node:internal/timers:476:21) {
  statusCode: 404,
  error: { message: 'Not Found', error: 404 },
  options: {
    gzip: true,
    json: true,
    headers: {
      'user-agent': 'web:expanse:v=1.2.11 (hosted by u/<username>)'
    },
    baseUrl: 'https://oauth.reddit.com',
    qs: { raw_json: 1, count: 9999, limit: 5 },
    auth: {
    },
    resolveWithFullResponse: true,
    timeout: 30000,
    transform: [Function: transform],
    uri: 'u/<username>/gilded/given',
    method: 'GET',
    callback: [Function: RP$callback],
    simple: true,
    transform2xxOnly: false
  },

The /gilded/given uri is what feeds the awarded category. It appears that this retrieval is now failing and causing the content to not update. I believe this has something to do with the rework to Reddit Gold that they announced a few days before I first saw this issue.

https://support.reddithelp.com/hc/en-us/articles/360043034412-What-is-a-Reddit-premium-membership-

It seems like this app may need to remove the awarded category.

After some time I get a 401 error and I need to relogin and reimport everything

When I'm not visiting my expanse instance in a while I get an 401 error and I have to login again. After logging back in all my before saved posts are gone.
These are the logs from the console:

getting user (myusername)
deserialize error (myusername)
Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
destroyed session (myusername)

I haven't changed my password in the meantime or changed anything else on my reddit account. This is also not the first time that this is happening

Cannot create environment variable files in step iii of setup

The third step in the setup "create environment variables file: cp ./backend/.env_example ./backend/.env_prod" does not do anything. No new environment variable file is created no matter how many time I put in the command. I am using macOs Monterey 12.6.8 and Terminal for the setup. For the requirements, I have git via Xcode and Docker Desktop installed.

Expanse isnt being updated

After importing my reddit csv files, docker has been running for the past week without updating expanse. I've tried restarting the docker container and it continues to import. There is maybe 50,000 links from all the cvs imported.

Frontend Container never runs

Just tried to clone the repo and follow the steps in your readme, however the frontend container never seems to come up.

I've tried resetting and manually running commands from your run.sh script but it still is not successful. The frontend container never starts and shows no logs. The build and run process using your script has the error. Different line numbers are shown based on the run.sh command you use

./run.sh: line 30: return: can only `return' from a function or sourced script

Save scores and perform analytics

Hey, I've been using your app to save explanatory posts from reddit that I write. I'm active in homelab, selfhosted, networking, other IT subs. I miss the feature to order by score. It's quite an important thing because some of my explanations are article-like and gather higher scores.

I've came across this site: https://reddit-user-analyser.netlify.app/

It analyzes all available comments and provides analytics - activity, scores, wording, readability. It's a nice thing to have, especially when it's combined with expanse's long-term storage, beyond the 1000 item mark.

www.reddit.com/prefs/apps does not exist

From ./backend/.env_prod:

# go to https://www.reddit.com/prefs/apps ➔ "create another app..." ➔ name: "expanse", type: "web app", redirect uri: "http://localhost:1301/callback". then fill in these values

However this URL doesn't seem to exist (redirects you to reddit) and looking through "Settings" on reddit I am not seeing anything that allows me to create an app.

Expanse data is not beeing updated

After some time I've run into an issue where the data is not getting updated anymore: last updated: 15h ago. I've imported my Reddit data with csv files which lead to an overall saved count of 4410. Also, my pihole fetched about 38000 requests to oauth.reddit.com.
In the console I got this error while it's trying to update the data:

RequestError: Error: ETIMEDOUT
    at new RequestError (/backend/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/backend/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/backend/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/backend/node_modules/request/request.js:185:22)
    at Request.emit (node:events:513:28)
    at Timeout.<anonymous> (/backend/node_modules/request/request.js:851:16)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  cause: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
  error: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
�      
  options: {
    gzip: true,
    json: true,
    headers: { 'user-agent': 'web:expanse:v=1.0.0 (hosted by u/myusername)' },
    baseUrl: 'https://oauth.reddit.com',
�      
{
      raw_json: 1,
      count: 9999,
      limit: 9007199254740991,
      before: 't3_xazslb'
    },
    auth: { bearer: 'hidden for privacy' },
    resolveWithFullResponse: true,
    timeout: 30000,
    transform: [Function: transform],
    uri: 'user/myusername/upvoted',
    method: 'GET',
    callback: [Function: RP$callback],
    simple: true,
    transform2xxOnly: false
  },
  response: undefined
}

At the moment I try to recreate the error:

I've imported csv files to it (I think all of the possible ones). After that it didn't updated anymore.

localhost vs local IP

Hello!
I just gave this a try and I am running the project on my headless server.

When I follow your directions, it is using "localhost" so when I try to access that from the laptop and not the main server machine, it obviously doesn't work. When connecting reddit to it, I had to replace the callback url so that it used the local IP and not "localhost" like the project is setup to do.

Should I open PR with a modified readme that includes a disclaimer? Are there steps in the chain where this could be a variable that the user sets and the default is just "localhost"?

Sync stopped working from 14/09/2023. Possibly related to Awarded category API changes

I am not sure if this is related to the API changes or not but the app stopped syncing data.
I have verified that the app credentials are correct.

App Logs below:

getting user (REDACTED)
updating user (REDACTED)
StatusCodeError: 404 - {"message":"Not Found","error":404}
    at new StatusCodeError (/app/backend/node_modules/request-promise-core/lib/errors.js:32:15)
    at /app/backend/node_modules/request-promise-core/lib/plumbing.js:97:41
    at tryCatcher (/app/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/app/backend/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/app/backend/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/app/backend/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/app/backend/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/app/backend/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/app/backend/node_modules/bluebird/js/release/async.js:102:5)
    at Async.drainQueues [as _onImmediate] (/app/backend/node_modules/bluebird/js/release/async.js:15:14)
    at process.processImmediate (node:internal/timers:471:21) {
  statusCode: 404,
  error: { message: 'Not Found', error: 404 },
  options: {
    gzip: true,
    json: true,
    headers: { 'user-agent': 'web:expanse:v=1.2.9 (hosted by u/REDACTED)' },
    baseUrl: 'https://oauth.reddit.com',
    qs: { raw_json: 1, count: 9999, limit: 5 },
    auth: {
      bearer: REDACTED
    },
    resolveWithFullResponse: true,
    timeout: 30000,
    transform: [Function: transform],
    uri: 'u/REDACTED/gilded/given',
    method: 'GET',
    callback: [Function: RP$callback],
    simple: true,
    transform2xxOnly: false
  },
  response: <ref *1> IncomingMessage {
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: [],
      flowing: true,
      ended: true,
      endEmitted: true,
      reading: false,
      constructed: true,
      sync: false,
      needReadable: false,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      destroyed: true,
      errored: null,
      closed: true,
      closeEmitted: true,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      dataEmitted: true,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: false
    },
    _events: [Object: null prototype] {
      end: [Array],
      close: [Array],
      data: [Function (anonymous)],
      error: [Function (anonymous)]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'oauth.reddit.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'oauth.reddit.com',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [ClientRequest],
      timeout: 30000,
      write: [Function: writeAfterFIN],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 909262,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: Timeout {
        _idleTimeout: -1,
        _idlePrev: null,
        _idleNext: null,
        _idleStart: 36558288,
        _onTimeout: null,
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: true,
        [Symbol(refed)]: false,
        [Symbol(kHasPrimitive)]: false,
update all completed        [Symbol(asyncId)]: 909273,
        [Symbol(triggerId)]: 909272

      },
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: false,
      [Symbol(kSetKeepAliveInitialDelay)]: 0,
      [Symbol(kBytesRead)]: 2232,
      [Symbol(kBytesWritten)]: 1276,
      [Symbol(connect-options)]: [Object]
    },
    httpVersionMajor: 1,
    httpVersionMinor: 1,
    httpVersion: '1.1',
    complete: true,
    rawHeaders: [
      'Connection',
      'close',
      'Content-Length',
      '38',
      'x-ua-compatible',
      'IE=edge',
      'expires',
      '-1',
      'cache-control',
      'private, s-maxage=0, max-age=0, must-revalidate, no-store',
      'x-ratelimit-remaining',
      '885',
      'x-ratelimit-used',
      '111',
      'x-ratelimit-reset',
      '187',
      'content-type',
      'application/json; charset=UTF-8',
      'x-moose',
      'majestic',
      'Accept-Ranges',
      'bytes',
      'Date',
      'Fri, 22 Sep 2023 07:56:54 GMT',
      'Via',
      '1.1 varnish',
      'Strict-Transport-Security',
      'max-age=31536000; includeSubdomains',
      'X-Content-Type-Options',
      'nosniff',
      'X-Frame-Options',
      'SAMEORIGIN',
      'X-XSS-Protection',
      '1; mode=block',
      'Set-Cookie',
      'loid=0000000000000n3ii0.2.1429864439786.Z0FBQUFBQmxEVWpGOXh5YlVLY1ZVQlZNejVNazlBaEFSQktOX2ZobHgwWGNuZkpya2VNV3ZiMm1pblc3a09rWlBva2lESUR5QUczMDNpOWtyeVp2UjJxY2ozLWt0UDZvRXlnbnpZYVdjVmpEWWhNZ0dJN19EYTQ3SDlqMUlZQ1A1dTRiSUh6cXpXeU8; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sun, 21-Sep-2025 07:56:53 GMT; secure; SameSite=None; Secure',
      'Set-Cookie',
      'session_tracker=aifpaeajkddcgelrlb.0.1695369413844.Z0FBQUFBQmxEVWpGVGlKNlRTMUpGLWpNdk95OGExN3ljMFcyQk11MkIzTUFEam8wNVBWNEowNjlBTkFMdENYV2RJODVQZEJmeUpLYnlpV25mMGRGQ0UyUmFsb2M0dlR2UDlBWUM2bEdIS0d1a2lYa3Rta190czlKNDVLVHgySmRhdmxpNmhRVGVaSTc; Domain=reddit.com; Max-Age=7199; Path=/; expires=Fri, 22-Sep-2023 09:56:53 GMT; secure; SameSite=None; Secure',
      'Set-Cookie',
      'redesign_optout=true; Domain=reddit.com; Max-Age=94607999; Path=/; expires=Mon, 21-Sep-2026 07:56:53 GMT; secure',
      'Set-Cookie',
      'csv=2; Max-Age=63072000; Domain=.reddit.com; Path=/; Secure; SameSite=None',
      'Set-Cookie',
      'edgebucket=5ROveAagHA7U5mmgDi; Domain=reddit.com; Max-Age=63071999; Path=/;  secure',
      'Server',
      'snooserv',
      'Report-To',
      '{"group": "w3-reporting-nel", "max_age": 14400, "include_subdomains": true,  "endpoints": [{ "url": "https://w3-reporting-nel.reddit.com/reports" }]}, {"group": "w3-reporting", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting.reddit.com/reports" }]}, {"group": "w3-reporting-csp", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting-csp.reddit.com/reports" }]}',
      'NEL',
      '{"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 1.0, "failure_fraction": 1.0}'
    ],
    rawTrailers: [],
    aborted: false,
    upgrade: false,
    url: '',
    method: null,
    statusCode: 404,
    statusMessage: 'Not Found',
    client: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'oauth.reddit.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'oauth.reddit.com',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [ClientRequest],
      timeout: 30000,
      write: [Function: writeAfterFIN],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 909262,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: Timeout {
        _idleTimeout: -1,
        _idlePrev: null,
        _idleNext: null,
        _idleStart: 36558288,
        _onTimeout: null,
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: true,
        [Symbol(refed)]: false,
        [Symbol(kHasPrimitive)]: false,
        [Symbol(asyncId)]: 909273,
        [Symbol(triggerId)]: 909272
      },
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: false,
      [Symbol(kSetKeepAliveInitialDelay)]: 0,
      [Symbol(kBytesRead)]: 2232,
      [Symbol(kBytesWritten)]: 1276,
      [Symbol(connect-options)]: [Object]
    },
    _consuming: true,
    _dumped: false,
    req: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'GET /user/REDACTED/gilded/given?raw_json=1&count=9999&limit=5 HTTP/1.1\r\n' +
        'user-agent: web:expanse:v=1.2.9 (hosted by u/REDACTED)\r\n' +
        'authorization: Bearer REDACTED' +
        'accept-encoding: gzip, deflate\r\n' +
        'accept: application/json\r\n' +
        'referer: https://oauth.reddit.com/u/REDACTED/gilded/given?raw_json=1&count=9999&limit=5\r\n' +
        'host: oauth.reddit.com\r\n' +
        'content-length: 0\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/user/REDACTED/gilded/given?raw_json=1&count=9999&limit=5',
      _ended: true,
      res: [Circular *1],
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'oauth.reddit.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: true,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(kUniqueHeaders)]: null
    },
    request: Request {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      gzip: true,
      headers: [Object],
      resolveWithFullResponse: true,
      timeout: 30000,
      transform: [Function: transform],
      uri: [Url],
      method: 'get',
      readable: true,
      writable: true,
      explicitMethod: true,
      _qs: [Querystring],
      _auth: [Auth],
      _oauth: [OAuth],
      _multipart: [Multipart],
      _redirect: [Redirect],
      _tunnel: [Tunnel],
      _rp_resolve: [Function (anonymous)],
      _rp_reject: [Function (anonymous)],
      _rp_promise: [Promise [Object]],
      _rp_callbackOrig: undefined,
      callback: [Function (anonymous)],
      _rp_options: [Object],
      setHeader: [Function (anonymous)],
      hasHeader: [Function (anonymous)],
      getHeader: [Function (anonymous)],
      removeHeader: [Function (anonymous)],
      localAddress: undefined,
      pool: {},
      dests: [],
      __isRequestRequest: true,
      _callback: [Function: RP$callback],
      proxy: null,
      tunnel: true,
      setHost: true,
      originalCookieHeader: undefined,
      _disableCookies: true,
      _jar: undefined,
      port: 443,
      host: 'oauth.reddit.com',
      url: [Url],
      path: '/user/REDACTED/gilded/given?raw_json=1&count=9999&limit=5',
      _json: true,
      httpModule: [Object],
      agentClass: [Function: Agent],
      agent: [Agent],
      href: 'https://oauth.reddit.com/user/REDACTED/gilded/given?raw_json=1&count=9999&limit=5',
      ntick: true,
      timeoutTimer: null,
      response: [Circular *1],
      originalHost: 'oauth.reddit.com',
      originalHostHeaderName: 'host',
      _started: true,
      req: [ClientRequest],
      responseContent: [Circular *1],
      _destdata: true,
      _ended: true,
      _callbackCalled: true,
      [Symbol(kCapture)]: false
    },
    toJSON: [Function: responseToJSON],
    caseless: Caseless { dict: [Object] },
    body: { message: 'Not Found', error: 404 },
    [Symbol(kCapture)]: false,
    [Symbol(kHeaders)]: {
      connection: 'close',
      'content-length': '38',
      'x-ua-compatible': 'IE=edge',
      expires: '-1',
      'cache-control': 'private, s-maxage=0, max-age=0, must-revalidate, no-store',
      'x-ratelimit-remaining': '885',
      'x-ratelimit-used': '111',
      'x-ratelimit-reset': '187',
      'content-type': 'application/json; charset=UTF-8',
      'x-moose': 'majestic',
      'accept-ranges': 'bytes',
      date: 'Fri, 22 Sep 2023 07:56:54 GMT',
      via: '1.1 varnish',
      'strict-transport-security': 'max-age=31536000; includeSubdomains',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'SAMEORIGIN',
      'x-xss-protection': '1; mode=block',
      'set-cookie': [Array],
      server: 'snooserv',
      'report-to': '{"group": "w3-reporting-nel", "max_age": 14400, "include_subdomains": true,  "endpoints": [{ "url": "https://w3-reporting-nel.reddit.com/reports" }]}, {"group": "w3-reporting", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting.reddit.com/reports" }]}, {"group": "w3-reporting-csp", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting-csp.reddit.com/reports" }]}',
      nel: '{"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 1.0, "failure_fraction": 1.0}'
    },
    [Symbol(kHeadersCount)]: 50,
    [Symbol(kTrailers)]: null,
    [Symbol(kTrailersCount)]: 0
  },
  extras: { category: 'awarded' }
} ```

[Feature] Plugin system

I have an idea to include some sort of plugin system to process the posts that are being saved.

This way expanse can be decoupled of some features users might want to add and focus on only storing data directly retrieved from Reddit's API.

For example, saving images of posts, a plugin could be implemented for whatever image storage service a user might want to use (to disk, imgur, imgbb, any other cloud provider, etc)
Other might be retrieve and save the content of any link.

The plugins could be registered for all post being saved (i.e. inserts to item table), or for users to decide which ones they want to enable for their posts (i.e. inserts to user_item table)

Content isn't getting updated

The content of my user accounts stopped getting updated after a day. I set it up on Wednesday and the last update occurred on Thursday. Here's an excerpt of the logs:

last Thursday at 9:49:37 PM updating user (Keagel)
last Thursday at 9:49:39 PM updated user (Keagel)
last Friday at 11:40:53 AM backed up db to file (23-09-2022_09꞉40꞉53꞉AM_UTC.sql)
last Saturday at 11:40:53 AM backed up db to file (24-09-2022_09꞉40꞉53꞉AM_UTC.sql)
yesterday at 11:40:53 AM backed up db to file (25-09-2022_09꞉40꞉53꞉AM_UTC.sql)
today at 11:27:37 AM socket (1W4pcrj0y8-iKP_OAACZ) connected
today at 11:27:37 AM getting user (Keagel)
today at 11:27:37 AM deserialized user (Keagel)

Restarting the container forces the updates though.

Originally posted by @Keagel in #1 (comment)

Use same api structure as reddit

I know this is a tall order but...

If you refactored the backend api endpoints to mirror the existing reddit api, at least as a subset of the api, developers could use your application as a proxy. Basically as a cache using the same api. Would enable existing applications to leverage your app with only a URL change.

Container exiting

When I try to run the container, it will stop with the following log:

> docker
> node -r dotenv/config controller/server.mjs dotenv_config_path=/data/.env_prod
node:internal/process/esm_loader:108
    internalBinding('errors').triggerUncaughtException(
                              ^
Error: getaddrinfo ENOTFOUND undefined
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'undefined'
}
Node.js v19.2.0

Database is up & reachable, volume & env-file are available & readable, too. Am I missing something here?

[Config] Using expanse with podman

Bringing up the containers with podman-compose works as intended.
However, expanse_db_1 throws an error on start: "FATAL: zero-length delimited identifier at or near """" at character 12".
This is due to the quotation marks around the three "POSTGRES" lines in the .env_prod
Remove the quotation marks and everything work perfectly.

I've verified, that the quotation marks are also not relevant in a docker context. It works with and without them, so they could be removed entirely from .env_prod which would make expanse work with podman and docker.

Instruction Confusion

Hi any and all,
When I get to filling out env_prod's values I am somewhat confused. I have docker desktop for windows, and docker compose in that, but when i attempt to open env_prod to fill in what is needed no applications seem to want to open it. I am unsure how to proceed from there, as all the previous steps have been followed successfully. Any help in understanding my misstep would be greatly appreciated.

Sincerely,
~Marytha

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.