What volumes should I mount on my docker-compose to persist data about pact_broker-docker HOT 12 CLOSED

I guess you could look at https://github.com/pact-foundation/pact_broker-serverless and use it with EFS (https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/)?

For completeness, there is also https://pactflow.io/ (but I understand that may not be an option for you).

from pact_broker-docker.

Well, you've made me re-evaluate my thoughts on using SQLite for prod. I just did some googling and came across this page https://www.sqlite.org/whentouse.html

Situations Where A Client/Server RDBMS May Work Better
Client/Server Applications
If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite. SQLite will work over a network filesystem, but because of the latency associated with most network filesystems, performance will not be great. Also, file locking logic is buggy in many network filesystem implementations (on both Unix and Windows). If file locking does not work correctly, two or more clients might try to modify the same part of the same database at the same time, resulting in corruption. Because this problem results from bugs in the underlying filesystem implementation, there is nothing SQLite can do to prevent it.
A good rule of thumb is to avoid using SQLite in situations where the same database will be accessed directly (without an intervening application server) and simultaneously from many computers over a network.

High Concurrency
SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writers queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.

Is the data separated from the application by a network? → choose client/server

I guess if your usage is quite light, then the likelihood of having concurrent transactions is quite low. I've been using it as the test database since it was first written, and I know that on start up, I occasionally get "file locked" errors when it gets hit too hard, but you probably wouldn't be starting and stopping it that often. If it's worked for you, then that's great!

from pact_broker-docker.

Yes. There's nothing else stored on the file system.

from pact_broker-docker.

@bethesque Could you also mention what paths need to be mounted as volumes for working with sqlite?

from pact_broker-docker.

Where ever you put your Sqlite database. Only you know that! It's the value that you'll have supplied to thePACT_BROKER_DATABASE_NAME.

from pact_broker-docker.

Awesome. That information would have been nice to have in the readme.

After digging around and poking through the docker image I found that the app itself is under /home/app/pact_broker and I had used the value provided in the readme as the PACT_BROKER_DATABASE_NAME and my problem was that it's a file that doesn't exist and will be created upon first startup and it is in the folder where the rest of the app is so mounting a docker volume is not so straight forward.

Probably a solution would be to put the PACT_BROKER_DATABASE_NAME=/some_sub_folder_to_mount/pact_broker.sqlite

As a workaround I used a binding instead of a volume like so:

version: '3.2'
services:
  pactbroker:
    image: dius/pact-broker
    ports:
      - "7000:80"
    environment:
      - PACT_BROKER_DATABASE_ADAPTER=sqlite
      - PACT_BROKER_DATABASE_NAME=pact_broker.sqlite
      - PACT_BROKER_BASIC_AUTH_USERNAME=*****
      - PACT_BROKER_BASIC_AUTH_PASSWORD=*****
      - PACT_BROKER_BASIC_AUTH_READ_ONLY_USERNAME=****
      - PACT_BROKER_BASIC_AUTH_READ_ONLY_PASSWORD=****
      - PACT_BROKER_PUBLIC_HEARTBEAT=true
    volumes:
      - type: bind
        source: "/data/pact_broker.sqlite"
        target: "/home/app/pact_broker/pact_broker.sqlite"
    restart: always
networks:
  default:
      ipam:
        driver: default
        config:
          - subnet: 10.222.0.0/24

from pact_broker-docker.

Can I ask what problem you're trying to solve? It's not normal to use an SQLite database for ongoing usage for this application. It's only used for running the test suites, or starting up the broker for demo purposes.

from pact_broker-docker.

We've been using it like this for over a year now, without a problem, until for some reason the container restarted a couple of times and we lost contracts published and it was annoying to have to re-run all consumers again to re-publish all the contracts.

I get that this is not the way it was intended, and would love to hear what are the reasons a postgres or mysql would be mandatory......but up until know I thought, if it gets the job done, I don't see why I would have to add the extra cost of infrastructure for a database engine or the hassle to monitor the db to see that it's up.

Tbh, I would love to have the broker as a serverless function and keep the state in a file based db or managed serverless db service if possible.

At some point, it might be the only thing left in my application's landscape that is still requiring a VM to live on. (well, I guess I could use something like aws fargate....but it feels like overkill for just the broker)

from pact_broker-docker.

I've found tools like terraform have exposed concurrency issues when using pact broker backed by sqllite which makes sense given it is going to do a lot of writing. Choose what makes sense for you I guess!

from pact_broker-docker.

I've been getting this error a LOT on my local development machine recently. I don't know how this doesn't happen in production!

16:20:21 E {cognito_sub: 96b88866-8c12-489e-9aec-5e7b3850c2eb} -- SQLite3::BusyException: database is locked: SELECT * FROM `pact_versions` WHERE `id` = 677
16:20:21 E {cognito_sub: 96b88866-8c12-489e-9aec-5e7b3850c2eb} -- Error reference qupXcXPHqI -- Exception: Sequel::DatabaseError: SQLite3::BusyException: database is locked

from pact_broker-docker.

well....context might be different. On your local machine might there by other software that would get a lock on that file? antivirus, IDE, etc....

from pact_broker-docker.

It happens when the React UI recompiles and it refreshes in the browser, so a bunch of requests fire off at the same time. Now I think about it, you're on the old server side rendered UI, so you wouldn't get situations with a heap concurrent requests from a browser. That probably explains it.

from pact_broker-docker.