Giter Club home page Giter Club logo

request-baskets's Introduction

Request Baskets Build Status Coverage Status Go Report Card

Request Baskets is a web service to collect arbitrary HTTP requests and inspect them via RESTful API or simple web UI.

It is strongly inspired by ideas and application design of the RequestHub project and reproduces functionality offered by RequestBin service.

Table of Contents

Introduction

Request Baskets service is available on our demonstration server: https://rbaskets.in

However, we encourage you to set up your own server and keep control over the information passed through and collected by Request Baskets service.

Features

Distinguishing features of Request Baskets service:

  • RESTful API to manage and configure baskets, see Request Baskets API documentation in interactive mode
  • All baskets are protected by unique tokens from unauthorized access; end-points to collect requests do not require authorization though
  • Individually configurable capacity for every basket
  • Pagination support to retrieve collections: basket names, collected requests
  • Configurable responses for every HTTP method
  • Alternative storage types for configured baskets and collected requests:
    • In-memory - ultra fast, but limited to available RAM and collected data is lost after service restart
    • Bolt DB - fast persistent storage for collected data based on embedded bbolt database (maintained fork of Bolt), service can be restarted without data loss and storage is not limited by available RAM
    • SQL database - classical data storage, multiple instances of service can run simultaneously and collect data in shared data storage, which makes the solution more robust and scaleable (PostgreSQL and MySQL are only supported at the moment)
    • Can be extended by custom implementations of storage interface

Screenshots

Basket requests overview: Screenshot of the Request Baskets application showing multiple requests: DELETE, GET and POST. The POST requests shows also the request Body. In this case some JSON.

Configuration of basket responses: Screenshot of the Request Baskets application showing the possibility how a response can be created for a specific HTTP Method and location. The sample Body is a JSON example.

Install

Build from source

Build latest:

$ go get github.com/darklynx/request-baskets

Run:

$ export PATH=$PATH:$GOPATH/bin
$ request-baskets

Run docker container

$ docker pull darklynx/request-baskets
$ docker run -p 55555:55555 darklynx/request-baskets

Configuration

Request Baskets service supports several command line configuration parameters. Use -h or --help to print command line help:

$ request-baskets --help
Usage of bin/request-baskets:
  -db string
      Baskets storage type: "mem" - in-memory, "bolt" - Bolt DB, "sql" - SQL database (default "mem")
  -file string
      Database location, only applicable for file or SQL databases (default "./baskets.db")
  -conn string
      Database connection string for SQL databases, if undefined "file" argument is considered
  -l string
      HTTP listen address (default "127.0.0.1")
  -p int
      HTTP service port (default 55555)
  -page int
      Default page size (default 20)
  -size int
      Initial basket size (capacity) (default 200)
  -maxsize int
      Maximum allowed basket size (max capacity) (default 2000)
  -token string
      Master token, random token is generated if not provided
  -basket value
      Name of a basket to auto-create during service startup (can be specified multiple times)
  -prefix string
      Service URL path prefix
  -mode string
      Service mode: "public" - any visitor can create a new basket, "restricted" - baskets creation requires master token (default "public")
  -theme string
      CSS theme for web UI, supported values: standard, adaptive, flatly (default "standard")

Parameters

List of command line parameters with corresponding ENVVAR for docker container:

  • -p port (PORT) - HTTP service listener port, default value is 55555
  • -l IP address (LISTEN) - HTTP listener IP address, default 127.0.0.1 (docker default: 0.0.0.0)
  • -page size (PAGE) - default page size when retrieving collections
  • -size size (SIZE) - default new basket capacity, applied if basket capacity is not provided during creation
  • -maxsize size (MAXSIZE) - maximum allowed basket capacity, basket capacity greater than this number will be rejected by service
  • -token token (TOKEN) - master token to gain control over all baskets, if not defined a random token will be generated when service is launched and printed to stdout
  • -db type (DB) - defines baskets storage type: mem - in-memory storage (default), bolt - bbolt database (docker default), sql - SQL database
  • -file location (FILE) - location of Bolt database file, only relevant if appropriate storage type is chosen
  • -conn connection (CONN) - database connection string for SQL databases, if undefined -file argument is considered
  • -basket value (BASKET) - name of a basket to auto-create during service startup, this parameter can be specified multiple times
  • -prefix URL path prefix (PATHPREFIX) - allows to host API and web-UI of baskets service under a sub-path instead of domain ROOT
  • -mode mode (MODE) - defines service operation mode: public - when any visitor can create a new basket, or restricted - baskets creation requires master token
  • -theme theme (THEME) - CSS theme for web UI, supported values: standard, adaptive, flatly

Usage

Open http://localhost:55555 in your browser. The main page will display a list of baskets that may be accessed if the basket token is known. It is possible to create a new basket if the name is not in use.

If basket was successfully created the authorization token is displayed. It is important to remember the token because it authorizes the access to management features of created basket and allows to retrieve collected HTTP requests. The token is temporary stored in browser session to simplify UI integration and improve user experience. However, once browser tab is closed, the token will be lost.

To collect HTTP requests send them (GET, POST, PUT, DELETE, etc.) to http://localhost:55555/<basket_name>

To view collected requests and manage basket:

  • Open basket web UI http://localhost:55555/web/<basket_name>
  • Use RESTful API exposed at http://localhost:55555/api/baskets/<basket_name>/...

It is possible to forward all incoming HTTP requests to arbitrary URL by configuring basket via web UI or RESTful API.

Bolt database

By default Request Baskets service keeps configured baskets and collected HTTP requests in memory. This data is lost after service or server restart. However a service can be configured to store collected data on file system. In this case the service can be restarted without loosing created baskets and collected data.

To start service in persistent mode simply configure the appropriate storage type, such as bbolt database (maintained fork of Bolt):

$ request-baskets -db bolt -file /var/lib/request-baskets/baskets.db
2016/01/08 23:15:28 [info] generated master token: abcdefgh1234567...
2016/01/08 23:15:28 [info] using Bolt database to store baskets
2016/01/08 23:15:28 [info] Bolt database location: /var/lib/rbaskets/baskets.db
2016/01/08 23:15:28 [info] starting HTTP server on port: 55555
...

Any other kind of storages or databases (e.g. MySQL, MongoDb) to keep collected data can be introduced by implementing following interfaces: BasketsDatabase and Basket

PostgreSQL database

The first attempt to implement SQL database storage for Request Baskets service is now available for evaluation. Even though the logic to organize the data within SQL database is written in the generic SQL dialect, the code make use of parametrized SQL queries that unfortunately do not have standard to express parameter placeholders across different databases.

Current implementation is based on PostgreSQL syntax. So running Request Baskets service with PostgreSQL database as a storage is fully supported.

Use following example to start the Request Baskets service with PostgreSQL database:

$ request-baskets -db sql -conn "postgres://rbaskets:pwd@localhost/baskets?sslmode=disable"
2018/01/25 01:06:25 [info] generated master token: mSEAcYvpDlg...
2018/01/25 01:06:25 [info] using SQL database to store baskets
2018/01/25 01:06:25 [info] SQL database type: postgres
2018/01/25 01:06:25 [info] creating database schema
2018/01/25 01:06:25 [info] database is created, version: 1
2018/01/25 01:06:25 [info] HTTP server is listening on 127.0.0.1:55555
...

The documentation of Go driver for PostgreSQL provides detailed description of connection string and its parameters.

If no configured instance of PostgreSQL server is available to test the Request Baskets service with, there is a quick way to launch one using Docker with following command:

$ docker run --rm --name pg_baskets -e POSTGRES_USER=rbaskets -e POSTGRES_PASSWORD=pwd \
    -e POSTGRES_DB=baskets -d -p 5432:5432 postgres

# following command will stop and destroy the instance of PostgreSQL container
$ docker stop pg_baskets

MySQL database

Added driver and application support within the SQL basket database for MySQL (or MariaDB) database.

Use following example to start the Request Baskets service with MySQL database:

$ request-baskets -db sql -conn "mysql://rbaskets:pwd@/baskets"
2018/01/28 23:39:59 [info] generated master token: aPgyuLxw723q...
2018/01/28 23:39:59 [info] using SQL database to store baskets
2018/01/28 23:39:59 [info] SQL database type: mysql
2018/01/28 23:39:59 [info] creating database schema
2018/01/28 23:39:59 [info] database is created, version: 1
2018/01/28 23:39:59 [info] HTTP server is listening on 127.0.0.1:55555
...

The documentation of Go driver for MySQL provides detailed description of connection string and its parameters.

If no configured instance of MySQL server is available to test the Request Baskets service with, there is a quick way to launch one using Docker with following command:

$ docker run --rm --name mysql_baskets -e MYSQL_USER=rbaskets -e MYSQL_PASSWORD=pwd \
    -e MYSQL_DATABASE=baskets -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d -p 3306:3306 mysql

# following command will stop and destroy the instance of MySQL container
$ docker stop mysql_baskets

Docker

Build docker image

$ docker build -t request-baskets .

This will create a docker image using multi-stage docker builds approach with 2 stages: compiling the service and packaging the result to a tiny alpine container. The resulting size of built image is ~12 Mb.

Note: since the first stage is using golang:latest container to build the service executable, there is no need to have Go lang SDK installed on the machine where process of building the container is taking place.

See docker folder for alternative docker builds with detailed explanation of the process for every variant.

Run container as a service

$ docker run --name rbaskets -d -p 55555:55555 request-baskets
$ docker logs rbaskets

Cleanup

Stop and delete docker container:

$ docker stop rbaskets
$ docker rm rbaskets

Delete docker image:

$ docker rmi request-baskets

request-baskets's People

Contributors

darklynx avatar fergusean avatar gregorwolf avatar jackymancs4 avatar lavode avatar nextrevision avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

request-baskets's Issues

Add allow/deny feature for forward URLs

we operate the application in kubernetes and want to avoid access to internal ressources via request-baskets, as it can be turned into an open proxy. the most flexible use is probably to provide the ability to allow/deny domains and ip ranges.

example use cases:

  • request-baskets --allow-domain example.com --deny-domain dev.example.com
    baskets can only be configured with URLs with example.com
    as domain (e.g. https://api.prod.example.com),
    except URLs pointing to dev.example.com
    (e.g. https://api.dev.example.com)

  • request-baskets --deny-domain cluster.local
    baskets can forward any URL except to URLS under the
    cluster.local domain

  • request-baskets --deny-address 127.0.0.0/8
    baskets can not forward URLs pointing to the loopback
    interface

  • request-baskets --deny-private-address
    shortcut for denying forwards to all addresses designated
    for private use (192.168.0.0/16, 172.16.0.0/12,
    10.0.0.0/8) by the IANA

  • request-baskets --allow-address 192.168.1.0/24 --deny-address 192.168.1.1/32 --deny-address 192.168.1.254/32
    only URLs resolving to a class B private net are allowed,
    except 192.168.1.1 and 192.168.1.254

by default, everything is allowed and nothing is denied (just as
before). denials are applied after allowances, so that you
can allow on a broad scale and deny granular.

given that the validation only has to be performed on basket
creation/updates, the resource overhead for domain resolution
should be relatively low.

Handling Multipart

Could multipart/form-data be handling in a better way? Showing the content is fine, but maybe provide the option to download the files?

image

SSL Support

Hi, Mr.Vladimir L. Thank you for sharing. May I ask how to add and set an SSL certificate(Let's encrypt)?

Swagger spec warning

GET https://raw.githubusercontent.com/darklynx/request-baskets/master/doc/api-swagger.yaml
- code: EXTRA_REFERENCE_PROPERTIES
  message: 'Extra JSON Reference properties will be ignored: description'
  path:
    - definitions
    - Request
    - properties
    - headers

you can solve this by wrapping $ref with allOf.

allOf:
  $ref: '#/...'

Allow proxying responses when forwarding

When setting a forward URL, have an additional setting that allows for the request to complete and pass the response back to the client. I have this minimally setup in my fork, but would probably need to do some work to make it scale.

If this is something you're interested in, I can polish up my fork and submit a PR.

[Feature] Support create basket and configure response

We can provision a basket with BASKET param, but the response config is only in interface/api. To docker containers make senses configure automatically responses to a provisioned basket.

Something like:

request-baskets -basket myapi -response ./response-config.json

Bootstrap Upgrade/Dark Mode

Hi!

Currently the site is running off Bootstrap 3. If the site is upgraded to Bootstrap 5+, dark mode could be easily added. In a recent bootstrap update, dark-versions of all the main components have been added, and can be toggled with an attribute on the body element.

I also want to thank you for making such an amazing project.

Allow hosting the webapp as non-root

Hi, and thanks for this software! It's extremely useful.
I'm facing the following problem: I want to put request-baskets under a reverse proxy (e.g. NGINX).
I hold a domain (let's say example.org) and I want to host request-baskets under a subpath of my domain (e.g. https://example.org/baskets/).

I can use a proxy_pass directive with a rewrite so that request-baskets will receive a request similar to 127.0.0.1/web instead of 127.0.0.1/baskets/web. However, it will generate links and redirects such as /web instead of /baskets/web. This breaks the user experience noticeably. I can still host under a subdomain and avoid this problem, but this is not my preferred solution.

Would it be possible to add an option to relativise all URLs or to prepend a custom string to all the endpoints?

basket expiration

Feature Description

It would be great, if a basket could be configured with an expiration time or lifespan. After expiration, the basked would be scheduled for deletion. This could be beneficial to in-memory request storage as automatic clean-up.

The idea is inspired by the 30 minute expiration of https://postb.in/.

Impact on Existing User Workflows

I have not yet checked the code (I can read it, but would not call myself a Gopher) to give an idea where this feature would have impacts and therefore could cause some concerns.
Regarding interfaces, I could imagine to have the possibilities to give a default value for all baskets on the server config, and also pass a basket specific value on the creation request. The expiration time should also appear in the basket info which would add an extra field to the basket response model. Logically, the backward compatible default would be a never value.

Restrict creation of new baskets

Would it be possible to restrict creation of new baskets (and potentially restrict all access to the /api and /web paths) to clients that present the main token? This would potentially make it easier to host Request Baskets on public-facing servers. Right now anyone who visits the domain it is hosted on can immediately create an unlimited number of baskets.

Allow for storing and displaying responses

Optionally allow for storing and displaying response data like headers and content. I'm using this with the response proxy. Not sure if you want to support something like this, but if so, I can polish up my fork and submit a PR.

Error installing version 1.2.0

Installing version v1.2.0 results in the following error, version v1.1.0 works as expected

$ go install github.com/darklynx/request-baskets@latest
go: downloading github.com/darklynx/request-baskets v1.2.0
go: github.com/darklynx/request-baskets@latest: github.com/darklynx/[email protected]: parsing go.mod:
	module declares its path as: darklynx/request-baskets
	        but was required as: github.com/darklynx/request-baskets

Migrate to go modules

Hello @darklynx
I really like your implementation of a request bin.
I'm planning to add a some features that I need, meant mainly for personal use in my own fork but that you may like as well.

Instrumental to this is updating the codebase to use the modern Go Modules (https://blog.golang.org/migrating-to-go-modules) instead of go packages.

I have a PR ready to be submitted if you are interested, the content of which can be seen https://github.com/Jackymancs4-Fork/request-baskets/tree/33895a324ebac4bc2086d7eb8778264a70667225

In that repo I replaced Travis CI with Github Actions worklows, with all tests passing.
If you're willing, this can be another issue.

Anyway, thank you.

add ability to get full raw GET parameters in the "GET" green header

Currently the green GET header displays a truncated GET request only.

image

In some situations it is necessary to copy+paste the whole request, and currently this is only possible by manually copying the parsed parameters as seen in the "Query Params" box.

A suggestion: why not make the GET header expandable, so that if the user clicks on it, it shows the complete GET request as received by the basket?

Host header is missing from the Headers section

When I send a request with a different Host header then the request is logged successfully but the Host is missing.

curl -H "Host: example.com" https://rbaskets.in/3vgdn65

Shows:

image

Testing with webhooks.site the host header is listed:

image

This prevents debugging host-related issues.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.