hoppscotch / proxyscotch Goto Github PK

Hello, I would like to ask how I use docker to deploy on a self-built host, because I want to manage users and their own related data, thank you

Running ./build.sh linux fails on Linux with the error

cp: cannot stat 'resources/linux': No such file or directory

Tested on Arch Linux.

It would seem (according to an email report) that upstream self-signed certificates do not work, this needs to be checked.

Then, we need to figure out whether self-signed or invalid certificates should be allowed by default. I don’t see a security risk from doing that but it might make debugging harder or more confusing because we would bypass those issues.

I’m thinking the best option is to add it behind a flag.

➜ /home/user_00 >j proxy
/home/web/postwoman-proxy
➜ /home/web/postwoman-proxy git:(master) >pwd
/home/web/postwoman-proxy
➜ /home/web/postwoman-proxy git:(master) >export GOPATH=/home/web/postwoman-proxy:$GOPATH
➜ /home/web/postwoman-proxy git:(master) >./build.sh linux server
server/server.go:6:2: cannot find package "postwoman.io/proxy/libproxy" in any of:
/usr/local/go/src/postwoman.io/proxy/libproxy (from $GOROOT)
/home/web/postwoman-proxy/src/postwoman.io/proxy/libproxy (from $GOPATH)
/home/user_00/go/src/postwoman.io/proxy/libproxy

As per hoppscotch/hoppscotch#2095.

Thanks

I'm having an issue trying to install this (desktop tray version) on Windows (10), . I cloned the package, open git bash in the root postwoman-master folder and get this when running the bash script:

./build.sh: line 50: go: command not found
./build.sh: line 65: go: command not found
./build.sh: line 67: rsrc: command not found
./build.sh: line 68: go: command not found
rm: cannot remove 'rsrc.syso': No such file or directory```

Hello :)

Just a quick question: would it be possible for you to provide the official hoppscotch/proxyscotch image for ARM too?

Ubuntu VMs for CI have been updated to Jammy Jellyfish (22.04) from 18.04 which is now EOL.

It seems libappindicator dependency is once again moved or otherwise causing problems so need to fix dependency and ideally in future avoid this occuring.

Currently, the support is only for hoppscotch.io but not for self-hosted ones.
It would be great to have the option to configure the base URL and deploy the changes.

root@server:~/proxyscotch/out/linux-server# ./proxyscotch-server-linux-amd64-v0.1.1 --host="127.0.0.1:9159" --token="XXXX"
2023/11/16 06:30:19 Starting proxy server...
2023/11/16 06:30:19 [ready=true] Listening on http://127.0.0.1:9159/
2023/11/16 06:30:24 An unauthorized request was made.
2023/11/16 06:31:48 An unauthorized request was made.

My first time setup proxyscotch. How can i ensure that my request was authorized from hoppscotch.io ? i have been getting this error and not sure how to plug in the token value on the hoppscotch.io web. When I removed the token its working fine though. Sorry cause not that familiar with proxyscotch.

When sending body data though the pwa, the server crashes with the following error:

2020/01/04 18:57:48 http: panic serving 127.0.0.1:33336: assignment to entry in nil map
goroutine 7 [running]:
net/http.(*conn).serve.func1(0xc4200988c0)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0xd0
panic(0x693b00, 0x720c60)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
net/textproto.MIMEHeader.Set(0x0, 0xc420154134, 0xc, 0xc42014a160, 0x1f)
	/usr/lib/go-1.10/src/net/textproto/header.go:22 +0xb0
net/http.Header.Set(0x0, 0xc420154134, 0xc, 0xc42014a160, 0x1f)
	/usr/lib/go-1.10/src/net/http/header.go:31 +0x53
postwoman.io/proxy/libproxy.proxyHandler(0x724620, 0xc4201600e0, 0xc420148100)
	/home/felix/go/src/postwoman.io/proxy/libproxy/proxy.go:136 +0x545
net/http.HandlerFunc.ServeHTTP(0x6ff280, 0x724620, 0xc4201600e0, 0xc420148100)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0x8796c0, 0x724620, 0xc4201600e0, 0xc420148100)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x130
net/http.serverHandler.ServeHTTP(0xc42008b2b0, 0x724620, 0xc4201600e0, 0xc420148100)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4200988c0, 0x724820, 0xc420066540)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x27b
2020/01/04 18:57:48 http: panic serving 127.0.0.1:33340: assignment to entry in nil map
goroutine 34 [running]:
net/http.(*conn).serve.func1(0xc4201a2000)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0xd0
panic(0x693b00, 0x720c60)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
net/textproto.MIMEHeader.Set(0x0, 0xc4201ae084, 0xc, 0xc4201980e0, 0x1f)
	/usr/lib/go-1.10/src/net/textproto/header.go:22 +0xb0
net/http.Header.Set(0x0, 0xc4201ae084, 0xc, 0xc4201980e0, 0x1f)
	/usr/lib/go-1.10/src/net/http/header.go:31 +0x53
postwoman.io/proxy/libproxy.proxyHandler(0x724620, 0xc4201c6000, 0xc4201b8000)
	/home/felix/go/src/postwoman.io/proxy/libproxy/proxy.go:136 +0x545
net/http.HandlerFunc.ServeHTTP(0x6ff280, 0x724620, 0xc4201c6000, 0xc4201b8000)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0x8796c0, 0x724620, 0xc4201c6000, 0xc4201b8000)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x130
net/http.serverHandler.ServeHTTP(0xc42008b2b0, 0x724620, 0xc4201c6000, 0xc4201b8000)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4201a2000, 0x724820, 0xc4201b0040)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x27b

steps:

1. Clone repo
2. sudo docker build -t proxyscotch
3. sudo docker run -it proxyscotch

the build runs fine but I get this error at run:
sh: /etc/postwoman-proxy/out/linux-server/postwoman-proxy-server: not found

Indeed:

computer:~/dev/proxyscotch$ sudo docker run -it proxyscotch sh
/etc/postwoman-proxy $ cd out/
/etc/postwoman-proxy/out $ cd linux-server/
/etc/postwoman-proxy/out/linux-server $ ls
proxywoman-server-linux-v0.0.1

I'm not sure what the best way to fix this given I don't have any go background.
I also have related questions:

• Is there an official build on docker hub ?
• If not, could you add instructions on how to build and run the docker image in README ?

Access to XMLHttpRequest at 'https://127.0.0.1:9159/' from origin 'https://hoppscotch.io' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://postwoman.io' that is not equal to the supplied origin.

I have a quick question in regards to where it says ./server --host="<hostname>:<port>" --token="<token_or_blank>"

On windows, this doesn't seem to do anything. I just get "./server is a directory" in bash. If I need to just manually add the host and token values before building it, that's fine I was just curious if the command would work or not, as it would be handy.

Also, in regards to hosting: I've so far been able to get this working (with my own copy of Postwoman) locally. I would really like to host the proxy but am a bit confused on what's going to happen. My question is if the server, with an IP of 123.123.123.123 is running the proxy (pointing at localhost:9159 for host) then would I be able to reach the proxy at 123.123.123.123:9159?

Thanks a million for all of the work that has gone into this and Postwoman!

The build.sh script cannot be used on Linux because it's full of CRLF. I've created PR #21 to fix it.

Hi,

I installed proxyscotch with docker but when I try to access it, it's directly redirecting to https://postwoman.io

It was also mentioned here hoppscotch/hoppscotch#1380

Often, while writing new clients, or debugging existing API's, it is often useful to use a tool such as Charles Proxy. It allows one to save traces (request + response) and reload them again for reviewing as well.

Would be useful if Postwoman came up with a way to act as such a proxy, and one could connect existing apps / browsers on them, and look through the requests / responses. Would also be helpful to generate user signed certificates if one wishes to debug through TLS encrypted requests as well.

Ask:

1. A standard way to load and import traces, to begin with. Please support existing HTTP capture trace formats, such as Charles' .trace files etc.
2. Look at actually enable the proxying feature.
3. Enable support for saving traces as well, so they can be reused later across app sessions.

Alternatives:
N/A

Additional Context:
N/A

Issue transferred from hoppscotch/hoppscotch#325

I would like to have a Dockerimage for the postwoman proxy. It would also be great to integrate it into the official docker-compose of postwoman.

I can also offer to implement it, just give me some input on how you would like it and what the best way is to bring this together with postwoman.

I have an HTTPS service with a certificate issued based on a domain name, but proxyscotch tells me that it cannot validate the certificate when I access it from an IP address in a development environment. Is there any option or configuration to turn off certificate validation? Just like Insomnia, I can uncheck "Validate certificates".

proxyscotch  | 2024/02/26 15:10:51 Failed to write response body: Post "https://10.10.10.109:10010/robot/controlTask": tls: failed to verify certificate: x509: cannot validate certificate for 10.10.10.109 because it doesn't contain any IP SANs

Here's my compose config file

version: '3.8'

services:
  proxyscotch:
    image: hoppscotch/proxyscotch:latest
    container_name: proxyscotch
    hostname: proxyscotch-1
    ports:
      - '9159:9159'
    restart: always
    networks:
      - default
networks:
  default:
    external: true
    name: demo_default

When setting a header in the progressive web app, the server crashes

2020/01/01 17:36:37 http: panic serving 127.0.0.1:60756: assignment to entry in nil map
goroutine 7 [running]:
net/http.(*conn).serve.func1(0xc4200988c0)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0xd0
panic(0x694b00, 0x721cc0)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
net/textproto.MIMEHeader.Add(0x0, 0xc42001af90, 0x9, 0xc420016a20, 0x24)
	/usr/lib/go-1.10/src/net/textproto/header.go:15 +0xec
net/http.Header.Add(0x0, 0xc42001af73, 0x9, 0xc420016a20, 0x24)
	/usr/lib/go-1.10/src/net/http/header.go:24 +0x53
postwoman.io/proxy/libproxy.proxyHandler(0x725680, 0xc420170000, 0xc42016e000)
	/home/felix/go/src/postwoman.io/proxy/libproxy/proxy.go:137 +0x63d
net/http.HandlerFunc.ServeHTTP(0x700288, 0x725680, 0xc420170000, 0xc42016e000)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0x87a6c0, 0x725680, 0xc420170000, 0xc42016e000)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x130
net/http.serverHandler.ServeHTTP(0xc42008cf70, 0x725680, 0xc420170000, 0xc42016e000)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc4200988c0, 0x725880, 0xc420066500)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x27b
2020/01/01 17:36:37 SessionId 5cb4acf5-3a1f-4203-b4d9-93e091ebf96c
2020/01/01 17:36:37 http: panic serving 127.0.0.1:60760: assignment to entry in nil map
goroutine 9 [running]:
net/http.(*conn).serve.func1(0xc420098960)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0xd0
panic(0x694b00, 0x721cc0)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x229
net/textproto.MIMEHeader.Add(0x0, 0xc42018a0a0, 0x9, 0xc4201a0060, 0x24)
	/usr/lib/go-1.10/src/net/textproto/header.go:15 +0xec
net/http.Header.Add(0x0, 0xc42018a083, 0x9, 0xc4201a0060, 0x24)
	/usr/lib/go-1.10/src/net/http/header.go:24 +0x53
postwoman.io/proxy/libproxy.proxyHandler(0x725680, 0xc4201aa000, 0xc420198000)
	/home/felix/go/src/postwoman.io/proxy/libproxy/proxy.go:137 +0x63d
net/http.HandlerFunc.ServeHTTP(0x700288, 0x725680, 0xc4201aa000, 0xc420198000)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0x87a6c0, 0x725680, 0xc4201aa000, 0xc420198000)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x130
net/http.serverHandler.ServeHTTP(0xc42008cf70, 0x725680, 0xc4201aa000, 0xc420198000)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0xbc
net/http.(*conn).serve(0xc420098960, 0x725880, 0xc42018e040)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x27b

related: hoppscotch/hoppscotch#175

Is there an existing issue for this?

• I have searched the existing issues

Current behavior

On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests, when I create the following request:

CURL equivalent:

curl --request POST \
  --url 'https://hookb.in/YVkaPWyrBbCQjy0QmeZp?param=test' \
  --header 'content-type: application/json' \
  --header 'origin: https://mycustomorigin.com' \
  --header 'referer: https://mycustomreferrer.com/' \
  --header 'user-agent: mycustom-UA' \
  --header 'x-source: mycustom-source' \
  --data '{}'

Then I inspect the request on the destination, and get the following result instead:

## HTTP HEADERS 
accept-encoding: gzip
content-length: 2
content-type: application/json
host: hookb.in
origin: https://mycustomorigin.com
referer: https://mycustomreferrer.com/
user-agent: Proxyscotch/1.1
x-source: mycustom-source

## QUERY STRING
param: test

## BODY
{}

unexpectedly some of the headers are modified:

• from user-agent: mycustom-UA to user-agent: Proxyscotch/1.1.
  Looks like the Proxyscotch is overwriting it.

Why is it concerning?

• In some valid use case, you really want to send a custom HTTP headers of user-agent. e.g.
  • in case like to debug API issue, or
  • to reproduce exact same request created by other client.
  • You want to emulate browser request, with device specific user-agent).

So having this is important.

Maybe instead of Proxy scotch overwriting the actual user-agent header, it should add its own headers e.g. x-requested-via-proxy-agent: Proxyscotch/1.1

Steps to reproduce

1. Prepare endpoint that can inspect HTTP request. I used https://hookbin.com/. e.g. I will create endpoint & get URL https://hookb.in/YVkaPWyrBbCQjy0QmeZp
2. On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests
3. import CURL provided above, modify the URL with URL you get from step 1.
4. Send request
5. Inspect the HTTP request result on the destination (in my case I refreshed the Hookbin page I created on step 1.)
6. You will get same result as mine above.

Environment

Production

Version

Cloud

always alert "commond not found",either linux or windows

Hi , I'm appreciate about your hard work.
Currently, i'm trying to use latest proxyscotch docker image to resolve CORS https request from hoppscotch web client.
But i received error response from proxy like under picture.

Could you help me or tell me what I am doing wrong ?

There is my docker-compose to deploy proxy

version: "3.2"
services:
  proxy_server:
    image: hoppscotch/proxyscotch
    command: ["sh", "-c", "/etc/proxyscotch/out/linux-server/proxyscotch-server-* --host 0.0.0.0:9159 --allowed-origins *"]
    ports:
      - 9159:9159

Json responses from orig servers becomes stringified json whenever it comes through proxywoman.
The part that i can't understand that is https://postwoman.apollosoftware.xyz/ proxy works fine and it gets data by json.

Is there any chance that i could be missing something?

I've tested with both same link which is https://projects.propublica.org/nonprofits/api/v2/search.json?order=revenue&sort_order=desc

How to reproduce (Tested with Linux / Windows / Docker(Linux))

1. Start Linux/Windows Proxywoman server.
2. Request https://projects.propublica.org/nonprofits/api/v2/search.json?order=revenue&sort_order=desc
3. Check response from proxy server

Sample response (Truncated organizations) from my local proxywoman

{
  "success": true,
  "data": "{\"total_results\":1968539,\"organizations\":[{\"ein\":464744976,\"strein\":\"46-4744976\",\"name\":\"0 DEBT EDUCATION INC\",\"sub_name\":\"0 DEBT EDUCATION INC\",\"city\":\"SANTA ROSA\",\"state\":\"CA\",\"ntee_code\":\"P51\",\"raw_ntee_code\":\"P51\",\"subseccd\":3,\"has_subseccd\":true,\"have_filings\":true,\"have_extracts\":true,\"have_pdfs\":true,\"score\":1.0}],\"num_pages\":100,\"cur_page\":0,\"page_offset\":0,\"per_page\":100,\"search_query\":null,\"selected_state\":null,\"selected_ntee\":null,\"selected_code\":null,\"data_source\":\"ProPublica Nonprofit Explorer API: https://projects.propublica.org/nonprofits/api/\\nIRS Exempt Organizations Business Master File Extract (EO BMF): https://www.irs.gov/charities-non-profits/exempt-organizations-business-master-file-extract-eo-bmf\\nIRS Annual Extract of Tax-Exempt Organization Financial Data: https://www.irs.gov/uac/soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data\",\"api_version\":2}",
  "status": 200,
  "statusText": "OK",
  "headers": {
    "Age": "5394",
    "Alt-Svc": "h3-27=\":443\"; ma=86400, h3-28=\":443\"; ma=86400, h3-29=\":443\"; ma=86400",
    "Cache-Control": "public, max-age=86400, s-maxage=31536000, stale-while-revalidate=600, stale-if-error=600",
    "Cf-Cache-Status": "HIT",
    "Cf-Ray": "5af22d0e9e6ee9f0-ICN",
    "Cf-Request-Id": "03cb3a7d1c0000e9f0208e3200000001",
    "Content-Type": "application/json; charset=utf-8",
    "Date": "Tue, 07 Jul 2020 14:15:09 GMT",
    "Etag": "W/\"38320e44c0b00757dc75821be0287618\"",
    "Expect-Ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
    "Server": "cloudflare",
    "Set-Cookie": "__cfduid=dd646369072c7864ab415e4b3240333cb1594131309; expires=Thu, 06-Aug-20 14:15:09 GMT; path=/; domain=.propublica.org; HttpOnly; SameSite=Lax",
    "Vary": "Accept,Accept-Encoding,Content-Type",
    "X-Content-Type-Options": "nosniff",
    "X-Frame-Options": "SAMEORIGIN",
    "X-Request-Id": "b2fe267c-ab18-489b-88bf-b4f6c90dd9b3",
    "X-Runtime": "0.139345",
    "X-Xss-Protection": "1; mode=block"
  }
}

See title. High priority.

As part of this, we’ll need to determine what aspects of proxyscotch should be tested and how (e.g., unit tests).

Then, CI needs to be set up for pull requests - etc.

go build server/server.go does not work on linux server

@NBTX, it turns out that there are a bunch of files in your commit that introduced CRLF, and it's breaking the build in bigger ways. For example, the binary that's built has a CR in the filename, and many of the source files do. This is a bigger change than I have time to fix right now.

It would be great if we could package the server for Mac(primary), Linux and Windows(In the near future). Goreleaser could be used with Travis CI or GH Actions.

I'll be happy to send a PR,

When creating a certificate, there is an error in the judgment logic that prevents key.pem from being generated, and the judgment logic of whether the file exists or not and whether the write is successful or not needs to be modified.

Sorry, I can't find the license file for this project

It would be nice to have a list of the dependencies to build and run the proxy. The build script downloads a lot of dependencies, but it does not install libappindicator.

I got the following while building on my machine:

# pkg-config --cflags  -- ayatana-appindicator3-0.1
Package ayatana-appindicator3-0.1 was not found in the pkg-config search path.
Perhaps you should add the directory containing `ayatana-appindicator3-0.1.pc'
to the PKG_CONFIG_PATH environment variable
Package 'ayatana-appindicator3-0.1', required by 'virtual:world', not found
pkg-config: exit status 1

Made it work installing the following package on Arch:

sudo pacman -Sy libayatana-appindicator

There where multiple related packages so I needed to check the packages file lists to figure the right one. Pehaps there should be a "dependencies" doc section?

Currently, the latest proxyscotch release is 0.0.1 but this release is from the proxywoman (old hoppscotch name).
This causes the hopscotch application to fail the request for proxy because cors is not allowed (only allowed in postwoman.io)

To solve this problem, I would like to request a new proxyscotch release for the GitHub release if possible.
I built my own one, In some ways, some features do not work in my release.

Websocket cannot be linked using a custom port
ws://121.40.165.18:8800