The Caterpillar project aims to be a universal web debugging proxy. Currently, Fediverse-related features (such as spam filters) are embedded within the code without being modularized.

There is a need to separate this.

Add new strategy: SearchEngine5

• https://github.com/Ahwxorg/librey/

PHP Parse error: Unmatched ')' in index.php on line 17

This improvement retrieves the requested URL from cache services (such as Wayback Machine, Google Search Cache, Bing Search Cache, etc.) and delivers the response through a proxy.

• Wayback Machine APIs https://archive.org/help/wayback_api.php
• Google Search Cache
• Bing Search Cache

Some web hosting providers require HTTP Basic Authentication when accessing their subdomains rather than standalone domains.

Moreover, even in cases where it's not necessarily required, there might be potential instances where utilizing HTTP Basic Authentication is advantageous.

If we implement stateful relay, we can get the effect of improving speed, reliability, and bypass the capacity limit setting of the web server (e.g., max_upload_size) to allow large-capacity transmission.

These proposals are messages that the client (e.g., server.py) will send to the server (e.g., index.php).

Proposal: Stateless relay

This is a case where the client cannot expose the port to the outside, it is the same already implemented.

{
    "jsonrpc": "2.0",
    "method": "relay_request",
    "params": {
        "data": <base64 encoded data>,
        "compressed": <e.g. deflate, none>,    // proposal
        "client": <address of the client>,
        "server": <address of the remote server>,
        "port": <port number of the remote server>,
        "scheme": <scheme (e.g. http, https, ssl, tls),
        "url": <URL>,
        "length": <length of data>,
        "chunksize": <size of buffer (e.g. 8192)>,
        "datetime": <datetime (e.g. %Y-%m-%d %H:%M:%S.%f)>
    },
    "id": 3
}

Proposal: Stateful relay

This is a case where the client can expose the port to the outside, which works similarly to tunneling.

{
    "jsonrpc": "2.0",
    "method": "relay_connect",
    "params": {
        "client": <address of the client>
        "port": <port number of the client>,
        "chunksize": <size of buffer (e.g. 8192)>,
        "datetime": <datetime (e.g. %Y-%m-%d %H:%M:%S.%f)>
    },
    "id": 3
}

Summary

(Here's a story from Canada.)

There is an issue in infrastructure management tools (e.g., OpenShift or similar software) where TCP and HTTP are not clearly distinguished. This has led to minor conflicts between infrastructure engineers and developers.

As a result, there is a need for a solution that allows maintaining the business logic while flexibly switching the communication method between TCP (socket) and HTTP.

My Opinion

TCP and HTTP are distinctly different concepts, but since HTTP is the most well-known application protocol running on top of TCP, the mistake of not distinguishing between them often occurs. To better understand this, you can refer to the OSI 7-layer model, which explains this distinction well.

Upon hearing this story, Caterpillar Proxy has decided to work on enabling the endpoint communication method to be switched flexibly between TCP (socket) and HTTP.

Currently, we support existing plugins through the web.py file.

Java based worker

https://github.com/gnh1201/caterpillar/tree/worker-java

alembic integration

https://alembic.sqlalchemy.org/en/latest/

Add support the HTTPS proxy

Summary

In 2022, I conducted a PoC on executing WASM(WebAssembly) binaries using PHP and WAMR(WebAssembly Micro Runtime) on top of the LAMP stack. As the Caterpillar project is currently underway, it seems feasible to integrate the efforts made during that time into the Caterpillar project.

The Caterpillar project will provide a method for injecting WASM runtimes into shared hosting servers in the future.

Related Links

• https://github.com/gnh1201/wp-wamr
• bytecodealliance/wasm-micro-runtime#1017
• bytecodealliance/wasm-micro-runtime#802
• bytecodealliance/wasm-micro-runtime#1058
• bytecodealliance/wasm-micro-runtime#1012
• bytecodealliance/wasm-micro-runtime#1166
• bytecodealliance/wasm-micro-runtime#803
• https://gist.github.com/gnh1201/b9c4ee4a98fbb369237b3e447a9550a2
• https://bytecodealliance.zulipchat.com/#narrow/stream/206238-general/topic/A.20cost-effectiveness.20perspective.20of.20WASM.28and.20I.20need.20advice.29/near/360674718
• https://github.com/asmnext/wasm-integration-design/blob/main/php-iwasm.md (Korean)

Gateways that do not rely on well-known protocols such as WiFi or 3G/LTE provide data transmission only through serial or Bluetooth. For instance, gateways supporting LoRaWAN technology or satellite communication rarely, if ever, directly support Ethernet or WiFi, and even if they do, they are usually expensive.

We need to assess whether we can support serial or Bluetooth in this project.

This project did not prioritize ThreadPool implementation as it was focused on Proof of Concept (PoC). ThreadPool implementation is necessary to resolve thread hell when a large number of connections occur.

I need to implement an SMTP relay. Caterpillar will receive SMTP requests first and then forward them to a relay server written in PHP. The relay server written in PHP will then send the email to the actual recipient.

I propose the following flow:

SMTP Client (Sender) <- Caterpillar (Python) <-> Relay Server (PHP) <-> Actual recipient

Add proxy cache like the squid-cache

HSTS only applies to software that fulfills all the specifications as a web browser. Therefore, in communications where there is no web browser involved, typical SSL MITM poses no issue.

However, if you intend to use a web browser, HSTS policies can cause inconvenience. Thus, here are some alternatives:

These alternatives are based on the assumption that we won't alter the web browser's settings. Disabling the HSTS feature by adjusting the browser settings can resolve the issue more easily than expected.

1. Removing HSTS-related headers.
2. Proxying with an actual web browser.

I'll add more ideas if they come up in the future.

In the Caterpillar Proxy, we support a stateful method of tunneling using network sockets to bypass capacity limit settings in HTTP requests.

However, there's an idea for resolving capacity limits even in stateless mode.

It involves storing the data to be requested in a separate Object Storage. This way, capacity limits can be addressed even in Stateless mode.