swhitty / flyingfox Goto Github PK

Overview

If the data passed to Socket.write is a slice with a non-zero startIndex, memory after the end of the data buffer will be leaked to the recipient.

Cause

The issue is on line 229 of Socket.swift:

let sent = try write(buffer.baseAddress! + index, length: data.endIndex - index)

The code assumes that buffer.baseAddress! + index correctly gets the byte at index in the data, however baseAddress points to the byte at startIndex not at index 0. For example, consider the following code:

let data = "abcd".data(using: .utf8)!
let slice = data[2...] // contains "cd"

let index = slice.startIndex
try slice.withUnsafeBytes { buffer in               
    _ = try write(buffer.baseAddress! + index, length: data.endIndex - index)
    //                   (1)            (2)            (3)
    //
    // 1. baseAddress points to "c"
    // 2. after adding startIndex (which is 2), the pointer points to the byte after the end of the buffer
    // 3. length is 2 (4 - 2)
    //
    // In this example scenario, the server accidentally sends two bytes of
    // the memory after the end of the data buffer to the client, which could
    // lead to sensitive data being leaked in certain setups. It could also potentially
    // be combined with certain other types of vulnerabilities to execute arbitrary code.
}

Proof of concept

First, run the following command to start a tcp listener that simply prints out any data it receives.

nc -l 8080

Next, run this code snippet with swift run for the highest chance of reproducing, because that's how I ran it:

@main
struct FlyingFoxDataTest {
    static func main() async throws {
        // Generate some dummy data and make a slice
        let data = String(repeating: "abcd", count: 32).data(using: .utf8)!
        let slice = data[64...]
        
        // This length of string seems to work consistently on my laptop
        let secretPassword = "thisismyverylongandsecuresecretpasswordthisismyverylongandsecuresecretpasswordthisismyverylongandsecuresecretpassworditissogood!".data(using: .utf8)!
        
        // Attempt to send the slice through the socket
        let socket = try await AsyncSocket.connected(to: .inet(ip4: "127.0.0.1", port: 8080), pool: .polling)
        try await socket.write(slice)
    }
}

When I run that snippet, I get the following output:

$ nc -l 8080
thisismyverylongandsecuresecretpasswordthisismyverylongandsecure

As you can see, it sent the first half of secretPassword instead of the contents of the data slice. This bug could have pretty bad side-effects if it appeared in any unfortunate situations.

Mitigation

Make the following change:

- let sent = try write(buffer.baseAddress! + index, length: data.endIndex - index)
+ let sent = try write(buffer.baseAddress! + index - data.startIndex, length: data.endIndex - index)

I'll make a PR to fix this soon.

And I know this full on bug report might be a bit overkill, but I had fun getting that proof of concept to work, so I did it anyway.

We currently only support for

public mutating func appendRoute(_ route: HTTPRoute, to handler: HTTPHandler) {
        handlers.append((route, handler))
    }

    public mutating func appendRoute(_ route: HTTPRoute,
                                     handler: @Sendable @escaping (HTTPRequest) async throws -> HTTPResponse) {
        handlers.append((route, ClosureHTTPHandler(handler)))
    }

Can we also have the function that inserts at 0 or remove all handlers.
The scenario we encounter is that we use it on the UI test, while we need to mock a response change, we will need to change the handlers array to make it happen, if append is the only function, we will need to rebuild a server again to make the change.

I'm using this in a UI test environment. It seems like there's a silent issue at times after starting a server for UI testing where continued execution ceases at the log "starting server port". I have tried:

1. Starting a server once in a task as documented in readme and stopping it once with the test bundle for the entire run. Issues launching the test app persist using this method.

2. Starting a server directly using the async setup methods for a given test case. In this case it hangs up on execution.

3. Starting a server in a task alongside launching the app in a setup method. This method works when I launch the app first, although not always.

Is anyone else using FF as a dependency of UI testing and have found a happiest path for using in such an environment?

Althought we can use nginx/apache with a SSL certificate I am wondering if there are plans to add it here in the future.

Is FlyingFox able to handle large file uploads?

Thanks for the great software. A small leak found with leaks:

   public static func makeAddress(from addr: sockaddr_storage) throws -> Address {
       switch Int32(addr.ss_family) {
       case AF_INET:
           var addr_in = try sockaddr_in.make(from: addr)
           let maxLength = socklen_t(INET_ADDRSTRLEN)
           let buffer = UnsafeMutablePointer<CChar>.allocate(capacity: Int(maxLength))
           try Socket.inet_ntop(AF_INET, &addr_in.sin_addr, buffer, maxLength)
           return .ip4(String(cString: buffer), port: UInt16(addr_in.sin_port).byteSwapped)

       case AF_INET6:
           var addr_in6 = try sockaddr_in6.make(from: addr)
           let maxLength = socklen_t(INET6_ADDRSTRLEN)
           let buffer = UnsafeMutablePointer<CChar>.allocate(capacity: Int(maxLength))
           try Socket.inet_ntop(AF_INET6, &addr_in6.sin6_addr, buffer, maxLength)
           return .ip6(String(cString: buffer), port: UInt16(addr_in6.sin6_port).byteSwapped)
...
    }

buffer is leaked. Should be an easy fix.

Can we setup a local HTTPS Server in Mac OS with FlyingFox?

First of all, thanks for this really good library!

With this setup:

let server = HTTPServer(port: 8080)

let root = URL(fileURLWithPath: "/Users/myUser/myStaticDir")

let route = HTTPRoute(method: .GET, path: "/static/*")
let handler = DirectoryHTTPHandler(root: root, serverPath: "static")

await server.appendRoute(route, to: handler)

try await server.start()

When I perform:

curl -L -vv http://localhost:8080/static/index.html

I'm always getting a 404.

That's because:

To make it work I had to write the handler like this:

let handler = DirectoryHTTPHandler(root: root, serverPath: "/static/")

But then I found out that if I want to also handle the default file to be index.html when not explicitly specifying it:

curl -L -vv http://localhost:8080/static/

or

curl -L -vv http://localhost:8080/static

I have to also add:

let fileRoute = HTTPRoute(method: .GET, path: "/static")
let fileHandler = FileHTTPHandler(path: root.appendingPathComponent("index.html"), contentType: "text/html")
await server.appendRoute(fileRoute, to: fileHandler)

Then I thought this is a very common scenario one would expect to work with minimal work, so I written this extension:

extension HTTPServer {
    func appendStaticRoute(_ route: String, root: URL, defaultFileName: String = "index.html", withContentType contentType: String = "text/html") {
        appendRoute(HTTPRoute(method: .GET, path: "/\(route)/*"), to: DirectoryHTTPHandler(root: root, serverPath: "/\(route)/"))
        appendRoute(HTTPRoute(method: .GET, path: "/\(route)/"), to: FileHTTPHandler(path: root.appendingPathComponent(defaultFileName), contentType: contentType))
        appendRoute(HTTPRoute(method: .GET, path: "/\(route)"), to: .redirect(to: "/\(route)/"))
    }

    func appendStaticRoute(_ route: String, bundle: Bundle = .main, defaultFileName: String = "index.html", withContentType contentType: String = "text/html") throws {
        guard let root = bundle.resourceURL else { throw HTTPUnhandledError() }
        appendStaticRoute(route, root: root, defaultFileName: defaultFileName, withContentType: contentType)
    }
}

Which allows you to achieve the same with only:

let server = HTTPServer(port: 8080)

let root = URL(fileURLWithPath: "/Users/myUser/myStaticDir")

await server.appendStaticRoute("static", root: root)

try await server.start()

And so here I am, posting this. If you think it's a valuable addition, feel free to add it in the library 🍻

JSON responses are common for a web server. With this extension (serializing dates as millis by default to ease JS clients):

extension HTTPResponse {
    static let jsonTimestampMillisEncoder: JSONEncoder = {
        let jsonEncoder = JSONEncoder()
        jsonEncoder.dateEncodingStrategy = .millisecondsSince1970

        return jsonEncoder
    }()

    static func json(_ encodable: Encodable, jsonEncoder: JSONEncoder = jsonTimestampMillisEncoder) throws -> HTTPResponse {
        HTTPResponse(statusCode: .ok, body: try jsonEncoder.encode(encodable))
    }
}

one can easily return a JSON ready to be consumed by another application:

struct Cat: Codable {
    let birthDate: Date
    let name: String
}

struct MyHandler : HTTPHandler {
    func handleRequest(_ request: HTTPRequest) async throws -> HTTPResponse {
        try .json(Cat(birthDate: Date(), name: "Loris"))
    }
}

If you think it's a valuable addition, feel free to add it in the library 🍻

What is a path traversal vulnerability?

A path traversal vulnerability occurs when paths aren't properly normalised by the server, allowing attackers to access any file on the server, not just files within the served directory.

Proof of concept

The following code snippet for creating a simple file server is vulnerable to path traversal (and so is any other code that uses DirectoryHTTPHandler).

let directoryHandler = DirectoryHTTPHandler(root: URL(fileURLWithPath: "."), serverPath: "/")
let server = HTTPServer(port: 80)
await server.appendRoute("GET *", to: directoryHandler)
try await server.start()

To observe the effect of this vulnerability, run the code snippet above and then run the following command in terminal to see that the server has exposed the contents of your machine's /etc/passwd file (and all other files for that matter):

curl --path-as-is http://127.0.0.1/../../../../../../../../../../../etc/passwd

Any user that can access the web server can run a similar command on their machine to access any file on the server (within the limits of the privileges that the server is running with).

Mitigation

Option 1 (easy, but isn't as nice as option 2)

Abort any requests to the directory handler that contain ../ in their path. This is how vapor's file server middleware avoids path traversal (the following snippet is taken from vapor source). This only fixes the issue for the built-in directory handler, but the check could be applied to all requests (not just those handled by the directory handler) to fix that.

// protect against relative paths
guard !path.contains("../") else {
    return request.eventLoop.makeFailedFuture(Abort(.forbidden))
}

I don't like this solution because it feels like a bit of a quick fix, but another way of looking is that its simplicity makes it very hard to mess up.

Option 2

Normalize request paths in HTTPDecoder to remove any .. components. Refer to the RFC specification on removing dot components (https://www.rfc-editor.org/rfc/rfc3986#section-5.2.4).

This solution is a bit more involved, but it results in nicer behaviour in my opinion, and I think it's the approach that most http servers take.

Conclusion

Let me know which solution you prefer. If you would like me to implement the fix, I'd be happy to implement either solution. Personally I prefer the behaviour of the second solution but the simplicity of the first, so it's up to you.

Hello,

for one of my usecases i need url which uses percent encoded spaces, eg: http://host/some%20folder,
currently such url wont get matched and handled, far from ideal but i got around it for now by editing HTTPDecoder and replacing the "%20" with "+", then i get the route matched and handle it accordingly.

let comps = status
            .trimmingCharacters(in: .whitespacesAndNewlines)
            .replacingOccurrences(of: "%20", with: "+")
            .split(separator: " ", maxSplits: 2, omittingEmptySubsequences: false)

any ideas on how to solve it nicely?

thanks for the great work!

If I try to use FlyingFox and embed some HTML with CSS in my app and then load the content via a WebKit browser instance, the HTML content does not load correctly because the CSS does not get loaded.

It appears that the makeContentType method in FileHTTPHandler does not return the right type for CSS files. If you add the following case to the method, the HTML content renders correctly:

case "css":
	return "text/css"

I was migrating an HTTP Server to FlyingFox and I encountered a client side error "Socket write failed". I spent some time to discover this line:

The client app didn't send the "keepalive" header in requests. I think it would be great to mention "keepalive" in readme.

Thank you for this great lightweight server :)

When it comes to cross-origin requests, I've noticed that the requests don't even reach the server, so I can't set the necessary headers for cross-origin as I usually would. How can I handle this?

private func startListener() async{
    do {
        let headers = [
                       HTTPHeader("Access-Control-Allow-Origin"):"*",
                       HTTPHeader("Access-Control-Allow-Headers"): "*",
                       HTTPHeader("Access-Control-Allow-Methods"): "OPTIONS, GET, POST",
                       HTTPHeader("Access-Control-Allow-Credentials") : "true"
                      ]

          await server.appendRoute("/hook") { request in
          //  Don't receive any message.
          return HTTPResponse(statusCode: .ok,headers: headers)
        }

        try await server.start()

        print("Server is running on port xxxx")

    } catch {
        print("Server start error: \(error)")
    }
}

When multiple endpoints are defined, one ends up with:

await server.appendRoute(HTTPRoute(.GET, "/cat"), to: MyCatGetHandler())
await server.appendRoute(HTTPRoute(.GET, "/dog"), to: MyDogGetHandler())
await server.appendRoute(HTTPRoute(.GET, "/fish"), to: MyFishGetHandler())

and then implementations contains only the handler:

struct MyCatGetHandler : HTTPHandler {
    func handleRequest(_ request: FlyingFox.HTTPRequest) async throws -> FlyingFox.HTTPResponse { }
}

struct MyDogGetHandler : HTTPHandler {
    func handleRequest(_ request: FlyingFox.HTTPRequest) async throws -> FlyingFox.HTTPResponse { }
}

struct MyFishGetHandler : HTTPHandler {
    func handleRequest(_ request: FlyingFox.HTTPRequest) async throws -> FlyingFox.HTTPResponse { }
}

Maybe it's my personal taste, but I find it better to have both the route and the handler in the same file, so by adding those extensions:

protocol RouteHandler : HTTPHandler {
    func route() -> HTTPRoute
}

extension HTTPServer {
    func appendRoutes(_ routeHandlers: RouteHandler...) {
        routeHandlers.forEach {
            appendRoute($0.route(), to: $0)
        }
    }
}

I'm able to write each handler like this (which I find convenient, specially when the route has placeholders, so I can write a single common function and define shared constants between the route and the request handler):

struct MyCatGetHandler : RouteHandler {

    func route() -> HTTPRoute {
        HTTPRoute(method: .GET, path: "/cat")
    }

    func handleRequest(_ request: FlyingFox.HTTPRequest) async throws -> FlyingFox.HTTPResponse {
        // implementation
    }
}

and also be able to add them all like this:

await server.appendRoutes(
    MyCatGetHandler(),
    MyDogGetHandler(),
    MyFishGetHandler()
)

If you think it's a valuable addition, feel free to add it in the library 🍻

This is my third proposal so far (#50 #49), and the fact I was able to write everything I needed as an extension denotes really good design choices on your side, so kudos for the great work! 💯

I'd like to use this server in some tests. The tests might be executed in parallel, so I'm not able to hard-code specific port numbers.

If you bind a socket address with port number 0, the system will assign a unique port number for you, which is ideal for these situations. Unfortunately, AFAICT, there is no way to get the port number or socket from HTTPServer, so I wouldn't be able to actually make requests to that server instance.

Hi Simon,

I'm running into an issue with FlyingFox when building my project. The problem only arises when I specifically direct Swift to build it for macOS 12.0 (the reason I'm doing that is because I'm installing my tool through mint, which automatically targets the machine's os version). Here's the error:

$ swift build -c release -Xswiftc -target -Xswiftc x86_64-apple-macosx12.0
Building for production...
/Users/stackotter/Desktop/Projects/Swift/Scute/.build/checkouts/FlyingFox/Sources/Handlers/ProxyHTTPHandler.swift:51:42: error: ambiguous use of 'data'
        let (data, response) = try await session.data(for: req)
                                         ^
/Users/stackotter/Desktop/Projects/Swift/Scute/.build/checkouts/FlyingFox/Sources/URLSession+Async.swift:42:10: note: found this candidate
    func data(for request: URLRequest, forceFallback: Bool = false) async throws -> (Data, URLResponse) {
         ^
Foundation.URLSession:3:17: note: found this candidate
    public func data(for request: URLRequest, delegate: URLSessionTaskDelegate? = nil) async throws -> (Data, URLResponse)
                ^
[3/7] Compiling FlyingFox AsyncSequence+Extensions.swift

It seems like you may have to rename your method or change the function signature in some other way so that it doesn't clash with the macOS 12.0 API additions,

Cheers,
stackotter

Hi! First of all, You've done an amazing job with this library. Thank you very much.

I was wondering if there's a cleaner way to see if HTTPServer is active or not. Current work around seems to be either using a custom wrapper around HTTPServer or use self.listeningAddress != nil.

Let me know what you think

First, thank you for developing this fantastic lightweight server!

We've encountered a recurring crash, primarily when the iOS app is running in the background. The crash log is as follows:

Crashed: com.apple.root.user-initiated-qos.cooperative
0  libsystem_kernel.dylib         0x1e64 close + 8
1  App                          0x71c300 Socket.close() + 4384424704
2  App                          0x6f7eb8 HTTPServer.start() + 4384276152
3  libswift_Concurrency.dylib     0x41948 swift::runJobInEstablishedExecutorContext(swift::Job*) + 416
4  libswift_Concurrency.dylib     0x42868 swift_job_runImpl(swift::Job*, swift::ExecutorRef) + 72
5  libdispatch.dylib              0x15944 _dispatch_root_queue_drain + 396
6  libdispatch.dylib              0x16158 _dispatch_worker_thread2 + 164
7  libsystem_pthread.dylib        0xda0 _pthread_wqthread + 228
8  libsystem_pthread.dylib        0xb7c start_wqthread + 8

To provide further context, here are relevant code snippets related to how we start and stop the server:

 func start() {
        Task {
            do {
                if await !server.isListening {
                    try await server.start()
                }
            } catch (_ as CancellationError) {
                print("Task cancelled")
            } catch {
                logger.error("Error local server: \(error.localizedDescription)")
            }
        }
    }

    func stop() {
        let task = Task {
            await server.stop()
        }
        task.cancel()
    }

Thanks!

First of all thanks for developing this amazing package!

I wanted to ask if it is possible to use http 2 with this package as I see the server is currently using HTTP/1.1.