rustls / rustls-ffi Goto Github PK

View Code? Open in Web Editor NEW

120.0 120.0 29.0 744 KB

Use Rustls from any language

License: Other

Makefile 0.46% Rust 66.64% C 10.14% C++ 22.13% CMake 0.62%

cryptography ffi rust ssl tls

rustls-ffi's Introduction

Rustls is a modern TLS library written in Rust.

Status

Rustls is used in production at many organizations and projects. We aim to maintain reasonable API surface stability but the API may evolve as we make changes to accommodate new features or performance improvements.

We have a roadmap for our future plans. We also have benchmarks to prevent performance regressions and to let you evaluate rustls on your target hardware.

If you'd like to help out, please see CONTRIBUTING.md.

Changelog

The detailed list of changes in each release can be found at https://github.com/rustls/rustls/releases.

Documentation

https://docs.rs/rustls/

Approach

Rustls is a TLS library that aims to provide a good level of cryptographic security, requires no configuration to achieve that security, and provides no unsafe features or obsolete cryptography by default.

Rustls implements TLS1.2 and TLS1.3 for both clients and servers. See the full list of protocol features.

Platform support

While Rustls itself is platform independent, by default it uses aws-lc-rs for implementing the cryptography in TLS. See the aws-lc-rs FAQ for more details of the platform/architecture support constraints in aws-lc-rs.

ring is also available via the ring crate feature: see the supported ring target platforms.

By providing a custom instance of the crypto::CryptoProvider struct, you can replace all cryptography dependencies of rustls. This is a route to being portable to a wider set of architectures and environments, or compliance requirements. See the crypto::CryptoProvider documentation for more details.

Specifying default-features = false when depending on rustls will remove the dependency on aws-lc-rs.

Rustls requires Rust 1.63 or later. It has an optional dependency on zlib-rs which requires 1.75 or later.

Cryptography providers

Since Rustls 0.22 it has been possible to choose the provider of the cryptographic primitives that Rustls uses. This may be appealing if you have specific platform, compliance or feature requirements that aren't met by the default provider, aws-lc-rs.

Users that wish to customize the provider in use can do so when constructing ClientConfig and ServerConfig instances using the with_crypto_provider method on the respective config builder types. See the crypto::CryptoProvider documentation for more details.

Built-in providers

Rustls ships with two built-in providers controlled with associated feature flags:

aws-lc-rs - enabled by default, available with the aws_lc_rs feature flag enabled.
ring - available with the ring feature flag enabled.

See the documentation for crypto::CryptoProvider for details on how providers are selected.

Third-party providers

The community has also started developing third-party providers for Rustls:

rustls-mbedtls-provider - a provider that uses mbedtls for cryptography.
boring-rustls-provider - a work-in-progress provider that uses boringssl for cryptography.
rustls-rustcrypto - an experimental provider that uses the crypto primitives from RustCrypto for cryptography.
rustls-post-quantum: an experimental provider that adds support for post-quantum key exchange to the default aws-lc-rs provider.

Custom provider

We also provide a simple example of writing your own provider in the custom-provider example. This example implements a minimal provider using parts of the RustCrypto ecosystem.

See the Making a custom CryptoProvider section of the documentation for more information on this topic.

Example code

Our examples directory contains demos that show how to handle I/O using the stream::Stream helper, as well as more complex asynchronous I/O using mio. If you're already using Tokio for an async runtime you may prefer to use tokio-rustls instead of interacting with rustls directly.

The mio based examples are the most complete, and discussed below. Users new to Rustls may prefer to look at the simple client/server examples before diving in to the more complex MIO examples.

Client example program

The MIO client example program is named tlsclient-mio. The interface looks like:

Connects to the TLS server at hostname:PORT.  The default PORT
is 443.  By default, this reads a request from stdin (to EOF)
before making the connection.  --http replaces this with a
basic HTTP GET request for /.

If --cafile is not supplied, a built-in set of CA certificates
are used from the webpki-roots crate.

Usage:
  tlsclient-mio [options] [--suite SUITE ...] [--proto PROTO ...] [--protover PROTOVER ...] <hostname>
  tlsclient-mio (--version | -v)
  tlsclient-mio (--help | -h)

Options:
    -p, --port PORT     Connect to PORT [default: 443].
    --http              Send a basic HTTP GET request for /.
    --cafile CAFILE     Read root certificates from CAFILE.
    --auth-key KEY      Read client authentication key from KEY.
    --auth-certs CERTS  Read client authentication certificates from CERTS.
                        CERTS must match up with KEY.
    --protover VERSION  Disable default TLS version list, and use
                        VERSION instead.  May be used multiple times.
    --suite SUITE       Disable default cipher suite list, and use
                        SUITE instead.  May be used multiple times.
    --proto PROTOCOL    Send ALPN extension containing PROTOCOL.
                        May be used multiple times to offer several protocols.
    --no-tickets        Disable session ticket support.
    --no-sni            Disable server name indication support.
    --insecure          Disable certificate verification.
    --verbose           Emit log output.
    --max-frag-size M   Limit outgoing messages to M bytes.
    --version, -v       Show tool version.
    --help, -h          Show this screen.

Some sample runs:

$ cargo run --bin tlsclient-mio -- --http mozilla-modern.badssl.com
HTTP/1.1 200 OK
Server: nginx/1.6.2 (Ubuntu)
Date: Wed, 01 Jun 2016 18:44:00 GMT
Content-Type: text/html
Content-Length: 644
(...)

$ cargo run --bin tlsclient-mio -- --http expired.badssl.com
TLS error: InvalidCertificate(Expired)
Connection closed

Server example program

The MIO server example program is named tlsserver-mio. The interface looks like:

Runs a TLS server on :PORT.  The default PORT is 443.

`echo' mode means the server echoes received data on each connection.

`http' mode means the server blindly sends a HTTP response on each
connection.

`forward' means the server forwards plaintext to a connection made to
localhost:fport.

`--certs' names the full certificate chain, `--key' provides the
RSA private key.

Usage:
  tlsserver-mio --certs CERTFILE --key KEYFILE [--suite SUITE ...] [--proto PROTO ...] [--protover PROTOVER ...] [options] echo
  tlsserver-mio --certs CERTFILE --key KEYFILE [--suite SUITE ...] [--proto PROTO ...] [--protover PROTOVER ...] [options] http
  tlsserver-mio --certs CERTFILE --key KEYFILE [--suite SUITE ...] [--proto PROTO ...] [--protover PROTOVER ...] [options] forward <fport>
  tlsserver-mio (--version | -v)
  tlsserver-mio (--help | -h)

Options:
    -p, --port PORT     Listen on PORT [default: 443].
    --certs CERTFILE    Read server certificates from CERTFILE.
                        This should contain PEM-format certificates
                        in the right order (the first certificate should
                        certify KEYFILE, the last should be a root CA).
    --key KEYFILE       Read private key from KEYFILE.  This should be a RSA
                        private key or PKCS8-encoded private key, in PEM format.
    --ocsp OCSPFILE     Read DER-encoded OCSP response from OCSPFILE and staple
                        to certificate.  Optional.
    --auth CERTFILE     Enable client authentication, and accept certificates
                        signed by those roots provided in CERTFILE.
    --crl CRLFILE ...   Perform client certificate revocation checking using the DER-encoded
                        CRLFILE. May be used multiple times.
    --require-auth      Send a fatal alert if the client does not complete client
                        authentication.
    --resumption        Support session resumption.
    --tickets           Support tickets.
    --protover VERSION  Disable default TLS version list, and use
                        VERSION instead.  May be used multiple times.
    --suite SUITE       Disable default cipher suite list, and use
                        SUITE instead.  May be used multiple times.
    --proto PROTOCOL    Negotiate PROTOCOL using ALPN.
                        May be used multiple times.
    --verbose           Emit log output.
    --version, -v       Show tool version.
    --help, -h          Show this screen.

Here's a sample run; we start a TLS echo server, then connect to it with openssl and tlsclient-mio:

$ cargo run --bin tlsserver-mio -- --certs test-ca/rsa-2048/end.fullchain --key test-ca/rsa-2048/end.key -p 8443 echo &
$ echo hello world | openssl s_client -ign_eof -quiet -connect localhost:8443
depth=2 CN = ponytown RSA CA
verify error:num=19:self signed certificate in certificate chain
hello world
^C
$ echo hello world | cargo run --bin tlsclient-mio -- --cafile test-ca/rsa-2048/ca.cert -p 8443 localhost
hello world
^C

License

Rustls is distributed under the following three licenses:

Apache License version 2.0.
MIT license.
ISC license.

These are included as LICENSE-APACHE, LICENSE-MIT and LICENSE-ISC respectively. You may use this software under the terms of any of these licenses, at your option.

Project Membership

Joe Birr-Pixton (@ctz, Project Founder - full-time funded by Prossimo)
Dirkjan Ochtman (@djc, Co-maintainer)
Daniel McCarney (@cpu, Co-maintainer - full-time funded by Prossimo)
Josh Aas (@bdaehlie, Project Management)

Code of conduct

This project adopts the Rust Code of Conduct. Please email [email protected] to report any instance of misconduct, or if you have any comments or questions on the Code of Conduct.

rustls-ffi's People

Contributors

Stargazers

Watchers

rustls-ffi's Issues

rustls_error can cause undefined behavior when a non-error is passed

Steps to reproduce:

  char buf[256];
  size_t n;
  fprintf(stderr, "watch this:\n");
  rustls_error(0, buf, sizeof(buf), &n);

Result:

watch this:
Segmentation fault (core dumped)

This is because it's undefined behavior in Rust to have an enum where the contents are out of range for the enum definition, and there's no rustls_result corresponding to 0.

To fix this, we need to do some validation any time we accept a rustls_result from C. I think that's just in rustls_error.

Change rustls_*_session_write_tls to take callback

Right now, rustls_*_session_write_tls takes a buffer, and moves bytes from the session's internal buffer into the user provided one, for the user to send on to the network. As described in curl/curl#7007, this poses a problem: now the user code has to keep track of the bytes it received, in case they can't all be written to the network right away.

It would be simpler (and more efficient) to borrow bytes from rustls' internal buffer, write as many as we can, and tell rustls how many we consumed. In fact, that's how rustls works in the Rust world. write_tls takes a &dyn mut Write. The Write trait receives a borrowed buffer and returns the number of bytes it accepted.

The closest thing to traits in C is callbacks, and we have some well-established patterns for using callbacks in crustls to implement traits. We should do this with rustls_*_session_write_tls: take a callback and turn it into an &dyn mut Write implementation.

This would be a breaking API change. I don't think it's worth keeping around a function that offers the old copy-to-buffer functionality because it's easy to make mistakes with and hard to use correctly.

How to translate rustls config builders

The upcoming rustls release introduces config builders that use a typestate pattern to move between various partially-configured states into a final configuration: https://hoffman-andrews.com/rustls/rustls/struct.ConfigBuilder.html. This introduces a bunch of new types. Ideally we should translate these in a way that (a) preserves type safety to prevent mistakes, and (b) preserves the linker-friendly property introduced by rustls: if a program configures only 1 cipher suite, the linker should be able to eliminate all other cipher suites at link time (and similar for other dead code paths).

I wrote out what this would look like under our current idioms for translating Rust types and method names into C:

struct rustls_config_builder_with_suites;
struct rustls_config_builder_with_kx_groups;
struct rustls_config_builder_with_versions;
struct rustls_config_builder_with_suites;

struct rustls_client_config_builder;
struct rustls_client_config_builder_with_cert_verifier;
struct rustls_client_config_arc;

struct rustls_server_config_builder;
struct rustls_server_config_builder_with_client_auth;
struct rustls_server_config;
struct rustls_server_config_arc;

struct rustls_config_builder_with_versions *
rustls_config_builder_with_safe_defaults_new();

struct rustls_config_builder_with_suites *
rustls_config_builder_with_cipher_suites_new(ciphersuites);

struct rustls_config_builder_with_suites *
rustls_config_builder_with_safe_default_cipher_suites_new();

struct rustls_config_builder_with_kx_groups *
rustls_config_builder_with_suites_with_safe_default_kx_groups(
  struct rustls_config_builder_with_suites *rcb);

struct rustls_config_builder_with_versions *
rustls_config_builder_with_kx_groups_with_protocol_versions(
  struct rustls_config_builder_with_kx_groups *rcb,
  ...protocol versions...);

struct rustls_config_builder_with_versions *
rustls_config_builder_with_kx_groups_with_safe_default_protocol_versions(
  struct rustls_config_builder_with_kx_groups *rcb);

struct rustls_client_config_builder *
rustls_config_builder_with_versions_for_client(
  struct rustls_config_builder_with_versions *rcb);

struct rustls_server_config_builder *
rustls_config_builder_with_versions_for_server(
  struct rustls_config_builder_with_versions *rcb);

struct rustls_server_config_builder_with_client_auth *
rustls_server_config_builder_with_no_client_auth(
  struct rustls_server_config_builder *rcb);

struct rustls_server_config_builder_with_client_auth *
rustls_server_config_builder_with_client_cert_verifier(
  struct rustls_server_config_builder *rcb,
  ...verifier...);

struct rustls_server_config *
rustls_server_config_builder_with_client_auth_with_single_cert(
  struct rustls_server_config_builder_with_client_auth *rcb,
  ...certified key...);

struct rustls_server_config *
rustls_server_config_builder_with_client_auth_with_cert_resolver(
  struct rustls_server_config_builder_with_client_auth *rcb,
  ...resolver...);

struct rustls_client_config_builder_with_cert_verifier *
rustls_client_config_builder_with_root_certificates(
  struct rustls_client_config_builder *rcb,
  ...roots...);

struct rustls_client_config_builder_with_cert_verifier *
rustls_client_config_builder_with_custom_certificate_verifier(
  struct rustls_client_config_builder *rcb,
  ...verifier...);

struct rustls_client_config *
rustls_client_config_builder_with_cert_verifier_with_single_cert(
  struct rustls_client_config_builder_with_cert_verifier *rcb,
  ...certified key...);

struct rustls_client_config *
rustls_client_config_builder_with_cert_verifier_with_no_client_auth(
  struct rustls_client_config_builder_with_cert_verifier *rcb);

Fairly complex and burdensome. We could trim some of the length from the names by breaking our pattern of "methods on types are represented by a C function starting with 'rustls_' plus the name of the type." Instead we could start these all with rustls_config_ and let the type of the first argument be our indicator, but I think it doesn't improve the overall picture that much.

CARGO_PKG_VERSION: move to build.rs

Currently we load CARGO_PKG_VERSION by calling env! at runtime. However, from @jsha in Slack, it sounds like we could hardcode it.

The trick is to use build.rs, which I was reluctant to do previously. But now we have a build.rs for other reasons, so it would be fine to do that there.
Alternatively, we could just hardcode the CARGO_PKG_VERSION and make sure to update it in two places whenever we bump the version number.

The advantage to doing this would be that we could construct the version as a static string, which would avoid the need to allocate memory at runtime.

Rename `_config_builder_` functions to just `_config_`

I'm considering some renames. Specifically, we have all these functions named rustls_{client_server}_config_builder_foo. I introduced the "builder" concept to distinguish a ServerConfig from Arc without making the C user think about Rust's Arc type. But this didn't work out - to understand how these things work, you need to know a little about what Arc is doing, and we explain it in the comments.

We could make our function names a little shorter, and map more cleanly onto Rust types and names, if we called the config_builder just the config, and called the thing you finally build it into arc_config. So for instance:

struct rustls_server_config;
struct rustls_arc_server_config;

struct rustls_server_config* rustls_server_config_new();
const struct rustls_arc_server_config* rustls_arc_server_config_new(rustls_server_config*);

This would be an API break, but we are still at semver major 0 so that's okay:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

I'm interested in input on this!

Add client handshake example

The current model in tests/client.c has a workflow that looks like:

Create a rustls_connection (which does not involve reading or writing from the socket)
Write unencrypted bytes to the connection
In a for loop, alternate writes and reads based on the socket state and connection state. The handshake is performed implicitly, followed by the write of the unencrypted bytes.

Some of the C libraries I've seen have a different workflow - they have API's that look like

Socket connection and TLS handshake (cr_connect_nonblocking in curl)
API for writing data (cr_send in curl, pgtls_write in Postgres libpq/fe-secure.c)
API for reading data (cr_recv in curl, pgtls_read in Postgres)

It might be helpful to add a second example, or break out the logic in the current example so the handshake is performed separately. Especially since as implemented in vtls/rustls.c, the logic to perform a handshake is not obvious (call the write method with an empty buffer), and there's no explicit "handshake" method in crustls.h

Even if it's the same workflow under the hood, conceptually this might help when porting code that currently exists in C.

client certificate in outgoing connections

Thinking about adding support in rustls-ffi for client configs. The rustls functions are there, the question is how to make them concise int the ffi part:

ServerConfigs work on CertifiedKeys and ffi has functions to load these.
a CertifiedKey has a Vec<Certificate> and a SigningKey
ClientConfig's set_single_client_cert() gets a Vec<Certificate> and PrivateKey

I can see no way to downgrade a SigningKey to a PrivateKey. So, if applications load a CertifiedKey they seem to be unable to pass that to set_single_client_cert().

Correct?

Add a log facility

Rustls has some useful log lines. We should provide a way for C code to register a handler to receive those logs. Note that a single global handler isn't great, because this is a library. Also some downstream code (e.g. curl) may not have global state that we can hang log facilities off of.

curl+hyper+crustls fails to build on macOS due to Security.framework linkage

@kevinburke reports a failure building curl:

autoreconf -fi && ./configure --with-hyper=/Users/<pii>/src/github.com/hyperium/hyper --with-rustls=/Users/<pii>/src/github.com/rustls/rustls-ffi/target --without-ssl --without-secure-transport --without-nghttp2 --enable-debug --prefix=/Users/<pii>curl && make && make install && rehash && ~/curl/bin/curl --version

configure:27602: gcc -o conftest -Werror-implicit-function-declaration -g -O0 -pedantic -Wall -W -Wpointer-arith -Wwrite-strings -Wunused -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissin
g-prototypes -Wno-long-long -Wbad-function-cast -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wold-style-defi
nition -Wstrict-aliasing=3 -Wcast-align -Wtype-limits -Wold-style-declaration -Wmissing-parameter-type -Wempty-body -Wclobbered -Wignored-qualifiers -Wconversion -Wno-sign-conversion -Wvla -ftree-vrp -Wdo
uble-promotion -Wformat=2 -Warray-bounds=2 -Wshift-negative-value -Wshift-overflow=2 -Wnull-dereference -fdelete-null-pointer-checks -Wduplicated-cond -Wunused-const-variable -Wduplicated-branches -Wrestr
ict -Walloc-zero -Wformat-overflow=2 -Wformat-truncation=2 -Wimplicit-fallthrough=4 -Wno-system-headers -Wenum-conversion  -I/Users/<pii>/src/github.com/hyperium/hyper/capi/include  -I/Users/<pii>/src/git
hub.com/rustls/rustls-ffi/target/include  -framework CoreFoundation -framework SystemConfiguration -L/Users/<pii>/src/github.com/hyperium/hyper/target/debug  -L/Users/<pii>/src/github.com/rustls/rustls-ff
i/target/lib conftest.c -lcrustls -lpthread -ldl -lldap -lz -lhyper -ldl -lpthread -lm  >&5
conftest.c:46:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
   46 | char rustls_connection_read ();
      | ^~~~
Undefined symbols for architecture x86_64:
  "_SecRandomCopyBytes", referenced from:
      __ZN4ring4rand6darwin4fill17hef096156cdbb5e22E in libcrustls.a(ring-059d35c0cff8849f.ring.4rb6t1i9-cgu.13.rcgu.o)
      __ZN4ring2ec7suite_b5ecdsa7signing12EcdsaKeyPair3new17h59971278221fac36E in libcrustls.a(ring-059d35c0cff8849f.ring.4rb6t1i9-cgu.14.rcgu.o)
  "_kSecRandomDefault", referenced from:
      __ZN4ring4rand6darwin4fill17hef096156cdbb5e22E in libcrustls.a(ring-059d35c0cff8849f.ring.4rb6t1i9-cgu.13.rcgu.o)
      __ZN4ring2ec7suite_b5ecdsa7signing12EcdsaKeyPair3new17h59971278221fac36E in libcrustls.a(ring-059d35c0cff8849f.ring.4rb6t1i9-cgu.14.rcgu.o)
ld: symbol(s) not found for architecture x86_64

I believe the problem here is that the gcc invocation is missing -framework Security, needed to resolve SecRandomCopyBytes. That symbol is needed by ring::rand::SecureRandom on macOS (see also briansmith/ring#149). However, building curl with --without-secure-transport causes it to not link Security.framework when building on Darwin.

QUIC support

rustls supports QUIC (now published as an RFC). OpenSSL still does not have QUIC support upstream. It would be nice if we exposed QUIC support in the bindings.

dtolnay/cxx?

I searched for an evaluation of https://github.com/dtolnay/cxx, but didn't find one. I've had really good luck with it, and it might work well in this project alongside a C header.

Licensing

Hi there, I just noticed I couldn't find licensing information in this repository.

Document all function calls

I can go to e.g. https://docs.rs/rustls/0.18.1/rustls/ and get an overview of all of the function calls and types in the rustls package. It would be nice if there was a page that documented these for rustls-ffi that was available from the README or Github homepage.

Implement macro for opaque struct + CastPtr + BoxCastPtr

We have a recurring pattern in this repository:

pub struct rustls_server_config_builder {
    // We use the opaque struct pattern to tell C about our types without
    // telling them what's inside.
    // https://doc.rust-lang.org/nomicon/ffi.html#representing-opaque-structs
    _private: [u8; 0],
}

impl CastPtr for rustls_server_config_builder {
    type RustType = ServerConfig;
}

impl BoxCastPtr for rustls_server_config_builder {}

We should define a macro such that we can simply all this boilerplate to:

c_type!(rustls_server_config_builder, ServerConfig);

Or, to include BoxCastPtr:

c_type_boxed!(rustls_server_config_builder, ServerConfig);

We should make sure that doccomments right above the macro invocation wind up in crustls.h associated with the struct declaration.

Expected type did not match the received type.

https://github.com/abetterinternet/crustls/blob/88564644d8fe24a2bf95e1f55ef5e976221be26e/src/client.rs#L465-L469

    let session: &Sess = try_ref_from_ptr!(session);
    match session.session.get_protocol_version() {
        Some(v) => v.get_u16(),
        None => 0
    }

https://github.com/abetterinternet/crustls/blob/88564644d8fe24a2bf95e1f55ef5e976221be26e/src/client.rs#L487-L499

    let session: &Sess = try_ref_from_ptr!(session);
    let protocol_out = try_mut_from_ptr!(protocol_out);
    let protocol_out_len = try_mut_from_ptr!(protocol_out_len);
    match session.session.get_alpn_protocol() {
        Some(p) => {
            *protocol_out = p.as_ptr();
            *protocol_out_len = p.len();
        },
        None => {
            *protocol_out = null();
            *protocol_out_len = 0;
        }
    }

All PR's currently fail CI

It looks like there is a check related to pernos.co that is currently failing all builds, which can obscure the presence of legitimate failures. Could we disable the check, or get it working again?

Ciphersuite selection philosophy

In #56 we're working towards exporting ciphersuites so that servers can select the ones they want to support. However, we should consider if this is a feature of the rustls API that we want to implement. The rustls default is to offer all of the ciphersuites it supports, which is a good, modern selection without known issues. So the purpose of configuring custom ciphersuites would presumably be to disable a ciphersuite that is found to be broken at some point in the future. However, if that turns out to be the case, rustls would (presumably) ship a new release removing that ciphersuite, and crustls would do that same; so the fix, like fixes for other CVEs, would be to upgrade your libraries.

One argument in favor of supporting configuration of ciphersuites is that rustls supports it, and since we are a translation layer for rustls, we should support it to. But crustls is (for now) a subset of rustls and we focus on implementing the features that are most needed.

Adam Langley has a post about cryptographic agility and its cost here: https://www.imperialviolet.org/2016/05/16/agility.html.

/cc @icing

rustls_connection_get_peer_certificate() broken

If I read this correctly now, the rustls v0.19.x returns session.get_peer_certificates() as copy. This will be changed in the next release to a reference.

The crustls rustls_connection_get_peer_certificate() returns pointers to these copies in 0.6.0, any use is use after free.

Either these need to be Arced or the next rustls is needed, I guess?

error: ‘content_length’ may be used uninitialized

rustls-ffi 0.7.0 fails to build in our CI due to this error. The code seems to look like this in master as well.

cbindgen --lang C > src/crustls.h
WARN: Skip crustls::RUSTLS_CRATE_VERSION - (not `pub`).
WARN: Missing `[defines]` entry for `unix` in cbindgen config.
WARN: Missing `[defines]` entry for `unix` in cbindgen config.
cc -o target/client.o -c tests/client.c -Werror -Wall -Wextra -Wpedantic -g -I src/ -O3
tests/client.c: In function ‘send_request_and_read_response’:
tests/client.c:319:42: error: ‘content_length’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
            conn->data.len >= headers_len + content_length) {
                              ~~~~~~~~~~~~^~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Makefile:40: recipe for target 'target/client.o' failed
make: *** [target/client.o] Error 1

Support compiling crustls-demo on Windows

I recently switched from epoll to select to be more broadly compatible, and added macOS builds in CI: #14.

I'd also like to provide Windows builds in CI. Besides the change to use select, it looks like I would need some additional tweaks to use winsock: https://tangentsoft.net/wskfaq/articles/bsd-compatibility.html. I haven't done network programming on Windows, and don't have a Windows dev environment yet, so it would probably take me a while. @gvanem, you provided the starter Makefile.Windows (thanks!); do you have any interest in doing the winsock changes?

Homebrew package for curl with rustls support

Hi,
I put together a Homebrew package for curl with rustls+hyper support built in. You can try it for yourself here:

brew install --HEAD meterup/safe/curl
==> Cloning https://github.com/curl/curl.git
Updating /Users/kevin/Library/Caches/Homebrew/curl--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at e992770 test433: clear some home dir env variables
==> Reinstalling meterup/safe/curl
==> autoreconf -fi
==> ./configure --prefix=/usr/local/Cellar/curl/HEAD-e992770 --without-ca-bundle --without-ca-path --with-secure-transport --with-gssapi --with-libidn2 --with-libmetalink --with-librtmp --without-libpsl --with-hyper=/usr/local/opt/hyper --with-rustls=/usr/local/opt/crus
==> make install
==> make install -C scripts
==> Caveats
curl is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have curl first in your PATH, run:
  echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc

For compilers to find curl you may need to set:
  export LDFLAGS="-L/usr/local/opt/curl/lib"
  export CPPFLAGS="-I/usr/local/opt/curl/include"

For pkg-config to find curl you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/curl/lib/pkgconfig"


zsh completions have been installed to:
  /usr/local/opt/curl/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/curl/HEAD-e992770: 481 files, 11.5MB, built in 2 minutes 47 seconds

$ curl --version
curl 7.76.0-DEV (x86_64-apple-darwin19.6.0) libcurl/7.76.0-DEV SecureTransport (crustls/0.2.0/rustls/0.19.0) zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.0 nghttp2/1.43.0 librtmp/2.3 Hyper/0.14.4
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli Debug GSS-API HTTP2 IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TrackMemory UnixSockets zstd

You'll need the --HEAD until curl 7.76.0 is released.

You can see the package source code here: https://github.com/meterup/homebrew-safe.

One thing to note is curl gets a "Bad" rating from How's My SSL.

curl https://www.howsmyssl.com/a/check | python -mjson.tool
{
    "given_cipher_suites": [
        "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
        "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
        "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
        "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
        "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
        "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384",
        "TLS_DHE_RSA_WITH_AES_256_CBC_SHA256",
        "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
        "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
        "TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
        "TLS_GOST2012256-GOST89-GOST89",
        "TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256",
        "TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA",
        "TLS_GOST2001-GOST89-GOST89",
        "TLS_RSA_WITH_AES_256_GCM_SHA384",
        "TLS_RSA_WITH_AES_256_CBC_SHA256",
        "TLS_RSA_WITH_AES_256_CBC_SHA",
        "TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256",
        "TLS_RSA_WITH_CAMELLIA_256_CBC_SHA",
        "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
        "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
        "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
        "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
        "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
        "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256",
        "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256",
        "TLS_DHE_RSA_WITH_AES_128_CBC_SHA",
        "TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256",
        "TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA",
        "TLS_RSA_WITH_AES_128_GCM_SHA256",
        "TLS_RSA_WITH_AES_128_CBC_SHA256",
        "TLS_RSA_WITH_AES_128_CBC_SHA",
        "TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256",
        "TLS_RSA_WITH_CAMELLIA_128_CBC_SHA",
        "TLS_ECDHE_RSA_WITH_RC4_128_SHA",
        "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA",
        "TLS_RSA_WITH_RC4_128_SHA",
        "TLS_RSA_WITH_RC4_128_MD5",
        "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA",
        "TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA",
        "TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA",
        "TLS_RSA_WITH_3DES_EDE_CBC_SHA",
        "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
    ],
    "ephemeral_keys_supported": true,
    "session_ticket_supported": false,
    "tls_compression_supported": false,
    "unknown_cipher_suite_supported": false,
    "beast_vuln": false,
    "able_to_detect_n_minus_one_splitting": false,
    "insecure_cipher_suites": {
        "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA": [
            "uses RC4 which has insecure biases in its output"
        ],
        "TLS_ECDHE_RSA_WITH_RC4_128_SHA": [
            "uses RC4 which has insecure biases in its output"
        ],
        "TLS_RSA_WITH_RC4_128_MD5": [
            "uses RC4 which has insecure biases in its output"
        ],
        "TLS_RSA_WITH_RC4_128_SHA": [
            "uses RC4 which has insecure biases in its output"
        ]
    },
    "tls_version": "TLS 1.2",
    "rating": "Bad"
}

Build broken on Windows by #112

When I build now, I get:

   Compiling rustls v0.19.1=======>      ] 19/24: sct, webpki, ring
error[E0432]: unresolved imports `crate::io::rustls_write_vectored_callback`, `crate::io::VectoredCallbackWriter`, `crat
e::io::VectoredWriteCallback`
 --> src\connection.rs:8:5
  |
8 |     rustls_write_vectored_callback, CallbackReader, CallbackWriter, ReadCallback,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |     |
  |     no `rustls_write_vectored_callback` in `io`
  |     help: a similar name exists in the module: `rustls_write_callback`
9 |     VectoredCallbackWriter, VectoredWriteCallback, WriteCallback,
  |     ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^ no `VectoredWriteCallback` in `io`
  |     |
  |     no `VectoredCallbackWriter` in `io`

What!?

Originally posted by @gvanem in #112 (comment)

Build a container for miri tests

Right now we're getting this error for the miri tests:

error: component 'miri' for target 'x86_64-unknown-linux-gnu' is unavailable for download for channel nightly
Sometimes not all components are available in any given nightly. If you don't need the component, you can remove it with:

    rustup component remove --toolchain nightly --target x86_64-unknown-linux-gnu miri

We should build a container with a given version of nightly that we know has the miri component, so we're not vulnerable to breakages like this. That would also be beneficial for build times, since right now we build some things for the miri test job, and that's slow.

Define an array that contains all of the TLS versions

Postgres has configuration flags that let you set the minimum and the maximum TLS versions. I am thinking about implementing this with logic like this:

min_version = conn->ssl_min_version or 'TLS 1.2'
max_version = conn->ssl_max_version or None

versions = []
for version in rustls_tls_versions:
    if version >= min_version and (max_version is None or version <= max_version):
        versions.push(version)
rustls_client_config_builder_set_versions(versions)

The question is how to implement rustls_tls_versions... I can define all of the constants myself, in an array, but then if rustls adds TLS1.4, or TLS2, my array is going to be out of date, and people can't select a higher version, even if rustls can negotiate it..

It might be better if rustls-ffi declares an array containing all of the TLS versions. I can use that in my code to ensure that I'm not accidentally bounding the maximum TLS version that can be negotiated.

Alternatively, rustls-ffi could have a function that accepts a minimum and a maximum and returns me a uint16_t*.

git-remote-https exits with 11 return code

Steps to reproduce:

install Git with Hyper+Rustls support (I set this up at brew install meterup/safe/git if you want to give it a try)
check out the Go source tree from https://go.googlesource.com/go
Try to run "git pull origin master"

Running with GIT_TRACE=1 GIT_CURL_VERBOSE=1 I get the following error message:

$ GIT_TRACE=1 GIT_CURL_VERBOSE=1 glob
12:27:20.721657 git.c:444               trace: built-in: git rev-parse --symbolic-full-name --abbrev-ref HEAD
12:27:20.728892 git.c:444               trace: built-in: git pull origin master
12:27:20.731244 run-command.c:664       trace: run_command: git fetch --update-head-ok origin master
12:27:20.735904 git.c:444               trace: built-in: git fetch --update-head-ok origin master
12:27:20.739273 run-command.c:664       trace: run_command: GIT_DIR=.git git remote-https origin https://go.googlesource.com/go
12:27:20.743706 git.c:730               trace: exec: git-remote-https origin https://go.googlesource.com/go
12:27:20.744143 run-command.c:664       trace: run_command: git-remote-https origin https://go.googlesource.com/go
12:27:20.754890 http.c:756              == Info: STATE: INIT => CONNECT handle 0x7fcaf5821608; line 1646 (connection #-5000)
12:27:20.755407 http.c:756              == Info: Couldn't find host go.googlesource.com in the .netrc file; using defaults
12:27:20.755422 http.c:756              == Info: Added connection 0. The cache now contains 1 members
12:27:20.755480 http.c:756              == Info: STATE: CONNECT => WAITRESOLVE handle 0x7fcaf5821608; line 1692 (connection #0)
12:27:20.758742 http.c:756              == Info: family0 == v4, family1 == v6
12:27:20.758779 http.c:756              == Info:   Trying 74.125.137.82:443...
12:27:20.758864 http.c:756              == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x7fcaf5821608; line 1774 (connection #0)
12:27:20.789167 http.c:756              == Info: Connected to go.googlesource.com (74.125.137.82) port 443 (#0)
12:27:20.789179 http.c:756              == Info: STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x7fcaf5821608; line 1840 (connection #0)
12:27:20.792759 http.c:756              == Info: ALPN, offering http/1.1
12:27:20.792876 http.c:756              == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x7fcaf5821608; line 1858 (connection #0)
12:27:20.845354 http.c:756              == Info: TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
12:27:20.845566 http.c:756              == Info: Server certificate: *.googlecode.com
12:27:20.845678 http.c:756              == Info: Server certificate: GTS CA 1O1
12:27:20.845729 http.c:756              == Info: Server certificate: GlobalSign
12:27:20.845736 http.c:756              == Info: STATE: PROTOCONNECT => DO handle 0x7fcaf5821608; line 1877 (connection #0)
12:27:20.846111 http.c:756              == Info: Time for the Hyper dance
12:27:20.851404 http.c:703              => Send header, 0000000052 bytes (0x00000034)
12:27:20.851419 http.c:715              => Send header: GET /go/info/refs?service=git-upload-pack HTTP/1.1
12:27:20.851890 http.c:703              => Send header, 0000000027 bytes (0x0000001b)
12:27:20.851901 http.c:715              => Send header: Host: go.googlesource.com
12:27:20.851908 http.c:703              => Send header, 0000000024 bytes (0x00000018)
12:27:20.851910 http.c:715              => Send header: User-Agent: git/2.30.1
12:27:20.851914 http.c:703              => Send header, 0000000013 bytes (0x0000000d)
12:27:20.851917 http.c:715              => Send header: Accept: */*
12:27:20.851935 http.c:703              => Send header, 0000000098 bytes (0x00000062)
12:27:20.851939 http.c:715              => Send header: Cookie: o=<redacted>
12:27:20.851944 http.c:703              => Send header, 0000000033 bytes (0x00000021)
12:27:20.851947 http.c:715              => Send header: Accept-Language: en-US, *;q=0.9
12:27:20.857064 http.c:703              => Send header, 0000000018 bytes (0x00000012)
12:27:20.857089 http.c:715              => Send header: Pragma: no-cache
12:27:20.857120 http.c:703              => Send header, 0000000025 bytes (0x00000019)
12:27:20.857127 http.c:715              => Send header: Git-Protocol: version=2
12:27:20.857131 http.c:703              => Send header, 0000000002 bytes (0x00000002)
12:27:20.857135 http.c:715              => Send header:
12:27:20.858702 http.c:756              == Info: STATE: DO => DO_DONE handle 0x7fcaf5821608; line 1935 (connection #0)
12:27:20.858715 http.c:756              == Info: STATE: DO_DONE => PERFORM handle 0x7fcaf5821608; line 2056 (connection #0)
12:27:21.411592 http.c:756              == Info: HTTP 1.1 or later with persistent connection
12:27:21.411706 http.c:703              <= Recv header, 0000000017 bytes (0x00000011)
12:27:21.411717 http.c:715              <= Recv header: HTTP/1.1 200 OK
error: git-remote-https died of signal 11

The most obvious issue I can think of is a connection that's using TLS 1.1 or older but that doesn't seem to be the case here - it says TLS 1.2.

Let me know what other information you'd need to help track this down. I tried running with RUST_BACKTRACE=1 but that didn't yield anything useful.

Run tests on Windows

#120 enables CI builds for Windows, but we don't yet run tests. Makefile.Windows needs to be updated to build tests/*.c instead of src/client.c, and there's a comment in there claiming that it doesn't work anyway due to a dependency on epoll(7) which we should look into.

[Win32] Crashes with ASAN

I've built and used CrustTls in the demo and libcurl for some time with great success.

But enabling AddressSanitizer (ASAN) feature of latest MSVC-2019 (cl ver. 19.28.29912 for x86),
both curl https://whatever ... and a crustls-demo.exe www.vg.no / > test.file abort with this trace:

Sending:
GET / HTTP/1.1
Host: www.vg.no
...
ClientSession wants us to write_tls.
ClientSession wants us to read_tls. First we need to pull some bytes from the socket
=================================================================
==15612==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x07487200,0x07487735) and [0x0748727f, 0x074877b4) overlap
    #0 0x611a1dcf  (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x10031dcf)
    #1 0x106beb in _ZN6rustls4msgs8deframer15MessageDeframer4read17hd804cc7f869bdd48E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x406beb)
    #2 0x1129aa in _ZN74_$LT$rustls..client..ClientSession$u20$as$u20$rustls..session..Session$GT$8read_tls17h1cfd14ff7a451970E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x4129aa)
    #3 0x102673 in rustls_client_session_read_tls (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x402673)
    #4 0x24e247 in copy_tls_bytes_into_client_session F:\MingW32\src\inet\Crypto\Crustls\src\main.c:217
    #5 0x24e5d3 in do_read F:\MingW32\src\inet\Crypto\Crustls\src\main.c:345
    #6 0x24f621 in send_request_and_read_response F:\MingW32\src\inet\Crypto\Crustls\src\main.c:434
    #7 0x24ebf0 in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
    #8 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #9 0x7656fa28  (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
    #10 0x771d7c7d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
    #11 0x771d7c4d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)

0x07487200 is located 0 bytes inside of 18437-byte region [0x07487200,0x0748ba05) allocated by thread T0 here:
    #0 0x611aa74c  (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x1003a74c)
    #1 0x140d8c in __rdl_alloc /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\alloc.rs:356
    #2 0x112685 in _ZN6rustls6client13ClientSession3new17habc44068009b2037E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x412685)
    #3 0x158a95 in _ZN7crustls6client25rustls_client_session_new28_$u7b$$u7b$closure$u7d$$u7d$17hfc2e02940a78157fE.llvm.16205319766425828457 (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x458a95)
    #4 0x101b7d in rustls_client_session_new (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x401b7d)
    #5 0x24ebbb in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
    #6 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #7 0x7656fa28  (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
    #8 0x771d7c7d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
    #9 0x771d7c4d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)

0x0748727f is located 127 bytes inside of 18437-byte region [0x07487200,0x0748ba05) allocated by thread T0 here:
    #0 0x611aa74c  (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x1003a74c)
    #1 0x140d8c in __rdl_alloc /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4\/library\std\src\alloc.rs:356
    #2 0x112685 in _ZN6rustls6client13ClientSession3new17habc44068009b2037E (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x412685)
    #3 0x158a95 in _ZN7crustls6client25rustls_client_session_new28_$u7b$$u7b$closure$u7d$$u7d$17hfc2e02940a78157fE.llvm.16205319766425828457 (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x458a95)
    #4 0x101b7d in rustls_client_session_new (F:\MingW32\src\inet\Crypto\Crustls\crustls-demo.exe+0x401b7d)
    #5 0x24ebbb in main F:\MingW32\src\inet\Crypto\Crustls\src\main.c:591
    #6 0x26038e in _scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #7 0x7656fa28  (C:\WINDOWS\System32\KERNEL32.DLL+0x6b81fa28)
    #8 0x771d7c7d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c7d)
    #9 0x771d7c4d  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e7c4d)

SUMMARY: AddressSanitizer: memcpy-param-overlap (f:\gv\VC_2019\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\clang_rt.asan_dynamic-i386.dll+0x10031dcf)
==15612==ABORTING

CFLAGS used: -fsanitize=address.
Ref: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes#16.9.0

A clang-cl v11 version with the same CFLAGS reports the same. The crash happens right after the 1st recv():

  * 0.292 sec: src/main.c(319) (do_read+149):
    recv (860, 0x0073E890, 2048, 0) --> 2048 bytes.
    0000: 16 03 03 00 7A 02 00 00 76 03 03 46 77 60 1C 0A  ....z...v..Fw`..
    0010: 29 CA 39 FE 7B 37 6E 30 97 77 9C 69 F5 A5 51 2A  )-9¦{7n0ùw£i)ÑQ*
    0020: 8E B8 0B 64 CC A5 64 56 62 6F 15 20 54 3D 19 2B  Ä+.d¦ÑdVbo. T=.+
    0030: 5B 16 7C 0B C7 34 60 9A C0 88 A5 A8 D5 D6 92 71  [.|.¦4`Ü+êÑ¿++Æq
    0040: A2 98 E2 13 2E 3D 57 33 30 7B FA 69 13 03 00 00  óÿG..=W30{·i....
    0050: 2E 00 33 00 24 00 1D 00 20 80 AB 91 09 58 98 98  ..3.$... Ç½æ.Xÿÿ
    0060: 74 19 EB D8 97 98 EF 37 C5 88 80 E5 5A 72 17 10  t.d+ùÿn7+êÇsZr..
    0070: 6C 52 23 7E 49 B1 7A ED 53 00 2B 00 02 03 04 14  lR#~I¦zfS.+.....
    0080: 03 03 00 01 01 17 03 03 09 B7 C2 6D 9F 4B C1 92  .........+-mƒK-Æ
    0090: 1E E6 5D BF 27 7F 44 C5 03 40 A8 E5 2C EC F8 0B  .µ]+'.D+.@¿s,8°.
    00A0: 3D CC 0C 13 27 11 6A 46 07 2A 40 2A 76 1F 17 36  =¦..'.jF.*@*v..6
    00B0: BA 6A DD A5 2E 76 7D AA 08 1D C4 ED AF ED 87 C9  ¦j¦Ñ.v}¬..-f¤fç+
    00C0: A6 01 5F 80 8D 6E 97 1B 33 7F EB D0 28 50 14 46  ª._Çìnù.3.d-(P.F
    00D0: C8 90 73 1F 04 CF 05 9D 8A A1 5D 3F 3C 35 B1 7B  +És..-.Øèí]?<5¦{
    00E0: 86 69 C5 B0 40 E0 7E 01 FB B4 5B 0D 85 E7 5E 0A  åi+¦@a~.v¦[.àt^.
    00F0: 52 42 31 49 0B 0B E5 77 F5 C4 B0 55 0B E5 99 CA  RB1I..sw)-¦U.sÖ-
    0100: D5 44 3F D0 A9

(courtesy of my Wsock-trace library)

Add panic handling to each function

Since unwinding across an FFI boundary is undefined behavior (rust-lang/rust#58794), we should handle panics and return errors instead of continuing to unwind.

Readme mentions nonexistent function

Curl configure checks for rustls by looking for a function named rustls_client_session_read in crustls.h, and says it can't find rustls if that function doesn't exist, even if crustls.h is in the correct place.

On main, that function is only described in the README, but not in the code - I believe it disappeared after v0.5.0.

"unspecific protocol error detected" making request to go.googlesource.com

Hi, not sure which library is the right place for this one, so making a placemaker issue... I'm compiling curl on Mac using the Homebrew formula available at github.com/meterup/homebrew-safe. You can install using brew install meterup/safe/curl.

$ curl --version
curl 7.78.0 (x86_64-apple-darwin19.6.0) libcurl/7.78.0 crustls/0.7.1/rustls/0.19.0 zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.2 librtmp/2.3 Hyper/0.14.11 OpenLDAP/2.5.5
Release-Date: 2021-07-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli Debug GSS-API HSTS HTTP2 IDN IPv6 Kerberos Largefile libz SPNEGO SSL TrackMemory UnixSockets zstd

$ curl https://go.googlesource.com
curl: (56) Hyper: [1] error reading a body from connection: protocol error: unspecific protocol error detected

Here's the verbose logs:

$ curl https://go.googlesource.com -vvv
* STATE: INIT => CONNECT handle 0x7f944c010408; line 1780 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => RESOLVING handle 0x7f944c010408; line 1826 (connection #0)
* family0 == v4, family1 == v6
*   Trying 142.251.2.82:443...
* STATE: RESOLVING => CONNECTING handle 0x7f944c010408; line 1908 (connection #0)
* Connected to go.googlesource.com (142.251.2.82) port 443 (#0)
* STATE: CONNECTING => PROTOCONNECT handle 0x7f944c010408; line 1973 (connection #0)
* offering ALPN for HTTP/1.1 and HTTP/2
* rustls_connection wants us to write_tls.
* cr_send 0 bytes of plaintext
* cr_send wrote 264 bytes to network
* rustls_connection wants us to read_tls.
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* reading would block
* Curl_socket_check: reading would block
* STATE: PROTOCONNECT => PROTOCONNECTING handle 0x7f944c010408; line 1991 (connection #0)
* rustls_connection wants us to read_tls.
* cr_recv read 2830 bytes from the network
* reading would block
* Done handshaking
* ALPN, negotiated h2
* STATE: PROTOCONNECTING => DO handle 0x7f944c010408; line 2010 (connection #0)
* Time for the Hyper dance
* cr_send 24 bytes of plaintext
* cr_send wrote 6 bytes to network
* cr_send wrote 58 bytes to network
* cr_send wrote 46 bytes to network
> GET / HTTP/2
> Host: go.googlesource.com
> User-Agent: curl/7.78.0
> Accept: */*
>
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* cr_recv got 0 bytes of plaintext
* cr_send 40 bytes of plaintext
* cr_send wrote 62 bytes to network
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* cr_recv got 0 bytes of plaintext
* cr_send 42 bytes of plaintext
* cr_send wrote 64 bytes to network
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* cr_recv got 0 bytes of plaintext
* STATE: DO => DID handle 0x7f944c010408; line 2068 (connection #0)
* STATE: DID => PERFORMING handle 0x7f944c010408; line 2187 (connection #0)
* cr_recv read 639 bytes from the network
* cr_recv copied out 49 bytes of plaintext
* cr_recv got 0 bytes of plaintext
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* cr_recv got 0 bytes of plaintext
* cr_send 9 bytes of plaintext
* cr_send wrote 31 bytes to network
* cr_recv read 117 bytes from the network
* cr_recv copied out 95 bytes of plaintext
* cr_recv got 0 bytes of plaintext
* sread: EAGAIN or EWOULDBLOCK
* cr_recv read 0 bytes from the network
* cr_recv got 0 bytes of plaintext
* HTTP/2 found, allow multiplexing
< HTTP/2 400 Bad Request
< content-type: text/html; charset=UTF-8
< referrer-policy: no-referrer
< content-length: 1555
< date: Fri, 23 Jul 2021 04:16:34 GMT
<
* Hyper: [1] error reading a body from connection: protocol error: unspecific protocol error detected
* multi_done
* The cache now contains 0 members
* Closing connection 0
* cr_send 0 bytes of plaintext
* cr_send wrote 24 bytes to network
* Expire cleared (transfer 0x7f944c010408)
curl: (56) Hyper: [1] error reading a body from connection: protocol error: unspecific protocol error detected

Requests to some other HTTPS URL's work okay, for example jsonip.com or meterapi.com.

Update wiki "building crustls" instructions

The repository names are no longer correct. I would update it myself but apparently I need access in order to edit.

Bazel

I started working on getting this project to compile with Bazel and BuildBuddy. This effort will result in reproducible builds and cached build results (meaning that something already done on a branch won't require a recompile).

I'll need to use cargo-raze to pull in the Cargo deps. But, this way, the project will have a build that pulls in a fixed Rust and C/C++ toolchain. (Bazel downloads it all, and doesn't depend on the local machine)

https://github.com/grafica/crustls/tree/buildbuddy

client certs (yes/no/how)

Lets develop some understanding if, and if yes how we want client certificates to work in crustls and the basic methods/struct that we want to have for it.

Looking at what rustls and webpki offer here, there are 2 areas where functionality seems to be lacking:

access to certificate details, to identify the cert (subject/email)
revocations checks

Point 2 is the hairy one, I guess. Without it, any client certificate authentication seems to be pointless and dangerous. rustls uses webpki's cert.verify_is_valid_tls_client_cert() and that checks the trust chain signatures only. So, there would need to be some mechanism to let the application do revocation checks on a signature-checked certificate chain (the intermediaries might need checking as well).

As an application example: mod_tls would not want to implement parsing and evaluating CLRs, I guess. OCSP could be added by extending mod_md for this, but this is not trivial.

Do we want to do this?

`rustls_error` equivalent for `rustls_io_result`

I can call rustls_error on a rustls_result to get a string explaining it, but I can't do the same for a rustls_io_result.

I believe that a rustls_io_result is supposed to represent values like EAGAIN, EWOULDBLOCK, EINTR etc, but the existing documentation is not really clear about that:

/// A return value for a function that may return either success (0) or a
/// non-zero value representing an error.

Retrieve the default set of ciphersuites

There's a new rustls_server_config_builder_new_with_safe_defaults helper. Unfortunately Postgres has a functionality that allows users to set the min and max TLS versions, so I still need to use rustls_server_config_builder_new, which looks like the only way to set those.

That function requires me to also pass in a list of cipher suites. I can get all of the ciphersuites, but as far as I can tell, there's no way to get the default set of ciphersuites that are used by e.g. the with_safe_defaults function, in C code?

Would it be possible to expose that default list? For example:

/**
 * Return the length of rustls' list of default cipher suites.
 */
size_t rustls_default_ciphersuites_len(void);

/**
 * Get a pointer to a member of rustls' list of supported cipher suites. This will return non-NULL
 * for i < rustls_default_ciphersuites_len().
 * The returned pointer is valid for the lifetime of the program and may be used directly when
 * building a ClientConfig or ServerConfig.
 */
const struct rustls_supported_ciphersuite *rustls_default_ciphersuites_get_entry(size_t i);

Test that all functions are wrapped in a panic boundary

Followup from #25. We should have automated tests to ensure newly added functions are treated with an ffi_panic_boundary_* macro.

Please consider using cargo-c

It has the logic to build header, library (including the dynamic variant) and pkg-config file.

https://crates.io/crates/cargo-c

Get a given ciphersuite as a string

As far as I can tell, it's impossible to translate between a rustls_supported_ciphersuite and the actual string value of that ciphersuite. For example if I am in a C program, and rustls has negotiated TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for me, I (or specifically psql) would like to display that to the user but the best I can get right now is a u16, I believe, unless I manually copy all of the constants out from the Rust program.

Add support for selecting ciphersuites and TLS versions

Remove defensive zeroing

In rustls_server_session_read and rustls_client_session_read, we receive a buffer from C, turn it into a &mut [u8], zeroize it, and then pass it to Session::read.

        let read_buf: &mut [u8] = unsafe {
            if buf.is_null() {
                return NullParameter;
            }
            slice::from_raw_parts_mut(buf, count as usize)
        };
        // Since it's *possible* for a Read impl to consume the possibly-uninitialized memory from buf,
        // zero it out just in case. TODO: use Initializer once it's stabilized.
        // https://doc.rust-lang.org/nightly/std/io/trait.Read.html#method.initializer
        for c in read_buf.iter_mut() {
            *c = 0;
        }

RFC 2930 explains this quite well:

The current design of the Read trait is nonoptimal as it requires that the buffer passed to its various methods be pre-initialized even though the contents will be immediately overwritten. This RFC proposes an interface to allow implementors and consumers of Read types to robustly and soundly work with uninitialized buffers.

However, the current defensive zeroing we have is actually incorrect. Per from_raw_parts_mut:

Behavior is undefined if any of the following conditions are violated:
...
data must point to len consecutive properly initialized values of type T.

In other words, if the C code passes us uninitialized memory, we will already have invoked undefined behavior when we called from_raw_parts_mut, before we zeroize the memory ourselves.

Also, the zeroizing we do is inefficient. @icing has measured a significant slowdown attributed to it. In practice, we don't need to zeroize memory with each call, we just need to be sure that the memory has well-defined contents (i.e. is not uninitialized). The C caller can do that by allocating a buffer with calloc and then reusing it across multiple read calls.

We should remove the zeroization on our side and thoroughly document the requirement that the caller MUST initialize the memory once before passing it to us (and can then reuse the same memory without re-initializing it).

Note that this is fairly risky: the requirement to initialize memory that's going to be overwritten is definitely unintuitive, and relying on documentation to make it happen may not be sufficient. Another approach would be to offer a new opaque struct, rustls_byte_buffer, that would be allocated and initialized in Rust. We would change the signature of the _read functions to take one of these, so we would be guaranteed to be working with initialized memory. Callers would request one of these with the appropriate size when they create a session, and hang on to it through the session's lifetime, reusing it for efficiency. The struct would be opaque so that users can't create one from uninitialized memory; we would provide access to the underlying bytes via a method.

Also worth noting: There is a semi-related issue that strange Read impls may technically read from their output slice in addition to writing it. We're not worried about that because we know that rustls' Read impls behavior normally and don't read from their input slice.

This seems useful and related but I haven't read it yet: https://www.ralfj.de/blog/2019/07/14/uninit.html (linked from RFC 2930).

Reduce scope of unsafe block for slice handling

Right now we build slices like this:

        let protocols: &[rustls_slice_bytes] = unsafe {
            if protocols.is_null() {
                return NullParameter;
            }
            slice::from_raw_parts(protocols, len)
        };

We should switch to:

        if protocols.is_null() {
            return NullParameter;
        }
        let protocols: &[rustls_slice_bytes] = unsafe {
            slice::from_raw_parts(protocols, len)
        };

Since the null check doesn't require unsafe, we move it outside the block. This reduces the scope of the unsafe block, focusing attention on what actually needs the unsafe. It also moves the flow control (return) outside of the block that returns a value.

Document default certificate behavior

If I try to just read and write TLS using certs that are descended from certs in my system's trust store, should they work without any additional configuration on my part? Or do I need to write some code in order to load the system certs?

I've read the documentation, the curl code in vtls/rustls.c and tests/*.c and I haven't found anything related to loading the system certs - there are examples to load from a file but not load the default set.

Support custom certificate validation for clients

This is particularly useful in implementing curl's --insecure / -k flag. It will also unlock passing some of the curl tests.

Note that even with a custom certificate validator, creating a client session for an IP address will fail, because webpki::DNSNameRef does not accept IP address certificates. One workaround until that happens is for client code to set a stand-in hostname like example.invalid when connecting to an IP address. Such client code should also disable SNI (functionality to disable SNI needs to be added to crustls).

Observation: call time costs

Update

The initially reported numbers result from a crustlsbuilt without compiler optimisations, as this is the current Makefile default. When building a release version with -O3, the calltimes are nearly identical between Rust and C functions.

Original Issue

I made a test of call times when invoking crustls functions from my mod_tlscode. I added 2 functions to crustls (code below) and a server global C function pointer. The server global function pointer prevents the compiler/linker to make any assumption about where it comes from. So, no inlining or LTO, as far as I know.

I called these functions in a loop 10 million times (sample code below) and got:

crustls /w panic_boundary:  434 ms
crustls /wo panic_boundary: 192 ms
C global function*:          10 ms

Unless there is some mistake on my part (possibly, that's why I document things here), the overhead seems to play a role.

It is negligible in functions invoked a few times on connection setup. However the API design of Rust components should probably avoid frequent calls back and forth in time critical sections when possible.

This issue is meant as an observation.

Sample code of calling:

  start = apr_time_now();
  for (i = 0; i < iters; ++i) {
      test_function(cc->rustls_connection, n, &n);
      if (n != 42) break;
  }
  end = apr_time_now();
  assert(i == iters);
  duration = end - start;

crustls test functions:

#[no_mangle]
pub extern "C" fn rustls_connection_try_get_int(
    conn: *mut rustls_connection,
    n: size_t,
    out_n: *mut size_t,
) {
    ffi_panic_boundary! {
        let _conn: &mut Connection = try_mut_from_ptr!(conn);
        let out_n: &mut size_t = try_mut_from_ptr!(out_n);
        *out_n = n;
    }
}

#[no_mangle]
pub extern "C" fn rustls_connection_get_int(
    conn: *mut rustls_connection,
    n: size_t,
    out_n: *mut size_t,
) {
    let _conn: &mut Connection = try_mut_from_ptr!(conn);
    let out_n: &mut size_t = try_mut_from_ptr!(out_n);
    *out_n = n;
}

C test function:

void tls_test_get_int(rustls_connection *conn, size_t n, size_t *out_n)
{
    (void)conn;
    *out_n = n;
}

Consider removing all the type annotations on let bindings

In places where these aren't needed, the code looks verbose and unidiomatic to me. How would you feel about removing these? In general I'm pretty sure we can trust the type system, and the compiler will point out when we need annotations (that said, I don't have much experience with FFI bindings, so maybe it's different here).

undefined reference to 'fmaf' on Ubuntu 18.04

The CI builds that verify curl+rustls has started to fail on Ubuntu 18.04 and I can reproduce in a manual build as well on Ubuntu 18.04.I'm not sure exactly when this problem started but it feels like within the last few weeks. (I've tried work-arounds to get it working again in curl/curl#7701 but so far it has failed)

This reproduces on master as well as v0.7.1 and v0.7.2.

It does not reproduce on my Debian Linux unstable.

The steps to reproduce:

cd $HOME
git clone https://github.com/rustls/rustls-ffi.git
curl https://sh.rustup.rs -sSf | sh -s -- -y
source $HOME/.cargo/env
cargo install cbindgen
cd $HOME/rustls-ffi
make
make DESTDIR=$HOME/crust install

# now build curl to use this rustls
cd $HOME
git clone https://github.com/curl/curl.git
cd curl
./buildconf
./configure --with-rustls=$HOME/crust

That configure script tries use the function rustls_connection_read when linking with crustls.a - but fails since it lacks the symbols fmafand fma. Here's a piece of the resulting config.log:

configure:26660: checking for rustls_connection_read in -lcrustls
configure:26682: gcc -o conftest -Werror-implicit-function-declaration -g -O0 -pedantic -Wall -W -Wpointer-arith -Wwrite-strings -Wunused -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long -Wbad-function-cast -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wold-style-definition -Wstrict-aliasing=3 -Wcast-align -Wtype-limits -Wold-style-declaration -Wmissing-parameter-type -Wempty-body -Wclobbered -Wignored-qualifiers -Wconversion -Wno-sign-conversion -Wvla -ftree-vrp -Wdouble-promotion -Wformat=2 -Warray-bounds=2 -Wshift-negative-value -Wshift-overflow=2 -Wnull-dereference -fdelete-null-pointer-checks -Wduplicated-cond -Wunused-const-variable -Wduplicated-branches -Wrestrict -Walloc-zero -Wformat-overflow=2 -Wformat-truncation=2 -Wimplicit-fallthrough=4 -Wno-system-headers    -I/home/zuul/crust/include -lm   -L/home/zuul/crust/lib conftest.c -lcrustls -lpthread -ldl -lzstd  -lbrotlidec   >&5
conftest.c:46:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 char rustls_connection_read ();
 ^~~~
/home/zuul/crust/lib/libcrustls.a(std-008055cc7d873802.std.cf1c8f7e-cgu.0.rcgu.o): In function `std::f32::<impl f32>::lerp':
/rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b//library/std/src/f32.rs:914: undefined reference to `fmaf'
/home/zuul/crust/lib/libcrustls.a(std-008055cc7d873802.std.cf1c8f7e-cgu.0.rcgu.o): In function `std::f64::<impl f64>::lerp':
/rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b//library/std/src/f64.rs:916: undefined reference to `fma'
collect2: error: ld returned 1 exit status
configure:26682: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "curl"
| #define PACKAGE_TARNAME "curl"
| #define PACKAGE_VERSION "-"
| #define PACKAGE_STRING "curl -"
| #define PACKAGE_BUGREPORT "a suitable curl mailing list: https://curl.se/mail/"
| #define PACKAGE_URL ""
| #define DEBUGBUILD 1
| #define CURLDEBUG 1
| #define PACKAGE "curl"
| #define VERSION "-"
| #define OS "x86_64-pc-linux-gnu"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define HAVE_LDAP_SSL 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_TIME_H 1
| #define HAVE_CLOCK_GETTIME_MONOTONIC 1
| #define HAVE_LIBBROTLIDEC 1
| #define HAVE_BROTLI_DECODE_H 1
| #define HAVE_BROTLI 1
| #define HAVE_LIBZSTD 1
| #define HAVE_ZSTD_H 1
| #define HAVE_ZSTD 1
| #define CURL_DISABLE_LDAP 1
| #define CURL_DISABLE_LDAPS 1
| #define ENABLE_IPV6 1
| #define HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID 1
| #define HAVE_WRITABLE_ARGV 1
| /* end confdefs.h.  */
| 
| 
| #ifdef __cplusplus
| extern "C"
| #endif
| char rustls_connection_read ();
| int main (void)
| {
| return rustls_connection_read ();
|  ;
|  return 0;
| }
configure:26691: result: no
configure:26706: error: --with-rustls was specified but could not find rustls.

client session wants read all the time

In rustls/src/client/mod.rs line 445, the client session wants to read before the handshake has started.

It would be nice if it did not do that, since my connection blocks on it. The remote side is waiting for my client hello.

Sure, i can hack around this, but I'd prefer avoiding to make assumptions about the TLS state engine. Even if this is not likely to change soon.

Factor out shared Session code

In rustls, ClientSession and ServerSession both implement the Session trait, which contains a number of shared methods. The C bindings for those methods are duplicated between client.rs and server.rs. We explored methods of de-duplicating them in #30, and concluded that:

It's worthwhile.
The ffi_panic_boundary_foo and the try_ref_from_ptr macros have to be in the outer function that calls the shared function.
Using a macro to generate code for each type of peer won't work because of token pasting issues.

We should define a new shared session.rs. Each method, e.g. rustls_client_session_read, will call a corresponding rustls_session_read(&mut impl Session, ...). For some methods, like rustls_session_wants_read, the body will be a single line. For these cases it's probably still worthwhile defining a shared version, just for consistency across methods of Session.

Implement Acceptor API

rustls 0.20.0 introduced the Acceptor API. This allows servers to read off a ClientHello before constructing a Connection, which makes selection of server certificates easier. For more details see rustls/rustls#89.

application session data

We install C callbacks into server_config which will later be invoked from server_session instances created for individual connections. These callbacks need access to connection related information for reasons of proper logging or use of correct memory pools for allocations.

The server_session should carry a void * application data add during its creation. Passing this to callback installed in the base server_config would allow an application easy and fast access to the connection context.

In the current design, this is only possible to achieve when the complete server_config is created again and again for each connection. Which seems not desirable. Alternatively, having the application maintain global maps of connection <-> server_session pairing seems overly complicated.