feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
package measurex
|
|
|
|
|
|
|
|
//
|
|
|
|
// TLS
|
|
|
|
//
|
|
|
|
// Wraps TLS code to write events into a WritableDB.
|
|
|
|
//
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
|
|
|
"crypto/tls"
|
|
|
|
"crypto/x509"
|
|
|
|
"errors"
|
|
|
|
"net"
|
|
|
|
"time"
|
|
|
|
|
2022-01-03 13:53:23 +01:00
|
|
|
"github.com/ooni/probe-cli/v3/internal/model"
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
"github.com/ooni/probe-cli/v3/internal/netxlite"
|
|
|
|
)
|
|
|
|
|
|
|
|
// WrapTLSHandshaker wraps a netxlite.TLSHandshaker to return a new
|
|
|
|
// instance of TLSHandshaker that saves events into the DB.
|
2022-01-03 13:53:23 +01:00
|
|
|
func (mx *Measurer) WrapTLSHandshaker(db WritableDB, thx model.TLSHandshaker) model.TLSHandshaker {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return &tlsHandshakerDB{TLSHandshaker: thx, db: db, begin: mx.Begin}
|
|
|
|
}
|
|
|
|
|
|
|
|
// NewTLSHandshakerStdlib creates a new TLS handshaker that
|
|
|
|
// saves results into the DB and uses the stdlib for TLS.
|
2022-01-03 13:53:23 +01:00
|
|
|
func (mx *Measurer) NewTLSHandshakerStdlib(db WritableDB, logger model.Logger) model.TLSHandshaker {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return mx.WrapTLSHandshaker(db, netxlite.NewTLSHandshakerStdlib(logger))
|
|
|
|
}
|
|
|
|
|
|
|
|
type tlsHandshakerDB struct {
|
2022-01-03 13:53:23 +01:00
|
|
|
model.TLSHandshaker
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
begin time.Time
|
|
|
|
db WritableDB
|
|
|
|
}
|
|
|
|
|
2021-11-05 10:46:45 +01:00
|
|
|
// QUICTLSHandshakeEvent contains a QUIC or TLS handshake event.
|
|
|
|
type QUICTLSHandshakeEvent struct {
|
|
|
|
CipherSuite string
|
|
|
|
Failure *string
|
|
|
|
NegotiatedProto string
|
|
|
|
TLSVersion string
|
|
|
|
PeerCerts [][]byte
|
|
|
|
Finished float64
|
|
|
|
RemoteAddr string
|
|
|
|
SNI string
|
|
|
|
ALPN []string
|
|
|
|
SkipVerify bool
|
|
|
|
Oddity Oddity
|
|
|
|
Network string
|
|
|
|
Started float64
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (thx *tlsHandshakerDB) Handshake(ctx context.Context,
|
|
|
|
conn Conn, config *tls.Config) (net.Conn, tls.ConnectionState, error) {
|
|
|
|
network := conn.RemoteAddr().Network()
|
|
|
|
remoteAddr := conn.RemoteAddr().String()
|
|
|
|
started := time.Since(thx.begin).Seconds()
|
|
|
|
tconn, state, err := thx.TLSHandshaker.Handshake(ctx, conn, config)
|
|
|
|
finished := time.Since(thx.begin).Seconds()
|
2021-11-05 10:46:45 +01:00
|
|
|
thx.db.InsertIntoTLSHandshake(&QUICTLSHandshakeEvent{
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
Network: network,
|
|
|
|
RemoteAddr: remoteAddr,
|
|
|
|
SNI: config.ServerName,
|
|
|
|
ALPN: config.NextProtos,
|
|
|
|
SkipVerify: config.InsecureSkipVerify,
|
|
|
|
Started: started,
|
|
|
|
Finished: finished,
|
2021-11-05 10:46:45 +01:00
|
|
|
Failure: NewFailure(err),
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
Oddity: thx.computeOddity(err),
|
|
|
|
TLSVersion: netxlite.TLSVersionString(state.Version),
|
|
|
|
CipherSuite: netxlite.TLSCipherSuiteString(state.CipherSuite),
|
|
|
|
NegotiatedProto: state.NegotiatedProtocol,
|
2021-11-05 10:46:45 +01:00
|
|
|
PeerCerts: peerCerts(err, &state),
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
})
|
|
|
|
return tconn, state, err
|
|
|
|
}
|
|
|
|
|
|
|
|
func (thx *tlsHandshakerDB) computeOddity(err error) Oddity {
|
|
|
|
if err == nil {
|
|
|
|
return ""
|
|
|
|
}
|
|
|
|
switch err.Error() {
|
|
|
|
case netxlite.FailureGenericTimeoutError:
|
|
|
|
return OddityTLSHandshakeTimeout
|
|
|
|
case netxlite.FailureConnectionReset:
|
|
|
|
return OddityTLSHandshakeReset
|
|
|
|
case netxlite.FailureEOFError:
|
|
|
|
return OddityTLSHandshakeUnexpectedEOF
|
|
|
|
case netxlite.FailureSSLInvalidHostname:
|
|
|
|
return OddityTLSHandshakeInvalidHostname
|
|
|
|
case netxlite.FailureSSLUnknownAuthority:
|
|
|
|
return OddityTLSHandshakeUnknownAuthority
|
|
|
|
default:
|
|
|
|
return OddityTLSHandshakeOther
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func peerCerts(err error, state *tls.ConnectionState) (out [][]byte) {
|
|
|
|
var x509HostnameError x509.HostnameError
|
|
|
|
if errors.As(err, &x509HostnameError) {
|
|
|
|
// Test case: https://wrong.host.badssl.com/
|
|
|
|
return [][]byte{x509HostnameError.Certificate.Raw}
|
|
|
|
}
|
|
|
|
var x509UnknownAuthorityError x509.UnknownAuthorityError
|
|
|
|
if errors.As(err, &x509UnknownAuthorityError) {
|
|
|
|
// Test case: https://self-signed.badssl.com/. This error has
|
|
|
|
// never been among the ones returned by MK.
|
|
|
|
return [][]byte{x509UnknownAuthorityError.Cert.Raw}
|
|
|
|
}
|
|
|
|
var x509CertificateInvalidError x509.CertificateInvalidError
|
|
|
|
if errors.As(err, &x509CertificateInvalidError) {
|
|
|
|
// Test case: https://expired.badssl.com/
|
|
|
|
return [][]byte{x509CertificateInvalidError.Cert.Raw}
|
|
|
|
}
|
|
|
|
for _, cert := range state.PeerCertificates {
|
|
|
|
out = append(out, cert.Raw)
|
|
|
|
}
|
|
|
|
return
|
|
|
|
}
|