feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
package measurex
|
|
|
|
|
|
|
|
//
|
|
|
|
// Resolver
|
|
|
|
//
|
|
|
|
// Wrappers for Resolver to store events into a WritableDB.
|
|
|
|
//
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
|
|
|
"strings"
|
|
|
|
"time"
|
|
|
|
|
2022-01-03 13:53:23 +01:00
|
|
|
"github.com/ooni/probe-cli/v3/internal/model"
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
"github.com/ooni/probe-cli/v3/internal/netxlite"
|
2022-08-27 15:47:48 +02:00
|
|
|
"github.com/ooni/probe-cli/v3/internal/tracex"
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
)
|
|
|
|
|
|
|
|
// WrapResolver creates a new Resolver that saves events into the WritableDB.
|
2022-01-03 13:53:23 +01:00
|
|
|
func (mx *Measurer) WrapResolver(db WritableDB, r model.Resolver) model.Resolver {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return WrapResolver(mx.Begin, db, r)
|
|
|
|
}
|
|
|
|
|
|
|
|
// WrapResolver wraps a resolver.
|
2022-01-03 13:53:23 +01:00
|
|
|
func WrapResolver(begin time.Time, db WritableDB, r model.Resolver) model.Resolver {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return &resolverDB{Resolver: r, db: db, begin: begin}
|
|
|
|
}
|
|
|
|
|
|
|
|
// NewResolverSystem creates a system resolver and then wraps
|
2022-01-10 11:25:52 +01:00
|
|
|
// it using the WrapResolver function.
|
2022-01-03 13:53:23 +01:00
|
|
|
func (mx *Measurer) NewResolverSystem(db WritableDB, logger model.Logger) model.Resolver {
|
2022-06-06 14:46:44 +02:00
|
|
|
return mx.WrapResolver(db, netxlite.NewStdlibResolver(logger))
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// NewResolverUDP is a convenience factory for creating a Resolver
|
|
|
|
// using UDP that saves measurements into the DB.
|
|
|
|
//
|
|
|
|
// Arguments:
|
|
|
|
//
|
|
|
|
// - db is where to save events;
|
|
|
|
//
|
|
|
|
// - logger is the logger;
|
|
|
|
//
|
|
|
|
// - address is the resolver address (e.g., "1.1.1.1:53").
|
2022-01-03 13:53:23 +01:00
|
|
|
func (mx *Measurer) NewResolverUDP(db WritableDB, logger model.Logger, address string) model.Resolver {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return mx.WrapResolver(db, netxlite.WrapResolver(
|
2022-06-01 11:10:08 +02:00
|
|
|
logger, netxlite.NewUnwrappedSerialResolver(
|
|
|
|
mx.WrapDNSXRoundTripper(db, netxlite.NewUnwrappedDNSOverUDPTransport(
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
mx.NewDialerWithSystemResolver(db, logger),
|
|
|
|
address,
|
|
|
|
)))),
|
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
type resolverDB struct {
|
2022-01-03 13:53:23 +01:00
|
|
|
model.Resolver
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
begin time.Time
|
|
|
|
db WritableDB
|
|
|
|
}
|
|
|
|
|
|
|
|
// DNSLookupEvent contains the results of a DNS lookup.
|
|
|
|
type DNSLookupEvent struct {
|
2021-11-05 10:46:45 +01:00
|
|
|
Network string
|
|
|
|
Failure *string
|
|
|
|
Domain string
|
|
|
|
QueryType string
|
|
|
|
Address string
|
|
|
|
Finished float64
|
|
|
|
Started float64
|
|
|
|
Oddity Oddity
|
|
|
|
A []string
|
|
|
|
AAAA []string
|
|
|
|
ALPN []string
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// SupportsHTTP3 returns true if this query is for HTTPS and
|
|
|
|
// the answer contains an ALPN for "h3"
|
|
|
|
func (ev *DNSLookupEvent) SupportsHTTP3() bool {
|
|
|
|
if ev.QueryType != "HTTPS" {
|
|
|
|
return false
|
|
|
|
}
|
2021-11-05 10:46:45 +01:00
|
|
|
for _, alpn := range ev.ALPN {
|
|
|
|
if alpn == "h3" {
|
|
|
|
return true
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
|
|
|
|
// Addrs returns all the IPv4/IPv6 addresses
|
|
|
|
func (ev *DNSLookupEvent) Addrs() (out []string) {
|
2021-11-05 10:46:45 +01:00
|
|
|
out = append(out, ev.A...)
|
|
|
|
out = append(out, ev.AAAA...)
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
func (r *resolverDB) LookupHost(ctx context.Context, domain string) ([]string, error) {
|
|
|
|
started := time.Since(r.begin).Seconds()
|
|
|
|
addrs, err := r.Resolver.LookupHost(ctx, domain)
|
|
|
|
finished := time.Since(r.begin).Seconds()
|
2021-11-05 10:46:45 +01:00
|
|
|
r.saveLookupResults(domain, started, finished, err, addrs, "A")
|
|
|
|
r.saveLookupResults(domain, started, finished, err, addrs, "AAAA")
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return addrs, err
|
|
|
|
}
|
|
|
|
|
2021-11-05 10:46:45 +01:00
|
|
|
func (r *resolverDB) saveLookupResults(domain string, started, finished float64,
|
|
|
|
err error, addrs []string, qtype string) {
|
|
|
|
ev := &DNSLookupEvent{
|
2022-08-27 15:47:48 +02:00
|
|
|
Network: tracex.ResolverNetworkAdaptNames(r.Resolver.Network()),
|
2021-11-05 10:46:45 +01:00
|
|
|
Address: r.Resolver.Address(),
|
|
|
|
Failure: NewFailure(err),
|
|
|
|
Domain: domain,
|
|
|
|
QueryType: qtype,
|
|
|
|
Finished: finished,
|
|
|
|
Started: started,
|
|
|
|
}
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
for _, addr := range addrs {
|
|
|
|
if qtype == "A" && !strings.Contains(addr, ":") {
|
2021-11-05 10:46:45 +01:00
|
|
|
ev.A = append(ev.A, addr)
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
if qtype == "AAAA" && strings.Contains(addr, ":") {
|
2021-11-05 10:46:45 +01:00
|
|
|
ev.AAAA = append(ev.AAAA, addr)
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
continue
|
|
|
|
}
|
|
|
|
}
|
2021-11-05 10:46:45 +01:00
|
|
|
switch qtype {
|
|
|
|
case "A":
|
|
|
|
ev.Oddity = r.computeOddityLookupHost(ev.A, err)
|
|
|
|
case "AAAA":
|
|
|
|
ev.Oddity = r.computeOddityLookupHost(ev.AAAA, err)
|
|
|
|
}
|
|
|
|
r.db.InsertIntoLookupHost(ev)
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func (r *resolverDB) computeOddityLookupHost(addrs []string, err error) Oddity {
|
|
|
|
if err != nil {
|
|
|
|
switch err.Error() {
|
|
|
|
case netxlite.FailureGenericTimeoutError:
|
|
|
|
return OddityDNSLookupTimeout
|
|
|
|
case netxlite.FailureDNSNXDOMAINError:
|
|
|
|
return OddityDNSLookupNXDOMAIN
|
|
|
|
case netxlite.FailureDNSRefusedError:
|
|
|
|
return OddityDNSLookupRefused
|
|
|
|
default:
|
|
|
|
return OddityDNSLookupOther
|
|
|
|
}
|
|
|
|
}
|
|
|
|
for _, addr := range addrs {
|
2021-11-02 12:20:04 +01:00
|
|
|
if netxlite.IsBogon(addr) {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
return OddityDNSLookupBogon
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return ""
|
|
|
|
}
|
|
|
|
|
2022-01-03 13:53:23 +01:00
|
|
|
func (r *resolverDB) LookupHTTPS(ctx context.Context, domain string) (*model.HTTPSSvc, error) {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
started := time.Since(r.begin).Seconds()
|
|
|
|
https, err := r.Resolver.LookupHTTPS(ctx, domain)
|
|
|
|
finished := time.Since(r.begin).Seconds()
|
|
|
|
ev := &DNSLookupEvent{
|
2022-08-27 15:47:48 +02:00
|
|
|
Network: tracex.ResolverNetworkAdaptNames(r.Resolver.Network()),
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
Address: r.Resolver.Address(),
|
|
|
|
Domain: domain,
|
|
|
|
QueryType: "HTTPS",
|
|
|
|
Started: started,
|
|
|
|
Finished: finished,
|
2021-11-05 10:46:45 +01:00
|
|
|
Failure: NewFailure(err),
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
Oddity: Oddity(r.computeOddityHTTPSSvc(https, err)),
|
|
|
|
}
|
|
|
|
if err == nil {
|
2021-11-05 10:46:45 +01:00
|
|
|
ev.A = append(ev.A, https.IPv4...)
|
|
|
|
ev.AAAA = append(ev.AAAA, https.IPv6...)
|
|
|
|
ev.ALPN = append(ev.ALPN, https.ALPN...)
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
}
|
|
|
|
r.db.InsertIntoLookupHTTPSSvc(ev)
|
|
|
|
return https, err
|
|
|
|
}
|
|
|
|
|
2022-01-03 13:53:23 +01:00
|
|
|
func (r *resolverDB) computeOddityHTTPSSvc(https *model.HTTPSSvc, err error) Oddity {
|
feat(measurex): refactored measurement library (#528)
This commit introduce a measurement library that consists of
refactored code from earlier websteps experiments.
I am not going to add tests for the time being, because this library
is still a bit in flux, as we finalize websteps.
I will soon though commit documentation explaining in detail how
to use it, which currrently is at https://github.com/ooni/probe-cli/pull/506
and adds a new directory to internal/tutorial.
The core idea of this measurement library is to allow two
measurement modes:
1. tracing, which is what we're currently doing now, and the
tutorial shows how we can rewrite the measurement part of web
connectivity with measurex using less code. Under a tracing
approach, we construct a normal http.Client that however has
tracing configured, we gather events for resolve, connect, TLS
handshake, QUIC handshake, HTTP round trip, etc. and then we
try to make sense of what happened from the events stream;
2. step-by-step, which is what websteps does, and basically
means that after each operation you immediately write into
a Measurement structure its results and immediately draw the
conclusions on what seems odd (which later may become an
anomaly if we see what the test helper measured).
This library is also such that it produces a data format
compatible with the current OONI spec.
This work is part of https://github.com/ooni/probe/issues/1733.
2021-09-30 01:24:08 +02:00
|
|
|
if err != nil {
|
|
|
|
return r.computeOddityLookupHost(nil, err)
|
|
|
|
}
|
|
|
|
var addrs []string
|
|
|
|
addrs = append(addrs, https.IPv4...)
|
|
|
|
addrs = append(addrs, https.IPv6...)
|
|
|
|
return r.computeOddityLookupHost(addrs, nil)
|
|
|
|
}
|