ooniprobe - Online in the Cloud

PROGRAM:

NAME

ooniprobe - a network censorship measurement tool.

SYNOPSIS

ooniprobe [-hnsp] [--version] [--spew] [-o reportfile] [-i testdeck] [-c collector] [-b
bouncer] [-l logfile] [-O pcapfile] [-f configfile] [-d datadir] test_name

DESCRIPTION

ooniprobe is tool for performing internet censorship measurements. Our goal is to achieve
a common data format and set of methodologies for conducting censorship related research.

OPTIONS

-h or --help
Display this help and exit.

-n or --no-collector
Disable reporting the net test results to an oonib collector.

-s or --list
List all of the available net tests.

-p or --printdeck
Print to standard output the specified command line options as an ooniprobe test
deck.

-o or --reportfile
Specify the path to report file to write.

-i or --testdeck
Specify as input a test deck: a yaml file containing the tests to run and their
arguments.

-c or --collector
Specify the address of the collector of net test results. It is advisable to always
specify and bouncer and let it return a collector for the test or test deck you are
running.

-b or --bouncer
Address of the bouncer that will inform the probe of which collector to use and the
addresses of test helpers. default: httpo://nkvphnp3p6agi5qq.onion

-l or --logfile
Path to the log file to write

-O or --pcapfile
Prefix to the pcap file name.

-f or --configfile
Specify a path to the ooniprobe configuration file.

-d or --datadir
Specify a path to the ooniprobe data directory

--spew Print an insanely verbose log of everything that happens. Useful when debugging
freezes or locks in complex code.

--version
Display the ooniprobe version and exit.

OONIPROBE

Is the tool that volunteers and researches interested in contributing data to the project
should be running.

ooniprobe allows the user to select what test should be run and what backend should be
used for storing the test report and/or assisting them in the running of the test.

ooniprobe tests are divided into two categories: Traffic Manipulation and Content
Blocking.

Traffic Manipulation tests aim to detect the presence of some sort of tampering with the
internet traffic between the probe and a remote test helper backend. As such they usually
require the selection of a oonib backend component for running the test.

Content Blocking are aimed at enumerating the kind of content that is blocked from the
probes network point of view. As such they usually require to have specified an input list
for running the test.

Threat Model
Our adversary is capable of doing country wide network surveillance and manipulation of
network traffic.

The goals of our adversary are:

· Restrict access to certain content, while not degrading overall quality of the
network

· Monitor the network in a way that they are able to identify misuse of it in real time

More specifc to the running of network filtering detection tests:

1. Detect actors performing censorship detection tests

2. Fool people running such tests into believing that the network is unrestricted

Note that while 2) => 1) it is not true that 1) => 2) as the identification of such actors
does not necessarily have to happen in real time. While our intention is to minimize the
risk of users running OONI probe to be identified, this comes with a tradeoff in accuracy.
It is therefore necessary in certain tests to trade-off fingerprintability in favour of
tests accuracy.

This is why we divide tests based on what risk the user running it can face, allowing the
user to freely choose what threat model they wish to adere to.

Installation
Read this before running ooniprobe!

Running ooniprobe is a potentially risky activity. This greatly depends on the
jurisdiction in which you are in and which test you are running. It is technically
possible for a person observing your internet connection to be aware of the fact that you
are running ooniprobe. This means that if running network measurement tests is something
considered to be illegal in your country then you could be spotted.

Furthermore, ooniprobe takes no precautions to protect the install target machine from
forensics analysis. If the fact that you have installed or used ooni probe is a liability
for you, please be aware of this risk.

Debian based systems
sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list'

gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4

gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add -

sudo apt-get update && sudo apt-get install ooniprobe

Linux
We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as well as
versions of Ubuntu such as natty and later releases. Running ooniprobe without installing
it is supported with the following commands:

git clone https://git.torproject.org/ooni-probe.git

cd ooni-probe

./setup-dependencies.sh

python setup.py install

Setting up development environment
On debian based systems this can be done with:

Vsudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper

mkvirtualenv ooniprobe

python setup.py install

pip install -r requirements-dev.txt

Other platforms (with Vagrant)
Install Vagrant and Install Virtualbox

On OSX:

If you don't have it install homebrew

brew install git

On debian/ubuntu:

sudo apt-get install git

1. Open a Terminal and run:

git clone https://git.torproject.org/ooni-probe.git

cd ooni-probe/

vagrant up

2. Login to the box with:

vagrant ssh

ooniprobe will be installed in /ooni.

3. You can run tests with:

ooniprobe blocking/http_requests -f /ooni/inputs/input-pack/alexa-top-1k.txt

Using ooniprobe
Net test is a set of measurements to assess what kind of internet censorship is occurring.

Decks are collections of ooniprobe nettests with some associated inputs.

Collector is a service used to report the results of measurements.

Test helper is a service used by a probe for successfully performing its measurements.

Bouncer is a service used to discover the addresses of test helpers and collectors.

Configuring ooniprobe
You may edit the configuration for ooniprobe by editing the configuration file found
inside of ~/.ooni/ooniprobe.conf.

By default ooniprobe will not include personal identifying information in the test result,
nor create a pcap file. This behavior can be personalized.

Running decks
You will find all the installed decks inside of /usr/share/ooni/decks.

You may then run a deck by using the command line option -i:

As root:

ooniprobe -i /usr/share/ooni/decks/mlab.deck

Or as a user:

ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck

Or:

As root:

ooniprobe -i /usr/share/ooni/decks/complete.deck

Or as a user:

ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck

The above tests will require around 20-30 minutes to complete depending on your network
speed.

If you would prefer to run some faster tests you should run: As root:

ooniprobe -i /usr/share/ooni/decks/fast.deck

Or as a user:

ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck

Running net tests
You may list all the installed stable net tests with:

ooniprobe -s

You may then run a nettest by specifying its name for example:

ooniprobe manipulation/http_header_field_manipulation

It is also possible to specify inputs to tests as URLs:

ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/
37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1

You can find the result of the test in your current working directory.

By default the report result will be collected by the default ooni collector and the
addresses of test helpers will be obtained from the default bouncer.

You may also specify your own collector or bouncer with the options -c and -b.

(Optional) Install obfsproxy
Install the latest version of obfsproxy for your platform.

Download Obfsproxy

Bridges and obfsproxy bridges
ooniprobe submits reports to oonib report collectors through Tor to a hidden service
endpoint. By default, ooniprobe uses the installed system Tor, but can also be configured
to launch Tor (see the advanced.start_tor option in ooniprobe.conf), and ooniprobe
supports bridges (and obfsproxy bridges, if obfsproxy is installed). The tor.bridges
option in ooniprobe.conf sets the path to a file that should contain a set of "bridge"
lines (of the same format as used in torrc, and as returned by
https://bridges.torproject.org). If obfsproxy bridges are to be used, the path to the
obfsproxy binary must be configured. See option advanced.obfsproxy_binary, in
ooniprobe.conf.

Setting capabilities on your virtualenv python binary
If your distributation supports capabilities you can avoid needing to run OONI as root:

setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python

Core ooniprobe Tests
The source for Content blocking tests and Traffic Manipulation tests can be found in the
nettests/blocking and nettests/manipulation directories respectively.

Content Blocking Tests
· DNSConsistency

· HTTP Requests

· TCP Connect

Traffic Manipulation Tests
· HTTP Invalid Request Line:

· DNS Spoof

· HTTP Header Field Manipulation

· Traceroute

· HTTP Host

Other tests
We also have some other tests that are currently not fully supported or still being
experimented with.

You can find these in:

· ooni/nettests/experimental

Tests that don't do a measurement but are useful for scanning can be found in:

· ooni/nettests/scanning

Tests that involve running third party tools may be found in:

· ooni/nettests/third_party

Reports
The reports collected by ooniprobe are stored on https://ooni.torproject.org/reports/0.1/
CC /

Where CC is the two letter country code as specified by ISO 31666-2.

For example the reports for Italy (CC is it) of the may be found in:

https://ooni.torproject.org/reports/0.1/IT/

This directory shall contain the various reports for the test using the following
convention:

testName - dateInISO8601Format - probeASNumber .yamloo

The date is expressed using ISO 8601 including seconds and with no : to delimit hours,
minutes, days.

Like so:

YEAR - MONTH - DAY T HOURS MINUTES SECONDS Z

Look here for the up to date list of ISO 8601 country codes

The time is always expressed in UTC.

If a collision is detected then an int (starting with 1) will get appended to the test.

For example if two report that are created on the first of January 2012 at Noon (UTC time)
sharp from MIT (AS3) will be stored here:

https://ooni.torproject.org/reports/0.1/US/2012-01-01T120000Z_AS3.yamloo
https://ooni.torproject.org/reports/0.1/US/2012-01-01T120000Z_AS3.1.yamloo

Note: it is highly unlikely that reports get created with the same exact timestamp from
the same exact ASN. If this does happen it could be index of some malicious report
poisoning attack in progress.

Report format version changelog
In here shall go details about the major changes to the reporting format.

version 0.1
Initial format version.

Writing OONI tests
The OONI testing API is heavily influenced and partially based on the python unittest
module and twisted.trial.

Test Cases
The atom of OONI Testing is called a Test Case. A test case class may contain multiple
Test Methods.

class ooni.nettest.NetTestCase
This is the base of the OONI nettest universe. When you write a nettest you will
subclass this object.

· inputs: can be set to a static set of inputs. All the tests (the methods starting
with the "test" prefix) will be run once per input. At every run the _input_
attribute of the TestCase instance will be set to the value of the current
iteration over inputs. Any python iterable object can be set to inputs.

· inputFile: attribute should be set to an array containing the command line
argument that should be used as the input file. Such array looks like this:
["commandlinearg", "c", "default value" "The description"]

The second value of such arrray is the shorthand for the command line arg. The
user will then be able to specify inputs to the test via:
ooniprobe mytest.py --commandlinearg path/to/file.txt

or
ooniprobe mytest.py -c path/to/file.txt

· inputProcessor: should be set to a function that takes as argument a filename and
it will return the input to be passed to the test instance.

· name: should be set to the name of the test.

· author: should contain the name and contact details for the test author. The
format for such string is as follows:
The Name <[email protected]>

· version: is the version string of the test.

· requiresRoot: set to True if the test must be run as root.

· usageOptions: a subclass of twisted.python.usage.Options for processing of
command line arguments

· localOptions: contains the parsed command line arguments.

Quirks: Every class that is prefixed with test must return a
twisted.internet.defer.Deferred.

If the test you plan to write is not listed on the Tor OONI trac page, you should add it
to the list and then add a description about it following the Test Template

Tests are driven by inputs. For every input a new test instance is created, internally the
_setUp method is called that is defined inside of test templates, then the setUp method
that is overwritable by users.

Gotchas: never call reactor.start of reactor.stop inside of your test method and all will
be good.

Inputs
Inputs are what is given as input to every iteration of the Test Case. Iflyou have 100
inputs, then every test case will be run 100 times.

To configure a static set of inputs you should define the ooni.nettest.NetTestCase
attribute inputs. The test will be run len(inputs) times. Any iterable object is a valid
inputs attribute.

If you would like to have inputs be determined from a user specified input file, then you
must set the inputFile attribute. This is an array that specifies what command line option
may be used to control this value.

By default the inputProcessor is set to read the file line by line and strip newline
characters. To change this behavior you must set the inputProcessor attribute to a
function that takes as argument a file descriptor and yield the next item. The default
inputProcessor looks like this:

def lineByLine(filename):
fp = open(filename)
for x in fp.xreadlines():
yield x.strip()
fp.close()

Setup and command line passing
Tests may define the setUp method that will be called every time the Test Case object is
instantiated, in here you may place some common logic to all your Test Methods that should
be run before any testing occurs.

Command line arguments can be parsed thanks to the twisted
twisted.python.usage.UsageOptions class.

You will have to subclass this and define the NetTestCase attribute usageOptions to point
to a subclass of this.

class UsageOptions(usage.Options):
optParameters = [['backend', 'b', 'http://127.0.0.1:57001',
'URL of the test backend to use']
]

class MyTestCase(nettest.NetTestCase):
usageOptions = UsageOptions

inputFile = ['file', 'f', None, "Some foo file"]
requiredOptions = ['backend']

def test_my_test(self):
self.localOptions['backend']

You will then be able to access the parsed command line arguments via the class attribute
localOptions.

The requiredOptions attributes specifies an array of parameters that are required for the
test to run properly.

inputFile is a special class attribute that will be used for processing of the inputFile.
The filename that is read here will be given to the
ooni.nettest.NetTestCase.inputProcessor method that will yield, by default, one line of
the file at a time.

Test Methods
These shall be defined inside of your ooni.nettest.NetTestCase subclass. These will be
class methods.

All class methods that are prefixed with test_ shall be run. Functions that are relevant
to your test should be all lowercase separated by underscore.

To add data to the test report you may write directly to the report object like so:

def test_my_function():
result = do_something()
self.report['something'] = result

OONI will then handle the writing of the data to the final test report.

To access the current input you can use the input attribute, for example:

def test_with_input():
do_something_with_input(self.input)

This will at each iteration over the list of inputs do something with the input.

Test Templates
Test templates assist you in writing tests. They already contain all the common
functionality that is useful to running a test of that type. They also take care of
writing the data they collect that is relevant to the test run to the report file.

Currently implemented test templates are ooni.templates.scapyt for tests based on Scapy,
ooni.templates.tcpt for tests based on TCP, ooni.templates.httpt for tests based on HTTP,
and ooni.templates.dnst for tests based on DNS.

Scapy based tests
Scapy based tests will be a subclass of ooni.templates.scapyt.BaseScapyTest.

It provides a wrapper around the scapy send and receive function that will write the sent
and received packets to the report with sanitization of the src and destination IP
addresses.

It has the same syntax as the Scapy sr function, except that it will return a deferred.

To implement a simple ICMP ping based on this function you can do like so (Taken from
nettest.examples.example_scapyt.ExampleICMPPingScapy)

from twisted.python import usage

from scapy.all import IP, ICMP

from ooni.templates import scapyt

class UsageOptions(usage.Options):
optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]

class ExampleICMPPingScapy(scapyt.BaseScapyTest):
name = "Example ICMP Ping Test"

usageOptions = UsageOptions

def test_icmp_ping(self):
def finished(packets):
print packets
answered, unanswered = packets
for snd, rcv in answered:
rcv.show()

packets = IP(dst=self.localOptions['target'])/ICMP()
d = self.sr(packets)
d.addCallback(finished)
return d

The arguments taken by self.sr() are exactly the same as the scapy send and receive
function, the only difference is that instead of using the regular scapy super socket it
uses our twisted driven wrapper around it.

Alternatively this test can also be written using the twisted.defer.inlineCallbacks()
decorator, that makes it look more similar to regular sequential code.

from twisted.python import usage
from twisted.internet import defer

from scapy.all import IP, ICMP

from ooni.templates import scapyt

class UsageOptions(usage.Options):
optParameters = [['target', 't', '127.0.0.1', "Specify the target to ping"]]

class ExampleICMPPingScapyYield(scapyt.BaseScapyTest):
name = "Example ICMP Ping Test"

usageOptions = UsageOptions

@defer.inlineCallbacks
def test_icmp_ping(self):
packets = IP(dst=self.localOptions['target'])/ICMP()
answered, unanswered = yield self.sr(packets)
for snd, rcv in answered:
rcv.show()

Report Format
###########################################
# OONI Probe Report for Example ICMP Ping Test test
# Thu Nov 22 18:20:43 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
start_time: 1353601243.0, test_name: Example ICMP Ping Test, test_version: 0.1}
...
---
input: null
report:
answer_flags: [ipsrc]
answered_packets:
- - raw_packet: !!binary |
RQAAHAEdAAAuAbjKCAgICH8AAAEAAAAAAAAAAA==
summary: IP / ICMP 8.8.8.8 > 127.0.0.1 echo-reply 0
sent_packets:
- - raw_packet: !!binary |
RQAAHAABAABAAevPfwAAAQgICAgIAPf/AAAAAA==
summary: IP / ICMP 127.0.0.1 > 8.8.8.8 echo-request 0
test_name: test_icmp_ping
test_started: 1353604843.553605
...

TCP based tests
TCP based tests will subclass ooni.templates.tcpt.TCPTest.

This test template facilitates the sending of TCP payloads to the wire and recording the
response.

from twisted.internet.error import ConnectionRefusedError
from ooni.utils import log
from ooni.templates import tcpt

class ExampleTCPT(tcpt.TCPTest):
def test_hello_world(self):
def got_response(response):
print "Got this data %s" % response

def connection_failed(failure):
failure.trap(ConnectionRefusedError)
print "Connection Refused"

self.address = "127.0.0.1"
self.port = 57002
payload = "Hello World!\n\r"
d = self.sendPayload(payload)
d.addErrback(connection_failed)
d.addCallback(got_response)
return d

The possible failures for a TCP connection are:

twisted.internet.error.NoRouteError that corresponds to errno.ENETUNREACH

twisted.internet.error.ConnectionRefusedError that corresponds to errno.ECONNREFUSED

twisted.internet.error.TCPTimedOutError that corresponds to errno.ETIMEDOUT

Report format
The basic report of a TCP test looks like the following (this is an report generated by
running the above example against a TCP echo server).

###########################################
# OONI Probe Report for Base TCP Test test
# Thu Nov 22 18:18:28 2012
###########################################
---
{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1-alpha,
start_time: 1353601108.0, test_name: Base TCP Test, test_version: '0.1'}
...
---
input: null
report:
errors: []
received: ["Hello World!\n\r"]
sent: ["Hello World!\n\r"]
test_name: test_hello_world
test_started: 1353604708.705081
...

TODO finish this with more details

HTTP based tests
HTTP based tests will be a subclass of ooni.templates.httpt.HTTPTest.

It provides methods ooni.templates.httpt.HTTPTest.processResponseBody() and
ooni.templates.httpt.HTTPTest.processResponseHeaders() for interacting with the response
body and headers respectively.

For example, to implement a HTTP test that returns the sha256 hash of the response body
(based on nettests.examples.example_httpt):

from ooni.utils import log
from ooni.templates import httpt
from hashlib import sha256

class SHA256HTTPBodyTest(httpt.HTTPTest):
name = "ChecksumHTTPBodyTest"
author = "Aaron Gibson"
version = 0.1

inputFile = ['url file', 'f', None,
'List of URLS to perform GET requests to']
requiredOptions = ['url file']

def test_http(self):
if self.input:
url = self.input
return self.doRequest(url)
else:
raise Exception("No input specified")

def processResponseBody(self, body):
body_sha256sum = sha256(body).hexdigest()
self.report['checksum'] = body_sha256sum

Report format
###########################################
# OONI Probe Report for ChecksumHTTPBodyTest test
# Thu Dec 6 17:31:57 2012
###########################################
---
options:
collector: null
help: 0
logfile: null
pcapfile: null
reportfile: null
resume: 0
subargs: [-f, hosts]
test: nettests/examples/example_http_checksum.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786317.0
test_name: ChecksumHTTPBodyTest
test_version: 0.1
...
---
input: http://www.google.com
report:
agent: agent
checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1
requests:
- request:
body: null
headers:
- - User-Agent
- - [Opera/9.00 (Windows NT 5.1; U; en), 'Opera 9.0, Windows XP']
method: GET
url: http://www.google.com
response:
body: '<!doctype html><html ... snip ... </html>'
code: 200
headers:
- - X-XSS-Protection
- [1; mode=block]
- - Set-Cookie
- ['PREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT-2GCkNAocyXlVa;
expires=Sat, 06-Dec-2014 13:31:57 GMT; path=/; domain=.google.com', 'NID=66=KWaLbNQumuGuYf0HrWlGm54u9l-DKJwhFCMQXfhQPZM-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN;
expires=Fri, 07-Jun-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly']
- - Expires
- ['-1']
- - Server
- [gws]
- - Connection
- [close]
- - Cache-Control
- ['private, max-age=0']
- - Date
- ['Thu, 06 Dec 2012 13:31:57 GMT']
- - P3P
- ['CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657
for more info."']
- - Content-Type
- [text/html; charset=UTF-8]
- - X-Frame-Options
- [SAMEORIGIN]
socksproxy: null
test_name: test_http
test_runtime: 0.08298492431640625
test_started: 1354800717.478403
...

DNS based tests
DNS based tests will be a subclass of ooni.templates.dnst.DNSTest.

It provides methods ooni.templates.dnst.DNSTest.performPTRLookup() and
ooni.templates.dnst.DNSTest.performALookup()

For example (taken from nettests.examples.example_dnst):

from ooni.templates.dnst import DNSTest

class ExampleDNSTest(DNSTest):
def test_a_lookup(self):
def gotResult(result):
# Result is an array containing all the A record lookup results
print result

d = self.performALookup('torproject.org', ('8.8.8.8', 53))
d.addCallback(gotResult)
return d

Report format
###########################################
# OONI Probe Report for Base DNS Test test
# Thu Dec 6 17:42:51 2012
###########################################
---
options:
collector: null
help: 0
logfile: null
pcapfile: null
reportfile: null
resume: 0
subargs: []
test: nettests/examples/example_dnst.py
probe_asn: null
probe_cc: null
probe_ip: 127.0.0.1
software_name: ooniprobe
software_version: 0.0.7.1-alpha
start_time: 1354786971.0
test_name: Base DNS Test
test_version: 0.1
...
---
input: null
report:
queries:
- addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16]
answers:
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14
ttl=782>]
- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16
ttl=782>]
query: '[Query(''torproject.org'', 1, 1)]'
query_type: A
resolver: [8.8.8.8, 53]
test_name: test_a_lookup
test_runtime: 0.028924942016601562
test_started: 1354801371.980114
...

For a more complex example, see: nettests.blocking.dnsconsistency

GLOSSARY

Here we will summarize some of the jargon that is unique to OONI.

Test Case: a set of measurements performed on a to be tested network that are logically
grouped together

Report: is the output of a test run containing all the information that is require for a
researcher to assess what is the output of a test.

Yamlooni: The format we use for Reports, that is based on YAML.

Input: What is given as input to a TestCase to perform a measurement.

Use ooniprobe online using onworks.net services