Similar to Lab 2, you can use telnet to open a connection to your proxy and send it HTTP requests. This is a simple and powerful tool for testing your proxy. For example, it allows you to construct and see how your proxy handles arbitrary requests, multiple requests over a persistent connection, or an idle client. You can also initiate multiple telnet sessions with your proxy to see how it handles concurrent connections.
The basic usage is:
$ telnet host port
Here is a simple example:
$ telnet 127.0.0.1 60893
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET http://www.example.com/ HTTP/1.1
Host: www.example.com
HTTP/1.1 200 OK
Content-Type: text/html
ETag: "84238dfc8092e5d9c0dac8ef93371a07:1736799080.121134"
Last-Modified: Wed, 14 May 2025 20:11:20 GMT
Cache-Control: max-age=2540
Date: Thu, 15 May 2025 00:32:32 GMT
Content-Length: 1256
Connection: keep-alive
X-N: S
Via: 1.1 z1234567
<!doctype html>
<html>
<head>
<title>Example Domain</title>
... (additional output concealed for brevity)
curl is a command-line tool for transferring data to or from a server. It supports various protocols like HTTP, HTTPS, FTP, and more, allowing users to fetch or send data, automate requests, test APIs, and download files.
The basic usage is:
$ curl [options / URLs]
It supports many options which you might like to explore by running man 1 curl. A brief description of some of the perhaps more relevant options are included in the FAQ. Verbose mode (-v) is particularly useful for monitoring its use of connections with the proxy. So when given multiple URLs it can demonstrate the proxy's ability to handle persistence, and additionally with the parallel option (-Z), the proxy's ability to handle concurrency.
Here is a simple example:
$ curl -x '127.0.0.1:60893' 'http://httpstat.us/418'
418 I'm a teapot
The most common HTTP user agent is a Web browser. Most browsers rely on the system settings to determine whether it should connect via a proxy, so if you change your system settings and your proxy is dysfunctional then you won't be able to browse the Web. Firefox, however, can be configured separately (see the FAQ for instructions). Thus, it's a good choice of browser for testing your proxy. You'll still be able to access websites using a second browser, which also allows you to easily compare the expected response for a particular URL. Firefox also includes Developer Tools, where you can inspect the HTTP requests/responses, and disable the browser cache to ensure all requests are actually sent to the proxy.
While it's important your proxy does work with a browser, it's recommended you defer using one for testing until your proxy is behaving as you'd expect with simpler tools like telnet and curl.
There are many HTTP libraries available to use the protocol programmatically. While you aren't permitted to use any within your proxy, they can be a great tool for testing your proxy. A popular choice is the Python requests library, as it allows you to send HTTP/1.1 requests extremely easily. Review the documentation for its full power, but here are some simple examples:
Without connection reuse (i.e. non-persistent):
import requests
MY_PROXY_PORT = 60893 # CHANGE ME!
proxies = {
"http": f"http://127.0.0.1:{MY_PROXY_PORT}",
"https": f"http://127.0.0.1:{MY_PROXY_PORT}",
}
# GET request over HTTP
response = requests.get("http://www.example.org", proxies=proxies, timeout=10)
assert response.status_code == 200
assert "Example Domain" in response.text
# GET request over HTTPS
response = requests.get("https://www.example.org", proxies=proxies, timeout=10)
assert response.status_code == 200
assert "Example Domain" in response.text
print("All tests passed!")
With connection reuse (i.e. persistent):
import requests
MY_PROXY_PORT = 60893 # CHANGE ME!
proxies = {
"http": f"http://127.0.0.1:{MY_PROXY_PORT}",
"https": f"http://127.0.0.1:{MY_PROXY_PORT}",
}
# Creating a session object allows the underlying TCP connection
# to be reused when making several requests to the same host/proxy.
session = requests.Session()
urls = [
"http://www.example.org",
"http://www.example.com",
"http://www.example.net",
]
for url in urls:
response = session.get(url, proxies=proxies, timeout=10)
assert response.status_code == 200
assert "Example Domain" in response.text
print("All tests passed!")
The Simple Web Server developed in Lab 3 is a good starting point for testing your proxy, as it gives you complete transparency and control over the origin server. Aside from handling simple requests, it could also be used to test specific scenarios, such as the proxy's behaviour when an origin server is using a non-standard port. It could also be modified and used to test various error conditions, such as when an origin server is slow to respond or closes a connection unexpectedly.
It's also a reasonable starting point for the side of your proxy code that handles persistent client connections.
Most origin servers no longer support (insecure) HTTP. Rather, they'll typically send a 3xx response redirecting the user agent to use HTTPS. Here is a list of some sites that still support HTTP (as of the beginning of this assignment):
Please do not communicate anything sensitive over HTTP.
Once you have correctly implemented the CONNECT method, you should be able to interact with any HTTPS origin server through your proxy just as you would when bypassing it.
httpstat.us supports both HTTP and HTTPS and provides a simple service for generating different HTTP response codes. It also allows you to insert additional headers in the response, and delay the server sending a response. These could be useful, for example, in testing how your proxy handles appending its proxy psuedonym in an existing Via header, how it handles a server that is slow to respond, or how it handles concurrency, by sending multiple parallel requests with varying delays.
Here is a simple example using curl:
$ curl -x '127.0.0.1:60893' -i \
-H "X-HttpStatus-Response-Via: 1.1 foo" \
"http://httpstat.us/420?sleep=5000"
HTTP/1.1 420 Enhance Your Calm
Content-Length: 21
Connection: keep-alive
Content-Type: text/plain
Date: Thu, 15 May 2025 07:23:30 GMT
Server: Kestrel
Set-Cookie: ARRAffinity=1c4ce6b282a4edef63e94171500d99e8b18888422937ab7168b9007141be8730;Path=/;HttpOnly;Domain=httpstat.us
Via: 1.1 foo, 1.1 z1234567
Request-Context: appId=cid-v1:3548b0f5-7f75-492f-82bb-b6eb0e864e53
420 Enhance Your Calm
httpbin.org is similar to httpstat.us, and also supports both HTTP and HTTPS, but provides a more powerful API, allowing you to test many aspects of your proxy. It also follows the OpenAPI Specification, so the automatically generated website provides a well documented and interactive interface for each of the supported endpoints. Additionally, the website provides curl commands that can be easily copied into the terminal, where you can then specify your proxy address (with -x) and any other relevant options.
Here is a simple example using curl:
$ curl -x '127.0.0.1:60893' "http://httpbin.org/base64/R29vZCBsdWNrIHdpdGggdGhlIENPTVAzMzMxLzkzMzEgYXNzaWdubWVudCEK"
Good luck with the COMP3331/9331 assignment!
If you visit webhook.site in a browser it will generate a unique URL where you can send and retrieve data over both HTTP and HTTPS.
Assuming the URL it generated was:
https://webhook.site/123-abc
Here's a simple example using curl:
$ curl -x '127.0.0.1:60893' \
--json $'{"Hello": "COMP3331/9331!"}' \
'http://webhook.site/123-abc'
This URL has no default content configured. <a href="https://webhook.site/#!/view/123-abc">View in Webhook.site</a>.
$ curl -x '127.0.0.1:60893' \
-H 'accept: application/json' \
'http://webhook.site/token/123-abc/request/latest/raw'
{"Hello": "COMP3331/9331!"}
See the webhook.site examples page for more information.
Similar to Lab 2, Wireshark can be a valuable tool to capture and inspect all messages sent and received by your proxy, both on the client side and the server side.
Some tips include:
tcp.port == 80 or tcp.port == <port>, where
<port> is replaced by the actual port your proxy is listening on.Important note: the CSE servers do not permit network traffic to be captured, so you would need to do this on your own device.
Scripts are provided through the class account to check your proxy's function and its log output.
Note, you should not rely on these to prove the correctness of your proxy. They are basic sanity checks only.
You can execute some very simple test cases by running:
$ 3331 proxy_check <proxy_port>
Where <proxy_port> is the port your proxy is listening on.
Note, this requires that your proxy is already running on the same CSE server.
By default, this script will issue requests for all 4 methods you are to support, but you can also pass one or more options to be more selective:
-c, --connect test the CONNECT method
-g, --get test the GET method
-i, --head test the HEAD method
-p, --post test the POST method
You are strongly encouraged to run this sanity check multiple times, using different proxy ports, to confirm that your port is not hard-coded.
You can confirm that your proxy logging is formatted correctly by saving the output to a file and running:
$ 3331 log_check <log_file>
Where <log_file> is the path to your log file.
Note, this is merely a syntactical check, which if successful, will provide some summary statistics. It in no way checks that your proxy is functioning correctly.