Princeton University
COS 333: Advanced Programming Techniques

Purpose

The purpose of this assignment is to help you learn or review three-tiered programming. More specifically, the assignment will help you to learn or review network and concurrent programming in Python.

Rules

Make sure you study the COS 333 Policies web page before doing this assignment or any of the course's assignments.

Your Task

As with Assignment 1, pretend that you're working for Princeton's Registrar's Office. You're given a database containing data about classes and courses offered during an upcoming Princeton semester. Your task is to create Python applications that allows Princeton students and other interested parties to query the database.

Your Assignment 1 programs are not particularly realistic. Specifically, it's not realistic for the Princeton Registrar's Office database to be present on the computers of Princeton students. Instead it would be more realistic for a single database to be present on a computer that is administered by the Registrar's Office, for the Registrar's Office to run a server program that makes the database accessible to client programs, and for Princeton students to have client programs on their computers that communicate with the server program to fetch data from the database. In this assignment your task is to compose an application that implements that three-tiered architecture.

For this assignment you must compose three programs: client programs named regoverviews.py and regdetails.py, and a server program named regserver.py. Your client programs must communicate with your server program. Your client and server programs might be running on the same computer, or might be running on different computers. That is, your application must be networked.

The Given Files

Browse to the TigerFile page for this assignment. Download the reg2.zip file. Then unzip that file to create files named reg.sqlite, ref_regoverviews.pyc, ref_regdetails.pyc, ref_regserver.pyc, testregoverviewsgiven.py, testregdetailsgiven.py, and replace.py. Subsequent sections of this document describe those files.

The Database

The database is identical to the one from Assignment 1. The specification of Assignment 1 provides a description. The database is stored in the given file named reg.sqlite.

The Communication Protocol

The given client programs and your client programs must communicate with the given server program and your server program. For that to be possible, the client programs and server programs must use a defined communication protocol. This section specifies that protocol.

Suppose a client wants to fetch class overviews. In that case the client must send to the server a JSON document representing a Python object. The object must be a list object. The first element of the list object must be the str object "get_overviews". The second element of the list object must must be a dict object. There must be four bindings in the list object, having keys "dept", "coursenum", "area", and "title". (The values of some of the bindings might be the empty string.) For example, a client might send to a server a JSON document representing this Python object:

['get_overviews', {'dept':'COS', 'coursenum':'2', 'area':'qr', 'title':'intro'}]

In response, the server must send to the client a JSON document representing a Python object. The object must be a list object. The first element of the list object must be a bool object — indicating whether the server handled the request successfully or not. If the bool object is False, then the second element must be a str object which is an error message. If the bool object is True, then the second element must be a list object. Each element of the list object must be a dict object containing the data for one class. For example, if the client sends the server the above request (and there are no errors), then the server must send to the client a JSON representation of this Python object:

[True, [
   {'classid':8308, 'dept':'COS', 'coursenum':'217', 'area':'QR',
      'title':'Introduction to C_Science Programming Systems'},
   {'classid':9240, 'dept':'COS', 'coursenum':'342', 'area':'QR',
      'title':'Introduction to Graph Theory'}]]

The classes within the list object must be sorted; the primary sort must be by dept in ascending order, the secondary sort must be by coursenum in ascending order, and tertiary sort must be by classid in ascending order.

Now suppose a client wants to fetch class details. In that case the client must send to the server a JSON document representing a Python object. The object must be a list object. The first element of the list object must be the str object "get_details". The second element of the list object must an int object which is a classid. For example, a client might send to a server a JSON representation of this Python object:

['get_details', 8321]

In response, the server must send to the client a JSON document representing a Python object. The object must be a list object. The first element of the list object must be a bool object — indicating whether the server handled the request successfully or not. If the bool object is False, then the second element must be a str object which is an error message. If the bool object is True, then the second element must be a dict object containing the details for the specified class. For example, if the client sends the server the above request (and there are no errors), then the server must send to the client a JSON representation of this Python object:

[True, {
   'classid':8321,
   'days':'TTh',
   'starttime':'11:00AM',
   'endtime':'12:20PM',
   'bldg':'FRIEN',
   'roomnum':'006',
   'courseid':3672,
   'deptcoursenums':[{'dept': 'COS', 'coursenum': '333'}],
   'area':'',
   'title':'Advanced C%Science Programming Techniques',
   'descrip':'This is a course about the practice of programming.  Programming is more than just writing code.  Programmers must also assess tradeoffs, choose among design alternatives, debug and test, improve performance, and maintain software written by themselves & others. At the same time, they must be concerned with compatibility, robustness, and reliability, while meeting specifications.  Students will have the opportunity to develop these skills by working on their own code and in group projects.',
   'prereqs':'COS 217 and COS 226.',
   'profnames': ['Brian W. Kernighan']
}]

Within the deptcoursenums list the elements must be sorted primarily by department and secondarily by course number. Within the profnames list the elements must be sorted.

The regoverviews.py Client Program

Compose your regoverviews.py program. Your regoverviews.py program must have the same behavior as the ref_regoverviews.pyc program.

When executed with -h as a command-line argument, your regoverviews.py must display a help message that describes the program's behavior:

$ python regoverviews.py -h
usage: regoverviews.py [-h] [-d dept] [-n num] [-a area] [-t title] host port

Registrar application: show overviews of classes

positional arguments:
  host        the computer on which the server is running
  port        the port at which the server is listening

options:
  -h, --help  show this help message and exit
  -d dept     show only those classes whose department contains dept
  -n num      show only those classes whose course number contains num
  -a area     show only those classes whose distrib area contains area
  -t title    show only those classes whose course title contains title

Your regoverviews.py must accept two command-line arguments. The first must be the host — the IP address or domain name of the computer on which the server is running. The second must be the number of the port at which the server is listening. Thereafter your regoverviews.py must accept optional arguments, as does the Assignment 1 regoverviews.py program.

When executed without -h as a command-line argument, and as indicated by the usage message, your regoverviews.py must send a request to its server using the prescribed protocol, receive a response from its server using the prescribed protocol, and write the response to its stdout in the proper format (as defined in Assignment 1). At this point the server must be the given one. Later you'll also be able to use your server.

Perform boundary and statement testing of your regoverviews.py, as described in the Assignment 1 specification. Automate the testing of your regoverviews.py. To do that, make a copy of testregoverviewsgiven.py; name the copy testregoverviews.py. Then use testregoverviews.py in this sequence:

(1) Run the given server on somehost at someport:

python ref_regserver.pyc someport

(2) Run testregoverviews.py on your regoverviews.py, and then on the given ref_regoverviews.pyc:

python testregoverviews.py regoverviews.py somehost someport > out1 2>&1
python testregoverviews.py ref_regoverviews.pyc somehost someport > out2 2>&1
python replace.py out2 ref_regoverviews.pyc regoverviews.py

(3) Make sure the contents of out1 and out2 are identical.

Then enhance testregoverviews.py by adding more tests, and repeat that procedure.

The regdetails.py Client Program

Compose your regdetails.py program. Your regdetails.py program must have the same behavior as the ref_regdetails.pyc program.

When executed with -h as a command-line argument, your regdetails.py must display a help message that describes the program's behavior:

$ python regdetails.py -h
usage: regdetails.py [-h] host port classid

Registrar application: show details about a class

positional arguments:
  host        the computer on which the server is running
  port        the port at which the server is listening
  classid     the id of the class whose details should be shown

options:
  -h, --help  show this help message and exit

Your regdetails.py must accept three command-line arguments. The first must be the host — the IP address or domain name of the computer on which the server is running. The second must be the number of the port at which the server is listening. The third must be a classid.

When executed without -h as a command-line argument, and as indicated by the usage message, your regdetails.py must send a request to the server using the prescribed protocol, receive a response from its server using the prescribed protocol, and write the response to its stdout in the proper format (as defined Assignment 1). At this point the server must be the given one. Later you'll also be able to use your servers.

Perform boundary and statement testing of your regdetails.py, as described in the Assignment 1 specification. Automate the testing of your regdetails.py. To do that, make a copy of testregdetailsgiven.py; name the copy testregdetails.py. Then use testregdetails.py in this sequence:

(1) Run the given server on somehost at someport:

python ref_regserver.pyc someport

(2) Run testregdetails.py on your regdetails.py, and then on the given ref_regdetails.pyc:

python testregdetails.py regdetails.py somehost someport > out3 2>&1
python testregdetails.py ref_regdetails.pyc somehost someport > out4 2>&1
python replace.py out4 ref_regdetails.pyc regdetails.py

(3) Make sure the contents of out3 and out4 are identical.

Then enhance testregdetails.py by adding more tests, and repeat that procedure.

The regserver.py Program

First compose a preliminary server program named regserverprelim.py. Your regserverprelim.py program must have the same behavior as the ref_regserver.pyc program, except that it must not use multiple threads.

When executed with -h as a command-line argument, your regserverprelim.py must display a help message that describes the program's behavior:

$ python regserverprelim.py -h
usage: regserver.py [-h]  port

Server for the registrar application

positional arguments:
  port              the port at which the server should listen

options:
  -h, --help        show this help message and exit

When executed without -h as a command-line argument, and as indicated by the usage message, your regserverprelim.py must listen for client requests on the given port. When it receives a request using the prescribed protocol, it must fetch data from the database, formulate a response using the prescribed protocol, and send the response to the client. Your regserverprelim.py must work with the given clients, and also with your clients.

Your regserverprelim.py must handle the SQLite database. Your client programs must not access the SQLite database. Assume that the client program is running on computer X, your reqserver.py is running on computer Y, and the SQLite database is located on computer Y where X may not be the same as Y.

Your regserverprelim.py must handle '%' and '_' as ordinary characters, not wildcard characters, just as described in the Assignment 1 specification.

Your regserverprelim.py must protect itself from SQL injection attacks by using SQL prepared statements.

The connection between your regserver.py and the database must not be persistent. That is, it must not be the case that your regserver.py creates a database connection upon startup, and uses that database connection during the entire execution of your regserver.py.

Instead the database connection must be transient. Each time a client contacts your regserver.py, your regserver.py must create a database connection, fetch data from the database using that connection, and then close that connection.

Test your regserverprelim.py program by repeating your testing of your regoverviews.py and regdetails.py programs — this time using your regserverprelim.py instead of the given server.

After composing your regserverprelim.py program, compose program named regserver.py. Your regserver.py must have the same behavior as your regserverprelim.py program, except that it must use multiple threads. That is, each time it receives a client request, it must spawn a new thread to handle that request. Thus your regserver.py must have the same behavior as ref_regserver.pyc.

Test your regserver.py just as you tested your regserverprelim.py.

Delay Handling

Your regserverprelim.py and regserver.py programs must accept environment variables named IODELAY and CDELAY.

The IODELAY environment variable provides a way for you to observe the behavior of your application when it is experiencing I/O delays — such as when the server is waiting for responses from a slow database. If the user specifies an IODELAY of n seconds, then your server must "sleep" for n seconds by calling time.sleep(n). The delay must occur once for each client request, immediately before the application accesses the database. The IODELAY environment variable must default to 0.

The CDELAY environment variable provides a way for you to observe the behavior of your application when it is experiencing compute delays — such as when your server is doing some intense arithmetic computation. When the user specifies a CDELAY of n seconds, then your server must consume n seconds of processor time. (Lectures describe how to do that.) The delay must occur once for each client request, immediately before the application accesses the database. The CDELAY environment variable must default to 0.

Perform experiments with your regserverprelim.py and regserver.py server programs, using a variety of values of the IODELAY and CDELAY environment variables. (The lectures suggest some relevant experiments.) Then answer these questions in your readme file:

1. With respect to I/O and compute delays, under what circumstances is regserver.py better than regserverprelim.py?
2. With respect to I/O and compute delays, under what circumstances is regserver.py not better than regserverprelim.py?

Error Handling

Your application must be robust. It must be impossible for any client request to cause your server to exit. As with Assignment 1, each error message written by your application must be preceded with the name of the program, a colon, and a space. Generally, your application must implement this error handling strategy:

• If your server detects an error that is related to a client's request, then your server must send an error message to the client, and your client must write the error message to its stderr.
• If your server detects an error that is unrelated to the client's request (for example, a missing or corrupted database), then your server must write an error message to its stderr (thus reporting the error to the server's system administrators). Then it must send a generic error message to the client (thus, for security, not reporting the exact nature of the error to the client.) Your client must write the generic error message to its stderr.

More specifically, your application must handle the following errors:

Incorrect environment variables

If the value of the IODELAY or CDELAY environment variable cannot be converted to an integer, then your server silently must consider the value to be 0.

Incorrect command-line arguments

If your server or client is given command-line arguments that argparse can detect as incorrect, then argparse indeed must detect them as incorrect. In that case your program must write a descriptive message to its stderr and exit the process with status 2, as is the default when using argparse.

Unavailable port

If your server is given a command-line argument specifying a port that is unavailable (typically because it already in use), then it must write a descriptive error message — the one contained within the thrown exception object — to its stderr and exit with status 1.

Unavailable server

If your server is unavailable on the specified host at the specified port at the time your client sends a request, then your client must write a descriptive error message — the one contained within the thrown exception object — to its stderr and exit with status 1.

Non-existing classid

If your regdetails.py sends a "class details" request specifying a classid that does not exist in the database, then:

• Your server must send a descriptive error message to your regdetails.py and continue executing.
• Your regdetails.py must write the descriptive error message to its stderr and exit with status 1.

Database cannot be opened

If your server cannot open the database when your client sends a request, then:

• Your server must write a descriptive error message — the one contained within the thrown exception object — to its stderr, send a generic "A server error occurred. Please contact the system administrator." error message to your client, and continue executing.
• Your client must write the generic error message to its stderr and exit with status 1

Corrupted database

If the database is corrupted when your client sends a request such that on your server the SQLite driver's execution of a SELECT statement throws an exception, then:

• Your server must write a descriptive error message — the one contained within the thrown exception object — to its stderr, send a generic "A server error occurred. Please contact the system administrator." message to your client, and continue executing.
• Your client must write the generic error message to its stderr and exit with status 1.

Submission

Compose a readme file. Your readme file must contain:

• Your name and your teammate's name.
• A paragraph describing your contribution, and another paragraph describing your teammate's contribution. Please be thorough; we're looking for two substantial paragraphs, not a sentence or two.
• A description of whatever help (if any) you received from other people while doing the assignment, as prescribed by the course Policies web page.
• A description of the sources of information that you used while doing the assignment.
• An indication of how much time you spent doing the assignment.
• Your assessment of the assignment: Did it help you to learn? What did it help you to learn? Do you have any suggestions for improvement? Etc.
• (Optionally) Any information that will help us to grade your work in the most favorable light. In particular you must describe all known bugs.
• Your answers to the two questions noted previously.

Your readme file must be a plain text file. Don't create your readme file using Microsoft Word or any other word processor.

Submit your assignment files using the TigerFile page. Make sure you submit: