GSoC libtpproto2-py

Weekly update for July 18th

2009-08-18T17:59:00.000-07:00

This week's work was mostly on the port of tpclient-pytext. The majority of this work was modifying the code to work with the new generator-style API. Besides this was modifying the code to work with the new Object and Order implementations. This is still a little lacking, as the order insertion function hasn't been rewritten yet and the show function needs to be modified. I also discovered some older bugs with packing and unpacking DescStructures, which have been fixed.

Daily Update for August 16th

2009-08-17T08:03:00.000-07:00

After some work, I got all the descriptions working. All that I need to do now is work on the Order and Message methods, but I'm too tired to function any longer, so those will have to be handled tomorrow.

Daily update for August 15th

2009-08-16T09:56:00.000-07:00

Steady progress with tpclient-pytext. It still needs some work with descriptions, which will be my focus tomorrow.

Daily Update for Augist 13th

2009-08-14T02:33:00.000-07:00

I got tpserver-cpp running locally today, and with llnz's help I fixed the connection problem I had been having. The problem was that I was including the header's length in the packet length. This has now been fixed, but I haven't updated the unit tests yet. I will do that tomorrow before I work on extending the client.

Weekly update for August 11th

2009-08-11T18:03:00.000-07:00

This post will be extending more than a week back since I was unable to write my update for last week. As I mentioned yesterday, I've done a lot of work on common and client, which I will touch on separately:

common: While it was mostly functional, common could not pull subclasses of describable packets (order and object) correctly. This required modifying the code in parser to identify the type field in a description and save it so that common could look it up. After that work was mostly on expanding coverage, which is now fairly complete sans the SSL support. As I was unable to connect to any SSL servers with any program, there was no useful means to test it. The testing for this will be pushed to the integration tests with client, as time has become more pressing.

client: Probably more than half the lines in client had to be modified because of API changes over the course of the project. This includes differences in object names between libtpproto-py and protocol.xml, the transitions of objects to being an object rather than a singleton, and some logic changes that have occurred since the original writing. In the current state I don't believe any significiant changes need to be made, but because this is essentially protocol code, my own misunderstanding is a very real possiblity.

tpclient-pytext: This is still in a rudimentary state, as I'm having some difficulties porting it to the new library. As I mentioned before, I have been failing to connect to the servers with the client. The packet appears to be correctly constructed, however, so I fear I'm misunderstanding the protocol. This is presently my top priority. I will be attempting to set up a local server for a more controlled testing environment tonight.

Update for August 10th!

2009-08-10T20:56:00.000-07:00

Hello again, the internet! Thanks to ISP troubles I've been unable to connect over SSL or SSH for the last week. I don't know why, but it seems to have resolved itself.

I now have common in a mostly finished state and have client ready for testing. The insert methods in client require me to essentially cast a packet to another type and send it. This should be straightforward, but I've not implemented it yet.

I've started porting tpclient-pytext to use libtpproto2-py. I'm hung up on being unable to get a reply from the server upon logging in. I've manually determined that the server is not sending a reply, so I now suspect that there might be an error in the connect packet. i will investigate tomorrow.

Daily update for August 2nd

2009-08-03T17:48:00.000-07:00

Fixed the unpacking bug and made ConnectionCommon correctly identify subclasses of describable packets. Expanded coverage to get confidence that ConnectionCommon now works for all packets. Still need a good way to test SSL support, then work on Client.

Daily update for July 31st

2009-08-01T16:28:00.001-07:00

I believe all mentions of a packet's length have been corrected. However, I seem to have found problems with unpacking in the unit tests. I'll be looking into the cause of that tomorrow.

Daily update for July 30th

2009-07-30T22:50:00.000-07:00

I've found some issues with the length attribute of packets. To simplify calculating the length, length is a property that dynamically is calculated based on the data in the packet. This is supposed to be used in the outgoing packets (it is not) while we should be using the encoded length of properties received from the network (we use the calculated value, which is always the length of a header). I am working on fixing this and expect to have everything straightened out tomorrow.

Weekly update for July 28th

2009-07-28T17:18:00.000-07:00

The last few days have been somewhat slow; I've come down with a cold that has wrecked my brain. Common is now roughly done, but I'd like some better coverage on the socket and SSL wrappers, and I realized that the packet receiver does not properly create the subclasses of describable packets. I've got a fix mostly coded, and I think I'll have this written and tested tomorrow.

I've also started looking at the Client module for testing. I would like to actually have it talking to one of the existing TP servers for the tests, but I'm not sure I want that large a dependency for the tests.

Daily Update for July 26th

2009-07-27T00:30:00.000-07:00

Today was mostly code clean up of removing dead code and simplifying some functions. As a result the ConnectionCommon.__recvPacketGenerator method now knows the size of a packet and doesn't have to keep feeding bytes to the Header class until it works.

Tomorrow I'll try to solve my problems with the __process__ method not knowing what arguments to pass to __init__, as well as figuring out a good way to test the socket wrapping classes.

Daily Update for July 24th

2009-07-25T01:13:00.000-07:00

I found the origin of the failures I mentioned of the __get__ methods of structures. It turns out the __get__ method is only called when the structure is a member of the class; and the structures in Header were not actually class members. This is now remedied.

Daily Update for July 23rd

2009-07-24T00:31:00.000-07:00

Today I got ConnectionCommon._recvFrame working. There was some magic involved, such as giving Packets a second constructor. I also ran into some strange problems where accessing the structures of a packet returned the structure rather than the proxy. I am hoping to fix this tomorrow because it seems an afront to the laws of both man and God.

Daily Update for July 22nd

2009-07-22T20:47:00.000-07:00

Today's work was in ConnectionCommon. I'm presently working on the __recvFrameGenerator method. This method essentially pulls packets off the network as strings and translates them into the correct object. It also appears to be written with a different Packet class in mind, referencing, methods, properties, and exceptions that don't exist. Tomorrow I intend to have the logic gap bridged, and then I can figure out how to properly test the Connection class.

Weekly Update for July 21st

2009-07-21T16:20:00.000-07:00

This week has been working on Common and cleaning up Parser, Structures, and Xstruct. I've found that Common has some expectations of Parser that aren't (yet) true. I am working on reconcilliation of the two, but I want to get the previous branches merged with the master to minimize conflicts in the work.

For the more shareable work: Parser now has TP04 version numbering. Structures now has a more accessible API and unit tests that reflect it. Xstruct now is a port of the libtproto-py TP04 branch, with bug fixes and expanded unit tests to cover the new functionality and a few cases that I had previously overlooked.

Update for July 20th

2009-07-21T14:06:00.000-07:00

There's not much to say about the work on Common right now. I'm looking for the best way to test it, and the best answer I've come up with is using a dummy socket to capture the Connection I/O. For the protocol changes I mentioned before; the Connection objects now take a Protocol argument in addition to what they took previously.

In the other branches I've cleaned up the unit tests in Structures, and fixed the previously mentioned bug with lists that take only a single list in xstruct. I think those branches are ready to be merged into the master branch.

Daily Update for July 17th

2009-07-17T23:01:00.000-07:00

I've got StringQueue passing its UnitTests now. Working on the rest of the common module I've become aware that at several points the objects module is referenced. This module no longer exists, as Parser now creates a new object whenever it is run that has the contents of the former singleton. I'll discuss with mithro tomorrow how the module should be adapted for this.

Daily Update for July 15th

2009-07-16T02:30:00.000-07:00

Todays I got the new xstruct compliant with the unit tests. This included adding support for character and floating point types, and fixing a bug with {} lists. I also added some coverage tests for the new callback features.

I also became aware that the pack and unpack callback methods have an inconsistent API. I may change this after I speak with mithro tomorrow. Additionally, I have found that there's an edge case where a list whose only element is another list will have an inconsistent pack/unpack API. Once I figure out how to fix this I don't expect it to be more than a few lines.

Weekly Update fol July 14th

2009-07-14T17:45:00.000-07:00

This week the parser saw some flexiblity improvements. As I mentioned yesterday it's now encapsulated in a class and returns a unique protocol each time it's run so multiple protocols can be used simultaneously.

I also worked on improving some of the components I've already worked on. I've made some readability improvements to the structures module and have begun porting the xstruct module from libtpproto-py's TP04 branch. The xstruct module does not yet pass the unit tests, so it's not ready to be merged yet, but this should be remedied soon.

I've also been reading over the Common and Client modules to figure out how they can best be tested. There is not much to show for that yet, but I'm hoping to have something soon.

Daily Update for July 12th

2009-07-12T23:04:00.000-07:00

I've missed some daily updates; my ISP had been quite convinced that Blogger was not a real website. It seems better now, so I should be updating daily again.

At mithro's suggestion the parser returns a new protocol each time it's called rather than putting the definition in the objects module; thus multiple protocols can be used at once. I've also encapsulated the parser into a class, in response to some superstition about global variables or something, but this at least makes the parser threadsafe.

mithro has also informed me that the header that was previously in use doesn't work with TP04 because of the header format has changed to a major/minor version scheme. The minor version (the "Frame Type Version") isn't in protocol.xml, so implementing this is mostly waiting on thinking of a good answer.

What I've done so far.

2009-07-08T14:11:00.000-07:00

Mithro asked me to write a summary of what I've done so far in GSoC, so here I go:

I started on the xstruct module (an improved version of Python's struct module), as it did not depend on any other modules but was used by much of the rest of the library. My work here was mostly writing unit tests as xstruct was not a new module, but I did find flaws in a number of types wihch I presume simply hadn't been tested. This included incorrect unicode support for string encoding and floating-point types having been overlooked entirely.

After that I began work on the structures module. This module is something of a high-level wrapper of xstruct. It provides a kind of type-safe way to represent the structures of xstruct. I of course wrote unit tests for this, which uncovered quite a lot of bugs throughout that needed to be addressed. The bulk of code changes were in the Group and List structures, as they represented mutable and nestable types. These were in a pretty unusable state, but now do exactly what they're supposed to do.

In the last few weeks I've been working on the parser module of libtpproto2-py. This module is an XML parser which reads the protocol.xml file that specifies the Thousand Parsec protocol, and generates classes representing all the kinds of packets that might be transmitted. This module required quite a lot more work. The parser was SAX-based and took me a lot of work to be able to follow. Because a primary goal of libtpproto2-py is readability, I rewrote the parser from scratch using a DOM approach which I find to be much more readable. The previous parser also was incomplete in its parsing of protocol.xml. Most notable is that it had no support for parameters, which are used by the server to specify custom objects and orders for the game being played. Getting full parameter support has been the last major feature I've completed.

Weekly update for June 7th

2009-07-08T12:51:00.000-07:00

Parser should now be in a roughly finished state. All that should be remaining is getting 100% coverage on it, and improving readability until I get Mithro's okay to push.

Daily Update for July 6th

2009-07-07T01:40:00.000-07:00

I think everything in Parser works now. Tomorrow I'll clean everything up and, if I find nothing missing, will push it on to the main repository.

Daily Update for July 5th

2009-07-06T01:05:00.000-07:00

Only problem with DescStructure right now is that its check method requires state information it doesn't have access to in order to work. Mithro suggested that I write a GroupStructure that can encompass DescStructure and the field it needs to determine its structure, and let that Group handle checking. I'll work on that tomorrow, and finish making useparameters work (which I believe will mostly entail making sure I understand their behavior correctly.)

Dailf update for July 4th

2009-07-04T23:30:00.000-07:00

Well, testing says that Descstructure doesn't work. I've got it mostly hammered out now, hoping to figure out what is going on with it in the morning. My only explanation right now is "spooky action at a distance".

Daily Update for July 3rd

2009-07-04T00:48:00.000-07:00

I think descparameters work now. Tomorrow I'll properly test them and work on getting useparameters working.

Daily Update for July 2nd

2009-07-03T00:40:00.000-07:00

Today's work was on implementing packing packets as I touched on in yesterday's post. As usual, GroupStructure threw me off for a while, but has just been fixed. Tomorrow I'll write the Desc wrappers and tie it all together to make packets all packable.

Daily Update for July 1st

2009-07-01T23:26:00.000-07:00

The main focus today was how to handle the additional values in a packet's structure from parameters. Packets with use parameters are simple as they can simply be appended. Desc parameters are more difficult because they end up inside lists, giving them a heterogeneous structure. The solution I'm leading toward at present is creating a Structure class that can wrap a List or Group and which will defer to the wrapped class for most operations, but will provide pack and unpack methods which can handle the heterogeneity this requires. With luck this should be implemented tomorrow.

Weekly update for June 30th

2009-06-30T18:11:00.000-07:00

The bug with coverage.py has been confirmed. The exact nature of the bug hasn't been determined, but figleaf appears to be more reliable and will be the source of coverage reports from here on.

Parser is nearly finished now. All that remains is to get the logic in-line with the message protocol, which should not take more than a few days. Once this is complete work on Client/Server/common will begin; for real this time.

Daily Update for June 24th

2009-06-25T17:57:00.000-07:00

Mostly coverage work today. Wrote several XML files for all the verification tests. I'm also suspecting that I've run into a bug with coverage.py because of some strange holes in the reported coverage. Investigation will follow.

Weekly update for June 23rd

2009-06-23T18:43:00.000-07:00

This week most the work was on the DOM parser with a fix of the sorting method or ListProxy.

The parser now has more complete verification and several test files. Parameters are mostly implemented now, but are still lacking some of the semantic logic to tie them together to the packets. This should be remedied in the next few days, followed by work on Client/Server/common.

Weekly update for June 16th

2009-06-16T18:39:00.000-07:00

This week I pulled 100% coverage of structures and 99.5% coverage of xstruct, as well as finishing up the black magic of structures (Group and List.)

I'm on to Parser, where I have unit tests for the parts that I can confirm work (pretty much everything except parameterlists and enumerations.) Being sick the last few days has robbed me of the mental capacity to add the missing functionality to Parser, so I bided time by rewriting Parser in DOM rather than SAX. The DOM parser has enumeration support, but no paramatersets yet. I am waiting on mithro's approval before I decide whether or not to replace the SAX.

Work for June 13th

2009-06-14T14:43:00.000-07:00

Fixed up parser's imports so it runs. It has no support for parametersets, though. I also discovered that protocol.xml was buggy and grabbed a new version from the documents repository. Also found a couple instances of eval() to get members of modules. Replaced them with getattr.

Work for June 11th

2009-06-11T22:16:00.000-07:00

I think ListProxy now supports every list operation (I know I'm going to be shown wrong soon). Structures now has 100% test coverage, and xstruct has just 1 uncovered line related to traceback formatting.

Tomorrow I should be starting on parser.

Work for June 10th

2009-06-10T16:33:00.000-07:00

Most of today's work consisted of looking at segments of xstruct and structures that the unit tests haven't touched and finding a way to get them covered.

xstruct is now down to six uncovered lines, and structures down to 4, for an aggregate 98% coverage. Said lines are either very difficult/impossible to hit or of questionable utility, so I will seek mthiro's wisdom before proceeding.

How structures works

2009-06-09T18:02:00.000-07:00

structures contains much more deep magic than I was expecting when I started on it. For the benefit of anyone who might be working on it later, or the curious, I will try to explain

Basics

The structure classes all implement type-safe wrappers for data types. What this means for you is if your object has a StringStructure member, then you can pretty much treat it just like a string. You can plug it in wherever you would need a string and you can set it to any string value. The difference is if you try to set it to a non-string value an exception will be thrown. In libtpproto2-py this is used to catch wrong values for types as soon as possible rather than when we attempt to send them over the protocol.

How It Works

To achieve these properties we essentially have to change how Python acts when a Structure is get or set on an instance of a class. Fortunately for us, Python provides tools for doing just this. The Structure classes all override the __get__ and __set__ methods. These methods are called when a member of a class is read from or written to respectively. Each Structure stores a special variable in the class that contains the value it is storing. So, if have a StringStructure named "string" in an object A" and write to it, it first checks that the new value is a legal string. If this value is legal it stores it in A.__string. Likewise, when A.string is accessed, it actually returns A.__string.

Group and List

The above method works great for integers, times, and strings. This is because they may directly into an immutable Python type. Group, however acts more like an object, having mutable members, and List acts like a list of groups.

Suppose , for instance, an object A has a group g with a string s, and g returns a regular object with its members then:

A.g.s = "string"

Would modify s in the returned object, but this would not be reflected in A. Thus these mutable types need to return a more sophisticated object. We call this a Proxy.

Groups and their Proxies

A GroupProxy is created with awareness of what object it is a member of and what GroupStructure it proxies. It overrides __getattr__ and __setattr__ methods. These methods function like __get__ and __set__, but they belong to the parent object rather than the accessed child. These are overloaded to find the corresponding Structures within its Group.

Because it is relatively easy with Groups for Structures at different levels to share a name, we use a more complex naming scheme. In the example before A.g.s stores its values in A.__g_s rather than A.__s. This, combined with each Group ensuring that no two members share a name, means any Group is guaranteed to have unique qualifiers for every child.

The way Group achieves this is whenever it receives a name, for every child: it takes its own name, append an underscore, then adds the child's original name (stored as id in all Structures), and sets that as the child's new name. Notice that if any children are Groups this will propagate all the way down. This is important since the children will exist before the parent, since they are passed to the parent's constructor. It's also important to use the child's original name or g.g.s's name will be "s", then prepend "g_" for "g_s", then prepend "g_g_" for "g_g_g_s".

Lists and their Proxies

Naming threatened to get even trickier with lists. Suppose We have a list, A.l where every element of the list is a string. Using groups method, a list of strings would all want to store their special data somewhere like A.__l_s, which would mean they would all end up with the same value. Rather than try to work the index into the name as well (and force some name propagation nightmares) Everything is stored under A.__l. A.__l is a list of objects which have only one member: a GroupStructure named "group". The GroupStructure will contain whatever is stored in the list. Everything in Group i will be stored in A.__l[i] and not have to worry about stepping on anyone's toes.

ListProxies try to act as much like Python lists as possible. Rather than overriding __getattr__ and __setattr__ they override __getitem__ and __setitem__, which behave equivalently with indexing items in a list. Thus, when ListProxy.__getitem__(i) is called, it returns A.__l[i].group. This means it returns a GroupProxy, and can then access the members of the list following the rules set in GroupProxy.

Shortcomings

This method does its job well enough, essentially pushing strict typing on a loosely typed language without muddling the syntax, but it's not perfect. The biggest fault I can find with it right now is underscores in names can cause havoc. A structure named "getattr__" will overwrite its parent object's __getattr__ method and cause an exception to throw any time a member of the parent is read. Likewise clever usage of underscores could cause an object to share a qualifier with the child of a group at its level. The answer for this within my project is that Thousand Parsec's protocol isn't trying to destroy its applications, so this shouldn't come up. This is not a good answer, however. More practical solutions could mean restricting underscores from structure names, or finding somewhere else to store the data. For now I'll leave that for future generations to ponder.

Plans for June 16th

2009-06-09T17:54:00.000-07:00

Now that the foundational modules are well tested I'll be digging into the meatier parts of the project. For the next week I'll be working on the parser module, which reads protocol.xml and generates protocol information; essentially generating the protocol API at run time.

Update for June 9th

2009-06-09T17:43:00.000-07:00

Work on the structures package has been mostly completed now. The unit tests all check out and cover most of the code (about 70-80% by my estimates). I will be expanding that coverage as the project continues.

Plans for June 9th

2009-06-02T18:05:00.000-07:00

Now that the xstruct module is "done" (as much so as can be said of any code 2 weeks into a project), following Mithro's suggestion I'll be working on the structures module. This is something of a higher-level wrapper for xstruct. It should also be the level at which the data transmitted with the protocol will be defined at.

I expect this will be similar work to what I did for xstruct. Most the functionality is already there, so I'll be focusing on rigorous unit testing and fix any broken or missing functionality I find while testing.

Update for June 2nd

2009-06-02T17:30:00.000-07:00

This week I worked on the xstruct module. I wrote unit tests for pack and unpack that:

Ensured all types in the documentation could be packed and unpacked to get the original value.
Ensured pack generated the expected string from a tuple, and unpack generated the expected tuple from a string.
Ensured an exception was thrown for type mismatches.
Ensured an exception was thrown when an integer did not fit within its type.

In doing this, I corrected bugs that disallowed the 'c', 'f', and 'd' (single ASCII character, float, and double) types from being used. I also added bounds checking for all the integer types, as previously the module only checked that unsigned types did not have negative values.

Purpose

2009-05-26T18:33:00.000-07:00

This blog is for my Summer of Code project. I'll be working for Thousand Parsec on libtpproto2-py. That is, the 2nd Python library implementating the Thousand Parsec protocol. I'm writing this in a blog for several reasons:

To keep my mentor and anyone else who might be following my project as up to date as possible.
To force myself to know what it is I'm doing at every point.
To illucidate any future developers of any design decisions I'll have to make.
To give any curious readers an idea of what GSoC is really like.

I'll be posting any pertinent thought to those subjects here.