Daily update for July 30th

I've found some issues with the length attribute of packets. To simplify calculating the length, length is a property that dynamically is calculated based on the data in the packet. This is supposed to be used in the outgoing packets (it is not) while we should be using the encoded length of properties received from the network (we use the calculated value, which is always the length of a header). I am working on fixing this and  expect to have everything straightened out tomorrow.

Weekly update for July 28th

The last few days have been somewhat slow; I've come down with a cold that has wrecked my brain. Common is now roughly done, but I'd like some better coverage on the socket and SSL wrappers, and I realized that the packet receiver does not properly create the subclasses of describable packets. I've got a fix mostly coded, and I think I'll have this written and tested tomorrow.

I've also started looking at the Client module for testing.  I would like to actually have it talking to one of the existing TP servers for the tests, but I'm not sure I want that large a dependency for the tests. 

Daily Update for July 26th

Today was mostly code clean up of removing dead code and simplifying some functions.  As a result the ConnectionCommon.__recvPacketGenerator method now knows the size of a packet and doesn't have to keep feeding bytes to the Header class until it works.

Tomorrow I'll try to solve my problems with the __process__ method not knowing what arguments to pass to __init__, as well as figuring out a good way to test the socket wrapping classes.

Daily Update for July 24th

I found the origin of the failures I mentioned of the __get__ methods of structures. It turns out the __get__ method is only called when the structure is a member of the class; and the structures in Header were not actually class members. This is now remedied.

Daily Update for July 23rd

Today I got ConnectionCommon._recvFrame working. There was some magic involved, such as giving Packets a second constructor. I also ran into some strange problems where accessing the structures of a packet returned the structure rather than the proxy. I am hoping to fix this tomorrow because it seems an afront to the laws of both man and God.

Daily Update for July 22nd

Today's work was in ConnectionCommon. I'm presently working on the __recvFrameGenerator method. This method essentially pulls packets off the network as strings and translates them into the correct object. It also appears to be written with a different Packet class in mind, referencing, methods, properties, and exceptions that don't exist. Tomorrow I intend to have the logic gap bridged, and then I can figure out how to properly test the Connection class.

Weekly Update for July 21st

This week has been working on Common and cleaning up Parser, Structures, and Xstruct. I've found that Common has some expectations of Parser that aren't (yet) true.  I am working on reconcilliation of the two, but I want to get the previous branches merged with the master to minimize conflicts in the work.

For the more shareable work: Parser now has TP04 version numbering. Structures now has a more accessible API and unit tests that reflect it. Xstruct now is a port of the libtproto-py TP04 branch, with bug fixes and expanded unit tests to cover the new functionality and a few cases that I had previously overlooked.

Update for July 20th

There's not much to say about the work on Common right now.  I'm looking for the best way to test it, and the best answer I've come up with is using a dummy socket to capture the Connection I/O. For the protocol changes I mentioned before; the Connection objects now take a Protocol argument in addition to what they took previously.

In the other branches I've cleaned up the unit tests in Structures, and fixed the previously mentioned bug with lists that take only a single list in xstruct. I think those branches are ready to be merged into the master branch.

Daily Update for July 17th

I've got StringQueue passing its UnitTests now. Working on the rest of the common module I've become aware that at several points the objects module is referenced. This module no longer exists, as Parser now creates a new object whenever it is run that has the contents of the former singleton. I'll discuss with mithro tomorrow how the module should be adapted for this.

Daily Update for July 15th

Todays I got the new xstruct compliant with the unit tests. This included adding support for character and floating point types, and fixing  a bug with {} lists. I also added some coverage tests for the new callback features.

I also became aware that the pack and unpack callback methods have an inconsistent API.  I may change this after I speak with mithro tomorrow. Additionally, I have found that there's an edge case where a list whose only element is another list will have an inconsistent pack/unpack API. Once I figure out how to fix this I don't expect it to be more than a few lines.

Weekly Update fol July 14th

This week the parser saw some flexiblity improvements. As I mentioned yesterday it's now encapsulated in a class and returns a unique protocol each time it's run so multiple protocols can be used simultaneously.

I also worked on improving some of the components I've already worked on. I've made some readability improvements to the structures module and have begun porting the xstruct module from libtpproto-py's TP04 branch. The xstruct module does not yet pass the unit tests, so it's not ready to be merged yet, but this should be remedied soon.

I've also been reading over the Common and Client modules to figure out how they can best be tested. There is not much to show for that yet, but I'm hoping to have something soon.

Daily Update for July 12th

I've missed some daily updates; my ISP had been quite convinced that Blogger was not a real website. It seems better now, so I should be updating daily again.

At mithro's suggestion the parser returns a new protocol each time it's called rather than putting the definition in the objects module; thus multiple protocols can be used at once. I've also encapsulated the parser into a class, in response to some superstition about global variables or something, but this at least makes the parser threadsafe.

mithro has also informed me that the header that was previously in use doesn't work with TP04 because of the header format has changed to a major/minor version scheme. The minor version (the "Frame Type Version") isn't in protocol.xml, so implementing this is mostly waiting on thinking of a good answer.

What I've done so far.

Mithro asked me to write a summary of what I've done so far in GSoC, so here I go:

I started on the xstruct module (an improved version of Python's struct module), as it did not depend on any other modules but was used by much of the rest of the library. My work here was mostly writing unit tests as xstruct was not a new module, but I did find flaws in a number of types wihch I presume simply hadn't been tested. This included incorrect unicode support for string encoding and floating-point types having been overlooked entirely.

After that I began work on the structures module. This module is something of a high-level wrapper of xstruct. It provides a kind of type-safe way to represent the structures of xstruct. I of course wrote unit tests for this, which uncovered quite a lot of bugs throughout that needed to be addressed. The bulk of code changes were in the Group and List structures, as they represented mutable and nestable types. These were in a pretty unusable state, but now do exactly what they're supposed to do. 

In the last few weeks I've been working on the parser module of libtpproto2-py. This module  is an XML parser which reads the protocol.xml file that specifies  the Thousand Parsec protocol, and generates classes representing all the kinds of packets that might be transmitted. This module required quite a lot more work. The parser was SAX-based and took me a lot of work to be able to follow. Because a primary goal of libtpproto2-py is readability, I rewrote the parser from scratch using a DOM approach which I find to be much more readable. The previous parser also was incomplete in its parsing of protocol.xml. Most notable is that it had no support for parameters, which are used by the server to specify custom objects and orders for the game being played. Getting full parameter support has been the last major feature I've completed.

Weekly update for June 7th

Parser should now be in a roughly finished state. All that should be remaining is getting 100% coverage on it, and improving readability until I get Mithro's okay to push.

Daily Update for July 6th

I think everything in Parser works now. Tomorrow I'll clean everything up and, if I find nothing missing, will push it on to the main repository.

Daily Update for July 5th

Only problem with DescStructure right now is that its check method  requires state information it doesn't have access to in order to work.   Mithro suggested that I write a GroupStructure that can encompass DescStructure and the field it needs to determine its structure, and let that Group handle checking. I'll work on that tomorrow, and finish making useparameters work (which I believe will mostly entail making sure I understand their behavior correctly.)

Dailf update for July 4th

Well, testing says that Descstructure doesn't work. I've got it mostly hammered out now, hoping to figure out what is going on with it in the morning. My only explanation right now is "spooky action at a distance".

Daily Update for July 3rd

I think descparameters work now. Tomorrow I'll properly test them and work on getting useparameters working.

Daily Update for July 2nd

Today's work was on implementing packing packets as I touched on in yesterday's post. As usual, GroupStructure threw me off for a while, but has just been fixed. Tomorrow I'll write the Desc wrappers and tie it all together to make packets all packable.

Daily Update for July 1st

The main focus today was how to handle the additional values in a packet's structure from parameters. Packets with use parameters are simple as they can simply be appended. Desc parameters are more difficult because they end up inside lists, giving them a heterogeneous structure. The solution I'm leading toward at present is creating a Structure class that can wrap a List or Group  and which will defer to the wrapped class for most operations, but will provide pack and unpack methods which can handle the heterogeneity this requires. With luck this should be implemented tomorrow.