Archive for July, 2009
After 5 weeks, the core work on the Download Statistics project is coming to an end. It has been a fun project to work on with its many ups and downs. It is also refreshing to work on something that has real world value. Over these past 5 weeks, I have been tasked with creating a suite of applications that will retrieve download logs, parse them, and display them in a MindTouch installation. Aside from some optimization and tweaking, these goals have all been met. It is almost time for my suite to be graduated to release candidate.
Here’s a screen cap of what the Dekiscript front end will dynamically generate.
While working on this project, I have come to realize several things:
- If you can do it with a system library, there is likely a mindtouch.dream library that does it better.
- Libraries are made to be used, it is not academic dishonesty to use open source libraries from the Internet in projects.
- If you can’t find your problem/idea on Google, it is not because it doesn’t exist, it is because you are not specifying it correctly.
- It pays to structure the internal folders of the project ahead of time, pays roughly 6 hours of wages to be exact.
There are many more things that I have learned since my first days here, but the above are very directly related to the project that I have been working on. One of the hardest things for me to learn is that the methods inside System are very often time not the methods that I want to use. Coming from Java and structured school assignments, it is also hard to get over the fact that everything you every need isn’t already in the default libraries. Also going along with school assignments, curse not being conditioned to use auto completion!
Google is an amazing resource as long as the search terms are correct. No matter how amazing Google is, it can’t make the connection that when somebody searches for “functions’, they actually mean “methods”. Google happens to also always be right, which means that if it can’t be found on Google, then it is user error. Google is also very powerful and has more uses than is productive <http://lifehacker.com/339474/top-10-obscure-google-search-tricks>.
And lastly, structure program correctly in the beginning because an minor error when starting out can lead to several hours of tedious file moving and project properties tweaking down the line. The project was started in Mono Develop using the default solution creator which led to files and folders being stored in a hierarchy very different from the accepted standard. This in itself could be easily fixed, but when submitted to SVN, this folder structure refused to be changed via Mono Develop’s SVN interface. In the end, it took several hours and Arne’s help to just recreate the structure correctly.
Aside from these, the project went smoothly. Most of the deadlines were met with a bit of task juggling and a product is close to completion.
Post to come:
- Intern’s Advice to Future UCSD Lackeys
- Mono Develop vs Visual Studios
MindTouch 9.02.4, the latest incremental release for the MindTouch Lyons family of releases is now available for all open source and commercial deployments.
This release, which contains 14 bugfixes, should be considered as a high-priority update, as it contains critical fixes for two bugs that could cause data-loss; one which involves redirect pages, and the other which involves file revisions.
With this release, there’s now one change to the API that may affect applications built on top of the MindTouch platform. Bug #6560 prevents access to the POST:users feature without an API key – this is to prevent spammers from programatically hitting the API and creating fake users. If your application is utilizing POST:users, you will have to update your application to send in an API key (or admin credentials).
For savvy Linux users of MindTouch, you may have noticed that there was never a MindTouch 9.02.3 release. This was intentional – MindTouch 9.02.3 was the basis of our MindTouch 2009 for Windows release and did not contain enough critical bugfixes to warrant repackaging. When you download MindTouch 9.02.4, you’ll also get the MindTouch 9.02.3 bugfixes as well.
One of the projects that gets less mentions these days, but is still just as relevant to the ongoing development effort behind MindTouch Core/2009 is MindTouch Dream. Those who have been following MindTouch’s development for a long time know that Dream is the foundation for our web-oriented architecture. It’s a open-source framework (Apache 2.0 license) that makes building and interacting with RESTful web-services incredibly simple. It also includes a portable REST micro-server that enables us to run Core/2009 as a standalone application, a Windows service, or as an IIS handler, either under Windows or Mono. In short, it covers a lot of ground of what makes MindTouch Core/2009 such a unique and advanced product.
Ok, enough about buttering up Dream, but since it hasn’t had many mentions recently, it deserved a bit of attention. What’s more important are the cool new features being added that have much broader appeal than just for web-oriented programming, though they certainly help a lot with it as well. While this work is being done under the banner of Dream Profile 2–or Dream 2.0, the name is still in flux–the features described in this article will be part of Dream 1.7 “Garnet”, which will ship with MindTouch Core/2009 “Minneopa”.
The video below covers some of the new features, including the ElasticThreadPool class and its derived classes. In it I explain at a high-level how the thread pool works and how multiple instances interact. The video is less than 20 minutes long. I had to cut out about two minutes to make it fit within that time limit, otherwise Facebook would not allow uploading it.
Elastic Thread Pools
Anyone who has ever had to deal with the built-in .NET thread pool knows that it is an abomination. It scales poorly, it lacks configurability, and has no provisions for managing concurrent load. In short, it doesn’t cut it. This is not news and the situation is going to get a whole lot better in .NET 4.0 (see video on Channel 9 and blog post by Eric Eilebrecht). Unfortunately, that’s ways out and since Dream is cross-platform, we would still have to wait for Mono to follow suit.
Luckily, there was already a lot of published information about the improvements in .NET 4.0 and it turned out to be a fairly short exercise to create a custom implementation for Dream. Granted, this implementation may not be as sophisticated, but it’s available today (trunk) and it’s available under the Apache 2.0 open-source license. So you can borrow and improve upon it at will.
The ElasticThreadPool class in Dream is built using work-stealing queues to prevent threads from starving while others drown in work. An ElasticThreadPool instance can be configured to have a minimum number of reserved threads and a maximum number of parallel threads. This makes it easier to write applications that need to control the parallel fan-out when generating lots of work-items. Furthermore, all instances draw their threads from a common pool. Meaning, threads can be migrate between thread pools without incurring creation-destruction overhead. Lastly, most of the logic in the ElasticThreadPool is lock-free. Adding and removing threads from the thread pool is the only place where a lock is used. As a result, it scales up very nicely as the number of CPUs increase. Also, stealing work-items across threads will never block progress on another thread. A crucial property for a good thread pool. There is also an ElasticPriorityThreadPool that allows assigning priorities to work-items so the can get executed in a different order than they were scheduled.
Lock-Free Data Structures
Part of what makes the new thread pool in Dream so efficient is the use of lock-free data structures. Dream includes four classes in this category: LockFreeStack<T>, LockFreeQueue<T>, LockFreePriorityQueue<T>, and LockFreeWorkStealingDeque<T>.
I’m particularly happy with our choice of implementation of the work-stealing deque class. It’s entirely lock-free and has a great low-overhead memory profile. It’s based on a paper entitled “A dynamic-sized nonblocking work stealing deque” written by Danny Hendler, Yossi Lev, Mark Moir, and Nir Shavit.
We hope to add a lock-free hashtable and double-linked list by year end, but these require fairly complex algorithms and we don’t have use cases that necessitate them currently. No work has been on either of them apart from selecting relevant research papers. If solving these kinds of hairy problems appeals to you, let me know and I’ll share what we have so far.
Other Changes Ahead
In this post, I’ve covered two big topics that I’m really excited about, but that’s not all that’s coming down the pipes for Dream Profile 2. As always, we’re hard at work at improving the performance, lowering the memory requirements, and adding new features that have broad appeal. As part of this effort, we’re also cleaning up some of mistakes from the past by marking classes and methods as obsolete. Your applications and services will still compile and run until the official switch to Profile 2, but I recommend being proactive in abandoning the old patterns in favor of the new ones. All functionality that is being removed has alternative means to accomplish the same goal. Let us know if you need any help in the transition or if you recommendation on how to make it as painless a possible.
In an effort to encourage others to blog their support for MindTouch in the Sourceforge.net Community Choice Awards I’m providing this simple vote for MindTouch HTML snippet:
Please consider placing this in your blog sidebar, in a blog post, to your facebook wall or on your websites. Thanks!
I recently wrote an in-depth technical FAQ article on how to use jQuery AJAX to access the MindTouch API. The article was first posted on the forums for technical review and is now ready to be shared on the blog. I’m also happy to report that it has already inspired a few community members to experiment on their own and share their newly gained insights.
Over the coming months, we will continue to build new templates, mature existing ones, and also help community members like yourself create their own. My goal is to reach critical mass to launch an open app gallery where we can feature popular apps and where new ones can easily be shared. In the meantime, head over to the forums, join the fray, and participate in the rise of the MindTouch apps! ;)