Archive for the ‘DekiScript’ Category

23 May 11

Templating: Everything old is new again

I’ve hated mixing presentation with logic before it was cool and it’s always bothered me that for the last couple of years templating engines have steered primarily in the direction of native languages mixed 0with markup (erb, razor, ejs, etc.). However, #jsconf has given me some hope that separation of concerns is making a comeback.

<?php /* inject content here */ ?>

In the early days of the web, code was just emitting HTML as a side-effect of execution. The complete impossibility of managing the presentation layer with this approach spawned a hundred hand-rolled substitution/templating engines. I wrote at least 3 myself. PHP can be thought of as a direct descendant of these early templating engines, creating a syntax that natively intertwined code and markup.

<xsl:template match=”what-does-this-do”>

With the rise of XML, transformation of content into presentation took the form of XSLT. This was true separation of presentation and logic. However, XSLT, being a functional language, was unlike any other templating language  and authoring certainly was not designer friendly. The foreign syntax combined with requiring XHTML output and slow emergence of performant implementations kept XSLT from ever gaining mainstream acceptance.

$.template( “content”, “<h1>${Title}</h1>” );

Meanwhile XML and XHTML, at least partially driven by the configuration monsters being spawned in java enterprise land, lost popularity and developers turned towards simpler, closer to the metal templating approaches once again. Of these Ruby on Rails was clearly the resurgence leader. Using the MVC pattern, templating had much better business logic and presentation separation, but most templating engines did end up embedding Ruby in markup in some fashion or another. MVC frameworks for virtually all other runtime environments followed suit, eventually leading to templates with embedded javascript as the transformation engine on both client and server.

weld(document.querySelector(‘.content’), content)

This last development would seem rather counter to the otherwise popular unobtrusive javascript movement. Fortunately #jsconf showed at least two new templating engines with unobtrusive binding. While backbone.js and knockout.js (among others) are binding engines, they still litter the mark-up with enough javascript constructs to be  designer unfriendly. Enter weld.io and the (as yet unreleased) batman.js, two binding engines that modify plain-old HTML without any binding specific markup in the HTML:

<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
      <ul class='contacts'>
        <li class='contact'>
          <span class='name'>My Name</span>
          <p class='title'>Leet Developer</p>
        </li>
      </ul>
      <script src="lib/weld.js" type="text/javascript" charset="utf-8"></script>
      <script type="text/javascript" charset="utf-8">
var data = [{ name: 'hij1nx',  title : 'code slayer' },
            { name: 'tmpvar', title : 'code pimp' }];
weld(document.querySelector('.contact'), data);
      </script>
    </body>
</html>

wiki.template(“where do we go from here?”)

At MindTouch, we’ve had a more complex of story of separation of presentation and logic. At the application level, we’re a pure abstraction, where everything is available as application data via our REST APIs and is rendered into human/browser friendly HTML via a relatively thin PHP presentation layer. However, since our focus is on content editing and management, the edit content is fully formed HTML, which can include embedded DekiScript for HTML generation. At this time, DekiScript follows the common markup+code paradigm, as it is the simplest concept for non-programmers trying to add dynamic concepts.

So what does the future hold for templating at MindTouch? We have a considerable investment in our PHP presentation layer, but with the power of client-side javascript increasing steadily, the reliance on server-side transformation is likely to diminish with unobtrusive client-side templating techniques taking the lead. How this might influence the way we use DekiScript in page content is an unknown, but we’re exploring various approaches and would love to hear from the community how they would like DekiScript content generation, manipulation and templating to evolve.

08 Feb 11

DekiScript Page Profiler

speedy-gonzales-300x180

One of the exciting new features in the next major release (code-named Pipestone), is the DekiScript Page Profiler.  I’m confident this new feature will delight many DekiScript developers.

In the past, it has been a hassle to figure what part of a complicated, dynamically rendered page requires the most attention.  Often, the only method was to comment out sections one by one to determine the offending line or page inclusion.  This is now a thing of the past.  The page profiler provides a detailed view of every single DekiScript function called and every page included.  It’s your backstage pass to see exactly what’s going on!

The profiler output is split into 4 summaries:

  1. Rendered Content lists all the pages that were rendered and what functions were invoked on each page.  Function invocations are grouped together by location to show what each line is doing.  Additionally, each function includes an invocation count and total elapsed time.
  2. Function Summary lists all functions invoked across all pages with their total elapsed time, total invocation count, average time, and max time.
  3. Page Summary lists all pages with their total rendering time, inclusion count, average time, and max time.
  4. Data Access Summary lists how many database queries were issued and the performance of the optional cache.

Obtaining the page profiler report requires directly accessing the API, a task that shouldn’t be a hurdleto DekiScript developers.

Here is how you get your page profiler XML document:

  1. Identify the page ID of the page you want to profile.  The easiest is to look at the page source and look for the line that says:
    Deki.PageId = 123;
  2. The page meta-data for any page can be accessed by its ID:
    http://mysite/@api/deki/pages/123
  3. Now append /contents/explain and you’ll get the page profiler report:
    http://mysite/@api/deki/pages/123/contents/explain

Depending on how many operations your page performs, the returned XML can become fairly large, but that’s because it contains lots of details.  I would recommend looking at the function and page summaries first to get a sense for the potential culprits and use that information to track down the actual call that is causing the slowdown.

Finally, if your site your users report from sporadic slowdowns that you can’t reproduce, you can add this configuration key to have the profiler log its output when a page takes more than 3 seconds to render:

stats/slow-page-render-alert = 3.0

This will make it easier to capture those elusive moments where for some reason a page takes unusually long to render.

And remember when optimizing your pages: the fastest code is the one that never needs to run!

13 Jul 10

What’s New in DekiScript for Olympic

There is a lot of great new technology and capabilities in Olympic.  I and the rest of the team couldn’t be prouder of this release!  Beyond all the glitz and shiny newness that is getting all the attention is also an improved DekiScript engine with new capabilities.

Variable Scopes

For Olympic, we decided to make a change in how variables are scoped.  Pre-Olympic, variables have been scoped to their innermost defining context and therefore could shadow other variables without affecting them.  This is the scoping rule used by C#, Java, and many other programming languages.  JavaScript, on the other hand, scopes variables to the innermost function definition.  This design greatly simplifies the runtime in implementation and overhead.  After review, we decided to depart from our previous design and follow JavaScript’s lead.  Concretely, here is an example of what code will be affected by this change.

var x = 1;
if(condition) {
    var x = 2;
}
x;

Pre-Olympic, the above code will print 1, post-Olympic it prints 2 since the inner ‘x’ is not shadowing anymore.  The most positive impact of this change is that it is now possible to define variables at the beginning of a page and then refer to them anywhere in the body of the page following their definition.  This has been a much requested feature and I’m happy that it’s finally available!

Return Statement

Another part of the DekiScript runtime that has received significant attention is how output is accumulated and computed into a final value.  Not only is the new output buffer implementation much faster and cleaner, it also enabled the proper implementation of the semantics of the return statement, another much requested feature.

The return statement aborts the currently executing DekiScript block and produces an immediate value as output.  To some degree, the following three statements are identical.

1;
2;
return;
1;
return 2;
return (1; 2);

In all cases, the output will be 12.  In the first case, this output is produced by outputting 1 followed by 2.  In the second case, 1 is outputted and then 2 is appended by the return statement.  In the last case, nothing it outputted until the return statement appends the result of the compound expression.

However, there are cases where the output is quite different!  Consider the following three statements.

1;
<span>
    2;
    return;
</span>
1;
<span>
    return 2;
</span>
<span>
    return (1; 2;)
</span>

In the first case, only 1 is output.  More on that later.  In the second and third case, it will output 12 as well.  In all cases, the <span> element was ignored.  The reason for this is that the <span> element was not closed when the return statement was encountered and since partial XML is invalid, the entire XML block and its contents have been discarded.  That’s why in the first case, both the <span> and the inner 2 element have been omitted.  For the other two cases, the return statement ensures that the returned expression is appended to the output buffer before execution finishes for the current DekiScript block.  And hence, the expression is appended to the output that has successfully been accumulated so far.

You may wonder “why is this behavior useful?”  The answer is fairly simple: imagine you’re computing a table output, but you want to abort the entire table if an invalid value is found.  You have two choices.  Your first choice is to loop over each item in the table beforehand and validate it.  If validation fails, you show the error message. Alternatively, you have a way to abort outputting whatever you were currently computing and return an alternative result instead.  The latter case is exactly what the return statement enables you to do.  For example, the following code will do just that:

<table>
    foreach(var row in table) {
        <tr>
            foreach(var valuein row) {
                <td>
                    if(value is nil) {
                        return <strong> "A value was not set!" </strong>;
                    }
                    value;
                </td>
            }
        </tr>
    }
</table>

Try..Catch..Finally Statement

In addition to the return statement, we’ve also added the try..catch..finally statement.  It works pretty much as in any other language.  If an error occurs inside the try block, the catch block will be evaluated, otherwise the catch block is ignored.  Regardless, the finally block is always evaluated.

Since DekiScript is an expression-oriented language, we also added the !! operator to catch errors during expression evaluation.  Therefore, the following two statements are identical.

var x;
try {
    let x = f(123);
} catch {
    let x = 456;
}
var x = f(123) !! 456;

Triple-Quoted Strings

Ever found yourself in a pickle because you needed to create a string that contained both single- and double-quotes?  Up until now, you had to escape the offending character or experience parser errors.

For Olympic, we’ve added support for triple-quoted strings.  You can either use a single- or double-quote to create a triple-quoted string.  Unless you need the same triple-quotes again inside the string, you won’t need to worry about escaping anything.  Thus, the following strings are identical.

"The ‘quick’ brown fox, jumped over the \"lazy\" dog."
"""The ‘quick’ brown fox, jumped over the "lazy" dog."""

Triple-quoted strings can be used anywhere were regular strings are allowed, including for XML attributes.

Native JEM & CSS Blocks

Another improvement is the support for JEM and CSS blocks in the editor.  Just select either kind from the ‘Insert’ menu in the editor toolbar.  There is no need to encode these anymore in DekiScript blocks.  However, as before, you’ll need UNSAFECONTENT permission for JEM or CSS to be included in the page.

Conclusion

Olympic includes the best DekiScript engine we have ever shipped.  Not only does it add some great new features to the built-in scripting language that makes MindTouch so appealing, but behind the scenes, it’s new architecture is also the foundation for continued improvements in capabilities and performance.

I’m looking forward to seeing how you and the rest of the growing MindTouch community will take advantage of it!

18 Sep 09

Using tags to make powerful apps

Over the course of the last eight months or so I’ve been working quite diligently on the DekiScript “modules” that power WhoRunsGov.com.  I have learned quite a lot about implementing largly DekiScript powered projects and I am going to share some examples of how I used tags to develop very efficient reusable DekiScript templates.

The Use Case

WhoRunsGov.com is a user generated site focused on political profiles from all facets of the government.  With well over 700 profiles they are rapidly growing their content base and have successfully utilized DekiScript to offer a fluid and intuitive form of navigation.  Their solution includes a large number of “browse by” pages that display a list political profiles based on specified tags.   To make this more challenging there are about 150 different browse by pages so  I needed to come up with a solution that was both flexible and manageable.

The Solution

To solve this issue I created one centralized DekiScript template called ProfileList.  The ProfileList template is quite complex and is responsible for:

  • selecting the profiles
  • retrieving the profile content
  • identifying the most recently uploaded profile image
  • resizing it to fit the profile

In addition to these functions I also gave the template a number of different parameters.

  • $searchquery
  • $sortby
  • $limit

Template DekiScript

By adding these parameters I was able to make this template flexible enough to be called from all 150 browse by pages.  You can see that the code below uses the parameters to build a uri for the search API.   Using the search API is the fastest and most resource friendly approach to retrieving a set of pages.  Also, because lucene is so powerful you can filter the results with extreme control.

<h1>Template:ProfileList</h1>
{{
var qsortby = $sortby ?? '-tag:id-*';
var qlimit = $limit ?? 100; //

var results = web.xml(uri.build(site.api, [ 'site', 'search' ], { limit: qlimit, sortby: qsortby, format: 'search', q: $searchquery }), _, _, 900);
}}

‘Browse By’ DekiScript

When calling this template I simply passed in the $searchquery and the page displayed the appropriate profiles.

{{
profilelist{searchquery:'tag:"Governor" AND tag:id-*'}
}}

As you can see this searchquery is identifying pages that are tagged with ‘Governor’ and pages that are tagged with a tag that starts with ‘id-’.    Here are some more examples of how I used the ProfileList template for more advanced searches.

{{
profilelist{searchquery:'(tag:"administration official" OR tag:"obama administration official") AND tag:[id-a* TO id-i*]'}
}}

This query returns all pages tagged with ‘administration official’  or ‘obama administration official’ that are between A and I.

{{
profilelist{searchquery:'tag:"House member" AND tag:democrat* AND tag:congress AND tag:id-*'}
}}

This query returns all profiles that are tagged with ‘House member’, ‘congress’ and that also have a tag that starts with ‘democrat’.

Thanks for reading,

Damien Howley
@DamienH

02 Sep 09

How Did You Do That?? (Or, Introducing the DekiAPI JS Plugin)

Being able to interact with the MindTouch API without forcing users to use the WYSIWYG editor is something that comes up frequently. It’s the Holy Grail of MindTouch developers everywhere who want to make their applications built in MindTouch easy to use and professional in appearance. Questions like:

  • “How can I write to a page property without having to wade through page submenus?”
  • “How did that gauge update without reloading the whole page?”
  • “How the heck did you DO that?!?” (my perennial favorite)

all come up in the forums and in the MindTouch IRC channel. Previously we’ve seen some skillful jQuery ajax work from our frontend wizard Damien H to find ways to talk to the API. It’s a little granular and requires some knowledge as to what’s needed in order to direct the call, but it’s certainly the most customizable.

But what if you didn’t need to do things like check the xhr.status, and instead just wanted to do something like update a page property through the API?

Aspiring Dekiscripters, meet DekiAPI().

MindTouch has developed a very quick and easy way to accomplish many of the calls that coders have been asking how to do (or have been begging for an easier way to do!). The new javascript API extension allows Dekiscripters to do a number of API-related functions simply, including (but not limited to):

  • Read/update page contents
  • Create/read/update/delete page properties
  • Post messages to the API (which is used in the SendMessage template, discussed at the Developer site)
  • Reload individual DOM elements
  • Make sense of the health-care debate in America

Well, ok, not the last one. Not yet, anyways.

Previously if you wanted to say, update an existing page property, you’d have to slog about 50-60 lines of code and ajax calls to do it safely (if you wanted to do it unsafely, you could probably do it in about 10). Using the DekiAPI() functionality, it can be done in about 4 lines, 6 if you want pretty error messages should your update fail for some reason.

Let’s check out one of the more visually-appealing components of the DekiAPI script: the MindTouch.Deki.Reload() function. You can see the Reload in action here (part of the Down With Stagnant Corporate Intranets webinar):

(high-quality version can be seen here)

When a task is added or completed, the Task List automatically reloads, and the Progress Meter gauge is updated. This element-specific reload provides a much more fluid and seamless user experience, and projects a “polished” image for your application. Although there’s obviously a lot more going on behind the scenes as far as the task management and progress calculation in this example, the actual reload itself is quite simple.

No, really, how did you do that??

Step 1: Install and call DekiAPI().

Swing by the Deki AJAX API extension page to learn how to install it (you must have administrative privileges to access the control panel). After it’s successfully been installed and started, make sure you call it at the top of the page you want to reload by entering

{{ dekiapi() }}

or, if you’re using the Dekiscript <pre> formatting (which I highly recommend!),

dekiapi();

Bear in mind that you will need UNSAFECONTENT enabled for your user, group, or site to create pages that use this. If you’re creating a template, the scripting can still be maintained and used by users without UNSAFECONTENT so long as the template is being invoked (as opposed to inserted into) the page.

Step 2: Know what you want to reload.

This may seem obvious, but it’s about as fundamentally important as you can get. For example, take the following block:

<div id="outsideDiv">
    feed.list("foo");
    <div id="insideDiv">
        feed.list("bar");
    </div>
</div>

If you reload outsideDiv, both the foo and the bar RSS feeds will reload. Clearly that’s the desired result in this simple example, but  that can have some visually unappealing effects when you are reloading a div that contains, say, a table. If you have data within a table that you want to refresh, you’ll want to reload specific rows in that table, instead of forcing the entire table to redraw. It looks a lot more polished and less jarring to watch a single row/div/span/etc. reload instead of seeing the whole table disappear and reappear (causing the other elements around it to be affected as well).

Step 3: RELOAD!

Chances are, some event will be triggering an element reload. Whether it be an ONCLICK event, a JEM call from within a CTOR attribute, or a Javascript timer trigger, the invocation looks like this:

Trio Reloading

Photo by Photo Plod (flickr)

MindTouch.Deki.Reload(DOM_id, params, callback);

where DOM_id is the ID of the element you want to reload, params are any additional parameters you wish to include (this is generally null,the script inserts a dummy parameter to prevent certain browsers from presenting a cached version of the reloaded element), and callback is any additional function or activity you want to execute on complete. The last two arguments are optional; I often leave them out when using this particular function.

To reload the insideDiv from the example block above, let’s pretend that we’ve published a message on the “reload_Element” channel saying to reload. The complete script (assuming the use of the Dekiscript <pre> formatting) would be:

<script type="text/jem">"
    when(@reload_Element){
        MindTouch.Deki.Reload(#insideDiv);
    }
"</script>

Complex, huh? Seriously, that’s all there is to it.

For those not used to working inside the Dekiscript <pre> formatted blocks, those additional double-quotes are necessary, whereas they are not in plain source mode. This applies to <style type=”text/css”> blocks as well, and gave me some grief in the beginning until I got it figured out.

To see Reload in action, I built a simple example here: Reload Example

Additional Considerations

There are a few other things to consider when using the Reload function:

  • CTORs (JEM connectors inside of elements) lose functionality when reloaded. It’s an important thing to be aware of.  There are some ways around it, but that’s a little outside the scope of this post.
  • Images that have to be fetched will cause the elements around them to shift while the image is being retrieved. You can see that happening in the example above.
  • Reloading is still a page load. Or at least, a GET. If you’re reloading something big and complex every second, it can end up being expensive.

The Bottom Line

Simply put, the Reload function (as well as the other functions contained in the MindTouch.Deki.js file) is an easy way to make your MindTouch-developed applications look and behave in a polished and professional manner. Used in the right way, it is a fast and powerful tool that helps to bring your web app to life.

Copyright © 2011 MindTouch, Inc. Powered by