There has been a heated argument recently on the W3C Canvas API mailing list between accessibility advocates and browser vendors over a pretty tricky topic: should the Canvas API have graphical “objects” to make it more accessible, or should authors use SVG for that? I think it’s a false dichotomy, and I offer a proposal to suggests a way to improve the accessibility potential of the Canvas 2D API by defining how SVG and the Canvas 2D API can be used together.

This brings together some ideas I’ve had for a while, but with some new aspects. This is still a rough overview, but the technical details don’t seem too daunting to me.

Retained-Mode vs Immediate-Mode Graphics

First, some background…

SVG is a retained-mode graphics format, meaning that each shape persists as a separate object, with distinct characteristics such as location, size (bounding box), and precise outline (stroke). These characteristics can be acted upon within the object structure (DOM) as discrete entities (or groups of entities) for manipulation by script or by user-interface actions (e.g. mouse clicks or keyboard tabbing); one of these interactions may be hit-testing.

The Canvas 2D API is an immediate-mode API, meaning that it allows the programmer to create visual shapes as individual pixels that are unstructured; in other words, if a red circle and a red square are drawn which overlap one another, there is no way to distinguish these from one another, or from the background canvas element (though a programmer could maintain a separate structure in script).

With retained-mode graphics, each object which is added to the DOM increases the memory “weight” of the page or application, potentially slowing down the browser or otherwise decreasing performance; by contrast, immediate-mode graphics, which are not represented as objects in the DOM, do not decrease performance, which is an advantage over retained-mode graphics.

Some accessibility advocates have noted that hit-testing and zooming are critical accessibility features of graphics; this requires the ability to access the location, size, and other characteristics of individual graphics. Unfortunately, adding this retained-mode feature to an immediate-mode API effectively removes the primary advantage of using an immediate-mode graphics system.

However, by defining how SVG and the Canvas 2D API can be used together, a model may be created which allows authors to easily use both together, each for their appropriate strengths. This approach involves two aspects: defining the behavior, and a potential shared API syntax.

Defining Behavior

Currently, the Canvas 2D API is only defined to work with the <canvas> element defined in HTML5, and it does not allow SVG to be drawn or imported into the Canvas. However, it could be extended to work inside SVG or to allow SVG to be used inside Canvas.

Canvas in SVG

At its simplest, the Canvas 2D API could simply be defined to allow authors to draw on the SVG raster <image> element, just as it does in the HTML <canvas> element, but there does not seem to be a reason to limit it to this element. The Canvas API could be defined to draw to any SVG element, rendering or structural, as an overlay. It would be merely a visual effect, and would participate in the rendering structure, including “z-index”, only as its “target element” does.

SVG in Canvas

SVG shapes could be allowed to be imported into the <canvas> drawing context, such that they render there but are also inserted into the DOM or shadow DOM. Currently, as I understand it, this is not allowed for security restrictions involving content from potentially different sources, but for content that does not have external references (for example, a <use> element to another site, this should not be a security risk.

SVG in the Shadow Tree

The accessibility features of the Canvas 2D API include a “shadow tree” and focus ring for those elements, such as text, which are not otherwise accessible to users with limited vision. Each such “graphic piece” which is created and painted to the screen is also represented in this underlying shadow DOM (IIRC).

SVG elements used in conjunction with Canvas could also be placed in this shadow tree, so that navigability is preserved. In order to save memory usage, a concept like the <use> element in SVG could alternative be used to “point to” an SVG in the main DOM from its appropriate place in the shadow tree.

SVG elements could either be used as visual shapes themselves, or could be invisible “hot-spots” for specific areas of the screen.

Common API

I have suggested before that we create a simple graphics API that will allow either 2D Canvas (immediate-mode) or SVG (retained-mode) images with the same set of methods and parameters, with the only difference being which “mode” the author selects to have as the instantiated form. If the author draws a circle to the canvas context, it would simply be drawn to the screen with no structure; if the author draws that same circle to the SVG context, it creates a <circle> element with the appropriate attribute values and style and inserts it into the DOM.

There would be certain differences in some of the renderings, because of the different rendering and compositing models of SVG and Canvas, and there would be certain specialized functionality and features each would have, but the majority of the visual appearance would be the same.

This would allow authors to learn only one API for 2D graphics for the Open Web Platform. With this ease of learning and use, the author could decide on a case-by-case basis, even within the same application, which mode works best. It would provide incentive to authors to use each to its best effect.

I call this approach COG: Common Open Graphics.

Example

An example of the effective use of SVG and Canvas together is a hypothetical game in which the user pilots a spaceship amidst an asteroid field.

The background of the game includes animated nebulae, passing comets, rotating planets and moons, and other visual effects which are not interactive, and which do not affect the play of the game, but which add visual appeal. The foreground of the game (the “playable area”) includes asteroids, enemy ships, and energy beams that the player must dodge or shoot at, which increase in size the closer they get, and which benefit from hit-testing and keyboard focusability. The middle-ground of the game contains many smaller asteroids and ships that are not yet interactive, and so don’t need hit-testing or focusability.

Adding all of these graphics shapes as retained-mode objects in the DOM would unnecessarily increase the memory footprint of the game, degrading performance and playability. However, since only a small number of these elements (those in the playable area) need the retained-mode interactivity, the majority of them could be drawn with immediate-mode characteristics, then when a certain “distance threshold” is reached, changed to interactive SVG elements, perhaps using the same shared API. This preserves the accessibility of the game, while also causing minimal performance degradation, and possibly even helping performance, since the hit-testing doesn’t need to be calculated in script. From an authoring point of view, using the same shared API means that both types of shape, retained-mode and immediate-mode, are equally easy to generate and maintain.

To enhance the visual appearance of the SVG asteroids, the finer control of the Canvas API could even draw an overlay on the SVG stroke and fill areas, much as filters do, to show craters and other surface details in a more fine-grained manner then can be done efficiently in SVG. Since this would be drawn specifically on the SVG element, the element itself would still provide the geometry and position characteristics.

Caveats

This proposal has two caveats:

1. Canvas can only be extended to be as accessible as SVG is, in this model. Even with object retention, there are still many enhancements needed to make SVG truly widely accessible. However, as plans for SVG accessibility improvements are made, these would be largely importable directly into the Canvas API, rather than working in parallel on two unrelated but overlapping graphics models.
2. Simply because the Canvas 2D API may be made more accessible, does not mean that all Canvas API applications or content will be automatically accessible, any more than all HTML or SVG content is accessible. Authors will still need to use best practices and take specific steps to make their applications accessible. This proposal only provides the means and the incentive to do so.

Feedback

I’m interested in feedback on this, either in the comments section of this blog, or on the W3C Canvas mailing list, public-canvas-api@w3.org.

---

Update

Someone pointed out offlist that this post emphasizes the accessibility aspect of this proposal, because that was the context of the most recent discussion, but this is a win for anyone who wants to do graphics in the browser, for any reason, not just accessibility. Like any good accessibility feature, my proposal has general benefits, so it is more likely to be used in more content. In the example scenario above, the developer gets the hit testing for free, regardless of whether or not Accessibility Technology is involved. Other general advantages include ease of learning and use of graphics by developers (as mentioned above), sharing of features between modes (like using Canvas.toDataURL() to save an SVG as a PNG), pixel operations (like painting or detecting a color at a point) in SVG, metadata (like microdata, RDFa, or microformats) in select parts of a Canvas app, and so on. I’m sure that clever people will come up with a whole boatload of cool things to do with a combined retained-mode/immediate mode API that I couldn’t imagine here.