Chandler 0.6 ItemCollection Specification, v0.2

Authors: Ted Leung Last edited: June 13, 2005 10:24 AM Creation date: April 22, 2005
Reviewers:

Overview

Goals and Objectives

Ted Leung
Spec owner and contributor

Background

Item Collections are one of the key organizing mechanisms of the Chandler User Interface. They present the end user with a list of items. The items in this list are controlled by the user in several ways:

  1. The user may specify a rule or set of rules that controls which items are in the Item Collection. These rules may cause Items to be added to the Item Collection by background processes such as e-mail or RSS readers
  2. The user may manually remove items from an Item Collection via drag and drop or a keystroke command
  3. The user may manually add items to an Item Collection via drag and drop (or perhaps a key stroke command
  4. The user has control over the order in which the items appear in the item collection
  5. Item Collections are not limited to Items of a single kind, although an Item Collection may be filtered by Kind before it is presented to the user
  6. A single Item may be in more than one Item Collection

Item Collections appear in the Chandler User Interface in several places, including the sidebar and the summary view.

In Chandler, the sidebar displays a list of Item Collections. The user can select (by clicking) an Item Collection in the sidebar.

Once an Item Collection has been selected, the items in it will be displayed in the Summary View. The buttons in the Application Bar also control which items from the Item Collection are displayed in the Summary View. These buttons filter the Item Collection so that only items of a particular kind (associated with an "Application") are displayed in the Summary table.

Column Headings in the Summary table are annotated with the actual attribute redirected to by displaying that attribute in parentheses.

See the Sidebar Specification for user interface details of Item Collection.

Code Design

Requirements

API Requirements

Overview

Interfaces to parts of the Chandler User Interface

Item Collections contain data that needs to be communicated to three major portions of the Chandler User interface: the sidebar, the summary view, and the detail view.

Let's look at the Sidebar first:

Lets look at the Summary view

Now lets look at the Detail View:

The Detail View displays a single item, the item currently selected in the summary view. Ideally the detail view should not need to know what item collection is being displayed. The requirements for the Detail view are:

The requirement for attribute level notification is difficult to satisfy since it means checking every attribute access to every item. Furthermore, this functionality is needed inside the attribute editor framework, which encapsulates this behavior from the end user frameworks. Even enabling monitors on every item of a particular Kind looks like it would be expensive. Because the subscription is *so* targetted, I think it makes sense to use the Monitor mechanism to handle the detail view notifications.

Interface to Sharing

The sharing subsystem needs the following features in order to work with ItemCollections:

Also, the sharing system needs to keep track of which items are shared (and which collection they are shared from -- which means we can't use a scalar attribute on Items to detect whether or not they are shared). The current design calls for the sharing system to keep track of which items are shared by which collection (via the Share Item). Developers that want to know whether a particular Item is shared or not will have to use an API [Ted] (to be attached to Item?) to determine whether or not that item has been shared.

Interface to Background Tasks

There are two cases for background tasks:

The first case is that a background task creates new Items that should appear in a particular Item Collection. By updating the repository, Items which match the Item Collection's rule will be incorporated into the Item Collection (this means that when the background task commits, we must notify any queries/sets observing that kind that they must generate notifications -- hmm).

The second case is that the background task wants to update the UI thread on it's progress so that the UI thread can update a dialog box. This communication would presumably take place through an item -- Is this a monitor use case or an Item Collection Use case.

Notification

In order to make Item Collections work properly, we need a good mechanism for notifications. Notifications are used to tell an Item Collection when relevant items have changed, and are used by Item Collections to tell UI (and other) components that they need to (re)draw updated Item Collections.

Architectural Overview

Item Collection is built out of sets, queries/filters, and notification. Abstractly, the contents of an Item Collection is a set of items which is obtained by perfoming various set operations (filter/query, union, intersection, difference) on some number of base sets. These sets are "connected" via notification so that changing the contents of a set causes the contents of all derived sets to be updated as well.

The Item Collection abstraction itself is an ordered collection. This means that the Item Collection provides a thin layer on top of the set of items which preserves ordering for the user interface.

Usage Model

The usage model for ItemCollections looks like this:

You create an instance of ItemCollection. When you create this instance, you may specify a rule which determines the set of items that belong in the ItemCollection. If you specify a rule, then any Items which are created in the repository and which match the conditions of the rule, will be included in the ItemCollection. You do not need to do anything else in order to have your items included in the Item Collection.

A common idiom for writing the rule is to include items whose attribute(s) match a set of conditions. A special case of this idiom is to use an attribute which is a bi-directional reference.[Ted](possible implementation using birefs)

You may also explicitly include items which do not match the rule or exclude items which do match the rule by calling methods on the ItemCollection instance. You can have an ItemCollection that has no rule and whose contents are all explicitly described (via inclusion), however you should be aware that future versions of the Chandler UI may display explictly included or excluded members of an ItemCollection differently from members of an ItemCollection specified by a rule.

Once you have created the ItemCollection, you can then have other components subscribe to its notifications, so that those components can track changes to the ItemCollection. When Items are changed in a way that they no longer satisfy the ItemCollection rules or start satisfying the ItemCollection rules, subscribed components will be notified of these changes so that they can react appropriately.

Overview of ItemCollection implementation

Item Collections are implemented using Sets. An Item Collection contains three sets, which are combined to yield the contents of the item collection. The inclusions set contains those items which have been explicitly added to the ItemCollection, the exclusions set contains those items which have been explicitly removed from the ItemCollection, and the ruleSet contains those items which satisfy the rule associated with the collection.

The precise combination of these sets is difference(union(inclusions, ruleSet), exclusions)

The ItemCollection may also be filtered by kind in which case the result set would be filterbykind(kindlist, difference(union(inclusions,ruleSet)))

Order will be maintained by using an index on the items in the contents.

John proposes that we get rid of the ItemCollection wrapper abstraction. See the sample code section below.

Implementation of Sets

The important notion of sets is that sets of items fire events when items are added to or removed from the set. Reference collections have most of the properties that we need for sets, so we can either layer the needed notification code on top of reference collections or alter the reference collection implementation to perform notifications when needed. [Ted] to be resolved with Andi. Since Python already has a built in notion of Sets, we may end up calling this abstraction Collection just to avoid confusion. The alternative is to try and implement the entire contract for Python Sets, which is probably unnecessary.

[Ted] How does this integrate with PJE's Python schema API work.

We also need sets that represent the extensions of the various Kinds. That is, we need a way to work with "The set of all calendar events". These sets would correspond to the results of KindQueries in Chandler 0.5. The extents are necessary because most ItemCollections are composed over all Items of a particular kind. Andi has proposed that we not have extents and instead create sets to represent the "basis" set of an ItemCollection and then add items to the basis set as necessary. The problem with this is that the Chandler virtuality model will make it difficult to determine which basis sets an new Item should be placed into, although if the number of such basis sets is small, I suppose you could just check all basis sets every time that an Item is created, except that the check for membership in the basis set is going to be some condition which will need to be specified. The more I think about this approach, the less I like it because it seems non-extensible to me. Requiring item creation sites to know about all the bases that need to be updated is going to be problematic when the user and third party parcels can create bases as well. If we continue with an extent based approach, then we will need to have the extents, along with short names (see the addressing spec) for them.

What about Jeffrey's CalenderEventOccurrenceMixins? Can we even build an extent for them? -- They don't really exist? At one point I write "Suppose we allow addresses/names of Sets anywhere we allow a set, then we can put the name and handle that." but I have no idea what I meant.

Implementation of Queries

In Chandler 0.5, the set operations union, intersect, and difference are part of the Query package, along with functionality for filtering/selecting items. Since we are introducing a Set abstraction for Chandler 0.6, these operators ought to be associated with the Set abstraction and not left in the Query package. The Filter operation will move to Set as well. All of these operations will be modified to operate on Sets and produce Sets. The existing ItemCollection implementation manipulates the string form of queries. I plan to expose an API so that string manipulation is unnecessary in a lot of cases. Since we specify rules for ItemCollection as strings in parcel.xml, we will also have the existing string based method of constructing filters, although I plan to do some work to make the API more palatable

The sets/query operators must also propagate notifications. All sets can be instantiated with a flag that indicates that result set should be subscribed to events from the input set(s). The listener/observers for those events will be provided by the operators.

These are the notification handling rules for the various operators:

Implementation Notes: the filter operation will need to be able to detect that a date range query is being requested so that we can do the appropriate actions for virtual recurrence events. Handling these events may require some hooks to allow type specific or set/collection specific behaviors during filter processing. Also, the Chandler 0.5 implementation uses lazy evaluation to generate the initial query results. Since we are going to have extents or basis sets, we no longer need this functionality (except perhaps as part of the implementation of extents.

Implementation of Notification

A notification system is used to communicate changes in one part of the system to another part of the system. In the ItemCollection use cases, Notification is used to connect domain/content model Items with Sets and ItemCollections and to connect ItemCollections and Sets with CPIA blocks.

In Chandler, repository views provide a context where operations take place. Every operation on an Item occurs relative to a particular repository view. If we want to notify other components of changes to an Item or ItemCollection, it will be within the context of a repository view. So the changes to be communicated take place in a particular repository view, and will be communicated to other components in that repository view. A repository view also become aware of changes made in other repository views when the view calls the commit() or refresh() methods. Once this happens, it appears to the repository view that all of these changes happened in the original repository view. So communication/notification between repository views / threads takes place by changes to items in the repository. [Ted] for cases where two views wish to communicate more "directly", they can do so via an agreed upon Item (or set of Items). Locating this well known item then becomes a problem. We plan to handle this via the addressing specification which will deal with the issue of well known names for Items in the system.

You can think of the notification system working like this. Each repository view has a queue associated with it. When a component subscribes to an Item, Set, or ItemCollection, the observed component pushes its changes into the queue. At the application's convenience (which in 0.6 means from the Idle loop), it retrieves all the unprocessed changes from the queue and distributes them to their destinations. This allows the system to notify of changes at one rate, and process them at a different rate.

The kinds of changes that will be placed on the notification queue include Set add/remove operations, Item change operations (as recorded by the repository). Part of the notification system includes a commit time hook for each repository view which will record any changes to attribute values (if there is a subscription for item value changes)

Subscriptions will be managed in the same way that we manage them today, which means that the subscription lists are not persistent (since they refer to methods on Item). However, it should be possible for us to automatically hook up the notifications when building composite sets (sets constructed from other sets via union, intersect, difference, and filter). This will reduce the burden of subscription management substantially.

There may be opportunities to suppress duplicate events or perform other view specific processing (like 1/60th sec refresh limiting) of the notification queue. The notification system will allow registration of a queue filter function or equivalent so that the view can provide this behavior.

A future use of the per view notification queue is to serve as a source of input for an undo/redo log. This is not explicitly a goal for the notification system for 0.6 but PJE and I have discussed the possibility of this in the future.

Note that this proposal does not satisfy one of the primary requirements for the Detail View: synchronous notification. The needs of the detail view are different enough, that I think the best solution for the Detail View is to embed the use of Monitors in the Attribute Editor framework, perhaps in combination with the value model design pattern.

Implementation of "Mine/Not Mine"

Mine and Not Mine will be implemented as attributes on ItemCollections themselves. These attributes will be used to compose queries that display the correct sets of items.

Let A be "All CalendarEvents", let N1, N2 be a pair of Not Mine ItemCollections of CalendarEvents. Then the displayed ItemCollection "All My Events" = union(A, union(intersect(A, N1),intersect(A,N2)))

This should only be expensive when the query is initially populated. After that all updates should happen incrementally.

Implementation of "Shared/Not Shared"

Since an item can be shared in one collection and not shared in another collection, we cannot use a boolean attribute on the Item to indicate whether or not an Item is shared. There area number of possiblities, including a set-valued attribute containing the names(?) of the collections which are sharing the Items. Due to the design of the Sharing system, we are probably going to let the Sharing subsystem's Share abstraction handle "Shared/Not Shared". This means that clients interested in operating on shared items will need to consult the relevant Shares. We may be able to add some API to Item in order to make this determination less painful.

Set Public API

Set / ref collection methods, plus:

Query Public API - API version

[Ted]John was interested in a more data oriented API, so it looks like we will introduce the query operators as attributes of the Set kind. The class hierarchy in this section is no longer relevant and will be removed in the next version of this spec

We introduce one new class for each set/query operation.

ItemCollection Public API

See the sample code section for several possible designs for an API that includes or does not include an ItemCollection abstraction.

Potential additions:

  1. explicit control over inclusion/exclusion versus contents
  2. batch addition/removal
  3. quenching notifications temporarily

Notification Public API

The notification system has to store up generated notifications and deliver them when the application asks for delivery. The items in the repository are responsible for knowing the configuration of the notification "network". This configuration is stored as persistent values so that the network is restored along with the rest of the application state.

Sample code

Set/Query code

I have three different ways of creating an item collection from sets. I am ssuming that we have Extents, as indicated by repository.Set.Extent. There are at least three options. John wanted a more "data driven" API, rather than having a lot of methods to call, he wanted to have attributes to set, including the set operation (union, intersect, etc). This means that we cannot use subclasses to implement the various kinds of sets (if you view each kind of set as the result of one of the set operations). 

extent = repository.Set.NoteExtent

ruleSet = repository.Set(notify=True)
ruleSet.inputs = [ extent ]
ruleSet.operator = "filter:rulestring"

inclusionsSet = repository.Set(notify=True)

inclusionsRule = repository.Set(notify=True)
inclusionsRule.inputs = [ inclusionsSet, ruleSet]
inclusionsRule.operator = "union"

exclusionsSet = repository.Set(notify=True)
ic = repository.Set()
ic.inputs = [ inclusionsRule, exclusionsSet ]
ic.operation = "difference"

In constructor only form, that would look like this: 

extent = repository.Set.NoteExtent

ruleSet = repository.Set(notify=True, inputs=extent, operator="filter:rulestring")

inclusionsSet = repository.Set(notify=True)

inclusionsRule = repository.Set.(notify=True, inputs=[inclusionsSet, ruleSet], operator="union")

exclusionsSet = repository.Set(notify=True)
ic = repository.Set(notify=True, inputs=[ inclusionsRule, exclusionsSet ], operator="difference")

If we use subclassing, then the constructor form looks like this, which is a little bit cleaner, but has the disadvantage that we cannot change the operation associated with a set, without changing the Set Item instance (which we are trying not to do). This also means that we can use polymorphism rather than if/else blocks to determine the appropriate behavior for Sets based on a particular operator (I'm not sure how much we'll want/need to do this) 

extent = repository.Set.NoteExtent

ruleSet = repository.Set.Filter(notify=True, input=extent, rule="rulestring")

inclusionsSet = repository.Set(notify=True)

inclusionsRule = repository.Set.Union(notify=True, inputs=[inclusionsSet, ruleSet])

exclusionsSet = repository.Set(notify=True)
ic = repository.Set.Difference(notify=True, inputs=[ inclusionsRule, exclusionsSet ])

The last option shows how an ItemCollection wrapper abstraction might make the code a bit shorter. It also means that there are nice accessors/attribute names for the various sub pieces of the item collection. If all you are handed is the final composite set (ic in the examples above), then some amount of digging will be required to find the sets used for inclusion or exclusion. If that digging is repetitive it will get tossed into utility functions, and at that point you might as well just have the wrapper. Also if you don't have ItemCollection as a separate Kind, how will you findthem all? You don't have an extent, and you can't query by kind -- you can ask for all Sets, but that doesn't tell you which ones are ItemCollections. 

inclusionsSet = repository.Set(notify=True)

exclusionsSet = repository.Set(notify=True)

ic = ItemCollection(inclusions=inclusionsSet,rule="ruleString", exclusions=exclusionsSet)

Notification Code

There isn't that much notification code that should appear to clients. Calls to deliverNotifications() will be embedded in the OnIdle handler, so that will be in the framework code.

In the Item Collection case, the composite sets should be able to take care of wiring up their notification, so that client code should not have to call subscribe when compositing sets. Of course clients of the composite set (or its constituents) will still have to call subscribe or unsubscribe.

Simplifications

Never implemented

Special Considerations

QA / Test

We need a good set of headless unit tests for ItemCollection

API / Developer Platform

See Usage section above

Security

Internationalization / Localization

Accessibility

Build / Install


History

Author Edit date Description
Ted Leung April 22, 2005 First Draft