Customising Roundup

The dbinit.py module

The dbinit.py module contains two functions:

open

This function defines what your tracker looks like on the inside, the schema of the tracker. It defines the Classes and properties on each class. It also defines the security for those Classes. The next few sections describe how schemas work and what you can do with them.

init

This function is responsible for setting up the initial state of your tracker. It's called exactly once - but the roundup-admin initialise command. See the start of the section on database content for more info about how this works.

The "classic" schema

The "classic" schema looks like this (see below for the meaning of 'setkey'):

pri = Class(db, "priority", name=String(), order=String())
pri.setkey("name")

stat = Class(db, "status", name=String(), order=String())
stat.setkey("name")

keyword = Class(db, "keyword", name=String())
keyword.setkey("name")

user = Class(db, "user", username=String(), organisation=String(),
    password=String(), address=String(), realname=String(),
    phone=String())
user.setkey("username")

msg = FileClass(db, "msg", author=Link("user"), summary=String(),
    date=Date(), recipients=Multilink("user"),
    files=Multilink("file"))

file = FileClass(db, "file", name=String(), type=String())

issue = IssueClass(db, "issue", topic=Multilink("keyword"),
    status=Link("status"), assignedto=Link("user"),
    priority=Link("priority"))
issue.setkey('title')

What you can't do to the schema

You must never:

Remove the users class

This class is the only required class in Roundup. Similarly, its username, password and address properties must never be removed.

Change the type of a property

Property types must never be changed - the database simply doesn't take this kind of action into account. You can't just remove a property and re-add it as a new type either. If you wanted to make the assignedto property a Multilink, you'd need to create a new property assignedto_list and remove the old assignedto property.

What you can do to the schema

Your schema may be changed at any time before or after the tracker has been initialised (or used). You may:

Add new properties to classes, or add whole new classes

This is painless and easy to do - there are generally no repurcussions from adding new information to a tracker's schema.

Remove properties

Removing properties is a little more tricky - you need to make sure that the property is no longer used in the web interface or by the detectors.

Classes and Properties - creating a new information store

In the tracker above, we've defined 7 classes of information:

priority

Defines the possible levels of urgency for issues.

status

Defines the possible states of processing the issue may be in.

keyword

Initially empty, will hold keywords useful for searching issues.

user

Initially holding the "admin" user, will eventually have an entry for all users using roundup.

msg

Initially empty, will hold all e-mail messages sent to or generated by roundup.

file

Initially empty, will hold all files attached to issues.

issue

Initially empty, this is where the issue information is stored.

We define the "priority" and "status" classes to allow two things: reduction in the amount of information stored on the issue and more powerful, accurate searching of issues by priority and status. By only requiring a link on the issue (which is stored as a single number) we reduce the chance that someone mis-types a priority or status - or simply makes a new one up.

roundup-admin set issue23 assignedto=2

roundup-admin set issue23 assignedto=richard

Examples of adding to your schema

TODO

Detector API

Auditors are called with the arguments:

audit(db, cl, itemid, newdata)

where db is the database, cl is an instance of Class or IssueClass within the database, and newdata is a dictionary mapping property names to values.

For a create() operation, the itemid argument is None and newdata contains all of the initial property values with which the item is about to be created.

For a set() operation, newdata contains only the names and values of properties that are about to be changed.

For a retire() or restore() operation, newdata is None.

Reactors are called with the arguments:

react(db, cl, itemid, olddata)

where db is the database, cl is an instance of Class or IssueClass within the database, and olddata is a dictionary mapping property names to values.

For a create() operation, the itemid argument is the id of the newly-created item and olddata is None.

For a set() operation, olddata contains the names and previous values of properties that were changed.

For a retire() or restore() operation, itemid is the id of the retired or restored item and olddata is None.

Additional Detectors Ready For Use

Sample additional detectors that have been found useful will appear in the 'detectors' directory of the Roundup distribution. If you want to use one, copy it to the 'detectors' of your tracker instance:

newissuecopy.py

This detector sends an email to a team address whenever a new issue is created. The address is hard-coded into the detector, so edit it before you use it (look for the text 'team@team.host') or you'll get email errors!

creator_resolution.py

Catch attempts to set the status to "resolved" - if the assignedto user isn't the creator, then set the status to "confirm-done". "classic" Roundup doesn't have that status, so you'll have to add it. If you don't want to though, it'll just use "in-progress" instead.

email_auditor.py

If a file added to an issue is of type message/rfc822, we tack on the extension .eml. The reason for this is that Microsoft Internet Explorer will not open things with a .eml attachment, as they deem it 'unsafe'. Worse yet, they'll just give you an incomprehensible error message. For more information, see the detector code - it has a length explanation.

Auditor or Reactor?

Generally speaking, the following rules should be observed:

Auditors

Are used for vetoing creation of or changes to items. They might also make automatic changes to item properties.

Reactors

Detect changes in the database and react accordingly. They should avoid making changes to the database where possible, as this could create detector loops.

Vetoing creation of or changes to items

Auditors may raise the Reject exception to prevent the creation of or changes to items in the database. The mail gateway, for example, will not attach files or messages to issues when the creation of those files or messages are prevented through the Reject exception. It'll also not create users if that creation is Reject'ed too.

To use, simply add at the top of your auditor:

from roundup.exceptions import Reject

And then when your rejection criteria have been detected, simply:

raise Reject

Generating email from Roundup

The module roundup.mailer contains most of the nuts-n-bolts required to generate email messages from Roundup.

In addition, the IssueClass methods nosymessage() and send_message() are used to generate nosy messages, and may generate messages which only consist of a change note (ie. the message id parameter is not required).

New User Roles

New users are assigned the Roles defined in the config file as:

• NEW_WEB_USER_ROLES
• NEW_EMAIL_USER_ROLES

Changing Access Controls

You may alter the configuration variables to change the Role that new web or email users get, for example to not give them access to the web interface if they register through email.

You may use the roundup-admin "security" command to display the current Role and Permission configuration in your tracker.

Repercussions of changing the tracker schema

If you choose to change the tracker schema you will need to ensure the web interface knows about it:

1. Index, item and search pages for the relevant classes may need to have properties added or removed,
2. The "page" template may require links to be changed, as might the "home" page's content arguments.

How requests are processed

The basic processing of a web request proceeds as follows:

1. figure out who we are, defaulting to the "anonymous" user
2. figure out what the request is for - we call this the "context"
3. handle any requested action (item edit, search, ...)
4. render the template requested by the context, resulting in HTML output

In some situations, exceptions occur:

• HTTP Redirect (generally raised by an action)

• SendFile (generally raised by determine_context)

  here we serve up a FileClass "content" property

• SendStaticFile (generally raised by determine_context)

  here we serve up a file from the tracker "html" directory

• Unauthorised (generally raised by an action)

  here the action is cancelled, the request is rendered and an error message is displayed indicating that permission was not granted for the action to take place

• NotFound (raised wherever it needs to be)

  this exception percolates up to the CGI interface that called the client

Determining web context

To determine the "context" of a request, we look at the URL and the special request variable @template. The URL path after the tracker identifier is examined. Typical URL paths look like:

1. /tracker/issue
2. /tracker/issue1
3. /tracker/@file/style.css
4. /cgi-bin/roundup.cgi/tracker/file1
5. /cgi-bin/roundup.cgi/tracker/file1/kitten.png

where the "tracker identifier" is "tracker" in the above cases. That means we're looking at "issue", "issue1", "@file/style.css", "file1" and "file1/kitten.png" in the cases above. The path is generally only one entry long - longer paths are handled differently.

1. if there is no path, then we are in the "home" context. See the "home" context below for more information about how it may be used.
2. if the path starts with "@file" (as in example 3, "/tracker/@file/style.css"), then the additional path entry, "style.css" specifies the filename of a static file we're to serve up from the tracker "html" directory. Raises a SendStaticFile exception.
3. if there is something in the path (as in example 1, "issue"), it identifies the tracker class we're to display.
4. if the path is an item designator (as in examples 2 and 4, "issue1" and "file1"), then we're to display a specific item.
5. if the path starts with an item designator and is longer than one entry (as in example 5, "file1/kitten.png"), then we're assumed to be handling an item of a FileClass, and the extra path information gives the filename that the client is going to label the download with (i.e. "file1/kitten.png" is nicer to download than "file1"). This raises a SendFile exception.

Both b. and e. stop before we bother to determine the template we're going to use. That's because they don't actually use templates.

The template used is specified by the @template CGI variable, which defaults to:

• only classname suplied: "index"
• full item designator supplied: "item"

The "home" Context

The "home" context is special because it allows you to add templated pages to your tracker that don't rely on a class or item (ie. an issues list or specific issue).

Let's say you wish to add frames to control the layout of your tracker's interface. You'd probably have:

• A top-level frameset page. This page probably wouldn't be templated, so it could be served as a static file (see serving static content)

• A sidebar frame that is templated. Let's call this page "home.navigation.html" in your tracker's "html" directory. To load that page up, you use the URL:

  <tracker url>/home?@template=navigation

Serving static content

See the previous section determining web context where it describes @file paths.

Performing actions in web requests

When a user requests a web page, they may optionally also request for an action to take place. As described in how requests are processed, the action is performed before the requested page is generated. Actions are triggered by using a @action CGI variable, where the value is one of:

login

Attempt to log a user in.

logout

Log the user out - make them "anonymous".

register

Attempt to create a new user based on the contents of the form and then log them in.

edit

Perform an edit of an item in the database. There are some special form variables you may use.

new

Add a new item to the database. You may use the same special form variables as in the "edit" action.

retire

Retire the item in the database.

editCSV

Performs an edit of all of a class' items in one go. See also the class.csv templating method which generates the CSV data to be edited, and the '_generic.index' template which uses both of these features.

search

Mangle some of the form variables:

• Set the form ":filter" variable based on the values of the filter variables - if they're set to anything other than "dontcare" then add them to :filter.
• Also handle the ":queryname" variable and save off the query to the user's query list.

Each of the actions is implemented by a corresponding *XxxAction* (where "Xxx" is the name of the action) class in the roundup.cgi.actions module. These classes are registered with roundup.cgi.client.Client which also happens to be available in your tracker instance as interfaces.Client. So if you need to define new actions, you may add them there (see defining new web actions).

Each action class also has a *permission* method which determines whether the action is permissible given the current user. The base permission checks are:

login

Determine whether the user has permission to log in. Base behaviour is to check the user has "Web Access".

logout

No permission checks are made.

register

Determine whether the user has permission to register. Base behaviour is to check the user has the "Web Registration" Permission.

edit

Determine whether the user has permission to edit this item. Base behaviour is to check whether the user can edit this class. If we're editing the "user" class, users are allowed to edit their own details - unless they try to edit the "roles" property, which requires the special Permission "Web Roles".

new

Determine whether the user has permission to create (or edit) this item. Base behaviour is to check the user can edit this class. No additional property checks are made. Additionally, new user items may be created if the user has the "Web Registration" Permission.

editCSV

Determine whether the user has permission to edit this class. Base behaviour is to check whether the user may edit this class.

search

Determine whether the user has permission to search this class. Base behaviour is to check whether the user may view this class.

Special form variables

Item properties and their values are edited with html FORM variables and their values. You can:

• Change the value of some property of the current item.
• Create a new item of any class, and edit the new item's properties,
• Attach newly created items to a multilink property of the current item.
• Remove items from a multilink property of the current item.
• Specify that some properties are required for the edit operation to be successful.

In the following, <bracketed> values are variable, "@" may be either ":" or "@", and other text "required" is fixed.

Most properties are specified as form variables:

<propname>

property on the current context item

<designator>"@"<propname>

property on the indicated item (for editing related information)

Designators name a specific item of a class.

<classname><N>

Name an existing item of class <classname>.

<classname>"-"<N>

Name the <N>th new item of class <classname>. If the form submission is successful, a new item of <classname> is created. Within the submitted form, a particular designator of this form always refers to the same new item.

Once we have determined the "propname", we look at it to see if it's special:

@required

The associated form value is a comma-separated list of property names that must be specified when the form is submitted for the edit operation to succeed.

When the <designator> is missing, the properties are for the current context item. When <designator> is present, they are for the item specified by <designator>.

The "@required" specifier must come before any of the properties it refers to are assigned in the form.

@remove@<propname>=id(s) or @add@<propname>=id(s)

The "@add@" and "@remove@" edit actions apply only to Multilink properties. The form value must be a comma-separate list of keys for the class specified by the simple form variable. The listed items are added to (respectively, removed from) the specified property.

@link@<propname>=<designator>

If the edit action is "@link@", the simple form variable must specify a Link or Multilink property. The form value is a comma-separated list of designators. The item corresponding to each designator is linked to the property given by simple form variable.

None of the above (ie. just a simple form value)

The value of the form variable is converted appropriately, depending on the type of the property.

For a Link('klass') property, the form value is a single key for 'klass', where the key field is specified in dbinit.py.

For a Multilink('klass') property, the form value is a comma-separated list of keys for 'klass', where the key field is specified in dbinit.py.

Note that for simple-form-variables specifiying Link and Multilink properties, the linked-to class must have a key field.

For a String() property specifying a filename, the file named by the form value is uploaded. This means we try to set additional properties "filename" and "type" (if they are valid for the class). Otherwise, the property is set to the form value.

For Date(), Interval(), Boolean(), and Number() properties, the form value is converted to the appropriate

Any of the form variables may be prefixed with a classname or designator.

Two special form values are supported for backwards compatibility:

@note

This is equivalent to:

@link@messages=msg-1
msg-1@content=value

except that in addition, the "author" and "date" properties of "msg-1" are set to the userid of the submitter, and the current time, respectively.

@file

This is equivalent to:

@link@files=file-1
file-1@content=value

The String content value is handled as described above for file uploads.

If both the "@note" and "@file" form variables are specified, the action:

@link@msg-1@files=file-1

is also performed.

We also check that FileClass items have a "content" property with actual content, otherwise we remove them from all_props before returning.

Default templates

The default templates are html4 compliant. If you wish to change them to be xhtml compliant, you'll need to change the HTML_VERSION configuration variable in config.py to 'xhtml' instead of 'html4'.

Most customisation of the web view can be done by modifying the templates in the tracker 'html' directory. There are several types of files in there. The minimal template includes:

page.html

This template usually defines the overall look of your tracker. When you view an issue, it appears inside this template. When you view an index, it also appears inside this template. This template defines a macro called "icing" which is used by almost all other templates as a coating for their content, using its "content" slot. It also defines the "head_title" and "body_title" slots to allow setting of the page title.

home.html

the default page displayed when no other page is indicated by the user

home.classlist.html

a special version of the default page that lists the classes in the tracker

classname.item.html

displays an item of the classname class

classname.index.html

displays a list of classname items

classname.search.html

displays a search page for classname items

_generic.index.html

used to display a list of items where there is no *classname*.index available

_generic.help.html

used to display a "class help" page where there is no *classname*.help

user.register.html

a special page just for the user class, that renders the registration page

style.css.html

a static file that is served up as-is

The classic template has a number of additional templates.

Remember that you can create any template extension you want to, so if you just want to play around with the templating for new issues, you can copy the current "issue.item" template to "issue.test", and then access the test template using the "@template" URL argument:

http://your.tracker.example/tracker/issue?@template=test

and it won't affect your users using the "issue.item" template.

How the templates work

Information available to templates

The following behaviour is implemented by roundup.cgi.templating.RoundupPageTemplate

The following variables are available to templates.

context

The current context. This is either None, a hyperdb class wrapper or a hyperdb item wrapper

request

Includes information about the current request, including:

• the current index information (filterspec, filter args, properties, etc) parsed out of the form.
• methods for easy filterspec link generation
• user, the current user item as an HTMLItem instance
• form The current CGI form information as a mapping of form argument name to value

config

This variable holds all the values defined in the tracker config.py file (eg. TRACKER_NAME, etc.)

db

The current database, used to access arbitrary database items.

templates

Access to all the tracker templates by name. Used mainly in use-macro commands.

utils

This variable makes available some utility functions like batching.

nothing

This is a special variable - if an expression evaluates to this, then the tag (in the case of a tal:replace), its contents (in the case of tal:content) or some attributes (in the case of tal:attributes) will not appear in the the output. So, for example:

<span tal:attributes="class nothing">Hello, World!</span>

would result in:

<span>Hello, World!</span>

default

Also a special variable - if an expression evaluates to this, then the existing HTML in the template will not be replaced or removed, it will remain. So:

<span tal:replace="default">Hello, World!</span>

would result in:

<span>Hello, World!</span>

python:context['list']

python:context['journal']

"structure msg/content/hyperlinked"

python:context.creation.local(10)

python:context.activity.pretty('%Y-%m-%d')

context/title/field

request/form/name/value

python:request.form['name'].value

db/user
python:db.user

templates/name

templates[name]

templates/name/macros/macro_name

templates[name].macros[macro_name]

repeat/user
python:repeat['user']

python:utils.Batch(sequence, size, start, end=0, orphan=0,
overlap=0)

request/batch

<table class="otherinfo">
 <tr><th colspan="4" class="header">Existing Keywords</th></tr>
 <tr tal:define="keywords db/keyword/list"
     tal:repeat="start python:range(0, len(keywords), 4)">
  <td tal:define="batch python:utils.Batch(keywords, 4, start)"
      tal:repeat="keyword batch" tal:content="keyword/name">
      keyword here</td>
 </tr>
</table>

Displaying Properties

Properties appear in the user interface in three contexts: in indices, in editors, and as search arguments. For each type of property, there are several display possibilities. For example, in an index view, a string property may just be printed as a plain string, but in an editor view, that property may be displayed in an editable field.

Index Views

This is one of the class context views. It is also the default view for classes. The template used is "classname.index".

/issue?status=unread,in-progress,resolved&
    topic=security,ui&
    @group=priority&
    @sort=-activity&
    @filters=status,topic&
    @columns=title,status,fixer

Searching Views

This is one of the class context views. The template used is typically "classname.search". The form on this page should have "search" as its @action variable. The "search" action:

• sets up additional filtering, as well as performing indexed text searching
• sets the @filter variable correctly
• saves the query off if @query_name is set.

The search page should lay out any fields that you wish to allow the user to search on. If your schema contains a large number of properties, you should be wary of making all of those properties available for searching, as this can cause confusion. If the additional properties are Strings, consider having their value indexed, and then they will be searchable using the full text indexed search. This is both faster, and more useful for the end user.

If the search view does specify the "search" @action, then it may also provide an additional argument:

Item Views

The basic view of a hyperdb item is provided by the "classname.item" template. It generally has three sections; an "editor", a "spool" and a "history" section.

<table class="form">
<tr>
 <th>Title</th>
 <td colspan="3" tal:content="structure python:context.title.field(size=60)">title</td>
</tr>

<tr>
 <th>Priority</th>
 <td tal:content="structure context/priority/menu">priority</td>
 <th>Status</th>
 <td tal:content="structure context/status/menu">status</td>
</tr>

<tr>
 <th>Superseder</th>
 <td>
  <span tal:replace="structure python:context.superseder.field(showid=1, size=20)" />
  <span tal:replace="structure python:db.issue.classhelp('id,title')" />
  <span tal:condition="context/superseder">
   <br>View: <span tal:replace="structure python:context.superseder.link(showid=1)" />
  </span>
 </td>
 <th>Nosy List</th>
 <td>
  <span tal:replace="structure context/nosy/field" />
  <span tal:replace="structure python:db.user.classhelp('username,realname,address,phone')" />
 </td>
</tr>

<tr>
 <th>Assigned To</th>
 <td tal:content="structure context/assignedto/menu">
  assignedto menu
 </td>
 <td>&nbsp;</td>
 <td>&nbsp;</td>
</tr>

<tr>
 <th>Change Note</th>
 <td colspan="3">
  <textarea name=":note" wrap="hard" rows="5" cols="60"></textarea>
 </td>
</tr>

<tr>
 <th>File</th>
 <td colspan="3"><input type="file" name=":file" size="40"></td>
</tr>

<tr>
 <td>&nbsp;</td>
 <td colspan="3" tal:content="structure context/submit">
  submit button will go here
 </td>
</tr>
</table>