2013-12-18

Object Property Aliases

workorder:
  namespace: http://www.example.com/service
  attribute: workorder
documentType
  namespace: http://www.example.com/document
  attribute: documentType

Text 1: Defines two property aliases; "workorder" = "{http://www.example.com/service}workorder" and "documentType" = "{http://www.example.com/document}documentType".
Property aliases, defined in "/dav/Administration/PropertyAliases.yaml" provide short names for object properties. Primarily property aliases are used when storing documents via AttachFS when some meta-data needs to be stored along with the document. Use of property aliases is also supported for constructing the file-to path when a document is processed for auto-filing.
The normal representation of an object property's use of name-spaces makes specification of the property in a URI parameter cumbersome and unreliable. By allowing properties to be aliased to short-names they can be referenced more easily. It is expected that most object properties will not be aliases, but only those for which it is useful to refer to in URIs, etc...
In the upload URI for an AttachFS PUT the object property aliases can be referenced as parameters by using a "pa." prefix on the alias [object property aliases are case-sensitive]. An object property specified in the URL using a "pa." prefix but having no value will be assigned a value of "YES".
$ curl -u adam -T snapper.log -H 'Content-Type: text/plain' \ 'http://coils.example.com/attachfs/30330610/george.txt?mode=file&pa.workorder=123456&pa.documentType=note'
Text 2: Upload the document "george.txt" and setting object property values for the object properites with the aliases of "workorder" and "documentType".
Aside: In addition to setting defined object properties the URL parameters "abstract", "appointment", and "company" may be used to set the corresponding attributes of a document entity [when you are uploading a document via AttachFS]. "abstract" must be a string, this sets the long human-readable description of the document. "appointment" and "company" values must be integers; these set the linkage attributes of the document to establish a 1:1 relation between the specified entity and the document. Assigning a non-integer value to "appointment" or "company" parameters will result in an exception and the AttachFS opertation will fail; but beyond the type of the value the server does not validate that the provided value refers to an available appointment or company [contact or enterprise] entity. A client may discover object-property aliases by providing the alias information in the "DefinedProperties.yaml" file using the "alias" field of the defined property description.

2013-12-17

OpenGrouwpare As A CUPS Backend

A script suitable for use as a back-end for the CUPS print server package is included with OpenGroupware Coils. This back-end allows for queues to be created that perform HTTP PUT requests to the server using configurable [per queue] paths.  Using OpenGroupware Coils as a CUPS back-end easily enables client applications - and any application - to submit documents to work-flows or to store documents in project folders; the client simply prints via a print queue using a Postscript print driver and the CUPS server converts the document to PDF and pushes it to the server using the specified path.  An example of this is a receipt printer - printing to a project document folder allows the document to be filed electronically;  event driven document processing can then turn that same PDF around and submit it to an actual print queue if necessary.

Installation
To install the CUPs backend copy the “cups-to-ogo” script to the appropriate back-end directory on the print server and modify the permissions to permit the script to be executed by the CUPs user [typically “lp” or “root”]. 
This script does not require OpenGroupware components or the OpenGroupware package to be installed on the print server; all Python requirements are satisfied by the Python standard library.
A tool such as “ps2pdf” for converting Postscript to PDF will also be required; the script assumes all content posted to the server is in PDF format. These tools are generally already installed on any CUPS instance as CUPS itself uses the ghostscript tool-chain for processing the content of jobs.
When the script is executed by the CUPS server it will log messages to the local syslog server under the LPR facilty.

Configuration
[global]
uri=http://coils.
example.com/
auth_mech=plain
ps2pdf=/usr/bin/ps2pdf
username=lp
password=
theverysupersecretpassword

Text 2: An example cups-to-ogo.ini configuration file.
The script will attempt to read it's configuration from an INI file located at “/etc/cups/cups-to-ogo.ini”. This location may be overridden by setting the environment variable OGOCUPSCONFIGINI to the full path of the required INI file; in order to be effective that environment variable must be present in the environment of the CUPS server itself – the environment of the server is inherited by all filters and back-ends executed by the CUPS instance.
The INI file has a single global section. This section specifies the URI of the OpenGroupware Coils server, the authentication information , and the path to the postscript to PDF conversion tool. 
Section Parameter Description
global
uri
The base URI to the OpenGroupware COILS server. This URI should end with a “/”.
global
auth_mech
The authentication mechanism to use when authenticating to the server; at this time only “plain” is supported.
global
ps2pdf
The path to the executable postscript to PDF conversion tool. This tool must read its input from standard-input and write the converted document to standard-output. The tool must also be executable within the context of the CUPS server instance.
global
username
If auth_mech is “plain” this specifies the user-name to use when performing BASIC authentication to the CUPS server.
global
password
If auth_mech is “plain” this specifies the password to use when performing BASIC authentication to the CUPS server.
Table 2: cups-to-ogo configuration parameters supported in the cups-to-ogo.ini file.
Once the script has been installed and configured it should be reported when listing the enabled CUPS back-ends using the “lpinfo -v” command on the print server. 

$ lpinfo -v
network ipp
network socket
direct scsi
network smb
network http
network lpd
network beh
network https
serial serial:/dev/ttyS0?baud=115200
serial serial:/dev/ttyS1?baud=115200

network cups-to-ogo

network socket://192.168.1.10

Text 3: Listing the available CUPS back-ends using "lpinfo -v"
Security
When defining a target to receive documents from the CUPS integration do not forget to provision security ACLs of the resource to make it accessible to the user which the back-end authenticates as. In order to create a document within a folder the CUPS context must be able to realize the folder's project and have been granted list and insert permissions on the target folder.
It is recommended that a site create a specific intranet account for use by the CUPS server for authenticating to the OpenGroupware instance; this facilitates change control and auditing. This special purpose account should be excluded from the all-intranet team to avoid the possibility of inheriting additional privileges. The CUPS server should never be provided with the OpenGroupware administrative credentials or any credentials holding an administrative or developer role. Authentication and access control related to queuing jobs to queues defined on the CUPS server is entirely an issue relating to the administration and configuration of the CUPS instance,.
Creating Queues

Queues for pushing documents to the OpenGroupware Coils server can created using either the CUPS web interface or the lpadmin command line tool.
The recommended CUPS print driver for all cups-to-ogo queues is “Generic PostScript Printer Foomatic/Postscript”. The postscript input job will be converted to PDF using the configured conversion tool.
Illustration 1: CUPS web interface for creating a new print queue, via "Add Printer".
The utility of the cups-to-ogo backend is exercised through the URL portion of the printer definition. The URI specified for the queue is appended to the server URI provided in the INI configuration file. The print job, after being converted to PDF format, will be uploaded to the resulting URL by an HTTP PUT operation. Utilizing the standard HTTP PUT operation allows documents to be submitted to work-flow routes, uploaded via AttachFS, or placed in WebDAV collections.
In order to facilitate uploads into collections, via either AttachFS or WebDAV, as well as to allow capturing of all source meta-data replacement labels may be placed in the queue's URI.  These strings have the form of “${label}” where label is the label name;  each label in the queue's URI will be replaced with the appropriate value when a job is processed.
For example, to submit a print job [converted to a PDF document] to a work-flow route named “MailMeExample” and include the user name of the user who submitted the job to the CUPS server a queue URI of “cups-to-ogo://dav/Workflow­/Routes/MailMeExample/input?username=${user}” would be used. The content of the print job would be the input message of the new process [instance of the work-flow route] and an XATTR named “username” would be created and attached to the process, the value of that XATTR would be the name of the username of the user who authenticated to the CUPS server. 
Label Description
user
The user name of the context which submitted the print job [and potentially authenticated to the CUPS server]. This value will be URL quoted when the URL is constructed as user names may contain punctuation and whitespace.
id
The job id assigned to the job by the CUPS server.
title
The title attribute of the print job. This value will be URL quoted when the URL is constructed as job titles may contain punctuation and whitespace.
guid
Generates a GUID string. This is intended for use-cases when a unique identifier is required within the PUT URI.

Text 4: Substitution labels supported by the cups-to-ogo backend.

Another common use for the CUPS backend is to auto-file jobs to project folders. This can be accomplished by setting the URI of the queue to an AttachFS target indicating the folder in which to file the document. Note that auto-filing, auto-printing, etc... are still effective – those services do no consider the origin of the document. In order to file a document into a project folder with an object id of 8,891,874 a queue should be created with a URI of “cups-to-ogo:// attachfs/8891874/${guid}.pdf?mode=file”. Note how this is a standard AttachFS URI; the file mode is specified since we desire to create a document rather than an attachment. A unique file name within the folder is required so the guid substitution label is used; from this URI a unique named document will be created with a “pdf” file extension.
cupsJobUser:
  attribute: cupsJobUser
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046

cupsJobTitle:

  attribute: cupsJobTitle
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046

cupsJobId:

  attribute: cupsJobId
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046
Text 5: Example for adding the three CUPS meta-data related properties to the server's object property alias map.
Three object-properties have been defined in the document management name-space intended to contain the CUPS job information. To expose these properties for use in AttachFS URIs add them to the property aliases map of the server. If the properties are exposed as the default aliases the additional meta-data can be recorded as object properties of the new document by extending the URI of the previous example to “cups-to-ogo://attachfs/8891874/${guid}.pdf?mode=file&pa.cupsJobId=${id}&pa.cupsJobUser=${user}&pa.cupsJobTitle=${title}”.
Debugging

When the script is executed by the CUPS server it will log messages to the local syslog server under the LPR facilty.
Provided the back-end attempts to submit a job to the OpenGroupware server – meaning the input for the filter was processed successfully – the result provided back to the CUPS server will depend upon the server's response to the HTTP PUT.
A success response, any of 200 [OK], 201 [Created], 202 [Accepted], 204 [No Content], or 301 [Moved Permanently] result in a status 0 [OK] to the CUPS server indicating the job was successfully processed and the server may proceed to the next job.
A server response of 401 [Authentication Required], 419 [Authentication Time Out], 511 [Network Authentication Required], 496 [No Certificate], or 495 [Certificate Error] results in a status of 2 [Authenticated Required2] indicating to the CUPS server than an authentication or identity related issue occurred. CUPS will place the job on hold and add the “cups-held-for-authentication” keyword to the “jobs-reasons” CUPS job attribute.
If the server response indicates a temporary error by responding with 423 [Locked (WebDAV)], 409 [Conflict], 408 [Request Timeout], or 429 [Too Many Requests (Back Off)] the CUPS status of 6 [Retry] will be returned. This status indicates the CUPS server may retry the job at a later time; the server may still proceed to processing the next job in the queue.
A response value of either 418 [Teapot] or 451 [Unavailable For Legal Reasons] will result in CUPS status 5 [Cancel]. The CUPS server responds to a status 5 by automatically canceling the current job and proceeding to the next job.
Any other server response, most notably a 404 [Not Found] or 500 [Server Error], will return status 1 [Failed] to the server. The server will respond to this status using the queues defined error-policy attribute. Generally speaking the CUPS default error policy is to change the queue to a paused state when encountering a status 1 [Failed] response; a paused queue will continue to accept jobs but will not attempt to submit them to the backend.

2013-12-16

Getting A Route's Markup Via AttachFS

AttachFS provides a simple method to retrieve the representation or contents of an entity via a GET request for the object's identifier [even improper entities such as messages and attachments can be retrieved this way].  For simple clients this is often a more straight forward approach than WebDAV.
As of 0.1.49rc70 you can now retrieve the BPML markup of an OIE workflow by requesting the route's object id from AttachFS.

$ curl -vvv -u ogo:fred123 http://coils.example.com/attachfs/view/161560
...
> GET /attachfs/view/161560 HTTP/1.1

...
< Content-Length: 4228
< Content-Type: text/xml
< Set-Cookie: OGOCOILSESSIONID=20822efa-c714-40bb-bd17-283b6466e049-4cb5a34a-74cd-4c3e-9d1b-9f92382b2820-52d38676-c9ec-41f7-968d-add83df842ad; Domain=www.example.com; expires=Mon, 16-Dec-2013 23:34:21 UTC; Path=/
< X-OpenGroupware-Filename: V388TmpxrefrLoad.bpml
< etag: 161560:1
< Connection: close
<
<?xml version="1.0" encoding="UTF-8"?>

...
Example response when requesting the object id 161,560 using AttachFS.  This is a route so the markup is returned.  Note the inclusion of the X-OpenGroupware-Filename as well as the etag.

2013-12-15

StandardXML To JSON

The latest commits to OpenGroupware Coils add a new jsonWriteAction action for zOIE workflows.  OIE has long as readJSON for serializing arbitrary JSON documents into XML.  jsonWriteAction adds the ability to serialize StandardXML data into JSON documents [provided all the data-types represented in the StandardXML document can be persisted in JSON].  Date and date time values will be serialized as strings in the "YYYY-MM-DDTHH:MM:SS" format; date values will have a time value of "00:00:00'".  The jsonWriteAction requires no parameters - it simply transfoms the entire input message into JSON.

<action name="actionActivity" id="00020" extensionAttributes="XLSToJSONMailBack/00020">
  <input property="JSONData" formatter="StandardRaw"/>
  <output><source property="XLSData"/></output>
  <attributes>
    <extension name="activityName">writeJSONAction</extension>
  </attributes>
</action>

Text 1: The contents of the message "XLSData" is serialized to an application/json message labeled "JSONData".

The row elements from the StandardXML data stream will persisted into the rows key of the JSON document with each element of that array containing a keys and fields array.  Keys contain the attributes of the row which where indicated to be primary keys while fields contains the non-idempotent field values.  This maintains the meta-data at least of the record identity which in StandardXML would be encoded in element attributes [JSON having no concept of value attributes cannot express this information].
 {"formatClass": null,
 "formatName": null,
 "rows": [
     {"keys": {},
      "fields": {
        "comment": null,
        "fs_net_price": 72.0,
        "fs_discount": 0.4,
        "dexription": "Brace",
        "no_": "000-0005",
        "x2011_list": 120.0}},
     {"keys": {},
      "fields": {
        "comment": null,
        "fs_net_price": 85.98,
        "fs_discount": 0.4,
        "dexription": "Diamond Brush Template, 13\"",
        "no_": "000-0010",
        "x2011_list": 143.3}} ]
    ...
 "tableName": null,
 "rowCount": 6330}

Text 2:A JSON representation of a StandardXML message created via the xlsToXmlAction - a message having no format name, format class,  or target/source table.  No elements of the row are indicated to be keys so all columns are represented as fields.
The target/source table as well as the format information will be presented in the JSON data as outermost keys.  If the StandardXML was generated from arbitrary XML [such as the use of xlsToXmlAction] these fields may be null.   The total row count is also presented as an outermost key avoiding the need to clients to iterate the entire value of rows in order to present a count.
Thanks to the new writeJSONAction the job of shipping row and field oriented data off to a web application is now trivial.

2013-11-18

XPath Expression As Action Parameters

A new feature in the latest edition of OpenGroupware Coils [0.1.49rc67] is support for using XPath expressions as parameters for work-flow actions.  When a work-flow action parameter begins with "$XPath:" the input message (which must be XML) is interrogated for the corresponding value prior to the action being performed.  Evaluation of the XPath is performed after label substution allowing wild cards to be used to construct the query expression.
One use-case where this is frequently useful is when processing the "current" message within a foreach construct.

<foreach name="foreachActivity" id="1ee1e121"
         extensionAttributes="SendSomeNotifications/1ee1e121"
         select="/ResultSet/row/dispatched_to[text()!='none']/..">
  <source property="NotificationData"/>
  <attributes/>
  <action name="actionActivity" id="04334"
           extensionAttributes="SendSomeNotifications/04334">
    <output><source property="current"/></output>
    <attributes xmlns="">
      <extension name="activityName">sendMailAction</extension>
      <extension name="bodyText">$XPath:/row/message/text()</extension>
      <extension name="subject">$XPath:/row/subject/text()</extension>
      <extension name="from">opengroupware@example.com</extension>
      <extension name="to">$XPath:/row/dispatched_to/text()</extension>
      <extension name="asAttachment">NO</extension>
    </attributes>
  </action>
</foreach>

Via the XPath expression values can be retrieved from "current" as parameters to the action allowing the elements of the current iteration to drive work-flow actions.  In this example StandardXML rows [such as from an sqlSelectAction] are iterated producing e-mail messages based on the content of the rows.  The select element of the foreach construct chooses only those rows with a "dispathed_to" value other than "none".  The sendMailAction is then performed selecting the message body, subject, and target address from the element being iterated [the "current" message].
For efficiency the action's input message is deserialized only if an XPath expression is used,and it is only deserialized once for each iteration regardless of the number of XPath expressions used.

2013-10-31

Introducing the CIA Namespace

No, do not panic. This post is not about OpenGroupware collaborating with that nefarious organization which overthrew the democratically elected governments of Guatemala and Iran. CIA is a simply new RPC namespace added in OpenGroupware Coils 0.1.49rc64; CIA stands for "Coils Interogation API".
The role of the CIA namespace is to provide a home for esoteric RPC calls that are useful to developers and system administrators. Methods exposed in the CIA namespace are not intended for use by day-to-day end-user applications; these methods are for interrogating the state of the server. Having this namespace for methods of more esoteric purpose avoids crowding the zOGI, or any other, namespace with odd methods which may not conform to that namespace's establish conventions. For example the values returned by CIA method calls are not Omphalos representations of entities [as would be expected in the zOGI namespace], the signature of each response is specific to its method.
Currently the CIA namespace hosts a total on one method; additional methods may be added as their need becomes apparent.
That one method is getMaterializedRights(context_id:;integer, object_id::integer). getMaterializedRights allows a system administrator or workflow administrator [permissions to perform the method is limited by those roles] to inquire as to what rights a given context has on a given object. If a user should be granted access to a project, for instance, the administrator can verify after changing ACLs or assignment that the user actually has the expected access.
A call of cia.getMaterializedRights(1181110, 29680) returns a structure indicating the access which context [user] 1,181,110 has to object [a Project in this example] 29,680.

{'accountObjectId': '1181110',
'contextContextIds': ['1181110', 10735169, 10003, 959150, 955240],
'contextLogin': 'fred',
'contextRoles': [2010000],
'targetAccessRights': ['i', 'l', 'v'],
'targetEntityNamee': 'Project',
'targetObjectId': 29680}

Text 1: Response to the getMaterializedRights method.

The response contains the context ids of the specified contacts - this includes team membership, proxy-user assignments, etc... as well as the roles held by that user - in this case 2,010,000 [Helpdesk]. The target object having object id 29,680 is a Project and the specified context has insert, list, and view permissions in relation to that object. The ability for trusted users to visualize the rights granted on X to Y can facilitate the troubleshooting of access related issues; access control provisions in the enterprise can be a complicated.
Currently the CIA namespace is exposed only via XML-RPC (under the /RPC2 URI), a bug is open to expose the namespace for JSON-RPC as well.

2013-09-29

Dealing with not so potent Idempotent values.

Idempotent values are hard to come by [in relational database terms these are what we call "primary keys"]; they uniquely identify some record, or value, or object.  At least you are certain they do... until they don't.  The great big world has a habit of not respecting the idem-potency of what you declare to be idempotent; something out there just recycles it anyway.  This is no less true in IT systems like ERP applications than it is anywhere else.  
An ERP package I work with defines a workorder document to have a workorder number that is 9 digits.  And when you get to the end of that nine digit range... yea, you saw that coming... it starts over again at the bottom.  Nine digits is a lot, but with dozens of people pounding away 40+ hours a week nine digits only lasts about five years.  The trouble comes in that the data is exported from that ERP application and then sent to an OIE work-flow which performs [after some transformation] an sqlUpsertAction into a table in a relational database.  That data is then used for a variety of other work dispatch and decision support applications. 
Why an sqlUpsertAction rather than an sqlInsertAction?  An obvious question.  As any experienced data grinder knows: no process is ever quite as straight-forwards as it first appears it should be.  This ERP applications handles credits [you know - those things you have to issue when someone screws up] by reissuing the same workorder number the next day as all negative values, and then reissuing the same workorder number the day after that with the new corrected values.  So if you process the stream of the workorder's segments in date order you will end up with the correct final values [you really should use a CQRS model and journalize your data changes]. That is a bit odd, but not a problem, the primary key for the record in the relational database table is still workorder number + workorder segment number.  All the other fields acquired from the input stream can just be updated (including the date) and you'll be left with a coherent representation of the current state of the workorder.  Simple...  Until that workorder number rolls over. Then your sqlUpsertAction either finds records that do no actually represent the same document the user is looking at, or [virtuously] it fails with a referential constraint violation and an exception report is sent.
So workorder number + workorder segment number is only sorta kinda an idempotent value [primary key].  And the data stream, ultimately, does not contain any idempotent value which identifies THIS DOCUMENT definitively.  Or at least not a consistently idempotent value, it is still a valid primary key within a certain scope!  That it has a scope -  there in lies the solution to this type of problem.
Fortunately sqlUpsertAction has a "WHERE" parameter that allows 'manually' specified scope to be applied to the SELECT/INSERT/UPDATE command sequence that will be used to create the relational modification.  The keys and values are still used in the same manner - except the extra qualificiation specified in the WHERE parameter is added onto the query.  Previously my BPML sqlUpsertAction stanza looked like:

<action name="MXType2RecordCollector" id="000030" 
               extensionAttributes="MXType2RecordCollector/000030">
 <output>
  <source property="TransformedData"/>
 </output> 
 <attributes xmlns=""> 

  <extension name="activityName">sqlUpsertAction</extension> 
  <extension name="dataSource">miecr@BARNET</extension> 
  <extension name="description"/> 
 </attributes> 
</action>

With that stanza the SELECT statement looked like: "SELECT workorder, workorder_segment FROM tableName WHERE workorder = :0 AND workorder_segment = :1".  If one and only one record was found an UPDATE like "UPDATE tableName SET ... WHERE workorder = :0 AND workorder_segment = :1" would be performed, if zero records were found an appropriate INSERT would be performed, and if more than one records were found by the SELECT an error would be raised [indicating your use of they keys does not match the existing data - your keys are not idempotent].  This performed flawlessly for years.  Then the ERP application quietly one day reset the workorder number sequence.  BLAM!

The revised sqlUsertAction stanza using the WHERE parameter to apply scope to the SELECT and UPDATE is: 


<action name="MXType2RecordCollector" id="000030"
                extensionAttributes="MXType2RecordCollector/000030">
 <output>
  <source property="TransformedData"/>
 </output>   
 <attributes xmlns=""> 
  <extension name="activityName">sqlUpsertAction</extension>
  <extension name="dataSource">miecr@BARNET</extension> 
  <extension name="where">event_date > TODAY - 730 UNITS DAY</extension>
  <extension name="description"/> 
 </attributes>     
</action>


Now the SELECT looks like: "SELECT workorder, workorder_segment FROM tableName WHERE workorder = :0 AND workorder_segment = :1 AND event_date > TODAY - 730 UNITS DAY".  And a corresponding UPDATE, if a record is found, looks like: "UPDATE tableName SET ... WHERE workorder = :0 AND workorder_segment = :1 AND event_date > TODAY - 730 UNITS DAY".  
So with two clearly documented provisos -
  • The workorder sequence does not roll over in two years.
  • The life-cycle of no workorder [issue, credit, re-issue] exceeds two years
- this small change allows the data to continue to be streamed by the OIE workflow into the relational database table.  The condition provided by the WHERE attribute keeps the scope of the already existing data considered for update limited by currency [time].  The WHERE parameter allows the elegance of the sqlUsertAction to be deployed even when dealing with only-sorta-kinda primary keys; frequently those are the only 'primary keys' one has.