Building Collaborative Document Solutions with Connections Docs 2.0 and Box

 

Welcome back!  In previous posts, we explored the new 3rd party support in Connections Docs 2.0 and built a custom module to communicate with SharePoint 2013.  This time, we’ll be integrating with our friends at Box.  What’s nice about building this integration is that it encounters production-level questions:

  • How do I architect a hybrid (cloud and on-prem) solution?
  • How will authentication work (e.g. OAuth2, cookies, SAML)?
  • What’s the best way to initiate a co-editing session?  (e.g. build my own solution or use Box’s extension points)

The result is a solution that behaves like so.

Download Video: MP4

Admittedly, there’s one step I don’t like.  It’s the login to Docs.  I wrote the solution, but there’s one small issue I’m working through.  Once resolved, I’ll update this post.

The steps you see in the video have the following sequence.

Box Architecture

  1. User begins in Box and chooses the file to be edited in Connections Docs.
  2. Box sends the user to the Connections Docs server.  And in doing so, Box provides Docs with needed information like the file ID and the OAuth code.
  3. The user’s browser then connects to the Docs server (you’ll see the address change in the browser).
  4. The Docs application uses the OAuth code to retrieve an OAuth token directly from Box.  This token allows Docs to act on behalf of the Box user (e.g. downloading and uploading the file).
  5. The Docs application connects to my custom code.  This is needed because Box doesn’t yet know how to give the file to Docs.  So my code acts as that bridge.
  6. The file is downloaded from Box and opened in the Docs editor.
  7. On save, the file is then uploaded back to Box.

So what does it take to do all this?

Box SDK

Box has an easy to use Java SDK.  I’ll be using it to do most of the work to download a file, get the meta data, get information about the user, etc.  This actually makes my code minimal.  For example, here’s everything needed to download and upload the file.

 private BoxAPIConnection getApi() {
 String bearer = settings.get("Authorization");
 
 if(bearer != null && !bearer.equals("")) {
 bearer = bearer.substring(bearer.indexOf("Bearer") + 7);
 logger.finest("Using Box access token " + bearer);
 return new BoxAPIConnection(bearer);
 } else {
 logger.severe("No Authorization header found; OAuth not possible");
 }
 
 return null;
 }
 
 @Override
 public void open(String fileId, OutputStream out) throws IOException {
 logger.entering(BoxRepository.class.getName(), "open");
 
 BoxFile boxFile = new BoxFile(getApi(), fileId);
 BoxFile.Info info = boxFile.getInfo();
 
 logger.fine("Retrieved " + fileId + " from Box " + info.getID());
 
 boxFile.download(out);
 
 logger.exiting(BoxRepository.class.getName(), "open");
 }
 
 @Override
 public void save(String fileId, InputStream in) throws IOException {
 logger.entering(BoxRepository.class.getName(), "save");
 
 BoxFile boxFile = new BoxFile(getApi(), fileId);
 boxFile.uploadVersion(in);
 
 logger.exiting(BoxRepository.class.getName(), "save");
 }
 
 @Override
 public JSONObject getMeta(String fileId) {
 logger.entering(BoxRepository.class.getName(), "getMeta");
 
 JSONObject o = new JSONObject();
 
 BoxFile boxFile = new BoxFile(getApi(), fileId);
 BoxFile.Info info = boxFile.getInfo();
 
 if (info != null) {
 o.put(DocRepository.ID, fileId);
 o.put(DocRepository.VERSION, info.getVersion().getID());
 // TODO : Confirm works as expected
 if (info.getExtension() != null) {
 o.put(DocRepository.MIME, info.getExtension());
 }
 // FIXME : What if the extension is not present in name
 o.put(DocRepository.NAME, info.getName());
 o.put(DocRepository.DESCRIPTION, info.getDescription());
 o.put(DocRepository.SIZE, info.getSize());
 o.put(DocRepository.CREATED,
 DocRepositoryUtil.formatTime(info.getContentCreatedAt().getTime()));
 o.put(DocRepository.MODIFIED,
 DocRepositoryUtil.formatTime(info.getContentModifiedAt().getTime()));
 
 // created_by
 JSONObject c = new JSONObject();
 c.put(DocRepository.ID, info.getCreatedBy().getID());
 c.put(DocRepository.NAME, info.getCreatedBy().getName());
 c.put(DocRepository.EMAIL, info.getCreatedBy().getLogin());
 o.put(DocRepository.CREATED_BY, c);
 
 // modified_by
 JSONObject m = new JSONObject();
 m.put(DocRepository.ID, info.getModifiedBy().getID());
 m.put(DocRepository.NAME, info.getModifiedBy().getName());
 m.put(DocRepository.EMAIL, info.getModifiedBy().getLogin());
 o.put(DocRepository.MODIFIED_BY, m);
 
 // permissions
 JSONObject p = new JSONObject();
 EnumSet<Permission> permissions= info.getPermissions();
 
 if(permissions != null){
 p.put(DocRepository.READ,
 String.valueOf(permissions.contains(
 Permission.CAN_DOWNLOAD))); // String not bool!
 p.put(DocRepository.WRITE,
 String.valueOf(permissions.contains(
 Permission.CAN_UPLOAD))); // String not bool!
 } else {
 // FIXME : does this mean it's not shared or it's owned
 p.put(DocRepository.READ, "true");
 p.put(DocRepository.WRITE, "true");
 }
 
 o.put(DocRepository.PERMISSIONS, p);
 }
 
 logger.exiting(BoxRepository.class.getName(), "getMeta");
 
 return o;
 }

To use the SDK, I’ve placed the box-java-sdk-2.0.0 and it’s maven dependencies inside F:\IBM\Docs\WebSphere\AppServer\lib\ext.  The ext folder is part of WebSphere’s classloader lookup.  Since the various JARs are ~4MB, you can place them in the ext folder rather than inside your web application.

OAuth2

You may have noticed that in my getApi() function, I’m using the OAuth header.  This is my authentication mechanism.  Box tells Docs who the user is by way of the OAuth code.  Docs will then exchange the code for a token, and that token is used in my code.  You also saw me login to Docs.  I think this is unnecessary, and I’m currently doing it because I’m having a different issue.  Once I resolve it, the OAuth token is the user’s identity and Docs will use it to retrieve additional user information from Box.

Box OAuth

Box has a good write up on configuring OAuth on their end.  Read it here.  After setting up my application on Box, I need to get the OAuth information to configure Docs.

Box OAuth

Box assigns the client_id and client_secret for you.  Add the URL to Docs in the redirect_uri, for example https://docs.demos.ibm.com/docs/driverscallback.  One point of caution, just be consistent when entering this URL in various places.  OAuth will fail if the redirect_uri here is different than the one Docs sends.  Check your concord-config.json file for entries like

"docs_callback_endpoint" : "https://docs.demos.ibm.com/docs/driverscallback"

Whatever you have here should match the redirect_uri (or vice versa).

Docs OAuth

Next we need to configure Docs to use the client_id and client_secret values.  See the article here on doing so.  Admittedly, I did not even get this right.  Here are the examples from IBM.

./wsadmin.sh -lang jython -user xx -password xx -f customer_credential_mgr.py -action
add -customer docs.demos.ibm.com -key oauth2_client_id -value
"l7xxf61984f99f404575a781d47c6bfebdca"
./wsadmin.sh -lang jython -user xx -password xx -f customer_credential_mgr.py -action
add -customer docs.demos.ibm.com -key oauth2_client_secret -value
"cc692ce34451418e86d9b231ee34af65"

Some helpful points:

  • The value used for customer should match the “customer_id” : “docs.demos.ibm.com” entries in concord-config.json and viewer-config.json.
  • You will issue two wsadmin commands to store oauth_client_id and oauth2_client_secret.  In the beta, I think this was documented differently.
  • The wsadmin commands are storing data in CONCORDDB.CUSTOMER_CREDENTIAL should you need to verify.

You’ll also need to confirm that concord-config.json and viewer-config.json are properly set to use OAuth.  This is done in the following sections (concord-config.json on DMgr01 as the example).

{
 "id" : "external.rest",
 "class" : "com.ibm.docs.repository.external.rest.ExternalRestRepository",
 "config" :
 {
 "s2s_method" : "oauth2",
 "customer_id" : "docs.demos.ibm.com",
 "oauth2_endpoint" : "https://app.box.com/api/oauth2/token",
 "j2c_alias" : "",
 "s2s_token" : "123456789",
 "token_key" : "docstoken",
 "onbehalf_header" : "docs-user", 
 "media_meta_url" : "http://docs.demos.ibm.com/mydocs/DocServlet?id={ID}&mode=meta",
 "media_get_url" : "http://docs.demos.ibm.com/mydocs/DocServlet?id={ID}&mode=content",
 "media_set_url" : "http://docs.demos.ibm.com/mydocs/DocServlet?id={ID}&mode=content",
 "docs_callback_endpoint" : "https://docs.demos.ibm.com/docs/driverscallback",
 "repository_home" : "http://docs.demos.ibm.com/mydocs/DocServlet"
 }
 }
{
 "id" : "external.rest",
 "class":"com.ibm.docs.authentication.filters.ExternalAuth",
 "config" : {
 "s2s_method" : "oauth2" 
 } 
 }
{
 "id" : "external.rest",
 "class":"com.ibm.docs.directory.external.ExternalDirectory",
 "config" : {
 "profiles_url" : "",
 "s2s_method" : "oauth2",
 "customer_id" : "docs.demos.ibm.com",
 "oauth2_endpoint" : "https://app.box.com/api/oauth2/token",
 "j2c_alias" : "",
 "s2s_token" : "123456789",
 "token_key" : "docstoken",
 "onbehalf_header" : "docs-user", 
 "docs_callback_endpoint" : "https://docs.demos.ibm.com/docs/driverscallback",
 "keys" : {
 "id_key" : "id",
 "name_key" : "name",
 "display_name_key" : "display_name",
 "email_key" : "email",
 "photo_url_key" : "photo_url",
 "org_id_key" : "org_id",
 "url_query_key" : "userid"
 },
 "bypass_sso" : "false" 
 } 
 }

Note that the oauth2_endpoint is https://app.box.com/api/oauth2/token.  This is the second step in the OAuth dance – not the first.

SSL

I’ve used SSL (https) in a few places – most notably for the Box OAuth endpoint.  Docs (i.e. WebSphere) will be going out to Box to retrieve the OAuth code.  To do that over SSL, we must import the SSL certs into WebSphere.  We also need to do this because as you see in the architecture diagram, the Box SDK needs to connect to api.box.com and upload.box.come over SSL.  If you do not do this, you will see the SSL handshake exceptions in the log.

The steps to do so are documented in various places, for example here.  I’ve done the import into the Cell Default Trust store for app.box.com, upload.box.com, and api.box.com.  Just do all three for best practice – today Box is using the same cert for two of their servers, which is why you only see two certs listed in my trust store.

Box SSL

Box SSL 2

X-FRAME-OPTIONS

OMG there’s more?  Almost done.  But this one really confounded me.  Box is going to open the Docs app inside an IFrame.  For that to work correctly, we need certain security settings applied in the Docs HTML response.  I knew why the problem existed (the developer console in Firefox or Chrome will tell you it’s an issue), but I didn’t know how to resolve.  Basically, we need to change the X-FRAME-OPTIONS: SAMEORIGIN header to be ALLOW-FROM https://app.box.com.  All those blog posts that say you can do this with an Apache unset header are wrong.  It just didn’t work.  Fortuntely (after alot of decompiling), I found an undocumented way.  Add the following before the last brace at the end of your concord-config.json.

 "x-frame-options" :
 {
 "allow_option" : "ALLOW-FROM",
 "allow_uri" : "https://app.box.com"
 }

This will ensure that the header is set from the Docs application code.

Restart Docs and test.

This example was not easy.  So virtual high five for me, and if you run into issues, post a comment.

Happy coding …

Building Collaborative Document Solutions with Connections Docs 2.0 and SharePoint 2013

In a previous blog entry, I explored Connections Docs 2.0’s 3rd party support.  In this post, I’ll actually build support for a 3rd party – SharePoint 2013.  This post is pretty technical and is meant to provide working example code. If you have questions, feel free to leave a comment.

Getting Started

A few days ago, I had never used SharePoint.  But I was asked to connect Connections Docs and SharePoint 2013.  And where there’s an API, there’s a way.  SharePoint 2013 has RESTful APIs to get and store documents – specifically the Files and Folders API as documented on MSDN.

Let’s review.  To integrate a 3rd party we simply need to provide information about a file (metadata) and the file itself (binary).  We can use two SharePoint REST endpoints to do just that.

<app web url>/_api/web/getfilebyserverrelativeurl('/Shared Documents/filename.docx')
<app web url>/_api/web
 /getfilebyserverrelativeurl('/Shared Documents/filename.txt')/$value

SharePoint has a peculiar file identifier; it has spaces and slashes – just like a path on your desktop. Using “/Shared Documents/filename.txt” as an example, we’d end up with the following Docs URL.

http://docs.demos.ibm.com/docs/app/doc/external.rest//Shared%20Documents/filename.docx/edit/content

See those double slashes or notice that we have /filename.docx/edit/content?  Docs is going to fail if given this URL.

One possible solution is to encode the file identifier so that it doesn’t contain slashes or spaces.  I’ll use base64 encoding to demonstrate.

Docs Adapters

First, base64 encode the file ID (in green).  The result is a human unreadable value (in blue), but it will preserve the slashes.  I’m also going to slip in my adapter name (in orange).  This way my code can simultaneously get the correct file ID as well as the intended backend.  If we want a different backend, we just change the adapter name.  This approach also gives you a way to pass additional parameters to the 3rd party.  Pretty flexible.

Apache HTTPComponents Pitfall

I’ll be using the Apache HTTPComponents library to help with the REST communication.

Regarding SharePoint authentication, I’m simply supplying a username and password.  This is similar to basic authentication, but Microsoft does things a bit differently.  And this led to some technical frustration.

Docs ships with version HTTPComponents 4.1.1.  Unfortunately in version 4.1.1, there’s a bug in NTCredentials, which is how we need to authenticate to SharePoint.  And the latest version of HTTPComponents will not work (you’ll get issues due to Docs loading the older classes).  So I’m using httpclient-4.2.6 and httpcore-4.2.5.  You can get these from Maven.

If you’re using OAuth as the authentication strategy, YMMV.

Adapter Design

I have a few integration examples being developed.  To handle multiple 3rd parties, my code uses an adapter pattern (in software engineering speak).  The mechanics of my core code and Docs doesn’t vary.  What varies is the 3rd party.  And the adapter pattern allows me to delegate how communication to the 3rd partyis done.

Even though every adapter is different, each is going to need to do the same high level operations.

  1. Get the metadata about a file
  2. Open the file from the 3rd party
  3. Save the file to the 3rd party

So I’ve created an abstract class DocRepository to define these requirements.

abstract public void open(String fileId, OutputStream out) throws IOException;
abstract public void save(String fileId, InputStream in) throws IOException;
abstract public JSONObject getMeta(String fileId);

Notice the InputStream and OutputStream. These are sinks to the Docs server.  The OutputStream is used to write the file to Docs.  The InputStream is used to read the file from Docs.  The adapter’s job is to know how to speak “3rd party” – handling connections, authentication, error checking, etc.

SharePoint Adapter

Setting Up the HTTP Client

Let’s take a look at the code that communicates with SharePoint and sets up authentication, the HttpClient.

 private HttpClient getHttpClient() throws IOException {
 // TODO : You will want to do something more consumable; perhaps OAuth
 // figure out the user's password
 Properties props = new Properties();
 InputStream stream = getClass().getResourceAsStream(
 "/com/ibm/demos/docs/ext/SharePointUsers.properties");
 props.load(stream);
 
 // FIXME: a bit of a hack just to show functionality
 // use vstaub and not vstaub@demos.ibm.com
 String user = settings.get(settings.get(ONBEHALF_HEADER));
 user = user.substring(0, user.indexOf("@"));
 
 String password = (String) props.get(user);
 
 // set up the client with NTLM authentication
 CredentialsProvider credsProvider = new BasicCredentialsProvider();
 credsProvider.setCredentials(
 new AuthScope(AuthScope.ANY),
 // map the user to a stored password
 new NTCredentials(user, password, settings
 .get(HOST), "domain"));
 
 DefaultHttpClient httpClient = new DefaultHttpClient();
 httpClient.setCredentialsProvider(credsProvider);
 
 return httpClient;
 }

How does it know who to authenticate?  Recall that Docs will make a REST request to your implementation (3rd party).  This request will carry with it some headers.  Here’s a print out of what my servlet receives.

[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 Incoming request http://docs.demos.ibm.com/mydocs/DocServlet
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 docstoken=123456789
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 docs-user=vstaub@demos.ibm.com
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 User-Agent=Jakarta Commons-HttpClient/3.1
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 Host=docs.demos.ibm.com

Notice the docs-user header.  This is how Docs informs the 3rd party of the user’s identity.  I’m logged in as vstaub@demos.ibm.com on the Docs server.  My code can then take the value vstaub@demos.ibm.com and use it to authenticate me with SharePoint. I’ve chosen to do this by looking my password up in a property file.

“And I’m supposed to just trust that this docs-user is who he says he is?” Yep, and also notice the docstoken header.  This is a server to server (s2s) secret the 3rd party can use to validate the incoming request.  Keep in mind that this is Docs server to 3rd party interaction – not the user’s browser.  But if you need more assurances, there are other mechanisms – like using Cookies rather than a s2stoken.  See the documentation for more details.

** Update ** During a fresh deployment, I saw an error related to NTLM scheme not supported by the HttpClient. I did not see this on my server, but there were many code revisions, and it’s possible that something was stuck in the classloader.  If you run into this, you may want to add this line to the HttpClient.

httpClient.getAuthSchemes().register(AuthPolicy.NTLM, new NTLMSchemeFactory());

Getting Metadata from SharePoint

Next let’s look at the code that obtains metadata about the file.  (The code is incomplete – for example, the modification details and permissions are stubbed).

public JSONObject getMeta(String fileId) {
 logger.entering(SharePointRepository.class.getName(), "getMeta");
 
 String filename = DocRepositoryUtil.encodeSpaces(
 DocRepositoryUtil.getFilename(fileId));
 
 String url = settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('"
 + filename + "')";
 
 JSONObject o = new JSONObject();
 JSONObject d = getJson(url, Request.GET);
 
 if (d != null) {
 // ID is set by com.ibm.demos.docs.DocServlet
 // o.put(DocRepository.ID, "&lt;ID&gt;");
 
 // o.put(DocRepository.MIME, "application/msword");
 // if the extension is not in name, set mime above
 o.put(DocRepository.NAME, DocRepositoryUtil.getFilename(fileId));
 o.put(DocRepository.VERSION,
 d.get("MajorVersion") + "." + d.get("MajorVersion"));
 o.put(DocRepository.DESCRIPTION, d.get("Name"));
 o.put(DocRepository.SIZE, d.get("Length"));
 o.put(DocRepository.CREATED, d.get("TimeCreated"));
 o.put(DocRepository.MODIFIED, d.get("TimeLastModified"));
 
 // created_by
 JSONObject c = new JSONObject();
 c.put(DocRepository.ID, "sdaryn");
 c.put(DocRepository.NAME, "Samantha Daryn");
 c.put(DocRepository.EMAIL, "sdaryn@demos.ibm.com");
 o.put(DocRepository.CREATED_BY, c);
 
 // modified_by
 JSONObject m = new JSONObject();
 m.put(DocRepository.ID, "sdaryn");
 m.put(DocRepository.NAME, "Samantha Daryn");
 m.put(DocRepository.EMAIL, "sdaryn@demos.ibm.com");
 o.put(DocRepository.MODIFIED_BY, m);
 
 // permissions
 JSONObject p = new JSONObject();
 p.put(DocRepository.READ, "true"); // String not bool!
 p.put(DocRepository.WRITE, "true"); // String not bool!
 o.put(DocRepository.PERMISSIONS, p);
 }
 
 logger.exiting(SharePointRepository.class.getName(), "getMeta");
 
 return o;
 }

Getting the actual data from SharePoint is done by getJson().

 private JSONObject getJson(String url, Request type) {
 JSONObject d = null;
 
 try {
 HttpClient httpClient = getHttpClient();
 
 if (httpClient != null) {
 try {
 HttpRequestBase request = null;
 
 switch (type) {
 default:
 case GET:
 request = new HttpGet(url);
 break;
 case POST:
 request = new HttpPost(url);
 break;
 }
 
 request.setHeader("accept",
 "application/json; odata=verbose");
 
 logger.finest("Executing request "
 + request.getRequestLine());
 
 HttpResponse response = httpClient
 .execute(request);
 
 String json = EntityUtils
 .toString(response.getEntity());
 logger.finest("Received data from SharePoint " + json);
 d = JSONObject.parse(json);
 d = (JSONObject) d.get("d"); // nested object in d:
 
 } finally {
 httpClient.getConnectionManager().shutdown();
 }
 }
 } catch (Exception e) {
 e.printStackTrace();
 }
 
 return d;
 }

Reading the File from SharePoint

The code to obtain a file is actually quite succinct.  Notice that I’m reading the data from SharePoint and simply writing it to the OutputStream.  This effectively pipes data from SharePoint to the Docs server.

 public void open(String fileId, OutputStream out) throws IOException {
 logger.entering(SharePointRepository.class.getName(), "open");
 
 String filename = DocRepositoryUtil.encodeSpaces(
 DocRepositoryUtil.getFilename(fileId));
 
 HttpClient httpClient = getHttpClient();
 
 if (httpClient != null) {
 try {
 HttpGet get = new HttpGet(settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('" + filename
 + "')/$value");
 
 logger.finest("Executing request " + get.getRequestLine());
 
 HttpResponse response = httpClient.execute(get);
 
 logger.finest(response.getStatusLine().toString());
 
 // pipe the file data to the output
 out.write(IOUtils.toByteArray(response.getEntity()
 .getContent()));
 } finally {
 httpClient.getConnectionManager().shutdown();
 }
 
 }
 
 logger.exiting(SharePointRepository.class.getName(), "open");
 }

Writing the File to SharePoint

And finally, the write operation is a combination of the code seen earlier.  We need to getJson() to obtain a digest from SharePoint.  And then it’s used in the POST to SharePoint.

public void save(String fileId, InputStream in) throws IOException {
 logger.entering(SharePointRepository.class.getName(), "save");
 
 String filename = DocRepositoryUtil.encodeSpaces(
 DocRepositoryUtil.getFilename(fileId));
 
 // 1: Get the Digest for the POST
 JSONObject context = (JSONObject) getJson(
 settings.get(HOST) + "/_api/contextinfo", Request.POST).get(
 "GetContextWebInformation");
 String digest = (String) context.get("FormDigestValue");
 
 // 2: Send the data in POST
 
 HttpClient httpClient = getHttpClient();
 
 if (httpClient != null) {
 try {
 HttpPost post = new HttpPost(settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('" + filename
 + "')/$value");
 post.setHeader("X-HTTP-Method", "PUT");
 post.setHeader("X-RequestDigest", digest);
 
 ByteArrayEntity entity = new ByteArrayEntity(
 IOUtils.toByteArray(in));
 
 post.setEntity(entity); // pipe the input
 
 logger.finest("Executing request " + post.getRequestLine());
 
 HttpResponse response = httpClient.execute(post);
 
 
 logger.finest(response.getStatusLine().toString());
 } finally {
 httpClient.getConnectionManager().shutdown();
 }
 }
 
 logger.exiting(SharePointRepository.class.getName(), "save");
 }

And that’s it.  It may seem like a lot of code, but it’s not.  In fact, in my next post, we’ll see just how succinct this interaction can be.

In the mean time, happy coding!

Building Collaborative Document Solutions with Connections Docs 2.0

Connections Docs (formerly IBM Docs) is soon to release its next major update, Connections Docs 2.0.  And with it, the IBM team adds a new, significant capability: integration with third party document repositories.  For independent software vendors who are already storing, organizing, or sharing documents, you can now easily add document-centric collaboration with your existing offering.  Let’s get started.

Docs 2.0

Overview

The general idea is fairly basic.  Connections Docs will take care of all the real-time co-editing, commenting, tracking changes, conversion of file formats, etc.  You just need to supply the document to Connections Docs via a programmatic interface.  Your interface will be responsible for both retrieving and storing the document as well as returning some information about the document.  Connections Docs 2.0 supports two formats: CMIS and REST.  I’ll focus on the latter because with REST you’ll be able to code in Java, node.js – anything really – and communicate with any document repository.

Installation

Connections Docs 2.0 installs into a WebSphere Application Server cluster.  The process to do a WebSphere install is not covered here.  But I’ve linked to a few external posts on the major steps.

  1. Install WebSphere 8.5.
  2. Create a single node cluster.  (This link shows a custom profile.  You can just select Cell in the Environment Selection screen.)
  3. Add a web server to the cluster.  (This link is really good, but it’s specific to Portal. You don’t need to add the rewrite rules, and there will be no wp_profile.)

With the above pre-conditions, you can install Connections Docs.  All steps are as screenshots in the file Connections Docs 2.0 Install.  A couple of points as you go through the steps:

  • Install Packages
    • Ensure the “Other content management systems” package is used.  This is the option for third party ISVs.
    • I have not selected the Extension packages.  These are only used with IBM Connections.  Presumably, you will not be installing Connections Docs alongside IBM Connections.
  • Node Identification
    • I used the defaults for cluster and node names.
    • The webserver you installed earlier should be listed.  If you don’t want to do this step, you could likely enter a bogus URL in the “Enter URL” textfield and later access Docs using the internal ports.
  • Integration with Other Content Management Systems
    • This is the important screen.  I used my own implementation during my installation.  But since we’ll be using a sample provided by IBM later in this post, we’ll enter the following values into this screen.
    • Repository type: REST
    • URL for the file metadata: http://<your server information>/docs-sample/files/{ID}/meta
    • URL for getting/setting file content: http://<your server information>/docs-sample/files/{ID}/content
    • Call authentication method: s2s_token
    • Server-to-server token key: token
    • Server-to-server token value: 123456789
    • Act as a user: as-user
    • User profiles endpoint: <Leave this blank>
    • Repository home: http://<your server information>/docs-sample/files
  • Client-side mount points
    • I chose to use local directories rather than NFS shares.  Create these directories manually prior to installation.
  • Editor Server Cluster
    • Fully qualified host name and address of Connections file server: <add a bogus URL; this is a bug>
    • Fully qualified host name and address of email notification service: <add a bogus URL; this is a bug>
  • Restart Web Server
    • Yes

Now grab a cup of coffee because the install takes about an hour on my VM.

For those using the beta code, see the manual step in the troubleshooting section at the end of this post.

To confirm the installation, access the URL http://<your server information>/docs/api/list?method=fileType.  You should receive a JSON response with the following data.

{".ods":"20480",".xls":"20480",".odt":"20480",".pptx":"51200",".txt":"20480",".ppt":"51200",".xlsx":"20480",".csv":"5120",".doc":"20480",".odp":"51200",".docx":"20480"}

Security

Docs relies on the user’s Java security Principal.  There’s a few ways to approach security: LDAP, SAML, OAuth, etc.  When you installed the WebSphere server, security should have been enabled.  You can log in to WebSphere’s console to add users to the file based repository.  Access https://<deployment manager url>:<port>/ibm/console.  Then go to Users and Groups -> Manage Users.  Then use the create button to add new users.

Integration via REST

Using the settings from above as an example, the out-of-the-box document retrieval process works like this.

  1. The user begins in the 3rd party application and elects to edit a document.  The 3rd party application would perform any operations to do so.  For example, the document may need to be locked in the 3rd party repository or authentication performed.  Finally the 3rd party application should redirect the user to the Docs application.
  2. If the user is not authenticated in WebSphere, a redirect to the login page will occur.  Note, this may not occur or be necessary depending on configuration of your specific server.
  3. Connections Docs will request meta data about the document described by the file_id URL parameter.
  4. The 3rd party application must respond with specific JSON data describing the document and permissions that can be performed on the document.  Special note: include the extension in the “name” property.  If you do not, the mime type will not be recognized and a failure will occur.  Alternatively, you can add a “mime” JSON property with the extension as the value.
  5. Connections Docs will then request the actual document.
  6. The 3rd party application sends over the document.
  7. The user is redirected to the Docs application where the document is opened ready to be edited.

Docs Integration

 

There are optional, additional steps if you have integrated profiles.  This is not covered here [yet], but the process is essentially the same.  Given an endpoint to the 3rd party repository, Docs can query external user information in JSON format.

{
 "id" : "5c11a0c0-7f6f-1033-982d-eba7a40afa7a", 
 "name" : "docs_tester", 
 "display_name" : "docs_tester", 
 "email" : "docs_tester@mail.com", 
 "photo_url" : "https://domain/profiles/id/photo.png", 
 "org_id" : "default_org" 
}

Integration Sample

Fortunately, there is an IBM reference implementation located here.  This is boldfaced because I overlooked this fact in the beta documentation.  Don’t you do the same. IBM’s implementation is a servlet that retrieves and stores documents from a directory inside the web module.  It’s trivial to extend this example to store and retrieve from disk, database, etc.

Sample Configuration

Download the code and open it with your IDE.  We need to update the configuration file.  In the Java src folder, expand the package com.ibm.docs.api.rest.sample.filters.  You’ll see a config.json file.  Update the file with the following contents.

{
 "s2s_method": "s2s_token",
 "s2s_token": "123456789",
 "onbehalfof_key" : "as-user"
}

The above tells the sample to use the token mechanism and which header identifies the user.  The sample code is currently written to look for the “token” header and validate it with the s2s_token property in the config.json.  Note that these must match the same settings we used when installing the server.

  • Server-to-server token key: token
  • Server-to-server token value: 123456789
  • Act as a user: as-user

Sample Installation

Next export and install the web module (or EAR) on the IBMDocsMember1 server.

Install the WAR using WebSphere Console
Install the WAR using WebSphere Console
Ensure the web module is mapped to both the web server and IBMDocsMember1 server.
Ensure the web module is mapped to both the web server and IBMDocsMember1 server.
Docs Sample Install
Ensure the context matches the URLs used in the installation wizard.

 

And to be certain that everything is properly mapped in the HTTP server’s plugin, now is a good time to update the web server.  In WebSphere, do the following:

  • Generate Plugin
  • Propagate Plugin
  • Restart the HTTP server

Docs Web Server

Testing

If all goes well, you should be able to perform the following actions.

Download Video: MP4

Troubleshooting and Reference

Here are a few tips and tricks as you build your first integration.

Important URL Examples

  • https://docs.demos.ibm.com:9051/ibm/console
  • http://docs.demos.ibm.com/docs/login
  • http://docs.demos.ibm.com/docs/api/list?method=fileType
  • http://docs.demos.ibm.com/docs/driverscallback?repository=rest&file_id=test.ods

Beta Configuration Step

For beta users, you’ll need to create a mock <install root>\WebSphere\AppServer\profiles\AppSrv01\config\cells\docsCell01\LotusConnections-config\LotusConnections-config.xml file.  I’ve attached my LotusConnections-config for this purpose.  If you do not, you’ll see errors.  After you make the update, restart Docs.

[12/10/15 16:12:53:142 EST] 00000070 ConnectionsCo W com.ibm.connections.httpClient.ConnectionsConfigHelper loadConfig SONATA: Connections configuration file [F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\config\cells\docsCell01\LotusConnections-config\LotusConnections-config.xml] does NOT exist.

Script to Start Deployment Manager

@ECHO OFF

call time /t

echo Starting Deployment Manager …

F:\IBM\Docs\WebSphere\AppServer\profiles\Dmgr01\bin\startManager.bat

call time /t

PAUSE

Script to Start Docs

@ECHO OFF

call time /t

echo Starting Docs …

call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\startNode.bat

call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\startServer.bat IBMConversionMember1
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\startServer.bat IBMDocsMember1
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\startServer.bat IBMDocsProxyMember1
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\startServer.bat IBMViewerMember1

call time /t

PAUSE

Script to Stop Deployment Manager

@ECHO OFF

call time /t

echo Stopping Deployment Manager …

set username=wasadmin
set password=password

call F:\IBM\Docs\WebSphere\AppServer\profiles\Dmgr01\bin\stopManager.bat -username %username% -password %password%

call time /t

PAUSE

Script to Stop Docs

@ECHO OFF

call time /t

echo Stopping Docs…

set username=wasadmin
set password=password

call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\stopServer.bat IBMConversionMember1 -username %username% -password %password%
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\stopServer.bat IBMDocsMember1 -username %username% -password %password%
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\stopServer.bat IBMDocsProxyMember1 -username %username% -password %password%
call F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\bin\stopServer.bat IBMViewerMember1 -username %username% -password %password%
call time /t

PAUSE

Docs Configuration

If you find that you need to change a configuration setting.  Review the <install root>\WebSphere\AppServer\profiles\Dmgr01\config\cells\docsCell01\IBMDocs-config\concord-config.json file.  This contains all the settings used during the installation wizaLotusConnections-configrd.  Should you need to make a change, you’ll need to update this file and synchronize the nodeagent.  Then restart the Docs servers. The file located at <install root>\WebSphere\AppServer\profiles\AppSrv01\config\cells\docsCell01\IBMDocs-config must update and reflect your changes for the new settings to take effect.

Application Security

Make absolutely sure that Application Security is enabled.  If you do not, you will receive 401 errors when accessing Docs and errors in the log similar to the following.

[12/9/15 14:14:36:442 EST] 000000b3 ExternalAuth  W   Request is not authorized while accessing URL: /docs/api/list

This is an easy fix.

Docs Security Enabled

When All Else Fails

Review the SystemOut.logs in F:\IBM\Docs\WebSphere\AppServer\profiles\AppSrv01\logs.  Specifically see the logs inside IBMDocsMember1.