Building Collaborative Document Solutions with Connections Docs 2.0 and SharePoint 2013

In a previous blog entry, I explored Connections Docs 2.0’s 3rd party support.  In this post, I’ll actually build support for a 3rd party – SharePoint 2013.  This post is pretty technical and is meant to provide working example code. If you have questions, feel free to leave a comment.

Getting Started

A few days ago, I had never used SharePoint.  But I was asked to connect Connections Docs and SharePoint 2013.  And where there’s an API, there’s a way.  SharePoint 2013 has RESTful APIs to get and store documents – specifically the Files and Folders API as documented on MSDN.

Let’s review.  To integrate a 3rd party we simply need to provide information about a file (metadata) and the file itself (binary).  We can use two SharePoint REST endpoints to do just that.

<app web url>/_api/web/getfilebyserverrelativeurl('/Shared Documents/filename.docx')
<app web url>/_api/web
 /getfilebyserverrelativeurl('/Shared Documents/filename.txt')/$value

SharePoint has a peculiar file identifier; it has spaces and slashes – just like a path on your desktop. Using “/Shared Documents/filename.txt” as an example, we’d end up with the following Docs URL.

See those double slashes or notice that we have /filename.docx/edit/content?  Docs is going to fail if given this URL.

One possible solution is to encode the file identifier so that it doesn’t contain slashes or spaces.  I’ll use base64 encoding to demonstrate.

Docs Adapters

First, base64 encode the file ID (in green).  The result is a human unreadable value (in blue), but it will preserve the slashes.  I’m also going to slip in my adapter name (in orange).  This way my code can simultaneously get the correct file ID as well as the intended backend.  If we want a different backend, we just change the adapter name.  This approach also gives you a way to pass additional parameters to the 3rd party.  Pretty flexible.

Apache HTTPComponents Pitfall

I’ll be using the Apache HTTPComponents library to help with the REST communication.

Regarding SharePoint authentication, I’m simply supplying a username and password.  This is similar to basic authentication, but Microsoft does things a bit differently.  And this led to some technical frustration.

Docs ships with version HTTPComponents 4.1.1.  Unfortunately in version 4.1.1, there’s a bug in NTCredentials, which is how we need to authenticate to SharePoint.  And the latest version of HTTPComponents will not work (you’ll get issues due to Docs loading the older classes).  So I’m using httpclient-4.2.6 and httpcore-4.2.5.  You can get these from Maven.

If you’re using OAuth as the authentication strategy, YMMV.

Adapter Design

I have a few integration examples being developed.  To handle multiple 3rd parties, my code uses an adapter pattern (in software engineering speak).  The mechanics of my core code and Docs doesn’t vary.  What varies is the 3rd party.  And the adapter pattern allows me to delegate how communication to the 3rd partyis done.

Even though every adapter is different, each is going to need to do the same high level operations.

  1. Get the metadata about a file
  2. Open the file from the 3rd party
  3. Save the file to the 3rd party

So I’ve created an abstract class DocRepository to define these requirements.

abstract public void open(String fileId, OutputStream out) throws IOException;
abstract public void save(String fileId, InputStream in) throws IOException;
abstract public JSONObject getMeta(String fileId);

Notice the InputStream and OutputStream. These are sinks to the Docs server.  The OutputStream is used to write the file to Docs.  The InputStream is used to read the file from Docs.  The adapter’s job is to know how to speak “3rd party” – handling connections, authentication, error checking, etc.

SharePoint Adapter

Setting Up the HTTP Client

Let’s take a look at the code that communicates with SharePoint and sets up authentication, the HttpClient.

 private HttpClient getHttpClient() throws IOException {
 // TODO : You will want to do something more consumable; perhaps OAuth
 // figure out the user's password
 Properties props = new Properties();
 InputStream stream = getClass().getResourceAsStream(
 // FIXME: a bit of a hack just to show functionality
 // use vstaub and not
 String user = settings.get(settings.get(ONBEHALF_HEADER));
 user = user.substring(0, user.indexOf("@"));
 String password = (String) props.get(user);
 // set up the client with NTLM authentication
 CredentialsProvider credsProvider = new BasicCredentialsProvider();
 new AuthScope(AuthScope.ANY),
 // map the user to a stored password
 new NTCredentials(user, password, settings
 .get(HOST), "domain"));
 DefaultHttpClient httpClient = new DefaultHttpClient();
 return httpClient;

How does it know who to authenticate?  Recall that Docs will make a REST request to your implementation (3rd party).  This request will carry with it some headers.  Here’s a print out of what my servlet receives.

[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 Incoming request
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 docstoken=123456789
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1 User-Agent=Jakarta Commons-HttpClient/3.1
[1/27/16 11:57:57:223 EST] 000000c9 DocServlet 1

Notice the docs-user header.  This is how Docs informs the 3rd party of the user’s identity.  I’m logged in as on the Docs server.  My code can then take the value and use it to authenticate me with SharePoint. I’ve chosen to do this by looking my password up in a property file.

“And I’m supposed to just trust that this docs-user is who he says he is?” Yep, and also notice the docstoken header.  This is a server to server (s2s) secret the 3rd party can use to validate the incoming request.  Keep in mind that this is Docs server to 3rd party interaction – not the user’s browser.  But if you need more assurances, there are other mechanisms – like using Cookies rather than a s2stoken.  See the documentation for more details.

** Update ** During a fresh deployment, I saw an error related to NTLM scheme not supported by the HttpClient. I did not see this on my server, but there were many code revisions, and it’s possible that something was stuck in the classloader.  If you run into this, you may want to add this line to the HttpClient.

httpClient.getAuthSchemes().register(AuthPolicy.NTLM, new NTLMSchemeFactory());

Getting Metadata from SharePoint

Next let’s look at the code that obtains metadata about the file.  (The code is incomplete – for example, the modification details and permissions are stubbed).

public JSONObject getMeta(String fileId) {
 logger.entering(SharePointRepository.class.getName(), "getMeta");
 String filename = DocRepositoryUtil.encodeSpaces(
 String url = settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('"
 + filename + "')";
 JSONObject o = new JSONObject();
 JSONObject d = getJson(url, Request.GET);
 if (d != null) {
 // ID is set by
 // o.put(DocRepository.ID, "&lt;ID&gt;");
 // o.put(DocRepository.MIME, "application/msword");
 // if the extension is not in name, set mime above
 o.put(DocRepository.NAME, DocRepositoryUtil.getFilename(fileId));
 d.get("MajorVersion") + "." + d.get("MajorVersion"));
 o.put(DocRepository.DESCRIPTION, d.get("Name"));
 o.put(DocRepository.SIZE, d.get("Length"));
 o.put(DocRepository.CREATED, d.get("TimeCreated"));
 o.put(DocRepository.MODIFIED, d.get("TimeLastModified"));
 // created_by
 JSONObject c = new JSONObject();
 c.put(DocRepository.ID, "sdaryn");
 c.put(DocRepository.NAME, "Samantha Daryn");
 c.put(DocRepository.EMAIL, "");
 o.put(DocRepository.CREATED_BY, c);
 // modified_by
 JSONObject m = new JSONObject();
 m.put(DocRepository.ID, "sdaryn");
 m.put(DocRepository.NAME, "Samantha Daryn");
 m.put(DocRepository.EMAIL, "");
 o.put(DocRepository.MODIFIED_BY, m);
 // permissions
 JSONObject p = new JSONObject();
 p.put(DocRepository.READ, "true"); // String not bool!
 p.put(DocRepository.WRITE, "true"); // String not bool!
 o.put(DocRepository.PERMISSIONS, p);
 logger.exiting(SharePointRepository.class.getName(), "getMeta");
 return o;

Getting the actual data from SharePoint is done by getJson().

 private JSONObject getJson(String url, Request type) {
 JSONObject d = null;
 try {
 HttpClient httpClient = getHttpClient();
 if (httpClient != null) {
 try {
 HttpRequestBase request = null;
 switch (type) {
 case GET:
 request = new HttpGet(url);
 case POST:
 request = new HttpPost(url);
 "application/json; odata=verbose");
 logger.finest("Executing request "
 + request.getRequestLine());
 HttpResponse response = httpClient
 String json = EntityUtils
 logger.finest("Received data from SharePoint " + json);
 d = JSONObject.parse(json);
 d = (JSONObject) d.get("d"); // nested object in d:
 } finally {
 } catch (Exception e) {
 return d;

Reading the File from SharePoint

The code to obtain a file is actually quite succinct.  Notice that I’m reading the data from SharePoint and simply writing it to the OutputStream.  This effectively pipes data from SharePoint to the Docs server.

 public void open(String fileId, OutputStream out) throws IOException {
 logger.entering(SharePointRepository.class.getName(), "open");
 String filename = DocRepositoryUtil.encodeSpaces(
 HttpClient httpClient = getHttpClient();
 if (httpClient != null) {
 try {
 HttpGet get = new HttpGet(settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('" + filename
 + "')/$value");
 logger.finest("Executing request " + get.getRequestLine());
 HttpResponse response = httpClient.execute(get);
 // pipe the file data to the output
 } finally {
 logger.exiting(SharePointRepository.class.getName(), "open");

Writing the File to SharePoint

And finally, the write operation is a combination of the code seen earlier.  We need to getJson() to obtain a digest from SharePoint.  And then it’s used in the POST to SharePoint.

public void save(String fileId, InputStream in) throws IOException {
 logger.entering(SharePointRepository.class.getName(), "save");
 String filename = DocRepositoryUtil.encodeSpaces(
 // 1: Get the Digest for the POST
 JSONObject context = (JSONObject) getJson(
 settings.get(HOST) + "/_api/contextinfo", Request.POST).get(
 String digest = (String) context.get("FormDigestValue");
 // 2: Send the data in POST
 HttpClient httpClient = getHttpClient();
 if (httpClient != null) {
 try {
 HttpPost post = new HttpPost(settings.get(HOST)
 + "/_api/web/getfilebyserverrelativeurl('" + filename
 + "')/$value");
 post.setHeader("X-HTTP-Method", "PUT");
 post.setHeader("X-RequestDigest", digest);
 ByteArrayEntity entity = new ByteArrayEntity(
 post.setEntity(entity); // pipe the input
 logger.finest("Executing request " + post.getRequestLine());
 HttpResponse response = httpClient.execute(post);
 } finally {
 logger.exiting(SharePointRepository.class.getName(), "save");

And that’s it.  It may seem like a lot of code, but it’s not.  In fact, in my next post, we’ll see just how succinct this interaction can be.

In the mean time, happy coding!