How to Use Pivotal Device Server in Your Own Applications

Pivotal CRM has an application for the iPad and for the iPhone. Both applications talk to a middle tier called Pivotal Device Server. This middle tier is the one responsible of accessing the information on the Pivotal Business Server. The Device Server exposes REST services which are consumed by these applications.

At the moment, the Device Server is only used by these two iOS applications. However, having a REST endpoint to our Pivotal server is something that could be leveraged in other applications. For example, I could have a HTML5 web site that reads/writes to our Pivotal backend using the REST interface. I could also write any mobile app I like (e.g. for Windows Phone or Android) to access the information. In other words, I could use the REST endpoint for anything I like.

The problem is, there is no documentation (or at least I haven’t found any) on how to use the Device Server. How can I use the REST endpoint if I don’t know the services exposed, the lifecycle, and the parameters required? Well, after some digging I’ve found some information that will allows us to use the Device Server as a middle tier for any application we want. As an example, I provide a simple console application that gets a list of contacts and then gets the data for one particular contact (you can extrapolate this to more useful and fun applications such as a MVC HTML5 app or a mobile cross platform app for any of the phones out there).

Note: I don’t know how the licensing works on this. You should check with your Pivotal provider to see how you can license this service to be used as a middle tier for any non-Pivotal application. This is just a technical article on how to do things, and as with everything, this doesn’t mean you can do them.

After the little disclaimer, let’s get to work.

Discovering the Service Metadata

Services created with Microsoft WCF can display a help page to show a little of documentation about the services exposed. This can be disabled, but fortunately enough, the developers didn’t disable this. You can get the metadata for the web service by appending /help to your service endpoint.

All the Device Server service metadata should be exposed by a call to http://<server>/PivotalDeviceServer/App/HelpAll (and I knew this after I did the actions below), but this throws an error.

So, how can I found what services are being used if there is no documentation? So our first step is to find out what services are exposed. Since all the communication between the devices and the device server is through HTTP, the easiest thing to do is to put something in the middle to allow us to see what information is being exchanged. I used Fiddler to do this. I configured Fiddler as a reverse proxy and then I configured the URL on the Pivotal CRM iOS application (either iPad or iPhone) to point to Fiddler instead of the actual server.

In the image we can see the Pivotal CRM iPhone/iPad login screen. We can specify our own URL for the “Server URL” field". In this case we are pointing to the server where the Device Server is, but we are using the default Fiddler port 8888. Fiddler is installed on the same machine where the Device Server is installed, so it will then forward the request to the Device Server, but in the middle will show us what the request looks like. This will allow us to see what URLs are being called by the application, thus exposing the services endpoints.

After playing around with the app I found that the Device Server exposes four services:

• App
• User
• Data
• Metadata

If we navigate to the service metadata page (by appending the /help to the end of the url for the service) for each one of these services we will get a list of all the methods exposed by the service. The image shows the metadata for the Data service:

After doing this for each service, I got the following list:

/Data/Attachment/ByDataFormId/{dataformid}/{recordid}/{fieldid}
/Data/Attachment/ByDataFormName/{dataformname}/{recordid}/{fieldid}
/Data/ChoiceLists/ById/{tableid}?choicetype={choicetype}
/Data/ChoiceLists/ByName/{tablename}?choicetype={choicetype}
/Data/Command/CommandType/{commandtype}/CommandId/{commandid}/?returnSchema={returnSchema}
/Data/DataFormId/ByTableId/{tableid}/ByRecordId/{recordid}
/Data/Image/ById/{imageId}
/Data/Image/ByName/{imageName}
/Data/Images
/Data/Graph/ByGraphId/{graphId}/BySearchId/{searchId}
/Data/Graph/ByGraphName/{graphName}/BySearchId/{searchId}
/Data/ListDataForField/ByDataFormId/{dataformid}/{recordid}/{listformfieldid}
/Data/RecordAdd/ByDataFormId/{dataformid}?returnSchema={returnSchema}
/Data/RecordAdd/ByDataFormName/{dataformname}?returnSchema={returnSchema}
/Data/Record/ByDataFormId/{dataformid}/{recordid}?returnSchema={returnSchema}
/Data/Record/ByDataFormName/{dataformname}/{recordid}?returnSchema={returnSchema}
/Data/RecordCancel/ByDataFormId/{dataformid}/{recordid}
/Data/RecordDelete/ByDataFormId/{dataformid}/{recordid}
/Data/RecordDelete/ByDataFormName/{dataformname}/{recordid}
/Data/RecordLink/ByDataFormId/{parentdataformid}/{parentrecordid}/{linktableid}/{linkrecordid}?returnSchema={returnSchema}
/Data/RecordNew/ByDataFormId/{dataformid}?returnSchema={returnSchema}
/Data/RecordNew/ByDataFormName/{dataformname}?returnSchema={returnSchema}
/Data/RecordSecondary/ByDataFormId/{dataformid}/{recordid}/{secondarygroupid}?returnSchema={returnSchema}
/Data/RecordUnlink/ByDataFormId/{parentdataformid}/{parentrecordid}/{unlinktableid}/{unlinkrecordid}
/Data/RecordUpdate/ByDataFormId/{dataformid}/{recordid}?returnSchema={returnSchema}
/Data/RecordUpdate/ByDataFormName/{dataformname}/{recordid}?returnSchema={returnSchema}
/Data/Search/BySearchId/{searchId}?criteria={criteria}&returnSearchResultsListDef={returnSearchResultsListDef}&searchResultsListId={searchResultsListId}
/Data/Search/BySearchName/{searchName}?criteria={criteria}&returnSearchResultsListDef={returnSearchResultsListDef}&searchResultsListId={searchResultsListId}
/Data/Search/ByTableId/{tableId}?criteria={criteria}
/Data/Search/ByTableName/{tableName}?criteria={criteria}
/Data/SearchCount/BySearchId/{searchId}?criteria={criteria}
/Data/SearchCount/BySearchName/{searchName}?criteria={criteria}
/Data/SearchCount/ByTableId/{tableId}?criteria={criteria}
/Data/SearchCount/ByTableName/{tableName}?criteria={criteria}

/App
/App/HelpAll

/Metadata/AppLayout/BusinessObject
/Metadata/AppLayout/Dashboard
/Metadata/AppLayout/DataForm/ByTableName/{table}
/Metadata/AppLayout/All
/Metadata/AppLayout/Images
/Metadata/AppLayout/SearchList/ByTableName/{table}
/Metadata/AppLayout/Taskpads

/User/Login
/User/Logout
/User/Permissions

Ok, now I have some documentation. Next thing is how to use these methods.

Authentication

The first thing we need to do is login with the Device Server. If we use Fiddler, and login using the iPad/iPhone application, we will notice the following calls:

The first call is the most important. If we inspect the request and the response, we will see something like this:

------------------------ Request ------------------------
POST http://windows2008/PivotalDeviceServer/User/Login HTTP/1.1
Host: windows2008
X-Titanium-Id: a406e9f4-38bf-40ff-b6ed-ed1c17ee11eb
X-Requested-With: XMLHttpRequest
Accept-Encoding: gzip
Content-Type: application/json
Content-Length: 737
Connection: close
Cookie: ASP.NET_SessionId=phfvgrdugwvhyiu1gah54kzy
User-Agent: Appcelerator Titanium/3.1.3.GA (iPad/7.1.1; iPhone OS; en_ES;)

{
    "AppInstallationId": "32C74B8E-C640-43D3-8556-64D29065C872",
    "Domain": "",
    "Id": "PCS",
    "Password": "%27%3C%C3%8D%03yOy%C3%96%C2%96",
    "AppVersion": "6.0.5.0.4.2",
    "SecurityGroups": {
        "SecurityGroups": [{
            "Id": "0x8000000000000002",
            "Name": "ContactManagementAdministrator"
        },
        {
            "Id": "0x8000000000000003",
            "Name": "ContactManagementSuperUser"
        },
        {
            "Id": "0x8000000000000004",
            "Name": "ContactManagementUser"
        }]
    },
    "AuthenticatedToken": null,
    "ForceLogon": false
}


------------------------ Response ------------------------
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 08 May 2014 11:13:04 GMT
Connection: close
Content-Length: 1659

{
    "AppSchemaVersion": "1.0.3",
    "StatusCode": "Success",
    "CRMSystemName": "CRM",
    "DaysToKeepInHistory": 20,
    "DeviceServerVersion": "6.0.4.16",
    "DisableAutoLoginAfterIdleTimeout": "false",
    "DisableAutoLoginOnAppStart": "false",
    "DisableCachedMeta": "true",
    "DisableOffline": "false",
    "DoNotRememberUsernameOnAppStart": "false",
    "DoNotRunQuickSearchImmediately": "false",
    "ImagesChecksum": "817575750",
    "MetaDataSchemaVersion": "06.13",
    "MetaDataVersion": "826",
    "Permissions": {
        "SecurityGroups": [{
            "Id": "0x8000000000000002",
            "Name": "Contact Management Administrator"
        },
        {
            "Id": "0x8000000000000003",
            "Name": "Contact Management Super User"
        },
        {
            "Id": "0x8000000000000004",
            "Name": "Contact Management User"
        }],
        "TablePermissions": [{
            "CanDelete": true,
            "CanInsert": true,
            "CanQuickSearch": true,
            "CanRead": true,
            "CanUpdate": true,
            "Table": "Contact",
            "TableId": "0x800000000000003D"
        }]
    },
    "SecurityGroupsModified": "false",
    "StatusText": null,
    "SystemGlobalId": "17f65970-9322-47e7-b112-e424af3a430e",
    "SystemServerVersion": "6.0.13.16",
    "SystemType": "Pivotal",
    "UserToken": "4c8925a3753392da383f229f4ef4706ecb0ad9d3f8fcab016301e7f1fb2e7b2e8b8c171c4001dd6bcbba0057bd63b131ad744a286224d742b42788537063ea715e68407fc3062a165436bd6b95a36143818b9ed562b8833351b95a2f9aa68b6a5d64a74d4b9634c3ee00c1e374a0520ed35a999fcdc3df594d88817d495e5102"
}

The important information is highlighted. On the request we see that we need to specify a username and a password to authenticate with the Device Server. On the response we see that the Device Server returns an authentication token. This token will be used for all the subsequent requests to the Device Server.

The tricky part here is how to encrypt the password. The encryption used is some kind of the ARCFOUR algorithm. Here is an utility to encrypt the password:

using System;
using System.Text;
using System.Web;

namespace Pivotal
{
    public static class Util
    {
        public static string EncryptString(string valueToEncrypt)
        {
            if (string.IsNullOrWhiteSpace(valueToEncrypt))
            {
                return valueToEncrypt;
            }
            return Uri.EscapeDataString(RC4Encrypt("rzPeOQapG67sJfUqATJOfzAl74JXzyFdo0xTNYwVh", valueToEncrypt));
        }
        
        private static string RC4Encrypt(string key, string pt)
        {
            int num;
            int[] numArray = new int[256];
            for (int i = 0; i < 256; i++)
            {
                numArray[i] = i;
            }
            int num1 = 0;
            for (int j = 0; j < 256; j++)
            {
                int num2 = key[j % key.Length];
                num1 = (num1 + numArray[j] + num2) % 256;
                num = numArray[j];
                numArray[j] = numArray[num1];
                numArray[num1] = num;
            }
            int num3 = 0;
            num1 = 0;
            StringBuilder stringBuilder = new StringBuilder();
            for (int k = 0; k < pt.Length; k++)
            {
                num3 = (num3 + 1) % 256;
                num1 = (num1 + numArray[num3]) % 256;
                num = numArray[num3];
                numArray[num3] = numArray[num1];
                numArray[num1] = num;
                int num4 = pt[k];
                stringBuilder.Append(Convert.ToChar(num4 ^ numArray[(numArray[num3] + numArray[num1]) % 256]));
            }
            return stringBuilder.ToString();
        }
    }
}

With this we can authenticate with the server. Let’s create a simple console application to test the authentication. Since we are working with REST and JSON we will use the RestSharp and Json.Net libraries from NuGet.

static void Main(string[] args)
{
    string user = args[0];
    string password = args[1];
    
    password = Util.EncryptString(password));

    var client = new RestClient("http://windows2008/PivotalDeviceServer");
    
    // get the authentication token
    var request = new RestRequest("User/Login", Method.POST) { RequestFormat = DataFormat.Json };
    request.AddHeader("Accept", "application/json");
    request.AddBody(new
    {
        Id = user,
        Password = password
    });

    var response = client.Execute(request);
    JObject resultResponse = JObject.Parse(Util.DecodeJson(response.Content));
    string authenticationToken = (string)resultResponse["UserToken"];
}

We are reading the username and password from the command line (as mentioned above, you can extrapolate this to do more fun applications, this is just a simple example to explain things). In line 6, we encrypt the password using the utility presented above, and then, on line 16 and 17, we build a JSON body using RestSharp. Notice that we only care about the username and password, any other parameters we can skip. We then use Json.Net to parse the response and get the authentication token, in line 21, identified by the parameter UserToken. Notice also that we use an utility to decode the JSON response returned by the Device Server.

Not sure why (maybe is an iOS thing) but the JSON response has a lot of escape characters which are not standard JSON format. In order to get a valid JSON from the response, we need to trim all this escape chars. For this we use another utility method:

public static string DecodeJson(string json)
{
    json = json.Replace("\\\\\\\"","'").Replace("\\\"", "\"").Replace("\\\\/","/");
    return json.Substring(1,json.Length - 2);
}

We will use this utility each time we get a response back from the Device Server to format the returned JSON. If we don’t do this we will not be able to parse the JSON using Json.Net

Now that we have an authentication token, the next step is start using the API to work with the Device Server and the Pivotal backend. For our example we will get a list of contacts and then get the information from one of them. To get a list of records for a specific table we can use the method Data/Search/ByTableName/{tableName}. You can use Fiddler to see how to call this method. To save space, I won’t put the request and response output from Fiddler here, but instead I will show how to call this method from our sample program:

// get the list of contacts
request = new RestRequest("Data/Search/ByTableName/{tableName}", Method.POST) { RequestFormat = DataFormat.Json };
request.AddHeader("Accept", "application/json");
request.AddUrlSegment("tableName", "Contact");
request.AddBody(new
{
    AuthenticatedToken = authenticationToken
});
response = client.Execute(request);
resultResponse = JObject.Parse(JsonUtils.DecodeJson(response.Content));
JArray results = (JArray)resultResponse["SearchResult"];
List<Contact> records = results.Select(r => 
    {
        JArray fields = (JArray)r;
        int idFieldPosition = fields.Count - 1;
        return new Contact() 
        { 
            RnDescriptor = (string)fields[idFieldPosition]["Descriptor"],
            Id = (string)fields[idFieldPosition]["Value"]
        };
    }).ToList();

Notice, in line 4, how we specify the table name to be Contact. And in line 7, we use the authentication token we got from the first call and append this to the body of our request. I’m using a POCO object called Contact to store the information. This class is as follows:

public class PivotalRecord
{
    public string Id { get; set; }
    public string RnDescriptor { get; set; }
    public DateTime RnEditDate { get; set; }
    public DateTime RnCreateDate { get; set; }
    public string RnEditUser { get; set; }
    public string RnCreateUser { get; set; }

    public override string ToString()
    {
        return String.Format("{0} [{1}]",RnDescriptor, Id);
    }
}

public class Contact : PivotalRecord
{
    public string FirstName { get; set; }
    public string LastName { get; set; }        
}

We now have a list of all the contact Ids. Next, let’s see how we can get the data form a specific record. For this we will use the method Data/Record/ByDataFormName/{formName}/{recordId}

Contact record = records[0];
request = new RestRequest("Data/Record/ByDataFormName/{formName}/{recordId}?returnSchema=false", Method.POST) { RequestFormat = DataFormat.Json };
request.AddHeader("Accept", "application/json");
request.AddUrlSegment("formName", "Contact - Device");
request.AddUrlSegment("recordId", record.Id);
request.AddBody(new
{
    AuthenticatedToken = authenticationToken
});
response = client.Execute(request);                
resultResponse = JObject.Parse(JsonUtils.DecodeJson(response.Content));

record.RnDescriptor = (string)resultResponse["RecordDescriptor"];
record.RnEditDate = DateTime.Parse((string)resultResponse["RecordEditDate"]);
record.RnCreateDate = DateTime.Parse((string)resultResponse["RecordCreateDate"]);
record.RnCreateUser = (string)resultResponse["RecordCreateUserId"];
record.RnEditUser = (string)resultResponse["RecordEditUserId"];

JArray fieldList = (JArray)resultResponse["FormDataTableList"][0]["FormDataRowList"][0]["FormDataFieldList"];
record.FirstName = (string)fieldList[0]["Value"];
record.LastName = (string)fieldList[1]["Value"];

Notice again in line 8 how we specify the authentication token to be part of the body. The rest is just parsing the returned JSON to extract the information we need on the contact. Unfortunately, using the deserializer from Json.Net is not straighforward to do due to the way the JSON is constructed by the Device Server (it is a generic format used to represent any object).

And this is it. You can now use the Device Server as a REST endpoint to your Pivotal backend from any of your applications.