Category Archives: Technical Investigation

Search Schema Scoping in SharePoint Online

For solutions that are contained in a single site collection, or span a small number of site collections, or are in a tenant where the other solutions are not trusted or are unknown, then I have a strong preference to use site collection scoped search schema rather than tenant scoped.

Side note: I am yet to come across a situation where I would use site scoped search schema. In my mind, the existence of search schema at this level only serves to confuse.

Search Schema hierarchy is SharePoint Online.
Search Schema hierarchy is SharePoint Online. There is also site scoped search schema at lowest level which is not present here.

For those that aren’t fully aware, search schema (the set of managed properties that are accessible via the search framework) can be provisioned at the tenant, site collection, or site scope. These scopes are hierarchical such that managed properties are inherited from the tenant scope down to the site scope but can be overridden along the way. There are some good articles that delve into this in more detail.

By provisioning search schema at the site collection level you are mitigating the risks of errors related to other solutions changing the properties which your solution relies upon. This is especially relevant in SharePoint Online where all solutions in the tenant have to share a common set of RefinableTypeXX managed properties.

There are some important exceptions, of course.

  1. People Search, a.k.a User Profile Search, a.k.a Local People Results
    In SharePoint Online, people properties are indexed on a very slow schedule. We requested more information from Microsoft regarding this and were told that this schedule is ‘confidential’. I have found that when using site-collection scoped managed properties it can take *weeks* for them to get populated. I have found much better (although still poor) performance using tenant scoped properties (usually within a few days). Assuming you do require custom search schema for people properties I would still recommend provisioning all remaining managed properties (all those not mapped to people properties) at the site collection level.
  2. Many site collections
    Of course, having many site collections which require the same search schema is valid reason to go tenant scoped. This is purely due to management of the properties going forwards. A solid scripted deployment procedure should not care if you are provision search schema to 1 or 50 site collections – but anyone maintaining the solution will definitely care if they have update 50 schemas manually, or are suddenly required to script something which they feel should be *easy*. Even in this scenario you should still consider how much you trust other solutions in the tenant against the impact of finding out that one day your managed properties are mapped incorrectly. Depending on your solution this could lead to errors that are left undetected, or conversely obviously break your home page.


SharePoint Online remote authentication (and Doc upload)

The SharePoint REST API is touted as being the tool to provide inter-platform integration with SharePoint Online. However, outside of .NET the authentication piece is not so straightforward. App authentication solves this issue for registered apps but I want to show how remote user authentication can be achieved, regardless of platform.

The goal of this post is to provide examples of the HTTP requests which need to be made in order to authenticate SharePoint Online. It then provides an example of using the same technique to upload a document and update metadata just to prove it all works 🙂

The type of applications where this kind of approach may be necessary include: a Java application, a PHP application, or JavaScript application where there is otherwise no SharePoint Online authentication context and the decision has been made (for whatever reason) that user authentication is most appropriate (as opposed to app authentication).

Edit: This approach will not work in a JavaScript environment due to cross-domain restrictions enforced by browsers (unless of course you are on the same domain, in which case you don’t need to worry about any of this anyway). The ADAL.js library is available for the cross-domain JS scenario. I have posted an example here:

I wrote about using the SharePoint REST API here (and background here, and here).

I will be providing examples of the requests using the ‘Advanced REST Client’ Google Chrome extension.


The authentication piece comes in a few steps:

  • Get the security token
  • Get the access token
  • Get the request digest

Get the security token

First we must provide a username and password of a user with Contribute access to the Roster Data library and the URL at which we want access to the SharePoint Online Security Token Service.

This is done by POSTing the following XML as the request body to:

<s:Envelope xmlns:s=""
    <a:Action s:mustUnderstand="1"></a:Action>
    <a:To s:mustUnderstand="1"></a:To>
    <o:Security s:mustUnderstand="1"
    <t:RequestSecurityToken xmlns:t="">
      <wsp:AppliesTo xmlns:wsp="">

Requesting the security token
Requesting the security token

The response from the request includes the security token needed to get the access token. It is the value which has been stricken out in orange in the image below.

Response including the security token
Response including the security token

Get the access token

Once the security token has been retrieved it must be used to fetch the access token. The can be done by POSTing to the following URL with the security token as the request body:

Request to fetch the access token, passing the security token
Request to fetch the access token, passing the security token

The response from this request includes couple of cookies which must be passed as headers with all future requests. They are marked with the ‘Set-Cookie’ header. We need the ones beginning with rtFa= and FedAuth=. They can be seen the below image of the response headers.

Response includes the access token cookies
Response includes the access token cookies

Get the request digest

The request digest is a .NET security feature that ensures any update requests are coming from a single session. It must also be included with any POST requests.

The request digest is fetched by POSTing to:
The access token cookies must be included as Cookie headers with the request as shown in the image below.

Request to fetch the request digest, passing access tokens
Request to fetch the request digest, passing access tokens

The response from the request will include the request digest in the XML response as in the image below. The entire contents of the FormDigestValue tag will required, including the date time portion and timezone offset (-0000).

Response containing the request digest value
Response containing the request digest value

Upload a document with metadata

Upload the document

Now that we have all the authentication headers we can make update calls into SharePoint Online as the user whose credentials we originally supplied when fetching the security token.

In order to upload a document perform the following POST request:‘list name’)

A number of headers must be send with the request including the access token cookies, the request digest (X-RequestDigest) and the accept header as shown in the image below. The body of the request must contain the content of the document being uploaded.

Request to upload a document to SharePoint Online
Request to upload a document to SharePoint Online

The response of this request contains some minimal metadata about the file and can be safely ignored. However, for completeness here it is.

Response JSON from the document upload request
Response JSON from the document upload request

The unique ID property could be used to fetch the document in order to perform metadata updates rather than URL as done in the following example.

Update document metadata

The final step which needs to take place is update the document in SharePoint with the relevant metadata.

This can be done with yet another POST request. This time to the following URI:‘listTitle')

All the headers sent with the previous request must be sent with this request as well. The request body is a JSON object which defines the metadata fields to be updated. The fieldname and fieldValue properties must be updated as required. Note that the fieldname property must be equal to the field internal name not the field display name. An example of this is in the image below.

Request to set metadata on a document in SharePoint
Request to set metadata on a document in SharePoint

The response from this request provides success notification for each individual field update as shown below.

Response from the document metadata update request
Response from the document metadata update request

So, this should now be enough to write an application in any server-side language which supports web requests and work against SharePoint Online. I’d love to see some implementations of this, please comment if you’ve done it.

I’d like to acknowledge the following posts as they were invaluable references:


Key difficulties deploying a SharePoint Online solution using CSOM

I have been developing a console app that utilises the SharePoint C# CSOM to deploy a solution to SharePoint Online (a.k.a Office 365 SharePoint). The solution involves more than just a wsp (although it has one of those too). I have encountered a few difficulties during this process and this blog will discuss those:

  • (Re)creating a site collection
  • Importing a large-ish taxonomy
  • Uploading and installing a sandboxed solution (that contain only declarative elements)
  • Hooking up of taxonomy and (root site) lookup columns
  • Pre-creating a number of sites with specific features enabled (including the root site)

Before I go any further, for those of you reading this before doing something similar yourselves, please be aware of two constraints which caught me by surprise:

  • You can’t leverage the same import taxonomy function that is available in Term store management. If you already have files in that format you will need some custom code (I have an example later on) or you may want to import from a more robust XML formatted document
  • The CSOM does not support uploading or activating sandboxed solutions! However, there is a CodePlex project that assists with this. I also include the dll later in the post that I have rebuilt with references to the lastest v16 Microsoft.SharePoint.Client dlls.
  • The CSOM does not support activating web scoped features! You can active site scoped but not web scoped. You need to use web templates to achieve this. Again, I will cover this in some more depth later on.


Deleting and recreating a site collection

The initial step of the deployment process involves creating a new site collection (having deleted it first as required). In order to perform actions at this scope (tenant) you cannot create your client context in the same manner as usual (with a reference to the site collection; as it is yet to exist and the site collection delete and empty recycle bin require it too). Instead you must create the client context passing in tenant/admin site URL.
This is the one that looks like this: https://<tenant>

You can then create a Microsoft.Online.SharePoint.Tenant object by passing the ‘admin’ client content to its constructor. This object requires a reference to the Microsoft.Online.SharePoint.Client.Tenant assembly which is available by downloading and installing the SharePoint Server 2013 Client Components SDK. The assembly can then be found here: C:\Program Files\SharePoint Client Components\16.0\Assemblies

The tenant object provides the methods required to perform the create and delete site collection actions. This process involves a lot of waiting about for deletion to complete, and then provisioning to complete. Unfortunately you can’t continue with other actions until this has occurred. I found this to take upwards of three minutes.

A link to the relevant code that I used to achieve this can be found here:

Importing taxonomy

As mentioned above you can’t pass those CSV files directly to the CSOM and have it import it all for you. In my scenario we had already developed a lot (dozens) of term sets in the form of these CSV files so that were able to import them during a discovery phase so it was important that I could support the import of taxonomy in this form. I wrote code to support in the import of these files, but only to the point that it meets my immediate requirements. Please use the following as a rough guide only as it is not fully featured (or tested beyond the happy path).

The code I wrote to support his can be found here:

Uploading and activating a sandboxed solution

There is a CodePlex project that provides this functionality (as well as some authentication utilities) that I mentioned above. It performs web requests to UI and I am very glad someone else has already done this before me! It was originally created when SharePoint 2010 was present in the cloud and references the v14 assemblies of the Microsoft.SharePoint.Client assemblies accordingly. If you don’t mind maintain references to both v14 and v16 assemblies then this might be fine. I have instead rebuilt the source having replaced the references with the v16 equivalents.

You can download it here: SharePointOnline.Helper.dll

FYI: v14 is SharePoint 2010, v15 is SharePoint 2013, v16 is SharePoint 2013 Online specific

Activating web features

Actually there isn’t a lot more to say here other than you must use web templates if you need to create sites with features enabled as part of the deployment process as it can’t (currently) be done using the CSOM. I would recommend using the web template for nothing other than activating features and put all other declarative elements in a feature. This will provide the best upgrade experience in the future.

Hooking up taxonomy columns

The best place to start is almost certainly a reference to Chris O’Brien’s blog on this here. As I have the luxury of being able to run further deployment code after uploading/activating the sandboxed solution I opted to avoid having to rebuild the solution for various environments and instead hook-up the columns using the CSOM and a mapping. There is a catch with this though.

If your list instance is built from a list template which defines the managed metadata columns then updating the site column via the CSOM fails to push down the new SspID. To get around this, DO NOT include managed metadata column definitions as part of the list definition (in the fields element). When you run the CSOM to update the site columns it will update the content type and add the column to the list instance with the correct SspID.

Good luck building your SharePoint Online CSOM deployment framework!

Running SP.UI.Status.addStatus on page load

This post is about a specific issue when running the SP.UI command ‘addStatus’ on page load as well as short discussion on the JavaScript page life cycle.

Initial Problem
When running the SP.UI.Status.addStatus command upon page load, the status message was being hidden almost immediately in Chrome but worked as expected in IE.

We have a concept of ‘archived’ sites. A control is embedded on the page layout for site home pages (default.aspx) which renders javascript which should display a ‘this site is archived’ message. The message should be displayed using SP.UI.Status.addStatus and was initially implemented as follows (the script is being rendered dynamically using an ASP literal control, I’ll just be considering the final output):

// This is an example of what NOT to do
ExecuteOrDelayUntilScriptLoaded(function () {
    SP.UI.Status.addStatus("This is an archived site", "removed for brevity");
}, 'sp.js'););

This worked as expected in IE, however the status message would appear for a split second and then disappear in Chrome.


The first piece of the the puzzle: What is happening?
After getting deep with Chrome DevTools I found the script responsible for hiding the status message. SharePoint utilises the document.onreadystatechange handle to run a function called fnRemoveAllStatus. I think you can guess what it achieves. Why this is being run at this point is beyond me. Importantly, I don’t want to prevent it running in case it serves a purpose that I’m unaware of.

The second piece of the the puzzle: How’s that work?
If a function is assigned to document.onreadystatechange it will be run as many as four times (depending on when in the cycle the assignment occurs), once for each transition between the following sequence of states:

  1. ‘uninitialized’
  2. ‘loading’
  3. ‘interactive’
  4. ‘complete’

Good practice would have the function check for the current state and only act once, when in the correct state. Naturally this logic is absent from the fnRemoveAllStatus function.

The third piece of the the puzzle: What is running when?
$(document).ready vs $(window).load vs ExecuteOrDelayUntilScriptLoaded
The difference between these options in regards to what we have just discussed is that $(document).ready and ExecuteOrDelayUntilScriptLoaded run when document.readyState is ‘interactive’ whereas $(window).load runs when document.readyState is ‘complete’.

Laying the final puzzle piece: Why’s it working in IE but not Chrome?
When running the code in IE, rather than executing the script block during the ‘interactive’ readyState is was being executed after transitioning to the ‘complete’ readyState and this meant that is was running after the fnRemoveAllStatus call as we desire. I believe this happens because the sp.js file is being added via a script link control with ‘LoadAfterUI’ set to true which is only understood by IE. I haven’t investigated this last comment, if I am wrong please leave a comment about it below.

So, the solution is rather simple once you have this understanding. Wrap the command in $(window).load to ensure it occurs after the fnRemoveAllStatus method is called during the transition into the ‘complete’ state. Like this:

$(window).load(function() {
    ExecuteOrDelayUntilScriptLoaded(function () {
        SP.UI.Status.addStatus("This is an archived site", "removed for brevity");
    }, 'sp.js'););

NB: ExecuteOrDelayUntilScriptLoaded will also load scripts which are marked to load on-demand. If you are testing in Chrome you may begin to believe it unnecessary to use ExecuteOrDelayUntilScriptLoaded when using $(window).load as all scripts have loaded by then. This is true for browsers other than IE. In IE we must use this function to ensure that the script is loaded at all.

As a final note I’d like to add that apart from this specific case I would suggest using $(document).ready rather than $(window).load as it will mean that the page loads faster (unless of course your script requires all resources to be loaded before acting. e.g. you are working with images of undefined sizes).

Excel data connection with Access 2013 App

I have had a hard time creating data connections with an Access 2013 App database. After a good few hours spent scouring the internet for a solution, and a good few more hours uncovering a “solution” that is underwhelming at best, I am happy to share with you my findings. I really hope that someone will leave a comment with a better solution at some point in the future.

This blog post will provide step-by-step guide on how to achieve a data connection from an Excel workbook (which can be hosted in SharePoint) to the SQL database behind an Access 2013 App. Once this is achieved, a good BI developer should have no trouble visualising the data captured via the Access App with the help of pivot tables, slicing and graphing.

The first step is to identify the server address and database to connect to along with the credentials required to authenticate.

  1. This can be done by navigating to the Access App, clicking the ‘settings’ icon, then clicking ‘Customize in Access’

    Launching the app database in Access
    Launching the app database in Access
  2. Download the .accdw file and open it to launch access
  3. Click ‘FILE’ in the ribbon
  4. Click ‘Manage’
  5. In the drop-down menu ensure that ‘From Any Location’ and ‘Enable Read-Only Connection’ are highlighted with pink squares. If not, click them

    Determining the Access database location and credentials
    Determining the Access database location and credentials
  6. Click ‘View Read-Only Connection Information’
  7. Take note of Server, Database, UserName, and Password from this dialog as you will need them all later

    Access connection information dialog
    Access connection information dialog

Next we use this information to create the data connection.

  1. Launch Excel
  2. Create a new external data connection ‘From Data Connection Wizard’

    Launching the Excel data connection wizard
    Launching the Excel data connection wizard
  3. Click ‘Other/Advanced’, then ‘Next’
  4. Click ‘SQL Server Native Client 11.0’, then ‘Next’
  5. On the ‘Data Link Properties’ dialog, uncheck the ‘Blank Password’ box and check the ‘Allow saving password’ box, then input the server name, user name, password, and database

    Configuring the data connection. Ensure you provide the database
    Configuring the data connection. Ensure you provide the database
  6. Test the connection, you should see a dialog box with ‘Test Connection Succeeded’
  7. Note that it is when you attempt to make a data connection without providing the database that you get the following error which I bet lead to to this post:

    Failure to connect to the Access 2013 App's SQL database
    Cannot open server ‘xxxxxxxxxx’ requested by the login. Client with IP address ‘’ is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.
  8. You can now click ‘Ok’
  9. Uncheck the ‘Use Trusted Connection’ checkbox and replace the existing password with the correct one. Click ‘Ok’
  10. Select a table and click ‘Next’. You can get fancy here later, let’s just get it working first.
  11. Click ‘Finish’
  12. Click ‘Ok’
  13. The data connection will fail with the following error:

    Connection error
    Initialization of the data source failed. Check the database server or contact your administrator. Make sure the external database is available, and then try the operation again. If you see this message again, create a new data source to connect to the database.

The final frustration!

On the next dialog, uncheck the ‘Use Trusted Connection’ checkbox and replace the existing password with the correct one. Click ‘Ok’.
The second time it works. This process of providing the connection credentials twice is required not only upon the creation of the connection but also every time the data needs to be refreshed. It makes for a rather poor UX and it is a pretty awful scenario to have to explain to a client.

I really want to believe that there is a setting (most probably under the ‘All’ tab on the ‘Data Link Properties’ dialog) that will workaround this issue however I am yet to find it. Please leave a comment if you find a solution to this issue.


Reading meeting invites from SPEmailMessage

SharePoint allows developers to create receivers for the EmailReceived event which occurs when a list receives an email. I have a use case which requires me to leverage this event in order to forward incoming email to a set of users according to a number of business specific rules. To achieve this I must create a custom email message object (we are using aspNetEmail to send email) from the message object received in the event receiver. I need to be able to extract all of the parts from the SPEmailMessage to create this new object. The SPEmailMessage object is pretty easy to work with; the attachments are in the attachments collection, the subject is in the subject property – you get the idea. However, there is one ‘property’ that isn’t as trivial to extract from the object: a meeting invite.


I will explain how to extract the meeting invite below but first let me provide a basic overview of how an email is stored in eml format. The eml format is relevant because the SPEmailMessage can be constructed from an eml stream and also because when SharePoint is configured to attach incoming emails to discussion items it does so using eml. The first lines of an email in eml format are the email headers (think properties). These are simply key value pairs and includes things like ‘to’, ‘from’, ‘date’, ‘subject’ and many other less obvious properties including threading info. Then comes the mime body parts. These should represent the ‘same’ content in different formats (mime types). Typically this includes a text/plain block and a text/html block. A client which supports HTML will render the latter body part where otherwise it might render the plain text body part as the email content. Finally, attachments are listed out with their own set of headers and the binary content (commonly represented as a base64 encoded string).

When a meeting invite is sent to a SharePoint list without attachments the meeting invite itself can be found in the attachments collection of the SPEmailMessage object. But don’t be fooled. Although it is present in this situation, if you send same meeting invite with an attached document then – sad face – the meeting invite is not in the attachment collection (the attached documents will be). Nor can the invite be found in any of the public properties on the email object. It’s not that strange that meeting invite isn’t present in the attachments collection; it is strange that it can ever be found there. I say this because if we consider the eml format, a meeting invite is stored as another mime body part (of type text/calendar) and not as an attachment at all.

Eventually, after much investigation and reflection, we discovered a way to read the mime body parts directly from the email using only Microsoft libraries with the help of reflection. Once we have the meeting invite as a memory stream we parse it into a dictionary of string properties. The dictionary contains keys such as “LOCATION”, “SUMMARY”, “DESCRIPTION”, “DTSTART”, “DTEND” and “UID”, along with any other data stored as part of the invite. See the example code below:

Finally, I’d like to note that we only have requirements to support Outlook clients at this point so please consider that your mileage may vary when you get it out into the real world. Good luck.

OData in SharePoint 2013 with jQuery

This is the third part of a series of posts and assumes knowledge of RESTful Services in jQuery and Understanding OData.
In the previous post I showed an example of calling an OData service in the browser using the URI: http://server/_api/web/lists?$filter=startsWith(Title, 'Calendar')&$select=Hidden. This is an example of calling the SharePoint 2013 OData web service or as it commonly known: the ‘_api’ (underscore A.P.I). The _api is, in fact, just an alias (or mapping) to a service that was already present in SharePoint 2010: _vti_bin/client.svc. Calling this service directly wasn’t supported in SharePoint 2010 (it wasn’t OData or REST compliant until now) but if you have had any experience coding against the Client Side Object Model (CSOM) then you have been calling this service indirectly. It is important that you always use the _api alias as Microsoft provides no guarantee that the service end point won’t be moved or renamed in future releases. Side note: The SP2010 asmx web services are deprecated in SP2013 and although ListData.svc is not officially deprecated the new _api should be used going forward.

SP OData service diagram
SP OData service diagram – Thanks to Ted Pattison’s SPC12 slide deck for this image

Thankfully Microsoft decided that OData support would be useful and I think that the vast majority of developers will now rarely use the CSOM when writing SharePoint Apps or other SharePoint interfacing applications (at least from web clients; it may still have a place in desktop applications). The service API has been extended to cover:

  • Search
  • User Profiles
  • BCS
  • Taxonomy
  • Workflow
  • Publishing
  • More… (including analytics, feeds, IRM and e-discovery)

All the new functionality that is available via the RESTful interface is all present via the CSOM as well, however, the inter-platform and familiar nature of a RESTful service greatly reduces the learning curve for developers compared to the CSOM.

So, now that you have an understanding of REST and OData, the challenge of efficiently leveraging the _api service comes down to learning the service ‘syntax’. That is; learning the resource paths and query options that will return the data (SharePoint assets) which you require. To begin with you need to select an access point dependent on what you are looking for.

Service Access Points

  • Site http://server/site/_api/site
  • Web http://server/site/_api/web
  • User Profile http:// server/site/_api/SP.UserProfiles.PeopleManager
  • Search http:// server/site/_api/search
  • Publishing http:// server/site/_api/publishing

From an access point you can traverse objects using the same hierarchy which you will be familiar with having coded against the server-side API. For example /_api/web/lists/GetByTitle('Calendar')/fields/getbyid('<guid>') would return information about a single field on the Calendar list. The ‘methods’ which the SharePoint OData service use (eg GetByTitle and GetById) are extensions to the OData protocol to make the service more usable. If you have experience with the CSOM you can leverage this knowledge as the OData service can be called using syntax that is very similar. A good reference on this can be found here.

Rather that going into great depth regarding the _api vocabulary I will present practical code examples to perform GET, ADD, DELETE and UPDATE actions. Once you have these examples up and running you can look elsewhere for the syntax to find specific objects if it hasn’t been covered here.

Debugging Tip
Before you start debugging these examples keep in mind that if you press F5 to test your project it will attach the Visual Studio debugger and prevent the browser debugger from being allowed to attach. Instead press CNTL+F5.

GET Example

function onGetCustomer(customerId) {
  // begin work to call across network
  var requestUri = _spPageContextInfo.webAbsoluteUrl +
                "/_api/Web/Lists/getByTitle('Customers')/items(" + customerId + ")";
  // execute AJAX request
    url: requestUri,
    type: "GET",
    headers: { "ACCEPT": "application/json;odata=verbose" },
    success: function(data){
    error:  function(){
      alert("Failed to get customer");

The _spPageContextInfo JavaScript object has been around since SharePoint 2007. It provides really easy way to start building the correct _api URI, as shown. It also includes a property for the current user Id among other values. The ACCEPT header is required when performing service call against the _api. I’ve explained the purpose of the ACCEPT header in a previous post but the point of note here is that the odata=verbose appended to the end is required in order for the service call to succeed. This is because the OData specification also supports odata=light (the default) but the _api doesn’t (as yet).

ADD Example

function onAddCustomer() {

    var requestUri = _spPageContextInfo.webAbsoluteUrl +

    var requestHeaders = {
      "ACCEPT": "application/json;odata=verbose",
      "X-RequestDigest": $("#__REQUESTDIGEST").val(),

    var customerData = {
      __metadata: { "type": "SP.Data.CustomersListItem" },
      FirstName: "Paul"

    requestBody = JSON.stringify(customerData);

      url: requestUri,
      type: "Post",
      contentType: "application/json;odata=verbose",
      headers: requestHeaders,
      data: requestBody,
      success: function(){
        alert("Created customer");
      error: function(){
        alert("Failed to create customer");

Explaining the form digest beyond “a security feature of ASP.NET” is outside of the scope of this article. The important thing to understand is that you must send the form digest along with any POST request or nothing will get updated. Luckily for us, the form digest is stored as the value of a well known element on the page: the element with id ‘__REQUESTDIGEST’ (the double underscore is not a typo). It must be passed as the value of the X-RequestDigest header, as shown.
This post is specifically about calling the _api with jQuery, however I will mention that in order to get the form digest in another context you would need to first perform a call to _api/contextinfo where it can be retrieved.

When adding a new entity you must provide the __metadata 'type' property with the list item entity type of the list. You can think of this like the content type of the list. The name of the list item entity type can often be predicted as it’s built based on the list name. E.g If you have a list called Customers then it will have list item entity type of “SP.Data.CustomersListItem”. However, this quickly ceases to hold true; If a second Customers list were to be created then the list item entity type for it would be different (an integer gets inserted). As such, it is bad practice to hard code this value and instead it should be the result of previous service call. However for the sake of code brevity I haven’t heeded my own warning in this example. In order to obtain the list item entity type you will need to perform a GET query using the resource path to the list with a URI like /_api/Web/Lists/getByTitle('Customers') and then inspect the ListItemEntityTypeFullName property of the resulting data object.

DELETE Example

function onDeleteCustomer(customerId) {

  var requestUri = _spPageContextInfo.webAbsoluteUrl +
                "/_api/Web/Lists/getByTitle('Customers')/items(" + customerId + ")";

  var requestHeaders = {
    "ACCEPT": "application/json;odata=verbose",
    "X-RequestDigest": $("#__REQUESTDIGEST").val(),
    "If-Match": "*" // delete the item even if another user has updated it since we last fetched it

    url: requestUri,
    type: "DELETE",
    contentType: "application/json;odata=verbose",
    headers: requestHeaders,
    success: function(){
      alert("Deleted customer");
    error: function(){
      alert("Failed to delete customer");


SharePoint supports an optimistic concurrency scheme which is what prevents users overwriting one-another’s changes when working concurrently. It works by only updating items which at the same version as when you last fetched it. This means that if another user updates the item before you can submit your changes, then your update will fail rather than wiping the other user’s changes. The If-Match header must define the eTag (version) which can be updated by the call. The “*” allows us to ignore the scheme and just overwrite no matter what. This is okay in this scenario only because we are performing a delete action. When performing an update you will need to query for the latest eTag first.

UPDATE Example

function onUpdateCustomer(customerId, firstName) {
  // first we need to fetch the eTag to ensure we have the most recent value for it
  var requestUri = _spPageContextInfo.webAbsoluteUrl +
                "/_api/Web/Lists/getByTitle('Customers')/items(" + customerId + ")";
  // execute AJAX request
    url: requestUri,
    type: "GET",
    headers: { "ACCEPT": "application/json;odata=verbose" },
    success: function(data){
      // got the eTag
      var etag = data.d.etag;
      var requestUri = _spPageContextInfo.webAbsoluteUrl +
                "/_api/Web/Lists/getByTitle('Customers')/items(" + customerId + ")";
      // set the MERGE verb so we only need to provide the update delta
      var requestHeaders = {
        "ACCEPT": "application/json;odata=verbose",
        "X-RequestDigest": $("#__REQUESTDIGEST").val(),
        "X-HTTP-Method": "MERGE",
        "If-Match": etag
      var customerData = {
        __metadata: { "type": "SP.Data.CustomersListItem" },
        FirstName: firstName
      requestBody = JSON.stringify(customerData);
        url: requestUri,
        type: "POST",
        contentType: "application/json;odata=verbose",
        headers: requestHeaders,
        data: requestBody,
        success: function(data){
          alert("Updated customer");
        error: function(data){
          alert("Failed to update customer");
    error:  function(data){
      alert("Failed to get customer eTag");

Note how the eTag is obtained and set appropriately.

The OData specification provides two verbs for performing updates: MERGE and PUT. Using PUT is impractical as replaces the target object with the object representation that it receives. This means that any properties that are not provided in the representation will be set to null. MERGE on the other hand only updates the properties that are present on the received object representation. This more efficiently utilises bandwidth and is easier to code. However, since MERGE is not one of the verbs that is defined in the HTTP specification, using the MERGE verb may not flow through network intermediaries as seamlessly as methods that are defined in HTTP. As such, the X-HTTP-Method header is set appropriately and we perform a POST. Conceptually, this should be thought of a MERGE rather than a POST operation.

A final note
These examples assume that your code is running in the context of a SharePoint App and is only accessing data in the App Web (or alternatively is running on a page deployed as part of a full trust solution). Should you wish to use the _api in a SharePoint App to access assets outside of the App Web you will need an OAuth enabled solution in order to authenticate beyond the scope of the App. I will undoubtedly blog about this in the future. Please subscribe to my feed to catch that post when it’s written.

Understanding OData

“The Open Data Protocol (OData) is a Web protocol for querying and updating data”

as defined on the OData site here. From a web developer’s point of view, I think it can be better defined as such:

‘A standardized syntax for RESTful web services’

As such, if you don’t have solid understanding of RESTful services and calling RESTful services with jQuery then I suggest you read the previous post in this series which addresses that topic specifically. I have also written a blog on OData in SharePoint 2013 specifically which assumes the knowledge laid out here.
OData protocol – agnostic of implementation

OData logo
OData logo

OData defines a standard syntax which defines how resource representations (entities) are queried via a RESTful URI: [the base URI (eg http://server/service.svc)] + [the resource path (eg /departments)] + [the query options(eg ?$select=name)]. A standardised approach makes the service easier to work with for anyone with previous OData experience. The resource path can be determined as it is structured logically and the query options, once learned, are repeatable across services.

The resource path is used to locate a collection of entities, a single entity or a single property that the service surfaces in a hierarchical manner. E.g.

  • http://server/service.svc/departments – collection of departments
  • http://server/service.svc/departments(1) – single department
  • http://server/service.svc/organisation – single organisation
  • http://server/service.svc/organisation/name – name of the organisation

It’s worth keeping in mind that the OData protocol is fundamentally similar to SQL as it aims to achieve the same goal of data access. It is the query options that allow us to perform a subset of SQL-like queries: select ($select), where ($filter), order by ($orderby), join ($expand) and paging ($top and $skip) operations. E.g.

  • http://server/service.svc/departments?$select=name – returns only the name element for each department
  • http://server/service.svc/departments?$filter=name eq HR – returns only the HR department element
  • http://server/service.svc/departments?$orderby=name – returns all department elements sorted by name
  • http://server/service.svc/departments?$expand=directors – returns the department elements and including the department director
  • http://server/service.svc/departments?$skip=40&top=20 – returns the third page of 20 results

Multiple query options can be used at once (?$filter=staffCount gt 100&$select=name) to create complex queries. Note that a dollar sign ‘$’ character is prepended to each of OData query option parameters.

OData compliant service being consumed directly in IE
OData compliant service being consumed directly in IE

The filter query option may also support a range of math, string and date operations. It is important to remember, however, that an OData service has no obligation to support all (or any) query options.

$filter logical and arithmetic functions
Using the OData $filter query parameter – Thanks to Ted Pattison’s SPC12 slide deck for this image
$filter string functions
Using the OData $filter query parameter – Thanks to Ted Pattison’s SPC12 slide deck for this image
$filter date and math functions
Using the OData $filter query parameter – Thanks to Ted Pattison’s SPC12 slide deck for this image

Fundamentally, that is OData. The specification has a lot more to it but if you are comfortable with it at this level then it is just a matter or reading up on the spec. The blog post here expands a little on the fundamentals I’ve just described or the full spec can found here but it’s a rather difficult read as it’s raw unstyled HTML.
Please see the next post in this series on OData in SharePoint 2013 with jQuery.

RESTful services with jQuery

The goal of this article is to provide a practical understanding of RESTful services with jQuery as it is a fundamental prerequisite to working with the OOTB OData implementation in SharePoint 2013. This is the first part of a series of posts on OData in SharePoint 2013. The other parts are Understanding OData and OData in SharePoint 2013 with jQuery.

For those of you who aren’t familiar with RESTful services, it is important that you first have an understanding of this technology as it’s the basis of the OData protocol. It’s also important to have experience using jQuery to interface with services now that Microsoft advocates this approach. If you already have these skills then you may want to skip ahead to the next post in this series.

jQuery logo - sourced from
jQuery logo – sourced from

Understanding RESTful services in jQuery

REST (REpresentational State Transfer) is a design model (or architectural style) for web services. Services implementing the REST design model are considered as being ‘RESTful’. These services are accessed via HTTP URIs (as opposed to complex mechanisms like SOAP) and provide a stateless, human readable and structured way of accessing data. REST services can be utilized from a web client with only fundamental knowledge of JavaScript and JSON (or XML) and as such are inherently cache-able  These services can support CRUD (Create Read Update Delete) by leveraging well known HTTP methods – GET, POST, PUT and DELETE – and hence limit the chance of being interferred with by a firewall. So that’s all lovely, but what does it look like?

That was an example of a call to a RESTful service. We will go over it piece by piece but before that I should start by making it abundantly clear that this is jQuery – you must have included a script reference to the jQuery library. Of course you could go back to basics and implement a solution in raw JavaScript (or many other client-side technologies) if you have a specific requirement. I would think very hard before making that decision. JQuery has been adopted by Microsoft and is now included as part of many Visual Studio item templates (including SP Apps and Office 2013 Apps) and greatly improves developer efficiency as well as largely mitigating browser compatibility issues.

The $.ajax is the magic method, but before we call it we have to define an object which represents the settings as to how the ajax call should be performed.

The url property
Of primary importance is the url property. This must identify an endpoint capable of handling the HTTP request. The endpoint in this example is an HTTP handler (.ashx) but it could be implemented in one of a number of ways including as a .NET web service (.asmx), a WCF web service (.svc) or any of a number of ways outside of the Microsoft stack. Implementing services is beyond the scope of this post, however HTTP handlers are great place to start for simple services being consumed from within your web applications but you’ll want to look into WCF services for large scale and public facing API services. In theory the endpoint could even be an aspx file but it would be impractical and has a high overhead.

A REST URI is made up of three parts:
[the base URI (the end point file as just discussed)] + [the resource path] + [the query options].
The resource path in this example is ‘/items’ and defines the resource. You’ll see more on this when we get to OData. Lastly, the query string allows us to refine the result set either specifically as in this case (get me the resource with an ID equal to 8) or notionally (e.g. get me resources that have been modified in the last 12 hours).

The type property
The type property defines the action to be performed as outlined next:

  • GET allows us to Read resource representations
  • POST allows us to Create new resources
  • PUT allows us to Update existing resources
  • DELETE allows us to Delete existing resources

In the case of GET and DELETE the URI itself provides enough information to perform the action. For POST and PUT a representation of the resource is required to be sent through the wire. This is where the data property comes into play.

The data property
The data property is quite self-explanatory and as previously mentioned should only be included when performing a create or update activity. Data can be sent in a variety of formats (see below) however I would recommend using a JSON representation as it is extremely lightweight, human-readable, easily created, easily serialised (stringified) and sent, and easily deserialised (parsed) and worked with. JQuery also makes working with XML reasonably straightforward should you decide to go in that direction.

Visual representation of a JSON object - sourced from
Visual representation of a JSON object – sourced from

The contentType property
Paired with the data property is the contentType property. It is only required when then data property is present and it tells the service what kind of data to expect. Some services may work without this (especially if they only accept one data type) but it should always be included. Most commonly one would expect this value to represent json or xml but a service could be written to accept pretty much anything. However, accepting the likes of SOAP would erode the goal of a RESTful implementation. The value for this property should be an Internet Media Type. For a full list of all available types see here but it is most likely that the only ones you’ll need are these:

  • application/json
  • application/xml
  • text/html

The accept header property
The properties discussed so far are all defining the request. The remaining properties define the response (or at least how to handle the response). Defining the accept header lets the service know what kind of data you expect it to return. It should be noted that the service must take the responsibility of reading the accept header and generating the appropriate output. There is no inherent transformation occurring based on this value and services may only support limited response types.

The dataType property
The dataType property defines the type of data that is returned. This value will determine the transformation that is applied to the data returned from the service. If the service returns a stringified JSON object then a dataType value of ‘json’ will mean that the success callback method will receive a JSON object as the returned value. A value of ‘xml’ will return an xml document that can then be parsed easily using jQuery. Other values include ‘text’, ‘html’ and ‘script’ and can be further investigated via the jQuery API documentation.

The callback properties
Finally there are the two callback functions: one for success and one for when an error occurs. Without the success callback a GET call is pointless as it is here that you would define the action to take with the service data (formatted according to dataType). You may decide to omit the success callback for a POST, PUT or DELETE if you don’t need user feedback (but this is very unlikely). Hopefully you understand the premise of callbacks because I’m not going to explain these further (if not, you’ll have no trouble finding good references elsewhere on the web). If you want to find out more about the $.ajax function in general (not specific to REST) try visiting the jQuery API documentation here.

A couple of tips:

  • When testing with any web services using jQuery, make sure you run $.ajax({cache: false}); first to prevent caching responses.
  • You can type GET requests straight into your browser to view the xml/json response for ease of testing. However, you need to prevent the browser from treating it as an RSS feed. In Chrome prepend ‘view-source:’ to the URI (eg view-source:http://hostname) and in IE go: Tools > Internet Options > Content > Feeds and Web Slices > Settings > uncheck ‘Turn on feed reading view’
An example of viewing a service response in IE
An example of viewing a service response in IE

Please read the successors to this article, Understanding OData and OData in SharePoint 2013 with jQuery.

If you’ve found this useful then please subscribe to blog feed.