Tag Archives: SharePoint Online

Calling the Office 365 Unified API from JavaScript using ADAL.js

The goal of this post is to provide very basic ‘hello world’ example of how to call the Office 365 Unified API (aka Graph API) using JavaScript and the ADAL.js library. This has recently become possible (May 2015) now that CORS is supported by the Office 365 APIs (most of the individual endpoints support it as well as the unified API).

The ADAL library simplifies the process of obtaining and caching the authentication tokens required to retrieve data from Office 365. It is possible to avoid the ADAL library and handle this yourself, although I would recommend doing so as a learning exercise only.

I failed to find a simple example of how to achieve this, my search results often filled with examples of calling the APIs from server-side code or else utilising the Angular.js framework. This example is based on a more complex example.

The following snippet will log to the browser console the results of a call the to files endpoint of the Office 365 unified API, which will return a JSON object containing information about the files in the current users’ OD4B.

Before it will work you must complete the following steps (as described in detail here):

  1. Register an Azure Active Directory App. Note that *every* Office 365 subscriptions comes with AAD and supports the creation of an app
  2. Associate the required ‘permissions to other services’, in this case ‘Read users files’ via the Office 365 Unified API
  3. Allow implicit flow

Not covered explicitly in the above article but also critical are the following steps:

  • Get the App’s Client ID and copy it into the snippet
The App Client ID
The App Client ID
  • Get the Azure Active Directory subscription ID and copy it into the snippet
The subscription ID in Azure Active Directory
The subscription ID in Azure Active Directory

Once the above steps have been completed, you can try out the snippet by embedding in a Script Editor web part, or you can run it externally to SharePoint as part of, say, a provider hosted app.

NOTE: I found that the call to files endpoint is failing for certain users. I am still unsure whether this is due to external vs internal users (is working for internal [.onmicrosoft.com] users) or whether it could be licencing issue. The /beta/me endpoint is working in all cases.

Paul.

GLOSSARY

CORS: Cross-Origin Resource Sharing
ADAL: Active Directly Authentication Library
OD4B: OneDrive 4 Business

Search Schema Scoping in SharePoint Online

For solutions that are contained in a single site collection, or span a small number of site collections, or are in a tenant where the other solutions are not trusted or are unknown, then I have a strong preference to use site collection scoped search schema rather than tenant scoped.

Side note: I am yet to come across a situation where I would use site scoped search schema. In my mind, the existence of search schema at this level only serves to confuse.

Search Schema hierarchy is SharePoint Online.
Search Schema hierarchy is SharePoint Online. There is also site scoped search schema at lowest level which is not present here.

For those that aren’t fully aware, search schema (the set of managed properties that are accessible via the search framework) can be provisioned at the tenant, site collection, or site scope. These scopes are hierarchical such that managed properties are inherited from the tenant scope down to the site scope but can be overridden along the way. There are some good articles that delve into this in more detail.

By provisioning search schema at the site collection level you are mitigating the risks of errors related to other solutions changing the properties which your solution relies upon. This is especially relevant in SharePoint Online where all solutions in the tenant have to share a common set of RefinableTypeXX managed properties.

There are some important exceptions, of course.

  1. People Search, a.k.a User Profile Search, a.k.a Local People Results
    In SharePoint Online, people properties are indexed on a very slow schedule. We requested more information from Microsoft regarding this and were told that this schedule is ‘confidential’. I have found that when using site-collection scoped managed properties it can take *weeks* for them to get populated. I have found much better (although still poor) performance using tenant scoped properties (usually within a few days). Assuming you do require custom search schema for people properties I would still recommend provisioning all remaining managed properties (all those not mapped to people properties) at the site collection level.
  2. Many site collections
    Of course, having many site collections which require the same search schema is valid reason to go tenant scoped. This is purely due to management of the properties going forwards. A solid scripted deployment procedure should not care if you are provision search schema to 1 or 50 site collections – but anyone maintaining the solution will definitely care if they have update 50 schemas manually, or are suddenly required to script something which they feel should be *easy*. Even in this scenario you should still consider how much you trust other solutions in the tenant against the impact of finding out that one day your managed properties are mapped incorrectly. Depending on your solution this could lead to errors that are left undetected, or conversely obviously break your home page.

Paul.

Suite Bar logo is not displayed after applying theme

In SharePoint Online I have detected a bug that prevents the suite bar logo (configured in the o365 tenant admin center) from being rendered after applying a theme with a non-default colour scheme. The logo is not just hidden or not visible; the HTML is no longer rendered to the page at all.

NOTE: This issue has been raised with Microsoft and they have acknowledged it as a bug. They have told is that a fix is planned for June 2015 and should take a couple of weeks to reach all tenants.

Update 06/08/2015

I was advised last week that a fix this issue had officially started rolling out. Today I can confirm that one of our test tenants no longer exhibits this behavior! At the time of writing some of our tenants have not yet received the fix, but it is definitely rolling out and should be with you soon if not already.

When exactly is this occurring?

There is a colour scheme file that nominated as the default for each master page. Only when using this colour scheme in conjunction with the given master page will the logo be present. Specifically, only a colour scheme which is nominated as the default colour scheme will show the suite navigation logo.

This is unrelated to any customisation work and can be observed on any OOTB SharePoint Online publishing site.

The default colour scheme for a given master page is nominated within that master page’s preview file. The preview file is the one which sits alongside the master page in the master page gallery and has the same name name as the master page but with an extension of ‘.preview’ instead of ‘.master’ (e.g., seattle.preview). The first line of the contents of this file identifies the default colour scheme for the master page. When this colour scheme is applied, the theming engine assumes that the CSS is already themed appropriately and does *not* act.

Identifying the default colour scheme for a master page
Identifying the default colour scheme for a master page

I still want the logo on every page, what can we do?

  1. As I have said, a fix is on the way – just wait a few months!
  2. We can inject the logo using JavaScript (or CSS, but we were unable to achieve a CSS only solution which meets our responsive design requirements)
    • It is against best practice
    • This approach is brittle as the suite bar is a living, changing thing – Microsoft is updating it all the time and any changes they make will likely break this functionality.
    • The logo will no longer be configurable via tenant admin settings.
  3. We can create and use multiple master pages – one for every theme colour
    • Partially defeats the purpose of master pages which is to maintain all common pages elements in a single place.
    • Master pages may fall out of synch
    • Adds complexity for support

I want to reproduce this myself, how do I do it?

  1. Create an new site collection in a tenant that has a suite navigation logo configured, using the Publishing Site template
  2. Note that the logo is present
  3. Via Site Settings, “Change the look”
  4. Choose the “Office” composed look
    [Specifically, we want the seattle master page and colour pallette 001]
  5. Change the colour scheme to anything except the default
  6. Apply the look
  7. Note that the logo is no longer present
  8. Re-apply the theme with the default colour scheme and logo will reappear

 

Paul.

PowerShell Add-Type Error: Could not load file or assembly XXX or one of its dependencies.

If you are attempting to run the PowerShell command Add-Type with a Path parameter that references an assembly which has been emailed or downloaded from the internet (e.g. Add-Type -Path "C:\Microsoft.SharePoint.Client.dll") then it is likely that you will encounter the following error:

Could not load file or assembly ‘file:///’ or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)

PS_AddType

To confirm that the following steps are appropriate. Check the inner exception:

PS> $error[0].Exception.InnerException

“An attempt was made to load an assembly from a network location which would have caused the assembly to be
sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable
CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly,
please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more
information.”

Not using SharePoint Client Components?

First check the execution policy under which PowerShell is running. This can be done by running the Get-ExecutionPolicy cmdlet. If you are not running under an Unrestricted policy, try running the Add-type cmdlet after setting the execution policy to Unrestricted (Set-ExecutionPolicy “Unrestricted”). If this resolves the issue then you’ll need to look into getting the assembly signed by a trusted publisher if you can’t continue to run PowerShell in this lowered state of security.

Next ensure that the assembly hasn’t been blocked. The easiest way to achieve this is by right-clicking on the assembly and checking on the General tab that there isn’t the option to unblock the assembly at bottom of the dialog. There is also the Unblock-File cmldlet to achieve this action via PowerShell.

Unblocking an assembly
Unblocking an assembly

Finally, PowerShell supports a config file which by default is not present. It allows configuration of a few things like the .NET versions which are supported and, importantly for us, whether or not PowerShell is allowed to load from remote sources. In order to add/update this file, ensure that you are logged in as a local administrator. The config file needs to reside adjacent to the PowerShell exe file. So, by default, it ought to be created here: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe.config

Note: Do not confuse this with powershell_ise.exe.config although you may want to update this as well if you use ISE.

The content of the powershell.exe.config should be as follows. If you already have a file present, ensure that you merge the XML rather than just replacing it.

<?xml version="1.0"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedruntime version="v4.0.30319"/>
    <supportedruntime version="v2.0.50727"/>
  </startup>
  <runtime>
    <loadfromremotesources enabled="true"/>
  </runtime>
</configuration>

Using SharePoint Client Components?

If you are adding the SharePoint Client assemblies required for the execution of CSOM requests then you should take a different approach. Rather than bundling these assemblies with your script, instead insure that the correct SharePoint Client Components package is installed on the executing machine and reference the assemblies from their default location:

Download Client Components:
SharePoint Online Client Components
SharePoint 2013 Client Components

Reference Accordingly:
C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\assembly.dll
C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\assembly.dll

Paul.

SharePoint Online Supporting High Resolution Background Images

When you import an image to use as a background image as part of a composed look in SharePoint Online, the image is re-scaled to 1024 x 768 and compressed as low quality jpg. If you have an image which still looks reasonable after these modifications then please continue to use composed looks as the ensures that the page weight is kept to a minimum.

A full HD image
A full HD image
The same image as above after being processed by the Composed Look framework. Admittedly the compression is pretty good in this example.
The same image as above after being processed by the Composed Look framework. Admittedly the compression is pretty good in this example.

SharePoint provides no configuration around this that I am aware of and even you provide a background image URL in the composed look to a non-compressed image, the image will be compressed and the compressed image will be referenced when the composed look is applied.

The only way to support a high resolution background image is to reference it in CSS, overriding the composed look background image. I have found that the best way to do this is with the following two CSS rules. The second prevents the background being used in dialogs which occasionally don’t use the transparent overlay and can become unusable otherwise.

If the image needs to be changed later, a new image with the same name can uploaded to replace the current one. The issue with this approach is that client browsers may have cached the background image and will continue to use the cached image until the cache expires (could be years, not sure what the default cache is) or the user manually clears their browser cache. In order to force the browser to re-fetch the image a query string value should be appended (or amended, in the current example we are using ‘v=1.0’).

Note that if you are using theme-able CSS you will need to reapply the theme in order to apply changes to the CSS.

Paul.

SPO CSOM Error: For security reasons DTD is prohibited in this XML document

I found myself encountering the following error when authenticating to SharePoint Online using CSOM from PowerShell:

DTD_error
Exception calling “ExecuteQuery” with “0” argument(s): “For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.”

I believe that there a number of causes for this issue some of which are firewall and ISP related. This may only resolve a subset of the cases where this issue has been arising, even under the same circumstances.

In my scenario, I found that this issue was only arising when the credentials I was passing were being federated. That is, when the username was *not* in form <me>@<domain>.onmicrosoft.com but rather something like <me>@<domain>.co.uk. It is also possible that this issue resolves itself after a single successful authentication has occurred. Try providing credentials for a *.onmicrosoft.com account, and if that works try again with a federated account. This is discussed more later.

I used Fiddler to compare the request/response trace from a successful authentication and one where this error occurs. It turns out that somewhere internally a request is made to msoid.<full-domain> where <full-domain> is the bit after the @ symbol from the username provided. In the case where this value is of the *.onmicrosoft.com variety, a 502 error (no DNS entry) is returned with no request body and the authentication proceeds successfully. In the other case, the ‘msoid’ URL is resolved and a response with a request body is returned.
In my case the response was a 301 error (permanent relocation), however I read of cases where a 200 (success) has been received. Importantly to note, is that the response, success or otherwise, returns an HTML body containing a DTD (Document Type Declaration), and in turn produces the rather unhelpful error message.

So how do you fix it? Well one way is to provide an entry in your hosts file which ensure that the msoid URL will be invalid. I found that providing a local host entry for it worked. Your hosts file can be found here:
C:\Windows\System32\drivers\etc

I added a line which looks like the following:

127.0.0.1        msoid.<domain>.co.uk

And it worked! Intriguingly I found that if I then removed this line from my hosts file, SharePoint Online authentication from PowerShell continued to work. It is for this reason that I suggested trying to use a *.onmicrosoft.com account first at the begging of this post – just in case it resolves the issue for you without touching the hosts file. Please comment if you have any success (or otherwise) with that approach.

Hope this helps! Good luck.

SharePoint Online remote authentication (and Doc upload)

The SharePoint REST API is touted as being the tool to provide inter-platform integration with SharePoint Online. However, outside of .NET the authentication piece is not so straightforward. App authentication solves this issue for registered apps but I want to show how remote user authentication can be achieved, regardless of platform.

The goal of this post is to provide examples of the HTTP requests which need to be made in order to authenticate SharePoint Online. It then provides an example of using the same technique to upload a document and update metadata just to prove it all works 🙂

The type of applications where this kind of approach may be necessary include: a Java application, a PHP application, or JavaScript application where there is otherwise no SharePoint Online authentication context and the decision has been made (for whatever reason) that user authentication is most appropriate (as opposed to app authentication).

Edit: This approach will not work in a JavaScript environment due to cross-domain restrictions enforced by browsers (unless of course you are on the same domain, in which case you don’t need to worry about any of this anyway). The ADAL.js library is available for the cross-domain JS scenario. I have posted an example here: http://paulryan.com.au/2015/unified-api-adal/

I wrote about using the SharePoint REST API here (and background here, and here).

I will be providing examples of the requests using the ‘Advanced REST Client’ Google Chrome extension.

Authenticate

The authentication piece comes in a few steps:

  • Get the security token
  • Get the access token
  • Get the request digest

Get the security token

First we must provide a username and password of a user with Contribute access to the Roster Data library and the URL at which we want access to the SharePoint Online Security Token Service.

This is done by POSTing the following XML as the request body to:
https://login.microsoftonline.com/extSTS.srf

<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"
      xmlns:a="http://www.w3.org/2005/08/addressing"
      xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
  <s:Header>
    <a:Action s:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</a:Action>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
    <a:To s:mustUnderstand="1">https://login.microsoftonline.com/extSTS.srf</a:To>
    <o:Security s:mustUnderstand="1"
       xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <o:UsernameToken>
        <o:Username>[username]</o:Username>
        <o:Password>[password]</o:Password>
      </o:UsernameToken>
    </o:Security>
  </s:Header>
  <s:Body>
    <t:RequestSecurityToken xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
      <wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
        <a:EndpointReference>
          <a:Address>[endpoint]</a:Address>
        </a:EndpointReference>
      </wsp:AppliesTo>
      <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType>
      <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>
      <t:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</t:TokenType>
    </t:RequestSecurityToken>
  </s:Body>
</s:Envelope>

Requesting the security token
Requesting the security token

The response from the request includes the security token needed to get the access token. It is the value which has been stricken out in orange in the image below.

Response including the security token
Response including the security token

Get the access token

Once the security token has been retrieved it must be used to fetch the access token. The can be done by POSTing to the following URL with the security token as the request body:
https://yourdomain.sharepoint.com/_forms/default.aspx?wa=wsignin1.0

Request to fetch the access token, passing the security token
Request to fetch the access token, passing the security token

The response from this request includes couple of cookies which must be passed as headers with all future requests. They are marked with the ‘Set-Cookie’ header. We need the ones beginning with rtFa= and FedAuth=. They can be seen the below image of the response headers.

Response includes the access token cookies
Response includes the access token cookies

Get the request digest

The request digest is a .NET security feature that ensures any update requests are coming from a single session. It must also be included with any POST requests.

The request digest is fetched by POSTing to: https://yourdomain.sharepoint.com/_api/contextinfo
The access token cookies must be included as Cookie headers with the request as shown in the image below.

Request to fetch the request digest, passing access tokens
Request to fetch the request digest, passing access tokens

The response from the request will include the request digest in the XML response as in the image below. The entire contents of the FormDigestValue tag will required, including the date time portion and timezone offset (-0000).

Response containing the request digest value
Response containing the request digest value

Upload a document with metadata

Upload the document

Now that we have all the authentication headers we can make update calls into SharePoint Online as the user whose credentials we originally supplied when fetching the security token.

In order to upload a document perform the following POST request:
https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘list name’)
/rootfolder/files/add(url='filename.csv',overwrite=true)

A number of headers must be send with the request including the access token cookies, the request digest (X-RequestDigest) and the accept header as shown in the image below. The body of the request must contain the content of the document being uploaded.

Request to upload a document to SharePoint Online
Request to upload a document to SharePoint Online

The response of this request contains some minimal metadata about the file and can be safely ignored. However, for completeness here it is.

Response JSON from the document upload request
Response JSON from the document upload request

The unique ID property could be used to fetch the document in order to perform metadata updates rather than URL as done in the following example.

Update document metadata

The final step which needs to take place is update the document in SharePoint with the relevant metadata.

This can be done with yet another POST request. This time to the following URI:
https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘listTitle')
/rootfolder/files/getbyurl(url='serverRelFileUrl')/listitemallfields/validateupdatelistitem

All the headers sent with the previous request must be sent with this request as well. The request body is a JSON object which defines the metadata fields to be updated. The fieldname and fieldValue properties must be updated as required. Note that the fieldname property must be equal to the field internal name not the field display name. An example of this is in the image below.

Request to set metadata on a document in SharePoint
Request to set metadata on a document in SharePoint

The response from this request provides success notification for each individual field update as shown below.

Response from the document metadata update request
Response from the document metadata update request

So, this should now be enough to write an application in any server-side language which supports web requests and work against SharePoint Online. I’d love to see some implementations of this, please comment if you’ve done it.

I’d like to acknowledge the following posts as they were invaluable references:

 

Setting Default Page Layout, Available Page Layouts, and Available Web Templates via CSOM (JSOM)

The publishing infrastructure provides a site with many features including page layouts. A default page layout, a restricted set of available page layouts, and a restricted set of web templates can be set. The page in the UI looks like this:

config_pageLayouts

In order to perform these configurations programmatically, methods are provided as part of the the server side object model via the PublishingWeb object. However setting these properties via CSOM is less obvious. The equivalent methods are not available (at least not at the time of writing. It is worth noting that the CSOM model gets extended as part of CUs as well as in SPO).
The property information is stored solely as web properties. I have verified this by reflecting the Microsoft.SharePoint.Publishing assembly, and ensured that the server side object model methods perform no other action than this.

The following JSOM code demonstrates how to set the web properties with valid XML values.

The methods can be called like this:

// The page layout is defined by its filename
var filename = "NewsArticlePage.aspx";
setDefaultPageLayout(filename, onSuccess, onFailure);


// The page layouts are defined by an array of filenames
var filenames = ["NewsArticlePage.aspx"];
setAvailablePageLayouts(filenames, onSuccess, onFailure);

// The web templates are defined by their long name,
//the guid is that of the feature which deploys the web template
var webTemplates = ["{BDACFFF5-05DF-4446-9907-B4C39F15F1D7}#WT_VanitySite"];
setAvailableWebTemplates(webTemplates, onSuccess, onFailure);

 Note – these samples reference both the jQuery and underscore libraries.