OAuth On-Behalf-Of Flow: Getting a user access token without user interaction. Includes ADAL sample.

OAuth 2.0 (and hence Azure Active Directory) provides the On-Behalf-Of flow to support obtaining a user access token for a resource with only a user access token for a different resource – and without user interaction.
This supports the scenario where a secured Web API acts as an interface to other resources (a.k.a endpoints) secured by the same identity provider and that require user context. As a practical example, a mobile client accesses some resources via a middle tier API which provides services such as data processing, caching, API simplification/optimisation, joining of datasets, etc.

The OAuth flow that achieves this is called the On-Behalf-Of flow; this makes sense as we’re facilitating the middle tier to act on behalf of the client when it accesses the resources farther down.

Using the on-behalf-of flow to access a resource via a middle tier API

Some background

Authentication with an OAuth 2.0 identity provider (such as AAD) produces JWT tokens. These tokens include information such as which claims (permissions) the user should be granted and the particular resource at which the token is valid (such as graph.microsoft.com). The OAuth 2.0 framework is specified such that a given token can only ever be valid for a single resource. This means that the token received by an endpoint (such as an Azure App Service Web API) cannot be used to directly authenticate to ‘another resource’. This is because the token’s resource will be that of the Web API and not the ‘other resource’. To see this I recommend checking out jwt.io and cracking open some tokens.

For completeness, the ‘other resource’ could be accessed using app-only authentication if it supports it, and if user context is not required (i.e. the return value will be the same regardless of the user) although this may greatly increase complexity in a multi-tenant scenario.

Configuring AAD for on-behalf-of

Before we get to the code the first hurdle is configuring AAD app registrations correctly. Initially it may be tempting to consider having both the Client and Web API layers utilise a single AAD app registration. After all, they are same holistic ‘app’ and how else can we get a user to consent to the permissions required by the Web API app when there is no interactive interface at that point? The latter point is resolved by explicitly binding the app registrations so that both are consented to as one. I mention how this is done below. By having two app registrations the flexibility of configuration is improved; we can have a Native app registration for the client and a Web API app registration for the Web API, we can have implicit flow configured for one app and not the other, and generally have granular control over configuration. Most vitally, an app registration can’t issue tokens valid for its own resource so two app registrations is a requirement.

I’ll avoid stepping though the configuration of the app registrations here as this is available elsewhere including this GitHub project. I will give a high level overview of what needs to happen.

1. Create app registration for the Web API
  - Assign permissions to the downstream resources (e.g. Microsoft Graph, a custom Web API, etc)
  - If supporting multi-tenant authentication ensure availableToOtherTenants is set to true in the manifest
2. Create app registration for the Client
  - Assign permissions to the app registration created above for the Web API
  - If supporting multi-tenant authentication ensure availableToOtherTenants is set to true in the manifest
3. Associate app registrations
  - In the manifest for Web API app registration, configure knownClientApplications to reference the App ID for the app registration created for the Client. E.g. "knownClientApplications": ["9XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXc"]
    This binds the app registrations such that the Web API app registration is consented to as part of a single consent dialog displayed to a user when they authenticate to the Client app registration.

Before and after the app registrations are associated. Note how ‘Access Mobile App Backend’ is no longer present and instead is expanded to show the individual permissions required by that app.

Access Token Broker code

The following code is a .NET example of how to use the Active Directory Authentication Library (ADAL) to achieve the On-Behalf-Of flow.

using Microsoft.IdentityModel.Clients.ActiveDirectory; using System.Security.Claims; using System.Threading.Tasks; namespace PR.Azure.Core.AccessTokenBrokers { public class AdalAccessTokenBroker //: IAccessTokenBroker { public async Task<string> GetAccessTokenOnBehalfOfAsync( string accessToken, string appId, string appKey, string resource) { ClientCredential clientCred = new ClientCredential(appId, appKey); string tenantClaim = "http://schemas.microsoft.com/identity/claims/tenantid"; string authority = string.Format("https://login.microsoftonline.com/{0}", ClaimsPrincipal.Current.FindFirst(tenantClaim)?.Value); // NOTE : Implement a Redis token cache or token will be fetched from AAD every time. // E.G. https://blogs.msdn.microsoft.com/mrochon/2016/09/19/using-redis-as-adal-token-cache/ //TokenCache tokenCache = new RedisTokenCache(ClaimsPrincipal.Current.FindFirst(ClaimTypes.NameIdentifier)?.Value); //AuthenticationContext authContext = new AuthenticationContext(authority, tokenCache); AuthenticationContext authContext = new AuthenticationContext(authority); UserAssertion userAssertion = new UserAssertion( accessToken, "urn:ietf:params:oauth:grant-type:jwt-bearer", ClaimsPrincipal.Current.FindFirst(ClaimTypes.Upn)?.Value); // NOTE : May want to handle retry in case of transient errors AuthenticationResult result = await authContext.AcquireTokenAsync(resource, clientCred, userAssertion); return result?.AccessToken; } } }

Credits and further reading

This GitHub project was a very useful resource and a recommended starting point:
https://github.com/Azure-Samples/active-directory-dotnet-webapi-onbehalfof-ca

Microsoft docs OAuth 2.0 On-Behalf-Of flow
https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols-oauth-on-behalf-of

OAuth 2.0 specification
https://tools.ietf.org/html/rfc6749

Redis Token Cache example
https://blogs.msdn.microsoft.com/mrochon/2016/09/19/using-redis-as-adal-token-cache/

Vardhaman Deshpande – always helpful
http://www.vrdmn.com/

I was inspired to write this post not because this information isn’t available but because the information is hard to find if you aren’t familiar with the term “On-Behalf-Of”. Hopefully this post will be found by those of you searching for terms like “trade access token for new resource”, “change token resource”, “use access token with multiple resources/endpoints”, “access Microsoft Graph via Web API”, etc.

Paul.

Request Signing, Amazon Product Advertising API, .NET C#

The Amazon Product Advertising API documentation provides some code samples for its use but none using ASP.NET. A personal interest brought me to play with it and as it wasn’t entirely trivial to create a signed request as required associate authentication I thought I’d share some working code samples.

Some notes

The API does surface a WSDL file and as such a Web Reference could be used to generate classes to interact with the API. The sample I am providing here does not take advantage of this and is instead submitting raw REST requests.

I see the most valuable part of this sample as the request signing piece. This sample should not be seen as a best practice for interacting with the API but rather as a utility for request signing.

The order of the query string parameters that are included in the signed string is crucial. They must be ordered by character code (in practise this equates to alphabetically, but with all upper case letters coming before any lower case letters). The API documentation suggests string splitting, sorting, and string joining. This is definitely the approach I would take if you find yourself writing queries that use parameters dynamically but I struggle to see the use-case. This sample just uses a hard-coded string with the relevant parameters in the correct order.

Although I haven’t looked in detail yet, the approach taken to sign requests here appears very similar if not identical to that required by the Instagram API, and I am sure many other (social media) APIs.

Requests to APIs which require the signing of a secret key cannot be made securely directly from the client (e.g. using JavaScript) as it would require your secret key to be available in plain text on the client. If you want to run ajax commands against the API you need execute requests to an intermediary service. This is the approach that the sample code below facilitates.

You can read about the Amazon Product Advertising API here: Product Advertising API

The code

Below you will find a class called the AmazonApiHelper. Further below is an ashx HttpHandler as an example of calling the utility functions provided by the helper class. You’ll need to provide you own values for the following constants:
private const string awsSecretKey = "Your secret key goes here"; private const string awsAccessKeyId = "Your access key Id goes here"; private const string associateTag = "Your associate tag goes here";

The helper class

using System; using System.Security.Cryptography; using System.Linq; using System.Text; using System.Web; using System.IO; using System.Net; using System.Web.Script.Serialization; using System.Xml.Linq; namespace AmazonApp.Utilities { public class AmazonApiHelper : IDisposable { #region Constants private const string EndPoint = "webservices.amazon.com"; private const string RequestUri = "/onca/xml"; private const string qsService = "AWSECommerceService"; private const string qsOperation = "ItemSearch"; private const string qsSearchIndex = "All"; private const string qsResponseGroup = "Images,ItemAttributes,Offers"; private const string canonicalQsFormat = "AWSAccessKeyId={2}&AssociateTag={3}&Keywords={5}&Operation={1}&ResponseGroup={6}&SearchIndex={4}&Service={0}&Timestamp={7}"; private const string dateFormat = "yyyy-MM-ddTHH:mm:ss.000Z"; private const string stringToSignFormat = "GET\n{0}\n{1}\n{2}"; private const string signedUriFormat = "http://{0}{1}?{2}&Signature={3}"; #endregion private HMACSHA256 hmac = null; public string AwsSecretKey { get; private set; } public string AwsAccessKeyId { get; private set; } public string AssociateTag { get; private set; } public AmazonApiHelper(string associateTag, string awsAccessKeyId, string awsSecretKey) { AssociateTag = associateTag; AwsAccessKeyId = awsAccessKeyId; AwsSecretKey = awsSecretKey; } public string GetRequestUri(string keywords) { string qsKeywords = HttpUtility.UrlPathEncode(keywords); string qsTimestamp = DateTime.UtcNow.ToString(dateFormat).Replace(":", "%3A"); string canonicalQs = string.Format(canonicalQsFormat, qsService, qsOperation, AwsAccessKeyId, AssociateTag, qsSearchIndex, qsKeywords, qsResponseGroup.Replace(",", "%2C"), qsTimestamp); string stringToSign = string.Format(stringToSignFormat, EndPoint, RequestUri, canonicalQs); byte[] hashedSecretString = hmacSHA256(stringToSign, AwsSecretKey); string qsSignature = Convert.ToBase64String(hashedSecretString).Replace("+", "%2B").Replace("=", "%3D"); string signedUri = string.Format(signedUriFormat, EndPoint, RequestUri, canonicalQs, qsSignature); return signedUri; } public string ExecuteWebRequest(string uri, string keywords) { string responseJson = null; WebRequest webRequest = WebRequest.CreateHttp(uri); using (HttpWebResponse webResponse = webRequest.GetResponse() as HttpWebResponse) using (Stream dataStream = webResponse.GetResponseStream()) using (StreamReader reader = new StreamReader(dataStream)) { string responseFromServer = reader.ReadToEnd(); responseJson = GetWebResponseJson(uri, keywords, responseFromServer, webResponse); } return responseJson; } #region Utilities private string GetXmlValue(XElement el, params string[] elNames) { if(el == null) { return null; } if(elNames == null || elNames.Length < 1) { return el.Value; } XElement currentNode = el; foreach (string elName in elNames) { currentNode = currentNode.Elements().FirstOrDefault(e => e.Name.LocalName == elName); if(currentNode == null) { // Path is not valid return null; } } string valueOfFinalEl = currentNode.Value; return valueOfFinalEl; } private string GetWebResponseJson(string requestUri, string keywords, string responseFromServer, HttpWebResponse webResponse) { var xmlDoc = XDocument.Parse(responseFromServer); var itemsEl = xmlDoc.Descendants().First(d => d.Name.LocalName == "Items"); var itemEls = itemsEl.Elements().Where(e => e.Name.LocalName == "Item"); var items = itemEls.Select(i => new { asin = GetXmlValue(i, "ASIN"), productUrl = GetXmlValue(i, "DetailPageURL"), productImgUrl = GetXmlValue(i, "MediumImage", "URL") ?? GetXmlValue(i.Descendants().FirstOrDefault(e => e.Name.LocalName == "MediumImage"), "URL"), title = GetXmlValue(i, "ItemAttributes", "Title"), price = GetXmlValue(i, "OfferSummary", "LowestNewPrice", "FormattedPrice"), offersUrl = GetXmlValue(i, "Offers", "MoreOffersUrl") }); JavaScriptSerializer jsonSerializer = new JavaScriptSerializer(); var json = jsonSerializer.Serialize(new { keywords = keywords, // requestUri = requestUri, responseArray = items, statusDescription = webResponse.StatusDescription, statusCode = webResponse.StatusCode.ToString(), isError = webResponse.StatusCode != HttpStatusCode.OK, isFromCache = webResponse.IsFromCache }); return json; } public static string GetEmptyResponseJson(string keywords, string message, HttpStatusCode code) { JavaScriptSerializer jsonSerializer = new JavaScriptSerializer(); var json = jsonSerializer.Serialize(new { keywords = keywords, // requestUri = requestUri, responseArray = new object[0], statusDescription = message, statusCode = code.ToString(), isError = code != HttpStatusCode.OK, isFromCache = false }); return json; } private byte[] hmacSHA256(string data, string key) { if (hmac == null) { hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key)); } return hmac.ComputeHash(Encoding.UTF8.GetBytes(data)); } #endregion #region IDisposable Support private bool disposedValue = false; protected virtual void Dispose(bool disposing) { if (!disposedValue) { hmac.Dispose(); disposedValue = true; } } public void Dispose() { Dispose(true); } #endregion } }

Calling the helper class from a web handler

<%@ WebHandler Language="C#" Class="Handler" %> using System; using System.Net; using System.Web; using RankazonSPA.Utilities; public class Handler : IHttpHandler { private const string awsSecretKey = "xXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX"; private const string awsAccessKeyId = "xXxXxXxXxXxXxXxXxXxX"; private const string associateTag = "xXxXxXxXxXxXxX-21"; public void ProcessRequest(HttpContext context) { var keywords = context.Request.QueryString["k"]; string responseJson = string.Empty; if (string.IsNullOrEmpty(keywords)) { responseJson = AmazonApiHelper.GetEmptyResponseJson(string.Empty, string.Empty, HttpStatusCode.OK); } else { using (AmazonApiHelper apiHelper = new AmazonApiHelper(associateTag, awsAccessKeyId, awsSecretKey)) { string requestUri = apiHelper.GetRequestUri(keywords); try { responseJson = apiHelper.ExecuteWebRequest(requestUri, keywords); } catch (Exception ex) { responseJson = AmazonApiHelper.GetEmptyResponseJson(keywords, ex.Message, HttpStatusCode.InternalServerError); } } } context.Response.ContentType = "application/json"; context.Response.Write(responseJson); } public bool IsReusable { get { return false; } } }

Good luck advertising those products!

Paul.

SPO CSOM Error: For security reasons DTD is prohibited in this XML document

I found myself encountering the following error when authenticating to SharePoint Online using CSOM from PowerShell:

Exception calling “ExecuteQuery” with “0” argument(s): “For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.”

I believe that there a number of causes for this issue some of which are firewall and ISP related. This may only resolve a subset of the cases where this issue has been arising, even under the same circumstances.

In my scenario, I found that this issue was only arising when the credentials I was passing were being federated. That is, when the username was *not* in form <me>@<domain>.onmicrosoft.com but rather something like <me>@<domain>.co.uk. It is also possible that this issue resolves itself after a single successful authentication has occurred. Try providing credentials for a *.onmicrosoft.com account, and if that works try again with a federated account. This is discussed more later.

I used Fiddler to compare the request/response trace from a successful authentication and one where this error occurs. It turns out that somewhere internally a request is made to msoid.<full-domain> where <full-domain> is the bit after the @ symbol from the username provided. In the case where this value is of the *.onmicrosoft.com variety, a 502 error (no DNS entry) is returned with no request body and the authentication proceeds successfully. In the other case, the ‘msoid’ URL is resolved and a response with a request body is returned.
In my case the response was a 301 error (permanent relocation), however I read of cases where a 200 (success) has been received. Importantly to note, is that the response, success or otherwise, returns an HTML body containing a DTD (Document Type Declaration), and in turn produces the rather unhelpful error message.

So how do you fix it? Well one way is to provide an entry in your hosts file which ensure that the msoid URL will be invalid. I found that providing a local host entry for it worked. Your hosts file can be found here:
C:\Windows\System32\drivers\etc

I added a line which looks like the following:

127.0.0.1 msoid.<domain>.co.uk

And it worked! Intriguingly I found that if I then removed this line from my hosts file, SharePoint Online authentication from PowerShell continued to work. It is for this reason that I suggested trying to use a *.onmicrosoft.com account first at the begging of this post – just in case it resolves the issue for you without touching the hosts file. Please comment if you have any success (or otherwise) with that approach.

Hope this helps! Good luck.

SharePoint Online remote authentication (and Doc upload)

The SharePoint REST API is touted as being the tool to provide inter-platform integration with SharePoint Online. However, outside of .NET the authentication piece is not so straightforward. App authentication solves this issue for registered apps but I want to show how remote user authentication can be achieved, regardless of platform.

In a .NET environment please refer to the ADAL library for authentication rather than writing it yourself.

The goal of this post is to provide examples of the HTTP requests which need to be made in order to authenticate SharePoint Online. It then provides an example of using the same technique to upload a document and update metadata just to prove it all works 🙂

The type of applications where this kind of approach may be necessary include: a Java application, a PHP application, ~~or JavaScript application~~ where there is otherwise no SharePoint Online authentication context and the decision has been made (for whatever reason) that user authentication is most appropriate (as opposed to app authentication).

Edit: This approach will not work in a JavaScript environment due to cross-domain restrictions enforced by browsers (unless of course you are on the same domain, in which case you don’t need to worry about any of this anyway). The ADAL.js library is available for the cross-domain JS scenario. I have posted an example here: https://paulryan.com.au/2015/unified-api-adal/

I wrote about using the SharePoint REST API here (and background here, and here).

I will be providing examples of the requests using the ‘Advanced REST Client’ Google Chrome extension.

Authenticate

The authentication piece comes in a few steps:

Get the security token
Get the access token
Get the request digest

Get the security token

First we must provide a username and password of a user with Contribute access to the Roster Data library and the URL at which we want access to the SharePoint Online Security Token Service.

This is done by POSTing the following XML as the request body to:
https://login.microsoftonline.com/extSTS.srf

The response from the request includes the security token needed to get the access token. It is the value which has been stricken out in orange in the image below.

Get the access token

Once the security token has been retrieved it must be used to fetch the access token. The can be done by POSTing to the following URL with the security token as the request body:
https://yourdomain.sharepoint.com/_forms/default.aspx?wa=wsignin1.0

Request to fetch the access token, passing the security token

The response from this request includes couple of cookies which must be passed as headers with all future requests. They are marked with the ‘Set-Cookie’ header. We need the ones beginning with rtFa= and FedAuth=. They can be seen the below image of the response headers.

Response includes the access token cookies

Get the request digest

The request digest is a .NET security feature that ensures any update requests are coming from a single session. It must also be included with any POST requests.

The request digest is fetched by POSTing to: https://yourdomain.sharepoint.com/_api/contextinfo
The access token cookies must be included as Cookie headers with the request as shown in the image below.

Request to fetch the request digest, passing access tokens

The response from the request will include the request digest in the XML response as in the image below. The entire contents of the FormDigestValue tag will required, including the date time portion and timezone offset (-0000).

Upload a document with metadata

Upload the document

Now that we have all the authentication headers we can make update calls into SharePoint Online as the user whose credentials we originally supplied when fetching the security token.

In order to upload a document perform the following POST request:
https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘list name’) /rootfolder/files/add(url='filename.csv',overwrite=true)

A number of headers must be send with the request including the access token cookies, the request digest (X-RequestDigest) and the accept header as shown in the image below. The body of the request must contain the content of the document being uploaded.

Request to upload a document to SharePoint Online

The response of this request contains some minimal metadata about the file and can be safely ignored. However, for completeness here it is.

Response JSON from the document upload request

The unique ID property could be used to fetch the document in order to perform metadata updates rather than URL as done in the following example.

Update document metadata

The final step which needs to take place is update the document in SharePoint with the relevant metadata.

This can be done with yet another POST request. This time to the following URI:
https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘listTitle') /rootfolder/files/getbyurl(url='serverRelFileUrl')/listitemallfields/validateupdatelistitem

All the headers sent with the previous request must be sent with this request as well. The request body is a JSON object which defines the metadata fields to be updated. The fieldname and fieldValue properties must be updated as required. Note that the fieldname property must be equal to the field internal name not the field display name. An example of this is in the image below.

Request to set metadata on a document in SharePoint

The response from this request provides success notification for each individual field update as shown below.

Response from the document metadata update request

So, this should now be enough to write an application in any server-side language which supports web requests and work against SharePoint Online. I’d love to see some implementations of this, please comment if you’ve done it.

I’d like to acknowledge the following posts as they were invaluable references: