I found myself encountering the following error when authenticating to SharePoint Online using CSOM from PowerShell:
Exception calling “ExecuteQuery” with “0” argument(s): “For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.”
I believe that there a number of causes for this issue some of which are firewall and ISP related. This may only resolve a subset of the cases where this issue has been arising, even under the same circumstances.
In my scenario, I found that this issue was only arising when the credentials I was passing were being federated. That is, when the username was *not* in form <me>@<domain>.onmicrosoft.com but rather something like <me>@<domain>.co.uk. It is also possible that this issue resolves itself after a single successful authentication has occurred. Try providing credentials for a *.onmicrosoft.com account, and if that works try again with a federated account. This is discussed more later.
I used Fiddler to compare the request/response trace from a successful authentication and one where this error occurs. It turns out that somewhere internally a request is made to msoid.<full-domain> where <full-domain> is the bit after the @ symbol from the username provided. In the case where this value is of the *.onmicrosoft.com variety, a 502 error (no DNS entry) is returned with no request body and the authentication proceeds successfully. In the other case, the ‘msoid’ URL is resolved and a response with a request body is returned.
In my case the response was a 301 error (permanent relocation), however I read of cases where a 200 (success) has been received. Importantly to note, is that the response, success or otherwise, returns an HTML body containing a DTD (Document Type Declaration), and in turn produces the rather unhelpful error message.
So how do you fix it? Well one way is to provide an entry in your hosts file which ensure that the msoid URL will be invalid. I found that providing a local host entry for it worked. Your hosts file can be found here: C:\Windows\System32\drivers\etc
I added a line which looks like the following:
127.0.0.1 msoid.<domain>.co.uk
And it worked! Intriguingly I found that if I then removed this line from my hosts file, SharePoint Online authentication from PowerShell continued to work. It is for this reason that I suggested trying to use a *.onmicrosoft.com account first at the begging of this post – just in case it resolves the issue for you without touching the hosts file. Please comment if you have any success (or otherwise) with that approach.
The SharePoint REST API is touted as being the tool to provide inter-platform integration with SharePoint Online. However, outside of .NET the authentication piece is not so straightforward. App authentication solves this issue for registered apps but I want to show how remote user authentication can be achieved, regardless of platform.
In a .NET environment please refer to the ADAL library for authentication rather than writing it yourself.
The goal of this post is to provide examples of the HTTP requests which need to be made in order to authenticate SharePoint Online. It then provides an example of using the same technique to upload a document and update metadata just to prove it all works 🙂
The type of applications where this kind of approach may be necessary include: a Java application, a PHP application, or JavaScript application where there is otherwise no SharePoint Online authentication context and the decision has been made (for whatever reason) that user authentication is most appropriate (as opposed to app authentication).
Edit: This approach will not work in a JavaScript environment due to cross-domain restrictions enforced by browsers (unless of course you are on the same domain, in which case you don’t need to worry about any of this anyway). The ADAL.js library is available for the cross-domain JS scenario. I have posted an example here: https://paulryan.com.au/2015/unified-api-adal/
I wrote about using the SharePoint REST API here (and background here, and here).
First we must provide a username and password of a user with Contribute access to the Roster Data library and the URL at which we want access to the SharePoint Online Security Token Service.
This is done by POSTing the following XML as the request body to: https://login.microsoftonline.com/extSTS.srf
Requesting the security token
The response from the request includes the security token needed to get the access token. It is the value which has been stricken out in orange in the image below.
Response including the security token
Get the access token
Once the security token has been retrieved it must be used to fetch the access token. The can be done by POSTing to the following URL with the security token as the request body: https://yourdomain.sharepoint.com/_forms/default.aspx?wa=wsignin1.0
Request to fetch the access token, passing the security token
The response from this request includes couple of cookies which must be passed as headers with all future requests. They are marked with the ‘Set-Cookie’ header. We need the ones beginning with rtFa= and FedAuth=. They can be seen the below image of the response headers.
Response includes the access token cookies
Get the request digest
The request digest is a .NET security feature that ensures any update requests are coming from a single session. It must also be included with any POST requests.
The request digest is fetched by POSTing to: https://yourdomain.sharepoint.com/_api/contextinfo
The access token cookies must be included as Cookie headers with the request as shown in the image below.
Request to fetch the request digest, passing access tokens
The response from the request will include the request digest in the XML response as in the image below. The entire contents of the FormDigestValue tag will required, including the date time portion and timezone offset (-0000).
Response containing the request digest value
Upload a document with metadata
Upload the document
Now that we have all the authentication headers we can make update calls into SharePoint Online as the user whose credentials we originally supplied when fetching the security token.
In order to upload a document perform the following POST request: https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘list name’)
/rootfolder/files/add(url='filename.csv',overwrite=true)
A number of headers must be send with the request including the access token cookies, the request digest (X-RequestDigest) and the accept header as shown in the image below. The body of the request must contain the content of the document being uploaded.
Request to upload a document to SharePoint Online
The response of this request contains some minimal metadata about the file and can be safely ignored. However, for completeness here it is.
Response JSON from the document upload request
The unique ID property could be used to fetch the document in order to perform metadata updates rather than URL as done in the following example.
Update document metadata
The final step which needs to take place is update the document in SharePoint with the relevant metadata.
This can be done with yet another POST request. This time to the following URI: https://yourdomain.sharepoint.com/subweb/_api/web/lists/getbytitle(‘listTitle')
/rootfolder/files/getbyurl(url='serverRelFileUrl')/listitemallfields/validateupdatelistitem
All the headers sent with the previous request must be sent with this request as well. The request body is a JSON object which defines the metadata fields to be updated. The fieldname and fieldValue properties must be updated as required. Note that the fieldname property must be equal to the field internal name not the field display name. An example of this is in the image below.
Request to set metadata on a document in SharePoint
The response from this request provides success notification for each individual field update as shown below.
Response from the document metadata update request
So, this should now be enough to write an application in any server-side language which supports web requests and work against SharePoint Online. I’d love to see some implementations of this, please comment if you’ve done it.
I’d like to acknowledge the following posts as they were invaluable references:
