“SafeQueryPropertiesTemplateUrl” error when calling the SharePoint _api

If you call the SharePoint 2013 REST API in your applications ensure that any requests originating from the client are sent from the current web base URL to avoid returning a SafeQueryPropertiesTemplateUrl error.

If the current site is
https://tenant.sharepoint.com/sites/mysitecollection/subsite1/subsite2 then it is very important that you submit API requests as
https://tenant.sharepoint.com/sites/mysitecollection/subsite1/subsite2/_api
and NOT as any of:

  • https://tenant.sharepoint.com/_api or
  • https://tenant.sharepoint.com/sites/mysitecollection/_api or even
  • https://tenant.sharepoint.com/sites/mysitecollection/subsite1/_api

The reason for this is that the current user must have access to the site addressed by the base URL of the API request (the bit before the _api). If the user cannot access this site then the request will fail. Unfortunately it doesn’t fail in the manner you might expect (i.e. a 401 access denied exception). A request that fails in this manner will return a 500 error. The specific exception details are as follows:

Exception class:
Microsoft.Office.Server.Search.REST.SearchServiceException

Exception message:
The SafeQueryPropertiesTemplateUrl "The SafeQueryPropertiesTemplateUrl "{0}" is not a valid URL." is not a valid URL.

SafeQueryPropertiesTemplateUrl Error received when attempting to hit _api endpoint from web to which the current user does not have access
Error received when attempting to hit _api endpoint from web to which the current user does not have access

Paul.

HTTP 405 Error in .NET Web App

I was recently told that an web app I had developed was returning an HTTP 405 error upon being freshly deployed. It took me way too long to realise that cause of the issue came down to missing files. Specifically, the complete folder structure had been deployed however the files at the top level web root were missing. These are files are rather critical.

They are the web.config and global.asax

If you are seeing this error, ensure these files have been deployed correctly and aren’t corrupt as a first point of call.

Receiving an HTTP 405 in IE11
Receiving a 405 in IE11

For SEO HTTP 405

  • Chrome: The page you are looking for cannot be displayed because an invalid method (HTTP verb) is being used.
  • IE: HTTP 405 The website has a programming error. This error (HTTP 405 Method Not Allowed) means that Internet Explorer was able to connect to the website, but the site had a programming error.
  • Edge: HTTP 405 error That’s odd… Microsoft Edge can’t find this page

Paul.

DateTime validation message colour

There is a minor style bug in SharePoint 2013 (including SharePoint Online). The error message on a required DateTime field is not displayed in a manner consistent with other control validation errors. No it’s not just you, and no it’s not due to some conflicting CSS – it is a SharePoint bug.

Specifically I am referring to the page layout edit experience. A user fails to provide a value for a required DateTimeField control and the validation message is shown in the default text colour – ‘You must specify a value for this required field.’

redvalidation

For all other validation messages the SharePoint controls add the ms-formvalidation which sets a CSS rule to set the red colour. This is the only rule which the ms-formvalidation class sets and as such this is the only rule that should be applied to fix the issue.

I use the following CSS selector to resolve this issue:

Paul.

Dynamically generating complex pre-refined search result page URLs

I while ago I blogged about creating a static link to a pre-refined (pre-filtered) search page. This post follows that idea to it’s natural conclusion by providing a number of JavaScript functions which can dynamically create search result page URLs. These URLs will look something like this:

https://tenant.sharepoint.com/search#Default=%7B%22k%22%3A%22article%22%2C%22r%22%3A%5B%7B%22n%22%3A%22RefinableString20%22%2C%22t%22%3A%5B%22%5C%22%C7%82%C7%824275696c64%5C%22%22%5D%2C%22o%22%3A%22OR%22%2C%22k%22%3Afalse%2C%22m%22%3A%7B%22%5C%22%C7%82%C7%824275696c64%5C%22%22%3A%22Build%22%7D%7D%2C%7B%22n%22%3A%22RefinableString21%22%2C%22t%22%3A%5B%22%5C%22%C7%82%C7%824c6f6e646f6e%5C%22%22%5D%2C%22o%22%3A%22OR%22%2C%22k%22%3Afalse%2C%22m%22%3A%7B%22%5C%22%C7%82%C7%824c6f6e646f6e%5C%22%22%3A%22London%22%7D%7D%5D%7D

The provided scripts support filtering on:

  • a search term
  • multiple refiners
  • multiple values for a refiner, or
  • any combination of the above

It would be worth reading the intro of my earlier article to get a better understanding of what is happening in the snippets provided in this post.

Default Enterprise Search Centre
Default Enterprise Search Centre

OF NOTE:

  • As the most common usage will surely be to produce search result page URLs that are refined on a single value, I have written an ‘overload’ function that simplifies calling the method in this scenario
  • The ‘search page URL’ can be provided to the functions in a number of ways including:
    • “/search” : to the web. The default page for that web. In the case of an Enterprise Search Centre this will be the ‘Everything’ search results page
    • “/search/Pages/peopleresults.aspx” : to the page
    • Use an absolute URL if you are out of the context of the SharePoint Online tenant in which the search page resides. This will be true for provider hosted add-ins (apps)
    • If you are writing your own refiner, then pass an empty string and set window.location.hash to the result of the function
  • This script has no dependencies on other libraries (jQuery, SP.js, etc)
  • The hex encoded string must be UTF-8 encoded. JavaScript is natively UTF-16. The particular scenario where this raised an issue for me was the wide-ampersand character which is often used instead of a standard ampersand as it is XML friendly. ‘unescape’ returns a UTF-8 encoded string and is used to force the required encoding. Thanks to ecmanaut for this solution
  • I took inspiration for the stringToHex method from a post by pussard

The functions:

var getPreRefinedSearchPageUrl = function (searchPageUrl, searchTerm, managedPropertyName, managedPropertyValue) {
  return getComplexPreRefinedSearchPageUrl({
    searchPageUrl: searchPageUrl,
    searchTerm: searchTerm,
    refiners: [
      {
        managedPropertyName: managedPropertyName,
        managedPropertyValues: [
          managedPropertyValue
        ]
      }
    ]
  });
};

// input:
// {
//   searchPageUrl: "/search/Pages/results.aspx",
//   searchTerm: "",
//   refiners: [
//     {
//       managedPropertyName: "RefinableString08",
//       managedPropertyValues: [
//         "Human Resources"
//       ]
//     }
//   ]
// }
var getComplexPreRefinedSearchPageUrl = function (data) {
  var searchObj = {
    "k": data.searchTerm,
    "r": []
  };
  for (var i = 0; i < data.refiners.length; i++) {
    var refiner = data.refiners[i];
    var searchObjRefiner = {
      "n": refiner.managedPropertyName,
      "t": [],
      "o": "OR",
      "k": false,
      "m": {}
    };
    for (var j = 0; j < refiner.managedPropertyValues.length; j++) {
      var refinerValue = refiner.managedPropertyValues[j];
      // Force UTF8 encoding to handle special characters, specifically full-width ampersand
      var managedPropertyValueUTF8 = unescape(encodeURIComponent(refinerValue)); 
      var managedPropertyValueHex = stringToHex(managedPropertyValueUTF8);
      var managedPropertyValueHexToken = "\"ǂǂ" + managedPropertyValueHex + "\"";
      searchObjRefiner.t.push(managedPropertyValueHexToken);
      searchObjRefiner.m[managedPropertyValueHexToken] = refinerValue;
      searchObj.r.push(searchObjRefiner);
    }
  }
  var seachObjString = JSON.stringify(searchObj);
  var searchObjEncodedString = encodeURIComponent(seachObjString);
  var url = data.searchPageUrl + "#Default=" + searchObjEncodedString;
  return url;
};

var stringToHex = function (tmp) {
  var d2h = function (d) {
    return d.toString(16);
  };
  var str = '',
    i = 0,
    tmp_len = tmp.length,
    c;
  for (; i < tmp_len; i += 1) {
    c = tmp.charCodeAt(i);
    str += d2h(c);
  }
  return str;
};

These are examples of how to call the function that are defined above.

var complexUrl = getComplexPreRefinedSearchPageUrl({
  searchPageUrl: "/search/Pages/results.aspx",
  searchTerm: "article",
  refiners: [
    {
      managedPropertyName: "RefinableString20",
      managedPropertyValues: [
        "Build", "Land"
      ]
    },
    {
      managedPropertyName: "RefinableString21",
      managedPropertyValues: [
        "London"
      ]
    }
  ]
});
var basicUrl = getPreRefinedSearchPageUrl("/search/Pages/results.aspx", "", "RefinableString20", "Build");

Paul.

Calling the Office 365 Unified API from JavaScript using ADAL.js

The goal of this post is to provide very basic ‘hello world’ example of how to call the Office 365 Unified API (aka Graph API) using JavaScript and the ADAL.js library. This has recently become possible (May 2015) now that CORS is supported by the Office 365 APIs (most of the individual endpoints support it as well as the unified API).

The ADAL library simplifies the process of obtaining and caching the authentication tokens required to retrieve data from Office 365. It is possible to avoid the ADAL library and handle this yourself, although I would recommend doing so as a learning exercise only.

I failed to find a simple example of how to achieve this, my search results often filled with examples of calling the APIs from server-side code or else utilising the Angular.js framework. This example is based on a more complex example.

The following snippet will log to the browser console the results of a call the to files endpoint of the Office 365 unified API, which will return a JSON object containing information about the files in the current users’ OD4B.

Before it will work you must complete the following steps (as described in detail here):

  1. Register an Azure Active Directory App. Note that *every* Office 365 subscriptions comes with AAD and supports the creation of an app
  2. Associate the required ‘permissions to other services’, in this case ‘Read users files’ via the Office 365 Unified API
  3. Allow implicit flow

Not covered explicitly in the above article but also critical are the following steps:

  • Get the App’s Client ID and copy it into the snippet
The App Client ID
The App Client ID
  • Get the Azure Active Directory subscription ID and copy it into the snippet
The subscription ID in Azure Active Directory
The subscription ID in Azure Active Directory

Once the above steps have been completed, you can try out the snippet by embedding in a Script Editor web part, or you can run it externally to SharePoint as part of, say, a provider hosted app.

NOTE: I found that the call to files endpoint is failing for certain users. I am still unsure whether this is due to external vs internal users (is working for internal [.onmicrosoft.com] users) or whether it could be licencing issue. The /beta/me endpoint is working in all cases.

Paul.

GLOSSARY

CORS: Cross-Origin Resource Sharing
ADAL: Active Directly Authentication Library
OD4B: OneDrive 4 Business

Search Schema Scoping in SharePoint Online

For solutions that are contained in a single site collection, or span a small number of site collections, or are in a tenant where the other solutions are not trusted or are unknown, then I have a strong preference to use site collection scoped search schema rather than tenant scoped.

Side note: I am yet to come across a situation where I would use site scoped search schema. In my mind, the existence of search schema at this level only serves to confuse.

Search Schema hierarchy is SharePoint Online.
Search Schema hierarchy is SharePoint Online. There is also site scoped search schema at lowest level which is not present here.

For those that aren’t fully aware, search schema (the set of managed properties that are accessible via the search framework) can be provisioned at the tenant, site collection, or site scope. These scopes are hierarchical such that managed properties are inherited from the tenant scope down to the site scope but can be overridden along the way. There are some good articles that delve into this in more detail.

By provisioning search schema at the site collection level you are mitigating the risks of errors related to other solutions changing the properties which your solution relies upon. This is especially relevant in SharePoint Online where all solutions in the tenant have to share a common set of RefinableTypeXX managed properties.

There are some important exceptions, of course.

  1. People Search, a.k.a User Profile Search, a.k.a Local People Results
    In SharePoint Online, people properties are indexed on a very slow schedule. We requested more information from Microsoft regarding this and were told that this schedule is ‘confidential’. I have found that when using site-collection scoped managed properties it can take *weeks* for them to get populated. I have found much better (although still poor) performance using tenant scoped properties (usually within a few days). Assuming you do require custom search schema for people properties I would still recommend provisioning all remaining managed properties (all those not mapped to people properties) at the site collection level.
  2. Many site collections
    Of course, having many site collections which require the same search schema is valid reason to go tenant scoped. This is purely due to management of the properties going forwards. A solid scripted deployment procedure should not care if you are provision search schema to 1 or 50 site collections – but anyone maintaining the solution will definitely care if they have update 50 schemas manually, or are suddenly required to script something which they feel should be *easy*. Even in this scenario you should still consider how much you trust other solutions in the tenant against the impact of finding out that one day your managed properties are mapped incorrectly. Depending on your solution this could lead to errors that are left undetected, or conversely obviously break your home page.

Paul.

Follow documents from search hover panel

There is a somewhat confusing logic behind when the FOLLOW button is displayed on the search results hover panel (a.k.a document preview).

A document hover panel with both the POST and FOLLOW buttons present
A document hover panel with both the POST and FOLLOW buttons present

What I am talking about?

If you are building a solution that relies on the following of documents but you are using Yammer rather than the SharePoint social feed then you may be wondering why, from the search results hover panel, you can follow pages, users, sites, but not most document types.

NB. If you are finding that you can’t following anything, check that web scoped feature ‘Follow Content’ has been activated on each site which contains content you wish to be able to follow.

NB. You can still follow the document types in question by clicking ‘view in library’ and using the library item menu to follow.

In many cases, wanting both POST and FOLLOW doesn’t make a lot sense as a primary reason of following documents is to populate the activity feed which is not available when Yammer is being used as the enterprise social experience. As such, please consider why you want this behaviour at all. In my scenario the user’s list of followed documents is promoted to the home page and bookmarking documents is a key user story.

What is going on?

The search results hover panel is built from a number of display templates which you can read about in more depth here (TechNet) or here (Chris O’Brien) or many other places.

Importantly, there is a display template which defines the common actions (buttons) across the bottom of the hover panel and when to display them. The display template is called Item_CommonHoverPanel_Actions and can be found here:

Site Settings > Master Pages and Page Layouts > Display Templates > Search > Item_CommonHoverPanel_Actions.html

If you inspect this display template you will find an if else block around the rendering of the POST and FOLLOW buttons. The logic can be summarised as:
The POST button is visible if Yammer is enabled and the result type supports it, otherwise the FOLLOW button is visible if the result type supports it, at no time will both buttons be visible.

If you download a copy of the display template HTML file, update it to remove the ‘else’ as in the code snippet below, and then upload it again, you will find the both the POST and FOLLOW buttons will be displayed in the search hover panel when supported. Success!

But is it okay update that file?

The short answer is yes. Take care as this file is used by every hover panel in SharePoint (to my knowledge, there may be some completely unique ones) and so changes could break something that isn’t obvious.

The major risk is that if Microsoft decide to update the hover panel which require them to produce a new version of the display template file (they have done this previously when introducing the POST button). In the case that you have modified this file, then your changes will be lost. This can happen without warning (unless you have a second tenant on first release to catch these issues before they hit production – you should be doing this!).
For very minor updates such as this, and to support non-critical functionality, it may be okay to make these changes and be prepared to re-implement them should Microsoft issue an update.

The alternative is to make a copy of the display template with a new name. This approach means that your changes will not get overridden but it also means that your solution will not get the updates that would otherwise be pushed to this file. We call this ‘customisation tax’ and it is a trade off as to which way you’d rather push changes.
In this particular scenario this latter approach is not very practical as every result type references the existing display template. You would be required to make copies of all the result type display templates that are applicable (possibly a dozen or more), and update the result types themselves to use your new templates. Unless you are bypassing result types and using a single display template for all results, this feels overly complex for such a minor change, but major changes will necessitate the effort.

EDIT: A colleague of mine, Luis Manez, pointed out that with a little JS you can force a custom hover panel to be rendered for all result types. You can read about it (approach one) and some other approaches to associating custom hover panels here (Elio Struyf).

Paul.

Increase Search Result Limit Beyond 50

Content Search Web Parts (CSWPs) and Search Result Web Parts (SRWPs, a.k.a Result Script Web Parts RSWPs) prevent more than 50 results being returned at once. This is true with or without paging enabled. This is true even if you provision the web part using XML. In older versions of SharePoint, it may allow you to enter a row limit greater than 50 but will then default to a mere 10 results.

increaseSearchResultsRowlimit

I’m sure Microsoft has very good UX and technical reasons behind this limitation, however at 100 items I don’t feel that either UX nor performance suffer. I imagine that if using this technique for returning large page sizes (>100) that there is very real potential for bad performance and the UX is most likely appalling.

Importantly – this code should only be considered a learning tool. This code is entirely unsupported and generally just a really bad idea. Please be responsible 🙂

Recognition to Matt Stark who provided this solution. All I have done is rewritten it a bit for safety and I wanted to prefix it with a bit more discussion than it had. Original discussion is here.

The web parts which you want to act upon must be set to load asynchronously (this is not the default mode).
Edit Web Part > Change Query > Settings > Loading Behaviour :: Async option

Edit Web Part > Change Query > Settings > Loading Behaviour :: Async option
Edit Web Part > Change Query > Settings > Loading Behaviour :: Async option

Take note:

  1. There are a few available solutions to this, none of which are much good, and even this one is should be considered a hack and be treated with caution.
  2. I will only discuss this option as it is the best I have come across when considering the limitations of SharePoint Online.
  3. Please only include this code on specific pages using a Content Editor or a Script Editor, and NOT globally.
  4. I have amended the original code such that it only changes the row limit for those web parts which are set to return 50 items exactly.
  5. I haven’t found any issues with this code, however I am using on a page that does nothing but display a list and in a situation where failure has limited impact on the system.
  6. Apparently doesn’t work for anonymous users

And finally the code…

// <script type="text/javascript">
var CC = window.CC || {};
CC.CORE = CC.CORE || {};

CC.CORE.IncreaseSearchResultsMax = (function () {
    "use strict";

    var $ocreate = null;
    var newMaxItems = 100;
    var oldMaxItems = 50; // web part must be set to show this many items

    // on application initialization 
    // steal the global create variable and 
    // intercept calls to create UI widgets.
    Sys.Application.add_init(function() {
        $ocreate = $create;
        $create = updateResultCountCreate;
    });

    // listen to UI widget calls for CBS & DP
    var updateResultCountCreate = function (a,b){
        var ps = Array.prototype.slice.call(arguments, 0);  
        if(a === Srch.ContentBySearch && b.numberOfItems === oldMaxItems) { 
            b.numberOfItems = newMaxItems;
        }
        if(a === Srch.DataProvider && b.resultsPerPage === oldMaxItems) {
            b.resultsPerPage = newMaxItems;
        }
        $ocreate.apply(this,ps);  // apply the original $create method that we stole
    };
    return true;
})();
// </script>

Paul.

SPO CSOM Error: For security reasons DTD is prohibited in this XML document

I found myself encountering the following error when authenticating to SharePoint Online using CSOM from PowerShell:

DTD_error
Exception calling “ExecuteQuery” with “0” argument(s): “For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.”

I believe that there a number of causes for this issue some of which are firewall and ISP related. This may only resolve a subset of the cases where this issue has been arising, even under the same circumstances.

In my scenario, I found that this issue was only arising when the credentials I was passing were being federated. That is, when the username was *not* in form <me>@<domain>.onmicrosoft.com but rather something like <me>@<domain>.co.uk. It is also possible that this issue resolves itself after a single successful authentication has occurred. Try providing credentials for a *.onmicrosoft.com account, and if that works try again with a federated account. This is discussed more later.

I used Fiddler to compare the request/response trace from a successful authentication and one where this error occurs. It turns out that somewhere internally a request is made to msoid.<full-domain> where <full-domain> is the bit after the @ symbol from the username provided. In the case where this value is of the *.onmicrosoft.com variety, a 502 error (no DNS entry) is returned with no request body and the authentication proceeds successfully. In the other case, the ‘msoid’ URL is resolved and a response with a request body is returned.
In my case the response was a 301 error (permanent relocation), however I read of cases where a 200 (success) has been received. Importantly to note, is that the response, success or otherwise, returns an HTML body containing a DTD (Document Type Declaration), and in turn produces the rather unhelpful error message.

So how do you fix it? Well one way is to provide an entry in your hosts file which ensure that the msoid URL will be invalid. I found that providing a local host entry for it worked. Your hosts file can be found here:
C:\Windows\System32\drivers\etc

I added a line which looks like the following:

127.0.0.1        msoid.<domain>.co.uk

And it worked! Intriguingly I found that if I then removed this line from my hosts file, SharePoint Online authentication from PowerShell continued to work. It is for this reason that I suggested trying to use a *.onmicrosoft.com account first at the begging of this post – just in case it resolves the issue for you without touching the hosts file. Please comment if you have any success (or otherwise) with that approach.

Hope this helps! Good luck.