SharePoint Search Out of the Box is NOT Enough

Introduction

With SharePoint 2013/16, Microsoft offers a significant increase over previous versions in Search functionality that might suggest that it is the ultimate Enterprise Search solution. Many organizations like to think about SharePoint 2013 Search as a “working out-of-the-box” solution, rather than a platform.

Considering SharePoint 2013/16 Search as a ready-to-go solution might seem to be easy, but organizations that take the approach of “Search as a platform” are much more successful with their Search projects. Unfortunately, many Search needs are not easily met using SharePoint 2013/16 out of the box. Luckily, companies can extend the SharePoint platform using third-party tools to much more fully meet their Search requirements. There are three key areas where many companies find that SharePoint 2013/16 out of the box Search does not meet their needs:

  • Content: Your solution needs to connect to it, classify it and make it easily available to users regardless of where the content resides.

  • Search management and maintenance: SharePoint 2013/16 Search is complex and advanced Search management or maintenance operations requires writing PowerShell script.

  • User experience: Users want a USEFUL Search experience or they’ll complain. They need the ability to quickly,easily, and meaningfully refine search results, they want custom visualizations of the results and they want to be able to save queries to reuse.

Content

One of the most important pillars of Enterprise Search in SharePoint 2013/16 is that SharePoint can connect to many kinds of content from many different sources. When considering and planning Search, the first step is to define the content sources to get connected to. This can be file repositories, document management systems, databases, ERP or CRM applications, or any other target applications.

Getting Connected

In order to find any results in Search, we must connect SharePoint 2013/16 to the Content Sources. SharePoint 2013/16 offers several connectors out-of-the-box (OOTB), for example, to SharePoint, file shares, Exchange Public Folders or websites. We can also get connected to application databases and Web Services by using Business Connectivity Services (BCS).

OOTB SharePoint connectivity options mentioned above provide a good “starting kit,” although real-world organizations often have other types of systems where content with high business value is stored. Integrating these external content sources into SharePoint Search can be critical to increase employees productivity and provide better findability in a shorter time. Which is the whole purpose of a good search platform.

But getting connected to these external systems is not always easy. Although SharePoint offers Business Connectivity Services, it’s very limited in permission management and performance at enterprise scale.

The real solutions usually lie in custom connectors, which can connect to the particular external content source system, usually by using its standard API's or direct database access. After enumerating the content, the connector sends it to the standard SharePoint Content Processor for further processing.

However, writing these connectors has several challenges. One has to know not only SharePoint and the best practices associated with SharePoint development, but also the connected system’s data and security models, APIs, database architecture, etc. Besides the data model, understanding the permission management may also be critical as many systems have different, non-AD (Active Directory) based security models with custom permission features. These all have to get mapped to the users in SharePoint who will initiate Search. Providing security trimmed results is always a must.

Classifying and Unifying Cross-System Data

Getting connected and being able to “pull in” the content is essential, although not enough. The next challenge is to classify and unify data coming from various systems.

Classification is the part of content processing during which we put additional metadata on the content (in the Search index) based on its existing characteristics or content. This requires extracting the existing content, transforming it by our pre-defined rules and generating the new metadata to be added.

Preparing and generating unified data with auto-classification solutions is obvious, but the data coming from heterogeneous systems can be even more heterogeneous. The names of metadata fields, as well as the value sets, might be different, which results in a confusing user experience in the end. This is why we need to unify everything. With this, classification tools can help again with transforming the heterogeneous data into the needed unified standard(s). However, SharePoint’s out-of-the-box capabilities are very limited, thus we need to look to third-party tools to fill in these gaps.

Hybrid Scenarios

In hybrid environments, we feed Search data coming from on-premises as well as from the cloud. It is a challenge to unify the information from hybrid environments – but (surprisingly or not), getting connected is almost as a big a challenge. Federation is the most widely used technique, but it leaves us with some difficult compromises (for example, the lack of aggregating results from on-prem and cloud using out-of-the-box SharePoint).

Unfortunately, getting “really” connected and providing a unified user experience is still not possible by using out-of-the box components only. To achieve this, a custom connector is needed.

Search First

Besides the “traditional” Search-driven integration scenarios, there’s one more scenario to discuss. In many cases, content migration is a long and painful process. Providing a transparent, solid experience to users is a huge challenge. They have to know what has already been migrated to the new environment and what can still be found in legacy systems. Moreover, this list constantly changes as the migration process goes forward.

The "Search First" approach comes into the picture now. Using Search First adds both the legacy and the new systems to Search, so users don’t have to know where the content they’re searching for is stored. They just have to use Search, which is always up-to-date by default.

You can use the Search First approach when you migrate content to SharePoint or Office 365 (from file shares, or from another content management system). More generally, however, it also can be used when migration happens from any system to any system – by connecting both to SharePoint Search.

Search Management and Maintenance

Due to the complexity of SharePoint Search, its management and maintenance is also complex. Although SharePoint provides administration options on the Central Administration as well as on Site Collection and Site Administration levels, these admin UIs don’t support all the capabilities and options.

If we want to do advanced Search management or maintenance operations, then we have to write PowerShell scripts. Also, if we want to make an operation sequence repeatable, for example on staging environments (development, test, production), then we again need PowerShell.

In SharePoint 2013/16, the complexity gets even bigger than in previous SharePoint version. Due to the delegated Search administration on Site Collection and Site levels, we lose the option to control and manage everything on one aggregated and integrated user interface. While this is a significant improvement in terms of delegation to power users within the business, it also creates a large gap, from both a maintenance and governance perspective. For example, getting the full list of Managed Properties (MP) across the farm we would need to write a script that enumerates the farm- level MPs as well as the MPs defined on each site collection.

 
 

Besides the provided results and their relevance, what end users mostly care about is the user experience: how they can interact with Search, how they can filter, reorder and group the results, and how these results get presented.

 
 

User experience is critical from the perspective of end users: if they don’t like it, they complain, strike or even ban using it - even if Search provides the perfect results. We have to plan it carefully, configure wisely and choose the proper add-ons if and where needed.

Moreover, we might need a different user interface in different use cases. The marketing department, for example, seeks information that may be quite different from Manufacturing or Distribution. IT developers need different documents than sales. In a pre-sales phase, there may be a need for different customer information than what is needed in a sales follow-up. All of these use cases might require access to similar content sources, but with different filters and different views. This is one of the most important things to understand: the synergy of users, context and content.

With this holistic approach, the first thing we have to identify is how many and what kind of search user interfaces we need (search centers, search pages, embedded results, etc.) and what their main goals are (users, context and content). Who will use it? In what context? Which content do they expect as results? As a next step, we have to analyze the requirements and needs for each, one by one, separately.

Custom Refinement

One of the most important modules that provide interactivity on the user interface is the Refinement Panel. SharePoint provides a set of default refinements when creating our Search Center, but they are no more than handy examples. File Type, Author, Last Modified Date (all part of "Dublin Core" metadata) – this is what we get.

Of course these might be useful, but these refinements are seldom enough. Every organization has its own, specific set of business data that needs to be used in Search too, in order to provide good findability. We discussed the importance of good and consistent unified metadata earlier. Presenting some of the metadata elements as refinements on the UI is at least as important as generating them. These refinements have to make sense to the end users. They have to be presented in a useful and ergonomic way.

What we get out of the box in SharePoint is a set of display templates for refiners. These display templates define how the refiners have to be displayed, one by one. We can also create our own display templates, of course, given that they are defined in HTML and JavaScript. Or, in more complex scenarios like a hierarchical display or chart, graph or geographical (map) refiners, we can add in third-party solutions.

Query Operations

Typical users have repeating queries in which they need the results every now and then. SharePoint 2013/16 provides a feature called "Subscription" to support this need: users can get alerts when a new result appears in the index for their queries.

But, if the users would like to see the result set of the same query again and again on the Search UI, then they have to re-enter the query repeatedly. Saving and sharing queries is a feature that is available only in third-party solutions.

Search Based Applications

The powerful concept of Search Based Applications is getting more and more attention with the need of extending SharePoint’s out-of-the-box capabilities getting stronger and stronger.

Conclusions

SharePoint Search is often considered a “solution” that provides a “ready-to-go” solution to the organization’s search needs. However, it’s more beneficial to think about it as a platform that has some out-of-the-box capabilities, but even more importantly, it can be extended to support and fulfill every organization’s unique needs.

In this article we discussed the most important points where this type of extensibility is most commonly needed and confirmed that, with proper requirements gathering and planning, the limits of Enterprise Search in SharePoint are the sky! But getting there can be a rough path sometimes. To make it easier, you have to address several key considerations, including:

  • Getting connected to third-party systems, without OOTB connectors in SharePoint

  • Auto-tagging and classifying data in search, unifying tags and classification across various systems’ content

  • Providing a solution to on-prem/online hybrid scenarios

  • Providing custom refinement controls like charts, bars or even maps

  • Providing custom display of results like charts, bars or maps, as well as complex SBAs

  • Repeatable query options

  • Search management and maintenance

One option is to develop your own, custom solution to each, but this way might be very painful and needs a deep understanding of each Search component and their relationships. In most cases, Enterprise Search can be most easily and effectively extended to meet all but the most extreme cases through the judicious use of some of the array of third-party add-on solutions.

Making a case to acquire and implement 1 or more of these solutions is almost certain to pay big dividends for YOUR organization!

by Lorne Rogers Comment