PowerShell HowTo: Setting start addresses of an existing content source

In a previous post we created a completely new content source. Now we are going to modify an existing content source.

The root of all crawling

It's a simple cmdlet, Set-SPEnterpriseSearchCrawlContentSource,  taking a reference to the search application, the access protocol and the content source. The latter will be identified using its name. Here's the code:

     
  $SearchApp = Get-SPEnterpriseSearchServiceApplication
  $StartAddresses = "protocol1://localhost/?entity=sheep,protocol1://localhost/?entity=bird,protocol1://localhost/?entity=fish"
  Set-SPEnterpriseSearchCrawlContentSource -SearchApplication $SearchApp -Identity "Content Source Name" –StartAddresses $StartAddresses -CustomProtocol "protocol1"

Above code sets three start addresses for a single content source:

• protocol1://localhost/?entity=sheep
• protocol1://localhost/?entity=bird
• protocol1://localhost/?entity=fish

In code they are separated by colon. Noteworthy detail: in the configuration page for the content source these addresses will appear in reverse order.

Accessing a content source's properties

If you want to modify a content source to e.g. add start addresses instead of replacing them all you have to get the content source. Get-SPEnterpriseSearchCrawlContentSource ist the cmdlet to use:

     
  $SearchApp = Get-SPEnterpriseSearchServiceApplication
  $ContentSource = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $searchapp "Content Source Name"

The collection $ContentSource.StartAddresses contains all currently configured addresses. Combine them with yours and use the first script to update the configuration.

SharePoint 2010 SP1 changes the account used for indexing content. Not.

After updating SharePoint 2010 from RTM state (14.0.4763.1000) to Service Pack 1 (SP1, 14.0.6029.1000) a content source of the Search Service Application (SSA) suddently stopped indexing content.

The content source in question was of type CustomRepository and used a custom indexing connector to access an external system. It downloaded data via web service. And this suddenly seemed to fail.

What was going on?

Somebody cannot access something

A look into the ULS log revealed errors which happened every time the connector tried to access the web service. The crawl history showed a single top level error and the crawl log had the following entry:

"Error while crawling LOB contents. ( Credentials were not found for the current user within the target application '...'. please set the credentials for the current user. )"

The error message pointed into one direction: the Secure Store. All credentials for accessing the external web service were saved in the secure store. And one account was allowed to get these credentials. The message was suggesting that another account than the allowed one was trying to get the credentials.

But what is the "current" user? Shouldn't the user be the Default Content Access Account of the SSA as configured in the Crawl Rules?

Identity crisis

After looking into the task manager I decided to give credential access to one account: the account mssdmn.exe runs under, which is the account of the SharePoint Server Search 14 service.

And it seemed like

• before updating to SP1 the Default Content Access Account (as configured in the Crawl Rules) was used to access the secure store credentials
• after updating to SP1 this account apparently changed to the account of the SharePoint Server Search 14 service.

So the solution was simple, yet mysterious: I changed the account allowed to access the web service credentials. And it worked.

But stop!

Resolution? Confusion.

After a few days I deleted the content source previously affected and added it again. And the indexing stopped again. Same error as before: "Credentials were not found for the current user within the target application '...'. please set the credentials for the current user." What was going on this time?

The account used by Search to access the secure store credentials changed again. To what was set prior to installing SP1: the Default Content Access Account of the SSA. As one would expect.

Strange.

PowerShell HowTo: Create Content Source of Type CustomRepository

In this post I explain how to create a SharePoint content source of type CustomRepository via PowerShell. There are some pitfalls which I will highlight.

CustomRepository

Content sources of type CustomRepository are used to index external data sources which aren't supported by any of the built-in indexing connectors. These content sources are to be used in conjunction with Custom Indexing Connectors, which use a custom protocol to access the external system.

The property page of an already created content source of type CustomRepository looks like in Figure 1.

Figure 1: Content source; properties relating to type CustomRepository are highlighted

You can see that a custom connector named Custom Protocol with scheme protocol1 is used to access the external system. How to register the custom connector will be topic of another blog post.

Scripting

You can use PowerShell to script the creation of this type of content source. It is pretty straightforward if you know about the pitfalls.

Use the New-SPEnterpriseSearchCrawlContentSource cmdlet. Here is a working example:

        
  $contentSource = New-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssa -Type CustomRepository -Name "My Content Source" -CustomProtocol "protocol1" -StartAddresses "protocol1://localhost"

Pitfalls

The StartAddresses parameter is not really optional

Don't forget to specify the StartAddresses parameter! The cmdlet's documentation states that this parameter is optional. Well, this is true in the sense of skipping this parameter won't stop the content source from being created. But it will be broken:

Figure 2: "The custom connector used by the content source has been removed or undeployed."

The error message hints that the custom protocol used by the indexing connector is not available anymore. This is misleading. The correct error message would be "There are no start addresses provided and I am freaking out about it for no reason.".

There is also no way to correct this. The radio box next to Custom Protocol cannot be clicked and the OK button of the dialog is disabled. And won't be enabled again. The only way to fix the content source is to delete it.

So remember to provide a start address when creating a content source via cmdlet. 

Don't use start addresses with empty host part

Using a start address like "protocol1://" (with empty host part) can also lead to the above error. I say "can" because this sometimes seemed to work, sometime not.

So to be sure you should always specify a host in your start address.