SharePoint 2010 Enterprise Search and FS4SP

FAST Search for SharePoint Relevancy - Part 1

2012-01-12T19:21:00.001-08:00

Topic: FAST Search for SharePoint 2010 Relevancy.

Subject: Understanding Relevancy and FS4SP.

Problem: When I perform a search the items are not returned in the order I expect?

Response: Relevancy is fairly complex but FS4SP has some very powerful capabilities once a user understands how relevancy can be tailored to meet a specific organizations requirements. The rank of any item under index is made up of 2 distinct parts:

1. Static Rank

2. Dynamic Rank

Total Rank = Static Rank + Dynamic Rank

Relevancy is the order which items will be displayed based on their individual rank when queried from the index.

Static Rank:

The static rank is determined at CRAWL time and will not change unless an item is re-crawled and environmental factors have changed since the last crawl. The static rank is calculated from 4 components.

1. Urldepthrank: Rank points given to boost shorter URLs.

2. Docrank: Rank points given based on the number of and relative importance of links pointing to an item.

3. Siterank: Rank points given based on the number of and relative importance of links pointing to the items on a site.

4. Hwboost: FAST Search Server 2010 for SharePoint placeholder for generic usage of quality rank points.

Dynamic Rank:

The dynamic rank is determined at QUERY time and can change for any item retrieved from the index depending on how the item is retrieved. The majority of dynamic rank is calculated from several components.

1. Managed Property Context: Associated with Full Text index map.

2. Freshness: Age of a document

3. Managed Property Field Boosts: Additional rank points give to specific managed property values.

4. Authority Weight (Anchor Text): Weight associated with anchor text associated with hyperlinks

5. Query Authority Weight: Click through relevancy weight

6. Stop Word Threshold: Associated with how “Managed Property Context” is calculated.

*** Side Notes:

1. Rank is not Unique

Many items within an index will have the same Rank. When rank is calculated many items may have the same criteria which when calculated will have the same rank number.

2. Rank is Dynamic

A single item will not have the same rank for every search performed. Rank is dynamic meaning an item’s rank will change depending on how the item is retrieved from the index. Example: With OBB settings, if an item is retrieved from the index based on a hit on the title it will have a higher rank than if it is retrieved by a hit on the body or other managed properties.

The good news is these are all adjustable to meet the needs of different organizations.

In the solution\example section I will focus on #1 Managed Property Context for a couple of reasons. 1) Brevity. Trying to write a blog showing a hands-on example of how relevancy works completely would be extremely long (I will try to follow up on other relevancy topics), 2) there happens to be an OOB issue with one setting Managed Property Context settings.

Solution\Example:

1. Let's take a look at the Managed Property Context and the relationship to rank and relevancy.

2. Open SharePoint Central Administration

a. General Application Settings

b. Search -> Farm Search Administration

c. Select the FAST Query SSA

d. Left hand Navigation Select FAST Search Administration

e. Select Managed properties

f. Select the first managed property.

i. In my case I select “Account”

g. Scroll to the bottom and select “View Mappings”

3. This is the OOB Full-text Index Map used by the Managed Property Context.

a. As can be seen the map is broken into 7 levels.

b. Any managed property can be added to this map.

c. Dynamic rank points are added based on the search performed.

Example: A search is performed and 2 items are returned from the index. Item A was returned because the search term was found in the title managed property and Item B was returned because the search term was found in the body managed property. When calculating the total rank Item A would receive more rank point than Item B based on the Full-text Index Map. Remember there are several components that comprise the total rank but assuming the two items had identical scores from all other rank components Item A would outrank Item B in terms of relevancy and therefore be presented first in the search center.

4. Let’s look at how the Managed Property Context calculates Rank and contributes to the total rank.

5. Setup Content Source

a. I have setup a content source which contains 3 documents:

RELEVANCE SAMPLE – ONE.pdf

RELEVANCE SAMPLE – TWO.doc

RELEVANCE SAMPLE – THREE.docx

*** Side Note: For the purpose this example you want to make sure the content of each file will not interfere with the search criteria. “LEVELTEST” or “SAMPLE” except in title and metadata.

b. I have made sure that everything about the 3 Files are the same (Modified Dates, etc) and how I crawl them (by putting them in the same content source) are identical to ensure that all the relevancy will be the same except for the Full-Text Index Mapping

c. I created 3 crawled properties to associate with the Documents when crawling.

i. LEVEL1

ii. LEVEL2

iii. LEVEL3

***Side Note: I am using a custom crawler to crawl my content source but I could have just as easily created a document library with 3 custom fields. If you use a document library you will end up with more data in the index but you should still be able to achieve the same end results.

d. I created 3 Managed properties using the same name as my crawled properties. I have mapped them to the crawled properties and added each of them to the Full-Text Index Map.

***Side Note: If you are not created the crawled properties via PowerShell or another tool you may need to populate all the custom fields and crawl first before being able to setup the managed property.

New Properties:

New Full-Text index Map:

e. For this pass I have only populated 1 crawled property for each item in my content source.

i. RELEVANT TEST – ONE.pdf

1. Crawled Property: LEVEL1 Value: LEVELTEST

ii. RELEVANT TEST – TWO.doc

1. Crawled Property: LEVEL2 Value: LEVELTEST

iii. RELEVANT TEST – THREE.docx

1. Crawled Property: LEVEL3 Value: LEVELTEST

6. Crawl the Content Source

7. Open a FAST Search

a. Perform a Search for the term “SAMPLE”

b. I have modified my search center to display 4 managed properties: RANK, LEVEL1, LEVEL2, and LEVEL3.

c. With this search all 3 of my items where returned based on the “Title” property hit. With the freshness date and static rank properties being equal all 3 items have the same Rank of 1991 and therefore the same relevancy. Do not be surprised if your results display in a different order. This is the order in which my items made it into the index. With the same rank the order is the position the item went into the index. FIFO.

8. Change the Search Term to “LEVELTEST”

a. From what we have talked about in the Full-Text Index Map the order shouldn’t change. Each will be retrieved but from their associated Managed Property: LEVEL1, LEVEL2, and LEVEL3. We expect LEVEL3 to have a higher priority than LEVEL2 and LEVEL2 higher than LEVEL1.

b. The search Results reveal a different story. We should have seen exactly what we expected. We can see that our Rank for each individual item has changed based on how they matched to the Full-Text index map. Seeing the different ranks between the 2 search terms on the same items shows how rank and relevancy is truly dynamic.

9. Next we will take a look how Rank is calculated and why we didn’t get our expected results.

10. I am going to use a fantastic tool on CodePlex FAST Search fro Sharepoint 2010 Query Logger developed by MIKAEL SVENSON who has a worthwhile reading blog @ http://techmikael.blogspot.com/.

a. I have modified the source a little for the purpose of this blog and will point out where.

b. There is no need to make the modification if you are following the steps. I made the change for clarity.

c. Static Score = urldepthrank + docrank + siterank + hwboost.

d. The original output has Static Score = urldepthrank + docrank + siterank + hwboost + context score.

***Side Note: We will see later that Managed Property Context isn’t completely dynamic but has a static portion to it.

e. I also broken out the individual components of the static score

11. Download and Start the Query Logger tool by double clicking on the FS4SPQueryLogger.exe.

12. Click Start Logging

13. Resubmit Search Terms: “LEVELTEST”

14. We should have captured the query in the Query Logger.

a. Select the Term and Click on the Rank Log Tab

b. Note the highlighted notes showing how total Rank is calculated in this example.

Title: RELEVANCE SAMPLE - ONE

Query term: 'leveltest'

Context score.................: 236

Number of hits/score................: 1/20

Importance level/score.................: 1/216

XRANK score...................: 0 Term: fileextension:csv

XRANK score...................: 0 Term: fileextension:zip

XRANK score...................: 0 Term: fileextension:rtf

XRANK score...................: 0 Term: fileextension:vsd

XRANK score...................: 0 Term: fileextension:oft

XRANK score...................: 0 Term: fileextension:msg

XRANK score...................: 0 Term: fileextension:txt

XRANK score...................: 0 Term: isemptylist:true

XRANK score...................: 0 Term: islistitem:true

Static rank score...............: 270

urldeptrank.......: 270

docrank...........: 0

siterank..........: 0

hwboost...........: 0

Freshness score...............: 1045

Total Rank score............: 1551

**** TOTAL Rank = Context Score + XRANK Score + Static Score + Freshness Score ****

1551 = 236 + 0 + 270 + 1045

############################

Title: RELEVANCE SAMPLE - THREE

Query term: 'leveltest'

Context score.................: 164

Number of hits/score................: 1/20

Importance level/score.................: 3/144

XRANK score...................: 0 Term: fileextension:csv

XRANK score...................: 0 Term: fileextension:zip

XRANK score...................: 0 Term: fileextension:rtf

XRANK score...................: 0 Term: fileextension:vsd

XRANK score...................: 0 Term: fileextension:oft

XRANK score...................: 0 Term: fileextension:msg

XRANK score...................: 0 Term: fileextension:txt

XRANK score...................: 0 Term: isemptylist:true

XRANK score...................: 0 Term: islistitem:true

Static rank score...............: 270

urldeptrank.......: 270

docrank...........: 0

siterank..........: 0

hwboost...........: 0

Freshness score...............: 1045

Total Rank score............: 1479

**** TOTAL Rank = Context Score + XRANK Score + Static Score + Freshness Score ****

1479 = 164 + 0 + 270 + 1479

############################

Title: RELEVANCE SAMPLE - TWO

Url:

Query term: 'leveltest'

Context score.................: 92

Number of hits/score................: 1/20

Importance level/score.................: 2/72

XRANK score...................: 0 Term: fileextension:csv

XRANK score...................: 0 Term: fileextension:zip

XRANK score...................: 0 Term: fileextension:rtf

XRANK score...................: 0 Term: fileextension:vsd

XRANK score...................: 0 Term: fileextension:oft

XRANK score...................: 0 Term: fileextension:msg

XRANK score...................: 0 Term: fileextension:txt

XRANK score...................: 0 Term: isemptylist:true

XRANK score...................: 0 Term: islistitem:true

Static rank score...............: 270

urldeptrank.......: 270

docrank...........: 0

siterank..........: 0

hwboost...........: 0

Freshness score...............: 1045

Total Rank score............: 1407

**** TOTAL Rank = Context Score + XRANK Score + Static Score + Freshness Score ****

1407 = 92 + 0 + 270 + 1121

############################

15. The only difference between the 3 items is the Context Score “Managed Property Context”

16. Looking closer at the Context Score

Title: RELEVANT TEST - ONE

Context score.................: 236

Number of hits/score................: 1/20

Importance level/score..............: 1/216

############################

Title: RELEVANT TEST - THREE

Context score.................: 164

Number of hits/score................: 1/20

Importance level/score..............: 3/144

############################

Title: RELEVANT TEST - TWO

Context score.................: 92

Number of hits/score................: 1/20

Importance level/score..............: 2/72

############################

a. Item 1

i. Earned 20 points for a single hit in the Full-Text index Map.

Number of hits/score................: 1/20

ii. Earned 216 Points for a hit in Level 1

Importance level/score..............: 1/216

iii. Total 236 = 20 + 216

b. Item 2

i. Earned 20 points for a single hit in the Full-Text index Map.

Number of hits/score................: 1/20

ii. Earned 144 Points for a hit in Level 3

Importance level/score..............: 3/144

iii. Total 164 = 20 + 144

c. Item 3

i. Earned 20 points for a single hit in the Full-Text index Map.

Number of hits/score................: 1/20

ii. Earned 92 Points for a hit in Level 3

Importance level/score..............: 2/72

iii. Total 92 = 20 + 72

17. Now that we know how it is calculated the “BIG” question becomes “How did a ‘Lowest Priority’ hit jump the ‘Very Low priority’ AND ‘Low priority’ hits in the Context Map?”

18. Let’s look deeper into how the points are granted and not just how the points are calculated and applied to the total rank (or relevance)

19. On a FAST Server Open a FAST Command Shell as Administrator

20. Execute:

$RankProfile = Get-FASTSearchMetadataRankProfile –Name default

$content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}

$content

FullTextIndexReference : content

ProximityWeight : 140

ContextWeight : 50

21. Execute:

$content.GetImportanceLevelWeight(1)

$content.GetImportanceLevelWeight(2)

$content.GetImportanceLevelWeight(3)

$content.GetImportanceLevelWeight(4)

$content.GetImportanceLevelWeight(5)

$content.GetImportanceLevelWeight(6)

$content.GetImportanceLevelWeight(7)

OOB you should get the values

Level1 - 30

Level2 - 10

Level3 - 20

Level4 - 30

Level5 - 40

Level6 - 50

Level7 - 60

22. It is apparent where the problem is. The Weight on level 1 is set equal to the weight on level 4.

23. Create a new Rank Profile

a. It is always recommended you create and use a new RankProfile when making changes. This gives you the ability to compare differences between Profiles when making changes and before implementing any changes in production.

b. From the FAST Command Shell Execute:

New-FASTSearchMetadataRankProfile -name default1

c. Any new Rank Profile will inherit from the default Profile unless specified

i. You can execute the commands from #20 and #21 replacing “default” with “default1”.

ii. The two rank profiles should be identical

24. Update the “default1” profile to set Importance Level1 to an appropriate value

a. I choose to set the Level 1 weight to 5. I could have just as easily changed them to 10 through 70 but for this example I want to change as little as possible.

b. From the FAST Command Shell Execute:

$RankProfile = Get-FASTSearchMetadataRankProfile -Name default1

$content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}

$content.SetImportanceLevelWeight(1, 5)

$content.Update()

$content.GetImportanceLevelWeight(1)

25. Expose new Profile in the Search Center

a. On the Search Center edit the “Search Action Links” web part and enable the new default rank profile.

26. Change the Sort by to the new “default1” profile and execute the Search “LEVELTEST”

a. Notice quickly that the results are now as expected.

27. Rank Log Results

Title: RELEVANCE SAMPLE - THREE

Query term: 'leveltest'

Context score.................: 164

Number of hits/score................: 1/20

Importance level/score.............: 3/144

Total Rank score............: 1479

############################

Title: RELEVANCE SAMPLE - TWO

Query term: 'leveltest'

Context score.................: 92

Number of hits/score................: 1/20

Importance level/score.............: 2/72

Total Rank score............: 1407

############################

Title: RELEVANCE SAMPLE - ONE

Query term: 'leveltest'

Context score.................: 56

Number of hits/score................: 1/20

Importance level/score.............: 1/36

Total Rank score............: 1327

############################

28. I used a pretty simple example to show how the Managed Property Context map works so I will use another quick example.

29. I populate all three Crawled properties with the value of “LEVELTEST” for all three documents and re-crawled.

30. The follow are the search results and rank.

31. Note the difference in the Rank Calculation.

a. All Levels where hits matched the Full-Text Index map contribute to the Rank.

b. This will not always be the case. There are some advance situations where not all levels will be applied depending on the search term and how many items are under index. I will leave the advanced settings and how, when, and why for a follow-up post.

Title: RELEVANCE SAMPLE - THREE

Query term: 'leveltest'

Context score.................: 278

Number of hits/score................: 3/26

Importance level/score.................: 3, 2, 1/252

Total Rank score............: 1593

############################

Title: RELEVANCE SAMPLE - TWO

Query term: 'leveltest'

Context score.................: 278

Number of hits/score................: 3/26

Importance level/score.................: 3, 2, 1/252

Total Rank score............: 1593

############################

Title: RELEVANCE SAMPLE - ONE

Query term: 'leveltest'

Context score.................: 278

Number of hits/score................: 3/26

Importance level/score.................: 3, 2, 1/252

Total Rank score............: 1593

############################

32. Let’s re-visit some the Weight Properties whether from the content level or the individual importance levels.

FullTextIndexReference : content

ProximityWeight : 140

ContextWeight : 50

a. As you probably noticed weights to not directly relate directly to points. The weights within individual Dynamic Rank calculations are based on how important they are to other rank calculations in the overall Rank calculation. If we changed the “ContextWeight” from 50 to 100 we would see the Rank Points produced from the Managed Property Context double meaning it would become more important in the overall rank calculation.

33. Final Important Note: The Managed Property Context is considered part of the dynamic portion of relevancy but it does have a static portion to it. The Full-Text Index Mappings are static. If you want to add or re-arrange the Map you must re crawl the content for it to take effect.

Conclusion: The Managed Property Context is one portion of how Rank is calculated and relevancy is determined. I never tell people what they should do but it is pretty obvious that the Importance Level 1 is not set correctly OOB. The Managed Property Context can be tailored to help an organization improve relevancy from the stand point of what managed properties are added to the Full-Text index map and how much the Managed Property Context itself and individual levels should weigh against other relevancy factors. It is extremely difficult to adjust relevancy with 20 million items in the index but it is possible. If I had tried to look at the examples I provided with 20 million items I probably would not have noticed the erroneous setting. Fortunately the Managed Property Context is a part of the dynamic ranking calculation and multiple Rank Profiles are available so trying adjustments and comparing results most times does not required re-crawling content.

KORITFW

FS4SP Primary Indexer Redundancy

2011-10-05T00:45:00.000-07:00

Topic: SharePoint 2010, FS4SP, Fast Search Server 2010

Subject: Recovering from Disaster, Primary Indexer Redundancy, FS4SP Backup Strategy?

Problem: The FS4SP server for my Primary Index Column has become corrupt and un-stable and my FS4SP has an un-recoverable error. What is the fastest way to recovery? What backup plan should I have put in place?

Response:

Most customers tend to focus on Search uptime and therefore setup backup columns for the primary index columns. If a FS4SP primary column server encounters an un-recoverable error the backup columns become responsible for serving queries but what about indexing? Content is changing and new content is being created daily. We can’t just stop indexing and we certainly can’t index once we have lost a primary column.

There are three responses to this problem and all revolve around repairing or rebuilding the primary index column.

1. Rebuild the Primary Column Server and perform a Full Re-crawl.

a. This would be acceptable for a small corpus but if you are crawling millions of items this could take days or even weeks to recover.

2. Rebuild the Primary Column Server and use the FS4SP backup and restore procedures from the last backup.

a. This would probably take less time than option #1 but comes with a different set of issues.

i. The Crawl databases from the FAST Content Connector SSA would also need to be backed up and restored.

ii. If the Crawl databases were backed up after by FS4SP index or vice versa, the actual index could contain more or less items than the Crawl Databases.

iii. If the Crawl databases and index can’t be backed up at idle time so they have the identical content the preference would be to have the Crawl databases backed up earlier. In this case when the Content SSA is up and running an incremental would pick up items which already exist in the FS4SP index but those items would merely replace those that already exist.

3. Prompt a backup index column to become a primary column.

a. This would allow the quickest turn around to have the index being populated with content again.

As option #3 would be the quickest road to recovery I will use the Solution\Example section to simulate a disaster and recovery.

In this example I will use a 3 server FS4SP Farm: 1 Admin, 1 Primary Column, and 1 Backup Column and three File Share Folders for crawling content. I will also reset my index to make looking at the index a clean a possible.

Starting FS4SP Farm Deployment

<?xml version="1.0" encoding="utf-8" ?>

<instanceid>FAST Search Server POC</instanceid>

<connector-databaseconnectionstring></connector-databaseconnectionstring>

<!—Admin Node -- >

</host>

<!—Primary Column -- >

<content-distributor id="0" />

<indexing-dispatcher />

</host>

<!—Back Column -- >

<document-processor processes="4"/>

<content-distributor id="1" />

<indexing-dispatcher />

</host>

</searchcluster>

</deployment>

(3) Create 3 folders accessible to the FAST Connector Content SSA for Crawling. My contain 100 items (mixed PDF, DOC, etc)

\\myfileshare\Folder1

\\myfileshare\Folder2

\\myfileshare\Folder3

Solution\Example:

1. Setup and crawl a File Share within the FAST Content SSA.

a. In this example \\myfileshare\Folder1

2. Ensure the Index is Idle and primary and backup are identical

a. Open a Microsoft FAST Search Server 2010 Command Shell as Administrator

b. Execute: indexerinfo –a status

<?xml version="1.0"?>

<indexer hostname="fast2.mydomain.local" port="13050" cluster="webcluster" column="0" row="0"

Running" exclusion_interval="60" time_since_last_index="0">

<column_role state="Master" backups="1"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315333741336367000" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic"

</partition>

<document_api number_of_elements="0" last_sequence="92764" frequence="0.000000"

<queue_size current="0"/>

<operations_processed api="0"/>

</document_api>

</indexer>

<?xml version="1.0"?>

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster" column="0" row="1" factory_type="fi

nterval="60" time_since_last_index="0">

<column_role state="Backup" backups="0"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315333741336367000" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic"

</partition>

<document_api number_of_elements="0" last_sequence="92764" frequence="0.000000"

<queue_size current="0"/>

<operations_processed api="0"/>

</document_api>

</indexer>

Site Note: I truncated the output a little for readability.

c. The yellow highlight shows partition “0” contains the 101 items (100 items + 1 folder) in both the Primary and backup column.

d. The blue highlight shows that all 101 items have been indexed.

i. When actively crawling the “not_indexed” count will not be constantly zero. As batches are processed, stored and the indexed at intervals.

e. The green highlight shows that current status is idle for each partition within the Column Index.

f. In the real world we will not get that lucky as our server crash will probably not occur while we are idle and if we lose a server we will not be able to get this information for the primary and backup columns. Only the backup.

3. Let’s lose the Primary Column. (i.e. simulate the Server has crashed and is un-recoverable)

a. Open the FAST Command Shell as Administrator on the Primary Column Server

i. In this Example: fast2.mydomain.local

b. Execute: nctrl stop

i. This will stop all FAST services

c. Alternatively we could issue: “nctrl stop indexer” which will stop just the RTS Indexer but I want to simulate the server is gone.

d. Setup a 2^nd File Share Crawl in the FAST Content SSA

i. Start a Full Crawl

e. From the FAST Shell as Administrator

i. Execute: psctrl status

1. Note that no items are being submitted to the FS4SP pipeline. (this will only show results if there are Document Processors assigned to the server that are not stopped)

ii. Execute: indexerinfo –a status

1. Note: we have lost the Primary column and no counts have changed on the backup column.

2. The Primary and Backup Columns do NOT work as a fail-over in which content will be indexed to the backup and replicated to the primary columns once a primary column has been re-established

3. It would not be un-expected for the Backup Columns and Primary Columns to have different counts if the primary server crashed while active crawling was occurring. To be able to recover our primary column without having to restore from a backup or re-crawl millions of items and to eliminate any long downtime is worth losing a few items that would need to be accounted for once we are back up and crawling.

iii. Stop the Crawl from Step d.

Side Note: A reset of the OSearch14 service may be required if the crawl status does not go to idle.

4. Prompt the Backup Column to become the Primary Column

a. Stop all Crawls

b. Stop the Web Analyzer and Relevancy Admin

i. The WebAnalyzer runs on a schedule. To avoid any updates or processing causing changes to the index we will suspend the services from processing.

ii. Logon to the FS4SP server which hosts the WebAnalyzer.

1. In this example: fast1.mydomain.local

iii. Open FAST Command Shell As Administrator

iv. Execute: waadmin showstatus

1. The Overall Status needs to be running before we can suspend it.

v. If the Status is paused

a. Execute: waadmin enqueueview

b. Repeat Steps iv.

vi. Execute: waadmin AbortProcessing

vii. Execute: spreladmin AbortProcessing

c. Stop FS4SP Nodes

i. Perform the Node shutdown on each FS4SP server

1. In this example “fast1.mydomain.local” & “fast3.mydomain.local”

d. Modify the Existing Farm Configuration

i. On the FAST Admin Server

ii. Use Windows Explorer to navigate to <FAST Install Driver>\FASTSearch\etc\config_data\deployment.

iii. Edit the deployment.xml file

1. After initial FS4SP configuration this becomes the master deployment file

2. Comment out the current Primary Column.

a. fast2.mydomain.local in this example

3. Prompt the Backup Column by changing the row from “1” to “0”

a. fast3.mydomain.local in this example

4. Remove the Backup Row from the Farm by commenting out row=”1” from the search cluster.

iv. The new deployment file will look as follows with changes highlighted.

<?xml version="1.0" encoding="utf-8" ?>

<instanceid>FAST Search Server POC</instanceid>

<connector-databaseconnectionstring></connector-databaseconnectionstring>

<!—Admin Node -- >

</host>

<!— Old Primary Column -- >

<!-- <host name="fast2.mydomain.local">

<content-distributor id="0" />

<indexing-dispatcher />

</host> -- >

<!—New Primary Column -- >

<document-processor processes="4"/>

<content-distributor id="1" />

<indexing-dispatcher />

<!--<searchengine row="1" column="0" /> -- >

</host>

<!-- <row id="1" index="secondary" search="true" /> -- >

</searchcluster>

</deployment>

v. Starting with the Admin Server

1. In this example: “fast1.mydomain.local”

vi. Open FAST Command Shell As Administrator

vii. Execute: Set-FASTSearchConfiguration

viii. Execute: nctrl start

ix. Execute: nctrl status

1. Wait for the Node to respond that all services are running.

x. Repeat steps vi. – ix. for each non-admin node

1. In this example only “fast3.mydomain.local”

2. Remember “fast2.mydomain.local” crashed so we will not do anything with that server.

xi. Execute: indexerinfo –a status

1. Note fast3 is column=”0” row=”0”, State=”standalone”, backup=”0”

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster" column="0" row="0"

Running" exclusion_interval="60" time_since_last_index="162">

<column_role state="standalone" backups="0"/>

e. Re-crawl the 2^nd File Share

i. In this example I will crawl \\myfileshare\Folder2

f. Re-Execute: indexerinfo –a status

i. Validate the counts total include the new crawl.

<indexer hostname="trickypoc3.trickydomain.local" port="13050" cluster="webcluster" column="0" row="0"

Running" exclusion_interval="60" time_since_last_index="162">

<column_role state="standalone" backups="0"/>

5. Re-configure a New FS4SP server to join the FS4SP Farm.

a. In this example I will merely re-use fast2.mydomain.local

b. Stop FS4SP Nodes

i. Perform the Node shutdown on each FS4SP server

1. In this example “fast1.mydomain.local” & “fast3.mydomain.local”

c. Modify the Existing Farm Configuration to include the new Backup Column

i. On the FAST Admin Server

ii. Use Windows Explorer to navigate to <FAST Install Driver>\FASTSearch\etc\config_data\deployment.

iii. Edit the deployment.xml file

1. Un-Comment new FS4SP Server to be used as the backup Column

a. fast2.mydomain.local in this example

2. Change the row from “0” to “1”

a. Fast2.mydomain.local in this example

3. Re-implement the Backup row=”1” in the search cluster.

iv. The new deployment file will look as follows with changes highlighted.

<?xml version="1.0" encoding="utf-8" ?>

<instanceid>FAST Search Server POC</instanceid>

<connector-databaseconnectionstring></connector-databaseconnectionstring>

<!—Admin Node -- >

</host>

<!—New Backup Column -- >

<content-distributor id="0" />

<indexing-dispatcher />

</host>

<!—New Primary Column -- >

<document-processor processes="4"/>

<content-distributor id="1" />

<indexing-dispatcher />

<!--<searchengine row="1" column="0" /> -- >

</host>

</searchcluster>

</deployment>

v. Copy the fixml from the new Primary Column (fast3) to the new Backup Column (fast2) with the following commands. As I am using my existing server I will delete the fixml from “fast2” first.

robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <<em>source_path</em>>\data\data_fixml <FASTSearchFolder>\data\data_fixml

copy <<em>source_path</em>>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage

In this example:

robocopy /E /MT:100 /NFL /COPYALL \\fast3\c$\FASTSearch\data\data_fixml C:\FASTSearch\data\data_fixml

copy \\fast3\c$\fastsearch\data\ftStorage\processed_checkpoint.txt C:\FASTSearch\data\ftStorage

Side Note: Why copy the FIXML if replication between a Primary Column and Backup Column will occur automatically when we implement the new backup column and resume indexing? So on that note, I will not use the robocopy in this example. When we implement the backup column and resume crawling FS4SP certainly appears to take care of the replication as we end up with the same number of items in both the Primary and Backup Column. I highlighted in RED two numbers to look at in those results which are the Total Counts for each Column and not just the Counts for each partition. When using a Primary and backup column strategy not only is the index kept in sync but also the FIXML. As we crawled \\myfileshare\Folder2 before re-implementing our backup column we want to make sure our FIXML stays in sync. If you choose not to perform the robocopy like I did browse around the FIXML folders on the 2 machines and you will quickly notice they do not appear to be identical. I encourage trying it both ways.

vi. Starting with the Admin Server

1. In this example: “fast1.mydomain.local”

vii. Open FAST Command Shell As Administrator

viii. Execute: Set-FASTSearchConfiguration

ix. Execute: nctrl start

x. Execute: nctrl status

1. Wait for the Node to respond that all services are running.

xi. Repeat steps vii. – ix. for each non-admin node

1. In this example “fast2.mydomain.local” and “fast3.mydomain.local”

2. “fast2.mydomain.local” is our new backup server.

xii. Execute: indexerinfo –a status

1. Note fast3 is no longer column=”0” row=”0”, State=”standalone”, backup=”0” but state=”Master” backups=”1” while fast2 is column=”0” row=”1” state=”Backup” backups=”0”

d. Resume WebAnalyzer and Relevancy Admin

i. Open FAST Command Shell As Administrator on the FS4SP Admin node or the Server with the WebAnalyzer service enabled.

ii. Execute: waadmin EnqueueView

iii. Execute: spreladmin Enqueue

e. Setup the 3^rd File Share Crawl and Crawl

i. In this example I will crawl \\myfileshare\Folder3

ii. When the Crawl has completed

iii. Open FAST Command Shell As Administrator on any FS4SP server

iv. Execute: indexerinfo –a status

v. Note the highlighted sections in the results below:

1. fast3 has become the Primary Column

2. fast2 has become the Backup Column

3. both columns have the same number of expected documents

4. As replicated does not occur immediately you may have to give the backup column time before it becomes in sync with the Primary Column. Using the robocopy in the SideNote above will probably get you in sync faster if you have a large corpus of documents. It this example using a small corpus of 303 items it occurs relatively quick.

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster

Running" exclusion_interval="60" time_since_last_index="103166">

<column_role state="Master" backups="1"/>

<index_frequence min="8.538462" max="25.250000"/>

<partition id="0" index_id="1315606415333091000" status="idle" type="dynamic" ti

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic" ti

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic" ti

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic" ti

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic" ti

</partition>

<document_api number_of_elements="0" last_sequence="92865" frequence="0.000000"

<queue_size current="0"/>

<operations_processed api="101"/>

</document_api>

</indexer>

<?xml version="1.0"?>

<indexer hostname="fast2.trickydomain.local" port="13050" cluster="webcluster

nterval="60" time_since_last_index="0">

<column_role state="Backup" backups="0"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315606415333091000" status="idle" type="dynamic" ti

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic" ti

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic" ti

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic" ti

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic" ti

</partition>

<document_api number_of_elements="0" last_sequence="92865" frequence="0.000000"

<queue_size current="0"/>

<operations_processed api="0"/>

</document_api>

</indexer>

Conclusion: Though all the services in a FS4SP farm, with the exception of the Admin service, can be implemented in a scaled/redundant fashion there is no way to implement indexer redundancy for your Primary Columns. If you lose a Primary Column you will not be able to index. It would be nice if the backup columns automatically worked as a fail-over for both query and indexing but they do not. When planning a disaster recovery strategy you can choose a couple of options: 1) using the built-in fs4sp backup and restore commands or 2) backing up the fixml folder and rebuilding the index from that restore but if you are spending the cost to implement high-availability of search (i.e. primary columns have backups) it may be wise to think about putting a plan in place to prompt backup columns to primary columns for disaster recovery. There could be several advantages: 1) the storage space has already been allocated for the backup columns saving the cost of additional storage for FS4SP backups, 2) you will probably get backup and indexing faster, and 3) you can immediately resume indexing by prompting the backup column and worry about building a new backup column later. (Standing up new hardware can take time especially if there is not hardware in sight to use)

KORITFW

Rebuilding the FAST Search Server 2010 index (FS4SP)

2011-09-14T21:31:00.000-07:00

Topic: SharePoint 2010, FS4SP, FAST Search Server 2010, FIXML

Subject: Index corruption and rebuilding the FS4SP index from FIXML

Problem: My FS4SP index has become corrupt. It took a week to original perform our full crawls. Is there anything we can do?

Response: Though this situation tends to be rare there may come a time when you need to rebuild a FS4SP index. When FS4SP is indexing items it not only stores the physical index itself <FASTInstall Drive>\FASTSearch\data\data_index but it also stores FIXML <FASTInstall Drive>\FASTSearch\data\data_fixml for each item indexed. The FIXML contains all the information which is to become that item within the index. Search results are retrieved from the index so why keep the FIXML around? Well one reason is for this exact problem: rebuilding an index. There are two ways of fixing a corrupt index and I will use the Solution\Example section to cover both. (In addition rebuilding an index is very important in prompting a backup column to a primary column) In this example I am using a three server single Column FS4SP farm.

FS4SP Farm: 1 Admin, 1 Primary Column, and 1 Backup Column.

Note: “fast3.mydomain.local” is the Primary Column Server.

Starting FS4SP Farm Deployment

<?xml version="1.0" encoding="utf-8" ?>

<instanceid>FAST Search Server POC</instanceid>

<connector-databaseconnectionstring></connector-databaseconnectionstring>

<!—Admin Node -- >

</host>

<!—Primary Column -- >

<content-distributor id="0" />

<indexing-dispatcher />

</host>

<!—Back Column -- >

<document-processor processes="4"/>

<content-distributor id="1" />

<indexing-dispatcher />

</host>

</searchcluster>

</deployment>

Solution\Example:

Repairing a corrupt index using FIXML

1. Optional (To allow for ease of observation)

a. Clear Event Viewer entries on the Primary Column Server

b. Clear FS4SP Logs

i. Open a FAST Command Shell as Administrator

ii. Execute: nctrl stop

iii. Execute: net stop fastsearchmonitoring

iv. Delete all files under <FASTInstallDrive>\FASTSearch\var\log

v. Execute: nctrl start

2. Resetting the Index

a. An index reset can be performed from any server in the FS4SP farm and on any Column or Row by specifying the Column, Row in the reset command. For this example, I will perform the command from the server which hosts the Primary Column and will specify the Row and Column even though the FS4SP farm only has a single column.

b. An index reset does not rebuild the Column from scratch. The indexer will validate each item within the FS4SP column against the original FIXML. Any item which is deemed not in sync or corrupt will be updated in the FS4SP column/index. This will take less time than rebuilding the index column from scratch.

3. Open the FAST Command Shell as Administrator on the FS4SP Admin Server

a. In this example: fast1.mydomain.local

b. Make sure all crawls are stopped or paused and the FS4SP Column is idle

i. Execute: indexerinfo --row=0 --column=0 status

ii. Look at the status of each partition status

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster" column="0" row="0"

<column_role state="Master" backups="1"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315841940082287000" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic"

</partition>

<document_api number_of_elements="0" last_sequence="99610"

<queue_size current="0"/>

<operations_processed api="0"/>

<document_api number_of_elements="0" last_sequence="99610" frequence

</document_api>

</indexer>

c. Stop the Web Analyzer and Relevancy Admin

i. The WebAnalyzer runs on a schedule. To avoid any updates or processing causing changes to the index we will suspend the services from processing.

ii. Logon to the FS4SP server which hosts the WebAnalyzer.

1. In this example: fast1.mydomain.local

iii. Open FAST Command Shell As Administrator

iv. Execute: waadmin showstatus

1. The Overall Status needs to be running before we can suspend it.

v. If the Status is paused

a. Execute: waadmin enqueueview

b. Repeat Steps iv.

vi. Execute: waadmin AbortProcessing

vii. Execute: spreladmin AbortProcessing

4. Issue an Index reset

a. Logon to the Primary Index Column Server

i. In this example: fast3.mydomain.local

b. From the FAST Command Shell as Administrator

c. Execute: indexeradmin --row=0 --column=0 resetindex

d. Execute: indexerinfo --row=0 --column=0 status

i. You may have to issue the command several times to see the work being performed.

ii. The index reset will work through each partition of the FS4SP column. If you can keep executing the command to watch the progress. Note: all the items are originally in partition 1 (they will not end up there. At least in my example because of the number of items I have under index).

iii. When the index reset finishes all the items end up in partition 4. (This will vary depending on the partition spread. Just an interesting observation as to how the reset is performed under the covers)

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster" column="0" row="0"

<column_role state="Master" backups="1"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315841940082287000" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="1315328535885114000" status="indexing (6%)" type="dynamic"

</partition>

<document_api number_of_elements="0" last_sequence="99610"

<queue_size current="0"/>

<operations_processed api="0"/>

<document_api number_of_elements="0" last_sequence="99610" frequence

</document_api>

</indexer>

e. Alternative ways to watch the process. (Especially if you experiment with very little data in the system as the index reset will work fast enough that you may not see the status changing)

i. From Event Viewer

1. Open the Windows Event Viewer

2. Expand the Applications and Services Logs Node

3. Open the FAST Search Logs

4. You will find several entries similar to the following:

indexer_admin_servant: Reset index requested.

state::runtime: Indexing suspended

state::runtime: Indexing resumed

work_order 4_1315848480044895000: Index State Reset - Not using incremental indexing

master_indexing_thread (p:4,j:1): Completed Index State Reset. Normal indexing enabled.

f. From FAST Logs

i. Using Windows Explorer Navigate to <FASTInstallDrive>\FASTSearch\var\log\indexer folder

ii. Open indexer.txt

iii. Search for “Reset index requested” and you will see something similar to the following:

INFO indexer indexer_admin_servant: Reset index requested.

VERBOSE indexer rts::indexing::util: Suspending indexing

INFO indexer state::runtime: Indexing suspended

INFO indexer state::runtime: Indexing resumed

VERBOSE indexer percentage_file_distributor: Partition 4 should have 100% of the docs. Num docs 7046 target : 6000000. New range: 1-714

INFO indexer work_order 4_1315848480044895000: Index State Reset - Not using incremental indexing

VERBOSE indexer index_producer (4): Index 1315848480044895000 completed. OK docs: 7046, failed docs: 0, errors: 0, range: 1-714, 0 exclusionlisted

VERBOSE indexer percentage_file_distributor: Partition 3 has empty range.

INFO indexer work_order 3_1315848737366889000: Index State Reset - Not using incremental indexing

VERBOSE indexer index_producer (3): Index 1315848737366889000 completed. OK docs: 0, failed docs: 0, errors: 0, range: 0-0, 0 exclusionlisted

VERBOSE indexer percentage_file_distributor: Partition 2 has empty range.

INFO indexer work_order 2_1315848740067419000: Index State Reset - Not using incremental indexing

VERBOSE indexer index_producer (2): Index 1315848740067419000 completed. OK docs: 0, failed docs: 0, errors: 0, range: 0-0, 0 exclusionlisted

VERBOSE indexer percentage_file_distributor: Partition 1 has empty range.

INFO indexer work_order 1_1315848743251859000: Index State Reset - Not using incremental indexing

VERBOSE indexer index_producer (1): Index 1315848743251859000 completed. OK docs: 0, failed docs: 0, errors: 0, range: 0-0, 0 exclusionlisted

VERBOSE indexer percentage_file_distributor: Partition 0 has empty range.

INFO indexer work_order 0_1315848746358249000: Index State Reset - Not using incremental indexing

VERBOSE indexer index_producer (0): Index 1315848746358249000 completed. OK docs: 0, failed docs: 0, errors: 0, range: 0-0, 0 exclusionlisted

VERBOSE indexer search_controller_holder: Activating index set '0_1315848746358249000,1_1315848743251859000,2_1315848740067419000,3_1315848737366889000,4_1315848480044895000', 7046 active docs, 0 exclusionlisted.

INFO indexer master_indexing_thread (p:4,j:1): Completed Index State Reset. Normal indexing enabled.

VERBOSE dictionary_producer Loading rc-file: 'C:\FASTSE~1\var/etc/findexrc'

VERBOSE dictionary_producer Reading index configuration from C:\FASTSearch\data\data_index\fast3.mydomain.local.normalized.temp\index.cf

VERBOSE indexer dictionary_builder: New dictionaries ready (trickypoc3.trickydomain.local.normalized.1315848819)

g. Resume WebAnalyzer and Relevancy Admin once the index reset completes.

i. Open FAST Command Shell As Administrator on the FS4SP Admin node or the Server with the WebAnalyzer service enabled.

ii. Execute: waadmin EnqueueView

iii. Execute: spreladmin Enqueue

h. Test your Search Center

Rebuilding an index from FIXML

1. Open the FAST Command Shell as Administrator on the FS4SP Admin Server

a. In this example: fast1.mydomain.local

b. Make sure all crawls are stopped or paused and the FS4SP Column is idle

i. Execute: indexerinfo --row=0 --column=0 status

ii. Look at the status of each partition status

<indexer hostname="fast3.mydomain.local" port="13050" cluster="webcluster" column="0" row="0"

<column_role state="Master" backups="1"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="1315841940082287000" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="1315328545925813000" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="1315328542666333000" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="1315328539447778000" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="1315328535885114000" status="idle" type="dynamic"

</partition>

<document_api number_of_elements="0" last_sequence="99610"

<queue_size current="0"/>

<operations_processed api="0"/>

<document_api number_of_elements="0" last_sequence="99610" frequence

</document_api>

</indexer>

c. Stop the Web Analyzer and Relevancy Admin

i. The WebAnalyzer runs on a schedule. To avoid any updates or processing causing changes to the index we will suspend the services from processing.

ii. Logon to the FS4SP server which hosts the WebAnalyzer.

iii. Open FAST Command Shell As Administrator

iv. Execute: waadmin showstatus

1. The Overall Status needs to be running before we can suspend it.

v. If the Status is paused

a. Execute: waadmin enqueueview

b. Repeat Steps iv.

vi. Execute: waadmin AbortProcessing

vii. Execute: spreladmin AbortProcessing

2. Rebuild the Primary Column Index

a. Logon to the Primary Index Column Server

b. In this example: fast3.mydomain.local

c. From the FAST Command Shell as Administrator

d. Execute: nctrl stop

e. Using windows explorer navigate to <FASTInstallDrive>\FASTSearch\data\

f. Delete the data_index folder

g. Execute: nctrl start

h. Execute: indexerinfo --row=0 --column=0 status

i. Notice the total=”7046”, indexed=”0” and not_indexed=”7046” as these numbers are generated from the FIXML.

ii. Much like the index reset you will see that status of the partitions changes

iii. Keep re-issuing the indexerinfo --row=0 --column=0 status to watch the progress

iv. Note the difference between index reset and rebuilding the index. The index reset moved the items from one partition to another while keeping the index populate. The rebuilding of the index from scratch started with an indexed count: indexed=”0”.

<indexer hostname="fast3.myomain.local" port="13050" cluster="webcluster" column="0" row="0"

ndex="0">

<column_role state="Master" backups="0"/>

<index_frequence min="0.000000" max="0.000000"/>

<partition id="0" index_id="0" status="idle" type="dynamic"

</partition>

<partition id="1" index_id="0" status="idle" type="dynamic"

</partition>

<partition id="2" index_id="0" status="idle" type="dynamic"

</partition>

<partition id="3" index_id="0" status="idle" type="dynamic"

</partition>

<partition id="4" index_id="0" status="indexing (8%)"

</partition>

<document_api number_of_elements="0" last_sequence="99610" frequence="0.000000"

<queue_size current="0"/>

<operations_processed api="0"/>

</document_api>

</indexer>

i. Much like the index reset above the alternative ways to watch the progress are through the Event Viewer and the indexer.txt log. The messages will differ but the same end result will occur.

j. Resume WebAnalyzer and Relevancy Admin once the index rebuild completes.

i. Open FAST Command Shell As Administrator on the FS4SP Admin node or the Server with the WebAnalyzer service enabled.

ii. Execute: waadmin EnqueueView

iii. Execute: spreladmin Enqueue

k. Test your Search Center

Conclusion:

For each item indexed into the FS4SP index a FIXML file is created representing the item within the index. The FIXML items stored on the FS4SP Servers does take up additional storage but it has several beneficial functions from validating security, debugging crawled/managed properties and as in this example fixing a corrupted index or even rebuilding an index.

Special Note: Rebuilding the index from scratch requires a lot more free disk space than an index reset. Temporary files are created and released and you may consume twice as much disk space as the final index. If you need to rebuild from scratch watch the amount of free disk space. If it runs out (about 2GB min) the process will fail and immediately start again. Until the lack of free storage space is addressed the process will continue to cycle.