Results tagged “Search” from Bill Benac

Last month I shared a diagnostic application for the WCI API Service. It worked great on my machine, but...



Performance was terrible when I deployed it to my customer's environment. It commonly would take 8 seconds (8300 ms) to load. Why the hang up? On my laptop it only took 20 ms.

I looked at the spy logs, and I found that nearly all the processing time went to this  command from the API Service when it tried querying users:

SQueryRequest.execute() executing query: "(((not null) TAG phraseQ OR null) AND ((subtype:"PTUSER")[0])) AND ((((@type:"PTPORTAL")[0]) OR ((@type:"PTCONTENTTEMPLATE")[0])) AND (((ptacl:"u1") OR (ptacl:"9994") OR (ptacl:"9992") OR (ptacl:"51") OR (ptacl:"1"))[0]) AND (((ptfacl:"u1") OR (ptfacl:"9994") OR (ptfacl:"9992") OR (ptfacl:"51") OR (ptfacl:"1"))[0])) METRIC logtf [1]"

The customer has 1.1 million users in the database, so maybe the search index is just very slow? I deleted a few hundred thousand users, and the query could then regularly return in about 6200 ms, so this was a major improvement. Also, I used the standard administrative UI to search for all users, and it took a similarly long time to return.

Ahh, so the problem is that the search index is doing a very bad job querying against a large set of objects.

Does anyone know whether there is a way to overcome this problem? Some uninspired attempts to tune the search service's cache size gave me no material change.

So in the absence of good tuning knowledge? I changed the apicheck diagnostic app so that now it verifies it can query communities (of which there are hundreds) instead of the users. The performance problems for my portlet have magically gone away

I'm working with a Collab deployment where we want to disable the search feature. This involves two things: disabling index requests from Collab to the Search Server and removing the search form from Collab's web UI.

Disabling Index Requests from Collab to Search

The easy part is disabling index requests from Collab to the Search Server. Just open %ALUI_HOME%\ptcollab\4.5\settings\config\config.xml and change the value for "search enabled" from "yes" to "no."

Removing Search from Project Explorer UI

For whatever reason, the Project Explorer view of Collab shows a search form even when the app is configured to not have its content index. This is shown below:

Let's get to know how to easily do what the Collab installer doesn't let you do easily: manage its individual components.

At my current client, we're upgrading ALUI Collab 4.2 to 4.5 as part of our upgrade to ALUI 6.5. One feature of the new ALUI suite is that it offers a "common notification service" that replaces the collab-specific notification service that had been part of 4.2. Accordingly, we don't need the old Collab notification service. The official upgrade documentation says "If you have an existing Notification Service from your previous installation of Collaboration, disable or uninstall it." But how do you do that?

If you run the Collab 4.2 or 4.5 uninstaller on a machine with more than one Collab component, it will remove every component. That doesn't work well since you're trying to upgrade the Collab piece and remove the notification piece. Similarly, you may realize that you installed Collab 4.5's Search Service on the Collab box when you really want it on the Search box, so how do you remove just the Search Service?

In either case, you can use the batch file that is part of the undesired component, remove the service, then delete the files from the file system. In the case of the 4.2 notification, use this batch file: %ALUI_HOME%ptnotification\4.2\bin\ptnotificationserverd.bat, and in the case of the Search Service, use this batch file: %ALUI_HOME%searchservice\1.1\bin\searchservice.bat. Pass in the parameter "remove" and you're done.

Enjoy!

I recently posted about how to restore a search cluster that has failed. Step 1 was to make sure all the nodes are running. But what if one of the nodes won't start? What do you do when a node itself is corrupted?

Ironically, you can't use the tools to reset a node if the node is broken and won't start. But you can go on the file system and clean it up with these steps--though make sure the node really is stopped before you do this.

In shorthand for the Unix-minded:

set search_home=d:\bea\alui\ptsearchserver\6.1
set node=alui-ss-0101
rm -rf %search_home%\%node%\index\*
mkdir %search_home%\%node%\index\1
echo 1 > %search_home%\%node%\index\ready
cd %search_home%\%node%\index\1
..\..\..\bin\native\emptyarchive lexicon archive
..\..\..\bin\native\emptyarchive spell spell

Or more prosaically for those who prefer Windows and a mouse:

1. Go into the index folder of the node. On my machine, it's d:\bea\alui\ptsearchserver\6.1\bbenac0201\index.
2. The index folder will contain a folder named 1. Open this folder and delete all its contents.
3. The index folder may also contain a folder named 2. If this is the case, delete the folder 2.
4. The index folder should contain a file named "ready." Open it and make sure it contains just the number 1. This "ready" file tells the node to look within folder 1 for its content.
5. Open a command prompt within the folder 1.
6. Run these commands, the first of which should create about 8 files, the second of which should create about 7:

   ..\..\..\bin\native\emptyarchive lexicon archive
   ..\..\..\bin\native\emptyarchive spell spell

At this point, you should be able to start your node. It will contact the search cluster and populate itself with the current search index.

Ever have problems with your search cluster getting corrupted? One way to fix it is to wipe it out and reindex everything, but that may take 24 hours. A better approach is to use the feature of scheduling checkpoints to backup your cluster daily, then restoring that checkpoint if corruption occurs.

If your ALUI system isn't set to do daily checkpoints, configure them as follows:

1. Browse to the portal's admin section, then select the utility "Search Cluster Manager."
2. In the left navigation, select "Checkpoint Manager."
3. On the far right, click "Initiate Checkpoint" to open the Checkpoint Scheduler.
4. Select the "Scheduled" radio button, select today's date, set a time, and set it to repeat every 1 day. Click OK.
5. After the Checkpoint Scheduler closes, you'll see in the "Checkpoint Activity Status" section when the next scheduled checkpoint will run.
6. Click "Finish" and your system will then backup your search index daily into checkpoints.

If at some point you realize your cluster is corrupt and you need to restore it, and you've been creating checkpoints periodically, then:

1. Makes sure all the nodes in the search cluster are started. If one of the nodes won't start, you might want to use these instructions to revive it.
2. Browse to the portal's admin section, then select the utility "Search Cluster Manager."
3. In the left navigation, select "Checkpoint Manager."
4. In the center of the screen you'll see a list of checkpoints. The most recent ones will show themselves as "Available" in the last column.
5. Click on the checkpoint you want to restore. Its row will change from white to light green.
6. Click the "Restore Checkpoint" button on the right side of the screen below the list of checkpoints.
7. Watch the "Checkpoint Activity Status" section to see status. Use the refresh button at the top of the screen to update status. It may show messages like "Node pswwwlab-0301 completed copying - 0%."
8. When it completes, you'll see in the "Named Restore Status" area something like this:

   Cluster is currently in a named restore state.
   Cluster restored from c:\\bea\\alui\\ptsearchserver\\6.1\\cluster\checkpoints\0_1_0.
   The named restore state - SUCCEEDED.
   The named restore must either be discarded or committed.

9. Finally, you must "Commit" the checkpoint by using the "Commit" button at the bottom right of the screen.
10. At this point the search cluster will be restored.

Note that by default, the most recent three search checkpoints are stored. So on the fourth day, the first checkpoint gets deleted. In some cases, you may prefer to have more or fewer checkpoints. If this is the case, then edit the cluster.ini file in your search cluster. Add the following new parameter to set the desired number of saved checkpoints, in this case, 2:

RF_NUM_CHECKPOINTS_SAVED=2

If you're lucky, you'll never need to restore a checkpoint. But you'd better be prepared. So make sure you've set this up!

The ALUI search component underwent a major redesign for the 6.1.0.0 release. In earlier versions, the indexing component of search was a single point of failure. The querying component though could be made highly available by replicating its index to secondary servers. Release 6.1 brought the important improvement of allowing both indexing and querying to be redundant. A customer can install two search servers to act as nodes of the same partition, and the index takes care of itself.

So there is no longer a need to write special scripts to replicate the 6.1 search index, right? Not if you are just trying to make your live index redundant. So the 6.1 product dropped the old "replicate" tool.

But larger customers have a use case that still requires index replication. If the customer has a failover datacenter for use when the primary system is unavailable for one reason or another, then the customer needs to replicate the search index to that site, and how do you this? It used to be very easy. Consider the following that did the job on ALUI 6.0 (in this case on RHEL). It was two easy commands:

SEARCHHOME=/usr/local/plumtree/ptsearchserver/6.0

echo ----------- copy the master search index to a backup directory
$SEARCHHOME/bin/replicate -incr_backup aqlsearch 15244 $SEARCHHOME/indexmaster $SEARCHHOME/incr_backup

echo ----------- restore the backup index to the failover search server
$SEARCHHOME/bin/replicate -restore aqlsearchfail 15244 $SEARCHHOME/index $SEARCHHOME/incr_backup

You can still attain the same result in the ALUI 6.1 releases, but hold onto your hat. It's a wild ride.
• Copy the %searchcluster%\checkpoints and %searchcluster%\requests folders from the origin server to a temporary directory on the destination server
• Make sure all search nodes and services are running properly
• Run the following command to empty the checkpoints folder and set the requests folder to only have an indexQueue.segment:
  
         %searchhome%\bin\native\cadmin.exe purge --remove
  
  
• Stop all nodes and services
• Copy the origin checkpoints and requests folders over their corresponding destination folders, including the indexQueue.segement:
• Replace the cluster.nodes file(s) in the checkpoints folder(s) with the one from destination cluster base. For example, copy %searchcluster%\cluster.nodes over %searchcluster%\checkpoints\0_106_30976\cluster.nodes
• Start all nodes and services. The nodes will restore from the checkpoint. If you have a large index, the nodes may take a while to start (five minutes for 1.2 million objects). If you have multiple nodes, they may take another several minutes to move from stall/recover.

Other migration methods may have trouble migrating checkpoints from a system with low-numbered checkpoint folders to a destination system with higher-numbered folders. This method however has no such problem.

These steps have been tested only on systems that use a single partition. The number of nodes in the partition is not significant; these steps have been tested when restoring the cluster to destination systems with different numbers of nodes than the origin.

The directories can be copied from the source while all services are up, and this should not cause synchronization issues later during the restore. So if you take a checkpoint, then later make several changes to the index, then later copy the checkpoints and requests folders to the destination, the destination will have all the changes in its restored index, including those made after the checkpoint.

This process was created with significant input from Dax Farhang, the product manager responsible for the search product. If we're lucky, he'll cook this feature into the 6.5 ALUI product so that this post will become obsolete. In the meantime, enjoy.

The reality is load balancing can be confusing. Some applications require sticky load balancing (like the portal), some require shared file system access (like the document repository), some require an external load balancer (like the image server), and some are load balanced internally by the portal (like most custom portlets). So not surprisingly, some customers approach load balancing with trepidation, or perhaps they just expect implementing it will require mad voodoo engineering skills.

Enter Grid Search. The 6.1 version of the search server was rewritten with a raft of design changes including improved scalability. Previously it didn't fully scale. Now it has a tool called the Search Cluster Manager, which tells the story, right?

Actually, though the Search Cluster Manager's name correctly says "we can cluster," it gives many people a false idea about how clustering works. Administrators use the Search Cluster Manager to add a new search node to the cluster, but that's it. The Search Cluster Manager isn't required at runtime for end users who need their queries managed across the multiple nodes in the cluster. This component would be better named something like "Search Cluster Administration." Those who are used to configuring components to sit behind load balancers frequently expect the portal needs to connect to the cluster manager to serve content from the nodes.

In fact, when an administrator installs a new search node and adds it to the cluster through the Search Cluster Manager, the product goes on auto-pilot, and the load balancing is done. When the portal had only one search node, it was configured in the Search Server Manager to use that single node as the search cluster contact node. The magic is that after adding a second node, the contact node doesn't need to change. The search server is smart enough to notify the portal of any other node participating in the cluster. So the portal knows all the servers toward which it can distribute search traffic.

When you click the Search Service Manager's "Show Status" button, you can verify that it knows of all your nodes even though it's configured to use just one of them as the Cluster Contact Node. Note that at the bottom of the page, each node is listed:

You can read a full document I put together that several people have told me provides useful enhancement to the standard documentation. It discusses installation, creation of nodes, the rationale behind your choices as you do this, and a little bit on load balancing. Feel free to download the file.