Tools for Monitoring and Troubleshooting FRS

Microsoft provides a number of tools for monitoring and troubleshooting FRS that vary in their ability to perform real-time monitoring and troubleshooting or point-in-time troubleshooting. These tools also vary in their complexity and installation requirements and are thus suited for specific scenarios. The following topics provide a brief overview of each tool and when to use it.

Ultrasound

Ultrasound is a powerful monitoring and troubleshooting tool used to measure the health of FRS replica sets. Ultrasound shows health ratings and historical information about replica sets and allows administrators to monitor the progress of replication and detect problems that can cause replication to become backlogged or stopped. Ultrasound also provides drill-down views for troubleshooting and a framework for customizing alerts and views to suit your organizational needs.

Because Ultrasound requires the deployment of WMI providers on all servers to be monitored, Ultrasound is best suited for organizations that want to regularly monitor FRS, even when FRS is working normally. For more information about Ultrasound, see Ultrasound Overview.

Sonar

Sonar collects broad statistics about servers in a replica set, allowing administrators to monitor traffic levels, backlogs, free space, and other issues. In addition to providing basic monitoring capabilities, Sonar can also be quickly deployed on a single server to identify FRS problems.

Sonar works well for small organizations do not need a dedicated monitoring system and for organizations that have not deployed Ultrasound. Sonar also works well in scenarios where you want to quickly identify FRS problems.

Sonar provides less detailed troubleshooting information compared to Ultrasound because Sonar provides information on a per-member level. Sonar is quite useful when you need to quickly get a view of the health of the distributed system and determine possible problem areas. For more information about Sonar, see Monitoring FRS Using Sonar.

FRS Diagnostics (FRSDiag)

FRSDiag replaces the Health_Chk tool used to collect information about FRS from various sources, including Active Directory, event logs, FRS debug logs, and more. FRSDiag provides a graphical user interface for choosing which information to collect and analyzes the results to identify common problems.

After FRSDiag completes its collection and analysis, you can review the errors that FRSDiag identified and the text files that contain the information that FRSDiag collected. You can also provide these text files to Microsoft Product Support Services for assistance in diagnosing and resolving replication problems.

FRSDiag is typically used as a diagnostic and troubleshooting tool, not a monitoring tool. It is often used with Sonar or Ultrasound as a way to diagnose an unhealthy replica member. For more information about using FRSDiag, see Troubleshooting FRS Using FRSDiag.

FRS Troubleshooter

This troubleshooting guide provides step-by-step procedures that help you find the root cause of a number of FRS problems. These procedures are the same as those used by Microsoft Product Support Services to solve FRS problems and include tasks such as verifying network connectivity and DNS, verifying Active Directory replication, and examining output from FRSDiag. If you are not familiar with these tasks, we recommend that contact Microsoft Product Support Services for assistance. To use the troubleshooter, see FRS Troubleshooter.

Perl Scripts

The Perl scripts TopChk, Connstat, and IoLogSum are available as part of the Windows Support Tools. These scripts parse through the output of Ntfrustil commands to create a summary of relevant FRS data that is useful when troubleshooting FRS. For more information about running these scripts and interpreting their output, see FRS Data Collection Scripts.


FRS Overview

File Replication Service (FRS) is a multimaster replication service used to replicate files and folders in SYSVOL file shares and in DFS link targets. FRS works by detecting changes to file and folders in a replica set, and then replicating those changes to other replica members, which are connected in a replication topology. Because FRS is a multimaster replication service, any member of a replica set can generate changes, and FRS can resolve file and folder conflicts to make data consistent among the replica members.


Glossary of FRS Terms

The following terms are used to describe the components and processes of FRS.

Authoritative restore (also called D4) The process whereby one replica member is made the authoritative member, and all the data in its replica tree is replicated to all other replica members. This procedure should be used only in extreme circumstances under the supervision of your support provider or Microsoft Product Support Services. This process is informally referred to as a "D4" in reference to the registry setting used to invoke this process.

Change order A message that contains information about a file or folder that has changed on a replica member. The change order is sent to the member’s outbound partners. If the outbound partners accept the change, the partners request the associated staging file. After installing the change on their individual replica trees, the partners propagate the change order to their outbound partners.

Burflags Short for "backup restore flags," an FRS-related registry setting used to change the behavior of the FRS service at startup. Common values are D2 and D4.

Downstream partner For a given replica member, the set of servers to which the replica member provides data for a given replica tree. Downstream partners are also called outbound partners.

File event time The time at which a change to a file or directory is made. This might not be the same as the file create or last-write time. For example, restoring a file from a backup tape preserves the file create and last-write times, but the file event time is the time when the actual file restoration was performed.

File GUID An identifying property of a file or directory in a replica tree. FRS creates and manages file GUIDs, which, along with the replication version number and event time, are stored in the File ID table in the FRS database. Corresponding files and directories across all replica set members have the same file GUID.

File ID Table A table in the FRS database that contains an entry with version and identity information for each file and folder in the replica tree.

File Version Number A property of a file and directory in a replica tree that is incremented each time the file or directory is updated. The file version number is used to resolve concurrent updates originating from more than one member of the replica set. The version number is only incremented by the member that originated the file update. Other members that propagate the update do not change the version number.

Inbound connection For a given replica member, a component of the NTFRS_MEMBER object in Active Directory that identifies inbound partners. An inbound connection exists for each inbound partner.

Inbound log A table in the FRS database that stores pending change orders to be processed. As entries are processed, acknowledgments are sent to the inbound partners.

Inbound partners For a given replica member, the set of servers providing data to that replica member. Inbound partners are also known as upstream partners.

Knowledge Consistency Checker (KCC) A built-in process that runs on all domain controllers and generates replication topology for the Active Directory forest. The KCC creates separate replication topologies depending on whether replication is occurring within a site (intrasite) or between sites (intersite). The KCC also dynamically adjusts the topology to accommodate new domain controllers, domain controllers moved to and from sites, changing costs and schedules, and domain controllers that are temporarily unavailable.

Link In a DFS namespace, a component in a DFS path that lies below the root and maps to one or more link targets.

Link target In a DFS namespace, the mapping destination of a link, typically a shared folder on a server. FRS can be used to keep link targets (shared folders) synchronized on different servers.

Local change order A change order that is created because of a change to a file or directory on the local server. The local server becomes the originator of the change order and constructs a staging file.

MD5 checksum A cryptographically secure one-way hashing algorithm that is used by FRS to verify that a file on each replica member is identical.

Morphed directory A directory that FRS creates when resolving a name conflict between two or more identically named directories that originate on different replica members. FRS identifies the conflict during replication, and the receiving member protects the original copy of the folder and renames (morphs) the later inbound copy of the folder. The morphed folder names have a suffix of "_NTFRS_xxxxxxxx," where "xxxxxxxx" represents eight random hexadecimal digits.

Nonauthoritative restore (also called D2) The process whereby a given replica member is reinitialized by obtaining a replica tree from another replica member. This process is often used to resynchronize a replica member’s replica tree with one of its inbound partners, such as after a server failure. This process is informally referred to as a "D2" in reference to the registry setting used to invoke this process.

Originator GUID A globally unique identifier that associated with each replica member. All change orders produced by a given replica member carry the replica member’s originator GUID, which is saved in the file ID table.

Outbound connection For a given replica member, a component of the NTFRS_MEMBER object in Active Directory that identifies outbound partners. An outbound connection exists for each outbound partner.

Outbound log A table in the FRS database that stores pending change orders to be sent to outbound partners. The changes can originate locally or come from an inbound partner. These change orders are eventually sent to all outbound replica partners.

Outbound partner For a given replica member, the set of servers to which the replica member provides data for a given replica tree. Outbound partners are also called downstream partners.

Preinstall directory A hidden subdirectory under the root of the replica tree. When a newly created file or directory is replicated to a downstream partner, the file or directory is first created in the preinstall directory. After the file or directory is completely replicated, it is renamed to its target location in the replica tree. This process is used so that partially constructed files are not visible in the replica tree.

Remote change order A change order received from an inbound (or upstream) partner that originated elsewhere in the replica set.

Replica member A server that participates in a given replica set.

Replica partner The immediate upstream and downstream partners of a given replica member. (Upstream partners are also referred to as inbound partners, and downstream members are also referred to as outbound partners.)

Replica set Two or more servers configured to replicate a replica tree. The individual servers are known as replica members.

Replica tree The contents of a directory that is replicated among replica members in a replica set.

Retry change order A change order that is in some state of completion but was blocked for some reason and must be retried later.

Staging directory A directory that acts as a buffer by retaining copies of updated files until replication partners successfully receive them and move them into the replica tree. The staging directory acts as a queue for changes to be replicated to downstream partners. After the changes are made to a file and the file is closed, the file content is compressed, written to the staging directory, and replicated according to schedule. Any further use of that file does not prevent FRS from replicating the staging file to other members. In addition, if the file is replicated to multiple downstream partners or to members with slow data links, using a staging file ensures that the underlying file in the replica tree can still be accessed.

Staging file In a replica tree, a file or directory that is constructed in the staging directory and is replicated to all replica members. Staging files are generated as a result of a file or directory change.

SYSVOL On a domain controller, a shared directory that stores a copy of the domain’s public files, including system policies and Group Policy settings, which are replicated to all other domain controllers in the domain by using the File Replication service.

Upstream partner For a given replica member, the set of servers providing data to that replica member. Upstream partners are also known as inbound partners.

USN A monotonically increasing sequence number for each volume. The USN is incremented every time a modification is made to a file on the volume.

USN journal On NTFS volumes, a persistent log that tracks all changes on the volume, including file creations, deletions, and changes. The USN journal has a configurable maximum size and is persistent across reboots and system crashes. FRS uses the USN journal to monitor changes made in the replica tree.

USN journal wrap An error that occurs when large numbers of files change so quickly that the USN journal must discard the oldest changes (before FRS has a chance to detect the changes) to stay within the specified size limit. To recover from a journal wrap, you must perform a nonauthoritative restore on the server to synchronize its files with the files on the other replica members.

Version vector A vector of USNs, with one entry per replica set member. All change orders carry the originator GUID of the originating member and the associated USN. As each replica member receives the update, it tracks the USN in a vector slot that is assigned to the originating member. This vector describes whether the replica tree is current with each member. The version vector is then used to filter updates from inbound partners that may have already been applied. The version vector is also delivered to the inbound partner when the two members join. When a new connection is created, the version vector is used to scan the File ID table for more recent updates that are not seen by the new outbound partner.

Version vector join The process whereby a new replica member joins a replica set.


Common Scenarios for FRS

FRS is widely used by Windows 2000 and Windows Server 2003 for a variety of scenarios, from SYSVOL replication to application and document distribution.

SYSVOL

In a Windows 2000 or Windows Server 2003 Active Directory domain, there are two important folders shared out by each domain controller:

  • \\<domain_name>\SYSVOL, used by Windows XP and Windows 2000 clients to obtain Group Policy.
  • \\<domain_name>\NETLOGON, used by the Windows logon process and by clients accessing legacy logon scripts and policy

Distributed File System (DFS) is automatically used to create this \\domain_name\... network name. These file shares actually map via DFS to shared folders hosted by each of the domain controllers in the domain domain_name. For example, the DFS link resolves to the file shares \\domain_controller_X\SYSVOL and \\domain_controller_X\NETLOGON.

Active Directory requires the contents of these individual file shares to be consistent throughout the domain, and FRS is used to achieve this goal.

In large domains, the SYSVOL replica set can span hundreds or even thousands of computers. SYSVOL is used for a key part of Active Directory infrastructure, including default domain policy and other parts of group policy. For this reason, there are specific recommendations for the type of data that should and should not be stored in SYSVOL:

  • Policy files and login scripts should be kept in SYSVOL.
  • Login scripts should be designed so that clients do not hold them open for indefinite periods of time. For example, do not use login scripts that can be delayed while waiting for a user to press a key to continue.
  • Application or hotfix install points should not be kept in SYSVOL. We recommend that you set up a separate DFS namespace for these.

For SYSVOL, FRS uses the same connection topology that is manually or automatically created for Active Directory replication. Because the connection objects are the same, the schedule and topology for intrasite and intersite replication are the same for FRS and Active Directory. Like Active Directory replication, FRS compresses all replicated content between sites, uses a trigger replication scheme, and implicitly uses an always-on schedule between members in the same site. However, unlike Active Directory replication, FRS also compresses replicated content within a site.

Replicating DFS Link Targets

Unlike SYSVOL replication, which is enabled by default, you must explicitly enable replication in a DFS namespace by using the Distributed File System snap-in. Only targets of roots and links in a domain-based DFS can use FRS; stand-alone DFS does not support automatic file replication using FRS.

You can enable replication of files and folders between computers by using the Configure Replication Wizard in the Distributed File System snap-in. The replication policy can be different for each root and link in the DFS namespace. You must have at least two targets configured before you can enable replication.

Important

  • We recommend that you enable FRS replication only on DFS links. Avoid enabling replication on DFS roots.

Publishing Applications

FRS and DFS are two separate services that can be combined to build a way of distributing and publishing applications. DFS provides the means to construct a global namespace spanning multiple servers and file shares. It provides the capability to have any portion of the namespace (DFS links) hosted by multiple servers (DFS link targets) in one or more sites. In this way it provides availability, load balancing, and reduced latency and bandwidth usage by referring the client to a server in a local or nearby site.

In the DFS/FRS publication scenario, a distribution topology is defined (typically hub-and-spoke, a multilevel tree, or a redundant hub-and-spoke) and by convention, new applications are deployed or updated on a single computer (usually the hub or root of the topology or another centrally located computer), and FRS then propagates the changes to the spoke computers.

These configurations have proven very effective in the field, as long as the rules described in the Designing an FRS Deployment are adhered to.

Note

  • Some organizations have adopted a policy whereby they run applications from a network file share instead of installing them onto local systems. For this scenario, and where Windows XP clients are in use, administrators can take advantage of Offline Folders in order to cache application files on the local hard disks on client computers and thus reduce server load and increase application availability.

Publishing Data

An almost identical scenario is publication of data across an enterprise. Common examples include documents, diagrams, operational procedures, and historical/test result data. This scenario follows the same advice given for publication of data.

Another method of publishing data is known as reverse publication, where the data flows from a number of dispersed computers to a centralized data collection server. One typical scenario is when collections of log files and reports generated on individual computers are collected to a central site. Another similar scenario when data is collected to a central server for backup purposes.

Typically this is configured using a replica set for each (hub, remote site) pair, because the intent is that the data flows to the central data collection server, and not to each of the servers generating the data. This is sometimes described as a petal topology, because the diagram for such a configuration shows a central hub and each replica set looks like the petal of a flower replicating between the hub and a single computer.

Ensuring the Availability of User Data

An alternate set of scenarios revolve around supporting a large number of independent users who want to read and write data in their own personal folders on a file server. To provide availability against disk or system failure, it is possible to use FRS as a way to replicate data for the purpose of data availability and use DFS to provide failover

However, these scenarios need more careful consideration since they are not a core scenario for DFS/FRS; instead, it is a core scenario for Microsoft Cluster Services (MSCS).

  • FRS uses a "last writer wins" algorithm for files. This algorithm is applied in two situations: when the same file is changed on two or more servers, and when two or more different files with the same name are added to the replica tree on different servers. The most recent update to a file in a replica set becomes the version of the file that replicates to the other members of the replica set, which might result in data loss if multiple masters have updated the file. In addition, FRS cannot enforce file-sharing restrictions or file locking between two users who are working on the same file on two different replica set members. (Note that deletes are a "last write" event and can ripple through all members of the set, taking a single, directory or the entire tree with it.)
  • FRS uses a "last writer wins" algorithm when a folder on two or more servers is changed, such as by changing folder attributes. However, FRS uses a "first writer wins" algorithm when two or more identically named folders on different servers are added to the replica tree. When this occurs, FRS identifies the conflict during replication, and the receiving member protects the original copy of the folder and renames (morphs) the later inbound copy of the folder. The morphed folder names have a suffix of "_NTFRS_xxxxxxxx," where "xxxxxxxx" represents eight random hexadecimal digits. The folders are replicated to all servers in the replica set, and administrators can later merge the contents of the folders or take some other measure to reestablish the single folder.

Temporary data inconsistency due to replication latency is more likely to occur in geographically diverse sites with infrequent replication across slow WAN links. If you want to use replication among servers in the same site, consistency is probably not an issue, because the replication can occur quickly after the file changes — assuming that only one user makes changes to the data. If two users make changes to the data, replication conflicts occur and one user loses those changes.


Appropriate Scenarios for Using FRS

FRS works well in the following scenarios.

When the data is read-only or changes infrequently

Because changes occur infrequently, the data is usually consistent. In addition, FRS has less data to replicate, so network bandwidth is not heavily affected.

When the sites are geographically dispersed and consistency is not an issue

Geographically dispersed sites might have slower bandwidth connections, but if your organization does not require the data in those sites to always be consistent with each other, you can configure replication in those sites on a schedule that make sense for your organization. For example, if your organization has sites in Los Angeles and Zimbabwe, you can place one or more replicas of the data in servers in those sites and schedule replication to occur at night or during periods of low bandwidth use. Because in this scenario replication could take hours or days to update every member, the delay must be acceptable to your organization.

When each file is changed by only one person from one location

Replication conflicts rarely occur if only a single user changes a given file from a single location. Some common scenarios for single authorship are redirected My Documents folders and other home directories. Conversely, if users roam between sites, replication latency could cause the file to be temporarily inconsistent between sites.

When replication takes place among a small number of servers in the same site

Replication latency is reduced by frequently replicating data using high-speed connections. As a result, data tends to be more consistent.

As a file server fail over configuration, if some data inconsistency between servers can be tolerated

It is possible to use DFS and FRS to replicate read-write user data so that if one file server fails, another can take its place. However, before deploying such a scenario, the following factors must be taken into account to determine if DFS and FRS are appropriate in the context of the planned scenario.

The issues to consider are:

  • DFS does not guarantee which file server a client will be referred to; there are clear rules around how DFS load balancing and site selection work, but depending on transient network issues, a DFS client might attach to any candidate server that advertised the file share used by an enabled DFS link target.
  • The FRS last-writer-wins conflict resolution model means that if two client computers (with either the same or different users logged in) access a replicated DFS link and are directed to different link targets, then these two clients can make changes to the two copies of the same file without being aware of each others’ locks on the files. One of these two clients will silently lose the changes they make.
  • FRS can only replicate file changes after the file is closed.
  • FRS replicates whole files each time; if a file is only modified in one small area, then FRS still transmits the complete file. This is acceptable for many files, but, depending on bandwidth, may not be appropriate for files such as .PST files, which are large but typically undergo change in small areas of the file.

In some scenarios, this can still be acceptable. The key question is how likely it is that conflicting edits may be made by two different client computers to the same file before the data has had time to replicate.

Another concept is to consider a mechanism (such as scripts) whereby only one of the link targets raises its shared folder at a time. In this case, DFS can only ever successfully route a client computer to one file server, and so such write conflicts cannot occur. In this case, failover is provided by deciding one other member of the replication set can raise its file share, and the failing member is disconnected and has its file share lowered.

The final issue to consider in this scenario is bandwidth usage. Because users are updating files, there is no clear bound to how much replication traffic they may generate, and this should be considered carefully in replica sets that are intended to span a wide area network (WAN).


Inappropriate Scenarios for Using FRS

Replication should not be used in the following scenarios.

In organizations with no operations group or dedicated administrators

Organizations that do not have the staff or the time to monitor FRS event logs on each replica member should not implement FRS. Organizations must also have well-defined procedures in place to prevent the accidental or unintentional deletion of data in the replica set, because deleting a file or folder from one replica member causes the file or folder (and its contents) to be deleted from all replica members. In addition, if a folder is moved out of the replica tree, FRS deletes the folder and its contents on the remaining replica members.

Note

  • In Windows Server 2003, to avoid having to restore the files or folders from backup, you can enable shadow copies on some of the replica members so that you can easily restore a file or folder that was accidentally deleted.

In organizations that do not update virus signatures or closely manage folder permissions

A virus in FRS-replicated content can spread rapidly to replica members and to clients that access the replicated data. Viruses are especially damaging in environments where the Everyone group has share permissions or NTFS permissions to modify content. To prevent the spread of viruses, it is essential that replica members have FRS-compatible, up-to-date virus scanners installed on the servers and on clients that access replicated data.

For information about running antivirus software on SYSVOL, see article 822158, "Virus Scanning on a Domain Controller" at http://support.microsoft.com/?id=822158. For information about antivirus software that is compatible with FRS, see 815263, "Antivirus, Backup, and Disk Optimization Programs That Are Compatible with the File Replication Service" at http://support.microsoft.com/?id=815263.

When the rate of change exceeds what FRS can replicate

If you plan to schedule replication to occur during a specified replication window, verify that FRS can replicate all the changed files within the window. Replication throughput is determined by a number of factors:

  • The number and size of changed files
  • The speed of the disk subsystem
  • The speed of the network
  • Whether you have optimized the servers by placing the replica tree, the staging directory, and the FRS data on separate disks.

Each organization will have different FRS throughput rates, depending on these factors. In addition, if your data compresses extremely well, your file throughput will be higher. To determine the replication rate, perform testing in a lab environment that resembles your production environment.

If the amount of data changes exceeds what FRS can replicate in a given period of time, you need to change one of these factors, such as increasing the speed of the disk subsystem (number of disks, mechanical speed, or disk cache) or network. If no change is possible, FRS is not recommended for your organization.

In organizations that always use clustered file servers

Some organizations use clustered file servers regardless of whether the server contains business-critical data. Although it might seem that storing FRS-replicated content on the shared cluster storage of a clustered file server would increase the availability of the data even more, combining clustering and FRS is not recommended because you then have the weaker guarantees of asynchronous file replication, but the more stringent configuration requirements of a cluster. In addition, Windows Server 2003 and Windows 2000 Server do not support configuring FRS to replicate data on cluster storage.

In organizations that use Remote Storage

Remote Storage is a feature that automatically copies infrequently used files on local volumes to a library of magnetic tapes or magneto-optical disks. Organizations that use Remote Storage must not use FRS on the same volume. Specifically, do not perform any of the following tasks:

  • Do not create a replica set on a volume that is managed by Remote Storage.
  • Do not add a volume that contains folders that are part of an FRS replica set to Remote Storage.
  • If you use Remote Storage for volumes that contain FRS replica sets, backup tapes might be damaged or destroyed if FRS recalls a large number of files from Remote Storage. The damage occurs because FRS does not recall files in media order. As a result, files are extracted randomly, and the process can take days to complete and might damage or destroy the tape in the process. Random extraction from magneto-optical disks can also be extremely time consuming.

When locks by users or processes prevent updates to files and folders

FRS does not replicate locked files or folders to other replica members, nor does FRS update a file on a replica member if the local file is open. If users or processes frequently leave files open for extended periods, consider using clustering instead of FRS.

FRS does not replicate files until the file has been closed for 3 seconds. If files are changed but are then held open (or reopened) exclusively, before FRS is able to create a staging file of that change, then FRS cannot replicate the file until it has been closed again.

When the data to be replicated is on mounted drives

If a mounted drive exists in a replica tree, FRS does not replicate the data in the mounted drive.

When the data to be replicated is encrypted by using EFS

FRS does not replicate files encrypted by using EFS, nor does FRS warn you that EFS-encrypted files are present in the replica set.

When the FRS jet database, FRS logs, and staging directory are stored on volumes where NTFS disk quotas are enabled

If you plan to store a replica set on a volume where disk quotas are enabled, you must move the staging directory, FRS jet database, and FRS logs to a volume where disk quotas are disabled.


Designing an FRS Deployment

There are a small number of golden rules that should be followed when planning a FRS to ensure a consistent service levels.

After you have chosen FRS as the means to replicate data, you should support that service and not supplement or override it

If replication stops for some reason, the very worst thing you can do is to copy files manually to replication partners. This will cause additional replication traffic, backlog, and possible replication conflicts.

The correct action is to find the root cause that stopped replication from progressing and resolve that. Common causes include low disk free space, poor connectivity, excessive file updates, and files that are in use and cannot be replicated.

Proactively monitor the status of replication by using Sonar or Ultrasound.

If a system problem has prevented replication from progressing, then it is important to determine this and remedy the issue in a timely manner.

If replication outages persist unnoticed for long periods of time, then failures start to compound upon each other. For example, backlogs of data on one computer can cause accumulation of files in the staging directory to radiate out to other computers and progressively cause network-wide congestion, and also can affect other replica sets on the same computers.

For more information about monitoring FRS, see Monitoring FRS Using Sonar and Ultrasound Overview.

Look for unexpected sources of replication traffic and file locks

Any application can potentially alter a file. Some applications can unexpectedly alter many files on a regular basis. Examples include disk defragmentation tools, antivirus software, and file system policy templates.

Some applications might hold files open for long periods of time, preventing updated files from being applied from another computer. The most common example of this is "press-a-key" prompts and message boxes in login scripts where the user goes away for a long period of time after logging in.

An operator in one part of a system might make changes without realizing that another operator is making conflicting changes in the same area from some other computer. This is known as "dueling admins" and in some cases a manual replication feedback loop is created as the admins keep reapplying their changes, unaware that another is also making changes.

Build an FRS deployment plan that handles bandwidth availability, topology definition, data quantity, data change rate, and monitoring procedures

There are no fixed limits to how much data or data change rate that FRS can support; however, with more data and more volatile data, the system designer needs to consider how to optimize the topology and replica member configuration to support the required level of replication traffic.

Take the time to understand how FRS works

A significant amount of detailed FRS training material is now available. If you are using FRS in an advanced manner (large/complex topology, large amounts of replication traffic, and so on) then nominate a staff member to undergo this training so that they can be aware of potential issues and how to troubleshoot and repair any that occur.


Building a Deployment Plan for an FRS Replica Set

This section discusses how to plan a single replica set. The aim is to ensure that the rate of change in the data set can be accommodated by the available communication bandwidth and schedule.

Planning for bandwidth availability

The first aspect to consider is the bandwidth availability between nodes in the replication set:

  • Determine the raw available bandwidth between nodes, especially the bandwidth between the most poorly connected nodes.
  • Define a replication schedule for FRS. Each FRS connection can have its own schedule defined on an hourly basis.

If you are using schedules and hubs with many outgoing connections, it is a best practice to stagger the schedules so that the hub computer does not try to deliver a backlog of replication traffic to all partners at once. This reduces the amount of simultaneous disk I/O that the hub server needs to perform and will help to reduce timeouts from partners. The extent to which the schedules should be staggered should be based on number of connections to the hub, and the CPU and disk performance of the hub server.

Note that scheduling is the only mechanism that FRS provides for throttling replication traffic.

Planning for data size and change rate

When a file is modified, FRS sends a complete copy of the resultant file to the computer’s replication partners.

If you have an existing data set that you want to replicate, use the following procedure to estimate how much replication traffic will be generated in a given time period:

  1. Use ATTRIB –A <path> /S
  2. Use the data set at <path> as normal
  3. After the chosen interval (for example daily), use XCOPY /A /S <path> <temp_dir>

    Note that both <path> and <temp_dir> should be on NTFS volumes.

  4. Ensure the files in <temp_dir> are compressed by executing COMPACT /C /S:<temp_dir>. When this command completes, the summary report indicates how many bytes are being used to store the files, for example:
    405 files within 11 directories were compressed.
19,862,282 total bytes of data are stored in 10,677,439 bytes
The compression ratio is 1.9 to 1.

Note that replication schedule also plays an important role here. If there are multiple versions of a file in the FRS replication queue from a computer, then FRS will only send the most recent version, not all of the intermediate versions. For this reason, a schedule coalesces file changes and reduces bandwidth usage at the expense of a short-term backlog.

Expected propagation requirements

If the topology has multiple levels from the master to its furthest node (based upon the number of hops in the replication topology), and the connections have schedules defined, then the system designer should consider the likely propagation delay for the file to reach that final target.

Planning the number of replica sets per server

When using FRS to replicate DFS links as replica sets, there is no fixed limit to the number of replica sets that a single file server can be involved in; however, we recommend that you host no more than 150 different replica sets on a single server to ensure replication performance. The optimal number of replica sets for servers in your organization depends on the CPU, memory, disk input/output (I/O) throughput, and the amount of data changed.

Configuring filters to exclude files and folders from replication

You can use the Distributed File System snap-in to create filters that exclude subfolders or files with certain extensions from replication. By default, the following files are excluded from FRS replication:

  • File names starting with a tilde (~) character
  • Files with .bak or .tmp extensions

Expected growth of replicated data

You need to know whether you plan to replicate larger and larger amounts of data over time so that you can ensure that your topology, schedule, and bandwidth can handle the additional data.

Expected increase in the number of replica members

If you plan to deploy a small number of servers at first and then deploy additional servers over time, you need to ensure that your topology and bandwidth can handle the new servers.


FRS Topologies

FRS supports arbitrary topologies subject to requirement of the replica set type.

  • SYSVOL replica sets require that each DC in the same domain have at least one inbound and one outbound connection to another DC in the same domain.
  • DFS replica sets only require that each member has a single inbound connection, so it is possible to have an all one-way topology for all members.
  • In any valid topology, there must be a direct or indirect path between any two members of the replica set.

FRS does not rely on broadcast technology and can provide fault tolerant distribution via multiple connection paths between members. If a given replica member is unavailable, the data will flow via a different route assuming one exists. Dampening logic prevents a data file from being sent more than once to any given member.

Replicating data between sites that are connected with slow WAN links requires careful planning of the topology and schedule. If the sites have a high-bandwidth connection, but business-critical databases and other applications use that connection as well, you will need to schedule replication so that it does not consume bandwidth needed for other uses.

A key part of topology definition involves choosing a trade-off between redundancy and scalability. Topologies with more redundant paths offer greater fault tolerance by providing the largest number of alternate routes for data transfer, but at the same time this adds administrative overhead (more connections to monitor and support) and potential liability (a single offline downstream partner now affects additional upstream partners).

The full mesh topology in which every node has a connection to every other is the most extreme case: it provides the most redundancy, but the least scalability. By contrast, the single-hub-and-spoke topology provides the least redundancy, but provides better scalability. Consequently, the best topology for a given scenario typically lies in a continuum between the extremes of full mesh and single-hub-and-spoke, and balances scalability and redundancy.

In the majority of cases (especially large ones) the optimal topology is often redundant hub-and-spoke with staggered schedules. In this arrangement, a small number of members (the hubs) have direct inbound and outbound connections with every other member (the spokes and the other hubs). There are no additional connections. The connections from the hubs to the spokes have staggered schedules so that the hubs do not get overloaded by replicating files to all partners at once. The hubs are well-connected to each other and typically have constant (or very frequent) schedules.

A case that is common within sites is the ring topology. In this case, all members are connected as if they were points on a circle, and each member has inbound and outbound connections with only its immediate neighbors.

Configuring the desired replication topology (DFS case)

Administrators should use the updated Distributed File System snap-in the Windows Server 2003 Administration Tools Pack. You can install this pack on computers running Windows XP with Service Pack 1 (SP1) or later and create FRS schedules and topologies on remote servers running Windows 2000. This snap-in offers a number of important benefits over the version of the tool that shipped in Windows 2000:

  • The updated tool provides the ability to define a customized replication topology for both Windows 2000 and Windows Server 2003 systems. In comparison, the Windows 2000 Distributed File System snap-in always created full-mesh topologies with a constant schedule.
  • The updated tool allows schedules to be defined on each connection.
  • The updated tool allows connection priority to be defined on each connection. This is an important parameter to set for redundant-hub-and-spoke topologies since it can be used to identify to a hub where the other hubs are. This information is also used during recovery process (non-authoritative restore, also known as the D2 process) to ensure that it sources data from the well-connected hub server.
  • Allows the administrator to specify staging directory location as each target is added to FRS replica set.

For more information, see article 304718, "Administering Windows 2000-Based and Windows Server 2003-Based Computers Using Windows XP Professional-Based Clients" at http://support.microsoft.com/?id=304718.

Configuring the desired replication topology (SYSVOL case)

The techniques and tools for configuring SYSVOL replication sets is described in length, along with tools and scripts, in the Windows 2000 Branch Office Deployment Guide available at http://go.microsoft.com/fwlink/?LinkId=16718.

It is essential that administrators with large configurations (over 100 sites) review the Branch Office Deployment Guide (especially the topology section) to decide if a custom, scripted topology is required. Even for smaller configurations, this is a very important and useful document and tool set.


Optimizing FRS Replication Schedules

You might need to change the default FRS replication schedules for SYSVOL and DFS replicas to maximize your network bandwidth. Keep in mind as you consider your replication schedule that FRS replicates whole files, and this can impact the network bandwidth that is available. You might decide, for example, to schedule replication during off hours in order to maximize your bandwidth during business hours. Becoming familiar with the procedures for configuring replication schedules for both DFS and SYSVOL will assist you in scheduling replication for your organization.

You can build a staggered and well-balanced topology using the MKDSX scripts from the Branch Office Deployment Guide available at http://go.microsoft.com/fwlink/?LinkId=16718.

Detailed discussion of this subject is beyond the scope of this document. However, for Active Directory replication, a rule of thumb is that a given domain controller that acts as a bridgehead server should not have more than 50 active simultaneous replication connections at any given time in a replication window. This was determined on a reference server that had four Pentium III Xeon processors with 2 gigabytes (GB) of RAM and 2 megabytes (MB) of L2 cache. Adjusting this rule to a limit of fewer than 50 servers will have significant positive impact on CPU utilization, network throughput, and I/O throughput on this domain controller

Note

  • If you use the MKDSX script, note that the existence of multiple domains in the forest and the new Application Partitions in Windows Server 2003 make use of manual topologies more complex. Also note that the updated KCC in Windows Server 2003 has improved scalability compared to the Windows 2000 version.

For additional information about scheduling replication, see Scheduling SYSVOL Replication and Scheduling FRS Replication for DFS Replica Sets.


Scheduling SYSVOL Replication

FRS replicates SYSVOL using the same intrasite connection objects and schedule built by the KCC for Active Directory replication. The connection object schedule is an attribute associated with each connection object. The connection object schedule contains a 7x24 array of bytes, one byte for each hour in a 7-day week. The low 4 bits of each byte are used to indicate the number of times replication is attempted in that hour. The upper 4 bits of each byte are reserved for future use. Intrasite SYSVOL replication occurs once per hour by default, unless changes are made that trigger replication. When information in the SYSVOL folder is modified, FRS replicates the information immediately to all replication partners.

The example connection object schedule below (each entry is 2 hexadecimal digits) would trigger both SYSVOL and Active Directory replication on the even hours of the week. Note that for SYSVOL, this schedule applies to intersite replication and not to intrasite replication. For intrasite SYSVOL connections, schedules are implicitly always on.

                                    Hour of the day
      00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
  sun=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
D mon=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
A tue=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
Y wed=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
S thu=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
  fri=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00
  sat=01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00 01 00

Note

  • Note that this is how the schedule information is interpreted by the service; the actual user interface does not present the schedule in this form. You can, however, view the connection object schedule for a particular computer by using Ntfrsutl.exe.

The SYSVOL replication schedule can be configured so that replication takes place four, two, one, or zero times per hour.

  • When set to four times per hour, replication starts at 0:00, 0:15, 0:30, and 0:45. This schedule is essentially continuous replication.
  • When set to two times per hour, replication at 15-minute intervals starting at 0:15 and 0:45.
  • When set to one time per hour, replication starts at 0:00 and ends at 0:15 assuming that all changes have been replicated.

Note that for SYSVOL, FRS will continue to replicate until all changes are replicated, regardless of whether the schedule closes.

Changing Schedules on SYSVOL Connection Objects

It is possible to change schedules on connection objects, but this should be done only after careful consideration of the implications and alternatives:

  • Note that schedules exist on both site links and connection objects.
  • If you change the schedule on a connection object, the connection then becomes a manual connection that cannot be managed by KCC.

To view and change the schedule on a connection object for SYSVOL

  1. In Control Panel, double-click the Administrative Tools icon.
  2. Double-click the Active Directory Sites and Services icon.
  3. In the console tree, expand Sites, and then expand the site of interest.
  4. Expand Servers, expand the server of interest, and click NTDS Settings.
  5. Right-click the connection object of interest, and click Properties.
  6. Click Change Schedule to view or change the schedule for that connection.


Scheduling FRS Replication for DFS Replica Sets

For DFS replica sets, FRS uses the connection objects, topology, and schedule built by the Distributed File System snap-in. However, a schedule for a DFS replica set can be assigned globally to the replica set or overridden on each individual connection. Generally, it is preferable to change the schedule on the replica set, unless you specifically want to have exceptions or want to stagger schedules. Connection object schedules override a schedule assigned to a replica set.

For example, assigning a schedule to the replica set might be more appropriate for a replica set with a large number of replicas, because it would be a tedious process to configure the schedules on all of the connection objects for a replica set with over 100 members.

You schedule replication for a DFS replica set by making replication either available or unavailable. When set to available, replication occurs four times per hour at 0:00, 0:15, 0:30, and 0:45, which is essentially continuous replication. Unlike SYSVOL replication, replication for a DFS replica set stops after the schedule closes, even if all changes have not been replicated.

To view or change the schedule on a connection object for a replica set in Windows 2000

  1. In Control Panel, double-click the Administrative Tools icon.
  2. Double-click the Active Directory Users and Computers icon.
  3. On the View menu, click Advanced Features.
  4. In the console tree, expand System, expand File Replication Service, and expand DFS Volumes.
  5. Expand the DFS root.
  6. Right-click the root/link of interest, and click Properties.
  7. Click Change Schedule to view the schedule for that connection.

Note again that if a schedule is assigned to a connection object within this replica set, it overrides the schedule assigned to the replica set.

To view or change the schedule on a connection object for a replica set in Windows Server 2003

  1. In the Distributed File System snap-in, right-click the link where you configured FRS, and then click Properties.
  2. Click the Replication tab, and then do one of the following:
    • If you want to set the same schedule for all connections, click Schedule.
    • If you want to set a schedule for each connection, click Customize. In Connections, select a connection, and then click Schedule.
  3. Adjust the schedule as necessary.


FRS Best Practices

The following sections describe best practices to help ensure trouble-free FRS operations.

  • Use the Current Service Packs and Post-Service Pack Releases
  • Build and Maintain a Picture of the Environment
  • Adjust the USN Journal Size Accordingly
  • Adjust the Staging Directory Size Accordingly
  • Choose the Location of the Staging Directory
  • Keep the FRS Service Running
  • Never Help the Replicator
  • Use Appropriate Hardware
  • Create Support Procedures and Train the Operations Staff
  • Regularly Monitor FRS to Detect Problems or Failures


Use the Current Service Packs and Post-Service Pack Releases

To reduce the likelihood of FRS errors, it is recommended that all replica members run the current service packs and post-service pack releases for either Windows 2000 Server or Windows Server 2003.

Windows 2000 Server

We recommend that you install one of the following:

  • Windows 2000 Server with SP3 and the post-Service Pack 3 release of Ntfrs.exe, described in article 815473, "File Replication Service Does Not Log Errors on Sharing Violations" at http://support.microsoft.com/?id=815473.
  • Windows 2000 Server with SP4.

Windows Server 2003

Windows Server 2003 and the pre-Service Pack 1 release of Ntfrs.exe as described in article 823230, "Issues That Are Fixed in the Pre-Service Pack 1 Release of Ntfrs.exe" at http://support.microsoft.com/?id=823230.

Note

  • For current information about FRS, see the DFS and FRS Web site at http://go.microsoft.com/fwlink/?LinkId=18421.


Build and Maintain a Picture of the Environment

To aid in monitoring and troubleshooting FRS, maintain a clear picture of your environment. A common approach is to create a Visio diagram that describes the replica set configuration. Include the following information in your diagram:

  • Replica members and their locations
  • Bandwidth between servers and sites
  • Replication topology
  • Master, hub, and spoke servers
  • Available disk space on members
  • How much data is in the replica set
  • How much data gets changed and how often (and therefore replicated on a daily basis)


Adjust the USN Journal Size Accordingly

When sizing the USN journal, use 128 MB for every 100,000 files on the volume. If files on the volume are changed or renamed frequently (regardless of whether they are part of the replica set), consider sizing the USN journal larger than these recommendations to prevent USN journal wraps, which can occur when large numbers of files change so quickly that the USN journal must discard the oldest changes (before FRS has a chance to detect the changes) to stay within the specified size limit. To recover from a journal wrap, you must perform a nonauthoritative restore (also called a D2) on the server to synchronize its files with the files on the other replica members.

When determining the USN journal size, also consider the following:

  • Multiple replica trees can exist on the same NTFS volume. Therefore, choose a USN journal size based on the total number of files managed by FRS on that volume.
  • Other files might also be stored on the same NTFS volume, and file operations on these files are tracked in the USN journal, even though FRS ignores these changes. As a general guideline, add an extra 8 MB per 100,000 files in this case, though this is an approximate number that depends on how much activity occurs on those files.
  • For best results, put replica trees on their own NTFS volume (or volumes) so that other file activity does not affect the USN journal on the volumes storing replica trees.

Use the following procedure to adjust the USN journal size. Note that the USN journal size applies to all volumes on the server.

To change the USN journal size

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters
  3. Right-click the Parameters subkey and choose New -> DWORD Value.
  4. Type NTFS journal size in MB as the new value name and press Enter.
  5. Right click NTFS journal size in MB and click Modify.
  6. Click Decimal, and then in the Value data field, type the size in megabytes that you would like to make the NTFS journal.
  7. Click OK, and then close the Registry Editor.
  8. At the command prompt, type net stop ntfrs & net start ntfrs.


Adjust the Staging Directory Size Accordingly

The staging directory acts as a queue for changes to be replicated to downstream partners. After the changes are made to a file and the file is closed, the file content is compressed, written to the staging directory, and replicated according to schedule. Any further use of that file does not prevent FRS from replicating the staging file to other members. In addition, if the file is replicated to multiple downstream partners or to members with slow data links, using a staging file ensures that the underlying file in the replica tree can still be accessed.

The size of the staging directory governs the maximum amount of disk space that FRS can use to hold those staging files and the maximum file size that FRS can replicate. The default size of the staging directory is approximately 660 MB, the minimum size is 10 MB, and the maximum size is 2 TB. The largest file that FRS can replicate is determined by the staging directory size on both the upstream partner (the replica member that sends out the changed file) and downstream partners (the replica member that receives the changed file) and whether the replicated file, to the extent that it can be compressed, can be accommodated by the current staging directory size. Therefore, the largest file that FRS can replicate is 2 TB, assuming that the staging directory size is set to the maximum on upstream and downstream partners.

Factors that Affect the Staging Directory Size

The ratio of staging area size to data set size depends upon a range of factors. For additional guidelines, see article 329491, "Configuring Correct Staging Area Space for Replica Sets" at http://support.microsoft.com/?id=329491.

Largest file to be replicated

The largest file that FRS can replicate is determined by the staging directory size on both the upstream and downstream computers. Therefore, the largest possible file that FRS can replicate is 2 TB, when the staging directory size has been set to this maximum value.

If using Windows 2000 SP2 or later

Windows 2000 SP2 and later compresses the data in the staging directory. Some file types (text, some binaries, documents) are more compressible than others (compressed archives, and multimedia files).

If using Windows 2000 SP3 or later service packs or Windows Server 2003

If you are using Windows 2000 SP2 or earlier, be aware that FRS stops replicating if the staging directory runs out of free space. This means that if a replica set member goes offline for an extended period of time, it does not block replication on an upstream member because the staging directory is filled. Consequently, you should use a generous estimate for staging directory size.

However, Windows 2000 SP3 and later service packs and Windows Server 2003 have an updated staging file management algorithm. On these servers, when FRS tries to allocate space for a staging file and is not successful (because there is not enough space or because the amount of space in use has reached 90 percent of the staging space limit parameter), FRS starts to delete staging files. Staged files are deleted (in the order of the longest time since the last access) until the amount of space in use has dropped below 60 percent of the staging space limit parameter. Consequently, it is not as critical to use as generous an estimate for staging directory size as it was for pre-Windows 2000 SP3 systems, but it is still advised to do so to prevent disk and CPU performance being consumed by repeatedly staging and deleting files.

The amount of changed files to be replicated

FRS replicates whole files that have changed, so the rate of change is the sum of the sizes of files modified, not the sum of the sizes of changes to files. There is also the issue of the multiple changes to the same file. FRS can enter a file into the staging directory multiple times — once for each time it was written and closed (but note that the FRS aging cache prevents more than one change order and staging file being generated within 3 seconds).

The ability of downstream computers to accept files

The ability of downstream partners to accept files is a key factor in determining staging directory size. Other factors here include:

  • Replication schedule between partners. How long must the files wait for a chance to be replicated?
  • Availability of partners. Issues like planned/unplanned downtime can cause backlogs. An outbound partner that has not connected for a while can cause a lot of staging space to be required.
  • Bandwidth available between partners. How long will it take to replicate the files?
  • Number of downstream partners. FRS staging space is decided by the slowest partner.

Adjusting the Staging Directory Size

Use the following procedure to adjust the staging directory size.

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters.
  3. Right-click the Staging Space Limit in KB entry and click Modify.
  4. Click Decimal, type the desired size of the staging directory in kilobytes, and click OK.
  5. Close the registry editor.
  6. At the command prompt, type net stop ntfrs & net start ntfrs


Choose the Location of the Staging Directory

You can choose the location of the staging directory when you use the Distributed File System snap-in in Windows Server 2003 or the Windows XP Administration Tools pack to configure a replica set on Windows 2000. If you configure a replica set using the Distributed File System snap-in in Windows 2000 Server, the staging directory location is chosen automatically.

By default, all replica sets on the server use the same staging directory, and the staging directory space limit applies to all staging directories on the local server. When the staging directory is shared among multiple replica sets, any one replica set can cause the staging directory to reach 90% capacity, at which point FRS begins to delete the oldest staging files for any of the replica sets. (Or in the case of Windows 2000 SP2 or earlier, FRS stops replicating if the staging directory runs out of free space.)

To simplify staging directory management and any future recovery procedures, it is recommended that you configure replica sets in one of the following ways:

  • Give each replica set its own staging directory, and place the staging directories on the same volume. This solution provides good recoverability while minimizing cost.
  • Give each replica set its own staging directory, and put each staging directory on its own volume. This solution provides better recoverability, but at a higher cost.

For information about moving the staging directory, see article 291823, "How to Reset the File Replication Service Staging Folder to a Different Logical Drive" at http://support.microsoft.com/?id=291823.

Do not store the staging directory on a volume where NTFS disk quotas are enabled.


Keep the FRS Service Running

To minimize the chance of encountering a USN journal wrap, keep FRS service running at all times, especially when bulk changes (adds, deletes, or changes) are being made to FRS-replicated files.


Never Help the Replicator

If it appears that files are not replicating correctly, never attempt to copy those files yourself. Instead, investigate why replication is not occurring. See Troubleshooting FRS for troubleshooting information.


Use Appropriate Hardware

FRS can consume considerable amounts of disk space, disk I/O, CPU, and bandwidth, depending upon the replication traffic it needs to manage. This applies especially to hub systems, and it is recommended that system designers consider providing hubs with faster CPU and faster and larger disk subsystems. In particular, using separate disks for the FRS database, staging directory, and replicated content will provide the most significant performance gains.


Create Support Procedures and Train the Operations Staff

All administrators who add or change content in FRS replica trees should understand the implications of multimaster replication, including the following points:

  • FRS supports a multi-master file replication model in which any computer accepts and replicates changes to any other computer taking part in the replication configuration.
  • Administrators should be aware of the bandwidth impacts of changes that they make in the replica tree.
  • Any changes made to a file in a replica tree will occur on all other members of the replica set. Administrators must never attempt to help FRS by making the same change on other members of the replica set.
  • If an administrator makes a change to a file, and it later seems the changes have been reversed, then it might be a case of "dueling administrators;" that is, another operator is making changes in the same file and has overwritten the earlier changes. In this case, administrators should try to contact other operators to see if this is the case.
  • Any files that administrators delete on one member will also be deleted on all other members.
  • If administrators rename a file or folder so that it is moved out of the replica tree, FRS treats that as a delete on the other replica members.
  • If two operators create identically named folder at the same time (or even before the first created directory has reached the other computer) on the same server, then morphed folders occur. Morphed folders occur when two or more identically named folders on different servers are added to the replica tree. FRS identifies the conflict during replication, and the receiving member protects the original copy of the folder and renames (morphs) the later inbound copy of the folder. The morphed folder names have a suffix of "_NTFRS_xxxxxxxx," where "xxxxxxxx" represents eight random hexadecimal digits.


Regularly Monitor FRS to Detect Potential Problems or Failures

FRS uses a number of finite resources — bandwidth availability, disk space, CPU — and yet FRS has no way of limiting how much change can be introduced in the file system. For this reason, it is essential that you monitor FRS regularly to ensure that the resources are matched to the replication demands and that the underlying technologies that FRS uses are available.

By using monitoring tools such as Sonar or Ultrasound, you can determine whether FRS is working optimally, in a potential error state, or not replicating. This proactive approach to monitoring FRS can help you identify problems early and decrease the amount of time you spend troubleshooting replication failures.

For more information about choosing an FRS monitoring tool that’s right for your organization, see Tools for Monitoring and Troubleshooting FRS.


Ultrasound Overview

Ultrasound is a monitoring and troubleshooting tool for the File Replication service (FRS). FRS is used to replicate files and folders in the SYSVOL file share on domain controllers and files in Distributed File System (DFS) targets. Ultrasound gives administrators in all sizes of organizations a powerful tool to measure the health of FRS replica sets by providing health ratings and historical information about replica sets. Ultrasound also allows administrators to monitor the progress of replication and detect problems that can cause replication to become backlogged or stopped.

Ultrasound works by installing WMI providers on replica members in an organization. These providers gather FRS status information, which is polled and gathered by the Ultrasound controller. The controller is the service component of Ultrasound that collects data about monitored replica sets, pushes the information into the database, and analyzes the data to look for problems or other issues that require administrator notification or intervention. By using the user interface portion of Ultrasound, known as the console, administrators can configure Ultrasound to alert them via email of serious problems and use an incident log in Ultrasound to keep track of changes or tasks they performed in response to alerts.


Glossary of Ultrasound Terms

The following terms are used to describe the features and components of Ultrasound.

Alert An indication of a significant event. Alerts appear in the Alerts pane. Ultrasound provides a number of preconfigured alerts to indicate issues or problems in typical replica sets. Experienced FRS administrators can also customize alerts to more granularly define the types of issues they want to monitor.

Alerts pane The area in the Ultrasound console where alerts are displayed.

Column set In the Advanced tab of the Ultrasound console, a set of viewable data shown in columns. For example, a column set can show information about a monitored replica set, such as the replica set name, the domain name, the data the replica set was created or changed, file and folder filters, and so on.

Console The user interface component of Ultrasound.

Controller The service component of Ultrasound that collects data about monitored replica sets and pushes the information into the database.

Database The component of Ultrasound that stores data about the monitored replica sets. The database is hosted on a server running MSDE or SQL Server 2000 with SP3.

Health metrics Rules that define whether an object (replica set, replica member, inbound connection, and outbound connection) is either red (critical), yellow (warning), or green (OK).

Health rating A rating that indicates the health of an object (replica set, replica member, inbound connection, and outbound connection). A health rating can be red (critical), yellow (warning), or green (OK).

Incident log In the Ultrasound console, a journal for administrators to take notes about the behavior of FRS or the actions the administrators take in response to alerts or other issues.

Notification The method that Ultrasound uses to notify administrators of an event. A notification can be an email message that is sent to administrators, an entry in the Application event log where the controller is installed, or both.

Propagation file A file used to test that replication is working. A propagation file with a unique name is created on a replica member that has outbound partners, and Ultrasound will verify that the file replicates throughout the topology to all replica members with inbound partners.

Propagation test A test that drops propagation files into the replica tree and monitors the replication of the propagation file to all replica members with inbound connections.

Provider The component of Ultrasound that is installed on replica members to collect FRS data. The provider is based on the WMI framework.

Reference domain controller The domain controller from which the controller gathers FRS data from Active Directory.

Reference replica member The replica member from which Ultrasound gathers the master list of files in the replica set.

Row filter An expression used to filter rows from the view. A row filter is the basis of alerts, notifications, and health ratings.

Schema An organized set of tables used to model the data that Ultrasound collects.

View A relationship between multiple tables. Row filters are defined on top of views.


Ultrasound System Requirements

The following sections describe the system requirements for running Ultrasound.

Hardware Requirements

Video adapter: The Ultrasound console (the user interface component of Ultrasound) is designed for viewing in 1024x768 or higher resolution.

Hard disk space: Hard disk space requirements vary according to the Ultrasound component to be installed:

  • Microsoft Data Engine (MSDE): <100 MB. MSDE is installed separately from Ultrasound.
  • Initial size of Ultrasound database: 5 MB. After FRS monitoring begins, the size of the database depends on the number and size of monitored replica sets. MSDE has 2-GB database size limit.
  • Console: 2.5 MB.
  • Controller: 1.5 MB.
  • Provider: 400 KB.

Operating System and Service Pack Requirements for FRS Replica Members

To reduce the likelihood of FRS errors, it is recommended that all replica members run the current service packs and post-service pack releases for either Windows 2000 Server or Windows Server 2003. The following sections describe the recommended service packs.

Note

  • For current information about FRS, see the DFS and FRS Web site at http://go.microsoft.com/fwlink/?LinkId=18421.

Windows 2000 Server

The minimum supported operating system for FRS servers monitored by Ultrasound is Windows 2000 Server with SP2 and the hotfix described in article 322141, "Ntfrs.exe Does Not Clean Up the Staging Folders on Members with No Outbound Partners in Windows 2000." Later service pack releases, however, offer significant FRS improvements and bug fixes. Therefore, we recommend that all replica set servers run one of the following:

  • Windows 2000 Server with SP3 and the post-Service Pack 3 release of Ntfrs.exe, described in article 815473, "File Replication Service Does Not Log Errors on Sharing Violations."
  • Windows 2000 Server with SP4.

Windows Server 2003

Install the pre-Service Pack 1 release of Ntfrs.exe as described in article 823230. This release contains a number of improvements and is recommended for all servers running FRS.

Ultrasound Installation Requirements

The Ultrasound controller can be installed on Windows 2000 SP3 and above, Windows Server 2003, and Windows XP.

The server running the Ultrasound console must have the following software installed:

  • The .NET Framework version 1.1. This is included in Windows Server 2003, or you can download it from http://go.microsoft.com/fwlink/?LinkId=16722. The .NET Framework 1.1 can be installed on Windows 2000 (with SP2 recommended) and Windows XP Professional.
  • Microsoft Data Access Components (MDAC) 2.6. This is included in Windows Server 2003, or you can install it from http://go.microsoft.com/fwlink/?LinkId=18365. Note that MDAC is automatically upgraded to version 2.6 when you install MSDE.

Ultrasound requires a database server to store FRS data. You must install the database server separately before you install the controller. You can download and install Microsoft SQL Server 2000 Desktop Engine (MSDE), which is available for free on http://go.microsoft.com/fwlink/?LinkId=16720, or you can use SQL Server 2000 with SP3a.

MSDE has a 2-GB limit and is appropriate for most small and medium organizations that have less than 100 replica members and that want to keep two weeks’ worth of data. (The default retention period is five days.) Other organizations should consider using SQL Server 2000 with SP3a depending on how much data they want to keep, the number of replica sets they want to monitor, and the number of files in the replica sets. For more information, see Installing MSDE.

Network Bandwidth Requirements

The controller, database, and console can all be installed on different computers. Therefore, the following bandwidth between computers is recommended:

Network connection between the controller and database: Highest bandwidth available, such as the bandwidth available between servers in the same data center.

Network connection between the console and database: High bandwidth, such as the bandwidth available between servers in the same site.

Network connection between the reference domain controller, reference replica member, and controller: High bandwidth, such as the bandwidth available between servers in the same site.

Network connection between the controller and each provider: The computer running the controller must be able to directly access each server running the provider. Ultrasound data does not flow through the same connections as does FRS data.

Supported FRS Topologies

Ultrasound works with any type of FRS topology. Because FRS is a distributed system that runs on multiple servers, it is important that you design a replication topology that works well in your network infrastructure and that can grow with your organization. It is also important that you correctly configure FRS-related settings, such as staging directory and USN journal sizes, to reflect the amount of data that is replicated in each replica set on a server. Follow the guidelines in FRS Best Practices to ensure smooth FRS operation.


Where to Get Help with Ultrasound

Use the following sources for information about FRS and Ultrasound.

DFS and FRS Web site

For links to information about Ultrasound, including the Ultrasound FAQ, see the DFS and FRS Web site at http://go.microsoft.com/fwlink/?LinkId=18421.

Newsgroup support

Use the microsoft.public.windows.file_system newsgroup to ask questions about Ultrasound.

Other resources

To help you design and deploy a new replica set for DFS link targets, see the chapter "Designing and Deploying File Servers" in the Planning Server Deployments book of the Windows Server 2003 Deployment Kit available at http://go.microsoft.com/fwlink/?LinkId=16513.

If you are designing SYSVOL replication topology in a large organization with more than 100 sites, review the Windows 2000 Branch Office Deployment Guide available at http://go.microsoft.com/fwlink/?LinkId=16718 to decide if a custom, scripted topology is required for your organization. This guide contains techniques and tools for configuring SYSVOL replication and is useful even to administrators who manage simple SYSVOL topologies.


How Ultrasound Works

Ultrasound works by installing WMI providers on replica members in an organization. These providers gather FRS status information, which is polled and gathered by the Ultrasound controller. The controller places this information in a database and analyzes the data to look for problems or other issues that require administrator notification or intervention.

By using the user interface portion of Ultrasound, known as the console, administrators can configure Ultrasound to notify them via email of serious problems. Ultrasound also shows health ratings and historical information about replica sets.

The following sections describe how Ultrasound works.

  • Ultrasound Architecture
  • What Ultrasound Collects
  • Ultrasound Security


Ultrasound Architecture

Ultrasound consists of the following components:

Console

The console is the Ultrasound user interface. The console allows administrators to interact with the system by setting data collection schedules, deploying providers, and more. The console also displays health ratings, alerts, the incident log, and detailed views of the database.

Controller

The controller collects data about multiple replica sets and pushes the information into the database. The controller has the following characteristics:

  • The controller is a service that runs 24 hours a day, seven days a week. A dedicated server is not required--you can run the controller on any server.
  • There is one controller per database, and one controller per server.
  • A controller can monitor multiple replica sets.
  • Communication between controller and database uses Windows authentication without encryption. To enable SSL encryption, see the following articles:
    • 316898, "HOW TO: Enable SSL Encryption for SQL Server 2000 with Microsoft Management Console" at http://support.microsoft.com/?id=316898.
    • 276553, "HOW TO: Enable SSL Encryption for SQL Server 2000 with Certificate Server" at http://support.microsoft.com/?id=276553.

The controller acts as the middle tier between the providers and console and performs several types of operations on behalf of the administrator:

  • Collecting data from providers and Active Directory.
  • Executing tests to verify that FRS is working correctly.
  • Processing data to trigger the appropriate alerts, notifications, and health ratings.

The procedure for collecting and processing data is as follows:

  1. The controller periodically queries the database to determine which tests are scheduled by the administrator.
  2. The controller contacts the respective data source (primarily the providers on replica members, but also Active Directory) and performs the data collection test.
  3. The controller inserts the information returned from the data source into the database.
  4. The controller post-processes and integrates the gathered data into the database, then queries the data to trigger the appropriate alerts, notifications, and health ratings.

For performance reasons, there can be a delay (typically less than five minutes) between the time the data is retrieved and when alerts are triggered.

Database

The database uses a highly schematized format to store FRS data. Ultrasound alerts, notifications, and health metrics are built on top of the schema to provide extensibility and flexibility.

The controller triggers a database purge on a configurable cycle. You can configure how often the purges take place, the time at which they occur, and how many days’ worth of data to maintain. If the controller is not running, the database is not purged. For more information on configuring database purge options, see To configure Ultrasound options.

Provider

Ultrasound uses Windows Management Instrumentation (WMI) as a way to communicate between the controller and the provider. The provider is installed by the Ultrasound console as described in Deploying Providers. When the Ultrasound controller polls the provider, the provider is started by the WMI service, during which the provider runs as Local System account and performs the following operations:

  • Accesses event logs, performance counters, the registry, and the FRS monitoring APIs.
  • Creates or deletes propagation files in the UltrasoundPropagationFiles.servername folder under the replica root.

During this process, it is possible that the provider can be polled by multiple controllers at the same time. However, because only a single provider can exist on a server, the versions of the multiple controllers must be compatible.

For more information about providers, see Deploying Providers.


Data That Ultrasound Collects

The Ultrasound controller and provider collect the following information.

Data collected by the controller

The controller gathers data collected by the provider and from Active Directory. When polling Active Directory, the controller collects topology information about monitored replica sets primarily from the reference domain controller, though the controller also collects data from other domain controllers, including those in other domains for the subscriber object. The data gathered includes the replica members, inbound connections, outbound connections, and the replication schedules for each connection. The controller also tracks changes to replica members, such as adding or removing replica members, and changes to inbound and outbound connections. Because the controller tracks these changes, Ultrasound can show changes that occurred within specified time intervals.

The controller is also responsible for monitoring information about propagation files. If you schedule a propagation test, the controller records where the propagation file is dropped, the size of the propagation file, and the time of the drop. The controller can detect whether the propagation file arrived on a particular replica member within the time-out period and the time of arrival. (You configure the time-out period when you schedule the propagation test.) For more information about propagation files, see Verifying Replication by Using Propagation files.

Data collected by the provider

The provider collects information about the server on which it is installed. When the FRS service is running on a server, the provider collects the following information:

  • Performance counters.
  • FRS state (such as Active, Starting, Stopped, or JRNL_WRAP_ERROR, among others).
  • FRS service state. (The FRS service does not need to be running for this information to be collected.)
  • SYSVOL status (either not SYSVOL, not shared, not a junction point, or OK).
  • Inbound connections and any sharing violations on those inbound connections.
  • Outbound connections and any backlogged files on those outbound connections.
  • Sharing violations in the staging directory.
  • Events from the File Replication Service event log. (The FRS service does not need to be running for this information to be collected.)
  • FRS-related registry values. (The FRS service does not need to be running for this information to be collected.)
  • USN journal status (the size of the USN journal and how much space is left).

In addition to the information listed above, the provider installed on the server that is designated as the reference member for a monitored replica set gathers information about files in the replica set, including files that are added, deleted, and renamed. Based on this master list of files, Ultrasound can compute the file name from the file GUID, the size of the files in the replica set, how often files are changed, and which replica member changes each file.

If the FRS service is not running, or if FRS is in a severe error state, the provider gathers information about that replica member from other data sources, such as the event log and registry entries. Ultrasound combines this information with information gathered while the service was running to fill in the gaps and by default triggers an alert to notify administrators that the FRS service is an unhealthy state.


Ultrasound Security

The following sections describe Ultrasound security.

Controller security

In Windows Server 2003 and Windows XP, the controller runs as a service under the Network Service account. In Windows 2000 Server, the controller runs as a service under the Local System account. The computer account where the controller is installed must be able to authenticate with any of the provider computers; that is, the controller must either be a member of the same domain as the provider computers, or be a member of a fully trusted domain. Communication with the provider and the database requires authentication of the computer account where the controller is installed. ACLs that allow the controller account access are configured during Ultrasound setup and provider deployment.

For security purposes, we recommend against installing the controller on critical servers, such as domain controllers. See the next section on database security for more information.

Database security

Although the controller is given access the database during Ultrasound setup, the database administrator must grant individual users or groups access to the database by running the USDBAccess.vbs script. This script, available in the Program Files\FRS Monitoring directory after you perform a complete installation or a console-only installation, gives read-only access or read-write access to the database. Those users can then install the console on their local computers and access the database on a remote server. For information about running this script, see Granting Database Access to Multiple Administrators.

Important

  • For the highest database security, regularly check the Microsoft Web site to obtain and install SQL Server service packs or other security updates when they become available.

If you install the controller and database on the same server running Windows 2000, be aware that whoever has read-write access to the database is implicitly given administrative access to the server on which the controller is installed. This access occurs because any user with read-write access to the database can modify a SQL command that is executed by the controller, a process known an SQL injection. A malicious user could append harmful code to SQL commands, and because the controller is running as the Local System account on Windows 2000, the user could gain administrative access to the server.

Note

  • More precisely, the command is executed by SQL using the credentials of the controller. When the database and controller are running on the same server running Windows 2000, the credentials of the controller are for Local System (administrator) on the database server. If the controller is not installed on a server running Windows 2000, or the controller and database are on different servers, SQL injection can still happen, but the credentials used are much more restricted.

The following table describes the credentials used by the controller in various installation scenarios.

Credentials Used in Different Controller and Database Installation Scenarios

Operating system where controller is installed | Controller and database on the same server? | Credentials used
Windows 2000 | Yes | Local System
Windows 2000 | No | Controller computer account
Windows Server 2003 | Yes | Network Service
Windows Server 2003 | No | Controller computer account

To reduce the risk of SQL injection in situations where the controller and database are installed on the same server running Windows 2000 Server, limit read-write database access to trusted users. To reduce security risks on critical servers, avoid installing the controller or database on a critical server, such as a domain controller.

Provider security

Because the provider can run on domain controllers, provider security is of utmost importance. The communication channel between the provider and controller is authenticated and encrypted to make sure both parties are members of a trusted domain. In addition, the WMI namespace for the FRS provider has permissions that limit access to only the controller(s). When a provider is deployed to a server, the setup package configures the FRS WMI namespace to allow access to the controller’s machine account. Because a provider can support multiple controllers, if the WMI namespace already exists, the setup package only adds the new controller’s computer account and does not configure the other settings.

Note

  • The WMI service starts the provider under the Local System account. This is required so that the provider has access to the file system to create and clean up propagation files.

WMI uses DCOM for remote communication. For DCOM to go through a firewall, you must open a set of ports. For more information, see http://go.microsoft.com/fwlink/?LinkId=16723.


Deploying Ultrasound

Ultrasound deployment consists of a number of steps, which are described in the following topics.

  • Installing Ultrasound Components
  • Choosing Replica Sets to Monitor
  • Deploying Providers
  • Configuring Default Notifications


Installing Ultrasound Components

Ultrasound installation consists of installing the console and controller, either together as a complete installation or on separate servers as a custom installation. Before you can install and run Ultrasound, however, you must first install MSDE or SQL Server 2000 with SP3a. This installation is separate from the Ultrasound installation, allowing you to manage the database and apply SQL security updates and other service packs as they become available. The provider installation is also a separate procedure that is performed after the console, controller, and database are installed. For more information about installing providers, see Deploying Providers.

The following sections describe the issues to consider before installing Ultrasound as well as the procedures for installing the Ultrasound components.

  • Deciding Where to Install Ultrasound Components
  • Determining How Many Ultrasound Environments to Install
  • Installing MSDE
  • Installing the Console and Controller
  • Granting Database Access to Multiple Administrators


Deciding Where to Install Ultrasound Components

Small and medium organizations with centralized management might choose to install the console, controller, and database on the same server. If your organization is large or has administrators in different sites, you can install the console on multiple servers or client computers.

If you plan to install the Ultrasound components on different computers, follow these guidelines:

  • Install the controller on a central server that has a high-bandwidth connection to the database, reference replica member, and frequently polled providers.
  • Make sure that the .NET Framework 1.1 and Microsoft Data Access Components (MDAC) 2.6 are installed on every computer that will run the console. For more information about these requirements, see Ultrasound System Requirements
  • Store the database on a central server that has high-bandwidth network connections to the controller and as many consoles as possible.


Determining How Many Ultrasound Environments to Install

An Ultrasound environment is a controller and database that are configured to communicate only with each other. The paired controller and database can monitor a single replica set or multiple replica sets. Use the following guidelines to make your choice:

Use a separate Ultrasound environment for each replica set if you have a combination of these factors:

  • Your organization has numerous replica sets that are geographically dispersed.
  • The replica set you want to monitor consists of a group of servers that are not part of other replica sets.
  • You want to keep the database size smaller.

Use a single Ultrasound environment to monitor multiple replica sets if you have a combination of these factors:

  • Your organization has a small number of replica sets.
  • You want to monitor multiple replica sets that exist on the same group of servers.

If you decide to use multiple Ultrasound environments, the controller for each environment must be installed on a separate computer and use a different database name, though you can use the same database server instance.


Installing MSDE

If your organization has SQL Server 2000 with SP3a and you do not want to use MSDE, you can proceed to the next section. If you want to install MSDE, do the following:

To install MSDE

  1. Go to http://go.microsoft.com/fwlink/?LinkId=16720. Select the appropriate language and download the SQL2KDeskSP3.exe file or the corresponding localized file. (Ultrasound uses the "US English" language envirnoment when connecting to SQL. This environment is installed by default in all localized versions of MSDE and SQL.)
  2. Double-click the file and follow the directions to unpack the installation files.
  3. At the command prompt, navigate to the folder that contains the unpacked files. In that folder, open the MSDE folder.
  4. At the command prompt, run the Setup program by typing the following:
    setup INSTANCENAME=instancename SAPWD=strongpassword DISABLENETWORKPROTOCOLS=0

    For instancename, substitute the desired instance name, such as "frsmon." To ensure database security, choose a strong password to replace strongpassword.

    If you are running the console, controller, and database on the same server, you can omit DISABLENETWORKPROTOCOLS=0. For more information about disabling network protocols for MSDE, see article 814130, "INF: How to Secure Network Connectivity for SQL Server 2000 Local Databases" at http://support.microsoft.com/?id=814130.

  5. At the command prompt, start the service by typing
    net start mssql$instancename


Installing the Console and Controller

If this is the initial installation of Ultrasound and you plan to install the controller and console on the same server, choose a complete installation. Otherwise, choose a custom installation to install the console on a separate computer or to install the controller on a server where the database is already installed.

Performing a Complete Installation

A complete installation installs the console and controller on the same server. MSDE or SQL Server 2000 SP3a must already be installed on a server before you begin installation.

To perform a complete installation

  1. Run Setup_Ultrasound.exe to begin installation.
  2. Follow the directions in the installation wizard and choose Complete for the setup type.
  3. When prompted to enter a database server, type server\DBInstance that corresponds to the server name where you installed the database and the instance name of the database. If you did not specify a database instance when you installed MSDE or SQL Server 2000, just type the server name.

    Do not type "localhost" to specify the database server name; use the actual name of the server instead. For example, if you are installing Ultrasound on a server called MyServer and this server has an MSDE database instance called "frsmon" already installed, specify "MyServer\frsmon" for the database server.

  4. For the database name, type a name to use as the database name or use the default database name "Ultrasound."
  5. Follow the instructions to complete the installation.

Performing a Custom Installation

A custom installation allows you to install the controller and console separately and is typically used to install the console on administrators’ client computers. You can also perform a custom installation to install the controller on a server that has access to a database on another server or the same server.

To perform a custom installation

  1. Run Setup_Ultrasound.exe to begin installation.
  2. Follow the directions in the installation wizard and choose Custom for the setup type.
  3. For the component that you do not want to install, click the drop-down arrow next to the component name and click This feature will not be available.
  4. Do one of the following:
    • If you are installing the console, follow the instructions to complete the wizard. The first time you start the Ultrasound console, you are prompted to enter a database server and database name.
    • If you are installing the controller, you are prompted to enter a database server and database name. Provide this information, and then follow the instructions to complete the wizard. If the database server or database instance name does not exist, you are notified that Ultrasound cannot connect to the server, and installation fails. Verify that the database service is running and accessible and try running setup again.


Granting Database Access to Multiple Administrators

Members of the Administrators group on the server where SQL or MSDE is installed implicitly have read-write access to the database. If other users need to access the database, the database administrator can run the script USDBAccess.vbs, available in the Program Files\FRS Monitoring directory after you perform a complete installation or a console-only installation, to give read-only access or read-write access to individual users or to groups of users. Those users can then install the console on their local computers and access the database on a remote server. For more information about tasks that users can and cannot perform when they have read-only access, see Monitoring Ultrasound in Read-Only Mode.

Important

  • If you install the controller and database on a server running Windows 2000, be aware that whoever has read-write access to the database is implicitly given administrative access to the server on which the controller is installed. For more information, see Ultrasound Security.

The following sections describe the USDBAccess.vbs syntax and parameters.

Syntax

cscript.exe USDBAccess.vbs dbserver dbname account [add | remove] [ro | rw]

Parameters

dbserver

Specifies the name of the database server instance. For example, in "sqlsrv1\frsmon," "frsmon" is the instance name hosted on the "sqlsrv1" server. If you did not specify the instance name when you installed MSDE, use only the server name.

dbname

Specifies the name of the database, usually called Ultrasound.

account

Specifies the name of the user or group account that needs access to the database in the form "domain\username."

[add | remove]

Specifies whether to add or remove the account name from the list of accounts that can access the database. If you do not specify this parameter, add is assumed.

[ro | rw]

Specifies whether to give the account read-only (ro) or read-write (rw) access to the database. If you do not specify this parameter, ro is assumed.


Choosing Replica Sets to Monitor

When you run Ultrasound for the first time after installing it, you are prompted to choose which replica sets you want to monitor. You can add other replica sets to monitor at any time by following the procedure To choose replica sets to monitor.


To choose replica sets to monitor

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, click Add/Remove Replica Sets.
  3. Under Domain, choose the domain that contains the replica sets you want to monitor.

    The list of domains includes all domains trusted by the forest where the console is installed. You can also type in a domain name that is externally trusted.

  4. Under FRS replica sets, click the replica set, and then click Add.
  5. After you click OK, you are prompted to choose the servers on which to install providers. Select the option you prefer.
  6. If you want to collect data from the providers immediately after they are installed, click Collect initial data from WMI providers after installation. For more information about the initial provider deployment, see Monitoring FRS Using Ultrasound.


Deploying Providers

After you choose the replica sets you want to monitor, you are prompted to install providers on the servers that are part of those replica sets. It is recommended that you install providers on every server to be monitored, though you might choose not to install providers on servers that have unreliable or low-bandwidth connections.

Note

  • When you are prompted to install providers, you also have the option of having Ultrasound perform an initial data collection after providers are deployed. If you do not select this option, you must wait for the data polling schedule to open for each provider, which can be up to two hours, or you can manually poll providers by using the procedure To manually poll providers.

You typically deploy providers by using the Ultrasound console. (This procedure is described in To install providers by using the console.) The provider deployment is done in the background after you choose the servers to monitor. Because the console deploys providers on remote servers on behalf of the user running the console, you must run the console when logged on using an account that is a member of the local Administrators group of the remote replica member or the Domain Admins group before deploying providers. The provider deployment is a one-time operation; after that, users who are given access to the database can use the console to monitor FRS, even if they are not administrators on the replica members. For more information about giving multiple users access to the database, see Granting Database Access to Multiple Administrators.

After you select servers on which to install providers, the provider deployment status changes from Uninstalled to Installed (Queued). A background operation in the console is started and selects servers that are in the Installed (Queued) state. The status is then changed to Installed (Pending) and the operation to install the provider begins. Multiple installations can occur at any one time.

After the provider setup package is copied to the %windir% folder on the remote server, the console triggers a remote installation, after which the server installs the provider setup package, registers the WMI namespace, and sets the appropriate permissions to allow the controller to access the provider. The console then queries the provider to validate the installation, at which time it will update the database to indicate a successful installation, and the provider deployment status changes to Installed.

Deploying providers can be a time-consuming process, because Ultrasound must contact each server. Expect to wait one hour for every 100 servers, though it’s possible that provider deployment can take as little as five minutes for 100 servers. Keep the console running until the deployment status is Installed for all selected servers. For more information, see Starting and Stopping Provider Deployment.

If Ultrasound is unable to install a provider, it will wait 30 minutes before trying again. This process will continue for a maximum of 60 attempts, after which the provider deployment status changes to Installed (Failed). A failure can occur for the following reasons:

  • The user logged onto the console does not have permissions to do the required operations on the remote server.
  • The console cannot contact the server because of network problems.
  • Firewall issues prevent the provider setup package from being copied to remote servers.
  • Problems on the remote server cause the deployment to fail, such as a lack of disk space. (The provider requires approximately 400 KB.)

If these issues occur, you must resolve them before the providers can be installed. See Troubleshooting Ultrasound Deployment Problems for more information. After you resolve the issue, click Install Provider again to restart the process. You also have the option of installing the providers on the local servers. For more information, see To locally install providers by using the command line.

The process for uninstalling providers is similar to the installation process. When you click Uninstall Provider, the provider deployment status changes from Installed to Uninstalled (Queued). The console then begins the uninstallation process and changes the status to Uninstalled (Pending). After the uninstallation is complete, the status changes to Uninstalled. If the uninstallation is not successful, Ultrasound waits 30 minutes before trying again with a maximum of 60 attempts.


To install providers by using the console

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, choose the replica set for which you want to install providers. (You can also press the CTRL key and click to select multiple replica sets.)
  3. Under Ultrasound WMI Provider Deployment Status, click the server on which you want to install the provider, and then click Install Provider. (You can also press the CTRL key and click to select multiple servers.)


To locally install providers by using the command line

  1. Copy the provider file USProvider.msi from the console server’s Program Files\FRS Monitoring folder to the remote replica members on which you want to install the provider.
  2. On the remote servers, at the command prompt, type the following:
    msiexec /i USProvider.msi CONTROLLERACCOUNT=ControllerComputerAccount

    3. In the above syntax, for ControllerComputerAccount substitute the computer account name of the server where the controller is installed. The expected format for the computer account name is ComputerName$@FullyQualifiedDomainName. For example, if the controller is running on a computer whose fully qualified DNS name is server1.microsoft.com, you must specify the controller account name as server1$@microsoft.com. Note that case is important for the CONTROLLERACCOUNT parameter.

  3. On the console server, run Ultrasound.
  4. On the Tools menu, click Configure Ultrasound.
  5. Under Monitored FRS replica sets, click the replica set whose members you installed providers on by using the command line. (You can also press the CTRL key and click to select multiple replica sets and replica members.)
  6. Under Ultrasound WMI Provider Deployment Status, right-click the replica member on which you installed the provider, point to Override Deployment State, and then click Installed.


To update remote providers by using the console

If you already have providers installed, you must use the Update Provider button to force a new installation, such as when you have a new version of the provider.

  1. Make sure that the newest version of the provider file, USProvider.msi, is located in the Program Files\FRS Monitoring folder.
  2. On the Tools menu, click Configure Ultrasound.
  3. Under Monitored FRS replica sets, choose the replica set for which you want to upgrade providers. (You can also press the CTRL key and click to select multiple replica sets.)
  4. Under Ultrasound WMI Provider Deployment Status, click the replica member on which you want to update the provider, and then click Update Provider. (You can also press the CTRL key and click to select multiple servers.)


To locally update providers by using the command line

  1. Copy the provider file USProvider.msi from the Program Files\FRS Monitoring folder on the console server to the remote replica members on which you want to update the provider.
  2. On the remote servers, at the command prompt, type the following:
    msiexec /fvamus USProvider.msi CONTROLLERACCOUNT=ControllerComputerAccount

    In the above syntax, for ControllerComputerAccount substitute the computer account name of the server where the controller is installed. The expected format for the computer account name is ComputerName$@FullyQualifiedDomainName. For example, if the controller is running on a computer whose fully qualified DNS name is server1.microsoft.com, you must specify the controller account name as server1$@microsoft.com. Note that case is important for the CONTROLLERACCOUNT parameter.


Starting and Stopping Provider Deployment

Providers are deployed only when the console is running. If you begin provider deployment and then attempt to exit the console, a warning message indicates that provider deployment will cease after you close the console. After you restart the console, you are prompted to start provider deployments by clicking the Start Provider Deployment button in the Configure Ultrasound dialog box. Provider deployments are not restarted automatically because Ultrasound cannot determine whether you are logged into the console using the administrative credentials required to deploy providers on remote servers. Before starting provider deployment, make sure that you are logged in using an appropriate account. See Deploying Providers for information about the credentials required for deploying providers.

If for some reason you want to stop provider deployments, see To start or stop provider deployment.


To start or stop provider deployment

  1. On the Tools menu, click Configure Ultrasound.
  2. Do one of the following:
    • To start provider deployment, click Start Provider Deployment.
    • To stop provider deployment, click Stop Provider Deployment.


Configuring Default Notifications

A notification is an e-mail message that is sent or event log entry that is created as a result of an alert. An alert occurs when the controller analyzes FRS data and detects an error or potential problem. Alerts are displayed in the Alerts pane in the Ultrasound console and are rated by severity, either informational, warning, or error. Each alert can be associated with the following actions:

  • Send no notifications.
  • Send a notification to a set of default destinations, which can be an event log entry on the server where the controller is installed and an e-mail message sent to one or more e-mail addresses. This is known as a default notification and is useful if you want to send e-mail notifications for a variety of alerts to the same group of people.
  • Send a notification to a set of custom destinations, which can be an event log entry on the server where the controller is installed and an e-mail sent to one or more e-mail addresses. This is known as a custom notification and is useful if you want to notify only certain people of a specific alert.

As part of your initial Ultrasound deployment, configure default notifications by specifying whether to notify administrators via email by using the simple mail transfer protocol (SMTP) host or by an event in the Application log on the server where the controller is installed.

Important

  • Before Ultrasound can send e-mail messages, you must specify the SMTP server. See To configure Ultrasound options for more information.

For more information about enabling or modifying alerts and notifications, see Customizing Row Filters.


To configure default notifications

  1. On the Tools menu, click Default Notifications.
  2. Click Add, and do any of the following:
    • To configure emails, click Send e-mail and type an e-mail address.
    • To log an event, click Log event. Events will be logged in the Application event log on the server where the controller is installed.

E-mail messages cannot be sent until you specify the SMTP server, and the SMTP server must support anonymous access. To specify the SMTP server, see To configure Ultrasound options.


Troubleshooting Ultrasound Deployment Problems

The following sections describe the problems you might encounter when installing Ultrasound and suggests methods of resolving those problems.

  • Troubleshooting Firewall Problems
  • Troubleshooting Permissions Problems
  • Troubleshooting Disk Space Problems

If problems persist, review the event logs on the remote servers to identify possible problems and see Where to Get Help with Ultrasound.


Troubleshooting Firewall Problems

Ultrasound deployment uses two protocols: SMB and DCOM. If a firewall blocks the SMB port, you cannot deploy providers, because Ultrasound uses SMB to copy the provider setup package to the remote servers. WMI is then used to invoke the remote process that executes the setup package.

If you have firewalls in your environment, and the SMB port is blocked, you should locally copy and install the provider package on the remote server by using the procedure To locally install providers by using the command line.

If your firewall blocks DCOM, this will prevent Ultrasound from collecting data through WMI. For DCOM to go through a firewall, you must open a set of ports. For more information, see http://go.microsoft.com/fwlink/?LinkId=16723.

Note

  • To check the status of provider deployment, click the Provider deployment status view on the left side of the Advanced tab. If provider deployment fails, review the description field for information about why the deployment failed.


Troubleshooting Permissions Problems

As described in Deploying Providers, provider deployment is performed on behalf of the user who uses the console to begin provider deployment. If you attempt to use the console to install a provider on a server where you are not a member of the local Administrators group or the Domain Admins group, provider deployment fails. To resolve this, either have your user account added to the required groups on the replica members or log into the console server using an account that already has membership in required groups. You can also have another administrator for the replica members install the provider locally by using the command prompt. See To locally install providers by using the command line for the procedure.


Troubleshooting Disk Space Problems

Ultrasound setup can fail if there is not enough disk space for the components. To resolve this, free up disk space and try installing Ultrasound again.


Uninstalling Ultrasound

If you want to uninstall Ultrasound, you need to uninstall each of its components separately. For procedures to uninstall each component, see the following sections:

  • Uninstalling Providers
  • Uninstalling the Controller
  • Uninstalling the Database
  • Uninstalling the Console


Uninstalling Providers

The provider is a component that is only invoked by request, so if it’s left on a server, it will not cause any harm. There are three ways to uninstall providers:

  • Remotely uninstall providers by using the console.
  • Uninstall providers locally by using the command line.
  • Uninstall providers locally by using Add or Remove Programs in Control Panel.

Before uninstalling providers, note that multiple controllers can share a single provider. Therefore, before uninstalling a provider, make sure that no controllers are using the provider.

To remotely uninstall providers by using the console

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, choose the replica set for which you want to uninstall providers. (You can also press the CTRL key and click to select multiple replica sets.)
  3. Under Ultrasound WMI Provider Deployment Status, click the replica member on which you want to uninstall the provider, and then click Uninstall Provider. (You can also press the CTRL key and click to select multiple servers.)

To uninstall providers locally by using the command line

  1. Copy the provider file USProvider.msi from the Program Files\FRS Monitoring folder on the console server to the servers on which you want to uninstall the provider.
  2. On the servers, at the command prompt, type the following:
    msiexec /x Installerpackage

    In the above syntax, for Installerpackage substitute the path and file name of the file USProvider.msi (for example, F:\Data\USProvider.msi).

    If the installer package is not available, use the following command instead:

    msiexec /x {632B88AD-6ABC-489D-8B8D-CB84C4987251}
  3. On the console server, run Ultrasound.
  4. On the Tools menu, click Configure Ultrasound.
  5. Under Monitored FRS replica sets, click the replica set whose members you uninstalled providers by using the command line.
  6. Under Ultrasound WMI Provider Deployment Status, right-click the replica member on which you installed the provider, point to Override Deployment State, and then click Uninstalled.


Uninstalling the Controller

To uninstall the controller, use Add or Remove Programs in Control Panel to remove the entry for Ultrasound. When the controller is uninstalled, the database is dropped and deleted. This means all collected data and Ultrasound settings are removed. The console is also removed if it was included in the installation.

Important

  • Dropping a database deletes a database from a device and destroys all data it contains. It is an irreversible process.

If you want to uninstall the controller but keep the database, which can be used if you want to move the controller from one computer to another, please refer to the Ultrasound FAQ described in Where to Get Help with Ultrasound.


Uninstalling the Database

If you want to uninstall the Ultrasound database, not the MSDE service, you must uninstall the controller or refer to the Ultrasound FAQ about how to delete the database. To uninstall the MSDE service, use Add or Remove Programs in Control Panel to remove the entry for Microsoft SQL Server Desktop Engine.

For more information about the Ultrasound FAQ, see Where to Get Help with Ultrasound.


Uninstalling the Console

To uninstall the console, use Add or Remove Programs in Control Panel to remove the entry for Ultrasound. When the console is uninstalled, the controller is also uninstalled. Because uninstalling the controller causes the database to be dropped and deleted, all collected data and Ultrasound settings will be lost.

Important

  • Dropping a database deletes a database from a device and destroys all data it contains. It is an irreversible process.


Monitoring FRS Using Ultrasound

Data begins appearing in the Ultrasound console after the first polling of the providers. The first polling can take place after one of the following events:

  • When you select replica sets to monitor, you are prompted to install providers and choose whether to have Ultrasound perform an initial data collection after providers are deployed. If you choose this option, data begins appearing in the console soon after the providers are deployed.
  • If you do not select this option, you must wait for the provider polling schedule to open for each provider, which can be up to two hours. Until this polling occurs, you will see limited information about monitored replica sets. See Changing Polling Schedules for more information.
  • You manually poll the providers. See To manually poll providers for more information.

After polling begins, use the Ultrasound console to monitor replica sets in your organization. The console is divided into three parts: the Workspace pane, the Alerts pane, and the incident log.

  • The Workspace pane consists of five tabs that display the data collected from monitoring, health ratings, alert history, and so on. For information about these tabs, see Viewing Tabs in the Workspace Pane.
  • The Alerts pane shows all active alerts and can be filtered to show a subset of the alerts, such as those that are assigned to the user viewing the console. For information about using the alerts pane, see Responding to Alerts.
  • The incident log shows entries added by administrators to document the behavior of FRS or the actions taken in response to alerts or other issues. For information about using the incident log, see Making Temporary Notes in the Incident Log.


Viewing Tabs in the Workspace Pane

The Workspace pane contains the following tabs:

  • Health Tab
  • Details Tab
  • Alert History Tab
  • Summary Tab
  • Advanced Tab


Health Tab

The Health tab provides an at-a-glance view of the health of the objects in each monitored replica set. Health is indicated by icons that are red, yellow, or green.

  • Red indicates a critical condition that requires immediate attention.
  • Yellow indicates a warning condition that might not result in replication problems but should still be investigated or monitored.
  • Green indicates that no problems with replication were detected.

For information about events or conditions that cause replica set health to be red (critical) or yellow (warning), see Understanding Health Ratings.

In the Health tab, you can:

  • View the health of all replica sets, or use the Health filter drop-down list to view a subset of replica sets based on health.
  • View how long ago the last data collection or health change occurred by selecting Last update or Current health in the Show drop-down list. (If no data collection or health change has ever occurred, they are shown as "Not Set.")
  • View the current health of all replica sets, or use the Date filter drop-down list to view past health, such as 24 hours ago, last week, or a custom set of dates.


Details Tab

The Details tab provides in-depth information about replica set topologies, the files in the replica tree, and propagation tests. After selecting a replica member on the left side of the screen, click the Show drop-down list and select one of the following options:

  • Click Topology to view the replica members in the replica set and the inbound connections and outbound connections for the selected server.
  • Click Topology Changes to view the changes to the inbound and outbound connections.
  • Click Files to see a list of all files in the replica tree on the reference replica member. You can also view the following information:
    • The number of files in the replica tree.
    • The total size of the replica tree.
    • The full path and file size of any file in the replica tree.
    • Where the most recent changes to the file originated.
    • Any pending changes to the selected file.
    • The total number of backlogged files and sharing violations in the selected replica set.
  • Click Propagation Tests to see a list of propagation tests and statistics about those tests, such as the time required for the propagation file to reach all monitored members of the replica set. For more information about propagation tests, see Verifying Replication by Using Propagation Tests.


Alert History Tab

The Alert History tab displays history for all alerts and the incident log. In the Alert History tab, you can:

  • View a list of all alerts and incident log entries, or use the Date filter drop-down list to view a subset of alerts and entries, such as those that occurred in the last 24 or 48 hours or during a custom set of dates.
  • View alerts based on their severity, such as informational, warning, or error.
  • After selecting an individual alert, change the status, severity, or owner of that alert. (To do this, right-click or double-click the alert and choose the desired setting.)


Summary Tab

The Summary tab displays an overview of all aspects of monitored replica sets, making it easy for you to see whether any parts of the replica set, such as replica members or inbound or outbound connections, require attention. The Summary tab shows the following information:

  • All monitored replica sets.
  • All unhealthy or recently updated connections.
  • All unhealthy or recently updated members.
  • Recently completed propagation tests.
  • Recent active notifications.

To view additional details about an object shown in this tab, such as a replica member, right-click the object and then click Go To Details or Go To Health.


Advanced Tab

The Advanced tab is useful for administrators who want to create custom views of database information and customize row filters and health metrics. For more information about using the Advanced tab, see How Views and Row Filters Appear in the Advanced Tab.


Understanding Health Ratings

Health ratings describe the overall health of replica sets, replica members, and inbound and outbound connections. Health ratings are shown on the Health tab and Summary tab and use different icons in the following colors:

  • Red indicates an error condition that requires immediate attention.
  • Yellow indicates a warning condition that might not result in replication problems but should still be investigated or monitored.
  • Green indicates replication is working normally.

Ultrasound provides default health ratings based on metrics that typically indicate warning or error conditions in most replica sets. For example, if Ultrasound detects that the system clock is moved ahead on one replica member, that member can no longer replicate files until the clock is corrected. In this situation, an alert is triggered and the health of the individual replica member turns to red to indicate a critical error that has caused replication to stop. In other situations where replication is still working but there is a potential problem that needs to be monitored, such as a backlog of files on a particular connection, an alert is triggered and that connection might turn yellow to indicate a warning condition.

The default health metrics will likely work well in small or medium organizations and do not need to be modified. However, in large organizations, especially in organizations where administrators are experts in managing FRS in their environment, administrators might want to customize health metrics to fine-tune when health ratings change from one status to another, specify which conditions trigger alerts and notifications, and so on. For more information, see Customizing Row Filters and Customizing Health Metrics.


Verifying Replication by Using Propagation Tests

Propagation files are files of varying sizes that are dropped into a replica tree to verify that replication is working. Because Ultrasound also tracks the amount of time required for the propagation file to replicate to all replica members, propagation files are also useful for identifying inefficient topologies or other replication delays.

You set up a propagation test to schedule propagation files to be dropped into the replica tree. You have a number of options when setting up propagation tests, including the following:

  • Scheduling an immediate one-time test or a recurring test at the interval and time you choose.
  • Choosing the size of the propagation file, ranging from 0 bytes to 1 MB.
  • Specifying when the propagation test times out if replication does not complete, ranging from four hours to eight weeks.

Propagation files are located in a hidden directory under the replica root. The directory name is UltrasoundPropagationFiles.servername, where servername is the name of the server where the propagation file is dropped. The propagation file is named UltrasoundPropagationFile_GUID, where GUID is a unique identifier.

Ultrasound only tracks the replication of propagation files on servers that have providers installed on them. When a propagation test starts, a snapshot list of monitored replica members is created. This list defines the servers on which propagation files are tracked.


To schedule a propagation test

  1. On the Tools menu, click Propagation Tests.
  2. Next to Scheduled tests, click Add.
  3. Choose the options for this test, such as the replica set, the server on which to drop the propagation file, and the size of the propagation file. (Only servers on which providers have been successfully deployed are included in the Server drop-down list.)
  4. Choose the frequency at which the propagation test will run, either weekly at the specified schedule or immediately. If you want to use a recurring test, click Schedule to specify the time and day for the test to occur. Note that times shown in the Schedule Propagation Test dialog box reflect the local time of the server where the console is installed.
  5. Choose the time out interval to specify when the propagation test is stopped if the file is not replicated to all monitored replica members.


To update a propagation test

  1. On the Tools menu, click Propagation Tests.
  2. Under Scheduled tests, click the test that you want to update, and then click Schedule.
  3. Do one or more of the following:
    • To change the schedule, click Schedule and choose the desired schedule. Note that times shown in the Schedule Propagation Test dialog box reflect the local time of the server where the console is installed.
    • To change the time out, click the Time out drop-down menu and select a new time out.


To delete a propagation test

  1. On the Tools menu, click Propagation Tests.
  2. Under Scheduled tests, click the test that you want to delete, and then click Remove.


To view statistics about completed propagation tests

  1. On the Details tab, in the Show drop-down list, click Propagation Tests.
  2. On the left side of the screen, click the replica set whose propagation tests you want to review.
  3. Under Propagation Tests, click the test for which you want to review the statistics.


Responding to Alerts

Alerts appear in the Alerts pane in response to a condition or some other event that requires the attention of administrators. Each alert is ranked by severity, either informational, warning, or error. The Alerts pane always shows the alerts sorted by date and time with the most recent alerts at the top. The alerts that appear in a day are separated by a thick dividing line from alerts that occur on other days.

There are a number of steps you can take when responding to alerts.

  • View a description of the event or condition that triggered the alert.
  • View the number of times the event or condition occurred in a predefined interval. (Alerts that occur within this interval are grouped into a single alert. For more information, see How Alert Grouping Works.)
  • Assign an owner to the alert (by default, the alert is marked Unassigned).
  • Change the severity of the alert if the condition or event in your organization is less or more important than the default severity.
  • Diagnose and repair the condition or event that caused the alert. Detailed steps for troubleshooting the alert are displayed when you click the More button in the Alert Details dialog box.
  • After the condition or event is repaired, change the alert status from Active to Resolved.
  • Edit the row filter that triggered the alert to change how future alerts are displayed. For more information, see Customizing Row Filters.
  • Edit the comments field to annotate the row filter.

Note

  • Alerts that have their status set to Resolved are not displayed in the Alerts pane. You can view all alerts, both Active and Resolved, on the Alert History tab.

Within a few days of an initial Ultrasound deployment, you might see a flood of alerts for problems that you did not know existed. In many cases the replica sets, members, and connections might have red (critical condition) or yellow (warning condition) health ratings. We recommend the following strategy for dealing with an initial flood of alerts and health issues:

  1. Review alerts regarding provider deployments to ensure that all providers were deployed successfully. Any unsuccessful deployments could cause an inaccurate picture of the overall health of the replica set. Investigate and repair provider deployment failures using the advice given in the alert and corresponding Help topic. After that, you should receive an alert that indicates that provider deployment was successful.
  2. Use the Summary tab to investigate any objects that are red (critical condition) or yellow (warning condition) and repair those issues. In many cases a corresponding alert will describe the problem. See Understanding Health Ratings for more information.
  3. Resolve all alerts by using the procedure To respond to an alert, and then wait for new alerts to occur. If these alerts are informational, address them when you have time. Many informational alerts can be safely resolved without following the advice in the alert, though in some cases following the advice in an informational alert could result in better performance and reliability of your replica sets.
  4. If you continue receiving alerts or health ratings that are not applicable to your organization, you can customize or disable them by using the procedure To respond to an alert.


To respond to an alert

  1. In the Alerts pane, double-click the alert to which you want to respond.
  2. Do any of the following:
    • Click the Status drop-down list to change the status from Active to Resolved. (To change the status of multiple alerts at the same time, go to the Alert History tab, select the alerts you want to resolve, right-click them, point to Change Status To, and then click Resolved.)
    • Click the Severity drop-down list to change the severity to Error, Informational, or Warning.
    • In Owner, type the name of the person responsible for investigating or resolving the alert.
    • Click the More button to view more information about the alert and suggested methods of resolving the error or condition that triggered the alert. In addition to using the Help topic that appears, use the troubleshooting information provided in the Ultrasound Help file for further investigation. If you cannot solve a problem, contact Microsoft Product Support Services and provide them with the details of the problem.
    • If you no longer want the alert to appear, right-click the alert, and then click Row Filter Editor. On the Alert tab, click to clear the Enable alert check box.
    • If you no longer want the health to change as a result of the condition that caused the alert, right-click the alert, and then click Row Filter Editor. On the Health Metrics tab, click to clear the Enable health metrics check box.
  3. Advanced users can get additional information about alerts by doing the following:
    • On the Advanced tab of the alert, click the Edit Filter button to view or edit the row filter that caused the alert. You can also update the Comments field to note steps you performed to troubleshoot the alert for future reference. See Customizing Row Filters for more information.
    • On the Alert History tab, view the history of alerts to identify alert trends.
    • Right-click an alert, and then click Go To Related View to open the view that was used to trigger the alert.


How Alert Grouping Works

To minimize the number of alerts that appear for a given event or condition, Ultrasound groups related events that occur within a predefined time period. For example, if you successfully deploy providers to fifty servers, only one "Successful deployments" alert appears within a one-week time period. The alert in the Alerts pane shows the number of times the alert has occurred, and you can see the individual alerts that are grouped under a single alert by double-clicking the alert and viewing the Advanced tab of the Alert Details dialog box.

Alert grouping changes in the following scenarios:

  • Only alerts marked Active are grouped. If you change the alert status of an alert from Active to Resolved, a new alert will appear if the event or condition occurs again, even if it occurs within the predefined time period.
  • If a view or row filter that contains the alert is changed in any way, a new alert is created instead of being grouped with existing alerts.

For information about customizing alert grouping, see To customize an existing row filter.


Making Temporary Notes in the Incident Log

Use the incident log to make temporary notes about observations or steps taken in response to FRS conditions or events. This is useful in organizations where multiple people administer FRS and want other administrators to be aware of changes made to the topology, troubleshooting steps, or general observations about FRS behavior.

Entries in the incident log are stored in the database and are not kept indefinitely. Instead, they are purged according to the purge parameters described in To configure Ultrasound options. Therefore, do not store critical information in the incident log.


To add an entry to the incident log

  1. If the incident log is not displayed in the console, click the View menu, and then click Incident Log.
  2. Right-click anywhere in the incident log, such as on WrittenBy, and click Add.
  3. Type your name and the notes you want to enter, and then click OK.


Monitoring Ultrasound in Read-Only Mode

Users who are granted read-only database access can use Ultrasound to monitor FRS and perform basic tasks that do not require updating the database. When a user with read-only database access runs Ultrasound, the console title bar shows [Read Only] and the user can perform the following tasks:

  • View information in all tabs and dialog boxes.
  • Start the Ultrasound Help file.
  • View entries in the incident log.

Users with read-only database access cannot perform the following tasks:

  • Make changes in the Configure Ultrasound dialog box, including:
    • Adding or removing replica sets to monitor.
    • Changing reference members.
    • Installing, updating, and uninstalling providers.
    • Scheduling domain controller and provider polling.
  • Manually poll providers for data.
  • Add comments to the incident log.
  • Respond to alerts.
  • Create or update propagation tests.
  • Customize row filters or health metrics.
  • Configure default notifications.
  • Configure Ultrasound options.

For more information on granting users access to the database, see Granting Database Access to Multiple Administrators.


Customizing Ultrasound

You can customize Ultrasound by doing any of the following:

  • Changing Reference Servers
  • Changing Polling Schedules
  • Configuring Ultrasound Options
  • Customizing Row Filters
  • Customizing Health Metrics


Changing Reference Servers

Ultrasound uses two servers for reference: a replica member and a domain controller. Ultrasound chooses these servers based on logic described in these sections, but in some situations you might want to change the reference servers.

Reference Replica Member

The reference replica member contains what Ultrasound considers the master set of files in a replica set. The reference member is used to collect per-file metadata, such as the full path, change history, file size, and last change time. With this metadata, the reference member is able to translate file GUIDs into file names and provide rich information about backlogged files, excessive replication, and sharing violations. This information is collected only from the reference replica member for performance and bandwidth reasons.

For a given replica set, Ultrasound chooses a default reference replica member that is in the same site as the controller. If multiple replica members exist in the same site as the controller, Ultrasound chooses the reference replica member by using the following criteria:

  1. Ultrasound chooses a replica member that is already a reference member for another monitored replica set.
  2. If multiple replica members meet the criteria in Step 1, Ultrasound chooses the reference replica member that has the most outbound connections.
  3. If multiple replica members meet the criteria in Step 2, Ultrasound chooses the reference replica member has the most inbound connections.

If you change the reference replica member, pick one that is in the same site as the controller. The reference member should be a highly available server, preferably where the majority of file changes originate. For single-master scenarios, choose the reference member acting as the single master. For information about changing the reference replica member, see To change the reference replica member.

Note

  • After you change the reference replica member, the list of files in the replica set shown in the Details tab is still based on the previous reference replica member. The file list will be updated after the next scheduled data collection or you can use the procedure To manually poll providers.

If no replica member exists in the same site as the controller, Ultrasound does not choose a reference member and sends an alert to notify you of this situation. Until you choose a reference replica member, Ultrasound cannot track files or calculate some aggregate file size values.

Reference Domain Controller

The controller uses the reference domain controller to retrieve topology information from Active Directory. When you install Ultrasound, it chooses the closest domain controller as the reference domain controller. You might want to change the reference domain controller if you have experienced Active Directory replication issues; in this case, you might prefer to choose the PDC emulator master instead. If possible, use the same reference domain controller for multiple replica sets to ensure that Ultrasound shows a synchronized view of the FRS topology information retrieved from Active Directory for all the monitored replica sets.

By default, the controller polls the reference domain controller every eight hours to collect FRS topology information from Active Directory, though you can change the schedule if you want. When the database is updated, Ultrasound updates the delta (change) tables to reflect the differences. This information is useful if you need to understand what, if any, topology changes occurred around the same time as the FRS problem appeared.

If the reference domain controller fails or is unavailable due to network problems or other issues, an alert notifies you that the reference domain controller is unavailable. If a reference domain controller is demoted, Ultrasound chooses a new reference domain controller and sends an alert to notify you of this operation.

For information about changing the reference domain controller, see To change the reference domain controller.


To change the reference replica member

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, choose the replica set for which you want to change the reference replica member.
  3. Under Ultrasound Reference Sources, next to Reference member, click the Change Member button.
  4. Under Change Ultrasound reference member to, click the replica member to act as the reference replica member.
  5. Under Apply change to, click one of the following:
    • Current selected replica set to change the reference replica member for only the selected replica set
    • All replica sets with a reference member on the same server as the current member to select a new reference replica member for all replica sets that use the server shown in Current member as a reference replica member. Choosing this setting is useful if you are decommissioning the current member and want a number of replica sets to use another reference replica member that is common to those replica sets.
    • All replica sets on the same server as the current member to use the newly selected reference member as the reference member in any replica set in which this server is a replica member.

For information about changing the frequency at which the reference replica member is polled, see To change the provider polling schedule.


To change the reference domain controller

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, choose the replica set for which you want to change the reference domain controller.

    If you have recently made changes to the FRS topology and those changes don’t appear in this dialog box, click the Sync Topology button to force a manual poll of Active Directory information for the selected replica sets

  3. Under Ultrasound Reference Sources, next to Reference DC, click the Change DC button.
  4. Under Change Ultrasound reference DC to, click the domain controller to act as the reference domain controller.
  5. Under Apply change to, click one of the following:
    • Current selected replica set to change the reference domain controller for only the selected replica set
    • All replica sets referring to the current DC to select a new reference domain controller for all replica sets that use the server shown in Current DC as a reference domain controller. Choosing this setting is useful if you are decommissioning the current domain controller and want a number of replica sets to use another reference domain controller that is common to those replica sets.
    • All replica sets in the current domain to use the newly selected reference domain controller for all replica sets in the domain.

For information about changing the frequency at which the reference domain controller is polled, see To change the reference domain controller polling schedule.


Changing Polling Schedules

By default, the controller polls the reference domain controller once every eight hours and the providers once every two hours to collect FRS topology information from Active Directory and FRS data. The schedule is staggered to minimize peaks in bandwidth use during polling.

You might want to change the polling schedule for several reasons:

  • You want to fine-tune the staggered schedule to spread out the load with respect to network bandwidth or server load.
  • You want to collect data more or less frequently.
  • You want to collect data on a specific time and day, or avoid collecting data during business hours or periods of high network usage.

When changing the polling schedule, keep in mind that polling begins at the specified time and continues until polling is complete. Avoid doing high-frequency polling during periods when you need low-latency replication. If performance is an issue, schedule polling to occur when the replication schedule closes.

Tip

  • To view the amount of data collected and the time required to collect the data, go to the Advanced tab and click the Ultrasound updates view.


To change the reference domain controller polling schedule

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Monitored FRS replica sets, choose the replica set for which you want to change the domain controller polling schedule.
  3. Click Schedule DC, and then click the box that corresponds to the day and time you want to schedule polling from the reference domain controller.
  4. Click either None, Once per Hour, Twice per Hour, or Four Times per Hour to specify the frequency at which polling occurs.

    Note that times shown in the dialog box reflect the local time of the server where the console is installed.


To change the provider polling schedule

  1. On the Tools menu, click Configure Ultrasound.
  2. Under Ultrasound WMI Provider Deployment Status, click the replica member or set of replica members for which you want to schedule provider polling. (To select multiple replica members, hold the CTRL key and click each replica member.)
  3. Click Schedule Provider, and then click the box that corresponds to the day and time you want to schedule polling.
  4. Click either None, Once per Hour, Twice per Hour (every thirty minutes), or Four Times per Hour (every fifteen minutes) to specify the frequency at which provider polling occurs.

    Note that times shown in the dialog box reflect the local time of the server where the console is installed.


To manually poll providers

  1. On the Advanced tab, under General (in the list of views on the left side of the screen), click Servers.
  2. In the list of monitored servers, click the server whose provider you want to poll. (You can also press the CTRL key and click to select multiple servers.)
  3. Right-click the selected servers, point to Collect Data From, and then click Server.

To verify that polling has occurred, do the following:

  1. Press the F5 key to refresh the information in the Servers view.
  2. In the ProviderState column (in the Servers view), look for any of the following states: Queued (Manual), Queued, or Running. If you do not see these states, press F5 periodically. The ProviderState column should transition through these states, then to Idle.
  3. To verify that the update succeeded, under General (in the list of views on the left side of the screen), click Ultrasound updates.
  4. Set the Date filter to Current.
  5. Click the UpdateName column to sort the rows by update name.
  6. Locate the rows that correspond to the servers whose providers you polled manually and verify that the Successful cells for those servers contains a 1.

Depending on the amount of data collected and how busy the controller and database are, there might be a delay between the time when the provider state is Idle and the time when the update is actually completed.


Configuring Ultrasound Options

There are a number of Ultrasound settings that affect how Ultrasound operates. These settings include UI refresh parameters, notification parameters, and purge parameters. You can also configure Ultrasound to refresh data in the console automatically and specify whether the console shows friendly or verbose column names.

  • To configure Ultrasound options
  • To enable auto refresh
  • To change how Ultrasound displays column names


To configure Ultrasound options

  1. On the Tools menu, click Options.
  2. Change any of the following options:

    General

    • Auto Refresh Interval. Specifies how often the user interface is refreshed with updated information. Alerts are always updated according to this interval. For other user interface elements, this interval is only used only when Auto Refresh is selected on the View menu.
    • SQL Connection Timeout. Specifies how long Ultrasound waits to connect to the SQL database.
    • SQL Command Timeout. Specifies how long Ultrasound waits to execute a SQL command before it times out.

    Notifications

    • SMTP Server. Specifies the name of the SMTP server. The default IP port is 25. The SMTP server must support anonymous access or else Ultrasound cannot send email notifcations.
    • SMTP Timeout. Specifies how long Ultrasound waits before the message times out.

    Database Purge

    • Frequency. Specifies how often (in days) the controller purges data from the database.
    • Time to Run. Specifies the time (in local console time zone) at which the controller purges data from the database.
    • Retain. Specifies the number of days to retain data in the database.


To enable auto refresh

  • On the View menu, click Auto Refresh.

After you enable Auto Refresh, the data shown in the console is refreshed according to the Auto Refresh Interval option described in To configure Ultrasound options. (Even when Auto Refresh is not enabled, the alerts are still refreshed according to the Auto Refresh Interval. The Auto Refresh menu option only affects when the tabs in the Workspace pane are refreshed.)


To change how Ultrasound displays column names

  • On the View menu, click Database Schema Column Names.

For information about this setting, see Row Filter Syntax.


Customizing Row Filters

A row filter is a set of criteria applied to records to show a subset of the records. For the subset of records exposed by the row filter, you can customize the alerts, notifications, and health ratings associated with those records using the following topics:

  • Options for Customizing Row Filters
  • Customizing Health Metrics

Before you customize row filters, it is important to understand the database architecture used by Ultrasound. The following table describes the components of the database and whether those components can be customized.

Components

Component | Description | Customizable?
Database schema | The collection of tables that contain data in the database. A table is a two-dimensional object, consisting of rows and columns, used to store data in a relational database. | No
View | An alternate way of looking at data from one or more tables in the database. A view can be thought of as either a virtual table or a stored query. The data accessible through a view is not stored in the database as a distinct object. | Yes
Row Filter | A set of criteria applied to records to show a subset of the records. (This is the WHERE clause of a SQL query) | Yes
Alerts and Health Metrics | An alert is a row filter with a severity rating. A health metric is an object defined as a view that meets the row filter and has an applied rating. | Yes
Notifications | A row filter that has a notification enabled. | Yes


How Views and Row Filters Appear in the Advanced Tab

The Advanced tab shows preconfigured views on the left size of the screen. These views are categorized as either general (apply to all monitored replica sets) or replica set specific. When you click a view, the columns defined in that view appear. For example, if you click the Replica Sets view under General, the columns displayed include ReplicaSet, Domain, #Files, TotalFileSize, and so on.

For a selected view, you can apply row filters. Row filters further reduce the amount of data shown in the views. For example, if you click the Ultrasound updates view under General, you see a series of columns such as UpdateName, UpdateType, Successful, and so on. If you click the Row filter drop-down list, you can filter the results to show only Failed updates, Successful updates, and so on.

Tip

  • To view the SQL query that defines the view, click a view, and then click the Details button.


Options for Customizing Row Filters

As described in Customizing Row Filters, a row filter is a set of criteria applied to a view to show a subset of the records. Part of these criteria is whether the row filter triggers alerts, notifications, and changes to health ratings. Ultrasound provides a number of predefined row filters that trigger alerts, notifications, and health changes, and these row filters work well for most small and medium organizations. If you want to customize row filters, you can do so by opening the Row Filter Editor dialog box, where you have a number of options:

Modify the row filter and add comments

On the General tab, you can modify the filter and add comments. For more information, see Row Filter Syntax and To customize an existing row filter.

Enable or customize alerts

On the Alert tab, you can do the following:

  • Enable or disable an alert for the row filter.
  • Specify alert severity (informational, warning, or error).
  • Group duplicate alerts that occur within a given time period. For more information, see How Alert Grouping Works.
  • Enable default and custom notifications.

For information about enabling or modifying alerts on existing row filters, see To customize an existing row filter.

Enable health metrics

Some row filters allow you to create new or modify existing health metrics that are used to determine the health rating of an object. Health metrics are enabled on the Health Metrics tab. For more information about customizing health metrics, see Customizing Health Metrics.


Row Filter Syntax

Before you create or edit row filters, enable the Database Schema Column Names option on the View menu to make sure that you use the right column names. Every column in the Ultrasound database has a friendly column name and a verbose column name. By default, Ultrasound shows a shortened "friendly" column name to make columns easier to read. In some cases a single friendly column name might be used for columns that have different verbose column names. For this reason, you cannot use friendly names when creating row filters; you need to know the verbose column names instead.

It is also helpful to review the description of the underlying view on which the filter will be implemented. You do this on the Advanced tab by clicking the desired view on the left side of the screen and then clicking the Details button. The View Details dialog box shows the SQL query (on which the view is based) and the available columns.

When you create or edit row filters in Ultrasound, the syntax you use is similar to the WHERE clause in ADO.NET and SQL. Some example search conditions include:

  • Comparison: >, <, >=, <=, =, <>, and IN
  • Arithmetic: + - * / %
  • Operators: AND OR NOT

The following list contains some sample filters:

  • A filter for stopped services: ServiceRunning = 0
  • A filter to detect whether Ultrasound encounters consistent errors when collecting data: (NumRetryAttempts > 5) and (TimeSinceLastSuccessfulCollection > (24*60*60))

    This filter will show all servers where Ultrasound has attempted data collection and failed for greater than five times and where the time since the last successful collection was over 24 hours ago (computed as 24 hours per day times 60 minutes per hour times 60 seconds per minute.)

Note

  • If you want to edit or create row filters, you should be familiar with writing SQL queries. For relevant topics, review the SQL SDK documentation for Microsoft SQL Server 2000 on http://msdn.microsoft.com/.


To customize an existing row filter

  1. On the Advanced tab, click the view whose row filter you want to customize.
  2. Click the … button next to the Row filter drop-down list.
  3. Select the row filter you want to modify, and then click Change.
  4. Do any of the following:
    • Update the text in the Filter text box (advanced users only). See Row Filter Syntax for help in customizing the filter.
    • Update the text in Comments with information that is specific to your organization.
  5. On the Alert tab, do any of the following:
    • Click Enable alert and specify the severity of the alert. If the Enable alert check box is unavailable, you cannot create an alert for this row filter because the UpdateID column is not available in the underlying view.
    • Click Group duplicates and specify the method of grouping alerts and the time period in which to group similar alerts.
    • Click Enable notifications and specify either default or custom notifications.
  6. On the Health Metrics tab, click Enable health metrics to configure health metrics.

    7. If the Enable health metrics check box is unavailable, you cannot enable health metrics for this row filter because the UpdateID column is not available in the underlying view.

For more information about health metrics, see Customizing Health Metrics.


To create a new row filter

  1. On the Advanced tab, click the view where you want to create a new row filter.
  2. Click the … button next to the Row filter drop-down list.
  3. Click Add.
  4. Do any of the following:
    • Type a name for the row filter in Name.
    • Specify the new filter in the Filter text box (advanced users only). See Row Filter Syntax for help in creating the new filter.
  5. On the Alert tab, do any of the following:
    • Click Enable alert, and then specify the severity of the alert. If the Enable alert check box is unavailable, you cannot create an alert for this row filter because the UpdateID column is not available in the underlying view.
    • Click Group duplicates, and then specify the method of grouping alerts and the time period in which to group similar alerts.
    • Click Enable notifications, and then specify either default or custom notifications.
  6. On the Health Metrics tab, click Enable health metrics to configure health metrics. If the Enable health metrics check box is unavailable, you cannot enable health metrics for this row filter because the UpdateID column is not available in the underlying view.

For more information on health metrics, see Customizing Health Metrics.


To view a list of all row filters

To view all row filters for all views, do the following:

  • On the Tools menu, click Row Filters.


Customizing Health Metrics

Ultrasound provides default health metrics that work well for most small and medium organizations. If you want to fine-tune existing health metrics to better match conditions in your organization, you should be familiar with issues regarding FRS, Active Directory, and network connectivity that are common or expected in your organization.

You customize health metrics by using the Advanced tab in the Workspace pane. Health metrics can be configured to indicate health ratings for the following objects:

  • Replica sets
  • Replica members
  • Inbound connections
  • Inbound members
  • Outbound connections
  • Outbound members

You create or customize health metrics for these objects by using a procedure similar to customizing row filters. You select a view in the Advanced tab, which displays the row filters for a selected view, and then add or customize a row filter. However, not all row filters can have health metrics. To determine whether a row filter can have health metrics, perform the following procedure:

  1. In the Advanced tab, click the view for which you want to create or customize health metrics.
  2. Click the … button next to the Columns drop-down list. (You might have to minimize the Alerts pane to see this.)
  3. In the Column Sets dialog box, click Add.
  4. Determine whether any of the objects in the following table are listed under Available columns. If they are, you can create or customize health metrics for the row filter.

Objects That Can Be Assigned Health Metrics

If the view contains this column name… | You can assign a health metric to this object
ReplicaSetID | Replica set
InboundMemberID | Member
OutboundMemberID | Member
MemberID | Member
InboundConnectionID | Inbound connection
OutboundConnectionID | Outbound connection

To customize or create health metrics, see To customize health metrics for an object and To create new health metrics for an object.


To customize health metrics for an object

  1. On the Advanced tab, click the view whose row filter you want to customize health metrics for.
  2. Click the … button next to the Row filter drop-down list.
  3. Select the row filter you want to modify, and then click Change.
  4. On the Health Metrics tab, under Enable health metrics, select the health metric you want to change, and then click Change.

    If Enable health metrics is not already selected, you need to create a new health metric. See To create new health metrics for an object for more information.

  5. In the Rating drop-down box, select the desired rating.


To create new health metrics for an object

  1. On the Advanced tab, click the view whose row filter you want to add health metrics for.
  2. Click the … button next to the Row filter drop-down list.
  3. Select the row filter you want to create new health metrics for, and then click Change.
  4. On the Health Metrics tab, click Enable health metrics, and then click Add.

    5. If the Enable health metrics check box is unavailable, you cannot enable health metrics on this object because the UpdateID column is not available in the underlying view. For more information, see Customizing Health Metrics.

  5. Select the object for which you want to enable health metrics.
  6. In the Rating drop-down box, select the rating you want.


Monitoring FRS Using Sonar

Sonar is a tool designed for monitoring key statistics about FRS members in a replica set. Sonar can collect basic status information from FRS running on either Windows 2000 (all service packs), or Windows Server 2003. You can use Sonar to easily watch coarse statistics on a replica set, and to monitor traffic levels, backlogs, free space, and other issues. Sonar allows definition of filters that define rules for which rows to display, and also allows definition of column sets that can be viewed. Sonar does not modify any settings on the computers that you monitor; it just passively reads information.

Note that the Sonar display is member-oriented — it tries to roll-up statistics into a per-member view, instead of a connection-specific view. This is by design; Sonar provides a way of monitoring a set at a high level for members that are in trouble. However, the actual trouble might be connection-specific, so troubleshooting often involves first finding a member that requires attention by using Sonar, and then drilling into the connection-specific issues by using other tools, such as FRSDiag or the Perl scripts. For more information about these tools, see FRS Data Collection Scripts.


Deploying Sonar

Sonar is implemented using Microsoft Visual Studio .NET and requires the computer on which it is running to have the .NET Framework version 1.1 installed. To download .NET Framework 1.1, visit http://go.microsoft.com/fwlink/?LinkId=16722.

Note

  • It is not necessary to have .NET Framework 1.1 on the computers that are being monitored.

Sonar requires ntfrsapi.dll on the local computer system. If you want to run Sonar on a Windows 2000 Professional or Windows XP system, you need to copy this DLL from a Windows 2000 Server system.

By default, you need to copy the Sonar.exe to a local folder before running it.

To enable .NET Framework 1.1 to execute code from a remote server, go to Microsoft .NET Framework 1.1 Configuration in the Administrative Tools menu, right-click Runtime Security Policy, click Adjust Security, and set the Local Intranet policy to Full Trust.

The tool assumes you have Administrator access, and are able to access WMI, Performance Counters, Active Directory, and the FRS service on the members to be monitored.

To minimize network traffic and status calls into FRS servers, Sonar should be run by a small number of users in your organization at any one time.

  • It is recommended to leave one system running Sonar continuously in logging and notification mode for a given replication set. Note that the CYCLE counters are not available until Sonar has been running for 24 hours.
  • It is recommended that an administrator use Sonar interactively at least daily to observe the traffic and status of the replica set using the suggested methods in this section.

When Sonar is started with no command line options, it allows a query to be defined. The administrator can choose a domain and a replica set within the domain. Administrators can also choose to just view the hub machines in the domain; in this case, Sonar queries the FRS topology in Active Directory and then limits the view to just those computers with a larger than average number of connections. Note that members can be explicitly added or removed later.

After the replica set is selected (or an existing query is loaded), Sonar displays replica member status in a grid view. The users should now click Refresh All to collect the data from the member systems.


Using the Sonar User Interface

For descriptions of components that make up the Sonar user interface, see the following sections:

  • Sonar Tool Tips
  • Sonar Status Bar
  • Sonar Column Types
  • Sonar Menus


Sonar Tool Tips

Many of the cells in the grid display have additional information available as tool tips. These are displayed when the mouse is over a cell, and held stationary for a few seconds. For example:

  • Column headers have tool tips that describe the column contents in more details.
  • In case of a Data Collection failure by Sonar, the Data Collection State cells have tool tips that show which part of data collection failed.
  • The OutConnections column has a tool tip that lists the replica partners, and shows which ones are currently joined.


Sonar Status Bar

A status bar at the bottom of window shows:

  • Queued. Number of requests queued up
  • Pending. Number of requests currently being serviced (up to 25)
  • Succeeded. Number of requests which succeeded
  • Failed. Number of requests failed
  • Notify. Notifications enabled/disabled.
  • Refresh in. Time to next scheduled refresh
  • Log. Indicates if the log is enabled (started) or disabled (stopped).


Sonar Column Types

Some columns are expressed in terms of totals, deltas, or cycles:

  • Total means data that is displayed as an absolute value or string. For some parameters it is defined as the total number of occurrences since the server last started; for others it is a quantity expressed by the service (for example, staging space in use).
  • Delta means show delta during this refresh period: the difference between the absolute between the last two refresh intervals.
  • Cycle means the total number of occurrences within a 24 hours period. If a 24-hour period has not elapsed (i.e., the server started less than 24 hours ago) the value is null.


Sonar Menus

The following sections describe the menus in Sonar.

File menu

The File menu provides a way to create a new query, open an existing query, or save the current query. Queries are stored as XML files with the file type .SNR.

The File menu also provides a way to log output to a Comma Separated File, so that it may be analyzed by another tool.

Options menu

The Options menu allows definition of servers in the query, filters on which rows to display, and definition of column sets. These definitions form part of the query saved in the .SNR files.

The Options menu also allows selected rows to be either copied to the clipboard (for pasting into applications such as Word, Excel, or Notepad), or for the selected rows to be refreshed.

Select Servers submenu

The Select Servers submenu provides a way to define which members of the replica set will be queried for status.

The left-hand column lists servers in the replica set which could be added to the query. This may show candidates either (a) from the list of all members of the replica set, or (b) just members of the replica set with a greater than average number of connections – these are considered to be hubs.

The right-hand column lists the members of the replica set that are part of the current query.

This ability to select servers is useful for limiting network bandwidth usage on a WAN by excluding computers that are connected by low-bandwidth links.


Configuring Sonar

For descriptions of how you can customize Sonar, as well as methods for viewing data in Sonar, see the following sections:

  • Editing Filters and Notifications
  • Selecting a Column Set
  • Refreshing Information
  • Sorting and Resizing Columns
  • Associating Programs with Rows


Editing Filters and Notifications

The current active filter and column set is selected from a drop-down box above the grid tool.

The filters shown above are default filters that are created in any new query, but you can define your own filters in the Options menu.

The rules for filtering rows (using a syntax similar to the "Where" command in SQL) are as follows:

  • Comparison: > < >= <= > LIKE IN
  • Arithmetic: + - * / %
  • String Ops: * LIKE
  • Aggregate: Sum Avg Min Max Count StDev Var
  • Functions: IIF (if operator)
  • Operators: AND OR NOT

Examples:

  • VerCompiledOn < MAX( VerCompiledOn )
  • SharingViolations > 0

Note that filters can also be defined as notifications; if enabled, at the end of each refresh cycle, Sonar executes each query marked as a notification. An email message is sent listing all filters that result in a nonempty row set.

The Min Notify Interval shows the minimum time between Sonar sending notifications by email. An example of a Sonar notification is:

From: SONAR [mailto:SONAR]
Sent: Wednesday, August 28, 2002 7:28 PM
To: FRSMON
Subject: SONAR FRS Notification [8/28/2002 7:28:25 PM]: Replica Set 'FRSDog|Pub1|Top' in domain test.reskit.com
Message from SONAR status viewer for Windows File Replication Service
- Issued by X:\frs\sonar\Sonar.exe on FOO.test.reskit.COM
- User: test\frsmon
- Query:
- Domain: test.reskit.com
- Replica set: 'FRSDog|Pub1|Top'
The following filters have been triggered as part of a data refresh:
--------------------------------------------
Filter: Currently involved in join
Servers: frsdogb
--------------------------------------------
For more details, use the SONAR tool to investigate the status.


Selecting a Column Set

Column sets are selected from another drop-down box, and again there are a number of built-in column sets added to any new query. You define your own column sets by clicking Edit Columns on the Options menu.


Refreshing Information

You can refresh all servers by clicking the Refresh All button, or you can select rows by right-clicking them, then choosing to just refresh the selected ones by clicking Refresh on the Options menu. In addition, the default refresh period can be modified from the final drop-down box on this part of the window.


Sorting and Resizing Columns

The grid control can be sorted by clicking on a column heading, and columns can be resized by dragging the boundary between columns.


Associating Programs with Rows

When you double-click a row, Sonar can call out to a program defined in the HKEY_LOCAL_MACHINE\Software\Microsoft\FRSMon\SONAR\Expand registry value (REG_SZ), and will execute "program name" with the parameters "<local computer name> <domain name> <replica root> <column name>". The user can thus define what program actually is executed; for example, it could be Health_chk, or a Connstat report, and this can even be column-sensitive.


Monitoring FRS by Using Sonar Column Sets

When monitoring FRS, work through the defined column sets for Sonar for the replica sets you are investigating, and then work through some filters for specific issues. The following sections describe the column sets in Sonar.

  • Error Conditions Column Set
  • Backlog Column Set
  • Replication Status Column Set
  • Resource Usage Column Set
  • Settings Column Set
  • Sources of Change Column Set


Error Conditions Column Set

This column set shows the most common situations that require attention. This is the default column set that Sonar displays, and issues that show on this column set should be addressed first.

DataCollectionState

This is not an FRS-reported error. Instead, it indicates that Sonar was unable to collect one or more pieces of status information about the FRS system. The details of the failure can be determined by either hovering the mouse pointer over the cell, or by looking at the DataCollectionError column in the All Columns view:

  • PerfCtr. Failed to read Performance counter

    Known to fail on Windows 2000 SP2 systems and Windows Server 2003 RC1 systems. On other platforms, issues are normally resolved with a refresh.

  • Registry. Failed to read registry
  • SCM. Failed to query Service Control Manager
  • DS. Failed to query Active Directory
  • TimeZone. Failed to get TimeZone information
  • Proc. Failed to collect Process information
  • WMI. Failed to query information via WMI
  • EventLog. Failed to query Event log
  • SYSVOL. Failed to connect to SYSVOL file share
  • FRSVer. Failure of the FRS Version RPC interface
  • FRSInlog. Failure of the FRS Inlog RPC interface
  • FRSSets. Failure of the FRS Sets RPC interface

FRSState

Servers are normally in the ACTIVE state, look for servers in other states:

  • ALLOCATED. Replica set is in an initializing state
  • INITIALIZING. Replica set is in an initializing state
  • STARTING. Starting the replica set
  • STOPPED*. Replica set is now stopped, typically because the set no longer has connections to any partners
  • ERROR*. Replica set is stopped due to an error
  • JRNL_WRAP_ERROR*. Replica set is stopped due to data loss in the NTFS journal
  • REPLICA_DELETED. Replica set is marked as deleted

If a member is in one of the states marked with an asterisk (*), it requires attention before it can replicate again

SCM State

Shows if the FRS Service is running according to the Windows Service Control Manager. Look for servers that are not in the "Running" state, and investigate. FRS servers should normally be left running.

SYSVOLShared

If the replica set is a SYSVOL, look for "Not shared" or "Not a junction." For troubleshooting information, see Verifying SYSVOL Junction Points.

LongJoinCycle

High numbers indicate servers that have had problems joining with a replication partner. For troubleshooting details, see Troubleshooting FRS Events 13508 without FRS Event 13509.

ExcessiveReplicationCycle

Shows servers that have detected files that have been touched by some process, but not updated. If this is non-zero, see Excessive Replication and High CPU and Disk Usage by Ntfrs.exe.

Huge File Cycle

This shows the number of 13523 events logged on this member in the last 24 hours. This indicates that a file could not be replicated since it was larger than the staging directory size, and so cannot be replicated until the staging area has been reconfigured.

Increase the staging directory size on computers in the replica set so that these files can be replicated.

To increase the size of the staging directory

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters.
  3. Right-click the Staging Space Limit in KB entry and click Modify.
  4. Click Decimal, type the desired size of the staging directory in kilobytes, and click OK.
  5. Close the registry editor.
  6. At the command prompt, type net stop ntfrs & net start ntfrs.

Staging Full Cycle

This shows the number of 13522 (Staging Area Full) events logged on this member in the last 24 hours. This indicates that a file could not be replicated because the staging directory was full. To resolve this, increase the size of the staging directory.

ChangeOrdersMorphedCycle and ChangeOrdersMorphedTotal

Shows the number of directories morphed to foldername_NTFRS_xxxxxxxx names due to conflicting updates. See Morphed Folders for details on the causes and cleanup of this situation.

CommTimeoutDelta, CommTimeoutCycle, and CommTimeoutTotal

Look for computers with high numbers of communication timeouts. This is an indication of network connectivity problems, and if it persists, should be investigated.


Backlog Column Set

After addressing issues shown in the Error Conditions column set, the next column set to investigate is Backlog. This column set shows the backlog most common situations that require attention. The additional information in this column set is:

OutConnections

Shows the number of outbound connections for this member in this replica set. This is helpful for identifying which computers are hubs in the replication topology.

BacklogConnections

This indicates that a number of connections from this member is in a state where the maximum number of files are being replicated in parallel. If this persists, troubleshoot as described below and in the next chapter.

BacklogFiles and BacklogSize

Look for servers with unusually high numbers of files backlogged awaiting replication, or large total quantities of data to replicate. If this persists, troubleshoot as described below and see the section Backlogged Files.

  1. Backlogs are actually a per-connection property. Because Sonar just shows per-member information, it only displays the most backlogged connection for this member.
  2. Number of files backlogged is determined by examining the difference between the change order sequence numbers: BacklogFiles = NextChangeOrderSeq – TrailingChangeOrderSeq. However, there are some cases where this metric gets distorted: A vvjoin operation creates directed change orders, which only go to one partner, but Sonar does not take account of this possibility in this metric.

LongJoinCycle

See Error Conditions Column Set for details.

SharingViolations

This actually shows how many files are waiting to be installed on this computer (this list is known as the inlog or inbound logs). Files are most commonly in this state if there is a sharing violation preventing a file being installed; however, there are other potential reasons why a change order might wait in the inlog. See Sharing Violations for more information.

VVjoinsActive

This shows computers that are in the process of joining (performing an initial synchronization) with their replication partners. Note that this process can occur in normal operation, and might cause a temporary replication backlog.

VVJoinsTotal

This shows the total number of VVjoins (or attempted VVjoins) this computer has been involved in since it restarted. This is not an indication of an error, because VVjoins can occur in normal operation.


Replication Status Column Set

This is a more expanded column set that is useful when drilling down into issues with backlogs. It shows more context about what the cause of a backlog might be. New information in this column set includes:

InConnections

Shows the number of inbound connections for this member in this replica set. This is helpful for identifying which computers are hubs in the replication topology.

InJoinedConnections and OutJoinedConnections

This shows how many of the Inbound/Outbound connections are currently joined (and hence able to replicate).

LastInJoinInterval

Time interval since the last of this member’s inbound partners joined to replicate.

LastOutJoinInterval

Time interval since the last of this member’s outbound partners joined to replicate.

LocalChangeOrdersCycle

Shows changes initiated by this member in the last 24 hours. Regularly look to see that changes are being initiated from computers you expect. Look for unusual cases of many members suddenly originating changes, or for one computer unexpectedly initiating a lot of change.

StagingFilesRegeneratedCycle

On Windows 2000 SP3 and later, this shows that the staging directory was too small and that files are being purged from this cache. If this condition occurs frequently:

  • Confirm that all direct outbound replication partners receiving updates from this member are online and receiving updates
  • Verify that the replication schedule for receiving partners is open or "on" for a sufficient window of time to accommodate the number of files being replicated
  • Consider increasing the staging directory size to improve system performance:

To increase the size of the staging directory

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters.
  3. Right-click the Staging Space Limit in KB entry and click Modify.
  4. Click Decimal, type the desired size of the staging directory in kilobytes, and click OK.
  5. Close the registry editor.
  6. At the command prompt, type net stop ntfrs & net start ntfrs.


Resource Usage Column Set

The next column set to Monitor is the Resource Usage set, which shows the disk space available to support FRS.

  • DiskSpaceReplicaRoot. MB of free space on the replica root.
  • DiskSpaceStagingRoot. MB of free space in the staging directory.
  • DiskSpaceDB. MB of free space on the volume holding the FRS database.
  • StagingFilesRegeneratedCycle. See Replication Status Column Set for details.


Settings Column Set

This column set shows some key parameters of the FRS members. New information in this column set includes:

UpdateTime

The time Sonar last collected data from this member.

VerLatestChanges

The FRS version string for this member.

BURFLAGS

The Backup Restore Flags are active for this member. This is useful for confirming that the entire replica set has been correctly configured during the Authoritative Restore process.

USNJournalSize

The FRS registry setting for the NTFS USN journal size. If 0, then this indicates default. The default sizes are as follows:

  • Windows 2000 SP2 and earlier: 32 MB
  • Windows 2000 SP3: 128 MB
  • Windows 2000 SP3 with the post-Service Pack 3 release of Ntfrs.exe, described in article 815473: 512 MB
  • Windows 2000 SP4: 512 MB
  • Windows Server 2003: 128 MB
  • Windows Server 2003 with the pre-Service Pack 1 release of Ntfrs.exe described in article 823230: 512 MB.

Note that 128MB of journal is recommended per 100,000 files in the replica set. For more information about the staging directory, see Adjust the Staging Directory Size Accordingly.


Sources of Change Column Set

This column set indicates statistics about sources of change in the replica set. In normal use, most systems will have a limited number of systems that generate change in the replica set. Use this column set on a regular base to confirm that change is being originated from the expected systems. New information in this column set includes:

UpdateTime

The time Sonar last collected data from this member

ExcessiveReplicationCycle

Shows servers that have detected files that have been touched by some process, but not updated. For troubleshooting details, see Excessive Replication and High CPU and Disk Usage by Ntfrs.exe.

LocalChangeOrdersDelta, LocalChangeOrdersCycle, and LocalChangeOrdersTotal

Shows changes initiated by this member in the last 24 hours. Regularly look to see that changes are being initiated from computers you expect. Look for unusual cases of many members suddenly originating changes, or for one computer unexpectedly initiating a lot of change.

ChangeOrdersMorphedCycle and ChangeOrdersMorphedTotal

Shows the number of directories morphed to foldername_NTFRS_xxxxxxxx names due to conflicting updates. See Morphed Folders for details on the causes and cleanup of this situation


Troubleshooting FRS

When troubleshooting FRS, you have a number of options:

  • If you do not know what the problem is, you can follow the steps in the FRS Troubleshooter to diagnose and repair the problem.
  • If you have reviewed the File Replication Service event log and know the event ID of the FRS problem, see Troubleshooting FRS Using Event Logs for a list of common FRS problems and links to information on how to repair those problems. You can also search this Help file for specific event IDs.
  • If you are using Sonar or Ultrasound and have detected an FRS problem, the documentation for those tools will often refer you to a topic in the FRS Troubleshooting Topics for specific steps to repair the problem.
  • You can contact Microsoft Product Support Services for assistance.


FRS Troubleshooter

The FRS Troubleshooter is a step-by-step guide to diagnosing and repairing FRS replication problems. This troubleshooter is useful if your servers are experiencing replication problems but you are not sure what the problem is or where to begin. If you are using Sonar or Ultrasound, you might not need to use this troubleshooter because Sonar and Ultrasound can often pinpoint the problem and provide appropriate troubleshooting topics in this Help file.

Begin the FRS Troubleshooter.


Troubleshooting FRS Using FRSDiag

FRSDiag is an easy-to-use tool that replaces the Health_Chk tool used to gather FRS information. FRSDiag provides a graphical user interface for choosing which information to gather and analyzes the results to identify common FRS and Active Directory problems.

FRSDiag is recommended in the following scenarios:

  • You need to gather additional information about a replica member after Sonar or Ultrasound detects a problem.
  • You want to investigate FRS error events in the File Replication Service event log.
  • You are prompted to do so when you work through the FRS Troubleshooter.

The following sections describe FRSDiag.

Minimum Requirements

You can run FRSDiag on Windows 2000 Server, Windows Server 2003, Windows XP, and Windows 2000 Professional. To run FRSDiag, you must have .NET Framework version 1.1 installed. The .NET Framework 1.1 is included in Windows Server 2003, or you can download it from http://go.microsoft.com/fwlink/?LinkId=16722. The .NET Framework 1.1 is included in Windows Server 2003 and can be installed on Windows 2000 (with SP2 recommended) and Windows XP Professional.

Installing and Running FRSDiag

Run FRSDiag.exe to extract the files, and then run the extracted FRSDiag.exe to start the program.

Information FRSDiag Collects By Default

  • Event logs
  • Ntfrsutl output
  • FRS debug logs
  • FRS registry dump
  • SYSVOL dump
  • Repadmin /showreps and /showconn

Additional Information that FRSDiag Creates

  • FRS debug logs error scan
  • FRSDiag.txt contains the results of tests
  • FRSDiag_Log.txt contains information about which tests were requested
  • Connstat.txt parses out the sets .txt into a more readable format
  • IDTable.txt parses out the IDTable into a more readable format with full file paths
  • A .cab file that contains all collected files

FRSDiag Tools

  • GUID2Name builds a server GUID-to-name correlation (output in a .txt file) against selected server(s).
  • ForceReplication forces replication on target servers, which causes the target servers to replicate all changes from parent (pull replication) regardless of intersite schedules set. This feature only works if you are running at least Windows 2000 SP3 with 811370, Windows 2000 SP4, or Windows Server 2003. This tool only works against SYSVOL, not DFS volumes.
  • Propagation File Tracer can be used to check the following in SYSVOL file shares. (This feature is not available in DFS replica sets.)
    • Check for the existence of a single (Propagation) file in SYSVOL. (Note this is not related to the propagation file functionality in Ultrasound.)
    • Current FRS replication consistency (number of files/folders existing in each server)
    • Look for morphed folders in the replica set
  • Look for no longer needed Pre_Existing folders
  • Other files and folders as specified by using wild cards. You can also use the retry feature to keep checking servers that fail the specified checks.

Tests FRSDiag Performs

FRSDiag performs the following tests. An asterisk (*) next to a test means that a failure might not necessarily indicate a problem.

FRS Event Log*

  • Tests for Errors and Warnings (filters when newer 13516 exists)
  • Tests 13508 warnings for unmatched 13509
  • Filters when event if over 10 days old AND is not one of the latest 7 events

DS Event Log*

  • Tests for Errors in the last 12 days

Ntfrs_version

  • Tests for minimum FRS version requirements

Ntfrs_DS Services

  • Tests for (null) or broken serverReference objects
  • Tests for member servers with no connection objects
  • Tests for disabled connection objects*
  • Tests for null or improperly set fRSMemberReference on SYSVOL volumes
  • Tests for (null) values on fRSMemberReference on DFS volumes

Ntfrs_Replica Sets

  • Warns about file filters not set to default values*
  • Warns about directory filters not set to default values*

Ntfrs_Connstat*

  • Warns when number of backlog files is greater than 30 files
  • Errors when number of backlog files is greater than 100 files

Ntfrs_Config Table:

  • FRS structure (if SYSVOL Dump is selected)
  • Tests for existence of SYSVOL folder
  • Tests for existence of SYSVOL junction point
  • Tests for existence of staging area folder
  • Tests for existence of staging area junction point

Inlog/Outlog*

  • Warns when 15% of change orders were triggered by "Security" change
  • Errors when 40% of change orders were triggered by "Security" change

IDTable

  • Errors when an IDtable entry (file or directory) is found with no parent (missing parent issue); details about the IDtable entry are written.

FRS Debug Logs

  • ERROR_SHARING_VIOLATION errors*
  • ERROR_ACCESS_DENIED errors*
  • IBCO_STAGING_RETRY errors
  • IBCO_FETCH_RETRY errors
  • IBCO_INSTALL_RETRY errors
  • Missing Parent Issue Fixed in 811217
  • INVALID Join GUID Fixed in 811217

Services and File shares

  • Ensures the following needed services are running:
    • NtFrs
    • W32Time
    • Netlogon
    • Server and Workstation
    • RpcSs (RPC Service)

Tests if SYSVOL and NETLOGON file shares exist and are accessible*

Repadmin /showreps

  • Tests for any failed Active Directory replication events.
  • Options
    • Auto-Retry Target Server(s) that failed to grab data. This option lets you enable Auto-Retry when processing multiple servers. This is helpful for large environments where a server may be unavailable at that time but you must have information from all servers therefore keep trying until the server comes back up. It can be toggled on/off during execution.
    • Attempt to DnsResolve prior to starting. This option checks if you are able to resolve the Target Server(s) name and if that server is accessible at this time. It should only be turned off in large environments where you’d like to keep retrying for down servers and you are confident that name resolution is working correctly.
    • Create “Logs” Directory for data output. This option is to create a Logs directory where all output data will go to (if this is not checked, output is generated from the location Frsdiag.exe was called from, usually on the root of where it resides).


Troubleshooting FRS Using Event Logs

If you are not using an FRS monitoring tool, the File Replication Service event logs are a key source of monitoring information. The following table summarizes the main File Replication Service event log entries that should be monitored on a regular basis and the actions required to troubleshoot each case.

FRS Event Log Messages

Event ID | Priority | Summary | Actions Required
13508 | (depends) | Unable to RPC to a replication partner | Wait for 13509. If no 13509 follows within 4 hours (rule of thumb) then investigate. See Troubleshooting FRS Events 13508 without FRS Event 13509.
13509 | (none) | Able to RPC to a replication partner | No action; this event indicates that the 13508 wait is over.
13511 | P1 | FRS database is out of disk space | Make more space available for FRS database. See Not Enough Disk Space for the FRS Database.
13512 | (none) | Enabled disk cache detected | Typically no action required.
13522 | Windows 2000 SP2: P1 Windows 2000 SP3: P3 | Staging area full | On Windows 2000 before SP3, requires administrator intervention to clear replication backlog. In Windows Server 2003 and Windows 2000 SP3 and later, automatic processes trim the size of the staging area and treat it like a cache. Administrators should investigate increase as an optimization if this is repeatedly logged. See Staging Directory is Full.
13526 | P1 | SID cannot be determined from the distinguished name. This can occur when the domain controller is unreachable. | Restart FRS on that system. See Domain Controller is Unreachable.
13548 | P1 | Systems clocks are too far apart | Correct clocks on one or more replica members. See Clock Skew.
13557 | P1 | Duplicate connections configured | Delete the unnecessary connection object.
13567 | P2 | Excessive replication detected and suppressed | Investigate what is causing excessive replication work; in the meantime, the FRS server uses CPU and disk resources to damp the replication traffic by comparing files to previously sent copies. See Excessive Replication and High CPU and Disk Usage by Ntfrs.exe.
13568 | Windows 2000 SP2: P2 Windows 2000 SP3: P1 | Journal wrap | Depends on FRS version. See USN Journal Wrap.
13573 | P1 | Sharing violation | A file is held open on a server, preventing the file from replicating. See Sharing Violations.


FRS Troubleshooting Topics

Each of the FRS troubleshooting topics in this section addresses a particular FRS problem. The topics are alphabetized according to their titles and are usually linked to from some other source within this Help file, such as the Sonar documentation, or directly from alerts reported in Ultrasound. Due to the large number of topics, they are not meant to be browsed as a way of troubleshooting FRS. It is much easier to use the search function to locate topics related to event IDs or other common FRS problems, such as morphed folders, excessive replication, USN journal wraps, and so forth. If you are not sure where to begin, try the FRS Troubleshooter.


All Domain Controllers Are Currently Unavailable

Ultrasound has detected that all domain controllers are unavailable. This alert is triggered when Event ID 1566 appears in the Directory Service event log.

Resolution

Use the topic Verify FRS Dependencies to check for DNS, Active Directory replication, and network connectivity problems.


Assertion/Failure in FRS: FRS Failure with Event ID 13506 or 13555

FRS event ID 13506 occurs when FRS has detected an internal consistency problem.

Resolution

FRS will restart automatically at a later time. If this problem persists, a subsequent FRS event ID 13555 describes the recovery procedure that should be carried out.


Backlogged Files

Ultrasound has detected that files have not replicated to other partners. This problem could be caused by the following factors:

  • The replication schedule is too short to allow all data to replicate to other members.
  • Network bandwidth is affecting the speed at which files replicate, causing a delay.
  • A downstream partner is unavailable due to network problems or other issues.

This alert might also be triggered when a D2 is performed on a downstream partner.

Resolution

If a D2 was not performed on a downstream partner, look for failure indicators at either the upstream or downstream partners. If you cannot find failure indicators, reexamine the schedule and network bandwidth on this connection to ensure that enough replication time is scheduled to allow the data to replicate.


Bad SMTP Server

Ultrasound could not send email notifications using the specified SMTP server.

Resolution

See To configure Ultrasound options to specify the SMTP server used to send email notifications. If the SMTP server is valid, try increasing the SMTP timeout.


Propagation File Failed Drop to Initial Members

Ultrasound is attempting to create a hidden propagation file whose name resembles UltrasoundPropagationFile_{2696CDC3-17AA-4206-8F45-27BF61242B8D} in a hidden directory called UltrasoundPropagationFiles. <SERVER_NAME> located in the SYSVOL\domain folder (for domain controllers) or directly under the replica tree root (for DFS) but failed.

Resolution

Check local system permissions to the folder as well as other physical causes, such as disk space issues.


Propagation File Failed to Replicate to All Members

Ultrasound has detected that the specified propagation test did not complete within the timeout period.

Resolution

Use the procedure To view statistics about completed propagation tests to view detailed information about propagation file propagation.


Changes Made On Member with No Outbound Connection

Ultrasound has detected that files or folders were changed on a replica member that has no outbound connections. These changes will not be replicated.

Resolution

If the replica member was purposefully configured without outbound connections, investigate why these changes were made and determine whether you need to make the same changes on a replica member with outbound connections.

If the replica member was configured incorrectly, add the necessary outbound connections so that the changes replicate to other members.


Clock Skew

FRS event ID 13548 occurs when the specified replica member has a system clock that is plus or minus 30 minutes different from one of its replication partners. This error could be caused by the selection of an incorrect time zone on the local server or its replication partner.

This replica member cannot replicate with its partner until the system clock is corrected.

Resolution

Check that the time zone and system clock are correctly set on both servers; the clocks must be within 30 minutes of each other, but preferably much closer.


Clocks Moved Ahead

FRS event ID 13549 occurs when FRS is unable to replicate from a partner computer because the event time associated with the file to be replicated is too far into the future. This can happen if the system time on the partner computer was set incorrectly when the file was created or updated. To preserve the integrity of the replica set, this file update will not be performed or propagated further.

If this time difference is close to a multiple of 60 minutes, then it is likely that this file might have been created or updated on the partner computer while the computer was set to the incorrect time zone when its computer time was initially set.

Resolution

Check that the time zone and time are correctly set on the partner computer.


Confirmation of Volume Change Required

FRS event ID 13559 occurs when FRS has detected that the replica root path has changed.

Resolution

If this is an intentional move, you must create a file with the name NTFRS_CMD_FILE_MOVE_ROOT under the new root path. See event ID 13559 for more information.


Corrupted Member Object

Ultrasound has detected that a member object in Active Directory is corrupted.

Resolution

Refer to the following articles for more troubleshooting procedures.

  • 811219, "Recovering FRS Objects and Files Using System State Restores" at http://support.microsoft.com/?id=811219
  • 312862, "Recovering Missing FRS Objects and FRS Attributes in Active Directory" at http://support.microsoft.com/?id=312862
  • 296183, "Overview of Active Directory Objects That Are Used by FRS" at http://support.microsoft.com/?id=296183


D2 Detected

Ultrasound has detected that the originator GUID for this server has changed, indicating that a D2 operation has occurred.

Resolution

No resolution is required. Other alerts are typically generated on upstream partners as a result of a D2, including Staging Files Regenerated and Very Old Backlogged Files. If these alerts appear on an upstream partner, they can be resolved.


DNS Name Cannot Be Resolved

FRS cannot find the DNS name for a replication partner.

Resolution

Verify that DNS is working correctly and that this system is able to resolve the DNS names of its replication partners. Check general network connectivity and ensure there are no firewalls between the members. For more details about this error, look at the text for event ID 13525 in the File Replication Service event log on this system and look for DsCrackNames failures in the FRS debug logs on this member.


Domain Controller is Unreachable

FRS event ID 13526 occurs when a domain controller becomes unreachable. This problem occurs because FRS polls Active Directory at regular intervals to read FRS configuration information. During the polling, an operation is performed to resolve the security identifier (SID) of an FRS replication partner. The binding handle might become invalid if the bound domain controller becomes unreachable over the network or restarts in a single polling interval (default 5 minutes).

Resolution

To resolve this issue, restart FRS on the computer logging the error message.


Duplicate VSN Detected

FRS event ID 13571 occurs when FRS has detected that one or more volumes on this computer have the same volume serial number. FRS does not support this configuration and will not replicate until the volume serial numbers on disks on this system are unique.

Resolution

Correct this problem by dismounting one of the duplicate volumes or by using a tool that can change the volume serial number of a volume.


Error Collecting Data from Active Directory

An error occured when Ultrasound attempted to collect topology data from Active Directory. This alert can be caused by network connectivity issues, authentication or authorization issues, or other problems.

Resolution

Investigate and resolve network or security issues.


Excessive Replication and High CPU and Disk Usage by Ntfrs.exe

FRS event ID 13567 occurs when FRS detects and suppresses excessive replication. Excessive replication is typically caused by applications that change all or most of the files in a replica tree on a regular basis without actually altering the file content — in other words, these applications perform writes that do not change the file. Because FRS monitors the USN journal for changes, if FRS finds a change, then it replicates the file. Excessive replication can cause FRS to use excessive amounts of disk I/O or CPU as it tracks this activity. Common cases of applications that cause extensive replication are those that rewrite the existing ACL (file security policy, antivirus software) or rewrite the existing file (defragmentation programs).

The effect of excessive replication depends on the version of FRS being used:

  • From Windows 2000 Service Pack 3 and later and Windows Server 2003, FRS event ID 13567 records that this kind of non-change was suppressed to prevent unnecessary replication. However, the underlying cause should still be investigated, because it is causing unnecessarily high disk and CPU usage on that replica member.
  • For Windows 2000 versions prior to SP3, excessive replication generators were the most common cause of replication backlogs and for staging directories to fill up. On these versions of FRS, administrators should investigate this problem urgently.

High CPU usage without event ID 13567 can be caused in the following situations:

  • The number of connections is not balanced between hub servers.
  • There are more than 50 incoming connections to the server.
  • The KCC chose a server that was not intended to be a hub server.
  • Other applications, such as Exchange or SQL Server, are running on the server.
  • The staging directory is too small, which causes staging files to be regenerated and increases CPU and disk usage.

Resolution

Identify the cause of excessive replication and high CPU and disk usage by doing the following:

  • If the connections aren’t balanced or the server has too many simultaneous incoming connections, adjust the schedule by using the guidelines in Optimizing FRS Replication Schedules.
  • If high CPU usage is caused by other applications, upgrade the hardware or move the application to another server.
  • If the staging directory is too small, see Adjust the Staging Directory Size Accordingly.
  • If FRS event ID 13567 is present in the File Replication Service event log, do the following:
    • Selectively turn off common causes of excessive replication, such as antivirus products, backup programs, and defragmentation tools, to determine if excessive replication declines.
    • Determine whether your organization has set file system policy on replicated data.
    • Use the FileSpy tool from the Windows 2000 Resource Kit to identify file information.
    • Run FRSDiag.exe and inspect the Ntfrs_outlog.txt file to see which files are being replicated. See "IOLogSum Remarks" for more information on interpreting this file.

For additional information, see the following articles:

  • 815263, "Antivirus, Backup, and Disk Optimization Programs That Are Compatible with the File Replication Service" at http://support.microsoft.com/?id=815263.
  • 284947, "Antivirus Programs May Modify Security Descriptors and Cause Excessive Replication of FRS Data in Sysvol and DFS" at http://support.microsoft.com/?id=284947.
  • 822158, "Virus Scanning on a Domain Controller" at http://support.microsoft.com/?id=822158.
  • 282791, "FRS: Disk Defragmentation Causes Excessive FRS Replication Traffic" at http://support.microsoft.com/?id=282791.
  • 279156, "The Effects of Setting the File System Policy on a Disk Drive or Folder Replicated by the File Replication Service" at http://support.microsoft.com/?id=279156.
  • 307777, "Possible Causes of a Full File Replication Service Staging Area" at http://support.microsoft.com/?id=307777.


Excessive VVJoins

Ultrasound has detected that this replica member is currently synchronizing (VVJoin) with a large number of downstream partners. While this occurs, performance will be impacted on this server.

Resolution

Microsoft recommends that a server synchronize with a small number of downstream partners (typically less than five) at a time, particularly for large replica sets. Some workaround include:

  • Reduce the number of connections to the member to be joined.
  • Stagger the schedule on the outbound connections so that the new member connects to one server at a time.
  • For DFS replica sets, configure connection priorities to specify which server the new members should connect to first. For more information about connection priorities, see the chapter "Designing and Deploying File Servers" in the Planning Server Deployments book of the Windows Server 2003 Deployment Kit available at http://go.microsoft.com/fwlink/?LinkId=16513.


Failed Ultrasound Server Updates

Ultrasound cannot contact this server due to a network outage or other problem. This alert is triggered in a number of cases:

  • After five or more failed attempts, Ultrasound has been unable to collect data from the server for more than one day.
  • Ultrasound has been unable to collect data from this server for more than one week.

Resolution

On the specified server, do the following:

  • Determine whether you can contact this server from the server where the controller is running.
  • Verify that the Windows Management Instrumentation service is running on the server.
  • If no network connectivity problems exist and the provider service is running, see the FRS Troubleshooter for additional troubleshooting procedures.


Files Not Replicating

Files can fail to replicate for a wide range of underlying reasons: DNS problems, firewalls, topology problems, insufficient disk space, the FRS service being stopped, FRS servers in an error state, or sharing violations. When troubleshooting this problem, focus on how to enable FRS to run again. Do not start replicating data using some additional, out-of-band mechanism, such as manual copying of files. If replication stops for some reason, the very worst thing you can do is to copy files manually to replication partners — this will cause additional replication traffic, backlog, and possible replication conflicts (see Morphed Folders for more details).

Important

  • If you need to manually copy folders, ensure that you fully understand the implications of this action, or perform this task under the advice of your Microsoft Product Support provider.

The correct course of action is to find the root cause. Common causes for files not replicating include lack of disk space for data or staging, poor connectivity, critical FRS objects/attributes missing in Active Directory, or files and directories that are in use and cannot be replaced.

A general procedure for troubleshooting FRS when files appear to not be replicating between any two direct replication partners A and B consists of the following steps:

  1. Examine the event logs on the computers involved. The most common causes of replication failure are logged in the File Replication Service event log.
  2. Verify that both Computer A and Computer B are available on the network. Because FRS uses the fully qualified domain name (FQDN) of the replica members, a good first check is to use a ping command specifying the fully qualified name of the problem replicas. From the console of Computer A, send a ping command with Computer B’s FQDN. Then, from the console of Computer B, send a ping command to Computer A’s FQDN. Verify that the addresses returned by the ping command are the same as the addresses returned by an ipconfig /all command carried out on the command line of the destination computer.
  3. Verify remote procedure call (RPC) connectivity between Computer A and Computer B. A good method to do this to execute the command-line command NTFRSUTL VERSION FQDN_of_other_computer from both servers.
  4. Use the Active Directory Sites and Services snap-in to verify the replication schedule on the connection object, and to confirm that replication is enabled between the computers.
  5. Check for files that are larger than the amount of free space on the source or destination server, larger than free space on the staging directory volume, or the size of the staging directory limit in the registry. Resolve the disk space problem or increase the maximum staging file space. See Staging Directory is Full and Huge File Added to Replica Tree for more details.
  6. Check whether the source file was excluded from replication by FRS file and folder filters (search for "filter" in the Ntfrs_ds.txt file generated by FRSDiag or in Ultrasound see the Replica sets view in the Advanced tab under General.) Confirm that the file is not EFS encrypted, a reparse point, marked as temporary or excluded by a file or folder filter on the originating replica member (see article 296944, "HOW TO: Use File Replication Service File and Folder Filters in Windows 2000" at http://support.microsoft.com/?id=296944 and article 229928, "Design Decisions, Defaults and Behavior for FRS File and Folder Filters in DFS and SYSVOL Replica Sets" at http://support.microsoft.com/?id=229928.) If any of these conditions are true, FRS does not replicate the file or directory.

    By default, the following files and folders are excluded from FRS replication:

    • File names starting with a tilde (~) character.
    • Files with .bak or .tmp extensions.

    Regardless of the filters you set, FRS always excludes the following from replication:

    • NTFS mounted drives.
    • Files encrypted by using EFS.
    • Any reparse points except those associated with DFS namespaces. If a file has a reparse point used for Hierarchical Storage Management (HSM) or Single Instance Store (SIS), FRS replicates the underlying file but not the reparse point.
    • Files on which the temporary attribute has been set.
  7. Check whether the file on the originating server is locked on the target computer. See Sharing Violations for details on this process.
  8. As a last resort you might try a service restart on either the upstream and/or the downstream partner.


Files with Invalid Parent GUID

Ultrasound has detected that an inconsistency has been detected in the File Replication Service database. There is a file in the FRS database with no matching parent.

Resolution

To resolve this condition, obtain FRS hotfix 811370 from Microsoft Product Support Services. For more information, see article 811370, "Issues That Are Fixed in the Post-Service Pack 3 Release of Ntfrs.exe" at http://support.microsoft.com/?id=811370.


FRS Database Disk Space Less than 256 MB

Ultrasound has detected that the volume where the FRS database is stored has less than 256 MB free. If the volume fills up, replication will stop and FRS event ID 13511 will appear in the File Replication Service event log.

Resolution

Free up some disk space on the volume containing the FRS database. If this is not possible, move the database to a larger volume with more free space. For more information about how to move the database to a larger volume, see article 221093, "How to Relocate the NTFRS Jet Database and Log Files" at http://support.microsoft.com/?id=221093.


FRS Database Corrupt

FRS event ID 13513 occurs when replication has stopped on the specified replica member because the FRS database is corrupt. You must restore the FRS state by performing a nonauthoritative restore (D2 operation)

Resolution

Perform a nonauthoritative restore on the server. For more information, see Performing a Nonauthoritative Restore.


FRS in Mismatched Root, Replica, or Journal ID

FRS event ID 13539 occurs when FRS is in an error state or the replica root object ID or the USN journal ID or a file ID has been altered and does not match the values in the FRS database. Possible reasons are reimaging the volume or applications that change the file IDs without the knowledge of FRS.

Resolution

Restore the FRS state on the server by performing a nonauthoritative restore (D2 operation). For more information, see Performing a Nonauthoritative Restore.


FRS Service Stopped or Disabled

Ultrasound has detected that the FRS service on this server is either disabled or has not been running for 2 hours. The FRS service should be running at all times, except during upgrades and while preparing for a recovery process (D2 or D4).

Resolution

Investigate if there is a legitimate reason why FRS is not running on this server. To restart the service, type net start ntfrs at the command prompt.


FRS in Mismatched Volume Serial Number

FRS event ID 13571 occurs when the volume serial number from the FRS database does not match the volume serial number from the file system. Possible reasons are reimaging the volume.

Resolution

The state of FRS will need to be restored on the server by performing a nonauthoritative restore (D2 operation). For more information, see Performing a Nonauthoritative Restore.


Full Mesh Topology Not Recommended

A full mesh topology is not recommended for replica sets with more than five members.

Resolution

Adjust the topology using the guidelines in FRS Topologies.


Health Metric Definition Is Invalid

Ultrasound has detected that a health metric definition is invalid and cannot compute the health metric.

Resolution

Use the row filter editor in the Ultrasound console to correct this health metric


Huge File Added to Replica Tree

FRS event ID 13523 occurs when a file cannot be replicated because it is larger than the staging directory size. The file will not be replicated until you adjust the size of the staging directory.

Resolution

Use the procedure Adjust the Staging Directory Size Accordingly to increase the size of the staging directory on each member of the replica set so that the staging directory is larger than the file.


Inbound and Outbound Partners Are The Same

Ultrasound has detected that a connection has the same inbound and outbound partner.

Resolution

Adjust the inbound and outbound partners for this replica member. For DFS, use the Distributed File System snap-in to change the topology. For SYSVOL, use the Active Directory Sites and Services snap-in to change the topology.


KCC Encountered an Error While Adding a Connection Object

Ultrasound has detected that the KCC has encountered an error while adding a connection object. This alert is triggered when Event ID 1131 is logged in the Directory Service event log.

Resolution

Refer to the following articles for troubleshooting procedures:

  • 307593, "How to Troubleshoot Event ID 1311 Messages on a Windows 2000 Domain" at http://support.microsoft.com/?id=307593.
  • 214745, "Troubleshooting Event ID 1311: Knowledge Consistency Checker" at http://support.microsoft.com/?id=214745.
  • 819249, "You Receive an "Event ID 1311" Message Every 15 Minutes in the Directory" at http://support.microsoft.com/?id=819249.
  • 244368, "How to Optimize Active Directory Replication in a Large Network" at http://support.microsoft.com/?id=244368.


KCC Has Detected Problems with a Directory Partition

Ultrasound has detected that the KCC has detected problems with a directory partition. This alert is triggered when Event ID 1311 is logged in the Directory Service event log.

Resolution

Refer to the following articles for troubleshooting procedures:

  • 307593, "How to Troubleshoot Event ID 1311 Messages on a Windows 2000 Domain" at http://support.microsoft.com/?id=307593.
  • 214745, "Troubleshooting Event ID 1311: Knowledge Consistency Checker" at http://support.microsoft.com/?id=214745.
  • 819249, "You Receive an "Event ID 1311" Message Every 15 Minutes in the Directory" at http://support.microsoft.com/?id=819249.
  • 244368, "How to Optimize Active Directory Replication in a Large Network" at http://support.microsoft.com/?id=244368.


KCC Unable to Complete the Topology

Ultrasound has detected that the KCC was unable to complete the topology.

Resolution

Refer to article 249256, "HOW TO: Troubleshoot Intra-Site Replication Failures" at http://support.microsoft.com/?id=249256.


Long VVJoin

Ultrasound has detected that a VVJoin operation has been running for more than twelve hours. A VVJoin is the algorithm that FRS uses to synchronize two members of a replica set on a connection. VVJoins can take a long time to complete if the number of files in the replica set is very large, or if many VVJoin are being performed in parallel against the same upstream member.

Resolution

Look for the backlog indicators in Ultrasound and make sure that the backlog is decreasing. If the backlog is not decreasing, look for:

  • Network connectivity issues.
  • Backlogged files as reported in Ultrasound
  • Low disk space on the staging directory volume
  • Sharing violations
  • Schedule is too short (for DFS replica sets)
  • Multiple VVJoins occurring on the specified member


Missing Computer Reference

Ultrasound has detected that a computer object is missing from Active Directory.

Resolution

Refer to the following articles for more troubleshooting procedures:

  • 811219, "Recovering FRS Objects and Files Using System State Restores" at http://support.microsoft.com/?id=811219.
  • 312862, "Recovering Missing FRS Objects" at http://support.microsoft.com/?id=312862.
  • 296183, "Overview of Active Directory Objects That Are Used by FRS" at http://support.microsoft.com/?id=296183.


Missing Inbound or Outbound Connections

Ultrasound has detected that a SYSVOL member is missing an inbound or outbound connection. Unless you have a single master topology, you should always have at least one inbound connection if you have an outbound connection.

Resolution

Create an inbound or outbound connection to the SYSVOL member.


Missing NTDS Settings Reference

Ultrasound has detected that NTDS settings objects are missing from Active Directory.

Resolution

NTDS settings objects are described in TopChk Remarks and Verify the FRS Topology As Stored in Active Directory. For troubleshooting procedures, refer to the following articles:

  • 811219, "Recovering FRS Objects and Files Using System State Restores"
  • 312862, "Recovering Missing FRS Objects"
  • 296183, "FRS Attributes in Active Directory and Overview of Active Directory Objects That Are Used by FRS"


Missing Subscriber Reference

Ultrasound has detected that subscriber objects are missing from Active Directory.

Resolution

  • 811219, "Recovering FRS Objects and Files Using System State Restores"
  • 312862, "Recovering Missing FRS Objects"
  • 296183, "FRS Attributes in Active Directory and Overview of Active Directory Objects That Are Used by FRS"


Morphed Folders

FRS has detected that two or more identically named folders on different servers were added to the replica tree. FRS identifies the conflict during replication, and the receiving member protects the original copy of the folder and renames (morphs) the later inbound copy of the folder. The morphed folder names have a suffix of "_NTFRS_xxxxxxxx," where "xxxxxxxx" represents eight random hexadecimal digits.

There are two common causes of this condition:

  • A folder with the same name is created on multiple computers in the replica set before the folder has been able to replicate. This could be due to the administrator or application creating duplicate folders on multiple FRS members, perhaps in an effort by the administrator to make data consistent among all members with a manual copy.
  • You initiated an authoritative restore (D4) on one server and either:
    • Did not stop the service on all other members of the reinitialized replica set prior to restarting the NTFRS service following the authoritative restore
    • Did not set the D2 registry key on all other members of the reinitialized replica set prior to such a server replicating outbound changes to reinitialized members of the replica set.

Note

  • If you are running Ultrasound and see alerts for Morphed change order created, these could be false alerts. Use the steps in Resolution below to determine whether morphed folders exist.

Resolution

Locate the morphed folders in the replica tree. (To see which folders are morphed in Ultrasound, use the Files in replica set view on the Advanced tab. Sort the view by the MorphedDirectory column. You may need to collect data from the reference replica member first.)

To recover from this condition, the best practice is to rename the original folders and the morphed folders and wait for the renamed folders to propagate to all replica members. This will ensure that the folders have a common name throughout the replica set and have file GUIDS that match on all members. Note that you cannot just delete the unwanted folder and rename the other one as this could lead to even more naming conflicts.

After the rename has propagated, choose the one you want to keep, and rename it back to the original name, and then you can safely delete the remaining morphed folders.

For more information about cleaning up morphed folders, see article 328492, "Folder Name Is Changed to FolderName_NTFRS_<xxxxxxxx>" at http://support.microsoft.com/?id=328492.

Important

  • Before deleting any of the folders, it is a best practice to ensure you have a backup of the original (and complete) folder.


NETLOGON Is Not Shared Correctly

Ultrasound has detected that the NETLOGON file share is not shared correctly.

Resolution

Follow the troubleshooting steps in the following articles:

  • For Windows Server 2003, see article 327781, "How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server 2003 Domain Controllers" at http://support.microsoft.com/?id=327781.
  • For Windows 2000 Server, see article 257338, "Troubleshooting Missing SYSVOL and NETLOGON Shares on Windows 2000 Domain Controllers" at http://support.microsoft.com/?id=257338.


No Reference Replica Member

Ultrasound has detected one of the following conditions:

  • There is no reference replica member defined for this replica set. Ultrasound might have been unable to automatically choose a reference member for this replica set.
  • The reference replica member is defined but does not have a provider deployed on it.

Resolution

If no reference replica member is defined, follow the procedure To change the reference replica member. If a provider is not deployed on the reference replica member, see To install providers by using the console or To locally install providers by using the command line.


Not Enough Disk Space for the FRS Database

FRS event ID 13511 occurs when the FRS database is almost out of disk space.

Resolution

To correct this situation, free up some disk space on the volume containing the FRS database. If this is not possible, consider moving the database to a larger volume with more free space. For more information about how to move the database to a larger volume, see article 221093, "How to Relocate the NTFRS Jet Database and Log Files" at http://support.microsoft.com/?id=221093.


Not Enough Disk Space for FRS Logs

FRS event ID 13564 occurs when the volume holding the FRS debug logs is running out of disk space.

Resolution

Use the troubleshooting steps provided in event ID 13564.


Old 811370 FRS version

Microsoft discovered an issue with this version of NTFRS, resolved it, and re-released the 811370 hotfix under the same article name. Under certain conditions, Excel files are renamed to an eight-character hexadecimal name. There is no data loss associated with this issue. The problem occurs when a user opens an Excel (.xls) file within the replica tree, edits the file, and saves changes. FRS then renames the file to an eight-character hexadecimal name. The contents of the file remain unchanged but the name of the file is changed. This eight-character name is the name of the temp file that Excel creates during the editing process. This issue can occur if the server is using version 5.0.2195.6628 of NTFRS.

Resolution

Microsoft fixed this issue in the latest version of the 811370 QFE. The new version of NTFRS included in this fix is 5.0.2195.6743. You can request this hotfix by call placing a free call to Microsoft Product Support Services. This fix will also be part of Windows 2000 Service Pack 4 (SP4). If you are not experiencing this particular issue or if you do not keep any Excel (.xls) files in your replica tree, you can opt to wait until SP4 is released and install that version instead of using this hotfix.

For more information, see article 811370, "Issues That Are Fixed in the Post-Service Pack 3 Release of Ntfrs.exe" at http://support.microsoft.com/?id=811370.


Pre-811370 FRS Version

The specified replica member does not have the latest post-service pack releases installed. The installed FRS service is older than 811370 (5.0.2195.6628). Microsoft recommends that all replica members running Windows 2000 have Service Pack 3 (SP3) and the post-Service Pack 3 release of Ntfrs.exe, described in article 811370, or later service packs when they are available.

Resolution

Install the post-Service Pack release of Ntfrs.exe described in article 811370 at http://support.microsoft.com/?id=811370.


Pre-823230 FRS Version

Ultrasound has detected that the specified replica member does not have the most recent version of FRS running. A new release of FRS exists for servers running Windows Server 2003 and is described in article 823230, "Issues That Are Fixed in the Pre-Service Pack 1 Release of Ntfrs.exe" at http://support.microsoft.com/?id=823230.

Resolution

Install the version of FRS described in article 823230.


Pre-SP2 FRS Version

The specified replica member does not have the latest service pack and post-service pack releases installed. The installed FRS service is older than Windows 2000 Server SP2 (5.0.2195.2721). Microsoft recommends that all replica members running Windows 2000 have Service Pack 3 (SP3) and the post-Service Pack 3 release of Ntfrs.exe, described in article 815473, or Windows 2000 Server with SP4.

Resolution

Install SP3 and the post-Service Pack release of Ntfrs.exe described in article 815473 at http://support.microsoft.com/?id=815473.


Overloaded Server

Ultrasound has detected that this server is a member of more than 50 replica sets and might be overloaded.

Resolution

Check to see whether performance is degraded. If this server has not experienced performance degradation, you can disable this alert by doing this following:

  1. Right-click the alert, and then click Row Filter Editor.
  2. On the Alert tab, click to clear the Enable alert check box.
  3. If you also want to disable health metrics for this alert, on the Health Metrics tab, click to clear the Enable health metrics check box.


Provider Deployment Failure

Ultrasound has failed installing or uninstalling the WMI provider on the server to be monitored. Ultrasound will retry this operation after 30 minutes. For more information, go to the Advanced tab and select the Provider deployment status view. This view provides a list of successful and failed provider deployments and describes why the deployment failed.

Resolution

If this problem persists, you can manually install the provider by using the procedure To locally install providers by using the command line.


Provider Deployment Successful

Providers have successfully been installed or uninstalled on these servers.

Resolution

No resolution is required. If you do not want to see this alert after other successful provider deployments, do the following:

  1. Right-click the alert, and then click Row Filter Editor.
  2. On the Alert tab, click to clear the Enable alert check box.
  3. If you also want to disable health metrics for this alert, on the Health Metrics tab, click to clear the Enable health metrics check box.


Replica Set Deleted

Ultrasound has detected that a replica set has been deleted, but Active Directory indicates the replica set still exists. This could be an Active Directory replication issue. This alert might be caused when Ultrasound is not correctly receiving data from the reference domain controller.

Resolution

If you recently deleted a replica set, wait and see if the problem disappears in a few hours. If so, you can ignore this alert. If the replica set was not deleted, use FRSDiag and the FRS Troubleshooter to determine the problem. If you have failed Active Directory updates, this is probably because Ultrasound cannot get updated information from Active Directory. Therefore Ultrasound is working on stale data. If you don’t have failed Active Directory updates, then it’s probably an Active Directory replication issue.


Replica Set in Stopped State

Ultrasound has detected that FRS on the specified server is stopped. This might indicate that there was a USN journal wrap or that the server no longer has connections to any partners. This could also indicate that replication was purposefully disabled on a DFS replica set in the Distributed File System snap-in.

Resolution

Verify whether replication was purposefully disabled. In the case of a journal wrap, refer to the topic USN Journal Wrap. Otherwise, verify the state of FRS by running FRSDiag.


Replica Set Not Found

Ultrasound has detected that this replica set could not be found. This problem could be an Active Directory replication issue where this server should be a member of a replica set, but the server is not part of the replica set according to Active Directory.

Resolution

If you recently added this member and the problem disappears in a few hours, you can ignore this alert. If the member is not new, use the FRS Troubleshooter to troubleshoot this problem.


Replica Set Removed

Ultrasound has detected that one of the monitored replica sets no longer exists in Active Directory. This can occur when an administrator removes the replica set from the Distributed File System snap-in.

Resolution

Review existing Ultrasound data for the replica set to determine if it has historical value. If it does not, remove the replica set from monitoring. To do this, perform the following procedure:

  1. On the Tools menu, click Configure Ultrasound.
  2. Click Add/Remove Replica Sets, and then under Monitored FRS replica sets, click the replica set that you no longer want to monitor, and then click Remove.


Replica Tree Disk Space Is Almost Full

Ultrasound has detected that the volume where the replica tree is stored is running out of disk space. This alert occurs in the following conditions:

  • Replica root space is less than 32 MB
  • Replica root space is less than 512 MB

Resolution

Free up or add disk space to the volume storing the replica tree.


Row Filter Definition Is Invalid

Ultrasound has detected that the specified row filter or view has an invalid syntax.

Resolution

Refer to the topic Row Filter Syntax for information about row filter syntax and use the row filter editor in the Ultrasound console to correct the filter. It is also helpful to review the description of the view on which the filter is implemented. To do this, go to the Advanced tab, click the view whose filter you are modifying, and then click Details.


Schedule Not Recommended

Ultrasound has detected that the specified server is scheduled to replicate with more than 50 servers at the same time. This could cause performance problems and replication delays on this server.

Resolution

Review the sections FRS Topologies and Optimizing FRS Replication Schedules and adjust the schedule or topology.


Sharing Violations

A file held open on a replication target by some user or process can prevent FRS from replacing the file with updated content. In this case, the file will be staged on the target computer, and FRS will retry installation of the file. In the case of a folder, a sharing violation on a folder can block changes that reside in and under that locked folder.

If the file is locked on the computer from which the replication should originate, then FRS will be unable to read the file to generate the staging file, hence delaying replication. If the file is locked on the replication target, then FRS will be unable to update the file. In this case FRS continues to retry the update until it succeeds. The retry interval is 30 to 60 seconds.

Resolution

Identify and close the files causing the sharing violations by using the following methods.

Event Logs

Sharing violations are logged in the File Replication Service event log in the following operating systems:

  • Windows 2000 Server with the hotfix described in article 815473, "File Replication Service Does Not Log Errors on Sharing Violations" at http://support.microsoft.com/?id=815473.
  • Windows Server 2003 with the pre-Service Pack 1 release of Ntfrs.exe described in article 823230 at http://support.microsoft.com/?id=823230.

Sharing violations are logged as FRS event ID 13573. This event provides the file name and file GUID of the file held open and provides steps to resolve the sharing violation.

FRSDiag

FRSDiag shows which files have sharing violations. For more information about FRSDiag, see Troubleshooting FRS Using FRSDiag.

Ultrasound

Ultrasound shows two types of alerts for sharing violations: fetch sharing violations and install/retry sharing violations. These alerts display the file name, path, and file GUID of the file held open. The type of alert helps you determine which server has the file held open. For fetch sharing violations, investigate files held open on the outbound partner. For install/retry sharing violations, investigate files held open on the inbound partner.

You can also view current sharing violations by viewing the Advanced tab, expanding the list of views for the desired replica set, and then clicking any of the following views:

  • Current sharing violations (connection)
  • Current sharing violations (events)
  • Current sharing violations (member)
  • Current sharing violations (replica set)

Note

  • The sharing violation alerts and views do not provide file-related information if this information was not known to FRS on the reference replica member at the time of the last collection from the reference replica member.

If you want to reduce the number and frequency of sharing violation alerts that Ultrasound displays for servers running Windows Server 2003, install the latest versions of FRS described under Event Logs and use the instructions in article 816493, "How to Configure the File Replication Service to Allow Fewer Sharing Violations That Block Replication" at http://support.microsoft.com/?id=816493 to enable the Install Override feature.

Other Tools

The following methods can be used to confirm that sharing violations are the cause of the file not replicating.

Methods for Detecting Sharing Violations

Method | Details
Net file command | Available as part of the Windows 2000 and Windows Server 2003 operating systems, the net file command displays the names of all open shared files on a server and the number of file locks, if any, on each server. Net file also closes individual shared files and removes file locks. Although the net file command will tell you which users are holding the file open via the network, this tool does not report any files being held open by local processes.
Oh.exe | Available as part of the Windows 2000 Resource Kit tools, Oh.exe shows the handles of all open windows. The tool can also be used to show only information relating to a specific process, object type, or object name. This feature is useful for finding the process that has a file open when a sharing violation occurs.

For relevant KB articles, see the following:

  • 816493, "How to Configure the File Replication Service to Allow Fewer Sharing Violations That Block Replication."
  • 822300, "FRS Encounters ERROR_SHARING_VIOLATION Errors When It Tries to Replicate Data That Is Still in Use."
  • 815473, "File Replication Service Does Not Log Error on Sharing Violations."
  • 823230, "Issues That Are Resolved in the Pre-Service Pack 1 Release of Ntfrs.exe."


SMTP Error

Ultrasound could not send email notifications using this SMTP server. The SMTP server name might be incorrect, or the SMTP server might be set up incorrectly.

Resolution

See To configure Ultrasound options to specify the correct SMTP server.


Staging Directory is Full

On Windows 2000 before SP3, FRS event ID 13522 indicates that the FRS service has paused because the staging directory is full. Replication will resume if staging space becomes available or if the staging directory size is increased.

On Windows 2000 before SP3, you must clear the replication backlog. There are four common reasons why the staging directory fills up:

  • One or more downstream partners are not accepting changes. This could be a temporary condition due to the schedule being turned off and FRS waiting for it to open, or a permanent state because the service is turned off, or the downstream partner is in an error state.
  • The rate of change in files exceeds the rate at which FRS can process them.
  • There are no obvious changes made but the staging are is filling up anyway. This is typically the excessive replication issue described in Excessive Replication and High CPU Usage by Ntfrs.exe.
  • A parent directory for large number of changes is failing to replicate in so all changes underneath it are blocked.

Often, there is a combination problems; that is, changes cannot be replicated and the amount of change orders does not map to the amount of changes made.

Resolution

Increase the size of the staging directory by following this procedure:

To increase the size of the staging directory

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters.
  3. Right-click the Staging Space Limit in KB entry and click Modify.
  4. Click Decimal, type the desired size of the staging directory in kilobytes, and click OK.
  5. Close the registry editor.
  6. At the command prompt, type net stop ntfrs & net start ntfrs.

For more information about the staging directory, see Adjust the Staging Directory Size Accordingly.


Staging Directory Volume is Almost Full

The Staging Directory Volume is Almost Full

Ultrasound has detected that the volume where the staging directory is stored is running out of disk space. This alert occurs in the following conditions:

  • The staging directory volume has less than 512 MB of free disk space.
  • The staging directory volume has less than 32 MB of free disk space.

Resolution

Free up or add disk space to the volume storing the staging directory, or move the staging directory to another volume with more disk space. For more information, see article 291823, "How to Reset the File Replication Service Staging Folder to a Different Logical Drive" at http://support.microsoft.com/?id=291823.


Staging Files Regenerated

On Windows 2000 SP3 and later and Windows Server 2003, this alert occurs when staging files are regenerated in the staging directory. FRS deleted the files from the staging directory after the staging directory reached 90% of its size limit. FRS must regenerate the staging files before sending them to downstream partners, and this regeneration can negatively impact server performance.

This alert might also be generated after a server in a replica set has a D2 operation performed on it.

Resolution

If a D2 was not performed on a server in the replica set, increase the size of the staging directory to avoid the need for FRS to purge and regenerate staging files. For sizing guidelines, see Adjust the Staging Directory Size Accordingly.


SYSVOL is Not Shared Correctly

Ultrasound has detected that the SYSVOL file share is not shared correctly.

Resolution

Follow the troubleshooting steps in the following articles:

  • For Windows Server 2003, see article 327781, "How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server 2003 Domain Controllers" at http://support.microsoft.com/?id=327781.
  • For Windows 2000 Server, see article 257338, "Troubleshooting Missing SYSVOL and NETLOGON Shares on Windows 2000 Domain Controllers" at http://support.microsoft.com/?id=257338


SYSVOL is Not a Junction Point

Ultrasound has detected that the SYSVOL file share is not a junction point.

Resolution

To correct this problem, perform the following procedure.

  1. Use the Resource Kit tool Linkd.exe to recreate the SYSVOL junction point. For more information about creating junction points, see article 205524, "How to Create and Manipulate NTFS Junction Points" at http://support.microsoft.com/?id=205524.
  2. Restart FRS.

If this procedure does not fix the issue, contact Microsoft Product Support Services to discuss other methods of restoring the junction point. One possible procedure is a nonauthoritative restore, which will fix the junction point but will require the entire replica tree to be replicated to this server. You can prestage the data to avoid replicating data across the network.

For more information about SYSVOL junction points, see article 324175, "Best Practices for SYSVOL Maintenance" at http://support.microsoft.com/?id=324175.


Total Staging Directory Capacity is 90 Percent of Limit

Ultrasound has detected that the amount of staging directory space in use for all staging directories on the server is 90% or more of the staging directory limit set in the registry.

Resolution

Increase the size of the staging directory by following this procedure:

To increase the size of the staging directory

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters.
  3. Right-click the Staging Space Limit in KB entry and click Modify.
  4. Click Decimal, type the desired size of the staging directory in kilobytes, and click OK.
  5. Close the registry editor.
  6. At the command prompt, type net stop ntfrs & net start ntfrs.

For more information about the staging directory, see Adjust the Staging Directory Size Accordingly.


Troubleshooting FRS Events 13508 without FRS Event 13509

Event 13508 in the FRS log is a warning that the FRS service has been unable to complete the RPC connection to a specific replication partner. It indicates that FRS is having trouble enabling replication with that partner and will keep trying to establish the connection.

A single event ID 13508 does not mean anything is broken or not working; simply look for event ID 13509 to make sure that the problem was resolved. Based on the time between event IDs 13508 and 13509, you can determine if there is a real problem that needs to be addressed.

Note

  • If FRS is stopped after a 13508 interval, and then later started at a time when the communication issue has been resolved, no 13509 is entered in the FRS event log. Therefore, an event indicating that FRS has started, without a 13508 message, indicates replication connections are being made correctly

Because FRS servers gather their replication topology information from their closest Active Directory domain controller (itself on a domain controller that is also an FRS member), there is also an expected case where a replica partner in another site will not be aware of the replica set until the topology information has been replicated to domain controllers in that site. When the topology information finally reaches that distant domain controller, the FRS partner in that site will be able to participate in the replica set and lead to FRS event ID 13509. Note that intra-site Active Directory replication partners replicate every 5 minutes. Intersite replication only replicates when the schedule is open (shortest delay is 15 minutes). In addition, FRS polls the topology in the active directory at defined intervals — 5 minutes on domain controllers, and 1 hour on other member servers of a replica set. These delays and schedules (and especially in topologies with multiple hops) can delay propagation of the FRS replication topology

Resolution

  1. Examine the 13508 event in the File Replication Service event log to determine which computer that FRS has been unable to communicate with.
  2. Determine whether the remote computer is working properly, and verify that FRS is running on it. A good method to do this to execute the following command: NTFRSUTL VERSION <FQDN_of_remote_DC_name> from the computer logging the 13508 event.
  3. If this fails, check network connectivity by pinging the <FQDN_of_remote_DC_name> from the computer logging the 13508 event. If this fails, then troubleshoot the problem as a DNS or TCP/IP issue. If it succeeds, confirm the FRS service is started on the remote domain controller.
  4. Determine whether FRS has ever been able to communicate with the remote computer by looking for 13509 in the event log and reviewing recent change management to networking, firewalls, DNS configuration, and Active Directory infrastructure to see if there is a correlation.
  5. Determine whether there is anything between the two computers that is capable of blocking RPC traffic, such as a firewall or router.
  6. Confirm that Active Directory replication is working.


Ultrasound Cannot Resolve DNS Name of Server

Ultrasound cannot resolve the DNS name for the specified server.

Resolution

Verify that DNS is working correctly. Check general network connectivity and ensure there are no firewalls between the Ultrasound controller and the FRS servers.


Ultrasound Database is Almost Full

The current size of the Ultrasound database is greater than 80% of the MSDE 2-GB limit. If the database reaches 2 GB, the controller stops collecting and processing data. To determine the current size of the database, click the Advanced tab, and on the left side of the screen under General, click Ultrasound self diagnostic.

Resolution

Increase the frequency of database purging or decrease the number of days that data is retained by using To configure Ultrasound options. You can avoid the 2-GB MSDE limit by installing SQL Server 2000 with SP3.

If the database reaches the 2-GB limit, start the Ultrasound console, decrease the number of days that data is retained, and then restart the controller by restarting the Ultrasound Controller service in the Services snap-in.


Ultrasound Has Chosen a New Reference Domain Controller

Ultrasound has detected that the previous reference domain controller was demoted and is no longer a domain controller. Ultrasound has chosen a new domain controller to act as the reference domain controller.

Resolution

Verify that the new reference domain controller is acceptable or choose a new reference domain controller by following the procedure To change the reference domain controller


Ultrasound WMI Provider Could Not Read FRS Registry

The Ultrasound WMI provider was unable to read FRS configuration data from the registry.

Resolution

Check the registry on the affected FRS server computer to ensure that it is accessible (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Ntfrs).


Ultrasound WMI Provider Error (Internal)

An internal error occurred while reading data from the Ultrasound WMI provider.

Resolution

See Where to Get Help with Ultrasound for support information.


Ultrasound WMI Provider Problem

The Ultrasound controller has a problem contacting the WMI provider for the specified server. This problem could be caused the following issues:

  • The version of the Ultrasound provider on the server is not compatible with the controller.
  • The Ultrasound WMI provider is not installed correctly.
  • The Ultrasound WMI provider is disabled.

Resolution

If the problem persists, update the provider by using the procedures To update remote providers by using the console or To locally update providers by using the command line. If updating the provider does not fix the problem, see Where to Get Help with Ultrasound for support information.


Ultrasound WMI Provider Could Not Read File Replication Service Event Log

The Ultrasound WMI provider was unable to read events from the File Replication Service event log.

Resolution

Check the File Replication Service event log on the affected FRS server to verify that it is accessible.


USN Journal is Almost Full

Ultrasound has detected that changes are being made to the volume faster than FRS can detect them in the USN journal and log them in the FRS database. The recommended USN journal size is 128 MB per 100,000 files on the volume. If the USN journal is too small for the number of files on the volume, including those outside of the replica tree, a USN journal wrap error can occur, which causes replication to cease and requires a nonauthoritative restore to correct.

Default USN journal sizes are as follows:

  • Prior to Windows 2000 SP3, the default journal size is 32 MB and the maximum journal size is 128 MB.
  • In Windows 2000 SP3, the default journal size is 128 MB, and the maximum journal size is 10,000 MB.
  • In Windows 2000 SP3 with the post-Service Pack 3 release of Ntfrs.exe, described in article 815473, and Windows 2000 SP4, the default journal size is 512 MB.
  • In Windows Server 2003, the default journal size is 128 MB. After you install the pre-Service Pack 1 release of Ntfrs.exe as described in article 823230, the default journal size is 512 MB.

You can increase the size of the USN journal by modifying the registry, but keep in mind that after you increase it, you should not lower it again because this will cause a USN journal wrap.

Resolution

Increase the USN journal size to 128 MB per 100,000 files on the volume.

  1. In the Run dialog box, type regedit.exe, and click OK.
  2. Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters
  3. Right-click the Parameters subkey and choose New -> DWORD Value.
  4. Type NTFS journal size in MB as the new value name and press Enter.
  5. Right click NTFS journal size in MB and click Modify.
  6. Click Decimal, and then in the Value data field, type the size in megabytes that you would like to make the NTFS journal.
  7. Click OK, and then close the Registry Editor.
  8. At the command prompt, type net stop ntfrs & net start ntfrs

Caution

  • Do not edit the registry unless you have no alternative. The registry editor bypasses standard safeguards, allowing settings that can damage your system, or even require you to reinstall Windows. If you must edit the registry, back it up first and see the Registry Reference on the Microsoft Windows Server 2003 Resource Kit companion CD or at http://www.microsoft.com/reskit.


USN Journal Wrap

FRS event ID 13568 occurs when the specified replica member enters a journal wrap condition. The NTFS file system maintains a special log called the USN journal, which is a high-level description of all the changes to files and directories on an NTFS volume. FRS uses this mechanism to track changes to the file system in directories of interest, and queue those changes for replication to other computers.

The USN journal has defined size limits and will discard old log information on a first in, first out basis to maintain its correct size. If FRS processing falls behind the USN journal, and if USN journal information that FRS needed has been discarded, then FRS enters a journal wrap condition. FRS then needs to rebuild its current replication state with respect to the file system and other replication partners.

FRS can encounter USN journal wrap conditions in the following cases:

  • Many files are added at once to a replica tree while FRS is very busy, starting up, or not running.
  • On a server that is being used for authoritative restore, or as the primary for a new replica partner, there is a lot of file activity at the start of this process and this can consume USN journal records.
  • An NTFS volume needs to be processed with Chkdsk, and Chkdsk corrects the file system structure. In this case, NTFS creates a new USN journal for the volume or deletes the corrupt entries from the end of the journal.
  • The USN journal is deleted or reduced in size.
  • The FRS service is in an error state that prevents it from processing changes in the USN journal.

Resolution

If FRS is experiencing journal wrap errors on a particular server, it cannot replicate files until the condition has been cleared. To continue replication, you must stop FRS on that server and perform a nonauthoritative restore of the data so that the system can synchronize with its replication partners. Note the following about nonauthoritative restores:

  • Windows 2000 SP1 cannot perform this process automatically.
  • In Windows 2000 SP2, FRS performs this process automatically.
  • In Windows 2000 SP3 and later and Windows Server 2003, FRS does not perform this process automatically. The reason for this change was that it was typically being performed at times that were not planned by administrators. However, there is a registry setting available that allows FRS to perform the automatic nonauthoritative restore, just as in Windows 2000 SP2. However, Microsoft recommends leaving this as a manual process.

For more information about nonauthoritative restores, see Performing a Nonauthoritative Restore.

For information about sizing the USN journal to avoid journal wraps, see Adjust the USN Journal Size Accordingly.


View Definition Is Invalid

Ultrasound has detected an error when processing the view.

Resolution

Verify that the view matches SQL syntax and use the view editor in the Ultrasound console to correct this view.


View Definition Is Too Large

Ultrasound has detected that the number of characters used to define a view is limited to less than 24,000 characters.

Resolution

Use the view editor in the Ultrasound console to shorten this view.


WMI Access Denied

Ultrasound has detected that the controller does not have access to the WMI provider.

Resolution

Update the provider by using the procedures To update remote providers by using the console or To locally update providers by using the command line. If updating the provider does not fix the problem, see Where to Get Help with Ultrasound for support information.


WMI Service Is Disabled

Ultrasound has detected that the WMI service is disabled on the server. The WMI service must be enabled on an FRS server for the Ultrasound provider to work.

Resolution

Enable the Windows Management Instrumentation service on the server where the provider is deployed.


Recovering FRS Using Nonauthoritative and Authoritative Restores

The following sections describe how to perform nonauthoritative restores, also called D2 restores, and authoritative restores, also called D4 restores.

  • Performing a Nonauthoritative Restore
  • Performing an Authoritative Restore

Important

  • Perform these restores only after you discover FRS dependencies and you understand and resolve the root cause. If you do not resolve the issues that initially caused the replication problem, the members will typically revert back to the problematic situation as replication continues.


Performing a Nonauthoritative Restore

The nonauthoritative restore (D2) is used to bring the local FRS replica set back up to date with its partners. Nonauthoritative restores are often necessary to resolve the following issues:

  • Assertions in the FRS service
  • Corruption of the local FRS jet database
  • Journal wrap errors
  • FRS replication failures

The nonauthoritative restore process brings a replica member up to date by comparing all the local files in a replica set with the ones of the upstream partners. After you set the Burflag to D2 and restart the service, FRS will move all the data into a "pre-existing" directory and then start to compare the file IDs and the files’ MD5 checksums from the upstream partner with the local ones. If the file ID and the MD5 checksum match, FRS copies this file from the pre-existing directory into its original location. If they don’t match, then FRS copies the new file from the partner. Before Windows 2000 SP3, this work is performed with each of upstream partner at once; with Windows 2000 SP3 and later and Windows Server 2003, this process is performed sequentially — one partner at a time.

To perform a nonauthoritative restore, use the following procedure:

To perform a nonauthoritative restore

  1. At the command prompt, type net stop ntfrs.
  2. Click Start, click Run, type regedit, and then click OK.
  3. Locate the following subkey in the registry:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NtFrs\Parameters\ Backup/Restore\Process at Startup

  4. In the right pane, double-click BurFlags.
  5. In the Edit DWORD Value dialog box, type D2 and then click OK.
  6. Close Registry Editor.
  7. At the command prompt, type net start ntfrs.

When the FRS service restarts, the following actions occur:

  • The value for BurFlags registry key returns to 0.
  • Files in the reinitialized FRS folders are moved to a folder named NtFrs_PreExisting_See_EventLog.
  • The FRS database is rebuilt.
  • The member performs an initial join of the replica set from an upstream partner or from the computer that is specified in the Replica Set Parent registry key if a parent has been specified for SYSVOL replica sets.
  • The reinitialized computer performs a full replication of the affected replica sets when the relevant replication schedule begins.

The placement of files in the NtFrs_PreExisting_See_EventLog folder on reinitialized members is a safeguard in FRS designed to prevent accidental data loss. Any files destined for the replica that exist only in the local PreExisting folder and did not replicate in after the initial replication can then be copied to the appropriate folder. When outbound replication has occurred, delete files in the PreExisting folder to free up additional disk space.

For more information about performing a nonauthoritative restore, see article 290762, "FRS: Using the BurFlags Registry Key to Reinitialize File Replication Service Replica Sets" at http://support.microsoft.com/?id=290762.


Performing an Authoritative Restore

The D4 (authoritative restore) makes the replica tree on one member authoritative for all the members of this replica set. This is done by stamping a new unique file ID on every single file and folder. To get these new files replicated to all the partners, you must also initiate a nonauthoritative (D2) restore on every member of the replica set.

This D4 (authoritative restore) procedure should be performed only in consultation with your Microsoft Product Support provider. The D4 process is a last resort only used in selected situations. For more information about performing an authoritative restore, see article 290762, "FRS: Using the BurFlags Registry Key to Reinitialize File Replication Service Replica Sets" at http://support.microsoft.com/?id=290762.

Important

  • The FRS service must be stopped on all replica members of a replica set before you perform this D4 process. Failure to do so will cause morphed folders to appear on all replica partners. You can use the Sonar tool with the Settings column set to confirm that all members are stopped and have the correct D2 BURFLAGS set.


Appendix

The topics in this section are useful for advanced analysis and troubleshooting of FRS by using the FRS debug logs. This section also contains a comprehensive list of all FRS event log messages in the File Replication Service event log.

  • Setting the Debug Log File Size and Verbosity Level
  • FRS Data Collection Scripts
  • Analyzing FRS Status Reports and Logs
  • FRS Event Log Messages


Setting the Debug Log File Size and Verbosity Level

FRS debug logs are used for troubleshooting FRS. FRS creates these text-based log files in the %Systemroot%\Debug folder. For instructions on changing the debug log settings, see article 221112, "NT File Replication Service Log File Size and Verbosity Settings" at http://support.microsoft.com/?id=221112.


FRS Data Collection Scripts

The Windows Support Tools contain a number of scripts that you can run to collect data about FRS. The following sections briefly describe these scripts.

Health_Chk

Health_Chk collects a range of event, error log, Ntfrsutl status, and other information useful for monitoring or troubleshooting FRS. Note that Health_Chk has been replaced by FRSDiag.exe. For more information, see Troubleshooting FRS Using FRSDiag.

Topchk.cmd

TopChk processes the output generated when the NtfrsUtl tool is run with the ds parameter. Using the NtfrsUtl ds output as its data source, TopChk creates a report that summarizes the replication topology of one or more replica sets. This report includes the following information: topology, server reference table, server host table, and server inbound and outbound partners. For more information, see Topchk.cmd: DFS and SYSVOL Replication Topology Analysis Tool.

Connstat.cmd

Connstat processes the output generated when the NtfrsUtl tool is run with the sets parameter. Using the NtfrsUtl sets output as its data source, ConnStat creates a report that summarizes the status of the FRS replication connections to and from a specified computer. For more information, see Connstat.cmd: FRS Connection Status Report.

Iologsum.cmd

Iologsum processes the output generated when the NtfrsUtl tool is run with the inlog, outlog, or IDtable parameters.

Running NtfrsUtl inlog or NtfrsUtl outlog creates a summary inbound or outbound NtfrsUtl log report showing the pending change orders for the replica member. Running NtfrsUtl IDtable creates a report with version information for each file in the replica tree; this version information tracks the last change order accepted for that particular file or directory. Running IOLogSum then sorts and formats this NtfrsUtl output. For more information, see Iologsum.cmd: FRS Inbound and Outbound Logs Summary Report Tool.


Health_chk.cmd: FRS Health Check

This command-line tool retrieves FRS-related state information from the specified domain controller or server. It collects and provides administrators with in-depth data that can be used to monitor or troubleshoot an FRS system.

Health_Chk calls a series of tools for the output they produce. It can be run against a remote server, however, running the script directly on the server to be analyzed enables the tools to gather additional useful data not available when the script is run remotely.

  • For monitoring, Health_Chk should be run regularly against hub servers and a small selection of spoke servers. Building a simple script that invokes these health checks and collects them to a single server for analysis is very valuable - especially when scheduled to run on a daily basis.
  • For troubleshooting, Health_Chk should be run against the server that shows the symptoms of replication failures, but also, ideally, against all other related servers. It may not be possible to collect data for all servers in a large enterprise, but including direct replication partners or, if this is still too much, a good sample set of them, will provide more complete information about your topology.

Corresponding UI

There is no corresponding user interface for this tool.

Concepts

The File Replication Service (FRS) is a multi-master file replication service for NTFS that allows member machines to be interconnected in an arbitrary topology and uses last-writer-wins conflict resolution. It uses authenticated Remote Procedure Call (RPC) with encryption as the communication protocol between members, and can also support disconnected operation. FRS supports multiple replication sets per computer, each with potentially distinct topology.

System Requirements

  • Windows 2000 Server or Windows Server 2003
  • File Replication Service running on the target server(s)
  • Administrator priveleges on the account running Health_Chk.

Related Topics

  • Health_Chk Remarks
  • Health_Chk Syntax
  • Health_Chk Examples


Health_Chk Remarks

Logs and Output Files

Health_Chk retrieves FRS-related state information from the specified domain controller or server and stores this information in text files. These text files provide administrators with in-depth data that can be used to monitor or troubleshoot an FRS system.

Health_Chk can take quite a while to run. The script may collect huge amounts of data. This is especially true if you encounter problems with full staging areas, though staging area problems are not common on computers running Windows 2000 SP3 and later.

The text files are written to a specified directory using the name of the target machine.

Note

  • Organizations typically have well-established tools for collecting event logs from Windows systems, and these should also continue to be used. The event log data collected by Health_Chk contains only a subset of the full event information.

The following lists the specific text files that Health_Chk creates, with a description of the data they contain:

File | Description
ds_showconn.txt | The output from repadmin /showconn.
ds_showreps.txt | The output from repadmin /showreps.
evl_application.txt | Application event logs in text format.
evl_dns.txt | DNS server event logs in text format.
evl_ds.txt | Active Directory event logs in text format.
evl_ntfrs.txt | FRS event logs in text format.
evl_system.txt | System event logs in text format.
ntfrs_config.txt | Output from ntfrsutl configtable: Lists all replica sets.
ntfrs_ds.txt | The output from ntfrsutl ds: Displays the machine (FRS) topology definition in the local Domain Controller, from the perspective of the machine.
ntfrs_errscan.txt | A scan of error messages from the FRS debug logs.
ntfrs_inlog.txt | The output from ntfrsutl inlog: Shows inbound Change Orders (CO's).
ntfrs_machine.txt | A listing of free disk space on all drives.
ntfrs_outlog.txt | The output from ntfrsutl outlog: Shows outbound Change Orders (CO's).
Ntfrs_reg.txt | The registry data from HKLM\system\currentcontrolset\services\NtFrs and below.
Ntfrs_sets.txt | The output from ntfrsutl sets: A list of inbound and outbound connections and their status.
Ntfrs_sysvol.txt | The output from ‘dir /s *.*’ of the SYSVOL share.
Ntfrs_version.txt | The output from ntfrsutl version: Shows FRS version information.

Configuration Settings

Health_Chk does not modify any configuration settings either on the target computer or in the Active Directory. The script only reads configuration and status information.


Health_Chk Syntax

Health_chk uses the following syntax:

One Step health_chk.cmd result_dir [target_computer]

Parameters

result_dir
Created if it does not exist. No trailing backslash.
target_computer
Default is current computer. Valid computer names include the NetBIOS name with no leading slashes, or a full DNS name, xxx.yyy.zzz.com


Health_Chk Examples

In this example, the remote computer, DfsRepA, is part of a DFS replica set. It is not a domain controller, so the warnings about no SYSVOL are reasonable.

C:\tools>health_chk logdir DfsRepA

When given this command, Health_Chk runs a number of tools and scripts, generates text output files in the logdir folder, and then exits.

Please WAIT....

 NTFRSUTL checks are running ...

 Dumping FRS inbound and outbound logs ..

 Dumping FRS registry parameters ...

SYSVOL check is running

The network path was not found.

SMB/DFS check is running

repadmin /showreps frsdoga is running

repadmin /showconn frsdoga is running

dcdiag /v /s:frsdoga is running

 Scanning eventlogs ...

 Scanning FRS debug logs for error/warning info ...

 Done ...
After Health_Chk completes, twenty text log files are created. The data in these generated text files can be used to monitor and troubleshoot the FRS system. For a description of the contents of the text files, see the Logs and Output Files section on the Remarks page.

C:\tools>dir logdir\DfsRepA

 Volume in drive C is OS_VOL

 Volume Serial Number is 4D59-4447

 Directory of C:\Program Files\Support Tools\logdir\DfsRepA

07/21/2002  09:07 PM    <DIR>          .
07/21/2002  09:07 PM    <DIR>          ..
07/21/2002  09:06 PM               389 dcdiag.txt
07/21/2002  09:06 PM                92 ds_showconn.txt
07/21/2002  09:06 PM                91 ds_showreps.txt
07/21/2002  09:06 PM            16,216 evl_application.txt
07/21/2002  09:07 PM               161 evl_dns.txt
07/21/2002  09:07 PM               175 evl_ds.txt
07/21/2002  09:06 PM             3,998 evl_ntfrs.txt
07/21/2002  09:07 PM            18,398 evl_system.txt
07/21/2002  09:06 PM               766 net_view.txt
07/21/2002  09:06 PM            13,270 ntfrs_config.txt
07/21/2002  09:06 PM            21,562 ntfrs_ds.txt
07/21/2002  09:07 PM           731,201 ntfrs_errscan.txt
07/21/2002  09:06 PM                94 ntfrs_inlog.txt
07/21/2002  09:06 PM               337 ntfrs_machine.txt
07/21/2002  09:06 PM                95 ntfrs_outlog.txt
07/21/2002  09:06 PM             4,913 ntfrs_reg.txt
07/21/2002  09:06 PM            24,244 ntfrs_sets.txt
07/21/2002  09:06 PM                82 ntfrs_stage.txt
07/21/2002  09:06 PM                 0 ntfrs_sysvol.txt
07/21/2002  09:06 PM               465 ntfrs_version.txt
              20 File(s)        836,549 bytes
               2 Dir(s)   7,297,257,472 bytes free


Topchk.cmd: DFS and SYSVOL Replication Topology Analysis Tool

This command-line tool processes the output generated when the NtfrsUtl tool is run with the ds parameter. Using the NtfrsUtl ds output as its data source, TopChk creates a report that summarizes the replication topology of one or more replica sets. This report includes the following information:

  • Topology
  • Server Reference Table
  • Server Host Table
  • Server Inbound/Outbound partners
  • Summary

Corresponding UI

There is no corresponding user interface for this tool.

Concepts

The File Replication Service (FRS), first introduced in Windows 2000, is an essential element of the Microsoft Active Directory technologies. The Active Directory service uses FRS for SYSVOL replication. FRS can also be used with the Microsoft Distributed File System (DFS) technologies for automatic replication of content between alternating targets.

Running NtrfsUtl ds produces output listing the raw definition of the FRS replication topology stored in Active Directory. A replica set consists of two or more computers configured to replicate the contents of a file folder. The individual computers are referred to as replica members.

System Requirements

The following are the system requirements for TopChk:

  • Windows 2000 Server or Windows Server 2003
  • PERL 5.005 521e or 522e

Files Required

  • Topchk.cmd
  • Ntfrsutl.exe

Related Topics

  • TopChk Remarks
  • TopChk Syntax
  • TopChk Examples


TopChk Remarks

Before Using TopChk

If Perl is not installed on your system, if you are connected to the Internet, you can download a free version from the ActiveState Web site (http://go.microsoft.com/fwlink/?LinkID=9247).

TopChk Report

The TopChk report is used to verify that configuration information for FRS replica sets has been defined and stored correctly within the Active Directory. It is useful both as a regular health-check tool to confirm that replication is correctly configured, and as a troubleshooting tool for investigating possible causes of FRS replication issues.

Familiarity with FRS concepts and terminology is required in order to interpret TopChk reports.

The report consists of two main sections: The Subscriber List Summary followed by a Topology Analysis for each replica set. There is a section that summarizes the report following these sections.

Subscriber List Summary

This part of the report displays one row of information for each replica set that this member participates in. For example:

SUBSCRIBER: DOC|SALES      Root: f:\docs\sales                     Stage: e:\frs-stg
SUBSCRIBER: DOC|SUPPORT    Root: f:\docs\support                   Stage: e:\frs-stg

Each row shows the replica set name, the location on this computer where the replica set data exists, and the location of the FRS staging directory for this replica set.

Replica Set Topology Analysis

This part of the report contains a separate topology section for each replica set that this member participates in. The beginning of each section is indicated by the heading:

SET: <replica set name>

There may follow any of several possible sections, depending on the topology of the given replica set. The following sections can be included in the TopChk report:

SERVERS MISSING INBOUND CONNECTIONS

This part of the topology report appears if any FRS member servers have outbound replication partners but no inbound connection objects. There may be several reasons for this condition:

  • There are no connection objects under the NTDS Settings object for this server. This is an error.
  • The ServerReference Attribute for this server is null. This is an error.
  • The FRS member object may be missing. This is an error.
  • The server may be in a different domain, therefore no FRS member object can be identified for it. Domain controllers that hold a copy of the Global Catalog and are in a different domain are also listed this way. Such servers very likely do have inbound connections to replicate parts of the local domain naming context, but they are not members of the SYSVOL replica set. As a result, these servers are listed but they are not in an error state.

SERVERS MISSING OUTBOUND CONNECTIONS

This part of the topology report appears if any FRS member servers have inbound replication partners but no outbound connection objects.

MISSING NTDS SETTINGS REFERENCES

This part of the topology report appears if any FRS member objects have no Server Reference to an NTDS Settings object. For more information, see article 312862, "Recovering Missing FRS Objects and FRS Attributes in AD" at http://support.microsoft.com/?id=312862.

MEMBERS MISSING COMPUTER REFERENCE

This part of the topology report appears if any FRS member objects have no computer reference. For more information, see article 312862, "Recovering Missing FRS Objects and FRS Attributes in AD" at http://support.microsoft.com/?id=312862.

MEMBERS MISSING CONNECTION OBJECTS

This part of the topology report appears if any FRS member objects have no inbound connection objects. This is most commonly caused by an Administrator manually defining a replication topology, and not creating a connection object.

In this situation, an Administrator should check for NTDS connection objects. If none exists, the Administrator can create one by using Active Directory Sites & Services. For more information, see article 257338, "Troubleshooting Missing SYSVOL and NETLOGON Shares" at http://support.microsoft.com/?id=257338.

MEMBERS WITH SELF-REFERENCE CONNECTION OBJECTS

This part of the topology report appears if any FRS member objects have connection objects that refer back to themselves. This is most commonly caused by an Administrator manually defining a replication topology and mistakenly creating this condition. In this situation, the topology must be manually corrected.

MEMBERS WITH 168 HOUR CONNECTIONS

This part of the topology report appears if the FRS member object has a connection object with a 168-hour (that is, 24x7) replication schedule. This section intentionally lists connections that are continuously enabled; it does not indicate an error.

Summary Report Section

This section provides a statistical overview of the other reports. Each potential entry for this section is described here.

Member objects with no NTDS Settings reference

This entry represents the number of FRS objects that did not replicate their SYSVOL share and need to be further investigated.

Member objects with no connection objects

This entry represents the number of NTDS Settings objects with no connection objects, meaning that the servers shown here are not replicating inbound. Note that each SYSVOL member must have one inbound and one outbound connection object. DFS replica sets are permitted to have inbound connection objects with no outbound connection object when using hub and spoke or custom topologies.

Connection objects set to disabled

This entry represents the number of disabled replication connections. The Administrator should investigate these to see which connections have been disabled, and determine if this is intended. The section titled Connection Summary printed as part of the Summary Report can be used for this purpose.

Monday Schedule

This section appears if TopChk finds a connection object scheduled to replicate at least daily (Monday in this case is assumed to be representative of the schedule for the business week). This provides some insight into whether the schedules have been staggered across connections, which is a best practice for large configurations.

Number connections with per-week active replication hours

This section displays replication hours for each connection object. It begins with a number of rows showing (a) number of servers, then (b) the number of hours the schedule is enabled for those servers.

Note

  • This is approximate. Any hour in which the schedule byte is non-zero is counted as an active replication hour.

It is possible that objects used for intra-site communication may show high replication hours (in the range of 160 or over); however, in most cases this level of network use is not advised.

Connection Summary

This section lists each connection, along with its Monday schedule, replication hours, and enabled/disabled status in order to facilitate further investigation of schedules.

Note

  • A connection with a GUID, for example "cxtion: CC9BA0F4-3B12-4ADE-86BA-C405D86D4D59" indicates a connection that has been auto-generated by the Active Directory Knowledge Consistency Checker (KCC) or that has been created by using the DFS MMC snap-in utility. When the connection is associated with a name such as "cxtion: DC22," this indicates the connection was manually generated.

Server Inbound/Outbound Partner Report

The Server Inbound/Outbound Partner report (the final part of the Summary) lists the total number of inbound and outbound connections for a given server. Each server shows two entries: one with '<<' signifying an inbound connection list, and one with '>>' signifying an outbound connection list.

This part of the report is beneficial for investigating two aspects of the topology definition:

  • It shows whether SYSVOL members have the required minimum of one inbound and one outbound connection object.
  • It helps determine if the topology is reasonable with respect to the number of direct partners for each member, the amount of content being replicated (size * change), the link speed, and so forth.

An excessive number of connections to or from any one partner should be evaluated in order to see if the topology is balancing replication load appropriately.


TopChk Syntax

TopChk uses the following syntax:

One Step topchk DataFile

Parameters

DataFile
Specifies a file that was created by running NtfrsUtl with the ds parameter. This is a required parameter.

Note

  • The Perl interpreter should be installed and in the path for this command to function. See Before Using TopChk for details on obtaining Perl.

Sample Usage

topchk ntfrs_ds.txt > top.txt


TopChk Examples

Example 1: Generating the NtfrsUtl DS Data File

TopChk organizes the output of NtfrsUtl ds. Therefore, when running the topchk command you must specify a file that contains this output. Type the following at the command line to create the NtfrsUtl output file.

ntfrsutl ds >ntfrsout.txt

This command statement generates an output file called Ntfrsout.txt in the current directory.

Example 2: Using Topchk to Parse a Data File

This example parses the data file generated in Example 1: Generating the NtfrsUtl DS Data File and then displays the results to the screen. Type the following at the command line to save the TopChk report to a file in the current directory:

topchk.cmd ntfrsout.txt >topchkout.txt

This command runs the TopChk script against the Ntfrsout.txt file, which was generated by the NtfrsUtl ds command and saves the resulting output in a file named Topchkout.txt in the current directory.


Connstat.cmd: FRS Connection Status Report

This command-line tool processes the output generated when the NtfrsUtl tool is run with the sets parameter. Using the NtfrsUtl sets output as its data source, ConnStat creates a report that summarizes the status of the FRS replication connections to and from a specified computer.

Corresponding UI

There is no corresponding user interface for this tool.

Concepts

The File Replication Service (FRS), first introduced in Windows 2000, is an essential element of the Microsoft Active Directory technologies. The Active Directory service uses FRS for SYSVOL replication. FRS can also be used with the Microsoft Distributed File System (DFS) technologies for automatic replication of content between alternating targets. For more information on FRS and Active Directory replication please see the Windows Server 2003 Resource Kit.

A replica set consists of two or more computers configured to replicate the contents of a file folder. The individual computers are referred to as replica members.

FRS maintains logs of "change orders" that are either received from replication partners (the inbound log), or are to be sent to outbound replication partners (the outbound log).

The inbound log stores pending change orders to be processed. As entries are processed, FRS sends acknowledgment to the inbound partners. Data stored in the inbound log includes the globally unique identifier (GUID) of the change order, the file name, the object ID, the parent object ID, the version number, and the event time.

The outbound log stores pending change orders to be sent to outbound partners. Change orders remain in the outbound log until all outbound partners receive and acknowledge the change. Data stored in the outbound log is the same as that stored in the inbound log. Also in the outbound log is the leading (next change) and trailing (last acknowledged) index for each partner.

The outbound logs can become quite large, particularly when replica members are unavailable, links between replicas are slow, replication hours are restricted, or a large number of changes occur. For example, even when a member for a given replica set is unavailable, outbound log entries are maintained until the member becomes available. When the changes are then sent, the inbound partner sends all changes in log file order.

System Requirements

  • Windows 2000 Server or Windows Server 2003
  • Perl 5.005 521e or 522e

Files Required

  • Connstat.cmd
  • Ntfrsutl.exe

Related Topics

  • ConnStat Remarks
  • ConnStat Syntax
  • ConnStat Examples


ConnStat Remarks

Before Using ConnStat

If Perl is not installed on your system, if you are connected to the Internet, you can download a free version from the ActiveState Web site (http://go.microsoft.com/fwlink/?linkid=9247).

Using the ConnStat Report

The Connstat report consists of three main sections, the Report Header section, the Inbound Connections section, and the Outbound Connections section. Familiarity with FRS concepts and terminology is required in order to interpret these reports.

The Report Header Section

The first line of the Report Header lists the file specified in the ConnStat datafile parameter as well as the date that the file was last modified. The remainder of the Report Header gives information about the FRS service on the host where the input file was generated. The following table describes this information:
Report Element | Description | Comment
Replica | Displays the replica set name and GUID. | SYSVOL replica sets are always called "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)." Replica sets are uniquely identified by the GUID.
Member | Displays the member name. Individual computers in a replica set are referred to as replica members. | Note that for DFS replica sets this name is often a GUID.
ServiceState Shows the state of the FRS service. Valid return values are between 0 and 7.
  • 0 - Allocated: indicates an initializing state
  • 1 - Initializing: indicates an initializing state
  • 2 - Starting: indicates an initializing state
  • 3 - Active: the replica set is functioning
  • 4 - Stopped: the replica set is stopped with an error
  • 5 - Error: the replica set is stopped with an error
  • 6 - Jrnl_Wrap_Error: the replica set is stopped with an error
  • 7 - Replica_Deleted: indicates the replica set is marked for deletion
 The friendly name is displayed in parentheses next to the return value, for example 3 (ACTIVE).
OutLogSecNum | Displays the current outbound log sequence number.
OutLogCleanup | Displays the outbound log sequence number where the next clean-up will begin.
Delta | Represents the difference between OutLogSecNum and OutLogCleanup. This is a good estimation of the number of change orders currently present in the outbound log.
Config Flags | Displays state information about the replica set. Possible values are: Multimaster, Primary, or Seeding. | In multi-master replica sets, any domain controller accepts and replicates directory changes to any other domain controller. A primary flag means the host was a primary member when the replica set was created. The seeding flag, used for SYSVOL only, indicates the replica set is in the initial, seeding state.
Root Path | The root path used by the replica member.
Staging Path | The staging path used by the replica member.
File Filter | Lists file extensions that were excluded during replication, if any.
Directory Filter | Lists directories that were excluded during replication, if any.

The Inbound Connections Section

This section gives state information for all inbound connections for the given member machine. The following table lists columns and descriptions for this section:
Column Heading | Description | Comments
Partner | Displays the name of the immediate inbound partner for this replica member (member machine). | To participate in a valid FRS replica set, each member must have at least one inbound partner.
I/O | Specifies whether the connection is inbound (In) or outbound (Out). | This column appears in both the Inbound and Outbound report sections.
State | Specifies the current inbound connection state. | See the Connection State table in the Connection State Definitions remark for a list of valid Inbound connection states and their meanings.
Rev Represents operating system version information for the communication protocol of the inbound partner. Valid values include the following:
  • 0 - The connection is not currently joined.
  • 3 - The partner is running either Windows 2000 v. 2195, Windows 2000 v. 2195 SP1, or Windows 2000 v. 2195 SP2 with compression disabled.
  • 4 - The partner is running Windows 2000 v. 2195 SP2 with compression enabled.
  • 7 - The partner is running Windows 2000 v. 2195 SP3.
  • 8 - The partner is running a member of the Windows Server 2003.
LastJoinTime | Displays a timestamp showing when the replica member last successfully joined with this partner. | This data is available only for partners running Windows 2000 SP2 or later.

The Outbound Connections Section

This section gives state information for all outbound connections for the given member machine. The following table lists columns and descriptions for this section.

Note

  • The first several columns are identical to the Inbound connection report, with some additional information in the State column (see the Connection State table in the Connection State Definitions remark for more information). The columns starting with OLog State contains outbound log-related state information for the connection.
Column Heading | Description | Comments
Partner | Displays the name of the immediate outbound partner for this replica member.
I/O | Specifies whether the connection is inbound (In) or outbound (Out).
State Specifies the current outbound connection state.

See the Connection State table in the Connection State Definitions remark for a list of valid Outbound connection states and their meaning.

 In the outbound connections report, the appearance of the tag "-vv" means that this connection is performing a Version Vector-based join operation. Typically this is done when the outbound partner is first added to the replica set or if it is in the process of doing a non-authoritative restore (a D2). Once this initial sync is complete, the connection leaves the VVJoin state.
Rev Represents operating system version information for the outbound partner’s communication protocol.

See the Inbound Connection table in the Inbound Connections Section remark for definitions of the operating system versions.
LastJoinTime | Displays a timestamp showing when the replica member last successfully joined with this partner. | This data is available only for partners running Windows 2000 SP2 or later.
OLog State Specifies the state of outbound log processing for this connection.
  • OLP_UNJOINED - The partner is not joined.
  • OLP_ELIGIBLE - The partner can accept further change orders (Connection joined, and "change orders out" is less than the max limit).
  • OLP_STANDBY - The partner is ready to join the eligible list.
  • OLP_AT_QUOTA - The partner is at max quota for outstanding change orders.
  • OLP_INACTIVE - The partner is not accepting change orders.

Note

  • OLP_AT_QUOTA is a normal condition when FRS is actively replicating because the service always tries to keep eight change orders outstanding on each joined outbound connection.
Leadx | Specifies the index into the outbound log for the next change order to be processed for this connection.
Send Delta Specifies the difference between OutLogSeqNum (index of the most recent change order in the outlog) and the Leadx value for this connection. In other words, Send Delta specifies the number of change orders that remain to be processed by this connection.

A Send Delta of zero means that this connection is fully synchronized at this time. Unless there is very little activity in the replica set or the connection schedule is always on, you are unlikely to find connections with a Send Delta of zero.

Trailx | Specifies the index into the outbound log of the oldest unacknowledged outbound change order. Each change order sent to the partner for this connection must eventually be acknowledged.
Cleanup Delta | Specifies the difference between the Trailx value for this connection and OutLogCleanup index, the latter being the minimum Trailx value for all outbound connections. | Partners having connections with small cleanup deltas should be examined carefully to verify their operational status, since they are often the ones who are furthest behind in replicating content from this member.
LMT Specifies the difference between the values in the Leadx and Trailx columns. This maximum value is limited by the size of the AckVector, typically 128.

A value of 127 in this column indicates that the connection may require investigation.

 If the value of LMT isnbsp;127, wait for a while to see if the condition changes. If it does not, first try to delete and recreate the related connection object in the directory service. If the condition persists, then you may need to run a non-authoritative restore on the outbound partner.
COs Out | Specifies the number of active unacknowledged change orders pending at the partner. By default, FRS allows up to 8 change orders to be outstanding at a time on each outbound connection. | This limit is controlled by a registry parameter, which you may want to increase in an environment with high latency communication links. Note that FRS on the partner will try to initiate fetch requests for staging files for all eight change orders concurrently, so increasing this registry parameter will increase the load on the upstream member. See the Microsoft Windows Server 2003 Resource Kit (http://go.microsoft.com/fwlink/?linkid=4546) for more details.
Last VVJoin | Specifies the last date and time that a Version Vector-based join was done on the connection.

Connection State Definitions

The following table lists Connection State values and their meaning:
Connection State | Description
Init | Specifies that the connection is newly allocated.
Unjoined | Specifies that the connection is not yet joined to a partner.
Start | Specifies that the inbound partner has requested a join (StartJoin).
Starting | Specifies that the join has started.
Scanning | Specifies that FRS is scanning the inbound log for the replica set.
SendJoin | Specifies that the scan was completed, and the join request has been forwarded to the partner.
WaitJoin | Specifies that FRS is waiting for a reply from the partner.
Joined | Specifies that the join request succeeded.
Unjoining | Specifies that the join request failed after a series of retries.
Deleted | Specifies that the connection has been deleted.


ConnStat Syntax

ConnStat uses the following syntax:

One Step connstat [-sort=Option [-sort=Option]...] DataFile

Parameters

-sort=Option
Specifies how the NtfrsUtl sets output is sorted. The following values are valid for the -sort parameter:
Option | Description
send | Sorts outbound connections in ascending order by the Send Delta column of the ConnStat report.
name | Sorts outbound connections in ascending order by the Server Name column of the ConnStat report.
clean | Sorts outbound connections in ascending order by the Cleanup Delta column of the ConnStat report.
lmt | Sorts outbound connections in ascending order by the LMT (Leading minus Trailing) column of the ConnStat report.
lastjointime | Sorts outbound connections in ascending order by the LastJoinTime column of the ConnStat report.
lastvvjoin | Sorts outbound connections in ascending order by the Last VVJoin column of the ConnStat report.
DataFile
Specifies the file that was generated by running NtfrsUtl with the sets parameter.

Sample Usage

connstat ntfrs_sets.txt
connstat sort=lastvvjoin ntfrs_sets.txt


ConnStat Examples

Example 1: Generating the NTFRS SETS Output

ConnStat organizes the output of the NtfrsUtl tool, specifically the output generated when NtfrsUtl is run with the sets parameter. This example creates output that will be used by ConnStat. Type the following at the command line:

ntfrsutl sets >ntfrs_sets.txt

This command generates an output file called Ntfrs_sets.txt in the current directory. The command must be run on a computer that is a replica set member.

Example 2: Using ConnStat to Parse an NtfrsUtl sets Datafile

This example parses the data file generated in Example 1: Generating the NTFRS SETS Output and then displays the results to the screen. Type the following at the command line to save the ConnStat report to a file in the current directory:

connstat.cmd ntfrs_sets.txt >connstat_out.txt

This command runs the ConnStat script against the Ntfrs_sets.txt file, which was generated by the NtfrsUtl sets command and saves the resulting output in a file named Connstat_out.txt in the current directory.


Iologsum.cmd: FRS Inbound and Outbound Logs Summary Report Tool

This command-line tool processes the output generated when the NtfrsUtl tool is run with the inlog, outlog, or idtable parameters.

Running NtfrsUtl inlog or NtfrsUtl outlog creates a summary inbound or outbound NtfrsUtl log report showing the pending change orders for the replica member. Running NtfrsUtl idtable creates a report with version information for each file in the replica tree; this version information tracks the last change order accepted for that particular file or directory. Running IOLogSum then sorts and formats this NtfrsUtl output.

Corresponding UI

There is no corresponding user interface for this tool.

Concepts

The File Replication Service (FRS), first introduced in Windows 2000, is an essential element of the Microsoft Active Directory technologies. The Active Directory service uses FRS for SYSVOL replication. FRS can also be used with the Microsoft Distributed File System (DFS) technologies for automatic replication of content between alternating targets. For more information on FRS and Active Directory replication please see the Windows Server 2003 Resource Kit.

A replica set consists of two or more computers configured to replicate the contents of a file folder. The individual computers are referred to as replica members.

FRS maintains logs of change orders that are either received from replication partners (the inbound log), or are to be sent to outbound replication partners (the outbound log).

The inbound log stores pending change orders to be processed. As entries are processed, FRS sends acknowledgment to the inbound partners. Data stored in the inbound log includes the change order's globally unique identifier (GUID), file name, object ID, parent object ID, version number, and event time.

The outbound log stores pending change orders to be sent to outbound partners. Change orders remain in the outbound log until all outbound partners receive and acknowledge the change. Data stored in the outbound log is the same as that stored in the inbound log. Also in the outbound log is the leading (next change) and trailing (last acknowledged) index for each partner.

The outbound logs can become quite large, particularly when replica members are unavailable, links between replicas are slow, replication hours are restricted, or a large number of changes occur. For example, even when a member for a given replica set is unavailable, outbound log entries are maintained until the member becomes available. When the changes are then sent, the inbound partner sends all changes in log file order.

System Requirements

  • Windows 2000 Server or Windows Server 2003
  • Perl 5.005 521e or 522e

Files Required

  • Iologsum.cmd
  • Ntfrsutl.exe

Related Topics

  • IOLogSum Remarks
  • IOLogSum Syntax


IOLogSum Remarks

Before Using IOLogSum

If Perl is not installed on your system, if you are connected to the Internet, you can download a free version from the ActiveState Web site (http://go.microsoft.com/fwlink/?linkid=9247).

IOLogSum Report

When used with NtfrsUtl inlog and NtfrsUtl outlog, the IOLogSum report can be used to determine which files are involved in FRS replication, and what has caused them to replicate. When used with NtfrsUtl idtable, IOLogSum can generate a report describing the state of all files and folders known to an FRS member.

Familiarity with FRS concepts and terminology is required in order to interpret IOLogSum reports.

Each IOLogSum report starts by identifying the name of the file that it processed, and the sort options used to generate the report. Then, for every replica set in which the computer participates, the report lists one row per change order.

The following table lists and describes each column in the report.

Column Heading | Description
SeqNumber | Specifies the unique sequence number for the change order. For NtfrsUtl idtable reports, this is always reported as 00000000.
EventTime | Specifies the USN Journal Entry Timestamp; in other words, the time the file was changed on the originating machine.
FileVersNum | Specifies the file version number, incremented each time the file is closed.
FileUsn | Specifies the USN that the file must match on the Fetch request.
FileSize | Specifies the file size (in KB).
FileGuid | Specifies the object ID of the file or directory. For brevity, only the last 8 digits are printed.
OriginatorGuid | Specifies the GUID of the originating member. For brevity, only the last 8 digits are printed. The originator GUID can be mapped back to the server name by searching the NtfrsUtl sets output.
Cxtion | Specifies the GUID of the remote connection – for brevity, only the last 8 digits are printed
FileName | Specifies the Unicode file name.
Flags Specifies a valid Change Order Flag. Valid flags are shown in the following list.
  • Abort - Set when CO is being aborted.
  • VVAct - Set when VV activate request is made.
  • Content - Valid content command.
  • Locn - Valid location command.
  • LclCo - CO is locally generated.
  • Retry - CO needs to retry.
  • InstallInc - Local install not completed.
  • Refresh - CO is an upstream-originated file refresh request.
  • OofOrd - Do not check/update version vector.
  • NewFile - If CO fails, delete IDTable entry.
  • DirectedCo - This CO is directed to a single connection.
  • DemandRef - CO is a downstream demand for refresh.
  • VVjoinToOri - CO is from vvjoin to originator.
  • MorphGen - CO generated as part of name morph resolution.
  • MoveinGen - This CO was generated as part of a sub-dir MOVEIN.
  • OidReset - This CO reset the Object ID (OID) back to the FRS defined value.
  • CmpresStage - The stage file for this CO is compressed.
  • MD5 - The MD5 hash of the file is included.
  • DELETED - The file has been deleted. On idtable reports, this indicates that the record is a tombstone, ensuring the file remains deleted throughout the replica set.
LocationCmd Specifies the Change Order Location command. Valid Location Commands are shown in the following list.
  • CREATE - Create a new file or directory.
  • DELETE - Delete a file or directory.
  • MOVEIN - Rename a file or directory from a point outside the Replica Tree, into a Replica Tree on the same volume.
  • MOVEOUT - Rename a file or dir from a point inside the Replica Tree to a point outside the Replica Tree on the same volume.
  • MOVERS - Rename a file or directory from one Replica Tree to another Replica Tree on the same volume (currently unsupported).
  • MOVEDIR - Rename a file or directory from one directory to another in the same Replica Tree.
ContentCmd Specifies the Change Order content changes from the USN reason. Flags set in the NTFS change log describing modifications to the file.
  • Close - Change log close record.
  • Create - File or directory was created.
  • Delete - File or directory was deleted.
  • RenNew - File or directory was renamed.
  • DatOvrWrt - Main file data stream was overwritten.
  • DatExt - Main file data stream was extended.
  • DatTrunc - Main file data stream was truncated.
  • Info - Basic info change (attrib, last write time, etc.).
  • Oid - Object ID change.
  • StreamNam - Alternate data stream name change.
  • StrmOvrWrt - Alternate data stream was overwritten.
  • StrmExt - Alternate data stream was extended.
  • StrmTrunc - Alternate data stream was truncated.
  • EAChg - Extended file attribute was changed.
  • Security - File access permissions changed.
  • IndexableChg - File change requires re-indexing.
  • HLink - Hard link change.
  • CompressChg - File compression attribute changed.
  • EncryptChg - File encryption changed.
  • Reparse - Reparse point changed.

By default, the records are sorted by sequence number, but it is possible to sort by other columns.

  • Sorting by File GUID makes it easy to detect repetitive changes to single files and the times at which they occurred (for example, once per hour).
  • Sorting by Event Time is useful for identifying older change orders that may indicate a particular partner has been unavailable for a while. The IOLogSum report, coupled with the report that ConnStat generates, can tell you what change orders are pending for the delinquent partner.


IOLogSum Syntax

IOLogSum uses the following syntax:

One Step iologsum [-sort=Option] DataFile

Parameters

-sort=Option
Specifies a sort order for the IOLogSum report. The following table lists are valid values for Option:
Option | Description
seqnum | Sorts output by sequence number. This is the default sort order.
version | Sorts output by file version number.
filename | Sorts output by file name.
size | Sorts output by file size.
fileguid | Sorts output by file GUID.
origguid | Sorts output by originator GUID.
cxtion | Sorts output by cxtion name/GUID.
eventtime | Sorts output by event time.
DataFile
Specifies a file that was created by running NtfrsUtl with the inlog, outlog or idtable parameters. This is a required parameter.

Note

  • The Perl interpreter should be installed and in the path for this command to function. See Before Using IOLogSum for details on obtaining Perl.

Sample Usage

iologsum ntfrs_inlog.txt > in.txt
iologsum –sort=fileguid ntfrs_outlog.txt > out.txt


Analyzing FRS Status Reports and Logs

This section describes how to analyze the data that has been collected in the FRS logs.

  • Verify Active Directory Replication
  • Verify the FRS Topology As Stored in Active Directory


Verify Active Directory Replication

FRS replication depends on Active Directory replication. The first step is always to ensure you have correct monitoring processes for Active Directory replication. The simplest way to check for any suspicious Active Directory issues is to execute

FINDSTR "FAIL" ds_showreps.txt

If any issues are found, see article 229896, "Using Repadmin.exe to Troubleshoot Active Directory Replication" at http://support.microsoft.com/?id=229896.


Verify the FRS Topology As Stored in Active Directory

Each FRS server reads the replica set topology information from the closest available domain controller, not from some specified master DC. If the topology information is changing, and if there are delays in Active Directory replication due to backlog or schedule, then it is possible that different FRS servers will temporarily be out of sync.

Therefore, the first task in topology analysis is to decide the locations from which to check and compare the topology. This will typically include an FRS server from the main sites from which topologies are updated, along with a handful of branch sites, or any sites experiencing replication or join issues.

For each FRS server you choose to inspect topology from, run the Ntfrsutl DS command to obtain the raw topology information, then run the Topchk tool as described in Topchk.cmd: DFS and SYSVOL Replication Topology Analysis Tool.

Then, for each server from which to compare topology, follow the procedures described in this section.

Detecting a Missing NTDS Settings Reference

One possible reason for the differences could be that NTDS settings objects are missing in Active Directory.

The ServerReference attribute on the FRS member object of a SYSVOL replica set points to the Distinguished Name (DN) of that member’s NTDS Settings object. If the NTDS Settings object was deleted, the Server Reference attribute is not rebuilt. This means that either (i) the NTDS Settings object is missing or (ii) the link to it in the ServerReference attribute does not exist.

In the top.txt file this situation is reported in the following manner (only on the servers where the ntds settings object is missing):

M I S S I N G   N T D S   S E T T I N G S   R E F E R E N C E S
The following FRS Member objects have no Server Reference to an NTDS Settings Object
XYZA0314S01
XYZA0699S01
XYZA0281S01

Repairing a Missing NTDS Settings Reference

To recover from these problems you have to know if the NTDS Settings object was removed intentionally (so the server should have been demoted) or by accident. If it was intentional, remove the member object (DN=ERNI-VMDC1,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=erni,DC=com) with ADSIEDIT or LDP. Also remove the whole server object (CN=ERNI-VMDC1 ,CN=Servers, CN=Site1, CN=Sites, CN=Configuration, DC=erni, DC=com). NTDSUTL should do this job, if not try to delete is in the sites and services MMC or use ADIEDIT or LDP again.

Important

  • This process involves directly editing the FRS configuration objects in the Active directory – and must be performed with great care.

You’ll find a detailed description on how to recover missing or corrupted FRS Active Directory objects in the following articles:

  • 312862, "Recovering Missing FRS Objects and FRS Attributes in Active Directory" at http://support.microsoft.com/?id=312862.
  • 216498, "HOW TO: Remove Data in Active Directory After an Unsuccessful Domain Controller Demotion" at http://support.microsoft.com/?id=216498.

Investigating and Repairing Connection Balancing and Schedules

In this section, we look for 4 specific issues:

  • For SYSVOL replica sets, ensure the requirement that every inbound connection is matched by an outbound connection is satisfied.
  • Ensure that no single domain controller is overloaded with an excessive number of inbound or outbound connections.
  • Ensure that connections are ideally balanced among Domain Controllers in the hub site.
  • Ensure that schedules are reasonable for the number of partners, speed of network links and amount of data being replicated.

Here is an example from a top.txt file that shows a case where hubs are not well balanced – DC08CCAN has significantly more partners than the others:

Servers referenced from cxtions (From List)
XYZ\DC06CCAN               47  63
XYZ\DC07CCAZ               78  78
XYZ\DC08CCAN              152 162

To remedy such situations, you can manually rebuild the topology or use the MKDSX script (in both cases refer to the Windows 2000 Branch Office Deployment Guide available at http://go.microsoft.com/fwlink/?LinkId=16718. You can also use this script to set an appropriate schedule.

The Topchk report includes a section that shows a schedule that has been set unnecessarily high (actually continuous since 24x7 = 168 hours). For a larger site with > 100 Domain controllers in as many Active Directory sites it would be advisable to correct this.

M E M B E R S   W I T H    1 6 8   H O U R   C O N N E C T I O N S
The following FRS Member objects have connection objects with 168 hour replication schedules
Member:   DC10CCAN   cxtion:   03A1E67D-FCD3-4E6F-A17B-A81149A19206   host: CCA\DC10CCAN
Member:   DC10CCAN   cxtion:   40A91E15-9572-4B18-8B74-BF9B06DDD3C0   host: CCA\DC10CCAN
Member:   DC10CCAN   cxtion:   40EAAD73-511F-4D67-8555-EC69F2232370   host: CCA\DC10CCAN

The schedule as reported in the Topchk report represents every hour of Monday – only Monday is shown in order to reduce the size of the report, by making the assumption that the Monday schedule is representative of the normal daily schedule:

RepHrs: 168  Sched: 111111111111111111111111

In this example FRS replicates once per hour in 168 hours of the week (24*7=168). Every single number of the above represents one hour of the day as a decimal 4 bit value. Each single bit represents 15 minutes of this hour. So if we have "1" in decimal, then one bit is set in binary (0001) and we replicate once per hour.

If the decimal value is 5 (0101 in binary) we replicate twice per hour, for example:

RepHrs: 168  Sched: 555555555555555555555555

Finally if it is F (1111) we replicate 4 times per hour:

RepHrs: 168  Sched: FFFFFFFFFFFFFFFFFFFFFFFF

A connection can be disabled (enabled: FALSE) or the schedule could be all "0" or "none". If the connection is disabled, the FRS service does not use it and therefore does not replicate; this is true for both DFS/FRS and SYSVOL/FRS replication. If the schedule has 24 zeros, FRS also does not replicate. However, if it is set to "(none)" FRS will replicate 4 times per hour, because this is the default schedule on replica set objects.

Important

  • Domain controllers that hold a copy of the Global Catalog and are in a different domain are listed under "server missing inbound connections." These servers most likely have inbound connections to replicate parts of the local domain naming context, but they are not members of the SYSVOL replica set. As a result these servers are listed but they are not in an error state.

Every Member Must Have at Least One Inbound Connection

Every member of a replica set has to have at least one inbound connection. This is always true for SYSVOL and therefore Active Directory replication as otherwise no new users or even password changes could replicate. A possible exception could be a custom DFS topology where changes are only made on one of the participating servers.

In the case of servers missing inbound connections, the following warning is shown in the top.txt report:

S E R V E R S   M I S S I N G   I N B O U N D   C O N N E C T I O N S

The following FRS Member servers have outbound replication partners but no inbound connection objects. There could be several reasons for this:
1. There are no connection objects under the NTDS Settings object for this server.  This is an error.
2. The ServerReference Attribute for this server is null. This is an error.
3. This server could be in a different domain so there will be no FRS member object for it.
4. The FRS member object may be missing.  This is an error.
XYZ\DCFRCAN
XYZ\DCFRCAZ
XYZ\NLDC001

Members Missing Computer Reference

S E R V E R S   M I S S I N G   C O M P U T E R   R E F E R E N C E

This part of the topology report appears if any FRS member objects have no computer reference. For more information, refer to article 312862, "Recovering Missing FRS Objects and FRS Attributes in Active Directory" at http://support.microsoft.com/?id=312862.

Members Missing Connection Objects

S E R V E R S   M I S S I N G   C O N N E C T I O N   O B J E C T S

This part of the topology report appears if any FRS member objects have no inbound connection objects. This is most commonly caused by an administrator manually defining a replication topology, and not creating a connection object.

In this situation, check for NTDS connection objects. If none exists, create one by using the Active Directory Sites and Services snap-in. For more information, refer to article 257338, "Troubleshooting Missing SYSVOL and NETLOGON Shares on Windows 2000 Domain Controllers" at http://support.microsoft.com/?id=257338 or article 327781, "How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server 2003 Domain Controllers" at http://support.microsoft.com/?id=327781.

Members With Self-Reference Connection Objects

This part of the topology report appears if any FRS member objects have connection objects that refer back to them. This is most commonly caused by an administrator manually defining a replication topology and mistakenly creating this condition. In this situation, the topology must be manually corrected by deleting the object that points back to the server.


FRS Event-Log Messages

This section contains a list of all event messages that FRS logs in the File Replication Service event log.


13501

File Replication Service is starting


13502

The File Replication Service is stopping


13503

The File Replication Service has stopped


13504

The File Replication Service stopped without cleaning up


13505

The File Replication Service has stopped after taking an assertion failure.


13506

The File Replication Service failed a consistency check (%3) in "%1" at line%2. The File Replication Service will restart automatically at a later time. If this problem persists a subsequent entry in this event log describes the recovery procedure. For more information about the automatic restart right-click My Computer and then click on Manage, System Tools, Services, File Replication Service, and Recovery.


13507

The File Replication Service cannot start replica set %1 on computer %2 for directory %3 because the type of volume %4 is not NTFS 5.0 or later. The volume’s type can be found by typing "chkdsk %4". The volume can be upgraded to NTFS 5.0 or later by typing "chkntfs /E %4".


13508

The File Replication Service is having trouble enabling replication from %1to %2 for %3 using the DNS name %4. FRS will keep retrying. Following are some of the reasons you would see this warning.[1] FRS can not correctly resolve the DNS name %4 from this computer.[2] FRS is not running on %4.[3] The topology information in the Active Directory for this replica has not yet replicated to all the Domain Controllers. This event log message will appear once for each connection. After the problem is fixed you will see another event log message that indicates that the connection has been established


13509

The File Replication Service has enabled replication from %1 to%2 for %3 after repeated retries.


13510

The File Replication Service on the computer %1 cannot communicate with the File Replication Service on the computer %2. Verify that the computer %2 is up and running. Verify that the File Replication Service is running on %2 by typing "net start ntfrs" on %2. Verify that the network is functioning between %1 and %2 by typing "ping %1" on %2 and "ping %2" on %1. If the pings succeed then retry the failed operation. If the pings fail then there may be problems with the DNS server. The DNS server is responsible for mapping computer names to IP addresses. The commands "ipconfig" and "nslookup" help diagnose problems with the DNS server. Typing "ipconfig /all" will list the computer’s IP address and the IP address of the computer’s DNS servers. Type "ping DNS server’s IP address" to verify that a DNS server is available. The DNS mapping for %2 or %1 can be verified by typing "nslookup" and then typing"%2" and then "%1" on %1 and %2. Be sure to check out the DNS server on both%1 and %2; a DNS problem on either computer will prevent proper communication. Some network problems between %1 and %2 can be cleared up by flushing the DNSResolver Cache. Type "ipconfig /flushdns". Some network problems between %1and %2 can be cleared up by renewing the IP address. Type "ipconfig /release" followed by "ipconfig /renew". Some network problems between %1 and %2 can be cleared up by resetting the computer’s DNS entry. Type "net stop NetLogon" followed by "net startNetLogon". Some problems between %1 and %2 can be cleared up by restarting the File Replication Service. Type "net stop ntfrs" followed by "net startntfrs". Some problems between %1 and %2 can be cleared up by restarting the computers %1 and %2 after closing running applications, especially dcpromo. Click Start, Shutdown, select Restart, and click on OK. Other network and computer problems are beyond the scope of this event log message.


13511

The File Replication Service is stopping on computer %1 because there is no free space on the volume containing %2. The available space on the volume can be found by typing "dir %2". Once free space is made available on the volume containing %2, the File Replication Service can be restarted immediately by typing "net start ntfrs". Otherwise, the File Replication Service will restart automatically at a later time. For more information about the automatic restart right click on My Computer and then click on Manage, System Tools, Services, File Replication Service, and Recovery. For more information about managing space on a volume type "copy /?", "rename/?", "del /?", "rmdir /?", and "dir /?".


13512

The File Replication Service has detected an enabled disk write cache on the drive containing the directory %2 on the computer %1. The File Replication Service might not recover when power to the drive is interrupted and critical updates are lost.


13513

The File Replication Service on computer %1 is stopping because the database%2 is corrupted. The database can be recovered by typing "esentutl /d %2". Once the database has been successfully recovered the File Replication Service can be restarted by typing "net start ntfrs".


13514

File Replication Service is initializing the system volume with data from another domain controller. Computer %1 cannot become a domain controller until this process is complete. The system volume will then be shared as SYSVOL.

To check for the SYSVOL share, at the command prompt, type:

net share

When File Replication Service completes the initialization process, the SYSVOL share will appear.

The initialization of the system volume can take some time. The time is dependent on the amount of data in the system volume, the availability of other domain controllers, and the replication interval between domain controllers.


13515

The File Replication Service may be preventing the computer %1 from becoming a domain controller while the system volume is being initialized and then shared as SYSVOL.

Type "net share" to check for the SYSVOL share. The File Replication Service has stopped preventing the computer from becoming a domain controller once the SYSVOL share appears.

The initialization of the system volume can take some time. The time is dependent on the amount of data in the system volume.

The initialization of the system volume can be bypassed by first typing regedit and setting the value of SysvolReady to 1 and then restarting the Netlogon service.

WARNING - BYPASSING THE SYSTEM VOLUME INITIALIZATION IS NOT RECOMMENDED.

Applications may fail in unexpected ways.

The value SysvolReady is located by clicking on HKEY_LOCAL_MACHINE and then clicking on System, CurrentControlSet, Services, Netlogon, and Parameters.

The Netlogon service can be restarted by typing "net stop netlogon" followed by "net start netlogon".


13516

The File Replication Service is no longer preventing the computer %1 from becoming a domain controller. The system volume has been successfully initialized and the Netlogon service has been notified that the system volume is now ready to be shared as SYSVOL. Type "net share" to check for the SYSVOL share.


13517

The File Replication Service will not check access to the API "%1". Access checks can be enabled for "%1" by running regedt32. Click Start, Run, and type regedt32. Click the window entitled HKEY_LOCAL_MACHINE. Double click on SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, Access Checks,"%1", and "%2". Change the string to Enabled. Permissions can be changed by highlighting "%1" and then clicking the toolbar option Security, and then Permissions.


13518

The File Replication Service did not grant the user "%3" access to the API"%1". Permissions for "%1" can be changed by running regedt32. Click Start, Run, and type regedt32. Click the window entitled HKEY_LOCAL_MACHINE. Double click on SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, Access Checks, and highlight "%1". Click the toolbar option Security and then Permissions. Access checks can be disabled for "%1". Double click on "%2" and change the string to Disabled.


13519

The File Replication Service could not grant an unknown user access to the API "%1". Access checks can be disabled for "%1" by running regedt32. Click Start, Run, and type regedt32. Click the window entitled HKEY_LOCAL_MACHINE. Double click on SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, Access Checks, "%1", and "%2". Change the string to Disabled. Permissions can be changed by highlighting "%1" and then clicking on the toolbar option Security, and then Permissions.


13520

The File Replication Service moved the preexisting files in %1 to %2. The File Replication Service may delete the files in %2 at any time. Files can be saved from deletion by copying them out of %2. Copying the files into %1 may lead to name conflicts if the files already exist on some other replicating partner. In some cases, the File Replication Service may copy a file from %2into %1 instead of replicating the file from some other replicating partner. Space can be recovered at any time by deleting the files in %2.


13521

The File Replication Service cannot enable replication on the computer %1until a backup/restore application completes. A backup/restore application has set a registry key that prevents the File Replication Service from starting until the registry key is deleted or the system is rebooted. The backup/restore application may still be running. Check with your local administrator before proceeding further. The computer can be rebooted by clicking Start, Shutdown, and selecting Restart. WARNING - DELETING THE REGISTRY KEY IS NOT RECOMMENDED! Applications may fail in unexpected ways. The registry key can be deleted by running regedt32.Click Start, Run, and type regedt32. Click the window entitled HKEY_LOCAL_MACHINE. Double click on SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, Backup/Restore, "Stop NtFrs from Starting". On the toolbar, click on Edit and select Delete. Be careful! Deleting a key other than "Stop NtFrs From Starting" can have unexpected side effects


13522

The File Replication Service paused because the staging area is full. Staging files are used to replicate created, deleted or modified files between partners. FRS will automatically remove least recently used files from this staging area (in the order of the longest time since the last access) until the amount of space in use has dropped below 60% of the staging space-limit, after which replication will resume.

If this condition occurs frequently:

Confirm that all direct outbound replication partners receiving updates from this member are online and receiving updates.

Verify that the replication schedule for receiving partners is open or "on" for a sufficient window of time to accommodate the number of files being replicated.

Consider increasing the staging area to improve system performance.

The current value of the staging space limit is %1 KB. To change the staging space limit, run regedit: Click Start -> Run and type REGEDIT. Expand HKEY_LOCAL_MACHINE, SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, and the value "Staging Space Limit in KB".


13523

The File Replication Service paused because the size of a file exceeds the staging space limit. Replication will resume only if the staging space limit is increased. The staging space limit is %1 KB and the file size is %2 KB. To change the staging space limit, run regedt32. Click Start, Run and typeregedt32. Click the window entitled HKEY_LOCAL_MACHINE. Double click on SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, and the value "Staging Space Limit in KB".


13524

The File Replication Service is stopping on the computer %1 because a universally unique ID (UUID) cannot be created. The SDK function UuidCreate()returned the error "%2". The problem may be the lack of an Ethernet address, token ring address, or network address. The lack of a network address implies an unsupported net card. The File Replication Service will restart automatically at a later time. For more information about the automatic restart right click on My Computer and then click on Manage, System Tools, Services, File Replication Service, and Recovery


13525

The File Replication Service cannot find the DNS name for the computer %1 because the "%2" attribute could not be read from the distinguished name "%3". The File Replication Service will try using the name "%1" until the computer’s DNS name appears


13526

File Replication Service cannot replicate %1 with the computer %2 because the computer’s SID cannot be determined from the distinguished name "%3". The File Replication Service will retry later


13527

The RPC binding failed in the Open function of the FileReplicaSet Object. The counter data for this object will not be available. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13528

The RPC binding failed in the Open function of the FileReplicaConn Object. The counter data for this object will not be available. The FileReplicaConn object contains the performance counters of the connections over which files are being replicated by the File Replication Service.


13529

The RPC call failed in the Open function of the FileReplicaSet Object. The counter data for this object will not be available. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13531

The RPC binding failed in the Collect function of the FileReplicaSet Object. The counter data for this object will not be available till the binding succeeds. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13532

The RPC binding failed in the Collect function of the FileReplicaConn Object. The counter data for this object will not be available till the binding succeeds. The FileReplicaConn object contains the performance counters of the connections over which files are being replicated by the File Replication Service.


13533

The RPC call failed in the Collect function of the FileReplicaSet Object. The counter data for this object will not be available till the call succeeds. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13534

The RPC call failed in the Collect function of the FileReplicaConn Object. The counter data for this object will not be available till the call succeeds. The FileReplicaConn object contains the performance counters of the connections over which files are being replicated by the File Replication Service.


13535

The call to VirtualAlloc failed in the Open function of the FileReplicaSet Object. The counter data for this object will not be available. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13536

The call to VirtualAlloc failed in the Open function of the FileReplicaConn Object. The counter data for this object will not be available. The FileReplicaConn object contains the performance counters of the connections over which files are being replicated by the File Replication Service.


13537

The call to the registry failed in the Open function of the FileReplicaSet Object. The counter data for this object will not be available. The FileReplicaSet object contains the performance counters of the Replica sets whose files are being replicated by the File Replication Service.


13538

The call to the registry failed in the Open function of the FileReplicaConn Object. The counter data for this object will not be available. The FileReplicaConn object contains the performance counters of the connections over which files are being replicated by the File Replication Service.


13539

The File Replication Service cannot replicate %1 because the pathname of the replicated directory is not the fully qualified pathname of an existing, accessible local directory.


13540

The File Replication Service cannot replicate %1 because the pathname of the customer designated staging directory: %2 is not the fully qualified pathname of an existing, accessible local directory


13541

The File Replication Service cannot replicate %1 because it overlaps the File Replication Service’s logging pathname %2


13542

The File Replication Service cannot replicate %1 because it overlaps the File Replication Service’s working directory %2.


13543

The File Replication Service cannot replicate %1 because it overlaps the staging directory %2.


13544

The File Replication Service cannot replicate %1 because it overlaps the replicating directory %2.


13545

The File Replication Service cannot replicate %1 because it overlaps the staging directory %2 of the replicating directory %3.


13546

The File Replication Service could not prepare the root directory %1 for replication. This is likely due to a problem creating the root directory or a problem removing preexisting files in the root directory. Check that the path leading up to the root directory exists and is accessible.


13547

The File Replication Service detected an invalid parameter value in the registry. %1.

The expected registry key name is "%2".

The expected value name is "%3".

The expected registry data type is %4.

The allowed range for this parameter is %5

The data units for this parameter value are %6.

The File Replication Service is using a default value of "%7".

To change this parameter, run regedit.

Click Start, Run and type regedit.

Expand HKEY_LOCAL_MACHINE.

Click down the key path: "%8"

Double click on the value name - "%9" and update the value.

If the value name is not present you may add it with the New function under the Edit Menu item. Type the value name exactly as shown above using the above registry data type. Make sure you observe the data units and allowed range when entering the value.


13548

The File Replication Service is unable to replicate with its partner computer because the difference in clock times is outside the range of plus or minus %1 minutes.

The connection to the partner computer is: "%2"

The detected time difference is: %3 minutes.

Note: If this time difference is close to a multiple of 60 minutes then it is likely that either this computer or its partner computer was set to the incorrect time zone when the computer time was initially set. Check that the time zone and the system time are correctly set on both computers.

If necessary, the default value used to test for computer time consistency may be changed in the registry on this computer. (Note: This is not recommended.)

To change this parameter, run regedit.

Click Start, Run and type regedit.

Expand HKEY_LOCAL_MACHINE.

Click down the key path: "System\CurrentControlSet\Services\NtFrs\Parameters"

Double click on the value name "Partner Clock Skew In Minutes" and update the value.

If the value name is not present you may add it with the New->DWORD Value function under the Edit Menu item. Type the value name exactly as shown above.


13549

The File Replication Service is unable to replicate from a partner computer because the event time associated with the file to be replicated is too far into the future. It is %1 minutes greater than the current time. This can happen if the system time on the partner computer was set incorrectly when the file was created or updated. To preserve the integrity of the replica set this file update will not be performed or propagated further. The file name is: "%2" The connection to the partner computer is: "%3" Note: If this time difference is close to a multiple of 60 minutes then it is likely that this file may have been created or updated on the partner computer while the computer was set to the incorrect time zone when its computer time was initially set. Check that the time zone and time are correctly set on the partner computer.


13550

The File Replication Service is unable to open the customer designated staging directory for replica set %1. The path used for the staging directory is, "%2"

The customer designated root path for this replica set is: "%3"

The service is unable to start replication on this replica set. Among the possible errors to check are:

  • An invalid staging path.
  • A missing directory.
  • A missing disk volume.
  • A file system on the volume that does not support ACLs.
  • A sharing conflict on the staging directory with some other application.

Correct the problem and the service will attempt to restart replication automatically at a later time.


13551

The File Replication Service is unable to open (or create) the pre-install directory under the customer designated replica tree directory for replica set %1.

The path used for the pre-install directory is "%2"

The customer designated root path for this replica set is: "%3"

The service is unable to start replication on this replica set. Among the possible errors to check are:

  • An invalid root path.
  • A missing directory.
  • A missing disk volume.
  • A file system on the volume that does not support NTFS 5.0.
  • A sharing conflict on the pre-install directory with some other application.

Correct the problem and the service will attempt to restart replication automatically at a later time.


13552

The File Replication Service is unable to add this computer to the following replica set: "%1"

This could be caused by a number of problems such as:

  • An invalid root path.
  • A missing directory.
  • A missing disk volume.
  • A file system on the volume that does not support NTFS 5.0.

The information below may help to resolve the problem:

Computer DNS name is "%2"

Replica set member name is "%3"

Replica set root path is "%4"

Replica staging directory path is "%5"

Replica working directory path is "%6"

Windows error status code is %7

FRS error status code is %8

Other event log messages may also help determine the problem. Correct the problem and the service will attempt to restart replication automatically at a later time.


13553

The File Replication Service successfully added this computer to the following replica set: "%1"

Information related to this event is shown below:

Computer DNS name is "%2"

Replica set member name is "%3"

Replica set root path is "%4"

Replica staging directory path is "%5"

Replica working directory path is "%6"


13554

The File Replication Service successfully added the connections shown below to the replica set:

More information may appear in subsequent event log messages.


13555

The File Replication Service is in an error state. Files will not replicate to or from one or all of the replica sets on this computer until the following recovery steps are performed:

Recovery Steps:

[1] The error state may clear itself if you stop and restart the FRS service.

This can be done by performing the following in a command window:

net stop ntfrs

net start ntfrs

If this fails to clear up the problem then proceed as follows.

[2] For Active Directory Domain Controllers that DO NOT host any DFS alternates or other replica sets with replication enabled:

If there is at least one other Domain Controller in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.

If there are NO other Domain Controllers in this domain then restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and choose the Advanced option which marks the sysvols as primary.

If there are other Domain Controllers in this domain but ALL of them have this event log message then restore one of them as primary (data files from primary will replicate everywhere) and the others as non-authoritative.

[3] For Active Directory Domain Controllers that host DFS alternates or other replica sets with replication enabled:

(3-a) If the DFS alternates on this DC do not have any other replication partners then copy the data under that DFS share to a safe location.

(3-b) If this server is the only Active Directory Domain Controller for this domain then, before going to (3-c), make sure this server does not have any inbound or outbound connections to other servers that were formerly Domain Controllers for this domain but are now off the net (and will never be coming back online) or have been fresh installed without being demoted.

To delete connections use the Sites and Services snap-in and look for

Sites->NAME_OF_SITE->Servers->NAME_OF_SERVER->NTDS Settings->CONNECTIONS.

(3-c) Restore the "system state" of this DC from backup (using ntbackup or other backup-restore utility) and make it non-authoritative.

(3-d) Copy the data from step (3-a) above to the original location after the sysvol share is published.

[4] For other Windows servers:

(4-a) If any of the DFS alternates or other replica sets hosted by this server do not have any other replication partners then copy the data under its share or replica tree root to a safe location.

(4-b) net stop ntfrs

(4-c) rd /s /q %1

(4-d) net start ntfrs

(4-e) Copy the data from step (4-a) above to the original location after the service has initialized (5 minutes is a safe waiting time).

Note: If this error message is in the eventlog of all the members of a particular replica set then perform steps (4-a) and (4-e) above on only one of the members.


13556

The File Replication Service has detected what appears to be an attempt to change the root path for the following replica set: "%1"This is not allowed. To perform this operation you must remove this member from the replica set and add the member back with the new root path.

It is possible that this is a transient error due to Active Directory replication delays associated with updating FRS configuration objects. If file replication does not take place after an appropriate waiting time, which could be several hours if cross site Active Directory replication is required, you must delete and re-add this member to the replica set.

Information related to this event is shown below:

Computer DNS name is "%2"

Replica set member name is "%3"

The current Replica set root path is "%4"

The desired new Replica set root path is "%5"

Replica staging directory path is "%6"


13557

The File Replication Service has detected a duplicate connection object between this computer "%6" and a computer named "%1".

This was detected for the following replica set: "%2"

This is not allowed and replication will not occur between these two computers until the duplicate connection objects are removed.

It is possible that this is a transient error due to Active Directory replication delays associated with updating FRS configuration objects. If file replication does not take place after an appropriate waiting time, which could be several hours if cross site Active Directory replication is required, you must manually delete the duplicate connection objects by following the steps below:

[1] Start the Active Directory Sites and Services Snap-in.

[2] Click "%3, %4, %5, %6, %7".

[3] Look for duplicate connections from "%1" in site "%8".

[4] Delete all but one of the connections.


13558

The File Replication Service has detected a duplicate connection object between this computer "%7" and a computer named "%1".

This was detected for the following replica set: "%2"

This is not allowed and replication will not occur between these two computers until the duplicate connection objects are removed.

It is possible that this is a transient error due to Active Directory replication delays associated with updating FRS configuration objects. If file replication does not take place after an appropriate waiting time, which could be several hours if cross site Active Directory replication is required, you must manually delete the duplicate connection objects by following the steps below:

[1] Start the Active Directory Users and Computers Snap-in.

[2] Click the view button and advanced features to display the system node.

[3] Click "%3, %4, %5".

[4] Under "%5" you will see one or more DFS related replica set objects.

Look for the FRS member object "%6" under the subtree for replica set "%2".

[5] Under "%6" look for duplicate connections from "%1".

[6] Delete all but one of the connections.


13559

The File Replication Service has detected that the replica root path has changed from "%2" to "%3". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path. This was detected for the following replica set: "%1" Changing the replica root path is a two step process which is triggered by the creation of the NTFRS_CMD_FILE_MOVE_ROOT file. [1] At the first poll, which will occur in %4 minutes, this computer will be deleted from the replica set. [2] At the poll following the deletion this computer will be re-added to the replica set with the new root path. This re-addition will trigger a full tree sync for the replica set. At the end of the sync all the files will be at the new location. The files may or may not be deleted from the old location depending on whether they are needed or not.


13560

The File Replication Service is deleting this computer from the replica set "%1" as an attempt to recover from the error state,

Error status = %2

At the next poll, which will occur in %3 minutes, this computer will be re-added to the replica set. The re-addition will trigger a full tree sync for the replica set.


13561

The File Replication Service has detected that the replica set "%1" is in JRNL_WRAP_ERROR. Replica set name is: "%1" Replica root path is: "%2" Replica root volume is: "%3" A Replica set hits JRNL_WRAP_ERROR when the record that it is trying to read from the NTFS USN journal is not found. This can occur because of one of the following reasons. [1] Volume "%3" has been formatted. [2] The NTFS USN journal on volume "%3" has been deleted. [3] The NTFS USN journal on volume "%3" has been truncated. Chkdsk can truncate the journal if it finds corrupt entries at the end of the journal. [4] File Replication Service was not running on this computer for a long time. [5] File Replication Service could not keep up with the rate of Disk IO activity on "%3". Following recovery steps will be taken to automatically recover from this error state. [1] At the first poll, which will occur in %4 minutes, this computer will be deleted from the replica set. [2] At the poll following the deletion this computer will be re-added to the replica set. The re-addition will trigger a full tree sync for the replica set.


13562

Following is the summary of warnings and errors encountered by File Replication Service while polling the Domain Controller %1 for FRS replica set configuration information. %2.


13563

The File Replication Service has detected that the staging path for the replica set %1 has changed.

Current staging path = %2

New staging path = %3

The service will start using the new staging path after it restarts. The service is set to restart after every reboot. It is recommended that you manually restart the service to prevent loss of data in the staging directory. To manually restart the service, do the following:

[1] Run "net stop ntfrs" or use the Services snap-in to stop File Replication Service.

[2] Move all the staging files corresponding to replica set %1 to the new staging location. If more than one replica set are sharing the current staging directory then it is safer to copy the staging files to the new staging directory.

[3] Run "net start ntfrs" or use the Services snap-in to start File Replication Service, followed by "net start ntfrs".


13564

The File Replication Service has detected that the volume holding the FRS debug logs is running out of disk space. This will not affect replication unless this volume hosts database, staging, or replica root paths as well. Path to the logs directory = %1

You can change the number and size of logs by adjusting the following registry values. Sample values are shown below. These values are under the registry key "HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/ Services/NtFrs/Parameters

Debug Log Files REG_DWORD 0x5

Debug Log Severity REG_DWORD 0x2

Debug Maximum Log Messages REG_DWORD 0x2710

You can also change the path to the logs directory by changing the following value at the same location.

Debug Log File REG_SZ %windir%\debug

Changes to the registry values will take affect at the next polling cycle.


13565

File Replication Service is initializing the system volume with data from another domain controller. Computer %1 cannot become a domain controller until this process is complete. The system volume will then be shared as SYSVOL. To check for the SYSVOL share, at the command prompt, type net share. When File Replication Service completes the initialization process, the SYSVOL share will appear. The initialization of the system volume can take some time. The time is dependent on the amount of data in the system volume, the availability of other domain controllers, and the replication interval between domain controllers.


13566

File Replication Service is scanning the data in the system volume. Computer %1 cannot become a domain controller until this process is complete. The system volume will then be shared as SYSVOL. To check for the SYSVOL share, at the command prompt, type net share. When File Replication Service completes the scanning process, the SYSVOL share will appear. The initialization of the system volume can take some time. The time is dependent on the amount of data in the system volume


13567

File Replication Service has detected and suppressed an average of %1 or more file updates every hour for the last %2 hours because the updates did not change the contents of the file. The tracking records in FRS debug logs will have the file name and event time for the suppressed updates. The tracking records have the date and time followed by:T: as their prefix. Updates that do not change the content of the file are suppressed to prevent unnecessary replication traffic. Following are common examples of updates that do not change the contents of the file. [1] Overwriting a file with a copy of the same file. [2] Setting the same ACLs on a file multiple times. [3] Restoring an identical copy of the file over an existing one. Suppression of updates can be disabled by running regedit. Click Start, Run and type regedit. Expand HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters and create or update the value "Suppress Identical Updates To Files" to 0 (Default is 1) to force identical updates to replicate


13568

The File Replication Service has detected that the replica set "%1" is in JRNL_WRAP_ERROR.

Replica set name is: "%1"

Replica root path is: "%2"

Replica root volume is: "%3"

A Replica set hits JRNL_WRAP_ERROR when the record that it is trying to read from the NTFS USN journal is not found. This can occur because of one of the following reasons.

[1] Volume "%3" has been formatted.

[2] The NTFS USN journal on volume "%3" has been deleted.

[3] The NTFS USN journal on volume "%3" has been truncated. Chkdsk can truncate the journal if it finds corrupt entries at the end of the journal.

[4] File Replication Service was not running on this computer for a long time.

[5] File Replication Service could not keep up with the rate of Disk IO activity on "%3".

Setting the "Enable Journal Wrap Automatic Restore" registry parameter to 1 will cause the following recovery steps to be taken to automatically recover from this error state.

[1] At the first poll, which will occur in %4 minutes, this computer will be deleted from the replica set. If you do not want to wait %4 minutes, then run "net stop ntfrs" followed by "net start ntfrs" to restart the File Replication Service.

[2] At the poll following the deletion this computer will be re-added to the replica set. The re-addition will trigger a full tree sync for the replica set.

WARNING: During the recovery process data in the replica tree may be unavailable.

You should reset the registry parameter described above to 0 to prevent automatic recovery from making the data unexpectedly unavailable if this error condition occurs again.

To change this registry parameter, run regedit.

Click Start, Run and type regedit.

Expand HKEY_LOCAL_MACHINE.

Click down the key path: "System\CurrentControlSet\Services\NtFrs\Parameters"

Double click on the value name "Enable Journal Wrap Automatic Restore" and update the value.

If the value name is not present you may add it with the New->DWORD Value function under the Edit Menu item. Type the value name exactly as shown above.


13569

The File Replication Service has skipped one or more files and/or directories during primary load of the following replica set. The skipped files will not replicate to other members of the replica set.

Replica set name is: "%1"

A list of all the files skipped can be found at the following location. If a directory is skipped then all files under the directory are also skipped.

Skipped file list: "%2"

Files are skipped during primary load if FRS is not able to open the file. Check if these files are open. These files will replicate the next time they are modified.


13570

The File Replication Service has detected that the volume hosting the path %1 is low on disk space. Files may not replicate until disk space is made available on this volume.

The available space on the volume can be found by typing

"dir /a %1".

For more information about managing space on a volume type "copy /?",

"rename /?", "del /?", "rmdir /?", and "dir /?".


13571

The File Replication Service has detected that one or more volumes on this computer have the same Volume Serial Number. File Replication Service does not support this configuration. Files may not replicate until this conflict is resolved.

Volume Serial Number: %1

List of volumes that have this Volume Serial Number: %2

The output of "dir" command displays the Volume Serial Number before listing the contents of the folder.


13572

The File Replication Service was unable to create the directory "%1" to store debug log files. If this directory does not exist then FRS will be unable to write debug logs. Missing debug logs make it difficult, if not impossible, to diagnose FRS problems.


13573

File Replication Service has been repeatedly prevented from updating:

File Name: "%1"

File GUID: "%2"

due to consistent sharing violations encountered on the file. Sharing violations occur when another user or application holds a file open, blocking FRS from updating it. Blockage caused by sharing violations can result in out-of-date replicated content. FRS will continue to retry this update, but will be blocked until the sharing violations are eliminated.

Possible reasons for a sharing violation are other sources that may have opened the file to be replicated in on the target machine.

To determine the full path of the file in sharing violation open the Computer Management, Shared Folders, Open Files from compmgmt.msc for the file in question and the user that has the filed opened. Search for the file listed above, Right click on the file, Select the close option to forcibly close the file. Note if multiple files with the same name are held open you may need to close all or perform more detailed steps listed in the KB article to determine file with full path that matches the GUID reported in the event. The install override feature in the Windows Server 2003 version of ntfrs.exe can be used to avoid this blockage by renaming the locked file to allow replication to occur. For more information on the install override feature in the Windows Server 2003 version of ntfrs.exe and its behavior, please refer to http://support.microsoft.com/?id=816493

To disable logging of this event, edit the following registry keys as follows:

Click Start, Run, and then type regedit.

Expand HKEY_LOCAL_MACHINE, SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, and update the value of "Enable Sharing Violation Logging" to FALSE.

To change the frequency of reporting relative to the number of retries performed by FRS, edit the following registry key as follows:

Click Start, Run, and then type regedit.

Expand HKEY_LOCAL_MACHINE, SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, and create (REG_DWORD, min =1, max=30, default=10) or update the value "Sharing Violation Events Counter" to a larger value to reduce the frequency of reporting.

To change the number of events reported per hour, edit the following registry key as follows:

Click Start, Run, and then type regedit.

Expand HKEY_LOCAL_MACHINE, SYSTEM, CurrentControlSet, Services, NtFrs, Parameters, and create (REG_DWORD, min =1, max=60, default=10) or update the value "Max Sharing Violation Event".

FRS will only report at most 60 sharing violation blockages every hour, beyond this limit, additional sharing violations may be unreported as events. If you suspect that FRS is hitting more sharing violations than those reported per hour causing replication to be backlogged, you may need to follow the detailed steps listed in the KB article.


All Servers Have an Incomplete Replica Tree

If all servers have incomplete data, you will need to perform an authoritative restore. See Performing an Authoritative Restore for more information.


At Least One Domain Controller Has the Needed Data But FRS is Not Working

If you have verified that all FRS dependencies are working, then the problem is likely FRS. On the problem server, run FRSDiag.exe, which collects and analyzes FRS data. After running FRSDiag, review the FRSDiag.txt output file. If FRSDiag identifies any of the problems below, click the appropriate link for troubleshooting information.

  • ERROR_SHARING_VIOLATION
  • ERROR_ACCESS_DENIED
  • IBCO_STAGING_RETRY
  • IBCO_FETCH_RETRY
  • IBCO_INSTALL_RETRY
  • Junction Point Problems
  • Missing Subscriber Objects
  • SYSVOL Not Shared
  • FRS event ID 13568: USN Journal Wrap

If FRSDiag.txt shows other errors/warnings in the File Replication Service event log, see Troubleshooting FRS Using Event Logs for a list of common FRS errors. You can also search this Help file for the event ID to locate troubleshooting topics.


At Least One Server Has a Complete Replica Tree

If you have verified that all FRS dependencies are working, then the problem is likely FRS. Run FRSDiag.exe, which collects and analyzes FRS data. After running FRSDiag, review the FRSDiag.txt output file. If FRSDiag identifies any of the problems below, click the appropriate link for troubleshooting information.

  • ERROR_SHARING_VIOLATION
  • ERROR_ACCESS_DENIED
  • IBCO_STAGING_RETRY
  • IBCO_FETCH_RETRY
  • IBCO_INSTALL_RETRY
  • Missing Subscriber Objects
  • FRS event ID 13568: USN Journal Wrap
  • FRS event ID 13567: Excessive Replication

If FRSDiag.txt shows other errors/warnings in the File Replication Service event log, see Troubleshooting FRS Using Event Logs for a list of common FRS errors. You can also search this Help file for the event ID to locate troubleshooting topics.


Choose the Replication Type

Click the appropriate link below to specify the type of replication problem.

  • SYSVOL Replication Problems on Domain Controllers
  • DFS-Related Replication Problems on Servers


Data is Incomplete on All Domain Controllers

If the data is damaged on all domain controllers, we recommend that you contact Microsoft Product Support Services to assist you in the FRS recovery process, which might also require rebuilding the default policies. FRS recovery will mostly depend on the amount of time you have (or don’t have) and on the existence of a good recent backup.

Important

  • Rebuilding SYSVOL should only be used as a last resort because it is a very intrusive process. It is critical to verify that all FRS dependencies are working before proceeding.


DFS-Related Replication Problems on Servers

Can you access the file share by using \\FQDN\Sharename? If you cannot, investigate access issues using article 257346, "Access This Computer from the Network User Right Causes Tools Not to Work" at http://support.microsoft.com/?id=257346.

If you can access the file share, click the link below that best applies to your situation.

  • At Least One Server Has a Complete Replica Tree
  • All Servers Have an Incomplete Replica Tree


ERROR_ACCESS_DENIED in FRSDiag.txt

If ERROR_ACCESS_DENIED is shown in FRSDiag.txt, verify permissions on the replica set. In some cases, antivirus software modifies the permissions on upstream and downstream partners. Required permissions are as follows:

  • System: Full Control
  • Administrator: Full Control
  • Authenticated users: Read & Execute

This problem might be easily resolved by selecting Allow Inherited permissions from parent on the child folder.


FRS is Working but Not Optimally

If files are replicating but you notice other problems, click the appropriate link below for troubleshooting information.

  • Excessive Replication and High CPU and Disk Usage by Ntfrs.exe
  • Morphed Folders
  • Sharing Violations


FRS Missing Member and Subscriber Objects

FRS relies on essential containers, objects, and attributes stored in Active Directory and replicated among domain controllers in a given domain. Critical objects include FRS member and subscriber objects. Optional attributes include the Schedule attribute, the FRS-File-Filter attribute, the FRS-Folder-Filter attribute, and the FRS database location. For more information about these objects, see article 296183, "Overview of Active Directory Objects That Are Used by FRS" at http://support.microsoft.com/?id=296183

To troubleshoot this problem, do the following:

  1. Check for existence of FRS event ID 13562 or "Missing Subscriber Object" in FRSDiag.txt.
  2. Run FRSDiag, and then run TopChk on NTFRS_DS.txt. For instructions, see TopChk Remarks.
  3. In Ntfrs_ds.txt (generated by FRSDiag), find which objects are missing.
  4. Do one of the following:
    • If referenced server still exists, follow the steps listed article 312862, "Recovering Missing FRS Objects and FRS Attributes in Active Directory" at http://support.microsoft.com/?id=312862.
    • If the referenced server no longer exists: use Metadata Cleanup to remove the referenced server. Use Replmon to trigger replication after the deletion, and run TopChk afterward. For more information, see article 216498, "HOW TO: Remove Data in Active Directory After an Unsuccessful Demotion" at http://support.microsoft.com/?id=216498.


IBCO_FETCH_RETRY in FRSDiag.txt

IBCO_FETCH_RETRY means that FRS is waiting to retry remote change order staging file fetch. This error might be caused by connectivity problems with the upstream partner. To resolve this problem, check connectivity with the upstream partner by doing the following:

  • If upstream partner is in VVJoin with its parent, run FRSDiag on the upstream partner.
    • To find the upstream partner, use TopChk.
    • To find if partner is in VVJoin, use Connstat output.

For more information about running these scripts and interpreting their output, see TopChk Remarks and ConnStat Remarks.


IBCO_INSTALL_RETRY in FRSDiag.txt

IBCO_INSTALL_RETRY means that file installation into the replica set is being retried and is thus causing a sharing violation. Check for third-party software that might be holding open the files in question. For more information, see Sharing Violations.

If problem persists, contact Microsoft Product Support Services and increase the severity of debug logs:

  • HKEY_LOCAL_MACHINE\system\currentcontrolset\services\NtFrs Parameters
    • Debug Log Severity = REG_DWORD 4
    • Debug Log Files = REG_DWORD 5
    • Debug Maximum Log Messages = REG_DWORD 200000


IBCO_STAGING_RETRY in FRSDiag.txt

IBCO_STAGING_RETRY means that FRS is waiting to retry local staging file generation. This could indicate that the staging directory is full. See Staging Directory is Full for details.

Note that in Windows 2000 SP3 and Windows Server 2003, the staging directory does not fill up. See Adjust the Staging Directory Size Accordingly and article 329491, "Configuring Correct Staging Area Space for Replica Sets" at http://support.microsoft.com/?id=329491.

If problem persists, contact Microsoft Product Support Services and increase the severity of debug logs as follows:

  • HKEY_LOCAL_MACHINE\system\currentcontrolset\services\NtFrs Parameters
    • Debug Log Severity = REG_DWORD 4
    • Debug Log Files = REG_DWORD 5
    • Debug Maximum Log Messages = REG_DWORD 200000


Install the Latest Version of FRS

Before you troubleshoot FRS, if at all possible, upgrade all replica members to the latest version of FRS.

  • For Windows 2000, install Windows 2000 Server with SP4 or Windows 2000 Server with SP3 and the post-Service Pack 3 release of Ntfrs.exe, described in article 815473, "File Replication Service Does Not Log Errors on Sharing Violations" at http://support.microsoft.com/?id=815473.
  • For Windows Server 2003, install the pre-Service Pack 1 release of Ntfrs.exe as described in article 823230, "Issues That Are Fixed in the Pre-Service Pack 1 Release of Ntfrs.exe" at http://support.microsoft.com/?id=823230.

Note

  • For current information about FRS, see the DFS and FRS Web site at http://go.microsoft.com/fwlink/?LinkId=18421.

Click Verify FRS Dependencies to continue.


SYSVOL Not Shared

This problem occurs when SYSVOL is not shared correctly. To troubleshoot this problem, do the following:

  1. If you have not done so already, check DNS. See Verify FRS Dependencies for more information.
  2. Refer to the following articles:
    • For Windows 2000 domain controllers, see article 257338, "Troubleshooting Missing SYSVOL and NETLOGON Shares on Windows 2000 Domain Controllers" at http://support.microsoft.com/?id=257338.
    • For Windows Server 2003 domain controllers, see article 327781, "How to Troubleshoot Missing SYSVOL and NETLOGON Shares on Windows Server 2003 Domain Controllers" at http://support.microsoft.com/?id=327781.

If this problem occurred after you restored a domain controller from backup, see article 316790, "The SYSVOL and NETLOGON Shares Are Missing After You Restore a Domain Controller from Backup" at http://support.microsoft.com/?id=316790.


SYSVOL Replication Problems on Domain Controllers

To begin troubleshooting SYSVOL problems, check to see if you can access SYSVOL by using \\FQDN\SYSVOL.

If you cannot access SYSVOL, see article 257346, "Access This Computer from the Network User Right Causes Tools Not to Work" at http://support.microsoft.com/?id=257346.

If you can access SYSVOL, click the link below that best applies to your situation.

  • FRS is Working But Not Optimally
  • At Least One Domain Controller Has the Needed Data But FRS is Not Working
  • Data is Incomplete on All Domain Controllers


Verify FRS Dependencies

Before you can troubleshoot FRS, you must verify that the FRS dependencies are working properly on the server experiencing problems. If any of the dependencies are not working, any time spent troubleshooting FRS is futile.

After you verify these dependencies, proceed to Choose the Replication Type. If you are not sure how to check these dependencies, contact Microsoft Product Support Services for assistance.

DNS

  • Use NETDiag to check for errors. (NETDiag is part of the Windows Support Tools.)
  • If possible, point all FRS servers to one Windows 2000 DNS server for testing. For additional information about DNS, see the following articles:
    • 291382, "Frequently Asked Questions About Windows 2000 and Windows Server 2003 DNS" at http://support.microsoft.com/?id=291382.
    • 260371, "Troubleshooting Common Active Directory Setup Issues in Windows 2000" at http://support.microsoft.com/?id=260371.
    • 257623, "Domain Controller’s Domain Name System Suffix Does Not Match Domain Name" at http://support.microsoft.com/?id=257623.
    • 258832, "Cannot Join Windows 2000 Client to a Windows NT Domain" at http://support.microsoft.com/?id=258832.
    • 203009, "DNS Zone Transfer Between Windows 2000 and Windows NT 4.0 Fails" at http://support.microsoft.com/?id=203009.
    • 229840, "DNS Server’s Root Hints and Forwarder Pages Are Unavailable" at http://support.microsoft.com/?id=229840.
    • 246804, "How to Enable/Disable Windows 2000 Dynamic DNS Registrations" at http://support.microsoft.com/?id=246804.
    • 275278, "DNS Server Becomes an Island When a Domain Controller Points to Itself for the _Msdcs.ForestDnsName Domain" at http://support.microsoft.com/?id=275278.
    • 235689, "How to Troubleshoot 7062 Errors Logged in DNS Event Log" at http://support.microsoft.com/?id=235689.
    • 259277, "Troubleshooting Netlogon Event 5774, 5775, and 5781" at http://support.microsoft.com/?id=259277.

Active Directory replication

  • Use Replmon and Dcdiag to check Active Directory replication. (Replmon and Dcdiag are part of the Windows Support Tools.)
  • Check for NTDS connection objects by running TopChk and making sure objects are present, enabled, and that the replication schedule is on. For more information, see TopChk Remarks and Verify the FRS Topology As Stored in Active Directory.

Network connectivity

  • FRS needs reliable links to move data around. If the links are slow or busy, you will run into replication problems.
  • Use NETDiag to check to networking-related errors. (NETDiag is part of the Windows Support Tools.)
  • Check for firewall and black hole routers. See the following articles for details:
    • 314825, "How to Troubleshoot Black Hole Router Issues" at http://support.microsoft.com/?id=314825.
    • 159211, "Diagnoses and Treatment of Black Hole Routers" at http://support.microsoft.com/?id=159211.
  • Use Event Viewer from remote computers to check for RPC connectivity.
    • 319553, "How to Restrict FRS Replication Traffic to a Specific Static Port" at http://support.microsoft.com/?id=319553.
  • Use Terminal Server session (if possible) to check for network connectivity.


Verifying SYSVOL Junction Points

Run FRSDiag against the domain controller and verify the junction points by viewing Ntfrs_sysvol.txt. The junction points look similar the following:

12/06/2001  02:53p      <DIR>          .
12/06/2001  02:53p      <DIR>          ..
12/21/2000  10:10a      <JUNCTION>     Contoso.COM
0 File(s)              0 bytes

If <JUNCTION> exists, do the following:

  1. Check to see if someone restored SYSVOL to two or more locations or copied SYSVOL to another location. If so, you have too many junction points. Because they are all linked, deleting data in one location will delete data from other folders.
  2. Find all locations where SYSVOL was restored or copied.
  3. Use the Resource Kit tool Linkd.exe to delete junctions, and then delete the folder.
  4. Test replication and run FRSDiag again.

If <JUNCTION> does not exist, note that FRS does not function without the junction point. To correct this, perform the following procedure.

  1. Use the Resource Kit tool Linkd.exe to recreate the SYSVOL junction point. For more information about creating junction points, see article 205524, "How to Create and Manipulate NTFS Junction Points" at http://support.microsoft.com/?id=205524.
  2. Restart FRS.
  3. If it works, test replication and run FRSDiag again.

If this procedure does not fix the issue, contact Microsoft Product Support Services to discuss other methods of restoring the junction point. One possible procedure is a nonauthoritative restore, which will fix the junction point but will require the entire replica tree to be replicated to this server. You can prestage the data to avoid replicating data across the network.

For more information about SYSVOL junction points, see article 324175, "Best Practices for SYSVOL Maintenance" at http://support.microsoft.com/?id=324175.