I have spent quite a bit of time understanding how Perforce works under the hood and experimented with various configurations in an attempt to tame this wild beast. Here is a collection of things that I have learned throughout the years which might or might not be known to the average person configuring a Perforce server. All of this information have been gathered on a Windows setup of the Perforce server with clients on Windows and Linux(Mac). Most of this information is also applicable to Linux installations of Perforce.

Background

Most operations involving the Perforce Server involves memory, CPU, network bandwidth/throughput and foremost disc I/O. Very few operations are bound by CPU in my experience. Instead it is the disc I/O and network I/O that have been the bottlenecks. Figuring out what data to send to the client and then the actual retrieval and sending of the data is the main areas of concern.

Database Locking

[http://www.perforce.com/perforce/doc.current/schema/]

Every Perforce operation involves one or more database files in order to service the request. To ensure atomic operations and the integrity of the database, each operation locks all necessary database files with either a sharable(read) or an exclusive(write) lock. The address above shows the order in which these files are attempted to be locked. The files are always locked in the same order to avoid deadlocks. If a file can’t be locked the operation waits a while for the lock. At this point different versions of the server behaves differently. Once all locks have been acquired the relevant changes are made followed by the releasing of all locks.

Locking Information

[http://kb.perforce.com/article/630/debugging-perforce-database-locking]

Adding the global option ‘-Ztrack’ to any Perforce command will display the database locking information for that command. Adding the ‘-vtrack=1’ and ‘-v server=3’ as server parameters will give you lots of useful logging on the server side.

Example

prompt$ p4 -Ztrack counter change

22

— lapse .016s

— rpc msgs/size in+out 0+1/0mb+0mb himarks 64836/524924

— db.counters

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 1+0+0 0+0

— db.user

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 1+0+0 0+0

— db.group

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 0+3+5 0+0

— db.domain

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 1+0+0 0+0

— db.view

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 0+1+2 0+0

— db.protect

—?? pages in+out+cached 2+0+1

—?? locks read/write 1/0 rows get+pos+scan put+del 0+1+3 0+0

— db.monitor

—?? pages in+out+cached 3+4+2

—?? locks read/write 0/2 rows get+pos+scan put+del 0+0+0 2+0

The listing above shows the database activity for the ‘p4 counter change’ command where I read the counter named ‘change’. Let’s decipher the output. The listing shows the files in the order in which they were locked.

— db.protect

—?? pages in+out+cached 2+0+1

The line above indicates that the server read 2 pages, wrote 0 pages and cached 1 page of the db.protect database. The size of a page can be found by running the command ‘p4 dbstat db.protect’. For me this size is 8 KB per page.

—?? locks read/write 1/0 rows get+pos+scan put+del 0+1+3 0+0

Now, let’s look at the line containing the details of the operation. It indicates how many read/write locks were acquired during the servicing of the request. In this case only 1 sharable read lock was acquired. Next we see how many rows of the database were read. ‘get’ indicates the retrieval of a single row, ‘pos’ and ‘scan’ is used together to position and read a block of rows. In this case 3 rows were read(scan) in 1 operation(pos). No rows were added or deleted. [http://kb.perforce.com/article/883/interpreting-server-log-files].

Analyzing the situation

Examining the server logs and using the ‘-Ztrack’ flag will give you insight into how your server is performing. For very large sites you might have many millions of files. If your users also have very wide client specs you might have many hundreds of thousands or rows read during a simple ‘p4 sync’. Using other system tools such as Process Monitor you can examine the network utilization and file accesses done by the server. In the past I have noticed that the default installation of a Perforce server is reading and writing data in small block sizes both to/from disc as well as the network. Fortunately this can be detected using Process Monitor and can be addressed using ‘tunables/configurables’.

Tunables

[www.perforce.com/perforce/conferences/eu/2010/Presentations/Michael_Shields-Tunables.slides.pdf]

These are configuration parameters that you can set to modify certain parameters and buffer sizes in how your Perforce server/client. One thing that I have noticed was that a Perforce server took many times longer to transfer a 20 MB file then it took to copy the file using a straight up Windows file copy. The reason ended up being small TCP transfer buffers. By increasing the tunable net.buf
size and net.tcpsize to 32K or 64K for both the server and clients our transfer speeds got 5 times faster for large files. Another good tunable is the size of each commit when reading/writing revision files to(client) and from(server) disc, filesys.bufsize. It defaults to 4K. Increasing this to 32K or 64K greatly improves performance of transferring large files.

A little bit of this and that…

Create a spec depot

Enable versioning of all client specs, user configurations, labels, permissions table, depot listings and more
Create a new depot by the name ?spec? and of type ?spec? and map it to some local location. Once done you run the command ?p4 admin updatespecdepot -a? to take all existing information and add it to the spec depot. Very useful to have a history for some of these files.

Checkpoint often… and practice recovering…

?p4 admin checkpoint? is all you need to run to checkpoint your database. Don’t forget to also backup all the version files. Recovering works if the versioned files are newer or equally old as the checkpoint but a newer checkpoint than the version files can yield an inconsistent database. Other commands are prevented from running during the process so run it in the middle of the night or on the weekend. Practice restoring the database so that you don?t feel scared about such an operation if suddenly needed.

Permissions

Using P4Admin is a really convenient way for managing users, groups and permissions.? I strongly advice to create however many groups you need and then only add permissions to groups. This simplifies the permissions table a lot.
Also, using this approach will prevent automatic creation of users as you only give explicit permission to individual groups. A user not assigned to any group can not do anything which is exactly what you want. Beware of ?user * *? permission table entries.

Memory Hog

Have ample of memory on your Perforce machine… especially if it is under Windows. In cases where a large number of users all sync the same files and the number of files is large you can see some serious spikes in memory usage and processing time. The reason for this is that each user will need to have his/her records of what files they already have read into memory from disc(db.have). Each record can be estimated to be maybe 256 – 512 bytes. Also, each serviced user have a list of the files needed which is an additional ~1K per file.
I have seen cases where syncing 50000 new files by 50 users brought the memory load of the server to over 15 GB. The server started swapping to the page file and things got even slower. Due to the db files now being swapped to disc the read times and locking times got enormous (minutes).
Most of this could have been avoided by having much more memory and by setting the MaxLockTime for all groups to something sane (30000 ms).

Multiple automatic build machines?

If you currently are using one user per build machine there are a few things you can do.
Ask Perforce support to grant you a ?background user? or ?build machine user?. This will effectively give you one extra user added to your licensed user count. Now, have all your tools run as this one user and create multiple client specs for them to allow different views for each build machine. Having one user for all build machines saves money and admin time. Don?t forget to prevent your build machines from accidentally mess things up by setting up the permissions properly. One way is to add your build machine user to a group that only have ?review? permissions. This will only allow them to read files but also modify the ?p4 counters? and ?p4 reviews? but nothing else.

The broke-ass ?p4 verify? command

Perforce suggests that you do a ?p4 verify //…? before making a checkpoint. The fact of the matter is that this can take a REALLY long time. The reason for this is the excessive reading of data from disc. I tried modifying the ?filesys.bufsize
? tunable to 512K to reduce the disc load. Instead it got absurdly slow as the server would now read 512K, use 1K of it and then throw it out, read another 512K offset by 1K and then throw it out… After 30 hours I shut it down and emailed Perforce suggesting a buffered file approach. Perforce acknowledged the issue, added it to the bug database but seems to have no plans on fixing this in the near future.
I am so tired of software treating disc I/O as reading any normal piece of memory (Visual Assist, Virus Scanners and more are notorious for reading 1-12 bytes at a time from files. *sigh*)

Useful Links

http://www.perforce.com/perforce/conferences/sessionindexwho.html
http://www.perforce.com/perforce/technical.html