When it comes to performance the list of variables can be quite long and the solutions do not always bubble up to be what you think they should be.
For instance we can start at the user level and trickle all the way up to the SQL or Application Server, but you can also have application, custom code and network bandwidth that comes into the formula. Mix that in with CPU, Memory Utilization and Disk I/O and the battle is on.
If we start with the user's machine then we must realize that Microsoft Dynamics CRM is a web application that is authenticating and using IE. There are key settings that you can make to insure that things are not slowing on the user's machine and Microsoft has provided a number of white papers and recommendations regarding these. (see below)
When using the CRM Web Access option these include items such as insuring the web pages are in the trusted zone, understanding your latency and confirming that your anti-virus software is not fighting with the application.
When looking at the user machine and the CRM Outlook client the volume of variables expands. You want to take into consideration all the different "other applications" that can impact Outlook. For instance the numerous toolbars that can get installed or the size of the Outlook data footprint and if the user has numerous PST files in addition to their Exchange mailbox. You also want to check the settings on the anti-virus/anti-spyware/anti-yuck software which often wants to interact with anything going on. Another consideration is the version of Outlook that you are using. Outlook 2007 is significantly different than Outlook 2003 for instance.
For both of the above another consideration comes into play and this has to do with the amount of memory and the hardware being used by the user. Despite the fact that many times hardware is not the solution, hardware can make a huge difference if limited.
We also have the User's Operating system as a variable. Outlook 2007 for instance was built and designed for VISTA and yet many companies are running Windows XP. This means that Outlook 2007 must take into consideration a completely different platform than it was designed for and although I have not seen any specific issues related to this it is a variable.
Now despite all these factors you might find that one small thing makes a huge difference. From my personal experience I have both run the outlook client without issue (perfect performance) and have run into massive headaches (resulting in extended effort to fix things).
Here are some links that might help if you have to dive into this area.
First the Microsoft White Papers and posts
Optimizing and Maintaining Microsoft Dynamics CRM 4.0
Optimizing the Performance of MS Dynamics CRM 3.0
Microsoft Dynamics CRM 4.0 Performance and Scalability White Papers
Improving Microsoft Dynamics CRM Performance and Securing Data with Microsoft SQL Server 2008
Boost performance with Pre-generated XmlSerializers
Best Practices for Better Performance from Your Custom Code
Performance Odds and Ends
Investigating CRM V4 Performance
Performance Study - Dynamics CRM 4.0 WAN Performance using Cisco Wide Area Application Services
Tips for faster and better performance on Microsoft Dynamics CRM 4.0
Other Blog Posts on Microsoft Outlook and CRM Performance
The CRM Outlook Client Diagnostic Utility
Improving the Outlook Experience with CRM
You receive Outlook sync messages when you view a shared calendar or a shared contacts folder in Microsoft Dynamics CRM 4.0 client for Outlook
and Blog posts on Performance with a Server Focus
Enterprise Scalability and Performance with Microsoft Dynamics CRM 4.0
Ronald Lemmen on CRM Performance
MS Dynamics CRM and Perfmon
Optimizing IIS Performance for Microsoft CRM, Part 1
Improving the Performance of IIS 6.0 Applications
What is interesting is that you can easily get into tuning on SQL or IIS that really can make a difference and yet are significantly far away from CRM Expertise.