Stjepan Groš

Using Zimbra on CentOS

My goal was to install additional Zimbra server for a disaster recovery site of a certain company. This company already had primary mail server which also runs Zimbra. In this document I'm describing the setup process as well as some notes about user migration. Parts of this document can be used as a guideline for primary mail server installation.

Additional goal of this setup is to have test server. Namely, when Zimbra publishes new release of it's server, I wanted to have place where I can try to upgrade server and see how it behaves. Only after it turns good, I'll install main Zimbra server.

So, suppose we have four zones. DMZ with mail.dmz.domain.local primary mail server, DR site with mail.dr.domain.local mail server, local network colocated with DMZ and Internet. All the users have mail address of the form username@domain.local and/or name.surname@domain.local.

In this document I'm using CentOS 5.5 and Zimbra 6.0.8. Notes for the older versions, as I update this document, will be placed at the end.

Installing Zimbra

Preinstallation steps

The first pre-installation step is for you to determine storage and CPU requirements for the Zimbra. You can find pointers to more details in the links section below. Here, I'll only give a brief overview.

Storage requirements

It is important that you properly size disk storage, or at least leave a room for extensions in case you run into a filesystem storage problems.

First off, you have to have at least 5G free space in /opt/zimbra directory. Then, you have to add to this storage for mailboxes. This is approximately calculated as follows (taken from the references!):

  1. Let U be the number of users. Note that you should count the number of users you'll have at least in a year from now, and more preferably in three years from now. If you don't know how many user there will be, try to look at numbers that show how many users were there year ago, two years ago, etc. and then try to guess how many will be in the next three years.
  2. Estimate how much each user will have data on mail server, let's call this Su (in MB). Again, the best way to do this is to look at the data on the existing server. Even more better is if you have some historical data so that you can make predictions few years in advance.
  3. For storage requirements you need S = U x Su MB for mailboxes, 15% x S for MySQL data, 20G for Zimbra logs, and 20% x S for indexes. Add all this and you'll get approximate storage requirements.

As one final point, I was talking about planning three years in future but actually this depends on you purchase plans and maintenance plans. If there is no problem for you to assign additional 300G to a server any time it is necessary and you have enough man-power to do this along with the scheduled server downtime then you can have smaller window for dimensioning server. As usual, each environment has it's own specifics and peculiars so modify this advice to suite your needs.

Distributing disk load

CPU requirements

Base OS install

For the base OS install, please look at the CentOS Installation page.

Note that Zimbra installs into /opt/zimbra directory where it also places all the data, including mail messages, so be careful to assign enough space for this file system, either /opt or if you insist separate /opt/zimbra partition. Another directory that we'll be stressed is /var/log because mail logging is also done via syslog into zimbra.log file. You have the option to change that, but it's default so be careful.

The following additional packages are necessary to be installed as Zimbra depends on them: 

anacron vixie-cron perl sudo libidn fetchmail gmp
compat-libstdc++-296 compat-libstdc++-33 libtool-ltdl
sysstat

If you are installing Zimbra on 64-bit system then invoking yum with the previous list will cause some 32-bit binaries to be installed which are not necessary (i.e. 32-bit perl interpreter). To avoid installing unnecessary components use the following list: 

anacron.x86_64 crontabs.x86_64 vixie-cron.x86_64 perl.x86_64 sudo.x86_64
libidn.x86_64 fetchmail.x86_64 gmp.x86_64 compat-libstdc++-296.x86_64
compat-libstdc++-33.x86_64 libtool-ltdl.x86_64 unrar.x86_64 unzip.x86_64
compat-libstdc++-296.i386 compat-libstdc++-33.i386 libstdc++-4.1.2-44.el5.i386
sysstat.x86_64

Note that 32-bit C++ runtime libraries are mandatory to install.

Some packages are optional to install but it is recommended that you install them, e.g. anti spam filter in Zimbra depends on them and some might make your life easier: 

unrar unzip

Installing all this will also pull exim SMTP server as a dependency, as we are installing mail server that has SMTP server included, remove exim using rpm command. You'll have to use --nodeps option, otherwise rpm will refuse to remove the packet: 

rpm -e --nodeps exim

Now it is a right time to reboot the machine because kernel will almost certainly be updated and possibly some other core system libraries.

Before continuing, be certain that hostname of the machine you are installing is properly bounded to it's IP address in hosts file. Furthermore, you have to insert the name in the following format or otherwise installation program will fail: 

<ip> <FQHN> <HN>

DNS setup

You should configure your DNS in order to properly route mail messages to your mail server. If you only plan to set up outgoing SMTP server it is not necessary to modify DNS configuration.

The minimum change necessary is to add MX record for the domain that points to the mail server's hostname and appropriate A record to associate IP address with the mail server name.

Note that if you have NAT (some incorrectly call it firewall, even though those are two separate functions usually bundled together) between your mail server and the rest of the network then A record's IP address will be different than your mail server's true IP address.

Alternatively, you can install DNS on the mail server itself. Look at this page on some instructions on how to do that.

Firewall setup

Firewall is a must have! I'll describe here what has to be done to properly configure access permissions to the mail server. Stay tuned... :)

In the mean time note that if default CentOS firewall is turned on you'll not be able to access Zimbra so you have to modify it!

Install Zimbra

Copy Zimbra archive you downloaded from the Zimbra site to the mail server using scp tool. In case you already unpacked the archive you'll have directory tree instead of a single file so it's better to use rsync tool than scp.

Now, on the mail server enter the directory with Zimbra installation files and start the installation process using the following command: 

./install.sh --platform-override

The option --platform-override is necessary as Zimbra supports RHEL and we are installing on CentOS. If you miss this option than the installer will notify you that it is necessary to start again with the given option and terminate further execution.

During installation you'll see prerequisites checking. Pay close attention that everything is found or else you might have some unexpected problems with the installation. Be careful that installer doesn't check for cron and if you don't have it installed it will not install crontab and there will be some side effects. 

Checking for prerequisites...
     FOUND: NPTL
     FOUND: sudo-1.7.2p1-8
     FOUND: libidn-0.6.5-1.1
     FOUND: gmp-4.1.4-10
     FOUND: /usr/lib64/libstdc++.so.6
Checking for suggested prerequisites...
    FOUND: perl-5.8.8
    FOUND: sysstat
Prerequisite check complete.

Next, you'll be asked for the components you wish to install. This installation has two specifics. First it is a mail server without shell access for users or any other possibility to interact with a system. The other specific is that no central user database exists. So, LDAP should be selected, but it's good to select all the components of the Zimbra.

You have to answer few more questions with defaults N so change that and the installation of RPM packages will begin.

After RPMs are installed, installer will notify you that the domain name has no MX record, but the problem is that domain name is actually hostname, i.e. if you have domain dr.domain.local and your mail server is called mail.dr.domain.local installer will think that the domain is called mail.dr.domain.local which is obviously wrong. So, correct this when it asks you if you want to change domain name, in our case to dr.domain.local.

At the end of the installation process you are presented with the menu to configure Zimbra. Only the admin password is mandatory to change. This password is highly sensitive, so chose it wisely. :) 

Main menu

   1) Common Configuration:                                                  
   2) zimbra-ldap:                             Enabled                       
   3) zimbra-store:                            Enabled                       
        +Create Admin User:                    yes                           
        +Admin user to create:                 admin@example.local         
******* +Admin Password                        UNSET                         
        +Enable automated spam training:       yes                           
        +Spam training user:                   spam.3qbfqigdhy@example.local
        +Non-spam(Ham) training user:          ham.ktmfh5bjx@example.local 
        +Global Documents Account:             wiki@example.local          
        +SMTP host:                            mail.example.local          
        +Web server HTTP port:                 80                            
        +Web server HTTPS port:                443                           
        +Web server mode:                      http                          
        +IMAP server port:                     143                           
        +IMAP server SSL port:                 993                           
        +POP server port:                      110                           
        +POP server SSL port:                  995                           
        +Use spell check server:               yes                           
        +Spell server URL:                     http://mail.example.local:7780/aspell.php
        +Configure for use with mail proxy:    FALSE                         
        +Configure for use with web proxy:     FALSE                         
        +Enable version update checks:         TRUE                          
        +Enable version update notifications:  TRUE                          
        +Version update notification email:    admin@example.local         
        +Version update source email:          admin@example.local         

   4) zimbra-mta:                              Enabled                       
   5) zimbra-snmp:                             Enabled                       
   6) zimbra-logger:                           Enabled                       
   7) zimbra-spell:                            Enabled                       
   8) Default Class of Service Configuration:                                
   r) Start servers after configuration        yes                           
   s) Save config to file                                                    
   x) Expand menu                                                            
   q) Quit                                    

Address unconfigured (**) items  (? - help)

Select 3 then 4 and type in administrator's password. Return to the main menu, save configuration and Zimbra will start. I suggest that you allow sending notification about your installation to Zimbra.

After installation process has finished, if you want SELinux enabled (I believe you should), then do the following: 

chcon -t texrel_shlib_t /opt/zimbra/httpd-2.2.8/modules/libphp5.so

This will get rid of some annoying warning messages.

After this change you shoud restart Zimbra services. Note that this has to be done as a zimbra user: 

$ zmcontrol stop
Host mail.dr.domain.local
	Stopping stats...Done
	Stopping mta...Done
	Stopping spell...Done
	Stopping snmp...Done
	Stopping archiving...Done
	Stopping antivirus...Done
	Stopping antispam...Done
	Stopping imapproxy...Done
	Stopping mailbox...Done
	Stopping logger...Done
	Stopping ldap...Done
$ zmcontrol start
Host mail.dr.domain.local
	Starting ldap...Done.
	Starting logger...Done.
	Starting mailbox...Done.
	Starting antispam...Done.
	Starting antivirus...Done.
	Starting snmp...Done.
	Starting spell...Done.
	Starting mta...Done.
	Starting stats...Done.

Now you can access administration console on: 

https://mail.dr.domain.local:7071/

and Web mail on: 

https://mail.dr.domain.local/

Install 32-bit Zimbra on 64-bit system

There could be cases in which it is necessary to install 32-bit Zimbra on a 64-bit OS. For example, I had a situation in which backup server was 32-bit and new server was 64-bit. Eventually I decided to install 64-bit OS on a new server, but then to put 32-bit Zimbra on it in order to be able to sync the whole Zimbra tree on a secondary server.

This is a combination that will not be allowed by the Zimbra installer. So you have to cheat a bit. On a 64-bit CentOS 5.4, in order to install 32-bit Zimbra 6.0.6 do the following:

Note that some 64-bit packages are not necessary in this case, but probably the simplest way for now is to install them and later try to optimize. In general, the rule is simple. If something is a binary then 64-bit version is necessary (e.g. sysstat tools) and if something is a library then 32-bit version is a necessary. The perl above is a bit specific. It is a binary, but the problem is that it has libraries that Zimbra tries to use so 32-bit version has to be installed.

Cluster installations

There are several possibilities if you need Zimbra in a cluster, each with its own pros and cons. Obviously, the most optimal way is to purchase Network Edition. But in case you can not do that (e.g. lack of money) you can try one of the possibilities defined in the following subsections.

All the cluster installations can be grouped according to the following parameters:

Note that I don't say anything about load balancing. This, as far as I know, is only possible when using Network Edition.

Migrate users from another Zimbra

The idea here is that if you already have one Zimbra, to add another one as a backup server you have to a) migrate users, and b) set up appropriate mail routing.

The steps to migrate users from main Zimbra server are:

  1. Dump LDAP database on the source Zimbra

    Go to the primary mail server and become zimbra user. Then, in the /opt/zimbra directory, run the following command: 

    $ ./openldap/sbin/slapcat -f /opt/zimbra/conf/slapd.conf -l /tmp/ldap.ldif
The first database does not allow slapcat; using the first available one (2)

    After this, you'll have a file ldap.ldif in your /tmp directory which contains all the information about your users. But, it also contains data about your mail server that you have change, or transfer to DR mail server. Copy this file to DR mail server before making any changes.

  2. Modify domain names

    You should modify domain and host names in LDIF, otherwise Zimbra won't work. For a setup described in the introduction you need to run the following command: 

    sed 's/dmz.domain.local/dr.domain.local/g' ldap.ldif > ldap.ldif.dr

    Note that it might become more complex that this because of the way Zimbra stores aliases into LDAP. For example, if you had zimbra.dmz.domain.local domain which you want to remove on the new server simple grep -v won't help. There is a separate tree in LDAP, dc=zimbra,dc=dmz,dc=domain,dc=local which won't be affected by the grep command and you'll have to either write script to handle this or manually edit the ldif file.

  3. Stop the target Zimbra 
    $ zmcontrol stop

    Just in case, check if LDAP is running. If so kill it using the kill command

  4. Clean /opt/zimbra/openldap-data directory. DO NOT remove logs and db subdirectories and also keep DB_CONFIG file!
  5. Import ldif database 
    $ cd /opt/zimbra
$ ./openldap/sbin/slapadd -f /opt/zimbra/conf/slapd.conf -l /tmp/ldap.ldif.dr
The first database does not allow slapadd; using the first available one (2)

    The warning you see is the only one you should see and I think you can safely ignore it.

  6. Get LDAP passwords from source Zimbra using: 
    $ zmlocalconfig -s ldap_amavis_password ldap_postfix_password ldap_replication_password ldap_root_password zimbra_ldap_password
ldap_amavis_password = password
ldap_postfix_password = password
ldap_replication_password = password
ldap_root_password = password
zimbra_ldap_password = password

    All the passwords are same, I don't know if it every time but for now I'll suppose it is. In case I'm wrong, modify the following steps appropriately.

  7. You should start Zimbra on DR mail server now because otherwise it is not possible to change password. Note that there will be errors but we ignore them: 
    $ zmcontrol start
Host mail.dr.paba.hr
	Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.
  8. Now modify all the passwords: 
    $ zmldappasswd <password from source Zimbra>
Updating local config and directory
$ zmldappasswd -l <password from source Zimbra>
Updating local config and directory
$ zmldappasswd -p <password from source Zimbra>
Updating local config and directory
$ zmldappasswd -a <password from source Zimbra>
Updating local config and directory
$ zmldappasswd -r <password from source Zimbra>
Updating local config and directory
Updating slapd.conf
  9. Now stop and start Zimbra again and it should work normally. Here is how netstat -ltn output should look: 
    $ netstat -ltn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 192.168.1.218:389           0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:10024             0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:10025             0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:7306              0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:7307              0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:3310                0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:465                 0.0.0.0:*                   LISTEN      
tcp        0      0 192.168.1.218:53            0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      
tcp        0      0 :::7072                     :::*                        LISTEN      
tcp        0      0 :::7777                     :::*                        LISTEN      
tcp        0      0 :::993                      :::*                        LISTEN      
tcp        0      0 :::995                      :::*                        LISTEN      
tcp        0      0 :::7780                     :::*                        LISTEN      
tcp        0      0 :::5222                     :::*                        LISTEN      
tcp        0      0 :::5223                     :::*                        LISTEN      
tcp        0      0 :::7335                     :::*                        LISTEN      
tcp        0      0 :::110                      :::*                        LISTEN      
tcp        0      0 :::143                      :::*                        LISTEN      
tcp        0      0 :::80                       :::*                        LISTEN      
tcp        0      0 :::7025                     :::*                        LISTEN      
tcp        0      0 :::5269                     :::*                        LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      
tcp        0      0 ::1:953                     :::*                        LISTEN      
tcp        0      0 :::443                      :::*                        LISTEN      
tcp        0      0 :::7071                     :::*                        LISTEN

Few notes to be aware of:

  1. You should be aware that by copying LDAP you also copied SSH keys and possibly other configuration data which you might prefer to be different. For SSH keys use zmsshkeygen.
  2. During migration process you probably removed default domain, so you have to log in to administration console using admin@mail.dr.domain.local username and set default domain to either mail.dr.domain.local or dr.domain.local. This is done in Global settings option on the administration Web interface.

After user migration, in order for the mail routing to work properly, you should add to each user an alias of the form username@dr.domain.local. You can use the following simple script to do that for each user: 

#!/bin/bash

USERS=`zmprov gaa mail.local`

# First, add new domain
zmprov cd dr.mail.local

# Then, for each user add new alias
for i in $USERS
do
	echo "Processing user $i"
	zmprov aaa $i ${i/mail.local/dr.mail.local}
done

In order to set up routing I suggest you first modify only several users to test the installation, and when it's proven to work correctly then modify all the users using a simple shell script.

Updating Zimbra

Sooner or later you'll have to update Zimbra. In my case I updated first the DR mail from version 5.0.8GA to 5.0.10GA. Everyting went well, except I noticed that in the administration console, after selecting Configuration -> Servers -> mail.dr.domain.local I get the following message: 

Message: system failure: exception during auth {RemoteManager: mail.dr.domain.local->zimbra@mail.dr.domain.local:22} Error code: service.FAILURE Method: GetServerNIfsRequest Details:soap:Receiver

I don't know if this error was caused by the migration process, or after updating the Zimbra, but the fix is documented in the following post on Zimbra forums:

http://www.zimbra.com/forums/installation/21647-solved-server-section-error.html

Using Zimbra as an authentication server

Zimbra can be used as an authentication server for other services. The primary reason someone would like to do something like that is that Zimbra allows user to easily change password, and also, allows password policies to be set. Furthermore, Zimbra doesn't allow automatic user provisioning from some external source (like AD) without much hassle.

In this section I'll list services I managed to connect to Zimbra and also give some gotchas while doing it.

But, before describing each service in turn I'll note that Zimbra's LDAP doesn't allow anonymous searches. Thus, any service that does search before bind should have credentials to bind to LDAP. For this reason I'll create an account, called zmsearch with only search rights on a certain attributes. All the following steps have to be done as a zimbra user unless otherwise noted!.

For a start, create a file with the following contents: 

dn: uid=zmsearch,cn=appaccts,cn=zimbra
uid: zmsearch
objectClass: zimbraAccount
objectClass: organizationalPerson
cn: zmsearch
sn: zmsearch
zimbraAccountStatus: active
zimbraIsSystemResource: TRUE
zimbraId: 20EBF77A-119B-41A7-BEE8-6CC29F982630
description: The search account
userPassword: {SSHA}x

First, note the attribute zimbraId. This attribute has to have a unique value. Since I was unable to find out how it is generated I created a temporary user, took his zimbraId and then removed this temporary user. Second, for the attribute userPassword you have to think of some password and then convert it into SSHA using the /opt/zimbra/openldap/sbin/slappasswd command like this: 

$ /opt/zimbra/openldap/sbin/slappasswd -s newpassword
{SSHA}xg/mWDH8lTjoIzXyh8QocZulqYtcJ2qK

Then c/p the output into the file with a new system account (zmsearch). Now, find out the master LDAP password with the following command: 

$ zmlocalconfig -s ldap_root_password

And import this new user into LDAP: 

$ ldapadd -f <filename> -x -H ldapi:/// -D cn=config -W
Enter LDAP Password: 
adding new entry "uid=zmsearch,cn=appaccts,cn=zimbra"

Don't forget to change filename with a name of file you used. When asked for a password enter the LDAP's master password you obtained using zmlocalconfig command.

This user now has search right. (TODO: Maybe restrict it more by modifying ACL-s for a security reasons?)

Apache

For security reasons, in any file that contains password for a LDAP user that allows searches change group and user ownership to root and and restrict read and write permission to only owner of a file!

LINK LINK

Samba

Shell access

Alfresco

http://forums.alfresco.com/en/viewtopic.php?f=9&t=21698

Asterisk

Supporting Zimbra

In this section I'll list what I have to do as a part of the maintenance phase in a certain company. All this can be found on the Internet (primarily on the excelent Zimbra's Wiki) but this is further customized for the setup I have.

Problems using Web console

One of the reasons that Web console doesn't work is that SSH keys are not valid. For example, when you get the following error message: 

2011-05-11 14:36:40,048 INFO  [btpool0-0] [name=admin@localhost.localdomain;mid=2;ip=192.168.1.240;ua=ZimbraWebClient - FF3.0 (Linux);] SoapEngine - handler exception
com.zimbra.common.service.ServiceException: system failure: exception during auth {RemoteManager: localhost.localdomain->zimbra@localhost.localdomain:22}
ExceptionId:btpool0-0:1305117400048:ed8467682ca1df93
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:251)
        at com.zimbra.cs.rmgmt.RemoteManager.getSession(RemoteManager.java:193)
        at com.zimbra.cs.rmgmt.RemoteManager.execute(RemoteManager.java:127)
        at com.zimbra.cert.GetCert.handle(GetCert.java:93)

It means that SSH keys are not correct. There is test command you can run to find out if the keys are correct: 

ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@localhost.localdomain

In case you are asked for a password, then the problems is with the keys. To recreate keys and deploy them use the following two commands (run them as zimbra user!): 

zmsshkeygen
zmupdateauthkeys

More information you'll find on Zimbra Wiki pages.

Recreating Zimbra self-signed certificate

Periodically you'll have to renew your self-signed certificates. This is easy to do from the command line and it boils down to the following sequence of commands: 

    

  1. su -

  2. su - zimbra

  3. su

  4. zmcertmgr createcrt -new -days 1095

  5. zmcertmgr deploycrt self

  6. zmcertmgr viewdeployedcrt

The su commands are run in that sequence in order to have Zimbra environment correctly set (su - zimbra) and then to have administrative privileges but still with Zimbra environment variables (e.g. PATH) correctly defined.

The first zmcertmgr creates self-signed certificate that will be valid for three years (3*365 days). The next one deploys new certificate. Finally, the last command shows currently deployed certificate so that you can check if everything went OK.

More information about manipulating certificate you can find on Zimbra Wiki pages.

Blocking senders based on their e-mail address

Sometimes you'll wish to block senders that have certain email address. This could be done per user, but in case you want to do it for a whole domain then you can modify salocal.cf.in file in /opt/zimbra/conf directory. There, you should place a line like the following one (you can append it at the end of the file): 

blacklist_from user@some.domain

Then, restart spam filter using the following commands (as a zimbra user): 

zmmtactl restart && zmamavisdctl restart

Finally, try to send test mail using telnet command and check the logs (/var/log/zimbra.log). You should see a line like the following one (note that the line was edited for readability but it is a single line in a log file!): 

Apr 11 14:51:01 mail amavis[25309]: (25309-02) Blocked SPAM, [192.168.0.1] [192.168.0.1] <user@some.domain>
	-> <somerecepient@some.domain>, Message-ID: <20110411125040.AB2A93B7C042@some.domain>,
	mail_id: NiSZ6q-grTy4, Hits: 97.1, size: 403, 347 ms

This is a confirmation that the message was received by a mail system but was discarded as a spam.

Be aware that it can happen that this change isn't preserved when you upgrade to a new version!

Finally, you can modify postifx configuration to achieve the same effect. The difference will be that the mail server will reject mail after MAIL FROM command and the sender will receive failed notification. If you use the modification of salocal.cf.in no notification will be sent to the sender.

Hunting missing mails

Well, when some user reports that her mail isn't delivered you'll have to find it. Alternatively, if someone claims that he didn't receive mail you'll have to prove that he is. All that assumes some digging throught log files.

I usually start with a question whether the user received back any report about undelivered mail. If he is, the reason why the message hasn't been delivered is stated in the report. Otherwise, log files to the rescue.

In /var/log/maillog you can track the offending message from the moment it entered into the mail system to the moment it left it, in any way. The possibilities are: it was delivered, it was dropped by spam/antivirus filter, or it was handed to some other mail server.

When antispam filter decides to drop a message you'll see a line like the following one: 

Nov 21 04:30:26 mail postfix/smtp[24616]: 484F63B7C069:
	to=<testuser@testdomain.com>,
	orig_to=<testuser@testdomain.com>,
	relay=127.0.0.1[127.0.0.1]:10024, delay=6.6, delays=0.52/0.01/0/6,
	dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=02542-08 - SPAM)

What this tells us is that the message that had ID 484F63B7C069 while in the mail system, was dropped. There is additional identifier 02542-08 that allows one to look into amavis logs to see why exactly this mail was marked as a spam. Note that marking message as a spam but letting it to proceed will not generate a message like this!

Amavis performs logging into /var/log/zimbra.log (actually, this file also includes maillog file). So, you can search there for a reason why this mail message was dropped. This is an example: 

Nov 23 16:36:47 mail amavis[6072]: (02542-08) Blocked SPAM,
	[10.0.0.5] [172.16.50.30]
	<no-reply@rolex.com> -> <testuser@testdomain.com>,
	Message-ID: <201011231536.oANFag6P024967@somemailhost.com>,
	mail_id: HI29Ev5BfUhw, Hits: 28.937, size: 3824, 73 ms

First IP address (10.0.0.5) is address of mail server that delivered mail to us, while the second IP address (172.16.50.30) is originating computer, i.e. the one that actually generated the message.

When email message was delivered to a local user in some folder, you'll find the message that looks something like this (broken into multiple lines for readability): 

Nov 23 16:39:28 mail postfix/lmtp[14662]: 1822B3B7C083:
	to=<testuser@testdomain.com>,
	relay=mail.testdomain.com[192.168.1.2]:7025, delay=0.17,
	delays=0.05/0/0/0.11, dsn=2.1.5, status=sent (250 2.1.5 Delivery OK)

This means that the message that had ID 1822B3B7C083 while processed in the mail system was finally delivered to a local user.

Other important log file is /opt/zimbra/log/mailbox.log. There you'll find when the particular user connected to check/receive new mail and using which protocol, but you'll also find the information about delivered mail. For example, this is how it looks when a mail message has been delivered into Inbox (of course it's obfuscated in order to protect innocent :)): 

2010-11-23 00:00:02,504 INFO  [LmtpServer-9243]
	[name=testuser@testdomain.com;mid=55;ip=192.168.1.2;]
	mailop - Adding Message: id=421441,
	Message-ID=<20101122230001.584623B7C059@mail.testdomain.com>,
	parentId=-1, folderId=2, folderName=Inbox.

What this message says is that message with Message-ID 20101122230001.584623B7C059@mail.testdomain.com has been delivered into folder named Inbox at a specified time. This message ID you can find in the headers of the email message. On the other hand there is another ID (id=), 421441, that says the message's ID on a local file system. You'll find it in /opt/zimbra/store, but note that this is a substring in the whole name!

One very important note. Email message, from the moment it enters mail system to the moment it exits will have two or three IDs in the form like in the examples above, i.e. 484F63B7C069 and 1822B3B7C083

Occasional problems with spam filter

It happened to me that one of the Zimbra users reported a problem with Zimbra. Some mails weren't delivered, while others did. This is a very specific setup so at first, I thought that it might be related to some wrong configuration option on a client machine. Also, I thought that Outlook client marked mail as junk. It turned out that Zimbra flagged message as spam and stored it into Junk folder. Now, clients use POP and thus access only Inbox folder so they didn't noticed it. I finally found out that it was delivered into Junk (see previous note for how to find missing mails).

Anyway, it turned out that this is an older Zimbra installation and that Spamassassin rules are not automatically updated. Thus, one rule was wrong giving all email messages higher score and in the end, some of them ended up as a Junk. The offending rule is related to OpenWhois server (DNS_FROM_OPENWHOIS), and maybe some others. Google for it and you'll find relevant information.

Anyway, the morale of this story is that you have to be careful and update SpamAssassin rules regularity. The easiest way to achieve this is to install distribution's SpamAssassin package and use it to retrieve updates. Place than updates into /opt/zimbra/conf/spamassassin/ directory taking care that permissions are good.

Different log messages and explanations/solutions

2010-09-28 12:38:23,336 WARN  [btpool0-18] [] log - javax.net.ssl.SSLException: Received fatal alert: unknown_ca

This one started to occur for some unknown reason after I regenerated CA and certificate on Zimbra server. I didn't manage to find a way to get CA using some Zimbra provided tool, but CA is stored in the /opt/zimbra/ssl/zimbra/ca directory under the name ca.pem. Using OpenSSL you can easily check the validity of your CA: 

openssl x509 -in ca.pem -noout -text

In my case it turned out that CA expired, even though certificates signed by this CA were still valid. 

2010-09-28 13:00:40,531 ERROR [btpool0-27] [] log - /zimbra/
org.mortbay.io.RuntimeIOException: java.io.IOException: Closed
        at org.mortbay.io.UncheckedPrintWriter.write(UncheckedPrintWriter.java:183)
        at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:181)
        [cut]

This seems to be hurmless debuging output fixed in versions post 6.0.1. See this link and this bugzilla entry. 

$ zmcontrol start
Host gw.centrala.plastform.hr
        Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

See this page.

Encrypted ZIP files

Encrypted ZIP files are blocked by the antivirus/antispam filter. There are two possibilities to resolve this issue, depending on how frequently you send/receive such messages:

  1. In case this is rarely and infrequent, then you can leave ZIP file blocking and manually retrieve the zip files from the backups.
  2. If the number of messages is high, then you can either pass encrypted ZIP messages, which is dangerous. Or you can teach your users to remove, or otherwise modify file extension, before sending it via email and recover it upon reception.

    Encrypted archives are passed by modifying option in Web administration console, Global settings -> AS/AV -> Block encrypted archives.

Auditing Zimbra management

Sometimes it is necessary to know who did something in Zimbra. For example, you might want to know who and when created some user account. In general, everything is stored in the /opt/zimbra/log directory, specifically, audit.log files.

In that file, you can find out the following information:

  1. Adding/removing aliases

    Look for the lines of the following format for adding and removing aliases:

    2008-11-19 09:34:27,886 INFO  [btpool0-509] [name=sgros@localhost.localdomain;mid=8;
ip=192.168.1.8;ua=ZimbraWebClient - FF3.0 (Linux);] security - cmd=ModifyAccount;
 name=testuser@localhost.localdomain;
2008-11-19 09:34:28,421 INFO  [btpool0-509] [name=sgros@localhost.localdomain;mid=8;
i p=192.168.1.8;ua=ZimbraWebClient - FF3.0 (Linux);] security - cmd=AddAccountAlias;
name=testusers@localhost.localdomain; alias=test.user@localhost.localdomain;

    This line informs you that at the given time (2008-11-19 at 9:34am) user sgros@localhost.localdomain modified account testuser@localhost.localdomain. The lines that follow the given line inform you what the modification was. In this case an alias was added to the account.

  2. Adding new user account

    When new account is added you'll see something like this: 

    2008-11-17 09:15:05,491 INFO  [btpool0-430] [name=sgros@localhost.localdomain;mid=8;
ip=192.168.1.8;ua=ZimbraWebClient - FF3.0 (Linux);] security - cmd=CreateAccount;
 name=testuser@localhost.localdomain; ou=User;
zimbraMailCanonicalAddress=test.user@localhost.localdomain; street=Some street;
zimbraMailStatus=enabled; l=Zagreb; company=Some company; postalCode=10000;
sn=Test; zimbraAccountStatus=active; zimbraPasswordMustChange=TRUE; co=Croatia;
givenName=Test; displayName=Test User;
2008-11-17 09:15:05,763 INFO  [btpool0-430] [name=sgros@localhost.localdomain;mid=8;
ip=192.168.1.8;ua=ZimbraWebClient - FF3.0 (Linux);] security - cmd=AddAccountAlias;
name=tuser@localhost.localdomain; alias=test.user@localhost;

    In this case we see that the user sgros@localhost.localdomain accessing Zimbra from host with the IP address 192.168.1.8 created new user with a name testuser@localhost.localdomain. Also, this user must change address when loggs in (zimbraPasswordMustChange=TRUE).

Log files are very important to know what happened to you installation. Default is to keep something like last 8 days of logs. It is good practice to increase this. But, keep you eye on disk usage, otherwise you can have real problems.

One other thing is very important. You should force all your users that are allowed to create and modify account to log into Zimbra using their "normal" username. If you have shared account (default is admin) then you'll have hard time determining who messed something.

Adding disclaimer

The following text is valid for Zimbra version prior to cca. 6.0.4. For later versions see the next subsection.

One additional big fat warning! On some servers with disclaimer implemented via method described in this section it happens that mails are cut off (see this and this). While I still don't know what is the reason for such behavior, I strongly suspect that it is altermime to be blamed for that, but I don't have any proof.

Zimbra has several shortcomings, one of which is the inability to set domain wide disclaimer on all the outgoing mail messages. There are, of course, workarounds and I used the follwing one:

http://wiki.zimbra.com/index.php?title=Adding_a_disclaimer_(altermime)_or_footer

On this page it is suggested that the altermime source package is downloaded and installed, which is further elaborated in the steps 1 to 3. But, in my opinion, it is not good practice to compile packages from the source, unless absolutely necessary as this adds maintenance burden, i.e. you have to monitor changes and, more important, security bugs of these packages and react properly when bugs are found. Unfotunatelly, there is no altermime package in the CentOS (and very probably RHEL) repository. But, there is one in the DAG repository so I decided to use it.

To install DAG repository on the CentOS I have (CentOS 5.2 on i386), I used the following command: 

rpm -Uvh http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

and after that installation of altermime was simple invocation of yum command: 

yum install altermime

Steps 4 to 6 should be done as described on the page.

Note that if you apply the step 7 then all the mails that passes through the mail server will have discaimer added. I believe that few people need this so probably you'll want to use step 7a. In the step 7a it is assumed that all your users have configured outgoing mail server to be 192.168.0.2, while all the external mails come via 192.168.0.1 IP address. Thus, the mail host has two IP addresses. The line that starts with 127.0.0.1 is mandatory to allow sending mail via loopback interface.

So, to cut this story short, change the first line that starts with smtp from: 

smtp    inet  n       -       n       -       -       smtpd

into 

smtp    inet  n       -       n       -       -       smtpd
        -o content_filter=dfilt:

And immediatelly after that line add the following two lines: 

dfilt   unix  -       n       n       -       -       pipe
        flags=Rq user=filter argv=/opt/zimbra/postfix/conf/disclaimer -f ${sender} -- ${recipient}

This solution, while simple, can be problematic to some setups. For example, if you don't have an easy way of adding another IP address. There is of course an alternative, which is modeled after the different disclaimers per domain described on the linked page above. The point is that the script disclaimer accepts several parameters, of which $2 contains the mail address of the sender. Thus, you can check in the script whether the argument $2 contains your domain naime, and if it does, then you can add disclaimer, otherwise you simply pass the mail unchanged.

For that matter I'll give here a bare bones filter, the one that simply passes the mail received on the standard input to it's standard output. This might be handy in case you want to add some specific processing of each mail message that passes through the mail server. 

#!/bin/bash
INSPECT_DIR=/var/spool/filter
SENDMAIL=/opt/zimbra/postfix/sbin/sendmail
FOLDER_DISCLAIMER=/opt/zimbra/postfix/conf

# Exit codes from <sysexits.h>
EX_TEMPFAIL=75
EX_UNAVAILABLE=69

# Clean up when done or when aborting.
trap "rm -f in.$$" 0 1 2 3 15

# Change to working directory where temporary files will be created
cd $INSPECT_DIR || { echo $INSPECT_DIR does not exist; exit $EX_TEMPFAIL; }

# cat will transfer all the data from stdin of the script into the
# file named in.$$. Instead of $$ a PID of the process will be
# inserted so that unique file name is used.
cat > in.$$ || { echo Cannot save mail to file; exit $EX_TEMPFAIL; }

# Here you insert your processing. Note that $2 is the value of the
# from field, while $4 is a value of the to field

# This will send file back to the mail server
$SENDMAIL "$@" < in.$$

exit $?

Using Zimbra builtin mandatory signature for disclaimer

Starting from somewhere around 6.0.4 version of ZCS you can use Zimbra's built in support for mandatory mail signature. There are two shortcomings to this version:

  1. Mandatory signature has to be defined via CLI (using command zmprov), and
  2. You can not choose which mails and when will be modified, i.e. all mails will have added signature.

In case the second problem is of concern, you have to use "manual" way. Otherwise, proceed with the reading. :)

To enable mandatory signature, use the following command (as Zimbra user!): 

zmprov mcf zimbraDomainMandatoryMailSignatureEnabled TRUE

Note that in some forum posts you'll see attribute zimbraDomainMandatoryMailSignature but it is an error. Next, to define what signature will be, use the following two commands: 

zmprov mcf zimbraDomainMandatoryMailSignatureText "This is text signature"
zmprov mcf zimbraDomainMandatoryMailSignatureHTML "<b>This is HTML signature</b>"

In case you have signatures written in files (e.g. signature.txt for text signature and signature.html for HTML signature), you can use also the following form: 

zmprov mcf zimbraDomainMandatoryMailSignatureText "`cat signature.txt`"
zmprov mcf zimbraDomainMandatoryMailSignatureHTML "`cat signature.html`"

Note that after executing the previous commands you need to at least restart Amavis in order for the changes to take effect. In case that no signature has been added to mails (when they are received!), then look into the log file (/var/log/zimbra.log). In my case I found the following two lines in there: 

Jan 20 17:08:20 mail amavis[2123]: (02123-01) (!!)collect_results from [6195] (/opt/zimbra/altermime/bin/altermime): DIED, signal 11 (000b) mime_alter.c:1350:ERROR - Cannot stat '/opt/zimbra/data/altermime/global-default.txt' (No such file or directory)\nmime_alter.c:1350:ERROR - Cannot stat '/opt/zimbra/data/altermime/global-default.html' (No such file or directory)\nAttempting to add disclaimer\n
Jan 20 17:08:20 mail amavis[2123]: (02123-01) (!)mangling by altermime failed: Program /opt/zimbra/altermime/bin/altermime failed: 11, mime_alter.c:1350:ERROR - Cannot stat '/opt/zimbra/data/altermime/global-default.txt' (No such file or directory)\nmime_alter.c:1350:ERROR - Cannot stat '/opt/zimbra/data/altermime/global-default.html' (No such file or directory)\nAttempting to add disclaimer, mail will pass unmodified

For some reason, restart didn't generate files global-default.txt and global-default.html. I created them manually and after that everything worked.

Controlling access to distribution lists

Zimbra unfortunatelly doesn't provide easy way to control who can send an email to a distribution list. This is very problematic and there is a high probability that you'll have to fix it. The only way it can be done (up to and including Zimbra 6.0.1) is by modifying postifx configuration files as documented on the following link:

RestrictPostfixRecipients

There is also another possibility, to integrate Mailman with Zimbra.

Accessing Zimbra through SOAP in Python

I'm writing a Pyhton class that will allow one to access Zimbra in Python script. Details can be found on the development page

Pulling groups/distribution lists from ActiveDirectory

Manipulating mailboxes from CLI

The command that manipulates mailboxes from CLI is zmmailbox. The following features were important to me:

Moving Zimbra to another server

It is possible to move Zimbra to another server by (almost) simply copying directory hierarchy. This can come up handy when you want to make additional identical installation, or when you have backup copy that you wish to activate. The drawback is that server have to be almost identical, including the name. Still, IP addresses can be different as not much in Zimbra configuration is bound to IP address itself.

First, be sure that all the required packages on the target server are installed. For required packages see the beginning of this document.

Then copy /opt/zimbra to another server in the same directory or, in case it is copied to some other location, create appropriate links! Note that rsync is the most appropriate tool for this step as it preserves ownership details.

Create zimbra user and group with the home in /opt/zimbra. Also add postfix user and group. Note that they have to have the same UID and GID as on the source Zimbra server.

Copy whole or parts of the following files to the new machine:

Modify hosts file to contain the same name as on the original host. The IP address can be different. This step is only necessary if you plan to run parallel version of Zimbra server.

Start the new server.

Aliases for external accounts

After I integrated mailman with Zimbra I had a problem that one list needed two different names. This is usually accomplished using aliases but in my case it didn't work. I tried to modify /etc/aliases and add there something like: 

alias:	existinglist

But for some strange reason it didn't work. Then I tried to find if Mailman supports list aliases but this also failed. Then, I thougth I could modify aliases file from the mailman such that I copy parts of existing list and only change the alias name (the name before colon). But this didn't work either.

I'm still refusing to add new account just to be able to have an alias.

Old Zimbra versions

Here are the notes that document differences between newest Zimbra version described on this page and the previous versions:

Zimbra links

Zimbra MTA - Anti-Virus Protection
Ajcody-Server-Topics
HOWTO replace AD+Exchange with Samba+Zimbra
Howto: Highly available Zimbra cluster using Heartbeat and DRBD
6 Tips for a Smooth Zimbra Server Install
SistemNET d.o.o.