Title: Recent Changes

<!-- This is part of the Python Milter Applications documentation.
 Copyright (C) 2001,2002,2003,2004,2005-2011 Business Management Systems, Inc.
 See the file license.ht for copying conditions.  -->

<h2> Recent Changes for the <code>pymilter</code> project </h2>

<h3> 1.0 </h3>
Remove start.sh to track EPEL repository, use daemonize as replacement
in RPMs using pymilter.
Allow ACCEPT as untrapped exception policy.
Support optional dir for getaddrset and getaddrdict in Milter.config to
reduce config clutter.
Show registered milter name in untrapped exception message.
Include selinux subpackage (FIXME: selinux subpackage should not care about pymilter version).
Support sqlite3 for greylisting, and provide Milter.greylist export and Milter.greysql import to migrate data.


<h3> 0.9.8 </h3>
Add Milter.test module for unit testing milters.  Fix typo that prevented
setsymlist from being active. Change untrapped exception message to:
"pymilter: untrapped exception in milter app"

<h3> 0.9.7 </h3>
Raise RuntimeError when result != CONTINUE for @noreply and @nocallback
decorators.  Remove redundant table in miltermodule (low level change).

<h3> 0.9.6 </h3>
Raise ValueError on unescaped '%' passed to setreply (setreply arg
is ultimately passed to printf by libmilter, usually resulting in a
coredump if it contains % escapes).

<h3> 0.9.5 </h3>
Print milter.error for invalid callback return type. (Since stacktrace is
empty, the TypeError exception is confusing.)  Fix milter-template.py.
It is not in the test suite and stopped working at some point - not
good for example code.

<h3> 0.9.4 </h3>
Handle IP6 in Milter.utils.iniplist().  Support (and require for 
RPM packages) python-2.6.

<h3> 0.9.3 </h3>
Handle source routes in Milter.util.parse_addr().
Fix unitialized optional arg in chgfrom().  Disable negotiate callback
when libmilter &lt; 8.14.3 (runtime API version 1.0.1)

<h3> 0.9.2 </h3>
Change result of @noreply callbacks to NOREPLY when so negotiated.
Cache callback negotiation.  Add new callback support: data,negotiate,unknown.
Auto-negotiate protocol steps.  Fix missing address of optional param to
addrcpt().

<h3> 0.9.0 </h3>
Spec file change for Fedora:
stop using INSTALLED_FILES to make Fedora happy,
remove config flag from start.sh glue,
own /var/log/milter,  use _localstatedir.
<p>

<code>pymilter-0.9.0</code> is the first version after separating 
<code>milter</code> and <code>pymilter</code>.  This will allow easier
reuse by other projects using pymilter to wrap libmilter.
In addition, we now support chgfrom and addrcpt_par in the milter API.
NS records are now supported by Milter.dns.  I suspect that it might
be useful to track reputation by nameserver to fight throwaway domains.

<h2> Recent Changes for the <code>milter</code> project </h2>

<h3> 0.8.18-2 </h3>
Default logdir to datadir for compatibility with old configs.
Add logdir to sample config.

<h3> 0.8.18-1 </h3>
<ul>
<li>test cases and bug fixes for spfmilter.
<li>configure untrapped_exception policy for spfmilter.
<li>reject numeric HELO for spfmilter.
<li>from_words from file feature for bms milter
<li> ban mailbox, not entire domain, for configured email_providers
<li> Use pysqlite (included in python) for greylist database
<li> banned_domains and ips moved back to logdir for bms milter
<li> straighten out datadir vs logdir for bms milter
</ul>

<h3> 0.8.17-2 </h3>
Include logrotate for dkim-milter.

<h3> 0.8.17-1 </h3>
<ul>
<li> report keysize of DKIM signatures.
<li> simple DKIM milter as another sample
<li> basic DKIM signing support
<li> Implement DKIM policy in access file.  
<li> Parse Authentication-Results header to get dkim result for feedback mail.  
<li> Let DKIM confirm domain for missing or neutral SPF result.
</ul>

<h3> 0.8.16 </h3>
Experimental DKIM support. Reference templated URLs in error messages.

<h3> 0.8.15 </h3>
Support (and require for RPM packages) Python2.6.

<h3> 0.8.14 </h3>
Ignore zero length keywords (from_words, porn_words) - a disastrous typo.
Ban generic domains for common subdomains.
Allow illegal HELO from internal network for braindead copiers.
Don't ban for multiple anonymous MFROM.
Trust localhost not to be a zombie - sendmail sends from queue on localhost.
Ban domains on best_guess pass.

<h3> 0.8.13 </h3> 
Default internal_policy to off.  Experimental banned domain list.
Block DSN from internal connections, except for listed internal MTAs.
BAN policy in access file bans connect IP.  Use DATA callback to improve SRS
check.  

<h3> 0.8.12 </h3>
Use the pid file in the initscript.  Fix bugs with greylisting config
and adjust demerits for HELO fail.  Add an SPF Pass policy.
Can be used to ban a domain.

<h3> 0.8.11 </h3>
Greylisting is now supported.  Messages from the 'vacation' program
are now recognized as autoreplies.  IPs of trusted relays (secondary
MXes, for instance) are never banned.  Added ban2zone.py to 
convert banned IP lists to BIND zonefile data.

<h3> 0.8.10 </h3>
SRS rejections now log the recipient.  
I have finally implemented plain CBV (no DSN).  The CBV policy
will do a plain CBV from now on, and the DSN policy is required
if you want to send a DSN.
I started checking the MAIL FROM fullname (human readable part
of an email) for porn keywords.  There is now a banned IP database.
IPs are banned for too many bad MAIL FROMs or RCPT TOs, and remain banned
for 7 days.

<h3> 0.8.9 </h3>
I use the <code>%ifarch</code> hack to build milter and milter-spf
packages as noarch, while pymilter is built as native.

I removed the spf dependency from dsn.py, so pymilter can be used without
installing pyspf, and added a Milter.dns module to let python milters do
general DNS lookups without loading pyspf.

<h3> 0.8.8 </h3>
Programs do not belong in the /var/log directory.  I moved the
milter apps to /usr/lib/pymilter.  Since having the programs and
data in the same directory is convenient for debugging, it will
still use an executable present in the datadir.

Several general utility classes and functions are now in the Milter package
for possible use by other python milters.  In addition to the trivial example
milter, a simple SPF only milter is included as a realistic example.

The spec file now build 3 RPMs:

<ul>
<li> pymilter is the milter module and Milter package for use by all python
     milters.
<li> milter is the all-singing, all-dancing python milter application, with
     supporting <code>/etc/init.d</code>, logrotate and other scripts.
<li> milter-spf is the simple SPF only milter application.
</ul>

<h3> 0.8.7 </h3>

The spf module has been moved to the 
<a href="http://cheeseshop.python.org/pypi/pyspf">pyspf</a> package.
Download <a href="http://sourceforge.net/project/showfiles.php?group_id=139894&amp;package_id=191419">here</a>.

<h3> 0.8.6 </h3>

Python milter has been moved to 
<a href="http://sourceforge.net/projects/pymilter/">pymilter Sourceforge
project</a> for development and release downloads.

<h3> 0.8.5 </h3>

Release 0.8.5 fixes some build bugs reported by Stephen Figgins.  It
fixes many small things, like not auto-whitelisting recipients of
outgoing mail when the subject contains "autoreply:".  There is a 
simple trusted forwarder implementation.  If you have more than
2 or so forwarders, we will need a way to "compile" SPF records into an
IP set and TTL for it to be efficient (like libspf2 does).

<h3> GOSSiP </h3>
An alpha release of <a href="pygossip.html">pygossip</a> has been commited to
CVS, module pygossip. A version of the bms.py milter has been commited to CVS
which supports calling GOSSiP to track domain reputation in a local database.

<h3> New website design </h3>

Hey, I'm no artist, so I just used the
<a href="http://ht2html.sourceforge.net/">ht2html</a> package
by <a href="http://barry.wooz.org/">Barry Warsaw</a>.  The mascot
is by <a href="http://alphard.ethz.ch/hafner/lebl.htm">Christian Hafner</a>,
or maybe his wife.  I chose Maxwell's daemon because it tirelessly
and invisibly sorts molecules, just as milters sort mail.
Christian has also provided a fun 
<a href="http://alphard.ethz.ch/hafner/PPS/PPS2002/Maxwell/simulation.htm">
simulation</a> that lets you try your hand at sorting molecules.

<h3> 0.8.4 </h3>

Release 0.8.4 makes configuring SPF policy via access.db actually work.
The honeypot idea is enhanced by auto-whitelisting recipients of
email sent from selected domains.  Whitelisted messages are then used
to train the honeypot.  This makes the honeypot screener entirely self
training.  The smfi_progress() API is now automatically supported when present.
An optional idx parameter to milter.addheader() invokes smfi_insheader().

<h3> 0.8.3 </h3>

Release 0.8.3 uses the standard logging module, and supports configuring
more detailed SPF policy via the sendmail access map.  SMTP AUTH connections
are considered INTERNAL.  Preventing forgery between internal domains is
just a matter of specifying the user-domain map - I'll define something
for the next version.  We now send DSNs when mail is quarantined (rejecting
if DSN fails) and for SPF syntax errors (PermError).  There is an
experimental option to add a Sender header when it is missing and the From
domain doesn't match the MAIL FROM domain.  Next release, we may start
renaming and replacing an existing Sender header when neither it nor the
From domain matches MAIL FROM.  Since bogus MAIL FROMs are rejected
(to varying degrees depending on the configured SPF policy), and
both Sender and From and displayed by default in many email clients,
this provides some phishing protection without rejecting mail based
on headers.

<h3> 0.8.2 </h3>

Release 0.8.2 has changes to <a href="http://openspf.net">SPF</a> to bring it
in line with the newly official RFC.  It adds 
<a href="http://ses.codeshare.ca/">SES</a>
support (the original SES without body hash) for pysrs-0.30.10, and honeypot
support for pydspam-1.1.9.  There is a new method in the base milter module.
milter.set_exception_policy(i) lets you choose a policy of CONTINUE, REJECT, or
TEMPFAIL (default) for untrapped exceptions encountered in a milter callback.

<h3> 0.8.0 </h3>

Release 0.8.0 is the first <a href="http://sourceforge.net/">Sourceforge</a>
release.  It supports Python-2.4, and provides an option to accept mail
that gets an SPF softfail or fails the 3 strikes rule, provided the
alleged sender accepts a DSN explaining the problem.  Python-2.3 is
no longer supported by the reworked mime.py module, although API changes
could be backported.  There are too many incompatible changes to the
python email package.

<h3> Older Releases </h3>

Release 0.7.2 tightens the authentication screws with a "3 strikes and
you're out" policy.  A sender must have a valid PTR, HELO, or SPF record
to send email.  Specific senders can be whitelisted using the
"delegate" option in the spf configuration section by adding a
default SPF record for them.  The PTR and HELO are required
by RFC anyway, so this is not an unreasonable requirement.
There is now a coherent policy for an SPF softfail result.  A softfail
is accepted if there is a valid PTR or HELO, or if the domain
is listed in the "accept_softfail" option of the spf configuration section.
A neutral result is accepted by default if there is a valid PTR or
HELO, (and the SPF record was not guessed), unless the domain is listed in the
"reject_neutral" option.  Common forms of PTR records for dynamic IPs are
recognized, and do not count as a valid PTR.  This does not prevent anyone
from sending mail from a dynamic IP - they just need to configure a
valid HELO name or publish an SPF record.
<p>
As SPF adoption continues to rise, forged spam is not getting through.  So
spammers are publishing their SPF records as predicted.  The 0.7.2 RPM
now provides the <code>rhsbl</code> sendmail hack so that spammer domains
can be blacklisted.  With the RPM installed, add a line like the following
to your <code>sendmail.mc</code>.
<pre>
HACK(rhsbl,`blackholes.example.com',"550 Rejected: " $&{RHS} " has been spamming our customers.")dnl
</pre>
<p>
Of course, spammers are now starting to register
throwaway domains.  The next thing we need is a custom DNS server,
in Python, that
can recognize patterns.  For instance, one spammer registers ded304.com,
ded305.com, ded306.com, etc.  We also need the custom DNS server to
let SPF classic clients check SES (which will be part of pysrs).  
The <a href="http://twistedmatrix.com/products/twisted">Twisted Python</a>
framework provides a custom DNS server - but I
would like a smaller implementation for our use.
<p>
The RPM for release 0.7.0 moves the config file and socket locations to
/etc/mail and /var/run/milter respectively.  We now parse Microsoft CID records
- but only hotmail.com uses them.  They seem to have applied for a patent on
the brilliant idea of examining the mail headers to see who the message is
from.  We aren't doing that here, so not to worry - but I am not a lawyer, so
if you are worried, change spf.py around line 626 to return None instead of
calling CIDParser().  There is a new option to reject mail with no PTR
and no SPF.
<p>
Microsoft is pushing an anti-opensource license for their pending patent
along with their sender-ID proposal before the IETF.
It is royalty free - but requires anyone distributing a binary they've
compiled from source to sign a license agreement.  The Apache Software
Foundation <a
href="http://www.apache.org/foundation/docs/sender-id-position.html"> explains
the problem with sender-ID</a>, and Debian <a
href="http://www.debian.org/News/2004/20040904">concurs</a>.  Since 
the <a href="http://download.microsoft.com/download/4/3/9/439b024b-09fd-44ee-8ff0-10e834004c36/senderid_FAQ.PDF">Microsoft license</a> is
<a href="http://www.circleid.com/article/732_0_1_0_C/">incompatible with free
software in general</a> and the <a
href="http://www.imc.org/ietf-mxcomp/mail-archive/msg03678.html">GPL in
particular</a>, Python milter will not be able to implement sender-ID in its
current form.  This was, no doubt, Microsoft's intent all along.
<p>
Sender-ID attempts to do for RFC2822 headers what SPF does for RFC2821 headers.
Unlike SPF, it has never been tried, and is encumbered by a stupid patent.  I
recommend ignoring it and continuing to implement and improve SPF until a
working and unencumbered proposal for RFC2822 headers surfaces.

<p>
<a href="http://openspf.com">
<img src="art/SPF.gif" align=left alt="SPF logo"></a>
Release 0.6.6 adds support for <a href="http://openspf.com/">SPF</a>,
a protocol to prevent forging of the envelope from address.  
SPF support requires <a href="http://pydns.sourceforge.net/">pydns</a>.
The included spf.py module is an updated version of the original 1.6
version at <a href="http://www.wayforward.net/spf/">wayforward.net</a>.
The updated version tracks the draft RFC and test suite.
<p>
The FAQ addresses <a href="faq.html#spf">how to get started with SPF</a>.
<p>
Release 0.6.1 adds a full milter based dspam application.
<p>
I have selected the <a href="http://www.nuclearelephant.com/projects/dspam/">
dspam bayes filter project</a> and <a href="dspam.html">
packaged it for python</a>.
Release 0.6.0 offers a simple application of dspam I call "header triage",
which rejects messages with spammy headers.  
To use header triage, you must have <a href="dspam.html">DSPAM</a> installed,
and select a dictionary that is well moderated by someone who gets
lots of spam.  That dictionary can be used to block spam that is 
obvious from the headers (e.g. X-Mailer and Subject) before it ties
up any more resources.  I have yet to see any false positives from this
approach (check the milter log), but if there are, the sender will
get a REJECT with the message "Your message looks spammy."

