The Sender Policy Framework (SPF)
source link: https://www.netmeister.org/blog/spf.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
The Sender Policy Framework (SPF)
August 30th, 2022
If you ever feel like you understand something, something that you think is simple, I recommend that you go ahead and read the RFC and you'll find out whoo boy, there's more to it than you thought. And then you implement it and find out all the ways in which people are violating the RFC...
Take the Sender Policy Framework (SPF), for example. Seems pretty straight forward, right? If you're, say, a Mail Transfer Agent (MTA) for a popular public email service, and a new connection is made to you trying to deliver mail, you perform a few checks and, by way of SPF, determine whether or not you should even bother to accept the mail:
$ ifconfig xennet0 | sed -n -e 's/.*inet \(.*\)\/.*/\1/p'
166.84.7.99
$ telnet $(dig +short mx yahoo.com | awk '{print $2; exit;}') 25
Trying 67.195.204.72...
Connected to mta5.am0.yahoodns.net.
Escape character is '^]'.
220 mtaproxy312.free.mail.bf1.yahoo.com ESMTP ready
helo localhost
250 mtaproxy312.free.mail.bf1.yahoo.com
mail from: <[email protected]>
250 sender <[email protected]> ok
rcpt to: <[email protected]>
250 recipient <[email protected]> ok
data
354 go ahead
From: <[email protected]>
To: <[email protected]>
Subject: this should fail SPF
spf fail?
.
554 5.7.9 Message not accepted for policy reasons.
See https://postmaster.yahooinc.com/error-codes quit
221 2.0.0 Bye
Connection closed by foreign host.
Yahoo's mail server rejected this mail after having checked whether the sender coming from 166.84.7.99 is authorized to send mail on behalf of microsoft.com. It did that (in part) by looking up the SPF policy for microsoft.com:
$ dig +short txt microsoft.com | grep spf
"v=spf1 include:_spf-a.microsoft.com include:_spf-b.microsoft.com
include:_spf-c.microsoft.com include:_spf-ssg-a.microsoft.com
include:spf-a.hotmail.com include:_spf1-meo.microsoft.com -all"
$ Since the sending IP address does not match, and since the policy ends in a "fail" directive (-all), Yahoo rejects the mail.1 (For a slightly more verbose example including DMARC, please see this video.)
If the SPF policy had allowed the mail to be delivered, then your SMTP Authorization-Results headers might display the desired results:
Authentication-Results: receiving-server
spf=pass [email protected];
dkim=pass header.d=netmeister.org header.s=2022;
dmarc=pass header.from=netmeister.org
Received: from panix.netmeister.org (panix.netmeister.org [166.84.7.99])
by receiving-server with ESMTPS id whatever
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256)
So far, so good. But now let's take a look at the syntax of the SPF policy as specified in RFC7208. It defines the qualifiers and mechanisms that may occur in the policy, and for the most part, that seems all pretty straight forward:
The Basics
You mostly encounter a, mx, ip4, ip6, and include mechanisms, ending with an all, each optionally prefixed with a qualifier (one of +?~-). For example, the most trivial SPF policy might be:
$ dig +short txt netmeister.org "v=spf1 a -all" $
The + qualifier for each mechanism is implicit, so this policy simply states that the IP address of the domain netmeister.org is allowed to send mail (a), and nobody else (-all). What about the IPv6 address of netmeister.org, you ask? Ok, we allow that one, too, per the a directive, because in this context here "a" doesn't mean "A record", but "either A or AAAA record". Yay!
But you can also designate a different set of IP addresses using the a mechanism, as the policy allows this syntax:
v=spf1 a:example.com -all
A domain with that policy would allow all IP addresses of example.com to send mail, and nobody else. Or you could invert it, because the use of qualifiers for each mechanism (not just all) provides for some flexibility. So we could, for example, say "anybody in the world can send mail on our behalf, except for the IP addresses of example.com":
v=spf1 -a:example.com all
Similarly, you can specify the MX records of either the domain the policy applies to (mx) or any other domain (mx:example.com). Only... a DNS MX lookup will yield any number of hostnames, so your mail server will have to then perform another lookup for those names to get an IP address for each.
Allowing specific IP addresses individually can be done using the ip4 / ip6 mechanisms, and rather conveniently you can also use those to allow (or deny, softfail, or be neutral about) entire CIDRs. This policy...
v=spf1 ?a:example.com ~mx:example.net -ip4:192.0.2.0/22 ip6:2001:db8::f351:bd9:42ff:65e2 -all
...marks the IP addresses of example.com as "neutral", the IP addresses of whatever names the MX lookup of example.net yields as "softfail", the 1024 IP addresses from 192.0.2.0/22 as "fail", the single IPv6 address 2001:db8::f351:bd9:42ff:65e2 as "pass", and then denies all others.
So that's useful: you'd commonly find domains allowlisting their usual IP space via a few CIDRs and deny the rest. What else is there?
A and MX can also use a CIDR length!
Wait, what? How does that work?
Both the A and MX mechanisms can, in addition to taking domain, also accept a CIDR length:
$ dig +short txt example.com "v=spf1 a a:example.org a/24 mx:example.net/16//64"
With this mechanism, you can then expand the scope of the resolved domain to a larger network, such as the /16 of all MX hosts for example.net. (This approach does not seem to be very widely used, however.)
Enter recursion!
We've already seen the include directive up there in Microsoft's SPF policy. And this is where the fun really begins. On the one hand, the mechanism is convenient, because you can easily delegate who is authorized to send mail on your behalf without having to manage other people's records or networks.
So any domain that uses e.g., Google to send mail can then include:_spf.google.com and call it a day. When the client connects to send mail, the mail server will then perform a lookup of _spf.google.com and do what that says:
$ dig +short txt example.com
"v=spf1 include:_spf.google.com -all
$ dig +short txt _spf.google.com
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"
$ dig +short txt _netblocks.google.com
"v=spf1 ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20
ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16
ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19 ~all"
$ dig +short txt _netblocks2.google.com
"v=spf1 ip6:2001:4860:4000::/36 ip6:2404:6800:4000::/36 ip6:2607:f8b0:4000::/36
ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all"
$ dig +short txt _netblocks3.google.com
"v=spf1 ip4:172.217.0.0/19 ip4:172.217.32.0/20 ip4:172.217.128.0/19 ip4:172.217.160.0/20
ip4:172.217.192.0/19 ip4:172.253.56.0/21 ip4:172.253.112.0/20 ip4:108.177.96.0/19
ip4:35.191.0.0/16 ip4:130.211.0.0/22 ~all"
$ As you can see here, a domain's SPF policy may include multiple other policies, which in turn may include yet other policies. So that of course yields the possibility of infinite recursion:
$ dig +short txt spfloop.dns.netmeister.org "v=spf1 include:spfloop.dns.netmeister.org -all" $
For this reason (as well to overall limit the burden of performing DNS lookups), RFC7208 requires servers to implement DNS lookup limits -- in practice, no more than 10 lookups should be performed; exceeding that limit MUST yield a permerror, meaning the policy is not evaluated. And it turns out that a number of SPF tools in use get this wrong.
Math is hard
Let's consider the following example:
$ dig +short txt example.com "v=spf1 mx include:_spf.example.com -include:example.org a -all" $ dig +short mx example.com 10 mx1.example.com 20 mx2.example.com 30 mx3.example.com $ host mx1.example.com mx1.example.com has address 192.0.2.227 mx1.example.com has IPv6 address 2001:db8::d723:cf25:bfd:b0e0 $ host mx2.example.com mx2.example.com has address 203.0.113.142 mx2.example.com has address 203.0.113.143 $ host -t a mx3.example.com mx3.example.com has no A record $ host -t aaaa mx3.example.com mx3.example.com has no AAAA record $ dig +short txt _spf.example.com "v=spf1 ~include:example.net ~all" $ dig +short txt example.net "v=spf1 +ip4:198.51.100.0/24" $ dig +short txt example.org "v=spf1 +ip4:198.51.100.227" $ dig +short a example.com 198.51.100.227 $ dig +short aaaa example.com 2001:db8::56b5:141b:acdc:590b $
There's a lot going on in this example, but let's start by looking at how many DNS lookups are performed here. First we perform the MX lookup (total: 1), which yields three names. Then we need to perform one lookup for each of these names (total: 4). Next, we perform a TXT lookup for _spf.example.com (total: 5). That policy has another include mechanism, so we have to make another TXT query for example.net (total: 6). We then also need to look up example.org (total: 7), and finally lookup the A / AAAA records for example.com itself per the a mechanism. That yields a total of 8 DNS lookups.
Wait... 8? Why not... 12, if we lookup both A and the AAAA addresses for the a directive and mx results? Well, turns out we don't really need to perform both lookups: we know whether the client connected using IPv4 or IPv6, so we can eliminate one of these lookups and remain at 8. However, many SPF validators appear to not count the lookups necessary to turn the MX results into IP addresses at all and count only 5 lookups.
This is problematic when your SPF policy yields 10 DNS lookups when counted incorrectly, but more than that when counting MX resolution correctly. In addition, if your domain (or one of the domains you included) contains an MX directive that yields more than 10 results, this is considered a permanent error. In either case, your SPF policy just became invalid and is ignored in its entirety.
Inconceivable include
include - we keep using that word. I do not think it means what you think it means. In fact, the RFC itself notes:
In hindsight, the name "include" was poorly chosen. Only the evaluated result of the referenced SPF record is used, rather than literally including the mechanisms of the referenced record in the first.
This has a number of rather unobvious consequences. Let us again consider our contrived example from above, with a client connecting from 198.51.100.227. Would that host be allowed to send email on behalf of example.com?
| Number | Mechanism | Result |
| 1. | mx | no-match |
| 2. | include:_spf.example.com
~include:example.net ~all ip4:198.51.100.0/24 |
no-match softfail pass |
| 3. | -include:example.org
+ip4:198.51.100.227 |
fail pass |
| 4. | a | pass |
So which one is it?
198.51.100.227 is not one of the addresses of any of the MX records for example.com, so 1. doesn't match. That's straight forward. But then:
198.51.100.227 falls into the 198.51.100.0/24 CIDR, so example.net's policy evaluates to "pass", which means that the ~include:example.net mechanism matches and in turn evaluates to "softfail". If that was all, we'd stick with the "softfail", but this was inside another "include", and a "softfail" from an include evaluates to "no match", meaning include:_spf.example.com will not evaluate to "pass" nor "softfail" as you might expect. So 2. doesn't match, and we move on.
198.51.100.227 in example.org's policy evaluates as +ip4:198.51.100.227 as "pass", thus becoming a "match" for the "include", but the mechanism was "-include", and so because we matched, we evaluate to "fail" in 3..
4. would evaluate to "pass", since 198.51.100.227 is indeed the A record for example.com, but we never evaluate that mechanism, since our previous mechanism already evaluated to "fail". So our final result will indeed be "fail".
Oh, and here's another thing that may not be obvious: any all statement in an include mechanism does not actually get included in your policy and only applies within the context of that domain's policy. That is, the following does not do what you might hope:
$ dig +short txt example.com "v=spf1 include:example.net" $ dig +short txt example.net "v=spf1 a -all" $
You might think that this translate to a:example.net -all, but since example.com is missing an all directive itself, this evaluates effectively to a:example.net +all overall.
Look over there!
Ok, so include doesn't actually "include" the policy, but fortunately for you, RFC7208 lets you do something else. It allows for a redirect modifier:
$ dig +short txt example.com "v=spf1 redirect=example.net" $ dig +short txt example.net "v=spf1 a -all" $
This does evaluate to a:example.net -all for example.com. But redirect only applies if all other mechanisms (anywhere in the policy, regardless of order) fail to match, and are completely ignored if an all directive is present (since that necessarily always matches):
$ dig +short txt example.com "v=spf1 a redirect=example.net ~all" $ dig +short txt example.net "v=spf1 -ip6:2001:db8::/32" $
Here, the IP address for example.com is permitted, and all others are marked as "softfail", with the policy from example.net being completely ignored.
Similarly, any mechanism following an all mechanism are ignored as well, and having one that is not at the end is in all likelihood a misconfiguration. In the following example, any attempts to send mail from 2001:db8::/32 will be marked "fail" since the "include" mechanism is never evaluated:
$ dig +short txt example.com "v=spf1 a mx ip4:198.51.100.0/24 -all include:_spf.example.com" $ dig +short txt _spf.example.com "v=spf1 ip6:2001:db8::/32" $
Reverse lookups via PTR
In addition to the above mechanisms, RFC7208 also include the PTR mechanism, which it then promptly recommend against using. It performs a reverse lookup match of the IP address against the specified domain:
$ dig +short txt example.com "v=spf1 ptr:example.com -all" $
The idea here might be that we'd like to allow any IP address that reverses into your domain. Since anybody can trivially add any entry they like into the reverse in-addr.arpa / ip6.arpa zone that's delegated to them based on their IP space allocation, the mail server then also needs to perform a forward lookup of the resulting PTR record, which may yield any number of IP addresses, which it then needs to match again against the client IP address.
(For the same reason, a DNS lookup failure for PTR yields a "no-match", while any other DNS failures would yield a "temperror", thereby possibly terminating the process and (temporarily) rejecting the message.)
Macros
If the above complexity isn't enough for you yet, hold on to your butts, because RFC7208 has another surprise for you: any domain name in use in any of the mechanisms or modifiers can be expanded using specific macros, whereby "%{}" formatted strings are replaced with e.g., the sender's domain name, the sender's IP address, the SMTP HELO/EHLO domain, and so on.
This lets you do all sorts of wild things such as implementing dynamic IP reputation lookups via DNSBLs using either the include or the exists mechanism (e.g., exists:%{ir}._dnsbl.%{d} would be expanded to the <reverse IP address of sender>._dnsbl.<domain>; the exists mechanism matches if the lookup returns an A record (not a AAAA record, regardless of whether the client connected via IPv4 or IPv6!).
You can also use this to add very flexible and customized error messages using the exp= modifier, which allows for the generation of an explanation string to accompany a "fail" result. This string is, of course, determined via yet another DNS TXT lookup, the result of which is also macro expanded:
$ dig +short txt example.com
"v=spf1 mx -all exp=explain._spf.%{d}"
$ dig +short txt explain._spf.example.com
"%{i} is not one of %{d}'s designated mail servers."
$ Size matters
Ok, so with all these capabilities, you'll find that many SPF policies grow to include many domains and IP CIDRs. With the number of DNS lookups restricted, you might be tempted to exhaustively list large numbers of IPs or CIDRs explicitly, but then you risk running into the DNS size limitations for UDP packets, risking failing over to TCP.
RFC7208 recommends to fit your SPF policy into under 450 octets -- another restriction not many SPF validators monitor. But DNS record size is not the only concern; you may also be concerned about the sheer number of IP addresses you (inadvertendly) grant permission to send mail on your behalf.
Now there really is no restriction on how many CIDRs or IPs you permit (or deny) via your SPF policy, but I was curious just what common domains allow. To better get an idea what this looks like, I wrote a a small tool to expand a domain's SPF policy. Its output looks like this:
$ spf youtube.com
youtube.com:
policy:
include:google.com mx -all
valid
pass:
include (1 domain):
google.com
mx (1 name):
smtp.google.com
mx (9 IPs):
142.251.163.27
172.253.115.26
172.253.115.27
172.253.122.26
172.253.122.27
2607:f8b0:4004:c06::1a
2607:f8b0:4004:c06::1b
2607:f8b0:4004:c09::1a
2607:f8b0:4004:c09::1b
google.com:
policy:
include:_spf.google.com ~all
valid
pass:
include (1 domain):
_spf.google.com
_spf.google.com:
policy:
include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all
valid
pass:
include (3 domains):
_netblocks.google.com
_netblocks2.google.com
_netblocks3.google.com
_netblocks.google.com:
policy:
ip4:35.190.247.0/24 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16 ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19 ~all
valid
pass:
ip4 (11 CIDRs / 215296 IPs):
108.177.8.0/21
173.194.0.0/16
209.85.128.0/17
216.239.32.0/19
216.58.192.0/19
35.190.247.0/24
64.233.160.0/19
66.102.0.0/20
66.249.80.0/20
72.14.192.0/18
74.125.0.0/16
All others: softfail
_netblocks2.google.com:
policy:
ip6:2001:4860:4000::/36 ip6:2404:6800:4000::/36 ip6:2607:f8b0:4000::/36 ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all
valid
pass:
ip6 (6 CIDRs / 2.97105609428491e+28 IPs):
2001:4860:4000::/36
2404:6800:4000::/36
2607:f8b0:4000::/36
2800:3f0:4000::/36
2a00:1450:4000::/36
2c0f:fb50:4000::/36
All others: softfail
_netblocks3.google.com:
policy:
ip4:172.217.0.0/19 ip4:172.217.32.0/20 ip4:172.217.128.0/19 ip4:172.217.160.0/20 ip4:172.217.192.0/19 ip4:172.253.56.0/21 ip4:172.253.112.0/20 ip4:108.177.96.0/19 ip4:35.191.0.0/16 ip4:130.211.0.0/22 ~all
valid
pass:
ip4 (10 CIDRs / 113664 IPs):
108.177.96.0/19
130.211.0.0/22
172.217.0.0/19
172.217.128.0/19
172.217.160.0/20
172.217.192.0/19
172.217.32.0/20
172.253.112.0/20
172.253.56.0/21
35.191.0.0/16
All others: softfail
All others: softfail
All others: softfail
All others: fail
SPF record for domain 'youtube.com': valid
Total counts:
Total number of DNS lookups : 7
pass:
Total # of include directives : 5
Total # of mx directives : 1
Total # of ip4 CIDRs : 21
Total # of ip4 addresses : 328965
Total # of ip6 CIDRs : 6
Total # of ip6 addresses : 2.97105609428491e+28
All others: fail
As you can tell, this tool shows you a bit of the complexity as it expands the domain's policy. The very simple include:google.com mx -all policy ended up incurring 7 DNS lookups and allows almost 330,000 IPv4 addresses from a total of 21 CIDRs.
Let's look at some domains...
I then took a look at the Alexa Top Internet sites which... oh, right, that list doesn't exist anymore. But fortunately there are alternatives. Anyway, so I took the top 100K domains, and ran my tool against each. (If you want to redo this exercise, I recommend setting up a local caching resolver first, as your ISP might not appreciate your sudden spike in DNS traffic; otherwise, if you're interested, you can download the data I collected from here; this 33 MB compressed tarball extracts 1 GB of data in two collections: plain text output and json.)
Some fun findings (all out of this one-time sample of 100K domains, so, you know, grains of salt etc.):
- 29809 domains have no SPF policy published
- 518 domains yielded a SERVFAIL error when looking up the domain
- many domains have an invalid SPF policy; some common reasons:
- 8556 domains have > 10 DNS lookups (highest number: 77, atcc.org (which includes an MX record with 59 names); 50 heinemann.com)
- 657 domains have typos or other errors, for example:
- zdnet.com accidentally glued an ip4 and an include directive together: ip4:216.239.114.222/23include:_spf.google.com
- simplecloud.ru includes itself
- gjsentinel.com tries to 'include' an IP: include:104.131.184.230
- spf05.xaas.jp tries to lookup a domain using ip4:aspmx.googlemail.com
- many domains have a problematic SPF policy:
- 2083 domains have policies > 450 octets in size (largest record: _netblocks.temple.edu with 8404 bytes)
- 251 domains have mechanisms following an all directive that are thus ignored
- 1780 domains use an mx directive without having MX records for that domain
- 87 domains use a redirect= modifier that's being ignored due to an all mechanism being present
- 1157 domains use an include mechanism where the included domain does not have a TXT record or no SPF policy published
- What defaults to domains fall through?
- 2438 domains block all senders (i.e., the full policy is v=spf1 -all)
- 45 domains' policies end in explicit 'pass' (+all), e.g., ubuntu.com
- 2408 domains' policies end in implicit 'neutral' (no all)
- 3147 domains' policies end in explicit "neutral" (?all)
- 27473 domains' policies end in "fail" (-all)
- 36682 domains' policies end in "softfail" (~all)
- 8352 domains' policies make use of macros (7085 via exists, 1240 via include, 20 via a, 7 via mx)
- 54 domains make use of exp= (some included domains also have an exp= modifier, but that is ignored)
- the most frequently included policies are:
- 13601 spf.protection.outlook.com
- (13257 spfd.protection.outlook.com)
- 13053 _spf.google.com
- 13390 _netblocks.google.com
- 13367 _netblocks2.google.com
- 13363 _netblocks3.google.com
- 5504 amazonses.com
- 4500 sendgrid.net
- 4074 mail.zendesk.com
- 3800 servers.mcsv.net
- 3634 spf.mandrillapp.com
- 3042 _spf.salesforce.com
- 13601 spf.protection.outlook.com
- the largest number of IPv4 addresses permitted is:
- 4294967296 for ccc.de
- 2148480115 polycom.com
- 1074539351 aba.com
- the largest number of IPv4 CIDRs permitted is 648 (fujifilm.com)
Summary
Well, there you have it. As I had promised at the beginning, nothing eviscerates your illusion of understanding a protocol or mechanism like actually reading the RFC and perhaps building a tool following the spec.
While SPF seems really straight forward, there are a number of surprises lurking. A few that I thought were particularly interesting (or worth mentioning because they might not be obvious) are:
- a missing all directive implies a "neutral" evaluation
- a performs A or AAAA lookups depending on the client IP version
- a and mx can take an additional domain
- a and mx can take a CIDR length
- an include that evaluates to a "softfail" becomes a "no match"
- all statements inside an include mechanism do not carry over into the parent
- exists will fail if the domain only returns AAAA records
- order matters, except for e.g., redirect= modifiers
- SPF records SHOULD remain under 450 octets (see also: DNS Response Sizes)
- SPF evaluation is subject to DNS lookups limits, and many online services at least get that wrong
- SPF evaluation can only succeed at the ingress MTA; any intermediate server will necessarily fail SPF validation -- this may require you to set disable SPF hard fail on the next hop
It's also worth pointing out that the use of TXT records is far from optimal: SPF policies are often set on the second-level domain, which also tends to be used for a gazillion other purposes, causing a lot of useless data to be returned to the mail server. A dedicated DNS resource record might make sense, but of course at this point the TXT record is already in use everywhere.
And finally, you should consider setting an SPF policy not only on your primary domain, but on your various other domains you own as well: attackers will happily use yourdomain.net to phish your employees when yourdomain.com is SPF protected.
In fact, I generally recommend a number of default settings for your entire second-level domain inventory and would suggest you include v=spf1 -all in your default domain template.
August 30th, 2022
[1] This isn't the whole truth: Yahoo rejects the mail based on it honoring/enforcing Microsoft's DMARC policy, which a failing SPF contributes to here.
SPF without DMARC only gets you so far. Blocking mail purely on SPF without a DMARC policy is risky from an email provider's point of view. There's a lot more to cover here, but this blog post already ballooned well beyond it's anticipated word count. Perhaps another time I'll build on this and cover DMARC.
Links:
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK