I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days.
The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.
I did a quick scan of rafa.eu.org and found the following:
- SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.
- A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.
- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).
- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this.
So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.
No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail.
And if you read the rest of that mailop thread, you'll see that the broader IP reputation of the /24 where your IP is located is a key part of whether or not your email gets delivered.
I would venture to guess that a substantial portion of people reporting issues delivering at small scale are sending from a low-reputation netblock. (Assuming auth is all set up optimally.) So it's not that it's not possible to send at low volume, it's just that most people don't know to look at the netblock, it's hard to evaluate netblock reputation, and it's not always possible to select a preferred netblock within a given provider (especially VPS providers with low-cost options).
The bottom line is: it's hard to run your own mail server, and much of the reason for the success of services like Sendgrid is a) it's easier to set up auth properly with most of them, and b) they actively manage the email reputation of their netblocks (though some do this better than others).
If you're approaching things from a practical perspective instead of making this your hill to die on, you just use an email service provider, and for the vast majority of senders, life is a little bit easier. There's plenty of ESPs to pick from, and there's a very good chance you can find a free one that delivers better than an average VPS or IP of a residential ISP.
My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.
Doing all of that from a popular low-cost hosting provider like Hetzner is still insufficient to be able to send emails to new contacts. You still end up in spam.
Same here. I also installed https://mailinabox.email on a server and can send email to gmail recipients without trouble. I had some trouble with one of my domains a couple of years ago but it was only related to misconfiguration on my side.
How are you verifying this? I encourage you to try sending to a gmail address you have never sent to before, I virtually guarantee you are going to end up spamboxed.
I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time.
Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.
I have verified it by helping people sign up for the small service I run and seeing the e-mails pop in their primary inbox. I guess I might be lucky then shrug.
If you would have actually read the link in the post you're replying to, you would know that Gmail is not accepting mail from me despite that I've spent a huge amount of time in verifying that everything is configured correctly.
Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.
There's plenty of low effort ways you could provide proof while maintaining relative anonymity (if you actually wanted to support your claims with evidence). For example, you can run a GlockApps test (5 minutes, free) and then take a screenshot of the results with identifying information blanked out.
Because OP talked about Google not allowing small players to send e-mail here's my server's deliverability results to Google: https://i.imgur.com/sceZogI.png
If you're wondering about the blacklist number, that's an error - the source site claims it knows nothing.
> Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.
I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim -- I can understand that they often block legitimate mail like in your case, but that doesn't mean that everyone has the same issue. I have occasional issues sending to Gmail but most of the time they accept it and don't mark it spam.
>I'm sure most people who run low-volume mail servers here have no problems sending to Gmail.
Based on direct experience and >testing< (I have talked to about 20 people with personal mail servers about this, and got them to attempt to send me email, on an account that had never interacted with their domain before), 100% of their emails were spam boxed.
Lots of people test gmail with their personal accounts, and think that email to gmail in general is delivered.
In my experience, it basically never is if you are a small sender.
> I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim
Where are all these magical people with these magical abilities? How curious that not a single one of them is able to post any kind of evidence to support these incredulous claims. And funny how my comments on HN are getting downvoted and hidden when I ask for proof.
Sounds like there's a niche business and/or OSS project there. Create gmail accounts. Install email send bot on email server. Send messages to Gmail accounts. Figure out how to avoid spam.
Its inaccurate to say that SPF, DMARC and DKIM setup this way is a "misconfiguration", It still allows anybody you are mailing to verify that email is from your mail server, and your domain, which is good enough for anti-spam purposes.
Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.
In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam.
> Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.
Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.
An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc.
> The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough.
A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does.
The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here.
> I did a quick scan of rafa.eu.org and found the following:
All the issues you list are non-issues, and merely the perpetuation of the security theatre.
SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either.
Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected.
Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.
The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.
I did a quick scan of rafa.eu.org and found the following:
- SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.
- A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.
- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).
- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this.
So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.
[0] https://www.mailhardener.com
[1] https://www.mailhardener.com/tools/spf-validator?domain=rafa...
[2] https://www.mailhardener.com/tools/dmarc-validator?domain=ra...
[3] https://testssl.sh/