The very short version: if you contact an open source project anonymously, you may not get the best help. Feel free to also reach out privately and share that your post from gmail.com is actually (say) from a very interesting deployment. Now read on for the long version!
Some months ago, the fine people from CloudFlare blogged about their new DNS implementation, and in that post they noted:
“While PowerDNS got us a long way, it started to run into issues as we scaled and dealt with an increasing number of large denial of service attacks … To their credit, the PowerDNS community responded to the first two problems with some efforts at rate limiting and other abuse detection”.
For us, this represented a sadly familiar pattern. We had lost CloudFlare as a user, and we had not been able to work with them sufficiently well to address their needs. How did it happen, and why does it matter?
For an open source project, it is important to have happy ‘lighthouse’ users. For years, we’ve proudly served the geodirection needs of Wikipedia for example. We’ve done a lot of work to keep various big users happy with their PowerDNS deployment. If people know or find out your product is being used by ‘name brand’, large scale, deployments, this helps tremendously with adoption and acceptability.
In fact, next to development resources, a great community and funding, having impressive deployments is one of the most important things for an open source project.
So when we lost CloudFlare, other users contacted us to ask what had gone wrong, and if PowerDNS was still suitable for their needs. We quickly reached out to CloudFlare, and Matthew Prince and co-workers sent us an impressive post-mortem on what had happened. They also graciously gave us permission to share the story, which is much appreciated.
It turns out they actually had shared their issues with the PowerDNS community, and they had gotten some help from us.. but not enough (by our own estimation). Importantly, they had not declared themselves as being a large or impressive deployment.
Now, we try to help everybody of course. We strive to be the friendliest name server community out there, and I think we are succeeding. But we can’t devote infinite resources to everyone. Major users, major customers, interesting deployments need and get more attention. (It should be noted that we are also spoilers for users politely asking for help and willing to run test versions of our software, by the way. Nearly infinite help awaits you in that case!)
But, back to the subject of this post, here’s the problem. Large or interesting deployments generally contact open source projects anonymously, mostly from gmail accounts, often even using fake names. And this is to be understood – in most large places (enterprise, public companies, government), sending out email to a world readable mailing list from a work email account is a sure way to get unwanted attention within your organisation (and get laughed at for your ridiculous multipage disclaimer).
Legal departments are likely to get their panties in a bunch – did you just share proprietary information? Security departments might ask if it was wise to publicly post company implementation details. Even the communication & marketing people might get in on the act. So email from gmail.com is what we get.
But this effectively means you might have a 500 server PowerDNS deployment doing really interesting things, things we’d love to help you with.. but we don’t know. Not only is this a matter of scale, we may also not realise the nature of your challenge. If we know what you are doing, our thinking about your problem may run among different lines.
Another reason why employees at big corporations are loath to contact open source projects is that they assume they won’t get help, and need to pay for support. And, while it is true that someone needs to pay the bills here, we realise that for many users, a support agreement is just not going to happen. Corporate IT might not even know the whole company is relying on an open source project. Questions might be asked. For such organisations, we are fine with providing free help on the public mailing lists. But to give proper weight and context to your issue, it helps if we know who you are.
So what about PowerDNS and CloudFlare? From our conversation, we now fully understand why they wrote their own server. But if at the time we had known the scale of their challenge, we would have loved to have tried to meet their needs. We do very much appreciate their help in the ‘post mortem’ of our silent breakup. As Matthew said “[PowerDNS] is a great piece of software that met our needs well for almost 4 years. Today our needs are extremely unique and it wouldn’t make a lot of sense for you to try and design for them.”
Concluding, on behalf of PowerDNS, and I expect a lot of open source projects: If you are a large or interesting deployment, please do post to the mailing lists. We do understand you will not be doing so from your corporate email account. But please do feel free to in parallel contact the project privately (IRC channels are typically a great way), where you can share more details of who you are and what you are doing.
Both you and the open source project will probably end up better that way.
