How to talk to an open source software project as a large scale or otherwise interesting user

The very short version: if you contact an open source project anonymously, you may not get the best help. Feel free to also reach out privately and share that your post from gmail.com is actually (say) from a very interesting deployment. Now read on for the long version!

Some months ago, the fine people from CloudFlare blogged about their new DNS implementation, and in that post they noted:

“While PowerDNS got us a long way, it started to run into issues as we scaled and dealt with an increasing number of large denial of service attacks … To their credit, the PowerDNS community responded to the first two problems with some efforts at rate limiting and other abuse detection”.

For us, this represented a sadly familiar pattern. We had lost CloudFlare as a user, and we had not been able to work with them sufficiently well to address their needs. How did it happen, and why does it matter?

For an open source project, it is important to have happy ‘lighthouse’ users. For years, we’ve proudly served the geodirection needs of Wikipedia for example. We’ve done a lot of work to keep various big users happy with their PowerDNS deployment. If people know or find out your product is being used by ‘name brand’, large scale, deployments, this helps tremendously with adoption and acceptability.

In fact, next to development resources, a great community and funding, having impressive deployments is one of the most important things for an open source project.

So when we lost CloudFlare, other users contacted us to ask what had gone wrong, and if PowerDNS was still suitable for their needs. We quickly reached out to CloudFlare, and Matthew Prince and co-workers sent us an impressive post-mortem on what had happened. They also graciously gave us permission to share the story, which is much appreciated.

It turns out they actually had shared their issues with the PowerDNS community, and they had gotten some help from us.. but not enough (by our own estimation). Importantly, they had not declared themselves as being a large or impressive deployment.

Now, we try to help everybody of course. We strive to be the friendliest name server community out there, and I think we are succeeding. But we can’t devote infinite resources to everyone. Major users, major customers, interesting deployments need and get more attention. (It should be noted that we are also spoilers for users politely asking for help and willing to run test versions of our software, by the way. Nearly infinite help awaits you in that case!)

But, back to the subject of this post, here’s the problem. Large or interesting deployments generally contact open source projects anonymously, mostly from gmail accounts, often even using fake names. And this is to be understood – in most large places (enterprise, public companies, government), sending out email to a world readable mailing list from a work email account is a sure way to get unwanted attention within your organisation (and get laughed at for your ridiculous multipage disclaimer).

Legal departments are likely to get their panties in a bunch – did you just share proprietary information? Security departments might ask if it was wise to publicly post company implementation details. Even the communication & marketing people might get in on the act. So email from gmail.com is what we get.

But this effectively means you might have a 500 server PowerDNS deployment doing really interesting things, things we’d love to help you with.. but we don’t know. Not only is this a matter of scale, we may also not realise the nature of your challenge. If we know what you are doing, our thinking about your problem may run among different lines.

Another reason why employees at big corporations are loath to contact open source projects is that they assume they won’t get help, and need to pay for support. And, while it is true that someone needs to pay the bills here, we realise that for many users, a support agreement is just not going to happen. Corporate IT might not even know the whole company is relying on an open source project. Questions might be asked. For such organisations, we are fine with providing free help on the public mailing lists. But to give proper weight and context to your issue, it helps if we know who you are.

So what about PowerDNS and CloudFlare? From our conversation, we now fully understand why they wrote their own server. But if at the time we had known the scale of their challenge, we would have loved to have tried to meet their needs. We do very much appreciate their help in the ‘post mortem’ of our silent breakup. As Matthew said “[PowerDNS] is a great piece of software that met our needs well for almost 4 years. Today our needs are extremely unique and it wouldn’t make a lot of sense for you to try and design for them.”

Concluding, on behalf of PowerDNS, and I expect a lot of open source projects: If you are a large or interesting deployment, please do post to the mailing lists. We do understand you will not be doing so from your corporate email account. But please do feel free to in parallel contact the project privately (IRC channels are typically a great way), where you can share more details of who you are and what you are doing.

Both you and the open source project will probably end up better that way.

10 comments

  1. jwheeler79

    basically, it sounds like you’re placing the blame on cloudflare for not yelling at you loud enough because, hey, you do facebook, ya know, and you’ve got big priorities. well, now you have one less.

  2. Joe Martinez

    I think the prior comment is way off base. Running a large OSS project, you get hundreds nay thousands of requests, comments, etc. You prioritize them by the value you can deliver. In this case, these issues that affect thousands were not at the top of the stack because the source was not clearly communicated.

    • Justin Alan Ryan

      I have interfaced with large open-source projects from a seat inside a company that donated significant amounts of equipment and bandwidth to them, and let me tell you – there is nothing as fucking annoying as a project where people only help ME because they know that I am some kind of “important”.

      I know there are lots of requests, but if you have a vested interest in people using your software, interface with them – not just the guy from the big company.

      What if the next CloudFlare just needs a little help? Maybe they should get their investors to arrange a private intro to the organizers of an open-source project, right? 😉

      • berthubert

        Justin – I see your point, but as stated in the article, we try to help everybody. And our mailing lists bear this out. But without context, I can’t tell if your problem has a wider impact for example.

  3. grin

    Really, Bert is one of the nicest open source authors out there, and he is responsive to requests from anyone. (For one, I am nowhere as important to insteresting as anyone else here around, and I usually get a meaningful answer pretty fast.) I do not believe the statemet that “you’re not important if you’re not big enough” suits this situation at all. And you, Justin, seem to miss an important point: motivation. People are limited by time and resources but most of them really like challanges and working for a purpose. A big company like cloudflare is not just a business potential and not just a good PR potential but (perhaps more importantly for Bert since he’s a maniac deep inside :)) a great challange and a great way to make your own program even better. And in the long run the bests programs win – and people win with them as well. Heck, *I* want Bert to find great challanges because the program *I* use just gets better. Even better.

  4. UX-admin

    The PowerDNS developers are some of the nicest people I have ever had the privilege of working with (although I privately regret that I have not had enough time to work with them (or for them) even more. What Bert wrote is absolutely true, they have helped numerous people in the community without a paid support contract, myself being one of them. When I did have the privilege to work with them, it was a joy.

    My stance is that the PowerDNS developers set the collaboration standard by which all other open source projects could be measured. And I have barely scratched the surface by saying that.

  5. Pingback: 45 Github Issues Dos and Don’ts | Unico Digital Singapore

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s