Sponsored By Aspose - File Format APIs for .NET

Aspose are the market leader of .NET APIs for file business formats – natively work with DOCX, XLSX, PPT, PDF, MSG, MPP, images formats and many more!

Chocolatey Community Feed Update!

Average approval time for moderated packages is currently under 10 hours!

In my last post, I talked about things we were implementing or getting ready to implement to really help out with the process of moderation.  Those things are:

  • The validator – checks the quality of the package
  • The verifier – tests the package install/uninstall and provides logs
  • The cleaner – provides reminders and closes packages under review when they have gone stale.

The Cleanup Service

We’ve created a cleanup service, known as the cleaner that went into production recently.

  • It looks for packages under review that have gone stale – defined as 20 or more days since last review and no progress
  • Sends a notice/reminder that the package is waiting for the maintainer to fix something and that if another 15 days goes by with no progress, the package will automatically be rejected.
  • 15 days later if no progress is made, it automatically rejects packages with a nice message about how to pick things back up later when the maintainer is ready.

Current Backlog

We’ve found that with all of this automation in place, the moderation backlog was quickly reduced and will continue to be manageable.

A visual comparison:

12/18/2015 - 1630 packages ready for a moderator

December 18, 2015 – 1630 packages ready

 

01/01/2016 - 7 packages ready for a moderator

January 16, 2016 – 7 packages ready

Note the improvements all around! The most important numbers to key in on are the first 3, they represent a waiting for reviewer to do something status. With the validator and verifier in place, moderation is much faster and more accurate, and the validator has increased package quality all around with its review!

The waiting for maintainer (927 in the picture above) represents the bulk of the total number of packages under moderation currently. These are packages that require an action on the part of the maintainer to actively move the package to approved. This is also where the clean up service comes in.

The cleaner sent 800+ reminders two days ago. If there is no response by early February on those packages, the waiting for maintainer status will drop significantly as those packages will automatically be rejected. Some of those packages have been waiting for maintainer action for over a year and are likely abandoned. If you are a maintainer and you have not been getting emails from the site, you should log in now and make sure your email address is receiving emails and that the messages are not going to your spam folder. A rejected package version is reversible, the moderators can put it back to submitted at any time when a maintainer is ready to work on moving the package towards approval again.

Statistics

This is where it really starts to get exciting.

Some statistics:

  • Around 30 minutes after a package is submitted the validator runs.
  • Within 1-2 hours the verifier has finished testing the package and posts results.
  • Typical human review wait time after a package is deemed good is less than a day now.

We’re starting to build statistics on average time to approval for packages that go through moderation that will be visible on the site.  Running some statistics by hand, we’ve approved 236 packages that have been created since January 1st, the average final good package (meaning that it was the last time someone submitted fixes to the package) to approval time has been 15 hours. There are some packages that drove that up due to fixing some things in our verifier and rerunning the tests. If I change to only looking at packages since those fixes have went in on the 10th, that is 104 packages with an average approval within 7 hours!

About Rob Reynolds

Rob Reynolds has been programming in .NET since the early days of 1.0... [Continue reading... at http://about.me/ferventcoder] On this blog you'll find just about anything I find interesting that has to do with development including Puppet and Chocolatey. Because we are all polyglot programmers in our own way, you may see anything from NHibernate to some custom CSS. Nobody is perfect and occasionally I may post something you find incorrect. Please keep me straight! With your help and interaction the community will benefit.
This entry was posted in chocolatey and tagged , . Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • https://about.me/ferventcoder ferventcoder

    We’ve added the statistics. https://chocolatey.org/ (it may be different than reported in this post – I was accounting for the fixes since the 10th, but the statistic is on a two week rolling average.

  • kendaleiv

    Well done!

    “Average approval time for moderated packages is currently under 10 hours!” Does this include completely new packages, or is it only applicable to previously moderated approved packages with updates?

    • kendaleiv

      Although, perhaps there’s no real difference between those two cases (especially now)?

    • https://about.me/ferventcoder ferventcoder

      Any package under moderation. See my other comment on this post, the statistics have been added to the site!