Download count feature needs optimizing

[gbaz]

We keep a record for each download instead of summarizing. On the live instance this counts feature is especially slow to persist and initialize -- we should improve it to reduce memory usage and time spent in restarts.

[LaurentRDC]

It has come up multiple times now, including in the discussions surrounding haskellfoundation/tech-proposals#67, that the download count feature incurs a lot of performance overhead, much more than the utility it provides.

I am interested in helping with this. Is there a fundamental reason it is difficult to optimize the storage of download counts, vs. simply removing the feature altogether?

Related issue: #1276

Edit: After some investigation, it looks like there's two ways this can be optimized:

• Stop durably recording every download individually, and instead accumulate in an in-memory structure before batch-updating the state at regular intervals;
• Stop storing per-day downloads forever, and instead encode a rolling-window calculation

[LaurentRDC]

cc @Bodigrim who has thought about this before as well

[Bodigrim]

Perhaps with some work we can optimize this feature, but my view is that it is not useful at all. What does it measure?

For example, base has ~70000 downloads total and ~200 downloads in the past month. Which is a low number, but it's actually very surprising that it is non-zero: obviously no human at all ever downloads base.tar.gz. This is all bot activity.

As another data point, aeson has ~600000 downloads total and ~250 downloads in the past month. Which is more than base, but still implausible. Every CI build on GitHub of any package depending on aeson (which is like pretty much every package) would have to download aeson.tar.gz. OK, there is some caching on GitHub Actions side, but still I alone probably trigger more than 250 downloads a month. The numbers are so low because this load is split across CDNs, Hackage mirrors and Stackage to an extent that Hackage number is hardly representative of anything.

I'd say that the number of reverse dependencies is much better predictor of package's quality than downloads count.

[LaurentRDC]

When I look at download counts, I don't necessarily care about the correctness, but relative scales. For example, the packages related to beam correctly show that practically no one uses beam-mysql, especially compared to beam-postgres.

Removing the download count is easy, but it's kind of final. Optimizing the feature may not be that much more work, and I'm willing to do it!

Regarding correctness of the downloads in the face of a CDN: other websites do it (e.g. https://crates.io/). I think there's a way to do this relatively simply:

• Add the Cache-Control: no-store header for the download endpoint;
  • Log the download statistic.
• Return HTTP302 (Redirect) to another endpoint with the current CDN caching configuration.

This results in every download from Hackage to count in the download statistics, while the payloads themselves remain cached.

For this to work, we need the feature to be optimized. So I suggest two pieces of work:

1. Optimize the download count feature. The reported download counts remain dodgy, but Hackage startup becomes faster at least.
2. Implement the redirect scheme I suggested above to get (mostly) accurate download counts

[gbaz]

I support the redirect scheme. I proposed it before but we didn't proceed out of an overabundance of caution.

As to the clunkiness of the feature, I was especially worried about it when we had hackage problems due to fd exhaustion, since download counts as implemented now open a ton of fds. However, I think we pinpointed those problems as due to other sorts of lag and slowness and not the download recalculations. Its still a weird clunky feature implementation, but some experiments at NYHac revealed it may not actually be a core performance/stability block so much as just our overall use of acid state. (cf #1463)