Now for the other weird bit. If I enable secret mode, the website works fine.

most likely the bad 404 response was also cached in your web browser. try clearing your cache or doing a reload with ctrl+f5 or cmd+shift+r.

we had an issue related to incorrectly caching 404 responses earlier but that was a few hours before your post. it should already have been resolved when you posted this.

sorry about the late response, got a lot of stuff going on currently and it seemed like you got useful replies here already anyway when i checked before.

we currently have a rule in place that blocks traffic with too high of a threat score. this rule was implemented before i joined, i’ll have to check with the team about the original reason for this and if we want to relax this.

at least the error message should be improved if we can do that, i think that’s just returning a static message currently.

you’re probably looking for the !perchance@lemmy.world community

@tgxn@lemmy.tgxn.net fyi

from a quick look it doesn’t seem like the crawler uses any federation, it seems to just iterate over the community list api for each tracked instance, it probably doesn’t have logic to remove entries that no longer exist, considering that they’re still in there.

https://github.com/tgxn/lemmy-explorer/blob/87d5167c7885cf547b5fe27cac03fc1aeb25ced5/crawler/src/crawl/community.js#L239-L270

there seem to be two separate issues relating to that.

the number at the top includes “all” communities, including those marked as nsfw.

on a quick glance, it seems all the nsfw marked ones are correctly marked as such, in the sense of also being nsfw on lemmy.

there also are a large number of communities missing overall, but at least the number next to the community tab adds up with the number of listed communities when the filter is set to show nsfw communities as well.

there is also either some kind of data corruption going on or there may have been some strange spam communities on lemmy.world in the past, as it shows a bunch of communities with random numbers in the name and display names like oejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwef which don’t currently exist on lemmy.world.