WaLLy3K@infosec.pubtoExplain Like I'm Five@lemmy.world•ELI5: How are these massive adblock lists kept updated so regularly?English
14·
1 year agoAs someone who runs a popular blocklist collection, I’ve come to find that most of the MASSIVE lists are people who collate a whole bunch of lists together and then promote their “one size fits all” solution alongside their donation link. There are very few original high quality ad-blocking lists maintained (where originality is defined as a sizeable amount of unique entries not shared by other lists) and almost all don’t appear to openly discuss the magic sauce behind their lists, outside of the obvious case of user submissions.
awk
is pretty damn solid. When I was completely rewriting thegravity.sh
script from Pi-hole about six years back, it was easily the fastest for parsing and uniquely sorting content from files with a couple million lines. It made things much more usable on Raspberry Pi Zero hardware, since changing to another language like Python was out of the question.