Data Hoarder

11 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

communick@selfhosted.forum

How can I scrape every .sensorpanel attachment from this thread? (alien.top)

submitted 11 months ago by ocp-paradox@alien.top to c/datahoarder@selfhosted.forum

3 comments fedilink hide all child comments

https://forums.aida64.com/topic/667-share-your-sensorpanel/

AIDA64 sensor panels are not archived anywhere as far as I know and trying to find them is horrible, it's a nearly 500 page thread.

Gotta be a solution out there already right? I foumd this https://creativeandcritical.net/fups but doesn't work with Invision forums.

top 3 comments

sorted by: hot top controversial new old

[–] binaryriot@alien.top 1 points 11 months ago (1 children)

A little bit shell magic around a little Python3 helper will do the job quickly: for i in $(seq 1 10); do grablinks.py --fix-links 'https://forums.aida64.com/topic/667-share-your-sensorpanel/page/'"${i}" --search 'file/attachment.php?' -f 'wget -c -O '\''%text%'\'' '\''%url%'\' | fgrep '.sensorpanel'; done | tee fetchscript.sh, then verify the generated shell script, and finally: sh fetchscript.sh.

Happy waiting! :)

You can grab my grablinks.py Python3 script from here: https://github.com/the-real-tokai/grablinks

[–] ocp-paradox@alien.top 1 points 11 months ago

Wow thanks. I'll try this out tonight. Was just a last ditch effort here before I went and manually did 50 pages a day or something. Thanks again.

[–] mrdebacle99@alien.top 1 points 11 months ago

Gotta be a solution out there already right?

Have you tried wfdownloader, works well for scraping forums.