Now you can output basic HTML threads too!Swiss Anon strikes again!There is a real possibility that the internet could go down soon due to war or some AI fuckery... are you ready to browse the archives offline?>there's already more complicated code for it on github or FoolFuuka so you can run it on a serverFair enough, but I want to do it myself from scratch, as an exercise, with the functionalities that I desire and as simple as possible, under ~500 lines of code. If I can do it, you will be able to do it too if there's an internet blackout.The colors aren't final, still experimenting with that, but I've now set the font to Tahoma since it's more readable than the previous default serif shit (probably TNR). Still need to figure out which is better: Tahoma or Verdana.WHAT'S NEW:v. 0.03 Various code optimization here and there, including rewriting parts of read_post() and read_thread().Added some try/except blocks to several functions to prevent exceptions and unwanted stops of the runtime.Added first elaborate HTML output functions output() producing HTML files that look like what you would see in a real minimalistic imageboard.Added _make_link() function to elaborate HTML reply hyperlinks.Added getreplylinks(input_OP, l) function to get all the small "Quoted by" reply links like in the real thing.Added rt_html(input_OP, out_to_html, multi) function to get and render a single thread as HTML.Removed a bug in the csv splitter since we almost never need a csv header here.PASTEBIN LINK:pastebin.com/aLrgNNZm
Swiss anon delivers again.Make sure to support me if you like the offline browser.I would also like some of you to test the code and post screenshots of your outputs ITT so I can check if all works properly.As you can see what (You) can output now starts to look very close to my current version (see pic).
Also what I added to the 0.03 version is only an early and very minimal version of the single thread output function.Then there's the SQL code (to avoid loading everything in the RAM), Top # rankers for hashes, words, reply number and more stuff too.
GLOWIES ARE SLEEPINGTIME FOR SOME SHAMELESS SELF BUMPING
This is a Venn diagram showing the current situation for publicly accessible dumps as far as I know.
Reminder that a basic HTML searchresult() output function is also already available to you in the Pastebin code since v. 0.02, here's what a search report looks like (works on any 4plebs board.
Sometimes reply links in sticky threads are still slightly bugged but you can always see the post text.This is what a standard thread looks like in the newest version:
>one full hour of based archiving thread>zero repliesThings are not looking good here Swiss anon. You could probably go ahead and end your archive at 2025 and not miss anything. Did anything happen last year? Maybe 2024 just to catch our election.
>>546584846God bless you anon. Truly. I am but a phone posting fag, and don’t possess the skills you do; mine are elsewhere. But preserving the aggregate knowledge (like a librarian…) is something that is severely underrated (archivists are exceptionally based). Stay based.
>>546586345>didn’t anything happenYeah. A concerted multi year effort to shittify this place. Its documented. Must have cost a fortune.
>>546586345Hi based fren.It's still night in the US, I guess it's more or less normal.Indeed the signal to noise ratio in the archives after about 2022 has been decreasing a lot.4plebs still hasn't made anything after 2024 available (I'm not the one uploading the csv files to archive.org) but there can still be some useful information if you have some hundreds GB free on a disk.
>>546586271Ah I forgot to update a line after adding a column to the SQL database.Now it properly gets the reply links once again.
>>546586490You've probably answered this several times but how big is all of the /pol/ data dumps combined? With images of course.
>>546586417>>546586451Thanks fren, I appreciate it.If you can try downloading Spyder on your PC and try to run some of my code, as of now I didn't see a single screenshot from anyone else than me.I would really like to see some feedback from users.
>>546586741All with full size images? Probably ~30 TB.>Seagate IronwolfGreat HDD I'm using one of these too (8TB).The last full sized image dump is from 2022 and just that is about 20 TB (1 TB per part).Add 100-150 GB for text.There's also the smaller thumbnail dumps if you want to start smaller.
>>546586741Ah but wait only /pol/ is probably smaller, 30 TB is all boards combined.Let me check./pol/ is the biggest part of course, maybe you can manage to go down to ~20 TB all included.
The 2013-2026 /pol/ only thumbnails are "only" ~110 GB.I would start from there.
Otherwise you can start from older dumps.I didn't check this one but it appears to be 2013-2015 full size images and it's "just" about 40 GB.
>>546586846>>546586937Thanks but if I'm going to do this I want it all. 20TB really isn't bad. Less than a thousand dollars to own the complete collection of this show I've been watching for a decade.
>>546587090BASED.Good luck fren, I didn't try to add the images to the HTML threads yet but it's probably very easy starting from the hash or filename.
I wrote it in a previous thread but I was surprised to see that the most used noun on /pol/ from 2013 to 2017 wasPEOPLE/pol/'s hivemind is always right, don't ever doubt it for a moment.
>>546584846so, its like 4plebs or moe, just that its offline?
>>546584846pratty is going to have the last laugh
This was also pretty funny to see.BREXIT and REMAIN side by side.
>>546588704Nah we can filter that dirty jeet out of existence.
>>546584846Ty swiss fren. I wish I had frens to help me get all the archive threads in an offline format in an easy ctrl f format. I’m worried some of my favorite posts got deleted because their were so many threads on /x/ for one general in particular. I want to remember all the wonderful frens who came and shared fun things I want to remember and remind them too about. I learned to keep some stuff on cold storage or phones screens broke sometimes an stuffs. We all deserve to be appreciated for all our hard work we did. So coinkidinki if my favorite posts are all gone tho. It’s a new years resolution to share wonderful things with the world at this point. Yep gotta get thru Christmas first tho. God Bless
Oh I didn't see that an archived.moe dump for /pol/ until 2015 is also available.Well that's more or less redundant since we have the 4plebs dumps covering more time.Archived.moe dumps after 2015 are not publicly available as far as I know.
>>546588846Ah, no, /pol/ wasn't included in the archived.moe dump, that's why I made the Venn diagram that way.
>>546588681Some day, probably in the immediate aftermath of a false flag, the jews are going to remove all traces of antisemitism from the internet. It wasn't really practical to do so before but with AI they can now monitor every dark corner simultaneously. This will probably come at the same time as digital ID so the jewish Mother AI can see exactly who is doing what. We'll be forced to share politically incorrect memes on microSDs like trading cards.
>>546588792Ty, and don't worry fren, nothing was lost after November 2013. Probably there's even some fragments before that.Now we must make sure that there's at least 10 copies of each dump on every continent so it's possible to recreate archived 4plebs clones everywhere.Pic related is the first archived thread (on position 0 in the .csv). After that all is saved (with only some minor csv errors but didn't see many).
>>546584846Based bump
>>546588681Yes exactly.It's similar but portable (only ~500 lines of code without considering the documentation) and for the moment without images, text only.But you can keep it on a nice Ironwolf HDD or similar like this anon >>546587090 plans to do and load it locally on your machine even if the internet goes offline for some reason.
>>546589048>the jews are going to remove all traces of antisemitism from the internetSure, they won't probably shut up everything but restrict the results you get from search engines, LLMs or similar in their usual twisted orwellian way.We win by decentralizing information and bypassing jewish censorship in an intercontinental way.
My home internet plan says I have "unlimited" data. Can I really just download 20+TB? Has anyone ITT downloaded several TB before?
>>546590106Yes never had a problem with that, dunno how it works with US providers tho.Better read the contract.
>>546590106Oh and of course use the torrent option so you can resume it later etc.
>>546585192You can't bump your own thread.>>546590106Duh obviously.
>>546590137Contract just names the plan I'm paying for and the plan just lists the advertised speeds so I guess no data limit.>>546590485>Duh obviouslyNo, not obviously. Maybe it's different over there but in Burgerland so-called "unlimited" mobile plans are notorious for fucking people who go over the data limit they didn't know was hidden in the fine print.
>>546590485>You can't bump your own thread.Go back to India, retard.
There is a small bug in the rt_html() function due to switching a global variable to local, I will fix it and update the paste.
>>546584846OP is the only Anon on this board actually doing anything to fight the elites.Everyone ITT better thank him
Alright now it works.The corrected code is:# In rt_html, line ~555:all_rep.append(output(post, timestamp, op, htmlpost, title, linklist_2, index, htmlreply_index, htmlreply))# In output, line ~405:def output(post, timestamp, op, htmlpost, title, linklist_2, index, htmlreply_index, htmlreply):I will substitute these two lines in the paste too.
THANKS OP!
>>546590878>>546590937Thanks bro, I would be happy if you test this code and paste some output (either .txt or HTML) and give me feedback.PASTE UPDATED NOW IT SHOULD WORK
>>546591035Upload, I mean upload some output screenshot here ITT.Too much coding, I must take a break.
>>546591035Perhaps I can’t test once I get home to my machine. I bumped and saved your v0.02 thread as well ~<3
>>546591079>can’tcan*
bumperino
Imagine they’d archived the graffiti of the ancient world. Such as the walls of Pompeii.Our shitposts will live on forever in the digital realm, so long as it is kept.
wtf is this skizzo shityou niggers didnt even knowreal eyes realize real liessomebody call the medic itt>stat
bummmmmmmm......p
bump :)
>>546591903shillbot fail
>>546593421is it?
>>546593970Unless that was you with a proxy pretending to be a bot, I think definitely not human.
"[spoiler]Test[/spoiler]"
>>546584846neat
>>546594091Just here for the ride, going mobile, have a bump, Fren.
Also remember, to find the OP index number in the big list, look at the op_indexes or the .txt output:
Otherwise use the search function to get a keyword's position, then use it to locate the OP.Example:
Utility 2 is your friend to locate OPs and thus threads.I should add some kind of "next" button or something to go from a thread to the next chronologically. Maybe I'll add that in the future.Would be nice also to have a simulated catalog at a certain date and time.
>>546594455Checked and screenshotted
>>546584846Bumping the only thread that actually matters. Cheers
Q)What happens if you feed to read_thread() or rt_html() a post index that's not an OP?A) At the moment I didn't add an exception message, but the thread that post belongs to will be saved on disk starting from that post and ignoring all previous ones.So even if you write random numbers in the function (as long as they're in the main list) you will always get something as output.You recognize a non-OP post by the absence of the (OP) label (see pic).
Same but in the console.If the first post you see doesn't have the "(OP)" label then you're not reading a full thread and you should change the index number.
This is what you're supposed to see when you get an OP post as starting point of your output: