[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/pol/ - Politically Incorrect

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
Flag
File
  • Please read the Rules and FAQ before posting.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: minimal html 0.03.png (63 KB, 1893x791)
63 KB PNG
Now you can output basic HTML threads too!

Swiss Anon strikes again!
There is a real possibility that the internet could go down soon due to war or some AI fuckery... are you ready to browse the archives offline?

>there's already more complicated code for it on github or FoolFuuka so you can run it on a server
Fair enough, but I want to do it myself from scratch, as an exercise, with the functionalities that I desire and as simple as possible, under ~500 lines of code. If I can do it, you will be able to do it too if there's an internet blackout.

The colors aren't final, still experimenting with that, but I've now set the font to Tahoma since it's more readable than the previous default serif shit (probably TNR). Still need to figure out which is better: Tahoma or Verdana.

WHAT'S NEW:

v. 0.03

Various code optimization here and there, including rewriting parts of read_post() and read_thread().
Added some try/except blocks to several functions to prevent exceptions and unwanted stops of the runtime.
Added first elaborate HTML output functions output() producing HTML files that look like what you would see in a real minimalistic imageboard.
Added _make_link() function to elaborate HTML reply hyperlinks.
Added getreplylinks(input_OP, l) function to get all the small "Quoted by" reply links like in the real thing.
Added rt_html(input_OP, out_to_html, multi) function to get and render a single thread as HTML.
Removed a bug in the csv splitter since we almost never need a csv header here.

PASTEBIN LINK:

pastebin.com/aLrgNNZm
>>
Swiss anon delivers again.
Make sure to support me if you like the offline browser.
I would also like some of you to test the code and post screenshots of your outputs ITT so I can check if all works properly.

As you can see what (You) can output now starts to look very close to my current version (see pic).
>>
File: hashes.png (114 KB, 894x662)
114 KB PNG
Also what I added to the 0.03 version is only an early and very minimal version of the single thread output function.
Then there's the SQL code (to avoid loading everything in the RAM), Top # rankers for hashes, words, reply number and more stuff too.
>>
File: glow cia.jpg (61 KB, 500x500)
61 KB JPG
GLOWIES ARE SLEEPING

TIME FOR SOME SHAMELESS SELF BUMPING
>>
File: Venn_2.png (264 KB, 2000x2000)
264 KB PNG
This is a Venn diagram showing the current situation for publicly accessible dumps as far as I know.
>>
Reminder that a basic HTML searchresult() output function is also already available to you in the Pastebin code since v. 0.02, here's what a search report looks like (works on any 4plebs board.
>>
File: thread HTML new.png (71 KB, 1899x897)
71 KB PNG
Sometimes reply links in sticky threads are still slightly bugged but you can always see the post text.

This is what a standard thread looks like in the newest version:
>>
>one full hour of based archiving thread
>zero replies
Things are not looking good here Swiss anon. You could probably go ahead and end your archive at 2025 and not miss anything. Did anything happen last year? Maybe 2024 just to catch our election.
>>
>>546584846
God bless you anon. Truly. I am but a phone posting fag, and don’t possess the skills you do; mine are elsewhere. But preserving the aggregate knowledge (like a librarian…) is something that is severely underrated (archivists are exceptionally based). Stay based.
>>
>>546586345
>didn’t anything happen
Yeah. A concerted multi year effort to shittify this place. Its documented. Must have cost a fortune.
>>
File: fren.png (636 KB, 637x539)
636 KB PNG
>>546586345
Hi based fren.
It's still night in the US, I guess it's more or less normal.
Indeed the signal to noise ratio in the archives after about 2022 has been decreasing a lot.
4plebs still hasn't made anything after 2024 available (I'm not the one uploading the csv files to archive.org) but there can still be some useful information if you have some hundreds GB free on a disk.
>>
File: thread HTML new.png (72 KB, 1899x899)
72 KB PNG
>>546586271
Ah I forgot to update a line after adding a column to the SQL database.
Now it properly gets the reply links once again.
>>
File: ironwolf-pro-12tb-front.png (168 KB, 1000x1000)
168 KB PNG
>>546586490
You've probably answered this several times but how big is all of the /pol/ data dumps combined? With images of course.
>>
>>546586417
>>546586451
Thanks fren, I appreciate it.
If you can try downloading Spyder on your PC and try to run some of my code, as of now I didn't see a single screenshot from anyone else than me.
I would really like to see some feedback from users.
>>
File: image dump.png (23 KB, 968x305)
23 KB PNG
>>546586741
All with full size images? Probably ~30 TB.

>Seagate Ironwolf
Great HDD I'm using one of these too (8TB).

The last full sized image dump is from 2022 and just that is about 20 TB (1 TB per part).
Add 100-150 GB for text.
There's also the smaller thumbnail dumps if you want to start smaller.
>>
File: pol parts.png (35 KB, 962x582)
35 KB PNG
>>546586741
Ah but wait only /pol/ is probably smaller, 30 TB is all boards combined.
Let me check.

/pol/ is the biggest part of course, maybe you can manage to go down to ~20 TB all included.
>>
File: 2026 thumbs files.png (40 KB, 1140x648)
40 KB PNG
The 2013-2026 /pol/ only thumbnails are "only" ~110 GB.
I would start from there.
>>
File: image dump 2015.png (55 KB, 1071x771)
55 KB PNG
Otherwise you can start from older dumps.
I didn't check this one but it appears to be 2013-2015 full size images and it's "just" about 40 GB.
>>
>>546586846
>>546586937
Thanks but if I'm going to do this I want it all. 20TB really isn't bad. Less than a thousand dollars to own the complete collection of this show I've been watching for a decade.
>>
>>546587090
BASED.
Good luck fren, I didn't try to add the images to the HTML threads yet but it's probably very easy starting from the hash or filename.
>>
File: people.png (3 KB, 753x48)
3 KB PNG
I wrote it in a previous thread but I was surprised to see that the most used noun on /pol/ from 2013 to 2017 was

PEOPLE

/pol/'s hivemind is always right, don't ever doubt it for a moment.
>>
>>546584846
so, its like 4plebs or moe, just that its offline?
>>
>>546584846
pratty is going to have the last laugh
>>
File: brexit remain.png (6 KB, 752x92)
6 KB PNG
This was also pretty funny to see.
BREXIT and REMAIN side by side.
>>
>>546588704
Nah we can filter that dirty jeet out of existence.
>>
>>546584846
Ty swiss fren.
I wish I had frens to help me get all the archive threads in an offline format in an easy ctrl f format.
I’m worried some of my favorite posts got deleted because their were so many threads on /x/ for one general in particular.
I want to remember all the wonderful frens who came and shared fun things I want to remember and remind them too about. I learned to keep some stuff on cold storage or phones screens broke sometimes an stuffs.

We all deserve to be appreciated for all our hard work we did. So coinkidinki if my favorite posts are all gone tho.

It’s a new years resolution to share wonderful things with the world at this point.

Yep gotta get thru Christmas first tho.
God Bless
>>
Oh I didn't see that an archived.moe dump for /pol/ until 2015 is also available.

Well that's more or less redundant since we have the 4plebs dumps covering more time.
Archived.moe dumps after 2015 are not publicly available as far as I know.
>>
File: no pol.png (9 KB, 365x399)
9 KB PNG
>>546588846
Ah, no, /pol/ wasn't included in the archived.moe dump, that's why I made the Venn diagram that way.
>>
>>546588681
Some day, probably in the immediate aftermath of a false flag, the jews are going to remove all traces of antisemitism from the internet. It wasn't really practical to do so before but with AI they can now monitor every dark corner simultaneously. This will probably come at the same time as digital ID so the jewish Mother AI can see exactly who is doing what. We'll be forced to share politically incorrect memes on microSDs like trading cards.
>>
File: thread_zero_new.png (89 KB, 1895x887)
89 KB PNG
>>546588792
Ty, and don't worry fren, nothing was lost after November 2013. Probably there's even some fragments before that.
Now we must make sure that there's at least 10 copies of each dump on every continent so it's possible to recreate archived 4plebs clones everywhere.
Pic related is the first archived thread (on position 0 in the .csv). After that all is saved (with only some minor csv errors but didn't see many).
>>
>>546584846
Based bump
>>
>>546588681
Yes exactly.
It's similar but portable (only ~500 lines of code without considering the documentation) and for the moment without images, text only.
But you can keep it on a nice Ironwolf HDD or similar like this anon >>546587090 plans to do and load it locally on your machine even if the internet goes offline for some reason.
>>
File: counter psyops.png (188 KB, 913x455)
188 KB PNG
>>546589048
>the jews are going to remove all traces of antisemitism from the internet
Sure, they won't probably shut up everything but restrict the results you get from search engines, LLMs or similar in their usual twisted orwellian way.
We win by decentralizing information and bypassing jewish censorship in an intercontinental way.
>>
File: 1789632510435781.jpg (24 KB, 272x292)
24 KB JPG
My home internet plan says I have "unlimited" data. Can I really just download 20+TB? Has anyone ITT downloaded several TB before?
>>
>>546590106
Yes never had a problem with that, dunno how it works with US providers tho.
Better read the contract.
>>
>>546590106
Oh and of course use the torrent option so you can resume it later etc.
>>
>>546585192
You can't bump your own thread.

>>546590106
Duh obviously.
>>
>>546590137
Contract just names the plan I'm paying for and the plan just lists the advertised speeds so I guess no data limit.

>>546590485
>Duh obviously
No, not obviously. Maybe it's different over there but in Burgerland so-called "unlimited" mobile plans are notorious for fucking people who go over the data limit they didn't know was hidden in the fine print.
>>
>>546590485
>You can't bump your own thread.
Go back to India, retard.
>>
There is a small bug in the rt_html() function due to switching a global variable to local, I will fix it and update the paste.
>>
>>546584846
OP is the only Anon on this board actually doing anything to fight the elites.
Everyone ITT better thank him
>>
Alright now it works.

The corrected code is:

# In rt_html, line ~555:
all_rep.append(output(post, timestamp, op, htmlpost, title, linklist_2, index, htmlreply_index, htmlreply))

# In output, line ~405:
def output(post, timestamp, op, htmlpost, title, linklist_2, index, htmlreply_index, htmlreply):


I will substitute these two lines in the paste too.
>>
File: IMG_4971.png (230 KB, 500x324)
230 KB PNG
THANKS OP!
>>
>>546590878
>>546590937
Thanks bro, I would be happy if you test this code and paste some output (either .txt or HTML) and give me feedback.

PASTE UPDATED NOW IT SHOULD WORK
>>
>>546591035
Upload, I mean upload some output screenshot here ITT.
Too much coding, I must take a break.
>>
>>546591035
Perhaps I can’t test once I get home to my machine. I bumped and saved your v0.02 thread as well ~<3
>>
>>546591079
>can’t
can*
>>
bumperino
>>
File: IMG_4306.jpg (85 KB, 1184x778)
85 KB JPG
Imagine they’d archived the graffiti of the ancient world. Such as the walls of Pompeii.
Our shitposts will live on forever in the digital realm, so long as it is kept.
>>
wtf is this skizzo shit
you niggers didnt even know
real eyes realize real lies
somebody call the medic itt
>stat
>>
File: 1794868074501697s.jpg (2 KB, 83x125)
2 KB JPG
bummmmmmmm...
...p
>>
bump :)
>>
File: IMG_4638.jpg (409 KB, 1000x732)
409 KB JPG
>>
>>546591903
shillbot fail
>>
>>546593421
is it?
>>
>>546593970
Unless that was you with a proxy pretending to be a bot, I think definitely not human.
>>
"[spoiler]Test[/spoiler]"
>>
>>546584846
neat
>>
>>546594091
Just here for the ride, going mobile, have a bump, Fren.
>>
File: op.png (21 KB, 960x571)
21 KB PNG
Also remember, to find the OP index number in the big list, look at the op_indexes or the .txt output:
>>
File: searchresults html.png (245 KB, 1878x668)
245 KB PNG
Otherwise use the search function to get a keyword's position, then use it to locate the OP.
Example:
>>
File: utility 2.png (32 KB, 883x459)
32 KB PNG
Utility 2 is your friend to locate OPs and thus threads.
I should add some kind of "next" button or something to go from a thread to the next chronologically. Maybe I'll add that in the future.
Would be nice also to have a simulated catalog at a certain date and time.
>>
>>546594455
Checked and screenshotted
>>
>>546584846
Bumping the only thread that actually matters. Cheers
>>
File: not op post.png (22 KB, 793x745)
22 KB PNG
Q)
What happens if you feed to read_thread() or rt_html() a post index that's not an OP?

A) At the moment I didn't add an exception message, but the thread that post belongs to will be saved on disk starting from that post and ignoring all previous ones.

So even if you write random numbers in the function (as long as they're in the main list) you will always get something as output.

You recognize a non-OP post by the absence of the (OP) label (see pic).
>>
File: console.png (5 KB, 361x191)
5 KB PNG
Same but in the console.
If the first post you see doesn't have the "(OP)" label then you're not reading a full thread and you should change the index number.
>>
File: console thread.png (7 KB, 299x171)
7 KB PNG
This is what you're supposed to see when you get an OP post as starting point of your output:



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.