Swiss Anon strikes again!There is a real possibility that the internet could go down soon due to war or some AI fuckery... are you ready to browse the archives offline?>there's already more complicated code for it on github or FoolFuuka so you can run it on a serverFair enough, but I want to do it myself from scratch, as an exercise, with the functionalities that I desire and as simple as possible, under ~500 lines of code. If I can do it, you will be able to do it too if there's an internet blackout.The Python/HTML browser is starting to look good, I finally managed to get the correct reverse reply links to work too.These little niggers are hard to implement if you never tried a similar task beforehand.I also added flags (text based for now), GREENTEXT and image hashes.The colors aren't final, still experimenting with that, but I've now set the font to Tahoma since it's more readable than the previous default serif shit (probably TNR). Still need to figure out which is better: Tahoma or Verdana.If (You) can, try building something similar for yourself, otherwise I will release my source code a little at a time. You can even partially vibecode it if you know how to do it.NEW (more or less):I’ve published on Pastebin a second version of the code, with some more basic functions I made last month, including a search function with a somewhat polished HTML output (see pic).Here’s the link so you can try it:pastebin.com/Y1SY0Z6z
A basic HTML searchresult output function is already available to you for free in the Pastebin code, here's what a search report looks like (works on any 4plebs board, this result is from /trv/ for instance):
I now implemented a solution that turns .csv files into SQL databases so RAM memory is not filled by huge datasets.Have a look at this, I managed to load a 83 GB file despite having only 16 GB RAM.It works.
>>546428793Continue elaborating
I also made a Venn diagram showing the current situation for publicly accessible dumps as far as I know.Don't forget that I'm also accepting monero donations... just in case.
>>546428793I'm okay with it
>>546428793This would be like in I Am Legend with Will Smith watching old news on repeat, but I can lurk instead?
Shouldn't somebody make a torrent?
>>546428957You can selectively search for keywords so you cut through all the automated glowie / shill spam.In an internet where glowie AI spambots dominate, having an archive of the old internet will be extremely valuable.I will eventually release most of the code for free, but first it would be nice to receive some moneros too.
>>546429054Fair enough. Id personally just vibeslop something so I could see random old threads in case shit died. Ideally with images if some are saved. Full searching would be useful though, as some of the archives have that disabled
>>546428999You can download the archives as torrents.This is the smallest dataset (~20 GB compressed, ~25 GB uncompressed text only, no images):archive.org/details/4plebs-org-data-dump-2017-01
>>546428793Are you only taking /pol/, or the other boards too? Early /s4s/ is great fun
>>546429158see >>546428898Everything listed in that picture is publicly available.
>>546429158/x/ is such a goldmine that the solution to the world's problems is probably in there somewhere
>>546429112Nvm, should've read the threadI'll toss you a few bucks for your public service in the next few days
>>546429112Neat. I think I have space for that.
>>546429201Thanks bro, a fucker sent me the equivalent of 0.10$ some days ago let's hope something more arrives.
>>546429199/x/ has also less glowie spam compared to /pol/ (ironically enough).Here's a screenshot of an old Top 1000 by reply number I made last week. It works.Now I made an upgraded version of the ranker that works with the SQL engine.
I mean, most of the spam on /x/ is made by genuine schizos, but these can be interesting.
Back when I was in a prepper club people were downloading offline copies of wikipedia. That was like 10 years ago.
This shit can create great OC stories.I can access all of it now, thanks to less than 500 lines of code.
>>546428793Wow that's fucking useful, thanks!Hope this works with firlin's /mlp/ dumps
>>546429589>89>revengeExtremely appropriate considering the current situation in the West.
>>546429614I left /mlp/ out of my Venn diagram EXACTLY to trigger /mlp/ fags.Let's see if some more of them materialize in my thread.
>>546429668well you've got me itt, so that's something I guess?I must say that having text-based flags is already an improvement over foolfuuka for /mlp/, as the thing's frontend doesn't have anything for /mlp/ memeflags despite properly archiving them
>>546429101>as some of the archives have that disabledNow even 4plebs doesn't allow you to go full depth anymore unless you do a massive effort.My code bypasses all of this and it is yours, forever.
Now I finally managed to find a solution to avoid setting Omsk as timezone.This new code should give you always Swiss time ignoring winter time etc. (in theory).For some reason UTC-6 means UTC+6 in the timezone library, don't ask me why.
The new HTML search output is also starting to look better now (but still minimalistic because that's what I want).
It seems Langley glowies / shills are sleeping, how unfortunate!
>>546429054Ill blip you something if you made torrents of the data
>>546429112Why it's so big? If it just one year.
>>546430125No need these are already publicly available (not made by me).I may adapt warosu databases in the future tho.See >>546429112
>>546428793would you please add support for ND-JSON format? Apparently that's what fuuka and firlin's dumps useFor reference:https://github.com/firlin123/desu-mlp
>>546430149No it's nov 2013 - jan 2017.
>>546429112come on. you can do better than that
>>546430306I haven't even tried to repack the archives for now. It's already pretty compressed tho, 100 million rows in 20 GB is not that bad.Might consider it later.
>>546430492i just meant I'll wait for the entire archive torrent. i have enough space for it and then some
>>546428793>real possibility that the internet could go down soon due to war or some AI fuckeryWell you faggots give the conspiratard paranoia a fucking rest already. The day happened and nothing happened. How long are you keep on this bs for the rest of the month. You morons don't even know let alone actually understand their gameplan to begin with and I'm not gonna tell you because you don't even deserve to know. When you're least concerned you will then need to be concerned. Figure it out or don't faggot.
>>546431172It's a three day latency
>>546428793I really hope so im going to kill so many pakis and niggers the second the power cuts begin
Bumping and thanking.
>>54642911220GB for 4 years of /pol/ really isn't bad. I assume this is with no images? You can use the image hash to retrieve them from a separate dump?>>546429258> 0.10$Sounds like someone testing your wallet.>>546429551We still do. Download Kiwix, it's an offline reader for wikipedia and other things. The current wikipedia is 120GB and that's with images although the image files are very small. Reminds me of Old Internet.https://lb.download.kiwix.org/zim/wikipedia/?C=S;O=D"wikipedia_en_all_maxi_2026-08.zim" is the newest.
>>546428872riiiiiightYou crated an index on a fieldYour total database size is 83.2GBRAM will be filled when you query the databaseif the dataset is huge and your query returns the entire dataset, it will be huge in ramYou're unlikely querying the database in its entirety with a single query thus you don't need 82.3GB of ram to query itand even if you didn't have a enough ram, the query might just page to disk.You are CLEARLY vibe coding.But if you're having fun building something that's cool anon. Seems like your heart is in the right place
>>546431243>electricity is stopping meYOU WON'T DO SHEEEIIIT!!!
How the fuck do you use this shit. You didnt put a damn tutorial together
>>546431812install gentoo
>>546431750That video is a good representation of the typical greedy and entitled nigger.
No one has ever posted anything on this site, that has made me change my life for the better.And you're archiving it.
>>546431920that you are terrified of and WON'T DO SHEEEIIIT about.
>>546428793def split_huge_csv(input_file, n_parts): """ Splits a massive CSV file into 'n' approximately equal parts by byte size. It processes the file line-by-line to guarantee low memory usage and ensures that rows are never split mid-line. """ # Calculate total file size and target size for each chunk file_size = os.path.getsize(input_file) target_part_size = file_size // n_parts # Integer division print(f"Total file size: {file_size / (1024**3):.2f} GB") print(f"Target size per part: {target_part_size / (1024**2):.2f} MB")hey bud, you don't need to split files into small pieces to load them into the database. The function/method is called readline()>Loading files into a database using readline lets you stream large files line by line without exhausting system memory. Instead of loading the entire file at once, you read a single line, parse its fields, and insert the data in batches.Although if the db ingress is broken you'll need to start the stream/file read over again. Up to you if you account for than and how you do that. General advice is>PRE-optimization is the root of all evil.https://stackify.com/premature-optimization-evil/Which you've done splitting files for ingress.Unless you're ingressing the files in parallel. I don't see that happening in your pastebin. If you refactor this code to readline() a csv, i.e. stream each record/row of a csv into he db table, you'll be able to reduce your lines of code - which you said you want to keep under 500 yes?Bonus: You create indexes on fields/columns you query the most or/and fields which return queries slow. Indexes improve query speed/return on the fields/columns which have indexes. If your queries are working fine, performing fine, then no need to index.>no need to pre-optimize.You didn't askbut HTH
>>546432035Yeah that is the old version, now I'm streaming to a SQLite .db file bypassing the memory.Coming soon to the pastebin.
>>546432074when will the torrent be available?
>>546432133I'm not planning to release a single torrent at the moment, all you need is on archive.org and pastebin. Also more anonymous than seeding a torrent.
build a better 4chan on gno.land, and you won't need to archive, it's on the chain. which means you can avoid ip tracking while viewing or posting too.
Alright after some experiments I saw that sometimes the old csv splitter (the one on Pastebin) corrupts the files for some reason if you try splitting parts repeatedly.Only use the splitter once on a big file and select a larger number of parts, that causes less corruption.Don't try to split the parts once again.
>>546430079Unfortunately this is just the sad state of 4chan these days. No lust provoking image means no clicks but what really slows down threads like this is bots don't know what to make of it so they aren't bumping. Anyway I hope you don't let that discourage you as the few humans still lurking really appreciate this project. Truth is you're one of very few people here still working towards something or posting any kind of OC at all.
>>546431925
>>546432035Thanks for the article about premature optimization.Good stuff.
This is an example of a simple thread output you can already get with the current pastebin code.I would really like some of you to try it and post your results so I know it works well on other OS etc (it should).
>>546434517>Anyway I hope you don't let that discourage you as the few humans still lurking really appreciate this project.Don't worry, as I said in a previous thread I suffer from almost pathological perseverance once I start a project.Even if I stop posting for some reason you can still vibecode the rest.
>>546431576That is useful too.I have full Wikipedia, full Grokipedia plus other stuff, soon full 4chan 2013-2026.
So this is what an output thread in HTML looks like now with the new code.Not bad innit?
Compare it with the original on 4plebs.Of course it's more minimalistic, but it works and now (after several iterations) it's finally fast.archive.4plebs.org/pol/thread/23594684/
Oh i forgot to change the timezone engine so it's off by 1h.Don't forget to use the new code:tz = ZoneInfo("Etc/GMT-6")
>>546431653Yeah I was calling "loading" taking csv data and putting it into the SQL DB file on disk, I don't know how the details of the CSfag lingo work.
>>546428793Nothing ever happens