[a / b / c / d / e / f / g / gif / h / hr / k / m / o / p / s / t / u / v / vg / vm / vmg / vr / vrpg / vst / w / wg] [i / ic] [r9k / s4s / vip] [cm / hm / lgbt / y] [3 / aco / adv / an / bant / biz / cgl / ck / co / diy / fa / fit / gd / hc / his / int / jp / lit / mlp / mu / n / news / out / po / pol / pw / qst / sci / soc / sp / tg / toy / trv / tv / vp / vt / wsg / wsr / x / xs] [Settings] [Search] [Mobile] [Home]
Board
▼ Settings Mobile Home
/pol/ - Politically Incorrect

Name
Options
Comment
Verification
4chan Pass users can bypass this verification. [Learn More] [Login]
Flag
File
  • Please read the Rules and FAQ before posting.

08/21/20New boards added: /vrpg/, /vmg/, /vst/ and /vm/
05/04/17New trial board added: /bant/ - International/Random
10/04/16New board for 4chan Pass users: /vip/ - Very Important Posts
[Hide] [Show All]


[Advertise on 4chan]


File: SQL thread HTML new.png (113 KB, 1891x859)
113 KB PNG
Swiss Anon strikes again!
There is a real possibility that the internet could go down soon due to war or some AI fuckery... are you ready to browse the archives offline?

>there's already more complicated code for it on github or FoolFuuka so you can run it on a server
Fair enough, but I want to do it myself from scratch, as an exercise, with the functionalities that I desire and as simple as possible, under ~500 lines of code. If I can do it, you will be able to do it too if there's an internet blackout.

The Python/HTML browser is starting to look good, I finally managed to get the correct reverse reply links to work too.
These little niggers are hard to implement if you never tried a similar task beforehand.
I also added flags (text based for now), GREENTEXT and image hashes.

The colors aren't final, still experimenting with that, but I've now set the font to Tahoma since it's more readable than the previous default serif shit (probably TNR). Still need to figure out which is better: Tahoma or Verdana.

If (You) can, try building something similar for yourself, otherwise I will release my source code a little at a time. You can even partially vibecode it if you know how to do it.

NEW (more or less):
I’ve published on Pastebin a second version of the code, with some more basic functions I made last month, including a search function with a somewhat polished HTML output (see pic).

Here’s the link so you can try it:

pastebin.com/Y1SY0Z6z
>>
File: searchresults html.png (245 KB, 1878x668)
245 KB PNG
A basic HTML searchresult output function is already available to you for free in the Pastebin code, here's what a search report looks like (works on any 4plebs board, this result is from /trv/ for instance):
>>
I now implemented a solution that turns .csv files into SQL databases so RAM memory is not filled by huge datasets.
Have a look at this, I managed to load a 83 GB file despite having only 16 GB RAM.
It works.
>>
>>546428793
Continue elaborating
>>
File: Venn_2.png (264 KB, 2000x2000)
264 KB PNG
I also made a Venn diagram showing the current situation for publicly accessible dumps as far as I know.
Don't forget that I'm also accepting monero donations... just in case.
>>
File: Mel Gibson.jpg (243 KB, 868x506)
243 KB JPG
>>546428793
I'm okay with it
>>
>>546428793
This would be like in I Am Legend with Will Smith watching old news on repeat, but I can lurk instead?
>>
Shouldn't somebody make a torrent?
>>
>>546428957
You can selectively search for keywords so you cut through all the automated glowie / shill spam.
In an internet where glowie AI spambots dominate, having an archive of the old internet will be extremely valuable.
I will eventually release most of the code for free, but first it would be nice to receive some moneros too.
>>
>>546429054
Fair enough. Id personally just vibeslop something so I could see random old threads in case shit died. Ideally with images if some are saved. Full searching would be useful though, as some of the archives have that disabled
>>
File: torrent.png (62 KB, 1332x446)
62 KB PNG
>>546428999
You can download the archives as torrents.
This is the smallest dataset (~20 GB compressed, ~25 GB uncompressed text only, no images):

archive.org/details/4plebs-org-data-dump-2017-01
>>
>>546428793
Are you only taking /pol/, or the other boards too? Early /s4s/ is great fun
>>
>>546429158
see >>546428898
Everything listed in that picture is publicly available.
>>
>>546429158
/x/ is such a goldmine that the solution to the world's problems is probably in there somewhere
>>
>>546429112
Nvm, should've read the thread
I'll toss you a few bucks for your public service in the next few days
>>
>>546429112
Neat. I think I have space for that.
>>
>>546429201
Thanks bro, a fucker sent me the equivalent of 0.10$ some days ago let's hope something more arrives.
>>
File: x part 1 old code.png (66 KB, 857x645)
66 KB PNG
>>546429199
/x/ has also less glowie spam compared to /pol/ (ironically enough).
Here's a screenshot of an old Top 1000 by reply number I made last week. It works.
Now I made an upgraded version of the ranker that works with the SQL engine.
>>
File: x part 1 old code 2.png (75 KB, 795x708)
75 KB PNG
I mean, most of the spam on /x/ is made by genuine schizos, but these can be interesting.
>>
Back when I was in a prepper club people were downloading offline copies of wikipedia. That was like 10 years ago.
>>
File: x OC roll.png (1.08 MB, 2709x1100)
1.08 MB PNG
This shit can create great OC stories.
I can access all of it now, thanks to less than 500 lines of code.
>>
>>546428793
Wow that's fucking useful, thanks!
Hope this works with firlin's /mlp/ dumps
>>
>>546429589
>89
>revenge
Extremely appropriate considering the current situation in the West.
>>
>>546429614
I left /mlp/ out of my Venn diagram EXACTLY to trigger /mlp/ fags.
Let's see if some more of them materialize in my thread.
>>
>>546429668
well you've got me itt, so that's something I guess?
I must say that having text-based flags is already an improvement over foolfuuka for /mlp/, as the thing's frontend doesn't have anything for /mlp/ memeflags despite properly archiving them
>>
>>546429101
>as some of the archives have that disabled
Now even 4plebs doesn't allow you to go full depth anymore unless you do a massive effort.
My code bypasses all of this and it is yours, forever.
>>
File: new timezone settings.png (10 KB, 904x133)
10 KB PNG
Now I finally managed to find a solution to avoid setting Omsk as timezone.
This new code should give you always Swiss time ignoring winter time etc. (in theory).
For some reason UTC-6 means UTC+6 in the timezone library, don't ask me why.
>>
File: search_HTML_new.png (80 KB, 1899x911)
80 KB PNG
The new HTML search output is also starting to look better now (but still minimalistic because that's what I want).
>>
File: US timezones.png (126 KB, 840x580)
126 KB PNG
It seems Langley glowies / shills are sleeping, how unfortunate!
>>
>>546429054
Ill blip you something if you made torrents of the data
>>
>>546429112
Why it's so big? If it just one year.
>>
>>546430125
No need these are already publicly available (not made by me).
I may adapt warosu databases in the future tho.
See >>546429112
>>
>>546428793
would you please add support for ND-JSON format? Apparently that's what fuuka and firlin's dumps use

For reference:
https://github.com/firlin123/desu-mlp
>>
>>546430149
No it's nov 2013 - jan 2017.
>>
>>546429112
come on. you can do better than that
>>
>>546430306
I haven't even tried to repack the archives for now. It's already pretty compressed tho, 100 million rows in 20 GB is not that bad.
Might consider it later.
>>
>>546430492
i just meant I'll wait for the entire archive torrent. i have enough space for it and then some
>>
>>546428793
>real possibility that the internet could go down soon due to war or some AI fuckery
Well you faggots give the conspiratard paranoia a fucking rest already. The day happened and nothing happened. How long are you keep on this bs for the rest of the month. You morons don't even know let alone actually understand their gameplan to begin with and I'm not gonna tell you because you don't even deserve to know. When you're least concerned you will then need to be concerned. Figure it out or don't faggot.
>>
>>546431172
It's a three day latency
>>
>>546428793
I really hope so im going to kill so many pakis and niggers the second the power cuts begin
>>
Bumping and thanking.
>>
File: 1779578127909698.png (27 KB, 1057x594)
27 KB PNG
>>546429112
20GB for 4 years of /pol/ really isn't bad. I assume this is with no images? You can use the image hash to retrieve them from a separate dump?

>>546429258
> 0.10$
Sounds like someone testing your wallet.

>>546429551
We still do. Download Kiwix, it's an offline reader for wikipedia and other things. The current wikipedia is 120GB and that's with images although the image files are very small. Reminds me of Old Internet.

https://lb.download.kiwix.org/zim/wikipedia/?C=S;O=D

"wikipedia_en_all_maxi_2026-08.zim" is the newest.
>>
>>546428872
riiiiiight
You crated an index on a field
Your total database size is 83.2GB
RAM will be filled when you query the database
if the dataset is huge and your query returns the entire dataset, it will be huge in ram
You're unlikely querying the database in its entirety with a single query thus you don't need 82.3GB of ram to query it
and even if you didn't have a enough ram, the query might just page to disk.
You are CLEARLY vibe coding.
But if you're having fun building something that's cool anon. Seems like your heart is in the right place
>>
File: 1793994144408978.mp4 (2.81 MB, 720x1280)
2.81 MB
2.81 MB MP4
>>546431243
>electricity is stopping me
YOU WON'T DO SHEEEIIIT!!!
>>
How the fuck do you use this shit. You didnt put a damn tutorial together
>>
>>546431812
install gentoo
>>
>>546431750
That video is a good representation of the typical greedy and entitled nigger.
>>
No one has ever posted anything on this site, that has made me change my life for the better.

And you're archiving it.
>>
>>546431920
that you are terrified of and WON'T DO SHEEEIIIT about.
>>
>>546428793
def split_huge_csv(input_file, n_parts):
"""
Splits a massive CSV file into 'n' approximately equal parts by byte size.
It processes the file line-by-line to guarantee low memory usage
and ensures that rows are never split mid-line.
"""
# Calculate total file size and target size for each chunk
file_size = os.path.getsize(input_file)
target_part_size = file_size // n_parts # Integer division

print(f"Total file size: {file_size / (1024**3):.2f} GB")
print(f"Target size per part: {target_part_size / (1024**2):.2f} MB")


hey bud, you don't need to split files into small pieces to load them into the database. The function/method is called readline()

>Loading files into a database using readline lets you stream large files line by line without exhausting system memory. Instead of loading the entire file at once, you read a single line, parse its fields, and insert the data in batches.

Although if the db ingress is broken you'll need to start the stream/file read over again. Up to you if you account for than and how you do that. General advice is
>PRE-optimization is the root of all evil.
https://stackify.com/premature-optimization-evil/

Which you've done splitting files for ingress.
Unless you're ingressing the files in parallel. I don't see that happening in your pastebin. If you refactor this code to readline() a csv, i.e. stream each record/row of a csv into he db table, you'll be able to reduce your lines of code - which you said you want to keep under 500 yes?

Bonus: You create indexes on fields/columns you query the most or/and fields which return queries slow. Indexes improve query speed/return on the fields/columns which have indexes.

If your queries are working fine, performing fine, then no need to index.
>no need to pre-optimize.

You didn't ask
but HTH
>>
>>546432035
Yeah that is the old version, now I'm streaming to a SQLite .db file bypassing the memory.
Coming soon to the pastebin.
>>
>>546432074
when will the torrent be available?
>>
>>546432133
I'm not planning to release a single torrent at the moment, all you need is on archive.org and pastebin. Also more anonymous than seeding a torrent.
>>
build a better 4chan on gno.land, and you won't need to archive, it's on the chain. which means you can avoid ip tracking while viewing or posting too.
>>
Alright after some experiments I saw that sometimes the old csv splitter (the one on Pastebin) corrupts the files for some reason if you try splitting parts repeatedly.
Only use the splitter once on a big file and select a larger number of parts, that causes less corruption.
Don't try to split the parts once again.
>>
File: 1780512896506382.jpg (16 KB, 594x336)
16 KB JPG
>>546430079
Unfortunately this is just the sad state of 4chan these days. No lust provoking image means no clicks but what really slows down threads like this is bots don't know what to make of it so they aren't bumping. Anyway I hope you don't let that discourage you as the few humans still lurking really appreciate this project. Truth is you're one of very few people here still working towards something or posting any kind of OC at all.
>>
File: 1793161226124840.png (938 KB, 1073x974)
938 KB PNG
>>546431925
>>
>>546432035
Thanks for the article about premature optimization.
Good stuff.
>>
File: txtoutput.png (57 KB, 1847x903)
57 KB PNG
This is an example of a simple thread output you can already get with the current pastebin code.
I would really like some of you to try it and post your results so I know it works well on other OS etc (it should).
>>
>>546434517
>Anyway I hope you don't let that discourage you as the few humans still lurking really appreciate this project.
Don't worry, as I said in a previous thread I suffer from almost pathological perseverance once I start a project.
Even if I stop posting for some reason you can still vibecode the rest.
>>
>>546431576
That is useful too.
I have full Wikipedia, full Grokipedia plus other stuff, soon full 4chan 2013-2026.
>>
File: thread.png (89 KB, 1895x887)
89 KB PNG
So this is what an output thread in HTML looks like now with the new code.
Not bad innit?
>>
File: thread_zero_4plebs.png (186 KB, 1885x683)
186 KB PNG
Compare it with the original on 4plebs.
Of course it's more minimalistic, but it works and now (after several iterations) it's finally fast.
archive.4plebs.org/pol/thread/23594684/
>>
Oh i forgot to change the timezone engine so it's off by 1h.
Don't forget to use the new code:

tz = ZoneInfo("Etc/GMT-6")
>>
>>546431653
Yeah I was calling "loading" taking csv data and putting it into the SQL DB file on disk, I don't know how the details of the CSfag lingo work.
>>
>>546428793
Nothing ever happens



[Advertise on 4chan]

Delete Post: [File Only] Style:
[Disable Mobile View / Use Desktop Site]

[Enable Mobile View / Use Mobile Site]

All trademarks and copyrights on this page are owned by their respective parties. Images uploaded are the responsibility of the Poster. Comments are owned by the Poster.