>literally nothing to report or discuss
>opening a new thread anyway

>>43393714
An Anon can dream.

>>43393714
>literally nothing to say
>bumping the thread anyway

>>43393714
>>43393730
it's about sending a message.

>finally twibooru is usable again!
>importer stuck
reeee

>>43394635
The importer was fine: it's image processing that got stuck.

>>43395175
will the images be reprocessed or something?
or there will be another 1000 image gap

File: a janny's salary.png (212 KB, 900x866)

212 KB PNG

>>43395240
As the fault was limited to a single step subsequent to the importer's successful operation, no such gap exists:
https://twibooru.org/3820842?q=derpibooru_id%3A3862973
https://twibooru.org/3820870?q=derpibooru_id%3A3862974

>>43395567
>no such gap exists
Then explain why for example these are missing:
https://derpibooru.org/images/3863319
https://derpibooru.org/images/3863317
https://derpibooru.org/images/3863316
And a lot more.

Twibooru should really re-import everything from derpy. The importer crapped itself countless times and tens of thousands of images are missing.

There's a "tantabus import" tag on Twibooru, but it only has a handful of images. Is it being used for manual imports?

>>43396087
Since the "elsewhere" button does not appear, I assume it is manual import.

Speaking of imports, how can someone upload an image so the elsewhere button appears?

>>43393576
Honestly, to get some momentum with this stuff, I've been pushing it to the TikTok side of the Fandom. G4 edits and other content gets like a million likes over there. These resources need a wider user base if we want good news with them.

>>43396223
>TikTok side of the Fandom
I'm not sure we want snowflakes here.

>>43396230
We're already flooded to the brim with snowflakes on both ends of the political cancer horseshoe. Is some tiktok zoomie/alphag who knows of pony as le edgy infection au really any worse than hordes of schizos shitting themselves over whatever political/culture war thing (real or imagined) they got triggered by?

>>43396403
At least we watched the show. And know history.

>>43396230
Most of those can't handle 4chan culture anyway, so after getting a couple "kill yourself faggot" comments they'll flee back to their hugbox to wash away the bad vibes.

>>43396223
>TikTok
Why would you bear this curse to us?

Is it possible on ponepaste to change username? Has anyone ever changed their username?
I don't want to change it, it is just a theoretical question.

>>43395977
My apologies, I was mistaken. Some disruption does appear to have ocurred at the importer level, and I'm working to rerun affected import jobs.
>>43395978
I've considered various approaches to filling in these gaps; I'm not a database expert by any means, but I'd like to hash something out with Filly.
>>43396087
>>43396192
"Tantabus imports" are indeed manual user uploads and not automatic imports by way of Moondancer, Twibooru's importer. Automatic importing would be hard to justify, both due to slopification concerns and Tantabus's particular creator:* namespace tagpoisoning.

>>43397310
>filling in these gaps
If you solve this, then
>rerun affected import jobs
Is not needed.
What I would do is simply grab all image IDs from derpi (just the IDs) and if it is not found on twibooru by "derpibooru_id:xxx" and is not a deleted image from twibooru, then add it to the importer's queue. Or an even stupider approach is to grab the latest ID and iterate over all possible IDs, add to the queue which is not on derpi. If the importel fails on non existent images (on derpi) then ignore the error.
>tantabus
I would really like to see twibooru being an archive and archive anything and everything. If slop is a concern, then hide ai stuff by the default filter. Tag "poisoning" can be solved by a rewrite rule, or just let it go and allow the "creator:" tags and automatically alias them to artist:* or whatever.

Yay actually someone working on Twibooru improvement! Thanks.

>Internal Server Error
>Internal Server Error
>Internal Server Error
>Internal Server Error
>Internal Server Error
I want Dario and Scam to be murdered brutally.

>>43393576
Manebooru is the best one

I'm trying to process the ponepaste archives.
It's a fucking mess.
The 2021.01.30 dump has no IDs nor authors. It's just the title (even that is partial...) and the content. I have not checked the content, but I assume that's a mess too in some cases. Not possible to find a paste by id, title or author. Basically useless.
The same issue with the 2021.05.18 dump, but only in some files. Some files have no ID because the title contained characters which broke the filename calculation function or IDK. This time the filename do have author names, but the separator character is underscore. Guess what some authors have in their name? Underscore! So it's not possible to separate the filename without some serious hacks.
The 2021.11.30 dump which is in a csv has fucked up escaping. It forgot to escape double quotes when there was a preceding backslash. The csv reader treats this as a new column -> content is fucked. This is repairable at least. But some paste content in it is completely messed up. Like http://ponepaste.org/2948 is fine on ponepaste, but in the dump it's just completely random characters. White noise level stuff. Not a single word can be spotted in it.
Same issue with the 2022.03.26 csv dump.
And I have not yet started to parse the .tar.gz dumps, but there is a readme stating that for a solid year or two the dumps were completely fucked up. Great.

Sad, because there are a quite a few deleted pastes. Like aftercase's (ponepaste's original creator!) NMP changeling story.

>>43396230
We all came here from somewhere else, even you. The weak ones will leave and the good ones will stay. You want youth in the Fandom. I'm 30 years old and have a wife and kid, I cannot match the level of activity and contribution that I could in 2012 but they totally can.

>>43396403
Not to mention, we were edgy. I was all over Rainbow Factory and Sweet Apple Massacre as a teenager. I think the infected au thing was honestly kinda cool. Better to be an edgelord than the newgen puritan revival.

>>43396993
The people you are concerned about can't complete the captcha.

>>43397771
And apparently you're too retarded to make multiple replies in one post. You must be 18 or older to post on 4chan.

>>43397774
Eh it's been a while, my b

>>43397756
this
>>43397764
is
>>43397771
not
>>43397780
reddit or tiktok

I don't want a bunch of 14 year olds polluting this place with skibidi toilet and tung tung sang or whatever that is. The core of the fandom was already 20 year old when the first episode aired. Including me. And do you know where I come from? Techeblog when that was a thing. Not a 10 second ADHD satisfier platform. You completely missing the point why this fandom even started to exist. It was not because of 14 year old kids posting memes.

>>43397825
I mean, I remember BronyCon 2013-2015 being a mixed bag between 15 to 30. Regardless, they don't even need to come here. Hell I haven't been here in ten years. I just think they should be made familiar with the resources. Big stuff like Equestria Daily or the various media like music might've been the domain of the older people, but the social prevalence was due to the younger side.

>>43397633
So even the latest DB dump (https://ponepaste.org/dumps/latest.tar.gz) is corrupt. Great.
Filly / Floorb check data/2948 in the ponepaste dump. It's not just character encoding issue in the file itself, because if you check it with a hex editor it is 90% 0x3F.

So is ponispics dead?

File: mare metrics.png (49 KB, 883x806)

49 KB PNG

>>43397853
The botspam has escalated to the point where Poner's essentially getting DDoSed by countless scrapers. Bing alone made around 100k requests.
For now, Under Attack Mode has been turned on, which has restored the site to working order. Apologies for the inconvenience if you get Cuckflared.

>>43397995
>Apologies for the inconvenience if you get Cuckflared
Looks like the captcha has also been applied to the api. Porter imports will be paused while this is the case.

>>43397995
I started complaining about the same two weeks ago: >>43355582
CF WAF captcha on first visit works well, no need for under attack mode. Just add a rule with the expression (http.host eq "ponerpics.org"), choose action: interactive challenge. Then in the settings pick Challenge passage and apply some large value (eg 1 week/month).
For the API bypass edit the expression and add some disable case like user agent is "ponerpics importer" or something (or just a cookie, or a random header, etc). That way the other boorus can quickly modify the importers.
Or if you don't observe the API being hammered, exclude that entirely from the captcha.
From the logs I did not see any request being made to API endpoints, only scraping through any link they could find.

>>43393714
I mean, Pony.Tube needing more servers is important.

>>43397825
>this fandom needs to just be bitter assholes who seethe about everything and raid funeral and convention threads just to shit things up
Nah.
>>43398186
Yeah, and is anyone ponying up ^:) the funds/volunteer help/infra to help with that, or is it just being brought up to occasionally state something?

>>43397756
>We all came here from somewhere else, even you
I've been almost exclusively using 4chan since 2005, so no.

>>43397995
Just checked and I got a CF automatic captcha so I guess CF doesn't remember if you recently visited another site behind their defenses, like was suggested in the previous thread.

>>43398227
>strawman
I agree with anon, we don't need people who come from platforms that discourage mental activity.

>>43397995
Thanks for the heads up.

Reminder that desuarchive is deleting old images and imperfectly archives text content. Some posts are missing lines.

>>43398691
>imperfectly archives text content. Some posts are missing lines.
Examples? Is it a bug or deliberate?

>>43398695
Some kind of formatting bug. Probably on the renderer side.
Example:
https://desuarchive.org/mlp/thread/43398640/#43398663
>>43398663

I made a bugreport thread like a year ago, no response. https://desuarchive.org/desu/thread/9760/

>>43397841
Filly / Floorb pls respond.

>>43398777
Oh yeah, that bug that nobody seems to be willing to take care of. I wonder if the people at Desuarchive are even actively involved any longer. Didn't it take them days to respond to abuse reports as well?

>>43398782
Even more reason to make an alternative.

>>43398797
Dunno, there's not much worth archiving on /mlp/ any longer. Hardly any OC left and if it is made and relatively good it'll usually get uploaded to the boorus (or ponepaste if it's a green). For the most part I only use Desuarchive to look for what legitimate posts 4chan tranny jannies deleted. I think it's the same for most people, and it serves its purpose well enough for that, even if it isn't a perfect archive.

>>43398807
He didn't go back on his recent behavior, if that's what you're asking.

>>43398804
>not much worth archiving on /mlp/
This is what was said in 2012 too. And 2013. And so on. Just because you find it worthless, it does not mean everyone finds it worthless.

>>43398819
Alright, then be the change. I still think a better use of your time would be finding unarchived OC on the board and uploading it to the boorus or ponepaste.

>>43398782
>Didn't it take them days to respond to abuse reports as well?
You're supposed to check on ur web sites once per year, semiyearly tops. The only reason humanity decided to bother with ugly ass semiconductors instead of something sleek like vacuum tubes or pneumatic computing to begin with is to avoid babysitting this shit all day. If you want to revert progress back to 1940s, I DEMAND vacuum tube logic back as well. Imagine all the nerd jokes. Evacuate this, hermeticism that.

>>43398314
But anon, that would exclude everyone here post-2014.

>>43399166
Before 2014 most people used to come from IRL, so technically they don't qualify either.

Look what the cat dragged in:
https://fanlore.org/wiki/Category:My_Little_Pony_Boorus

>>43399496
>Ponerpics is a -booru type imageboard for the My Little Pony fandom.
>Their about section claims that they specifically catering to AI generated images, stating that their name is Sweetiebotoru.
>Sweetiebotoru is an image booru for uploading and sharing AI generated images. AI is the future of art, and we embrace the future.
>—Ponerpics about page, https://ponerpics.org/pages/about
>However, an X post implies that this might be an April fools joke that they either forgot or actively chose not to change back.[1]
>A thread on another MLP imageboard implies the existance of the site before the era of generative AI.[2]

Lol, what a bunch of dense retards.

>>43399731
A fair approximation.

>>43399731
I don't think it's retardation, I think it's just trying to be "wiki-like" in that they state the facts as other sources write them.

File: 1782745754392.png (31 KB, 702x169)

31 KB PNG

>>43400095
>I don't think it's retardation, I think it's just trying to be "retardation-like"

>>43400408
Rare Wikipedia W

>>43400408
>inappropriate
Weasel words.

>>43400671
Teaching kids to masturbate/mutilate themselves GOOD, don'cha know.

And twibooru's derpi importer is stuck. For 5+ hours already. Again.
Ponepaste's dumps are garbage, full of files containing random white noise.

>>43401048
I am once again asking if there is anybody who is already 1) a relatively well-versed linux or bsd sysadmin and 2) can prove they are already well-known/trusted within this community
I do not have the energy or time to teach anyone right now, you need to be good at taking vague descriptions of someone else's mess and figuring out what to do with it.

>>43401326
>relatively well-versed linux or bsd sysadmin
>well-known
Check.
> good at taking vague descriptions of someone else's mess and figuring out what to do with it
That's basically any sw dev/engineer. Check.
>trusted
That's going to be an issue.

>>43401326
>well-known/trusted
You're asking this on an anonymous imageboard.

>>43401326
I am once again asking you to feel better, faget.

>>43401401
Yeah, and if I'm giving someone keys to my server, we're not going to be communicating only over an anonymous imageboard.
>>43401585
I don't see that as likely any time soon! But thank you.
>>43401379
Are you volunteering?

>everyone wants to demand
>nobody wants to step up and do
>bro wtf why do all our sites not work and die out???

It's no wonder all your sites rot from the inside out when you only ever get your Discord buddies and other attention whoring individuals to help you out. This is the same reason why 4chan jannies are all cancer: no actual 4channer has a "reputation" to (((prove))) that he is reliable, because he will eschew creating a name and identity for himself on the web.

And over 9000 other excuses you can use to pretend you're actually a super smart, capable, and successful person that just can't help with a project because X Y and Z.

>>43401975
There's a very simple reason I'm not helping your project: I don't care.

>Ponepaste down with a 502/504 gateway errors
>>43401740
>Are you volunteering?
Maybe? I would just fix the few things that annoy me personally then fuck off immediately never to be seen again. At least that's my track record with contributing to other peoples projects. Plus I don't use anything else but e-mail for communication if we don't count 4chan.
Why don't you properly open source the sites properly (aka git, issue manager, etc)? Or are you afraid it is full of holes (like last time)?

>>43402273
>Why don't you properly open source the sites properly (aka git, issue manager, etc)? Or are you afraid it is full of holes (like last time)?
All of these things have existed for between 1 and 5 years:
https://github.com/Neetpone/ponepaste
https://github.com/Neetpone/foalfetch
https://git.twibooru.org/Twibooru/.git/
https://github.com/Twibooru/twibooru-issues
>Ponepaste down with a 502/504 gateway errors
Looking into it.
>e-mail
That's fine, but I'm not really comfortable accepting systems administration help from people without some kind of public presence to provide evidence they're not just here to fuck my shit up. Code contributions are welcome from anyone, though.

>ponepaste down
>ponebin down
Coincidence?

>>43402463
Ponerpics too.

File: 3814179.png (56 KB, 605x600)

56 KB PNG

>certified check-cashing retard
>need to learn computers so my autism support sites stop going down
ucu

Perhaps a stupid question, but is there any conceivable future where the operating of these sites will become easier or cheaper? What would need to be done to move towards this?

>>43403279
>easier or cheaper
There is a continuous war ongoing that attempts to destroy small independent websites, so I doubt that. In the past hardly any site had to worry about getting DDoS'd, while nowadays every site needs to guard itself against that, for example.

>>43403333
This is why PonePaste was down last night. I had to implement caching of rendered pastes because scrapers were maxing out the CPU usage on my server with just rendering (eg: markdown, greentext, highlight) pastes. I had to request the help of someone who is better at administrating PHP than me to actually figure out that was the problem in the first place.

>>43403731
>PHP
This is your problem. ^;)

File: fuck ai.png (48 KB, 1267x267)

48 KB PNG

>>43403731
Yep, scrapers stepped up their game big time.
ponebin also hit 100% cpu usage (>>43375820), even through I aggressively cache all rendered pages already but when the scrapers request every paste, every format, every version the cache quickly gets full of entries used only once.
What solved for me is enabling Cloudflare interactive captcha (see pic). It starts to die down, but at the same time I noticed that some are starting to get through the captcha. If that happens I have to implement some persistent caching or something, not just a 4Gb memory pool.

Btw, please fix your dump script (https://ponepaste.org/dumps/scripts/dump_pastes.php.txt) because >>43397841
The following pastes are corrupt: 496,497,499,500,501,502,503,504,505,506,507,508,509,510,511,546,588,623,1047,1049,1068,1070,1071,1072,1073,1074,1075,1076,1078,1081,1082,1083,1084,1085,1086,1249,1250,1253,1254,1277,1278,1279,1284,1370,1410,1421,1586,1590,1825,2628,2629,2630,2697,2937,2948,2949,3001,3002,3067,3068,3089,3090,3091,3092,3093,3094,3095,3096,3098,3291,3371,3533,3610,3613,3849,3890,3891,4053,4054,4068,4176,4228,4403,4530,4547,4760,4889,4902,4938,4950,4967,5003,5315,5327,5409,5423,5460,5492,5495,5498,5499,5507,5509,5513,5514,5516,5519,5520,5526,5527,5528,5698,5701,5755,5775,6110,6182,6183,6184,6244,6768,6812,6924,7177,7195,7199,7217,7218,7298,7326,7416,7447,7626,7730,7794,7795,7984,8042,8067,8218,8238,8334,8402,8403,8507,8613,8665,8727,8729

>>43403780
I'm 90% sure Cloudflare runs the scrapers just so people pay them for "protection"
basically a extortion racket.

>>43403783
If this were the case and public would find out, that would be the end of cloudflare.
So far all they do is randomly extort money from already paying customers, if they don't pay then they nuke their content immediately. https://robindev.substack.com/p/cloudflare-took-down-our-website It happened a few times already.
My bet is on AI scrapers using residential proxies. I would not mind scraping (more poni fed to AI the better) but slow the fuck down. Especially when they request the same page a few days apart and find no change then why do they keep repeating it every few days?

>>43403783
Possibly. Same with AV companies, I find it hard to trust them. Especially since they perform censorship by marking perfectly fine content as "infected".

>>43403801
>why do they keep repeating it every few days?
Because people lied in their lastmods too many times. Violence begets violence.
Or because scraper writers are retarded.
Or because somebody convinced them that legacy internets don't matter in the glorious new era they're supposedly bringing.
Or >>43403783 is right and it's a false flag. But in this case my money are on legalism-promoting glowniggers.

>>43403731
>>43403780
I have manually checked a few more. A lot of pastes with non ascii characters are corrupted. For the non ascii quotation marks are widely used (“”‘’„) and these are messed up in the dumps.

>>43403872
I am working on the issue; I believe I understand some of the problem. I have temporarily paused the generation of dumps.

>>43404033
No need to pause it. At least it's doing something. I mean something is better than nothing.
A paste where only a single character is messed up:
https://ponepaste.org/1884#707 "clichés"

Rendering issue (not dump related):
The renderer messes up the pastebin link in 3637. And any similar where the link is followed by a ]. Both the resulting href and the display text includes the ] character.

>>43404097
I believe I have fixed the encoding issue, at least as presented in paste 1884. There is another issue pertaining to unencrypted pastes; I'm not sure if it's fixed yet but I tried to improve it. Please, check the dumps and tell me. Also, I moved all the old ones to an old/ folder.

>>43404216
1884 is fixed, but almost all which I listed in >>43403780 is missing. Eg 496,497,499,500,501. These are not unlisted, nor private.

Also about unlisted pastes: Can you include them in it too? Since the IDs are consecutive it does not take much to scrape it, but I rather avoid writing (more like running) a scraper.

>>43404216
>>43404623
So I wrote a scraper. Sorry.
But at least it exposed even more issues. Why is https://ponepaste.org/8 missing from the dump? Is it unlisted or something?

I also to scrape the internet archive anyway for deleted ones. Plus write a tool with which I can search by content, so maybe I can process the earliest dumps somewhat (which do not contain ID, author or proper title...).

>>43404742
>I also to scrape
>I also *want* to scrape
At 3AM my brain skips words.

Btw the ones I listed in >>43403780 were at some point in one of the dumps.

Also, the scraper revealed server side issues: sometimes the /raw/ returns 404, sometimes it returns 200 with an error page.
Also sometimes it returns internal server error, but if I query it again it works. Pretty inconsistent.

>>43405337
If you can provide as much information as possible it will help me investigate the issue. eg: paste IDs and what the error is.

>>43405353
The dump is missing 8, 496,497,499,500,501. These can be viewed on the site.

From the logs, I got a 500 internal server error for 5456, 5457, 5461, 5464, 5469, 5471-5483 but now it returns 404. Does not seem to be DB connection issue in your backend, because some succeeded (with normal error codes or the content) between these IDs. I re-ran the scraper on these and all is fine (returning 404) now.

If a paste is private, it returns 200 ok with the error page html on the /raw/, instead of 401/403. That confused me yesterday, I patch it temporarily but would be nice if it returned a proper error code.

And maybe in the metadata of the response the author name, paste title, creation/modification date. Because now I have download the normal rendered html too for these.

>>43405366
I added a few more retry mechanisms and stopped relying on the returned error code. Seems like I managed to scrape all the pastes. Web archive next!

But I would like to avoid using scraping this way, as it runs for ~1 hour to get all the pastes while hammering your backend. And hammering it daily for 1 hour is not nice. Plus if you enable CF captcha this won't work.

Further progress report: so far I'm missing ~2400 pastes (so about 25% of the total pastes what was once on ponepaste). About half of these are private/moderated/password protected, the other half is deleted.
I ran out of easily processible archives/dumps where I can at least pair the ID of the paste to the content.
So far about 60Gb of archives/dumps were processed. After deduplication (while keeping variants) it is only 400Mb.
Next one is internet archive.

p10 bump

>>43405834
Where do you put these scraped pastes?

>>43405883
Will release a DB dump once I finished importing all available sources.
Plus ponebin will also handle ponepaste paste ids and user pages.

desuarchive is down
damn, just when I wanted to use it

p9 bump

>>43403856
Etag

>>43405834
So, webarchive barely yielded anything (about 20 pastes), but it is better than nothing! On the other hand it is a pain in the ass to process the html back to a raw file. In one case it is a rendered markdown and half a megabyte of text (and it is valuable). That's going to be fun to convert back to markdown.
On the other hand the webarchive API is great. I only wish I could filter the urls more granularly, instead of just a prefix.

Also I noticed that the https://ponepaste.org/5985 pastedown cheat sheet has some rendering issues. The greentext, redtext, purple text is not escaped properly.

>>43407602
>greentext, redtext, purple text
gpt 5.6 Luna spent 15 minutes fixing this once I wrote a test harness for it. At least it will render correctly in the archive.

p10 bump

>>43403731
Block all datacenter and VPN IPs and all proxies, and that'll fix the problem.
>but muh privasee
I'm sure the schizos will live, given they're posting on a site that disallows using any of those to access it right now.

>>43408377
>anti-privacy sentiment
This is not your thread.

>>43408377
>Block all
GLHF blocking tens of millions of random residential IP addresses from all over the world.

File: 1760619188731757.png (113 KB, 588x576)

113 KB PNG

>>43408381
You're literally posting on a site that prevents you from using VPNs, proxies, any anonymizing network tool. Being too schizo to go to a pony paste website with SSL security unless it conveniently lets you use the features that allow aislop scrapers to rape your bandwidth to death is retarded.

Just wondering, but why is it considered such a problem to implement a Captcha to access the site? I understand the anti-CF arguments, but your site is already cuckflared, so that can't be the reason. If the Captcha is getting bypassed, just make your own Captcha: grab a random image of a M6 pony from Twibooru on each first request and have the user type which pony it is. Every user will be able to complete the challenge, and scrapers won't bother updating their code just to scrape some random pony sites (and if they do that would at least be a very interesting find).

>>43408392
>We should adopt 4chan's bad policies
No. Go back.

>>43408400
Automated scraping is sometimes beneficial. For example internet archive, or if a random Anon wants to scrape it.
Also imagine a ponypilled AI. It saying "talking with moonhorse", "consulting with Twilight" and similar instead of "bribing the hamster", "summoning clippy", "talking to the goblins" would be based.

>>43408655
>if a random Anon wants to scrape it.
They could, though? Just solve the captcha, save the cookie and start scraping. Or am I missing something here?

>>43408400
Twibooru is the only site I sysadmin for that uses Cloudflare, and I intend to keep it that way. The real solution is not a CAPTCHA, it's to make the site resistant to such attacks - which I am working on with PonePaste. The attack from the other day is still ongoing, but the site is a lot more stable now.

File: poner.png (87 KB, 1065x793)

87 KB PNG

I'm not a robot, God damn it. It's been three days now. If Twibooru goes down, there won't be a single altbooru that actually works.

>>43408727
CF will probably detect scraping and issue a new captcha. I'm unsure.
Also what if someone wants to do this automated? Would that person has to solve the captcha daily, save the cookie, etc?
Also fuck the captcha here, I get 3 to solve every single damn time.

>>43405982
It's live on ponebin. Will do a DB dump later, I have been working on this for like the past 5 days already. I tried to dump it to iwiftp but it disconnects mid upload every time.
Whatever dumps, archives, etc I have found (and can be imported) were imported. I still have two dumps (one from 2021.01.30 and one from 2021.05.18), which would be worth importing but it's missing the paste ids, authors (and titles are incomplete too). I have gone through a few manually, searching in desu, labeling, etc, but it's time consuming work. Workflow here would be to reverse find by content, keep it if it is not present, then reverse fuzzy search, if still not found keep it and look at it manually.
For example https://ponepaste.org/78 (deleted) -> https://poneb.in/78 is available again.
Also pastedown is supported: https://poneb.in/5985 renders perfectly. On https://poneb.in/5493 footnotes work too, I'm unsure if "colour banding weeeeeeeeeeee" should work or not (currently it does not change color mid line). 4chan does not change color mid line either.
Tags and user badges were not are not imported. Maybe I will extend the scraper to grab those too.

I have to set the scraper up to periodically scrape and import.
And I should grab all new pastebin IDs which were posted here in the past 5 years, scrape pastebin, scrape the webarchive, etc.

>>43409066
Please do not scrape the rendered paste pages. Use a combination of https://ponepaste.org/api/ajax_pastes.php and /raw/, eg: https://ponepaste.org/raw/1234

>>43409139
>ajax_pastes.php
This does not return all pastes. For example https://ponepaste.org/8 or https://ponepaste.org/11110.
I already mentioned #8 in >>43404742

>>43409139
>>43409228 (me)
Or 4410.

>>43409228
That's because they're unlisted. Please do not scrape the rendered paste pages. That's all I'm asking. Or, maybe I should be a bit more specific - please do not mass-scrape all of the rendered paste pages again. I don't care if you poll (highest number you've seen + 1) to see if there's a new paste, or whatever.
I had to solve 21 individual CAPTCHA puzzles (19 4Chan, 2 hCAPTCHA) to make this post. I don't want my sites to become like that.

>>43409139
Wait a sec, you render them on the server? Why?

>Twibooru rankings filled with barbieslop
Did some outsider retards just discover that shitty Jap artist or something?

>>43409755
Anon, it's always been that way on Twibooru. Pony is a lower rated category on there. I don't like it, but that's just the way Twibooru is.

>>43409776
Maybe, but all those (old) pictures by that same artist suddenly appearing in the top charts makes me believe some outside group discovered the site.

>>43409779
I checked and it just seems like another day that ends in "y". Nothing unusual. The biggest downside of Twibooru has always been it's userbase that votes this garbage to the top.

>>43409713
Because the pastes are not written in HTML, and your browser only speaks HTML, so they need to be translated to HTML at some point. Unless you like reading raw Markdown, or having your greentext not be green, I guess.

>>43409066
Cloudflare free plan has a 100mb upload limit so make sure your file is below that

>>43409393
>That's because they're unlisted.
There are users who post a single public paste as an index, and all other pastes are unlisted.
I can show you two examples:
https://ponepaste.org/user/grapenut (all unlisted but one)
https://ponepaste.org/user/pentapony (a lot of unlisted)
The unlisted/private/password protected paste feature were a mistake to add in the first place. Especially for guests. Instead a draft function would have been better aka registered user can save, edit, then publish.

The best I can do currently is avoid avoid scraping the rendered pages for the pastes which are available in the dump/ajax api. (daily dump is not running btw) I need the metadata: author, title, creation, modification date, tags.
Any of these would work:
- (1) make them available in the daily dump ( + tags, + badges for authors, etc)
- (2) make them available in the ajax_api.php (or another api endpoint) for all pastes
- (3) make a separate endpoint where which returns the metadata of a single paste, is it private, is it password protected, was it deleted by moderation (aka not just a 404 error in these cases)
- (4) return the metadata with the /raw/ endpoint in the http headers (not the best option, as this returns the raw paste, even if unchanged.
Option 1 is obviously the best for general archival purposes.
Option 2 is the best for my purpose, that way I can calculate the delta and only download what changed.
Option 3 would require a lot of requests.
Option 4 require a lot of request plus pointlessly returning the paste content. Maybe it can be combined with option 3: if the client sends a HEAD request then only return the metadata in the headers, if it sends a GET return the content too.
If I implement any of this (I would implement #2,4,1 in this order, in separate PRs) for you, will you merge them?

>21 individual CAPTCHA puzzles
WTF. I only have to solve 6 on the worst days (1 hcaptcha, 5 4chan). It was only 1 4chan captcha every 3 posts, currently it is 3 4chan captcha per post since like a week or so ago.

>>43409713
This comes down to the debate about what's the purpose of the browser. Do you treat it as a document viewer (server side rendering) or as an application runtime (making API calls, assembling the DOM itself). Both has its advantages.

>>43410006
It's ~3Gb. That's why I can't serve it directly. Will figure out something (like uploading it to mega).

Up.

Added some whatever random improvements came to my in my dreams
https://vcl.yerf.org/?q=twilight_sparkle+source:ponerpicsorg
https://vcl.yerf.org/post/2893223?q=copyright:%22my_little_pony%22+type:text+character:%22twilight_sparkle%22

Q: Why is it slow?
A: Because it's running along 50 other things on decade old hardware

>>43410430
What is that?

>>43410470
Ur moms nudes

>>43409776
>>43409787
I always down vote no hooves when I see it
I'm doing my part

>>43408403
>please leave the loopholes open so the ai scrapers can ruin your service
No.

>>43410013
>disable privacy options on your site so I can rape it to death with my ai scraper
Are you genuinely retarded?

>>43410671
Please provide a counterargument to >>43408387 that you conveniently keep ignoring.

>>43410674
Unlisted pastes were never a privacy feature. There's roughly 10,000 pastes on the site. You can enumerate them all in a few minutes just by asking for paste 1, followed by paste 2, followed by paste 3, etc. Unlisted pastes are only designed to avoid having them in the index for whatever reason the author might have. If you're treating unlisted pastes as a privacy feature, I'm sorry to hear that - should I make it more clear with eg: a message when you select "unlisted" saying that it doesn't prevent people from guessing the sequential 5-digit ID number?
>>43410013
I can't promise I'll merge anything - the code could be really bad. I would look at them and consider them, though. But, before you put in the work - I'm feeling alright today, I'm going to look into implementing a couple of those things (probably 1 and 2) later today. I also have an idea for how to make scraping rendered paste pages less of a load on the server, now that I have a few minutes to think and I'm not in "site is down, get it back up as fast as possible" mode.

>>43408387
>>43410686
NTA, but detecting residential proxies is actually pretty easy if you have JS. Once you figure out how, it feels almost silly how easy it is in concept (the implementation is a bit more troublesome.)

>>43410715
Maybe but AFAIK Filly doesn't want to require Javascript on his sites.

>>43410701
Thanks for your work anon.

>>43410674
The pool of IDs is tiny and sequential. It takes only about ten minutes to scan the whole site via the /raw/[0-9]+ uri
Unlike pastebin which uses a pool of 218340105584896 possible IDs.
>>43410701
>the code could be really bad
Thanks for judging my skills without knowing who am I, how long I have been a software engineer, etc.
>(probably 1 and 2) later today
That would be great.
>>43410715
Yes it's called a captcha. Also what should I do with that info on the client side? When all the time consuming calculation happen on the server side. A scraper sends one (1) request from a given IP, to a random page. That's it. GLHF determining just from the request alone if it's a residential proxy or not.

Also random status report, search paging is fixed (no longer ignores which site to search on subsequent pages), markdown headers also fixed on some pastes, images no longer overflow. I'll fix some of the automatic linkification of pastes because it misses a few. Plus I try to fine-tune the markdown format recognition as some markdown pastes gets rendered as greentext.

>>43410901
>Thanks for judging my skills without knowing who am I, how long I have been a software engineer, etc.
Not judging. I'm saying I'm avoiding blindly agreeing "yeah, if you write the code, I'll definitely merge it" explicitly because I CANNOT judge your skills or the quality of the code before I see it.
>GLHF determining just from the request alone if it's a residential proxy or not.
Believe it or not, it can be done. But indeed I do not want to do it, because it requires JS and I don't want to require JS, as another anon said.

Should I consider most of my paste hits as just bots and mass scrapers at this point? Do anons go to these sites to read or is it just a series of revolving archivists?

>>43411011
>Should I consider most of my paste hits as just bots and mass scrapers at this point?
I do some very rudimentary bot filtering; that is, I do not count hits for any client that explicitly identifies themselves as a bot or a crawler. This, of course, doesn't filter out those who pretend to be a regular user with a regular browser, but there's only so much I can do.

>>43410917
>Not judging.
You could have said "if you follow my coding guideline/patterns" or something like that, that's not judging. Saying " the code could be really bad, but maybe i consider it though" is explicitly telling the potential contributor to "fuck off I won't even look at your code".
>it can be done
From the request and its parameters alone? All you get is the GET request and it's parameters. No JS is being executed on the machine which made the request, as it is the first time it accesses the server.
I don't think it is possible without a second request. For example cookie challenge, redirect challenge, session establish redirect.
>because it requires JS
None of these require JS. But they do require a second request (and session handling on server side). And most crawlers bypass them if they behave like a browser (store cookies, follow redirects).

>>43411018
>explicitly telling the potential contributor to "fuck off I won't even look at your code"
I suppose it can be read that way if you ignore the very next sentence, which reads "I would look at them and consider them, though." Sorry if my autism rubbed against your autism the wrong way. Didn't mean anything bad by it.
>residential proxy blocking
I'd love to chat about it with you, but not here. It's an interesting concept that I love to talk about when I'm in the right mood.
>ponepaste API
Currently working on option #2 as suggested, and then if I still have time I'll work on #1.

>>43411018
>>43411061
Check "API", check latest dump, tell me if there are still any faults. Please and thank you.

>>43411018
>explicitly telling the potential contributor
I know English isn't your first language, but that sentence does not "explicitly" say that. What you want is a word like "tacitly".

>>43410901
bro you admitted to using GPT earlier in this thread, if site developers wanted slop contributions they could just do that on their own

>>43411150
The API seems good! I will cross check it with my DB, because I know some pastes have funky titles (I can't recall which is the one which causes the debugger console in the IDE to hang up, but there is one...). Plus I stored intermediate results of the imports, I will cross check with that too. But seems like this would work well. Do the modification dates work properly? I recall there was once a bug where ponepaste displayed the current date as the modification date or something like that.

The latest dump is still missing the unlisted pastes (#8), while now it contains some which was deleted by moderation eg #3034. The API does not return 3034, that's good. I will also cross check it with my list of known deleted/moderated/password protected/expired pastes.

I will check them in detail tomorrow.

>>43411322
Ah, an AI hater.
It is a tool. I'm not going to spend many hours digging through the markdown parser's barely documented extension API and retarded behavior like starting a new paragraph (encapsulating the extension in an additional parser block) if an extension's registered key matches. I rather spend 30 minutes writing a test harness, sample inputs and outputs which forces the AI to iterate on the extension and debug the internal behavior of the library. And I don't want to spend countless hours debugging browser behavior like closing the search overlay when a radio button was clicked or not selecting the radio button when the search overlay first opens and it's not getting focus properly. I just write a playwright test for it and let the AI iterate on the javascript until it figures out that it needed to block certain events, 2 lines of code change. Could I have done it? Sure, but it would have took me many hours to figure out. I wrote the test in 10 minutes, AI wrote the fix in 2 minutes.
Also I rather sketch out the algorithms, structures, interfaces, DB layout and write the tests first then let the AI do the actual implementation against my unit tests small piece by piece, then against my integration tests. It's not like I just prompt it to "develop the whole application, no mistakes". Also good luck have fun learning language #13514, library #62451 and digging through a ton of documentation on how to set up and configure application #1525513, etc. The language I choose barely reassembles what it was 20 years ago when I first learned it. You were probably not even alive when I was already shipping products. Even this small project already contains 7 different languages, 5 of which I don't use in my dayjob at all.
More than 50% of the code in the repository is used only once to import a specific dump to the DB. Basically throwaway code. I don't care how the AI writes throwaway code. I test it line by line in the debugger and verify that it does what I want from it to do. Which I would have done anyway if I would have implemented it manually.
Currently the project is like 85% written by me and 15% AI. And every single line of AI generated code I reviewed. It's not a vibe coded project. The AI would not make this huge of a fucking mess like I did with the JS and CSS...
If you want feel free to stick and stones and be stuck in the past.
Also can you show anything what you have contributed to the community besides shitposts? No?

>>43411493
>I don't want anti-ai features proven to work implemented on pony sites so I can keep killing them with scraping
>also please please please let me ruin your code with unwanted gpt-generated code

>>43411571
I understand your concern, but he is not the one killing PonePaste with scraping. I asked him not to scrape the rendered pages because it places additional load on the site, but there are also measures I could take from my end to improve that - or do what I'm doing now, which is make it so he doesn't need to do that.
I don't regularly use LLMs myself, other than the one built into most JetBrains IDEs for full-line code completion, but honestly if someone wants to use an LLM to make small but meaningful contributions to software I maintain and it passes my code review, I can't really complain. I won't review large obviously-LLM-generated PRs though. Funnily enough, last time I used an LLM to write a significant amount (>5 lines, I think this was about 150 lines) of code, it was a few months ago in an attempt to fix a bug in PonePaste, and I discarded all of it and just wrote it myself because the slop was too bad (which is impressive, if you've seen PonePaste's code quality in general.)
>>43411486
>The latest dump is still missing the unlisted pastes (#8), while now it contains some which was deleted by moderation eg #3034.
Will check this a bit later, good catch. Next step after getting that sorted will be to add the additional author information you requested, to the dumps.

>>43411571
You're that same guy that kept saying Catbox is not deleting files, aren't you? You have the same kind of unproductive responses and believe in conspiracies.

Or maybe there's two of you and you're both just shartniggers.

File: 4up.jpg (1.82 MB, 2000x1500)

1.82 MB JPG

>>43403279
Back when I had dreams of running my own booru, a thought I had that was interesting was to load up an SBC with a bunch of storage (say mirrored 8tb ssds) and take advantage of cheap Rasperry Pi colocation offers for hosting. The upfront cost would be larger, but the ongoing costs relatively low assuming minimal hardware failure.

>>43411686
For a niche small booru that'd be awesome. For something like a mirror of Derpibooru, there's a few issues with that (8TB isn't enough storage, the speed of image and video processing may be so slow that you can't handle realtime imports, etc) but they could perhaps be optimized around (store less different thumbnail sizes, compress images more, import the processed version of images/video instead of doing local processing.)

>>43411714
I think the total size of full images is still less than 8tb, though getting somewhat close. My goals were always going to be a subset of the full database anyway. The RK3588 is relatively performant for an ARM chip, though somewhat middling comparing to x86. I think it'd probably be able to keep up with real time imports. Of course, one could potentially take advantage of hardware acceleration too. A fun thought experiment anyway. Maybe one day I'll set up my own personal booru at my house.

>>43411686
RAM will be tight. 8TB is probably not enough. What I would do instead of mirroring is setting up an unorthodox RAID configuration: both a mirror and a stripe at the same time with different partitions.
The most valuable things (DB, configurations, etc) living on the mirrored partitions while the content + generated contents (thumbnails) residing on the striped partition.
Eg both drive having a 1TB partition and a 7TB one, the 1TB ones mirrored, the 7TB ones striped. That gives you 1TB + 14TB.
Processing power might be enough if you can use hw acceleration for image/video encoding/compression.

File: top.jpg (2.36 MB, 2000x1500)

2.36 MB JPG

>>43411870
The board I have has 24gb of ram, though you can get up to 32gb on the top end model. The software would have been Sunbooru, based around SQLite which is fairly efficient on the ram. Also note that this board actually has a third nvme slot (2242 I think?) on the top so a third non-replicated ssd could be used for thumbnails and whatnot. The split mirrored/striped partitions is a very interesting idea as of course a separate backup would be needed anyway. Not something that I thought of before.

>43411571
>total strawman of the argument
okay anon
>>43411639
>the one built into most JetBrains IDE
That's like absolutely garbage level. I don't even know why they bother with their own AI models or plugin. Try any cutting edge model released by big players in the past year. gpt 5.6 Luna or claude sonet with VSC copilot plugin (no need to pay microslop, you can configure it with a "3rd party" vendor). There is the copilot plugin in IDEA, it is usable (I use it daily) but the one in VSC is an order of magnitude better.
I was against AI in the early days with similar experiences, it generated unusable shit, hallucinated a lot, etc. But in the last year it completely changed and become a tool I use daily as a force multiplier. One just has to learn how to use it, how to feed it context, how to build a knowledge tree for it that it can use, what/how to write the coding guidelines/documentation, how to give it opportunity to verify/debug what it is doing, etc. Instead of just expecting it to be an AGI.

>>43411878
>Sunbooru
Are you the dev of that?
>fairly efficient
>python
Python and efficiency are mutually exclusive.

>>43411955
>Are you the dev of that?
Yes.

>python
Python's not really going to be the bottleneck, it's going to be SQLite that's doing most of the heavy lifting.

>>43411668
>not slandering catbox or wanting hallucinated aislop code makes you a sharty user
>not supporting slopcode and scraping is unproductive

>>43411947
I think you misunderstand. The unusable slop I mentioned in my anecdote was created by Claude Opus 4.7. Only reason I mentioned the tiny models built in to the IDEs I use is so some pedant didn't come around and say "WELL IVE SEEN YOU USE PHPSTORM, DONT YOU KNOW THE CODE COMPLETION TECHNICALLY USES AN LLM?!?!?!!!! SO YOU CANT SAY YOU DONT USE LLMS!!!"

>>43412384
Opus 4.7 is great. You just have to prompt it right. It's a skill, it took me months to learn how to use it like a tool and not just "vibe code" and expect great results. Also a LOT depends on what plugin you use in the IDE, this is why I suggest the copilot plugin to start with.

Btw I found the paste with the strange title: https://ponepaste.org/2692
When the title is printed out to the console the console freezes until the application terminates. The json correctly contains the title (with the whacky characters), so it's not an issue.

>>43412313
Ah, looks like I was right. Our own dedicated threadshitter. I'd tell you to get a tripcode so I can filter you, but your posts stand out on their own anyway so they're easily dismissed.

>>43412483
literally the slogan of the blog https://pivot-to-ai.com/
>It can’t be that stupid, you must be prompting it wrong

>>43412571
NTA but it's actually true, at least in part.

>>43412571
>i hate ai so much i have a random site on shortcut to prove how superior my baby dick is

>>43412662
>i use ai so much to do all my "thinking" that i cant even fathom someone having enough cognitive capacity to remember the url of a blog

ITT: people who just discovered meme arrows

>>43412511
>the truth is threadshitting
>lying and being a cunt is "contributing"
In the fight between the catbox owner's words and yours, I'll side with the guy running the service who actually knows what he's talking about vs a projecting autist with a chip on his shoulder.

>>43412483
Alternatively, you can learn to code so you aren't relying on hallucinating palantir bots to shart out some trash code. I don't get why you evangelists are so desperate you stat looking for niche pony websites to shill your trash to.

>>43412722
Your contribution has been recognized and summarily dismissed.

>>43412483
I suspect that title is mojibake from when I converted the encoding from some strange legacy encoding to UTF-8. If you see any other titles that look like that, or you have a way to find them, do let me know please. I have a very old copy of the database that I can likely use to figure out what they are supposed to be.
I believe I have fixed the dump to include unlisted pastes, and exclude pastes deleted by moderation.

>>43411947
>how to build a knowledge tree
Build what now?

File: Mommy.png (21 KB, 157x156)

21 KB PNG

I'm a good Twiboy... but I want Box to punish me for being naughty...

>>43412982
You had to wake up and type that so that i had to wake up and read that.
I was having a good dream too, there were mares.
Are you happy now? Is my disappointment clear?

>>43412732
>mojibake
Learned a new word today.

>>43413001
why are you berating someone who gets off on it

>>43413029
In the hopes that after he's had his one bit of fun he won't do it again.

>>43412732
So I finished the scraper, it makes a total of 4 requests if nothing is changed.
One request to fetch the ajax_pastes and the other 3 to /raw/ are caused by funky entries.
11645, 10198, 10197 appear in the json with "Removed by moderator" as the title. I could filter to this but 11939 has "Removed by moderator" title but at the same time is not removed, it just has 0 length content, and was "silently" changed. The updated_at remained the same but the content changed. So the scraper makes 3 requests for 11645, 10198, 10197, but does not for 11939 as it assumes it is unchanged.
I think this is fine for now. I do not scrape the rendered pages, just the /raw/ and only if the json says it changed.
No other moderated/deleted pastes appear in the ajax_pastes.
Thanks!

>>43412732
Title is most likely https://ponepaste.org/2692#6. Just the usual non ascii quotation marks.

>>43412734
Brainfart, it's knowledge graph.
Basically a fancy long term memory for the agents where it can store, tag, connect information nodes, search in them, etc.

>>43412732
>dump to include unlisted pastes, and exclude pastes deleted by moderation
Will check this one too later.

>>43412982
She has a penis (acquired)

p10

>>43413058
>Brainfart, it's knowledge graph.
Still not something I know how to use in the context of agents and stuff. Do you happen to have any cool intros on hand?
I don't know if I'm legit too lazy to look it up myself or just crave human communication. Maybe it secretly always was the same thing modulo modulae.

>>43413712
TLDR AI has a limited context window (what info it can use for prediction), and most of the time it wants to do something it needs to look up stuff. If it has to look up thousands of lines of code/documentation/web search/whatever then by the time it finished looking up the info it filled its context window and forgot everything else.

The base concept (RAG) is pretty old:
https://en.wikipedia.org/wiki/Retrieval-augmented_generation?useskin=vector
KGs aren't new either, but combining them with AI is.
https://theneuralmaze.substack.com/p/building-agent-memory-with-knowledge
https://pub.towardsai.net/your-ai-agent-got-it-right-but-did-it-reason-right-c9a8ad875f8c
There are hundreds of mcp capable microservices which implement KG + MCP for RAG, here is one:
https://github.com/aouicher/graphmind
I use whatever the copilot plugin has built in. It works well enough for the type of things I use it for.

>>43411014
>This, of course, doesn't filter out those who pretend to be a regular user with a regular browser
I wonder how high the percentage of this catgeory is in the total site traffic.

>>43414072
The scrapers did present themselves as normal users. A random up to date / close to up to date chrome/firefox/safari user agent.

I will do a db dump soon (maybe today), but first I have to fix some issues with the titles and also figure out what caused it. Luckily I don't have to reprocess everything.

Also chapter links (markdown) are fixed. Actually it was not broken, but no one managed to use them correctly. So now the backend tries to figure out what was the intention and fixes it. For example : https://poneb.in/11138
Footnotes are also enabled/fixed.
Markdown detection is fine tuned, but there are still cases which dont get detected as one, eg #1400.

>>43412732
Could you also add a field to the api which returns the format? (plain/green/pastedown)
Thanks!

Anything new?

>10

>>43415397
This thread only cares about dumps. We all like dumps. Dumps from here, dumps from there. I want to see your dump. Can you put your dump somewhere for me? I'm going to inspect your dump. Did you look at his dump? Now that's a big dump. A massive dump. How did he dump that much? How does your new dump compare to your old dump? I've been inspecting your dump. I did a deep dive in your dump. I found something inside your dump. In fact your dump is kinda gross. You should do a better job with your dumps. Can I help you with your dump?

Now if you'll excuse me, I need to take a dump.

>Days since the last time ponerpics was literally unusable
>0

File: I keep screaming but God (...).png (46 KB, 937x492)

46 KB PNG

>>43416182

Speaking of dumps
https://mega.nz/file/8nIxjQxJ#9w_lGVP0WVOijXFWngg9aGB-btZWuvmbMqjA5K-5eHI
ponebin DB dump. Can be imported into postgres 18.
Includes pastebin (until 2020 December) and ponepaste (until yesterday 2020.08.06).
I manually deduped ~1000 ponepaste versions yesterday (checking them one by one...) which were most likely caused by character encoding issues.

To get a ponepaste paste:

select ppv.title, ppc."content"
from ponepaste_pastes pp
join ponepaste_paste_versions ppv on pp.id = ppv.id and pp.default_version = ppv."version"
join ponepaste_paste_content ppc on ppv.content_hash = ppc.hash
where pp.id = 8

Ponepaste is utf8 encoded, no exceptions. The content is a raw byte array, you need to decode it.

To get a pastebin paste:

select ppv.title, ppc."content", pas.processor
from pastebin_pastes pp
join pastebin_paste_versions ppv on pp.id = ppv.id and pp.default_version = ppv."version"
join pastebin_paste_content ppc on ppv.content_hash = ppc.hash
join pastebin_archive_source pas on ppv."source" = pas.id
where pp.id = 'uA3K0w9B'

The if the third column is "utf8-sign" parse the content column (byte array) as such. Else parse it as utf8. If that fails replace every byte under 0x20 and over 0x80 except 0x0A and 0x0D with question marks or spaces or whatever you prefer. Then parse it as ascii or cp1252. Some version of the same paste might be utf8, some might have fucked up encoding, it's not guaranteed that the latest version is the best.
The pastebin content is not deduped, there are a lot of duplicate versions. Once I start importing more pastebin pastes I will do something about it, and try to also come up an automated way of handling encoding errors and remove the versions if an older/newer one is better.

>>43416684
>but God doesn't answer
What is this god you speak about? I only know www.celestianism.com

>>43416684
Don't you just love AI?

Actually I do like AI, but I hate corporatism.

>>43416692
You might want to know about https://ftfy.readthedocs.io/en/latest/ which tries to automatically fix encoding issues.

>>43416692
>(until yesterday 2020.08.06)
*2026.08.06
>>43416935
I will test it thanks!

P10 bump

https://www.youtube.com/watch?v=N25C4UVnsMU
Should we fear pony content getting removed from the internet archive?

>>43417612
What's the tl;dw?

>>43417634
IA being IA, shitniggers being shitniggers, and a macrocosm of the abuses Derpi has already demonstrated ad nausaem within pony.

>>43417645
I don't think IA can really be counted as a proper "archive" the way a site like Twibooru or Ponerpics can anyway. Most of the content that stays up there that's not strictly supposed to be there probably only does so because they have a shitload of data and it just happens to not be attracting unwanted attention. I guarantee that at the slightest complaint of a copyright holder or some DMCA notice they'd probably wipe every upload associated with an account that got targeted by the copyright troll.

>>43417319

File: 313681.jpg (371 KB, 1152x1691)

371 KB JPG

>>43416694
Idolatry is a sin, y'know.

>>43417612
>>43417645
Remember how an aussie battery company got them to purge their site from the archive when it surfaced that they changed the warranty terms and tried gaslighting their customers
IA are turbocucks who will bow at the slightest pressure

>>43419318
Yeah, I had an inkling that might be the case. I uploaded my copy of the lain.la /mlp/ archive to it and will upload the catbox.moe archive I'm still working on to it as well, but I'm going to keep a personal copy of both of them on one of my HDDs too in case they ever decide to be retards about it. I'd suggest everyone else who uploads content to it to do the same, just keep a personal copy just in case, because relying on IA as the sole archive of data seems dicey.

>>43419552
Rome moved the site archives to IA. Don't know if he still keeps a copy somewhere or not.

He should have deleted/moved the /gif/ archive instead. I doubt anyone wants to browse a random dump of gif images without the threads/posts. It's 22TB compressed.

>>43419318
IA is a big enough operation that someone might sue them for the money. They don't have any real choice here, because the moment lawyers get involved, it all goes to shit.

Is there a fimfic alternative which scrapes all the images, groups, comments, ratings, etc?
Not just the content.

Derpi seems to getting scraped too, I get an "i'm not a robot" page about every hour or so.

>>43420242
Is that why their torsite has been down for weeks?

p8 bump

>>43420718

>>43421309

>>43421757

>>43420316
The what?

Sorry for asking it here, but is there a good image archive place for hentai/cartoon stuff that isnt bitchy about uploading permissions?
Im sick of getting my bookmarks deleted on pixiv.

>>43422815
Derpi forcibly redirects all Tor traffic to their hidden service, which requires people to log in if they want to use the site via Tor. This would annoying enough if the service actually stayed up for more than ten minutes at a time.
It does track with their staff LARPing as the Stasi.

>>43423633
Yeah, Derpi's staff is beyond any rhyme or reason. They've fallen off the deep end a long time ago.

>>43423633
>>43423695
Tor is an abuse vector with thousands of IP addresses that can be used to anonymously abuse the site, including in highly illegal ways such as uploading child porn. The options are #1: to block Tor entirely from interacting with the site, either all at once, or slowly by means of banning individual Tor exits as they get used for abuse on the site. or #2: to support Tor use natively by means of providing a hidden service and requiring an account to interact with the site, so that the account can be banned if it is used for abuse, rather than a Tor exit IP, which is innocent and ephemeral.

>>43423633
As long as you can register while Tor'd up, I think this is reasonable

File: I really like her TAILS.png (104 KB, 2562x1463)

104 KB PNG

>>43423708
Read-only Tor access with login required to make any changes is a perfectly fine compromise, one which this forced redirect denies. You're unable to even view the front page without logging in, which you can't do when the hidden service is down, like it is 99% of the time.
>>43423710
You can't. The advice I got from a former staff member was to use a VPN for registration and Tor henceforth, but the complete and utter disregard for stewardship exhibited by Derpi has somewhat obsoleted that guidance.

>>43423633
Does it prevent you from just viewing the site?
I don't mind the block of anonymous posts from tor/vpn.

File: this isn't the derpi dram(...).png (86 KB, 1203x310)

86 KB PNG

>>43423919
I don't know how many times I have to repeat this, but yes. Derpi forces all Tor users to sign in via its .onion hidden service to access any part of the site. This hidden service is exceedingly unreliable. Result: Tor access is virtually impossible.
Mind you, they also gave me a permanent-turned-51-day ban for the simple act of using Tor, so this apparent incompetence could very well be a veil over malice.

>>43423708
If it were just Tor, I might agree with you. However, the staff has proven time and time again that they're nutjobs, not worthy of any respect.

How hard is it to set up a desuarchive alternative?

>>43424645
Not that hard for private purposes. The hard part is getting it online and then making sure nothing lawbreaking exists on it.

File: LilPolish.png (244 KB, 708x727)

244 KB PNG

>>43424653
>making sure nothing lawbreaking exists on it
You'd do that regardless of putting it up online.
...Right, anon?

>>43425008
I don't archive 4chan data, but I would do that but only if I came across it, I wouldn't want to bother sifting through swaths of gorespam or pizza or other trash. I value my sanity too much for that.

File: hard.gif (869 KB, 498x371)

869 KB GIF

>>43412483
>the console freezes until the application terminates
I have no idea if you knew this, but often you can blindly type "reset" and hit enter to recover from such situation without closing the terminal window.

>>43425138
User input is only processed when something reads it. If the application does not take the user input from the pipe then when it terminates the next application will grab it. For example the shell. But while the application is running the inputs just queue up. So blindly typing reset won't do anything.

>>43425171
Then hit ^C and then type reset, or ^Z and then type reset.

>>43425189
Good luck with that when the application is running inside an IDE where there is no shell underneath.

>>43425195
>using an IDE
>instead of writing your code in a terminal using here documents
ngmi

File: 1762246591351559.png (287 KB, 962x846)

287 KB PNG

apparently trixiebooru got blocked in China, so users are discovering that twibooru exists as well

File: herro!.png (169 KB, 503x602)

169 KB PNG

>>43426088
Well, let me be the first to welcome them.

>>43426088
>The site run by commies gets blocked in China
Kek.

>>43426088
ANOTHER W FOR THE GLORIOUS KIRINESE COMMUNIST PARTY

>>43426117
That's a good one.

>>43426088
>ponies blocked in china
>meanwhile g4 is huge in china

>>43426897
Even China recognizes the folly in letting Marxists control pony.

>>43423633
>>43426088
Curious.

Bumo.

>>43428475

Is anyone ITT hosting a panel at Mare Fair?

Small issue with the PonerPics scraper, I am getting a 403 response from Derpibooru trying to access for example https://derpibooru.org/api/v1/json/images/1. I can access it from the browser just fine, but when performing requests.get from within python 403 seems to be all I get. I didn't see anything in the api documentation about this. My scraper respects all posted limits. Any suggestions? This seems to have been an issue over the past few days.

>>43429188
>Any suggestions?
Copy paste your browser's user agent and cookies. Will work temporarily.

>has to scrape 120k pastes from pastebin
>1 request/sec API limit and I have to make 2 calls for each paste
>can only scrape for ~12 hour a day
It's going to take a while...

>>43429451
If you give me the list, I can help.

>>43430056
It's not just an easy single list. Its a moderately complex script which first tries to fetch it from pastebin, then known paste dumps, then the internet archive while also handling issues like passworded pastes or generic errors. And the internet archive might contain multiple snapshots of a given paste.
It will take like a week or two at this rate.

>>43429316
Looks like the issue mostly went away on it's own. I considered doing this, but was wondering if there might have been a more subtle cause that could be avoided. Scraping has been resumed, but I believe it's seeing more of the middleware challenges so it's going fairly slowly. Whenever the scraper sees one it currently backs off for 30min before trying again.

>>43429185
I am
It's about Twilight Sparkle

>>43430429
If it's the "I'm not a Robot" challenge where you just click a button, all you have to do is submit the form with the fields that are already there in hidden inputs. It does literally nothing else. You don't even need to store a cookie, it whitelists you by IP.

>>43430657
Yes, I am referring to the "I'm not a robot" challenge. There does not seem to be an IP whitelist as I've completed the challenge multiple times. Per the API docs this challenge will be issued based upon global API usage, and not your individual use. It says all you need to do is back off for 5 seconds before continuing. (The 30 minute delay is my catch all for any issues that I haven't made a special case for).

>>43430676
I get the same just by using the site normally (clicking on links from eqd). Does not seem to whitelist my ip, at least not longer than a few hours.

>>43430657
>>43430676
>>43431008
Literally every cuckflare website is getting hammered recently, it looks like, including 4chinz captcha.

>>43431012
Every website is getting hammered. Not a flaredhorsecock exclusive issue.

>>43431012
"I'm not a Robot" on Derpibooru is not Cloudflare. It's their own separate thing.

>>43428852

>>43431102
Who would do that to destroy the Internet?

>>43432618
Anon, I don't know if you ever heard of governments wanting to have total control over what people can see. One way to make it impossible for "small" sites to function is to ddos them constantly. Now in the age of the AI it's simpler than ever and the excuse is "AI needs to be fed".

File: death colgate.gif (3.08 MB, 389x452)

3.08 MB GIF

>>43432874
I was being facetious and forgot to attach the image for it. Pardon my retardation, i had just woken up.

>>43431950

With Linux having fully embraced AI coding, how long until pony sites accept machine-written code?

>>43433628
Philomena (most of the boorus) is written in a language nobody understands but one guy who wrote it: elixir.
Booru-on-rails (twibooru), danbooru (lyrabooru) is written in another language with little adoption: ruby.
Foalfetch is also ruby.
Both are hard for AI since there is barely any samples it can learn from. Okay maybe it would work on an ok level with ruby, but definitely not elixir.
Ponepaste is php. AI could work with that pretty well.
SunBooru is python. AI can work with python extremely good.

For front end (html/css/js) AI is extremely good.
What makes you think the devs don't already use AI tools? One already admitted it he does.

/rant
I recently tried claude at work and it is fucking amazing. I have a codebase which is 2+ million lines of java. To set the project up you need like a good week at first. It is that fucking bad and unnecessarily complex. I had a deployment issue today and spend a solid 4 hours trying to solve it. Then I fed it to claude and in 15 minutes if figured out I have two conflicting lines across about 100 configuration files which made the class loader do funny things. Would have spent like a day or two to figure this out myself. Oh and the AI did not get any help besides a brief "fix it" prompt and how to start the application. The best part is that these config files are hardcoded in the repository, but no one spotted the issue because no one was stupid enough to try running tests on CI. This is a 25+ year old project btw. Without proper CI tests. This is what enterprise software looks like.
/rant

>>43433784
>little adoption
Yeah, sadly Ruby was only ever used by tiny, little-known startups, like AirBnB, GitHub, Twitch and GitLab.

>>43433797
Anon, just because a few big companies use it that does not mean it's widely adopted.
By this logic, Fortran is the most widely adopted programming language! All the banking systems use it without exception!
Oh and ADA too! The F22 aircraft's code is in ADA! They spent over 50 billion dollars on it so it must be extremely widely adopted!

>>43433628
I don't really think there's any problem with machine-generated code as long as a human curates it. I'm not a coder though, so I have no experience with how well code produced by AI works. It could very well be it makes the same coding "isms" as it does when roleplaying a story, stuff that would make any proper coder roll his eyes and think "Of course it'd choose that brainlet solution to the problem".

Is there a proper front-end to the pony archive? It is almost completely useless, impossible to find anything on it.
By search I mean for example I want to search for a specific youtube video by title or author by name. Or even by video ID. The current channel ID folder names are impossible to use if the channel was deleted.
Also it does not seem to be updated in the past 2+ years.

>10

File: 3379993.jpg (1.47 MB, 1922x1079)

1.47 MB JPG

Take my hand, alt-sites
Let me show you the way out of Page 10

>>43435320
>>>/trash/

File: file.png (150 KB, 1603x750)

150 KB PNG

>>43433784
>Both are hard for AI since there is barely any samples it can learn from. Okay maybe it would work on an ok level with ruby, but definitely not elixir.
am I missing something? both elixir and ruby have been handled well enough to be workable with frontier models for at least a year
>danbooru (lyrabooru)
danbooru even has an AGENTS.md and CLAUDE.md files available in their repo https://github.com/danbooru/danbooru/blob/master/AGENTS.md
their contributors use it frequently enough at least

also for elixir you can check that some are using it right now too at least in public repos https://github.com/search?q=%22Co-authored-by%3A+Claude%22+language%3AElixir&type=pullrequests (they're the ones disclosing it at least, since you could commit the code yourself without the coauthor thing)

>>43435485
>you could commit the code yourself without the coauthor thing
Don't be silly, anon. That would be lying! And we all know how honest our code monkeys are, don't we?

>>43433784
>spoiler
Based. If you don't know how to leverage AI, you will be left behind.
With that in mind, Tantabus importer when?

>>43436072
>If you don't know how to leverage AI, you will be left behind.
The people who don't understand this are the people who confuse "shoving AI into everything" and using AI tools.
>Tantabus
I would really like an importer too. Tantabus is full of shit, but there are a few gems in it.

>>43436072
The bus is going to fucking Auschwitz and we're going to be the passengers, not the driver. I'm more than happy with being left behind.

>>43433994
Yes, I can't remember the name of the site, though
Sorry

>>43436202
One of these days I'll bring an Aryanne plush to Auschwitz.

Whoever made the report on ponebin: it has been taken care of.
53 similar (pone)pastes got deleted too.

>>43436987
Context?

>>43437848
pizza order links

>>43437928
can't believe poneb.in is going after the ukraine's primary mode of income

File: delivery.jpg (40 KB, 640x361)

40 KB JPG

>>43438129

>>43435429
And yet the images that get the most faves and upvotes on the boorus are all human porn. Curious!
>>43435485
It's kinda sad that Twibooru is the only option, considering how Chinese users were kind enough to give explanations and add details on images to make them better. We'll lose all context now.

>>43438129
Kek