Torrents of full images from /gif/ - Adult GIF, released monthly. >>>/gif/ is a NSFW 4chan board. Info:- Contents: porn, random videos, LiveLeak-esque videos, and other videos which were too interesting for weak video sharing sites like YouTube.- Stats so far: more than half a million files, totals to more than one terabyte.- Why? Among other reasons, no one else is archiving /gif/, so I decided to.Previous: >>1231730
== History ==/gap/ began in 2022-10. Archive files from 2022-10 to 2024-03 only have full image files and not threads (plain text). Starting in this thread, including 2024-05, each release going forward should have threads.== How does it work? ==Every 24 hours, this happens:1. Full image links obtained 24 hours ago get downloaded.2. All /gif/ threads get downloaded in API/JSON format.3. Full image links get extracted from the JSONs. Goto 1.
*more than half a million video files (gif, webm)
bump, not sure if u did but it would be good to publish your exact setup and workflow for this just in case someone else might want to use it for another board or do it for gif themselves or continue doing it one day if you disappear
>trying to find archive of two specific threads, grab torrents>it's a zip instead of folders i can pick through properly but whatever>open one today>it's a flatfile dump>no index file, no metadata, filenames are presumably md5>go to archivedmoe to find the thread to pull the hashes out>the archive for that is 404 for some reasonI don't want to bitch too much because this is all still better than nothing, but please, I beg, an SQLite file, OR simply putting the thread number in the name.
>>1311696>flatfile dumpIs that what that's called? In the past I called a similar thing a "simple many-file folder" (which contains zero folders).>simply putting the thread number in the [file]name.Sounds like a bad idea. One reason: same files in multiple threads. Could have thread number of only the first thread it showed up in, but still.More data stuff - unrelated to your post... I run an ipfs node which is consistently online in one computer. I run another IPFS node which is inconsistently/temporarily online in another computer. I have a cid which is only in the temp-online one. After recursively providing that cid to the dht in the temp one (for hours, probably still doing it) I saw that its storage went from 666 gb total to 669 gb. Conclusion: recursively providing a dag/CID which you don't have to the DHT seems to make you download it (if running as read-write which you are almost certainly doing).
>>1310152is 2024_05 available yet?
>>1310152Fuckin piece of shit, I found a fault that shows that 4chan /gif/ 2024-06 wasn't being downloaded for days. I think I fixed it now. I could check on logs and stuff going forward to see that it's working correctly.
>>1311996 = day that I fixed that problem, has worked fine since then.
>>1310152What are you using for this? GChan?
>>1310152Where is the magnet link?
>>1314207See the previous thread. I didn't share 2024-06 and 2024-05 yet.>>1313666No
>>1310155Post source code plz
>>1314635>See the previous thread. I didn't share 2024-06 and 2024-05 yet.But the previous thread is 404 already
Fuck cloudflare captcha
>>1310152>>1310815>>1314645Setup: GNU/Linux computer, set .sh files to executable by running "chmod +x file.sh". I use simple/crappy code to download this stuff. My code does not enable /gif/ users to do remote code execution because it parses JSON in such a way that it deals with privileged bytes that are part of the JSON structure and not the contents (same thing with older versions of the code that I used which parsed HTML instead).Folder: /path/4changifFolder: /path/4changif/testFolder: /path/4changif/threads- Maybe sure you have those 3 folders created (replace "/path/4changif" with whatever you have.)File: cron.sh- ipfs://bafkreidzrvqgtjebkj7uqzibqjz6jri3ei3tvsjscf5i2lebejr5s3wgga / https://web.archive.org/web/20240628134551/https://sabrig1480.xyz/FSrXIjIxT1z4ndG9dx018rGq6MGD3cF6BUPNTXs2lu4- Checks if cron0.sh is running, if not run cron0.sh. Specify the full path to "cron0.sh".File: cron0.sh- ipfs://bafkreiclpya533snasw6crtg4f6dgdoikkuudsgnmvz4uw4s53f572ugvy / https://aralper.xyz/ciw3Aigohm54YTN-EwQO0C3xPOckjS3BjjYCipRjqf0- Main downloader. Change "basepath1='/path/4changif'" to whatever folder you have set up to download /gif/. Manually run lines 11 through 37 when first running it to kick things off. Make it so 11-37 lines are one command then run that command. After doing that, the downloads all happen automatically.- In depth on each line. 1=Bash. 2=basepath1 variable. 3=HDD history. 6=if statement 1, 86400-second wait between links obtained and downloading the files of those same links, runs if above that number. 9=runs commands to download files, logs it. 10=clears commands to run. 12=Does stuff, gets thread OP numbers from https://a.4cdn.org/gif/catalog.json . 14=threadcount variable which is a number of all OPs. 16=while loop 1, to go over all the OPs. 18=selects a specific OP (var ii). 20=debug output. 22=Downloads a thread https://a.4cdn.org/gif/thread/$ii.json > $basepath1/threads/$ii.json.$now1/?
>>131488624=imgcount variable which is a number of all images in a thread ii - calculated as a count of JSON parts>jq ".posts[].ext, .posts[].tim, .posts[].md5" | grep -v "^null$"divided by 3. 26=filename variable - array of POSIX time filenames from the middle $imgcount lines of those JSON parts. 28=ext variable - array of file extensions from the top $imgcount lines of those JSON parts. 30=md5 variable - array of Base64(MD5) strings from the bottom $imgcount lines those JSON parts (formatted to "standard" URL-safe strings). 32=while loop 2: saves commands to download each image into a text file (downloads as "TZ=UTC wget -nc https://i.4cdn.org/gif/${filename[$n]}${ext[$n]}" -> "$basepath1/test/${md5[$n]}" - cmds in $basepath1/torun.txt); end while loop 2. 34-39=iterate while loop 1, end while loop 1, end if statement 1.Ignore the/this "In depth on each line" section if you just want to use it and don't care about how exactly the code works. You can also replace each case of "/gif/" with "/mlp/" if you want to download another board. This skips downloading janny-deleted files, which is good and bad. Bad if it was some harmless video that got deleted because the poster was too based and got his post deleted due to politics. There's no HTTP archive of HTTP 4chan /gif/ files, so no thing to fall back on and check if it's a harmless file. In the /mlp/ example, there is. I don't have a thing to specifically record "found deleted", but you can look at cronlog2.txt for 404'd files if you want to use this script on other boards then check found-deleted against what's saved in desuarchive.org, for example. And since I brought it up, here's a one-terabyte torrent that an anon (not me) downloaded from desuarchive /mlp/ and other captures of /mlp/:>magnet:?xt=urn:btih:9671fb0855c7931fe98f03f7612c18010fb10121&dn=4chan-mlp&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a6969%2fannounce2/?
>>1314887Run "crontab -e" (NOT as sudo) and put this in there:>0 * * * * /path/4changif/cron.shcrontab runs hourly, cron*.sh runs daily. I guess I could simplify it to not be hourly->daily and just have crontab run it daily, but what I use works to only download it daily, so whatever.File: 404.txtFile: addext.shFile: howto.txtFile: 4chan_gif_2024_03_empty.txt- see the latest torrent, magnet:?xt=urn:btih:84b2a6b0865a26bac9b7deef0ba63f893d6931c4&dn=4chan_gif_2024_03.zipFile: cronlog2.txtFile: cronlog1.txt- automatically created, see the latest (4chan_gif_2024_03) for one of thoseFile: time.txtFile: chkcmd.txtFile: torun.txt- automatically created3/3 for now.
>>1310152>>1314207>>1314635Hey OP not to be mean or anything but why did you create this thread, just post the fucking magnet link, the previous thread is long gone from the archive
== Links to 4chan /gif/ 2022-10 to 2024-03 files? ==Those are all in thread #1231730:- with working CSS/JS (WARC in a parent folder): https://bafybeifn7bxeg34zc725kkjzfuxpbf2ftb5lgjpdh5r7hrdrrpf3zpat2m.ipfs2.eth.limo/raws/boards.4chan.org/t/thread/1231730.html- as a text file: https://gateway.pinata.cloud/ipfs/bafybeicqxg64e6u3ws3ietrlxao7nxjwuxkbc542gh5xw53quazkgttpbu - includes .gz version at https://shadow39.online/OdSsPIeOso2q8kY6lvckhJuSzB4YGI9hdOqum0AncVQ- folder bafybeic...tpbu also includes a text file which only contains the magnet links posted in that thread, with 4chan /gif/ ones at the top: https://utkububa.xyz/JsoPMGJhzMrtMCDblJjbXH8XFoa_9vfqzuKGNBfwNSw = https://gateway.pinata.cloud/ipfs/bafkreig3hnbw65gikxi2j62q7bgpsn2npx7dkxmfyzut5emvtumzzpwdzeAnchor: >>1310152Replied to: >>1314207 >>1314778 >>1315187
>>1315342yeah I'm not clicking that shit glowie bot
>>1315353Wow, you are dumb. Here's that same text file:https://files.catbox.moe/3nd5c2.txt>text file which only contains the magnet links posted in that thread, with 4chan /gif/ ones at the topwhich is also here:https://ipfs.hypha.coop/ipfs/bafybeia7nmoydnj2d4gymp6rlpdpusozcip7x5znpax7gfprpfgx3wiaii/kill_all_retards.txtNow go back to watching CNN.
>>1314896Thank you. Not trying to be annoying but, putting this somewhere like a GitHub equivalent would be cool (we should probs stop using GitHub at this point because u are training an LLM with every commit)
>>1310152Great initiative anon.Looking forward to 05 and 06. Got all the other ones already. Im archiving some threads myself, mostly wsg and pol stuff.
Done: stuff such as segmenting 06 into its own folder, created folders for 07, updated howto.txtTodo: info on 05 and whatever
>>1314207>1231730>>1314635https://archiveofsins.com/t/thread/1231730/#1311161
>>1316804Guess I will work on 05 "soon". It's here:/zc/z9/4chan.org/gif//zc/vid_4mb/>4mb/wsg/ seems to have raised their max file size limit to like 6MB:https://boards.4ch an.org/wsg/thread/5612597/pol-politically-incorrect>>>/wsg/5618193 - can upload a video where upstream size=6MB, or is that derived size?Maybe /gif/ raised that limit too.
>>13179984chan_gif_2024_05 is like 10 GB for certain reasons, and I'm just gonna have to accept that for now. In order to get the rest of I need like 100 USD so I myself can do a HDD repair, 1000 USD to get a "professional" HDD repair. I wish this was a data ransom situation because that would mean I have all of the data for that month and am pretending not to have it. I don't have a big part of it, so less motivation to release 2024_05 which I felt like releasing before releasing subsequent months. It's been 2 or 3 months and I haven't got on 05, now at the acceptance part of the stages of grieving, so I guess I will be more likely to get on it soon. Trump catching an octopus AI video from the following (can't attach extensionless WebM file then post it to 4chan):>4chan_gif_2023_05 , https://gateway.ipfs.cybernode.ai/ipfs/bafybeigkplwidoyprmm7vyb2qlna7o2sgq26ydrg55nrqhf4xfcga3gxsuFUCK 4CHAN'S KEKFLARE CAPTCHA!!!!FIX THIS SHIT MOOT!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>1320059Oh look, a new wordfilter. I hate this website even more now. It's C U C K -> KEK.
>>1320059For 4chan_gif_2024_05, I'm on step 9 out of 29.
>>13205744chan_gif_2024_05 is available in IPFS:file:///ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4- packed and non-packed version- extra dataIt'll take some time to share a torrent of it. A highlight from that folder:https://gateway.pinata.cloud/ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4/music/xdSuXide_UCLnUxuqdNQeoh3YvH5mlKKA.partial/MLP_-_Friendship_is_Witchcraft_Pinkie_s_Brew_By_Sherclop_Pones_PMV_Lyrics-xdSuXide-20230526-youtube-1280x716-shFfnG6GRhA.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-05:magnet:?xt=urn:btih:dc88d6317891957e462a0af085c2f5d1cee33127&dn=4chan_gif_2024_05.zip&tr=udp%3A%2F%2Ftracker.deadorbit.nl%3A6969%2Fannounce&tr=http%3A%2F%2Fcn.pcfreetime.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.srv00.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mirrorbay.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fbandito.byterunner.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.darkness.services%3A6969%2Fannounce&ws=https://ipfs.ssi.eecc.de/ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4/imageboard/4chan_gif_2024_05.ziphttps://ipfs.apillon.io/ipfs/bafybeiel5yavn7jicgloferdbzwrxsojeeaqnhrkhglu3bw74xny23hqli/4chan_gif_2024_05.zip.torrenthttps://files.catbox.moe/4kqtzd.torrent----Anchor: >>1310152. Music for this release - "Pinkie's Brew" by Sherclop Pones:https://gateway.pinata.cloud/ipfs/bafybeidhaffpxuf7ar524yhswvtvsddxiakzyd4nijd777jkb5onjchzaiThese various musical "Pinkie's Brew" videos are related to time traveling into the past. I kinda wish I could go into the past and prevent myself from making certain mistakes, including as related to this /gif/ release for the month of May.
Release for 4chan_gif_2024_06 shouldn't be missing any files...
How do you know what webms came from what thread? It's a pain to open each video and delete the unnecessary ones.
>>1310155Was 2024_04 never done?
>>13211474chan_gif_2024_06 is missing roughly 2 or 3 days >>1311996. I'm creating the .zip for 2024_06: it'll be like 70 GB.
>>1323239I finished making that zip some time ago. Also finished creating a dag of the packed+nonpacked versions some time ago.Search became horrible in 4Chan archives.- https://archive.4plebs.org/pol/search/text/twitter/type/op/ - cuc kflare; archived.moe=cu ckflare+search not enabled for /pol/- https://archive.4plebs.org/tv/search/text/%22your%20honor%22 - c uckflareI have some positive experience using Selenium, so I can still get those webpages in a semi-automated way, even with those annoying anti-human and anti-good-bots captchas.
4chan_gif_2024_06:BitTorrent release: todoIPFS release:/ipfs/bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74- has extra data; in MFS, not pinned- packed + non-packed version = 138.94 GiB- previous version/pin:>ipfs pin update --unpin=false bafybeibie5qrgjalyghiczrj7q4dwserzpci6bsrytl63x7d67wx4fp3ge bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74- previous + this = 170 GiBIt'll take some time to share a torrent of it. A highlight from that folder:https://gateway.pinata.cloud/ipfs/bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74/music/yt/Kim_Jae_keyoung_UCv-in54UcjPqdcLuRLxdKLg/My_Little_Pony_AppleJack_-_Apple_Jack_Lisa_McHugh-DWr1SKWRM7c.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-06:magnet:?xt=urn:btih:b90ef006b7f7e4d069d310472f9a32413813f4a3&dn=4chan_gif_2024_06.zip&tr=udp%3A%2F%2Fmoonburrow.club%3A6969%2Fannounce&tr=udp%3A%2F%2Faarsen.me%3A6969%2Fannounce&tr=http%3A%2F%2Fdht.dhtclub.com%3A666%2Fannounce&tr=udp%3A%2F%2Fnew-line.net%3A6969%2Fannounce&tr=http%3A%2F%2F1337.abcvg.info%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.skynetcloud.site%3A6969%2Fannounce&ws=https://bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74.ipfs2.eth.limo/imageboard/4chan_gif_2024_06.ziphttps://eu.starton-ipfs.com/ipfs/bafybeid5cl6sc47bwg4rpuyxwrlvxfs55uof7yqa23az5xzfvcgibowpp4/4chan_gif_2024_06.zip.torrenthttps://files.catbox.moe/a708r6.torrent----Anchor: >>1310152. Music for this release - PMV of Apple Jack by Lisa McHugh ("My Little Pony AppleJack - Apple Jack(Lisa McHugh)"):https://gateway.pinata.cloud/ipfs/bafybeieybwwjoesqzune4aixpxab55723hfcd24wnichykrudz357zki6yThat's a folder to a copyright-deleted YouTube channel which contains the original WebM + derive MP4 (>>1323761).
*derived MP4. Size of 4chan_gif_2024_06.zip = 69 GiB (contains threads+full images). 4chan_gif_2024_07 shouldn't be missing any files, but I think it extended too far into 4chan_gif_2024_08...
>>1310152Serious question, deleted videos are there too? some images are illegal in some countries.
Upload on 4chan_gif_2024_06.zip was slower than wanted: avg. ~165 KiB/s. 100% of it finished uploading roughly 16 hours ago. Upload for 4chan_gif_2024_07 should be faster: I have a couple solutions for this. If one doesn't work, I can pause for a while to work on plan B.
4chan_gif_2024_07: on step 10 out of 28. Captcha: DART
>>13250704chan_gif_2024_07 is >100 GiB; step: 25 out of 28.>>1308979 [~2024-06 post in previous /gap/ thread]>Hope to put those blocks back online sometime this month.I was listening to tracks in this small folder, such as>/ipfs/bafybeiexmw4i57jgr3esuda7q2n4i45erll7vkaiask4t2tbpkxacaxove/keygen-music/KEYGENMUSiC MusicPack/!Others/909DEAD - Adobe CS6 All Products activator.it>Title: Pokemon RBY Lavender Townand was reminded of that. 4.7-GB keygen_music folder is back online: /ipfs/bafybeibeuoggietcp2dt3qtnsp6ul7cwcuubo4mmvswwzyb652me2cs6ou/ (also in 2 HDDs). Just put it together (realized it) today: hovering over a color word in Konsole version 21.12.3 shows a box of that color. I was confused about "randomly" seeing that GUI square in the past. So if you hover over "lavender", "green", "red", etc. (text in Konsole) then it will show a square filled in with that color.
>>13253172024_07: writing metadata. Will make another copy after that finishes.>>1325317>small folderThis track sounds familiar: "ANGELS + DEFJAM - Ranx English intro.xm" ("Title: daley").>CID back onlineAlso back online - contents of deleted upload https://archive.org/download/mega-dance-hits-collection-2-1990-2001 which I downloaded some time ago: /ipfs/QmWRdfw7YxaXY38wnFNvYaR9J1oNtJEeo6jNzsn827mbX2 (13 GB). It's more than 1000 dance music tracks (in 2 HDDs). Like bafybeib...s6ou, it would have been better if I stored it as raw blocks (it's not), but whatever. (Neither are repinned, yet - will maybe do that later after doing stuff with /gif/ 07.)
4chan_gif_2024_07:BitTorrent release: todoIPFS release:/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli- has extra data; in MFS, not pinned- previous + this = 424 GiB- pin for this month only: /ipfs/bafybeidjxg6jqs5p5oakwfflaijvisxx2d5fhjbw64xtcgrhgmna37g7mu- packed + non-packed version = 226.39 GiBHighlights from that folder:- https://gateway.pinata.cloud/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/music/BFUtEoT/MLP_S7_D4.ISO-VTS_10_1.VOB.mp4- https://gateway.pinata.cloud/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/music/BFUtEoT/Montgomery_Gator_UCeaHgmeXQg_5o_Zrmlupsig.partial/Best_Friends_Until_The_End_Of_Time-Montgomery_Gator-20200902-youtube-1200x1200-AAWSawGr0dk.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-07:magnet:?xt=urn:btih:352c3ee75397c6ffde9b6786e01441990bc57641&dn=4chan_gif_2024_07.zip&tr=udp%3A%2F%2Fipv4.rer.lol%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.ddunlimited.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fseedpeer.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.artixlinux.org%3A6969%2Fannounce&tr=http%3A%2F%2Fopen.tracker.ink%3A6969%2Fannounce&tr=http%3A%2F%2Ftracker.dump.cl%3A6969%2Fannounce&ws=https://gateway.ipfs.cybernode.ai/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/imageboard/4chan_gif_2024_07.ziphttps://eu.starton-ipfs.com/ipfs/bafybeifu7oeuip4rtt2vh4pknt4ymfpje6wsicztazsahpmticwd6yjxy4/4chan_gif_2024_07.zip.torrenthttps://files.catbox.moe/gsjz1h.torrent----Anchor: >>1310152. Music for this release - "Best Friends Until The End Of Time" from MLP:FIM (incl. singalong video from the DVD):https://gateway.pinata.cloud/ipfs/bafybeifr7ksfksrdvett6b6va6bkprvhbmvgnvi3u7djynz6f3t2zhsdde4chan_gif_2024_07.zip.torrent created in about 13 hours for 121,465,843,003 bytes (113.12 GiB) = 2.6 MB/s.
What are all these magnet links about?
About 15 hours ago my average upload speed on 07 was approx. 256 KB/s. Had something annoying happen recently (which is still a problem or an annoyance): not messing with that now, so the upload speed should be better. (300 K/s avg. should be the minimum goal, and perhaps I can get that on 4chan_gif_2024_08.)
Someone please guide me through how to use the MD5 hashes from the file name to find the related thread. Archive.Moe is giving me the "Whoops, looks like something went wrong." error.
>>1327935Look in ./4chan_gif_2024_07/threads/ - JSONs show which post in which thread has which MD5 filename.
>>1328080Thanks!! And for the older torrents that don't have those JSON files?
>>13282372024-05 and after all have threads. For before that, tell me a good way to mass download those threads if I don't bother to figure it out myself. Archived.moe disabled search months ago. Maybe with their API thing I can still get all pre-2024-05 threads from /gif/ which archived.moe didn't delete. Unlikely, because IIRC the api requests for search are as weighty as the non-api or web scraping requests for search.
>>1328242Yeah I'm no help there :(. Is there a way to convert MD5 hash to plain text? I just want a way to organize these files, It's like 40,000 webms per archive.
>>1328254>Is there a way to convert MD5 hash to plain text?So you mean this, basically: Base64(MD5) -> MD5? How to do that:>$ wget -qO- http://15.185.172.73:8084/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/imageboard/4chan_gif_2024_07/_tAmk9anxa1bJKWrAjLzXA.webm | md5sum - # video about Jewish scammers next to the Wailing Wall in Israel>fed02693d6a7c5ad5b24a5ab0232f35c ->$ echo -n _tAmk9anxa1bJKWrAjLzXA | sed "s/_/\//g" | sed "s/-/+/g" | base64 -d - 2>/dev/null | xxd -p>fed02693d6a7c5ad5b24a5ab0232f35c>$ # matchesSo, MD5+B64 string -> /_/\// -> /-/+/ -> Base64 decode -> hexdump
>>1328694Then what do I do with the hexdump file (fed02693d6a7c5ad5b24a5ab0232f35c)? Hex to text converters don't seem to work. Unless I'm stupid and supposed to be doing something else?
Guess I'll work on 4chan_gif_2024_08 after certain parts of HDD z2 finishing copying into HDD zc.>>1328724What exactly are you trying to do? What text did you expect to get out of that hexadecimal string? The original filename? Like "cupcakes.webm", for example? I think you misunderstand some things. Filenames in the packed files in these torrents are in this format, which is what 4chan and 4chan archive HTTP websites uses:>filename = urlSafe(base64Encode(reverseOperation(md5(file))))Explanation:- file: a webm or gif video.- md5: this is a non-cryptographically-secure hashing/checksum algorithm; it does mathematical operations on a segment of data, like a file, then outputs a set-length deterministic hexadecimal string (32 characters). MD5 hashes, each hex string, are basically non-reversable, and when hashing a file, the hash does not cover any external metadata for the data, such as the file name and file mtime (timestamp). Deterministic means that exactly the same data = always the same hash; cryptographically secure algos include SHA256, which I think is used in BitTorrent v2, and SHA512.- reverseOperation: takes a hexadecimal plain text string as input then converts that to binary data- base64Encode: in this case, this takes said binary data as input and encodes it as Base64 plain text- urlSafe: does certain text character replacements (different from a $urlsafe "programming module" which I wrote). Base64 can output these characters -- + and / -- which don't work so well as-is in filesystem paths and in URLs. Base58 is used in IPFS and it's like a safe text and more readable version of Base64. Base32 is also used in ipfs, and it's lowercase text-safe characters only.Read about MD5 message-digest [toy] algorithm with no donation begging:https://dweb.link/ipns/en.wikipedia-on-ipfs.org/wiki/MD5Or, I could elaborate.
Correction:filename = urlSafe(base64Encode(reverseOperation(md5(file)))) + "." + fileExtension
>>1325318>dance music set of deleted MP3s from IA "MEGA Dance Hits Collection 1990 - 2001 (Lossless)" back onlineI see that I am missing some of those blocks, but I think I have most of that data. If I can't get those blocks in some other HDD, then I hope that the probably-matching torrent is still online:https://gateway.pinata.cloud/ipfs/QmYdjVNCqqLt8PFypWK9TzdQMNMS9TG3twppjwoRJUQKnr
>>1328871>Guess I'll work on 4chan_gif_2024_08 after certain parts of HDD z2 finishing copying into HDD zc."Finished" some time ago, so...
>>1310152Ah shit, checked crontab and saw it was empty for some reason = missing >20 days = 2024-09 will be missing a bunch of files. 2024-08 is missing ~4 days.
4chan_gif_2024_08:BitTorrent release: todoIPFS release:ipfs://QmQn4yuGcGBwXBvt3prxZ3NtFgQxQARt1fA7cTZvc1Uc5N/root/video/4changif/imageboard- has extra data; in MFS- previous + this = 435 GB- packed + non-packed version = 22 GBHighlights from that folder:- Friendship is Magic fan-made video with music: https://ipfs.hypha.coop/ipfs/QmQn4yuGcGBwXBvt3prxZ3NtFgQxQARt1fA7cTZvc1Uc5N/root/video/4changif/music/Alexander_Lexiel_UCDBpby7C_XlAVzqtFh1hxAQ.partial/Andrew_W._K._-_Ready_to_Die_Cupcakes_~_Lyric-Alexander_Lexiel-20130925-youtube-1280x720-w1HAkWopRwU.mp4- Crackhead strength: https://ipfs.ssi.eecc.de/ipfs/QmQn4yuGcGBwXBvt3prxZ3NtFgQxQARt1fA7cTZvc1Uc5N/root/video/4changif/imageboard/4chan_gif_2024_08/lTXTzYOAq4c9VSLqnpo3Og.webmTOR release:http://ponypalsh4y6olziyjlswfv674utokqhz3y6beym2erqtstcgadmacid.onion:81/ipfs/QmQn4yuGcGBwXBvt3prxZ3NtFgQxQARt1fA7cTZvc1Uc5N/root/video/4changif/imageboard/
>>1331632>Crackhead strengthAttached. See that link for the original version which has audio commentary.
>>1331632the fuck is IPFS
>>1331772Inter-planetary file system, a different type of decentralized file hosting from torrents.
== Presenting ==Full images of 4chan /gif/ - 2024-08:magnet:?xt=urn:btih:985c29c86c49b3464a9c22239fd91e7cc05bd64e&dn=4chan_gif_2024_08.zip&tr=http%3A%2F%2Ftracker.ipv6tracker.org%3A80%2Fannounce&tr=https%3A%2F%2Ftr.ready4.icu%3A443%2Fannounce&tr=udp%3A%2F%2Fconcen.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.therarbg.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.dler.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fodd-hd.fr%3A6969%2Fannounce&ws=http://15.185.172.73:8084/ipfs/QmQn4yuGcGBwXBvt3prxZ3NtFgQxQARt1fA7cTZvc1Uc5N/root/video/4changif/imageboard/4chan_gif_2024_08.ziphttps://ipfs.cyou/ipfs/bafybeiglcqp6i3adwwuwhm3dsfsjw4s32uqxm7igpkwb4na4sqbayrpcxu/4chan_gif_2024_08.zip.torrenthttps://files.catbox.moe/zr6k4s.torrent----Anchor: >>1310152. Music for this release - "Andrew W. K. - Ready to Die (Cupcakes) ~ Lyric" (fan-made MLP:FIM video with lyrics):https://ipfs.hypha.coop/ipfs/bafybeiainvo2hj2xxepb3haaxmqmh66wkqytqyrrxjgecygdmrdiowhohm
>>1331816I was reading this comparison sometime ago, found it to be kinda interesting:>https://ipfs.hypha.coop/ipns/docs.ipfs.tech/concepts/comparisons/>IPFS comparisons>IPFS is a general-purpose file system that uses a distributed hash table (DHT) to route and transfer content-addressed data. This sets it apart from other solutions with a more specific focus or use of a specific data storage mechanism. For example:>- BitTorrent (opens new window) is a peer-to-peer (P2P) file-sharing protocol that uses a centralized tracker to manage the distribution of files among peers. It focuses on file-sharing rather than file storage.>- Storj (opens new window) and Sia (opens new window) are decentralized cloud storage platforms that use distributed networks of nodes for data storage. They focus on providing cloud storage services rather than a general-purpose distributed file system.>- Arweave (opens new window) is a decentralized, permanent storage platform that uses a novel data structure called a "blockweave" for data storage. It focuses on providing permanent storage rather than a file-sharing system.>- Filecoin (opens new window) is a decentralized storage network that allows users to rent out disk space. It focuses on providing a decentralized storage marketplace. It uses a proof-of-replication consensus mechanism and supports payment in various cryptocurrencies.> Filecoin is built on IPFS and uses the IPFS network for data storage and retrieval. Filecoin and IPFS are complementary technologies providing decentralized and efficient storage solutions.>- Hypercore (opens new window) is a decentralized data-sharing tool that uses a distributed hash table (DHT) for data storage. It focuses on enabling data sharing and collaboration.>- Holo (opens new window) is a decentralized hosting platform that uses a unique data storage and sharing mechanism called Holochain. It allows users to host and run web-based applications on a peer-to-peer network.>- Swarm (ope[...]
I guess that torrents will be released soonish!
Text in the previous thread:>Convert Unix time to UTC time = IDK, so here's the other way around:>$ date -d "2024-04-02T13:37:00.536419346Z" +%s>1712065020Unix Epoch -> GMT:>$ TZ=UTC date -d @1700000000>Tue Nov 14 10:13:20 PM UTC 2023
>>1335046I said 11 days ago...I was working on this one project but it ended up being fucking sad or disappointing, so to speak. Aiming to work on /gif/ torrent today or tomorrow.
>>1331632>BitTorrent release: todo>09/24/24its never coming, is it, anon?
>>1337578look 4 posts below that post, it's right there: >>1331848
4chan_gif_2024_09: on step 11 out of 26
>>1337720when
To add to "howto.txt":view found>$ #cat ../404.txt | xargs -d "\n" sh -c 'for args do stat --format=%n "$args" 2>/dev/null; done' _ | xargs -d "\n" sh -c 'for args do echo "$args"; mpv "$args"; done' _delete filenames from missing list>$ cat ../404.txt | xargs -d "\n" sh -c 'for args do stat --format=%n "$args" 2>/dev/null; done' _ | xargs -d "\n" sh -c 'for args do sed -i "s/..$(echo $args | sed "s/^..//g")//g" ../404.txt; done' _>>1337727I'm now on step 17/26.
video from upcoming release
>>1337735Also these:./9rwJ4y16gHSl03OwqnFJPQ Russians vs. Ukrainians, not a boring video./CMD212AhOEAtCqp-odyGQA attached./ETMgECUp203fvJ1LAXU3yg next attached
>>1337736
>>1337737More videos of some interest./FICPFAQuEog_OO3exnHu2Q attached./Gqk0-4GhnrY1JABB8iq2NQ a Russian made this homosexual compilation video, TF2, video games./HNZN5bBhLP-WnjRBQwTHlQ trap/shemale./HRK8uXglhcM34_Z1R-Z41Q trap/shemale./KOTYi0-nzcMvaJjgtL3Nyw amateur porn, better with audio, next attached
>>1337738
>>1337738./LXx_Yk4LEXbbImDLTc3tMA boys, don't hit girls; girls, don't be a bitch and pick a fight with a boy because you'll likely loose./LjLTHA0-UiDL4w1MGptueA "[...]Allah would side with a white Christian man over a Muslim woman[...]that bitch[...]it's a fella's religion"./MYH4LznPJcnUP_H6UtOg-g watermarked with "Do Not Share"; irony: it's a dumb farting video./NPpJKiYOEkvIe-8Zx0GAQg next attached, music, better with audio
>>1337741Pajeet sings the song of his people./RJcauKXxHuv2ARnECpQ2Ew shemale on male, BDSM./ShqorByqG4Ta416XpzgJfw not much debris left at the foundation/base from WTC 9/11 event: because the towers were exploded with explosives planted inside of the buildings, better with audio
>>1337734finished step 17>>1337742>ShqorByqG4Ta416XpzgJfwattached
4chan_gif_2024_09: 34,793 files amounting to 83G>>1337738./TE8qanyb2Hz3llCOzGpf2g sex toy, material science, rubber/plastic/silicon, position./ULHpKdxDwkW4QUkbTzuzmQ Peter Griffon obtained crack from a nigger
>>1337738./VTBcoV8jx8WxRC5rxySKVA Russians vs. Ukrainians, odd, odd music./YovS1vfyFyWKep4L6P0jGQ Ukrainian, attached, better with audio, shortened to <1min
4chan_gif_2024_09: on step step 22 out of 26more videos of interest./_j51cxFfN9Vuvljbvp9YlA hunting video, domed, animals./ci00UiQ7gTi5qcW97A5HVg WTC, next attachment
>>1337746./dmEyubO43BwMFJs7GRHXAA bitch whore in office job, unprofessional./eRQtCzdIytKCdSKl4rBbQA Hillary Clinton: "we're all going to die"./gdozM46nWWVR_yQFC38lJA black shemale./hJArS5vMDjoGbRedHxvnbg unique sex position, next attachment
>>1337747./iwxkKo3SLVaBHdEy23AWXw my body's neat, your body's neat, no trannies./jS1V2YIdKfP4_9nEVSIfdA colorful/beautiful horsecock dildo, interesting/informative video, next attachment
>>1337748>jS1V2YIdKfP4_9nEVSIfdAover here: https://files.catbox.moe/4k8c5p.webm./l7_0fH-Qppe_MVDYlOdmEA sex with a shemale, would be better with no negro./liuJ2uNS1ZHtDyDjs4qe7w WTC, 3m25s, "free energy" nonsense IIRC./otCg2f82eTmcdZZImCC7rQ female
>>1337749./pi5NPS6FYYFOlbCTGSziOQ balls./sB1PHCu11HXTSzdBbW7bBg Trump is bad for Jews./tKjZUHLjW1xK-ZC3p8B0wg dumb child gets ran over by a car./xS8u_SR0CyN-iSYVaZa_XA shemale, maybe better without the pissing part./xkt_d7rQVv_ZkYDdYH534w shemale, small penis, next attachment
>>1337750
He reminds me of Heavy Weapons Guy from "Team Fortress 2". from a previous release
>>1337746closer to release now. I have: packed + nonpacked versionsUse vim to parse a text file for IPFS CIDs>:%s/\(Qm[a-zA-Z0-9]\{44}\|bafyb[a-z0-9]\{54}\|bafk[a-z0-9]\{55}\|bafyk[a-z0-9]\{57}\)/\r===-notthistext-===\1/g Copy them to these locations in MFS:>/shared/cid/$args
4chan_gif_2024_09:BitTorrent release: todo, see belowIPFS release:ipfs://QmYehbqHKvPGWgpCdDFqaDdQKzd9XyRBrEdcsAr5cRiSTh/4changif/imageboard/- has extra data; in MFS- previous + this = 624 GB- packed + non-packed version = 176 GBHighlights from that folder:- GIDDY UP: https://flk-ipfs.xyz/ipfs/QmYehbqHKvPGWgpCdDFqaDdQKzd9XyRBrEdcsAr5cRiSTh/4changif/music/gu/Tweek_Studio_UCUxJcaaE2x3s-AaZNyLkV6g.partial/Giddy_Up_-_PMV_Animation-Tweek_Studio_Animation-20150717-youtube-1920x1080-G79UKneSjDQ.mp4- Based on: https://4everland.io/ipfs/bafybeigb3ddefqvwi2v42q5gj2ohxxx5ap3lteswrr7pqe447rxlbrmn5y/music/gu/EquestriaInhabitants_UCAT2JZazPSioOf1CAV1szrQ.partial/MLP_-_FiM_Season_3_Commercial_2_1080p-EquestriaInhabitants-20121111-youtube-1920x1080-2QopwDXDomQ.mp4Tor release:http://ponypalsh4y6olziyjlswfv674utokqhz3y6beym2erqtstcgadmacid.onion:8080/ipfs/Qmf4HbBPX3tFY8iHR1PxM3ZKyYwCsSb1fnqXXTo3KDuZEn/a/root/video/4changif/imageboard
== Presenting ==Full images of 4chan /gif/ - 2024-09:magnet:?xt=urn:btih:a987a4f0a35757c848c44923e3e743d766121b98&dn=4chan_gif_2024_09.zip&tr=udp%3A%2F%2Fipv6.fuuuuuck.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftr4ck3r.duckdns.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fu6.trakx.crim.ist%3A1337%2Fannounce&tr=https%3A%2F%2Fp2p.azu.red%3A443%2Fannounce&tr=https%3A%2F%2Ft.peer-exchange.download%3A443%2Fannounce&tr=udp%3A%2F%2Fwww.torrent.eu.org%3A451%2Fannounce&ws=https://ipfs-comet.cybernode.ai/ipfs/QmYehbqHKvPGWgpCdDFqaDdQKzd9XyRBrEdcsAr5cRiSTh/4changif/imageboard/4chan_gif_2024_09.ziphttps://gateway.ipfs.chaingpt.org/ipfs/QmaAT63FujckgM7oca5TGovdDRqUUgt87h3VvfE9xnzZLc/4chan_gif_2024_09.zip.torrentmirrored at https://files.catbox.moe/nqo00s.torrent----Anchor: >>1310152. Music for this release - Pony Music Video of Network Music Ensemble's "Giddy Up" (this little horsy kissing me):https://ipfs.desmos.network/ipfs/bafybeiazdgsf2j3xalgx2y3ynbzm423bwzf5kfs47hgyhqoveogeh2jcye
I had two copies of this:>81.94 GiB, /ipfs/bafybeig36aba5qldy2wk3tnpu24ll32hvfin6zlrcxlxrkfadyujscgkju 4chan_gif_2024_09>81.95 GiB, /ipfs/bafybeiekhe6uxj2x3z5msvj5ime4np43avewro6faylkyhxe2wqkr6ag5i 4chan_gif_2024_09.zipdeleted one.
Not only do I despise or dislike certain 4chan user(s), the 4chan system itself keeps getting worse. I looked at the log file and saw this a bunch or too many times:>HTTP request sent, awaiting response... 429 Too Many Requestson https://i.4cdn.org/gif/...Gotta add a delay in, more shitty work I don't really want to do.
>>1340333Easy fix - for "cron0.sh" (->"torun.txt") was:> n=0; while [ $n -lt $imgcount ]; do echo TZ=UTC wget -nc https://i.4cdn.org/gif/${filename[$n]}${ext[$n]} -O $basepath1/test/${md5[$n]}; n=$(expr $n + 1); done >> $basepath1/torun.txtchanged to> n=0; while [ $n -lt $imgcount ]; do echo TZ=UTC wget -nc https://i.4cdn.org/gif/${filename[$n]}${ext[$n]} -O $basepath1/test/${md5[$n]}; n=$(expr $n + 1); echo sleep 1; done >> $basepath1/torun.txtNewer (cron.sh->) cron0.sh:http://5d6zyscsdarzytpbi3mypv37uoaro2to7oqru4a3oga42bwt3jkof4qd.onion:8080/ipfs/bafybeibvrdwa2hfbgnr3xnotul6ygb6qxebcaqb2hc4oqoqrqlt4aquxze/cron0.sh--or--https://anti-mage01.store/7J4GahGzfKPLGPio4LK3yQRDVgfxtAGlsvbgJa_ZQ1gPositivity: a mirror zpool was created to host/store stuff such as /gif/ data. Using two different 18-terabyte HDDs for redundancy, I think it's a two-way mirror. Start time:>$ TZ=UTC zpool status zc>[...] in progress since Sat Nov 9 19:07:40 2024> 72.4G / 14.8T scanned at 125M/s, 37.0G / 14.8T issued at 63.8M/s> 37.7G resilvered, 0.24% done, 2 days 19:18:58 to go[...]Resilvered 8 hours earlier than initially estimated:>$ TZ=UTC zpool status zc>[...] scan: resilvered 15.0T in 2 days 08:20:30 with 0 errors on Tue Nov 12 03:28:10 2024[...]
Very curious to see how the size of the monthly archive change now that they removed five pages of /gif/. Archive was also removed, but it looks like that have returned.
>>1340348>>1340333Even with a fat one-second delay I still got http 429, begrudgingly will up it to two seconds.
4chan_gif_2024_10: on step 12 out of 26. Month 10 = 37 GB from 15,394 videos.Not sure if I mentioned this before, but I had some interest in a /gif/ video in a previous torrent. The video was like fortnite zoomers back-flip away from covid cougher. Perhaps originally from ShitTok. It's similar to story expressed by dance or choreography like a play. I feel a tad dumb for posting or thinking this...Ever wondered about not sleeping to a degree where it's unhealthy? In the past, I was talking or communicating to some Internet friend(s) about that. I had read some like science news article on it, and said something to the effect of "it's like an overly aggressive garbage collection mechanism in programming or execution thereof." Here's a recent (November 11, 2024) video on that titled "How Not Sleeping Actually Kills You":https://inv.nadeko.net/watch?v=MbuzrFb7spw&listen=false
>>1340555>The video was like fortnite zoomers back-flip away from covid cougherAn IRL/AFK video (not in a video game), I think when they did the flip it put their hoods on their heads.>Ever wondered about not sleeping to an unhealthy degree?Months ago I typed something like "if you don't sleep" in DuckDuckGo (intended to search "if you don't sleep will you die"). One of the search suggestions was something like "if you don't sleep will your brain melt". I regrettably didn't take a screenshot and haven't been able to replicate it since then, so all you got is my memory here.
>>1340542Even with a big two-second delay I still got some HTTP 429 errors, begrudgingly will up it to three seconds. Delay between successive downloads of files at https://i.4cdn.org/gif/[...]. Also, what I need is zero delay if wget says>File ‘$path’ already there; not retrieving.Otherwise it takes too damn long.Some 4chan_gif_2024_10 files:./JgMl10-puf179JULUlaCHQ.webm politics, has audio./JpWCPCAe7Bgz860i1Aj6vw.webm politics, has audio./RyQvhKM9FK7P3FHn_zFK5w.webm politics, has audio./4chan_gif_2024_10/./././d6XjxmfAzSaYX77A0cxEXA.webm attached, fellatio, human
>>1340982
>>1340982Dammit, Wget has the same exit code with not there vs. is there:>$ TZ=UTC wget -nc https://files.catbox.moe/vmqtde.gif>[...]2024-11-17 07:08:17 (2.52 MB/s) - ‘vmqtde.gif’ saved [503181/503181]>$ echo $?>0>$ TZ=UTC wget -nc https://files.catbox.moe/vmqtde.gif>File ‘vmqtde.gif’ already there; not retrieving.\>$ echo $?>0>$ # both zeroGuess I gotta do some gay shit like have wget delay or not based on previous wget stdout/stderr.
>>1340984Huh, perhaps not too difficult to reprogram GNU Wget to do what I want:> if (opt.noclobber && file_exists_p(opt.output_document, NULL))> {> /* Check if output file exists; if it does, exit. */> logprintf (LOG_VERBOSE,> _("File %s already there; not retrieving.\n"),> quote (opt.output_document));> exit (WGET_EXIT_GENERIC_ERROR);> }just change it to> exit ([some exit code, maybe 9 or 99, however I do that.]);Source code from https://www.gnu.org/software/wget/ -> https://ftp.gnu.org/gnu/wget/?C=M;O=D -> wget-1.24.5.tar.gz -> ./wget-1.24.5/src/main.c . Exit codes enumerated in ./wget-1.24.5/src/exits.h:>enum> {> WGET_EXIT_SUCCESS = 0,> WGET_EXIT_GENERIC_ERROR = 1,> WGET_EXIT_PARSE_ERROR = 2,> WGET_EXIT_IO_FAIL = 3,> WGET_EXIT_NETWORK_FAIL = 4,> WGET_EXIT_SSL_AUTH_FAIL = 5,> WGET_EXIT_SERVER_AUTH_FAIL = 6,> WGET_EXIT_PROTOCOL_ERROR = 7,> WGET_EXIT_SERVER_ERROR = 8,>> WGET_EXIT_UNKNOWN> };Cum truck from 4chan_gif_2024_10.
>>1340984That's withGNU Wget 1.21.4 built on linux-gnu.VersionGNU Wget 1.24.5 built on linux-gnu.does the same thing, zero exit code with "$ /usr/local/bin/wget -nc https://files.catbox.moe/vmqtde.gif"I see that you can't upload more than 100 files at a time to a github.com repo. Took 30 minutes to upload all of folder "wget-1.24.5/" from "wget-1.24.5.tar.gz" to GitHub.Build/compile and install newer Wget:>[extract wget-1.24.5.tar.gz]>$ apt-cache search 'libgnutls.*-dev'>libgnutls28-dev - GNU TLS library - development files>$ sudo apt install libgnutls28-dev>[...]>$ cd wget-1.24.5 && ./configure && sudo make && sudo make install>[...]>$ # older = "/usr/bin/wget"; newer = "/usr/local/bin/wget"More 4chan_gif_2024_10 files:./o5mo3b4IyYspgmtYqlSfYg.webm politics./pmezHBWDPrqsf38fqWlbjg.webm politics./pf6oUNY8wmaacdhhDrQpQA.webm gay porn/trap/femboy, attached
>>1340984Getting Dumbledore to drink the insanely spicy vomit diarrhea mixture in order to possibly destroy part of Lord Voldemort.>>1340989I changed that single line of code to:> exit (WGET_EXIT_IO_FAIL); /* was WGET_EXIT_GENERIC_ERROR */then compiled it after deleting /usr/local/bin/wget, but I'm not getting any different exit code with that.
>>1340985>>1340989>>1340992>WGET_EXIT_GENERIC_ERROR outputs zeroThis does look like a bug in Wget. WGET_EXIT_GENERIC_ERROR should have an exit code of 1 and not 0.For example, this works:> if (opt.warc_filename != 0)> {>[...]> if (opt.spider)> {> fprintf (stderr,> _("WARC output does not work with --spider.\n"));> exit (WGET_EXIT_GENERIC_ERROR);> }->>$ src/wget --spider --warc-file=a https://example.com>WARC output does not work with --spider.>$ echo $?>1>$ # NOT ZERO
>>1340994Because that code in main.c does nothing. See https://github.com/ProximaNova/gnu-wget/commit/ceec242de42f743655727624215d57e8a246b104 - short link = https://github.com/ProximaNova/gnu-wget/commit/ceec242The real action is at http.c and exits.c:> /* TODO: perform this check only once. */> if (!hs->existence_checked && file_exists_p (hs->local_file, NULL))> {> if (opt.noclobber && !opt.output_document)> {> /* If opt.noclobber is turned on and file already exists, do not> retrieve the file. But if the output_document was given, then this> test was already done and the file didn't exist. Hence the !opt.output_document */> return RETRUNNEEDED;> }->> case RETRUNNEEDED:> /* The file was already fully retrieved. */> ret = RETROK;> goto exit;->> case RETROK:> return WGET_EXIT_SUCCESS;and I think that's where the bug is. Attached = random /gif/ video.
>>1341004Yes! I change that C code and now it works better for task >>1340984 >>1340982. See https://github.com/ProximaNova/gnu-wget/commit/405bef0 which is changed to "return PROXERR;" on lines 2354 and 4296 of http.c. Proof:>$ /usr/local/bin/wget -nc https://files.catbox.moe/vmqtde.gif>File ‘vmqtde.gif’ already there; not retrieving.\>$ echo $?>8>$ # Not zero :)An exit code of 8 is better than 0, but it should be an exit code of 1, which would be even better (1=WGET_EXIT_GENERIC_ERROR). Go tell these guys that they're dumb, so to speak:https://savannah.gnu.org/bugs/?func=additem&group=wget
>>1341004Also exits.h>>1341008"ERROR 429: Too Many Requests." also has an exit code of 8, so not really useful to me. Therefore, changed to exit code 7: "return PROXERR;" -> "return HEOF;". Looking over the code, it seems that there isn't a simple drop in thing to trigger WGET_EXIT_GENERIC_ERROR. There is a drop-in thing to trigger each of the following:>case HEOF: case HERR: case ATTRMISSING:> return WGET_EXIT_PROTOCOL_ERROR;and>case PROXERR: case GATEWAYTIMEOUT:> return WGET_EXIT_SERVER_ERROR;So I've used one of those. exits.c has no thing for generic error. I could add a thing for that, but that would entail more work, probably changing other files for that addition, and making sure it all works.next random /gif/ vid
>>1341009Learned more about that: If ran a second time,>$ wget -nc https://example.comshould have an exit code of 1, but doesn't. If ran a second time,>$ wget -nc https://example.com -O ashould have an exit code of 1 and it does. If I email them with this info, I should include that details. More info: >>>/g/103218639https://archive.is/2024.11.17-211903/>>>/g/103218639>Exit code for no clobber and file exists is contradictory or inconsistent and some of the code does nothingSo, I can do this:>$ wget -nc https://example.com/404; if [ $? -eq 8 ]; then echo exit 8; fiFor /gap/:> n=0; while [ $n -lt $imgcount ]; do echo "TZ=UTC wget -nc https://i.4cdn.org/gif/${filename[$n]}${ext[$n]} -O $basepath1/test/${md5[$n]}; if [ "'$?'" -eq 8 ]; then sleep 3; fi"; n=$(expr $n + 1); done >> $basepath1/torun.txt
*https://archive.is/2024.11.17-211903/https://boards.4ch an.org/g/thread/103198347/fglt-friendly-gnulinux-thread#p103218639 - short link: https://archive.today/HlXdy#p103218639Will this code update actually work? Will see in like 3 hours (zero hour UTC), check /zc/vid_4mb/cronlog2.txt
>>1341106For what it's worth, rate limiting is not just a "you" thing. Been using jdownloader to scrape the archive boards every few days without issue until ~ a week ago, and now I'm getting either a huge amount of 429s, and/or <50kb/s download speeds with only 5 simultaneous downloads (compared to 3-5 Mb/s before). Something has changed significantly for the worse.
Some porn of interest - possibly shared as WebMs in /gif/:- woman masturbating, top https://www.xvideos.com/video.lccita8c27/maria_rya_-_secretary_dreams- vagina https://www.xvideos.com/video.uievkfvcdfd/melena_maria_rya_in_sexy_little_black_dress- both linked from https://www.xvideos.com/tags/sexy-dress- watched this some time ago, forgot my opinion on it https://www.xvideos.com/video.kvucbbv3b85/fallout_4_whorehouse_and_beauties- female, sex https://twibooru.org/3386720How do I convert these links www.xvideos.com/video.kvucbbv3b85/fallout_4_whorehouse_and_beauties to the older format links like the following?>https://web.archive.org/web/20231126014906/https://www.xvideos.com/video60069137/strapped_legs_blonde_dp_machine_bangedlive version now redirects to>https://www.xvideos.com/video.kvuvohhbbc4/strapped_legs_blonde_dp_machine_bangedIs it possible? Other than using their search thing?>>1341106>if server error (exit 8), then sleep for 3 secondsThat's not right. It should be if not already exists (exit 1 if output file is specified, which it is), then sleep 3.>>1341112I could look through log file(s) for the first instance of HTTP error 429.
This lame update slowing down downloads of full images, along with the 15 minute wait on posts. Hella gay. I can say that I have 15 minutes now to formulate my ideas on irrelevant shit, lulz.I should get WARCs of certain links which I have an interest in but are not so much at hand. I do have a thing to check if I already captured the URL or not. I could even change the code to get WARCs of /gif/ thread JSONs. Would be worthwhile and would not use much more space since those are saved as gzipped automatically by wget.>>1341112>Something has changed significantly for the worse.I agree. Likely also a pain for websites such as desuarchive.org. Site archived.moe is getting a taste of it's own medicine since they overly use Cloudflae to block access. (archiveofsins.com kinda does that too.)>>1341175>How do I convert these links to the older format links?Nice, in the source code of that Xvideos webpage you can see>"id":60083129,"vmdata":{"pub_comments":7,"has_comments":true,"vote_good":"\/video-vote\/60083129\/GOOD\And therefore can go to>https://web.archive.org/web/20240104212633/https://www.xvideos.com/video60083129/fallout_4_whorehouse_and_beauties>Other than using their search thing?Why'd I say that? My mind was a bit misconceived when writing that.>>1340984>>1340992>"Harry Potter and the Deathly Hallows – Part 1"More like "Harry Potter and the Rise of Magical Hitler">"Harry Potter and the Deathly Hallows – Part 2"More like "Harry Potter and the Decline of Magical Hitler"Also, apparently gamers really liked the Harry Potter PS5 video game - "Hogwarts Legacy - Official Reveal Trailer | PS5":https://inv.nadeko.net/watch?v=1O6Qstncpnc
test>https://cdn5.filehaus.su/files/1732054758_27972/4chan_gif_2024_10.zip.torrent>sharethread https://desuarchive.org/mu/thread/63669289
== Presenting ==Full images of 4chan /gif/ - 2024-10:magnet:?xt=urn:btih:27c4953df3d947a3b2a22dd70ac90241cd09f538&dn=4chan_gif_2024_10.zip&tr=udp%3A%2F%2Fftp.pet%3A2710%2Fannounce&tr=udp%3A%2F%2Flulz.vpndb.org%3A3000%2Fannounce&tr=udp%3A%2F%2Ftracker4.leechshield.link%3A6969%2Fannounce&tr=http%3A%2F%2Ftracker.bittor.pw%3A1337%2Fannounce&tr=https%3A%2F%2Fpybittrack.retiolus.net%3A443%2Fannounce&tr=udp%3A%2F%2Fy.paranoid.agency%3A6969%2Fannounce&ws=https://gatewayipfs.577975ff8cb80eb4.dyndns.dappnode.io/ipfs/bafybeic3qc6zthjirfyzo275pgxmjezam3b6qxtptxg6gp4vkc7gtxzeqy/imageboard/4chan_gif_2024_10.ziphttps://gateway.ipfs.chaingpt.org/ipfs/bafybeifdsozl6226ltl3xw6likdhymw5tpuvru2ysoqz5f3mh3radelqmu/4chan_gif_2024_10.zip.torrentmirrored at https://cdn5.filehaus.su/files/1732054758_27972/4chan_gif_2024_10.zip.torrent----Anchor: >>1310152. Music for this release - "It'll Be OK" by Sherclop Pones (from "Friendship is Witchcraft"):- clearweb https://gateway.pinata.cloud/ipfs/bafybeifx5w7fnzfj5xirnnqahmc6kmlpfrp3vo6gghrrfkapr3msuuea3q/itllbeok- privacy http://ponypalsh4y6olziyjlswfv674utokqhz3y6beym2erqtstcgadmacid.onion:8080/ipfs/bafybeifdyivnv3ix4vtepxkuazlx5ujpp6f44dzzdd3noc5k6c7auomxle
