Torrents of full images from /gif/ - Adult GIF, released monthly. >>>/gif/ is a NSFW 4chan board. Info:- Contents: porn, random videos, LiveLeak-esque videos, and other videos which were too interesting for weak video sharing sites like YouTube.- Stats so far: more than half a million files, totals to more than one terabyte.- Why? Among other reasons, no one else is archiving /gif/, so I decided to.Previous: >>1231730
== History ==/gap/ began in 2022-10. Archive files from 2022-10 to 2024-03 only have full image files and not threads (plain text). Starting in this thread, including 2024-05, each release going forward should have threads.== How does it work? ==Every 24 hours, this happens:1. Full image links obtained 24 hours ago get downloaded.2. All /gif/ threads get downloaded in API/JSON format.3. Full image links get extracted from the JSONs. Goto 1.
*more than half a million video files (gif, webm)
bump, not sure if u did but it would be good to publish your exact setup and workflow for this just in case someone else might want to use it for another board or do it for gif themselves or continue doing it one day if you disappear
>trying to find archive of two specific threads, grab torrents>it's a zip instead of folders i can pick through properly but whatever>open one today>it's a flatfile dump>no index file, no metadata, filenames are presumably md5>go to archivedmoe to find the thread to pull the hashes out>the archive for that is 404 for some reasonI don't want to bitch too much because this is all still better than nothing, but please, I beg, an SQLite file, OR simply putting the thread number in the name.
>>1311696>flatfile dumpIs that what that's called? In the past I called a similar thing a "simple many-file folder" (which contains zero folders).>simply putting the thread number in the [file]name.Sounds like a bad idea. One reason: same files in multiple threads. Could have thread number of only the first thread it showed up in, but still.More data stuff - unrelated to your post... I run an ipfs node which is consistently online in one computer. I run another IPFS node which is inconsistently/temporarily online in another computer. I have a cid which is only in the temp-online one. After recursively providing that cid to the dht in the temp one (for hours, probably still doing it) I saw that its storage went from 666 gb total to 669 gb. Conclusion: recursively providing a dag/CID which you don't have to the DHT seems to make you download it (if running as read-write which you are almost certainly doing).
>>1310152is 2024_05 available yet?
>>1310152Fuckin piece of shit, I found a fault that shows that 4chan /gif/ 2024-06 wasn't being downloaded for days. I think I fixed it now. I could check on logs and stuff going forward to see that it's working correctly.
>>1311996 = day that I fixed that problem, has worked fine since then.
>>1310152What are you using for this? GChan?
>>1310152Where is the magnet link?
>>1314207See the previous thread. I didn't share 2024-06 and 2024-05 yet.>>1313666No
>>1310155Post source code plz
>>1314635>See the previous thread. I didn't share 2024-06 and 2024-05 yet.But the previous thread is 404 already
Fuck cloudflare captcha
>>1310152>>1310815>>1314645Setup: GNU/Linux computer, set .sh files to executable by running "chmod +x file.sh". I use simple/crappy code to download this stuff. My code does not enable /gif/ users to do remote code execution because it parses JSON in such a way that it deals with privileged bytes that are part of the JSON structure and not the contents (same thing with older versions of the code that I used which parsed HTML instead).Folder: /path/4changifFolder: /path/4changif/testFolder: /path/4changif/threads- Maybe sure you have those 3 folders created (replace "/path/4changif" with whatever you have.)File: cron.sh- ipfs://bafkreidzrvqgtjebkj7uqzibqjz6jri3ei3tvsjscf5i2lebejr5s3wgga / https://web.archive.org/web/20240628134551/https://sabrig1480.xyz/FSrXIjIxT1z4ndG9dx018rGq6MGD3cF6BUPNTXs2lu4- Checks if cron0.sh is running, if not run cron0.sh. Specify the full path to "cron0.sh".File: cron0.sh- ipfs://bafkreiclpya533snasw6crtg4f6dgdoikkuudsgnmvz4uw4s53f572ugvy / https://aralper.xyz/ciw3Aigohm54YTN-EwQO0C3xPOckjS3BjjYCipRjqf0- Main downloader. Change "basepath1='/path/4changif'" to whatever folder you have set up to download /gif/. Manually run lines 11 through 37 when first running it to kick things off. Make it so 11-37 lines are one command then run that command. After doing that, the downloads all happen automatically.- In depth on each line. 1=Bash. 2=basepath1 variable. 3=HDD history. 6=if statement 1, 86400-second wait between links obtained and downloading the files of those same links, runs if above that number. 9=runs commands to download files, logs it. 10=clears commands to run. 12=Does stuff, gets thread OP numbers from https://a.4cdn.org/gif/catalog.json . 14=threadcount variable which is a number of all OPs. 16=while loop 1, to go over all the OPs. 18=selects a specific OP (var ii). 20=debug output. 22=Downloads a thread https://a.4cdn.org/gif/thread/$ii.json > $basepath1/threads/$ii.json.$now1/?
>>131488624=imgcount variable which is a number of all images in a thread ii - calculated as a count of JSON parts>jq ".posts[].ext, .posts[].tim, .posts[].md5" | grep -v "^null$"divided by 3. 26=filename variable - array of POSIX time filenames from the middle $imgcount lines of those JSON parts. 28=ext variable - array of file extensions from the top $imgcount lines of those JSON parts. 30=md5 variable - array of Base64(MD5) strings from the bottom $imgcount lines those JSON parts (formatted to "standard" URL-safe strings). 32=while loop 2: saves commands to download each image into a text file (downloads as "TZ=UTC wget -nc https://i.4cdn.org/gif/${filename[$n]}${ext[$n]}" -> "$basepath1/test/${md5[$n]}" - cmds in $basepath1/torun.txt); end while loop 2. 34-39=iterate while loop 1, end while loop 1, end if statement 1.Ignore the/this "In depth on each line" section if you just want to use it and don't care about how exactly the code works. You can also replace each case of "/gif/" with "/mlp/" if you want to download another board. This skips downloading janny-deleted files, which is good and bad. Bad if it was some harmless video that got deleted because the poster was too based and got his post deleted due to politics. There's no HTTP archive of HTTP 4chan /gif/ files, so no thing to fall back on and check if it's a harmless file. In the /mlp/ example, there is. I don't have a thing to specifically record "found deleted", but you can look at cronlog2.txt for 404'd files if you want to use this script on other boards then check found-deleted against what's saved in desuarchive.org, for example. And since I brought it up, here's a one-terabyte torrent that an anon (not me) downloaded from desuarchive /mlp/ and other captures of /mlp/:>magnet:?xt=urn:btih:9671fb0855c7931fe98f03f7612c18010fb10121&dn=4chan-mlp&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a6969%2fannounce2/?
>>1314887Run "crontab -e" (NOT as sudo) and put this in there:>0 * * * * /path/4changif/cron.shcrontab runs hourly, cron*.sh runs daily. I guess I could simplify it to not be hourly->daily and just have crontab run it daily, but what I use works to only download it daily, so whatever.File: 404.txtFile: addext.shFile: howto.txtFile: 4chan_gif_2024_03_empty.txt- see the latest torrent, magnet:?xt=urn:btih:84b2a6b0865a26bac9b7deef0ba63f893d6931c4&dn=4chan_gif_2024_03.zipFile: cronlog2.txtFile: cronlog1.txt- automatically created, see the latest (4chan_gif_2024_03) for one of thoseFile: time.txtFile: chkcmd.txtFile: torun.txt- automatically created3/3 for now.
>>1310152>>1314207>>1314635Hey OP not to be mean or anything but why did you create this thread, just post the fucking magnet link, the previous thread is long gone from the archive
== Links to 4chan /gif/ 2022-10 to 2024-03 files? ==Those are all in thread #1231730:- with working CSS/JS (WARC in a parent folder): https://bafybeifn7bxeg34zc725kkjzfuxpbf2ftb5lgjpdh5r7hrdrrpf3zpat2m.ipfs2.eth.limo/raws/boards.4chan.org/t/thread/1231730.html- as a text file: https://gateway.pinata.cloud/ipfs/bafybeicqxg64e6u3ws3ietrlxao7nxjwuxkbc542gh5xw53quazkgttpbu - includes .gz version at https://shadow39.online/OdSsPIeOso2q8kY6lvckhJuSzB4YGI9hdOqum0AncVQ- folder bafybeic...tpbu also includes a text file which only contains the magnet links posted in that thread, with 4chan /gif/ ones at the top: https://utkububa.xyz/JsoPMGJhzMrtMCDblJjbXH8XFoa_9vfqzuKGNBfwNSw = https://gateway.pinata.cloud/ipfs/bafkreig3hnbw65gikxi2j62q7bgpsn2npx7dkxmfyzut5emvtumzzpwdzeAnchor: >>1310152Replied to: >>1314207 >>1314778 >>1315187
>>1315342yeah I'm not clicking that shit glowie bot
>>1315353Wow, you are dumb. Here's that same text file:https://files.catbox.moe/3nd5c2.txt>text file which only contains the magnet links posted in that thread, with 4chan /gif/ ones at the topwhich is also here:https://ipfs.hypha.coop/ipfs/bafybeia7nmoydnj2d4gymp6rlpdpusozcip7x5znpax7gfprpfgx3wiaii/kill_all_retards.txtNow go back to watching CNN.
>>1314896Thank you. Not trying to be annoying but, putting this somewhere like a GitHub equivalent would be cool (we should probs stop using GitHub at this point because u are training an LLM with every commit)
>>1310152Great initiative anon.Looking forward to 05 and 06. Got all the other ones already. Im archiving some threads myself, mostly wsg and pol stuff.
Done: stuff such as segmenting 06 into its own folder, created folders for 07, updated howto.txtTodo: info on 05 and whatever
>>1314207>1231730>>1314635https://archiveofsins.com/t/thread/1231730/#1311161
>>1316804Guess I will work on 05 "soon". It's here:/zc/z9/4chan.org/gif//zc/vid_4mb/>4mb/wsg/ seems to have raised their max file size limit to like 6MB:https://boards.4ch an.org/wsg/thread/5612597/pol-politically-incorrect>>>/wsg/5618193 - can upload a video where upstream size=6MB, or is that derived size?Maybe /gif/ raised that limit too.
>>13179984chan_gif_2024_05 is like 10 GB for certain reasons, and I'm just gonna have to accept that for now. In order to get the rest of I need like 100 USD so I myself can do a HDD repair, 1000 USD to get a "professional" HDD repair. I wish this was a data ransom situation because that would mean I have all of the data for that month and am pretending not to have it. I don't have a big part of it, so less motivation to release 2024_05 which I felt like releasing before releasing subsequent months. It's been 2 or 3 months and I haven't got on 05, now at the acceptance part of the stages of grieving, so I guess I will be more likely to get on it soon. Trump catching an octopus AI video from the following (can't attach extensionless WebM file then post it to 4chan):>4chan_gif_2023_05 , https://gateway.ipfs.cybernode.ai/ipfs/bafybeigkplwidoyprmm7vyb2qlna7o2sgq26ydrg55nrqhf4xfcga3gxsuFUCK 4CHAN'S KEKFLARE CAPTCHA!!!!FIX THIS SHIT MOOT!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>1320059Oh look, a new wordfilter. I hate this website even more now. It's C U C K -> KEK.
>>1320059For 4chan_gif_2024_05, I'm on step 9 out of 29.
>>13205744chan_gif_2024_05 is available in IPFS:file:///ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4- packed and non-packed version- extra dataIt'll take some time to share a torrent of it. A highlight from that folder:https://gateway.pinata.cloud/ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4/music/xdSuXide_UCLnUxuqdNQeoh3YvH5mlKKA.partial/MLP_-_Friendship_is_Witchcraft_Pinkie_s_Brew_By_Sherclop_Pones_PMV_Lyrics-xdSuXide-20230526-youtube-1280x716-shFfnG6GRhA.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-05:magnet:?xt=urn:btih:dc88d6317891957e462a0af085c2f5d1cee33127&dn=4chan_gif_2024_05.zip&tr=udp%3A%2F%2Ftracker.deadorbit.nl%3A6969%2Fannounce&tr=http%3A%2F%2Fcn.pcfreetime.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.srv00.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.mirrorbay.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fbandito.byterunner.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.darkness.services%3A6969%2Fannounce&ws=https://ipfs.ssi.eecc.de/ipfs/bafybeigamrivrveeoctw6qfuw6siiinmsgamekf4z4hmdqhghcoyzscze4/imageboard/4chan_gif_2024_05.ziphttps://ipfs.apillon.io/ipfs/bafybeiel5yavn7jicgloferdbzwrxsojeeaqnhrkhglu3bw74xny23hqli/4chan_gif_2024_05.zip.torrenthttps://files.catbox.moe/4kqtzd.torrent----Anchor: >>1310152. Music for this release - "Pinkie's Brew" by Sherclop Pones:https://gateway.pinata.cloud/ipfs/bafybeidhaffpxuf7ar524yhswvtvsddxiakzyd4nijd777jkb5onjchzaiThese various musical "Pinkie's Brew" videos are related to time traveling into the past. I kinda wish I could go into the past and prevent myself from making certain mistakes, including as related to this /gif/ release for the month of May.
Release for 4chan_gif_2024_06 shouldn't be missing any files...
How do you know what webms came from what thread? It's a pain to open each video and delete the unnecessary ones.
>>1310155Was 2024_04 never done?
>>13211474chan_gif_2024_06 is missing roughly 2 or 3 days >>1311996. I'm creating the .zip for 2024_06: it'll be like 70 GB.
>>1323239I finished making that zip some time ago. Also finished creating a dag of the packed+nonpacked versions some time ago.Search became horrible in 4Chan archives.- https://archive.4plebs.org/pol/search/text/twitter/type/op/ - cuc kflare; archived.moe=cu ckflare+search not enabled for /pol/- https://archive.4plebs.org/tv/search/text/%22your%20honor%22 - c uckflareI have some positive experience using Selenium, so I can still get those webpages in a semi-automated way, even with those annoying anti-human and anti-good-bots captchas.
4chan_gif_2024_06:BitTorrent release: todoIPFS release:/ipfs/bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74- has extra data; in MFS, not pinned- packed + non-packed version = 138.94 GiB- previous version/pin:>ipfs pin update --unpin=false bafybeibie5qrgjalyghiczrj7q4dwserzpci6bsrytl63x7d67wx4fp3ge bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74- previous + this = 170 GiBIt'll take some time to share a torrent of it. A highlight from that folder:https://gateway.pinata.cloud/ipfs/bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74/music/yt/Kim_Jae_keyoung_UCv-in54UcjPqdcLuRLxdKLg/My_Little_Pony_AppleJack_-_Apple_Jack_Lisa_McHugh-DWr1SKWRM7c.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-06:magnet:?xt=urn:btih:b90ef006b7f7e4d069d310472f9a32413813f4a3&dn=4chan_gif_2024_06.zip&tr=udp%3A%2F%2Fmoonburrow.club%3A6969%2Fannounce&tr=udp%3A%2F%2Faarsen.me%3A6969%2Fannounce&tr=http%3A%2F%2Fdht.dhtclub.com%3A666%2Fannounce&tr=udp%3A%2F%2Fnew-line.net%3A6969%2Fannounce&tr=http%3A%2F%2F1337.abcvg.info%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.skynetcloud.site%3A6969%2Fannounce&ws=https://bafybeianmslb3rcwovkbciuz7p27e5t6uzipjqhwzshwnzcsqk7uzzfk74.ipfs2.eth.limo/imageboard/4chan_gif_2024_06.ziphttps://eu.starton-ipfs.com/ipfs/bafybeid5cl6sc47bwg4rpuyxwrlvxfs55uof7yqa23az5xzfvcgibowpp4/4chan_gif_2024_06.zip.torrenthttps://files.catbox.moe/a708r6.torrent----Anchor: >>1310152. Music for this release - PMV of Apple Jack by Lisa McHugh ("My Little Pony AppleJack - Apple Jack(Lisa McHugh)"):https://gateway.pinata.cloud/ipfs/bafybeieybwwjoesqzune4aixpxab55723hfcd24wnichykrudz357zki6yThat's a folder to a copyright-deleted YouTube channel which contains the original WebM + derive MP4 (>>1323761).
*derived MP4. Size of 4chan_gif_2024_06.zip = 69 GiB (contains threads+full images). 4chan_gif_2024_07 shouldn't be missing any files, but I think it extended too far into 4chan_gif_2024_08...
>>1310152Serious question, deleted videos are there too? some images are illegal in some countries.
Upload on 4chan_gif_2024_06.zip was slower than wanted: avg. ~165 KiB/s. 100% of it finished uploading roughly 16 hours ago. Upload for 4chan_gif_2024_07 should be faster: I have a couple solutions for this. If one doesn't work, I can pause for a while to work on plan B.
4chan_gif_2024_07: on step 10 out of 28. Captcha: DART
>>13250704chan_gif_2024_07 is >100 GiB; step: 25 out of 28.>>1308979 [~2024-06 post in previous /gap/ thread]>Hope to put those blocks back online sometime this month.I was listening to tracks in this small folder, such as>/ipfs/bafybeiexmw4i57jgr3esuda7q2n4i45erll7vkaiask4t2tbpkxacaxove/keygen-music/KEYGENMUSiC MusicPack/!Others/909DEAD - Adobe CS6 All Products activator.it>Title: Pokemon RBY Lavender Townand was reminded of that. 4.7-GB keygen_music folder is back online: /ipfs/bafybeibeuoggietcp2dt3qtnsp6ul7cwcuubo4mmvswwzyb652me2cs6ou/ (also in 2 HDDs). Just put it together (realized it) today: hovering over a color word in Konsole version 21.12.3 shows a box of that color. I was confused about "randomly" seeing that GUI square in the past. So if you hover over "lavender", "green", "red", etc. (text in Konsole) then it will show a square filled in with that color.
>>13253172024_07: writing metadata. Will make another copy after that finishes.>>1325317>small folderThis track sounds familiar: "ANGELS + DEFJAM - Ranx English intro.xm" ("Title: daley").>CID back onlineAlso back online - contents of deleted upload https://archive.org/download/mega-dance-hits-collection-2-1990-2001 which I downloaded some time ago: /ipfs/QmWRdfw7YxaXY38wnFNvYaR9J1oNtJEeo6jNzsn827mbX2 (13 GB). It's more than 1000 dance music tracks (in 2 HDDs). Like bafybeib...s6ou, it would have been better if I stored it as raw blocks (it's not), but whatever. (Neither are repinned, yet - will maybe do that later after doing stuff with /gif/ 07.)
4chan_gif_2024_07:BitTorrent release: todoIPFS release:/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli- has extra data; in MFS, not pinned- previous + this = 424 GiB- pin for this month only: /ipfs/bafybeidjxg6jqs5p5oakwfflaijvisxx2d5fhjbw64xtcgrhgmna37g7mu- packed + non-packed version = 226.39 GiBHighlights from that folder:- https://gateway.pinata.cloud/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/music/BFUtEoT/MLP_S7_D4.ISO-VTS_10_1.VOB.mp4- https://gateway.pinata.cloud/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/music/BFUtEoT/Montgomery_Gator_UCeaHgmeXQg_5o_Zrmlupsig.partial/Best_Friends_Until_The_End_Of_Time-Montgomery_Gator-20200902-youtube-1200x1200-AAWSawGr0dk.mp4
== Presenting ==Full images of 4chan /gif/ - 2024-07:magnet:?xt=urn:btih:352c3ee75397c6ffde9b6786e01441990bc57641&dn=4chan_gif_2024_07.zip&tr=udp%3A%2F%2Fipv4.rer.lol%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.ddunlimited.net%3A6969%2Fannounce&tr=udp%3A%2F%2Fseedpeer.net%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.artixlinux.org%3A6969%2Fannounce&tr=http%3A%2F%2Fopen.tracker.ink%3A6969%2Fannounce&tr=http%3A%2F%2Ftracker.dump.cl%3A6969%2Fannounce&ws=https://gateway.ipfs.cybernode.ai/ipfs/bafybeibfcnmvsxigcbbrdknuf4h4ziwqey5c5rwtp4od3huznx2ucseuli/imageboard/4chan_gif_2024_07.ziphttps://eu.starton-ipfs.com/ipfs/bafybeifu7oeuip4rtt2vh4pknt4ymfpje6wsicztazsahpmticwd6yjxy4/4chan_gif_2024_07.zip.torrenthttps://files.catbox.moe/gsjz1h.torrent----Anchor: >>1310152. Music for this release - "Best Friends Until The End Of Time" from MLP:FIM (incl. singalong video from the DVD):https://gateway.pinata.cloud/ipfs/bafybeifr7ksfksrdvett6b6va6bkprvhbmvgnvi3u7djynz6f3t2zhsdde4chan_gif_2024_07.zip.torrent created in about 13 hours for 121,465,843,003 bytes (113.12 GiB) = 2.6 MB/s.
What are all these magnet links about?
About 15 hours ago my average upload speed on 07 was approx. 256 KB/s. Had something annoying happen recently (which is still a problem or an annoyance): not messing with that now, so the upload speed should be better. (300 K/s avg. should be the minimum goal, and perhaps I can get that on 4chan_gif_2024_08.)