mandelbot edition>quant newshttps://www.ft.com/quant-investing>quantopian lectureshttps://www.youtube.com/playlist?list=PLRFLF1OxMm_UL7WUWM31iynp0jMVf_vLW>event driven backtester designhttps://www.youtube.com/watch?v=OhGdVdolqkU [Embed] [Embed]>bookshttps://github.com/zslucky/algorithmic_trading_book>recommended brokersstocks and options - https://alpaca.marketsfutes - https://www.tradovate.comcrypto - https://www.kraken.comforex - idek>wiphttps://alphapulse.org
Bump
>>61115210>>61115405I don't have anything interesting to talk about today. I think I jumped the gun on developing my defi arb system, I should've done more research first and maybe a python mockup system to learn the complete data requirements end to end. Essentially, I didn't know that price and liquidity levels couldn't be computed from swap events alone and there's a separate sync event that contains that data. I thought swaps were essentially equal to orderbook/L2 deltas, so I could build the orderbook state by accumulating swaps. because im using a binary internal format for message passing with fixed byte size (so I can do zero copy), it's gonna be a bunch of breaking changes to modify to accommodate sync events and I'll probably just take it as an opportunity to do codebase cleanup and redesign. kind of out of available time to work on it much this week so won't have many updates. BUT this explains a ton and why my arb detection was all wonky.
>>61115210>nooooo the model said 95% confidence wtffffff>grrrrrrr the market is retarded >it HAS to find fair value soon! Soon!!>noooooooo liquidated again?!
>>61115479yeah dont have anything either reallyI guess uh.. does stop loss count? I just activated trailing stop loss because a stock I am in is going parabolic, am I stupid? I dont know. Please tell me if I am. Not sure if it counts as algorithmic really
>>61115479you get price and liquidity levels from token pair pools, right? question is how do you find all liquidity pools for a given token on a network?
>>61115574>you get price and liquidity levels from token pair pools, right?yes, but from different events and data sources. you need pool factory address, token contract address, swap event, sync event, gas/tips, etc. >question is how do you find all liquidity pools for a given token on a network?Luckily I have this part figured out for the most part.
>>61115511yes it counts. a first step toward algotradig might be backtesting the effects of static vs trailing stop and optimizing the trail %, then maybe the same thing for take profit.
>>61115770Not a whole lot of options for doing that in the brokerage the stock is held in unfortunately, they let you say trailing stop with trigger + sell price %, or static at price trigger and sale, but you have to manually change stuff and place it as an order
>>61115807yeah that's where the algo part comes in and you'd want a script plugged up to the brokers API
>>61115861Yeah, that does make a lot of sense. I will have to take a look using Alpacca on a paper account
Where's a good place to get (free or cheap) forex and commodities data? Specifically want historic gold/solver data (1m candles spanning as long as possible) and maybe some forex pairs.
>>61115921Did some quick research, Alpacca does 10 000 bars via API call, you can use next-page token to get the next 10 000 bars, but you are rate limited to 200 requests per minute, so I guess you could work something out with that, 2 million bars per minute? If that works out. Havent tried using next-page token or anything myself
>>61115921try dolthub? I'd grab some for you if I was still paying for my alpaca plan but I cancelled since I've been inactive lately.
>>61115887your broker might have an official API you can use too, most of them do these days aside from robinhood for whatever reason. but Alpaca's API is modern and clean, extremely easy to use. IBKR was a nightmare and I gave up, I didn't wanna install al local java server bloatware nonsense
>>61116008Ah I checked and they do, but it seems you have to pay or people couldnt find it? The documentation is also not very readable
>>61116021What brokerage?
>>61116043Nordnet if you ever heard of it
>>61115992>>61115997How far does the data go back? I already have a trading bot i use and wanna try that before I commit to a new framework
>>61116058>How far does the data go back"With the Alpaca Data API you can access historical minute data back 5 years - we currently do not have data further back than 2016. Although there are a few missing data points towards the beginning, this is an issue we are working towards solving as soon as possible."From the support site
>>61116058>I already have a trading bot i use and wanna try that before I commit to a new frameworkAlpaca isn't a framework, just a broker that you plug your bot up to, and they provide historical data too.
>>61116064>>61116074Thanks anons, I'm looking these now, if I can't find a db on dolthub I'll just get alpaca for a month and download everything I can.
>>61116373Alpaca doesn't have Forex or commodities FWIW, it's just stocks, options and some crypto (but the crypto trading volume is very low, no sense in using it's crypto API over say kraken which will give you like 10GB of data for free).
>>61115479>I didn't know that price and liquidity levels couldn't be computed from swap events alone and there's a separate sync event that contains that data. I thought swaps were essentially equal to orderbook/L2 deltas, so I could build the orderbook state by accumulating swaps.As I did more research today, not all protocols even have sync events. So essentially, you need unique types for each protocol. Thankfully, my arbitrage engine is sufficiently decoupled from the venue specific data types, so after extracting the data I need from each venue, the arbitrage engine can still run unmodified. Spent most of today just researching and planning the new build. and cleaning up the codebase.
>>61116391Hm. Some etfs and mining companies track the price of gold very well though right? You think those could be close enough? I am purely crypto right now but want to expand, don't really care
>>61117658Yeah they totally could if you're just trying to get exposure. Miners in particular tend to act as leveraged bets on whatever commodity they mine, and I'm sure there are some currency ETFs or things provide similar exposure.
decent book so far
What exactly is the point of doing this versus deliberately and precisely executing it yourself not being facetious but this seems curious to me
>>61119132The ease of backtesting over a large universe of assets/instruments and timeframes, for one. Ain’t no one got time to manually try hundreds of thousands of permutations of some strategy manually.Similarly, you can’t really do machine learning strats manually, I mean I guess you could… Like stare at your screen until the model says buy, then click buy, etc but that’d be redonkulous. It’s also easier to get into different types of approaches this way, not locked into indicators on TradingView like every other discretionary trader. Essentially I think a lot of algo traders approach this as a data science problem, where you’ll want statistics to guide you. And in that domain you’ll want as many data points as you can get - more than you could ever accumulate manually.
>>61119132I guess that's a bit of a: Why and How does Warren Buffet beat traders, where he doesn't do the kind of stuff market makers and the like attempt to, and still beats them by a mileBut as >>61119204Put it>Ain’t no one got time to manually try hundreds of thousands of permutations of some strategy manually.Even Warren uses indicators and metrics to initially find stocks he can further investigate I suppose my own opinion on the matter isn't well formed as my trader and analytics tool is very early prototype stuff, all my gains has been from two buys in particular and hasn't been sold/bought/changed in two years
>>61119132its easier to create a set of rules that a machine must autistically follow than to try to apply them yourself in a market that exactly knows how to manipulate your emotions via price action (thousand year history going back to ancient grain markets) and put you into a state where you make bad calls.
>>61119383Further more...You can say using patterns are equivalent of using astronomy to predict the future, but at their core they are based on price floors, which is very much provable and realIt's the same for indicators and metrics, you can easily get into a world where you are overfitted or your indicators looks like they are doing something real. Then you actually buy stuff for a long enough time and they turn out to do nothing and predict nothing. And then there is the reason we try not to share our strategies publicly as we chase for Alpha and oversharing annihilates alpha
>>61119409the market also knows how to manipulate data in order to trick an autistic rule following machine, so its fun to learn how to counter also
>>61119204>>61119383>>61119409>>61119411>>61119413It seems like an awful lot of work and that the fun is probably in the building of complexity, I've made algorithmic music before and it obviously sounds quite similar to the process. How steep is the learning curve and how long did it take you lot to get comfortable doing this?
q52r4k here>>61119886>How long did it take to get comfortableI spent my summer learning everything from scratch, and building my own analytics and deployment tool from scratch, as I didn't get as many shifts as I had expectedI suppose it depends a lot on your previous knowledge on statistics, programming, maths and stock trade. Frankly though I am a scrub, and LLMs were a big help/crutch for me, although it hallucinates a lot and keeps screwing me over
>>61116008agree IBKR is a nightmareI got it to work a couple years back but then didnt want to fiddle with it anymore and my alpaca account got approvedstill would like to trade there to not be limited to okly US stocks, maxbe i‘ll give it another shot in a couple keksreally boggles my mind how they can be so backwards in 2025, i‘d think they can get a handful of pajeets to set up some proper APIs hell every thrid rate crypto exchange with no budget has it
>>61119886It’s not supposed to be fun, it’s supposed to optimize returns. The dream is being able to be afk for a month while it prints money. It’s not something I do just to “tinker”, for me it’s an attempt at essentially building a business where the product is software that extracts its revenue from the markets.
>>61119886And it can be as simple or as complex as you’d like. One algortihm for example might just be “buy and hold the S&P 500”. You might start there and then try tweaking methods to get better entries/exits using indicators, or start screening for value stocks, or doing some ML, etc etc. It can kinda sprawl out in whatever direction interest you or is most promising. There’s plenty of academic literature on the subject and papers you can implement too.
>>61119930>>61119970>>61119996I'll lurk a bit and go over the OP material, even if a money printer isn't the end result I've got a bit of spare time if there's anything else you want to recommend that isn't in OP let me know
bros I'm building a spot orderbook exchange that will expose some APIs for bot developers- place orders - cancel orders- get best 100 bids- get best 100 askswdythink?
>>61121684I guess my initial thought about it is well.. Doesn't most exchanges already have this built into their API?
>>61121772i guess what's different about mine is that no kyc, no market order, no stoploss, 0.1% trading fee, totally transparent as in you can see who is trading what and how much, etc.
>>61121684spot order book.. for what?>>61122334>no market order, no stoplosswhy do you say this like this it’s a feature? I guess like the other anon I don’t understand. Building an exchange is no small task, either..
Where's our crypto guy? He was the only poster with an active strategy.
https://www.youtube.com/watch?v=iWSDY8_5N3U
in general I think the "AI trading" (eg, just asking ChatGPT what to do al day) shit is super cringe but this is kinda fun to watch
>>61124465>Where's our crypto guy? He was the only poster with an active strategy.Crypto AI trader guy here. My trading server suffered an energy shortage during the night, and after that it shat the floor by losing 2 trades. Still in the green, but nothing interesting has happened since. I will update when cool shit happens (positive or negative).Anyways i have found new ways to improve my strategy, so i will get another server for testing purposes, so looks like my energy bill will be the only thing to go to the moon.
>>61125324This is the only cute little entry right now:
>>61125324>>61125333>energy shortagewdym, like a power outage? Crazy how resource intensive that can be. That's just because it's running ML inference locally?
>>61125393>wdym, like a power outage? Crazy how resource intensive that can be.Oh no, i live in the countryside, and a fucking falling branch cut my powerline.
>>61125393Yeah, i meant a power outage, not shortage, excuse my esl ass, lol.
>>61119204>Ain’t no one got time to manually try hundreds of thousands of permutations of some strategy manually.Sounds like overfitting to me
>>61125501you're misunderstanding then, because that example was just a brute force *search* - no fit applied. >>61125421>>61125454poop. but a good example of the real world risk. luckily you didn't have a massive open position get liquidated or something lol
>>61125575>because that example was just a brute force *search* - no fit applied.If you are brute forcing many permutations of some indicator and then selecting the ones that best *fit* the time series you are attempting to predict, that is overfitting.In general I don't backtest at all for my algos, I don't believe in backtesting and instead test live with fake money. Likewise, if my algorithm is not *fundamental* to the market, then it isn't sound. For example, if I find that the market is inefficient with regard to 48 hour SMAs, I won't even attempt to do anything with that data because there is *no discernible reason* why it would continue to be inefficient in the future. On the other hand, to give an example, if I find that certain options occasionally become mispriced such that I can extract a return significantly greater than the risk free return, then that is a perfectly fine strategy because it only works when the market misprices options and doesn't go against me when the market prices the options correctly since the algorithm won't even place a trade in such an environment.
>>61125675>selecting the ones that best *fit* the time series you are attempting to predict, that is overfitting.Again, I never even implied a selection process. And I'd argue that's not necessarily a good qualifier of what's overfit - that could be valid selection process if combined with walk forward validation.Not backtesting at all is foregoing the ability to leverage statistics for an edge. "Testing live" to see if it works is just backtesting but way slower.>For example, if I find that the market is inefficient with regard to 48 hour SMAs, I won't even attempt to do anything with that data because there is *no discernible reason* why it would continue to be inefficient in the future.I disagree here, I don't need to rationalize why an inefficiency would exist - i.e, if my backtesting implied there was an inefficiency to be extracted there, I'd pursue it. I wouldn't expect this to be a durable and long lasting strategy, however.. I agree backtesting isn't always necessary, the post you were replying to was using backtesting to answer a parent question of "why algotrade", and the ability to backtest and leverage statistics / ML is a great reason versus flying blind. But I'm not going to backtest a defi arbitrage strategy for example, there's no need - similar to your options mispricing (which is essentially an arbitrage).
>>61125675>If you are brute forcing many permutations of some indicator and then selecting the ones that best *fit* the time series you are attempting to predict, that is overfitting.What I tend to do here, fwiw, is select parameters not by best absolute returns but by finding ones within a profitable 'neighborhood', so that if tweaked slightly either direction they still yield good returns. But it's been awhile since I've been in this kind of workflow I'm sure I'm forgetting details or other quirks. I had a whole process for selecting parameters after running a massive grid search, prioritizing OOS robustness of course. The grid search then is essentially just a signal data generation phase, creating the data I need to actually make the selection - it's not the selection process itself.
>>61125324>power outageFuuuuck, if there has ever been a sign I need to automate my little pi boxes boot process it is thisAlso gotta say, I am so glad it basically runs on farts 24/7, less than a fridge by 1/10 a fridges energy consumption >>61115210Also OP, I am thinking I should just setup the community server myself (that I requested two days back) and post a link here if that's cool with you
>>61124962Could get that new Nvidia AI box (DGX spark) that costs $4000, it only runs 240w and has 120GB vram, so you can run multiple agents at once locally 24/7 at a very low power cost (although the initial investment into the box is quite high, there should be some cheaper options via Nvidia partners)
I've been reading day trading books recently. One mentioned algo trading and saying it's complete bs and you should not do it.
>>61127724Well if it's in a book it is definitely true, and you should trust his advice
