>>108598352
pretty print to stdout and ingest picrel to whatever db you use to store your logs

>>108598352
..

I have a more radical question: usecase for logging? Like sure, maybe when I'm seeing something is going wrong, I enable it for a while, but most of the time, it's a waste of resources and larping as "we are le serious about le security / servers / infrastructure".

>>108598364
Why don't you just print a message and read it on one line?

>>108599013
Because indexable fields are a pretty nice thing

>>108599056
Just grep the text with some regex

File: 1737116259027.jpg (46 KB, 720x720)

46 KB JPG

>>108598487
it's for when your software has users and they see something go wrong and they ask you about it and you go "wehhh ummm ahhh i don't knowwww i-i'll look into that ehem"

>>108598364
why would you use a database to store your logs

>>108599290
exactly this

>>108599304
it's for when you deploy your stuff to more than just one server and don't wanna ssh into every single one of them to search for the logs, so you use loki, clickhouse or something along those lines and store it in a centralized place. of course it's way too overkill for personal projects

>>108599376
why not just a fileserver though?

>>108599401
because that would be boring, also youd still have the problem that you somehow need to get the logs to the fileserver

File: file.png (142 KB, 3430x524)

142 KB PNG

>The search for a use case continues
the search for an employed /g/ poster or at least a decently knowledgeable opinion on infrastructure continues...

>>108598487
>Like sure, maybe when I'm seeing something is going wrong, I enable it for a while, but most of the time, it's a waste of resources
if you can predict the future and can accurately guess when to enable logs and where to find a rare bug, then by all means disable them
but most sysadmins are just normal humans without superpowers

>>108599401
you could do that in principle. in practice you'll spend time ssh-ing around and hating yourself.
say you look for what happened when user X had an http 503, you need in order:
1. to sift through the logs of whatever load balancer you use (probably gigabytes even on a small time range) to find out what backend handled the request,
2. then the same through the logs of the relevant backend,
3. then probably that backend was calling a database or redis or whatever the fuck, so you look for those logs now, on the right host for these,
this is already gonna be ass, but if you look for a pattern eg a percentage of users towards a percentage of servers... good luck with grepping around blindly
and to make this half-bearable, you better have really organized logs (split per host, per piece of software, per timerange, ... congrats you just rediscovered labels)
also remember to handle log rotation and retention (or discarding, more importantly), unless you got terabytes of SSDs to dedicate to it (see picrel for 24h of a fairly small infra; space is for *compressed* log files)
truth is if you designed it yourself, you'd probably arrive at least at something like Loki, which is p much distributed grep over files with a label+timerange=>file mapping in a database, then you'll realize that it's still slow (it's cheaper is why we use it) and wish you went for a DB with a bunch of indices to make semi-complex searches run in acceptable time

>>108599797
>the search for an employed /g/ poster or at least a decently knowledgeable opinion on infrastructure continues...
were my answers not good enough? :c

>>108598352
So it's logging except its a fucking json? Why must it be shoved into everything?

>>108599797
you are not very good at this

1-3 are all solved by using rsyslog to send all the logs to a logging host.

logs aren't that big when you're not inflating them with autistic json bullshit.

if you're getting *databases* involved you're just inept

all this shit is Extremely Windows Thinking and 90% of the tools you're describing are only needed because you chose poorly on the other 10%

rsyslog and grep are all you need until things get really complicated with multiple DCs and then you might also want awk

>>108599824
your answers were good, and I didn't quote you I think (?)

>>108599847
the point of json is just to be able to parse the log + metadata into DB columns without needing to do actual text processing with fragile things like regexes (you probably don't have the same metadata in every piece of software etc, so regexing your way around to extract it would require maintaining a bunch of them)

>>108599881
>1-3 are all solved by using rsyslog to send all the logs to a logging host.
rsyslog ships logs, doesn't organize the querying back
>logs aren't that big when you're not inflating them with autistic json bullshit.
we don't tho here (again, this is loki, for the poors cheaping out on storage), just got a bunch of users
also json longs aren't stored with the json syntax is the whole point of it -> so that they fit neatly as pre-made DB columns
>if you're getting *databases* involved you're just inept
I guess DBs are elitist now
>all this shit is Extremely Windows Thinking and 90% of the tools you're describing are only needed because you chose poorly on the other 10%
you addressed absolutely nothing in your stupid post except how to ship logs which is by far the easiest part (& rsyslog is a uniquely shit tool for that purpose as well; if you at least said syslog the protocol in general it'd have been defensible, but no you had to quote the shittest sink in existence lol)
>rsyslog and grep are all you need until things get really complicated with multiple DCs and then you might also want awk
DCs is less of an issue than just the amount of sources of logs
you can unclench your butt a bit, nobody said a hobbyist needs a fancy logging setup; and you're still writing a bunch of fragile regexes for various pieces of software when a jsonpath query would have solved your issue in a second; even works with awk!
however grep+awk doesn't cut it in practice, which is why nobody that actually deals with this at scale uses it nor advocates for it

>>108599881
>if you're getting *databases* involved you're just inept
the fs is also a db tho

>>108599958
this anon is probably 12, read a suckless article recently, and thinks his experience ricing his laptop means he's now a google architect
bringing "windows thinking" into it and thinking DCs is relevant here is kind of a dead giveaway he just doesn't know anything he's talking about...
alas, many such cases

>>108600013
nowadays i rather fear that it's just a llm generated response just to waste your time

>>108598352
so like windows event logging?

>>108600026
you might be right :(

>>108600030
yeah similar idea

>>108600059
>yeah similar idea
in that case the use case is easy filtering/querying/sorting of specific logs without having to parse a shitload of raw text
but you need categorized and parameterized events beforehand

>>108600088
>in that case the use case is easy filtering/querying/sorting of specific logs without having to parse a shitload of raw text
exactly
>but you need categorized and parameterized events beforehand
hence the jsonlog (or logfmt) formats: given a DB that supports dynamic columns (essentially all DBs people use for that) you can just parse the json and immediately get the schema of that log line and now you can query arbitrarily complex log lines over arbitrarily unique fields
naturally, there's processing, storage, and memory overheads, but as you rightly point out "parsing shitload of raw text" often gets way worse pretty quick