At my job they unilaterally decided that we no longer had access to our application logs in any way other than a single company wide grafana with no access control (which means anyone can see anything and seeing the stats and logs of only your stuff is a PITA).
Half the time the relevant log línes straight up don’t show up unless you use a explicit search for their content (good luck finding relevant information for an unknown error) and you’re extremely limited in how many log línes you can see at once.
Not to mention that none of our applications were designed with this platform in mind so all the logging is done in a legacy way that conforms to the idea of just grepping a log file and there’s no way the sponsors will commit to letting us spend weeks adjusting our legacy applications to actually log in a way that is useful for viewing in grafana and not a complete shitshow.
I’ve worked with a logstash/elastic/kibana stack for years before this job and I can tell you these solutions aren’t meant for seeing lines one by one or context searches (where seeing what happened right before and after matters a lot), they’re meant for aggregations and analysis.
It’s like moving all your stuff from one house to another in a tiny electric car. Sure technically it can be done but that’s not it’s purpose at all and good luck moving your fridge.
Are you sure it was set up correctly before? Kibana is the tool I’ve provisioned for dev log access for years so I don’t have to give them k8s perms. I have trained teams on debugging via Kibana and used Kibana myself for figuring out where prod errors were happening.
Your first paragraph is super shitty devX. That’s not okay. Your penultimate paragraph is really what I’m asking about.
I’m not sure if you’re serious or not.
At my job they unilaterally decided that we no longer had access to our application logs in any way other than a single company wide grafana with no access control (which means anyone can see anything and seeing the stats and logs of only your stuff is a PITA).
Half the time the relevant log línes straight up don’t show up unless you use a explicit search for their content (good luck finding relevant information for an unknown error) and you’re extremely limited in how many log línes you can see at once.
Not to mention that none of our applications were designed with this platform in mind so all the logging is done in a legacy way that conforms to the idea of just grepping a log file and there’s no way the sponsors will commit to letting us spend weeks adjusting our legacy applications to actually log in a way that is useful for viewing in grafana and not a complete shitshow.
I’ve worked with a logstash/elastic/kibana stack for years before this job and I can tell you these solutions aren’t meant for seeing lines one by one or context searches (where seeing what happened right before and after matters a lot), they’re meant for aggregations and analysis.
It’s like moving all your stuff from one house to another in a tiny electric car. Sure technically it can be done but that’s not it’s purpose at all and good luck moving your fridge.
And in the two prior posts, children, we can see the difference between trained and experienced.
Are you sure it was set up correctly before? Kibana is the tool I’ve provisioned for dev log access for years so I don’t have to give them k8s perms. I have trained teams on debugging via Kibana and used Kibana myself for figuring out where prod errors were happening.
Your first paragraph is super shitty devX. That’s not okay. Your penultimate paragraph is really what I’m asking about.
You can easily access raw live output from any source in kibana if you want to for observability
Ok…
So your point is that a bad logging implementation is bad. And I agree.
I’m not seeing how that’s extendable to implementations as a whole. You’re conflating your bad experience with "log aggregation is bad’.
Just because your company sucks at this doesn’t mean everyone else’s does.