Tell HN: ChatGPT is fantastic for finding and solving issues in logs

I used it to transform cryptic credit card statement items into company names, which then allowed me to query my Gmail archives for receipts and invoices from these vendors, automating a manual process of accounting backup discovery that is the bane of my very existence. I even got GPT-4 to assess whether an email likely relates to an invoice or payment so that I could limit the amount of noise extracted from my email archives.

I highly recommend considering GPT-4 every time you encounter a painful manual process. In nearly every case where I have applied GPT-4, it has been successful in one-shot or few-shot solving the problem.

I keep seeing anecdotes like this, and I wonder: How do you feel about the privacy aspect of this?

To do this, you had to feed your email into GPT-4, right?

I think there's a widely held misconception that anything you paste into GPT-4 will be used as raw training data by the model.

Some people even seem to believe that it's learning continuously, so something you paste in could show up in an answer for another user a few minutes later.

My mental model of how this works is somewhat different:

- It takes months to train a model on raw data, and OpenAI train new ones (that get released to the public) quite infrequently.

- OpenAI DO NOT WANT your private data in their training data. They put a great deal of work into stripping out PII from the training data that they do use already (this is described in their papers). They're not going to just paste in anything that anyone typed into that box.

Here's the problem though: they DO use ChatGPT interactions to "improve" their services. I don't think that means piping the data directly into training, but they clearly log everything and use those interactions as part of subsequent rounds for things like fine-tuning and RLHF.

Also they had that embarrassing bug a few weeks ago where some users could see the titles of conversations had by other users.

So it's not irrational to worry about pasting data into GPT-4 - it gets logged, and it could leak by accident.

But I'm confident that data passed to ChatGPT isn't being piped in as raw training data for subsequent versions of their live models.

(I hope I'm right about this though - I thought about blogging it, but OpenAI's transparency isn't good enough that I'd feel comfortable staking my reputation on this)

I'm absolutely horrified by people's willingness to submit private information (personal or corporate) even if it's not used for training. Data breaches happen all the time (targeted or accidental), and OpenAI is becoming a juicier target by the day.

You're right that OpenAI doesn't want the information. Consequently, OpenAI will not have security policies and processes geared for anonymization, or handling financial and health data as those are not a design goals. If I were an attacker, I'd go for the raw data rather than try to glean information off the model (in the hypothetical where user input were to be used for training)

I think it's more about being against the principle of piping potentially sensitive data to any third party.

True, OpenAI doesn't have any real motivation to randomly pluck your data and decide to do something horrible to you with it... but they could. More importantly, circumstances can and will change as time goes on. If your logs change hands as part of a buyout or cyberattack, you'll have no recourse.

> OpenAI doesn’t have any real motivation to randomly pluck your data and decide to do something horrible to you with it

They do have a motivation to use it for training, which could result in it being externally exposed to third parties, who might, OTOH, have the motivation when encountering it to do something horrible to you with it.

Could you imagine if they did?

Someone might ask it: "How do you I figure out if this person killed someone?" and it responds: "I can't be certain if they killed them but last week they asked me where they should hide the body."

But seriously, I think best argument for this is that the EU(or other euro nations) would not hesitate to go after a US company for collecting user data in violation of their data privacy laws. Even in the US, certain professionals are required to maintain confidentiality of certain records or face rather extreme penalties. OpenAI also doesn't have FAANG capital to grease Washington with yet and we know how kleptocrats love to leverage justice against newly emergent companies with valuable IP.

So if they say they don't, they had better not be or it would the likely be the end of them.

Yes re treating interactions as RLHF. Could imagine them developing a flow to automatically catalog interactions as successful and unsuccessful, then cluster those by domain + interaction flow. If someone has a successful interaction in a cluster that is normally unsuccessful, treat that as a 'wild-type' prompt engineering innovation that needs to be domesticated into the model.

I think you're right that blindly training on chats would bring back the olden days of google bombing ('santorum')

And also that any company with 'improve' in their TOS isn't committing to perfect privacy

> But I'm confident that data passed to ChatGPT isn't being piped in as raw training data for subsequent versions of their live models.

right now sure but they are almost certainly saving that data to send you targeted ads down the line. maybe not this company... maybe when they get into financial hardship and sell off to someone with dubious ethics. maybe not ads but something like that.

> OpenAI DO NOT WANT your private data in their training data

But they do want it. I can see many old chat logs.

Data is a liability. Does "clear conversations" in chat.openai.com actually remove them? Or jst mark them as "deleted", but they remain in a database. I just did a data export, then a clear conversation, then another data export. The second export was empty, which seems suspiciously fast to me

Would you be open to making this logic open source? I'd LOVE to do this as well. (I use LunchMoney.app for all spend tracking, but don't have my receipts logged)

How are you interfacing with gmail? I also wanted to clean up my inbox with some help and wasn’t sure the best touch point

That's pretty cool and interesting, I wonder how it would know credit card statement lines items or if that's in training data. Do you have an example of a cryptic item, like how is it totally unrelated to the purchase

You should turn this into a product (or open source it). I'd imagine this is a pain point for many, myself included.

Did you use the API for this, or did you copy and paste into the GUI?

Are you triggering on email received? I'd love to see how you went about this so I can solve some of my own headaches!

How does that work with logs? Logs are often... Huge? How many lines of logs can you paste? Because if I first need to narrow down the log to the problematic part, I kinda already have my problem right there no?

Or do you mean I do something like grab the lines with "error" in the log, hoping there aren't too many, then ask ChatGPT what it thinks about this:

    [ 0.135036] kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20210730/psobject-220)

That log line (with the four space at the front for HN formatting) is 40 tokens [1]. You can easily fit several hundred log lines with GPT4 8k context and with the incoming 32K context, you'll be able to fit close to a thousand log lines.

That's a lot of context, especially if you can prefilter from relevant services, nodes, etc. or provide a multi-node trace

[1] https://platform.openai.com/tokenizer

Yeah but… i can look through 100 log lines manually faster than writing a good gpt prompt. This would get useful if I can easily paste like 1M lines of logs (a few minutes of data for us), but even if that would work, it’d be prohibitively expensive I think.

In other words, I still don’t completely grok the use case that’s being shared here.

The use case here is looking through logs for software that aren't familiar, especially stuff that gets touched too infrequently to internalize like, say, driver logs on a Linux workstation.

If it's faster for you to read the logs yourself, you should continue to do that. If it's bespoke personal or commercial software, chances are GPT isn't going to be trained on its meaning anyway.

Most people aren't going to be familiar with arbitrary ACPI errors. Most people would have to Google or ask GPT to even understand the acronym.

Being able to pump a firehose of syslogs at a GPT and tell it to flag any problems would be great

200 lines @ 40 tokens per line equates to 8,000 tokens. That costs $1.60. for one query.

Thousands of log lines is actually pretty tiny though. I have some verbose logging in a testing lab and JUST network traffic from a few mobile devices can easily throw out several megs per hour in logs.

Why wouldn't you use a basic filter on logs before even reading them (or passing them elsewhere)? Maybe even get ChatGPT to write that for you if you're so inclined but it can just be a simple command as grep works fine.

After that, even if it's still a massive file, chunking it to ChatGPT should work within its limits (although I haven't personally used it for logs so I can't recommend this).

I let GPT-4 walk me through troubleshooting steps with my extremely slow read times on my SSD-backed KVM virtual machine. It told me the things it needed, I pasted relevant logs and other output, and finally I solved my issue. I was highly impressed! It parsed atop, top, and various other content, explaining exactly what everything meant.

Another benefit was that it was able to present a much more readable version of some of what I pasted. I may have to start using it for cleaning up hard-to-read output (looking at you, atop!) in the future, it really excels at that!

Also, the issue ended up being that I that I was reading from what turned out to be an NFS mount. Doh!

Used it recently to try and diagnose a weird networking issue on a server. It got me pretty far, but then the only next steps it could suggest involved installing some software that wasn't pre-installed. I don't like modifying prod servers by hand, especially because we do the whole infrastructure-as-code thing.

So I pinged a more senior engineer, they solved it. I asked what they did. They did exactly what GPT-4 had suggested for me to do. (Don't worry we ended up fixing it "properly" afterwards and re-launched the instance). I'm still not going to trust it when it tells me to do something I don't understand on a prod instance, but that was fun to see.

Even free one managed to decipher my 10 years old perl code and I was a bit surprised by it.

But in other instance I pasted same function but with parameter name changed (event -> events) and it just produced lies

Can any of the existing open source models do the same?

ChatGPT is great but I don't want all of my queries going to OpenAI.

I'd rather shell out a considerable sum to buy the equipment to run my own.

What’s wrong with “grep -i error”? That gets me to the source of the error, usually.

Yeah it's pretty awesome! I used GPT-4 last week to fix my corrupted SSD. Granted I already narrowed down the kernel logs to a few suspicious lines, but I just pasted those 10 lines in and asked for a fix. Pasting in GPT's arcane `fsck` incantations and boom -- fixed SSD. Saved me an hour or two of hassle reading man pages and stack overflow posts.

Holy crap, I've seen GPT models hallucinate api calls often enough, I would be shitting my pants when touching anything related to a drive.

This is a great feature and I did use it few times to test. However, be aware of potentially leaking your company's private or sensitive information when doing this.

https://openai.com/policies/api-data-usage-policies

OpenAI will not use data submitted by customers via our API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data.

Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law). The OpenAI API processes user prompts and completions, as well as training data submitted

In the link you provided:

> Note that this data policy does not apply to OpenAI's Non-API consumer services like ChatGPT or DALL·E Labs.

However, be aware of potentially leaking your company's private or sensitive information when doing this.

Nothing you said negates the potential for leaking your company's private or sensitive information by submitting it to a third party.

It truly seems to be the calculator for text.

It's a calculator for all human knowledge, albeit there are some rounding errors yet to be eliminated.

> there are some rounding errors yet to be eliminated

I can live with rounding errors. What it said about me was not rounding error - it was all lies that would sound correct to someone that didn't know better (which is the only reason to ask). We're screwed if ChatGPT starts getting used for background checks.

No, only knowledge that was on the internet and specifically on sites like Reddit that were crawled to train the model. This is definitely not all human knowledge!

Strong disagree. Will be writing a blog post how GPT-4 understands languages that don't even exist (at least not formally).

It's confabulating (or hallucinating) meaning. GPT has no understanding of anything. Given an input prompt (tell me what this made up language sentence means for example), it's reaching into its training set and stochastically parroting a response. There is no cognition or understanding of language.

We don’t know how many books and academic papers were in the dataset.

I can say with absolute certainty they did not include all books with all human knowledge.

For example I suspect chat GPT has zero knowledge of how to speak native American languages that have effectively died with no remaining speakers and no complete written history of their exact usage.

It also has massive gaps when it comes to problems when porting video games, concepts in game audio like procedural mixing, and even basic stuff like not generating a terribad NGINX config despite being fed the documentation. Anything niche, it's appalling. Anything you can otherwise Google? Sure.

Awesome! It is also fantastic for understanding chunks of Bash script, Perl, AWK etc where if you don't know something then it's next to impossible to search for it using a non-AI search engine.

Bonus: you can also use it to understand what the various flags in a command do.

Phind.com is so much better.

I just want to pay for the service in exchange of ensuring they won't use my data but I can't find how.

I used recently to navigate a quite complex Bash script. This is using Codex.

I'd go just below the line that I don't understand and type

    # Explain the line above in detail: << gpt explanation >>

And it'd write a very decent explanation that makes sense most of the time. Basically decrypting bash code, which I suck at.

However, there was one instance where it almost freaked me out as the output was quite human like:

    # Explain the line above: <<I don't understand it>>.

systemd is a very common log.

I'm curious whether you think this would work on logs for custom software that by necessity didn't have either its logs or writing about its logs in the training set.

Considering the fact that the RLHF for ChatGPT was done only in English but then worked just as well for every other language I would wager that specific types of logs being present in the training set is less important than it may seem.

> Considering the fact that the RLHF for ChatGPT was done only in English but then worked just as well for every other language

Does it work just as well for every other language, or does it work acceptably well for an important subset of other languages?

I speak a Slavic language spoken by ~2 million people, and I asked GPT-4 to tell me an old fable in my language. It did so fantastically, with no grammatical errors. I tried some more things -- it speaks it fluently, though admittedly not always idiomatically.

I tried to make it joke at the expense of Norwegians speaking a particular dialect, and it refused. (Language spoken by 5 million people).

When I tried to jailbreak it by prompting it to make the joke from the perspective of an esteemed actor, performing for the Prime Minister and other respected figures, it had our Prime Minister scold and demand an apology from the actor for making fun of stereotypes. The actor was contrite and toned down his humor.

It works well for a surprisingly large set. My tests involved Turkish and Azerbaijani and I'd assume it is distant enough that I was impressed.

No need to think, just.. try it and find out?

The author has chosen to 'Tell HN' so it seems reasonable to ask them for further details?

I have lots of hard-to-analyse logs but there are constraints which prevent me from sharing them with OpenAI. I am nonetheless curious about whether to do so would be worthwhile or not.

I concur ive been using it daily for all sorts of tasks, cant read Microsofts docs round it up get the important stuff i love it :)

Yesterday I dumped a broken markdown table and asked for the first "column" as CSV values. Works really well for this stuff too.

I love that it’s replacing sed, awk, and cut in the same way you’d bring artillery to a knife fight.

I've seen some really f**ed up text formatted files in the wild. In those cases I really see value in using something more "intelligent" (and declarative rather than imperative) rather than coding a non-trivial number of [python|bash] lines of code, especially when dealing with one-timers.

I think our operating systems are going to have this stuff installed one day. It might replace most uses of the CLI.

But sed, awk etc.results are predictable and reproducible.

That's not guaranteed with ChatGPT.

You could try asking it to output the sed/awx/etc. commands needed to do the desired transformation reproducibly. If it's not yet good at that it will be soon.

The question is if you can understand the solution if it's complex.

Just like the regex for valid email addresses, it may be correct but it's hard to understand it.

I see it more as a tool for doing the tedious but simple work, if it's getting complex you get a hard time checking the correctness of the result.

It is good at that and you can only paste so much data into the UI, so usually go this route for complex one liners.

Same! I had a directory of info coded horribly by some CRM into HTML DIVs (not a table) and copy/pasting it into a text file resulted in a single column of names, contact info, kids, grades, etc.

I asked ChatGPT to reformat it into a CSV, noted the useful breaks, requested some transformations and filtering, and specified a delimiter. After 5 minutes of experimentation, it worked like a charm. Absolutely amazing.

I use it for similar stuff all the time. Yesterday I had a csv full of multi-word hashtags with no spaces between the words. I needed to split them all into separate words, add spaces, and remove the #. It would have been pain to write code to reliably add spaces, especially because many of the hashtags contained non-standard words and abbreviations, but GPT-4 handled it no problem. Saved me hours.

I wish people would focus on these exceptional strengths of the model rather than blabbing about AGI or whatever.

Similarly to you, I have been able to find issues with logs, formatting, asking it quick query questions in [whatever flavor of query language XYZ service likes to use], etc.. and it's really, really good.

The alternative is to muscle through it, using a lot of energy, writing my own parser or something dumb, or to use Google - which basically isn't usable anymore!

But you have people who are like "GPT CODED MY ENTIRE WEBSITE" and "GPT TAUGHT ME QUANTUM PHYSICS" and I'm like... uh... big doubt my man...

Yes, especially given that nobody can adequately define what AGI even means.

But hey, if we did, then we couldn't have so many meandering, unproductive conversations about it on Fridman or Rogan.

Recently, I used GPT-4 to read my cousin's writings on his fictional world and generate a chart of the timeline and concepts using Mermaid syntax. I think one of the best things about LLMs at the moment is that it can convert things in an abstract way. Even if it doesn't get it totally right the first time, it can correct itself on instruction, and still saves time over coding something or downloading software.

Were you able to paste his writings all in one prompt without exhausting token space? Or did you have to do something tricky to get around that?

I think it does speed up learning a new language(programming) or library. It's not perfect but it's a good companion to a tutorial. or next step past a basic tutorial.

especially sample code. I created a script to connect to my email address and pull down my emails. connect to HN and pull done articles to put in sqlite.

really quick features, not because it's complicated but because it's tedious.

Right but you have to be really careful with that use.

Yeah, it only takes a small hallucination to delete emails instead of copy them

Yes, you should look at the code before you run it. and don't run development code on production resources like email addresses.

lol, indeed.

As long as the hallucination problem remains, I think we are going to see a significant hype bubble crash within a year or so.

Yes, it is still useful under proper guidance, but building things on full automation that are reliable doesn't seem to be something that is actually within realms of reality at present. More innovations will be required for that.

Never underestimate a hype bubble ;) The crypto one lasted for many years with far fewer use cases than LLMs, even accounting for hallucinations.

We don't even have to go to crypto. AI has had many boom/bust cycles. The term "AI Winter" dates back from the 80s!

Of course, at every cycle we get new tools. The thing is, once tools become mainstream, people stop referring to them as "AI". Take assistants (Amazon Echo, Cortana, Siri, etc). They are doing things that were active areas of "AI" research not long ago. Voice recognition and text to speech were very hard problems. Now people use them without remembering that they were once AI.

I predict that GPT will follow the same cycle. It's way too overhyped right now (because it's impressive, just like Dragon Naturally Speaking was). But people will try to everyday scenarios and – outside of niches – they will be disappointed. Cue crash as investments dry up.

Hopefully this time we won't have too many high profile casualties, like what happened with Lisp Machines.

I never upgraded to pro and I spent like $2 on credits so far this month. I could easily see them hitting a $1b/yr which to me isn't niche market or a hype bubble.

A large profitable business can still be overpriced or inflated by a bubble. Like Cisco/Intel in the 90s

My understanding is the base model is pretty good about knowing whether it knows stuff or not. it's human feedback training that causes it to lose that signal.

Base GPT-4 was highly calibrated. read open ai's technical paper.

also, this paper on gpt-4 performance of medical challenge problems confirmed the high calibration for medicine https://arxiv.org/abs/2303.13375

Like Simon said, it's a great language calculator!

Also, adapting to its quirks is important. Using the responses from both you and it to scaffold the final response is more effective than giving only one shot

That's actually the hidden secret. Millions of people are using it to solve their problems quietly - something Google and other companies whose lunch they're eating should be worried about rather than being worried about grand ideas like "AI Safety". Pretty much all of my non-tech family is using it in ways that I never expected: making meal plans, writing school essays, making assignments for students, writing emails to CRA (Canadian Tax Agency), fixing broken english and translation, etc.

I'm quite curious about some of these applications because I don't really understand what the cognitive work is that's being offloaded.

Making meal plans: what is being done here? making a list of meals to eat each day of the week? isn't this just a question of thinking what one would like to eat? why is it easier to have the meals chosen by someone else?

Writing school essays: what is the point of this? Aren't school essays only written in order to learn to write, or to learn about some other topic?

Writing emails to CRA: presumably you have to put all the pertinent information in the prompt. Can't you just copy that prompt into an email?

(The other couple do seen to make sense to me, fair enough.)

> Making meal plans: what is being done here? making a list of meals to eat each day of the week? isn't this just a question of thinking what one would like to eat? why is it easier to have the meals chosen by someone else?

Do you have children? Meal planning can be a quite tedious and frustrating task if you want to cook at home, eat healthy, eat tasty, vary the dishes and have meals that kids will accept.

I do have children. The thing is that I know what my children like to eat and don't like to eat, but ChatGPT does not.

If I had to direct, say, a human servant who is very good at cooking, but who doesn't know my kids, to plan meals for my family, I would suggest 4-6 meals that we eat frequently, 7-10 that we eat a bit less frequently, and then maybe mention a couple of things that my kids don't like. And specific dietary requirements if we had them.

I would expect that person to sort of randomly choose from the suggested meals, with the frequent ones more frequent, and then maybe try a couple of new things which don't match any of the not-likes. (And then ask us if we liked them before making them again.)

But it seems that the only hard parts are coming up with the spec to give to that person (which I do), and then varying it based on feedback (which the cook would do, but which ChatGPT doesn't do). What am I missing?

We tried it to get meal ideas and help manage a new diet with restrictions on calories/fat/protein.

Measuring each across meals and snacks each day and verifying each ingredient is time-consuming.

In addition, for recipes, it’s just a better Google. If I do “Give me a concise recipe for X” it gives me one. No fluff, no ads. Just ingredients and steps. For example. I asked for pasta carbonara, concise and then even more concise. Final result:

Quick Carbonara (4 servings)

Ingredients:

12 oz pasta

4 eggs

1 cup cheese

8 oz bacon

4 garlic

Salt, pepper

Parsley (opt)

Cook pasta, save water.

Mix eggs, cheese.

Fry bacon, garlic.

Combine, mix, season.

Serve.

Great if you’re grocery shopping and want to make sure you don’t forget anything.

Great if you want italians to get a stroke upon reading that you're putting bacon in carbonara, as well as your arteries dying at the thoughts of the sheer amount of pasta you've just made.

The regular prompt gives "pancetta, guanciale, or bacon". If I prompt for "traditional Italian" bacon is omotted and either guanciale or pancetta is suggested. If I ask for "European measurements" it suggests 350gr of pasta for 4 persons which seems reasonable.

Ahahaha as an Italian that made me laugh! Anyway I wonder if it would better to ask ChatGPT to give the recipe in Italian (so it pulls memories from the Italian blogosphere corpus) and then translate it to English (or your language of choice)

I wouldn't put it in carbonara, but Marcella Hazan's recipe[1] includes it and she's about as big of a "pasta sauce authority figure" as you're likely to find.

[1] https://www.latimes.com/recipe/marcella-hazans-spaghetti-car...

I'd pay for a service that let me take a few pictures of the contents of my fridge and pantry and then generate a list of recipes. Sounds like a good idea for a "ducktape AI" startup.

Anomaly detection is my current favorite use case for GPTs. Other problems can potentially be recast as anomaly detection that are not currently thought of as such. Regulation for instance.

There's a company that does this log analysis in real time for you called Zebrium

https://www.zebrium.com/

You better ask it to write software to analyze logs rather than analyze logs. Or your OpenAI bill will go to the roof.

also for fixing bugs in a chunk of code. at least, that has worked for me a couple of times. it can be frustrating if it's struggling, feels like you're stuck in a loop and it's hard to break out

I've been telling people the #1 thing I like about ChatGPT is that it breaks my writer's block both in prose and code.

wouldn't embeddings be a better than ChatGPT?