ChatGPT Scraper: How to Get Answers the API Leaves Out
A ChatGPT scraper pulls the web UI answer, including citation pills and shopping cards the ChatGPT API does not return. Point a real browser at chatgpt.com, keep one sticky residential session for the job, and parse the streamed reply. The API is the supported path for raw text. Use a scraper only when you need what the page shows.
Key Takeaways
- Web UI answers include citations and cards. The API does not.
- Datacenter IPs and cheap VPN exits hit Cloudflare and 403 first.
- Keep one sticky residential session for one scrape job. Rotate only between jobs. Mid-stream rotation drops the SSE reply.
- Rotate only between jobs. Mid-stream rotation drops the SSE reply.
- Official Completions is still the right tool if you only need text.
What is a ChatGPT scraper?
A ChatGPT scraper is a script or browser job that opens chatgpt.com, sends a prompt, and saves the web page reply. It is not the ChatGPT API. It is not a bot that asks ChatGPT to crawl Amazon for you. You run a real browser, wait for the streamed answer, and store the fields the page painted.
What you extract is the on-screen answer. That usually means the reply text, citation pills, shopping cards, and sometimes the search queries the page used. Those extras live on the website. Completions returns model text and does not ship the same pills.
Three jobs get mixed up under the same name. Completions or the Chat API is for raw model text through OpenAI. A ChatGPT-written crawler is a script the model drafted to hit some other site. An official Data Controls ZIP is a backup of your own export. None of those is a ChatGPT scraper.
Use this for public answers or prompts you typed. Do not steal another user’s session cookie. A logged-in thread behind someone else’s account is not a scraper job.
When you need the page, not Completions
You need the page when the question is what ChatGPT showed a user, not what the model would say through an API key. GEO teams watch citations. Brand teams watch whether the reply names a competitor. Researchers watch shopping cards and source pills that never appear in Completions.
If you only need the sentence the model would output, stop here and use Completions. A ChatGPT scraper adds Cloudflare, a live UI, and a stream that can stall. That cost is worth it only when the missing fields matter.
A web scraping service for AI training datasets is the wrong path for this job. Bulk training text belongs in Completions or in data you already have a license to use. Do not treat chatgpt.com as a corpus dump.
Do not ask ChatGPT to write a spider for Amazon and call that a ChatGPT scraper. That job uses a different keyword and a different target.
If chatgpt.com itself fails for everyone, check whether ChatGPT is down or only your session is before you debug a scraper.
Why scrapers get blocked: IP and stream
The prompt is rarely the first failure. chatgpt.com sits behind bot checks. Datacenter ranges and shared VPN exits share one ASN. Cloudflare scores that pattern fast. You see 403, a challenge page, or an empty stream.
The reply is a server-sent event stream. If the exit IP changes in the middle of that stream, the browser drops the connection. The JSON looks empty even though the prompt was fine. Keep one sticky residential session for the whole job. Rotate only after the answer is saved.
If the whole product is down, wait. Do not rotate proxies while it is down.
Route the browser through a sticky residential session so one scrape job keeps one exit. On IPOasis dynamic residential, a sticky session lasts up to 120 minutes. When that session ends, the IP can change. Start a new session for the next job.
Browser plus sticky residential proxy
How to scrape ChatGPT in practice is a real browser plus a pinned exit. chatgpt.com is a JavaScript app. Playwright, Puppeteer, or another real Chromium build can load it. A plain HTTP GET will not see the painted reply.
Use a sticky residential exit for the whole job. The IP can change when that session ends. One dedicated static ISP IP per account is the setup for signup and login errors. Do not use static ISP for this scrape job.
A free reverse proxy or a GitHub mirror forwards API traffic. It does not look like a household browser on chatgpt.com. It will not clear Cloudflare. Do not build one for this job.
- Launch a real Chromium browser. Playwright, Puppeteer, or a similar build is enough.
- Send that browser through one sticky residential proxy. Keep the same exit until the answer is saved.
- Open chatgpt.com and wait until the page paints. A challenge page means the exit is still a datacenter or VPN range. Stay on the same session and try again.
- Submit the prompt. Wait until the streamed reply stops growing. Do not open a second browser on a new IP while this stream is still running.
- Save the on-screen fields. Then start a new sticky session for the next job
Parse the streamed answer (JSON you can store)
Do not save the full HTML as your dataset. The layout will change. Store three fields you can re-read later: answer text, citations, and a timestamp. Add shopping cards only if your job needs them.
The UI will move. Treat selectors as something that will break. If a class name fails, wait for the stream to finish on the same sticky IP and adjust the parser. Do not rotate the exit to fix a selector.
Empty citations with a growing answer usually mean the SSE is still running. Wait on the same sticky session. A new IP starts a new page and you lose the half-written reply.
How to tell the job worked (and when to stop)
The job worked when one sticky session produced a record with answer text. If you need sources, the citations field is filled. The exit IP did not change while the stream was open.
Stop and use Completions when you only need model text and you do not need pills or cards. If you keep getting 403 or an empty stream, fix the exit first. Do not add more browsers on the same bad IP.
When the sticky session ends, or the IP already changed, open a new sticky session. Do not rotate in the middle of SSE. Do not stack machines to outrun a block.
FAQ
Does scraping ChatGPT violate the terms of use?
OpenAI’s supported path for text is Completions. Pulling the public website for prompts you typed, or for public answers, is a monitoring job. It can still sit outside what the terms allow. Do not steal another user’s cookie. Do not scrape behind a login wall that is not yours. If you only need text, use the API.
Can I use ChatGPT to scrape other websites with this setup?
No. That is a crawler the model wrote for a different host. Here, chatgpt scraper means chatgpt.com. Amazon, SERP pages, and shop sites need their own proxy plan and their own parser.
Why not datacenter IPs or a VPN exit?
Shared VPN and datacenter ranges hit Cloudflare first. A sticky residential exit looks like one household session for the whole job. It is an exit IP, not a speed booster. If the status page is red, wait. A new IP will not fix an outage.
Do I need to stay logged in?
Public answers and prompts you type are enough for this design. Someone else’s logged-in thread is not in scope. If signup itself is blocked with too many signups from the same IP, that is a static ISP problem, not a scraper session.
What if citations are empty but the answer text is there?
The page is still filling the stream. Wait on the same sticky IP. A rotation starts a new document and you lose both fields. Retry the parser after the reply stops growing.
Is a ChatGPT scraper legal for brand monitoring?
Watch how the public UI mentions your brand. That is the use case. Do not bypass a paywall. Do not farm accounts. If Completions already returns the sentence you need, skip the scraper.
Use a ChatGPT scraper only when you need citations and cards that the API leaves out. If you only need text, use the API. Run a real browser through a sticky residential session and keep the same IP throughout the streamed reply.
