Skip to content

Decodo/SERP-Scraping-API

Folders and files

NameName
Last commit message
Last commit date

Latest commit

History

97 Commits

Repository files navigation

SERP Scraping API

List of contents

Introduction

With our SERP Scraping API, you can scrape all popular search engines. Most notably and in depth, Google, but also Baidu, Bing, and Yandex.

Authentication

Once you have an active SERP subscription, you can try sending a request right from the dashboard SERP > Authentication method tab simply by entering your username, password, and clicking on Generate. You will also see an example of curl request generated right below your entered user:pass.

Note that this is only an example with preset values to get you on the right track for forming your own request, meaning you will not be able to change the request values in the dashboard itself – that will have to be done in your code.

Google

Google can be scraped using multiple different targets, including the Google search bar, direct URL, or more specific targets like Google suggest, Google ads, etc.

Below you will find every target, whether it can be parsed to JSON, its required parameters, and ready to use code examples for cURL, Python and PHP, where you only need to change the username and password for your SERP Scraping API user for the code to work.

For available parameters, you can refer to Parameters.

Google Search

API Link: https://scraper-api.decodo.com/v2/scrape

POST /scrape

Target: google_search (parseable)

Required parameters: query (history in this example)

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Response

First 2 organic results

"organic": [{"pos": 1, "url": "https://www.worldbank.org/en/home", "desc": "With 189 member countries, the World Bank Group is a unique global partnership fighting poverty worldwide through sustainable solutions.", "title": "World Bank Group - International Development, Poverty ...", "url_shown": "https://www.worldbank.org› ...", "pos_overall": 2 },{"pos": 2, "url": "https://www.who.int/", "desc": "WHO's primary role is to direct international health within the United Nations' system and to lead partners in global health responses.", "title": "WHO | World Health Organization", "url_shown": "https://www.who.int", "pos_overall": 3 } 

Examples

Programming LanguageExample locationDownload
Pythonpython/googlesearch.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googlesearch.py > googlesearch.py
PHPphp/googlesearch.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googlesearch.php > googlesearch.php
Node.jsnodejs/googlesearch.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googlesearch.js > googlesearch.js

Target: google (parseable)

Required parameters: url

ParameterTypeDescription
urlurlGoogle Search URL
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googlesearchurl.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googlesearchurl.py > googlesearchurl.py
PHPphp/googlesearchurl.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googlesearchurl.php > googlesearchurl.php
Node.jsnodejs/googlesearchurl.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googlesearchurl.js > googlesearchurl.js

Target: google_hotels (not parseable)

Required parameters: query

Returns search results from Google hotels. This target is not parseable, meaning it will return HTML.

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googlehotels.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googlehotels.py > googlehotels.py
PHPphp/googlehotels.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googlehotels.php > googlehotels.php
Node.jsnodejs/googlehotels.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googlehotels.js > googlehotels.js

Target: google_travel_hotels (not parseable)

Required parameters: query

Returns hotel search results from Google Travel service in HTML.

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target
starsintegerStar rating, 1-5 stars
guestsintegerGuest count
date_rangestringY-m-d,Y-m-d

Examples

Programming LanguageExample locationDownload
Pythonpython/googletravel.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googletravel.py > googletravel.py
PHPphp/googletravel.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googletravel.php > googletravel.php
Node.jsnodejs/googletravel.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googletravel.js > googletravel.js

Target: google_shopping_search (parseable)

Required parameters: query

Returns results from Google Shopping search.

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googleshopping.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googleshopping.py > googleshopping.py
PHPphp/googleshopping.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googleshopping.php > googleshopping.php
Node.jsnodejs/googleshopping.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googleshopping.js > googleshopping.js

Target: google_shopping_product (parseable)

Required parameters: query (as item ID from Google Shopping)

Returns results from Google Shopping based on supplied item ID.

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googleshoppingproduct.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googleshoppingproduct.py > googleshoppingproduct.py
PHPphp/googleshoppingproduct.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googleshoppingproduct.php > googleshoppingproduct.php
Node.jsnodejs/googleshoppingproduct.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googleshoppingproduct.js > googleshoppingproduct.js

Target: google_shopping_pricing (parseable)

Required parameters: query (as item ID from Google Shopping)

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googleshoppingpricing.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googleshoppingpricing.py > googleshoppingpricing.py
PHPphp/googleshoppingpricing.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googleshoppingpricing.php > googleshoppingpricing.php
Node.jsnodejs/googleshoppingpricing.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googleshoppingpricing.js > googleshoppingpricing.js

Target: google_images (not parseable)

Required parameters: query

Returns links of images similar to the provided image link.

ParameterTypeDescription
queryurlImage URL
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googleimages.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googleimages.py > googleimages.py
PHPphp/googleimages.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googleimages.php > googleimages.php
Node.jsnodejs/googleimages.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googleimages.js > googleimages.js

Target: google_suggest (not parseable)

Required parameters: query, geo in ISO 2 format

Returns Google keyword suggestions based on the supplied query.

ParameterTypeDescription
querystringGoogle Search query
geostringGeo-targeting location
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googlesuggest.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googlesuggest.py > googlesuggest.py
PHPphp/googlesuggest.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googlesuggest.php > googlesuggest.php
Node.jsnodejs/googlesuggest.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googlesuggest.js > googlesuggest.js

Target: google_trends_explore (not parseable,already returns structured data)

Required parameters: query

Retrieve Google Trends results. This target returns structured data so there is no need to use the parse parameter.

ParameterTypeDescription
querystringGoogle Search query
geostringGeo-targeting location
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googletrendsexplore.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googletrendsexplore.py > googletrendsexplore.py
PHPphp/googletrendsexplore.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googletrendsexplore.php > googletrendsexplore.php
Node.jsnodejs/googletrendsexplore.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googletrendsexplore.js > googletrendsexplore.js

Target: google_ads (parseable)

Required parameters: query

Returns Google search results with paid ads.

ParameterTypeDescription
querystringGoogle Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/googleads.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/googleads.py > googleads.py
PHPphp/googleads.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/googleads.php > googleads.php
Node.jsnodejs/googleads.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/googleads.js > googleads.js

Baidu

Baidu can be targeted either using direct URL or by using the query parameter to the Baidu search bar.

Target: baidu (not parseable)

Required parameters: url

Target by supplying direct URL link.

ParameterTypeDescription
urlurlBaidu Search URL
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/baidusearch.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/baidusearch.py > baidusearch.py
PHPphp/baidusearch.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/baidusearch.php > baidusearch.php
Node.jsnodejs/baidusearch.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/baidusearch.js > baidusearch.js

Target: baidu_search(not parseable)

Required parameters: query

Target by supplying your query to the Baidu search bar.

ParameterTypeDescription
querystringBaidu Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/baidusearchquery.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/baidusearchquery.py > baidusearchquery.py
PHPphp/baidusearchquery.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/baidusearchquery.php > baidusearchquery.php
Node.jsnodejs/baidusearchquery.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/baidusearchquery.js > baidusearchquery.js

Bing

Bing can be targeted either using direct URL or by using the query parameter to the Bing search bar.

Target: bing (parseable)

Required parameters: url

Target by supplying direct URL link.

ParameterTypeDescription
urlurlBing Search URL
targeturlScraping target

Response

{"results": [{"content":{"url": "https://www.bing.com/search?q=history", "page": 1, "results":{"paid": [], "organic": [{"pos": 1, "url": "https://www.history.com/", "desc": "Watch full episodes of your favorite HISTORY series, and dive into thousands of historical articles and videos. To know History is to know life. By submitting your information, you agree to ...", "title": "HISTORY | Watch Full Episodes of Your Favorite Shows", "url_shown": "https://www.history.com", "pos_overall": 1 },{"pos": 2, "url": "https://ejje.weblio.jp/content/history", "desc": "history とは 意味・読み方・使い方. 発音を聞く. プレーヤー再生. ピン留め. 単語を追加. 意味・対訳. 歴史、史学、歴史書、史書、史劇、 (学問・言語などの)発達史、変遷、経歴、来歴、沿革. 音節 his・to・ry 発音記号・読み方.", "title": "英語「history」の意味・使い方・読み方 | Weblio英和 …", "url_shown": "https://ejje.weblio.jp/content/history", "pos_overall": 2 },{"pos": 3, "url": "https://www.bing.com/profile/history", "desc": "Microsoft gives you tools to manage your privacy and data. L e a r n m o r e. S i g n i n to see your search history on different browsers and computers.", "title": "Search - Search History - Bing", "url_shown": "https://www.bing.com/profile/history", "pos_overall": 3 },{"pos": 4, "url": "https://jp.history.com/", "desc": "ヒストリーチャンネルは、上質な番組を放送する世界最大で日本唯一の歴史&エンターテインメント専門チャンネルです。 長期刑の判決を受けた犯罪者たちは、刑務所に収監される前の最後の24時間をどう過ごすのか?過ちを謝罪する者、幼い子どもの親権を手放す者、崩壊した家族を修 …", "title": "日本・世界の歴史&エンタメ | THE HISTORY CHANNEL JAPAN ...", "url_shown": "https://jp.history.com", "pos_overall": 4 },{"pos": 5, "url": "https://history.google.com/", "desc": "Welcome to My Activity. Data helps make Google services more useful for you. Sign in to review and manage your activity, including things you’ve searched for, websites you’ve visited, and videos you’ve watched. Learn more.", "title": "Welcome to My Activity", "url_shown": "https://history.google.com", "pos_overall": 5 },{"pos": 6, "url": "https://www.youtube.com/HISTORYjp", "desc": "世界最大の歴史エンターテイメントブランド「HISTORY」ヒストリーチャンネル の公式YouTubeチャンネルです。人気シリーズを続々公開!最新情報は ...", "title": "HISTORY公式 - YouTube", "url_shown": "https://www.youtube.com/HISTORYjp", "pos_overall": 6 },{"pos": 7, "url": "https://eow.alc.co.jp/search?q=history", "desc": "history 【名】 歴史、歴史学 過去のこと[もの]・I'm history. : 私の時代は終わった。/私は過去の人間...【発音】hístəri【カナ】ヒストリィ【変化】《複》histories - アルクがお届けするオンライン英和・和英辞書検索サービス。", "title": "historyの意味・使い方・読み方|英辞郎 on the WEB", "url_shown": "https://eow.alc.co.jp/search?q=history", "pos_overall": 7 },{"pos": 8, "url": "https://history.gt/index.html", "desc": "2022年07月22日 『HISTORY Advanced』新発売!. 他シリーズとの違いを徹底比較!. 2022年04月26日 新商品「Performanceシリーズ」についてYouNiqueのお二人に語って頂きました!. 2022年04月01日 皆様のおかげで10万本販売達成!. 感謝キャンペーンスタート!. 2021年11 …", "title": "HISTORY ヒストリー HISTORY -島村楽器", "url_shown": "https://www.history.gt", "pos_overall": 8 },{"pos": 9, "url": "https://jp.history.com/timetable/", "desc": "ヒストリーチャンネルは、上質な番組を放送する世界最大で日本唯一の歴史&エンターテインメント専門チャンネルです。 TEL: 0570-001-607(ナビダイヤル) ヒストリーチャンネルカスタマーセンター 受付時間(10:00 – 20:00/年中無休) ※PHS・IP電話等からは、03-4333-1031", "title": "日別番組表 | ヒストリーチャンネル", "url_shown": "https://jp.history.com/timetable", "pos_overall": 9 } ] }, "parse_status_code": 12000 }, "status_code": 200, "url": "https://www.bing.com/search?q=history", "task_id": "6970716517377789953", "created_at": "2022-08-31 12:18:47", "updated_at": "2022-08-31 12:18:51" } ] } 

Examples

Programming LanguageExample locationDownload
Pythonpython/bingsearch.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/bingsearch.py > bingsearch.py
PHPphp/bingsearch.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/bingsearch.php > bingsearch.php
Node.jsnodejs/bingsearch.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/bingsearch.js > bingsearch.js

Target: bing_search(parseable)

Required parameters: query

Target by supplying your query to the Bing search bar.

ParameterTypeDescription
querystringBing Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/bingsearchquery.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/bingsearchquery.py > bingsearchquery.py
PHPphp/bingsearchquery.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/bingsearchquery.php > bingsearchquery.php
Node.jsnodejs/bingsearchquery.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/bingsearchquery.js > bingsearchquery.js

Yandex

Yandex can be targeted either using direct URL or by using the query parameter to the Yandex search bar.

Target: yandex (not parseable)

Required parameters: url

Target by supplying direct URL link.

ParameterTypeDescription
urlurlYandex Search URL
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/yandexsearch.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/yandexsearch.py > yandexsearch.py
PHPphp/yandexsearch.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/yandexsearch.php > yandexsearch.php
Node.jsnodejs/yandexsearch.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/yandexsearch.js > yandexsearch.js

Target: yandex_search (not parseable)

Required parameters: query

Target by supplying your query to the Yandex search bar.

ParameterTypeDescription
querystringYandex Search query
targeturlScraping target

Examples

Programming LanguageExample locationDownload
Pythonpython/yandexsearchquery.pycurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/python/yandexsearchquery.py > yandexsearchquery.py
PHPphp/yandexsearchquery.phpcurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/php/yandexsearchquery.php > yandexsearchquery.php
Node.jsnodejs/yandexsearchquery.jscurl https://raw.githubusercontent.com/Decodo/SERP-Scraping-API/main/nodejs/yandexsearchquery.js > yandexsearchquery.js

Parameters

ParameterTypeDescription
targetstringData source. Available targets are listed here.
urlurlDirect URL (link)
domainurlAny available Google Top-level domain.
querystring
page_fromintegerStarting page number.
num_pagesintegerStarting page number.
localestringThis will change the Google search page web interface language (not the results). Example: – en-US – en-GB
geostringThe geographical location that the result depends on. City location names, state names, country names, coordinates and radius, Google’s Canonical
device_typestringDevice type and browser. Supported: desktop, desktop_chrome, desktop_firefox, mobile, mobile_android, mobile_ios.
parseboolean'true' will return parsed output in JSON format. Leave blank for HTML – not all data sources can be parsed.
google_nfprbooleanAuto-correction.
google_results_language stringShows results in a particular language. All of the supported languages are listed here.
google_tbm stringThis parameter lets you filter Google Search results for specific types of content (news, apps, videos...).
google_tbsstringThis parameter contains parameters, like limiting/sorting results by date.
google_safe_searchstringUsed to hide explicit content from the results.
starsinteger1-5 stars, used with google_travel_hotels target
guestsintegerUsed with google_travel_hotels target
date_rangestringY-m-d,Y-m-d used with google_travel_hotels target
headlessstringEnable JavaScript rendering. Supported: html, png

Targets

List of supported targets for the 'target' parameter

google_search google_hotels google_travel_hotels google_shopping_search google_shopping_product google_shopping_pricing google google_images google_suggest google_ads google_trends_explore baidu baidu_search bing bing_search yandex yandex_search 

Languages

ISO 639-1 CodeLanguage
afAfrikaans
arArabic
hyArmenian
beBelarussian
bgBulgarian
caCatalan
zh-CNChinese - Simplified
zh-TWChinese - Traditional
hrCroatian
csCzech
daDanish
nlDutch
enEnglish
eoEsperanto
etEstonian
tlFilipino
fiFinnish
frFrench
deGerman
elGreek
ieHebrew
hiHindi
huHungarian
isIcelandic
idIndonesian
itItalian
jaJapanese
koKorean
lvLatvian
ltLithuanian
noNorwegian
faPersian
plPolish
ptPortuguese
roRomanian
ruRussian
srSerbian
skSlovak
slSlovenian
esSpanish
swSwahili
svSwedish
thThai
trTurkish
ukUkrainian
viVietnamese

Response Codes

HTTP Response Codes

ResponseDescriptionSolution
200 - SuccessServer has replied and given requested response.Celebrate!
204 - No contentJob not completed yet.Wait a few seconds before trying again.
400 - Multiple error messagesBad structure of the request.Re-check your request to make sure it is in the correct format.
401 - Invalid / not provided authorization header (client not found)Incorrect login credentials or missing authorization.Re-check your provided credentials for authorization.
403 - ForbiddenYour account does not have access to this resource.Make sure the target is supported by us
404 - Not foundYour target was not found.Re-check your targeted URL.
429 - Too many requestsExceeded rate limit for your subscription.Make sure you still have at least one request left. Wait a couple minutes and try again. If you are encountering the error often – chat with us to see if your rate limit can be increased.
500 - Internal errorService unavailable, possibly due to some issues we are encountering.Wait a couple minutes and send another request. Contact us for more information.
524 - TimeoutService unavailable, possibly due to some issues we are encountering.Wait a couple minutes and send another request. Contact us for more information.

Parsed Result Response Codes

ResponseDescription
12000 - SuccessServer has replied and given the requested response.
12002 - ErrorParsing has failed completely.
12003 - Not supportedTargeted website parsing is not supported.
12004 - Response not fullSome fields were not parsed and are missing.
12005 - Response not fully parsedSome fields might not have been parsed and are returned unparsed.
12006 - ErrorUnexpected error. Let us know the task ID and we will check what went wrong.
12007 - UnknownWe could not determine whether the data was parsed correctly.
12008 - ErrorFailed to parse all the data.
12009 - ErrorTarget not found. Make sure the parameters you passed are correct and supported.

Postman Collection

Run in Postman

Explore more guides on scraping Google data:

Google Maps scraper

Google News Scraper

License

All code is released under MIT License