| Command status: | Active |
| Supported by OpenApps API: | Yes |
| Supported by Internal/Reseller API: | Yes |
| Possibly queued processing: | Yes (Always) |
This command allows for large scale backlinks retrieval beyond the maximum of 50000 links supported by GetBackLinkData.
This command will generate data equivalent to the data obtained through the download interface for Advanced Reports on the Majestic site.
Due to the high volume of data to process, this command will always work asynchronously, meaning it doesn't return the data instantly but instead requests the generation of a .GZ compressed CSV file which can either be uploaded to an FTP server on completion, or downloaded from the location specified in the corresponding GetDownloadsList command.
The GetDownloadsList command can also be polled to monitor the status of the download, although if you have a internet-facing application you may find the "NotifyURL" parameter a more suitable way to be informed when your download has completed.
| Resource | Description |
|---|---|
AnalysisResUnits |
This resource will be decreased by an amount dependent on the size of the analysed item. The precise amount is detailed in the response to GetIndexItemInfo under the data column DownloadBacklinksAnalysisResUnitsCost. |
RetrievalResUnits |
This resource will be decreased by the number of rows of data returned by this command. |
| Parameter | Description |
|---|---|
cmd |
Required: must be set to: DownloadBackLinks |
datasource |
Optional - defaults to historic |
Query |
Required: index item that is being queried using same convention as for GetIndexItemInfo, examples:
|
SkipIfAnalysisCostGreaterThan |
Analysis will be automatically aborted for any index items that have a greater analysis cost than specified. This parameter prevents accidental attempts to download backlinks for very large items, such as google.com. Important: This parameter defaults to 1000000. If you intend to get backlinks for items with a higher cost than this default, you will need to provide a maximum acceptable cost as a value for SkipIfAnalysisCostGreaterThan. Default: 1000000 |
NotifyURL |
Important: notification URL must be accessible from outside your intranet - do not specify internal servers that can't be assessed from our servers, if you do then you will never get the notification you expect! Optional: if specified this URL (with HTTP/HTTPS protocols only) will be requested to notify you that the download has been fully prepared. The URL you provide can contain query string parameters to help you identify the download request that was made, we will also substitute (if they are present) the following macro variables (case sensitive): %%DOWNLOAD_FILE%% - will be changed to the download filename %%DOWNLOAD_FILE_LOCATION%% - will be changed to the download filename location using PublicDownloadLocation variable below Your notify URL should respond with single piece of data: OK (2 characters - no HTML) to indicate that you have successfully received this notification. Any other response (including server failures on your side) will be treated as an error. In the case of an error, the notification URL will be called again a number of times using exponential backoff before failing. Example of notification url: http://www.example.com/mjseonotify.php (this URL doesn't actually exist!) |
UploadToFTP |
Important: this FTP URL must be accessible from outside your intranet - do not specify internal servers that can't be assessed from our side: if you do then you will never get the upload! Optional: if not specified prepared file with backlinks will be uploaded to majestic.com, this behaviour can be overridden by specifying your own FTP server using appropriate URI format such as: ftp://username:password@yourftpserver.com/public_html/uploads/ with the relative path of the upload directory or ftp://username:password@yourftpserver.com//home/user/public_html/uploads/ with the absolute path of the upload directory. Make sure that the user specified is allowed to write files in the directory that you designated for these uploads. If a trailing filename is specified, this will be prepended to the output filename when it is upload to your server. If not specified the upload will be made to www.majestic.com from where you will be able to download the file. |
PublicDownloadLocation |
Optional: by default, the prepared backlinks file will be available from http://downloads.majestic.com (note: directory listing is denied by design there). This parameters allows you to provide an alternative FTP location's public URL. |
Filtering rules can be provided to specify which data should be analysed. For example, a particular analysis request may wish to focus on backlinks marked as "nofollow" or exclude such links from any analysis. Filtering rules can be combined, ie: only analyse backlinks found in a particular time period, excluding any links marked as "nofollow".
| Rule name | Description |
|---|---|
Target URLs filtering rules |
|
URLs |
One or more Target URLs delimited by CR LF - only meaningful for domain level analysis, this will force analysis to run only on backlinks pointing to these URLs. Exact matching of URLs will be performed. |
IncludeMatchedURLs |
Comma delimited list of strings that will be required to be matched in Target URLs in order for them to be analysed. |
ExcludeMatchedURLs |
Same as IncludeMatchedURLs but matched Target URLs will be excluded. |
Source URL filtering rules |
|
FlagIncludeDeleted (also works old flag: FlagIncludeOldCrawl)
FlagIncludeNoFollow |
Includes into analysis source URLs that had at least one of the following flags set:
See more about Source Flags: http://www.majestic.com/glossary.php#SourceFlags
These are set by setting the parameter to 1 |
FlagExcludeDeleted (also works old flag: FlagExcludeOldCrawl)
FlagExcludeNoFollow |
Same as FlagInclude* filtering rules, only setting them will result in exclusion of source URLs with specified flags set. Recommended use: exclude non-rank passing backlinks such as those marked with Deleted, NoFollow, Mention, AltText, Frame, Redirect flags.
These are set by setting the parameter to 1 |
IncludeMatchedRefDomains |
If set then only backlinks for matching referring domains will be retrieved. Referring domains can be comma delimited to provide multiple referring domains of interest. Example of valid referring domains: example.com,example.net It's possible that there will be no backlinks for specified referring domains in which case empty file (with headers) will be produced. |
ExcludeMatchedRefDomains |
If set then backlinks for matching referring domains will be excluded. Referring domains can be comma delimited to provide multiple referring domains to be excluded. Example of valid referring domains: example.com,example.net It's possible that there will be no backlinks returned when these referring domains are excluded. |
IncludeMatchedRefURLs |
If set then only backlinks containing the specified text in the source URL will be returned, multiple text can be specified in a comma separated list Example of valid text: page,folder It's possible that there will be no backlinks for specified text in which case empty file (with headers) will be produced. |
ExcludeMatchedRefURLs |
Same as IncludeMatchedRefURLs but matched URLs will be excluded from the analysis. |
IncludeExactAnchorText |
If set then only backlinks with exactly matching (lower cased) anchor text will be selected. Example: "yahoo" (without quotes) - this will only match anchor texts that are exactly "yahoo", it won't match "yahoo!" or anything different. Use | (pipe) to separate multiple anchor texts, though this is not recommended. |
ExcludeExactAnchorText |
Same as IncludeExactAnchorText but matched anchor text items will be excluded from analysis. |
IncludeContainsAnchorText |
If set then only backlinks with anchor text containing (lower cased) specified anchor texts. Example: yahoo This will match anchor texts that contain that word, ie: "yahoo!", "click here to go yahoo" etc Use | (pipe) to separate multiple anchor texts, though this is not recommended. |
ExcludeExactAnchorText |
Same as IncludeExactAnchorText but matched anchor text items will be excluded from analysis. |
IncludeContainsAnchorText |
If set then only backlinks with anchor text containing (lower cased) specified anchor texts. Example: yahoo This will match anchor texts that contain that word, ie: "yahoo!", "click here to go yahoo" etc Use | (pipe) to separate multiple anchor texts, though this is not recommended. |
ExcludeContainsAnchorText |
Same as IncludeContainsAnchorText but matched anchor text items will be excluded from analysis. |
IncludeMatchedRefTLDs |
Comma delimited list of TLDs (Top Level Domains) that will be included in analysis, ie: edu,gov - note there is no . in front of TLD. |
ExcludeMatchedRefTLDs |
Same as IncludeMatchedRefTLDs but matched referring TLDs will be excluded from analysis. |
EnableBackLinkDateRange |
If set to 1 then Last Crawled date range analysis will be enabled. |
IncludeBackLinksDateFrom |
Date in format of DD/MM/YYYY (ie: 05/02/2008 for 5 February 2008) - links from Source URLs with Last Crawled date starting at that moment will be included in analysis. Requires EnableBackLinkDateRange to be set to 1. |
IncludeBackLinksDateTo |
Same as IncludeBackLinksDateFrom only this is the cut off Last Crawled date for backlinks analysis by date. Requires EnableBackLinkDateRange to be set to 1. |
EnableBackLinkFFDateRange |
If set to 1 then First Found date range analysis will be enabled. |
IncludeBackLinksFFDateFrom |
Date in format of DD/MM/YYYY (ie: 05/02/2008 for 5 February 2008) - links from Source URLs with First Found date starting at that moment will be included in analysis. Note: Fresh Index only includes data going back 30 days and it can only see when link was first found during that date range effectively making it impossible to say if the same link wasn't found 3 years ago. It is recommended to use historical index for first found dates. Requires EnableBackLinkFFDateRange to be set to 1. |
IncludeBackLinksFFDateTo |
Same as IncludeBackLinksFFDateFrom only this is the cut off First Found date for backlinks analysis by date. Requires EnableBackLinkFFDateRange to be set to 1. |
IncludeMatchedIPs |
Includes links from Source URLs with matched IP addresses of the resolved referring domains that matched comma delimited list of IPs. Prefix matching is used, you can match subnets by leaving out the end of the IP, ie: 212.34.4. |
ExcludeMatchedIPs |
Same as IncludeMatchedIPs but matched referring TLDs will be excluded from analysis. |
IncludeMatchedGeoCountry |
Includes backlinks referring domains of which were geo-located in specified countries. Use two letter country codes or NA for not geo-located IPs or unresolved domains. List of country codes: http://www.maxmind.com/app/iso3166 |
ExcludeMatchedGeoCountry |
Same as IncludeMatchedGeoCountry but matched referring TLDs will be excluded from analysis. |
MinRefACRank |
If specified only source URLs with ACRank higher than specified will be considered. |
MinCitationFlow |
If specified, only source URLs with a citation flow greater than the value provided will be considered |
MaxCitationFlow |
If specified, only source URLs with a citation flow less than the value provided will be considered |
MinTrustFlow |
If specified, only source URLs with a trust flow greater than the value provided will be considered |
MaxTrustFlow |
If specified, only source URLs with a trust flow less than the value provided will be considered |
CitationTrustDeltaChoice |
Enable/Disable the Citation / Trust flow delta filtering |
CitationTrustDeltaValue |
Apply filtering on trust/citation flows based on relative ratios to one another. |
UsePrefixScan |
This parameter indicates that you would like links to any page under the specified prefix
eg setting Default: 0 |
CreateBackLinkCountsOnly |
Adding this parameter with any value will create a list of known URLs within the query scope, summarised by the number of backlinks to them. eg CreateBackLinkCountsOnly=1 or CreateBackLinkCountsOnly=0 Default: parameter not sent |
MinBackLinks |
Used in conjunction with CreateBackLinkCountsOnly to specify the minimum number of external backlinks a URL should have to be included in the report. Default: 0 |
UniqueBackLinksOnly |
When set to 1 it will limit number of backlinks from the same source pointing to the same target URL. When used with Default: 0 |
This is a protocol-level example query that uses a special URL that was overridden to have zero cost of analysis (you will need to use your own API_KEY to analyse other urls):
<?xml version="1.0" encoding="UTF-8"?> <Result Code="QueuedForProcessing" ErrorMessage="" FullError=""> <GlobalVars DownloadJobID="FC19FC1BE7EC6FF277078A2073FE9562" IndexBuildDate="2017-09-04 13:42:54" IndexType="0" JobID="FC19FC1BE7EC6FF277078A2073FE9562" ReportName="DownloadBackLinks" ServerBuild="2017-10-12 15:22:34" ServerName="HUMMERR" ServerVersion="1.0.6494.25877" UniqueIndexID="20170904134254-HISTORICAL" UserID="895472" /> </Result>
{
"Code": "QueuedForProcessing",
"ErrorMessage": "",
"FullError": "",
"DownloadJobID": "CB8C36874702C69A767AC086AFD01B1B",
"IndexBuildDate": "2017-09-04 13:42:54",
"IndexType": 0,
"JobID": "CB8C36874702C69A767AC086AFD01B1B",
"ReportName": "DownloadBackLinks",
"ServerBuild": "2017-10-12 15:22:34",
"ServerName": "HUMMERR",
"ServerVersion": "1.0.6494.25877",
"UniqueIndexID": "20170904134254-HISTORICAL",
"UserID": 895472
}
This response indicates that the request was queued for asynchronous processing. Note: JobID value is returned in this case. This response indicates that this request has already been requested recently and data files were prepared for it. To see details on finding these files, please see the documentation regarding GetDownloadsList. Sample check query for this particular request data files (note: DownloadJobID value will be different - take it from XML returned by previous call):
| Return value | Description | Global variables |
|---|---|
| Code | Code indicating if this command successfully completed. |
| ErrorMessage | A message explaining the error. This will be blank if the code is "OK". |
| FullError | A verbose explanation of the error. |
| DownloadJobID | Unique identifier for this download. |
| IndexBuildDate | The date the index was built. |
| IndexType | The index queried (0=Fresh, 1=Historic). |
| JobID | Unique identifier for this download. |
| ReportName | Name of the report these files are downloaded for. |
| ServerBuild | Date/time on which the server was built. |
| ServerName | Name of the server where the file was stored. |
| ServerVersion | The version of the server when the file was deleted. |
| UniqueIndexID | Unique identifier for the index storing the deleted item. |
| UserID | ID of the user making the request. |
To see details on finding the files created by running DownloadBackLinks, please see the documentation regarding GetDownloadsList.
To see details on deleting this job, please see the documentation regarding DeleteDownloads.
To see details on how to obtain the cost of analysis, please see the documentation regarding GetIndexItemInfo.
| Problem | Solution |
|---|---|
| Making requests to very large domains such as google.com uses up resources very quickly. | Use SkipIfAnalysisCostGreaterThan parameter to avoid analysing too large domains and/or call GetIndexItemInfo command to get analysis cost first. |
| Repeated calls to the same domain with slightly varied parameters can quickly use up available resources. | Call this command once, then manipulate the data on your end. |
| Calls to this command yield few to no results. | Ensure that your parameters aren't too narrow. In particular, check date range filtering: please consider that our main index is not updated every day. |