afs_webservice_load - AFS - Reference Guides

AFS Filters Description

Product
AFS
Platform
7.12
Category
Reference Guides
Language
English

The web service connector allows to crawl XML web services. The web service connector does not allow to crawl non XML web services. The web service connector does not allow to crawl JSON web services (because JSON is not XML). It uses a URL and a POST or GET parameter, and generates one layer even with multiple XPathes.

The filter is declared with the afs_webservice_load type. It is in the antidot-paf-misc package. It is a processor filter.

This filter can be instantiated only once at any given moment. It will not read the "instances" parameter in the configuration.

The Webservice Load filter specifications are described in the following table:

Parameter name

Mandatory

Type

Default

Description

input_layer

No

layer

CONTENTS

The layer containing docs from which XPaths will be extracted.

output_layer

No

layer

USER_1

The layer where to store results.

form_url

Yes

string

N/A

The form URL of the web service, as: http://localhost/search

query_path

Yes

string

N/A

Query params with var to change, as: afs:service=1&afs:query=${keywords}&afs:feed=Foo. It is necessary to encode the "query_path".

query_params

Yes

map

N/A

The XPaths to fill the "query_path". Key is the variable name in "query_path", value is the XPath where to find the variable's value.

nsmap

No

map

Empty Map

The namespace map for Xpath interpretation.

user_agent

No

string

Python 3 user_agent, see python documentation

The user agent of each query.

method

No

string

GET

The method to be used. Can be GET or POST.

delayMs

No

integer

0

The delay between 2 requests in milliseconds.

raw_answer

No

boolean

false

Raw output mode. Each raw webservice call output is stored in a layer until there's no more call to make or all layers following the target layer with the same type have been filled.

Note: This filter is intended to be used as the first filter in a pipe. Received files cannot be processed.
Attention: The web service connector allows to crawl ONLY XML web services (it is not possible to crawl JSON web services with the web service connector).